Kullback–Leibler Divergence - KL Divergence For Normal Distributions

KL Divergence For Normal Distributions

The Kullback–Leibler divergence between two multivariate normal distributions of the dimension with the means and their corresponding nonsingular covariance matrices is:

D_\text{KL}(\mathcal{N}_0 \| \mathcal{N}_1) = { 1 \over 2 } \left( \mathrm{tr} \left( \Sigma_1^{-1} \Sigma_0 \right) + \left( \mu_1 - \mu_0\right)^\top \Sigma_1^{-1} ( \mu_1 - \mu_0 ) -\ln \left( { \det \Sigma_0 \over \det \Sigma_1 } \right) - k \right).

The logarithm must be taken to base e since the two terms following the logarithm are themselves base-e logarithms of expressions that are either factors of the density function or otherwise arise naturally. The equation therefore gives a result measured in nats. Dividing the entire expression above by loge 2 yields the divergence in bits.

Read more about this topic:  Kullback–Leibler Divergence

Famous quotes containing the word normal:

    “Everyone in the full enjoyment of all the blessings of his life, in his normal condition, feels some individual responsibility for the poverty of others. When the sympathies are not blunted by any false philosophy, one feels reproached by one’s own abundance.”
    —Elizabeth Cady Stanton (1815–1902)