In statistics, the Kullback–Leibler (KL) divergence is a distance metric that quantifies the difference between two probability distributions.

If we have two probability distributions, P and Q, we typically write the KL divergence using the notation KL(P || Q), which means “P’s divergence from Q.”

We calculate it using the following formula:

KL(P || Q) = ΣP(x) ln(P(x) / Q(x))

If the KL divergence between two distributions is zero, then it indicates that the distributions are identical.

We can use the function to calculate the KL divergence between two probability distributions in Python.

The following example shows how to use this function in practice.

Example: Calculating KL Divergence in Python

Suppose we have the following two probability distributions in Python:

Note: It’s important that the probabilities for each distribution sum to one.

#define two probability distributions
P = [.05, .1, .2, .05, .15, .25, .08, .12]
Q = [.3, .1, .2, .1, .1, .02, .08, .1]

We can use the following code to calculate the KL divergence between the two distributions:

from scipy.specialimport rel_entr

#calculate (P || Q)
sum(rel_entr(P, Q))

0.589885181619163

The KL divergence of distribution P from distribution Q is about 0.589.

Note that the units used in this calculation are known as , which is short for natural unit of information.

Thus, we would say that the KL divergence is 0.589 nats.

Also note that the KL divergence is not a symmetric metric. This means that if we calculate the KL divergence of distribution Q from distribution P, we will likely get a different value:

from scipy.specialimport rel_entr

#calculate (Q || P)
sum(rel_entr(Q, P))

0.497549319448034

The KL divergence of distribution Q from distribution P is about 0.497 nats.

Note: Some formulas use log base-2 to calculate the KL divergence. In this case, we refer to the divergence in terms of instead of nats.

Additional Resources

The following tutorials explain how to perform other common operations in Python: