In statistics, the Kullback–Leibler (KL) divergence is a distance metric that quantifies the difference between two probability distributions.
If we have two probability distributions, P and Q, we typically write the KL divergence using the notation KL(P || Q), which means “P’s divergence from Q.”
We calculate it using the following formula:
KL(P || Q) = ΣP(x) ln(P(x) / Q(x))
If the KL divergence between two distributions is zero, then it indicates that the distributions are identical.
The easiest way to calculate the KL divergence between two probability distributions in R is to use the KL() function from the philentropy package.
The following example shows how to use this function in practice.
Example: Calculating KL Divergence in R
Suppose we have the following two probability distributions in R:
#define two probability distributions
P
Note: It’s important that the probabilities for each distribution sum to one.
We can use the following code to calculate the KL divergence between the two distributions:
library(philentropy)
#rbind distributions into one matrix
x #calculate KL divergence
KL(x, unit='log')
Metric: 'kullback-leibler' using unit: 'log'; comparing: 2 vectors.
kullback-leibler
0.5898852
The KL divergence of distribution P from distribution Q is about 0.589.
Note that the units used in this calculation are known as nats, which is short for natural unit of information.
Thus, we would say that the KL divergence is 0.589 nats.
Also note that the KL divergence is not a symmetric metric. This means that if we calculate the KL divergence of distribution Q from distribution P, we will likely get a different value:
library(philentropy)
#rbind distributions into one matrix
x #calculate KL divergence
KL(x, unit='log')
Metric: 'kullback-leibler' using unit: 'log'; comparing: 2 vectors.
kullback-leibler
0.4975493
The KL divergence of distribution Q from distribution P is about 0.497 nats.
Also note that some formulas use log base-2 to calculate the KL divergence. In this case, we refer to the divergence in terms of bits instead of nats.
To calculate the KL divergence in terms of bits, you can instead use log2 in the unit argument:
library(philentropy)
#rbind distributions into one matrix
x #calculate KL divergence (in bits)
KL(x, unit='log2')
Metric: 'kullback-leibler' using unit: 'log2'; comparing: 2 vectors.
kullback-leibler
0.7178119
The KL divergence of distribution P from distribution Q is about 0.7178 bits.
Additional Resources
The following tutorials explain how to perform other common tasks in R:
How to Generate a Normal Distribution in R
How to Plot a Normal Distribution in R