Why Language Modeling Metrics Matter

As discussed in Chapter 1, language modeling has been around for decades, popularized by Claude Shannon in his 1951 paper "Prediction and Entropy of Printed English". The metrics used to guide the development of language models haven't changed much since then. Most autoregressive language models are trained using cross entropy or its relative, perplexity. When reading papers and model reports, you might also come across bits-per-character (BPC) and bits-per-byte (BPB); both are variations of cross entropy.

All four metrics — cross entropy, perplexity, BPC, and BPB — are closely related. If you know the value of one, you can compute the other three, given the necessary information.

Recall that a language model encodes statistical information (how likely a token is to appear in a given context) about languages. Statistically, given the context "I like drinking __", the next word is more likely to be "tea" than "charcoal". The more statistical information that a model can capture, the better it is at predicting the next token.

In ML lingo, a language model learns the distribution of its training data. The better this model learns, the better it is at predicting what comes next in the training data, and the lower its training cross entropy. As with any ML model, you care about its performance not just on the training data but also on your production data. In general, the closer your data is to a model's training data, the better the model can perform on your data.

Entropy

Entropy measures how much information, on average, a token carries. The higher the entropy, the more information each token carries, and the more bits are needed to represent a token.

Let's use a simple example to illustrate this. Imagine you want to create a language to describe positions within a square, as shown in Figure 3-4.

Figure 3-4. Two languages describe positions within a square. Compared to the language on the left (a), the tokens on the right (b) carry more information, but they need more bits to represent them.

Intuitively, entropy measures how difficult it is to predict what comes next in a language. The lower a language's entropy (the less information a token of a language carries), the more predictable that language. In our previous example, the language with only two tokens is easier to predict than the language with four (you have to predict among only two possible tokens compared to four). This is similar to how, if you can perfectly predict what I will say next, what I say carries no new information.

Cross Entropy

When you train a language model on a dataset, your goal is to get the model to learn the distribution of this training data. In other words, your goal is to get the model to predict what comes next in the training data. A language model's cross entropy on a dataset measures how difficult it is for the language model to predict what comes next in this dataset.

A model's cross entropy on the training data depends on two qualities:

Entropy and cross entropy share the same mathematical notation, $H$. Let $P$ be the true distribution of the training data, and $Q$ be the distribution learned by the language model. Accordingly, the following is true:

• The training data's entropy is, therefore, $H(P)$.
• The divergence of $Q$ with respect to $P$ can be measured using the Kullback–Leibler (KL) divergence, which is mathematically represented as $D_{\text{KL}}(P \parallel Q)$.
• The model's cross entropy with respect to the training data is therefore:

$H(P, Q) = H(P) + D_{\text{KL}}(P \parallel Q)$

Cross entropy isn't symmetric. The cross entropy of $Q$ with respect to $P$ — $H(P, Q)$ — is different from the cross entropy of $P$ with respect to $Q$ — $H(Q, P)$.

A language model is trained to minimize its cross entropy with respect to the training data. If the language model learns perfectly from its training data, the model's cross entropy will be exactly the same as the entropy of the training data. The KL divergence of $Q$ with respect to $P$ will then be $0$. You can think of a model's cross entropy as its approximation of the entropy of its training data.

Bits-per-Character and Bits-per-Byte

One unit of entropy and cross entropy is bits. If the cross entropy of a language model is $6$ bits, this language model needs $6$ bits to represent each token.

Since different models have different tokenization methods — for example, one model uses words as tokens and another uses characters as tokens — the number of bits per token isn't comparable across models. Some use the number of bits-per-character (BPC) instead. If the number of bits per token is $6$ and on average, each token consists of $2$ characters, the BPC is $6/2 = 3$.

One complication with BPC arises from different character encoding schemes. For example, with ASCII, each character is encoded using $7$ bits, but with UTF-8, a character can be encoded using anywhere between $8$ and $32$ bits. A more standardized metric would be bits-per-byte (BPB), the number of bits a language model needs to represent one byte of the original training data. If the BPC is $3$ and each character is $7$ bits, or $7/8$ of a byte, then the BPB is $3 / (7/8) = 3.43$.

Cross entropy tells us how efficient a language model will be at compressing text. If the BPB of a language model is $3.43$, meaning it can represent each original byte ($8$ bits) using $3.43$ bits, this language model can compress the original training text to less than half the text's original size.

Perplexity

Perplexity is the exponential of entropy and cross entropy. Perplexity is often shortened to PPL. Given a dataset with the true distribution $P$, its perplexity is defined as:

$\text{PPL}(P) = 2^{H(P)}$

The perplexity of a language model (with the learned distribution Q) on this dataset is defined as:

$\text{PPL}(P, Q) = 2^{H(P, Q)}$

If cross entropy measures how difficult it is for a model to predict the next token, perplexity measures the amount of uncertainty it has when predicting the next token. Higher uncertainty means there are more possible options for the next token.

Consider a language model trained to encode the 4 position tokens, as in Figure 3-4 (b), perfectly. The cross entropy of this language model is 2 bits. If this language model tries to predict a position in the square, it has to choose among $2^2 = 4$ possible options. Thus, this language model has a perplexity of 4.

So far, I've been using bit as the unit for entropy and cross entropy. Each bit can represent 2 unique values, hence the base of 2 in the preceding perplexity equation.

Popular ML frameworks, including TensorFlow and PyTorch, use nat (natural log) as the unit for entropy and cross entropy. Nat uses the base of $e$, the base of natural logarithm.

If you use nat as the unit, perplexity is the exponential of $e$:

$\text{PPL}(P, Q) = e^{H(P, Q)}$

Perplexity Interpretation and Use Cases

As discussed, cross entropy, perplexity, BPC, and BPB are variations of language models' predictive accuracy measurements. The more accurately a model can predict a text, the lower these metrics are. In this book, I'll use perplexity as the default language modeling metric. Remember that the more uncertainty the model has in predicting what comes next in a given dataset, the higher the perplexity.

What's considered a good value for perplexity depends on the data itself and how exactly perplexity is computed, such as how many previous tokens a model has access to. Here are some general rules:

Other than guiding the training of language models, perplexity is useful in many parts of an AI engineering workflow. First, perplexity is a good proxy for a model's capabilities. If a model's bad at predicting the next token, its performance on downstream tasks will also likely be bad. OpenAI's GPT-2 report shows that larger models, which are also more powerful models, consistently give lower perplexity on a range of datasets, as shown in Table 3-1. Sadly, following the trend of companies being increasingly more secretive about their models, many have stopped reporting their models' perplexity.

Table 3-1. Larger GPT-2 models consistently give lower perplexity on different datasets. Source: OpenAI, 2018.

Recall that the perplexity of a model with respect to a text measures how difficult it is for this model to predict this text. For a given model, perplexity is the lowest for texts that the model has seen and memorized during training. Therefore, perplexity can be used to detect whether a text was in a model's training data.

Perplexity and its related metrics help us understand the performance of the underlying language model, which is a proxy for understanding the model's performance on downstream tasks. The rest of the chapter discusses how to measure a model's performance on downstream tasks directly.

How to Use a Language Model to Compute a Text's Perplexity

A model's perplexity with respect to a text measures how difficult it is for the model to predict that text. Given a language model $X$, and a sequence of tokens $[x_1, x_2, \dots, x_n]$, $X$'s perplexity for this sequence is:

$P(x_1, x_2, \dots, x_n)^{-\frac{1}{n}} = \left( \frac{1}{P(x_1, x_2, \dots, x_n)} \right)^{\frac{1}{n}} = \left( \prod_{i=1}^n \frac{1}{P(x_i \mid x_1, \dots, x_{i-1})} \right)^{\frac{1}{n}}$

where $P(x_i \mid x_1, \dots, x_{i-1})$ denotes the probability that $X$ assigns to the token $x_i$ given the previous tokens $x_1, \dots, x_{i-1}$.