Evaluation Methodology

How to evaluate open-ended foundation models: language-modeling metrics, exact and subjective methods, AI as a judge, and their limitations.

Evaluation Methodology

The more AI is used, the more opportunity there is for catastrophic failure. We've already seen many failures in the short time that foundation models have been around.

Failures We Have Already Seen

Chatbot Encouraged Suicide

A man committed suicide after being encouraged by a chatbot.

Hallucinated Court Evidence

Lawyers submitted false evidence hallucinated by AI.

Airline Chatbot Misled a Passenger

Air Canada was ordered to pay damages when its AI chatbot gave a passenger false information.

Without a way to quality control AI outputs, the risk of AI might outweigh its benefits for many applications.

As teams rush to adopt AI, many quickly realize that the biggest hurdle to bringing AI applications to reality is evaluation. For some applications, figuring out evaluation can take up the majority of the development effort.

In December 2023, Greg Brockman, an OpenAI cofounder, tweeted that "evals are surprisingly often all you need."

Two Chapters on Evaluation

Due to the importance and complexity of evaluation, this book has two chapters on it.

This Chapter — Methods

Different evaluation methods used to evaluate open-ended models, how these methods work, and their limitations.

Next Chapter — Application

How to use these methods to select models for your application and build an evaluation pipeline to evaluate your application.

Evaluation in the Context of a Whole System

While I discuss evaluation in its own chapters, evaluation has to be considered in the context of a whole system, not in isolation.

Evaluation aims to mitigate risks and uncover opportunities.

Mitigate Risks

To mitigate risks, you first need to identify the places where your system is likely to fail and design your evaluation around them.

Uncover Opportunities

Evaluation also aims to uncover opportunities.

Often, this may require redesigning your system to enhance visibility into its failures. Without a clear understanding of where your system fails, no amount of evaluation metrics or tools can make the system robust.

Why People Skip Systematic Evaluation

Before diving into evaluation methods, it's important to acknowledge the challenges of evaluating foundation models. Because evaluation is difficult, many people settle for word of mouth (e.g., someone says that the model X is good) or eyeballing the results.

Word of Mouth

Someone says that model X is good. A 2023 study by a16z showed that 6 out of 70 decision makers evaluated models by word of mouth.

Eyeballing the Results

Also known as a vibe check.

This creates even more risk and slows application iteration.

Instead, we need to invest in systematic evaluation to make the results more reliable.

Language Modeling Metrics

Since many foundation models have a language model component, this chapter will provide a quick overview of the metrics used to evaluate language models, including cross entropy and perplexity.

Cross Entropy

Essential for guiding the training and finetuning of language models, and frequently used in many evaluation methods.

Perplexity

Essential for guiding the training and finetuning of language models, and frequently used in many evaluation methods.

Open-Ended Models Need Different Practices

Evaluating foundation models is especially challenging because they are open-ended, and I'll cover best practices for how to tackle these.

Human Evaluators

Using human evaluators remains a necessary option for many applications. Given how slow and expensive human annotations can be, the goal is to automate the process.

Automatic Evaluation

This book focuses on automatic evaluation, which includes both exact and subjective evaluation.

AI as a Judge

The rising star of subjective evaluation is AI as a judge — the approach of using AI to evaluate AI responses.

It's subjective because the score depends on what model and prompt the AI judge uses.

Rapid Traction

This approach is gaining rapid traction in the industry.

Intense Opposition

It also invites intense opposition from those who believe that AI isn't trustworthy enough for this important task.

I'm especially excited to go deeper into this discussion, and I hope you will be, too.

What This Chapter Covers

Challenges of Evaluating Foundation Models

Why evaluating foundation models is hard, including the limits of word of mouth and vibe checks, and best practices for open-ended models.

Understanding Language Modeling Metrics

A quick overview of cross entropy and perplexity — metrics essential for training, finetuning, and many evaluation methods.

Exact Evaluation

One half of automatic evaluation: exact methods, alongside subjective evaluation.

AI as a Judge

The rising star of subjective evaluation — using AI to evaluate AI responses, including why it is taking off and why it is contested.

Ranking Models with Comparative Evaluation

Ranking models with comparative evaluation.

Edit this pageorReport an issue

Summary

A recap of how training data, modeling choices, post-training, and sampling shape foundation model behavior.

Challenges of Evaluating Foundation Models

Why evaluating foundation models is harder than traditional ML — intelligence, open-ended outputs, black boxes, saturating benchmarks, and expanding scope.