Smarter Models Are Harder to Judge

The more intelligent AI models become, the harder it is to evaluate them. Most people can tell if a first grader's math solution is wrong. Few can do the same for a PhD-level math solution. It's easy to tell if a book summary is bad if it's gibberish, but a lot harder if the summary is coherent. To validate the quality of a summary, you might need to read the book first.

This brings us to a corollary: evaluation can be so much more time-consuming for sophisticated tasks. You can no longer evaluate a response based on how it sounds. You'll also need to fact-check, reason, and even incorporate domain expertise.

Open-Ended Outputs Break Ground Truth

The open-ended nature of foundation models undermines the traditional approach of evaluating a model against ground truths. With traditional ML, most tasks are close-ended. For example, a classification model can only output among the expected categories. To evaluate a classification model, you can evaluate its outputs against the expected outputs. If the expected output is category X but the model's output is category Y, the model is wrong.

However, for an open-ended task, for a given input, there are so many possible correct responses. It's impossible to curate a comprehensive list of correct outputs to compare against.

Models Are Treated as Black Boxes

Most foundation models are treated as black boxes, either because model providers choose not to expose models' details, or because application developers lack the expertise to understand them. Details such as the model architecture, training data, and the training process can reveal a lot about a model's strengths and weaknesses. Without those details, you can evaluate only a model by observing its outputs.

Public Benchmarks Saturate Fast

Publicly available evaluation benchmarks have proven to be inadequate for evaluating foundation models. Ideally, evaluation benchmarks should capture the full range of model capabilities. As AI progresses, benchmarks need to evolve to catch up. A benchmark becomes saturated for a model once the model achieves the perfect score. With foundation models, benchmarks are becoming saturated fast.

Evaluation's Scope Has Expanded

Last but not least, the scope of evaluation has expanded for general-purpose models. With task-specific models, evaluation involves measuring a model's performance on its trained task. However, with general-purpose models, evaluation is not only about assessing a model's performance on known tasks but also about discovering new tasks that the model can do, and these might include tasks that extend beyond human capabilities. Evaluation takes on the added responsibility of exploring the potential and limitations of AI.