Member-only story
A Guide to Effective Prompt Evaluations for Large Language Models
In the fast-changing world of artificial intelligence, large language models are now valuable tools for many businesses. But to get the most out of them, companies need more than just basic setup — they need to carefully test and improve how they use them. This is where prompt evaluations, or “evals,” are essential. This post aims to explain what evaluations are and how to write them.
Benchmarks
Before we jump into customer evaluations, let’s start with a type of evaluation that most people know: model benchmarks.
Model benchmarks are like the standardized tests of AI. Just as SAT scores give colleges a rough idea of a student’s academic skills, model benchmarks give us an overall sense of how well an AI model performs across different tasks.
Companies that build large language models use these benchmarks to showcase their models’ strengths. You might see high scores on tests with names like ARC, MMLU, or TruthfulQA. These benchmarks cover a range of skills — from basic reading to advanced reasoning and knowledge across various fields. They’re helpful for comparing models and tracking AI progress over time. Often, benchmark scores appear on model cards, like this:
