Engineering workflow showing an evaluation set feeding a CI regression gate and production monitoring for an LLM application

LLM Evaluation: Write the Eval Set Before the Prompt, Then Gate, Grade and Monitor

An evaluation-driven build loop for LLM applications: the eval set first, graders you can trust, regression gates in CI, graded human samples and production drift monitoring.

By Shayan Salehi 8 min read

Image: ColdAI

LLM evaluation works best when it comes first. Write the evaluation set before the prompt, so the prompt is built against a target. Run it as a regression gate in continuous integration on every change. Have people grade a sample of outputs against a written rubric. Then keep scoring a sample of production traffic, so drift shows up as a number and not as a complaint.

Why the order matters

A prompt written first and tested afterward gets tested on whatever the author thought of while writing it. That is the "vibe-based" evaluation OpenAI's evaluation best practices guide lists as an anti-pattern: using "it seems like it's working" as a strategy, or waiting until you ship before building any evals. The guide recommends the opposite habit, which it calls eval-driven development: evaluate early and often, and write scoped tests at every stage.

Evaluation-driven development is the same idea as test-driven development, adapted to a component whose output varies. Traditional tests assert one exact result. An LLM evaluation measures how often the system meets a standard across many cases, and whether that rate moves when you change something.

This article covers the build loop for one LLM application: the evaluation set, the graders, the CI gate, the human review sample and the production monitor. Release gates for autonomous agents involve more, such as tool permissions and red-team evidence, and are a separate topic.

Write the evaluation set before the prompt

Start with success criteria, because the set has to measure something. Anthropic's documentation on defining success criteria says good criteria are specific, measurable, achievable and relevant. Its example turns "the model should classify sentiments well" into an F1 score of at least 0.85 on a held-out test set of 10,000 diverse posts, with further targets on toxicity, error severity and response time.

Copy the structure and not the numbers. Your criteria should state the metric, the threshold, the data it is measured on and the slices that matter. "At least 95 percent of refund decisions match the policy on the held-out set, and no wrong approvals on the high-value slice" is a criterion. "Good answers" is not.

Then build the cases. OpenAI's guide lists the sources to draw from: production data, historical data, domain-expert-written examples, human-curated data and synthetic data. It advises that test data include typical, edge and adversarial cases, and that you use human expert labelers. It also says to log everything as you develop, so you can mine the logs for good eval cases.

Anthropic's documentation lists edge cases worth covering: irrelevant or missing input, overly long input, poor or harmful user input, and ambiguous cases where even humans would struggle to agree. Add the cases where the correct behavior is to decline or to hand off to a person.

Keep a held-out portion that nobody tunes against. Once a prompt author has seen every case, the score measures memory of the set, not quality.

Choose graders you can trust

Each case needs a grader. Use the cheapest one that is reliable for the task.

The Inspect framework, an open-source project from the UK AI Security Institute and Meridian Labs, shows the range. Its built-in scorers include substring and position matching, regular-expression extraction, exact match, F1 over token overlap, and model-graded scorers for open-ended answers. Use code-based graders wherever the answer can be checked by rule: valid JSON, a correct order number, a refusal phrase, a required citation.

For open-ended output, a model can grade against a rubric. Anthropic's tips are practical: write detailed, clear rubrics, ask for an empirical output such as correct or incorrect or a score from 1 to 5, and let the grader reason before it scores. OpenAI's guide adds that LLMs are better at discriminating between options, so evaluations should favor pairwise comparison, classification, or scoring against specific criteria over judging open-ended generation.

A model grader is a measuring instrument, and instruments need calibrating. OpenAI's guide says to maintain agreement, using human feedback to calibrate automated scoring. Have people grade the same 50 to 100 outputs, compare their labels with the model grader, and read every disagreement. Rewrite the rubric until the two agree on the cases that matter. Repeat whenever you change the grader model.

Measure more than accuracy

A system can pass on accuracy and still be unusable because it is slow, expensive or inconsistent. Anthropic's documentation says most use cases need multidimensional evaluation and lists task fidelity, consistency, relevance, tone, privacy, context use, latency and price among the dimensions.

Stanford CRFM's HELM project makes the same argument at research scale. Its paper measures seven metrics, accuracy, calibration, robustness, fairness, bias, toxicity and efficiency, for each of 16 core scenarios whenever possible, which it reports as 87.5 percent of the time. The authors say this multi-metric approach keeps metrics beyond accuracy from falling by the wayside and exposes trade-offs. Your application needs a smaller set, but the habit carries over: report several metrics side by side, and decide in advance which ones are gates and which are watch items.

Make the evaluation a gate in CI

Run the evaluation on every change that could alter behavior: a prompt edit, a model or version swap, a change to retrieval or tool definitions, a new guardrail. OpenAI's guide calls this continuous evaluation, running evals on every change and growing the set over time.

Design the gate so that it is trusted. Three rules help.

  • Compare against a stored baseline and fail on a regression beyond a stated tolerance. An absolute threshold alone hides a slow decline.
  • Gate on slices, not only the average. A 2-point gain overall can hide a loss on the slice with the most risk.
  • Account for noise. Outputs vary between runs, so a single run can swing across a threshold.

Inspect gives a model for the last point. Its command line has an epochs option that repeats each sample, and reducers such as mean, median, pass at k and at least n to combine the repeats. Its scorers report standard error alongside the mean.

The arithmetic is simple. With 100 cases and a true pass rate near 90 percent, the standard error is about 3 percentage points, so a drop from 91 to 88 is within noise. With 400 cases it falls to about 1.5. Size the set for the differences you need to detect, and run the cheaper subset on each pull request and the full set nightly or before release.

Finally, make the result visible to the people who approve releases. NIST's Generative AI Profile, AI 600-1, suggests sharing the results of pre-deployment testing with relevant actors, such as those with release approval authority. A gate that only engineers read is not a control.

Add graded human review

Automated graders scale, but they share blind spots. A sample of outputs graded by people against the written rubric gives you ground truth and finds the failures nobody thought to test.

Sample on purpose. Include random outputs, outputs near the pass threshold, outputs the model grader scored low, and outputs where users overrode or corrected the system. Have two reviewers grade a subset, and measure how often they agree. If humans disagree with each other, the rubric needs work before it can judge the model.

NIST AI 600-1 suggests organizations avoid extrapolating system performance from narrow, non-systematic and anecdotal assessments. A scheduled, rubric-based sample is the opposite of anecdote. It also notes that benchmark test sets can contain label errors, which is a reminder to audit your own labels and not assume they are right.

Feed what reviewers find back into the set. Every confirmed failure becomes a new case, tagged with its slice and the date it was found.

Monitor production and watch for drift

Behavior shifts after launch even when you change nothing. Users ask different questions, source documents change, and a provider updates or retires a model. Your own edits to prompts, retrieval indexes or tools shift results too.

OpenAI's guide recommends monitoring your application to identify new cases of nondeterminism and growing the eval set over time. AI 600-1 suggests evaluating performance in real-world scenarios to reveal issues that do not show in controlled testing, and monitoring and documenting instances where human operators override the system's decisions. It also notes that gaps between benchmarks and real use are likely to be wider because of prompt sensitivity and the variety of contexts of use.

In practice, score a random sample of live traffic every day with the same graders you use in CI, and chart the pass rate by slice. Track override and escalation rates, refusal rates, cost per task and latency next to it. Alert on a sustained move, for example a rolling pass rate below baseline minus your tolerance for several days, not on a single bad hour.

When the alert fires, run the full CI set against the current configuration. If CI is green and production is red, your set no longer matches real traffic, and the fix is new cases. If both are red, something you control or your provider changed.

Keep the evaluation portable

Store the dataset, rubrics and grader prompts in your own repository in a plain format such as JSON lines, and run them with a tool you can replace. OpenAI's deprecations page lists November 30, 2026 as the shutdown date for its Evals dashboard, API and graders for eval workflows, and recommends migrating to Promptfoo. Teams that kept their data and graders in their own repositories can move to another framework. Teams that kept them only in a hosted dashboard now have an export job.

Where to start

Pick one LLM feature that is live or close to launch and do these in order.

  1. Write the success criteria as metrics, thresholds, data and slices, and get the business owner to sign them.
  2. Build 100 cases from real logs and expert examples, tag each with its slice, and hold out a third.
  3. Choose a grader for each criterion. Use code where you can, and calibrate any model grader against 50 human-graded outputs.
  4. Run the set in CI with a stored baseline, slice thresholds and repeated runs, and block merges on regression.
  5. Schedule a weekly human review of 30 sampled outputs, with two reviewers on a subset, and add every confirmed failure to the set.
  6. Score a daily sample of production traffic with the same graders, and alert on sustained movement.

Build the set first for every LLM feature, because it forces the team to say what good means before anyone argues about wording.

Tags:llm-evaluationevaluation-driven-developmentci-regression-testingllm-as-judgehuman-reviewmodel-driftllm-monitoring

Your starting point

What would you like to move forward?

Start with the outcome that feels closest. We'll help give the next step a useful shape.

Let's find your next move

One conversation. A clearer direction.

Tell us what you want to improve. A few sentences are enough to start.

shayan@coldai.org