A prompt change looks better on the fourteen examples a team has been watching all week. The scores rise, the awkward answers disappear, and the release feels ready. There is one problem: the team also edited the examples, changed a grader and removed two difficult cases. The new score cannot be compared with the old one.
AI evaluation set versioning gives every result a stable reference point. The useful unit is not a spreadsheet called “final.” It is a recorded bundle of test cases, expected behavior, scoring rules and run settings that can be reproduced later.
OpenAI’s evaluation guidance describes a workflow that defines an objective, collects a dataset, defines metrics, compares runs and keeps evaluating as the system changes. NIST’s AI Risk Management Framework adds a broader requirement: document the test sets, metrics and tools used for testing, evaluation, verification and validation.
Start with the decision the evaluation supports
A useful evaluation has a release question. Can the support assistant classify refund requests without inventing a policy? Can the extraction workflow capture invoice totals across the documents customers actually upload? Can a retrieval system answer from the supplied sources and refuse when the evidence is missing?
Write that decision before collecting cases. A general “quality” score mixes unrelated failures and makes a release hard to judge. A focused objective lets the team select cases, labels and graders that match a real product boundary.
The deployment setting matters too. NIST’s framework calls for performance or assurance criteria to be demonstrated under conditions similar to where the system will operate. A polished set of short English prompts says little about a product that receives scans, long conversations or several languages.
Version four parts together
Treat an evaluation release as four linked artifacts:
- Cases. Store the input, relevant context, metadata and a stable case identifier.
- References. Record the expected answer, acceptable labels, source facts or human rubric used to judge the output.
- Graders. Version deterministic checks, model-based instructions, scoring thresholds and any human review form.
- Run settings. Capture the model or system version, prompt, tools, retrieval configuration, temperature and other settings that affect the output.
A result without these references is a snapshot, not a comparison. If a grader changes from exact matching to a semantic score, keep both results and name the new evaluation version. Do not redraw the historical chart as if the measurement never changed.
The same rule applies to human labels. Correcting a wrong reference is necessary, but it changes the test. Record why it changed, who reviewed it and which runs used the old label.
Keep a stable regression set and a growing discovery set
One dataset usually has two jobs that pull in different directions.
The regression set should remain stable enough to reveal whether a system change moved known behavior. The discovery set should grow as production reveals new phrasing, edge cases and harms. OpenAI’s guidance recommends continuous evaluation and adding cases as new nondeterministic behavior appears.
Separate the two.
Run the stable set on every meaningful change. Add newly discovered cases to a candidate pool, review them, then promote a dated batch into the next regression version. Report the old and new versions side by side during the transition. That makes improvement on existing cases visible without hiding the harder cases that just joined the suite.
Our guide to document extraction confidence uses a similar boundary: labeled tests establish where automation is reliable, while uncertain fields are routed for review.
Protect the test from tuning leakage
When developers repeatedly read every expected answer while editing a prompt, the evaluation set becomes part of the development material. Performance can rise because the prompt learned the examples rather than the underlying task.
Keep a development set for fast iteration and a held-out set for release decisions. Limit access to the held-out references where the product risk justifies it. For higher-stakes work, a separate reviewer can run the final evaluation and inspect failures without sharing every case in advance.
Production examples need another boundary. Remove or protect personal and confidential data, keep collection purpose and retention rules clear, and avoid copying sensitive conversations into an evaluation store simply because they contain useful failures.
Synthetic cases can widen coverage, but they should not be mistaken for production evidence. Label their origin and check whether they reproduce the language, constraints and failure modes the product actually sees.
Compare against a named baseline
A release result should answer “better than what?”
Name the baseline system, evaluation version and measurement window. Compare individual failure categories as well as the aggregate score. A two-point average gain can conceal a sharp drop in refusal accuracy or a new failure on long inputs.
Model changes deserve the same discipline. Our article on embedding model migration keeps old and new vector spaces separate through cutover. Evaluation comparisons should likewise keep the old system available long enough to run the same cases against both configurations.
Where outputs vary between runs, use repeated samples when the cost and risk warrant it. Report the sampling method and uncertainty instead of presenting one lucky run as a stable property.
Calibrate graders against people
Automated graders make frequent testing possible, but they are measurements with their own failure modes. Exact checks can reject valid paraphrases. Similarity scores can reward an answer that sounds close while getting a fact wrong. A model grader can inherit ambiguous rubric language.
Review a sample with people who understand the task. Measure agreement, inspect disagreements and revise the rubric before raising a score threshold. OpenAI’s guidance specifically recommends calibrating automated metrics with human feedback, while NIST calls for documented measurement methods and relevant domain input.
Keep the grader’s prompt and model version with the run. If the grader changes, rerun the baseline or start a new series. Otherwise the chart combines different instruments.
Run evaluations at the changes that matter
A practical release gate runs when the team changes:
- the system or developer prompt;
- the model or model snapshot;
- retrieval sources, chunking or ranking;
- tools and permissions;
- output schemas or post-processing;
- safety policy or refusal behavior;
- a grader, threshold or reference answer.
Not every change needs the full suite. A small fast set can catch obvious regressions during development, followed by the release set before deployment and scheduled production checks afterward.
The gate should name the action tied to a failure. Some misses block release. Some require a human review step. Others consume an accepted error budget and create a follow-up. A score with no decision rule becomes a dashboard decoration.
Keep a release record
For each approved release, retain:
- the system version or commit;
- the evaluation version;
- the exact run settings;
- pass and failure counts by category;
- reviewed disagreements;
- accepted exceptions and their owners;
- the final release decision.
This record lets a team answer a question months later: did the system change, did the test change, or did both change at once?
Sources and version note
Checked 28 October 2026 against OpenAI’s evaluation best practices, OpenAI’s dataset guide, and the NIST AI Risk Management Framework Core. Product interfaces and evaluation tools can change; the versioning and documentation controls described here are system-design practices.
