The old test is passing a new question

The answer changed because the job changed

The assistant still says seven days, and the old test still passes. The business has meanwhile changed the applicable reminder to three. Nothing needs to have changed in the model for the answer to become wrong: the job changed around it. Before interpreting a score, the evaluation needs to identify the rule it is judging, including the earlier cases that still belong to the old policy.

The worked example uses invented administrative rules for an onboarding information packet, not employment eligibility or an actual organisation's policy. Under task version one, onsite starts receive a reminder after three calendar days and remote starts after seven. Version two changes remote starts on or after 1 March 2025 to three days, while preserving the older rule for earlier starts. Unknown working mode requires clarification. The governing event is the onboarding start date, not the date on which somebody happens to ask the assistant.

A policy change is rarely a reason to replace every old expected answer. Some earlier records still need the old treatment, one existing case may legitimately change and an effective-date boundary may expose missing coverage. Move the specification, cases and expectations together so each difference has a reason. Renaming the set latest would save a few characters while hiding the very thing the next result needs to explain: what changed about being correct.

Read the answer-key diff before reading the score

The example's four existing cases have stable identifiers. C1 is onsite before the change and remains three days. C2 is remote before the boundary and remains seven. C3 is remote after the boundary and changes from seven to three. C4 lacks mode and still requires clarification. The new set adds C5, a hybrid-mode case the policy has not defined, and C6, a remote start exactly on 1 March. C5 requires clarification; C6 uses the new three-day rule because the boundary is inclusive.

The deterministic answer-key comparison found three unchanged existing expectations, one changed expectation and two additions. Its stored seven-day C3 answer fits version one and fails version two. No model predictions are involved: the criterion changed around that answer. Calling the difference a model regression would tell a causal story the comparison does not support. First decide whether the assistant received the revised rule and which start date governs this case, so the next investigation concerns an actual source or interpretation problem rather than a misleading label on the score.

The answer-key change needs to preserve useful old coverage. Replacing every seven with three would make C2 wrong under the grandfathering rule, while deleting old cases would stop checking earlier starts. Revising C3 alone would omit the exact-boundary case C6. Each retained, revised or added expectation therefore needs its own rule reason, so the next run can be understood as evidence about a specified task rather than a changed collection with an unexplained score.

The controlling date should be tested explicitly. An assistant might accidentally choose a policy based on current date, record creation date or reminder-send date. All can look reasonable in a loosely worded instruction. The specification in this example names start date, and the fixtures include dates on both sides and at the boundary. If the business instead intends the new rule to apply to every outstanding packet regardless of start date, that is another task version. The evaluation should not decide that policy by whichever interpretation a model finds convenient.

Version the things that determine what correct means

The manifest keeps task specification, effective date, controlling event, case-set identity, source set and output contract separate. A real run also needs model, prompt and relevant preprocessing or retrieval configuration. Those identities let a later reader ask which part changed: the rule defining correct, the evidence supplied or the answering behaviour. One pipeline version can identify a release while hiding that distinction. For C3, a later comparison should still reveal that the changed expectation came from the remote-start rule, rather than inviting someone to infer a model change from a new set name.

First ask whether the assistant received the new rule. If it did not, the seven-day answer may come from source selection rather than inability to follow the revised instruction. Supplying the expected source in a small diagnostic comparison can help separate those questions. The actual service still needs its end-to-end evidence path checked, because that is what supplies the user’s answer. A correct response under specially supplied context would not repair a retrieval route that keeps bringing back the old policy.

Do not update the answer key solely because a new model produced a more fluent justification. A changed target needs a task or evidence reason that the responsible people accept. Equally, do not refuse to revise a demonstrably obsolete target because the old benchmark has been useful for months. The artefact should preserve why the rule changed and which cases it affects. Evaluation maintenance is a form of product maintenance, with the same need to keep an intelligible history of decisions.

Leave historical scores with the manifest under which they ran. Regrading stored answers against version two can be useful, provided both the new grading and original interpretation remain visible. Otherwise the dashboard can show a model getting worse on a day when neither its prompt nor its behaviour changed. The recalculation may be arithmetically sound; the implied story about performance is what has drifted. Keeping the task identity beside the result prevents that story from borrowing certainty from the number.

Keep development and judgement separate as coverage evolves

A policy change may require further development of prompts, examples, retrieval metadata or routing. Cases discussed in that process become development information. Google's dataset guidance describes separate final tests and the loss of independence through repeated adaptation.[1] The same concern applies when tuning a prompt against these policy cases. New independent coverage is needed for a final judgement once the expected answers have informed the implementation.

These six published cases are teaching examples, with coverage roles such as regression, legacy and boundary rather than secret held-out status. They do not estimate real error rates. Authorised representative inputs and independent final coverage are later investments for an operational evaluation. The small deterministic diff is valuable earlier: it can expose a contradiction in the rule while the team can still resolve it on six readable records, before another prompt or model comparison makes the disagreement harder to locate.

When reporting a result, distinguish performance on unchanged cases from performance on the revised task and new coverage. That makes a practical discussion possible. Did ordinary onsite handling worsen? Are earlier remote starts still governed correctly? Does the exact boundary now work? A single overall percentage can hide a gain in new cases alongside a loss in old ones, especially when the number and mix of cases have changed. Preserve the case-level account before deciding what summary is useful.

There are cases where an old rule no longer needs operational support. Then a deliberate retirement can remove its legacy fixtures from the current release gate, while retaining their historical record. The important thing is to establish that those inputs can no longer arrive through live or replay paths. Calling a case old is not enough if stored records are still processed later. Coverage should follow the supported workload, with removal justified as carefully as addition.

Treat a policy change as an evaluation event

The smallest useful maintenance routine is a rule diff and an answer-key diff reviewed together. Name the affected population, preserve unchanged expectations, add boundary and unknown-information cases, and record the version under which the next result will be interpreted. That gives a model or prompt update something stable to be judged against. Without it, the team can spend time improving the answer while quietly disagreeing about the question.

Keep C2’s seven days beside C3’s revised three, with the different start dates visible. Both answers are correct under version two. That pair explains the policy change better than a new aggregate score, and it keeps the next model run attached to a question people have actually agreed. A later seven-day reply can then be investigated through source selection and interpretation, instead of becoming another unexplained movement on a performance chart.