What does the evaluation permit us to release?

The decision, the system and the cases

The report says forty-two of fifty supplier summaries passed. The proposed release would also route urgent enquiries automatically, although those routing decisions were never judged. The score is useful information about the summaries. Before approving the release, I would ask how it is meant to support the extra action. That connection should be visible in the report, rather than left for the reviewer to reconstruct from a general statement that the system is ready.

A useful report makes that disagreement possible. It separates observed results from interpretations and from the permission being requested. “The candidate passed these cases” is evidence. “That suggests it may help this task” is an inference. “We should allow it to perform this operation in production” is a decision with a wider consequence. The report should not slide through all three using one reassuring percentage.

The fictional assistant drafts supplier-response summaries for internal review. Automatic urgent routing would be a new effect beyond drafting. A reviewer needs to see that proposed expansion alongside the summary criteria, so that agreeing to try a useful drafting tool does not quietly become permission for it to direct urgent work.

Start with the decision this run was intended to inform: perhaps whether R-8 can draft standard supplier-enquiry summaries for a named internal review group. That gives the cases and results a job. The colleague reading them can judge that proposed use, while recognising that a new audience or automatic action would ask something different of the system. The report then has a proposition to examine before it acquires a readiness label.

Record model deployment, prompt, retrieval, tools, output schema and relevant application revision for the evaluated configuration. Link run identities and retained outputs under the appropriate access controls. If R-9 changes any material component, the report needs to identify the difference and evidence required to extend its conclusion. A configuration name alone cannot establish that the deployed candidate matches the one assessed.

Describe the cases in terms of the intended work. For the supplier assistant, say which kinds of enquiries, languages, document formats and difficulty groups are included. Record exclusions and how cases were selected. Fifty hand-selected cases can expose important failure mechanisms. They cannot automatically estimate the failure rate of every future supplier message.

Keep development examples separate from held-out release evidence. If a failed case was used to tune the prompt, retain it as a regression case but label its role. Repeating a known example is useful testing; presenting it as unfamiliar validation overstates what the result tells us about generalisation.

Record disagreement among reviewers rather than forcing every difficult judgement into an unexplained majority label. A disputed summary may reveal an unclear business rule or an evaluation criterion that does not match the intended user task. The report can preserve both interpretations and name the decision needed to resolve them. Agreement obtained by hiding the ambiguous case makes the evidence tidier while leaving the product question unanswered.

The expected outcome needs provenance too. Who defined the criteria, what source or business rule supports them and how were ambiguous cases resolved? A model-written reference answer can be an aid, but it should not silently become the authority for judging the same model family’s response. The report should expose the judgement procedure clearly enough that another reviewer can contest a label.

Show the denominator that the release depends on

For the invented fifty-case run, report all outcomes: forty-two accepted, five corrected and three unusable. If a chart shows forty-two successes out of forty-seven usable outputs, its 89.4% figure hides the three failures excluded by that denominator. Against all fifty attempted cases, the accepted fraction is 84%. Neither calculation should appear without its population and purpose.

Say what correction involved. Fixing punctuation and restoring a missing approval condition can both leave a usable final summary, but they ask very different things of the reviewer. Keep the intervention and the reason it was needed. If review belongs in the product, measure or explicitly estimate that work. A corrected answer shows that the process recovered; it should not be counted as an answer the assistant produced ready to use.

Small groups need their own visibility. Suppose six cases contain an urgent request and two of those are mishandled. A good overall summary score does not dissolve that pattern. Conversely, six selected urgent cases do not support a precise population rate for every future urgent message. Preserve the count, the mechanism and the sampling limit together.

Distinguish a failure to run from a judged answer failure. A timeout, parser rejection and unsupported input may each prevent a usable result, but they suggest different repairs. Excluding them from the headline can make the model look better while making the actual service harder to assess. The release boundary is completed usable work, including the routes for anything that cannot be completed.

Let findings authorise only the work they cover

Automatic urgent routing needs its own assessment of labels, timing and consequences. This run judged summary quality, so it leaves that decision open. That is a reason to plan a specific routing evaluation, not to discard the summary results or surround them with general warnings. The team has learned something about a drafting task and now needs to investigate the different effect it wants the release to add.

A limited drafting trial for a named internal group may follow if the failure mechanisms and review capacity are acceptable. The missing approval condition could instead block that scope until repaired. In either case, explain which observed defect determines the decision and what reviewers are expected to catch. Merely naming human review would leave its actual ability to manage the failure unexamined.

Write down what the approved trial includes: supported enquiry types, required review, exception handling and the observation period. Give someone responsibility for expanding it. Otherwise a useful drafting assistant can gradually acquire another audience or external action while everyone still points to the original result sheet. The evidence has not become worse; the product has moved beyond the work those fifty cases were selected to examine.

Anthropic’s effective-agents article encourages evaluation and iteration while warning about the costs and compounding errors of autonomous operation. The application-specific work is to turn that general advice into a bounded decision. A published architecture pattern cannot supply approval for a tool whose inputs, effects and failure handling the supplier has never seen.

Write a conclusion that can be checked later

A useful conclusion for the example might say: “R-8 produced acceptable unedited summaries in 42 of 50 selected cases. Five required correction and three were unusable. This supports a reviewed drafting trial for the stated population, subject to resolving the approval-condition defect. It does not support automatic urgent routing.” The exact release judgement remains a business decision; the structure makes its relationship to evidence visible.

Then name the next question rather than requesting more testing in general. If urgent cases reveal missing qualifications, collect or construct cases that distinguish detection from preservation of the condition. If parser rejection dominates, inspect the response contract before changing the model. The report can then identify a specific follow-up experiment and the defect its result is intended to resolve.

Retain enough information to compare a later run fairly. Record changed cases, changed criteria and changed configuration. An improved score after removing difficult cases may be a narrower product scope, which can be legitimate, but it should not be presented as the same task becoming more reliable. Explain the scope change and its practical effect.

Monitoring after release has a different population and purpose. It can reveal new inputs, drift and operating failures, but it does not retroactively turn a selected test set into a representative sample. Keep trial evidence and production observations connected without pretending they are interchangeable. Both can change the decision when their limits are understood.

Leave the forty-two accepted, five corrected and three unusable outputs available with their criteria and the drafting decision. A colleague can then challenge whether the approval-condition failure is acceptable or whether the review allowance is realistic. That disagreement is useful: it concerns a specific release rather than confidence in a percentage. If the team later wants urgent routing, the report also leaves a clear next question for a trial designed to answer it.