Agree the answer before improving the prompt

The prompt cannot settle an undecided policy

A leaking tap and a broken cupboard hinge give a repairs router two defensible answers. Plumbing follows one part of the message; joinery follows the other. If the software is allowed to choose only one trade, someone must decide what happens to mixed requests before judging its reply. Otherwise the answer key quietly makes that policy decision for everyone, and a prompt can be marked wrong for disagreeing with a rule nobody has actually discussed.

That is a good reason to build a small evaluation set before polishing the prompt. The first benefit may be agreement about the task rather than a score. A collection of examples can reveal whether the desired output is a trade, a review destination, a description of all issues, or a decision about what happens next. Those are related tasks with different correct answers. Changing between them halfway through prompt development makes apparent improvement difficult to interpret.

Use twelve invented messages to settle that first disagreement. The accompanying sheet proposes a single destination for routine repair enquiries, with alternative labels written by hand to expose ambiguous cases. It contains no model predictions or real annotator study, and is not emergency triage. Its immediate reader is the person deciding where these requests should go. They need an answer they can defend to the colleague receiving the job, before a prompt is rewarded for producing it.

Give the labels boundaries a person can explain

Give the coordinator five destinations: plumbing, electrical, joinery, review and out_of_scope. A clearly stated single-trade fault goes to its trade. A mixed or insufficiently specified repair goes to review; a message that does not request a new repair leaves this routing process. Review is doing a particular job here. It asks someone to resolve missing information or competing trades, rather than treating every difficult message as a failure with no owner.

The tap-and-hinge message, M5, exposes the first policy choice. The two hand-written alternatives pick different trades, while the proposed answer is review because no trade-priority rule exists. M6 asks about an invoice: its correct destination is out_of_scope, not repair review. M7 says only that there is a problem in the kitchen. It needs clarification, so review fits; plumbing would add a fault the sender never described. Those differences determine which colleague gets the next piece of work.

Keep the reasons beside those targets. An unexplained plumbing label for M7 could teach the prompt that kitchens always imply plumbing, then reward it for repeating the author’s assumption. The reason reveals the missing evidence. The out_of_scope label has a boundary too: M6 still deserves a response to its payment enquiry, just from another process. A repair-routing label should not acquire the authority to decide whether customer service deals with the sender at all.

With real annotators, give two knowledgeable people the policy and source messages independently, then retain both answers when resolving a disagreement. If M5 makes them choose a new mixed-repair rule, revisit earlier examples governed by that rule too. Showing the second person the first answer may make the sheet look more settled while concealing the disagreement the exercise was meant to find. A useful answer key records the reason the coordinator would accept, including the cases where policy changed during discussion.

Keep related messages on the same side of the split

A correction is not an independent example simply because it occupies another row. M2 reports a broken socket; M3 corrects it to a broken cupboard catch. They have different targets but share group g2, so both stay in development. The dripping-tap paraphrases M10 and M11 share g10 for the same reason. If one member of either pair is used to improve the prompt, judging the other as unseen work would give the later result more independence than it has earned.

The local bookkeeping check leaves seven development messages and five held-out messages, with no group crossing that boundary. Publishing the sheet makes the held-out role illustrative: anyone who tunes against these answers has already used them for development. The next release judgement then needs fresh independent cases. Keeping five familiar rows in another file would not restore that separation, however little of the prompt was changed after looking at them. Independence concerns what informed the work, not the name of a partition.

Google's current machine-learning guidance explains the role of separate development and final test data, including the problem of repeatedly adapting to test results.[1] Scikit-learn's documentation describes keeping dependent groups out of both sides of a split.[2] Those principles also help when adjusting prompts rather than training model weights. The source of dependence is domain-specific: a copied thread, a repeated customer template or several revisions of the same document may all reveal information that a simple random row split misses.

Grouping can leave a tiny held-out set with poor coverage of particular labels or ambiguities. More independent cases are then needed to cover those conditions. Scattering related messages across partitions may balance the counts while undermining the independence the split was meant to provide. The test plan should describe both its coverage and its remaining gaps, because a percentage allocation cannot supply examples of an input the set does not contain.

Score only after deciding what a result means

Once the policy and partition are frozen, compare each later prediction with its message’s target. Keep unusable outputs separate from valid but wrong destinations. An unknown label cannot be routed under this contract; plumbing for vague M7 is interpretable but unsupported. Review for mixed M5 is correct, because that is the policy agreed before the run. Counting it as a model failure would quietly replace the coordinator’s rule with a preference for more automatic decisions, without asking who will handle the resulting repair.

Keep a run attached to its original predictions, task-policy version, input set and configuration. If review uncovers a bad label, investigate it; do not quietly change the target merely because the model’s answer sounds persuasive. A genuine policy change deserves recording, along with which earlier results it makes incomparable. I would rather retain a slightly untidy history of what the system was asked than turn an old score into an answer to a task that did not exist when it ran.

A single accuracy number is particularly unhelpful at this size. Each held-out row would move a five-row result by twenty percentage points. More fundamentally, different errors create different work: a misrouted repair, an unnecessary referral and an unsupported claim about a vague message are not interchangeable operational outcomes. Begin with the row-level comparison and the reason each error matters. More elaborate cost-weighted metrics can come later, once the team knows what decisions those costs represent. Inventing precision before agreeing the labels only conceals the original uncertainty.

The first deliverable may be a better task

There are alternatives to the proposed single-destination policy. A workflow might split M5 into two work items, apply an explicit priority rule, or preserve multiple issue labels for a coordinator. Any could be reasonable with the right operational design. They should produce different evaluation targets deliberately, rather than compete within an answer key that assumes all three are equivalent. Keeping this early exercise narrow makes that disagreement visible without pretending to settle the entire maintenance process. Splitting a mixed request also changes the unit being counted: one incoming message now creates two routing decisions, so an old message-level score cannot be compared directly.

The next set should bring in authorised examples of how people actually report repairs: corrections, vague descriptions, payment enquiries and mixed jobs that the synthetic sheet cannot represent in full. Keep their expected destinations tied to the agreed rule, and reserve independent cases for the later judgement. For the tap-and-hinge request, the first unresolved question is still who should handle mixed repairs. Deciding that gives the next prompt something meaningful to improve, and gives the person receiving its answer a reason to trust the route.