hamsa softwareA model launch invites a paired migration experiment. The revealing result is often the case that got worse.
A new model can make a previously awkward task worth another attempt. Perhaps a maintenance summary keeps losing an approval condition, or a coding change needs to cross several components. That gives the trial a purpose. A launch announcement cannot tell the team whether replacing the model in its existing tool will improve that task, because the result still depends on the inputs, surrounding components and person who has to use it.
Anthropic announced Claude Opus 4.6 on 5 February 2026, describing improvements in coding and longer agentic work, alongside a million-token context window in beta. These are the supplier’s claims about a particular release. They give a team sensible questions to investigate.
The distinction matters because a migration changes a relationship between components. The prompt, retrieved material, output parser, time limit and human review process were all built around some previous behaviour. A stronger model may use those components differently. It can solve a harder task and still make your ordinary task slower, more expensive or less predictable in a way that matters to the person waiting for it.
I would give both configurations the same cases and read the disagreements. A paired comparison shows which old failures disappeared and which previously good answers became worse. The average still has a role, but it cannot tell a maintenance team that the extra pass came with a lost approval condition. That is the migration decision hiding inside an apparently straightforward improvement in score.
Suppose an internal assistant turns equipment-maintenance notes into a proposed service summary. The current system handles simple notes reasonably well but sometimes misses a qualification such as “inspect only; replacement requires approval”. The proposed benefit of migration is preserving those qualifications while producing a useful summary. That is narrower than “better reasoning”, and much easier to examine.
For that decision, collect ordinary notes alongside conflicting dates, a withdrawn instruction, a negated fault and an approval condition. Keep development material separate from the cases used for the release judgement. Otherwise successive prompt changes may adapt the candidate to the examples under review, leaving the team without evidence of how the revised system handles comparable notes it was not tuned against.
Before running anything, specify the unacceptable transition. For this assistant, dropping the approval condition is a serious regression even if the prose is clearer. A slightly longer summary may be acceptable. An invalid output structure might be repairable through validation, but that repair has a cost and should appear in the comparison. These are decisions about the application, not a universal ranking of models.
Record the source, prompt, schema, retrieved material, available tools and generation settings before comparing the models. If the candidate needs a different prompt or feature, give that change its own experiment. Redesigning the whole application may be worthwhile, but then the team is evaluating that redesign. Otherwise it becomes difficult to explain which improvement came from the substituted model and which came from the other work that accompanied it.
Here is a deliberately invented result sheet for twelve summaries. It illustrates how to read a comparison; it is not a measurement of Opus or any other model. The current system passes nine cases and fails three. The candidate passes ten and fails two. A headline announcing an improvement from 75% to 83.3% would be arithmetically correct and surprisingly uninformative.
Look at the pairs. Eight cases pass with both systems. Two fail with the current system and pass with the candidate. One passes with the current system and fails with the candidate. One fails with both. The candidate therefore repairs two failures in this teaching sheet but introduces one regression. The migration question now has shape: which failure did it introduce, and what happens if that failure reaches a user?
Suppose the two repairs concern awkward phrasing and an incomplete date, while the regression removes the approval requirement before replacement. Ten passes would be hard to defend as progress over nine. Reverse the consequences—an extra harmless sentence in exchange for preserving important constraints—and declining the change needs an explanation too. The arithmetic has stayed the same. Reading the paired outputs is what gives the release decision its meaning.
Preserve each paired output and the reason for its assessment so another reviewer can challenge the label without reproducing a configuration that may later be unavailable. Asking a second model for a reason after revealing the desired label weakens that independence. The report should expose where reasonable reviewers might disagree, especially when a fluent summary has omitted a consequential qualification.
A fictional paired comparison of twelve maintenance summaries. These are invented judgements, not model benchmark results.
Scroll sideways to see all columns.
| Current result | Candidate passes | Candidate fails |
|---|---|---|
| Pass | 8 cases unchanged | 1 regression |
| Fail | 2 repairs | 1 shared failure |
A model call is not the whole service-summary task. A faster first response that needs two repairs may produce a slower usable result. A longer initial response that preserves the approval condition may reduce review work. Measure the boundary the colleague experiences: from a ready input to a summary they can actually use, including validation failures, retries and required human correction.
Imagine ten successful old-configuration jobs taking twenty seconds each and ten successful candidate jobs taking sixteen. Those assigned times look attractive until two candidate failures spend ninety seconds retrying before referral. Keep those jobs in the account, with timeouts and unsuccessful results identified. Otherwise a reliability problem has helped produce the speed chart. The colleague needs to know how long a usable summary takes, including the runs that never produce one.
Use the same discipline for cost. Record request usage and any priced tools, then state the tariff and date used in the calculation. Keep human review minutes visible as a separate quantity unless you have an agreed way to value them. A blended financial estimate can be helpful, but an unexplained hourly rate adds confidence rather than evidence. Cheap model output and cheap completed work are not always the same thing.
A larger context window is a separate design opportunity. Test which additional documents are useful and current rather than assuming that including more will improve the maintenance summary. Keeping compact retrieval while using the candidate on difficult cases may be the better application change. That comparison needs its own configuration record because expanding the source material changes more than the model substitution alone.
The first decision need not be full replacement. The paired results might justify sending one well-defined task type to the candidate while retaining the existing route elsewhere. Or they may justify a larger investigation, with no production change yet. These are respectable outcomes. An experiment that discovers the reason not to migrate has still reduced uncertainty.
For the maintenance assistant, a release note could say that the candidate repairs two of the example’s failures but loses an approval condition on one previously passing case. The immediate decision is to retain the old production configuration, investigate that regression and repeat the comparison on held-out material after a proposed repair. This is a much more useful conclusion than “promising, needs more testing”, which has the unfortunate property of surviving almost any result.
Rollback needs the previous complete configuration, including a compatible prompt and parser. If the candidate's prompt changes the response structure, substituting the old model beneath it may fail to restore previous behaviour. Confirm that the earlier model remains available and decide how in-progress work is handled. Without those checks, a route labelled rollback may still leave the service in a configuration it has never evaluated.
For this assistant, I would make the approval condition decisive. Show whether the candidate retains it while repairing the summaries the current system mishandles. The result may support a narrow deployment, more investigation or staying with the existing configuration. Keep the paired outputs and the reason for that choice. When the next release appears, the team can reopen an actual argument about its work rather than repeat a vote on whether the new model sounds promising.
Working on a workflow like this? Hamsa helps finance and operations teams connect their tools and handle the exceptions, starting with one workflow.
See the Payment Operations PilotPrimary sources behind this article. Living documentation may change.
Practical engineering. A human perspective.
More about Hamsa Software and the person behind it. Editorial notes.A saved replacement, a running service and an attacker’s existing access can change at different times. Railway’s guidance explains why that matters; PingCAP and PHP show where the next job differs.
Computer use needs an observation contract for account, object and outcome, especially when a visible click produces only partial completion.
Local inference offers a different operating boundary. Evaluate the completed task, data routes and maintenance responsibility before buying into it.
