Claude 4 made the long task more tempting

A long run needs a definition of finished

The attractive part of a long coding run is the prospect of leaving a well-described job with an assistant and returning to a reviewable change. Anthropic’s 22 May 2025 launch of Claude Opus 4 and Sonnet 4, alongside Claude Code’s general availability, put sustained coding work near the centre of that promise. [1] For a small team, the question is how much work remains when the assistant says it has finished.

An assistant can spend that time improving an implementation and its tests while leaving the same compatibility mistake in both. More persistence might help it discover the mistake, or give it time to build further changes around the mistaken assumption. I would judge the long run by where an independent account of the required behaviour enters it. Otherwise the code and its reassuring tests can become a rather self-contained conversation.

Anthropic’s launch methodology described the tools used in its coding scaffold and distinguished ordinary results from a higher-compute procedure using multiple attempts and candidate selection. Those conditions affect how a reported result should be interpreted. [1] A team considering the tool needs to relate that evidence to its own repository, task context and review capacity; the headline score does not specify those conditions for them.

The example here is deliberately small: add validation to a customer importer while continuing to accept the export already in use. Both implementations were written for this explanation, not generated in a Claude trial. That lets us inspect the whole disagreement instead of guessing why a model behaved as it did. The old export supplies a useful test of completion because it has a claim on the result that the new code’s own tests might overlook.

Agree the awkward part before generating the patch

The importer accepts two alternative names for its identity column. Existing files use Customer ID; newer files use customer_id. Exactly one must appear. A row must supply a non-empty identity and have the same number of fields as the header. An optional notes column may remain. These are local rules for this example, chosen to make compatibility and rejection behaviour explicit rather than to represent every CSV integration.

A file containing both identity columns makes the compatibility rule more difficult. Choosing one header might seem reasonable while most rows contain the same value twice. But a row with C17 under one heading and C18 under the other has two account interpretations. This contract rejects the file rather than silently selecting one. Defining that case before implementation gives the reviewer a way to distinguish an intended rejection from a missing feature.

Python’s csv documentation explains its sequence and dictionary readers, including how dictionary reading maps fields and handles rows with extra or missing values. [2] Parsing a file into dictionaries does not enforce our business contract. The application must decide whether a familiar parser result is acceptable. This is precisely the sort of assumption a coding task should expose: using a standard library is sensible, but using it does not supply every missing rule.

It helps to leave storage changes, interface redesign and dependency updates out of this first task. Each would add a legitimate subject for review, but none is needed to settle the identity-column rule. Start from a known commit and write the expected outcomes where both the assistant and reviewer can see them. Independent cases are not a trap laid for the tool; they are a way to keep an existing customer file represented in the discussion.

Two patches, one revealing old file

Candidate A uses a dictionary reader, requires customer_id and checks that each resulting row has a value in that field. It is compact. It accepts the new format and rejects a missing or empty identity. If its tests cover only those examples, the implementation and tests can agree that the task is complete. The existing Customer ID export is nevertheless rejected before its first record is considered.

Candidate B reads the header as a sequence, locates recognised identity aliases and requires exactly one. It then checks row width and the selected value. The difference is not simply that B has more code. It implements an agreement A omitted: compatibility includes the old header, and ambiguity must be rejected rather than resolved by whichever dictionary property the author happened to prefer.

Check the parser’s quiet behaviour

The extra-value case is worth tracing separately from the old header. With customer_id and notes in the header, a row containing C17,a,b has three values for two names. The dictionary reader can retain the extra value under its configured remainder key. Candidate A asks only whether customer_id is non-empty, so it accepts the row while ignoring the unexpected shape. The parser has not promised our application that the import contract passed; it has represented the input according to its own documented rules.

Candidate B keeps the header positions and checks how many recognised aliases it found before choosing one. It also compares row width before accepting the identity. The order matters: after extra data has been ignored or one of two account columns silently selected, a later validator may no longer see the original ambiguity. Adding more checks downstream can make the patch look thorough without recovering the information discarded upstream.

The six-case fixture makes the difference concrete. A rejects the legacy export, accepts a file containing both aliases and accepts a row with an extra field. B produces the expected result in all six cases. The downloadable program lets you alter an input and follow the rule that produces its answer. Retain the expected outcome when trying the change: otherwise it is easy to make the test agree with the new parser while withdrawing the compatibility promise it was meant to protect.

A real importer has further obligations: character encoding, duplicate records, field lengths, safe reporting of invalid rows and the consequences of partially processing a file. The fixture does not settle those. It demonstrates a narrower problem with self-tests: cases based only on the new header can approve a change that rejects existing exports. Including a historical file makes the compatibility requirement independently checkable, even when the new-format tests pass.

Run the real comparison against the whole job

If you trial a coding assistant on this task, keep the packet stable: starting commit, allowed files, contract, relevant existing behaviour and model configuration. Save the actual proposed patch and test output. If the brief changes midway because the team discovers an unstated rule, record that change. A better second result may reflect better task preparation as well as a different model or a longer run; both are useful to understand.

The effort around the patch is part of its cost. Someone supplies the repository context, corrects assumptions, reads the diff and establishes whether the result works. A fast first patch may use more of that time than a slower, focused one. Counting only the interval while the assistant is running would be an unusually considerate way to measure its workload, particularly if the reviewer spends the afternoon repairing compatibility afterwards.

Use the final review to connect the code with the original request. GitHub’s pull-request documentation describes the changes, discussion and checks available together for that purpose. [3] Ask a reviewer to explain the alias policy and the ambiguous-input outcome from the diff, rather than accepting a summary saying validation is comprehensive. If the explanation requires private knowledge from the prompting conversation, put that knowledge into the task or repository where the next maintainer can find it.

Longer runs do not require a person to interrupt every minute. They do make it more valuable to agree what a finished result will look like before the work starts. Here, a historical export and six specified cases keep the original obligation visible throughout the run. If the final patch rejects that export, the reviewer has a concrete reason to return it, however many useful changes or passing new tests arrived with it.