When the documents disagree

Two good sources can disagree

The assistant has found the manual, the site procedure and an email discussing a trial. Every link works. It can still give the wrong answer about replacing the controller remotely, because those documents do different jobs. The manual describes what the equipment supports; the current site procedure says what this customer permits. A citation tells the colleague where a sentence came from, but someone still has to work out which decision it can support.

An organisation's knowledge base might contain supplier manuals, internal procedures, customer agreements and proposals circulated by email. Each can contribute relevant evidence without governing the same decision. For an assistant preparing service guidance, finding the closest passage is only the beginning: the output needs to distinguish technical capability from what the business is currently authorised or committed to provide.

Take a fictional equipment-support question: “Can we replace the controller remotely?” A supplier manual says remote replacement is supported for model R after firmware 3. An internal service note says all controllers at one customer site require an engineer visit. A later email suggests that restriction may be removed after a trial. The assistant has found relevant material. It has not yet found permission to promise remote replacement.

An answer that shows the disagreement can save useful work. The colleague no longer needs to find the three documents and discover why they appear to conflict; they can focus on the missing approval or equipment detail. I would rather receive that precise unfinished question than a fluent yes which leaves me responsible for finding the condition that quietly turns it into no.

Restore the context around the passage

A retrieved sentence can lose the conditions that made it true. “Remote replacement is supported” is an incomplete result if the heading says model R, the previous paragraph requires firmware 3 and a note excludes managed customer environments. The retrieval pipeline needs to preserve enough surrounding identity and scope for the answer to use the passage correctly.

Anthropic’s September 2024 contextual-retrieval article describes adding document-specific context to chunks before indexing. It is an engineering approach to reducing information lost when documents are split. It does not turn a retrieved passage into an authoritative business instruction. Better retrieval helps find the relevant disagreement; it does not decide which side governs the customer’s service.

Keep the document identity, section, revision and model with the controller passage, and distinguish the customer note from the general manual. A short generated summary can orient the colleague before they open the source. If the recommendation depends on an exact qualification, let them inspect that passage too. Otherwise the assistant can inherit an earlier summarisation mistake and later cite the same material as though it independently supported the conclusion.

Choosing how much source material to return requires looking at the conditions that actual retrieval failures lose. A sentence may need its model-specific heading, an exception on the next page or the column label of an extracted table. Returning more text indiscriminately can bury the same conditions. These cases help identify which surrounding context must remain attached to a passage for the conclusion to be checked.

Authority depends on the question

The supplier manual has authority about the product’s technical capability. The site-specific service agreement may govern which work the business has agreed to perform. An internal note might explain the current operating procedure. The email may be evidence of a proposed change without being the approval that makes the change effective. Ranking all four by publication date would answer a much simpler question than the one the colleague asked.

A proposed decision record could separate three claims. First: the product can support remote replacement under stated technical conditions. Second: the current site procedure still requires an engineer visit. Third: there is an unresolved proposal to change that procedure. The assistant can cite each claim and say that the proposal does not establish the present service route.

There is no source ranking that will settle every case. A supplier correction may supersede a technical manual; a signed customer exception may deliberately override an internal default. Decide which rules govern the decisions this assistant is meant to help with and who can resolve a conflict. Apply those rules in the application. The model should not have to infer organisational authority from whichever document sounds most confident about its own importance.

The bounded answer here is that remote replacement is technically supported under the manual's conditions, while the site's current procedure still requires a visit. That gives the colleague a usable present decision and identifies its limit. A general yes with an unspecified qualification would leave them to discover whether the site restriction counts among the conditions, recreating the source-resolution work the assistant was meant to reduce.

Show the unresolved part without drowning the reader

The assistant may also lack a fact about the equipment itself. If the installed firmware is unknown, the manual’s condition remains untested even after source authority is settled. Ask for that missing fact rather than treating the most recent product specification as a description of the device on the customer’s desk. Source resolution and case identification are separate work.

The answer need not reproduce every document. Lead with the permitted conclusion, then show the evidence that changes the action. In our example, the site restriction belongs near the recommendation, not in a footnote beneath a general description of remote maintenance. A reader who sees only the first paragraph should not receive the opposite practical instruction from a reader who opens all the links.

A compact evidence panel can list the technical capability, site restriction and proposed change separately. Each row should link to a stable source location where possible and record the revision used. Mark the conflict explicitly. A coloured confidence badge would be a poor substitute: confidence in retrieval, confidence in interpretation and authority to promise a service are different properties.

Here the escalation can be quite small: has the site restriction been formally removed, and from what effective date? The assistant has already separated the manual’s capability, the current procedure and the proposed change. Send those sources with that question. A colleague can seek the relevant approval rather than begin another broad review of whether the whole answer is trustworthy or repeat the original search from scratch.

Be careful with generated follow-up actions. Drafting a message that says remote replacement is available can undo the caution in the answer above it. The draft should preserve the same boundary, perhaps offering to confirm the current site arrangement. A good evidence panel is wasted if the one-click reply quietly promotes a proposal into a commitment.

Evaluate the promise, not just the citation

A useful test set for this assistant would include the controller question in several variations. Remove the customer restriction and see whether the answer changes appropriately. Make the email an explicit, authorised procedure update with an effective date and check that the result follows the documented rule. Change the firmware condition and confirm that a technically unsupported configuration does not receive the same answer.

These variations examine the causal role of the evidence. An answer that gives the same recommendation regardless of which decisive document is present may have failed to use the decisive source. Keep expected claim boundaries in the evaluation record, including what the assistant must leave unresolved. A correct refusal to promise remote work should not be counted as failure when the source conditions do not permit that commitment.

There is a cost to this discipline. More explicit source policy takes time to maintain, and some questions will require a colleague. For a low-impact explanation of a public feature, a lighter presentation may be sufficient. For customer commitments or operational changes, the ambiguity already exists whether or not the assistant shows it. Hiding it makes the answer shorter, not the decision simpler.

Leave the colleague with a present answer and the decision that could change it. Model R may support remote replacement under the manual’s conditions, but this site still requires a visit until the procedure is properly changed. The proposal email makes the next enquiry easier to target; it has not made a customer promise safe. That is a useful result for the assistant to hand over, even though it is less compact than a confidently sourced yes.