Should this old instruction still run?

The customer cancelled while the message waited

The consumer is repaired, and the old create-reservation message can finally run. Meanwhile the customer has cancelled the order. A dead-letter queue—a place to retain messages the normal consumer could not process—preserves that old instruction, not a promise that it still makes sense. Before sending it back, check what the business wants now and whether another route already completed or cancelled the action.

For order O-73, suppose create reservation refers to revision 4, but current revision 6 is cancelled. A proposed admission rule rejects the old create as superseded and records that cancellation is authoritative. It does not send another create merely because the payload is now parseable. If the current state is unknown, hold the item for investigation rather than guessing from the oldest available instruction.

Now consider a dependent change message waiting for the original create. If the order remains active, recovery may need to establish the reservation first, obtain its provider reference and then apply the change under the current rule. If the order is cancelled, both commands may be obsolete. Independent replay of each message cannot decide that relationship safely unless the consumer’s business-state checks already handle it.

For O-73, the report should explain that the old create was superseded by current cancellation, identify the relevant record revision and state that no new reservation was requested. If an external reservation might already exist, that is a reconciliation question requiring provider evidence. Rejecting the old create does not prove the external world contains no earlier effect.

Keep the old instruction attached to its decision

A queue service’s message identity may change during redrive. Amazon SQS documentation says redriven messages receive a new message ID and enqueue time. Keep the original business operation identity in the application’s record or message envelope where the design requires it. Otherwise a move intended as recovery can look like a brand-new command to downstream deduplication logic.

Record the original event time, source record revision, operation kind and any relevant dependency. Do not use the new queue enqueue time as the business age. For the reservation, the important fact is when the customer requested the operation and what has happened since, not when the recovery operator clicked the console control.

Choose what recovery is reconstructing. One route applies a bounded change from the receiver’s known state towards the object’s current desired state. Another replays events under a contract that defines their order and effects. The first can omit historical effects that still matter; the second needs the history and semantics to support replay. Decide which the reservation workflow requires before asking queue order to stand in for business history.

Ordering deserves explicit treatment. SQS warns that moving messages aside can break strict FIFO operation order, and its redrive documentation notes that recovered messages can interleave with new producer traffic. A recovery process cannot assume that oldest-first movement reconstructs the original business sequence at the destination.

The eligibility check and action also have a race. If the order can change again after the review, use the application’s supported conditional transition or revalidate at the effect boundary. A recovery manifest reviewed at 09:00 should not become permanent permission to act on an order changed at 09:05. Record the material revision and the response to a conflict rather than silently refreshing it.

Let repaired work share the available capacity

The old failure may have produced thousands of retained messages. Releasing them at once can overload the dependency that has just recovered, while current work is arriving too. This is a recovery-capacity question rather than a reason to keep increasing retries. Allocate room for live traffic and observe whether the replay makes progress without recreating the same failure.

For an illustrative dependency able to accept 60 operations a minute under the assumed pacing contract, reserve 40 for current work and admit at most 20 recovery operations a minute. A 600-item eligible replay set then has a thirty-minute lower bound for admission, before service time, retries or holds. It is not a completion guarantee. If live traffic rises, the recovery allowance may need to shrink.

Start with a small cohort and inspect its results before increasing admission within the policy. SQS redrive velocity controls message movement. It does not cap every API call the consumer makes afterwards, so a message that needs several calls must appear that way in the receiver-facing budget. Keep that distinction in the run view or the console can show a modest recovery pace while the repaired dependency receives much more work.

Cancellation of a redrive is also bounded. Stopping further movement does not recall items already moved, claimed or externally applied. Know how to pause the consumer or block a cohort if the recovered work reveals another problem. The recovery run should have an observable stop condition and a record of what had already crossed each boundary.

Account for the instructions that did not run

A useful cohort report divides messages into eligible, superseded, blocked by dependency and unresolved outcome. Each category has a next action or an accepted terminal explanation. Avoid one miscellaneous failed bucket that leaves the operator performing the same classification again tomorrow.

A replay manifest can identify the selected messages, their business operations, the repair applied and the admission result. Keep a relation between the original failed delivery and the recovery attempt. This allows an operator to explain why one item was replayed, another was superseded and a third remained under investigation without relying on the current queue count as the only record.

An empty queue can mean recovery, expiry, deliberate disposal or movement somewhere else. Give those outcomes different records so the next operator can tell which occurred. Otherwise the visible count can improve while the customer work merely becomes harder to locate. The stronger completion check follows each accepted operation to its outcome, including work deliberately withheld because a later business decision superseded it.

Exercise four cases before a broad replay: a current valid operation, a superseded operation, a dependent pair and an operation whose earlier external outcome is unresolved. Include a restart during the recovery itself. The resulting records should show which effects were attempted and why repeating the recovery will not turn a resolved item into new work.

There is a legitimate case for a simpler bulk redrive when every message is a repeat-safe projection of current state and dependencies are well behaved. Verify those properties rather than assuming them from the queue type. A message that sends a customer communication or creates a physical reservation has a different consequence from one that rebuilds a search document.

For the cancelled order, recovery succeeds by leaving the old create instruction unsent. Keep the cancellation revision and that decision when the message moves or disappears from the console.