How to stop an automation that still has work to do
Retire intake, accepted jobs and external authority in a deliberate order, so a late event cannot revive an obsolete process.
No completed jobs might mean no incoming work, a stalled process or lost monitoring. A small state model separates those possibilities before asking someone to respond.
No jobs completed. That can be a healthy quiet day, a stalled worker or a chart that has stopped hearing from the service. The identical zero calls for three different responses. Before waking someone up—or deciding nothing needs attention—I would ask whether any work arrived and whether the observations are still current. The completion line cannot answer either question just by staying flat.
Twelve known arrivals with no completions would describe work still owed. A failed completion collector could instead leave us without the records of work that carried on. Fresh input and completion observations with no arrivals may justify leaving an idle service alone. Those distinctions sound obvious when written out, but an empty chart gives them the same appearance. Its source, coverage and age have to supply the missing context.
Google’s SRE monitoring guidance distinguishes internal measurements from externally visible behaviour and focuses monitoring on useful operational questions. [1] For a small automation, that can begin with a modest set of signals. The original fixture in this article uses known input identities, completion identities, evidence freshness and worker heartbeat. It is a teaching model, not a deployed monitoring system or a universal alerting policy.
Look for demand somewhere that can see it before relying on the worker’s silence. Its processed counter will stay at zero if the input never reached it. A submission ledger or suitable source-system record can name the work expected, provided it covers the tenant, destination and period being checked. That gives the operator an intake to compare with completion rather than asking the worker to certify its own lack of business.
In the known-quiet case, the intake, completion observations and heartbeat are all fresh, and no jobs are expected. In the blind-source case, the counts also read zero, but the intake is stale. The model keeps that one visibility unknown. The distinction prevents a missing observation from being silently converted into a claim about business activity. The operator can leave the first service idle and investigate the second without pretending either empty chart tells the whole story.
A heartbeat tells us that a process or its reporting path emitted a recent signal. It does not tell us whether every submitted job arrived or whether the output was correct. A gap in that signal with no known demand calls for a different investigation from twelve customers waiting for work. Keeping those messages distinct gives the person on duty a useful first question instead of a service badge whose red state could mean almost anything.
Choose the freshness limit from how the source reports and how quickly this service needs to notice missing work. A collector that normally updates hourly cannot support a two-minute judgement on its own. The fixture represents freshness as a Boolean so the choice is visible. A deployed monitor has to compute it from the actual timestamps and agreed reporting behaviour, or the apparently simple fresh label conceals another assumption the operator cannot inspect.
The overdue case contains twelve known operations J01 through J12 and completions J01 through J07. Five specific operations remain: J08 to J12. The oldest has waited eighteen minutes against an illustrative fifteen-minute work budget. This is evidence of overdue work under the model, even if every recent measured execution span was fast and the worker heartbeat continues to arrive.
Compare the same identified work on both sides. Twelve arrivals today and twelve completions today may cover different jobs, with yesterday’s backlog finishing while a new request waits. Duplicate completion messages can inflate the count too. Keep stable operation identities, or another justified relationship, so the comparison can name what remains. Then account for valid endings and pauses: an authorised cancellation may mean the job is no longer owed, and a deliberate review hold may alter its time budget. Record those decisions instead of deleting inconvenient rows or leaving them permanently overdue. Otherwise the cohort model has reproduced the chart’s original weakness in a more elaborate form: it counts something faithfully without keeping the meaning of the work attached to it.
The complete case has all twelve identified completions. The pending case has only J01 completed, but its oldest outstanding operation is five minutes old and remains within the example budget. Calling every non-empty queue overdue would send the operator to work that is still proceeding on time. The downloadable model keeps complete and pending separate from the eighteen-minute wait, giving the alert wording a specific difference to preserve even when both views contain unfinished jobs.
The six results follow from the invented inputs; they do not recommend a fifteen-minute commitment for a real service. They also need a useful definition of completion. Acceptance of a request, generation of a file and a verified destination update are different stages. Pick the one this monitor is meant to watch and name it. A completed local job can otherwise be mistaken for delivered business work simply because both appear under the same reassuring heading.
Original cohort model. Input and completion evidence must be fresh before their absence is interpreted. A 15 minute illustrative work budget distinguishes pending from overdue; worker heartbeat is a separate signal.
Scroll sideways to see all columns.
| Case | Input /completed | Visibility or worker state | Result |
|---|---|---|---|
| Quiet | 0 /0 | All fresh | Known quiet |
| Overdue | 12 /7 | All fresh; oldest 18 minutes | J08–J12 overdue |
| Blind source | Apparent 0 /0 | Input evidence stale | Visibility unknown |
| Heartbeat gap | 0 /0 | Input/completion fresh; worker stale | Investigate worker visibility |
| Complete | 12 /12 | All fresh | Cohort complete |
| Pending | 12 /1 | All fresh; oldest 5 minutes | Within example budget |
A failed telemetry path can erase both error and success measurements. Keep an explicit way to detect missing observations. Prometheus documents absent and absent_over_time for detecting absent time series, including absence over a period. [2] Such functions support a technical check; they do not by themselves establish whether there was no demand or which business operations failed.
The three signals may share a collector. If it stops, intake, heartbeat and completion can all disappear together while the service itself continues. A genuinely independent observation route can help separate those possibilities, with its coverage and upkeep made clear. Copying the same feed into several dashboards cannot do that. It gives the same missing report more places to look at, without finding out whether the source or the collector is quiet.
A fresh timestamp can still accompany the wrong selection. The query may exclude a renamed tenant, or a changed label may remove the jobs this view is meant to watch. Check the covered source and effective filter as well as the extract’s age. Repeating a recently produced zero does not improve it if the observation has stopped including the relevant work; it only makes the wrong reassurance arrive promptly.
This is a good place for a safe synthetic check where the workflow supports it: submit a clearly identified test operation and verify the appropriate result through the destination path. Keep it distinct from customer activity and prevent unintended side effects. A test operation can exercise a route even during a quiet period, but it does not prove every input category or every tenant has the same outcome.
The monitoring model separates missing evidence from the service state being observed.
Intake, heartbeat and completion may all go silent
Extract timestamp can still be fresh
An idle service cannot exercise its whole route
One route worked for that case
An overdue-work alert can name J08–J12, their oldest age and where to start investigating. A visibility alert should instead identify the missing observation and the conclusion now unavailable. “All jobs failing” would send the operator towards the service when the known problem is a stopped completion collector. Describe what needs attention precisely enough that the person can choose a first action without having to disprove the alert’s wording first.
Whether a silence should interrupt somebody depends on what the service owes. A missing heartbeat with no current demand might allow a business-hours repair. Time-sensitive overdue operations may not. Those are commitments for the people running the service to agree, using the actual consequence and response arrangements.
Follow the alert beyond the condition too. Does it reach the responsible person, name a maintained next step and show how to recognise recovery? A correct calculation which never arrives is still useful monitoring work left unfinished. Acknowledging the notification is not the ending either. Check whether the jobs completed or the missing view returned, so closing the alert records the repair it was meant to bring about.
The six cases leave three very different reasons to act: J08–J12 need an overdue-work investigation, stale intake needs restored visibility, and fresh observations of an empty cohort justify quiet. Give the operator that reason, then verify recovery against it. Restored telemetry will not complete a stuck job, and completed jobs will not explain a source the monitor no longer sees. A quiet service can be good news once the person responsible can tell which kind of quiet it is.
Working on a workflow like this? Hamsa helps finance and operations teams connect their tools and handle the exceptions, starting with one workflow.
See the Payment Operations PilotPrimary sources behind this article. Living documentation may change.

Practical engineering. A human perspective.
More about Hamsa Software and the person behind it. Editorial notes.