Skip to content
Operations

When the Integration Changes Underneath You

A workflow that ran for months rarely breaks loudly. Something it depends on changes shape, and it keeps running on wrong data.


Mikhail Savchenko·August 28, 2026·4 min read
OperationsAutomationWorkflowProcess Audit

The failure that does not announce itself

An automation that stops is a small problem. It stops, somebody notices within the hour, the cause is obvious because the last thing that changed is the thing that broke.

The expensive failure is the one that keeps running. A field is renamed upstream, the code that reads it gets nothing, and nothing is a legal value in most processes - an empty note, an unset flag, a missing second line of an address. Nothing raises. The workflow produces records that look exactly like last week's records, and it does that for as long as it takes somebody to compare two numbers by hand.

What actually changes

Four shapes account for most of it.

A field is renamed or moved, usually as part of a tidy-up nobody thought was external. A payload gains a level of nesting when a vendor adds a wrapper for pagination or metadata. An enum gains a value - a new order status, a new document type - and the branch that handles the known values silently drops the unknown one. An API version sunsets, and the fallback turns out to be an older shape rather than an error.

None of these are outages. Every one of them is a Tuesday afternoon in somebody else's release notes.

Three defences, all cheap

One record, end to end, every morning. A synthetic item that goes the whole way and is checked at the far end against a known answer. It exercises the joins between systems, and the joins are where drift lives. Four separately healthy services can still be handing each other something that changed.

Assert on shape, not on status. A two hundred with a parseable body is not evidence that the body means what it meant last month. Check that the fields you read are present and typed as expected, and fail loudly when they are not. This is ten lines and it converts a silent wrong answer into a visible stop.

Alert on distribution, not on exceptions. If one in twenty items took the manual path last month and one in six takes it today, that is the signal - and no exception was raised to produce it. This only works if the ordinary numbers were written down first, which is the argument a measured baseline makes for itself.

Why this is a scoping question, not a maintenance one

Every integration is a dependency on somebody else's release schedule. That does not make it a bad idea; the rental delivery in where the four hours go reads availability from a system we do not control, and it still pays for itself. It makes the dependency a thing to price.

The practical version: when a workflow is scoped, list what it reads from outside itself, and for each one say what happens if the shape changes. Most answers will be "it stops, and that is fine". The ones where the answer is "it keeps going and we would not know" are the ones that need a canary before they ship, not after the first bad month.

The part nobody wants to name

A workflow that no person is responsible for is a workflow being checked by whoever happens to look, which in practice means after a customer complains. The name goes in the handover document with everything else, and it belongs to somebody who has opinions about the design rather than to whoever was free that week. The argument for treating that as a stage rather than an afternoon is in what a process audit must actually produce, and the reason it cannot be added later is that the person who forgot what was confusing cannot write it down.

Frequently Asked Questions
  • 01Why does an integration change break things silently?+

    Because most integrations are checked for success rather than for shape. The call returns two hundred, the body parses, and the workflow proceeds. If a field was renamed, the code that reads it now gets nothing and treats nothing as an ordinary empty value - a blank customer note, a missing second address line, an unset priority. Every one of those is a legal value somewhere in the process, so nothing raises. The workflow keeps producing records that look exactly like the ones it produced last week, and the only signal is that the outcomes have drifted, which nobody is watching for.

  • 02What is a canary record and why does it work?+

    One synthetic item pushed through the entire path every morning - created, routed, transformed, delivered - and checked at the far end against a result that is known in advance. It works because it exercises the joins between systems rather than any single system, and those joins are where drift lives. A monitoring check that pings each service separately will report four healthy services while the thing they hand to each other has changed underneath. The canary costs one record a day and answers a question none of the individual health checks can.

  • 03How do you tell drift from ordinary variation?+

    By having written down what ordinary looks like before the automation shipped. That is what the baseline stage exists for: volumes per week, the share of items that take the exception path, the spread of processing times. Drift shows up as a change in the shape of those numbers, not in their level - the mean holds and the tail moves, or the exception rate steps from one in twenty to one in six overnight. Without a baseline the same numbers are unreadable, because nobody can say whether this week is unusual.

  • 04Whose job is it to notice?+

    Somebody named, before the workflow ships. A process with no owner is checked by whoever happens to look, which in practice means it is checked after a customer complains. The name belongs in the handover document alongside what the workflow does, what it must never do, and which alert means what. That is why the handover stage is a stage rather than an afternoon at the end of the build - a system nobody has been made responsible for is a system running on the assumption that nothing will change, and something always changes.

Next step

Free AI Diagnostic

Fifteen minutes, no email required. It maps where your work actually goes and ranks what is worth automating first.

Start the free diagnostic

Starts immediately in the browser.

Fee
Free
Length
15 minutes

You keep the ranked list of candidates either way.

Keep reading