We have been called into eleven rescue engagements in the last three years. In nine of them the model already worked. What had failed was everything around it.
There is a specific and repeatable failure mode in industrial AI. A team builds something genuinely clever, demonstrates it on historical data, gets applause in a steering committee, and then spends eighteen months failing to deploy it. Eventually the budget is quietly reallocated and the organisation concludes that "AI does not work for us."
That conclusion is almost always wrong. The model worked. The path to production did not exist.
Reason one: the pilot used data that production will never have
This is the most common and most fatal. A pilot is built on a curated extract — cleaned, labelled, complete, pulled by a data scientist who had time to fix the gaps by hand. Production has none of those advantages. Sensors drop out. The MES writes a null where the historian writes a zero. A shift supervisor changes a batch code and nothing downstream is told.
The fix is not more data engineering after the fact. It is building the pilot against a live feed from week two, however ugly that feed is. You want to discover the missingness while you still have the freedom to redesign around it.
Reason two: nobody owns the decision the model informs
A defect-detection model produces a signal. Someone has to act on that signal. If the answer to "who acts, and what changes in their day" is vague at kickoff, it will still be vague at go-live — and the model becomes an interesting dashboard nobody opens.
Before we build anything, we write one sentence: "When the model says X, [named role] does Y instead of Z." If we cannot write that sentence, we do not start.
Reason three: the operators were not part of it
Line supervisors have usually seen at least one failed automation attempt. They are not being obstructive when they distrust the new system; they are applying reasonable priors. A model that arrives fully-formed and demands to be believed will lose that argument every time.
What works is shadow mode. Run the system in parallel with the existing process for six to eight weeks. Show its call and the human call side by side. Let operators see where it is right, where it is wrong, and — critically — let them override it and have that override recorded. Trust is earned by being visibly accountable, not by being accurate in a slide deck.
Reason four: no evaluation harness, so no way to argue
Without a fixed evaluation set, every conversation about model quality becomes anecdote versus anecdote. Someone remembers a bad call from Tuesday. Someone else remembers a good week. Nobody can settle it.
A golden dataset — a few hundred representative cases with agreed labels, held out and versioned — turns those conversations into engineering. It also gives you a regression suite, so the model you deploy in month nine is provably not worse than the one you validated in month two.
What a harness needs
Representative cases including the hard ones. Labels agreed by the people who own the decision, not by the data team alone. Version control. And a rule that the set is never used for training — the moment it leaks into training it stops telling you anything.
Reason five: the deployment target was never real
Plant floors are not cloud environments. There is often no reliable outbound connectivity, latency budgets are measured in milliseconds against line speed, and the hardware has to survive dust, vibration and temperature swings. A pilot built in a notebook on a laptop has made none of these decisions.
Decide the deployment target in week one and build toward it. Edge inference, network topology, failover behaviour when the model is unavailable — these are architecture questions, and answering them late means rebuilding.
What we do instead
- Name the decision first. One sentence, one owner, one metric. If we cannot write it, the use case is not ready.
- Live data from week two. Ugly, incomplete, real. We would rather find the gaps early.
- Build the harness before the model. Golden set, labelling protocol, regression suite.
- Deploy to the real target on a thin slice. One line, one defect class, one shift — end to end, in production, in weeks.
- Shadow mode until the operators sign off. Not until the data scientists do.
None of this is glamorous, and none of it makes for an impressive demo. It is, however, the difference between a system that runs for years and a slide deck that gets archived.
