The 5 things that break when your AI leaves the lab
Your model is rarely the problem. Five other things break first, in a predictable order.
Hello everyone,
If you’ve ever watched an AI proof-of-concept earn a standing ovation in a Tuesday demo and quietly fall apart by month two, you know the feeling we’re naming today. In this article we’re going to deep dive into what actually breaks when your model leaves the lab, in the order it breaks, and what your team can do before the first incident hits. Field notes, named cases, kitchen analogies. Let’s go.
The gap is older than the hype
The “POC works, production doesn’t” problem isn’t a 2025 problem. It’s a 2015 problem that finally got expensive.
In 2015, D. Sculley and team at Google published Hidden Technical Debt in Machine Learning Systems (NeurIPS), with the now-famous diagram where the ML model is a tiny box surrounded by giant boxes: data, configuration, serving, monitoring. Their warning: “ML systems have a special capacity for incurring technical debt… plus an additional set of ML-specific issues.” In 2017, Breck et al. shipped the ML Test Score: 28 production-readiness tests across data, models, infra, monitoring.
A decade later, the gap is still there… and the bills are bigger. RAND (Aug 2024) put the AI project failure rate at >80%, twice the rate of non-AI IT projects. Gartner forecast in July 2024 that 30% of GenAI projects would be abandoned after POC by end of 2025, revised to ~50% in 2025. McKinsey (Nov 2025) found 88% of orgs use AI somewhere, only ~6% are AI high performers; nearly two-thirds haven’t begun scaling. The space between “we built a model” and “it runs the business” is this article.
Test kitchen vs. Saturday night service
Before we list what breaks, we need a shared mental model for why it breaks.
A POC is built to prove the model can do the thing. Production is built to do the thing 24/7, on data the model has never seen, while a business depends on it, regulators watch, costs accumulate, and someone on call wakes up at 2 a.m. to fix it. Same artifact, different worlds. Almost nothing else around it is the same.
Think of a restaurant test kitchen versus Saturday night service. In the test kitchen, one chef cooks one plate at low pressure with the freshest mise en place, and the dish is brilliant. Saturday night, 12 plates fire at once, a sub’d ingredient shows up, the line is yelling, a guest has an allergy, the dishwasher breaks. The recipe didn’t change; the system around it did. That’s the gap, and it’s why Shankar et al. (ACM CHI 2024) titled their study of 18 ML engineers “We Have No Idea How Models Will Behave in Production Until Production.”
ℹ️ Small glossary
POC (proof of concept) = a small build that proves a model can do a task, usually offline, on a fixed dataset.
Production = the version of the system that actually serves real users, real traffic, real money, every day.
MLOps = the practice of running ML systems in production: deployment, monitoring, retraining, governance.
Drift = the slow (or sudden) change in the data your model sees once it’s live, compared with the data it was trained on.
So… what actually breaks first?
Here’s the twist most teams miss. When a POC fails in production, the model itself is rarely what breaks first. Five other things break first, in a predictable order. Each break creates the conditions for the next. Skip one and you don’t escape it; you just meet it later, with more damage.
A nuance: this order is the modal path, not the only one. Regulated industries (healthcare, finance) often hit governance first. And Shankar et al. push back on the “90% of models never make it to production” stat: constant experimentation should produce many versions, most of which never ship. Killing a POC fast can be a feature. The question we’re answering: when a POC should have shipped, what stopped it?
⚠️ When NOT to ship: if your evals are still moving, your data is thin, or no one will own the system in production, killing the POC now is the cheap decision. The five breakpoints below are for projects that earned their way to production — not a mandate to ship everything that demos well.
The 5 breakpoints, in the order they bite
Break 1. Data drift (Weeks 1–4)
Production data is not training data. The first divergence is almost always in the input distribution: new vocabulary, an upstream UI change, a seasonal pattern not in your training window, a customer segment that didn’t exist six months ago.
Think of a French crêpe recipe perfected in Paris with French flour, butter, eggs. Ship it to Mexico City and the same recipe produces a different pancake, not because the recipe is wrong, but because every input drifted. Data drift isn’t a bug; it’s geography catching up with your model.
Chip Huyen (Designing ML Systems, O’Reilly 2022) calls this the “ML systems are alive” problem: P(X), P(Y), and P(Y|X) all drift, often independently. No human has to do anything for drift to start ⇒ the moment real users touch the system, the clock starts. In the deployments I’ve seen, a POC at ~94% lab accuracy lands closer to ~62% in production within a month. Not a model problem; a the world moved problem.
What to do early:
Log every production input with timestamps, from day one.
Compute drift metrics weekly on the top 10 features (or top tokens, for LLMs).
Set the retraining trigger before you need it ⇒ never build it during an incident.
Break 2. The offline–online evaluation gap (Weeks 2–6)
Your offline metrics said the model was great. Online metrics say it’s mediocre or worse. The gap isn’t a bug in either measurement; they’re measuring different things, and you didn’t realize it.
This is driving lessons in an empty parking lot. No other cars, no kids on bikes, no rain. Hand over the keys for rush hour and the driver is technically qualified, technically failing.
Castells and Moffat (2022) put it bluntly: “The correlation between offline and online evaluation outcomes is often weak.” Eugene Yan argues for eval suites aligned to product behavior, not generic accuracy.
A tiny worked example of the gap (picture a model that suggests replies inside a customer-support tool):
Metric Offline (lab) Online (prod) What changed Top-1 accuracy 0.92 0.71 14% of real inputs are out-of-vocabulary CTR (click-through rate — the % of users who actually click a suggestion) n/a -3% vs. baseline Users skip “too confident” suggestions P50 latency 80 ms 240 ms Real traffic, real concurrency User trust n/a 2.8 / 5 Early wrong answers anchored perception
What to do early:
Run shadow mode for 4–8 weeks; duplicate production traffic to the new model and log it all.
Canary at 1% → 10% → 50% → 100% with explicit rollback gates (a canary release sends the new model to 1% of users first and ramps up only if it behaves).
Add at least one “business outcome” metric to your eval suite, not only accuracy.
Break 3. Monitoring and observability gaps (Months 1–3)
You can’t see what’s happening. By the time someone notices the outputs are getting worse, it’s because a customer complained, not because a dashboard alerted. You have logs of latency and uptime (DevOps observability) but nothing about prediction quality, drift, fairness, or degradation.
This is a car with no dashboard. It drives. It might be running on fumes, the temperature might be redlining, a tire might be soft. You’ll find out when it stops.
Drift and the eval gap can exist invisibly for weeks. They become visible only when monitoring catches them, and most POCs ship with no ML-specific monitoring, because the lab dataset was static.
Breck et al. dedicate 7 of their 28 ML Test Score tests to monitoring. Evidently AI ships 100+ observability metrics because the gap is that common. The National Institute of Standards and Technology (NIST) AI Risk Management Framework (AI RMF) formalizes “ongoing measurement” under MEASURE; even regulators now expect you to be watching. The budget is sneaky too: retraining, evals, drift monitoring, on-call tooling, compliance eat roughly 4–10× your model/inference budget in year one. POC at $100K ⇒ plan $400K–$1M for year one. Can’t get that? You have a demo.
Three layers to instrument from day one:
Infra: latency, uptime, cost per call.
Data: input/output drift, fairness slices.
Business: the metric the AI is supposed to move.
Most teams ship layer 1 only ⇒ they’re flying with a fuel gauge but no altimeter.
Break 4. Infrastructure, latency and cost at scale (Months 2–4)
The thing that ran on a laptop now runs on production traffic. P99 latency triples (P99 = the slowest 1% of requests; when it triples, roughly 1 user in 100 hits painfully slow responses even if most feel fast). The inference bill arrives. Someone realizes the embedding service is calling itself in a loop. The CFO asks why “the AI thing” is the third-largest line item in cloud spend.
This is a food truck going brick-and-mortar. Recipes work, vibe works. But suddenly you need a walk-in fridge, a dishwasher rated for 1,000 covers, a permit, payroll, HVAC. Same food, different operation.
Costs accumulate. First invoice small, second alarming, third triggers a meeting. Latency spikes as user load grows past the POC’s tested level, often 10–100×. The FinOps Foundation’s 2026 State of FinOps report named AI the fastest-growing new spend category, with 73% of teams reporting their AI costs ran over the original budget; CloudZero clocked average monthly AI spend jumping from $63K in 2024 to $85.5K in 2025 (+36% in a year). Gartner’s March 2026 outlook: 1T-parameter LLM inference will cost 90% less by 2030, but “as token consumption rises faster than token costs fall, overall inference costs are expected to increase.” In plain terms: the price per token falls ~90%, but because teams consume far more tokens, the total bill still rises.
What to do early:
Load-test at 10× and 100× expected traffic before launch.
Set per-feature cost budgets, alert at 70%.
Cache aggressively, batch where you can, pick the smallest model that clears the eval bar.
Break 5. Governance, ownership, and on-call (Months 3–6+)
Something goes wrong (a wrong answer, an unfair outcome, a regulator’s letter, a journalist’s email) and nobody owns it. The data scientist has moved teams. Legal has never seen the model card (a model card is a one-page spec documenting what the model does, its limits, its training data, and its known risks). No on-call, no incident playbook. The system is business-critical and orphaned.
This is a restaurant with no manager on duty. The food might be brilliant, the kitchen might hum. But when a customer slips on a wet floor, no one knows whose phone to call. Governance is a consequence function: it only fires once breakpoints 1 through 4 produce a visible incident, and when it fires, it’s the most expensive break of all.
The named cases aren’t subtle. In February 2024, the BC Civil Resolution Tribunal ruled in Moffatt v. Air Canada that “Air Canada still bore responsibility for all the information on its website, whether it came from a static page or a chatbot.” Damages were small ($812 CAD); precedent was huge: companies own their AI’s outputs.
Zillow Offers crashed harder in November 2021: a $569M write-down and 25% workforce cut (~2,000 employees) when the iBuying algorithm overvalued homes and no governance layer caught it in time.
The 2025 wave is more serious. By end-2025, at least 10 lawsuits had been filed against OpenAI and Character.AI for wrongful death, negligence, and product liability. In May 2025, Garcia v. Character Technologies let a US federal court rule that a chatbot can plausibly be a “product” for liability. And Europe is turning governance into a dated obligation. The EU AI Act lands in waves: prohibited practices and AI-literacy duties since 2 February 2025, GPAI-model rules and the penalty regime since 2 August 2025 (fines up to €35M or 7% of global turnover), and the bulk of the high-risk obligations (Annex III) due August 2026 — with some deadlines now pushed toward late 2027 under the 2026 “Digital Omnibus.” Pair that with ISO/IEC 42001 (Dec 2023, the first international AI-management standard) and NIST’s AI RMF, which puts GOVERN at its center on purpose, and the message for any team industrializing in Europe is blunt: the owner, the model card, and the kill-switch aren’t optional best practices anymore — they’re artefacts you must be able to produce on demand. Governance is no longer the boring last slide; it’s where projects die. A model without an owner is a liability with a UI.
What to do before launch:
Name the on-call (the named person on rotation who gets paged when the system breaks in production). Write the rollback runbook. Get legal to sign off on the model card.
Make sure customer support knows what the AI does and does not do.
Define the kill-switch and the criteria that pull it. Practice it.
Who owns each break in production, fill this in with real names, not role labels:
Breakpoint Owns the fix Gets the 2 a.m. call 1. Data drift Data / ML engineer MLOps on-call 2. Eval gap ML engineer + PM MLOps on-call 3. Monitoring MLOps / platform MLOps on-call 4. Infra & cost Platform + FinOps Engineering lead 5. Governance Product owner + Legal Incident commander
Is this a big thing?
Yes, and the framing matters more than any single tactic. Quick recap:
The model is rarely what breaks first. Drift, eval gap, monitoring, infra cost, governance break first ⇒ chasing model accuracy fixes the wrong problem.
The breaks arrive in order. Drift in weeks, eval gap soon after, monitoring exposes both, infra and cost catch up, governance bills the whole story. Skip one and it visits you later, louder.
Budget 4–10× your model cost for the surrounding system. Retraining, evals, monitoring, on-call, compliance. If you can’t, you have a demo.
Run shadow mode for 4–8 weeks, then canary in stages. Cheapest insurance against breaks 1 and 2 you’ll ever buy.
Stand up the on-call and incident playbook before launch. The most under-discussed practice in AI program management, and how the McKinsey ~6% of “AI high performers” actually behave.
The POC is done when the surrounding system is in place, not when the model works. That sentence is the difference between a demo and a product.
Most AI projects don’t fail because the model was wrong. They fail because we shipped the model and forgot to ship the system around it.
✅ The 5-minute self-diagnosis
One honest yes/no per breakpoint:
Drift: are you logging every production input, timestamped, since day one?
Eval gap: did you run shadow mode before sending real traffic to the model?
Monitoring: would a dashboard warn you about quality decay before a customer does?
Infra & cost: have you load-tested at 10× and set per-feature cost budgets?
Governance: can you name the on-call and produce the model card today?
Four or more “no”s? You have a demo, not a product.
Your turn
Which of these five has already bitten you? Hit reply and tell me the one that cost you the most!
Sources
Research / books
Sculley, D. et al. (2015). Hidden Technical Debt in Machine Learning Systems. NeurIPS. https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems
Breck, E. et al. (2017). The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction. IEEE Big Data. https://research.google/pubs/the-ml-test-score-a-rubric-for-ml-production-readiness-and-technical-debt-reduction/
Huyen, C. (2022). Designing Machine Learning Systems. O’Reilly. https://www.oreilly.com/library/view/designing-machine-learning/9781098107956/
Shankar, S. et al. (2024). “We Have No Idea How Models Will Behave in Production Until Production.” ACM CHI. https://arxiv.org/abs/2209.09125
News / industry
RAND (Aug 2024). The Root Causes of Failure for Artificial Intelligence Projects. https://www.rand.org/pubs/research_reports/RRA2680-1.html
Gartner (Jul 2024). 30% of GenAI Projects Will Be Abandoned After POC by End of 2025. https://www.gartner.com/en/newsroom/press-releases/2024-07-29-gartner-predicts-30-percent-of-generative-ai-projects-will-be-abandoned-after-proof-of-concept-by-end-of-2025
McKinsey (Nov 2025). The State of AI in 2025. https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai
FinOps Foundation (2026). State of FinOps 2026.
https://data.finops.org/
CloudZero (2025). The State of AI Costs in 2025. https://www.cloudzero.com/state-of-ai-costs/
Gartner (Mar 2026). By 2030, Inference on a 1T-Parameter LLM Will Cost GenAI Providers Over 90% Less Than in 2025. https://www.gartner.com/en/newsroom/press-releases/2026-03-25-gartner-predicts-that-by-2030-performing-inference-on-an-llm-with-1-trillion-parameters-will-cost-genai-providers-over-90-percent-less-than-in-2025
Official / standards / cases
NIST (Jan 2023). AI Risk Management Framework 1.0. https://nvlpubs.nist.gov/nistpubs/ai/nist.ai.100-1.pdf
BC Civil Resolution Tribunal (Feb 2024). Moffatt v. Air Canada. https://www.canlii.org/en/commentary/doc/2025CanLIIDocs1963
Evidently AI. What is data drift in ML, and how to detect and handle it. https://www.evidentlyai.com/ml-in-production/data-drift
Eugene Yan. Product Evals in Three Simple Steps. https://eugeneyan.com/writing/product-evals/