The demo works. Everyone in the room is impressed. Nine months later the project is quietly shelved. This happens often enough to be a pattern rather than bad luck, and the causes are consistent enough to be worth writing down.
A demo optimises for the wrong thing
A demo is built to be watched once, by people who want it to work, on inputs the builder chose. A production system is used repeatedly, by people who want to get on with their day, on inputs nobody anticipated. These are different engineering problems, and the first one teaches you almost nothing about the second.
The specific trap is that language models are unusually good at the demo case. They produce fluent, confident, plausible output on the happy path, which makes a prototype feel far closer to done than it is. Traditional software fails visibly when it is incomplete; an AI prototype fails invisibly, and the failure only surfaces at volume.
Nobody defined what correct means
Ask a team midway through a stalled AI project how accurate the system is, and the answer is usually a story about a good response someone saw last week. That is not a measurement, and without a measurement there is no way to tell whether a change helped.
The fix is unglamorous and it is the single highest-return thing on this list: build a test set from real cases before you build the feature. A hundred historical tickets, real documents, real questions, with the answer a competent human would give. Run it on every change. Track accuracy, refusal rate, and citation coverage as three separate numbers, because they trade against each other and a single score hides the trade.
Teams resist this because assembling the set takes a week that feels like it is not shipping. That week is what makes every subsequent week measurable.
Retrieval failures wearing a model's voice
When a grounded system gives a wrong answer, the instinct is to change the prompt. Most of the time the prompt was fine and the retrieval step simply did not surface the right passage — the model wrote a confident answer because that is what it does when handed insufficient context.
Tuning a prompt to compensate produces a system that appears fixed and breaks again as soon as the corpus changes, because the underlying gap was never addressed. Evaluate retrieval on its own, with its own test set: given this question, did the right passage come back, and where did it rank? Fix that number first, and a surprising share of the generation problems disappear without touching the prompt.
The system has no way to say no
A system that answers everything will confidently answer the things it has no basis for, and users find out. The trust cost of one fabricated answer is far higher than the utility of the fifty correct ones around it, because after the first fabrication every subsequent answer has to be checked — at which point the system has negative value.
Production systems need a threshold and a refusal path. When retrieval scores below it, say so and offer a handoff. Refusal rate then becomes a number you tune deliberately rather than a behaviour you discover, and users learn that an answer means something.
Cost and latency arrive as a surprise
Pilots run on the largest available model because it is the path of least resistance and the volume is negligible. The first real month of traffic then produces an invoice nobody budgeted for, or a p95 latency that makes the feature unpleasant to use — and the project gets re-scoped under pressure rather than by design.
Write cost per request and p95 latency down as requirements at the start, next to the functional ones. Model choice then follows the budget instead of the budget discovering the model choice later. Most mature systems route between a small fast model for the routine path and a larger one for genuinely hard cases, and the routing is measured rather than assumed.
Nobody decided what happens when it is wrong
Models are sometimes wrong, often slow, and occasionally unavailable. A pilot ignores all three; production cannot. Every path needs a defined fallback so the product degrades rather than breaks, and anything irreversible — money, records, messages to your customers — needs a person confirming it.
This is usually framed as a safety requirement. It is also a product requirement: a drafted action a human approves is shippable in a quarter of the time of an autonomous one, because the review step absorbs the residual error rate that would otherwise have to be engineered away.
What actually gets a system across
None of this is exotic. A test set from real cases, retrieval evaluated separately from generation, a threshold below which the system refuses, cost and latency written down as requirements, human approval on anything irreversible, and a defined fallback on every path.
Teams that do these six things ship AI features that are still running a year later. Teams that skip them ship a demo, and then spend three quarters discovering each item on this list one incident at a time.