Every product team has now built an LLM demo that made the room gasp. Far fewer have shipped one that survived real users, real data and a real SLA. The gap is not model quality — it is engineering discipline around the model.
This is the checklist we run before any AI feature reaches production in a client system.
Retrieval before generation
Nearly every useful business AI feature is retrieval wearing a generation hat. The quality ceiling is set by what you feed the context window, not by prompt cleverness. We spend roughly seventy percent of AI engineering time on ingestion, chunking, freshness and ranking — and thirty on the model call.
If retrieval is wrong, no prompt will save you. If retrieval is right, even a mid-tier model looks brilliant.
Evaluation is the feature
Before writing the feature, write the eval: a versioned dataset of real inputs with graded expected outputs, run on every change like a test suite. Teams that skip this ship regressions with every prompt tweak and never know it.
Our rule of thumb: no AI feature merges without an eval set of at least a hundred cases and a defined pass threshold.
Design for the failure, not the demo
The demo shows the happy path; production is the other paths. Every AI surface we ship has three designed states: confident answer with sources, hedged answer with an escape hatch to a human or search, and honest refusal. Users forgive "I am not sure about this one" — they do not forgive confident nonsense.
Add cost ceilings, latency budgets with streaming, and a kill switch that degrades the feature to its non-AI fallback. Boring, unglamorous — and the difference between a feature and an incident.