Why AI projects stall after the demo
The demo impressed; months later nothing is in use. The gap between a demo and production, and the five questions nobody asked in the room.
The short answer
The demo was impressive. Everyone in the room saw the future. Eight months later, nothing is in use, and nobody is quite sure why. The reason is the gap between what a demo shows and what production needs. A demo shows the best case, on chosen data, operated by someone skilled, with the awkward cases left out. Production is the average case, on your actual data, integrated with your systems, running when nobody is watching, owned by someone with a budget. The demo answered “can it?” The questions that decide whether anything ships were never asked in the room: on our data, at our volume, connected to what, owned by whom, at what monthly cost, measured how.
Demo versus production
| In the demo | In production | |
|---|---|---|
| Data | Selected, cleaned, small | All of it, messy, growing |
| Cases | The ones that work | The average, including the awkward |
| Operator | An expert who knows the prompts | Staff with other jobs |
| Integration | None, or a mock | Your CRM, your email, your accounting system, with errors and retries |
| Failure | Edited out | Visible to customers |
| Cost | Not mentioned | A monthly line, growing with use |
| Owner | The presenter | Somebody in your business, or nobody |
| Measurement | Applause | Baseline, evaluation set, scorecard |
Five questions to ask in the demo room
- “Can you run it on ten cases we choose right now, including three awkward ones?” The answer shows how the system handles reality.
- “What does it get wrong, and how often, on data like ours?” A confident vendor has numbers; a salesperson has reassurance.
- “What has to be connected to what for this to run in our business, and who builds that?” Integration is where the budget goes.
- “Who in our company would own this, and what would it cost per month in year one?” If the room goes quiet, the project is a demo with a timeline.
- “How would we know in six months whether it worked?” Baseline, metrics, dates. If none are proposed, propose them.
Leaving with a plan
Outcome one: a scoped proof of concept on your data, with a date and the question it will answer. Outcome two: a documented reason not to proceed now, usually data readiness, integration cost or the absence of an owner, with the condition under which to revisit. Outcome three: a date by which a decision will be made, with the information needed to make it assigned to named people. Any of the three is progress. Applause is not.
What this means for you
Enjoy the demo, then ask the five questions before anyone leaves: your cases, the failure rate, the integration, the owner and the monthly cost, the six-month measure. Insist on one of three outcomes. The projects that survive contact with reality are the ones that were scoped in the demo room, not admired there.
Frequently asked questions
The demo was on our own data. Why did it still stall?
Probably on a selected slice of your data, prepared for the occasion, with the vendor operating it. Ask what happened with the awkward cases, the messy records and the volume. The stall usually comes from integration, ownership and running costs, none of which a demo touches, however real the data looked.
How do we tell a good vendor demo from a misleading one?
A good one shows failures as well as successes, runs on cases you chose that morning, states what the system cannot do, and ends with a discussion of integration, evaluation, ownership and monthly cost. A misleading one is flawless, on the vendor's cases, and ends with a price for a pilot. The difference is whether the vendor is selling a system or a feeling.
Is it wrong to be impressed by a demo?
Not at all; demos exist to show what is possible, and the technology is genuinely impressive. The error is treating the impression as evidence about your business. Be impressed, then ask the five questions, and let the answers rather than the impression decide the next step.