The AI hype cycle: how to read vendor claims
Every AI product is sold in the same superlatives. How to turn claims into checkable statements, and the evidence a serious vendor can show.
The short answer
Every AI product is described in the same words: it understands, it learns, it is human-level, it is accurate, it plugs in. The technology behind many of them is genuinely capable and improving fast, which is what makes the words plausible and why they must be checked rather than believed or dismissed. The method is translation: turn each claim into a statement you can check on your own cases. On what data was this measured, how, compared with what, and at what cost? Products that survive translation are worth evaluating. Products that dissolve into adjectives are not. A serious vendor does the translation for you, unprompted.
Claims and their translations
| The claim | The question | What a good answer contains |
|---|---|---|
| ”It understands your documents” | Understands how? Retrieval with citations, or generation from memory? | Retrieval from your index, grounded answers, a way to say “not covered" |
| "It learns from your data” | Learns how? Fine-tuning, retrieval, or a phrase? What happens when a document changes? | Usually retrieval; changes reflected immediately; no retraining |
| ”Human-level accuracy” | On which task, measured against which humans, on whose data? | An evaluation on cases like yours with a method |
| ”No hallucinations” | How is that achieved and verified? | Grounding, output validation, a measured confident-wrong rate; never “none" |
| "Plug and play” | Plugged into what? Who builds the integration with our systems? | A named list of integrations and what each costs to set up |
| ”95 percent accurate” | Of how many cases, which kinds, labelled by whom, and what about the other five? | A denominator, a method, the failure types |
| ”Enterprise-grade security” | Which certifications, which data-processing terms, which region, which retention? | Documents, not adjectives |
The evidence a serious vendor can show
- An evaluation on cases like yours, with the method and the failure rate, and willingness to run on your cases.
- A list of what the product cannot do, offered before you ask.
- Integration described concretely: which systems, what effort, who does it.
- Ownership and data terms in writing: your data, your accounts, no training on your inputs, region, retention.
- Monthly cost at your volume, not a starting price.
- References you can call in businesses like yours, including one where it did not go smoothly.
A procurement habit
Before any AI purchase, write the vendor’s five biggest claims in one column and your translated questions in the next. Send the questions. Score the answers on whether they contain evidence, method and specifics. Then run twenty of your own cases through the product. The whole exercise takes a day and it is the difference between buying a capability and buying a description of one.
What this means for you
Treat AI claims as translation exercises: on what data, measured how, compared with what, at what cost. Ask for denominators, methods, integration specifics, data terms and monthly costs. Run your own cases. The capabilities are real, and the vendors who can show them on your data are the ones to buy from. The rest are selling adjectives, and adjectives do not ship.
Frequently asked questions
Is it all hype, then?
No. The capabilities are real and the pace of improvement is real, which is what makes the marketing plausible. The task is not to dismiss claims but to convert them into things you can check on your own cases. Products that survive that conversion are worth buying; the ones that dissolve into adjectives are not.
What does a fair accuracy claim look like?
A number with a denominator and a method: measured on this many cases of this kind, labelled by whom, with this definition of correct, including the rate of confident wrong answers. A number alone, however high, tells you nothing about your data. Ask for the method; a vendor with a real evaluation will be pleased you asked.
How do we compare two vendors making similar claims?
Give both the same twenty cases from your own work, including awkward ones, and score the results yourselves. Then compare integration effort, ownership terms, data handling and monthly cost at your volume. The vendor comparison that matters takes an afternoon and ignores the brochures entirely.