Choosing AI vendors and partners: the questions to ask
Every vendor demo works. The questions that reveal what happens with your data, what it costs at ten times the volume, and what you keep if you leave.
The short answer
Every AI vendor demo works, because it runs on the vendor’s data and the vendor’s questions. The decision depends on things the demo cannot show: what happens to your data, how the tool behaves on your material, what it costs when usage grows, and what you keep if you leave. Twelve questions, asked in writing, get to all of it. A vendor who answers them clearly is one you can work with; vague answers on data or exit end the conversation whatever the demo looked like.
The twelve questions
| Group | Question | A good answer |
|---|---|---|
| Data and privacy | 1. Where is our data processed and stored, and in which jurisdiction? | Specific regions; EU options if you need them |
| 2. Is our data used to train or improve your models? | No, by default and in the contract | |
| 3. Who at your company can see our data, and under what conditions? | Named roles, logged access, support only with permission | |
| Accuracy and control | 4. Will you run our fifty real examples and share the results? | Yes, before contract |
| 5. How does the system say “I do not know”, and how do we adjust it? | Explicit thresholds and configuration you control | |
| 6. What logs do we get, and can we review individual outputs? | Full logs, exportable | |
| Cost at scale | 7. What does this cost at ten times our current volume? | A number, with the pricing model explained |
| 8. What counts as a billable unit, and what drives it up? | Clear units; no surprises from long documents or retries | |
| 9. What happens when we hit a limit? | Graceful degradation, notification, no silent failure | |
| Exit and ownership | 10. If we leave, what do we take: data, configurations, prompts, workflows? | Everything we created, in usable formats |
| 11. Can we swap the underlying model or provider? | Yes, or a clear reason why not | |
| 12. What happens to our data after termination, and when? | Deleted within a stated period, confirmed in writing |
How to run the process
- Assemble the fifty examples first. Real documents, emails or questions, with the correct answer for each.
- Send the twelve questions to every candidate, in writing, before any demo.
- Run the examples with each candidate that answered acceptably. Compare results, not presentations.
- Score cost at your volume and at ten times, with the pricing model in writing.
- Read the exit answers as if you were leaving next year. Then decide.
The same questions for a partner
A partner building an AI system for you faces the same questions, slightly rephrased: which providers will you use and where does our data go; how will errors be caught and who reviews; what does it cost to run at ten times the volume; and what do we own when it is done, code, prompts, configurations and data. A partner who has clear answers has built these before.
What this means for you
Skip the demo-first process. Build the fifty examples, send the twelve questions, test on your own material, and read the cost and exit answers as commitments. It is a week of structure that prevents the two most common AI procurement outcomes: a tool that does not work on your data, and a tool you cannot afford to leave.
Frequently asked questions
Should we prefer a big vendor or a specialist?
Prefer the one that answers the twelve questions well. Big vendors often have clearer data terms and worse fit; specialists often fit better and have weaker terms. The questions expose both. Size is not a criterion; answers are.
How do we test accuracy before buying?
With your own material. Give the vendor fifty real examples with known correct answers, from your documents, your emails, your questions, and ask for the results. A vendor who declines is telling you what the trial would show.
What is the exit question?
If we stopped using this in a year, what would we keep? The data we put in, exported cleanly? The prompts and configurations we built? The workflows? A tool where the answer is 'nothing' is a tool that will raise its price.