Model choice as a strategic decision
Why which AI model you use matters less than how you structure around it, and the few cases where the choice genuinely is strategic.
The short answer
Which model a business uses feels like the strategic decision and usually is not. Models improve and change every few months, prices fall, and the leader on any given task rotates between providers. A strategy built around one specific model therefore has a short life. What is strategic is everything around the model: where data may be processed and under what contractual terms, how the system is evaluated so you know whether a change helps, whether your prompts, rules and evaluation sets live in your own code, and whether you could switch providers without rebuilding. Get those right and model choice becomes a routine decision you revisit quarterly. There are three cases where the choice itself is genuinely strategic: strict data residency or sovereignty requirements that only some providers meet, very high volumes where cost per task dominates the business case, and a capability only one provider offers that your product depends on. Outside those, evaluate on your own tasks, choose the cheapest model that is good enough for each, and keep the ability to change.
What is strategic and what is not
| Decision | Strategic? | Why |
|---|---|---|
| Which provider’s model to call today | Rarely | Changes every few months; switchable if designed for |
| Whether data may leave the EU | Yes | Legal and contractual; narrows the provider list |
| Contractual terms on training and retention | Yes | Determines what you may use AI for at all |
| Whether prompts, rules and evaluation sets are yours | Yes | Determines whether switching is possible |
| Having an evaluation set for your tasks | Yes | Without it you cannot judge any model or any change |
| Cost per task at your volume | Sometimes | Strategic at high volume, noise at low volume |
| A capability only one provider has | Sometimes | Strategic if your product depends on it; risky either way |
| Using a small model where sufficient | Operational, but valuable | Often the largest cost lever |
| Self-hosting an open model | Sometimes | Strategic under residency or volume pressure |
How to make the decision routinely
- Build an evaluation set of real examples with known good answers for each task the business uses AI for.
- Define the constraints: residency, terms, latency, budget.
- Score candidate models on your set for quality, cost and speed.
- Choose the cheapest sufficient model per task, not the best overall.
- Abstract the model behind one interface; keep prompts and rules in your repository.
- Re-run the evaluation when a new model appears or prices change, quarterly at least.
- Switch when the numbers justify it, and keep the previous configuration for a quick revert.
The cost lever people miss
A production AI system does not need the largest model for ordinary requests. Classification, routing, extraction, tagging and routine drafting are handled well by small, fast, cheap models, with the large model reserved for the hard cases or the final polish. Businesses that measure this and route accordingly often cut their AI costs substantially with no quality change that anyone notices. That measurement requires the evaluation set, which is another reason it is the first thing to build.
What this means for you
Treat model choice as a routine, revisitable decision and put the strategy into what surrounds it: data terms and residency, your own prompts and rules, an evaluation set built on your real tasks, and an abstraction that makes switching cheap. Choose the cheapest sufficient model per task, re-evaluate quarterly, and reserve genuinely strategic weight for residency requirements, high-volume cost pressure and capabilities your product truly depends on.
Frequently asked questions
Should we standardise on one AI provider?
Operationally yes, architecturally no. Using one provider keeps contracts, billing and staff training simple. But the system should be written so the model is behind an interface, with your prompts, rules, evaluation sets and data flows in your own code, so switching is a configuration change and a re-evaluation rather than a rebuild. That combination gives you simplicity now and options later.
How do we choose a model?
Build an evaluation set of thirty to a hundred real examples from your actual tasks with known good answers, and run candidate models against it, scoring quality, cost and speed. That takes a day and tells you more than any benchmark table, because your tasks are not the benchmark's tasks. Re-run it when a new model appears, which will be every few months.
Does a bigger model always give better results?
No. For classification, extraction, routing and routine drafting, smaller and cheaper models often match larger ones at a fraction of the cost and latency, and many production systems use a small model for ordinary requests and a large one for the hard cases. The right question is which model is sufficient for each task, not which is best overall.