The thesis in one line: the choice between open-source AI and a closed API almost never plays out where people think. It isn’t a battle of principles between «freedom» and «lock-in». It’s an engineering decision with two clocks —cost at scale and data sensitivity— and for the vast majority of companies the right answer on day one isn’t the ideological one. This article is about when it truly matters and when, honestly, it doesn’t.
Open-source AI vs closed API in your company: most people ask the question backwards
The conversation almost always arrives poisoned. Someone on the team read that Llama, Mistral or DeepSeek are open models, that you can download them and run them «at home», and suddenly the question stops being «what does my business need?» and becomes «are we the kind who depend on OpenAI, or the kind who self-host?». That isn’t a technical decision: it’s identity. And identity decisions get defended with pride, not data.
The useful question is far more boring: for the specific problem you want to solve, what does each option give you and what does each one cost —in money, in your team’s time, in risk? Framed that way, the flag falls on its own and what actually decides shows up: the volume at which you’ll call the model and how sensitive the data you’ll feed it is.
What each option really gives you (minus the flag)
Closed API: speed and zero maintenance
A closed API —OpenAI, Anthropic, Google— is a network call. You don’t provision GPUs, patch drivers or worry about the model falling over at 3am. You pay per token, start in minutes, and the provider improves the model underneath without you touching anything. In exchange, the data leaves your perimeter for theirs (contract in place), someone else sets the unit cost, and you’re tied to their roadmap and their pricing policy. You buy speed and peace of mind; you rent control.
Open source: control and cost at scale, in exchange for plumbing
An open model you run yourself —in your cloud or on your hardware— gives you the opposite: the data doesn’t move from where you say, the cost per million tokens at high volume can crater, and nobody deprecates the model on you by surprise. The price of that is that the infrastructure is now yours: serving the model, scaling it, monitoring it, updating it and answering when it breaks is your team’s job, forever. It isn’t «cheaper»: you’re swapping a provider’s invoice for an engineering payroll. For those with volume and staff, it pays off; for those without, it’s a pit.
| Axis | Closed API | Open source (self-hosted) |
|---|---|---|
| Time to start | Minutes | Weeks of setup |
| Where the data lives | At the provider (under contract) | Wherever you decide |
| Cost | Per token, provider sets it | Fixed (infra + team), cheap at scale |
| Maintenance | Zero, improves itself | Yours, forever |
| Control / lock-in | Low control, high lock-in | High control, no lock-in |
| Who it pays off for | Almost everyone at the start | High volume or sensitive data |
Why for 90% of SMEs the answer is to start with a closed API
It isn’t laziness or timid pragmatism: it’s arithmetic. Most SME AI projects die not from picking the wrong model but from never reaching production. And anything that lengthens the road to production raises the odds of death. Self-hosting an open model adds, up front, a whole infrastructure project before you’ve even validated the idea works. It’s optimizing the cost per token of something you don’t yet know you’ll use.
- Your volume is low at the start. Open source’s per-token saving only pays off past a lot of traffic. With a handful of calls a day, your own infrastructure costs more than the API, not less.
- You don’t have —or want— an MLOps team. Serving a model in production with high availability is a craft. If you don’t have it in house, what you save on tokens you pay in hires or in outages.
- Today’s open model isn’t your edge. Your differentiator isn’t which weights you run, it’s your process and your data. Starting on an API lets you focus on that instead of on GPU drivers.
- Migrating later is cheap; getting it wrong at the start is expensive. If you design the app decoupled from the provider, moving from a closed API to your own model later is a config change, not a rewrite.
The sane play is to start with a closed API, validate the case works in your operation, and leave the door open to drop down to open source the day the numbers ask for it. If the project is serious and will grow, that architecture —decoupled, with model routing and the data under your control— is exactly what we build in enterprise AI infrastructure: not to marry you to a provider, but so you can swap it painlessly.
When dropping to open source does make sense
The rule isn’t «never open source». It’s «open source when a real trigger justifies it», not when pride demands it. These are the triggers that actually tip the balance:
- The data can’t leave your perimeter. Healthcare, banking, legal, trade secrets: if by contract or regulation the data can’t touch a third-party server, self-hosting stops being a preference and becomes a requirement. Before you assume that’s your case, sort out what can and can’t leave with a usage policy for ChatGPT and external models in your company; often the sensitive data is a small fraction and the rest goes fine via API.
- Volume is large and sustained. When you call the model millions of times a month steadily, the API’s per-token cost starts to hurt and your own model amortizes. Here the arithmetic does favor open source —provided you already have someone to operate it—.
- You need to fine-tune the model with your knowledge. If your case demands a model specialized in your domain and your data, controlling the weights helps. Careful though: before fine-tuning, the real fix is almost always retrieval over your information, which we break down in training the agent on your own information.
- Latency or offline. If you need local response, no network, or guaranteed minimum latency, the model has to live near where it’s used. No closed API will do there.
The cost nobody tells you: maintenance isn’t free
The classic mistake when comparing is looking only at price per token and concluding open source is «free». It isn’t. A self-hosted model has a cost that shows up on no tariff: someone has to serve it with high availability, watch that it doesn’t degrade, update it when a better version ships, and be on call when it falls over. That cost is a payroll, not an invoice line, and it’s recurring. The closed API hides that cost inside the per-token price; open source hands it all to your team.
That’s why the honest comparison isn’t «expensive API vs cheap open source». It’s «variable cost you don’t manage vs fixed cost you do manage». For low or medium volume, the variable wins almost every time. For high volume with a dedicated team, the fixed one does. The same underlying logic that separates buy from build for any AI piece, which we already argued in buy or build AI agents.
What to do on Monday
- Start with a closed API and validate the case works in your real operation, not in a demo. Optimizing cost per token before that is optimizing something that doesn’t exist.
- Decouple from day one. Design the app so the model provider is config, not wiring. Then migrating later costs an afternoon, not a quarter.
- Mark your triggers. Write down, in black and white, what data can’t leave and at what volume the invoice would start to hurt. The day one is met, you have your signal to drop to open source —and only then—.