How many queries does it take for self-hosting to beat an API?
You’re evaluating whether to run a 7B-parameter open model on a single A100 GPU (£3,000 upfront) or pay for an API at £0.0001 per 1,000 tokens. At 100 tokens per query, the math is straightforward: self-hosting costs £3,000 ÷ 50,000 queries = £0.06 per 1,000 tokens. That’s 60x cheaper than the API’s £0.0001 rate—but only if you hit 50,000 queries/month.
The catch? That £3,000 server must run 24/7. Electricity for a single A100 adds £150/month, and you’ll need £500/month for cloud hosting if you’re not on-premise. At 10,000 queries/month, the API costs £10, while self-hosting burns £700. You break even at 49,000 queries/month, but latency becomes the real killer: unoptimized batching turns 0.8-second API responses into 1.2-second spikes during peak loads. Poor GPU utilization (e.g., 30% idle time) wastes £200/month in wasted compute.
What goes wrong: If your queries aren’t perfectly aligned with GPU capacity, you’ll either over-provision (buying 4 GPUs for 1’s workload) or throttle users during spikes. APIs handle this automatically.
Why does data residency make self-hosting a legal requirement?
UK GDPR Article 44 forces you to self-host if your LLM processes EU citizen data. A US-hosted API (even with EU data centers) triggers Schrems II risks unless you’ve obtained explicit consent or a derogation. For example, a healthcare chatbot answering patient queries cannot use an API unless every user signs off on data transfers to the US—something most won’t do.
Self-hosting in the UK adds £500/month for compliant cloud (e.g., AWS London region), but the legal risk is far higher. APIs like Together.ai offer EU endpoints, but their data residency guarantees are vague—you’re trusting their compliance team, not your own. A single audit failure could invalidate your entire project under GDPR’s liability clauses.
What goes wrong: If your API provider’s data centers are in a high-risk jurisdiction (e.g., US, China), you’re exposed to data access requests from foreign governments. Self-hosting in the UK avoids this, but only if you’ve documented the model’s data flow in your DPIA.
What’s the real cost of fine-tuning your own model?
Fine-tuning a 7B-parameter model on a single A100 costs £0.50/hour in electricity plus £200/month for cloud GPU time. That’s cheaper than an API’s £500 for 100k tokens—but the trade-off is accuracy. Quantization (e.g., 4-bit vs. 16-bit) cuts costs by 70%, but degrades performance by 5% on niche domains (e.g., legal contracts).
Self-hosted fine-tuning takes 48 hours vs. an API’s 2-hour turnaround, but APIs limit customization. You can’t upload private datasets to most APIs, forcing you to pre-process data locally—a step that adds another 12 hours of work. Overfitting is the biggest risk: if your training set is under 50k samples, the model will fail on edge cases (e.g., rare medical conditions in a healthcare bot).
What goes wrong: You spend £1,200 fine-tuning a model that only works for 80% of your use cases, then have to retrain when new data arrives.
When does an API’s ‘pay-as-you-go’ model actually cost more?
APIs advertise £0.0001 per 1,000 tokens, but hidden costs push the real price higher. Throttling fees (e.g., +20% overage charges) or per-endpoint pricing (e.g., £0.001 for embeddings vs. £0.00005 for completions) add up. A chatbot with 500 daily users costs £30/month on an API but £15/month self-hosted (assuming 50 queries/user/day).
The uptime trade-off is stark: APIs offer 99.9% SLA-backed reliability, while self-hosting guarantees 99%—and that 0.9% downtime costs you revenue. APIs also handle deprecations: if an endpoint is removed (e.g., a free tier), you’re forced to migrate mid-project, adding £2,000 in rework.
What goes wrong: You build a feature around an API’s free tier, then get hit with a £500/month bill when they sunset it.
How does self-hosting break when your traffic spikes?
A single GPU server maxes out at 100 concurrent queries. Scale to 3 GPUs for £1,200 in hardware, but now you need Kubernetes to manage load balancing. Vertical scaling (bigger GPUs) helps, but horizontal scaling requires orchestration tools like Ray or vLLM—adding £3,000/month in DevOps overhead.
During peak loads, GPU memory fragmentation causes OOM errors, crashing the inference service. APIs auto-scale to 10,000 concurrent users without manual intervention. Self-hosting forces you to either:
- Over-provision (buying 4 GPUs for 1’s workload), or
- Implement a queue system (adding latency).
What goes wrong: You spend £8,000 on 4 GPUs to handle 1,000 users, then realize you could’ve used an API for £200/month.
What’s the maintenance tax you’ll pay for self-hosted models?
Self-hosting isn’t just hardware—it’s 2 hours/week for monitoring, updates, and security patches. PyTorch vulnerabilities alone require monthly updates, and forgetting to patch leaves you exposed to prompt injection attacks. Downgrading from a bleeding-edge model (e.g., Llama 3) to a stable version (e.g., Mistral 7B) avoids breaking changes but limits capabilities.
The real cost? A dedicated DevOps role (~£60k/year) to manage the stack. APIs handle updates automatically, but self-hosting forces you to choose between:
- Bleeding-edge models (higher risk of crashes), or
- Stable models (lagging features).
What goes wrong: You skip a model update, and a new exploit lets users bypass your safety filters—costing you £5,000 in cleanup.
Can you really save money by self-hosting ‘free’ open models?
‘Free’ models like Llama 2 come with restrictive licenses. Meta’s terms ban commercial use without agreement, and reselling inference-as-a-service triggers legal action. The hard cost? £1,500 for a commercial license vs. £0 for ‘free’ models—but the latter may not cover your use case.
Support is another issue: ‘free’ models lack bug fixes. If the open-source project stops updates, your model becomes a liability. APIs offer SLAs and documentation, but self-hosting leaves you with no recourse if the community abandons the project.
What goes wrong: You build a product on an unlicensed model, then get a cease-and-desist—costing you £10,000 in legal fees.
What’s the fastest way to test if self-hosting is worth it?
Run a 30-day pilot with 10k queries on both self-hosted (Ollama + local GPU) and API (Together.ai). Self-hosting costs £50 in electricity vs. the API’s £10, but the latter includes support for setup issues. Compare:
- Latency: Self-hosted (1.5s) vs. API (0.9s)
- Accuracy: Self-hosted drops 3% on edge cases
- Uptime: Self-hosted fails during spikes; API doesn’t.
What goes wrong: You skip the pilot and over-provision hardware (e.g., buying 4 GPUs when 1 suffices), wasting £6,000.
Next step
If you’re still unsure, start with a £10 API trial to validate your query volume. For self-hosting, use our AI Development team to scope the hardware and compliance costs—we’ll give you exact figures for your workload.
Frequently asked
Yes, but it adds complexity. Use an API for public-facing queries (e.g., customer support) and self-host for internal tools (e.g., document processing). The trade-off is managing two different latency profiles—APIs will always be faster for edge cases. If you need consistency, stick to one approach.
A single A100 GPU (£3,000) with Ollama or vLLM is the lowest-cost option. For internal use, you can skip commercial licenses on ‘free’ models like Mistral 7B, but ensure your data never leaves your network. Electricity and cloud hosting will add ~£200/month—cheaper than an API only if you hit 50k queries/month.
Check the model’s license agreement (e.g., Meta’s Llama 2 terms). Commercial use without agreement is banned, and reselling inference-as-a-service violates most open licenses. For customer-facing queries, either buy a commercial license (£1,500+) or use an API with clear terms. Self-hosting ‘free’ models for customers is a legal risk.
Few APIs guarantee UK/EU-only data residency. Together.ai and Mistral offer EU endpoints, but their compliance documentation is vague. For true UK residency, self-hosting is the only option—APIs cannot match the legal certainty of on-premise or UK-cloud deployment.
Your model becomes unsupported. Security patches stop, and new vulnerabilities emerge. If the project dies, you’re left maintaining a fork—adding £10k/year in DevOps costs. APIs avoid this risk by handling updates centrally, but self-hosting gives you control over the model’s lifespan.
- ai-development
- cost analysis
- data-residency
- gpu-computing
- llm-integration
- scalability