A self-hosted LLM is an open-weight model running on infrastructure you control, on-premises or in your own cloud account, so prompts and data stay inside your boundary. Running one in production takes five commitments: a model chosen by evals on your own tasks and cleared for its license, GPUs sized for weights plus KV cache, a production inference engine such as vLLM, a serving layer with a gateway and observability, and security controls you now own. Each of them needs people to run it. A constraint such as data residency, an air-gap mandate, or sustained volume decides whether self-hosting is justified. Cost per token depends on GPU utilization, so an underused deployment can cost more per token than an API.
What Is a Self-Hosted LLM?
A self-hosted large language model runs on GPUs you control, using open weights you downloaded and inference software your own team operates. Those GPUs can sit on-premises, in a private cloud, or inside your own cloud account, and the arrangement is the same in all three. Private LLM and on-prem LLM describe variants of it, and buyers use the terms interchangeably.
Open-weight means downloadable model weights under a license, which is narrower than open code or published model training data. Calling them open source models without that qualification overstates the grant. The open source vs. open weight distinction decides what your legal team will sign.

“Self-hosted” gets applied to five arrangements, and only the first two are self-hosting in the strict sense. What separates them is where the model weights live, where data gets processed, and who holds the keys to the machine doing the processing.
Option | Where the Model Runs | Who Operates It | Data Leaves Your Boundary? |
|---|---|---|---|
On-prem LLM | Your data center or colocation | You | No |
Self-hosted in your cloud account (VPC) | Your AWS, GCP, Azure, or EU-cloud account | You | No, it stays in your account |
Managed open-weight inference | Provider’s GPUs running the same open weights | Provider | Yes |
Frontier API | Provider’s GPUs, proprietary model | Provider | Yes |
Hybrid via a gateway | Mix of the above, routed per request | You (routing) + provider | Only for routes you allow |
Option
On-prem LLM
Self-hosted in your cloud account (VPC)
Managed open-weight inference
Frontier API
Hybrid via a gateway
Where the Model Runs
Your data center or colocation
Your AWS, GCP, Azure, or EU-cloud account
Provider’s GPUs running the same open weights
Provider’s GPUs, proprietary model
Mix of the above, routed per request
Who Operates It
You
You
Provider
Provider
You (routing) + provider
Data Leaves Your Boundary?
No
No, it stays in your account
Yes
Yes
Only for routes you allow
Managed open-weight inference sits in the middle of that table: identical weights on somebody else’s GPUs, under a retention contract. It stays the natural fallback for any route that does not need the full boundary.
Why Do Companies Self-Host an LLM in 2026?
Most companies self-host an LLM because a constraint requires it: data residency, a network-boundary or air-gap mandate, sustained high volume, or control over model versions. Any one of them can be enough on its own. The assumption that self-hosting is cheaper by default is the one reason that rarely survives contact with a real workload.

It helps to sort the reasons into obligations and optimizations. An obligation decides whether you self-host at all, and it usually comes from legal or security before engineering has formed a view. An optimization only decides how much of your traffic runs that way.
Three obligations drive most self-hosted deployments in regulated industries. When one of them applies to your workload, the decision has already been made for you:
- Data residency and sovereignty, since a provider that receives prompts containing personal data acts as a processor under GDPR, and serving them outside the EEA triggers its transfer rules.
- Network-boundary or air-gapped environments in healthcare, defense, and critical infrastructure.
- Contractual no-retention or no-training requirements that a customer wrote into your agreement.
Optimizations behave differently because each trades effort for money or latency. Four come up in most build-or-buy discussions, and any of them can be bought back with an API if the effort stops paying.
- Cost at sustained volume.
- Latency and independence from provider rate limits.
- Version pinning, so a silent upgrade cannot change output quality overnight.
- Tuning a narrow model that beats a general one on your own task.
Vercel’s guide to self-hosting draws the same line: a mandate makes self-hosting an obligation, and cost only makes it an optimization. Whoever owns the budget can overrule an optimization.
Sustained volume pays for the optimization, because a GPU only becomes cheap once it stops sitting idle. Workloads with steady daily volume fit that shape: the trend-detection pipelines SPD Technology built for a global investment data platform, in a partnership now past ten years, process roughly 50,000 articles a day from more than 3,000 data sources.
Open-weight models are close to the best closed models without having caught them, and the distance moves in both directions. Stanford HAI’s 2026 AI Index put the top closed model 3.3% ahead of the top open model on the Arena Leaderboard in March 2026, up from 0.5% in August 2024. A few points on a public leaderboard matter only if they show up in your own tasks, so the properties of LLMs your workload actually depends on determine what that gap costs you.
What Does It Really Take to Self-Host an LLM?
Running a self-hosted LLM takes five commitments. Two are decided once, at procurement. You need a model validated by evals on your own tasks, and hardware sized for the memory the model actually consumes, which means VRAM, the video random access memory on the GPU card itself. That memory holds the weights plus the KV cache (key-value cache), where the server keeps attention state for every in-flight request.
The other three are operational, keep costing every month, and teams underestimate them. A production inference engine. A serving layer with a gateway and observability. Every security control that used to arrive bundled with somebody else’s API. All of it becomes your team’s property the day you download the weights.

LLM hosting turns into a platform job long before it becomes a model job. The people who keep it running come from MLOps consulting and development or an internal platform and machine learning team, and one of them answers the page at 2 AM.
-
A model chosen by evals on your own tasks and cleared for its license.
-
GPUs sized for the weights plus the KV cache at target concurrency.
-
A production engine benchmarked at real load, with a single-prompt test proving nothing.
-
A gateway, autoscaling, monitoring, and regression evals around the model.
-
Security, logging, and guardrails owned in-house from the first day of traffic.
Order matters as much as the list does. Model choice sets the hardware and the hardware narrows the engine, and self-hosting platforms package parts of that chain without moving the commitments off your team.
How Do You Choose an Open-Weight Model to Self-Host?
Match the model to the task first and to the hardware you can buy second. Model size and quantization drive cost far more than the engine does, so a smaller model that clears your quality bar stays cheaper every month after.
Start with the license. Some restrict use by region, others by user count or by the revenue of the company using them, and either can disqualify a model after a team has already benchmarked it. The split between Western vs. Chinese open-weight models raises procurement questions a regulated buyer is better off clearing in week one.
Evals beat leaderboards for a decision this specific. Define 50 to 200 representative tasks with a rubric scored on your own data, and test at the quantization you plan to ship. For agent workloads, the deciding score is tool calling, and the method matches how to evaluate AI agents before production anywhere else in your stack. Output evaluation judges the final answer, while trajectory evaluation judges the steps behind it.
Quantization stores the weights at lower numeric precision. Q4-class means roughly four bits per parameter, down from the sixteen a model usually ships with, and for general text work the quality loss is minimal. Structured output and long reasoning are the first things to wobble at lower bit widths, so put both in the eval set before anyone signs for a card. A narrow task often needs a smaller model than a leaderboard suggests, and fine-tuning one on domain data is the work behind generative AI development services.
What GPU Do You Need for an LLM?
Sizing a GPU for LLM work starts with model size, which sets the VRAM floor, and ends with memory bandwidth, which sets the speed once the model fits. Weights take the parameter count multiplied by bytes per parameter, the KV cache grows with the context window multiplied by concurrent requests, and 10 to 20% headroom covers fragmentation and activation memory. The arithmetic here covers text models; image generation workloads size differently.

The figures below follow Qovery’s rule of thumb and cover weights alone, so add the cache and the headroom before matching a model to a card.
Model Size | FP16 Weights | 8-bit | 4-bit | Typical GPU Class (4-bit) |
|---|---|---|---|---|
~8B | ~16 GB | ~8 GB | ~5–6 GB | 24 GB card (L4, RTX 4090) |
~32B | ~64 GB | ~32 GB | ~20 GB | 48 GB card (L40S) |
~70B | ~140 GB | ~70 GB | ~40–42 GB | One 80 GB GPU (A100/H100) |
Large MoE (100B+ total) | Set by total parameters | — | — | One or more 80 GB GPUs; speed set by active parameters |
Model Size
~8B
~32B
~70B
Large MoE (100B+ total)
FP16 Weights
~16 GB
~64 GB
~140 GB
Set by total parameters
8-bit
~8 GB
~32 GB
~70 GB
—
4-bit
~5–6 GB
~20 GB
~40–42 GB
—
Typical GPU Class (4-bit)
24 GB card (L4, RTX 4090)
48 GB card (L40S)
One 80 GB GPU (A100/H100)
One or more 80 GB GPUs; speed set by active parameters
Two things catch teams out here. The first is the KV cache, which at long context lengths and high concurrency can outgrow the weights and turn a comfortable fit into an out-of-memory error the moment real traffic arrives. The second is mixture-of-experts (MoE). MoE models route each token through a subset of expert networks, yet every expert stays resident in memory, so you buy VRAM for the total parameter count and get the speed of the active ones.
Rented capacity changes the bill’s shape without changing the operational load. Cloud GPUs move the initial investment to an hourly line and make resizing easy, while owned hardware in an on-premises data center amortizes over years and fixes your memory ceiling. Owned racks also pull power distribution and cooling into the conversation, which is a facilities decision before it is an engineering one. For dense models, offloading to system RAM costs too much generation speed to plan around.
vLLM vs. Ollama: Which Inference Engine Fits?
Ollama and llama.cpp are designed for development, local testing, and individual users. vLLM, SGLang, TGI, and TensorRT-LLM are built to handle concurrent production traffic. Put an engine tuned for single-user LLM inference behind ten concurrent users, and it queues, with utilization flat and latency climbing.
The difference is scheduling. Continuous batching adds and removes requests at every generation step, so a short request no longer waits for the longest one in its batch. A paged KV cache allocates attention memory in small blocks, so more requests fit on the same GPU. The PagedAttention paper presented at SOSP 2023 reported 2 to 4 times higher serving throughput than the serving systems of that time at the same latency.
Ollama has earned its place because one command puts a local model on a workstation, and LM Studio does much the same behind a graphical interface. Both leave a single developer with a local server to work against. llama.cpp goes further down the ladder, keeping local inference running without a data center GPU at all. Most of the decision below sits in the note column.
Engine | Best For | Concurrency | Note |
|---|---|---|---|
Ollama | Local development, prototypes | Low | Fastest start; never expose it unauthenticated |
llama.cpp | CPU, Apple Silicon, edge devices | Low | Runs without a data center GPU |
vLLM | General production serving | High (continuous batching, paged KV cache) | Broad model support; OpenAI-compatible |
SGLang | Shared-prefix workloads: RAG, multi-turn, agents | High | Benchmark against vLLM on your workload |
TGI | Hugging Face-centered stacks | High | Mature tooling |
TensorRT-LLM | Peak NVIDIA throughput | High | Benchmark against vLLM on your workload |
Engine
Ollama
llama.cpp
vLLM
SGLang
TGI
TensorRT-LLM
Best For
Local development, prototypes
CPU, Apple Silicon, edge devices
General production serving
Shared-prefix workloads: RAG, multi-turn, agents
Hugging Face-centered stacks
Peak NVIDIA throughput
Concurrency
Low
Low
High (continuous batching, paged KV cache)
High
High
High
Note
Fastest start; never expose it unauthenticated
Runs without a data center GPU
Broad model support; OpenAI-compatible
Benchmark against vLLM on your workload
Mature tooling
Benchmark against vLLM on your workload
Engine choice quietly fixes your quantization format as well. GGUF files run on llama.cpp and Ollama. vLLM is built around AWQ, GPTQ, and FP8, with only experimental GGUF support, and moving a model between the two camps usually means requantizing it before it will load.
What Does Serving an LLM in Production Involve?
Serving turns a working model into a service with uptime. At this stage, LLM deployment looks like any other stateful workload on Kubernetes, with a GPU-specific wrinkle in every layer. Each item below covers a failure that shows up in production rather than in a pilot.
- A GPU node pool with the device plugin installed and tested.
- Weights cached in object storage so a cold start never pulls from the public internet.
- Readiness probes tuned for slow model load, since large weights take minutes to become servable.
- Autoscaling keyed to queue depth, because CPU utilization says nothing useful about a saturated GPU.
- An OpenAI-compatible gateway holding keys, budgets, and SSO.
- Observability covering tokens per second, time to first token, queue depth, and GPU utilization.
- A regression eval suite run before every model, quantization, or engine change.
- Scheduled shutdown of non-production GPUs.
Observability usually arrives too late. A GPU at 95% utilization looks like success; a queue 40 requests deep at the same moment means users are waiting. Platform health metrics such as time to first token and queue depth tell those two states apart, which GPU metrics alone cannot.
Those metrics only pay off when somebody sizes the cluster against peak traffic, not the average. The support assistant SPD Technology built for a French fashion retailer runs retrieval-augmented generation on AWS with LangChain, a Mistral LLM, and a vector database holding the retailer’s product and policy data. It answers 99% of queries in under 10 seconds and held 100% uptime during peak hours at 30 requests per second.
Model churn applies steady pressure to the same layer. New open-weight versions ship within months of each other, and the gap to closed models has narrowed and widened again since 2024. Every swap means re-provisioning hardware and re-validating behavior, and the eval suite and the gateway are what keep that affordable.
Serhii Leleko
AI & ML Engineer at SPD Technology
“A pilot on one GPU hides the two things that break in production. The first is the KV cache: at long context and real concurrency it outgrows the weights, and the server starts queuing instead of generating. The second is that a model validated in March is not still validated in June, so without a regression eval suite every upgrade ships an untested system.”
What Security and Compliance Work Does a Self-Hosted LLM Need?
Security work on a self-hosted deployment is the work a provider used to do quietly in the background. Taking out the third-party data transfer also takes out everything bundled with that endpoint, from access management to supply-chain checks, and your team inherits all of it. The obligation doesn’t move. GDPR, the EU AI Act, and HIPAA apply exactly as they did before.
Each control below maps to that transfer of responsibility. The third column is where audits actually fail, since each entry describes a default somebody left in place.
Control | What It Covers | Common Miss |
|---|---|---|
Access control | SSO, per-team keys, RBAC on inference endpoints | An engine port exposed without authentication |
Network isolation | Private subnets, no public endpoint, egress rules | Model or gateway reachable from the internet |
Guardrails | Input and output filtering, PII redaction, policy checks | Assuming open weights ship with the safety layers hosted APIs bundle |
Prompt-injection defense | Tool permissions, allow-lists, output validation for agents | Giving an agent tools without scoping them |
Logging and audit | Tamper-evident logs of prompts, outputs, and access | Logs and traces become a new store of the sensitive data the deployment was meant to protect |
Supply chain | Weight provenance, license review, container scanning | Pulling unverified weights or images |
Retention and residency | Where logs, caches, and backups live | Backups or observability tools sitting outside the jurisdiction |
Control
Access control
Network isolation
Guardrails
Prompt-injection defense
Logging and audit
Supply chain
Retention and residency
What It Covers
SSO, per-team keys, RBAC on inference endpoints
Private subnets, no public endpoint, egress rules
Input and output filtering, PII redaction, policy checks
Tool permissions, allow-lists, output validation for agents
Tamper-evident logs of prompts, outputs, and access
Weight provenance, license review, container scanning
Where logs, caches, and backups live
Common Miss
An engine port exposed without authentication
Model or gateway reachable from the internet
Assuming open weights ship with the safety layers hosted APIs bundle
Giving an agent tools without scoping them
Logs and traces become a new store of the sensitive data the deployment was meant to protect
Pulling unverified weights or images
Backups or observability tools sitting outside the jurisdiction
Access control is the row with public evidence behind it. Disclosing CVE-2026-7482 in May 2026, Cyera Research estimated roughly 300,000 internet-exposed Ollama servers. Ollama ships without authentication, so any server reachable on a public network interface with nothing in front of it is open to whoever finds it. Open weights also arrive without the safety layers a hosted API folds into its endpoint, so input filtering, PII redaction, and policy checks become guardrails your own serving path has to carry.
These controls are architectural decisions, and regulated payments show why. For a US FSA/HSA payments company, SPD Technology keeps more than 200,000 eligible healthcare products in sync, with AI eligibility classification above 80% accuracy, and ran HIPAA and SOC 2 remediation alongside the product work on sub-one-day release cycles on AWS ECS.
Self-hosting also simplifies a DPIA without removing the need for one. The data privacy analysis still has to name every store of personal data, and EU AI Act transparency duties can apply to the system you built, depending on how it is used.
How Much Does a Self-Hosted LLM Cost?
The cost of a self-hosted LLM depends on how busy the GPU stays, because you pay for it by the hour whether queries arrive or not. There is no general break-even point: it moves with utilization, context length, and the models you choose. The fair comparison is managed inference for the same open weights, not a frontier API.
\text{Cost per 1M output tokens} = \frac{\text{GPU price per hour}}{\text{sustained output tokens per second} \times 0.0036}
The formula follows Qovery’s self-hosting guide, and its denominator is the only part a team can control after procurement. Idle GPUs bill at the same hourly rate as busy ones, so cutting sustained throughput in half doubles the cost per token. An API bill, by contrast, falls with traffic at night, because you pay only for the tokens you send. LLM inference cost is more about utilization than about the hardware used.

Procurement budgets have the first line below. Entries marked as often missed can turn a favorable spreadsheet into an unfavorable invoice.
Cost Line | Notes | Often Missed? |
|---|---|---|
GPU capacity, rented or owned | Hourly, reserved, or depreciated hardware | No |
Idle capacity | Nights, weekends, traffic troughs | Yes |
Storage and network egress | Weights, caches, logs, cross-zone traffic | Yes |
Engineering and on-call | Ongoing platform and MLOps time, not a one-time setup cost | Yes |
Re-validation per model swap | Evals, requantization, rollout | Yes |
Security and compliance work | Every control listed in the previous section | Yes |
Cost Line
GPU capacity, rented or owned
Idle capacity
Storage and network egress
Engineering and on-call
Re-validation per model swap
Security and compliance work
Notes
Hourly, reserved, or depreciated hardware
Nights, weekends, traffic troughs
Weights, caches, logs, cross-zone traffic
Ongoing platform and MLOps time, not a one-time setup cost
Evals, requantization, rollout
Every control listed in the previous section
Often Missed?
No
Yes
Yes
Yes
Yes
Yes
One cost belongs outside that table. The copilots, agents, and eval harnesses your engineers use to build the system sit in the engineering budget, and what AI-assisted development really costs is a separate calculation from what it costs to serve a model to your users.
The pattern holds for most workloads: a low or spiky load favors an API, while a high, steady load on one narrow task tips the arithmetic in self-hosting’s favor. Only your own traffic figures can make the case.
Measure that case against your current spend and traffic, the same discipline behind the ROI of AI-assisted software engineering.
How Do You Optimize LLM Inference on the Same GPUs?
LLM inference optimization starts with keeping your existing GPUs busy and memory-efficient. Batching, KV-cache management, quantization, and routing save more money than new hardware, and each is a configuration change before it is a purchase. Most LLM inference work at this stage is memory work.
Every technique below buys something specific and charges for it somewhere else.
Technique | What It Saves | Trade-Off |
|---|---|---|
Continuous batching | Higher throughput under concurrent load | Needs a production engine |
Paged KV cache (PagedAttention) | Less memory waste, larger batches | Engine-specific |
Prefix caching | Repeated system prompts and RAG context | Gains depend on prefix overlap |
Quantization | VRAM and cost per token | Quality loss on structured output at low bit widths |
Speculative decoding | Lower latency | An extra draft model to run |
Context discipline | A smaller KV cache per request | Requires prompt and retrieval design |
Model routing | Simple requests to small models, hard ones to larger or API models | Needs evals per route |
Scheduled shutdown of non-production GPUs | Idle cost | Cold starts |
Technique
Continuous batching
Paged KV cache (PagedAttention)
Prefix caching
Quantization
Speculative decoding
Context discipline
Model routing
Scheduled shutdown of non-production GPUs
What It Saves
Higher throughput under concurrent load
Less memory waste, larger batches
Repeated system prompts and RAG context
VRAM and cost per token
Lower latency
A smaller KV cache per request
Simple requests to small models, hard ones to larger or API models
Idle cost
Trade-Off
Needs a production engine
Engine-specific
Gains depend on prefix overlap
Quality loss on structured output at low bit widths
An extra draft model to run
Requires prompt and retrieval design
Needs evals per route
Cold starts
Routing carries the largest headroom of the eight, since a small model absorbing the easy majority of traffic changes the arithmetic more than any engine flag. The same logic applies to pipelines where different models handle different steps. A PDF extraction pipeline SPD Technology built for a B2B financial intelligence publisher combines NLP, YOLO, Camelot, AWS Textract, and a GPT-based step. Processing became 3x faster and 5x cheaper, while coverage grew from 25 to more than 50 source structures.
When Is Self-Hosting an LLM the Wrong Choice?
Self-hosting is the wrong choice in three situations. Low or erratic volume leaves your GPUs idle, and at full cost. Without an operations owner, your service won’t have a solution at 2 AM. And a hard requirement for frontier-model quality on the hardest tasks puts open weights behind from week one.

A hybrid setup keeps all three doors open. Put a gateway in front first, send sensitive or high-volume traffic to a model you host, and leave the hardest reasoning with an API. That split is the usual arrangement for AI agents that mix cheap steps with expensive ones, and it is spreading fast. McKinsey’s State of AI 2026 reports 40% of respondents at organizations above $1 billion in revenue are scaling AI agents, up from 27% a year earlier. Since applications still call one OpenAI-compatible endpoint, moving a route later is a configuration change.
Model-agnostic integration is an early design decision, which is why agent harness engineering treats the model as a swappable component and keeps tool definitions outside it. Done early, it lets you swap AI models without touching application code, and it also serves as insurance against vendor lock-in. Prototypes that started on a laptop carry their own AI MVP scaling risks, since the shortcuts that made a demo fast fail first at concurrency.
Starting on a managed open-weight endpoint running the same weights, then self-hosting that route once utilization justifies it, costs far less than discovering the constraint after the hardware arrives.
Inline element: For workloads that need frontier quality, see how SPD Technology approaches building production AI with Claude.
Self-Hosted LLM vs. Managed Open-Weight Inference vs. API: How Do They Compare?
The three options answer the same request with different trade-offs. Control and cost shape pull in opposite directions across them, and time to production tracks how much of the stack you own. Quality ceiling is the one factor set by model choice alone.
Most teams run more than one of these at once, which is the frame to read the comparison against. The frontier column assumes proprietary models that only the provider can change.
Factor | Self-Hosted LLM | Managed Open-Weight Inference | Frontier API |
|---|---|---|---|
Data control | Full; data stays within your boundary | Provider processes data | Provider processes data |
Upfront cost | GPUs, setup, team | None | None |
Running cost driver | GPU utilization | Tokens | Tokens |
Time to production | Weeks to months | Days | Days |
Team required | Platform or MLOps, plus security | Integration only | Integration only |
Model quality ceiling | Best available open weights | The same open weights | Frontier models |
Model upgrades | You re-provision and re-validate | Provider upgrades; you re-validate | Provider upgrades; you re-validate |
Fits | Residency, air gap, sustained volume, version control | Open weights without the ops burden | Hardest tasks, low or spiky volume |
Factor
Data control
Upfront cost
Running cost driver
Time to production
Team required
Model quality ceiling
Model upgrades
Fits
Self-Hosted LLM
Full; data stays within your boundary
GPUs, setup, team
GPU utilization
Weeks to months
Platform or MLOps, plus security
Best available open weights
You re-provision and re-validate
Residency, air gap, sustained volume, version control
Managed Open-Weight Inference
Provider processes data
None
Tokens
Days
Integration only
The same open weights
Provider upgrades; you re-validate
Open weights without the ops burden
Frontier API
Provider processes data
None
Tokens
Days
Integration only
Frontier models
Provider upgrades; you re-validate
Hardest tasks, low or spiky volume
How Do You Start: From Pilot to Production?
Prove the constraint and the model before you buy capacity, because self-hosted solutions start quickly and unwind slowly. A phased path puts every GPU commitment behind a gate you’ve already passed, and each phase ends with something someone can check. Skip the first phase, and it tends to resurface in a security review, by which point the hardware is on the floor.

Exit criteria matter more than phase names. Write each one in such a way that a security lead, an ML engineer, or a CFO can see immediately whether it is fulfilled.
Phase | What Happens | Exit Criterion |
|---|---|---|
1. Constraint and use case | Name the mandate or the volume case; define the workload and success metrics | Constraint confirmed by security and legal |
2. Model shortlist by evals | Build the eval set; test two or three license-cleared open-weight models at the chosen quantization | One model meets the quality bar |
3. Load benchmark | Run the engine at real concurrency and context length; measure cost per token at realistic utilization | Cost and latency within target |
4. Pilot behind a gateway | Serve one workload through an OpenAI-compatible gateway, with guardrails and logging | Pilot meets quality, latency, and security criteria |
5. Production hardening | Autoscaling, monitoring, on-call, regression evals, model-update process | Ready for the scale-out decision |
Phase
1. Constraint and use case
2. Model shortlist by evals
3. Load benchmark
4. Pilot behind a gateway
5. Production hardening
What Happens
Name the mandate or the volume case; define the workload and success metrics
Build the eval set; test two or three license-cleared open-weight models at the chosen quantization
Run the engine at real concurrency and context length; measure cost per token at realistic utilization
Serve one workload through an OpenAI-compatible gateway, with guardrails and logging
Autoscaling, monitoring, on-call, regression evals, model-update process
Exit Criterion
Constraint confirmed by security and legal
One model meets the quality bar
Cost and latency within target
Pilot meets quality, latency, and security criteria
Ready for the scale-out decision
Phase 2 is where eval engineering does the deciding, since two candidates that look identical on a public leaderboard will separate on your own tasks. Scoping phase 1 with a partner is much the same conversation as starting an AI-assisted engineering engagement, where the first deliverable is a constraint confirmed in writing.
Hardening in phase 5 resembles taking an AI MVP to production in 90 days, with autoscaling, on-call rotations, and a model-update process replacing the manual steps that carried the pilot.
Before go-live, check the deployment against our AI Production-Ready Checklist.
How SPD Technology Builds and Runs Self-Hosted LLMs
Our AI engineering experts help teams work out whether to self-host at all, then build and verify the stack so it survives production and the next model upgrade. Our AI/ML development services cover that path end to end, from constraint review through the regression evals that guard model changes.
- Regulated, high-stakes domains: AI and data work delivered for a global investment data platform, Morningstar, and Pie Insurance; payments and PII held to a dedicated security-review gate; Adyen service partner status for payments work.
- Reliability under load: a support AI for a European fashion retailer answers 99% of queries in under 10 seconds and held 100% uptime at peaks of 30 requests per second.
- Economics, measured: a document-processing pipeline that combines several models cut processing time 3x and cost 5x, the per-task model judgment behind our AI solutions development.
- Verification built in: evals act as the acceptance gate, and a named senior engineer approves every release, the way we run our agentic AI development services.
- You keep the system: eval suites, specs, and configuration live in your repository and stay yours by contract, so a model swap or a change of partner doesn’t reset the work.
- Model-agnostic by design: Claude-certified architects and Select-tier membership in the Claude Partner Network, with integrations built so the choice between a model you host and an API model stays open.
Conclusion
Self-host an LLM because a constraint requires it, not because a calculator promises savings. Once residency, an air gap, or sustained volume settles that question, the work is the five commitments: a model your evals approve, GPUs sized for real memory and real utilization, an engine built for concurrency, a serving layer you can observe, and security your team now owns. The gateway and the eval suite keep that decision reversible: when the next open-weight release lands, you re-run the evals and change a route, not the application. That is what lets a self-hosted LLM survive production past its first year.
If you need a team to build and run it, see how to choose an AI development partner.
Key Takeaways
- Self-hosting an LLM is justified first by constraints that do not bend — data residency, a network-boundary mandate, sustained volume, or version control — because cost alone rarely beats managed inference running the same open weights.
- The cost per token of a self-hosted LLM depends on how busy the GPU stays, so a deployment sized for peak traffic and idle most of the day can cost far more than an API.
- A model fits a GPU only if the weights and the KV cache both fit, and long context at high concurrency can make the KV cache larger than the weights.
- Ollama suits development and single users, while concurrent production traffic needs an engine with continuous batching and paged KV-cache management such as vLLM.
- Self-hosting removes third-party data transfer and puts every security control on your team, including the guardrails a hosted API bundles in by default and the logs that now hold sensitive data.
- New open-weight versions ship within months, so a regression eval suite and an OpenAI-compatible gateway keep a self-hosted deployment maintainable.
- Starting on managed inference for the same open weights, behind a gateway, lets a team prove quality and utilization before committing to GPUs.
FAQ
How much does it cost to self-host an LLM?
LLM inference cost comes down to the GPU hourly price and how busy that GPU stays. Cost per 1M output tokens equals the GPU price per hour divided by the product of sustained output tokens per second and 0.0036. Storage, network egress, monitoring, security work, and re-validation on every model swap are easy to forget. Engineering and on-call time is the line teams most often leave out, and in the first year it can rival the hardware bill.
What hardware do you need to run an LLM on your own infrastructure?
VRAM decides it, and memory bandwidth decides how fast it runs. Weights take roughly 2 GB per billion parameters at FP16, about 1 GB at 8-bit, and about 0.5 to 0.6 GB at 4-bit, with the KV cache and 10 to 20% headroom on top. A 4-bit 8B model fits a 24 GB card such as an L4, while a 4-bit 70B model needs an 80 GB GPU.
Is a self-hosted LLM cheaper than using an API?
A self-hosted LLM is cheaper only when the GPU stays busy with sustained and predictable traffic. Idle hours bill at the full rate, so low or spiky volume makes every token expensive, and managed inference for the same open-weight model usually wins there. The fair comparison uses managed open-weight pricing for identical weights, since frontier API prices measure a different product. Engineering and re-validation time belong in it too.
What is the best open-weight model to self-host?
No single best model exists; the right one is the smallest model that passes evals on your own tasks within your hardware and license constraints. Shortlist two or three license-cleared candidates in the size class your GPUs support, test them at the quantization you plan to run, and weight tool calling for agent workloads. Re-run those evals whenever a new version ships.
Is a self-hosted LLM more secure than a cloud API?
It removes third-party transfer risk and puts every other control in your team’s hands. Access control, network isolation, guardrails, prompt-injection defenses, logging, and supply-chain checks all become internal work with internal owners. The most common failure is an unauthenticated inference endpoint, since popular servers still ship without authentication enabled. GDPR and EU AI Act obligations apply exactly as they did before the move.
What is the difference between vLLM and Ollama?
Ollama is built for fast local setup and single users; vLLM is built for concurrent production serving. Ollama started as a wrapper around llama.cpp with simple model management, while vLLM uses continuous batching and paged KV-cache management to hold throughput across many concurrent requests. Both expose an OpenAI-compatible API, so moving application code between them is mostly a base URL and model name change. Teams commonly prototype on Ollama and serve production traffic on vLLM.
What is a private LLM?
A private LLM is any deployment whose prompts and outputs stay inside an organization’s controlled environment, usually open-weight models self-hosted on-premises or in its own cloud account. Some vendors also apply the term to dedicated managed instances, where a provider runs isolated capacity for one customer. Before accepting the label, check where data is processed and who operates the infrastructure.
Can you combine a self-hosted LLM with API models?
Yes, and many teams do. An OpenAI-compatible gateway routes sensitive or high-volume requests to a self-hosted model and the hardest reasoning tasks to an API model. Because applications call the gateway rather than a specific model, a route can change without code changes. Each route still needs its own evals and data-handling rules.