Quick answer

A self-hosted LLM is an open-weight model running on infrastructure you control, on-premises or in your own cloud account, so prompts and data stay inside your boundary. Running one in production takes five commitments: a model chosen by evals on your own tasks and cleared for its license, GPUs sized for weights plus KV cache, a production inference engine such as vLLM, a serving layer with a gateway and observability, and security controls you now own. Each of them needs people to run it. A constraint such as data residency, an air-gap mandate, or sustained volume decides whether self-hosting is justified. Cost per token depends on GPU utilization, so an underused deployment can cost more per token than an API.

What Is a Self-Hosted LLM?

A self-hosted large language model runs on GPUs you control, using open weights you downloaded and inference software your own team operates. Those GPUs can sit on-premises, in a private cloud, or inside your own cloud account, and the arrangement is the same in all three. Private LLM and on-prem LLM describe variants of it, and buyers use the terms interchangeably.

Open-weight means downloadable model weights under a license, which is narrower than open code or published model training data. Calling them open source models without that qualification overstates the grant. The open source vs. open weight distinction decides what your legal team will sign.

Where Does a Self-Hosted LLM Run
Where Can a Self-Hosted LLM Run

“Self-hosted” gets applied to five arrangements, and only the first two are self-hosting in the strict sense. What separates them is where the model weights live, where data gets processed, and who holds the keys to the machine doing the processing.

Option
Where the Model Runs
Who Operates It
Data Leaves Your Boundary?

On-prem LLM

Your data center or colocation

You

No

Self-hosted in your cloud account (VPC)

Your AWS, GCP, Azure, or EU-cloud account

You

No, it stays in your account

Managed open-weight inference

Provider’s GPUs running the same open weights

Provider

Yes

Frontier API

Provider’s GPUs, proprietary model

Provider

Yes

Hybrid via a gateway

Mix of the above, routed per request

You (routing) + provider

Only for routes you allow

Where the Model Runs

Your data center or colocation

Your AWS, GCP, Azure, or EU-cloud account

Provider’s GPUs running the same open weights

Provider’s GPUs, proprietary model

Mix of the above, routed per request

Who Operates It

You

You

Provider

Provider

You (routing) + provider

Data Leaves Your Boundary?

No

No, it stays in your account

Yes

Yes

Only for routes you allow

Managed open-weight inference sits in the middle of that table: identical weights on somebody else’s GPUs, under a retention contract. It stays the natural fallback for any route that does not need the full boundary.

Why Do Companies Self-Host an LLM in 2026?

Most companies self-host an LLM because a constraint requires it: data residency, a network-boundary or air-gap mandate, sustained high volume, or control over model versions. Any one of them can be enough on its own. The assumption that self-hosting is cheaper by default is the one reason that rarely survives contact with a real workload.

Why Do Companies Self-Host an LLM in 2026
Why Do Companies Choose Self-Hosting an LLM in 2026

It helps to sort the reasons into obligations and optimizations. An obligation decides whether you self-host at all, and it usually comes from legal or security before engineering has formed a view. An optimization only decides how much of your traffic runs that way.

Three obligations drive most self-hosted deployments in regulated industries. When one of them applies to your workload, the decision has already been made for you:

  • Data residency and sovereignty, since a provider that receives prompts containing personal data acts as a processor under GDPR, and serving them outside the EEA triggers its transfer rules.
  • Network-boundary or air-gapped environments in healthcare, defense, and critical infrastructure.
  • Contractual no-retention or no-training requirements that a customer wrote into your agreement.

Optimizations behave differently because each trades effort for money or latency. Four come up in most build-or-buy discussions, and any of them can be bought back with an API if the effort stops paying.

  • Cost at sustained volume.
  • Latency and independence from provider rate limits.
  • Version pinning, so a silent upgrade cannot change output quality overnight.
  • Tuning a narrow model that beats a general one on your own task.

Vercel’s guide to self-hosting draws the same line: a mandate makes self-hosting an obligation, and cost only makes it an optimization. Whoever owns the budget can overrule an optimization.

Sustained volume pays for the optimization, because a GPU only becomes cheap once it stops sitting idle. Workloads with steady daily volume fit that shape: the trend-detection pipelines SPD Technology built for a global investment data platform, in a partnership now past ten years, process roughly 50,000 articles a day from more than 3,000 data sources.

Open-weight models are close to the best closed models without having caught them, and the distance moves in both directions. Stanford HAI’s 2026 AI Index put the top closed model 3.3% ahead of the top open model on the Arena Leaderboard in March 2026, up from 0.5% in August 2024. A few points on a public leaderboard matter only if they show up in your own tasks, so the properties of LLMs your workload actually depends on determine what that gap costs you.

What Does It Really Take to Self-Host an LLM?

Running a self-hosted LLM takes five commitments. Two are decided once, at procurement. You need a model validated by evals on your own tasks, and hardware sized for the memory the model actually consumes, which means VRAM, the video random access memory on the GPU card itself. That memory holds the weights plus the KV cache (key-value cache), where the server keeps attention state for every in-flight request.

The other three are operational, keep costing every month, and teams underestimate them. A production inference engine. A serving layer with a gateway and observability. Every security control that used to arrive bundled with somebody else’s API. All of it becomes your team’s property the day you download the weights.

What It Really Takes to Self-Host an LLM
What Does It Take to Self-Host an LLM

LLM hosting turns into a platform job long before it becomes a model job. The people who keep it running come from MLOps consulting and development or an internal platform and machine learning team, and one of them answers the page at 2 AM.

Running a Self-Hosted LLM: Five Commitment Checklist
  • A model chosen by evals on your own tasks and cleared for its license.

  • GPUs sized for the weights plus the KV cache at target concurrency.

  • A production engine benchmarked at real load, with a single-prompt test proving nothing.

  • A gateway, autoscaling, monitoring, and regression evals around the model.

  • Security, logging, and guardrails owned in-house from the first day of traffic.

TIP: The checklist doubles as a budget forecast. A plan that covers only the GPUs runs short by the second quarter, because the three operating commitments keep spending long after the hardware lands, while evals and observability cut across all five and never finish.

Order matters as much as the list does. Model choice sets the hardware and the hardware narrows the engine, and self-hosting platforms package parts of that chain without moving the commitments off your team.

How Do You Choose an Open-Weight Model to Self-Host?

Match the model to the task first and to the hardware you can buy second. Model size and quantization drive cost far more than the engine does, so a smaller model that clears your quality bar stays cheaper every month after.

Start with the license. Some restrict use by region, others by user count or by the revenue of the company using them, and either can disqualify a model after a team has already benchmarked it. The split between Western vs. Chinese open-weight models raises procurement questions a regulated buyer is better off clearing in week one.

Evals beat leaderboards for a decision this specific. Define 50 to 200 representative tasks with a rubric scored on your own data, and test at the quantization you plan to ship. For agent workloads, the deciding score is tool calling, and the method matches how to evaluate AI agents before production anywhere else in your stack. Output evaluation judges the final answer, while trajectory evaluation judges the steps behind it.

Quantization stores the weights at lower numeric precision. Q4-class means roughly four bits per parameter, down from the sixteen a model usually ships with, and for general text work the quality loss is minimal. Structured output and long reasoning are the first things to wobble at lower bit widths, so put both in the eval set before anyone signs for a card. A narrow task often needs a smaller model than a leaderboard suggests, and fine-tuning one on domain data is the work behind generative AI development services.

What GPU Do You Need for an LLM?

Sizing a GPU for LLM work starts with model size, which sets the VRAM floor, and ends with memory bandwidth, which sets the speed once the model fits. Weights take the parameter count multiplied by bytes per parameter, the KV cache grows with the context window multiplied by concurrent requests, and 10 to 20% headroom covers fragmentation and activation memory. The arithmetic here covers text models; image generation workloads size differently.

What GPU Do You Need for an LLM
What GPU Do You Need for an LLM

The figures below follow Qovery’s rule of thumb and cover weights alone, so add the cache and the headroom before matching a model to a card.

Model Size
FP16 Weights
8-bit
4-bit
Typical GPU Class (4-bit)

~8B

~16 GB

~8 GB

~5–6 GB

24 GB card (L4, RTX 4090)

~32B

~64 GB

~32 GB

~20 GB

48 GB card (L40S)

~70B

~140 GB

~70 GB

~40–42 GB

One 80 GB GPU (A100/H100)

Large MoE (100B+ total)

Set by total parameters

—

—

One or more 80 GB GPUs; speed set by active parameters

FP16 Weights

~16 GB

~64 GB

~140 GB

Set by total parameters

8-bit

~8 GB

~32 GB

~70 GB

—

4-bit

~5–6 GB

~20 GB

~40–42 GB

—

Typical GPU Class (4-bit)

24 GB card (L4, RTX 4090)

48 GB card (L40S)

One 80 GB GPU (A100/H100)

One or more 80 GB GPUs; speed set by active parameters

Two things catch teams out here. The first is the KV cache, which at long context lengths and high concurrency can outgrow the weights and turn a comfortable fit into an out-of-memory error the moment real traffic arrives. The second is mixture-of-experts (MoE). MoE models route each token through a subset of expert networks, yet every expert stays resident in memory, so you buy VRAM for the total parameter count and get the speed of the active ones.

Rented capacity changes the bill’s shape without changing the operational load. Cloud GPUs move the initial investment to an hourly line and make resizing easy, while owned hardware in an on-premises data center amortizes over years and fixes your memory ceiling. Owned racks also pull power distribution and cooling into the conversation, which is a facilities decision before it is an engineering one. For dense models, offloading to system RAM costs too much generation speed to plan around.

vLLM vs. Ollama: Which Inference Engine Fits?

Ollama and llama.cpp are designed for development, local testing, and individual users. vLLM, SGLang, TGI, and TensorRT-LLM are built to handle concurrent production traffic. Put an engine tuned for single-user LLM inference behind ten concurrent users, and it queues, with utilization flat and latency climbing.

The difference is scheduling. Continuous batching adds and removes requests at every generation step, so a short request no longer waits for the longest one in its batch. A paged KV cache allocates attention memory in small blocks, so more requests fit on the same GPU. The PagedAttention paper presented at SOSP 2023 reported 2 to 4 times higher serving throughput than the serving systems of that time at the same latency.

Ollama has earned its place because one command puts a local model on a workstation, and LM Studio does much the same behind a graphical interface. Both leave a single developer with a local server to work against. llama.cpp goes further down the ladder, keeping local inference running without a data center GPU at all. Most of the decision below sits in the note column.

Engine
Best For
Concurrency
Note

Ollama

Local development, prototypes

Low

Fastest start; never expose it unauthenticated

llama.cpp

CPU, Apple Silicon, edge devices

Low

Runs without a data center GPU

vLLM

General production serving

High (continuous batching, paged KV cache)

Broad model support; OpenAI-compatible

SGLang

Shared-prefix workloads: RAG, multi-turn, agents

High

Benchmark against vLLM on your workload

TGI

Hugging Face-centered stacks

High

Mature tooling

TensorRT-LLM

Peak NVIDIA throughput

High

Benchmark against vLLM on your workload

Best For

Local development, prototypes

CPU, Apple Silicon, edge devices

General production serving

Shared-prefix workloads: RAG, multi-turn, agents

Hugging Face-centered stacks

Peak NVIDIA throughput

Concurrency

Low

Low

High (continuous batching, paged KV cache)

High

High

High

Note

Fastest start; never expose it unauthenticated

Runs without a data center GPU

Broad model support; OpenAI-compatible

Benchmark against vLLM on your workload

Mature tooling

Benchmark against vLLM on your workload

Engine choice quietly fixes your quantization format as well. GGUF files run on llama.cpp and Ollama. vLLM is built around AWQ, GPTQ, and FP8, with only experimental GGUF support, and moving a model between the two camps usually means requantizing it before it will load.

What Does Serving an LLM in Production Involve?

Serving turns a working model into a service with uptime. At this stage, LLM deployment looks like any other stateful workload on Kubernetes, with a GPU-specific wrinkle in every layer. Each item below covers a failure that shows up in production rather than in a pilot.

  • A GPU node pool with the device plugin installed and tested.
  • Weights cached in object storage so a cold start never pulls from the public internet.
  • Readiness probes tuned for slow model load, since large weights take minutes to become servable.
  • Autoscaling keyed to queue depth, because CPU utilization says nothing useful about a saturated GPU.
  • An OpenAI-compatible gateway holding keys, budgets, and SSO.
  • Observability covering tokens per second, time to first token, queue depth, and GPU utilization.
  • A regression eval suite run before every model, quantization, or engine change.
  • Scheduled shutdown of non-production GPUs.

Observability usually arrives too late. A GPU at 95% utilization looks like success; a queue 40 requests deep at the same moment means users are waiting. Platform health metrics such as time to first token and queue depth tell those two states apart, which GPU metrics alone cannot.

Those metrics only pay off when somebody sizes the cluster against peak traffic, not the average. The support assistant SPD Technology built for a French fashion retailer runs retrieval-augmented generation on AWS with LangChain, a Mistral LLM, and a vector database holding the retailer’s product and policy data. It answers 99% of queries in under 10 seconds and held 100% uptime during peak hours at 30 requests per second.

Model churn applies steady pressure to the same layer. New open-weight versions ship within months of each other, and the gap to closed models has narrowed and widened again since 2024. Every swap means re-provisioning hardware and re-validating behavior, and the eval suite and the gateway are what keep that affordable.

Sergii Leleko SPD Technology

Serhii Leleko

AI & ML Engineer at SPD Technology

“A pilot on one GPU hides the two things that break in production. The first is the KV cache: at long context and real concurrency it outgrows the weights, and the server starts queuing instead of generating. The second is that a model validated in March is not still validated in June, so without a regression eval suite every upgrade ships an untested system.”

Serhii Leleko, AI & ML Engineer at SPD Technology

What Security and Compliance Work Does a Self-Hosted LLM Need?

Security work on a self-hosted deployment is the work a provider used to do quietly in the background. Taking out the third-party data transfer also takes out everything bundled with that endpoint, from access management to supply-chain checks, and your team inherits all of it. The obligation doesn’t move. GDPR, the EU AI Act, and HIPAA apply exactly as they did before.

Each control below maps to that transfer of responsibility. The third column is where audits actually fail, since each entry describes a default somebody left in place.

Control
What It Covers
Common Miss

Access control

SSO, per-team keys, RBAC on inference endpoints

An engine port exposed without authentication

Network isolation

Private subnets, no public endpoint, egress rules

Model or gateway reachable from the internet

Guardrails

Input and output filtering, PII redaction, policy checks

Assuming open weights ship with the safety layers hosted APIs bundle

Prompt-injection defense

Tool permissions, allow-lists, output validation for agents

Giving an agent tools without scoping them

Logging and audit

Tamper-evident logs of prompts, outputs, and access

Logs and traces become a new store of the sensitive data the deployment was meant to protect

Supply chain

Weight provenance, license review, container scanning

Pulling unverified weights or images

Retention and residency

Where logs, caches, and backups live

Backups or observability tools sitting outside the jurisdiction

What It Covers

SSO, per-team keys, RBAC on inference endpoints

Private subnets, no public endpoint, egress rules

Input and output filtering, PII redaction, policy checks

Tool permissions, allow-lists, output validation for agents

Tamper-evident logs of prompts, outputs, and access

Weight provenance, license review, container scanning

Where logs, caches, and backups live

Common Miss

An engine port exposed without authentication

Model or gateway reachable from the internet

Assuming open weights ship with the safety layers hosted APIs bundle

Giving an agent tools without scoping them

Logs and traces become a new store of the sensitive data the deployment was meant to protect

Pulling unverified weights or images

Backups or observability tools sitting outside the jurisdiction

Access control is the row with public evidence behind it. Disclosing CVE-2026-7482 in May 2026, Cyera Research estimated roughly 300,000 internet-exposed Ollama servers. Ollama ships without authentication, so any server reachable on a public network interface with nothing in front of it is open to whoever finds it. Open weights also arrive without the safety layers a hosted API folds into its endpoint, so input filtering, PII redaction, and policy checks become guardrails your own serving path has to carry.

These controls are architectural decisions, and regulated payments show why. For a US FSA/HSA payments company, SPD Technology keeps more than 200,000 eligible healthcare products in sync, with AI eligibility classification above 80% accuracy, and ran HIPAA and SOC 2 remediation alongside the product work on sub-one-day release cycles on AWS ECS.

Self-hosting also simplifies a DPIA without removing the need for one. The data privacy analysis still has to name every store of personal data, and EU AI Act transparency duties can apply to the system you built, depending on how it is used.

How Much Does a Self-Hosted LLM Cost?

The cost of a self-hosted LLM depends on how busy the GPU stays, because you pay for it by the hour whether queries arrive or not. There is no general break-even point: it moves with utilization, context length, and the models you choose. The fair comparison is managed inference for the same open weights, not a frontier API.

\text{Cost per 1M output tokens} = \frac{\text{GPU price per hour}}{\text{sustained output tokens per second} \times 0.0036}

The formula follows Qovery’s self-hosting guide, and its denominator is the only part a team can control after procurement. Idle GPUs bill at the same hourly rate as busy ones, so cutting sustained throughput in half doubles the cost per token. An API bill, by contrast, falls with traffic at night, because you pay only for the tokens you send. LLM inference cost is more about utilization than about the hardware used.

How Much Does a Self-Hosted LLM Cost
How Much Does aIt Cost to Self-Host an LLM

Procurement budgets have the first line below. Entries marked as often missed can turn a favorable spreadsheet into an unfavorable invoice.

Cost Line
Notes
Often Missed?

GPU capacity, rented or owned

Hourly, reserved, or depreciated hardware

No

Idle capacity

Nights, weekends, traffic troughs

Yes

Storage and network egress

Weights, caches, logs, cross-zone traffic

Yes

Engineering and on-call

Ongoing platform and MLOps time, not a one-time setup cost

Yes

Re-validation per model swap

Evals, requantization, rollout

Yes

Security and compliance work

Every control listed in the previous section

Yes

Notes

Hourly, reserved, or depreciated hardware

Nights, weekends, traffic troughs

Weights, caches, logs, cross-zone traffic

Ongoing platform and MLOps time, not a one-time setup cost

Evals, requantization, rollout

Every control listed in the previous section

Often Missed?

No

Yes

Yes

Yes

Yes

Yes

One cost belongs outside that table. The copilots, agents, and eval harnesses your engineers use to build the system sit in the engineering budget, and what AI-assisted development really costs is a separate calculation from what it costs to serve a model to your users.

The pattern holds for most workloads: a low or spiky load favors an API, while a high, steady load on one narrow task tips the arithmetic in self-hosting’s favor. Only your own traffic figures can make the case.

Measure that case against your current spend and traffic, the same discipline behind the ROI of AI-assisted software engineering.

How Do You Optimize LLM Inference on the Same GPUs?

LLM inference optimization starts with keeping your existing GPUs busy and memory-efficient. Batching, KV-cache management, quantization, and routing save more money than new hardware, and each is a configuration change before it is a purchase. Most LLM inference work at this stage is memory work.

Every technique below buys something specific and charges for it somewhere else.

Technique
What It Saves
Trade-Off

Continuous batching

Higher throughput under concurrent load

Needs a production engine

Paged KV cache (PagedAttention)

Less memory waste, larger batches

Engine-specific

Prefix caching

Repeated system prompts and RAG context

Gains depend on prefix overlap

Quantization

VRAM and cost per token

Quality loss on structured output at low bit widths

Speculative decoding

Lower latency

An extra draft model to run

Context discipline

A smaller KV cache per request

Requires prompt and retrieval design

Model routing

Simple requests to small models, hard ones to larger or API models

Needs evals per route

Scheduled shutdown of non-production GPUs

Idle cost

Cold starts

What It Saves

Higher throughput under concurrent load

Less memory waste, larger batches

Repeated system prompts and RAG context

VRAM and cost per token

Lower latency

A smaller KV cache per request

Simple requests to small models, hard ones to larger or API models

Idle cost

Trade-Off

Needs a production engine

Engine-specific

Gains depend on prefix overlap

Quality loss on structured output at low bit widths

An extra draft model to run

Requires prompt and retrieval design

Needs evals per route

Cold starts

Routing carries the largest headroom of the eight, since a small model absorbing the easy majority of traffic changes the arithmetic more than any engine flag. The same logic applies to pipelines where different models handle different steps. A PDF extraction pipeline SPD Technology built for a B2B financial intelligence publisher combines NLP, YOLO, Camelot, AWS Textract, and a GPT-based step. Processing became 3x faster and 5x cheaper, while coverage grew from 25 to more than 50 source structures.

When Is Self-Hosting an LLM the Wrong Choice?

Self-hosting is the wrong choice in three situations. Low or erratic volume leaves your GPUs idle, and at full cost. Without an operations owner, your service won’t have a solution at 2 AM. And a hard requirement for frontier-model quality on the hardest tasks puts open weights behind from week one.

When Self-Hosting an LLM Is the Wrong Choice
When Self-Hosting an LLM Is Not the Best Option

A hybrid setup keeps all three doors open. Put a gateway in front first, send sensitive or high-volume traffic to a model you host, and leave the hardest reasoning with an API. That split is the usual arrangement for AI agents that mix cheap steps with expensive ones, and it is spreading fast. McKinsey’s State of AI 2026 reports 40% of respondents at organizations above $1 billion in revenue are scaling AI agents, up from 27% a year earlier. Since applications still call one OpenAI-compatible endpoint, moving a route later is a configuration change.

Model-agnostic integration is an early design decision, which is why agent harness engineering treats the model as a swappable component and keeps tool definitions outside it. Done early, it lets you swap AI models without touching application code, and it also serves as insurance against vendor lock-in. Prototypes that started on a laptop carry their own AI MVP scaling risks, since the shortcuts that made a demo fast fail first at concurrency.

Starting on a managed open-weight endpoint running the same weights, then self-hosting that route once utilization justifies it, costs far less than discovering the constraint after the hardware arrives.

Inline element: For workloads that need frontier quality, see how SPD Technology approaches building production AI with Claude.

Self-Hosted LLM vs. Managed Open-Weight Inference vs. API: How Do They Compare?

The three options answer the same request with different trade-offs. Control and cost shape pull in opposite directions across them, and time to production tracks how much of the stack you own. Quality ceiling is the one factor set by model choice alone.

Most teams run more than one of these at once, which is the frame to read the comparison against. The frontier column assumes proprietary models that only the provider can change.

Factor
Self-Hosted LLM
Managed Open-Weight Inference
Frontier API

Data control

Full; data stays within your boundary

Provider processes data

Provider processes data

Upfront cost

GPUs, setup, team

None

None

Running cost driver

GPU utilization

Tokens

Tokens

Time to production

Weeks to months

Days

Days

Team required

Platform or MLOps, plus security

Integration only

Integration only

Model quality ceiling

Best available open weights

The same open weights

Frontier models

Model upgrades

You re-provision and re-validate

Provider upgrades; you re-validate

Provider upgrades; you re-validate

Fits

Residency, air gap, sustained volume, version control

Open weights without the ops burden

Hardest tasks, low or spiky volume

Self-Hosted LLM

Full; data stays within your boundary

GPUs, setup, team

GPU utilization

Weeks to months

Platform or MLOps, plus security

Best available open weights

You re-provision and re-validate

Residency, air gap, sustained volume, version control

Managed Open-Weight Inference

Provider processes data

None

Tokens

Days

Integration only

The same open weights

Provider upgrades; you re-validate

Open weights without the ops burden

Frontier API

Provider processes data

None

Tokens

Days

Integration only

Frontier models

Provider upgrades; you re-validate

Hardest tasks, low or spiky volume

How Do You Start: From Pilot to Production?

Prove the constraint and the model before you buy capacity, because self-hosted solutions start quickly and unwind slowly. A phased path puts every GPU commitment behind a gate you’ve already passed, and each phase ends with something someone can check. Skip the first phase, and it tends to resurface in a security review, by which point the hardware is on the floor.

How to Start a Self-Hosted LLM
How to Start a Self-Hosted LLM in 2026

Exit criteria matter more than phase names. Write each one in such a way that a security lead, an ML engineer, or a CFO can see immediately whether it is fulfilled.

Phase
What Happens
Exit Criterion

1. Constraint and use case

Name the mandate or the volume case; define the workload and success metrics

Constraint confirmed by security and legal

2. Model shortlist by evals

Build the eval set; test two or three license-cleared open-weight models at the chosen quantization

One model meets the quality bar

3. Load benchmark

Run the engine at real concurrency and context length; measure cost per token at realistic utilization

Cost and latency within target

4. Pilot behind a gateway

Serve one workload through an OpenAI-compatible gateway, with guardrails and logging

Pilot meets quality, latency, and security criteria

5. Production hardening

Autoscaling, monitoring, on-call, regression evals, model-update process

Ready for the scale-out decision

What Happens

Name the mandate or the volume case; define the workload and success metrics

Build the eval set; test two or three license-cleared open-weight models at the chosen quantization

Run the engine at real concurrency and context length; measure cost per token at realistic utilization

Serve one workload through an OpenAI-compatible gateway, with guardrails and logging

Autoscaling, monitoring, on-call, regression evals, model-update process

Exit Criterion

Constraint confirmed by security and legal

One model meets the quality bar

Cost and latency within target

Pilot meets quality, latency, and security criteria

Ready for the scale-out decision

Phase 2 is where eval engineering does the deciding, since two candidates that look identical on a public leaderboard will separate on your own tasks. Scoping phase 1 with a partner is much the same conversation as starting an AI-assisted engineering engagement, where the first deliverable is a constraint confirmed in writing.

Hardening in phase 5 resembles taking an AI MVP to production in 90 days, with autoscaling, on-call rotations, and a model-update process replacing the manual steps that carried the pilot.

Before go-live, check the deployment against our AI Production-Ready Checklist.

How SPD Technology Builds and Runs Self-Hosted LLMs

Our AI engineering experts help teams work out whether to self-host at all, then build and verify the stack so it survives production and the next model upgrade. Our AI/ML development services cover that path end to end, from constraint review through the regression evals that guard model changes.

  • Regulated, high-stakes domains: AI and data work delivered for a global investment data platform, Morningstar, and Pie Insurance; payments and PII held to a dedicated security-review gate; Adyen service partner status for payments work.
  • Reliability under load: a support AI for a European fashion retailer answers 99% of queries in under 10 seconds and held 100% uptime at peaks of 30 requests per second.
  • Economics, measured: a document-processing pipeline that combines several models cut processing time 3x and cost 5x, the per-task model judgment behind our AI solutions development.
  • Verification built in: evals act as the acceptance gate, and a named senior engineer approves every release, the way we run our agentic AI development services.
  • You keep the system: eval suites, specs, and configuration live in your repository and stay yours by contract, so a model swap or a change of partner doesn’t reset the work.
  • Model-agnostic by design: Claude-certified architects and Select-tier membership in the Claude Partner Network, with integrations built so the choice between a model you host and an API model stays open.

Conclusion

Self-host an LLM because a constraint requires it, not because a calculator promises savings. Once residency, an air gap, or sustained volume settles that question, the work is the five commitments: a model your evals approve, GPUs sized for real memory and real utilization, an engine built for concurrency, a serving layer you can observe, and security your team now owns. The gateway and the eval suite keep that decision reversible: when the next open-weight release lands, you re-run the evals and change a route, not the application. That is what lets a self-hosted LLM survive production past its first year.

If you need a team to build and run it, see how to choose an AI development partner.

Key Takeaways

  • Self-hosting an LLM is justified first by constraints that do not bend — data residency, a network-boundary mandate, sustained volume, or version control — because cost alone rarely beats managed inference running the same open weights.
  • The cost per token of a self-hosted LLM depends on how busy the GPU stays, so a deployment sized for peak traffic and idle most of the day can cost far more than an API.
  • A model fits a GPU only if the weights and the KV cache both fit, and long context at high concurrency can make the KV cache larger than the weights.
  • Ollama suits development and single users, while concurrent production traffic needs an engine with continuous batching and paged KV-cache management such as vLLM.
  • Self-hosting removes third-party data transfer and puts every security control on your team, including the guardrails a hosted API bundles in by default and the logs that now hold sensitive data.
  • New open-weight versions ship within months, so a regression eval suite and an OpenAI-compatible gateway keep a self-hosted deployment maintainable.
  • Starting on managed inference for the same open weights, behind a gateway, lets a team prove quality and utilization before committing to GPUs.

FAQ

  • How much does it cost to self-host an LLM?

    LLM inference cost comes down to the GPU hourly price and how busy that GPU stays. Cost per 1M output tokens equals the GPU price per hour divided by the product of sustained output tokens per second and 0.0036. Storage, network egress, monitoring, security work, and re-validation on every model swap are easy to forget. Engineering and on-call time is the line teams most often leave out, and in the first year it can rival the hardware bill.