Self-Hosting LLMs vs Inference APIs: The True Cost Breakdown

Self-hosting an open model beats serverless pricing per token, but only if you can keep the hardware full. One owned eight-GPU node breaks even around 2.8 billion output tokens a month of the work it can actually serve, which is roughly 180 engineers running agentic workflows or 550 on lighter copilot-style use. Below that, serverless is cheaper and needs no capital, no lead time, and no on-call rotation.

That is the short answer. The rest is the arithmetic, with every assumption on the page so you can change one and get a different result. The worked examples are coding agents, following our guide to open-source coding models, but the method transfers to any workload: work out what your traffic sends, price it, and find where the lines cross. Our COO covered frontier versus open pricing in the cost and predictability piece; this one asks where the model you picked should run.

Three things decide that, and only one is cost. Cost is what most comparisons measure. Capability is whether the model you can afford is good enough at the specific job, often a reason to pay more. Control is residency, provenance, licence, and uptime you answer for, and where it applies it outranks the other two.

Most readers arrive from one of two places: a founder or engineer at an AI startup watching a token bill outgrow revenue, or an infrastructure lead at a company that has scaled into a build-versus-rent decision with a budget attached.

Disclosure: Hoonify serves open-weight models and builds dedicated and on-premises clusters, so we have an interest on both sides. Read the assumptions rather than the conclusion.

When to self-host an LLM: narrowing the field

Count the ways to run an open model and you reach a dozen. Most do not survive a production requirement, and the rest collapse into three, all of which present an OpenAI-compatible endpoint. What changes is who owns the hardware behind it and how you are billed.

Model	You pay for	You manage	Cost behaviour	Typically
Serverless inference	Tokens consumed	Nothing	Variable	Startups, early scale
Dedicated inference	Reserved capacity, monthly	Nothing	Fixed within the term, repriced at renewal	Scaled teams, steady load
Owned hardware	The box, up front	Everything	Fixed for the depreciation life	Control requirements, very high volume

One term from the pricing pages, since it looks like a fourth option and is not. Interruptible or preemptible capacity is GPU time sold cheaply because the provider can take it back mid-request when a full-price customer wants it. It is for work that retries from a checkpoint, not for live serving.

Serverless is variable. Owned hardware is fixed for three years. Dedicated sits between them, fixed while your term runs and repriced at renewal, so model it as a step rather than a flat line. Its pricing is negotiated rather than published, so what follows compares the two ends and leaves the middle to a quote. A fixed cost is not automatically a smaller cost, and most of this piece is about when it is and is not.

What actually drives LLM inference cost

Work out what your workload sends before you cost any hardware. Two things set the bill for a role: how input-heavy it is, and what its model charges.

Four coding roles, four different bills

Two things drive the cost of a role: how input-heavy it is, and what the model it runs charges.

Role	Model	List in / out	Ratio	All-in /1M out	Cached
The planner	Kimi K3	$3.00 / $15.00	20:1	$75.00	$31.80
The terminal operator	GLM-5.3	$1.40 / $4.40	10:1	$18.40	$9.28
The problem solver	DeepSeek V4 Pro	$1.31 / $3.96	4:1	$9.20	$5.15
The doer	Qwen3.8-27B	$0.35 / $2.55	3:1	$3.60	$2.88

All-in = what one million output tokens cost once the input arriving with them is billed too: (ratio × input price) + output price. List prices as published, August 2026, for the current checkpoint of each model; they vary by provider and by checkpoint. Ratios are our estimates.

Read the all-in column carefully, because it is not a list price. Kimi K3’s published output rate is $15.00 per million. The $75.00 is what a million output tokens cost once the twenty million input tokens arriving with them are billed too: (20 × $3.00) + $15.00. That is what the invoice reflects, and it is the figure that matters when sizing hardware, because output throughput is what a GPU is limited by.

The order follows the ratio, but the gaps do not. The planner’s ratio is 6.7 times the doer’s yet costs 20.8 times as much, because Kimi K3 also charges 8.6 times more per input token. Hold price constant and the ratio alone produces a 3.8x spread; add real prices and it becomes 20.8x. Model choice roughly quintuples the spread the ratio creates.

Two consequences carry into the hardware question.

Caching and prefill behave differently on your own box

First, caching pays differently depending on who owns the hardware. Cached input runs 80 to 90 percent below fresh input across all four models, so the planner’s bill drops by more than half. Self-hosted stacks do prefix caching too, but the saving arrives as throughput rather than as a discount, and if your hardware is not already at capacity, serving the same request faster costs what it cost before.

Second, input-heavy roles fill hardware inefficiently. Throughput is quoted in output tokens per second because decode is what gets benchmarked. A 20:1 workload means twenty tokens of prefill per token generated, on the same GPU. The role with the smallest self-hosted caching benefit also wastes the most capacity.

The line item nobody puts in the comparison

Per-token comparisons price GPU time and nothing else. Add one infrastructure engineer at 0.35 FTE on a fully loaded $275,000 a year, covering salary, benefits, and overhead, which is a conservative read of on-call, driver upgrades, model migrations, and capacity planning.

That engineer is 45% of what it costs to own the box, at every volume, because they cost the same whether it is busy or idle. The owning line falls as you fill the node and crosses serverless pricing at 2.8 billion output tokens a month. Below that crossing you pay more per token than you would to rent, and the gap widens fast: at 500 million a month owning runs $35.31 per million against $6.37 serverless.

Drive one cost low enough and it stops being what you pay for. The cheaper the per-token figure someone quotes, the more of their real cost sits outside it.

This is also the one line you can actually attack. A management layer that handles provisioning, upgrades, and failover cuts the hours a cluster demands, and its licence is a real annual cost you should put in the model alongside the engineer it displaces.

Self-hosting an LLM in production: what a box costs

At today’s prices, expect around $450,000 for a fully loaded eight-GPU H200 node: accelerators, dual server CPUs, several terabytes of system memory, 400Gb networking, chassis and power, and three years of support. The accelerators are roughly two thirds of it.

You need eight GPUs only when the model demands it. GLM-5.3 is 753 billion parameters with a published FP8 checkpoint: roughly one byte per parameter plus 20% for key-value cache, or about 900GB. That misses an eight-way H100 node at 640GB and fits an eight-way H200 node at 1,128GB.

Which leaves 224GB spare, and that matters more than it sounds. Qwen3.8-27B needs about 32GB, so both models sit resident on the same node with room left for key-value cache. You do not buy hardware per role; you buy one box and serve the stack from it. Any comparison pricing a separate machine per model is inflating the cost of owning.

What one node costs a year

Line item Annual
Amortized hardware	$105,000
Power	$10,591
Ops labor, 0.35 FTE	$96,250
Total fixed annual cost $211,841

One node, less a 30% residual, over three years: $450,000 × 0.7 ÷ 3 = $105,000. Power uses the US commercial average of $0.15 per kWh at 6.2 kW sustained and a 1.3 datacenter PUE; industrial rates nearer $0.10 cut that line by a third. Three-year amortization rather than five isdeliberate, since hyperscalers depreciate longer because hardware cascades from training to inference to batch, and one node has no cascade.

That is a single node with no redundancy. Failover is a choice, not a requirement: add a second node and hardware and power roughly double while ops does not, landing near $327,000. We price the single node because it is the honest floor, and state what redundancy costs rather than burying it in the base case.

Ops is 45% of that total before a single token is served, which is the line most comparisons leave out.

How many engineers does one node hold

Tokens per month is a bad unit for a decision most people make in headcount, so convert it. A node sustaining 6,300 output tokens per second serves roughly 126 concurrent streams at 50 tokens per second each, a conservative single-stream rate for a mixture-of-experts model this size.

What one engineer consumes depends entirely on how they work, and the gap between the two common patterns is wide enough to change the answer.

Copilot and agentic use are not the same workload

Usage pattern	Tokens per engineer/month	Serverless cost	Break-even
Copilot, human in the loop	~6M	$65/mo	~550 engineers
Agentic, two agents running past 5pm	~19M	$194/mo	~180 engineers

The agentic figure is the one worth trusting, and there is an independent check on it. Coding subscriptions are priced to stay profitable on heavy users, and at our blended rate the GLM Coding Plan Max tier at $168 a month implies about 16 million output tokens, while a $200 plan implies about 20 million. Nineteen million sits inside that band. A copilot user at 6 million would be paying $65 a month, which is nobody’s heavy tier.

Agentic use helps twice over. Each engineer generates three times the tokens, and agents that keep working past the end of the day stretch the hours the box is earning. A business-hours-only workload can use just 24% of the month’s seconds; agents running into the evening push that past half.

Only the doer and terminal operator run on the box, so break-even measures the box against what those two roles would cost on serverless, $6.37 per million, rather than against your whole bill. Crediting it with planner and problem-solver spend it never serves would roughly halve the threshold and be wrong.

What moves the threshold

Treat these as central estimates. The assumptions that move them most are per-stream speed, how many agents run concurrently, and how far past working hours they run. Across a reasonable range, break-even sits between 120 and 600 engineers. Add a second node for failover and both figures rise by about half.

Two things the arithmetic does not capture push against owning. GPU lead times of 36 to 52 weeks mean a decision now buys hardware arriving mid-2027 onto a three-year clock. And open-weight pricing keeps falling: Z.ai published GLM-5.3-Flash on 26 August 2026 at $0.15 input and $0.50 output per million. A three-year commitment bets that stops.

Which role belongs where

Roles do not contribute equally to your token bill. From what we see across coding stacks, the doer produces roughly 65% of output tokens, the terminal operator and problem solver about 15% each, and the planner around 5%. Those are observed estimates rather than measurements, but they are what weight the per-role prices into the $6.37 figure the break-even compares against, since the doer and terminal operator are the only two roles the box serves.

Two roles fill the box, two stay on serverless

Consolidating onto one node does not mean every role belongs on it, and for two of them the constraint is physical rather than economic. Kimi K3 is 2.8 trillion parameters and DeepSeek V4 Pro is 1.6 trillion; neither fits an eight-GPU node at any quantization you would want to serve from. The planner and problem solver stay on serverless because they will not fit, not because we prefer it that way.

• The planner is the worst candidate. It loses the most from caching when self-hosted, wastes the most capacity on prefill, and Kimi K3 needs more than one node anyway. Serverless.
• The problem solver runs at low volume by design, and capability matters more than unit cost here than anywhere else. Serverless.
• The terminal operator cannot tolerate preemption, since a stopped session is a broken session rather than a retried one. Owned or dedicated, never interruptible.
• The doer is the volume driver and the most output-weighted role. It is what fills the box.

The box earns its keep on doer traffic, with the terminal operator riding along because it needs somewhere reliable to run. The two roles that stay on serverless are 20% of your tokens and a far larger share of your bill, which is why routing by role beats sending everything one way.

When cost is not the deciding variable

The trade-offs that don’t show up in a price per token

No column wins every row. Which rows you weight is the actual decision.

	Open-weight API	Dedicated capacity	Owned hardware
Cost
Capital required	None	None	High
Cost behaviour	Variable	Fixed	Fixed
Cheapest at	Low to mid volume	Steady mid volume	Very high volume
Cost when idle	None	Full	Full
Capability
Model selection	Provider catalogue	Provider catalogue	Anything you can fit
Upgrades and patching	Automatic	Managed for you	Your project
Headroom for a spike	Effectively unlimited	Capped at reservation	Capped at hardware
Control
Where data sits	Provider’s boundary	Provider’s boundary	Yours
Version pinning	Provider decides	Provider decides	Guaranteed
Uptime accountability	Provider’s SLA	Provider’s SLA	Yours
Deployment time	Minutes	Days	36 to 52 weeks

Read down the column that matches your constraint.

Capability for the job. The cheapest model that clears a benchmark is not always the one that clears your tickets. A benchmark tells you what to test, not what to buy. Cost per accepted change is the metric; cost per token is a proxy that sometimes points the wrong way.

Licence terms, which move. GLM-5.2 shipped under MIT. GLM-5.3 ships under a custom Z.ai licence that permits self-hosting only below a stated annual revenue threshold. Nothing about the weights changed, but the terms you deploy under did, and a model adopted on one basis can arrive at its next version on another. Read the licence at each upgrade, not just at adoption.

Residency and provenance. If your data cannot leave a boundary, the comparison never runs. And some buyers are barred from models of a given national origin regardless of licence or benchmark, which rules out the cheapest and most widely adopted options first. Then the question becomes which model is eligible: Inkling Small is the US-origin, Apache-2.0 option that fits a single node, on the list for reasons unrelated to price per token.

Version pinning and uptime. A hosted model can change underneath a pinned name; a checkpoint on your disk cannot. A shared endpoint’s incident is your incident, on someone else’s timeline.

So which problem are you solving

If your pain is the bill, route by role. Planner and problem solver to serverless, doer and terminal operator onto whatever you own. Around 180 engineers running agents, or 550 on lighter use, one node pays for itself: run the baseline there and burst the overflow to serverless, since fixed capacity cannot absorb a spike. Below that, serverless needs no capital, no lead time, and no on-call rotation.

If your pain is capability, stop optimizing per-token cost and measure accepted-change rate on your own repositories instead.

If your pain is control, break-even is not your decision variable. Price the requirement. The analysis above still tells you the premium you are paying to satisfy it.

Cross the threshold and the next question is migration rather than cost: what it takes to move an existing integration onto an OpenAI-compatible open-weight endpoint. That deserves its own piece.

Run it against your own numbers

Every figure here is an assumption you can replace, and the ones that matter most are yours: your input to output ratio per role, your cache-hit rate, your duty cycle, and how much work runs overnight. Change any of them and the threshold moves.

Wiring it up is its own problem. All three options present an OpenAI-compatible endpoint, so routing by role is a question of pointing each one somewhere different, usually through a gateway. But your agent harness has to ask for different models per role before a gateway has anything to route on, and most of them were not built for that. It is solvable and it is fiddly.

Hoonify runs all three paths, so we have no stake in which one your arithmetic lands on: serverless to start, dedicated when load steadies, owned when headcount justifies it. If you want the break-even run against your own traffic, or help getting the routing to actually work in your harness, we will do it with you. We built the model; bring the inputs.

Frequently asked questions

When is self-hosting an LLM worth it?

Around 2.8 billion output tokens a month of doer and terminal-operator traffic on one owned eight-GPU node, which is roughly 180 engineers running agentic workflows or 550 on lighter copilot use. Add a second node for failover and both rise by about half. Residency, provenance, or version pinning requirements override the arithmetic entirely. Running self-hosted LLM inference in production also means owning the on-call rotation, which is 45% of the fixed cost before a token is served.

How much does an AI coding agent cost to run?

More depends on the role and its model than on either alone. A planner on Kimi K3 works out near $75 for every million output tokens it produces, because at a 20:1 ratio each one drags twenty million input tokens along at $3.00 per million on top of the $15.00 output rate. A doer on Qwen3.8-27B is about $3.60 on the same basis. Caching cuts the planner’s figure by well over half.

What GPU do I need to self-host an LLM?

Roughly one byte per parameter at FP8, plus about 20% for key-value cache, rounded up to a real serving node. A 27B model fits one H100 or H200. A 753B model needs eight H200s, because eight H100s do not have the memory.

---

Blake Kinnan, Co-Founder and VP of Strategic Intelligence at Hoonify, where he works on inference infrastructure and model evaluation for sovereign and on-premises deployments.

Built from published list pricing as of 28 August 2026, MLPerf Inference results, and vendor model cards. List prices vary by provider and change often; use your own. Hardware pricing is an estimate pending quotation; token-ratio and role-mix figures are our own observed estimates. Hoonify serves open-weight models including GLM-5.3, Inkling Small, Qwen3.8-27B, and Gemma 4 31B, and builds dedicated and on-premises clusters.