Guides → Playground & Guide → Local vs Cloud Inference - Serverless vs GPU Rental vs Self-Host
Meet Wei Zhang. Platform lead deciding how to serve a fine-tuned open-weight model in production. "We've outgrown casual serverless use. Do we stay on a per-token host, rent a dedicated GPU, or stand it up on our own AWS account?"
🔥 Three teams have three opinions and three spreadsheets. Nobody has put serverless, rented GPU, and own-cloud on the same axis at our real volume.
Serving an open-weight model is a three-way choice, not a binary. You can pay per token to a serverless host, rent a dedicated GPU by the hour, or run it on your own cloud account. Each is a different cost curve, and they cross at different volumes depending on how cheap the model's tokens are and how well you utilize a GPU.
Wei's model is past the experimentation phase. The serverless bill is climbing, but a dedicated GPU sits idle outside peak hours, and an own-cloud box adds ops the team has to own. The only way to decide is to price all three on the same workload - which is exactly what this does.
The deciding variables are volume, throughput and utilization. A GPU that serves 3,000 tokens/sec at 45% utilization has a very different effective cost than one at 80%. The calc sizes the GPU fleet for your volume, then compares the monthly cost of all three paths so the cheapest is a number, not a hunch.
It composes the same price sources as the rest of the fleet - the curated host table, the LiteLLM feed, and the resource-pricing GPU source of truth - so the three legs are consistent with the other calculators.
Open the full calculator. Pick a model, enter your tokens, see per-call, daily, monthly, and annual cost.
🚀 Open the full calculator →For an open-weight model and your workload, compare the three ways to serve it - serverless host, dedicated GPU rental, and self-hosting on your own cloud - and see which is cheapest at your volume.
| Input | Default | Typical ballparks |
|---|---|---|
throughputTokSec
|
2,500 | Naive / large model · 800 = 800 · Tuned vLLM 70B-class · 2,500 = 2,500 · Optimized batch / small model · 6K = 6,000 |
utilizationPct
|
45 | Real-world low · 20% = 20 · Typical · 45% = 45 · Well-batched · 70% = 70 |
monthlyOutputTokensM
|
500 | Growing product · 100M = 100 · Serious volume · 500M = 500 · Platform scale · 2,000M = 2,000 |
inputPct
|
25 | Generation-heavy · 15% = 15 · Typical chat · 25% = 25 · RAG / context-heavy · 40% = 40 |
family
|
glm | — |
gpuKey
|
aws:g5.48xlarge | — |
commitment
|
onDemand | — |
Ballparks are broad industry starting points (sourced ranges; * = rough estimate) — your result gets more accurate as you replace them with measured numbers.
Try them live in the calculator;
API & agent users get the same data from the MCP resource aicost://input-reference/local-vs-cloud-inference.
The winner depends on the model's token price. A cheap model (DeepSeek Flash, GPT-OSS) stays on serverless far longer; an expensive one crosses to dedicated GPUs at lower volume. Read the verdict for your actual model.
Utilization makes or breaks the GPU legs. Idle GPUs still bill. If your traffic is bursty and utilization is low, serverless usually wins; steady high utilization favors owning or renting.
Own-cloud is not automatically cheapest. It can be, with reserved or spot commitment and high utilization, but it adds ops you own. The number shows the cost; the overhead is yours to weigh.
Honest limitations. Every model is wrong; some are useful. Where this one falls short:
For these, use: Inference Serving Cost for a deeper own-cloud serving model. Hosting Cost Compare for the full serverless host spread.
The gaps we just listed are real, and they are the expensive ones: your actual prompts, your switching cost, your MLOps overhead. An AICost expert spends the hour on your AI and cloud costs, not a generic playbook. You leave with a written report: the way forward, in 30 days of concrete steps.
Not sure yet? The $39 AICost Blueprint credits toward a Session, and the Session fee credits toward any plan. You never pay twice for the same ground. See all pricing →
You have seen the shape of it. Open the calculator with your model, your tokens, your volume.
🚀 Open the full calculator →The full per-token spread across DeepInfra, Fireworks, Together, Novita, Groq.
Deep self-host math →Detailed own-cloud serving economics with utilization and commitment.
Is the host OK for your data? →Score the data-jurisdiction risk of the model/host for your data class.
Author: Subu Vdaygiri, Founder & CEO of CloudIntelligence.ai. 17 years Fortune 100 (Ingram Micro, Siemens). Wharton CTO program · Kellogg CPO program · 10× AWS+Azure certified.
Last-verified date is the most recent successful daily snapshot
(aicost_pricing_snapshots) or, when no snapshot exists yet,
the latest successful crawler run (aicost_crawler_runs).
10 of 10
vendors are currently verified. Aggregator services (TokenCost, AI Pricing Guru, etc.)
are not listed.
Derived from industry conventions, not directly published by the vendor. Typical conventions: cached input = 10% of base (90% off), Batch API = 50% of base (50% off).
| Vendor / Model | Field | Why it’s inferred |
|---|---|---|
| Anthropic — Claude Sonnet 4.6 | cachedInput |
Derived at 10% of input rate — Anthropic publishes 90% cache-hit discount on this tier. |
| Anthropic — Claude Sonnet 4.5 | cachedInput |
Derived at 10% of input rate; same 90% cache-hit convention as Sonnet 4.6. |
| Anthropic — Claude Sonnet 4.5 | batchInput |
Derived at 50% of standard input — Anthropic documents uniform 50% Batch discount. |
| Anthropic — Claude Sonnet 4.5 | batchOutput |
Derived at 50% of standard output — Anthropic documents uniform 50% Batch discount. |
| Anthropic — Claude Haiku 4.5 | cachedInput |
Derived at 10% of input rate — Anthropic 90% cache-hit discount convention. |
| OpenAI — GPT-5.4 Mini | cachedInput |
Derived at 10% of input — OpenAI documents automatic 90% discount on cache hits across GPT-5.x tier. |
| OpenAI — GPT-5.4 Nano | cachedInput |
Derived at 10% of input — OpenAI 90% cache-hit convention. |
| OpenAI — GPT-5.4 Nano | batchInput |
Derived at 50% of input — OpenAI Batch API uniform 50% discount. |
| OpenAI — GPT-5.4 Nano | batchOutput |
Derived at 50% of output — OpenAI Batch API uniform 50% discount. |
| OpenAI — GPT-5.4 Pro | cachedInput |
Derived at 10% of input — OpenAI 90% cache-hit convention. |
| OpenAI — GPT-5.4 Pro | batchInput |
Derived at 50% of input — OpenAI Batch API uniform 50% discount. |
| OpenAI — GPT-5.4 Pro | batchOutput |
Derived at 50% of output — OpenAI Batch API uniform 50% discount. |
| OpenAI — GPT-5.2 | cachedInput |
Derived at 10% of input; no residency uplift. |
| OpenAI — GPT-5.2 | batchInput |
Derived at 50% of input. |
| OpenAI — GPT-5.2 | batchOutput |
Derived at 50% of output. |
| OpenAI — GPT-5 | cachedInput |
Derived at 10% of input. |
| OpenAI — GPT-5 | batchInput |
Derived at 50% of input. |
| OpenAI — GPT-5 | batchOutput |
Derived at 50% of output. |
| OpenAI — GPT-5.5 Pro | cachedInput |
Derived at 10% of input — OpenAI does not publish a cached rate for *-pro models; using the family convention. |
| OpenAI — GPT-5.5 Pro | batchInput |
Derived at 50% of input. |
| OpenAI — GPT-5.5 Pro | batchOutput |
Derived at 50% of output. |
| OpenAI — GPT-5.2 Pro | cachedInput |
Derived at 10% of input — pro-tier convention. |
| OpenAI — GPT-5.2 Pro | batchInput |
Derived at 50% of input. |
| OpenAI — GPT-5.2 Pro | batchOutput |
Derived at 50% of output. |
| OpenAI — GPT-5.1 | batchInput |
Derived at 50% of input. |
| OpenAI — GPT-5.1 | batchOutput |
Derived at 50% of output. |
| OpenAI — GPT-5 Pro | batchInput |
Derived at 50% of input. |
| OpenAI — GPT-5 Pro | batchOutput |
Derived at 50% of output. |
| OpenAI — GPT-5 Nano | cachedInput |
Derived at 10% of input. |
| OpenAI — GPT-5 Nano | batchInput |
Derived at 50% of input. |
| OpenAI — GPT-5 Nano | batchOutput |
Derived at 50% of output. |
| Google — Gemini 3 Flash | cachedInput |
Derived at 10% of input — Google caching discount convention ~90%. |
| Google — Gemini 3.1 Flash-Lite | cachedInput |
Derived at 10% of input — Google caching convention. |
| Google — Gemini 3.1 Flash-Lite | batchInput |
Derived at 50% of input — Google Batch API uniform 50% discount. |
| Google — Gemini 3.1 Flash-Lite | batchOutput |
Derived at 50% of output — Google Batch API uniform 50% discount. |
| Google — Gemini 2.5 Pro | cachedInput |
Derived at 10% of input. |
| Google — Gemini 2.5 Flash | cachedInput |
Derived at 10% of input. |
| Google — Gemini 2.5 Flash-Lite | cachedInput |
Derived at 10% of input — Google caching convention. |
| Google — Gemini 2.5 Flash-Lite | batchInput |
Derived at 50% of input — Google Batch API uniform 50% discount. |
| Google — Gemini 2.5 Flash-Lite | batchOutput |
Derived at 50% of output — Google Batch API uniform 50% discount. |
| Google — Gemini 2.0 Flash | cachedInput |
Derived at 25% of input per Google 2.0 family caching rates. |
| Google — Gemini 2.0 Flash | batchInput |
Derived at 50% of input — Google Batch API uniform 50% discount. |
| Google — Gemini 2.0 Flash | batchOutput |
Derived at 50% of output — Google Batch API uniform 50% discount. |
| Google — Gemini 2.0 Flash-Lite | cachedInput |
Derived at 10% of input — Google caching convention. |
| Google — Gemini 2.0 Flash-Lite | batchInput |
Derived at 50% of input — Google Batch API uniform 50% discount. |
| Google — Gemini 2.0 Flash-Lite | batchOutput |
Derived at 50% of output — Google Batch API uniform 50% discount. |
| xAI — Grok 4 (legacy) | cachedInput |
Extrapolated at 25% of base. |
Pricing is cross-verified against the
LiteLLM community registry
when available. Daily snapshots are kept in aicost_pricing_snapshots;
every change is logged to aicost_price_changelog with old & new
values for full audit trail. Read the full methodology →