Guides → Playground & Guide → Inference Serving Cost Calculator - Self-Host $/1M Tokens vs API

Inference Serving Cost Calculator - Self-Host $/1M Tokens vs API

🚀 Open the calculator

Meet Marcus Reyes. Platform lead weighing moving a high-volume summarization workload off the API onto self-hosted GPUs. "At our volume, is it actually cheaper to serve an open model on our own H100s than to keep paying per token - and where's the break-even?"

🔥 Finance sees the API line growing every month and assumes self-hosting is the fix. Nobody has modeled what idle GPUs cost when traffic is bursty.

The story

Self-host cost is GPU $/hr divided by throughput. A GPU box bills the same whether it is saturated or idle, so the real per-token cost depends entirely on how fast you serve and how busy you keep it.

Utilization is the trap. The same H100 that looks cheap at 80% utilization is brutally expensive at 15% - you are paying for compute you are not using. Bursty, interactive traffic is exactly where self-host economics quietly break.

Marcus's workload: on an A10G box at ~45% utilization, self-hosting lands well under frontier API output prices - but a commodity API (DeepSeek-class) pushes the break-even volume so high it almost never wins. The model you compare against decides the answer.

🎮 Playground

Inference Serving Cost Calculator Playground

Here are the inputs that move the result the most. Play with the sliders and check it out. The number updates live.

What does self-hosted inference really cost per token?

Estimate the effective $/1M output tokens of serving an open model on your own GPUs (8×A10G default) — utilization-adjusted, the number that actually lands on your bill.

Effective $/1M output tokens (self-host)
—

💡Self-host cost is GPU $/hr ÷ throughput. Utilization is the killer: idle GPUs still bill, so dropping from 80% to 20% utilization quadruples your real per-token cost.

Ready to run the numbers?

Open the full calculator. Pick a model, enter your tokens, see per-call, daily, monthly, and annual cost.

🚀 Open the full calculator →

Top 3 right now

Verified 13 hours ago

About this calculator: Inference Serving Cost Calculator - Self-Host $/1M Tokens vs API

Estimate the effective dollar-per-million-output-tokens of serving an open model on your own GPUs - utilization-adjusted - plus the monthly bill and the token volume where self-hosting beats an API.

🎛 Inputs you control

Each input shapes the cost. Click an input on the calculator to set it. The explanations below match the live calculator field by field.

▸ GPU instance: The GPU instance type the model is served on.
How to choose: Match your serving hardware; A10G is cost-effective for smaller models, H100 for large/high-throughput.
▸ Commitment: On-demand, reserved (1yr), or spot pricing for the instance.
How to choose: Reserved cuts steady-serving cost; spot is cheapest but can be reclaimed mid-request.
▸ Instances: How many of this instance you run in parallel.
How to choose: Sets capacity and the monthly bill; per-token cost is unchanged by count.
▸ Sustained output tok/s: Measured output tokens/sec the deployment sustains at full load (per instance).
How to choose: Measure on your own traffic; vLLM on A10G ~2.5K, H100 ~8-10K for a small model. It directly divides cost.
▸ GPU utilization %: Share of provisioned time the GPUs actually serve requests.
How to choose: Steady production ~50-80%; bursty/interactive far lower. Idle time still bills, so be honest.
▸ Output tokens / month (M): Your workload volume in millions of OUTPUT tokens per month.
How to choose: Drives the API comparison and shows whether you fill the box capacity or pay for idle.
▸ Compare vs API model: The API model the break-even is computed against (its $/1M output).
How to choose: Pick what you would otherwise call. Frontier models make self-host look great; commodity models are the hard bar.
📋 Typical values & starting points Don’t know a value yet? Start with these broad, sourced ballparks — the calculator’s ▾ Typical menus pre-load the same options.

Context: Throughput is the whole game in self-serving: a tuned vLLM stack on H100 delivers ~2,200-2,400 tok/s for 70B FP8 at high concurrency (3-4x more for 8B-class). Delivered cost per token = GPU rate / (throughput x utilization) - low utilization quietly multiplies cost 3-5x.

Input Default Typical ballparks
throughputTokSec moves the needle 2,500 Naive / large model · 800 = 800 · Tuned vLLM 70B-class · 2,500 = 2,500 · Optimized batch / small model · 6K = 6,000
utilizationPct moves the needle 45 Real-world low · 20% = 20 · Typical · 45% = 45 · Well-batched · 70% = 70
monthlyOutputTokensM 500 Growing product · 100M = 100 · Serious volume · 500M = 500 · Platform scale · 2,000M = 2,000
instanceCount 1 Single node · 1 = 1 · Small cluster · 4 = 4 · 8-GPU node · 8 = 8
gpuKey aws:g5.48xlarge —
commitment onDemand —
compareApiModel claude-haiku-4-5 —

Ballparks are broad industry starting points (sourced ranges; * = rough estimate) — your result gets more accurate as you replace them with measured numbers. Try them live in the calculator; API & agent users get the same data from the MCP resource aicost://input-reference/inference-serving-cost.

Most popular

Everything above is the 80% case. The last 20% is where the money is.

The gaps we just listed are real, and they are the expensive ones: your actual prompts, your switching cost, your MLOps overhead. An AICost expert spends the hour on your AI and cloud costs, not a generic playbook. You leave with a written report: the way forward, in 30 days of concrete steps.

  • An hour with the people who built the engines
  • Report the same day
  • The fee credits toward any AICost plan
  • Two slots a week
Book a Solution Session: $299 → or $99 for small business →

Not sure yet? The $39 AICost Blueprint credits toward a Session, and the Session fee credits toward any plan. You never pay twice for the same ground. See all pricing →

Ready to run your own numbers?

You have seen the shape of it. Open the calculator with your model, your tokens, your volume.

🚀 Open the full calculator →

Methodology

Editorial gate
8-layer defense, see aicost.ai/ai-cost-economics
Last verified
7/20/2026, 8:00:00 PM

Author: Subu Vdaygiri, Founder & CEO of CloudIntelligence.ai. 17 years Fortune 100 (Ingram Micro, Siemens). Wharton CTO program · Kellogg CPO program · 10× AWS+Azure certified.

📖 Data sources & methodology 158 text models · 9 embeddings · 37 vision · 55 audio · 8 vector DBs across 10 vendor pages · last verified 2026-07-21

Methodology

  • All prices are USD per 1 million tokens, current as of 2026-07-21.
  • Vendor-published values have no mark. Inferred/extrapolated values are marked with * and listed below.
  • Batch API discounts are 50% off standard rates across providers that offer Batch mode.
  • Prompt caching discounts vary by provider (typically 80-90% off cached input tokens).
  • Regional data-residency surcharges (Anthropic 1.1x, OpenAI 1.1x, Google regional tiers) are NOT included in base rates.
  • Long-context pricing tiers apply when input exceeds model threshold.
  • Embedding prices are input-only (no output tokens generated).

Primary sources

Last-verified date is the most recent successful daily snapshot (aicost_pricing_snapshots) or, when no snapshot exists yet, the latest successful crawler run (aicost_crawler_runs). 10 of 10 vendors are currently verified. Aggregator services (TokenCost, AI Pricing Guru, etc.) are not listed.

Anthropic
2026-07-21
https://www.anthropic.com/pricing
Daily snapshot since Sep 2023 · 624 days captured
Anthropic Docs
2026-07-21
https://platform.claude.com/docs/en/about-claude/pricing
Daily snapshot since Sep 2023 · 624 days captured
OpenAI
2026-07-21
https://openai.com/api/pricing/
Daily snapshot since Sep 2023 · 625 days captured
Google AI
2026-07-21
https://ai.google.dev/gemini-api/docs/pricing
Daily snapshot since Dec 2023 · 600 days captured
Google Vertex
2026-07-21
https://cloud.google.com/vertex-ai/generative-ai/pricing
Daily snapshot since Dec 2023 · 600 days captured
DeepSeek
2026-07-21
https://api-docs.deepseek.com/quick_start/pricing
Daily snapshot since May 2024 · 539 days captured
xAI
2026-07-21
https://x.ai/api
Daily snapshot since Nov 2024 · 457 days captured
Mistral
2026-07-21
https://mistral.ai/pricing
Daily snapshot since Dec 2023 · 598 days captured
Cohere
2026-07-21
https://cohere.com/pricing
Daily snapshot since Sep 2023 · 624 days captured
Voyage AI
2026-07-21
https://docs.voyageai.com/docs/pricing

Inferred values (marked with * in calculator tables)

Derived from industry conventions, not directly published by the vendor. Typical conventions: cached input = 10% of base (90% off), Batch API = 50% of base (50% off).

Vendor / Model Field Why it’s inferred
Anthropic — Claude Sonnet 4.6 cachedInput Derived at 10% of input rate — Anthropic publishes 90% cache-hit discount on this tier.
Anthropic — Claude Sonnet 4.5 cachedInput Derived at 10% of input rate; same 90% cache-hit convention as Sonnet 4.6.
Anthropic — Claude Sonnet 4.5 batchInput Derived at 50% of standard input — Anthropic documents uniform 50% Batch discount.
Anthropic — Claude Sonnet 4.5 batchOutput Derived at 50% of standard output — Anthropic documents uniform 50% Batch discount.
Anthropic — Claude Haiku 4.5 cachedInput Derived at 10% of input rate — Anthropic 90% cache-hit discount convention.
OpenAI — GPT-5.4 Mini cachedInput Derived at 10% of input — OpenAI documents automatic 90% discount on cache hits across GPT-5.x tier.
OpenAI — GPT-5.4 Nano cachedInput Derived at 10% of input — OpenAI 90% cache-hit convention.
OpenAI — GPT-5.4 Nano batchInput Derived at 50% of input — OpenAI Batch API uniform 50% discount.
OpenAI — GPT-5.4 Nano batchOutput Derived at 50% of output — OpenAI Batch API uniform 50% discount.
OpenAI — GPT-5.4 Pro cachedInput Derived at 10% of input — OpenAI 90% cache-hit convention.
OpenAI — GPT-5.4 Pro batchInput Derived at 50% of input — OpenAI Batch API uniform 50% discount.
OpenAI — GPT-5.4 Pro batchOutput Derived at 50% of output — OpenAI Batch API uniform 50% discount.
OpenAI — GPT-5.2 cachedInput Derived at 10% of input; no residency uplift.
OpenAI — GPT-5.2 batchInput Derived at 50% of input.
OpenAI — GPT-5.2 batchOutput Derived at 50% of output.
OpenAI — GPT-5 cachedInput Derived at 10% of input.
OpenAI — GPT-5 batchInput Derived at 50% of input.
OpenAI — GPT-5 batchOutput Derived at 50% of output.
OpenAI — GPT-5.5 Pro cachedInput Derived at 10% of input — OpenAI does not publish a cached rate for *-pro models; using the family convention.
OpenAI — GPT-5.5 Pro batchInput Derived at 50% of input.
OpenAI — GPT-5.5 Pro batchOutput Derived at 50% of output.
OpenAI — GPT-5.2 Pro cachedInput Derived at 10% of input — pro-tier convention.
OpenAI — GPT-5.2 Pro batchInput Derived at 50% of input.
OpenAI — GPT-5.2 Pro batchOutput Derived at 50% of output.
OpenAI — GPT-5.1 batchInput Derived at 50% of input.
OpenAI — GPT-5.1 batchOutput Derived at 50% of output.
OpenAI — GPT-5 Pro batchInput Derived at 50% of input.
OpenAI — GPT-5 Pro batchOutput Derived at 50% of output.
OpenAI — GPT-5 Nano cachedInput Derived at 10% of input.
OpenAI — GPT-5 Nano batchInput Derived at 50% of input.
OpenAI — GPT-5 Nano batchOutput Derived at 50% of output.
Google — Gemini 3 Flash cachedInput Derived at 10% of input — Google caching discount convention ~90%.
Google — Gemini 3.1 Flash-Lite cachedInput Derived at 10% of input — Google caching convention.
Google — Gemini 3.1 Flash-Lite batchInput Derived at 50% of input — Google Batch API uniform 50% discount.
Google — Gemini 3.1 Flash-Lite batchOutput Derived at 50% of output — Google Batch API uniform 50% discount.
Google — Gemini 2.5 Pro cachedInput Derived at 10% of input.
Google — Gemini 2.5 Flash cachedInput Derived at 10% of input.
Google — Gemini 2.5 Flash-Lite cachedInput Derived at 10% of input — Google caching convention.
Google — Gemini 2.5 Flash-Lite batchInput Derived at 50% of input — Google Batch API uniform 50% discount.
Google — Gemini 2.5 Flash-Lite batchOutput Derived at 50% of output — Google Batch API uniform 50% discount.
Google — Gemini 2.0 Flash cachedInput Derived at 25% of input per Google 2.0 family caching rates.
Google — Gemini 2.0 Flash batchInput Derived at 50% of input — Google Batch API uniform 50% discount.
Google — Gemini 2.0 Flash batchOutput Derived at 50% of output — Google Batch API uniform 50% discount.
Google — Gemini 2.0 Flash-Lite cachedInput Derived at 10% of input — Google caching convention.
Google — Gemini 2.0 Flash-Lite batchInput Derived at 50% of input — Google Batch API uniform 50% discount.
Google — Gemini 2.0 Flash-Lite batchOutput Derived at 50% of output — Google Batch API uniform 50% discount.
xAI — Grok 4 (legacy) cachedInput Extrapolated at 25% of base.

Pricing is cross-verified against the LiteLLM community registry when available. Daily snapshots are kept in aicost_pricing_snapshots; every change is logged to aicost_price_changelog with old & new values for full audit trail. Read the full methodology →