Self-initiated · AI infrastructure

Forge

An LLM inference cost estimator. Pick a model, a GPU, and your traffic — get GPU count, latency, and the monthly bill. The same back-of-envelope math used before any serious serving deployment.

GPUs required

–

Est. monthly cost

–

p50 latency / req

–

Cost / 1M tokens

–

Memory per replica

Weights KV cache Headroom

How the estimate works

Decode is memory-bandwidth bound: tokens/sec per GPU ≈ HBM bandwidth × 55% ÷ model bytes. An H100 does ~24 tok/s on a 70B FP16 model — matching published vLLM numbers.
KV cache per token ≈ 2 × layers × hidden × bytes, with depth/width scaled from parameter count. It usually dominates memory at large batch.
Replicas = ceil(daily output tokens ÷ 86400 ÷ per-replica throughput). GPUs per replica grow until weights + KV cache fit in 90% of HBM.
Latency ≈ prefill (compute-bound guess) + output tokens ÷ per-request decode rate + 150ms overhead.

ROUGH ESTIMATES ONLY — real throughput depends on the serving engine (vLLM, TensorRT-LLM, SGLang), continuous batching, prefix caching, and your traffic shape. Validate with a benchmark before signing anything.