How to Reduce LLM Inference Costs With Quantization and Batching

Last updated: June 2026

LLM inference cost on self-hosted GPUs is cost per token: you rent the accelerator by the hour, so the bill divided by tokens served gives the unit you must lower. Quantization is converting model weights to a lower-precision format so the model uses less memory and computes faster. Batching is processing many requests together so the GPU does useful work on many sequences per pass instead of idling on one. The two compound: a quantized model fits a cheaper GPU and a batched serving engine fills that GPU, so tokens per dollar can rise several fold with no change to what the user sees.

This article is part of our AI, GPU and ML cluster. For the full picture start with the complete guide to AI and GPU cost optimization, the pillar this piece links up to. The idle-accelerator problem these techniques solve is explained in what is GPU utilization and why idle accelerators cost so much.

Why is self-hosted LLM inference so expensive?

Self-hosted inference is expensive when an underused GPU serves one request at a time. The accelerator is the cost, and a model that processes a single sequence leaves most of the GPU's parallel capacity idle while you pay the full hourly rate. Add an oversized model that demands the largest GPU, and cost per token climbs further. Quantization and batching attack both causes at once: a smaller model needs less GPU, and batching uses the GPU you have far more fully. For instance family selection, see the AWS accelerated computing documentation.

Step 1: How do I measure cost per token?

Divide your GPU instance cost by the tokens it serves per hour to get cost per token, the baseline every change is judged against. Instrument the serving engine to count input and output tokens, pull the instance hourly cost, and compute the unit at current throughput. Record latency at the same time so you can prove later changes did not breach the SLA. This is the See step: without a cost-per-token baseline you cannot tell whether quantization or batching actually helped.

Step 2: How does quantization lower cost?

Quantization converts the model's weights, and sometimes activations, from a higher-precision format to a lower one such as INT8 or FP8, so the model uses less memory and runs faster. A smaller model serves more tokens per second on the same GPU and often fits a cheaper accelerator entirely, both of which cut cost per token. The tradeoff is a small accuracy change, so quantize, then run the model against a quality benchmark representative of your workload and keep the precision level that holds quality within tolerance.

Step 3: How does continuous batching raise throughput?

Continuous batching processes many requests together and adds new ones to the running batch as slots free up, keeping the GPU saturated under variable traffic. Instead of the accelerator working on one sequence and idling, it does useful work on many sequences per pass, so throughput rises sharply and cost per token falls. Modern serving engines implement this as in-flight batching with paged attention for the KV cache. Enabling it is often the single largest throughput gain available on an existing GPU.

Step 4: How do I tune batch size and KV cache?

Raise batch size and configure the KV cache so the GPU stays saturated without exceeding memory or breaching latency. Larger batches improve throughput up to the point where GPU memory or the latency SLA becomes the limit, so sweep batch size and measure both cost per token and tail latency at each setting. The KV cache holds attention state for in-flight sequences and consumes memory that competes with batch size, so size it deliberately. Keep the configuration that gives the lowest cost per token while holding your latency target.

Step 5: How do I right-size the GPU to the quantized model?

Move the smaller, batched model to the cheapest GPU that still holds throughput and latency, because quantization often frees you from the largest accelerator. A model that needed a top-end GPU at full precision may run within SLA on a mid-tier GPU once quantized and batched, and the cheaper instance directly lowers the bill. Load test candidates and keep the least expensive that meets the SLA. The full instance-selection method is in how to right-size GPU instances for model serving.

Frequently asked questions

What is quantization in LLM inference?

Quantization is converting a model's weights and sometimes activations from a higher-precision format to a lower-precision one, such as 16-bit to INT8 or FP8. The smaller model uses less GPU memory and runs faster, so it serves more tokens per dollar and often fits a cheaper GPU. The tradeoff is a small, measurable accuracy change you validate before shipping.

How does batching reduce inference cost?

Batching processes multiple requests together so the GPU does useful work on many sequences per pass instead of sitting underused on one. Continuous batching adds and removes requests from the running batch dynamically, keeping the accelerator saturated. Higher throughput on the same instance means lower cost per token, since the hourly GPU cost is spread across far more tokens.

Does quantization hurt model quality?

Quantization causes a small accuracy change that is often negligible at INT8 or FP8 for most production tasks, but it must be measured. Run the quantized model against a quality benchmark representative of your workload and compare to full precision. Keep the quantization level that cuts cost while holding quality within your tolerance.

What is continuous batching?

Continuous batching, also called in-flight batching, is a serving technique that adds new requests to the GPU batch as soon as slots free up rather than waiting for a fixed batch to finish. It keeps the accelerator busy under variable traffic, raising throughput and lowering cost per token compared with static batching, without adding much latency.

The short version

Reduce LLM inference cost by measuring cost per token, quantizing the model, enabling continuous batching, tuning batch size and KV cache, and right-sizing the GPU. The unit falls because a smaller model on a fuller GPU serves many more tokens per dollar, and every change is validated against quality and latency. When you want that throughput work done and the cost-per-token drop proven, that is exactly what our FinOps implementation service delivers.

Written by Fredrik Filipsson and reviewed by Morten Andersen, applying the See, Cut, Lock, Run method. Independent and vendor neutral.