Right-size GPU instances for model serving in six steps: define the latency SLA and traffic profile, profile the model's GPU memory footprint, load test candidate GPUs from the smallest up, apply batching and quantization before sizing up, match the instance shape to the GPU count actually needed, and commit only the steady baseline. The right GPU is the cheapest one that holds the SLA at target concurrency, not the most powerful available. Tune the serving stack first, because batching and quantization often let a smaller, cheaper instance meet the same target.
Last updated: June 2026
GPU right-sizing for model serving is choosing the cheapest accelerator instance that meets your latency SLA at the concurrency you need, while fitting the model and its KV cache in GPU memory. The cost of an inference endpoint is the GPU you rent by the hour, so an instance with more memory and compute than the model uses is direct waste. Right-sizing reverses the common default of reaching for the largest GPU: you start from the smallest plausible accelerator, raise its effective throughput with batching and quantization, and move up only until the latency target is met.
This article is part of our AI, GPU and ML cluster. For the full picture start with the complete guide to AI and GPU cost optimization, the pillar this piece links up to. To understand why oversized accelerators are so costly, read what is GPU utilization and why idle accelerators cost so much.
How do I know if a GPU instance is oversized?
A GPU instance is oversized when it meets the latency SLA with large headroom in both memory and utilization. If the model uses a fraction of the GPU's memory and the accelerator stays well below saturation at peak traffic, a smaller GPU will likely hold the same SLA for less. Check GPU memory used versus available and accelerator utilization at peak; persistent low numbers on a large instance are the signal to test the next size down. AWS lists accelerated instance families and their memory in the accelerated computing documentation.
Step 1: How do I define the latency SLA and traffic profile?
Write down the latency target and the concurrency the endpoint must handle, because every instance is judged against that bar. Capture the p95 or p99 latency the product requires, the steady and peak requests per second, and the token lengths typical of your traffic. Without this, instance choices are guesses. This is the See step: a clear SLA and traffic profile make the rest of the process a measurement rather than an opinion.
Step 2: How do I profile the model's GPU memory footprint?
Measure how much GPU memory the model weights and the KV cache need at your batch size and sequence length, because memory sets the floor on which accelerators are even viable. A model that needs more memory than a GPU offers will not run there; one that fits with large headroom signals you can drop to a smaller GPU. Profile at realistic batch and context sizes, since the KV cache grows with both. This rules in and out candidate instances before you spend time load testing.
Step 3: How do I load test candidate GPUs?
Benchmark the smallest plausible GPU first and move up only until the latency SLA is met at target concurrency. Run representative traffic against each candidate, record p95 and p99 latency and cost per thousand requests, and stop at the first instance that holds the SLA. Starting small and moving up, rather than starting large and hoping to trim, is what keeps you from defaulting to an oversized accelerator. Keep the cheapest instance that passes.
Serving models on GPUs far larger than they need?
Our cost audit profiles each model, tunes batching and quantization, load tests candidate accelerators against your latency SLA, and right-sizes the fleet, then proves the saving before anything changes in production. On the performance model, you pay only from realized savings. No savings, no fee.
Book a cloud cost audit →Step 4: Why apply batching and quantization before sizing up?
Apply continuous batching and quantization before reaching for a bigger GPU, because both raise the throughput a given accelerator delivers and often let a smaller instance hold the SLA. Batching keeps the GPU saturated across many concurrent requests, and quantization shrinks the model so it computes faster and fits a cheaper accelerator. Sizing the GPU before tuning the serving stack locks in a larger instance than the workload needs. The full technique is in how to reduce LLM inference costs with quantization and batching.
Step 5: How do I match instance shape to GPU count?
Pick single-GPU instances for single-GPU models and reserve multi-GPU shapes for models that genuinely span GPUs, because paying for extra accelerators a model cannot use is pure waste. Many models fit on one GPU after quantization yet get deployed on multi-GPU instances by habit. Confirm whether the model actually requires tensor or pipeline parallelism across GPUs; if not, the single-GPU shape is cheaper and simpler. Consolidating several small models onto one instance, where memory allows, removes the one-instance-per-model tax.
Step 6: When should I commit the baseline?
Commit only the steady, always-on capacity after the workload is right-sized, so you discount a clean baseline instead of locking in waste. Once the fleet shows a persistent floor, cover it with Savings Plans or committed use discounts and leave variable peaks on on-demand with autoscaling. Reservations last, on a clean baseline, is the Cut sequence. The managed-endpoint version of this is in how to optimize SageMaker and Vertex AI endpoint costs.
The AI and GPU Cost Control Guide includes our GPU right-sizing worksheet for inference, mapping model footprint to accelerator choice. It is the downloadable companion to this article.
Frequently asked questions
How do I know if a GPU instance is oversized for inference?
A GPU instance is oversized when it meets the latency SLA with large headroom in GPU memory and compute utilization. If the model uses a fraction of available GPU memory and the accelerator sits well below saturation at peak traffic, a smaller GPU will likely hold the same SLA at lower cost. Load test the next size down to confirm.
What determines the right GPU for model serving?
The right GPU is the cheapest one that holds your latency SLA at target concurrency while fitting the model and its KV cache in memory. The decision is set by model memory footprint, required throughput, and the latency target, not by picking the most powerful accelerator available. Quantization and batching often let a smaller GPU meet the same SLA.
Should I right-size before or after batching?
Apply batching and quantization first, then right-size. Both raise the throughput a given GPU can deliver, which often lets a smaller, cheaper instance meet the latency SLA. Sizing the GPU before tuning the serving stack locks you into a larger accelerator than the workload actually needs.
When should I buy a commitment for GPU serving?
Buy a commitment only after the workload is right-sized and shows a stable always-on baseline. Cover that persistent floor with Savings Plans or committed use discounts and keep variable peaks on on-demand with autoscaling. Committing before right-sizing buys a discount on capacity you should have removed.
The short version
Right-size GPU instances for model serving by defining the SLA, profiling memory, load testing from the smallest GPU up, tuning batching and quantization first, matching the instance shape to real GPU count, and committing only the stable baseline. The right GPU is the cheapest that holds the SLA, not the largest available. When you want that right-sizing done and the saving proven, that is exactly what our FinOps implementation service delivers.
Written by Fredrik Filipsson and reviewed by Morten Andersen, applying the See, Cut, Lock, Run method. Independent and vendor neutral.
Cloud pricing and service behavior change frequently. Verify the specifics in this guide against the providers’ own current documentation and the FinOps Foundation: FinOps Foundation Framework ↗ and FinOps Rate Optimization capability ↗. This article also reflects Cloud Cost Room’s hands-on, vendor-neutral engagement experience.