Find underutilized GPUs and idle accelerators by measuring GPU compute utilization and GPU memory at the device level with NVIDIA DCGM or provider accelerator metrics, not host CPU, which says nothing about whether the GPU is busy. Separate idle accelerators (near-zero utilization, stop them) from underutilized ones (running but well below capacity, move to a smaller or fractional GPU or consolidate). Because accelerator instances cost many times a comparable CPU instance, even brief idle time is costly, so GPU utilization deserves dedicated monitoring. Match hardware to the work and add a continuous check so new idle accelerators are caught fast.
Last updated: June 2026
An underutilized GPU is an accelerator instance that is running but doing far less work than it is sized for, while an idle accelerator is one doing effectively no work at all. Both bill the full, premium hourly rate of GPU hardware regardless of how much of the accelerator is in use. This is the Cut step of FinOps pointed at the single most expensive line on most modern bills: clear idle accelerator spend and rightsize the rest, because a wasted GPU hour costs far more than a wasted CPU hour.
This article is part of our rightsizing and waste elimination cluster. For the full picture start with the complete guide to cloud rightsizing and waste elimination, the pillar this piece links up to. The same metric-first detection applies to databases, covered in how to find idle managed databases across AWS, Azure, and GCP.
How do I measure GPU utilization?
Measure GPU utilization at the device level using NVIDIA DCGM or the cloud provider's accelerator metrics, looking at GPU compute utilization and GPU memory used rather than host CPU. Host CPU is misleading because a GPU instance can show high CPU while the accelerator sits idle, or low CPU while the GPU is saturated. The two device metrics that matter are the percentage of time the GPU is performing compute and how much of its memory is in use. NVIDIA's Data Center GPU Manager (DCGM) exports both, and the major clouds surface equivalent accelerator metrics in their monitoring services.
What is the difference between an idle and an underutilized GPU?
An idle GPU shows near-zero compute utilization, so the accelerator is doing no work and the instance is pure waste, while an underutilized GPU is running real work but well below its capacity. A common underutilized case is a large GPU serving a small model at low throughput, where the work is real but a fraction of the hardware would do. The actions differ: idle GPUs should be stopped, and underutilized GPUs should be moved to a smaller or fractional GPU or consolidated so the work fits the hardware. Telling the two apart is why you need both the compute and memory metrics, not a single threshold.
How do I find underutilized GPUs, step by step?
Follow these six steps to find idle and underused accelerators and match each to the right fix.
- Inventory every accelerator instance. List all GPU and accelerator instances across regions and accounts with their type, device count, and hourly cost, so you know where the accelerator spend actually sits.
- Instrument GPU utilization. Collect GPU compute utilization and GPU memory used per device with DCGM or provider accelerator metrics, never relying on host CPU as a proxy.
- Distinguish idle from underutilized. Separate accelerators near zero utilization, which are idle, from those running but well below capacity, which are underutilized, because they need different actions.
- Find the cause. For each, identify whether it is an abandoned instance, an oversized GPU for the model, a low-throughput endpoint, or a data-loading bottleneck starving the GPU. The cause determines the fix.
- Choose the action. Stop idle instances, move to a smaller or fractional GPU where the model fits, consolidate low-traffic endpoints onto shared accelerators, or fix input pipelines that leave the GPU waiting on data.
- Add continuous accelerator monitoring. Schedule the utilization check and alert so newly idle or underused accelerators are caught quickly, which matters more here than anywhere because of the hourly cost.
GPUs billing premium rates while sitting half empty?
Our cost audit instruments device-level GPU utilization across your fleet, separates idle from underutilized, and right-sizes, consolidates, or stops each one so you pay for accelerators that actually work. On the performance model, you pay only from realized savings. No savings, no fee.
Book a cloud cost audit →Why are idle GPUs so much more expensive than idle servers?
GPU and accelerator instances are among the most expensive compute on any cloud, frequently many times the hourly rate of a comparable CPU instance, so an idle GPU wastes that premium every hour it runs empty. An accelerator left running after a job finishes, or a large GPU serving a tiny model, burns the same high rate as one doing useful work. That cost asymmetry is the reason GPU utilization deserves its own monitoring rather than being folded into general compute dashboards, and why even a small reduction in idle accelerator time moves the bill noticeably. Verify the exact rates for your instances against current vendor pricing, since accelerator pricing changes often.
What should I do with an underutilized GPU?
Match the hardware to the work rather than leaving a large accelerator mostly empty. Move a model that fits to a smaller GPU instance type or a fractional GPU, consolidate several low-traffic inference endpoints onto shared accelerators, or fix a data-loading bottleneck that is starving the GPU so it does more work per hour. For training, use spot capacity or job queues to keep accelerators busy between runs. The objective is high, steady utilization on right-sized hardware, which is the same goal that drives all of rightsizing and waste elimination, applied to the costliest hardware you run.
The Cloud Waste Audit Framework includes the accelerator utilization thresholds and the idle-versus-underutilized decision tree for GPU fleets. It is the downloadable companion to this article.
Frequently asked questions
How do I measure GPU utilization?
Measure GPU utilization at the device level with NVIDIA DCGM or the cloud provider's accelerator metrics, looking at GPU compute utilization and GPU memory used, not host CPU. Host CPU tells you nothing about whether the GPU is busy, because a GPU instance can show high CPU while the accelerator sits idle. The two device metrics that matter are the percentage of time the GPU is doing compute and how much of its memory is in use.
What is the difference between an idle and an underutilized GPU?
An idle GPU shows near-zero compute utilization, meaning the accelerator is doing no work and the instance is pure waste. An underutilized GPU is running real work but well below its capacity, for example a large GPU serving a small model at low throughput. Idle GPUs should be stopped; underutilized GPUs should be moved to a smaller or fractional GPU, or consolidated, so the work fits the hardware.
Why are idle GPUs so expensive?
GPU and accelerator instances are among the most expensive compute on any cloud, often many times the hourly rate of a comparable CPU instance, so even a short period of idle time costs far more than an idle CPU server. An accelerator left running with no work, or a large GPU serving a tiny model, wastes that premium rate every hour. That cost asymmetry is why GPU utilization deserves dedicated monitoring rather than being lumped in with general compute.
What should I do with an underutilized GPU?
Match the hardware to the work. Move a model that fits to a smaller GPU instance type or a fractional GPU, consolidate several low-traffic inference endpoints onto shared accelerators, or fix a data-loading bottleneck that is starving the GPU so it does more work per hour. For training, use spot or queue jobs to keep accelerators busy. The goal is high, steady utilization on right-sized hardware rather than a large GPU sitting mostly empty.
The short version
Find underutilized GPUs and idle accelerators by measuring device-level GPU compute and memory with DCGM or provider metrics, separating idle from underutilized, finding the cause, and matching the action: stop, rightsize, consolidate, or fix the pipeline. Because accelerator hours are the most expensive on the bill, the payoff is large. When you want your GPU fleet instrumented and right-sized, that is exactly what our rightsizing and waste elimination service delivers.
Written by Morten Andersen and reviewed by Fredrik Filipsson, applying the See, Cut, Lock, Run method. Independent and vendor neutral.
Cloud pricing and service behavior change frequently. Verify the specifics in this guide against the providers’ own current documentation and the FinOps Foundation: FinOps Foundation Framework ↗ and FinOps Rate Optimization capability ↗. This article also reflects Cloud Cost Room’s hands-on, vendor-neutral engagement experience.