Use spot GPUs for AI training in six steps: confirm the job is interruption-tolerant, write frequent checkpoints to durable storage, catch the interruption notice and write a final checkpoint, make the job resume automatically from the latest checkpoint, diversify instance types and zones so one capacity loss does not stall you, and use managed spot training where available. Spot GPUs run at up to 90% off on-demand, so the only real risk is lost progress, which frequent checkpointing and clean resume eliminate. Done right, spot is one of the largest cost levers in AI training.
Last updated: June 2026
A spot GPU is spare cloud accelerator capacity sold at a steep discount in exchange for the provider being able to reclaim it with short notice. On AWS, spot instances are available at up to a 90% discount compared to on-demand prices, with a two-minute interruption warning before reclaim, as documented on the Amazon EC2 Spot Instances page. AI training is a strong fit because most training is batch work that can stop and resume. The whole challenge reduces to one thing: never lose more than a few minutes of progress when an instance goes away. Frequent checkpointing and automatic resume turn an interruptible instance into a reliable, very cheap one.
This article is part of our AI, GPU and ML cluster. For the full picture start with the complete guide to AI and GPU cost optimization, the pillar this piece links up to. Once the run is right-sized, the serving side is covered in how to right-size GPU instances for model serving.
How much do spot GPUs save for AI training?
Spot GPUs save up to 90% versus on-demand because they use spare capacity the provider can reclaim. For interruption-tolerant training that checkpoints and resumes cleanly, that discount applies to the bulk of the run, which makes spot one of the largest single cost levers in AI training. The savings are real only if interruptions do not force you to repeat large amounts of work, which is exactly what the steps below prevent. AWS publishes the discount range and interruption behavior in its Spot Instances documentation.
Step 1: How do I confirm the job is interruption-tolerant?
Verify the training job can stop and resume before moving it to spot, because spot only suits work that survives being paused. Most batch training qualifies: it runs steps over data and can continue from saved state. Jobs that cannot checkpoint, or that hold non-reproducible in-memory state, are poor fits and should stay on on-demand or reserved capacity. This is the See step, deciding which workloads belong on spot at all.
Step 2: How do I checkpoint so an interruption is cheap?
Write model and optimizer state to durable object storage at a cadence short enough that recomputing the lost interval is cheap. Save checkpoints to storage that outlives the instance, not local disk, and keep them incremental so each write is fast. The cadence balances checkpoint overhead against work lost on interruption; every few minutes to every epoch is typical depending on step time. With frequent checkpoints, an interruption costs only the work done since the last save.
Step 3: How do I handle the interruption notice?
Catch the provider's interruption warning, two minutes on AWS, and trigger a final checkpoint so the very latest state is preserved before the instance stops. Listen for the spot interruption signal in your training loop or a sidecar, and on receipt flush a checkpoint and exit cleanly. This last-moment save turns the small remaining window into protection rather than lost progress, so resume picks up almost exactly where the job left off.
Training on on-demand GPUs when spot would do?
Our cost audit identifies which training jobs are spot-safe, builds the checkpoint, interruption, and resume handling, and diversifies capacity, then proves the saving against your on-demand baseline. On the performance model, you pay only from realized savings. No savings, no fee.
Book a cloud cost audit →Step 4: How do I make the job resume automatically?
Have the training script detect the newest checkpoint on restart and continue from it rather than starting over, so a replacement instance resumes the run unattended. On launch, the job should look for the latest checkpoint in durable storage, load model and optimizer state, and continue from that step. Automatic resume is what makes spot hands-off: when capacity returns, the orchestrator launches a new instance and the job carries on without manual intervention.
Step 5: Why diversify instance types and zones?
Request the GPU across multiple instance types and availability zones so capacity loss in one pool does not stall the job. Spot availability varies by instance type and zone, so a job pinned to one shape in one zone is exposed to that pool drying up. Specifying several compatible accelerator types across zones lets the scheduler place the job wherever spare capacity exists, raising the share of the run that stays on spot and reducing interruptions.
Step 6: When should I use managed spot training?
Use managed spot training when you want the discount without building interruption handling yourself. Services such as Amazon SageMaker managed spot training request spot capacity, react to interruptions, checkpoint, and resume on your behalf, and report savings against on-demand. It trades some control for far less engineering effort, and is the fastest way for most teams to capture the spot discount safely. Forecast the savings into your plan using how to forecast AI and GPU infrastructure spend.
The AI and GPU Cost Control Guide includes our spot training playbook with the checkpoint cadence and interruption-handling checklist. It is the downloadable companion to this article.
Frequently asked questions
How much do spot GPUs save for AI training?
Spot GPUs are available at up to a 90% discount compared to on-demand prices, because they use spare capacity that can be reclaimed with short notice. For interruption-tolerant batch training that checkpoints and resumes cleanly, that discount applies to the bulk of the run, which makes spot one of the largest single cost levers in AI training.
What happens when a spot GPU is interrupted during training?
When capacity is reclaimed, the provider sends a short interruption notice, two minutes on AWS, before stopping the instance. A well-built job catches that warning, writes a final checkpoint, and on restart resumes from the latest saved state. With frequent checkpointing the only loss is the work done since the last checkpoint, which is small.
How often should I checkpoint on spot training?
Checkpoint often enough that recomputing the lost interval costs less than the checkpoint overhead, typically every few minutes to every epoch depending on step time. The right cadence balances the cost of writing state to durable storage against the cost of redoing work after an interruption. Frequent, incremental checkpoints to object storage are the standard pattern.
Is managed spot training worth it?
Managed spot training is worth it when you want the spot discount without building interruption handling yourself. Services such as SageMaker managed spot training handle requesting spot capacity, reacting to interruptions, checkpointing, and resuming, and report the savings against on-demand. It trades some control for far less engineering effort.
The short version
Use spot GPUs for AI training by confirming the job is interruption-tolerant, checkpointing frequently to durable storage, handling the interruption notice, resuming automatically, diversifying instances and zones, and using managed spot training where it fits. Spot runs at up to 90% off on-demand, and clean checkpointing makes interruptions harmless. When you want spot training set up safely and the saving proven, that is exactly what our FinOps implementation service delivers.
Written by Morten Andersen and reviewed by Fredrik Filipsson, applying the See, Cut, Lock, Run method. Independent and vendor neutral.
Cloud pricing and service behavior change frequently. Verify the specifics in this guide against the providers’ own current documentation and the FinOps Foundation: FinOps Foundation Framework ↗ and FinOps Rate Optimization capability ↗. This article also reflects Cloud Cost Room’s hands-on, vendor-neutral engagement experience.