ETL and data pipeline costs come from three things that compound: the compute that runs each job, the data each job scans or reads, and the data movement between stages and regions. The reason pipelines get expensive is that most are built to be correct first and efficient never, so they reprocess the entire dataset on every run when only a sliver changed, scan whole tables when they need a few columns, and shuttle data across regions and clouds because that is where the source happened to sit. Each full reprocess is compute and scan you pay for again to recompute a result that barely moved. Reducing the cost is mostly about doing less work: process only what changed, read only what you need, and size the engine to the job rather than to the worst case. The compute right-sizing matters, but it is the smaller lever; the volume of data processed is the big one.
This article is part of our complete guide to cloud storage and data cost optimization, the cluster pillar it links up to. It feeds directly into how to optimize data warehouse costs, since the data a pipeline produces is what the warehouse then scans.
Most pipeline cost is reprocessing data that did not change. Move to incremental loads, shrink what each job reads, and right-size the compute last. The volume processed is the dominant lever.
Process only what changed
The single largest waste in most pipelines is full reprocessing: a job that reads and rebuilds the entire dataset every run even though only the latest day or the changed rows are new. Moving to incremental processing, where each run handles only new or changed data using change-data-capture, watermarks or partition awareness, can cut compute and scan by an order of magnitude on a large table, because you stop paying to recompute history that is already correct. Closely related is deduplicating work across pipelines: teams frequently build several jobs that each read the same raw source and derive overlapping results, so consolidating them to read once and branch, or materializing a shared intermediate layer, removes redundant scans. Avoiding repeated scans of the same data is the pipeline-side reason warehouse cost runs up, the same scan economics covered in optimizing data warehouse costs.
Shrink what each job reads and writes
Even an incremental job pays for the data it touches, so the next lever is reading and writing less per run. Push filters and column selection to the source so the job reads only the rows and columns it needs rather than pulling whole tables and discarding most of it. Store intermediate data in efficient columnar formats with partitioning so downstream stages scan less, the same partitioning discipline that cuts warehouse scans. Compress data in flight and at rest so each stage moves and stores fewer bytes. And prune the intermediate and staging datasets that pipelines leave behind, which otherwise accumulate as pure storage waste, the kind of leftover covered in storage waste from snapshots, orphaned disks and old backups. Reading and writing less cuts cost at every stage the data passes through.
| Lever | What it cuts | Typical impact |
|---|---|---|
| Incremental processing | Reprocessing unchanged data | Largest, can be an order of magnitude |
| Deduplicate overlapping jobs | Redundant scans of one source | High on multi-pipeline estates |
| Filter and prune columns early | Rows and columns read per run | High on wide tables |
| Columnar formats and partitioning | Bytes scanned downstream | Medium to high |
| Right-size and use spot compute | Engine cost per run | Medium, and safe for retryable jobs |
Are pipelines reprocessing the world every night?
Our cloud cost audit profiles your costliest jobs, moves full reprocesses to incremental loads, right-sizes the compute, and proves the saving against a clean baseline on AWS, Azure, GCP and OCI. On the performance model, you pay only from realized savings. No savings, no fee.
Book a cloud cost audit →Right-size the compute and use the cheap kind
Once the volume of work is minimized, the engine that runs it should match the job. Pipeline compute is often provisioned for the largest job and left at that size for everything, so right-sizing the cluster, the serverless allocation or the worker pool to the actual data volume per job is a clean win, the same discipline as rightsizing compute. Most batch ETL is interruptible and retryable, which makes it an ideal fit for spot or preemptible capacity at a large discount, since a lost worker simply reruns. Schedule non-urgent jobs for off-peak windows, and scale ephemeral processing clusters to zero between runs so you pay only while a job is actually executing rather than for an idle standing cluster. Verify the current pricing of your processing service and the spot discounts in the provider's documentation as of May 2026, since these move.
The Cloud Storage and Egress Cost Playbook includes the pipeline cost audit and the incremental-load checklist we use to cut reprocessing before touching the compute layer.
Orchestration, retries and idle clusters add up
The jobs themselves are not the whole bill; the machinery around them quietly adds cost that an audit often misses. Failed jobs that retry the full pipeline rather than resuming from a checkpoint pay the entire compute cost again on every retry, so a flaky pipeline can cost several times its nominal run, which makes reliability a cost lever as much as an availability one. Orchestration and scheduling layers sometimes hold a standing compute cluster alive between runs to reduce startup latency, and that idle cluster bills around the clock for work that happens for minutes a day, the same standing-idle waste described in the economics of idle. Over-frequent scheduling is another silent multiplier: a pipeline set to run every five minutes when the data updates hourly does twelve times the work for no extra freshness. Align the schedule to how often the source actually changes, resume failed runs from checkpoints instead of restarting, and scale the orchestration compute to zero between jobs, and the surrounding cost falls even before the jobs themselves are optimized.
The short version
Reduce ETL and data pipeline costs by doing less work, in order. Move full reprocesses to incremental loads so each run handles only what changed, which is the dominant lever. Deduplicate overlapping jobs that scan the same source, push filters and column selection to the source, and use columnar formats and partitioning so every stage reads fewer bytes. Then right-size the compute, run interruptible batch on spot capacity, and scale ephemeral clusters to zero between runs. Verify current service and spot pricing before committing. Process only what changed and read only what you need, and the compute follows down. When you want the costliest jobs found and the pipeline spend proven down across the estate, that is part of what our rightsizing and waste elimination service delivers.