Data lake storage cost is mostly a file-shape and tiering problem. Two fixes do the heavy lifting: compact millions of small files into fewer files of 128MB to 512MB so request volume, metadata overhead, and scan time all fall, then tier cold partitions into Infrequent Access or archive classes and expire data nobody reads. Do them in that order. Compaction first, because it often lowers query cost as well as storage, then tiering on the compacted, cold data. In the 500+ environments we have optimized, this sequence is the fastest path to a lower lake bill, and it is part of the work behind our 31% average reduction.
Last updated: June 2026 · Written by Fredrik Filipsson and reviewed by Morten Andersen · See, Cut, Lock, Run method
A data lake on object storage looks cheap per gigabyte, which is exactly why it is rarely managed. Two patterns let the bill creep upward unnoticed. The first is the small files problem: streaming and micro-batch pipelines write thousands of tiny objects per partition, and each one carries request and metadata cost that has nothing to do with the bytes inside it. The second is tiering neglect: data lands on the hottest, priciest storage class and never moves, even though most of it is never queried again after the first week. This guide is part of our complete guide to cloud storage and data cost optimization, the cluster pillar it links up to.
Why is my data lake storage bill so high?
A data lake bill is usually high because of small files and cold data on a hot tier, not because of the raw volume stored. Small files inflate the request line because every object write and read is a billable operation, and they inflate query cost because engines must list and open far more files than necessary. Cold data inflates the storage line because partitions that have not been queried in months still sit on the standard class. On Amazon S3, standard storage is roughly $0.023 per gigabyte-month while Standard-Infrequent Access is materially cheaper for data accessed less than monthly, per the Amazon S3 pricing page. The bill is high because nothing is moving the cold data down and nothing is consolidating the small files. Both are fixable with scheduled jobs and lifecycle rules rather than re-architecture.
How do I cut data lake storage costs step by step?
Cut the lake bill by reshaping the files first and tiering the cold data second. Work through these steps:
- Profile file sizes and partition layout. Use S3 Storage Lens and your table metadata to find partitions full of small files and to see which partitions are cold. You cannot fix a shape you have not measured.
- Compact small files into larger ones. Run a Spark or engine-native compaction job that rewrites many small objects into fewer files of 128MB to 512MB each. This cuts request volume, metadata overhead, and scan time in one pass.
- Tier cold partitions to cheaper classes. Apply lifecycle policies that move partitions past a known age boundary into Infrequent Access, and the genuinely archival data into Glacier classes. Tier by partition date, not by guesswork.
- Expire data nobody reads. Set expiration rules on temporary, staging, intermediate, and superseded data so it deletes automatically rather than accruing storage forever. Most lakes carry years of scratch data no one will ever query.
- Monitor and re-run on a schedule. Schedule compaction so new small files do not re-accumulate, and review tiering boundaries each quarter so the lake stays lean as access patterns shift.
This is the same waste-elimination work in our rightsizing and waste elimination service. The query-engine side of the same files, how layout and format drive scan cost, is covered in the sibling guide how to optimize Parquet and columnar storage for query cost.
Compacting before tiering means you move fewer, larger objects, so lifecycle transition request fees stay low. Transitioning a million tiny files into a cold class can cost more in transition requests than the storage it saves. Reshape first, then tier the clean result. Order matters, just as it does across our whole method.
How does file compaction lower storage cost?
Compaction lowers cost by replacing many small billable objects with a few large ones, cutting request volume and metadata overhead without changing the data. Each object in a data lake carries fixed per-request charges on write and read, and query engines pay a listing and open cost per file, so a partition of ten thousand one-megabyte files is far more expensive to store and scan than the same data in twenty 512-megabyte files. Compaction is a periodic job that reads the small files in a partition, rewrites them as larger files in an efficient columnar format, and removes the originals. For table formats such as Apache Iceberg, Delta Lake, and Apache Hudi, compaction is a first-class maintenance operation with built-in commands. The target size of 128MB to 512MB balances parallelism against per-file overhead, which is the range most query engines read most efficiently. Compaction usually lowers both the storage request line and the query scan time, which is why we pull it before any tiering work.
When should I tier data lake data to cheaper storage?
Tier a partition when its access frequency drops below the break-even point for the cheaper class, which for most lakes is a clear age boundary. Lake access is overwhelmingly recency-biased: the last few days or weeks of partitions get queried constantly, and older partitions get touched rarely or never. That makes age-based lifecycle rules a reliable proxy for access frequency. Move partitions older than the hot window into Infrequent Access, and partitions older than a longer threshold into archive classes, but only where retrieval latency and retrieval fees are acceptable, because cold classes charge to read the data back. The discipline is to tier by measured access, not optimism: if a so-called cold partition is read weekly, the retrieval fees will erase the storage saving. For the automated approach to this same decision across object stores, see the lifecycle work in our sibling guidance on optimizing Parquet and columnar storage for query cost.
Is your data lake quietly doubling in cost?
Our cost audit profiles file shape and partition access across the lake, runs the compaction and tiering plan, and proves the storage and query saving against a clean baseline before anything is committed. On the performance model, you pay only from realized savings. No savings, no fee.
Book a cloud cost audit →The Cloud Storage and Egress Cost Playbook includes the data lake compaction and tiering checklist we run, so small files and cold partitions stop inflating the bill.
Frequently asked questions
What is the small files problem in a data lake?
The small files problem is when a lake holds millions of tiny objects instead of fewer large ones. It inflates request costs, metadata overhead, and query time because the engine opens and lists far more files than necessary. Compaction fixes it by rewriting small files into larger ones, typically 128MB to 512MB each.
How big should compacted data lake files be?
Aim for files of roughly 128MB to 512MB for columnar formats like Parquet and ORC. That range balances parallelism against per-file overhead, so query engines read efficiently and you avoid both tiny-file overhead and oversized files that limit how much work can run in parallel.
Does tiering cold data save money in a data lake?
Yes, moving cold partitions to Infrequent Access or archive storage classes lowers the per-gigabyte rate. The saving is real when the data is genuinely rarely read, because cold classes add retrieval fees that can outweigh the storage saving if the data turns out to be hot. Tier by measured access, not optimism.
Will compaction reduce my query costs too?
Often yes. Fewer, larger files mean the query engine lists and opens less, so scan-based engines like Amazon Athena read more efficiently. Compaction lowers storage request volume and improves query performance at the same time, which is why it is usually the first lever to pull on a lake.
The short version
Data lake storage cost is a file-shape and tiering problem. Compact millions of small files into fewer files of 128MB to 512MB so request volume, metadata overhead, and scan time fall, then tier cold partitions into Infrequent Access or archive and expire data nobody reads. Compaction first, tiering second, monitored on a schedule so neither pattern returns. When you want the whole lake profiled and the saving proven down, that is the work our rightsizing and waste elimination service delivers.
Cloud pricing and service behavior change frequently. Verify the specifics in this guide against the providers’ own current documentation and the FinOps Foundation: FinOps Foundation Framework ↗ and FinOps Rate Optimization capability ↗. This article also reflects Cloud Cost Room’s hands-on, vendor-neutral engagement experience.