How to Optimize Streaming and Event Pipeline Storage Costs

Last updated: June 2026 · Written by Fredrik Filipsson and reviewed by Morten Andersen · See, Cut, Lock, Run method

A streaming pipeline, built on Apache Kafka, Amazon MSK, or Amazon Kinesis, holds a moving window of events so consumers can read and replay them. The storage cost is set by the retention window: how many hours or days of data sit on broker disks or stream shards. Because retention is configured once and rarely revisited, pipelines routinely keep weeks of events hot to support a replay need measured in hours, paying premium streaming storage rates for data that belongs in cheap object storage. This guide is part of our complete guide to cloud storage and data cost optimization, the cluster pillar it links up to.

What drives streaming and event pipeline storage cost?

Retention drives streaming storage cost: services bill for how long data stays on the broker or shard, so the window multiplies the per-GB rate regardless of throughput. Amazon MSK broker storage is about $0.10 per GB-month per the Amazon MSK pricing page, and Amazon Kinesis Data Streams, per the Kinesis Data Streams pricing page, includes 24 hours of retention then charges extra for extended retention up to 7 days and a separate long-term retention rate beyond that, plus retrieval. The same shape holds for self-managed Kafka on disks you provision and for other clouds. Because object storage costs a fraction of broker or shard storage per GB-month, the structural saving is to keep only the live replay window in the streaming layer and move everything older to a cheaper tier. Verify current rates against each provider before modeling, since streaming pricing changes by region and configuration.

How do I cut streaming storage cost step by step?

Cut streaming storage cost by right-sizing retention, tiering, and offloading cold data. Work through these steps:

1. Audit retention windows per topic and stream. Review the retention setting on every Kafka topic and Kinesis stream, since storage cost scales directly with how long data is kept on the broker or shard.
2. Shorten retention to what consumers need. Reduce retention to the window consumers actually replay, and meet longer-term needs from cheaper storage rather than keeping everything hot.
3. Enable tiered storage where available. Turn on MSK tiered storage and similar features so older log segments move to a lower-cost tier instead of staying on expensive broker storage.
4. Compact topics and remove duplicates. Use log compaction on keyed topics so only the latest value per key is retained, cutting storage for changelog-style streams.
5. Sink cold events to object storage. Land events in Amazon S3 or equivalent object storage for long-term query and replay at a fraction of streaming storage cost, then expire the hot copy.

This is the same waste-elimination work in our rightsizing and waste elimination service. The egress counterpart, where moving pipeline data between networks drives transfer cost, is covered in the sibling guide how to cut egress with private connectivity and peering.

Should I keep events in the stream or sink them to object storage?

Keep only the live replay window in Kafka or Kinesis and sink older events to object storage for long-term retention and query. The two layers serve different jobs, and using the streaming layer as a long-term store is the most common and most expensive mistake. Broker and shard storage is engineered for low-latency, ordered, replayable access, which is exactly what consumers need for the recent window, and you pay a premium for it. Object storage such as Amazon S3 is engineered for durable, cheap, queryable retention, which is what historical events need, and it costs a fraction per GB-month. A pipeline that sinks events to S3 continuously, keeps a short hot window on the broker, and queries history from the lake gets the best of both: fast replay where it matters and cheap retention everywhere else. The table contrasts the two layers.

Verdict: keep the hot replay window in the stream, tier where the service supports it, and sink long-term events to object storage.

Frequently asked questions

What drives streaming storage cost?

Retention is the main driver. Streaming services bill for how long data stays on the broker or shard, so a long retention window multiplies storage cost regardless of throughput. Amazon MSK broker storage is about $0.10 per GB-month, and Kinesis Data Streams charges extra for retention beyond the default 24 hours, so shortening retention and tiering old data are the largest savings.

How do I reduce Amazon MSK storage cost?

Shorten topic retention to what consumers replay, enable MSK tiered storage so older segments move to a lower-cost tier, and compact keyed topics so only the latest value per key is kept. MSK broker storage is about $0.10 per GB-month, and tiered storage holds cold segments more cheaply while keeping them readable.

Should I keep events in Kafka or sink them to object storage?

Keep only the replay window consumers need in Kafka or Kinesis, and sink older events to object storage like Amazon S3 for long-term retention and query. Broker and shard storage is far more expensive per GB-month than object storage, so long-term data belongs in the cheaper tier while the streaming layer stays lean.

What is the biggest hidden streaming cost?

Over-long retention and extended retention fees. Teams often keep weeks of data hot for a replay window measured in hours, and Kinesis charges separately for retention beyond 24 hours and for long-term retention, so a default left unexamined quietly multiplies storage cost.

The short version

Streaming systems bill for retention, so the window sets the cost. Audit retention per topic and stream, cut it to the consumer replay need, enable tiered storage, compact keyed topics, and sink cold events to object storage at a fraction of broker cost. When you want every pipeline reviewed and the saving proven down, that is part of what our rightsizing and waste elimination service delivers.