Managing retention in Elasticsearch can get complicated fast. Between Data stream lifecycle (DSL), Index lifecycle management (ILM), templates, and individual index settings, keeping policies consistent across data streams often takes more effort than it should.
Streams changes that. It introduces a clear, unified way to manage how long your data lives, whether you’re using DSL or ILM. You can visualize ingestion, understand where data sits across tiers, and adjust retention with confidence, applying updates to a single stream without worrying about unintended changes elsewhere, all from a single view.
Walkthrough: Exploring the Retention Tab
Retention management lives in the Retention tab of each stream. This is your control panel for understanding how much data you’re storing, how quickly it’s growing, and how your lifecycle policies are applied. It’s also where you can monitor and configure the Failure store, which tracks and retains documents that failed to be ingested.
Metrics at a glance
At the top of the view, you’ll find an overview of key metrics:
- Storage size: the total data volume currently held by the stream.
- Ingestion averages: calculated from the selected time range, Streams extrapolates both daily and monthly averages to give you a sense of long-term trends.
This combination of near-real-time and projected values helps you quickly spot when ingestion is ramping up and whether your retention policy aligns with it.
Ingestion over time
Below the metrics, a graph shows ingestion volume over time. This information is approximated based on the number of documents over time, multiplied by the average document size in the backing index.
Visualizing lifecycle phases
When an ILM policy is effective, the retention view becomes more visual. Streams displays a phase breakdown (hot, warm, cold, frozen) showing the data volume stored in each phase. This gives you a clear sense of how your data is distributed across the storage tiers and whether your lifecycle is doing what you expect.
Failure store
A failure store is a secondary set of indices inside a data stream, dedicated to storing documents that failed to be ingested. Within the Retention tab, you can toggle the Failure store on or off, and configure its own retention period. We’ll cover Failure store and Data quality in more detail in this article.
Updating Retention
Beyond visualizing your retention, Streams makes it easy to change how it’s managed.
Switching between DSL and ILM
You can freely switch a stream between DSL and ILM management, or update a DSL retention with just a few clicks. Streams takes care of updating the lifecycle settings at the data stream level, ensuring consistent retention across all existing backing indices, not just new ones.
Whether you prefer the simplicity of DSL or the fine-grained tiering of ILM, you can move between the two seamlessly.
Clicking “Edit data retention” opens a modal that allows you to update the stream’s configuration. From there you can update the ILM policy or set a custom retention period via DSL.
You can set a custom period, or pick an Indefinite retention for your data.
You can also update streams’ lifecycle via the Upsert stream or the Update ingest stream settings Kibana APIs.
Inherit or defer: different strategies for different stream types
Classic streams
For classic streams, you can default to the existing index template’s retention. Retention isn’t managed by Streams in this case, it follows the lifecycle configuration defined in the template just as it normally would.
This option is useful if you’re onboarding existing data streams and want to keep their lifecycle behavior intact while still benefiting from Streams’ visibility and monitoring features.
Wired streams
Wired streams live in a tree structure, and that hierarchy allows an inheritance model.
A child stream can inherit the lifecycle of its nearest ancestor that has a concrete policy (ILM or DSL). This keeps your configuration lean and consistent since you can set a single lifecycle at a higher level in the tree and let Streams automatically apply it to all relevant descendants.
If that ancestor’s lifecycle is later updated, Streams cascades the change down to all children that inherit it, so everything stays in sync.
In the figure below, we set a different retention for logs.prod and logs.staging environments. The child partitions of these environments automatically inherit the configuration.
How it works under the hood
When you apply or update a lifecycle, Streams calls Elasticsearch’s /_data_stream/_settings. This is a new API we’ve added in 8.19 / 9.1 for this purpose.
This API is key to keeping retention consistent:
- It applies the lifecycle directly at the data stream level, overriding any configuration from cluster settings or index templates.
- It propagates the retention update to all existing backing indices, not just new ones, so retention remains uniform across your historical and future data.
By centralizing lifecycle management at the data stream level and applying a consistent configuration across the backing indices, we remove the ambiguity that used to exist between template-level and index-level configurations. You always know which retention policy is actually in effect, and you can see it directly in the UI.
Wrapping Up
With Streams, retention management becomes clear and consistent. You can visualize ingestion, switch between DSL and ILM, or inherit policies across streams, all without diving into templates or manual index settings.
By unifying retention into a single view, Streams turns lifecycle management into something simple, predictable, and transparent.
Sign up for an Elastic trial at cloud.elastic.co, and trial Elastic's Serverless offering which will allow you to play with all of the Streams functionality.
Additionally, check out:
Read about Reimagining streams
Look at the Streams website
Read the Streams documentation