TopolearnLogs and metrics grow forever; data streams and lifecycle policies keep shards sized and storage under control.
A data stream is the recommended way to store append-only time-series data such as logs, metrics and events. You write to one name (logs-app-default); behind it Elasticsearch keeps hidden backing indices (.ds-logs-app-default-2026.10.17-000001...). Every document needs an @timestamp. Writes go to the newest backing index; searches cover all of them. Data streams accept only create operations, while updates and deletes go through _update_by_query and _delete_by_query or by targeting a backing index.
A data stream is created automatically on first write when a matching index template has "data_stream": {}.
Rollover starts a new backing index when the current one reaches a size, age or document count, keeping shards near the ideal size (for example max_primary_shard_size: 50gb).
Retention and tiering:
- ILM (index lifecycle management) policies move indices through phases: hot (rollover), warm (shrink, forcemerge, move to cheaper nodes), cold, frozen (searchable snapshots) and delete after a minimum age,
- the simpler data stream lifecycle sets just retention (
data_retention: "30d") and handles rollover automatically, a good default when you don't need tiers.
Deleting whole old backing indices is vastly cheaper than deleting documents by query, which is the core reason to organise time data this way.
Example
PUT /_ilm/policy/logs-30d
{ "policy": { "phases": {
"hot": { "actions": { "rollover": { "max_primary_shard_size": "50gb", "max_age": "1d" } } },
"warm": { "min_age": "7d", "actions": { "forcemerge": { "max_num_segments": 1 } } },
"delete": { "min_age": "30d", "actions": { "delete": {} } } } } }
PUT /_index_template/logs-app
{ "index_patterns": ["logs-app-*"], "data_stream": {},
"template": { "settings": { "index.lifecycle.name": "logs-30d" },
"mappings": { "properties": { "@timestamp": { "type": "date" },
"message": { "type": "text" } } } } }
POST /logs-app-default/_doc
{ "@timestamp": "2026-10-17T09:00:00Z", "message": "user login" }
GET /_data_stream/logs-app-default
# backing index ".ds-logs-app-default-2026.10.17-000001"Remember
- Data streams = one write name over rolling hidden backing indices
- Index templates with data_stream: {} create them on first write
- Rollover keeps shards sized; ILM or data stream lifecycle handles retention
- Dropping old indices beats delete_by_query for time data
Watch out for
- Storing years of logs in a single index and purging with delete_by_query.