TopolearnMemory usage, disk planning, slow restarts and data loss after crashes are all explained by the storage design.
Prometheus's local TSDB is organized in time blocks:
- The head block holds the most recent data (roughly the last two to three hours) in memory, with every active series and its recent samples. Samples are compressed with delta-of-delta timestamps and XOR-encoded values, averaging about 1 to 2 bytes per sample.
- Every incoming sample is first appended to the write-ahead log (WAL) on disk. After a crash or restart, Prometheus replays the WAL to rebuild the head, which is why restarts of large servers take minutes. Memory-mapped head chunks reduce that work.
- Every two hours the head is cut into an immutable persistent block: a directory with compressed chunks, an index (label to series postings), metadata and tombstones for deletions.
- Compaction merges small blocks into bigger ones (up to 10 percent of retention or 31 days), making queries over long ranges cheaper.
- Retention deletes whole blocks older than
--storage.tsdb.retention.time(default 15d) or beyond--storage.tsdb.retention.size.
Sizing rule of thumb: disk = retention seconds x samples per second x 1 to 2 bytes. Memory is driven mostly by the number of active series in the head, which is why cardinality matters more than retention.
Local storage is single-node and not replicated; use snapshots for backups and remote write for durable long-term storage. Use local SSD, not NFS.
Example
# Inspect the TSDB on disk
ls data/
# 01JA3K... 01JA6Q... chunks_head/ wal/ lock queries.active
promtool tsdb list data/
# BLOCK ULID MIN TIME MAX TIME DURATION NUM SAMPLES NUM CHUNKS NUM SERIES
promtool tsdb analyze data/ | head -20 # top metrics and labels by series count
# Disk estimate: 1M samples/s for 30 days at ~1.5 bytes/sample
python3 -c "print(1_000_000 * 30*24*3600 * 1.5 / 1e12, 'TB')" # 3.888 TB
# Key self-metrics (PromQL):
# prometheus_tsdb_head_series active series in memory
# rate(prometheus_tsdb_head_samples_appended_total[5m]) ingestion rate
# prometheus_tsdb_wal_corruptions_totalRemember
- Recent data lives in the in-memory head, protected by the WAL
- Two-hour blocks are compacted into larger ones and deleted by retention
- Memory scales with active series; disk with samples x retention
Watch out for
- Putting the TSDB on NFS or network storage, which the docs explicitly don't support.