TopolearnIndexing throughput, disk usage, deletes and near real-time behaviour all follow from Lucene's segment model.
Each shard is a Lucene index made of segments. A segment is an immutable set of files containing an inverted index (term to list of documents and positions) for text, doc values (columnar data) for sorting and aggregations, points (BKD trees) for numbers, dates and geo, stored fields for _source, and vector graphs.
Write path:
- a new document is added to an in-memory buffer and appended to the shard's translog, which is fsynced before the request is acknowledged (by default,
index.translog.durability: request), - a refresh turns the buffer into a new searchable segment in the filesystem cache, without an fsync; this is why search is near real-time,
- a flush performs a Lucene commit (fsyncing segments to disk) and trims the translog; after a crash, Elasticsearch replays the translog since the last commit.
Because segments are immutable, an update is a delete plus a new document, and a delete only marks the document in a live-docs bitmap. Space is reclaimed when background merges combine small segments into bigger ones and drop deleted documents. Many tiny segments (from very frequent refreshes) slow searches; merges cost I/O.
POST /index/_forcemerge?max_num_segments=1 is useful only for indices that will no longer be written to, such as yesterday's logs.
Immutability is also a strength: segments cache well in the OS page cache and need no locking for reads.
Example
GET /_cat/segments/products?v&h=shard,segment,docs.count,docs.deleted,size,committed,searchable
# shard segment docs.count docs.deleted size committed searchable
# 0 _0 1000 12 310kb true true
# 0 _1 40 0 22kb false true <- refreshed, not yet flushed
POST /products/_refresh # make recent writes searchable now
POST /products/_flush # Lucene commit + translog trim
# read-only index: merge down to one segment, purge deleted docs
POST /logs-2026.09/_forcemerge?max_num_segments=1
GET /products/_stats/refresh,flush,merge?filter_path=**.totalRemember
- Shards are Lucene indices made of immutable segments
- Refresh makes segments searchable; flush commits them and trims the translog
- The translog provides durability between commits
- Updates/deletes mark old docs; merges reclaim the space
Watch out for
- Force-merging an index that still receives writes, creating huge segments that are expensive to merge again.