WiredTiger storage engine

MongoDB · Expert · 12 min

Capacity planning and diagnosing slowdowns require knowing what the storage engine does with memory and disk.

WiredTiger is MongoDB's storage engine. Each collection and each index is a separate B-tree file on disk.

Cache. WiredTiger keeps uncompressed pages in its internal cache, sized by default to 50% of (RAM minus 1 GB), at least 256 MB. The rest of RAM is used by the OS file cache holding compressed pages. Performance is best when the working set, the documents and index pages you touch regularly, fits in cache. When it doesn't, reads hit disk and eviction threads work hard; under heavy pressure application threads are drafted into eviction and latency spikes.

Compression. Collections use snappy block compression by default (zstd and zlib are options), and indexes use prefix compression, so disk usage is usually much smaller than the logical data size.

Concurrency. WiredTiger uses document-level concurrency with optimistic MVCC: readers see a snapshot and writers to the same document conflict and retry instead of locking whole collections.

Durability. Writes go to an in-memory structure and the journal (write-ahead log, flushed every 100 ms by default or immediately with j: true). A checkpoint writes a consistent snapshot of all data files every 60 seconds. After a crash, MongoDB starts from the last checkpoint and replays the journal.

Watch db.serverStatus().wiredTiger.cache for bytes in cache versus maximum and for dirty data percentages.

Example

const c = db.serverStatus().wiredTiger.cache
c["maximum bytes configured"]          // e.g. 7.5 GiB on a 16 GiB machine
c["bytes currently in the cache"]
c["tracked dirty bytes in the cache"]
c["pages read into cache"]             // climbing fast = working set > cache

const s = db.orders.stats()
s.size                                 // logical, uncompressed data bytes
s.storageSize                          // on-disk bytes after compression
s.totalIndexSize

Remember

  • Default cache = 50% of (RAM - 1 GB); keep the working set in it
  • Snappy compression for data, prefix compression for indexes
  • Document-level concurrency with MVCC snapshots
  • Journal + 60-second checkpoints give crash recovery

Watch out for

  • Sizing servers by data size alone: what must fit in RAM is the working set, especially indexes.

Check yourself

  1. After a crash, how does WiredTiger recover?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: www.mongodb.com/docs/manual/core/wiredtiger/

Builds on

Unlocks

Nothing builds on this yet.