TopolearnExplaining slow queries, odd scores and hot shards requires knowing how a search executes across shards.
Any node receiving a search acts as the coordinating node. Execution happens in two phases by default (query_then_fetch):
- query phase: the request goes to one copy (primary or replica) of every relevant shard. Each shard runs the query locally and returns only the ids, sort values and scores of its top
from + sizehits, - the coordinator merges these into the global top hits,
- fetch phase: it asks only the shards holding the winners for their documents (
_source, highlights).
So a request with from: 9990, size: 10 makes every shard return 10,000 hits to merge, which is why deep paging is costly. Aggregations are computed per shard and reduced on the coordinator.
Scoring is per shard: IDF uses statistics from the shard, not the whole index. With few documents or skewed data, the same document can score differently depending on its shard. search_type=dfs_query_then_fetch gathers global term statistics first, at extra cost; with enough data the difference fades.
Routing decides a document's shard: shard = hash(_routing) % number_of_primary_shards (internally through routing shards), with _routing defaulting to _id. Custom routing (such as tenant id) lets a search target a single shard, but big tenants can create hot shards.
Adaptive replica selection sends each shard request to the copy expected to respond fastest. The profile API shows per-shard timings of query components.
Example
# all docs of tenant acme go to one shard; searches with the same routing hit only it
PUT /tickets/_doc/42?routing=acme
{ "tenant": "acme", "title": "Printer on fire" }
GET /tickets/_search?routing=acme
{ "query": { "match": { "title": "printer" } } }
# global IDF for small or skewed indices
GET /tickets/_search?search_type=dfs_query_then_fetch
{ "query": { "match": { "title": "printer" } } }
# where does the time go, per shard?
GET /tickets/_search
{ "profile": true, "query": { "match": { "title": "printer" } } }
# "profile": { "shards": [ { "id": "[node-1][tickets][0]", "searches": [ ... ] } ] }Remember
- The coordinating node fans out, merges, then fetches only the winners
- Each shard returns from + size candidates: deep paging is expensive
- IDF is per shard; dfs_query_then_fetch uses global stats
- Custom routing targets one shard but risks hot spots
Watch out for
- Diagnosing 'wrong' scores on a tiny test index with several shards: per-shard IDF differs.