Objects, arrays and nested fields

Elasticsearch · Advanced · 11 min

Searching arrays of objects gives wrong matches unless you know about flattening.

Elasticsearch has no special array type: any field can hold several values. But an array of objects is flattened into multi-valued fields. A document with "items": [ { "sku": "PEN", "qty": 1 }, { "sku": "PAD", "qty": 9 } ] is indexed like items.sku: [PEN, PAD] and items.qty: [1, 9]. The association between a sku and its qty is lost, so a query for sku = PEN and qty > 5 matches this document, wrongly.

The nested type fixes this by indexing each object as a separate hidden document tied to its parent. You then query with a nested query whose inner conditions must hold within one object:

{ "nested": { "path": "items", "query": { "bool": { "filter": [
  { "term": { "items.sku": "PEN" } }, { "range": { "items.qty": { "gt": 5 } } } ] } } } }

Matching nested objects can be returned with inner_hits, and there are nested and reverse_nested aggregations.

The costs: each nested object is an extra Lucene document (an order with 100 items is 101 documents), updating any part reindexes the whole parent, and there are limits (index.mapping.nested_objects.limit, 10,000 per document). Use nested only where the cross-field association matters; otherwise plain objects, or denormalising into separate documents, are cheaper. flattened is another option for arbitrary key-value maps.

Example

PUT /carts
{ "mappings": { "properties": {
  "items": { "type": "nested", "properties": {
    "sku": { "type": "keyword" }, "qty": { "type": "integer" } } } } } }

PUT /carts/_doc/1?refresh=true
{ "items": [ { "sku": "PEN", "qty": 1 }, { "sku": "PAD", "qty": 9 } ] }

GET /carts/_search
{ "query": { "nested": { "path": "items", "query": { "bool": { "filter": [
    { "term": { "items.sku": "PEN" } },
    { "range": { "items.qty": { "gt": 5 } } } ] } } } } }
# 0 hits: no single item is PEN with qty > 5
# (with a plain object mapping this would wrongly return the cart)

Remember

  • Arrays of objects are flattened; per-object associations are lost
  • nested indexes each object as a hidden document
  • Query nested fields with the nested query; inner_hits shows which matched
  • Nested adds documents and reindexing cost: use it deliberately

Watch out for

  • Querying nested fields with a normal query: it finds nothing, since nested fields need the nested query.

Check yourself

  1. Why can a cart with items [{sku: PEN, qty: 1}, {sku: PAD, qty: 9}] match 'sku PEN and qty > 5' under default mapping?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: www.elastic.co/guide/en/elasticsearch/reference/current/nested.html

Builds on

Unlocks

Nothing builds on this yet.