TopolearnSyncing data from a database or pipeline needs high-throughput indexing without overloading the cluster.
Indexing documents one request at a time wastes round trips. The _bulk API takes newline-delimited JSON (NDJSON): an action line (index, create, update, delete) followed, except for delete, by a source line. The body must end with a newline.
The response has "errors": true if any item failed, and per-item statuses, so always inspect it: bulk requests partially succeed. Client libraries provide bulk helpers that chunk, send in parallel and report failures.
Tips for throughput:
- batches of roughly 5 to 15 MB; measure rather than guess,
- several parallel bulk workers, backing off on
429(rejected because the write thread pool queue is full), - for big initial loads, set
"refresh_interval": "-1"and"number_of_replicas": 0, then restore them afterwards, - let Elasticsearch generate ids when you don't need your own; it skips a lookup.
Each refresh makes recent writes searchable by creating a new segment, so frequent refreshes cost CPU and I/O. The default 1s (applied only while the index is being searched) is a good balance; ?refresh=wait_for on a request waits for the next refresh.
For set-based changes on documents already in the index use _update_by_query and _delete_by_query with a query and, for updates, a script. They run as tasks, can be throttled, and report version conflicts.
Example
POST /_bulk
{ "index": { "_index": "products", "_id": "2" } }
{ "name": "Red pen", "price": 1.4, "category": "office" }
{ "create": { "_index": "products", "_id": "3" } }
{ "name": "Ink", "price": 3.0, "category": "office" }
{ "update": { "_index": "products", "_id": "1" } }
{ "doc": { "stock": 20 } }
{ "delete": { "_index": "products", "_id": "99" } }
# { "errors": false, "items": [ { "index": { "status": 201 } }, ...,
# { "delete": { "status": 404, "result": "not_found" } } ] }
POST /products/_update_by_query?conflicts=proceed
{
"query": { "term": { "category": "office" } },
"script": { "source": "ctx._source.price *= 1.1" }
}Remember
- _bulk sends many actions as NDJSON, ending with a newline
- Check errors and per-item status: bulk partially succeeds
- Disable refresh and replicas during big initial loads
- update_by_query / delete_by_query change sets of documents
Watch out for
- Ignoring the bulk response and silently losing documents that failed mapping validation.