TopolearnWhether a search finds 'running' for 'run' or 'café' for 'cafe' is decided by analysis.
When a text field is indexed, an analyzer turns it into terms in three steps:
- character filters clean the raw text (strip HTML, map characters),
- a tokenizer splits it into tokens (
standardsplits on word boundaries following Unicode rules,whitespaceon spaces,keywordkeeps one token), - token filters transform tokens:
lowercase,asciifolding(café to cafe),stopwords,stemmer(running to run),synonym_graph,edge_ngramfor autocomplete.
The default standard analyzer is the standard tokenizer plus lowercase. Language analyzers like english add stop words and stemming.
The same analysis normally runs on the query text too, which is why match works. You can set a different search_analyzer, for example index with edge_ngram for prefix autocomplete but search with standard, so query text isn't also n-grammed.
The _analyze API shows exactly which terms an analyzer produces, and it's the first tool to use when a search mysteriously doesn't match.
Analyzers are defined in index settings and referenced from the mapping. Changing an analyzer on existing data requires reindexing, since terms were computed at index time. For keyword fields, a normalizer (e.g. lowercase) applies a light single-token cleanup.
Example
PUT /articles
{
"settings": { "analysis": { "analyzer": {
"folded_english": {
"type": "custom",
"char_filter": ["html_strip"],
"tokenizer": "standard",
"filter": ["lowercase", "asciifolding", "stop", "porter_stem"]
} } } },
"mappings": { "properties": {
"body": { "type": "text", "analyzer": "folded_english" } } }
}
POST /articles/_analyze
{ "analyzer": "folded_english", "text": "<p>The Cafés are RUNNING late</p>" }
# tokens: "cafe", "run", "late"Remember
- Analyzer = char filters + tokenizer + token filters
- Query text is analyzed with the same (or a search_) analyzer
- _analyze shows the exact terms produced
- Changing analysis means reindexing
Watch out for
- Adding synonyms or stemming to the mapping of an existing index and expecting old documents to match differently without reindexing.