Search & Inverted Indexes

Intermediate · 17 min read · High-Level Design, Data Processing & Specialised Systems

Tokenisation, inverted indexes, relevance ranking, and running Elasticsearch at scale.

"Find products matching wireless noise cancelling headphones, ranked by relevance, filtered by price and brand, with typo tolerance, in 100 ms." A relational database's LIKE '%headphones%' can't do this: it scans every row, ignores relevance, and doesn't understand language.

The inverted index

A normal (forward) index maps documents to their words. An inverted index maps each term to the list of documents containing it — a postings list.

Text analysis

Before indexing — and identically at query time — text goes through an analyzer:

Relevance ranking

Matching documents must be ordered. Classic scoring is based on:

Architecture of a search cluster

Elasticsearch/OpenSearch (built on Apache Lucene) is the common reference:

Keeping the index in sync

The search engine is a secondary, derived store — never the source of truth.

Search vs other options

Interview questions

• Why can't you just use SQL LIKE for product search?
• How do you keep Elasticsearch consistent with your primary database?
• Why is search 'near real-time' rather than immediate?

Practice

Write your answer before opening the suggested one. If you mark it "Partly" or "Missed it", it comes back in your review deck.

Further reading

Key takeaways

• Search engines use inverted indexes: term → sorted postings list of documents.
• Analyzers tokenise, normalise, stem and add synonyms — identically at index and query time.
• BM25 ranks by term frequency, rarity and field length; products add behavioural signals, learning-to-rank or vector search.
• Clusters shard documents and scatter-gather queries; segments make search near-real-time.
• The index is a derived store: feed it via CDC/events, index idempotently, and keep reindexing possible.

Previous: Batch & Stream Processing · Next: Geospatial Indexing