Search Engines and Derived Indexes

An ordinary database index can locate exact values and ranges. Searching product descriptions also needs to interpret words, rank matches and apply filters. A search engine builds a representation suited to those queries.

That representation is usually a derived copy. A product update, privacy change or deletion must reach it, and the application needs a rule for the period before that happens.

An inverted index maps searchable terms to matching document IDs. Text analysis can split words, normalize case and apply language-specific rules. A search query can then gather candidates and rank them rather than scanning every description.

An exact keyword field and an analysed text field serve different queries. Keep identifiers, prices and permission fields in appropriate types. Relevance scores are not a proof that a result is correct for the visitor’s permissions.

Use a durable change stream, outbox or scheduled import according to the freshness requirement. Store document identity and version so repeated or reordered updates cannot restore an older state. Measure indexing lag and handle deletion events.

A rebuild may construct a new index, validate it and switch readers through a named alias. Coordinate updates during the rebuild so the new index does not miss intervening changes. A backup of the source and a replayable pipeline can support repair, but the rebuilding time still matters.

Decide from the query and freshness promise rather than assuming every application needs a new cluster.

PostgreSQL full-text search can fit applications already using that database. Apache Lucene is a search library. OpenSearch and Elasticsearch provide search services with distributed features; their licenses, hosting plans and capabilities differ, so verify the selected edition. Hosted search services shift some operation to a provider.

Use OpenSearch’s mapping documentation to understand field definitions. Benchmark document count and size, update rate, query mix, ranking, shard layout and acceptable indexing delay. “Documents indexed” alone is not a throughput limit.

Follow database indexing and copied fields for the broader cost of derived reads. Include deletion, private data, rebuild time and index storage in the plan.

Design the query, the derived update path and the access check together. Choose a search service only when its benefits justify synchronization and operation.