Elasticsearch — FAQ

Is Elasticsearch a good primary database, or should it always sit alongside a "real" database?

Generally the latter, and this is worth being explicit about since it's a common point of confusion for people newer to Elasticsearch. Elasticsearch is optimized for search and analytics — fast full-text queries and aggregations across large volumes of data — not for being the single source of truth for transactional application data. The standard pattern is a primary database (PostgreSQL, MySQL, MongoDB) holding the authoritative data, with relevant data synced into Elasticsearch specifically to power search/analytics features. Losing Elasticsearch's index should ideally mean "search is temporarily degraded, but no data is actually lost" — it can be rebuilt from the primary source — rather than "we lost data we can never get back."

Is Elasticsearch still a proprietary/closed-source product, given its licensing changes over the past few years?

As of a 2024 licensing change, Elasticsearch and Kibana are open source again (needs verification — confirm the exact date directly with Elastic's own announcement) — Elastic added AGPLv3 as a licensing option (alongside the existing SSPL and Elastic License 2.0), and AGPLv3 is an OSI-approved open source license. This followed a 2021 change (with the 7.11 release) that moved away from the original Apache 2.0 license specifically in response to concerns about cloud providers offering Elasticsearch as a managed service without contributing back — a dispute that also led directly to AWS forking Elasticsearch into OpenSearch. The practical licensing landscape today is genuinely a three-way choice (SSPL, ELv2, or AGPLv3) rather than the two-way "dual license" framing from the 2021-2024 period.

If Elasticsearch is open source again, is there still a reason to use OpenSearch instead?

It depends on your specific situation rather than one being universally better. OpenSearch remains a fully independent, actively-developed fork with its own roadmap, and some organizations have already built tooling, expertise, or contracts specifically around it (particularly common on AWS, where Amazon OpenSearch Service is the native managed offering). The "no licensing concerns" argument for choosing OpenSearch over Elasticsearch is genuinely weaker now that Elasticsearch offers an OSI-approved AGPLv3 option — but the two projects have been diverging in features and internals since the 2021 fork, so "which one has the features/ecosystem you need today" is now a more relevant question than the pure licensing argument alone.

Why does a `match` query sometimes score two seemingly-equal-relevance documents very differently?

Elasticsearch's default relevance scoring (BM25) accounts for several factors beyond simple term presence — term frequency (how often the search term appears in that specific document), inverse document frequency (rarer terms across the whole index score higher, since matching a rare term is more informative than matching a common one), and field length normalization (a match in a short field generally scores higher than the same match in a much longer field, since it represents a larger fraction of that field's content). Two documents that both "contain the search term" can score very differently once these factors are accounted for — this is usually intentional and correct behavior, though it can look surprising if you're expecting a simpler "matches or doesn't" scoring model.

Is it ever OK to just let dynamic mapping create fields automatically, or should every field always be explicitly mapped?

Dynamic mapping is genuinely fine and convenient for data with a reasonably stable, bounded set of fields, especially during early development or prototyping. It becomes a real risk specifically when field names (not just values) are unpredictable or externally controlled — arbitrary JSON keys from a webhook payload, user-controlled metadata, anything where the set of possible field names isn't something you control — since each new unique field name becomes a permanent addition to the mapping. This course's own Real World Scenarios material covers exactly this failure mode. The practical middle ground: use dynamic mapping for genuinely predictable data, and explicit mapping (with dynamic: false or strict for nested unpredictable data, or the flattened type) for anything where field names themselves could vary unpredictably.

Why does force-merging an index sometimes make things worse before they get better?

_forcemerge is a genuinely resource-intensive operation — it rewrites segments, consolidating many smaller ones into fewer, larger ones, which involves significant CPU and I/O while it runs. Running it during peak traffic can noticeably degrade search performance for the duration of the merge, even though the end result (fewer, more efficient segments) is an improvement. This is exactly why it's standard practice to run _forcemerge during low-traffic windows, and to generally reserve it for indices that are no longer being actively written to (a common pattern: force-merge a time-series index once it rolls over and stops receiving new writes, as part of an ILM warm-phase transition, rather than force-merging an actively-written index).

What's the practical difference between `must` and `should` in a bool query, beyond "AND vs OR"?

must clauses are required — a document not matching every must clause is excluded from results entirely. should clauses are optional in the sense that they don't exclude documents on their own (when there's no must clause present, at least one should typically needs to match, but the exact behavior depends on the minimum_should_match setting) — their main practical role is boosting the relevance score of documents that do match them, without requiring that match. A common real pattern: put required, structural conditions in must/filter, and put "nice to have, boost if present" signals (like a preferred brand or a promotional tag) in should to influence ranking without excluding documents that lack that specific signal.

Why do JVM heap size recommendations for Elasticsearch cap out at around 31GB, even on servers with much more RAM available?

This relates to a JVM feature called "compressed ordinary object pointers" (compressed oops), which lets the JVM use 32-bit references instead of 64-bit ones for heaps under a certain threshold (roughly 32GB), meaningfully reducing memory overhead per object. Once heap size crosses that threshold, the JVM switches to full 64-bit pointers, and the actual usable memory efficiency can paradoxically get worse despite the larger nominal heap size. This is why the standard recommendation is roughly 50% of available RAM for the JVM heap, but explicitly capped around 30-31GB regardless of how much RAM the server actually has — on a server with 128GB RAM, for instance, the right heap setting is still around 30GB, not 64GB, with the remaining RAM left for the OS page cache (which Elasticsearch/Lucene also relies on heavily for performance).

Is a "red" cluster status always a data-loss emergency?

It always indicates at least one primary shard is currently unassigned, which is serious and needs immediate attention — but "unassigned" and "permanently lost" aren't automatically the same thing. If a replica of that shard exists elsewhere in the cluster, promoting that replica to primary can resolve it without actual data loss, provided the cluster's allocation settings and available nodes allow that promotion to happen. Genuine, unrecoverable data loss specifically requires that no replica of the affected shard exists anywhere reachable — which is exactly why running with zero replicas (as some single-node or cost-minimized setups do) turns any single node failure into a true data-loss event, rather than a recoverable "red, but replicas exist" situation.

Why does this course emphasize measuring search relevance with actual metrics rather than just fixing individual complaints as they come in?

Because relevance tuning changes interact with each other in ways that aren't visible from looking at any single query in isolation — a change that fixes one reported bad result can simultaneously degrade many other, unreported queries that nobody happened to notice, and without measuring aggregate quality (click-through rate, a scored relevance judgment set), there's no way to know whether a series of individually-reasonable tuning changes has a net positive or net negative effect overall. This is exactly the failure mode covered in this course's own Real World Scenarios material — complaint-driven, unmeasured tuning can make average search quality worse even while every individual reported complaint gets "fixed."