Elasticsearch 9 and search engine auto-detection
- Introduced in:
OpenAEV [MigrationVersion]
Description of changes
The platform now supports Elasticsearch 9 next to Elasticsearch 8 and OpenSearch, and picks the right
client by itself. At startup it calls GET / on engine.url, reads the version the server reports and
instantiates the matching client:
| Server | Client used |
|---|---|
| Elasticsearch 8.19+ | Elasticsearch 8.x |
| Elasticsearch 9 | Elasticsearch 9.x |
| Elasticsearch Serverless | Elasticsearch 9.x (most recent client), recognised by its serverless build flavor |
| OpenSearch | OpenSearch |
Each Elasticsearch major needs its own client: the 9.x client stamps every request with an
application/vnd.elasticsearch+json; compatible-with=9 content type, which an 8.x server rejects
outright.
The engine.engine-selector parameter (ENGINE_ENGINE_SELECTOR) is removed. A value still set is
ignored, and the platform logs a warning at startup. When the detection cannot decide, for example behind a
proxy that blocks GET /, set engine.client (ENGINE_CLIENT) to elasticsearch8, elasticsearch9 or
opensearch: the platform then uses that client without probing. Its default, auto, keeps the detection. With AWS SigV4 authentication
(engine.engine-aws-mode set to es or aoss), the platform uses the OpenSearch client without
probing, as before.
The platform refuses to start, with an explicit message, when the server cannot be reached, rejects the credentials, or is older than Elasticsearch 8.
Migration
Nothing to do: remove engine.engine-selector from your configuration to get rid of the startup warning.
Do not copy its value to engine.client: leave engine.client unset unless the detection fails, otherwise
the client stays pinned when the cluster is upgraded.
To move from Elasticsearch 8 to Elasticsearch 9, upgrade the cluster following Elastic's upgrade documentation, then restart OpenAEV: it detects the new version and switches client. The indices are kept. Elasticsearch 9 only reads indices created by 8.x or later, so any index still carrying a 7.x creation version has to be reindexed or removed first — Elastic's Upgrade Assistant reports them.
The platform owns every index it queries and can rebuild them from PostgreSQL, so an alternative to migrating the data is to point OpenAEV at an empty Elasticsearch 9 cluster:
- Stop OpenAEV and set
engine.urlto the new cluster. - Delete the content of the
indexing_statustable in PostgreSQL. It records, per model, how far the data was already indexed: left in place, it makes the platform skip everything indexed into the old cluster, and the new indices only receive later changes. - Start OpenAEV: the indices, templates and lifecycle policy are recreated and reindexed from scratch.
Expect the initial reindex to take a while on a large dataset, and dashboards to be incomplete until it finishes.
Clusters served over HTTPS
With Elasticsearch 9, a cluster whose certificate is signed by an internal CA is trusted as soon as that
CA is dropped in openaev.extra-trusted-certs-dir, like every other outgoing connection of the platform
(see certificate validation). The Elasticsearch 8 client reads
the JVM default trust store only: import the CA into that trust store to reach such a cluster on
Elasticsearch 8. engine.reject-unauthorized=false also works, but it disables certificate and hostname
verification entirely, so keep it for tests.