Elastic Platform 8.18 and 9.0: ES|QL Lookup Joins are here and Lucene 10!

Elastic’s piped query language, ES|QL, continues to evolve, becoming more powerful and intuitive. These new features are designed to enrich data analysis, simplify complex queries, and improve performance in mission-critical workflows.

One of the most anticipated features in ES|QL is the Lookup Join in tech preview. This feature introduces JOIN capabilities similar to SQL's LEFT OUTER JOIN, allowing users to enrich security and observability data by performing lookups against any lookup index. Unlike traditional enrich policies, Lookup Joins make data enrichment simpler and more manageable — a lookup index is directly updateable.

This is a game-changer for security analysts, SREs, and detection engineers who rely on contextual data for investigations. 

Use Lookup Joins for Security:

• IP Threat Enrichment (Quickly see if any source IPs match known malicious addresses.)
  

Running long ES|QL queries can now retrieve partial results if a query is terminated due to a timeout. A new async/stop endpoint ensures that instead of failing outright, you can still work with available data.

ES|QL now improves performance when handling case-insensitive queries. TO_UPPER and TO_LOWER functions are pushed down to Lucene, significantly speeding up execution.

Standard Deviation function: Adds statistical support for identifying anomalies in security and observability data like high network usage or unusual number of logins

Elasticsearch logsdb index mode is now enabled by default for logs in new deployments of 9.0 and upgrades to 9.0 if the deployment doesn’t have existing usage of logs-* data streams. This helps Observability and Security users manage large volumes of logs more efficiently.

Logsdb index mode now supports custom routing of documents based on sort fields if you have the required subscription, leading to another 20% storage reduction in benchmarks compared to 8.17. Optimizing storage layout improves query performance and lowers storage costs — critical for teams dealing with petabytes of log data while staying on budget.

Setting: index.logsdb.route_on_sort_fields:true

A new option allows users to skip recovery source during indexing when synthetic source is enabled. The recovery source was used for replication and peer recovery and added a significant amount of disk IO during indexing with synthetic source. 

With synthetic recovery source, we don’t write it anymore, and instead generate it dynamically as needed. This eliminates unnecessary processing overhead and reduces IO, improving indexing throughput. Faster logs and metrics ingestion speeds directly impact security and observability pipelines, reducing lag between data generation and availability for search and analytics.

Elasticsearch 9.0 now runs on Lucene 10, bringing significant performance improvements:

• More efficient search parallelism for multi-core machines

• Better I/O handling for high-latency storage like object stores

• Sparse indexing improvements for CPU and storage efficiency

Search operations are now more efficient across different storage and compute environments, reducing latency for large-scale deployments. Read more about Lucene 10 in the Lucene release highlights.

We also made major improvements to the speed of downsampling for metrics in TSDS (time series data stream). By removing unnecessary reads of dimension fields, bulking operations, and avoiding unnecessary casts and number conversions, we’ve seen downsampling to 1 hour take 63% less time in our benchmarks. This also means less CPU usage in the tiers where you downsample. 

• Reindex Data Stream API: This API simplifies the process of upgrading backing indices for data streams during major version migrations. The Reindex Data Stream API can also be leveraged via the Kibana Upgrade Assistant as a part of broader upgrade preparations.

• Create index from source API: This API streamlines the reindexing process by ensuring target indices are created with compatible settings and mappings to the source index. Read more in the documentation.

These updates streamline upgrade processes and enhance security controls in multi-tenant environments.

• In-table search and highlight (CTRL+F-like): This allows users to search for terms across the entire table, not just visible rows.

• Log-level breakdown in histograms: This automatically adjusts breakdown selector based on log sources.

• AI Assistant for logs in Discover: AI-driven insights help surface critical log patterns and troubleshooting recommendations.

• Smooth transition from KQL/Lucene to ES|QL: Users migrating from traditional queries to ES|QL will now find their existing queries preloaded in Discover’s ES|QL mode.

• Case templates GA: Prebuilt templates streamline investigation workflows

• Case observables GA: Enables structured data collection and integration with third-party issue management tools

Security and observability teams can standardize investigations, improve response times, and better communication across teams.

Alongside Elastic Stack 8.18 and 9.0, we’re also releasing ECK 3.0 and ECE 4.0, which provide full compatibility with Elastic Stack 9.0 and the ability to orchestrate new deployments and upgrade existing ones to it.

Elastic 9.0 introduces powerful developer-focused enhancements across query performance, log management, and user experience:

• ES|QL introduces powerful new ES|QL commands and functions that enable real-time join, with Lookup Joins, partial query results, and advanced text grouping functions.

• Logsdb index mode is now the default in 9.0 for log storage, improving efficiency and reducing costs.

• Lucene 10, available in Elasticsearch 9.0, boosts performance, making search queries faster and more hardware-efficient.

• New APIs make index upgrades and management easier, reducing friction during migrations.These updates ensure Elastic delivers the best-in-class data retrieval and storage capabilities for scalable search, security analytics, and observability.

Read about these capabilities and more in the release notes.

Existing Elastic Cloud customers can access many of these features directly from the Elastic Cloud console. For the fastest and best cloud experience, try Elastic Cloud Serverless now. Start a free trial.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.