Apache Lucene development has always been vibrant, but the last few months have seen an especially high number of optimizations to query evaluation. There isn't one optimization that can be singled out, it's rather a combination of many improvements around mechanical sympathy and improved algorithms.
What is especially interesting here is that these optimizations do not only benefit some very specific cases, they translate into actual speedups in Lucene's nightly benchmarks, which aim at tracking the performance of queries that are representative of the real world. Just hover on annotations to see where a speedup (or slowdown sometimes!) is coming from. By the way, special thanks to Mike McCandless for maintaining Lucene's nightly benchmarks on his own time and hardware for almost 13 years now!
Key speedup benchmarks in Lucene
Here are some speedups that nightly benchmarks observed between Lucene 9.6 (May 2023) and Lucene 9.9 (December 2023):
- AndHighHigh: 35% faster
- AndHighMed: 15% faster
- OrHighHigh: 60% faster
- OrHighMed: 38% faster
- CountAndHighHigh: 15% faster
- CountAndHighMed: 11% faster
- CountOrHighHigh: 145% faster
- CountOrHighMed: 155% faster
- TermDTSort: 24% faster
- TermTitleSort: 290% faster (not a typo!)
- TermMonthSort: 7% faster
- DayOfYearSort: 25% faster
- VectorSearch: 5% faster
Lucene optimization resources
In case you are curious about these changes, here are resources that describe some of the optimizations that we applied:
- Bringing speedups to top-k queries with many and/or high-frequency terms (annotation FK)
- More skipping with block-max MAXSCORE (annotation FU)
- Accelerating vector search with SIMD instructions
- Vector similarity computations FMA-style
Lucene 9.9 was just released and is expected to be integrated into Elasticsearch 8.12, which should get released soon. Stay tuned!
Ready to try this out on your own? Start a free trial.
Elasticsearch and Lucene offer strong vector database and search capabilities. Dive into our sample notebooks to learn more.
Related content
April 7, 2025
Speeding up merging of HNSW graphs
Explore the work we’ve been doing to reduce the overhead of building multiple HNSW graphs, particularly reducing the cost of merging graphs.
April 7, 2025
Speeding up HNSW graph merge
We describe a new strategy we developed that reduces HNSW merge time by up to 70% whilst maintaining similar graph quality
February 27, 2025
Filtered HNSW search, fast mode
Explore the improvements we have made for HNSW vector search in Apache Lucene through our ACORN-1 algorithm implementation.
February 7, 2025
Concurrency bugs in Lucene: How to fix optimistic concurrency failures
Thanks to Fray, a deterministic concurrency testing framework from CMU’s PASTA Lab, we tracked down a tricky Lucene bug and squashed it
January 7, 2025
Early termination in HNSW for faster approximate KNN search
Learn how HNSW can be made faster for KNN search, using smart early termination strategies.