ES|QL, you know, for Search - Introducing scoring and semantic search

Search with ES|QL

With Elasticsearch 8.18 and 9.0, ES|QL adds a host of new functionalities, including:

support for scoring
semantic search
more configuration options for the match function
a new KQL function

In this blog, we will review these 8.18 features and other exciting new features that we plan to add to ES|QL, reinforcing our investment in making ES|QL a modern search language ready to fit your needs, whether you are building a search application powered by ES|QL or analyzing your data in Kibana Discover.

Introducing scoring

In 8.17 we added the ability to filter documents using full text functions. If you are unfamiliar with full text filtering in ES|QL, we suggest reading our original blog post about it.

With 8.18 and 9.0 we introduce support for scoring, making it possible to return documents in order of their relevance. To access the score for each document, simply add the metadata _score field to your ES|QL query:

FROM books METADATA _score
| WHERE match(title, "Shakespeare") OR match(plot, "Shakespeare")
| SORT _score DESC

We retrieve the same scores we get from the equivalent search API query:

GET books/_search
{
  "query": {
    "bool": {
      "should": [
        {
          "match": {
            "title": "Shakespeare"
          }
        },
        {
          "match": {
            "title": "Shakespeare"
          }
        }
      ]
    }
  }
}

Full text search functions such as match, qstr and kql can only be used in the context of a WHERE condition and are the only ones that contribute to the score.

The _score column can not only be used to sort documents by relevance, but also in custom scoring formulas. In the next example, we keep only the most relevant results using a score threshold and then add a score boost based on the reader rating:

FROM books METADATA _score
| WHERE match(title, "Shakespeare") OR match(plot, "Shakespeare")
| WHERE _score > 2 // we remove the documents with low scores
| EVAL new_score = _score + rating/5
| SORT new_score DESC

Improving the match function

In ES|QL, the match function simply translates to a Query DSL match query. In 8.18 and 9.0, we expanded the match function's capabilities to include all options that are currently available in Query DSL. It is now possible to set well-known match options such as boost, fuzziness and operator in ES|QL too:

FROM books METADATA _score
| WHERE match(title, "Dream of the red chamber", { "operator": "AND", "fuzziness": "AUTO", "boost": 0.75 })
        OR match(plot, "Dream of the read chamber", { "operator": "AND", "boost": 0.25 })
| SORT _score DESC

Enter semantic search

The 8.18 release comes with the exciting announcement that semantic search is now generally available. We've expanded the match function to support querying over semantic_text field types.

In ES|QL, executing a semantic query is now as simple as performing a full-text query, as shown in this example:

FROM books METADATA _score
| WHERE semantic_title:"Shakespeare"
| SORT _score DESC

In this example, we set semantic_title to use the semantic_text field type.

Mapping your index fields as semantic_text is all it takes to set up your index for semantic search.

Check our search with semantic text tutorial for more details.

Hybrid search with ES|QL

ES|QL makes it straightforward to do both semantic and lexical search at the same time. It is also possible to set different boosts, prioritizing results from semantic search or lexical search, depending on your use case:

FROM books METADATA _score
| WHERE match(semantic_title, "Shakespeare", { "boost": 0.75 }) OR match(title, "Shakespeare", { "boost": 0.25 })
| SORT _score DESC

Transitioning from KQL

If you are a long-term user of Kibana Discover and use KQL (Kibana Query Language) to query and visualize your data and you'd like to try ES|QL but don't know where to start, don't worry, we got you!

In 8.18 and 9.0, ES|QL adds a new function which allows you to use KQL inside ES|QL. This is as simple as:

FROM logs*
| WHERE KQL("http.request.method:GET AND agent.type:filebeat")

ES|QL is already available in Kibana Discover.

This way, you get the best of both worlds: you can continue to use KQL and start getting more familiar with ES|QL at your own pace.

Check out our getting started with ES|QL guide for more information.

Beyond 8.18 and 9.0

In future releases, we'll be adding more and more search capabilities to ES|QL, including vector search, semantic reranking, enhanced score customization options, and additional methods for combining hybrid search results, such as Reciprocal Rank Fusion (RRF).

Try it out yourself

These changes are available starting with Elasticsearch 8.18, but they are already available in Elasticsearch Serverless. For Elasticsearch Serverless, start a free trial cloud today or try Elastic on your local machine now!

Follow the Search and filter in ES|QL tutorial for a hands-on introduction to the features described in this blog post!

Ready to try this out on your own? Start a free trial.

Want to get Elastic certified? Find out when the next Elasticsearch Engineer training is running!