Filtering in ES|QL using full text search

ES|QL includes full text functions that can be used to filter your data using text queries. We will review the available text filtering methods and understand why these functions provide a better alternative. We will also look at the future improvements for full text functions in ES|QL.

Filtering text with ES|QL

Text data in logs is critical for understanding, monitoring, and troubleshooting systems and applications. The unstructured nature of text allows for flexibility in capturing all sorts of information.

Being unstructured, we need ways of isolating specific patterns, keywords, or phrases. Searching for an error message, narrowing down results using tags, or looking for a specific host name, are things that we do all the time to refine our results and eventually obtain the information we're looking for.

ES|QL provides different methods to help you work with text. Elasticsearch 8.17 added the full text functions MATCH and QSTR, and 8.18 added KQL, to help tackle more complex search use cases.

Limitations of text filtering

ES|QL already provided text filtering capabilities, including:

Text equality, to compare full strings directly using the equality operator.
String start and end, using the STARTS_WITH and ENDS_WITH functions.
Pattern and regex matching with the LIKE and RLIKE operators.

Text filtering is useful - but it can fall short on text-oriented use cases:

Multivalued fields

Using ES|QL functions with multivalued fields can be tricky - functions return null when applied to a multivalued field.

If you need to apply a function to a multivalued field, you first need to transform the value to a single value using MV_CONCAT so you can match on a single value:

FROM logs
| EVAL all_tags = MV_CONCAT(tags)
| WHERE all_tags == "production"

Analyzed text

Analyzers are incredibly useful for full text search as they allow transforming text. They allow us to extract and modify the indexed text, and modify the queries so we can maximize the possibility of finding what we're looking for.

Text is not analyzed when using text filtering. This means, for example that you need to match the text case when searching, or create regexes / patterns that address possible case differences.

This can become more problematic when looking for multilingual text (so you can't use ASCII folding), trying to match on parts of paths (path hierarchy), or removing stopwords.

Performance

Pattern matching and regexes take time. Lucene can do a lot of the heavy lifting by creating finite automata to match using the indexed terms dictionary, but nonetheless it's a computationally intensive process.

As you can see in our 8.17 release blog, using regular expressions can be up to 50-1000x slower than using full text functions for text filtering, depending on your data set.

Enter full text functions

Elasticsearch 8.17 introduced two new functions as tech preview for text matching: MATCH and query string (abbreviated QSTR). 8.18 introduced KQL. These functions are generally available (GA) as of Elasticsearch 9.1 (and of course Serverless)

These functions address some of the limitations that existed for text filtering:

They can be used directly on multivalued fields. They will return results when any of the values in a multivalued field matches the query.
They use analyzers for text fields. The query will be analyzed using any existing analyzers for the target fields, which will allow matching regardless of case. This also unlocks ASCII folding, removing stopwords, and even using synonyms.
They are performant. Instead of relying on pattern matching or regular expressions, they can directly use Lucene index structures to locate specific terms in your data.

MATCH function

MATCH allows matching a value on a specific field:

FROM logs
| WHERE match(message, "connection lost")

MATCH function uses a match query under the hood. This means that it will create a boolean query when multiple terms are used, with OR as the default operator for combining them.

You can use parameters for the MATCH function to achieve the same results of the match query parameters:

FROM logs
| WHERE match(message, "connection lost", {"fuzziness": "auto", "operator": "AND"})

MATCH function currently has some limitations:

It can only be used in WHERE and STATS clauses.
It can't be used after a STATS or LIMIT command

Match operator

The match operator (:) is equivalent to the MATCH function above, but it offers a more succinct syntax:

FROM logs
| WHERE message:"connection lost"

It is more convenient to use the match operator, but you can use whichever makes more sense to you.

Match operator has the same limitations as the MATCH function.

Query String function

Query string function (QSTR) uses the query string syntax to perform complex queries on one or several fields:

FROM logs
| WHERE qstr("message: 'connection lost' AND tags:'production'")

Query string syntax allows to specify powerful full text options and operations, including fuzzy search, proximity searches and the use of boolean operators. Refer to the docs for more details.

You can also specify options for QSTR to achieve the same results as the query parameters:

FROM logs
| WHERE qstr("connection lost", {"lenient": true, "fuzziness": 2})

Query string is a very powerful tool, but currently has some limitations, very similar to the MATCH function:

It can only be used in WHERE and STATS clauses.
It can't be used after STATS or LIMIT commands
It can't be used after commands that modify columns, like SHOW, ROW, DISSECT, DROP, ENRICH, EVAL, GROK, KEEP, MV_EXPAND, or RENAME

KQL function

KQL function (KQL) uses the Kibana query language syntax to perform complex queries on one or several fields:

FROM logs
| WHERE kql("http.request.method: GET OR http.response.status_code: 400")

KQL function allows using the KQL queries you already have directly in ES|QL

KQL has the same limitations as the QSTR function.

What's next

Full text functions were introduced in 8.1, and are GA as of 9.1. That doesn't mean that we're finished! These are the next upcoming features for full text functions:

Allow using them after LIMIT
Allow using text functions both in EVAL and after an EVAL clause

We also added scoring, so you can start using ES|QL for relevance matching and not just for filtering. This is quite exciting as this will define how the future of text search will be like in Elasticsearch!

Check out introducing scoring and semantic search for an overview of scoring for text search.

Give it a try

MATCH, QSTR and KQL are available as GA on 9.1, and as tech preview from Elasticsearch 8.17 on. And of course they are always up to date in Serverless.

What are you looking for in terms of text filtering? Let us know your feedback!

Happy full text filtering!

Ready to try this out on your own? Start a free trial.

Want to get Elastic certified? Find out when the next Elasticsearch Engineer training is running!