Excluding Elasticsearch fields from indexing

In Elasticsearch, indexing refers to the process of storing and organizing data in a way that makes it easily searchable. While indexing all fields in a document can be useful in some cases, there are situations where you might want to exclude certain fields from being indexed. This can help improve performance, reduce storage costs, and minimize the overall size of your Elasticsearch index.

In this article, we will discuss the reasons for excluding fields from indexing, how to configure Elasticsearch to exclude specific fields, and some best practices to follow when doing so.

Reasons for excluding fields from indexing

Performance: Indexing all fields in a document can lead to increased indexing time and slower search performance. By excluding fields that are not required for search or aggregation, you can improve the overall performance of your Elasticsearch cluster.
Storage: Indexing fields consumes storage space. Excluding fields that are not needed for search or aggregation can help reduce the storage requirements of your Elasticsearch cluster.
Index size: The size of an Elasticsearch index is directly related to the number of fields indexed. By excluding unnecessary fields, you can minimize the size of your index, which can lead to faster search and indexing performance.

Configuring Elasticsearch to exclude fields

To exclude a field from being indexed in Elasticsearch, you can use the “index” property in the field’s mapping. By setting the “index” property to “false”, Elasticsearch will not index the field, and it will not be searchable or available for aggregations.

Here’s an example of how to exclude a field from indexing using the Elasticsearch mapping:

PUT /my_index
{
  "mappings": {
    "properties": {
      "field_to_exclude": {
        "type": "text",
        "index": false
      }
    }
  }
}

In this example, we’re creating a new index called “my_index” with a single field called “field_to_exclude”. By setting the “index” property to “false”, we’re telling Elasticsearch not to index this field. The field will still be available in the source document, though.

Best practices for excluding fields from indexing

Analyze your data: Before excluding fields from indexing, it’s essential to analyze your data and understand which fields are necessary for search and aggregation. This will help you make informed decisions about which fields to exclude.
Test your changes: When excluding fields from indexing, it’s crucial to test your changes to ensure that your search and aggregation functionality still work as expected. This can help you avoid any unexpected issues or performance problems.
Monitor performance: After excluding fields from indexing, monitor the performance of your Elasticsearch cluster to ensure that your changes have had the desired effect. This can help you identify any additional optimizations that may be required.
Use source filtering: If you need to store a field in Elasticsearch but don’t want it to be searchable or available for aggregations, consider using source filtering. This allows you to store the field in the _source field but exclude it from the index.

Conclusion

Excluding fields from indexing in Elasticsearch can help improve performance, reduce storage costs, and minimize the overall size of your index. By carefully analyzing your data and understanding which fields are necessary for search and aggregation, you can make informed decisions about which fields to exclude. Always test your changes and monitor the performance of your Elasticsearch cluster to ensure that your optimizations have the desired effect.

Ready to try this out on your own? Start a free trial.

Want to get Elastic certified? Find out when the next Elasticsearch Engineer training is running!