Skip to main content

Less noise, more signal: How Elastic Defend slashed event volume

By

Daniel Ferullo

blogheaderimage-lessnoise-moresignal-howelasticdefendslashedev_177290.jpg

When an EDR tool generates too much endpoint telemetry, security teams quickly run into problems. Mountains of process events, network connections, and file operations can overwhelm analysts, making it harder to spot real threats in the noise. High data volumes drive up storage costs, slow down searches, and contribute to alert fatigue — leading to longer investigation times and potential blind spots. The challenge isn’t just collecting endpoint data; it’s ensuring the right signals stand out when they matter most. 

Going back a few releases to 8.14 we’ve been on a drive to reduce Elastic Defend’s default data volume. Now, with the release of 8.18, it is a good time to summarize those changes and give an example of their impact.

The best part of these changes is that none of them affect Elastic Defend’s protection level or visibility because we focused on just improving data efficiency. Our priority was making sure Elastic Defend continues to offer industry leading protection, just at a lower cost for users. Also, while these changes are the new defaults none of them are on by default for existing Elastic Defend policies. Every enhancement is controlled by a unique Elastic Defend advanced option, and all existing policies use those options to maintain the prior behavior after an upgrade.

Impact of Elastic Defend improvements

Before describing what we’ve done, let’s start by seeing their impact. With the cross platform workflow I selected (compiling the Elastic Defend source code), 8.13 Elastic Defend generated 2.18GB of uncompressed JSON on Linux, 2.17GB on macOS, and 714MB on Windows. For that same workflow, by default, 8.18 Elastic Defend generated 716MB on Linux, 949MB on macOS, and 379MB on Windows. That means we saw a 68% data volume reduction on Linux, 57% reduction on macOS, and 48% reduction on Windows!

Even better, by adding a single, simple process descendant event filter that matched on the command line used to compile Elastic Defend almost all data was suppressed, only 3MB–6MB of system background data was generated across each operating system.

Comparison between 8.13 and 8.18 event volume
Comparison between 8.13 and 8.18 event volume from Elastic Defend

It took a few changes to cause such a big reduction. Let me go over them.

What’s new in Elastic Defend

Looking back: 8.14 and 8.15

In 8.14, we started by no longer reporting Linux process capabilities in process events by default. Then in 8.15, we removed process ancestry from non-process events and reduced the default ancestry count. We started suppressing duplicate network events from the same process, and we began filtering out common, known-benign Windows registry activity. We also added process descendant event filtering — a key capability that lets you filter out all events from any child descendant of the matching process (for instance a script). As a reminder, event filters do not reduce Elastic Defend’s on-host visibility or protection level.

Today: 8.18

Now in 8.18, we’ve added more enhancements. Process lifecycle events (fork/exec/exit on POSIX and create/exit on Windows) will be merged into a single event for short lived processes. Similarly, network connect or accept events will be merged with their terminate event for short lived network connections. To save both CPU cycles and data volume, we’re no longer computing legacy MD5 or SHA1 hashes by default — just SHA256 hashes are included. Finally, we’re no longer including all host fieldset information in events but are still including it all in alerts and state management documents.

Try it out!

This isn’t the end of our effort to optimize Elastic Defend’s data volume, but it does mark an important milestone. We’re proud of reducing the default data volume and giving users a new event filter tool, ultimately driving down the cost of using Elastic Defend. If you haven’t already tried Elastic Defend, go check it out with a free trial and if you have tried it before, check it out again!

If you want to modify any of these new behaviors, use the Elastic Defend advanced options listed below. Each is described in detail in Kibana with a tooltip — to see them click Show advanced settings when editing an Elastic Defend policy.

  • [linux|mac|windows].advanced.events.process_ancestry_length

  • [linux|mac|windows].advanced.events.ancestry_in_all_events

  • [linux|mac|windows].advanced.events.deduplicate_network_events

  • [linux|mac|windows].advanced.events.deduplicate_network_events_below_bytes

  • windows.advanced.events.enforce_registry_filters

  • [linux|mac|windows].advanced.events.aggregate_process

  • [linux|mac|windows].advanced.events.aggregate_network

  • [linux|mac|windows].advanced.alerts.hash.md5

  • [linux|mac|windows].advanced.alerts.hash.sha1

  • [linux|mac|windows].advanced.events.hash.md5

  • [linux|mac|windows].advanced.events.hash.sha1

  • [linux|mac|windows].advanced.events.hash.sha256

  • [linux|mac|windows].advanced.set_extended_host_information

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Sign up for Elastic Cloud free trial

Spin up a fully loaded deployment on the cloud provider you choose. As the company behind Elasticsearch, we bring our features and support to your Elastic clusters in the cloud.

Start free trial