Introduction

Elastic has fully embraced OpenTelemetry as the backbone of its data ingestion strategy, aligning with the open-source community and contributing to make it the best data collection platform for a broad user base. This move benefits users by providing enhanced flexibility, efficiency, and control over telemetry data.

Why OpenTelemetry?

OpenTelemetry provides a powerful set of capabilities that make it a compelling choice for open-source-focused users. Elastic is re-architecting its data ingest tools around OpenTelemetry to offer users vendor-agnostic flexibility, performance optimization through OTel's efficient data model for correlating telemetry, and enhanced flexibility and control over data pipelines. This move brings the benefits of open-source telemetry to Elastic users.

Elastic engineers are active contributors to the Otel project in several areas of the project. Demonstrating its commitment to open source, Elastic continues to make significant contributions to OpenTelemetry.

OpenTelemetry as the Core of Elastic's Data Ingestion

Elastic is transforming its data ingestion strategy by basing all ingestion mechanisms on the OpenTelemetry components. Elastic currently supports the following OTel based ingest architecture, which support OTel SDKs and Collectors from OTel or Elastic's Distribution of OpenTelemetry (EDOT).

This marks a fundamental shift, ensuring a more standardized and scalable telemetry pipeline. All the existing Elastic ingest components will become OTel based.

Let's discuss how each component of Elastic's data ingestion platform will be based on an OpenTelemetry collector whilst still providing the same functionality to the user.

Beats

Elastic's traditional data shippers will be re-architected as OpenTelemetry Collectors, aligning with OTel's extensibility model. Current Beat architecture is essentially made up of a few stages in its pipeline, as shown in the diagram below. It consists of an Input, Processors for enrichments, Queuing of events and Output for batching and writing the data to a specific output.

Beatreceiver Concept

To ensure a smooth transition without major disruptions, a "beatsreceiver" concept is being implemented. These

(like or ) act as dedicated Beat inputs integrated into the OpenTelemetry Collector as native receivers. They support all existing inputs and processors, guaranteeing that the final architecture accepts the user's current configuration and delivers the same functionality as today's Beats, all without introducing any breaking changes.

An OTel based Beats architecture will see the Input phase embedded as an OTel receiver (eg. 

to represent the functionality of ). This receiver would only be available as part of Elastic's distribution of OTel in support of our current user base and not a functionality that would be available upstream.

All the remaining components of the pipeline will be based on OTel. The new Beat will accept the same filebeat configuration (as an example) and will transform it to an OTel based configuration in order to avoid any deployment disruption. It should be noted that in this architecture the Beats will continue to only support ECS formatted data. In order to keep the Beat functionality inline with what exists today, the Elasticsearch exporter (as an example) will output ECS formatted data only.

The following diagram illustrates the

concept by showing how a basic configuration is automatically translated into an OpenTelemetry-based configuration. This new configuration retains the original inputs and processors but leverages the native OpenTelemetry pipeline and exporter to achieve the same overall functionality. Existing configurations will be automatically converted, eliminating the need for manual adjustments or introducing breaking changes.

Elastic Agent

Elastic Agent is a unified agent for data collection, security, and observability. It can also be deployed in an OpenTelemetry only mode, enabling native OTel workflows. Elastic Agent is a supervisor that manages many other Beats as sub-processes in order to provide a more comprehensive data collection tool. It is capable of translating Agent Policy received from Fleet into configuration acceptable by the various sub-processes.

Expanding on the Beat receiver concept described above, the Elastic Agent, which currently can be deployed as an OTel collector (see blog), will be also modified to a much simpler OTel based architecture based on these receivers. As shown below, this architecture will streamline the components within the Elastic Agent and remove duplicated functionality such as queuing and output. Whilst supporting the current functionality, these changes will reduce the agent footprint and also present a reduction in number of connections opened to pipeline elements egress of the agent (such as Elasticsearch clusters, Logstash or Kafka brokers).

By moving to an OTel based architecture Elastic Agent is now able to operate as a truly hybrid Elastic Agent which provides not only the Beat functionality but also allows our users to create OTel native pipelines and take advantage of plethora of functionality available as part of the open source project.

Elastic's commitment to OpenTelemetry will deepen through increased contributions, resulting in OpenTelemetry receivers gradually superseding Beats receiver features. This evolution will eventually reduce the need for a distinct Beats receiver within the Elastic Agent architecture. The envisioned architecture will empower the Elastic Agent to transmit data in OTLP format as well, granting users the flexibility to select any OTLP-compatible backend, thereby upholding the principle of vendor neutrality.

Fleet & Integrations: Managing OpenTelemetry at Scale

Elastic's centralized management system will support OpenTelemetry-based configurations, making large-scale deployments easier to manage. Managing thousands of telemetry agents at scale presents a significant challenge. Elastic's Fleet & Integrations simplify this process by providing robust lifecycle management for these new OpenTelemetry-based Elastic agents.

Key Capabilities Offered:

• Scalability: Manage up to 100K+ agents across distributed environments.

• Automated Upgrades: Staged rollouts and automatic upgrades ensure minimal downtime.

• Monitoring & Diagnostics: Real-time status updates, failure detection, and diagnostic downloads improve system reliability.

• Policy-Based Configuration Management: Enables centralized control over agent configurations, improving consistency across deployments.

• Pre-Built Integrations: Elastic offers a catalog of 470+ pre-built integrations, allowing users to ingest data seamlessly from various sources. These will also include OTel based packages making configuration much more efficient across a large deployment.

The goal is for Fleet to also provide monitoring capabilities for native OTel collectors as well in a vendor agnostic fashion.

Conclusion

Elastic's adoption of OpenTelemetry marks a significant milestone in the evolution of open-source observability. By standardizing on OpenTelemetry, Elastic is ensuring that its data ingestion strategy remains open, scalable, and future-proof.

For open-source users, this shift means:

• Greater interoperability across observability tools.

• Enhanced flexibility in choosing telemetry backends.

• A stronger commitment to community-driven observability standards.

• Existing Beats and Elastic Agent users can seamlessly adopt OpenTelemetry without rearchitecting their pipelines.

• OpenTelemetry users can integrate with Elastic's observability stack without additional complexity.

Stay tuned for more updates as Elastic continues to expand its OpenTelemetry-based data collection capabilities! In the mean time here are some other references:

• Elastic Distributions of OpenTelemetry (EDOT) Now GA

• Dynamic workload discovery on Kubernetes now supported with EDOT Collector

• Introducing the OTTL Playground for OpenTelemetry