Modern applications often run across multiple services, containers, databases, and cloud environments. When a request becomes slow or an application starts returning errors, developers need reliable information to identify where the problem occurs.
Logs explain what happened, metrics show how a system behaves over time, and distributed traces reveal how a request travels across different services. Collecting this telemetry is only part of the challenge. Applications also need a reliable way to process, filter, and export it to monitoring platforms.
The OpenTelemetry Collector provides a vendor-neutral solution for handling telemetry data. Instead of configuring every application to send data directly to a monitoring backend, developers can send telemetry to a Collector that processes and routes it to the appropriate destinations.
This article explains how the OpenTelemetry Collector works, how to configure its pipelines, how to filter sensitive information, and how to deploy it for production monitoring.
What Is the OpenTelemetry Collector?
The OpenTelemetry Collector is a service that receives, processes, and exports telemetry data. It supports three main types of observability signals:
- Traces, which describe the path of a request through an application.
- Metrics, which measure values such as request duration, error rates, and resource usage.
- Logs, which record events generated by applications and infrastructure.
The Collector is part of the OpenTelemetry ecosystem, which provides APIs, SDKs, instrumentation, and protocols for generating and transporting telemetry.
A typical architecture looks like this:
Application A ──┐
│
Application B ──┼──> OpenTelemetry Collector
│ │
Application C ──┘ ├──> Metrics Backend
├──> Trace Backend
└──> Log Backend
Applications send telemetry to the Collector. The Collector can then apply configured processing rules before exporting the data to one or more destinations.
For example, a web application might send traces through the OpenTelemetry Protocol, commonly called OTLP. The Collector can receive those traces, batch them, remove selected attributes, and forward them to a compatible tracing backend.
This architecture separates application instrumentation from backend-specific configuration. If an organization changes its monitoring platform, it may be possible to update the Collector's exporters without changing every application.
However, the Collector does not automatically discover every application or generate complete telemetry by itself. Applications and infrastructure still need appropriate instrumentation and configuration.
Understand the Collector Architecture
A Collector pipeline consists of three primary components: receivers, processors, and exporters.
Receivers: Accept Telemetry
Receivers define how telemetry enters the Collector.
Common receiver types include:
- OTLP receivers for telemetry sent using the OpenTelemetry Protocol.
- Host metrics receivers for collecting supported host-level measurements.
- File-based receivers for supported log collection scenarios.
- Prometheus receivers for scraping compatible metrics endpoints.
Receiver availability depends on the Collector distribution. The OpenTelemetry Collector core and the OpenTelemetry Collector Contrib distribution do not contain exactly the same components.
Processors: Transform Telemetry
Processors operate on telemetry before it is exported.
Common processing tasks include:
- Batching telemetry to reduce export overhead.
- Filtering unwanted data.
- Adding or modifying attributes.
- Removing sensitive fields.
- Limiting memory usage.
- Sampling traces under defined conditions.
Processors execute in the order specified in the pipeline. Their configuration should reflect the intended processing sequence.
For example, a pipeline might remove sensitive attributes before sending telemetry to an external backend.
Exporters: Send Telemetry to Destinations
Exporters send processed telemetry to monitoring systems or other supported destinations.
An exporter might send traces to a tracing backend, metrics to a monitoring platform, or logs to a log aggregation service.
Some exporters support multiple signal types, while others support only specific signals. Authentication, endpoint configuration, and supported data formats depend on the exporter and destination.
A Collector can also use multiple pipelines to process different signals independently.
Install and Configure the Collector
The OpenTelemetry Collector can be deployed as a standalone process, a container, or part of a Kubernetes environment.
For a container-based deployment, use an appropriate Collector image and a configuration file. The following example illustrates the configuration structure.
Create a file named otel-collector.yaml:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
memory_limiter:
check_interval: 1s
limit_mib: 512
batch:
timeout: 5s
send_batch_size: 1024
exporters:
debug:
verbosity: basic
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [debug]
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [debug]
logs:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [debug]
This configuration defines an OTLP receiver for gRPC and HTTP traffic. It then creates separate pipelines for traces, metrics, and logs.
Each pipeline uses a memory limiter and batch processor before sending data to the debug exporter.
The debug exporter is useful during development because it helps verify that the Collector receives and processes telemetry. It is not a substitute for a production telemetry backend.
The memory limit is an example value, not a universal recommendation. Adjust it according to the container's memory limit, telemetry volume, and processor requirements.
Run the Collector
Run the configuration using a Collector distribution that includes the specified components.
For example, with a compatible container image:
docker run --rm \
-p 4317:4317 \
-p 4318:4318 \
-v "$(pwd)/otel-collector.yaml:/etc/otelcol/config.yaml:ro" \
otel/opentelemetry-collector-contrib:<pinned-version> \
--config=/etc/otelcol/config.yaml
Replace <pinned-version> with an actual version supported by your deployment.
The example uses the Contrib distribution because it includes a broader collection of components. For production, pin a specific image version rather than using an unqualified or floating version.
The command publishes the OTLP ports on all host interfaces by default. Do not expose these ports publicly without appropriate network restrictions, authentication, and transport security. For local-only testing, bind the ports to the loopback interface.
Configure Applications to Send Telemetry
The Collector can receive telemetry only when applications or other supported sources send it.
Consider a .NET application using OpenTelemetry instrumentation. The application can configure its telemetry SDK to export traces to the Collector's OTLP endpoint.
A simplified example using the OpenTelemetry .NET packages looks like this:
using OpenTelemetry;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
var builder = WebApplication.CreateBuilder(args);
builder.Services
.AddOpenTelemetry()
.ConfigureResource(resource =>
resource.AddService("orders-api"))
.WithTracing(tracing =>
{
tracing
.AddAspNetCoreInstrumentation()
.AddHttpClientInstrumentation()
.AddOtlpExporter(options =>
{
options.Endpoint =
new Uri("http://localhost:4317");
});
});
var app = builder.Build();
app.MapGet("/", () => "Orders API is running.");
app.Run();
Install the appropriate OpenTelemetry .NET packages before using this example. The exact package set depends on the instrumentation and exporter features required by the application.
The example configures ASP.NET Core and outgoing HTTP instrumentation, allowing supported operations to generate spans. It then exports traces using OTLP.
The endpoint assumes that the application and Collector can communicate through localhost. In a containerized environment, localhost refers to the current container, not another container. Use the appropriate service name or network address when the Collector runs separately.
Metrics and logs require their own instrumentation and exporter configuration. Configuring tracing alone does not automatically enable every telemetry signal.
Filter Sensitive Telemetry Before Exporting
Telemetry can contain sensitive information, especially when applications attach request attributes, user identifiers, or database-related details to spans.
Sending this information to an external backend without reviewing it can create privacy and compliance risks.
The Collector supports processors that can remove or filter telemetry attributes. For example, the following processor configuration demonstrates removing selected span attributes:
processors:
transform:
error_mode: ignore
trace_statements:
- context: span
statements:
- delete_key(attributes, "user.email")
- delete_key(attributes, "user.phone")
To use this processor, add it to the relevant trace pipeline:
service:
pipelines:
traces:
receivers: [otlp]
processors: [transform, batch]
exporters: [debug]
This example focuses on specific attributes. The Transform Processor's availability and configuration syntax depend on the Collector distribution and version.
The error_mode: ignore setting controls how the processor handles statement errors. It does not guarantee that all sensitive information is removed.
A production policy should identify sensitive fields at the source and remove them wherever necessary. For example, HTTP headers, query parameters, exception messages, and custom application attributes may expose confidential information even when the two example attributes are deleted.
Filtering rules should be tested with representative telemetry before deployment.
Control Telemetry Volume With Batching and Sampling
Collecting every span, log entry, and metric can increase network usage, storage requirements, and backend costs.
Two common strategies are batching and sampling.
Use Batching to Reduce Export Overhead
Batch processors group telemetry before exporting it. This can reduce the number of export operations and improve throughput.
For example:
processors:
batch:
timeout: 5s
send_batch_size: 1024
The timeout controls how long the processor waits before sending a batch under applicable conditions. The send batch size is a target rather than a strict maximum in all circumstances.
Batching should be tested against the expected traffic pattern. Large batches may improve throughput but can increase memory usage and the time before data reaches the backend.
Use Sampling for High-Volume Traces
Sampling reduces the number of traces retained or exported.
Head sampling makes a decision when a trace begins, while tail sampling evaluates traces after spans have been collected. Tail sampling can make decisions based on trace characteristics, but it requires the relevant spans to reach the same sampling decision point and requires sufficient buffering and memory.
A Collector deployment that uses tail sampling must be designed carefully for distributed workloads. If spans from the same trace are split across different Collector instances, sampling decisions may become incomplete or inconsistent unless the architecture routes those spans appropriately.
Do not apply sampling blindly. Security investigations, billing operations, and critical business workflows may require different retention policies from routine application requests.
Deploy the Collector in Production
A production deployment needs more than a working YAML file.
Protect Receiver Endpoints
Telemetry receivers should accept data only from trusted sources. Restrict network access, configure authentication where supported, and use TLS when telemetry crosses untrusted networks.
Do not assume that an OTLP endpoint is safe simply because it does not expose a conventional application API. An exposed receiver can be abused to inject data or consume system resources.
Monitor the Collector
The Collector is part of the observability infrastructure, so it also needs monitoring.
Track relevant indicators such as:
- Received telemetry volume.
- Exported telemetry volume.
- Export failures and retry activity.
- Queue sizes and dropped data.
- Memory consumption.
- Receiver and exporter errors.
The available metrics and names depend on the Collector version and component configuration. Use the telemetry and internal monitoring mechanisms supported by your deployment.
Alerts should identify when telemetry stops flowing, not only when an application reports an error. Otherwise, a broken monitoring pipeline may go unnoticed.
Plan for Backpressure and Backend Outages
A telemetry backend may become unavailable or respond slowly. Depending on the exporter and pipeline configuration, the Collector may retry requests, buffer data, or eventually drop telemetry.
Some exporters support sending queues and persistent storage options. These features must be configured explicitly when needed; they should not be assumed to be enabled automatically.
Persistent queues can reduce data loss during restarts, but they require storage capacity, operational monitoring, and a retention policy. They also cannot guarantee delivery if the storage itself fails or fills up.
Test backend outages and recovery behavior before relying on the Collector for production observability.
Common Mistakes to Avoid
Using Components That Are Not Installed
A configuration can reference a receiver or processor that is not included in the selected Collector distribution.
Check the components available in your chosen build and validate the configuration before deployment.
Treating the Debug Exporter as a Production Destination
The debug exporter is helpful for inspecting telemetry during development. Production deployments should use a suitable backend exporter and protect any credentials it requires.
Ignoring Memory Limits
Large batches, high traffic volumes, and trace sampling buffers can increase memory usage.
Configure the memory limiter, set appropriate container resource limits, and test the deployment under realistic workloads.
Assuming Telemetry Is Automatically Private
Telemetry may include personal information, request details, and internal system metadata.
Define a data-handling policy, avoid collecting unnecessary sensitive fields, and validate filtering rules before exporting data.
Failing to Test Exporter Errors
A Collector that accepts telemetry but cannot export it may silently lose visibility into application behavior if failures are not monitored.
Test destination outages, authentication failures, queue saturation, and recovery procedures.
Advantages and Limitations
Area | Advantages | Limitations |
|---|
Vendor independence | Separates instrumentation from backend-specific configuration | Backend-specific features may still require specialized configuration |
Processing | Supports batching, filtering, and other transformations | Incorrect rules can remove useful telemetry |
Scalability | Can be deployed across services and infrastructure | High-volume deployments require capacity planning |
Multiple signals | Supports traces, metrics, and logs through pipelines | Each signal may need different receivers, processors, and exporters |
Reliability | Queues and retry mechanisms can help manage temporary failures | Delivery is not guaranteed under every failure condition |
Security | Can filter attributes and use secured receiver and exporter configurations | Security depends on correct configuration and network controls |
Summary
The OpenTelemetry Collector provides a flexible way to receive, process, and export telemetry from distributed applications. Its receiver, processor, and exporter architecture allows teams to centralize telemetry handling without coupling every application directly to a specific monitoring backend.
A practical deployment begins with an OTLP receiver, adds batching and memory protection, and exports telemetry to a development or production destination. From there, teams can introduce filtering, sampling, authentication, and persistent queues based on their operational requirements.
For production systems, validate the configuration, restrict access to receiver endpoints, monitor export failures, and test backend outages. These measures help ensure that the observability pipeline remains useful when the applications it monitors encounter problems.
Join the conversation! Your thoughts help the community grow.