How to Propagate OpenTelemetry Trace Headers Over AWS Kinesis: Part 1

A user request reaches your application and generates an event. Instead of being processed immediately, the event is written to an AWS Kinesis stream, where a downstream application or AWS Lambda function processes it later. From a business perspective, it’s still one transaction. From an observability perspective, it often becomes two unrelated traces.

This happens because the trace context that identifies a distributed request doesn’t automatically travel with a Kinesis record. Unlike an HTTP request, which has headers for carrying trace context, Kinesis records don’t provide a dedicated place for OpenTelemetry to propagate the traceparent header. Without that context, the downstream consumer starts a new trace instead of continuing the original one. Engineers investigating an incident must then manually correlate logs, timestamps, and request identifiers to understand what actually happened.

This series explores practical approaches to solving that problem. In this article, we’ll examine why trace context is lost across AWS Kinesis, the tradeoffs of using the PartitionKey to preserve it, and what those tradeoffs mean for event ordering and application behavior.

Understanding trace context propagation

Trace context propagation is the mechanism that carries a request’s tracing information across service boundaries so telemetry from each service belongs to one continuous trace. In OpenTelemetry, that context can be represented by traceparent, by tracestate when it’s used, and by baggage when the application requires it.

The mechanism relies on a propagator. A standards-compatible propagator injects the context on the outbound side and extracts it on the inbound side, rather than simply attaching a trace ID and span ID and hoping the next service reads them. That inject-and-extract contract is what keeps the context intact as it moves.

What is trace context?

Trace context contains the identifiers and processing information needed to associate telemetry across service boundaries. Under W3C Trace Context, the traceparent value carries the trace ID, the parent span ID, a version, and trace flags. The tracestate value can carry vendor-specific state alongside it, and baggage can carry application-defined key-value data when a team needs it.

How is trace context propagated?

OpenTelemetry provides the propagation APIs and the propagators that define the format, but it doesn’t move the context on its own. Application code, or supported instrumentation, still needs an appropriate carrier to inject the context into on the way out and extract it from on the way in. When a service handles a request, it extracts the incoming context, records its own spans against it, and injects the context into whatever it sends downstream. Each service in the path contributes to one unified trace.

The role of cross-service tracing

With context propagated correctly, you get cross-service tracing. You can follow a request from where it starts, through every service it touches, to where it completes, and see the full path as a single connected view instead of a set of disconnected fragments.

How trace continuity improves troubleshooting in LogicMonitor

Preserving trace context is what keeps a request connected as it moves from one service to another. When OpenTelemetry sends trace data through the OpenTelemetry Collector to LogicMonitor Distributed Tracing, the entire request appears as a single trace instead of separate, disconnected ones.

That means when a problem occurs, engineers can follow the request from the producer to the consumer in one view, alongside the metrics and logs for the services involved. Instead of manually matching timestamps, request IDs, and log entries across multiple systems, they can quickly see where the request slowed down, failed, or spent most of its time. OpenTelemetry captures and exports the trace data, while LogicMonitor brings traces, metrics, and logs together to help teams investigate and resolve issues faster.

Interoperability and standardization in trace propagation

Because OpenTelemetry follows standardized formats like W3C Trace Context, different tracing tools and platforms stay compatible and interoperable. Integrating an observability solution, or switching between them, becomes much simpler.

The significance of trace propagation in distributed systems

In microservices or distributed architectures, a single user request often involves many services. Trace context propagation lets developers and operators see the request’s entire path, which makes it easier to diagnose issues, understand service dependencies, and improve performance.

Why automatic trace propagation doesn’t work with Kinesis

Monitoring distributed applications depends on keeping trace context intact as requests move between services. OpenTelemetry can automatically propagate trace context when the transport provides a standard place to carry it. With HTTP, that place is the request headers, allowing trace context to be injected and extracted without modifying the request body.

AWS Kinesis works differently. A Kinesis record doesn’t have a dedicated header where trace context can be stored. Each record contains only the payload, with no built-in location for tracing metadata. Because there isn’t a standard carrier for trace context, OpenTelemetry can’t automatically propagate it across the Kinesis boundary. If trace continuity is required, the application itself must define how trace context is passed from the producer to the consumer.

A baseline approach: embedding trace context in Kinesis records

One possible carrier strategy is to embed the trace context inside the record payload: the producer writes the context into the data it sends, and the consumer reads it back out. We introduce it here mainly to establish the challenges the later articles address, not as a universally recommended pattern.

Here’s how a producer-to-consumer flow looks with this approach:

1. An order service creates an OpenTelemetry producer span for the operation.
2. The producer injects the active context into a defined carrier inside the record.
3. The application sends the event to AWS Kinesis.
4. A Lambda consumer reads the event and extracts the context from the carrier.
5. The consumer creates its processing telemetry, associated with the upstream operation.
6. The spans are exported through an OpenTelemetry Collector and made available for investigation.

The steps below break down what happens on each side of that flow.

Embedding trace context in Kinesis records (producer side)

• When the producer sends a record, it includes the trace context in the record data, obtained from the OpenTelemetry SDK or a similar tracing tool.
• The context should follow a standard format, such as W3C Trace Context, for compatibility.

Sending records to AWS Kinesis

• Use the AWS SDK to send each record, making sure the trace context is included in the payload.
• Keep the context intact and readable so the consumer can interpret it correctly.

Extracting trace context in Kinesis records (consumer side)

• Applications reading from the stream extract the trace context from each record.
• This means parsing the record data to retrieve the context.
• Once extracted, the consumer uses the context to continue the trace, linking its processing to the trace the producer started.

Continuing the trace

• The consumer uses the extracted context to annotate its own processing, creating spans logically connected to the producer’s spans.
• A direct parent-child relationship can work for a simple one-message-to-one-consumer flow.
• Batched, delayed, retried, or fan-out processing may require span links or other messaging-specific modeling instead of a single parent-child link.

Part 2 and Part 3 use a simplified scenario, one message to one consumer, to keep the focus on the Kinesis propagation mechanism itself.

What are the challenges with modifying records to carry trace context over AWS Kinesis?

Writing trace context into the original content of a record introduces several concerns, since producers now embed the context and consumers must extract and interpret it correctly. The most important ones to keep in mind at this stage:

• Schema and contract compatibility: downstream systems that expect a specific structure can break when the payload shape changes.
• Record-size overhead: added fields increase each record’s size, which can affect throughput and cost.
• Signed, hashed, or immutable payloads: any change invalidates a signature or hash, or violates an immutability requirement.
• Sensitive data in baggage: context carried as baggage can accidentally include data that shouldn’t travel with the record.
• Producer and consumer version compatibility: both sides need to agree on where the context lives and how it’s encoded.
• Batch processing and multiple upstream contexts: a single batch can carry records from several upstream operations at once.
• Retries, fan-out, and delayed consumption: asynchronous patterns complicate how spans relate to one another.
• Partitioning and ordering effects: using alternative carrier fields, such as a partition key, can change how records are distributed and ordered.

On data integrity specifically, modifying the payload can conflict with strict contracts, signature verification, or requirements that business data remain unchanged. When you add a documented field to an extensible schema, the risk shifts from “always unsafe” to “it depends on what consumers validate and how they evolve their models.” In the rest of the series, we’ll look at carrier options that preserve trace continuity and keep those integrity guarantees intact.4210

What the rest of the series covers

This first part shows why Kinesis creates a trace‑context boundary and where payload‑based propagation starts to break down. In Part 2, we’ll move trace context into Kinesis parameters, and in Part 3 we’ll finish the implementation by reconciling trace continuity with partitioning and event ordering.

From trace continuity to observability with LM Envision

AWS Kinesis introduces a unique challenge for distributed tracing because it doesn’t provide the dedicated trace-context carrier available in HTTP. Preserving trace context across that boundary is essential for maintaining complete end-to-end traces, but the implementation must also respect application behavior and data integrity.

Once those traces are exported through OpenTelemetry, LM Envision brings them together with service metrics, infrastructure telemetry, and logs in a single view. Instead of manually correlating data across multiple systems, teams can follow a request across distributed services, identify where latency or failures occur, and move from isolated traces to a complete operational picture.

FAQs

By Denton Chikura

Technical Writer

Denton Chikura is a technical writer and longtime observability advocate focused on helping site reliability engineers and engineering teams discover the tools and capabilities that strengthen internet resilience. He works at the intersection of monitoring, performance, and infrastructure to make complex systems more understandable and usable, bridging the gap between deep technical detail and real‑world operations. His goal is to help teams build faster, detect issues earlier, and recover smarter, ultimately making the internet a better, more reliable place for everyone.

Disclaimer: The views expressed on this blog are those of the author and do not necessarily reflect the views of LogicMonitor or its affiliates.