The countdown to Elevate 2026 is on. Join us in Chicago, London, or Sydney.

Register here

Partners

Docs

LM Academy

LM Community

Pricing

Platform
Solution
Industry
Role
There is no result.
Try it free

14-day access to the full LogicMonitor platform

The countdown to Elevate 2026 is on. Join us in Chicago, London, or Sydney.

Register here

LOGICMONITOR BLOG

How to Propagate OpenTelemetry Trace Headers Over AWS Kinesis: Part 2

Walk through a hands-on baseline for propagating OpenTelemetry trace context in AWS Kinesis, and see why reusing the PartitionKey breaks event ordering.

5–8 minutes
Denton Chikura

NEWSLETTER

Subscribe to our newsletter

Get the latest blogs, whitepapers, eGuides, and more straight into your inbox.

SHARE

The quick download

If you reuse a field that already has a purpose in your application to carry trace context, you’ll break the existing behavior that depends on that field.

  • Without help, OpenTelemetry keeps producer and consumer in separate traces across AWS Kinesis, since trace context does not travel with an asynchronous message on its own.

  • Injecting the traceparent header into the Kinesis PartitionKey does connect both services into a single trace.

  • That same PartitionKey controls shard placement, so overwriting it scatters same-group events across shards and breaks ordering, opening the door to race conditions.

  • Propagate trace context through a field with no application semantics, then use your observability tooling to correlate those traces with underlying infrastructure and cloud signals.

In the first article of our series, we explored the importance of trace headers and the challenges of propagating them across asynchronous systems. In this second installment, we move from theory to practice.

When services communicate asynchronously, it’s easy to lose track of how a request flows from one service to another. When a producer hands an event to a queue and a consumer processes it later, trace context doesn’t automatically travel with the message. As a result, the producer and consumer end up in separate traces, making it harder to follow a single request across service boundaries, identify root causes, and troubleshoot distributed systems effectively.

Keeping trace context intact across asynchronous messaging is what makes reliable distributed tracing, and ultimately end-to-end observability, possible. It connects a user’s action to the code that served it and provides the context needed to understand how requests flow across distributed services. This article walks through a hands-on baseline implementation for propagating OpenTelemetry trace context in AWS Kinesis using the PartitionKey parameter.

Baseline exploration

To begin, the following sections examine the automated instrumentation and trace context propagation provided by the OpenTelemetry SDK, focusing on its default capabilities. The test scenario involves two interconnected services within AWS Kinesis: a producer, which generates and sends events, and a consumer, which receives and processes them. This initial scenario reveals the natural behavior of trace propagation between the services without any alterations to the system.

The test scenario and configuration are kept intentionally simple to focus on the main problem:

  • We have an AWS Kinesis stream with 2 shards.
  • We have an app named event-producer.
  • The app produces events and sends them to the AWS Kinesis stream by AWS SDK.
  • Events are partitioned across shards by event group ID, so events under the same group are put into the same shard and processed sequentially.
  • The event-producer app is automatically instrumented and traced by OTEL Java agent (version 1.30.1).
  • We have an app named event-consumer.
  • The app is an AWS Lambda function (otel-lambda-playground-kinesis-handler) and is triggered from the AWS Kinesis stream.
  • The batch size for the trigger was set to 1 to prevent multiple upstream trace link cases (the events are processed together here as a batch in the same invocation, but each event is put to the stream at different downstream traces in the event-producer application). We cover trace-linking for multi-event batches separately.
  • The event-consumer AWS Lambda function handler is wrapped and so automatically traced by OTEL Java AWS Lambda packages (opentelemetry-aws-lambda-core-1.0 and opentelemetry-aws-lambda-events-2.2).

When this test scenario runs, both the event-producer and event-consumer applications are automatically traced by OTEL SDK, but although they belong to the same flow, two different traces are created, independent of each other.

Image 1 – Trace of the event-producer application

Image 2 – Trace of the event-consumer application

1st Attempt – Propagate Trace Context Through “PartitionKey”

In the AWS Kinesis PutRecord request model, there are six parameters:

  • Data
  • ExplicitHashKey
  • PartitionKey
  • SequenceNumberForOrdering
  • StreamARN
  • StreamName

However, we cannot use or change many parameters here to propagate the traceparent header:

  • The Data parameter should not be changed, because earlier sections describe the kinds of problems that can occur if the request body is modified.
  • The StreamName and StreamARN parameters also cannot be changed, because they specify which AWS Kinesis stream receives the event.
  • The SequenceNumberForOrdering parameter likewise should not be changed, because it controls the ordering of events from the same client to the same shard; without a monotonically increasing sequence number, some records may be ignored without being processed.

Therefore, in this first attempt, the PartitionKey parameter is used to propagate the traceparent header. This should be treated as an experiment, not a production pattern. The goal is to test whether the field can carry trace context, not to recommend shipping it this way.

For this, we set the traceparent header to the PartitionKey parameter in W3C context header format manually, as in the sample code block below:

private PutRecordRequest injectTraceHeader(PutRecordRequest request){ 
    if (!TRACE_CONTEXT_PROPAGATION_ENABLED) { 
        return request; 
    } 
    Span currentSpan = Span.current(); 
    if (currentSpan == null) { 
        return request; 
    } 
    SpanContext currentSpanContext = currentSpan.getSpanContext(); 
    if (currentSpanContext == null) { 
        return request; 
    } 
    PutRecordRequest.Builder requestBuilder = request.toBuilder(); 
    String traceParent = String.format("00-%s-%s-%s", 
            currentSpanContext.getTraceId(), 
            currentSpanContext.getSpanId(), 
            currentSpanContext.getTraceFlags().asHex()); 
    requestBuilder.partitionKey(traceParent);  
    return requestBuilder.build(); 
}

With this approach, the traceparent header is passed to the event-consumer AWS Lambda function within the Kinesis event via the PartitionKey parameter. Because this is not a standard propagation mechanism, the traceparent header must be extracted manually from the PartitionKey parameter on the event-consumer and injected into the trace context. This allows the event-consumer trace to reuse the propagated trace ID so that both the event-producer and the event-consumer applications appear in the same trace.

public class KinesisHandler extends TracingRequestHandler<KinesisEvent, Void> {
    private static final String TRACE_PARENT = "traceparent"; 
    private static final String TRACE_PARENT_PREFIX = "00-"; 
    ... 
    
    @Override 
    protected Void doHandleRequest(KinesisEvent event, Context context) {
		... 
        return null; 
    } 
    @Override 
    protected Map<String, String> extractHttpHeaders(KinesisEvent event) { 
        List<KinesisEvent.KinesisEventRecord> records = event.getRecords(); 
        if (!records.isEmpty()) { 
            Map<String, String> headers = new HashMap<>(); 
            KinesisEvent.KinesisEventRecord record = records.get(0); 
            String partitionKey = record.getKinesis().getPartitionKey(); 
            if (partitionKey != null  
                    && partitionKey.startsWith(TRACE_PARENT_PREFIX)) { 
                headers.put(TRACE_PARENT, partitionKey); 
            } 
            if (!headers.isEmpty()) { 
                return headers; 
            } 
        } 
        return super.extractHttpHeaders(event); 
    }
}

After making these changes and then running our test scenario again, the event-producer and event-consumer applications, which were in separate traces in the previous test, are now included in the same trace because the trace ID is propagated via the PartitionKey parameter in the PutRecord request.

Image 3 – Trace of the event-producer and event-consumer applications with the same trace ID (broker partitioning)

Using this approach propagates the trace ID, but it also introduces a serious side effect. Kinesis assigns events to shards by hashing the PartitionKey, so overwriting the PartitionKey with the traceparent header causes each event to be assigned to a shard based on trace context rather than its original event group ID, and any consumer logic that depends on same‑group events reaching the same shard and being processed in order is no longer preserved.

Image 4 – event-consumer function invocation logs for processing of the 1st event with group ID group-1

Image 5 – event-consumer function invocation logs for processing of the 2nd event with group id group-1

The AWS CloudWatch logs from the event-consumer Lambda illustrate the problem. Events with the same group ID are partitioned into different shards because we replaced the PartitionKey with the traceparent value. Kinesis then assigns events to shards based on the traceparent rather than the original event group ID. As a result, events from the same group can be assigned to different shards, with each shard processed concurrently by a separate AWS Lambda microVM instance. This breaks the business logic in the event-consumer function, which expects same-group events to be processed in order. Concurrent processing of same-group events can therefore introduce race conditions.

This attempt clearly demonstrates the tradeoff. Trace continuity between the producer and consumer was preserved, but it came at the cost of the stream’s partitioning semantics, ordering guarantees, and the risk of race conditions. The takeaway for practitioners is simple: don’t use fields that already carry application semantics to propagate trace context. Because the PartitionKey determines shard placement, repurposing it for tracing changes application behavior and can break ordering guarantees.

Applying distributed tracing across asynchronous workloads

This first implementation demonstrates an important tradeoff: preserving trace continuity is valuable, but not if it changes application behavior. Reusing the PartitionKey keeps traces connected, but it also disrupts shard assignment and the ordering guarantees that many Kinesis workloads rely on.

That’s where observability platforms need to go beyond collecting traces. LM Envision builds on OpenTelemetry by correlating distributed traces with infrastructure, cloud services, and application health, helping teams investigate asynchronous workloads without sacrificing the correctness of the systems they’re monitoring. Preserving trace continuity is only part of the solution. The implementation must also respect the semantics of the underlying platform.

In the next and final post in this series, we’ll explore a safer propagation approach that preserves both event ordering and trace continuity, delivering complete end-to-end visibility without compromising application behavior.

Trace your asynchronous AWS Kinesis workloads end to end without breaking event ordering.

LogicMonitor correlates OpenTelemetry traces with infrastructure and cloud health, so you can follow requests across producers and consumers while preserving partitioning and ordering guarantees, from the application to the network path.

FAQs

Why do the producer and consumer end up in separate traces by default?

In asynchronous messaging, trace context does not automatically travel with the message when a producer hands an event to Kinesis and a consumer processes it later. The OpenTelemetry SDK instruments each application on its own, so the event-producer and event-consumer are recorded as two independent traces even though they belong to the same flow. Connecting them requires deliberately propagating the trace context between the two services.

Why does injecting the traceparent header into the PartitionKey break event ordering?

Kinesis assigns events to shards by hashing the PartitionKey. When you overwrite that field with the traceparent header, every span produces a different value, so events that share a group ID land on different shards. Each shard is processed concurrently by a separate Lambda instance, which breaks logic that expects same-group events to be handled in order and can introduce race conditions.

Which Put Record parameters are safe to use for trace propagation?

Data, StreamName, StreamARN, and SequenceNumberForOrdering all carry application meaning or routing behavior, so changing them risks corrupting payloads, misrouting events, or dropping records. The PartitionKey can technically carry the header, but it controls shard placement, which makes it unsafe in practice. The reliable approach is to propagate trace context through a field that has no existing application semantics, which the final post in this series explores.

By Denton Chikura

Technical Writer

Denton Chikura is a technical writer and longtime observability advocate focused on helping site reliability engineers and engineering teams discover the tools and capabilities that strengthen internet resilience. He works at the intersection of monitoring, performance, and infrastructure to make complex systems more understandable and usable, bridging the gap between deep technical detail and real‑world operations. His goal is to help teams build faster, detect issues earlier, and recover smarter, ultimately making the internet a better, more reliable place for everyone.

Disclaimer: The views expressed on this blog are those of the author and do not necessarily reflect the views of LogicMonitor or its affiliates.

By Denton Chikura

Technical Writer

Denton Chikura is a technical writer and longtime observability advocate focused on helping site reliability engineers and engineering teams discover the tools and capabilities that strengthen internet resilience. He works at the intersection of monitoring, performance, and infrastructure to make complex systems more understandable and usable, bridging the gap between deep technical detail and real‑world operations. His goal is to help teams build faster, detect issues earlier, and recover smarter, ultimately making the internet a better, more reliable place for everyone.

Disclaimer: The views expressed on this blog are those of the author and do not necessarily reflect the views of LogicMonitor or its affiliates.

© LogicMonitor 2026 | All rights reserved. | All trademarks, trade names, service marks, and logos referenced herein belong to their respective companies.

Related Blogs

Observability ROI: Real Savings From Real Deployments
Blog

Observability ROI: Real Savings From Real Deployments

Observability can cut alert noise, speed up incident response, reduce downtime, and give engineers more time for planned work. LogicMonitor customers have used those gains to lower costs and make better infrastructure decisions.
September 9, 2026
Learn more
Is HTTPS the Answer to Man in the Middle Attacks?
Blog

Is HTTPS the Answer to Man in the Middle Attacks?

See how synthetic monitoring exposed a hidden HTTP redirect attack, and why HTTPS, HSTS, and delivery-path visibility keep your users safe from interception.
September 9, 2026
Learn more
Stale DNS Glue Records: How to Diagnose Parent-Authoritative Mismatches
Blog

Stale DNS Glue Records: How to Diagnose Parent-Authoritative Mismatches

Your authoritative servers return the right IP, but users still hit the old one. Here’s how to find stale glue records and fix the parent-zone referral.
September 9, 2026
Learn more