Distributed Tracing

Distributed tracing

Modern software teams manage dozens or hundreds of independent services, each built and deployed separately.

Serving modern workloads, including richer pages, larger data volumes, and more concurrent users, pushes teams toward distributed architectures. The result is higher system reliability and improved scalability, both of which are common priorities for teams managing services at scale.

Although microservices, containerization, cloud computing, and serverless have facilitated application development, they’ve also introduced blind spots that traditional monitoring alone can’t cover. Application components and services are now being built and maintained by different teams. As a result, there’s less insight into the broader user journey across an entire application, from the end user through the Internet path to the underlying infrastructure.

The solution to effectively observe and monitor applications in a distributed world is to design a distributed tracing strategy. Distributed tracing extends your existing monitoring by adding an outside-in perspective: once you find a slowdown in the end-user experience, you can follow the user’s journey (the trace) to pinpoint the exact technical issue causing the delay. This ability to follow tracing across thousands of independent services lets engineering teams diagnose and resolve incidents faster.

Traces, metrics, and logs each contribute distinct signals to root cause analysis (RCA), and you need all three working together with your infrastructure, cloud, and network monitoring to get a complete picture.

What is distributed tracing?

Distributed tracing is a method for observing system requests as they flow through an application, from the front end to back-end services and data stores. The concept implies that your software architecture is implemented in a distributed computing environment.

What may start as a request to fetch a user’s cart total on a checkout page might involve a long chain of requests that include validating item availability, fetching payment options, and retrieving item prices before returning to the user. Many of these operations occur asynchronously and are abstracted from the end user. However, from an engineer’s perspective, they’re difficult to observe in log data because it’s impossible to know where a bottleneck occurred along the path.

Distributed tracing works by following an origin request and tracking it along its path until it reaches the end. The tracing framework assigns a unique identifier to each request along the path and labels each individual piece (e.g., an API call or a query to a data store) as a span, with the origin API call labeled as the root span.

Whenever a request enters a new service call, a child span is created, and each of its operations is identified and labeled. The root contains the entire execution path of the originating request, and each child can also serve as a root for its own nested spans. Each child’s unique identifier will include the original trace identifier, along with other relevant metadata, including error information and user information.

Using distributed tracing, support engineers and DevOps teams can get a clear picture of where a bottleneck happened, irrespective of who owns which service. There’s no longer a need to figure out where it happened because the trace shows it clearly.

It’s possible to drill down into span data in real time, giving support team members direct visibility into the end-user experience. When combined with infrastructure and network monitoring, these insights into the broader user journey, from the application layer to the underlying infrastructure, allow you to find, diagnose, and solve issues faster.

The following diagram from OpenTracing can serve as an example.

This visual representation shows how a request can have many child spans and various service calls along the way. With distributed tracing, you get a clear picture of the parent request’s journey, giving you visibility into which segment along the way you should dig deeper to find relevant information.

One last important concept in distributed tracing is sampling, the process of filtering trace data according to a rule to decide which data to keep and which to discard.

There are two main sampling approaches:

1. Head-based sampling: Starting at the root, randomly choose which subsequent trace data to collect and persist to storage.
2. Tail-based sampling: After the span has completed, retroactively decide which data to persist to storage.

Both have advantages and disadvantages to consider, but at a high level, head-based sampling is simpler and can be implemented more quickly. It works well for companies that don’t operate in a highly distributed environment.

Tail-based sampling gives more control over which data you retain because you decide after the trace completes. It does add operational complexity, and you’ll need infrastructure to buffer and process trace data before it’s stored.

Netflix has documented how they built their distributed tracing infrastructure, which is a useful reference if you’re planning at scale.

Key terms:

• Request: Communication at the API layer enabling applications, services, and machines in general to share information with each other.
• Trace: Relevant data about the requests in a system along some execution path.
• Span: API calls that belong to a trace.
• Root Span: The top span in a distributed trace.
• Child span: Any nested span relative to the root.
• Sampling: The process of deciding which trace data to store and which to discard. The two most popular are head-based and tail-based.

Sample use case

Let’s look at a simple implementation of OpenTracing’s tracing library in Python. We’ll follow its Python tutorial, which is hosted on GitHub. We’ll use a Dockerfile to run a visualization backend in Jaeger.

Prerequisites

1. This tutorial uses Python, pip, and virtualenv. Make sure you have these required dependencies before proceeding.
2. You’ll need the Docker CLI set up locally to complete this tutorial. Docker is free and easy to set up for your local machine here, or if on a Mac, simply run:

brew install –cask docker

The following tutorial will use several dependency packages, namely Jaeger, to serve as the tracing back-end. Once you have Docker installed, run the following command to open a port at https://localhost:16686 that runs Jaeger’s all-in-one binary.

docker run \
  --rm \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 16686:16686 \
  jaegertracing/all-in-one:1.7 \
  --log-level=debug

Now that that’s running, openhttps://localhost:16686.

You can see that Jaeger provides a clean UI for finding traces (left menu) and a search bar in the top menu. Once we start collecting some trace data, we’ll revisit this dashboard.

5. Next, you’ll want to clone the OpenTracing tutorial repo on GitHub and then navigate to the Python directory. Then execute the commands below.

git clone https://github.com/yurishkuro/opentracing-tutorial.git
cd opentracing-tutorial/python
virtualenv env
source env/bin/activate
pip install -r requirements.txt

6. Now, let’s walk through creating a Tracer and implementing a simple trace. We’ll create a simple “hello world” service that takes an argument and prints “Hello, {arg}!”. Create two files, one called hello.py and one called _init_.py.

git clone https://github.com/yurishkuro/opentracing-tutorial.git
cd opentracing-tutorial/python
virtualenv env
source env/bin/activate
pip install -r requirements.txt

7. Open hello.py and first create the service.

import sys
import time

def say_hello(hello_to):
       hello_str = 'Hello, %s!' % hello_to
       print(hello_str)

assert len(sys.argv) == 2

hello_to = sys.argv[1]
say_hello(hello_to)

8. This simple service will take a command-line argument, format it, and print the statement. To add tracing to it, use the tracing.py file from the python/lib directory to instantiate a Tracer. It contains code to interact with the Jaeger client. Recall that a trace is just a directed acyclic graph (DAG) of a bunch of spans. Each span in OpenTracing must contain, at a minimum, the name of the operation you’re tracing and the duration (start and end times). For simplicity, create a trace that consists of just one span.

import sys
import time
from lib.tracing import init_tracer

def say_hello(hello_to):
   with tracer.start_span('say-hello') as span:
       span.set_tag('hello-to', hello_to)

       hello_str = 'Hello, %s!' % hello_to
       span.log_kv({'event': 'string-format', 'value': hello_str})

       print(hello_str)
       span.log_kv({'event': 'println'})

assert len(sys.argv) == 2

tracer = init_tracer('hello-world') # initialize the tracer using the Jaeger client

hello_to = sys.argv[1]
say_hello(hello_to)

# yield to IOLoop to flush the spans
time.sleep(2)
tracer.close()

Let’s walk through what’s going on:

• start_span() on the Tracer instance starts the span and takes the operation name as an argument. Each span must be finished by calling finish(), and the start and end timestamps will be captured for you. Note that we’re using Python’s context manager (the with keyword), which is the same thing as writing this:

def say_hello(hello_to):
   span = tracer.start_span('say-hello')
   hello_str = 'Hello, %s!' % hello_to
   print(hello_str)
   span.finish()

• set_tag() on the span instance will add a tag to the span.
• log_kv() on the span instance will log values in a structured format. It’s important to be consistent in logging, as log aggregation systems need to process this information.
• init_tracer() is the function defined in python/lib/tracer.py that allows us to connect to the Jaeger client. It takes an identifier in and will mark all child spans as originating from our service.
• Lastly, we must introduce a delay into the program to allow enough time for the spans to flush to the Jaeger back-end before closing the tracer.

pytho n -m lesson01.tutorial.hello I❤️tracing

You’ll notice the following output:

...
Initializing Jaeger Tracer with UDP reporter
Using selector: KqueueSelector
Using sampler ConstSampler(True)
opentracing.tracer initialized to <jaeger_client.tracer.Tracer object at 0x10f619fd0>[app_name=hello-world]
Hello, I❤️tracing!
Reporting span 3c0e89bb739bc46:5f288b31bba91815:0:1 hello-world.say-hello
Using selector: KqueueSelector

Open the Jaeger UI to see the results.

You’ll get a time-series graph and all the traces that match your query. Click on the trace to drill down into it.

This will show you how many services this trace interacted with, the total depth, and total spans. It will also show you the associated logs and metadata.

If you want to challenge yourself further, follow the remaining tutorials on OpenTracing’s GitHub.

Distributed tracing best practices

A few practices make distributed tracing more reliable at scale.

1. Remember Sampling

Sampling is important because with hundreds to thousands of microservices, a large volume of trace data will be produced. To tackle this data storage problem and mitigate high costs and added complexity, choose a sampling strategy based on your environment’s scale, traffic patterns, and cost constraints.

2. Follow Well-Established Standards

Adopt established interoperability standards for trace context propagation. The W3C Trace Context specification defines how trace identifiers are passed across services and frameworks. Develop good practices early, so your tracing data is collected and organized in a way that scales and is easy to communicate across teams.

3. Consult with Professionals Across Your Team

Have discussions with your technical leaders and software architects to determine which path is feasible and makes sense for your use case. Some companies operating on a smaller scale don’t require many microservices or advanced observability techniques. Before adding distributed tracing infrastructure, confirm your traffic volume and operational needs justify the added complexity.

Available tools

Many tools, in addition to Jaeger and OpenTracing, are at your disposal for distributed tracing. There are great options in both the paid and unpaid categories.

Several widely used open source tools cover the range of distributed tracing needs:

1. Jaeger: As we’ve seen, Jaeger is an open source tool that provides distributed transaction monitoring, performance and latency optimization, RCA, service dependency analysis, and distributed context propagation. It provides an all-in-one executable to get you started fast with quick local testing.
2. OpenCensus: A framework that originated with Google. It supports many major back-end service providers and provides metrics and tracing solutions.
3. OpenTelemetry: OpenTelemetry is the modern successor to OpenTracing and OpenCensus, combining them into a single framework. It provides tools, SDKs, and APIs for gathering telemetry data across your application environment.
4. OpenTracing: OpenTracing provides APIs in many different popular programming languages. You can use its SDK as the tracing back-end or integrate it with Jaeger via Docker, but it’s mainly known as a tracing solution. Note that any new implementations of a distributed tracing library should use OpenTelemetry.
5. OpenZipkin: OpenZipkin is easy to use and provides features for both collection and data lookup.

Explore some of these open-source options and determine which best fits your use case. Gauge their capabilities across observability signals: logs, metrics, and traces.

Conclusion

Distributed tracing is most effective when you capture the right telemetry data and build a consistent instrumentation strategy from the start. It doesn’t replace your existing monitoring; it strengthens it by connecting end-user experience data to infrastructure, network, and application signals. Focus on traces, metrics, and logs together, and pair them with a sampling approach that keeps costs manageable without losing signal on critical paths.

FAQs