Traceroute InSession: A Traceroute Tool for Modern Networks

Traceroute has been a trusted network diagnostic tool for decades. When applications slow down or users experience connectivity issues, it’s often the first tool engineers use to determine whether the problem lies within their own infrastructure, somewhere on the Internet, or at the destination. But today’s Internet is far more complex than the one traceroute was originally designed for. Firewalls, load balancers, CDNs, and dynamic routing can all make traditional traceroute results incomplete—or even misleading.

Years of analyzing billions of traceroutes across backbone networks, cloud regions, enterprise environments, wireless providers, and endpoint devices have revealed both the strengths and limitations of traditional traceroute. This article explores those limitations, explains why they occur, and shows how Traceroute InSession addresses them.

Understanding the limitations of traditional traceroute

With standard traceroute, the results all too often look something like this:

This example shows a TCP traceroute from one of our backbone nodes in New York connected to Cogent Communications, to Amazon. As expected, Amazon uses a CDN to serve traffic from different edge locations, so the traceroute reaches five different destinations across repeated runs. This behavior is anticipated and normal.

Why firewalls interfere with traceroute

So what’s not expected? The red dots and all the crossed lines. The red dots show packet loss, but in reality the Internet isn’t that broken. Rather than packet loss, this is likely the result of firewalls blocking some of the traceroute packets. We’ll call this the firewalls problem.

Why traceroute can’t accurately reveal network paths

All the crossed lines suggest a very complex router topology, except that’s not true either. We’ll call this the paths problem. In fact, standard traceroute cannot reveal the path that a packet takes. That’s deeply counter-intuitive: traceroute is meant to trace the route, yet what it actually tells you is that it encountered one router X hops away and another router Y hops away. It has no way to prove whether the routers are connected or what’s between them.

To explain the paths problem further, let’s look at an individual traceroute run:

Traceroute to www.amazon.com (18.164.107.218), 30 hops max
 1  gateway (38.101.106.65) [AS174]  0 ms  0 ms  0 ms
 2  * * *
 3  te0-0-0-1.ccr31.jfk04.atlas.cogentco.com (154.54.5.177) [AS174]  1 ms  1 ms  1 ms
 4  be3363.ccr42.jfk02.atlas.cogentco.com (154.54.3.125) [AS174]  2 ms  1 ms  1 ms
 5  be3201.rcr52.ewr01.atlas.cogentco.com (154.54.90.62) [AS174]  2 ms  2 ms  2 ms
 6  38.142.215.210 (38.142.215.210) [AS174]  2 ms  2 ms  2 ms
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  15.230.208.23 (15.230.208.23)  2 ms  2 ms  2 ms
12  15.230.208.23 (15.230.208.23)  2 ms  2 ms  2 ms
13  server-18-164-107-218.jfk50.r.cloudfront.net (18.164.107.218) [AS16509]  2 ms  2 ms  2 ms

This output indicates that the traceroute went to the gateway, then to the jfk04 Cogent router, then the jfk02 router, and so on, right?

WRONG!

What it actually shows is that one packet from the traceroute took a path through the gateway and found that it was one hop away from the source. Another packet took a path through jfk04 and found that it was three hops away from the source. A third took a path through jfk02 and found that it was four hops away from the source, and so forth. Each packet in a traditional traceroute operates independently. The traceroute output reveals only the distance of each router from the source, not whether they are interconnected.

Another limitation is that a single traceroute cannot determine whether all possible routers have been identified. Catchpoint, now part of LogicMonitor, and others addressed this challenge long ago by running repeated traceroutes. Since each traceroute may discover a different subset of routers, combining the results from many runs provides a more complete view of the network path. That’s how the diagram at the top was generated. Each traceroute identified one router at each hop, and repeated runs made it possible to identify the full set of observed routers with a high degree of confidence.

How Traceroute InSession overcomes firewall and path limitations

Traceroute InSession was designed to address the two remaining challenges: firewalls causing so-called “packet loss” and the incorrect assumption that routers are adjacent when they might not be.

The result is a diagram that looks like this for the example above:

Far fewer red dots (none in this example) and much cleaner lines that provide a much higher degree of confidence that adjacent routers are actually connected.

How does it work? The following sections explain the approach and include a link to the open-source implementation. For readers interested in the technical background, the next section revisits the fundamentals of traceroute.

Those who prefer to skip ahead can jump directly to the solution.

A brief history of traceroute

Traceroute was developed as a diagnostic tool to identify the journey of a packet through an IP network and to assess the pathway’s performance. Before IP networks, traceroute didn’t need to exist. Maybe it could have, but the Internet’s simplicity at the time didn’t demand such a tool. To illustrate, this is what the Internet looked like in 1984:

Compare that to what the Internet looks like nowadays:

There’s a lot of complexity, and complexity calls for tools to decipher it. This principle is a recurring theme in our discussion.

In the 1980s, the Internet Protocol (IP), created in 1974 by Vint Cerf and Bob Kahn, was still a novel concept. Ping was introduced in 1983 to measure the latency between a client and a destination. Following this, in 1988, Van Jacobson wrote the now-famous “traceroute email”:

The level of complexity had reached a point where a specialized tool became necessary.

How does traceroute work?

Van Jacobson created the original traceroute in 1987 using the IP Time To Live (TTL) field. This field specifies the maximum number of routers a packet can pass through. Each time it’s routed, the field is decremented. When it reaches 0, an error message (ICMP TTL Exceeded for IPv4 or Hop Limit Exceeded for IPv6) is sent back.

Why modern networks challenge traditional traceroute

The original algorithm implemented by Van Jacobson works for a lot of scenarios today, but the complexities of modern networks mean the algorithm has to be enhanced to work properly. Specifically, traceroute struggles with firewalls and load balancers.

How firewalls affect traceroute accuracy

Security is a perpetual cat-and-mouse game. Each technological advance is met with exploitation by malicious actors, which leads to further innovations designed to close those loopholes. This is exactly what happened with traceroute.

ICMP traceroute often fails in modern networks because ICMP messages are generally used for diagnostic purposes and rarely carry application data. Many firewalls block them entirely.

Similarly, with UDP traceroute, firewalls often block it because it doesn’t look like application traffic. There isn’t a common UDP application that traceroute imitates, so firewalls block the packets.

For a while, TCP traceroute was the preferred approach. Because every TCP connection begins with a three-way handshake, firewalls couldn’t inspect application data at that stage, so SYN packets were generally allowed through. That changed as attackers began exploiting SYN-based denial-of-service attacks. As firewalls evolved to detect and mitigate this traffic, TCP traceroute became less reliable. (The extent of the impact depends on how SYN attack detection is implemented and where rate thresholds are configured.)

How load balancers complicate traceroute

The Internet is fragile. One of the technologies that helps improve its resilience is load balancers, specifically link load balancers.

In this context, a load balancer can be considered a specialized router that decides where to send a packet, not by the packet’s destination network (that’s what every router does) but by various other criteria. For example, cost-related policies might direct important application traffic over more expensive links for reliability, or resilience considerations might reroute traffic via a backup link if the primary route is too slow. These decisions lead to scenarios such as the following:

Looking at this diagram without any outside knowledge, it’s impossible to know which path a particular packet might take to reach the server. Maybe 1-2-5? Or 1-3-4? Or 1-2-4?

Load balancers are generally configured to balance flow traffic. If a TCP connection is established using a particular path, the load balancer will try to keep the same path for every packet in that connection. The same principles apply to UDP flows.

With traceroute, though, a “flow” is a single packet and response. Each probe sent out has different characteristics and may get load balanced to a different path.

This variability can lead to some very strange traceroute responses. For example, in the diagram above, Hop 1 will always be router 1. Hop 2 could be Router 2 or Router 3. Hop 3 could be Router 4 or Router 5. Hop 4 is the Server. So if you ran a traceroute and saw output like this:

1 (Router 1)

2 (Router 2)

(Router 3)

3 (Router 4)

(Router 5)

4 (Server)

How do you know what the actual path is? You could assume, for example, that a possible flow is 1-2-4, but that isn’t a path that exists.

Navigating complexity: Introducing Traceroute InSession

The complexities introduced by firewalls and load balancers have made the original traceroute less useful than it once was. To address these challenges, Catchpoint developed and open-sourced Traceroute InSession.

You can read our original blog post for a detailed explanation of how Traceroute InSession works. Below is a brief overview.

InSession imitates TCP connections by establishing a TCP connection to the destination server, so a firewall lets it through. Also, by using a single TCP connection, load balancers use a consistent path for every probe because each is treated as the same flow.

Here’s the result:

The first image shows many missing hops, with multiple routers returned for hops 5 to 9 and just one of three probes successfully reaching the destination (hop 13) because a firewall blocked the others.

The InSession version in the second image shows a marked improvement: most hops return data, the path is consistent with only one response per path, and all three probes succeed in reaching their destination.

Comparing Traceroute InSession with SideCar

Sidecar is the algorithm that inspired InSession. It essentially sets up an actual TCP connection or sends data over an existing one. By doing this, it also bypasses firewall detection and causes load balancers to use a consistent path.

The problem with Sidecar is that it sends actual data over a connection to a real server application, which means the destination server has to process that data. The data has to look real enough for the server to reply. With InSession, we use two useful TCP options to prevent this additional load from being added to the server: congestion control and SACK.

Congestion control means that if there’s a gap in the sequence numbers received by the server, the packets won’t be delivered to the application until the gap is filled, a gap that remains unfilled because we put it there deliberately.

SACK, or selective acknowledgment, lets the server respond to the client and tell us which sequence numbers made it and which didn’t. Without it, ACK packets wouldn’t be generated until the gap in sequence numbers is filled.

Comparing Traceroute InSession with Paris traceroute

Catchpoint offered Paris traceroute as an option for about ten years and found that customers generally didn’t use it, and when they did, it caused traceroute failures from locations that had previously operated successfully, more often than not. So the company decided to replace the Paris traceroute option with InSession.

Paris traceroute shares a common objective with InSession: to let traceroutes work amid routers that employ load balancing on packet header fields. This means that in our earlier New York to Amazon TCP traceroute scenario, the adjacent routers would be correctly identified.

However, Paris traceroute runs into two significant issues. First, it relies on a stream of SYN packets similar to traditional TCP traceroute, so firewalls often block it. In our New York to Amazon TCP traceroute scenario, the red dots would still be there. Second, the algorithm is fairly complex because it manipulates different fields inside the packet header:

Imagine explaining to your security team how this application works and why it isn’t malicious. In contrast, InSession simply transmits standard, unmanipulated TCP packets, albeit intentionally skipping one to create a sequence number gap.

While Paris traceroute is available for ICMP, UDP, and TCP protocols, InSession operates solely on TCP. Based on our observations, if firewalls block standard traceroute, they’ll block Paris ICMP and UDP as well. Since InSession runs over a mimicked TCP session, it has proven more reliable in environments where firewalls may drop packets due to SYN flooding, a vulnerability in regular TCP traceroute implementations, including Paris traceroute.

Other traceroute variants and where they fit

There are several other variants of traceroute. Dublin Traceroute is an add-on to Paris for NAT detection. Pamplona Traceroute attempts to identify IP responses that belong to the same router, also known as aliases.

Ultimately, you should use the tool that solves your problem and lets you navigate the complexity of your environment. We’re convinced that for most people, Traceroute InSession is the best choice.

Comparing traceroute variants at a glance

Important considerations when using Traceroute InSession

While Traceroute InSession addresses many of the limitations of traditional traceroute, it’s important to keep two considerations in mind:

• It handles per-flow load balancers, not per-packet ones. Per-packet load balancers are rare in our experience, so InSession still solves most users’ issues. As far as we know, no algorithm has solved the per-packet load balancer problem.
• It reports paths, it doesn’t discover them. Like all traceroute implementations, InSession only reports the paths it finds, so a single output can’t tell you whether every possible path has been found.

Access the code on GitHub, along with some pre-built binaries to explore further.

Extending traceroute with Internet Performance Monitoring

The way to get a complete picture of Internet routing is to run many traceroutes from multiple vantage points and across repeated runs, then view all of the paths in a single network graph. LogicMonitor Synthetics and Internet Performance Monitoring enable you to do just that. That path-level view becomes more useful inside the broader LogicMonitor platform, which unifies LM Envision, LM Internet Performance Monitoring, and Edwin AI, so teams can connect what’s happening on the Internet path to the health of their own infrastructure.

FAQs