Internet Exchange Points (IXPs) are a critical part of a complex network that makes up the Internet. In this article we will explain the role played by IXPs and how you can monitor the public Internet’s performance.

The Internet Explained

On a very high level, the Internet looks like this:

Don’t know what to think of that image? Don’t worry, we’ll break it down piece by piece in the following sections.

Autonomous Systems (AS)

An Autonomous System (AS) represents a set of IP prefixes that belong to a network and are managed by a single organization. Each AS is assigned an Autonomous System Number (ASN), which is unique to the network. 

The global Internet consists of tens of thousands of interconnected autonomous systems and a 32-bit (previously 16-bit) ASN is required for any interconnection between two ASs.

The Internet Assigned Numbers Authority (IANA) is responsible for the ASN’s allocation. IANA assigns the ASNs to the five regional Internet registries (RIR). The RIRs, in turn, allocate the ASNs to the organizations in their respective regions.

Internet Service Provider Tiers

As mentioned above, the Internet is a network of networks (AS). Most of these AS networks are enterprises, government entities, and universities. These AS networks are connected by Internet Service Providers (ISPs), which are also AS networks (one or more). Each ISP has its own routing policies and topologies that are used inside their AS. 

The ISPs transport Internet traffic for other ISPs and for their customers. 

Some ISPs are large, with extended reach across the world. Others are smaller — limited to a country or continent. This has lead to a 3-tier ISP model:

• Tier 1: These ISPs are the backbone of the Internet. They have a global reach by extending their network throughout the world. Tier 1 ISPs have private peering agreements with other Tier 1 ISPs and provide Internet transit to their customers. It is not uncommon that a Tier 1 ISP have multiple ASNs assigned, each being used for specific purpose and service.
• Tier 2: These ISPs use a combination of paid Internet transit from Tier 1 ISPs and peering with other Tier 2 ISPs. Usually they are national providers.
• Tier 3: These ISPs are only purchasing Internet transit from Tier 2 or Tier 3 ISPs to provide Internet connectivity to their end customers. They mostly provide local access to the Internet to their customers. They can be national or sub-regional (e.g., cities, metro) providers.

Internet Exchange Points

Internet Service Providers (ISPs), Content Delivery Networks (CDNs), web-scale, and SaaS companies require exchanging internet traffic to work. This exchange is usually made possible by connecting to internet eXchange Points (IXPs).

An IXP is nothing more than a location spread across one or more sites for redundancy where the members of the IXP are connected in a large Layer 2 network.

Various methods are available to create a single Layer 2 domain (across multiple switches, spread across multiple sites). For example, the process can be as simple as a regular VLAN configured across all switches or it can be something as complex as using Ethernet VPN (EVPN) as a control plane and Virtual Extensible LAN (VXLAN) as a data plane.

For instance, this is a representation of how various entities could be connected to an IXP:

The advantage of having a presence in an IXP is that the members are shortening their paths to other members of the IXP. This reduces latency, improves round-trip time, and lowers costs compared to traffic routed through a transit ISP, as demonstrated in the graphic below

As of January 2021, there are around 840 IXPs, according to PeeringDB. PeeringDB is a database containing networks. It has become the de facto location of Internet peering information.

Routing Protocols, and BGP In Particular

We noted previously that each AS owns a set of IP prefixes. These prefixes are exchanged between AS using routing protocols. A routing protocol allows the routers to choose the best path (IP prefix) to a destination and to adapt the routing table content in case of topology change events.

The protocol that runs the Internet is Border Gateway Protocol (BGP). The current BGP version is version 4. Thanks to multiprotocol extensions, version 4 can carry information about IP multicast routes and various Layer 3 address families like IPv4, IPv6, VPNv4, and VPNv6.

Each AS uses BGP protocol to exchange IP prefixes with other AS. There are two types of BGP: internal and external. The internal BGP (IBGP) is when the BGP session is configured between two peers within the same AS. The external BGP (EBGP) is when the BGP session is configured between two peers from different AS.

BGP, like any other routing protocol, prefers more specific routes rather than less specific routes. When dealing with prefixes, the longer the subnet mask, the more specific the prefix is. That means a 10.10.10.0/24 prefix is preferred over a 10.10.0.0/16 prefix when the router tries to communicate with 10.10.10.10 host.

When trying to decide which route is the best (in the event two identical routes are received from two BGP neighbours), BGP uses a best-path-selection algorithm that compares attributes and values of those prefixes.

The fact that BGP chooses the more specific prefix has led to massive Internet outages, some by mistake and some intentionally. For example, an enterprise could advertise (by mistake or intentionally) the Office365 prefixes as more specific prefixes than what Microsoft announces. Since every AS prefers the most specific routes possible, in a case like that, all of the traffic intended for Office365 applications would instead be directed to the enterprise.

To avoid such situations, each AS should allow from its BGP peers only the routes that are owned by those peer AS. This method is accomplished widely by using Internet Routing Registries (IRR) which are databases containing associations between AS and prefixes.

A newer mechanism to protect the route origin is Resource Public Key Infrastructure (RPKI), which is a way to cryptographically sign the associations between prefixes and originating AS of those prefixes. RPKI implementation effort is led by CDN providers, web scale providers and some of the major ISPs. 

DNS 

Domain Name System (DNS) translates domain names to IP addresses. All of the Internet runs on IPv4 or IPv6 protocol. 

Whenever a website is accessed, the host from where the website is accessed needs to know the IP address of the website. This process is transparent for the user and requires a DNS server to be configured on the host. The DNS server is able to make queries to find out what is the IP address associated with the website.

This type of translation is also called forward DNS.

There is another type of DNS translation called reverse DNS. Reverse DNS occurs when there is an attempt to find the name for an IP address. 

CDN 

A Content Delivery Network (CDN) is a network of distributed servers that minimizes content-loading delays by providing content from multiple locations. This process shortens the distance between each user and content server.

For example, if multiple users from Europe access a website from Asia, there is no point having each user retrieve the information from the origin server in Asia. It is more optimal to host a cached version of the content on a server in Europe. This will improve the latency and user experience. Should the content change on the origin server, the update will be reflected on the cached content.

While the web content is the main type of traffic delivered through CDN, there are other types of traffic that can be delivered as well, like audio streams, video streams, applications, and OS updates.

IPv6

An IPv4 address is a 32-bit value. This means there can be around 4.2 billion addresses. Some of those IPv4 addresses are reserved for private usage (RFC 1918), fr multicast usage, or research usage.

4.2 billion might sound like an impressive number. However, at this very moment, some RIRs cannot allocate any IPv4 because they’ve depleted their assigned public IPv4 blocks. This problem was inevitable as the internet grew, although various actions were taken to slow down the IPv4 depletion (such Network Address Translation (NAT)).

Ultimately, the decision was made to create a new protocol that accommodates all existing and future needs. Thus, IPv6 was developed. An IPv6 address has a 128-bit value—a vast improvement compared to IPv4.

Theoretically, each host, network device, IoT device, and sensor can be assigned multiple public IPv6 addresses and still be very far from depleting all public IPv6 addresses.

Unfortunately, IPv6 adoption has been slow. Only recently has adoption started to ramp up, with 30% of Alexa Top 1000 websites now reachable over IPv6. Google also predicts around 30% of their users have IPv6 connectivity.

SD-WAN

Software Defined Wide Area Network (SD-WAN) is a virtual architecture that allows the enterprises to use more efficiently their private links (MPLS), the broadband Internet, and LTE services.

Using these connections (called transport links or underlay), an SD-WAN solution creates an overlay network that is decoupled from the underlay network.

An SD-WAN solution has a centralized control plane. This control plane is responsible for the SD-WAN routes advertisements between branches that are part of the SD-WAN network.

Typically, any SD-WAN solution has a controller that handles SD-WAN routing. The controller is a management system that allows the configuration, monitoring, and troubleshooting of an SD-WAN network and any edge devices responsible for creating the overlay network. In some vendors, these two components (the SD-WAN and the controller) can be on the same device.  It also sends traffic over the overlay network according to whatever policies are in place.

Some of the advantages of SD-WAN include:

• Better application experience.
  • Multiple active links, which leads to increased high availability.
  • Application-aware dynamic routing.
• Increased security.
  • In-depth protection at branch level.
  • Secure connections over the public Internet and towards the public cloud.
• Optimized cloud experience.
  • Secured direct connection to the major public cloud providers.
  • Real-time monitoring for major public cloud providers.
• Simplified management.
  • A single pane of glass for configuration, monitoring, and troubleshooting.
  • Zero touch provisioning for any types of locations.

Monitoring a Public Network For Performance

There are multiple tools you can use to monitor the health or availability of public network resources.

It is important to understand that with any monitoring tool (be it ping, traceroute, or some other solution), the target host must allow the type of traffic used for monitoring or the intermediary network devices to permit this traffic to come through.

Sometimes, a firewall in the middle might drop all the monitoring traffic and the target host will appear as not available. Or other times, the target host will drop the traffic being monitored. Either of these outcomes can lead to false positive alarms. That is why it is important to use your monitoring system to develop a traffic baseline before setting up any types of alerts. 

Ping

Ping is the most basic tool that can be used to verify the availability of a device that is configured with an IPv4 or IPv6 address. The tool has been available for almost 40 years.

The ping utility works by sending Internet Control Message Protocol (ICMP) echo requests packets towards the target and waiting for ICMP echo replies. The tool records packet loss and any errors. It also provides a summary of the minimum, maximum, and average round-trip time.

Based on the implementation, additional options can be set when sending the ICMP packets. For instance, you can record the size of the packet, how many packets are sent, to not fragment the packets, to limit the number of network hops, how many packets to send, and  other more sophisticated options.

Traceroute

Traceroute is a monitoring tool that is used to reveal, in real-time, the path a packet takes to reach a certain target. The output shows the number of intermediary hops, as well as the time it took to reach all intermediary hops and the target. Most commonly, the traceroute tool uses ICMP packets. However, implementations using UDP or TCP packets are available as well. 

It is critical to know what kind of packets the tool is using. This knowledge enables you to understand what needs to be allowed through any potential middle firewall or at the target level. The tool sends packets with increasing TTL values for each set of probes sent (by default three packets).

Multiple traceroute commands executed from numerous sources towards the same destination can be used to track the availability of the host or application. This helps the owner of the application know if there is a local problem with specific users or a more general problem with the application itself.

This is part of what is called synthetic monitoring; a generic name applied to applications monitoring solutions.  A synthetic monitoring solution is able to connect from hundreds  of points to a specific application or website (mimicking real users) to monitor the availability, performance, and health of the application or website.

The purpose of synthetic monitoring can range from competition performance analysis, to third party performance evaluation, to discovering user experience improvements.

BGP Monitoring

The information about BGP AS and the prefixes that belong to each AS is public. For any given prefix, various AS will have a different view of that prefix and how they reach it. Collecting such information from various points can allow BGP monitoring tools to detect when changes in routing have appeared and if those changes caused any service impact.

In addition, the BGP monitoring can detect route hijacking, isolate bottlenecks, and route changes between customer locations and business applications.

DNS Monitoring  

DNS monitoring allows network connectivity testing between the authoritative name servers and local recursive servers. Also, DNS monitoring can ensure that the DNS records are not changed mistakenly due to human errors or malicious attacks.

With regards to the malicious attacks, two methods are more common:

• DNS poisoning: False information inserted in the DNS cache on a server
• DDOS attacks: Repeated requests targeting the DNS servers so that the servers become overwhelmed and fail to respond to legitimate requests

CDN Monitoring 

CDN monitoring allows to detect any performance issues between users, origin, and edge. A CDN monitoring solution measures the page load times, fetch times, latency, and loss and it can map users to specific CDN edge locations based on the user location.

Conclusion

Internet Exchange Points interconnect the backbone of the Internet. It is a challenging task to monitor such a vast delivery chain, which traverses across multiple ISPs and ASNs. The end results, though, are  better performing applications with happy end users. This translates into business outcomes such as more business closed, increased customer loyalty, improved employee productivity, and streamlined business processes.The solution to isolating performance bottlenecks is to insert probes at every step along the path of your public area network. The probes must be deployed in a global network of nodes to cast as wide and diverse of a reach as possible. Read our guide to Synthetic Monitoring to learn more.