Loading full navigation.

Autonomous IT is here. The alert loop ends now. LEARN MORE

Partners

Docs

LM Academy

LM Community

Pricing

Platform
Solution
Industry
Role
There is no result.

LOGICMONITOR BLOG

Outage Retrospective: How to Detect a Partial CDN Failure Before Users Do

A partial CDN failure can break images while your uptime dashboards stay green. See how outside-in synthetic monitoring pinpoints the real fault fast.

6–9 minutes
Denton Chikura

NEWSLETTER

Subscribe to our newsletter

Get the latest blogs, whitepapers, eGuides, and more straight into your inbox.

SHARE

The quick download:

A CDN can fail while your uptime dashboards stay green, and only outside-in monitoring reveals where the fault truly lives.

  • During a 2019 Facebook photo outage, feeds and timelines rendered normally while the photos inside them refused to load, so basic health checks kept passing

  • Synthetic checks from global vantage points isolated the failure to specific scontent-*.fbcdn.net photo hosts returning 502s, 503s, and connection timeouts

  • Comparing results across regions, ISPs, and IP versions moved the question from Facebook’s CDN placement toward the network paths carriers used, and even surfaced a chronic IPv6 routing inefficiency

  • SRE and network teams that adopt digital experience monitoring can locate the next partial CDN failure in minutes instead of losing time to blame-shifting

A partial CDN failure can strike while your uptime dashboards stay green. A content delivery network (CDN) or third-party dependency degrades, yet the page still loads and health checks still pass. Users, meanwhile, see broken images and stalled video.

The failure feels like an application problem, even though the responsible component remains outside your direct control. Teams can lose time deciding whether the fault is theirs, their provider’s, or the network’s.

A 2019 Facebook photo outage shows the pattern clearly. On July 3, 2019, users worldwide hit the same problem: Facebook and Instagram photos wouldn’t load, and WhatsApp couldn’t send images or video. The apps stayed up, so most people saw broken thumbnails and gray boxes.

LogicMonitor Digital Experience observed the incident from the outside in, running synthetic checks from vantage points around the world. That outside-in view located where the failure lived. Interfaces rendered while media delivery failed, which pointed at the CDN serving the images.

The findings below describe what the platform observed during the 2019 incident window. Read them as a case study in how to investigate this kind of failure.

Six steps for investigating a partial CDN failure

Partial CDN failures reward a repeatable method. Work from the user-visible symptom down to the responsible component, and confirm each layer with evidence before moving on. The Facebook incident followed six steps any team can reuse.

  1. Confirm the user-visible failure.
  2. Separate page and application availability from asset delivery.
  3. Group errors by hostname and dependency.
  4. Compare across regions, networks, and IP versions.
  5. Inspect response headers and network paths.
  6. Track recovery by edge rather than relying on a global average.

The sections below walk through each step using the Facebook data.

Step 1: Confirm what users actually saw

Start with the symptom users report, then reproduce it from monitoring. During the incident, feeds and timelines rendered normally while the photos inside them refused to appear.

On WhatsApp, text messages went through while image and video attachments failed to send. The problem held across regions, which suggested it reached beyond a single edge location.

A Facebook feed rendered while the photos inside it failed to load.

Step 2: Separate application health from asset delivery

A CDN is part of the application experience even when a third party operates it. When it degrades, users feel an app problem, though the failing component is outside the app owner’s control. Basic uptime checks stay green because the page still loads.

That’s what makes partial outages operationally dangerous. A team watching only page availability sees healthy dashboards while users stare at broken images. Ownership also gets murky, so application, network, and vendor teams can lose time pointing at each other.

The cost lands on the business. Images and video drive engagement, complete transactions, and keep distributed teams productive. When media delivery fails, companies face SLA exposure and eroded trust while the core application still reports healthy. Monitoring that can isolate third-party CDN failures closes that blind spot.

Step 3: Group errors by hostname and dependency

With the failure confirmed, cluster the errors by where they land. The monitoring data pointed at a specific set of CDN hostnames matching the pattern scontent-.*-\d.xx.fbcdn.net. Errors concentrated there rather than spreading evenly across Facebook’s stack.

Those hosts returned HTTP 503 and 502 status codes, and in some cases connection timeouts. Plotting the errors showed them clustered on the photo-serving domains across global vantage points.

Failures concentrated on scontent-*.fbcdn.net photo hosts across global vantage points.

Step 4: Compare across regions, networks, and IP versions

Next, compare the failing requests across geography, carrier, and IP version. Cities that routed to the same distant server shared one common factor: the ISP. Cities on the same provider tended to share a destination.

In these samples, that shifted the question from Facebook’s CDN placement toward the network paths the carriers used. The pattern appeared in the routes tested and isn’t a universal routing rule.

In the sampled routes, cities sharing an ISP tended to reach the same distant server.

One case involved NTT over IPv6, which routed traffic all the way to Hong Kong. The network path showed a peering relationship between NTT and Tata Communications steering the traffic there.

That triage surfaced a second story worth keeping. The investigation exposed a likely pre-existing IPv6 routing inefficiency with little to do with the day’s outage. Triage often uncovers chronic problems alongside the immediate failure.

The network path showed NTT to Tata Communications peering steering IPv6 traffic toward Hong Kong.

What the investigation revealed about the CDN design

The comparison work exposed how Facebook delivered photos during the tests. Photos came from dedicated scontent-*.fbcdn.net hosts. Each host mapped to a static IP behind a Unicast design, so a hostname pointed to one specific server instead of the nearest node.

As a result, requests were sometimes served from distant edges: Atlanta to Hong Kong, Bangalore to Singapore, and Seattle to Stockholm. When a request traveled that far, response times climbed, and the latency histograms reflected it.

The sampled design used Unicast, so a hostname pointed to one specific server.

Distant routing showed up as elevated response times.

Step 5: Inspect response headers and network paths

Individual responses carry more detail than a pass or fail result. LM Digital Experience engineers saw three distinct signals, and each says something different about the host.

SignalWhat it meansWhat it shows about the host
502 Bad GatewayActing as a gateway, the host received an invalid or malformed upstream response, not a valid error it could pass through.The host was reachable and answering.
503 Service UnavailableThe host responded and reported it couldn’t serve the request.The host was reachable and answering.
Connection timeoutThe request got no response at all.Reachability stays unconfirmed, because nothing came back.

The first two signals show the host was reachable and answering. A timeout doesn’t confirm network-layer reachability, because nothing came back to establish it.

A 503 Service Unavailable response: the host answered and reported it couldn’t serve the request.

A connection timeout: the request got no response at all.

A 502 Bad Gateway response: the host answered with an upstream error.

Response headers also pointed at a possible cause. The config version reported in the headers differed between the incident window and the period after recovery. The same test paths showed different config versions on the responding servers during and after the outage.

It helps to separate what the data established from what it suggested:

  • Observed: failures concentrated on specific photo-serving hosts, responses included 502, 503, and connection timeouts, and config-version headers changed after recovery.
  • Hypothesis: a configuration change contributed to the failures.
  • Not established: that a bad configuration was definitively the root cause.

The Dallas to Cogent path shows the header difference. The response during the incident appears below, followed by the response after recovery.

Dallas to Cogent during the incident, with the config version visible in the response headers.

The same Dallas to Cogent path after recovery, showing a different config version.

The London to Cogent path shows the same difference. The config version during the incident differs from the one seen after recovery.

London to Cogent during the incident.

London to Cogent after recovery, with a different config version than during the outage.

Step 6: Track recovery edge by edge

A global average can hide an uneven recovery, so watch each edge. Measurements returned to normal around 22:06 UTC (6:06 pm EST). That marks when the vantage points saw healthy responses again, and it doesn’t necessarily represent an official global resolution time.

What appeared to be a corrected configuration propagated gradually across Facebook’s CDN servers rather than flipping back all at once.

Performance climbed back as what appeared to be a corrected configuration propagated across the CDN.

Availability returned to normal around 22:06 UTC

What Digital Experience Monitoring gives you during a CDN failure

Digital Experience Monitoring helps a team quickly establish where a CDN failure lives. The fault could sit inside their own environment, in a provider dependency, or along the user’s network path. That answer shortens the argument between application, network, and vendor teams.

LogicMonitor Digital Experience observes services the way real users experience them, across the Internet paths that carry the traffic. Paired with infrastructure monitoring, that view helps correlate what a dashboard reports inside the firewall with what a user actually sees. LogicMonitor is designed to correlate Internet experience with infrastructure health, and that integration continues to deepen.

Get complete visibility into every CDN, region, and network path your users depend on

Partial CDN failures hide behind green dashboards until customers complain. LogicMonitor Digital Experience reveals what real users experience so you can locate and resolve the fault fast.

FAQs

Why do uptime dashboards stay green during a partial CDN failure?

Basic uptime checks confirm that a page loads, and during a partial CDN failure the page still renders. The images and video served by the CDN fail separately, so health checks keep passing while users see broken thumbnails and stalled media.

How does synthetic monitoring pinpoint where a CDN failure lives?

Synthetic checks run from vantage points around the world and watch services the way real users experience them. By grouping errors by hostname, comparing results across regions, ISPs, and IP versions, and inspecting response headers and network paths, teams can locate the fault across CDN hosts, carriers, and routes.

What did the 2019 Facebook incident reveal about CDN design?

During the tests, photos came from dedicated scontent-*.fbcdn.net hosts, each mapped to a static IP behind a Unicast design, so a hostname pointed to one specific server rather than the nearest node. That meant some requests traveled to distant edges, such as Atlanta to Hong Kong, which raised response times.

By Denton Chikura

Content Marketing Manager

Denton Chikura is a Content Marketing Manager for LogicMonitor Digital Experience, where he drives content on observability, Internet performance, and digital experience. He writes for SREs, IT operations teams, and business leaders, turning performance data into guidance they can act on.

Disclaimer: The views expressed on this blog are those of the author and do not necessarily reflect the views of LogicMonitor or its affiliates.

© LogicMonitor 2026 | All rights reserved. | All trademarks, trade names, service marks, and logos referenced herein belong to their respective companies.

Related Blogs

How LogicMonitor Pricing Works
Blog Observability

How LogicMonitor Pricing Works

Understand how LogicMonitor platform pricing works, from Hybrid Units to package selection, so you can estimate costs, compare options, and plan with confidence.
November 4, 2026
Learn more
What Are Domain-Specific ITOps Agents? How Specialized AI Works in IT Operations
Blog AIOps & Automation

What Are Domain-Specific ITOps Agents? How Specialized AI Works in IT Operations

Domain-specific ITOps agents are designed around the systems, data, and tasks involved in operating production IT. Learn how specialized agents work together and how Edwin AI applies them across incident investigation and response.
November 3, 2026
Learn more
Optimizing Website Performance: Harnessing the Power of Image Lazy Loading
Blog Digital Experience

Optimizing Website Performance: Harnessing the Power of Image Lazy Loading

Image lazy loading can speed up your site, or slow it down. Learn how to apply it selectively, protect Core Web Vitals, and measure the real-world impact.
October 30, 2026
Learn more

14-day access to the full LogicMonitor platform