Loading full navigation.

Autonomous IT is here. The alert loop ends now. LEARN MORE

Partners

Docs

LM Academy

LM Community

Pricing

Platform
Solution
Industry
Role
There is no result.

LOGICMONITOR BLOG

Outage Retrospective: What the AWS S3 Outage Still Teaches Us About Internet Resilience

See how one S3 region took down a slice of the Internet, and how outside-in monitoring pinpoints failures you don’t control before your users feel them.

7–10 minutes
Denton Chikura

NEWSLETTER

Subscribe to our newsletter

Get the latest blogs, whitepapers, eGuides, and more straight into your inbox.

SHARE

The quick download:

One storage service in one region sat behind a surprising slice of the day’s Internet, and most teams couldn’t tell whether the problem was theirs.

  • A single command error in AWS US-EAST-1 cascaded into EC2 launches, EBS, Lambda, and the S3 console, taking down Quora, Imgur, Trello, and more.

  • Infrastructure-only monitoring left teams guessing; Catchpoint’s synthetics located the failures in real time from outside the environment.

  • Even AWS’s status dashboard failed because it depended on S3, so incident-communication systems need their own failure domains.

  • Map your critical dependencies and monitor the third-party services you rely on directly, so you learn about failures from your own signals, not from your users.

On February 28, 2017, Amazon S3 in the US-EAST-1 region (Northern Virginia) began experiencing widespread issues at 12:37 PM EST. Amazon’s status dashboard reported only “high error rates with S3 in US-EAST-1.” That single line was the entire explanation available at the time, and it left thousands of teams guessing about what had just broken.

The reach went well beyond one region’s storage service. Popular online services that depend on S3, including Quora, Imgur, and Trello, suffered outages throughout the day. One storage service, in one region, sat behind a surprising slice of the day’s Internet.

A 2017 incident still matters in 2026 because cloud dependency chains have grown longer and more tangled since then.

A modern user experience can touch DNS providers, content delivery networks, identity providers, SaaS APIs, payment services, and now AI APIs. Each of those sits outside your direct control yet inside the experience your users judge you by.

When one link fails, the question is the same as it was in 2017: is the problem inside our environment, out on the Internet path, or at a dependency we do not operate?

A Single Region Took Down a Slice of the Internet

The disruption was scoped to S3 in US-EAST-1, not to S3 across the globe. According to AWS’s post-incident summary, an S3 team member ran a command during debugging of the S3 billing subsystem and entered incorrect input, which removed more servers than intended.

That mistake took down the S3 index subsystem, which stores metadata and location data for every object in the region, and the placement subsystem, which allocates storage for new objects.

The effects cascaded: new EC2 instance launches, EBS volumes that needed S3 snapshot data, AWS Lambda, and the S3 console were all affected.

This is a shared-responsibility story, and both parts are true at once. AWS’s own operational error caused the incident.

At the same time, the organizations that went dark still owned their resilience architecture, blast-radius control, and recovery strategy. The provider ran the region; the dependent teams still decided how much of their business rode on it.

S3 was effectively unavailable to users starting around 12:37 PM EST and began improving in the afternoon. That single regional service sat behind a surprising portion of the day’s digital experience.

Watching the Failures in Real Time

For most teams, the hardest part of a day like this is the blind spot. The failure lived outside their own infrastructure, so infrastructure-only monitoring could not tell them whether the problem started inside their environment, along the Internet path, or at an external dependency.

Application metrics might have shown dependency errors, but internal telemetry alone rarely locates the true source. Outside-in visibility is what closes that gap, and it is what enabled LogicMonitor Digital Experience to monitor the outage as it happened.

While Quora was unavailable, www.quora.com returned a “504 Gateway Timed Out” error. Using Catchpoint’s synthetic monitoring, we could see the failures occurring in real time.

The 504 for Quora’s homepage is visible in the waterfall chart below.

Mashable.com faced significant issues too, including images that failed to load because they were hosted in S3 buckets. An instant test of Mashable.com showed multiple images that never got served.

Tracing the Path to the Failure

The traceroute below ran from Catchpoint to one of the S3 buckets. Timeouts occurred closer to the destination.

On their own, traceroute timeouts do not prove an endpoint fault, since ICMP filtering or deprioritization can produce the same pattern.

Combined with the global TCP connection failures and HTTP errors we were already seeing, though, those timeouts were consistent with a destination-side problem. The value came from correlating several signals rather than trusting any one of them.

Two Phases of Downtime

From our outside-in vantage points, the downtime broke into two distinct phases.

  1. From about 12:35 PM EST to 3:35 PM EST: connection failures. We could not establish a TCP connection to the S3 endpoints from anywhere in the world, so this was not a geographic or network transit issue.
  2. From about 3:35 PM EST to 4:16 PM EST: high wait times and 500 errors, as service started to return.

Two timelines describe this recovery, and they answer different questions. Catchpoint’s outside-in view showed when user-visible failures cleared: we observed connection-level recovery by about 4:16 PM EST. AWS’s official summary describes when all API operations fully returned inside the region.

Partial GET, LIST, and DELETE service came back about 3:26 PM EST, the index subsystem fully recovered about 4:18 PM EST, and the placement subsystem recovered with S3 operating normally about 4:54 PM EST.

The two views are complementary: one measures the user’s experience, the other measures the provider’s internal state.

What This Outage Taught Us About Resilience

At the end of a hectic day, the practical takeaway held up. 100% uptime is not a realistic target, so teams need a plan for the moments when a dependency fails.

Monitoring your own services alongside the third-party services you rely on lets you catch problems early and resolve them quickly, which protects your users’ confidence. Clear communication matters as much.

AWS communicated across multiple channels during the incident, and that gave its users some relief during a chaotic day.

The communication itself carried a lesson. AWS could not update the individual service statuses on its Service Health Dashboard during the incident, because the dashboard depended on S3. It fell back to a global banner and other channels, and it could not resume normal per-service updates until about 2:37 PM EST.

That is a textbook dependency failure: the systems you use to communicate during an outage need their own infrastructure and failure domains, separate from the systems they report on. AWS afterward moved the dashboard to run across multiple regions.

The impact reached far beyond public websites. It stretched from consumer sites to IoT devices, since many companies rely on cloud services for nearly every part of their digital experience.

Our own operations were affected in several ways, a reminder that internal SaaS dependencies carry the same risk: the video conferencing systems Zoom and Bluejeans were down, our office door management system Kisi was inaccessible, and Duo Security along with several other daily tools, were unavailable.

A Risk-Based Approach to Resilience

Multi-cloud is not automatically the right answer, and spreading workloads across providers and zones is not a resilience strategy by itself.

The better approach is risk-based and tiered: invest in redundancy where the business impact justifies the cost, and keep it simple where it does not.

A practical checklist looks like this:

  • Map your critical dependencies and everything that relies on them, so you understand blast radius before an incident, not during one.
  • Define RTO and RPO requirements for each critical service, then let those targets drive the design.
  • Choose multi-Region or cross-provider redundancy only where the risk and the recovery objectives justify the added complexity.
  • Isolate status pages and incident-communication systems into separate failure domains from the systems they report on.
  • Monitor third-party dependencies directly with synthetics and status-page monitoring, so you learn about failures from your own signals.
  • Test failover regularly, because an untested plan is an assumption.

Questions to Ask Your Cloud and SaaS Providers

The incident also sharpened the due-diligence questions worth putting to any cloud or SaaS provider you depend on:

  • How is your service architected, and how transparent are you about the dependencies behind it?
  • How do you isolate failure domains so a single fault does not cascade?
  • What is your Region strategy, and where does my data and workload actually run?
  • What recovery objectives (RTO and RPO) can you commit to?
  • How often do you test failover, and what did the last test show?
  • How transparent is your status page, and does it run independently of the services it reports on?
  • What telemetry and monitoring hooks do you expose so I can watch the dependency myself?
  • How do you communicate during an incident, and through which channels?

Seeing Inside and Outside Your Environment

When a shared dependency like S3 fails, the answers a team needs are the ones they scrambled for that day: Is the problem inside our environment or external? Which users, regions, and workflows are affected? Which third-party dependency is failing, and what depends on it? What should responders work on first?

This incident shows how layered evidence answers those questions in sequence. Internet synthetics exposed the user-visible failures first, when Quora returned 504s and Mashable’s images would not load.

Global vantage points helped establish the failure domain, since the connection failures showed up worldwide rather than in one geography. Waterfall and protocol-level evidence separated the symptoms, distinguishing a page-level error from a failed asset, an HTTP 500 from a dropped TCP connection.

Infrastructure telemetry supplied the inside view of what a team’s own systems were doing. Correlating those signals is what turns a pile of alerts into a prioritized response.

LogicMonitor brings those layers together in one platform to monitor hybrid infrastructure, understand user experience across the Internet delivery path, and correlate signals to prioritize issues by business impact.

During an incident like this, that means less time arguing about whether the problem is yours and more time acting on where it actually lives.

The most important point from this incident is that we owe our customers the best service we can provide, under any circumstance. Tools take a team only so far. The rest is deciding, ahead of time, which dependencies your users rely on and making sure you can see and recover them.

See where a dependency failure actually lives before your users do.

LogicMonitor combines infrastructure telemetry with outside-in Internet visibility, so you can pinpoint whether the problem is yours, on the Internet path, or at a dependency you don’t operate.

FAQs

What caused the 2017 AWS S3 outage?

During debugging of the S3 billing subsystem in US-EAST-1, an AWS team member entered incorrect input to a command and removed more servers than intended. That took down the S3 index and placement subsystems, and the effects cascaded to EC2 launches, EBS, Lambda, and the S3 console.

Why did so many websites go down when only one AWS region failed?

Popular services like Quora, Imgur, and Trello depend on S3 for storage, so when S3 in US-EAST-1 failed, their sites failed too. Modern user experiences touch DNS, CDNs, identity providers, SaaS APIs, and more, and each dependency sits outside your control yet inside the experience users judge you by.

How can teams detect a third-party outage they don’t control?

Infrastructure-only monitoring can’t tell you whether a problem started inside your environment or at an external dependency. Outside-in synthetic monitoring from global vantage points, like Catchpoint captured during the S3 outage, exposes user-visible failures and helps establish the failure domain. Correlating that with infrastructure telemetry turns scattered alerts into a prioritized response.

By Denton Chikura

Content Marketing Manager

Denton Chikura is a Content Marketing Manager for LogicMonitor Digital Experience, where he drives content on observability, Internet performance, and digital experience. He writes for SREs, IT operations teams, and business leaders, turning performance data into guidance they can act on.

Disclaimer: The views expressed on this blog are those of the author and do not necessarily reflect the views of LogicMonitor or its affiliates.

© LogicMonitor 2026 | All rights reserved. | All trademarks, trade names, service marks, and logos referenced herein belong to their respective companies.

Related Blogs

How LogicMonitor Pricing Works
Blog Observability

How LogicMonitor Pricing Works

Understand how LogicMonitor platform pricing works, from Hybrid Units to package selection, so you can estimate costs, compare options, and plan with confidence.
November 4, 2026
Learn more
What Are Domain-Specific ITOps Agents? How Specialized AI Works in IT Operations
Blog AIOps & Automation

What Are Domain-Specific ITOps Agents? How Specialized AI Works in IT Operations

Domain-specific ITOps agents are designed around the systems, data, and tasks involved in operating production IT. Learn how specialized agents work together and how Edwin AI applies them across incident investigation and response.
November 3, 2026
Learn more
Optimizing Website Performance: Harnessing the Power of Image Lazy Loading
Blog Digital Experience

Optimizing Website Performance: Harnessing the Power of Image Lazy Loading

Image lazy loading can speed up your site, or slow it down. Learn how to apply it selectively, protect Core Web Vitals, and measure the real-world impact.
October 30, 2026
Learn more