Three Servers, One Outage: Why the Cloud Fails Where We Least Expect It

Imagine three houses on different streets. Each has sturdy walls, a new generator, and plenty of food. But there is only one bridge into town. If it closes, all three houses are cut off at once.

Cloud systems can work much the same way. We add another server, spread the application across different locations, and admire the green status indicators. Then one small shared component breaks, and users can no longer sign in or place an order.

The surprising thing about these outages is that the hardware may still be perfectly healthy. A running application has simply lost the ability to do the job it was built for.

A healthy server does not always mean a working store

Picture a small online store. Its application runs on three servers. If one shuts down, the other two keep responding. So far, it looks reliable.

But every order has to be saved to the same database. If the path to that database becomes unavailable, all three servers can still display a product page—and all three are equally unable to complete a purchase. Ten more copies of the application will not solve that problem.

In this fictional example, the database and the path to it have become a shared point of failure. Engineers use a broader term: failure domain. It is a group of resources that a single failure can affect together. Several programs on one computer depend on its power supply. Several locations may depend on the same sign-in service. And every copy of an application may be affected by one faulty update.

A useful question is: “What would only have to fail once to stop every copy from working?”

How an address book triggered a major outage

This is more than a thought experiment. In October 2025, an outage in the AWS Northern Virginia Region began in the DNS automation for DynamoDB. DNS helps clients find a service by name, much as an address book helps you find the right house.

According to AWS’s official incident report, a rare interaction between two update processes caused the IP addresses for DynamoDB’s regional endpoint to be removed. Clients and internal services could not establish new connections. Dependencies on DynamoDB then disrupted the launch of new EC2 virtual machines. EC2 instances that were already running remained healthy.

My takeaway is simple: an emergency plan that says “if we run short of capacity, we will quickly launch more servers” also depends on the system that creates those servers being available. The emergency exit can be behind a locked door, too.

Separate zones help, but shared dependencies remain

Availability Zones matter. In AWS, they are separate groups of data centers with isolated and redundant power infrastructure, networking, and connectivity within a Region. This arrangement helps isolate infrastructure failures. You can read more in AWS’s Availability Zone documentation.

But where your application runs and where its dependencies live are two different things. You can spread three servers across three zones while leaving them with one shared path to an external payment service. Or you can give all of them the wrong configuration at the same time. On the map, the copies are far apart. Their fate is still connected.

Start with a specific user action. For example: “Someone paid for an order and received confirmation.” Then trace the whole chain: opening the website, signing in, using the cart, paying, saving the order, and receiving a response. At each step, ask which services must respond and which tasks can temporarily wait.

Losing a zone does not make the work disappear

Suppose we have three identical zones. On a normal day, each uses 60% of its usable capacity. One zone disappears, and its requests are shared evenly between the two remaining zones.

Each surviving zone is now at 90% utilization: 60 × 3 ÷ 2 = 90. If normal utilization had been 70%, the result would be 105%. The system needs more resources than it has left.

This is a simplified model: the zones are identical, requests cost the same amount to process, demand stays constant, and traffic is redistributed immediately. In practice, some clients retry requests, caches need to warm up, and traffic spreads unevenly. Neat arithmetic is not enough. You need to measure how the service behaves when it loses capacity.

There is another catch: spare room on a CPU chart does not guarantee spare database connections, memory, or network bandwidth. You need headroom in whichever resource will become the bottleneck first.

Why “try again” can make things worse

When a service responds slowly, trying again feels natural. People do it. So do applications, payment libraries, and background jobs. If every layer retries independently, a small failure can turn into an avalanche of extra work.

AWS’s article on retries explains why systems need a limited number of attempts, pauses between them, and a random spread in retry timing. Without those safeguards, clients can all hit an overloaded service again at exactly the same moment.

Payments require particular care. A missing response does not prove that the money was never charged. A safe retry needs a mechanism that recognizes the same operation and prevents it from being carried out twice. The technical term is idempotency. For a customer, it is simply the promise that one purchase results in one charge.

A good fallback mode knows what matters most

A store can usually live without personalized recommendations for a few minutes. Reliable payment confirmation is much harder to do without. That difference needs to be built into the design ahead of time.

If recommendations are unavailable, for example, the store could show a standard selection instead. The confirmation email for an order that has already been saved could go into a queue with delivery tracking. But when a payment’s outcome is unknown, the store should not confidently say “your purchase failed” or keep asking the customer to pay again.

The idea of temporarily serving simpler responses during overload is discussed in Google’s SRE book. In practice, the technical fallback is only part of the job. The message also needs to explain what happened, whether the user’s data was saved, and whether they need to do anything.

“Your order has been saved. Confirmation will arrive later” is a good message only if the order really has been saved.

Prepare the emergency exit before the fire

AWS uses the term static stability for a system’s ability to keep working during a dependency failure without requiring urgent changes. One example is having capacity ready in advance rather than relying on launching new machines during an outage.

Think of it this way: when the lights go out, it is easier to take a charged flashlight out of a drawer than to assemble a generator in the dark. For a service, that “flashlight” might be ready-to-use reserve capacity, a tested failover path, or the ability to switch off a nonessential feature.

This costs money and effort, so there is no universal recipe. A small blog and a payment service need different margins of safety. The decision starts with two questions: how long can users wait, and which data must never be lost?

A conversation worth having with your team

You do not have to redesign your entire cloud at once. Start with one important operation and five questions:

1. What counts as success for the user? For example, “the order was durably saved,” rather than just “the server responded.”
2. Which shared dependency could stop every copy? Look at the database, DNS, sign-in, network paths, configuration, and external services.
3. Can the surviving resources handle the load? Check with measurements that include retries and recovery.
4. Can the backup take over without relying on the failed dependency? Test the recovery path itself.
5. What will the user see during the outage? The message must accurately describe the state of their operation.

Explore the answers through a small, controlled experiment. Start in a test environment, with a predefined limit on the impact and a way to restore normal operation. The goal is to find the fragile connection before your customers do.

Reliability starts with the connections you cannot see

It is easy to imagine the cloud as an endless warehouse of spare computers. For the user, something else matters more: can they finish what they came to do when one part of the system stops responding?

Sometimes the next step is another server. Sometimes it is a separate network path. And sometimes it is simply removing recommendations from the steps required to complete a purchase.

The most useful diagram of your system is the one that shows what can fail together. That is where real resilience begins.