Free lesson · AWS Solutions Architecture

Availability math - what 99.9% actually costs

Easy8 min readawssystem-designarchitecture

The Numbers First

Target	Per month	Per year	Roughly
99%	~7.2 hours	~3.65 days	A bad afternoon, monthly
99.9%	~43 minutes	~8.8 hours	One incident a month
99.95%	~22 minutes	~4.4 hours
99.99%	~4.3 minutes	~53 minutes	No human in the recovery path
99.999%	~26 seconds	~5.3 minutes	Almost nobody needs this

Important: Look at the 99.99% row and ask what could be accomplished in four minutes. Detection alone frequently exceeds it. A person acknowledging a page and opening a laptop exceeds it.

Four nines means most incidents must resolve without human involvement - automatic failover, automatic rollback, no manual step anywhere in the recovery path. That is achievable and it is a substantially different and more expensive architecture.

Every additional nine costs disproportionately more than the last, and the jump from three to four is where the architecture changes rather than being tuned.

Composite Availability

The arithmetic most architectures never do.

Dependencies in series multiply. If every component must work for the request to succeed:

A_total = A₁ × A₂ × A₃ × ...

Five components at 99.9% each:

0.999⁵ = 0.995  →  99.5%, or about 3.6 hours per month

Warning: A system built entirely from 99.9% components is not a 99.9% system. Every hard dependency you add reduces availability, and the reduction compounds silently.

This is why microservice architectures frequently have worse availability than the monoliths they replaced - a request that touched one process now touches eight, each with its own failure probability and its own network hop.

The corollary that matters most: reducing the number of hard dependencies is usually a larger availability lever than improving any single one. Taking a component from 99.9% to 99.99% is expensive; removing it from the critical path entirely is often cheaper and gains more.

Redundancy in parallel multiplies the failure probabilities instead:

A_total = 1 − (F₁ × F₂)

Two independent 99% components in parallel:

1 − (0.01 × 0.01) = 0.9999  →  99.99%

Two mediocre components, correctly redundant, beat one excellent component. That is the entire argument for multi-AZ, and it is why multi-AZ is the highest-return availability investment available.

Tip: The word doing the work in that formula is "independent," and it is where redundancy designs fail.

Two instances in the same AZ share a failure domain. Two services calling the same database are not redundant with respect to the database. Two Regions using the same deployment pipeline share a failure mode - a bad deployment reaches both.

Ask of every redundant pair: what do these share? Power, network, an AZ, a dependency, a deployment, a configuration source, a control plane. Shared components are where the independence assumption breaks, and the arithmetic silently stops applying.

Hard And Soft Dependencies

The distinction that changes the arithmetic, and the most powerful availability tool available.

A hard dependency must succeed for the request to succeed. It multiplies.

A soft dependency can fail without failing the request - you degrade instead. It does not multiply.

graph TD
    R[Request] --> A[Auth service<br/>**hard** - no auth, no request]
    A --> D[(Primary database<br/>**hard**)]
    D --> RESP[Response]
    R -.-> REC[Recommendations service<br/>**soft** - omit the panel]
    R -.-> PERS[Personalisation<br/>**soft** - serve the default]
    R -.-> AN[Analytics<br/>**soft** - queue it]
    REC -.->|fails| RESP
    PERS -.->|fails| RESP
    AN -.->|fails| RESP

This diagram distinguishes hard from soft dependencies in a request path. The authentication service and the primary database are hard dependencies - without them there is no response, so their availability multiplies into the total. The recommendations service, personalisation, and analytics are soft dependencies: if they fail, the response is still served with the recommendation panel omitted, a default in place of personalisation, and the analytics event queued. Soft dependencies do not reduce composite availability.

Important: Converting a hard dependency to a soft one is frequently the cheapest availability improvement available, and it is an architectural change rather than an infrastructure one.

A recommendations service at 99% availability, called synchronously with no fallback, costs you 1% of every request. The same service with a timeout and a default response costs nothing - the page renders without recommendations and the user largely does not notice.

The mechanism is a timeout plus a fallback, and both must exist. A timeout with no fallback still fails the request; a fallback with no timeout means the request hangs waiting for a dependency that will not answer.

Audit the critical path and ask of each dependency: must this succeed? The honest answer is "no" more often than the code implies, and each conversion removes a multiplier.

MTBF, MTTR, And Where To Spend

Availability decomposes:

Availability = MTBF ÷ (MTBF + MTTR)

Two ways to improve it: fail less often, or recover faster.

Tip: Reducing MTTR is almost always cheaper than increasing MTBF, and it is where most availability gains actually come from.

Making a component fail less often means better hardware, more testing, more redundancy - expensive, and with diminishing returns because some failures are outside your control.

Making recovery faster is largely an engineering and operational problem you fully control. Automatic failover instead of manual. Automatic rollback instead of a decision. Health checks that detect in seconds rather than minutes. Runbooks that are executable rather than prose.

MTTR includes detection, and detection is frequently the largest component. A system that recovers in two minutes but takes fifteen minutes to notice has a seventeen-minute MTTR. Improving detection is often the single highest-return availability investment, and it is comparatively cheap - Course 4 Module 02.

The decomposition worth measuring: time to detect, time to acknowledge, time to diagnose, time to mitigate, time to verify. Each is separately attackable, and the largest one is where to start.

What Each Nine Requires

Target	Architecture
99%	Single AZ, manual recovery, business-hours response
99.9%	Multi-AZ, automated instance replacement, on-call
99.95%	Multi-AZ throughout, no single points of failure, tested failover
99.99%	No human in the recovery path. Automatic failover and rollback everywhere. Possibly multi-Region.
99.999%	Multi-Region active-active, cell-based isolation, and an organisation built around it

The jump from 99.9% to 99.99% is where the architecture changes, not where it is tuned. Below that, a human can be part of recovery; above it, they cannot.

Note that your dependencies bound you. If a managed service in your critical path publishes a 99.95% SLA, your system cannot exceed that for requests requiring it - regardless of your own engineering.

AWS SLAs Are Not Your Availability

Warning: A service's published SLA is a commercial commitment with a credit remedy. It is not a prediction of your availability, and it is not compensation for your loss.

Three things follow, and each catches people:

The SLA applies to the service, not to your use of it. A multi-AZ-capable service used in one AZ does not get the multi-AZ SLA. The higher commitment is usually conditional on you deploying across AZs.

The remedy is a service credit, typically a percentage of that service's charges for the affected period. If a four-hour outage costs you a large sum in lost revenue, the credit will be a small fraction of your bill for that service.

Your composite availability is bounded by your dependencies' SLAs, and you have no mechanism to exceed them for hard dependencies.

Read the SLA for every service on your critical path, and note the conditions. The number is usually conditional on an architecture, and the architecture is the point.

Planned Maintenance And What Counts

Decide explicitly whether planned maintenance counts against your SLO, and get agreement.

The argument for excluding it: it is planned, communicated, and scheduled for low-impact windows.

The argument for including it: the user does not care why the service is unavailable.

Tip: Including planned maintenance in the SLO is the stronger position, and it produces better architecture.

If maintenance counts, the incentive is to eliminate maintenance windows - rolling deployments, blue/green, managed services with transparent patching, online schema changes. Those are all architecturally desirable and they get built when maintenance is expensive.

An SLO that excludes maintenance permits an architecture requiring downtime to change, which is a design nobody would choose deliberately and many arrive at by allowing the exclusion.

Choosing The Target

Two inputs, and neither is aspiration.

What does downtime cost per hour? Revenue, contractual penalties, regulatory exposure, and customer impact. This is a number the business can produce and rarely has been asked for.

What does the next nine cost? In infrastructure, in engineering time, and in ongoing operational burden.

Where the two curves cross is the target.

Different systems in the same organisation warrant different targets, and a single organisation-wide number is either unaffordable or unmet. Tier the services - Course 3 Module 05 Topic 3 makes the same point about RTO and RPO.

And measure current performance before committing. An SLO set below what you achieve is meaningless; one set far above it is aspirational. Course 4 Module 01 Topic 2 covers the measurement.

Common Failure Modes

Warning: Composite availability never calculated. Five 99.9% components is 99.5%.

Redundancy that shares a failure domain. The independence assumption breaks.

Every dependency treated as hard. Multipliers that need not exist.

No timeout or no fallback on soft dependencies. They remain hard in practice.

MTBF optimised, MTTR ignored. The expensive half.

Detection time excluded from MTTR. Frequently the largest component.

SLA read as expected availability.

SLA conditions unread. The higher commitment requires an architecture.

Credits treated as compensation.

One target for the whole organisation. Unaffordable or unmet.

Target set aspirationally. Missed monthly, then ignored.

Planned maintenance excluded. Permits an architecture requiring downtime.

Why This Matters In An Interview

Interview: Availability questions separate candidates who can do the arithmetic from those who name redundancy.

What interviewers are listening for:

Composite availability, calculated, with the series multiplication.

Hard versus soft dependencies as the cheapest lever.

MTTR and detection time over MTBF.

SLAs as commercial commitments, not predictions.

Common follow-ups:

"Your system has six services each at 99.9%. What is your availability?" → About 99.4% if all six are hard dependencies. Then the useful part: which of them actually need to succeed for the request to succeed? Converting even two to soft dependencies with timeouts and fallbacks recovers most of it.

"How do you get from 99.9% to 99.99%?" → Remove humans from the recovery path. Four minutes a month means automatic failover, automatic rollback, and detection in seconds. It is an architecture change, not tuning - and first check whether your dependencies' SLAs even permit it.

"Two instances in one AZ - is that redundant?" → Against instance failure, yes. Against AZ failure, no. Redundancy only multiplies failure probabilities when the components are independent, and these share a failure domain.

"Where would you invest to improve availability?" → Measure MTTR first and decompose it. Detection is usually the largest component and the cheapest to improve. Then convert hard dependencies to soft ones. Both beat improving component reliability.

Practice

Exercise: Starter. List every hard dependency in one critical path and multiply their availabilities. Compare against your stated target.

Intermediate. For each dependency in that path, ask whether it must succeed. For every "no," check whether a timeout and fallback actually exist.

Advanced. Decompose MTTR for your last five incidents into detect, acknowledge, diagnose, mitigate, verify. The largest component is where the next availability improvement is.