Free lesson · Microservices Interview
Deployment Strategies - Blue-Green, Canary, Rolling
Deployment Strategies - Blue-Green, Canary, Rolling
Module 07's exercise covered the routing mechanics of blue-green and canary deployments - how Istio VirtualService configurations shift traffic between versions. This lesson steps back to the deployment strategy itself: what each approach requires in terms of infrastructure, how long deployments take, what the risk profile is, and when each strategy is the right choice for a given service.
In a monolith, deployment is one event - one artifact, one server (or server cluster), one rollback unit. In microservices, deployment happens independently per service. The Order Service deploys version 2.4 while the Payment Service stays on version 3.1 and the Inventory Service is mid-deployment on version 1.8. Each service needs its own deployment strategy - and the strategy may differ by service based on risk, traffic volume, and dependency sensitivity.
At Amazon, we deploy thousands of times per day across hundreds of services. The deployment strategy for each service is chosen based on its blast radius - a payment service uses canary with automated rollback. An internal admin tool uses rolling deployment. A data pipeline uses blue-green. There is no single "best" strategy - there is the right strategy for each service's risk profile.
Rolling Deployment
How It Works
The existing instances are replaced with new instances one at a time (or in small batches). At any point during the deployment, some instances run the old version and some run the new version. When all instances are replaced, the deployment is complete.
In Kubernetes, a rolling deployment is the default behaviour of a Deployment resource. The maxUnavailable and maxSurge parameters control the pace:
- maxUnavailable: 1 - at most 1 pod is unavailable during the update (the pod being replaced)
- maxSurge: 1 - at most 1 extra pod is created above the desired count during the update
For a service with 4 replicas: Kubernetes creates 1 new pod (v2), waits for it to pass the readiness probe, then terminates 1 old pod (v1). Repeats until all 4 pods are v2. At every point during the deployment, at least 3 pods are handling traffic.
Kubernetes rolling update configuration:
The deployment strategy is configured in the Deployment spec:
strategy.type: RollingUpdate(default)strategy.rollingUpdate.maxUnavailable: 25%- at most 25 percent of pods are unavailable during the update. For 4 replicas, at most 1 pod is unavailable.strategy.rollingUpdate.maxSurge: 25%- at most 25 percent extra pods can exist during the update. For 4 replicas, at most 1 extra pod (total 5 pods) during the transition.minReadySeconds: 30- a new pod must be healthy for 30 seconds before the next pod is replaced. This catches pods that pass the readiness probe initially but crash after a few seconds (e.g., a connection pool that takes 20 seconds to exhaust).
The minReadySeconds parameter is frequently overlooked but critical for catching slow-crash bugs - a service that starts successfully but fails after processing a few requests. Without minReadySeconds, Kubernetes immediately proceeds to the next pod replacement - potentially replacing all pods with the buggy version before the crash is detected.
Risk Profile
Mixed-version traffic. During the deployment, some traffic goes to v1 instances and some to v2. If v2 changes the API response format, some clients receive the old format and some receive the new format - creating inconsistency. This is safe when v2 is backward-compatible with v1's API contract. It is unsafe when v2 introduces breaking changes.
No instant rollback. Rolling back requires another rolling deployment - replacing v2 instances with v1 instances one at a time. Rollback takes the same amount of time as the deployment. For a 10-instance service with 2-minute pod startup, rollback takes 10-20 minutes - during which the broken v2 instances continue serving traffic.
Partial failure state. If v2 has a bug that only manifests under load, the first few v2 instances might pass the readiness probe (low traffic) but fail when more traffic shifts to them (as v1 instances are terminated). By the time the failure is visible, half the instances are v2 and the blast radius is significant.
When to Use Rolling
Rolling deployment is appropriate for low-risk deployments: internal tools, non-critical services, services with comprehensive automated testing that catch bugs before production, and services where mixed-version traffic is safe (backward-compatible APIs).
Rolling is NOT appropriate for: services that handle financial transactions, services where mixed-version traffic causes data inconsistency, or services where rollback speed matters (the 10-20 minute rollback window is too long for critical services).
Blue-Green Deployment
How It Works
Two complete environments exist simultaneously: Blue (the current production version) and Green (the new version). Both environments are fully provisioned with the same number of instances. Traffic flows entirely to Blue while Green is deployed and tested.
When Green passes all validation (health checks, smoke tests, integration tests against the live environment), traffic is switched from Blue to Green in one atomic operation - a load balancer or DNS change. Blue remains running as the rollback target. If Green shows problems, traffic is switched back to Blue instantly.
Infrastructure Requirements
Blue-green requires double the infrastructure during the deployment. If the service normally runs 10 instances, blue-green requires 20 instances during the deployment (10 Blue + 10 Green). This doubles compute cost during the deployment window.
For services with expensive dependencies (dedicated database connections, GPU instances, licensed software), the cost of maintaining two complete environments may be prohibitive. Blue-green is most practical for stateless services where additional instances are cheap (standard compute pods in Kubernetes).
Database migration complication. If the new version requires a database schema change, both Blue and Green must work with the same database. The schema migration must be backward-compatible - Blue (old version) must function with the new schema, and Green (new version) must function with the old schema during the transition. This is the expand-migrate-contract pattern from Module 07's exercise:
- Deploy the schema migration (add column, create table) - backward-compatible with Blue
- Deploy Green with the new code that uses the new schema
- Switch traffic from Blue to Green
- After confirming Green is stable, clean up the old schema elements (if needed)
Risk Profile
Instant rollback. The primary advantage. If Green has a bug, switch traffic back to Blue. Rollback is a single load balancer or routing change - typically 5-10 seconds. Blue has been running the entire time with warm caches and established connections. No cold start, no connection pool warm-up.
No mixed-version traffic. All traffic goes to Blue or all traffic goes to Green - never a mix. This eliminates the API inconsistency risk of rolling deployments.
All-or-nothing validation. Green receives 100 percent of traffic immediately after the switch. If Green has a load-dependent bug (works at low traffic, fails at production traffic), 100 percent of users are affected immediately. This is mitigated by running load tests against Green before the switch - but load tests cannot perfectly replicate production traffic patterns.
When to Use Blue-Green
Blue-green is appropriate for services where instant rollback is critical: payment processing, user authentication, data-integrity-sensitive services. It is also appropriate when mixed-version traffic is unacceptable - when v1 and v2 have incompatible API contracts or incompatible database expectations.
Blue-green is NOT appropriate when the infrastructure cost of double provisioning is prohibitive, or when the service's dependencies cannot support two simultaneous environments.
Canary Deployment
How It Works
The new version is deployed alongside the existing version. A small percentage of traffic (1-5 percent) is routed to the canary. The remaining traffic continues going to the stable version. The canary is monitored for errors, latency, and correctness. If the canary is healthy, the traffic percentage is gradually increased (5 → 25 → 50 → 100 percent). If the canary shows problems, traffic is shifted back to 0 percent (rollback).
Module 07's exercise covered the Istio VirtualService configuration for canary traffic splitting. This section focuses on the strategy-level decisions.
The Canary Analysis Pipeline
The critical question in a canary deployment is: how do you decide whether the canary is healthy? Manual monitoring (an engineer watching dashboards) is error-prone - the engineer might miss a subtle degradation, might be distracted, or might hesitate to rollback. Automated canary analysis removes the human from the decision loop.
Flagger (Kubernetes canary controller) automates the canary analysis:
- Deploy the canary pods alongside the stable pods
- Flagger sets the initial traffic weight (e.g., 5 percent to canary)
- Flagger queries Prometheus for canary metrics: error rate, latency P99, custom metrics
- Flagger compares canary metrics against the stable version's metrics (not against absolute thresholds - relative comparison is more accurate)
- If the canary's error rate is less than 1.5× the stable version's error rate AND the P99 is less than 1.2× the stable version's P99, the canary passes. Flagger increases the traffic weight.
- If the canary fails any metric check, Flagger sets canary traffic to 0 percent and alerts the team.
- If all steps pass, the canary becomes the new stable version. The old stable is decommissioned.
Argo Rollouts provides similar functionality with a different configuration model. Both tools integrate with Istio, Linkerd, and Nginx for traffic splitting.
Risk Profile
Gradual blast radius. The primary advantage. At 5 percent traffic, only 5 percent of users are affected by a canary bug. At 25 percent, 25 percent are affected. The blast radius grows gradually - and at any point, rollback reduces it to 0 percent.
Statistical validation. The canary receives real production traffic - with real traffic patterns, real data, and real concurrency. This validates the deployment in a way that staging environments and load tests cannot match.
Longer deployment time. A canary deployment with 4 traffic steps and 15-minute soak times takes 1+ hour to complete - versus minutes for rolling or seconds for blue-green's traffic switch. For services that deploy multiple times per day, the canary pipeline must be fast enough to avoid blocking subsequent deployments.
Metric noise at low traffic. At 1 percent traffic, the canary might handle only 10 requests per minute. A single slow request makes the P99 look terrible. Statistical noise at low sample sizes can trigger false rollbacks or mask real problems. The minimum canary traffic should be high enough to produce statistically meaningful metrics - typically 5 percent or at least 100 requests per minute.
When to Use Canary
Canary is appropriate for any service where deployment risk justifies the longer deployment time: customer-facing services, services with complex dependency chains, services where bugs are hard to catch in staging, and services where the blast radius of a bad deployment is high.
Canary is the recommended default strategy for production microservices. It provides the best balance of safety (gradual blast radius), validation (real traffic), and automation (Flagger/Argo Rollouts).
Interview: The deployment strategy answer: "I would use canary deployment as the default for all customer-facing services. The canary receives 5 percent of traffic initially, with automated analysis comparing its error rate and P99 latency against the stable version. Flagger advances the canary through 5 → 25 → 50 → 100 percent if metrics pass, or rolls back to 0 percent if any metric breaches. For internal tools with low blast radius, I would use rolling deployment for simplicity. For payment services that need instant rollback, I would use blue-green with the canary analysis as a pre-switch validation."
Choosing the Strategy Per Service
| Factor | Rolling | Blue-Green | Canary |
|---|---|---|---|
| Rollback speed | Slow (10-20 min) | Instant (5-10 sec) | Fast (seconds to shift traffic) |
| Infrastructure cost | No extra capacity | 2× during deployment | 1.05-1.5× during deployment |
| Mixed-version traffic | Yes (during rollout) | No | Yes (canary + stable) |
| Deployment duration | Minutes | Minutes + validation | 1+ hour (with soak times) |
| Validation quality | Low (readiness probe only) | Medium (pre-switch testing) | High (real traffic analysis) |
| Best for | Low-risk internal services | Instant-rollback requirements | All customer-facing services |
Coordinating Deployments Across Services
In a monolith, deployment is one event. In microservices, 50 services deploy independently - but some deployments have cross-service dependencies. The Order Service v2.4 introduces a new API field that the Checkout BFF v3.1 depends on. If the BFF deploys first, it calls an API field that does not exist yet. If the Order Service deploys first, the new field exists but no one uses it - safe.
The backward-compatibility rule: Every deployment must be backward-compatible with the previous version of every service it communicates with. The Order Service v2.4 must work with both BFF v3.0 (old, does not use the new field) and BFF v3.1 (new, uses the new field). The BFF v3.1 must work with both Order Service v2.3 (old, does not have the new field - BFF treats it as optional) and v2.4 (new, has the field).
This means API changes follow the expand-contract pattern:
- Expand: The Order Service adds the new field as an optional addition (v2.4). Old clients ignore it. New clients can use it.
- Migrate: The BFF deploys v3.1, which starts using the new field.
- Contract: After all consumers are using the new field, the Order Service can make it required in a future version.
Breaking this rule - deploying a version that requires a field that does not exist in the current production API - is the most common cross-service deployment failure.
Deployment Anti-Patterns
Big-bang deployment. Deploying all 50 services simultaneously to "keep everything in sync." This creates a single failure event that affects the entire platform. If any one service has a bug, all 50 services are potentially broken. Independent deployment is the core principle of microservices - deploy services independently, not as a coordinated batch.
Manual deployment decisions. An engineer manually decides "this canary looks fine, promote it." Manual decisions are slow (the engineer must actively monitor for 30+ minutes), error-prone (fatigue, distraction, confirmation bias), and unscalable (you cannot have an engineer watching every deployment for 50 services). Use automated canary analysis (Flagger, Argo Rollouts) to remove the human from the promotion decision.
No deployment pipeline for rollback. The deployment pipeline is well-tested. The rollback procedure is a wiki page that nobody has read. Rollback must be as automated as deployment - a single button (or automatic trigger) that reverts to the previous version. If rollback requires manual steps (SSH into servers, modify config files, restart services), it will fail under the pressure of a production incident.
Deploying during peak traffic hours. A deployment during the daily traffic peak (8 PM for a consumer platform) maximises the blast radius of a bad deployment - the most users are affected. Deploy during low-traffic windows (2-4 AM, early morning) to minimise the user impact of deployment issues. Exception: if the deployment fixes a production incident that is currently affecting users, deploy immediately regardless of traffic.
Tip: In interviews, mention the deployment window and blast radius. "I would configure deployments to run during the low-traffic window (2-4 AM) and use automated canary analysis with Flagger. The canary receives 5 percent of traffic for 15 minutes, and if the error rate exceeds 1.5× the stable version's rate, Flagger automatically rolls back. This minimises the blast radius - at most 5 percent of the low-traffic user base is affected by a bad deployment."
What Comes Next
The next lesson covers containerisation and orchestration - not a Docker tutorial, but the microservices-specific Kubernetes concerns: sidecar patterns, pod design, resource limits, horizontal pod autoscaling, and namespace isolation.