Deploying to prod on each change can be scary, but not deploying to directly to prod is scarier.
This is the process I ran at Amazon, leading a team deploying to hundreds of millions of customers. Through my consulting, I've led engineering teams from deploying on bi-weekly scheduled releases to shipping on each merge.
Let's start with the obvious part:
You will cause an outage. It is not an if but a when.
No amount of unit testing, integration testing, dogfooding, end-to-end whatever, or sacrificing to the deploy gods will catch every bug.
The testing that was originally done on your feature one week ago was not tested with your teammate's latest changes.
Your testing was done against the pre-prod version of your teammates service, which has now changed and contains a backwards incompatible change.
The longer you wait, the more changes stack on a release. Should you need to rollback, you now need to rollback two weeks of changes rather than 1-2 hours worth.
If we accept the premise that an outage is inevitable, it makes much less sense to dedicate massive resources to QAing a release and instead focus our resources on monitoring & observing a release and being prepared to address an outage when it happens.
Now, onto how we get there:
Prerequisites:
CI/CD
Testing matters a lot less than you'd think. Tests can't prove your change is safe in production. Nothing can. What tests do is make failure cheap. A bug caught in CI costs minutes. A bug caught in production costs your evening doing a rollback.
So run the full suite on each merge, or at least as part of the pipeline: unit, integration, end-to-end tests. The further down the pipeline the bug travels, the more it costs to resolve.
Monitoring/Observability
The name of the game is to be able to catch a regression as soon as possible. To achieve this, you need to have excellent monitoring. This takes the shape of:
- metrics: errors, latency, availability
- logs w/ correlation ids
- alarms for sev-3 and sev-2 (paging) wired on the above two
There's a bit of an art and science to tuning your alarm thresholds. It is a balance between sensitivity and how fast you respond on a real incident. Your target sev-2 time to alert should be 5-10 mins.
Initially, you'll likely be wrong and likely be too sensitive. Unfortunately, this is learned mostly by trial and error, so you'll likely get a few 2am wakeups initially.
Feature Flags
For any change with risk, you should be shipping it behind a feature flag / remote config. A feature flag allows you to roll back and turn off any change within a few minutes, rather than needing to roll back the entire deployment. additionally, if your feature flag service allows it (it should), you can incrementally roll out the feature on a percentage or a cohort basis, further reducing the impact of a bad change.
This allows us to decouple the deployment of code and activation of code. Subtle, but a game changer for reducing risk.
Note: you will need a process for cleaning these up. Ideally, you create a removal ticket for each flag created. Otherwise, when your feature flag service goes down (it will), you'll have a significant regression. ask me how i know.
Automatic Rollback (Deploy time circuit breaker)
A deploy time circuit breaker is functionality that allows you to roll back the deployment if you see a count or percentage of errors as you are rolling it out across the fleet. Most cloud providers have this now with a simple checkbox.
Backwards Compatible Changes
You should already be doing this, but deploying on each commit forces the practice. During a rolling deployment, you will have the old version and the new version running at the same time. Every change needs to work alongside the previous version. Your trick of deploying at midnight to avoid this no longer works.
Deployment Strategies
Now, with those in place, we can walk through a few different deployment strategies that can help reduce risk as you roll out our changes.
One box (canary)
A one box deployment deploys your changes onto one box in the larger fleet. This allows you to reduce the impact of any bad changes to only one host.
you deploy and let this sit for a period of time, receiving a small fraction of the larger traffic. you have our monitoring and alerting configured onto this box that will alert if anything breaks.
Rolling Deployments
A rolling deployment allows you to roll out on a percentage basis over time such that if there is a catastrophic error, you will catch it before it affects all of the machines and we can then begin rolling them back.
Regional Rollout
As your company grows, you will end up having multi-region deployments. Rather than deploying to all of these regions simultaneously, you can deploy into one specific region first (typically the lowest-traffic).
Cases where this doesn't apply
App store
Shipping a mobile app is not totally compatible with this guidance. The app store review queue throttles your deployment cadence and requires a different strategy.
Certified Environments
Medical devices, avionics, industrial control etc. You can't continuously deploy if you need a regulator to certify the build.
On-prem / Self-hosted
You don't get to control the upgrade. You can still continuously deploy on everything you operate, however you still have to version each change and your customer determines when it is adopted.
Where to Start
Don't do all of this at once. Order matters:
- Get CI green and fast. Under 15 minutes ideally
- Get metrics and alarms on error rate, latency and availability. This is the most important part of the exercise
- Put any risky change behind a flag
- Add one-box + automated rollback
- Delete the release calendar
- Find a new use of all your extra time now that you aren't scheduling releases
Most teams I've worked with take about a quarter to get through this. The tooling is the easy part. The org process and breaking the illusion of scheduled releases being safe is the hard part.
If your team is on a release calendar and wants off it, that's the work I do. DM me.
