A deployment job turns green. The code reached production. But did the service stay healthy? Was recovery tested? Which sites received the release? Where is the evidence? Those questions should not require someone to reconstruct the release from Slack messages and pipeline logs.
I am working on change automation for GitHub-based releases. The goal is to connect the change record to the actual work: release instructions, qualification, deployment, validation and recovery. What follows is the workflow I am building toward, not a claim that every part is already implemented.
Build once. Prove it before promotion.
Start with a release tag or branch and produce a versioned artifact. Run unit tests and build checks before promoting it to pre-production.
Create the pre-production change with clear instructions, health checks and a recovery plan. Then deploy that artifact and test the service in its environment. Integration tests belong here, along with checks that dependencies and important service functions still work.
Recovery needs testing too. Can we roll back? Can we roll forward again? Do old and new versions work together during the transition? That last question matters for stateful services. Returning to an older binary does not automatically undo a database change. AWS’s guidance on rollback safety makes compatibility across versions a first-class deployment concern.
If qualification fails, block promotion. Roll back pre-production where configured and safe, and preserve the evidence.
Bake time needs something to observe
A 24-hour wait is not proof that a release works. The observation window should reflect the service and its workload. Are important operations being exercised? Are dependencies behaving normally? Are errors, latency and other predefined health signals within acceptable limits?
Keep the duration configurable. Twenty-four hours may be appropriate for a particular service, but it is not a universal rule. AWS describes production bake periods that also require sufficient request volume. The point is to collect useful evidence, not simply let a timer expire.
The proposed workflow
Open the diagram at full size.
Production earns its own confidence
Create the production change and attach the pre-production results. Apply the required policy checks and approvals. Promote the same artifact, not a fresh rebuild. Environment-specific configuration still needs its own versioning and review.
Start with limited production exposure. Depending on the service, that could be a small set of instances, a cell or a share of traffic. Why test again? Because pre-production cannot reproduce every customer workload. Google’s canary guidance explains how a small production deployment can expose defects before they reach everyone.
Then expand through defined sites or regions. Each wave follows the same sequence:
Deploy. Validate. Observe. Decide.
Choose the order around failure isolation and representative coverage, not convenience. Azure’s published deployment approach includes pilots across different hardware and configurations, with observation between stages.
Every health gate needs three possible outcomes:
- Pass: checks are satisfied and there is enough observation to proceed.
- Hold: metrics are missing or traffic is insufficient. Do not expand.
- Fail: stop expansion, execute the tested recovery plan and verify recovery.
No errors and no traffic is not a passing result.
Close the record with what actually happened
If a wave fails, do not proceed to the next one. The recovery plan must define which targets to recover and how. Rolling back the failing site and rolling back every previously updated site are different decisions.
Keep unresolved, partial or paused changes open. A rollback command finishing successfully is not enough; verify that the service recovered.
When every intended target passes final validation and observation, complete the production change with the evidence. If the release was rolled back and recovery is verified, record that actual outcome rather than calling it a successful release.
Passing tests earns a release limited production exposure. Production evidence earns it wider exposure.
This is not the only valid release process. Spotify has described an approach where most components did not use staging environments. The choices depend on the service and the strength of its safeguards.
For this workflow, the aim is straightforward: make the pipeline perform the checks, carry the evidence and stop when it cannot justify going further.
