How do we maintain a live cluster?
Maintenance should preserve the service, not merely finish an installation command. Plan the workload and recovery path before changing a live cluster.
Limit how far a bad change travels
- Plan and drain
Compatibility, jobs, and rollback
- Canary
A limited representative deployment
- Evaluate
Correctness, speed, errors, stop conditions
- Expand or recover
Proceed only on accepted evidence
Drain then test a small rollout
Draining stops new placement and deals safely with running work. Jobs may finish, checkpoint, or need a planned interruption.
A canary is a limited first deployment. Choose representative hardware and workloads, not just an idle node.
Compare correctness, performance, and errors with the baseline. Set stop conditions beforehand.
Know what rollback means
A container rollback does not reverse a firmware update. Some changes cannot be downgraded.
Record the supported combination of driver, firmware, operating system, and application. Check whether shared rack or fabric components widen the maintenance scope.
Try it yourself
A driver canary detects every GPU, but one production application becomes 20% slower. Should rollout continue?
Check your answer
Pause under the agreed criteria and investigate that application path. Device discovery is not workload acceptance. The plan needs scope, drain method, checks, stop conditions, recovery steps, and owners.