How do we manage incidents and recovery?
An incident ends when useful work is restored, not when the GPUs become reachable again.
Recovery is more than reachable devices
- Establish impact
Name the coordinator and technical owners
- Contain and restore safely
Protect people and affected workloads
- Stage job recovery
Avoid synchronized loading surges
- Verify useful results
Then review and assign follow-ups
Establish impact first
Which jobs failed? Which users cannot start? Are responses slow, unavailable, or incorrect?
Name a coordinator, technical owners, and a communication owner. Keep confirmed facts separate from hypotheses.
Contain the problem through the approved procedure. Safety protections and qualified facilities response take priority over preserving a running job.
Recover the work
A checkpoint must be complete, reachable, compatible, and restorable.
Restarting every job at once can overload shared storage or preparation capacity. Stage recovery and watch actual completions.
A post-incident review should produce specific owned actions, not simply “be more careful.”
Try it yourself
A fictional rack returns with 100 jobs waiting to reload their models. What could go wrong?
Check your answer
Simultaneous starts can create storage, network, CPU, or power surges. Use the approved staged plan and verify useful results. Ten-minute checkpoint intervals do not guarantee only ten minutes of lost time: failed checkpoints and recovery time also matter.