What happens when hardware fails?
An error message is evidence, not a verdict that the GPU card is broken. Software, links, power, and cooling can also cause failures.
From error to trustworthy capacity
- Capture the evidence
Device identity, time, jobs, and logs
- Contain suspect capacity
Follow the approved allocation procedure
- Diagnose and repair
Use the supported checks
- Validate before release
Health plus representative workload
Identify and contain
Record the stable device identity, host, jobs, timestamps, and related events. Preserve logs.
For NVIDIA errors, use the relevant Xid guidance rather than interpreting every code the same way. A corrected memory-error count is not a universal replacement threshold.
If the resource is suspect, follow the scheduler’s process to prevent new placement and handle running jobs appropriately.
Look beyond one device
Several GPUs failing together may share a dependency. A failure following one application across hosts suggests another investigation.
Neither pattern proves a cause, but both are more useful than repeatedly rebooting.
Try it yourself
A fictional node fails three jobs. Each reboot looks successful for a while. What should happen before a fourth job?
Check your answer
Contain the suspect capacity, inspect the history, and arrange approved diagnostics. Return it only after the required health and workload checks. A live reset is disruptive, not a harmless experiment.