What should we monitor?
Start monitoring with a question: are users getting correct results on time? Then follow the dependencies that explain the answer.
From user impact to physical cause
- Application
Correct output, completion time, errors
- Scheduler
Queue wait and placement
- GPU and server
Memory, clocks, health, input supply
- Network and facilities
Links, power, cooling, fresh measurements
Connect the layers
Look at application progress, queue wait, GPU health, server and network behavior, and relevant facility signals.
Keep identity attached: which GPU, host, rack, and job? A temperature without a location is hard to act on.
One sample is not a trend. A quiet GPU might be loading, waiting, checkpointing, or finished.
A read-only starting point
On an authorized NVIDIA Linux host with driver tools:
nvidia-smi --query-gpu=name,uuid,memory.used,memory.total,temperature.gpu,utilization.gpu --format=csvField availability varies. This is not an Apple GPU command.
An alert needs an owner and a next action. Missing measurements should appear as unknown, not healthy.
Try it yourself
A job slows down, but memory use stays steady. What else would you inspect?
Check your answer
Phase timings, GPU activity and clocks, input delays, communication, errors, and recent changes. Allocated memory does not prove that calculation is progressing. Avoid unbounded metric labels; detailed request history may belong in logs instead.