Is the cluster doing useful work?
A busy GPU is not necessarily doing useful work. It might be retrying failed tasks or producing answers too late.
Count the responses that qualify
Separate the measurements
Allocation is capacity assigned. Activity measures execution during a sampling period. Throughput counts completed work. Goodput counts work meeting the conditions that matter.
Those conditions might include correctness and a response-time target. Define them before comparing systems.
A utilization percentage is not a direct percentage of peak arithmetic performance.
Count the acceptable results
In a fictional comparison, A completes 1,000 responses per second but only 600 qualify. B completes 800 and 760 qualify.
A has higher throughput. B has higher goodput.
Keep the workload and quality target comparable. Deliberate recovery reserves can also justify some idle capacity.
Try it yourself
At equal hourly cost, A completes 10,000 tasks with 8,000 acceptable. B completes 9,000 with 8,550 acceptable. Which delivers more useful work?
Check your answer
B delivers 550 more acceptable tasks and has a lower cost per acceptable task. Include slow responses in the definition rather than hiding them behind an average.