Training versus inference
Training changes a model using examples. Inference uses that model to produce an answer. Both may use GPUs, but they are different jobs.
Learning and using a model
Predict → measure error → calculate gradients → update weights
Input → use learned weights → output
Learning versus answering
During typical training, the model predicts, measures error, calculates how its weights contributed, and updates them. Those calculated influences are called gradients.
Inference normally uses the resulting weights without that training update.
For a language model, prefill processes the input prompt. Decode generates subsequent tokens. A token is a unit of text, not necessarily a word.
Decide what success means
Training needs useful progress toward the required quality. An interactive inference service needs acceptable answers within a response-time target.
Batching groups items for processing. It may improve throughput while adding time spent waiting for the batch to form.
Try it yourself
One fictional configuration handles 100 requests per second with a 50 ms batch wait. Another handles 70 with no intentional batch wait. Which wins?
Check your answer
You need the latency target, total response times, output quality, and demand. A higher request count alone does not decide whether users receive a better service.