Local AI performance is often reported in tokens per second. The number is useful but easy to misread. It usually describes how quickly a model generates new output tokens after generation begins, not the complete time from clicking a button to receiving the finished answer.
What tokens per second measures
If a model produces 20 tokens per second, roughly 20 output units are generated each second. The number of words varies by language and text because one token is not the same as one word.
Prompt processing happens first
Before the first answer token appears, the model has to process the existing context. With a long PDF or a large conversation, this prefill phase can take noticeable time. A high generation rate afterward does not remove that delay.
Time to first token also matters
Perceived speed depends strongly on how long it takes before the first visible part of an answer appears. A system with a slightly lower token rate but very short startup latency can feel more responsive than a theoretically faster model with slow prefill.
What affects token rate?
- model size
- quantization
- CPU and GPU performance
- memory bandwidth
- VRAM or Unified Memory
- how many layers are offloaded
- runtime implementation and optimization
Smaller models are often much faster
A 3B or 7B model can generate far more quickly than a 14B or 32B model on the same computer. If the smaller model performs the task reliably, it may be more productive in everyday use.
More tokens per second is not always important
For a short summary, the difference between 30 and 60 tokens per second may barely matter. For long-form generation or code output, the difference becomes much more noticeable. The right speed depends on the workload.
Compare like with like
Benchmark numbers are hard to compare when model, quantization, context length, hardware, or runtime differ. A token-rate figure without those details is closer to marketing than a reproducible measurement.
A simple example
If a model produces a 600-token answer at 20 tokens per second, generation alone takes about 30 seconds. Prefill, file processing, and other runtime work still come on top.
Evaluating local AI realistically
Local AI should therefore be judged across three separate questions: Does the model fit in memory? Is the output quality sufficient? Is the speed acceptable for the actual workflow? See Local AI for more background.
Conclusion
Tokens per second is a useful measure of generation speed, not a complete performance score. Context processing, startup latency, model quality, and memory requirements are equally important to the real user experience.