Throughput

Throughput is the amount of work a system completes per unit of time — for AI inference, often measured in tokens or requests per second.

Higher throughput means more work done for the same hardware, which lowers cost per result and supports higher-volume AI labor.

Throughput trades off against latency: techniques like batching raise throughput but can increase the time any single request takes.