Pick a model, a quantization and a device — see whether it fits and how fast it runs.
Precision of the model weights. Lower uses less VRAM but costs quality.
KV cache precision. Dominates VRAM at long context.
Hardware
Select your GPU or configure a custom device.
Tensor-parallel replicas. Comms overhead is included.
Lets the model exceed VRAM — at host-bandwidth speed.
Workload
Sequences processed per step. Raises throughput, costs KV cache.
Tokens per sequence (prompt + generation). Drives KV cache.
Simultaneous requests. Multiplies KV cache, splits per-user speed.
0 GB
of 0 GB usable
Memory allocation
Assumes the device runs continuously at the estimated utilization.