Measured decode and prefill throughput for local GGUF models on one AMD Strix Halo box.
Decode is the number that governs how fast text appears. Bars are sorted fastest first.
Same dense model, four quantizations, same machine. Decode falls as the weights grow; prefill barely moves.
Prefill at 512 tokens, decode at 128, and decode with a 4096-token cache already warm.
One machine, one software stack — these numbers characterise this box at this ROCm and llama.cpp version, not Strix Halo in the abstract. Throughput is measured; quality is not, so nothing here tells you whether a fast quantization degrades output. Two models are absent because they could not be measured, not because they were dropped: a diffusion LM this llama.cpp build cannot load, and a 95 GiB quant that exceeds the 64 GiB VRAM allocation.
Data and harness: axjns/strix-halo-inference-bench · Kernels: axjns/strix-halo-kernels