§ 01
PROBLEM
PROBLEM
What it had to solve.
Quantization papers report accuracy on single devices. In practice, the "best" method changes with the hardware — memory bandwidth, tensor cores, and INT support all shift the tradeoff.
Quantization Benchmarks — picking the right knife for the device.
Cross-hardware benchmarking of AWQ, GPTQ, and SmoothQuant across NVIDIA A100, Jetson Orin, and Raspberry Pi. Derived device-specific deployment strategies for edge and cloud.
Methods tested
3
AWQ · GPTQ · SQ
Devices
3
A100 · Orin · Pi
Models
6
From 125M to 13B
Accuracy drop
<1
% on AWQ INT4
§ 02
APPROACH
APPROACH
How it works.
A unified harness runs each method on each device across six models, logging latency, memory, and downstream accuracy. Results are mapped onto a decision matrix that picks the right method for a target device + model size.
§ 03
RESULTS
RESULTS
What it delivered.
AWQ dominated on Orin for 7B-class models; SmoothQuant won on A100 for throughput; GPTQ remained competitive on CPU. All three held within 1% of FP16 accuracy on downstream tasks.