AutoRound Quantization for LLMs

When an LLM is quantized, it becomes possible to fit it onto a consumer GPU. Quantization is an important step in getting an LLM to work on many GPUs. Recently I investigated the AutoRound quantization from Intel.

AutoRound

In AutoRound, a sophisticated algorithm is used to determine the quantized weights. It claims to be state of the art ( SOTA ) for quantization. In my ongoing quest to improve my local LLM, I downloaded an AutoRound quantization of Qwen 3.6 from Hugging Face.

In my testing:

• The AutoRound quantization worked well and was consistent in its output.

• I managed to make some worthwhile code changes in OpenCode with it.

• I could not tell if it was better or worse than other same-bit quantizations in an objective sense, but I tended to prefer its output.

Also AutoRound was slightly smaller and faster than other quantizations. No quantization is going to be perfect, but AutoRound may be my new preferred quantization format (unless QAT is available, which is likely even better than AutoRound).