Sherry, Tequila and Fairies in Python

Quick Summary

This article focuses on SOTA quantization aware training (QAT) techniques to compress weights into 2-bits, 1.58-bits and 1.25-bits.

Subscribe if you find this useful

Code is available on Google Colab and GitHub.

Here are free Runpod GPU credits. First come first serve :)

1.0 Intro to Ternary Networks

The Era of 1-Bit LLMs: All Large Language Models Are in 1.58 Bits (Ma et al., 2024)1 introduces the BitNet b1.58 ternary networks for LLMs where:

1. Every weight is one of three digits (-1, 0, 1):
2. Matrix multiplication only involves addition and subtraction:

(Ma et al., 2024) introduce the absmean quantization function to constrain weights to the ternary range (-1, 0, 1) as given below:

The network is initially trained in FP32. Then quantization-aware-training (QAT) is applied only on the linear layers to obtain ternary weights.

The bitnet1.58 is super simple. Observe that the backward function is the Straight Through Estimator (STE):

#STE
class BitNetQuantSTE(torch.autograd.Function):
    @staticmethod
    def forward(ctx, w):
        scale = w.abs().mean()
        alpha = w.mean()
        return torch.where(w - alpha > 0, 1.0, -1.0).to(w.dtype) * scale
    @staticmethod
    def backward(ctx, grad_output):
        return grad_output

1.1 Sherry Quantization

Sherry (Sparse Hardware-Efficient Ternary Quantization) is an efficieny trick to make ternary neural networks SIMD efficient (Huang et al., 2026)2.

Sherry integrates a 3:4 sparsity constraint. In every block of 4 weights, exactly 3 are quantized to non-zero values and one is fixed to zero*.

This 3:4 sparsity constraint enables SIMD-friendly bitpacking where each 4-weight block is stored in 5 bits.

(Huang et al., 2026) associate the 3:4 constraint with weight-trapping problem: the challenge where weights accumulate in localized regions and gradients are homogenized.

Arenas (Annealing Residual Synapse modules) are their solution to weight-trapping. Arenas inject gradients during quantization aware training to re-introduce variance in the model weights.

It’s worth noting that only the weights are quantized while activations and KV-cache remain in BF16 or FP8.

C code is available from llama cpp. We use Python code from (Cen et al., 2026)3:

#Sherry
def _reshape_by_granularity(weight,granularity,group_size):
    original_shape=weight.shape
    if len(original_shape)!=2: raise ValueError("Special weight quantization expects a 2D weight tensor.")
    if granularity=="per_tensor": return weight.reshape(1,-1),original_shape
    if granularity=="per_channel": return weight.reshape(original_shape[0],-1),original_shape
    if granularity=="per_group":
        if group_size<=0 or original_shape[1]%group_size!=0: raise ValueError("per_group quantization requires a valid group_size.")
        return weight.reshape(original_shape[0],original_shape[1]//group_size,group_size),original_shape
    raise ValueError(f"Unsupported special quantizer granularity: {granularity}")

class SherryNMQuant(torch.autograd.Function):
    @staticmethod
    def forward(ctx,input,granularity,group_size,n,m):
        original_shape=input.shape
        if len(original_shape)!=2: raise ValueError("Sherry N:M quantization expects a 2D weight tensor.")
        if original_shape[1]%m!=0: raise ValueError(f"Input dimension {original_shape[1]} is not divisible by M={m}.")
        weight=input.reshape(original_shape[0],original_shape[1]//m,m)
        _,topk_indices=torch.topk(torch.abs(weight),n,dim=-1)
        mask=torch.zeros_like(weight,dtype=torch.bool)
        mask.scatter_(-1,topk_indices,True)
        sparse_weight=(weight*mask).reshape(original_shape)
        x,_=_reshape_by_granularity(sparse_weight,granularity,group_size)
        signed=torch.sign(x)
        denom=max(float(x.shape[-1])/float(m)*float(n),1.0)
        scale=x.abs().sum(dim=-1,keepdim=True)/denom
        return (signed*scale).reshape(original_shape)
    @staticmethod
    def backward(ctx,grad_output):
        return grad_output,None,None,None,None

#Sherry Linear layer
class SherryLinear(nn.Linear):
    def __init__(self,in_features,out_features,bias=True):
        super().__init__(in_features,out_features,bias)
        self.quant_step=0
    def forward(self,x):
        if self.quant_step<QUANT_START:
            self.quant_step+=1
            return F.linear(x,self.weight,self.bias)
        w=SherryNMQuant.apply(self.weight,"per_group",GROUP_SIZE,SHERRY_N,SHERRY_M)
        self.quant_step+=1
        return F.linear(x,w,self.bias)

1.2 Tequila Quantization

Tequila is a ternary quantization technique that contends with deadzone-trapping: the challenge where a large number of weights are zero due to uninformative gradients received from the Straight-Through Estimator (STE) during quantization (Huang et al., 2025)4.

Tequila quantization reactivates dead weights by repurposing them as biases during training. The bias term acts as a residual connection and this yields informative gradient signals for the dead weights (Cen et al., 2026):

2.1 Stretched Elastic Quantization

Stretched Elastic Quantization (SEQ) involves the use of quantization grids to determine configurations like (-2,-1,0,1) or (-1.5, -0.5, 0.5, 1.5) for 2-bit quantized networks (Liu et al., 2025)5.

The primary idea is to quantize weights to satisfy the equation below, where k denotes the number of quantization levels:

We use Python code from (Cen et al., 2026). Note that Tencent calls Tequila ‘Ultraquant’ and idk why:

#Actual Tequila Forward and Backward pass
class UltraQuantV2(torch.autograd.Function):
    @staticmethod
    def forward(ctx,input,quant_method,granularity,group_size,enable_zero_point,eps):
        original_shape=input.shape
        if granularity=="per_tensor":
            x=input.reshape(1,-1)
        elif granularity=="per_channel":
            x=input.reshape(original_shape[0],-1)
        elif granularity=="per_group":
            x=input.reshape(-1,group_size)
        else:
            raise NotImplementedError
        G_shape=x.shape
        scale,delta=absmean(x)
        A=torch.zeros_like(x).to(x.device)
        mask_A_pos=x>=delta
        mask_A_neg=x<=-delta
        A[mask_A_pos]=1
        A[mask_A_neg]=-1
        A=A*scale
        B=torch.zeros_like(x).to(x.device)
        mask_B=A==0
        B[mask_B]=eps*x[mask_B]
        A,B=A.reshape(original_shape),B.reshape(original_shape)
        ctx.save_for_backward(mask_B,eps,scale)
        ctx.other=G_shape
        return A,B
    def backward(ctx,grad_A,grad_B):
        mask_B,eps,scale=ctx.saved_tensors
        G_shape=mask_B.shape
        original_shape=grad_A.shape
        grad_A=grad_A.reshape(G_shape)
        grad_B=grad_B.reshape(G_shape)
        grad_output=grad_A
        grad_output[mask_B]=grad_output[mask_B]+eps*grad_B[mask_B]
        grad_output=grad_output.reshape(original_shape)
        return grad_output,None,None,None,None,None

class UltraQuantLinear(nn.Linear):
    def __init__(self,in_features,out_features,bias,quant_method="ultraquant",granularity="per_group",group_size=128,enable_zero_point=False,range_of_lambada=0.01,eps=1e-5):
        super(UltraQuantLinear,self).__init__(in_features,out_features,bias=bias)
        self.quant_method=quant_method
        self.granularity=granularity
        self.group_size=group_size
        self.enable_zero_point=enable_zero_point
        if self.quant_method in ["ultraquantv2","ultraquant"]:
            self.eps=eps
        elif self.quant_method in ["ultraquantv3","ultraquantv4"]:
            self.Lambada=nn.Parameter(torch.randn_like(self.weight)*range_of_lambada,requires_grad=True)
            self.optimizer=torch.optim.AdamW([self.Lambada],lr=1e-4)
    def update_lambada_v3(self,input,B):
        input=input.reshape(-1,input.shape[-1])
        T=input.shape[0]
        Y=nn.functional.linear(input,B)
        s=torch.sum(self.Lambada*B,dim=-1)
        loss=torch.sum((Y-s)**2)/T
        self.optimizer.zero_grad()
        loss.backward()
    def forward(self,input_):
        assert len(self.weight.size())==2
        real_weights=self.weight
        if self.quant_method=="ultraquant":
            eps=torch.tensor(self.eps,device=input_.device,dtype=input_.dtype)
            A,B=StaticQuaternaryQuant.apply(real_weights,self.quant_method,self.granularity,self.group_size,self.enable_zero_point,eps)
            A=A.to(input_.dtype)
            B=B.to(input_.dtype)
            ones=torch.sign(input_.detach())
            out=nn.functional.linear(input_,A)+nn.functional.linear(ones,B)
            if self.bias is not None:
                out+=self.bias.view(1,-1).expand_as(out)
            return out
        elif self.quant_method=="ultraquantv2":
            eps=torch.tensor(self.eps,device=input_.device,dtype=input_.dtype)
            A,B=UltraQuantV2.apply(real_weights,self.quant_method,self.granularity,self.group_size,self.enable_zero_point,eps)
        elif self.quant_method in ["ultraquantv3"]:
            assert not torch.isnan(self.Lambada).any(),f"{self.Lambada}"
            A,B=UltraQuantV3.apply(real_weights,self.quant_method,self.granularity,self.group_size,self.enable_zero_point)
            if self.training:
                self.update_lambada_v3(input_.detach(),B.detach())
            else:
                pass
            B=B*self.Lambada.detach()
        A=A.to(input_.dtype)
        B=B.to(input_.dtype)
        ones=torch.ones_like(input_,device=input_.device)
        out=nn.functional.linear(input_,A)+nn.functional.linear(ones,B)
        if self.bias is not None:
            out+=self.bias.view(1,-1).expand_as(out)
        return out

#Tequila Linear layer
class UltraWarmupLinear(nn.Module):
    def __init__(self,in_features,out_features,bias=True):
        super().__init__()
        self.linear=UltraQuantLinear(in_features,out_features,bias=bias,quant_method="ultraquantv2",granularity="per_group",group_size=GROUP_SIZE,enable_zero_point=False,eps=ULTRA_EPS)
        self.step=0
    def forward(self,x):
        if self.step<QUANT_START:
            y=F.linear(x,self.linear.weight,self.linear.bias)
        else:
            y=self.linear(x)
        self.step+=1
        return y

2.2 Fairy2I Complex (±1, ±i) Quantization

Fairy2i quantization converts real valued weights into complex numbers {±1,±i} (Wang et al., 2026)6. Each weight is replaced in the forward pass by:

During the forward pass, weights are multiplied by {−1, 0, +1}. Then multiplying by ±i is a simple sign flip.

A widely-linear complex representation is a mathematical technique to convert real layers into imaginary-valued weights. The authors simply represent the real and imaginary parts as column matrices lol:

The resulting widely-linear layer resembles the image below under the assumption the real linear matrix has even dimension:

(Wang et al., 2026) introduce PhaseQuant, a phase-based rule on the unit circle for selecting the nearest complex codeword given an angle and a magnitude:

They use Gauss’ multiplication algorithm to go from the 4 multiplications needed for a complex muls to only 3 multiplications and 5 additions:

They further perform recursive residual quantization during training. Here, the idea is to represent each complex weight as a short sum of very low-bit terms, where each term corrects the approximation error of the previous one (Wang et al., 2026).

The T=1 and T=2 Fairy2i quantization schemes resemble:

class PhaseQuantSTE(torch.autograd.Function):
    @staticmethod
    def forward(ctx, w_real, w_imag):
        phase = torch.angle(w_real + 1j*w_imag)
        real_pos = (phase >= -math.pi/4) & (phase < math.pi/4)
        real_neg = (phase >= 3*math.pi/4) | (phase < -3*math.pi/4)
        imag_pos = (phase >= math.pi/4) & (phase < 3*math.pi/4)
        imag_neg = (phase >= -3*math.pi/4) & (phase < -math.pi/4)
        mask_real = real_pos | real_neg
        mask_imag = imag_pos | imag_neg
        s_re = w_real[mask_real].abs().mean() if mask_real.any() else torch.tensor(1e-6, device=w_real.device)
        s_im = w_imag[mask_imag].abs().mean() if mask_imag.any() else torch.tensor(1e-6, device=w_imag.device)
        s_re = s_re.clamp_min(1e-6)
        s_im = s_im.clamp_min(1e-6)
        qr = torch.zeros_like(w_real)
        qi = torch.zeros_like(w_imag)
        qr[real_pos] = 1.0
        qr[real_neg] = -1.0
        qi[imag_pos] = 1.0
        qi[imag_neg] = -1.0
        return qr*s_re, qi*s_im
    @staticmethod
    def backward(ctx, grad_real, grad_imag):
        return grad_real, grad_imag

class PhaseQuantSTE_V2(torch.autograd.Function):
    @staticmethod
    def forward(ctx, wr, wi):
        qr1, qi1 = PhaseQuantSTE.apply(wr, wi)
        qr2, qi2 = PhaseQuantSTE.apply(wr-qr1, wi-qi1)
        return qr1+qr2, qi1+qi2
    @staticmethod
    def backward(ctx, gr, gi):
        return gr, gi 

3.0 Conclusion

All these quantization techniques are SOTA for different goals. Comparing quantization techniques is equivalent somewhat to comparing apples and oranges.

Find out what works best for you!

References

1

Ma, S., Wang, H., Ma, L., Wang, L., Wang, W., Huang, S., Dong, L., Wang, R., Xue, J., & Wei, F. (2024). The Era of 1-Bit LLMs: All Large Language Models Are in 1.58 Bits [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2402.17764

2

Huang, H., Wu, D., Hu, Q., Yu, G., Yang, J., Zhu, J., Liu, X., & Wu, D. (2026). Sherry: Hardware-Efficient 1.25-Bit Ternary Quantization via Fine-Grained Sparsification. ArXiv.

3

Cen, R., Hu, Q., Huang, H., Liu, H., Liu, S., Luo, X., Niu, L., Tan, Y., Wu, D., Xie, L., Yang, R., Yu, G., & Zhu, J. (2026). Angelslim: A More Accessible, Comprehensive, and Efficient Toolkit for Large Model Compression. arXiv. https://arxiv.org/abs/2602.21233

4

Huang, H., Wu, D., Cen, R., Yu, G., Li, Z., Liu, K., Zhu, J., Chen, P., Liu, X., & Wu, D. (2025). Tequila: Trapping-free Ternary Quantization for Large Language Models. ArXiv.

5

Liu, Z., Zhao, C., Huang, H., Chen, S., Zhang, J., Zhao, J., Roy, S., Jin, L., Xiong, Y., Shi, Y., Xiao, L., Tian, Y., Soran, B., Krishnamoorthi, R., Blankevoort, T., & Chandra, V. (2025). ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization. arXiv.

6

Wang, F., Tan, X., Huang, B., Zhang, Y., Wang, G., Cong, P., & Yang, T. (2026). Fairy2i: Training Complex LLMs from Real LLMs with All Parameters in ±1, ±i. arXiv. https://arxiv.org/abs/2512.02901