ReRound: Reconstructive Rounding to Resolve Midpoint Ambiguity in Calibration-Free LLM Quantization

H Kung, He-Yen Hsieh

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 16 functions out of this paper's own repositories and ran 13 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• louisYen/ReRound

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

ReRound (Reconstructive Rounding) is a post-training quantization method that addresses the midpoint ambiguity inherent in standard round-to-nearest (RTN) schemes when quantizing weights near the centers of quantization intervals. Starting from a pretrained LLM, ReRound trains a conditional diffusion model to produce continuous reconstructions of lowbit weights for the LLM. These reconstructed weights act as a guidance signal to disambiguate the rounding direction of weights located close to interval midpoints. To integrate this reconstruction-guided rounding with conventional RTN, ReRound introduces a tolerance metric measuring how far the quantized weight (not the final quantized integer) is away from the midpoint: quantized weights within a tolerance region around midpoints are quantized using diffusionbased reconstructions, whereas weights closer to quantization boundaries are quantized with RTN. By sweeping the tolerance parameter, ReRound generates multiple candidate quantized integer weight matrices and selects the de-quantized weight matrix candidate whose leading singular values most closely match those of the original full-precision weights. This selected candidate determines the tolerance parameter ReRound uses. ReRound is particularly effective for smaller LLMs. Across a range of such models, it consistently outperforms standard RTN for 3-bit and 4-bit weight quantization. ReRound achieves superior accuracy compared to an extensive set of calibration-free methods, remains competitive with calibration-dependent approaches, and operates entirely offline, introducing no additional overhead during low-bit inference. The ReRound strategy represents a new approach for low-bit quantization. The method applies to AI models beyond LLMs. This paper focuses on its applications to small LLMs. Code is available at https://github.com/louisYen/ReRound. Reconstructed Weight Matrix 𝑊 !"# Round Up (a) Midpoint Ambiguity: Round Down or Up? Round Down Round Up Floor Ceiling Midpoint 1 2 = 1.76 = 1.48 1.5 𝑤 " 𝑤 " !"# Diffusion-Prior Guidance from (b) (b) Diffusion-Prior Guidance Patches of LLM Weight Matrices Train U-Net Weight Matrix 𝑊 𝑤 " !"# Learned Diffusion Prior (U-Net) Quantized floating-point weight 𝑤 # Reconstructed quantized floating-point weight 𝑤 # !"# Map to quantized coordinates

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2608.11045")
get_code_for_paper("2608.11045")
have("2608.11045")

Connect an agent — have() is free.