CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs

Summary

The paper presents CAT-Q, a post-training quantization (PTQ) method that ternarises pretrained LLM weights to $\{-1, 0, +1\}$ using ~1M calibration tokens rather than the 100B-1T-token QAT recipes used by BitNet 1.58-bit, TriLM, and Tequila. The conceptual move is to couple two ideas on top of the SliderQuant sliding-layer framework: a Learnable Modulation (LM) that reparameterises the pretrained weight distribution and the ternary threshold with three scalar factors $(\delta_\mu, \delta_\alpha, \delta_\Delta)$ per group, and a Softened Ternarization (ST) that anneals a smooth transition function toward hard ternarization over the calibration epochs before switching to hard ternarization for the last $1-\gamma$ fraction. Both ideas are natural adaptations of established quantization tricks (LSQ-style learnable scales; differentiable soft quantization) to the ternary PTQ regime, and their combination is applied at unprecedented scale for ternary methods, including MoE checkpoints at 100B-235B parameters. If the numbers hold, CAT-Q would substantially widen the accessibility of ternary LLMs, since QAT-based ternary methods have so far been confined to sub-8B dense models.

The empirical case, however, is narrower than the framing implies. Evaluation is limited to five zero-shot commonsense reasoning benchmarks, and Appendix E quietly documents catastrophic collapse (Avg 3.44 vs FP16 73.94) on math and coding tasks, with a fix (CAT-Q+) deferred to an unreleased report. The headlined "100,000x reduction in training tokens" is arithmetically true but compares calibration-on-top-of-pretraining against training-from-scratch/from-checkpoint, hiding CAT-Q's dependence on a strong FP16 base - most transparently visible in the Qwen3-4B vs BitNetV1-3.9B comparison, where much of the delta plausibly reflects base-model quality. Statistically the paper is single-run throughout, with several sub-1 pp design decisions (without-$\mu$, group-size 128 vs 64, $s_0=30$ vs 100) drawn from ablations that lack variance estimates. A full-mode replication attempt further confirmed that the advertised code release currently ships no source at all: the entire pipeline had to be reimplemented from the manuscript, only the FP16 evaluation harness reproduced (within 0.7 pp on the two tested checkpoints), the transition function reproduced cleanly, but no CAT-Q accuracy row - headline or ablation - could be verified end-to-end within the available compute. Taken together, CAT-Q is a promising and well-motivated contribution whose evaluation slice, statistical rigor, and code artifact all need to catch up with its ambition.

Strengths

• Scalability of ternary PTQ. CAT-Q is the first ternary method reported to work end-to-end on 100B-235B MoE checkpoints in 8-60 hours on 8 A100-80GB GPUs; this is a real capability jump over BitNet/TriLM/Tequila which top out at 3-8B dense.
• Clean formulation. LM's decomposition $\mu = \mu_0 + \delta_\mu \alpha_0$, $\alpha = \delta_\alpha \alpha_0$, $\Delta = \delta_\Delta \Delta_0$ and ST's smooth-to-hard transition function are well motivated by the reconstruction-error picture in Figure 2 and give a tight, interpretable optimization surface for group-wise ternarization.
• Comprehensive ablations. $\gamma$, $s_0$, threshold, group size, sample count, LM factors, and ST are each swept individually (Tables 5-12), and the recommended defaults ($\gamma=0.8$, $s_0=30$, group=128, 512 samples) are justified rather than asserted.
• Transition function stands up to independent scrutiny. The properties claimed for $f(\cdot)$ - odd symmetry, near-identity at small $s$, monotone sharpening, $2\Delta$-wide zero region, and the $\max|f|\approx 1$ bound at $s=4.95$ - reproduce cleanly to $\mathcal{O}(10^{-4})$ in an independent reimplementation.
• End-to-end deployment. Table 13 pushes the artifacts through llama.cpp (TQ1_0/TQ2_0) on commodity CPU and GPU, showing memory/throughput gains vs Q4_K_M and Q2_K_M, which is what actually validates the "hardware-friendly" argument for ternary weights.
• Honest limitations. The group-size 128 choice is explicitly acknowledged as sub-optimal in exchange for a fair comparison against TriLM/Tequila, and Appendix G flags the missing kernel support and the reasoning-task gap.

Weaknesses

• Reasoning-task collapse buried in appendix. Appendix E's Table B shows vanilla CAT-Q averaging 3.44 vs FP16 73.94 on MATH-500 / GSM8K / Omni-MATH / HumanEval+ / MBPP+. Since the paper positions itself as a general PTQ ternarization method, this ceiling belongs in the main text and directly qualifies claims of "broad applicability."
• CAT-Q+ rescue is deferred to an unreleased report. The variant that raises the math/coding average from 3.44 to 45.60 is described as introducing a "new calibration data generation strategy" whose recipe is postponed to a future technical report, so the only positive story on reasoning-heavy tasks cannot currently be reproduced or evaluated on the paper's own merits.
• Framing of the 100,000x reduction. The comparison is calibration-on-top-of-pretraining vs training-from-scratch/from-checkpoint; CAT-Q inherits a strong FP16 base (e.g. Qwen3), which QAT baselines do not. The abstract does not disclose this asymmetry.
• Base-model confound in the QAT comparison. Table 2 pits Qwen3-4B + CAT-Q against BitNetV1-3.9B and TriLM-3.9B, which are trained from scratch. Without CAT-Q applied to the same base checkpoints (or the QAT baselines fine-tuned from Qwen3), the reader cannot separate "PTQ beats QAT" from "the base model is better."
• Bolded best result at the 3-4B tier is not the default configuration. In Table 2 the bolded Avg 58.80 uses the 2M-token (~1024-sample) setting rather than the paper's 512-sample default (57.06), which is actually below TequilaLLM-3B's 57.60. The 2M setting is never introduced in the surrounding text.
• Narrow evaluation slice. No WikiText-2 or C4 perplexity, no MMLU / TruthfulQA / long-context tasks in the main results. Commonsense zero-shot is a coarse target and can hide reconstruction quality differences that perplexity would reveal.
• No statistical rigor. Every table is a single point estimate. Several conclusions (without-$\mu$ vs with-$\mu$, group 128 vs 64, $s_0=30$ vs 100, 512 vs 1024 samples) rest on sub-1 pp gaps that are plausibly inside seed-to-seed noise.
• Contribution vs SliderQuant is under-quantified. CAT-Q sits inside SliderQuant's sliding-layer framework, but Tables 2-4 never show a ternary SliderQuant baseline. Only Table 5's first row separates the framework from LM/ST, and that is at ternary $W1.58$ where the "SliderQuant" row is really the framework without the paper's contribution.
• Overstated claims relative to the evidence. "Significantly boosts" for ST (last step in Table 5 is +1.1-1.5 pp, smaller than most LM increments); "improves consistently" with sample count (Table 11 Qwen3-4B WG peaks at 512, not 1024); "no PTQ ternarization matches QAT" (universal negative without citation); "originally proposed in TWN" (ternary schemes predate Li et al. 2016); attribution of the SA baseline to BitNet v1/v2 conflates the underlying quantizer with distinct architectural contributions.
• Ternary function in Figure 1 / Equation 2 has overlapping conditions. As written, the middle case $\hat W_i \le \Delta$ (mapped to 0) is not lower-bounded and overlaps with $\hat W_i < -\Delta$ (mapped to $-1$); the intended predicate is $|\hat W_i| \le \Delta$.
• Two-stage ST relay reuses last-iteration gradients through hard ternarization. This is unusual and the paper gives no analysis or ablation against a straight-through hard-ternarization continuation, so its stability rests on Figure B alone - and that figure only compares full-ST vs no-ST, not gradient-reuse vs STE.
• DL-worse-than-SA outliers left unexplained. Appendix F reports that Direct Learning, though initialised from SA, produces per-group errors larger than SA on some groups. This is an interesting optimisation instability that motivates LM, but no mechanism is proposed.
• Wall-clock lower bound is inconsistent. Abstract says "8 to 60 hours" for 14B-235B; introduction says "1 to 60 hours" for 1.7B-235B. Reconcilable but not stated.
• Monotone-with-scale trend is claimed under an ambiguous metric. Absolute Avg drops from Table 1 are not monotone with Qwen3 scale (Qwen3-4B drops more than Qwen3-1.7B), while relative drops are monotone but not the metric used to quote "3.81%."
• Table 4 baselines are not truly bit-width-matched. Dual-1-bit competitors (PB/BiLLM/DB-LLM) are noted to be "close to or larger than W2A16" and the remaining competitors are 2-bit; there is no genuinely 1.58-bit competitor row.

Reproducibility & code

The full-mode replication had to reimplement CAT-Q from the manuscript: the advertised repository ships only .gitignore, LICENSE, README.md, and images/logo.png (one commit, no code object ever in git history, Under Development badge). The following outcomes result:

• Available-but-empty repository. No calibration, quantization, evaluation, or GGUF-export scripts exist at the advertised IntelChina-AI/BitTern URL, so every Table 1-4, ablation, and Table 13 number is unverifiable from the release. This is the dominant reproducibility fact and it undermines nearly every downstream claim.
• FP16 baselines reproduce; no CAT-Q row does. With an independently written harness (lm-eval-harness, acc_norm for PIQA/ARC-e/ARC-c/HellaSwag and acc for Winogrande), the FP16 references reproduce closely - Qwen3-4B Avg 68.43 vs paper 68.25 and Qwen3-1.7B Avg 61.20 vs paper 61.42, both within stated tolerance. No W1.58 row was produced for any model (compute ceiling of 2xA100-80GB against the paper's 8xA100 configuration; ~4.1 h/model measured, never completed).
• Transition function reproduces cleanly (partial). Independent evaluation of Eq. 5 confirms odd symmetry, near-identity at $s=0.1$, monotone sharpening, a $2\Delta$-wide zero region, and $\max|f|=1.0001$ over $W\in[-100,100]$ at $s=4.95$. This is the strongest positive replication in the run.
• Loss-curve claim reproduces in direction but not magnitude (partial). With-ST final loss is lower than without-ST in all three logged sliding windows, but by only 1-2.5% relative rather than "substantially lower," and the with-ST trajectory falls monotonically rather than rising as the paper describes. The reduced 10-epoch schedule likely does not reach the hardening regime, but even in this regime the manuscript's qualitative story is only weakly confirmed.
• Reconstruction-error figure inconclusive. Only the SA .npy dump was completed, so the LM > DL > SA ordering could not be checked. Under-training rather than a refutation, but noted.
• Silent correctness trap discovered in reimplementation. The Eq. 7 target must be recomputed against the frozen FP16 network; naively reusing activations from windows already quantized silently corrupts 26 of 27 windows on Qwen3-4B and produces below-chance accuracies with no exception raised. The paper's text is internally consistent, but never discusses overlapping-window semantics - a reproducer without unit-tested infrastructure is likely to hit this and blame hyperparameters.
• Sliding window size is specified by pointer to an unreleased paper. Section 3.1 defers $l$ (and stride) to SliderQuant defaults, which are themselves unavailable. The reimplementer inferred $l=2$, stride 1; every table is sensitive to this choice.
• Eq. 5 convergence property lacks a measurement protocol. The literal $L_\infty$ reading is vacuous (pinned at $0.5$ by the $\pm\Delta$ discontinuity); a $\pm 0.02$ exclusion radius, invented by the replicator, does yield the monotone-to-zero picture the paper describes.
• Deployment (Table 13) is un-runnable from released artifacts. llama.cpp was built and TQ1_0/TQ2_0/Q4_K_M/Q2_K wired up, but no CAT-Q checkpoints exist to convert to GGUF; effective bits-per-weight for TQ1_0/TQ2_0, llama.cpp commit, packing granularity vs group size, CPU thread count, and CUDA/driver version are also not stated.
• Eval harness, prompt normalization, and calibration seed unspecified. The reproduced FP16 baselines match only because the reimplementer independently converged on the correct harness settings; a different reader will not know which choices to make.
• MoE handling underspecified. The excluded-routing-module set is described in prose only, without concrete module patterns or the fraction of unquantized parameters - critical for the 235B headline.
• W1.58A8 activation quantizer underspecified. Per-token vs per-channel, symmetric vs asymmetric, static vs dynamic scaling are all silent, which matters especially for the BitNetV2 comparison whose contribution is native-4-bit activations.
• CAT-Q+ (math/coding fix) is unreproducible. The calibration-data generation recipe that recovers Avg from 3.44 to 45.60 is deferred to a future report.

Recommended Changes

Essential

• Release the CAT-Q code and configs at the advertised URL. Publish the calibration/quantization pipeline, sliding-window boundary code, evaluation configs (harness version, seed, normalization), MoE routing-module patterns, and the GGUF export path. Ties to the empty-repository and eval-harness reproducibility points, and to the deployment-table reproducibility point.
• Bring the reasoning-task ceiling into the main text. Move Table B out of Appendix E, add vanilla CAT-Q on WikiText-2 / C4 perplexity, MMLU, and math/coding, and adjust the "broad applicability" framing so it is not contradicted an appendix later. Ties to the reasoning-collapse and narrow-evaluation weaknesses.
• Disclose the CAT-Q+ recipe or scope Table B as a limitation. Either describe the new calibration-data-generation strategy in enough detail to reproduce, or drop the CAT-Q+ column and reframe Table B as clean-cut evidence of vanilla CAT-Q's failure mode. Ties to the CAT-Q+ weakness.
• Reframe the 100,000x reduction and add a base-model-matched control. State explicitly that the comparison excludes CAT-Q's upstream FP16 pretraining cost, and add a control - CAT-Q on the same checkpoint the QAT baselines start from, or the QAT baselines fine-tuned from Qwen3. Ties to the framing and base-model-confound weaknesses.
• Report per-seed variance for the headline and every sub-1 pp ablation. At minimum, mean $\pm$ std over $\geq 3$ calibration seeds for Tables 1, 9, 11, and 12. Ties to the statistical-rigor weakness.
• Fix the ternarization function in Figure 1 / Equation 2. Rewrite the middle predicate as $|\hat W_i| \le \Delta$ so the mapping is unambiguous. Ties to the overlapping-conditions weakness.
• Specify sliding-window semantics. State the window size $l$, stride, and - crucially - that the reconstruction target is computed against the frozen FP16 network rather than the partially quantized cascade. Ties to the sliding-window and silent-correctness-trap reproducibility points.

Suggested

• Add a SliderQuant @ W1.58 row to Tables 2 and 3 so the reader can attribute CAT-Q's gain to LM+ST rather than the sliding-layer framework. Ties to the SliderQuant contribution weakness.
• Adjust Table 2 bolding to the 512-sample default, or introduce the 2M-token setting in prose and apply the same calibration budget to competitors. Ties to the bolded-best-result weakness.
• Provide a deployment appendix with TQ1_0/TQ2_0 effective bits-per-weight, llama.cpp commit and build flags, packaging tooling, and CPU/GPU config for Table 13, plus at least one exported GGUF. Ties to the deployment-table reproducibility weakness.
• Write out the activation-quantizer formula for the W1.58A8 rows (per-tensor/per-token, symmetric/asymmetric, static/dynamic). Ties to the activation-quantization weakness.
• List MoE routing-module patterns and shared-expert handling for Qwen3-30B-A3B / Qwen3-235B-A22B / RF2-100B-A6.1B, with the unquantized-parameter fraction. Ties to the MoE reproducibility weakness.
• Add a per-model wall-clock column (14B / 32B / 30B-A3B / 70B / 100B-A6.1B / 235B-A22B) and describe the distributed strategy, reconciling the abstract's "8 to 60 hours" with the introduction's "1 to 60 hours." Ties to the wall-clock inconsistency and MoE-reproducibility weaknesses.
• State the measurement protocol for the Eq. 5 convergence property (metric, exclusion radius around $\pm\Delta$, evaluation grid). Ties to the convergence-protocol reproducibility point.
• Soften overstated claims: "significantly boosts" $\rightarrow$ "adds a further +1.1-1.5 pp"; "improves consistently" $\rightarrow$ "best average"; "no PTQ ternarization matches QAT" $\rightarrow$ add "to our knowledge" or a citation; "originally proposed in TWN" $\rightarrow$ "formalised in TWN"; separate SA-as-quantizer from BitNet-specific contributions. Ties to the overstated-claims weakness.
• Add an ablation of the two-stage relay's frozen-gradient continuation against a straight-through hard-ternarization baseline. Ties to the two-stage-relay weakness.
• Add a mechanism sentence for the DL-worse-than-SA outliers (e.g. threshold-flipping instability), or add an $L_2$-regularised DL variant as a control. Ties to the DL-outliers weakness.
• Reconcile the "monotone drop with scale" metric. Pick either absolute or relative drop, apply it consistently across families, and adjust the prose in Section 3.3. Ties to the mixed-metric weakness.
• Add a matched-bit-width baseline to Table 4 (e.g. Wanda / QuIP / QuaRot at 2-bit weight-only with matched calibration), or reframe as efficiency-per-bit. Ties to the Table 4 baseline weakness.