Don't Force the Fit: Bounded Log-Likelihood Loss for Enhanced Reasoning in Large Language Models

Summary

The paper asks whether the token-level cross-entropy objective at the heart of supervised fine-tuning (SFT) is the right optimization target for reasoning-oriented tasks, where correctness is defined at the level of logical validity or final answer rather than surface token realization. The authors argue that the low-probability tail in reasoning demonstrations is dominated by stylistic or realization-specific artifacts, so CE's near-saturated corrective gradient on those tokens misallocates optimization pressure and can hurt generalization. From this diagnosis they propose the Bounded Log-Likelihood Loss BLL-Loss := $\log(2 - \pi_\theta(y_t \mid x))$, a parameter-free CE alternative that saturates at $\log 2$ as $\pi_\theta \to 0$. Two derivations are shown to converge on the same closed form: an alternating-series regularization of the CE Taylor expansion, and a $\beta \to 0$ limit of a log-odds decomposition that decouples "target promotion" from "complement suppression". Empirically, the paper backs the objective with a token-probability characterization of NuminaMath, a hard-masking diagnostic, main results on Qwen2.5-Math, Qwen3, and Llama-3 across math, code, and general-reasoning benchmarks, an ablation against learning-rate/gradient-clip controls, a data-scaling study, and an honest failure case for out-of-distribution knowledge injection. The reframing is timely and the objective is simple, well-motivated, and readily deployable. However, several central pieces of the argument are overstated or under-supported: the "natural" convergence of the two derivations depends on intermediate expressions (Eq. 9 and the $\beta$-loss form) that are never rendered, so the $\beta = 0$ limit is a design choice rather than a discovery; the "monotonic peak at $\beta = 0$" and "gains persist as model size increases" claims sit uneasily with the reported Qwen3-8B row and with the absence of per-seed variance; and the empirical scope leans on math-pretrained backbones where the stylistic-noise premise is favourably satisfied, a scoping the paper's own D.4 honestly hints at.

Strengths

• Clean, parameter-free objective. BLL-Loss = $\log(2 - \pi_\theta)$ is a drop-in replacement for CE with no tunable knob and a bounded loss ($\le \log 2$), which makes it appealing on its face and easy to adopt inside existing frameworks.
• Well-motivated diagnosis. The two-stage diagnostic — a token-probability landscape (Fig. 1) followed by a hard-masking experiment (Fig. 2) — is a much stronger motivation than dropping in a new loss and reporting gains. The finding that suppressing the low-probability tail improves reasoning is a genuinely counter-intuitive result that reframes SFT's optimization pressure.
• Ablation isolates the loss from step-size effects. Table 2's Lower-LR / Grad-Clip variants show that BLL-Loss is not simply a proxy for a smaller effective step, which is exactly the ablation a skeptical reader would ask for.
• Convergent theoretical framings. Even after the caveats below, having a series-convergence derivation and a log-odds decomposition end at the same closed form is a nice narrative anchor that gives the objective an interpretation as clipped importance-sampling reward (Table 4) rather than a heuristic.
• Honest failure case. The D.4 knowledge-injection failure at 0 accuracy is refreshingly explicit about where BLL-Loss breaks: when the base model lacks the requisite prior, suppressing low-probability tokens throws away the very signal that would inject new capability.

Weaknesses

• Unification claim rests on unshown algebra. The paper's "theoretical unification" between the series-convergence and log-odds derivations hinges on the exact $\beta$-balanced loss (Eq. 9/10), which is never displayed. Without the parametrised form the substitution $\beta = 0 \Rightarrow \log(2 - \pi_\theta)$ cannot be independently verified, and the alternating-sign construction in Perspective 1 is explicitly imposed rather than derived, so "emerges naturally" is stronger framing than the derivations support.
• Internal contradiction on gradient behaviour. Section 3.2 asserts that the gradient norm "vanishes ($|\nabla| \to 0$)" for high-surprisal tokens, but the Perspective-1 derivation shows the objective is asymptotically linear in $\epsilon = 1 - \pi_\theta$, so $dL/d\epsilon$ saturates at $1/2$ rather than vanishing. The vanishing property lives in logit space (through the softmax Jacobian $\pi(1-\pi)$), which CE also carries; the honest comparison is between modulation factors, not between "vanishing" and "maximal" gradient norms.
• "Gains persist as model size increases" not supported by the 8B row. The Qwen3-8B-Base row of Table 1 shows the smallest BLL $\Delta$ in the whole table (+6.84) and a BLL AIME cell (5.82) actually below the un-tuned Base model (8.55). The dominant pattern at 8B is that SFT/NEFT hurt the base model and BLL avoids that damage, not that BLL's gain scales up.
• "Monotonic peak at $\beta = 0$" oversells a noisy figure. Sensitivity curves across GSM8K, MATH500, Olympiad, Minerva, and AMC are famously high-variance — the paper itself averages 16 runs — yet the text claims strict monotonicity on all five and a peak strictly at the boundary, without confidence bands. Combined with the Figure 3 caption labelling $\beta = 0$ the "baseline" (while the text treats it as the optimum), the story is stronger than the evidence.
• Empirical scope leans on math-pretrained backbones. Six of the eight math backbones (Qwen2.5-Math-1.5B/7B, Qwen3-Base 1.7B/4B/8B) are already math-oriented, which is precisely the regime in which the Investigation-I premise ("the prior already aligns well with reasoning tokens") is easiest to satisfy. This makes the diagnostic-to-objective story partly self-fulfilling and leaves open whether the "stylistic-noise" argument generalises. D.4 honestly hints at this limit but does not resolve it.
• Overstated causal / mechanistic language. Related-Work claims BLL identifies "the mechanistic root cause" of SFT's reasoning limitations. In reality the argument rests on one token-probability plot, one word-cloud analysis (alphabetic tokens only, per D.1's own caveat), and one masking ablation — a correlational + interventional case for a mechanism, not the mechanism.
• Missing controls in the loss-design story. Focal Loss is invoked as the philosophical antithesis of BLL, and label smoothing is cited in Related Work, but neither is included as an empirical baseline. Given how naturally these live alongside BLL as token-reweighting objectives, their absence leaves the space of "simple gradient reshapings" under-mapped.
• Statistical support thin for close comparisons. No per-cell standard errors or confidence intervals accompany Tables 1–8 or Figures 3–4. Several headline gaps are small enough (Overall Avg 35.26 / 35.64 / 35.70 in Table 6; the Qwen3-8B $\Delta$ of +6.84; per-benchmark deltas in Table 5) that per-seed variance could plausibly swallow them.
• Percentage-point / percentage ambiguity in headline gains. Text like "+23.40% over SFT" and "+14.36% gain on BBH" are percentage-point differences, not relative improvements ($23.40/38.40 \approx 60.9\%$). The mixed convention will lead careful readers to under- or overestimate the size of the effect.
• "Consistently outperforms" wording overstates the tables. In Table 6 (Qwen3-4B) BLL-Loss is roughly 5 points behind -p and DFT on Minerva; in the MMLU breakdown Table 7 bolds SFT on multiple subjects (including a 6.4-point regression on one of them). "On average / on most subjects" is defensible; "consistently" is not.
• Threshold characterization confuses mass and count. Section 2.4 describes the masking thresholds as "bottom 5%, 10%, and 15% of the probability mass", but the underlying quantities in Investigation I and the masking rule itself are per-token percentiles — the bottom 15% of tokens carries far less than 15% of the total mass under a heavy-tailed distribution.
• Benchmark descriptions inconsistent with the scoring pipeline. AMC is described as "multiple-choice" and OlympiadBench as requiring "long, complex proofs", but the pipeline scores free-form final answers extracted from \boxed{}. Small clarifications would prevent readers from misinterpreting the reported accuracies.
• Loss name drift. Appendix B calls the objective "Bounded Log-Odds Loss" while the title and main text call it "Bounded Log-Likelihood Loss"; both share the acronym BLL, but the expansion is the definition of the central term of the paper and should be uniform.

Reproducibility & code

The evidence for this section comes from reading (not executing) the released code and data pointers implied by the paper's framework choices.

• Framework references are precise; training-specific artifacts are not released. The paper cites LLaMA-Factory, DeepSpeed ZeRO-3, and the Qwen2.5-Math official evaluation code, but no BLL-specific patch or module (a diff swapping CE for $\log(2 - \pi_\theta)$, including how $\pi_\theta \to 1$ is stabilised in bf16 and how prompt/padding masks compose with the target) is released alongside the paper. This is the single most reproducibility-relevant artifact and its absence forces re-implementers to guess.
• No seeds, subsample manifests, or checkpoint revisions. The 100k NuminaMath subset is described as "randomly subsampled" without a seed or index list, and the exact HuggingFace revisions of the Qwen and Llama backbones are not pinned. Combined with the two-decimal precision of Tables 1, 2, 5, and 6 (where several gaps are under 1 pp), exact reproduction of cell values from the paper alone is not realistic.
• Data-scaling comparison confounded by batch size. Appendix C.1 states that runs with fewer than 100k samples used global batch size 32 rather than 256, so the headline "BLL @ 10k > SFT @ 100k" claim mixes a data-size effect with a step-count effect. A matched-step or matched-batch comparison is missing.
• GEM baseline dropped without explanation on Qwen3 and Llama-3. Table 1's GEM column is empty for Qwen3-1.7B/4B/8B, and Appendix D.2's Llama-3 table omits GEM entirely. Whether GEM was evaluated and dropped, failed to converge, or was skipped is not documented.
• No variance reported for 16-run averaging. The 16-run mean-accuracy protocol is good, but per-cell standard errors are nowhere reported, so the reader cannot tell whether close gaps (particularly in Table 6 and in the Llama-3.2-1B sub-5% regime) are inside the sampling noise.
• OpenCompass evaluation not pinned. For MBPP, HumanEval, MMLU, BBH, and CommonsenseQA the paper defers to "OpenCompass built-in default settings" without pinning the framework revision, the MMLU shot count, the sandbox configuration, or the per-task templates. Given the large reported MBPP and BBH deltas, this is a material gap.
• Underlying values for Figures 2/3/4/6 not tabulated. The threshold-masking curves, the $\beta$-sensitivity plot, the data-scaling curve, and the training-loss dynamics each carry precise claims (monotonicity, cross-over, spike suppression) but ship without a CSV or an appendix table, so the arguments the text makes can currently only be checked visually.
• D.4 failure case underspecified. The "score of 0" on the 40K induced dataset is reported without a metric definition (exact match? F1?), without the CE baseline number on the same task, and without the training configuration used. This makes what is a genuinely useful boundary result hard to reproduce or interpret.
• Notation drift on the probing model. The Fig. 5 caption uses $p_\theta(y \mid x)$ where the main text carefully separates the pre-trained prior $\pi_{\theta_0}$ from the trainable policy $\pi_\theta$; the visualization is over the base model, so $\pi_{\theta_0}$ would be the consistent symbol.

Recommended Changes

Essential

• Display the $\beta$-balanced loss and the $\beta \to 0$ reduction. Render Eq. 9/10 explicitly and carry out the algebra that reduces the balanced loss to $\log(2 - \pi_\theta)$ at $\beta = 0$, so the "theoretical unification" of Perspectives 1 and 2 can be verified line-by-line rather than taken on faith (Weaknesses: unification claim).
• Reconcile the gradient behaviour claim. Rewrite the "$|\nabla| \to 0$" passage to attribute the vanishing to the softmax Jacobian rather than to the loss itself, so it no longer contradicts the "asymptotically linear" description in Perspective 1 (Weaknesses: internal contradiction on gradients).
• Add per-seed variance for all headline tables and the $\beta$ sweep. Report standard errors or bootstrap intervals for Tables 1, 2, 3, 5, 6 and for the curves in Figure 3, so claims like "strictly monotonic peak at $\beta = 0$", "consistently the highest on Overall Avg", and "gains persist as model size increases" become falsifiable (Weaknesses: statistical support thin; monotonic claim; 8B row).
• De-confound the data-scaling comparison. Rerun either SFT @ 100k at batch 32 or BLL @ 10k at batch 256 (or report the step-matched control) so the "10k BLL beats 100k SFT" claim is not mixing data size with optimization budget (Reproducibility: batch-size confound).
• Release the BLL loss implementation and configuration. Publish (i) a diff or module for the BLL loss inside LLaMA-Factory at a pinned commit, including numerical-stability handling near $\pi_\theta \to 1$ and the prompt/padding masking policy, (ii) seeds and subsample manifests for the 100k NuminaMath selection, and (iii) pinned OpenCompass revision, MMLU shot count, and per-task prompt templates (Reproducibility: framework references vs artifacts, seeds, OpenCompass).

Suggested

• Add Focal-Loss and label-smoothing baselines. Both are cited in the paper (Focal Loss explicitly as the philosophical opposite) but never run empirically, leaving the space of simple token-reweightings under-mapped (Weaknesses: missing controls).
• Fix the threshold characterization. Change "bottom 5%, 10%, and 15% of the probability mass" to "bottom 5%, 10%, and 15% of tokens by probability" in Section 2.4 (Weaknesses: threshold characterization).
• Soften over-strong claims to match the evidence. Replace "the mechanistic root cause" with "a specific mechanism", "consistently the highest" with the more accurate "highest on Overall Avg and Competition Math", and "capped at 0.5" with "floored at 0.5" (Weaknesses: overstated causal language; consistently-outperforms wording; capped-at-0.5 direction).
• Report percentage points explicitly. Change "+23.40% over SFT" and similar to "+23.40 pp" or "$\Delta = 23.40$", since the current phrasing invites misreading as relative improvement (Weaknesses: pp / % ambiguity).
• Unify the loss name. Pick either "Bounded Log-Likelihood Loss" or "Bounded Log-Odds Loss" and use it across the title, main text, and appendices; drop the drift in Appendix B.1 and B.3 (Weaknesses: loss name drift).
• Report the GEM baseline for Qwen3 and Llama-3 or explicitly document its omission. Currently GEM appears only for Qwen2.5-Math, and its absence elsewhere silently narrows the baseline comparison (Reproducibility: GEM baseline dropped).
• Extend the diagnostic figures to a genuinely non-math backbone. Re-run Figs. 1 and 5 with, e.g., Llama-3.2-1B-Base on NuminaMath, so the "low-probability tail = stylistic noise" observation is not exclusive to a math-pretrained prior (Weaknesses: empirical scope).
• Report the CE baseline and the metric for the D.4 failure case. Even one number (CE score on the 40K induced dataset with the same training recipe) would let readers tell whether "0" is a BLL-specific failure or a general difficulty (Reproducibility: D.4 underspecified).
• Clarify AMC and OlympiadBench scoring, and the GSM8K split. State that AMC is scored as free-form via \boxed{}, that the OlympiadBench subset used is answer-based (not proof-based), and that GSM8K scores are on the 1,319-problem test split (Weaknesses: benchmark descriptions; GSM8K split).
• Disambiguate the "Pass@1" reporting. State the estimator explicitly as $\Pr[\text{correct} \mid T = 1]$ with 16 samples per problem, and optionally add greedy-decoding numbers as a sanity check (Reproducibility: Pass@1 conflation).
• Fix Table 3 alignment and define "all valid metrics". The current layout smears some values into adjacent columns and the "valid metrics" qualifier is undefined; either define it or drop it (Weaknesses: "all valid metrics").
• Correct notation drift in Fig. 5. Change $p_\theta(y \mid x)$ to $\pi_{\theta_0}(y \mid x)$ so the caption matches the paper's careful $\pi_\theta$ vs $\pi_{\theta_0}$ distinction (Reproducibility: notation drift).