Reward-free Alignment for Conflicting Objectives

Summary

The paper proposes RACO, a reward-free multi-objective preference alignment method that keeps the DPO-style offline setup but replaces the naive weighted-loss update with a clipped variant of conflict-averse gradient descent. The conceptual move is to treat multi-objective LLM alignment as a gradient-conflict problem rather than a scalarization problem: per-objective preference tuples define per-objective DPO losses; CAGrad supplies a scalable Pareto-directional correction; and a new elementwise cap $\tilde p_i \leftarrow \min\{p_i, w_i\}$ prevents CAGrad from over-weighting an objective beyond its user-specified importance. Two theorems give a first-order convergence guarantee to a Pareto-critical point of the user-weighted loss (Theorem 3.1) and, for two objectives, a strictly stronger one-step decrease for the clipped update whenever clipping is active (Theorem 3.2). Empirically the paper studies Reddit summary (quality–conciseness and quality–faithfulness) and BeaverTails safety on Qwen3-4B, Llama3-8B, and Gemma3-4B (Base and Instruct), comparing against DPO-LW and AMoPO on Pareto frontiers and a GPT-5.1 win-rate table.

The framing — that gradient conflict, not the choice of loss family, is the real obstacle to multi-objective alignment — is well motivated, and the diagnosis in Figure 5 that vanilla CAGrad assigns a large correction weight to the less-preferred objective is a genuine and clarifying observation. The core technical contribution (the cap and its convergence analysis) is a small but substantive step beyond Liu et al. (2021). The main conceptual limitation is scope: every experiment operates at $m = 2$, so the framework is well tested on the two-objective slice the acceleration theorem covers but never exercised where the cap's sub-simplex behavior would matter most. The manuscript in its current form also has serious presentation defects — several central displays are missing or garbled (the per-objective loss $L_i$, the Theorem 3.1 rate chain, the Theorem 3.2 statement, Lemma B.3, the Appendix B.1 closed form) — and the strictness condition of Theorem 3.2 as printed contradicts the surrounding text.

Executing the released code partially strengthens and partially qualifies these judgments. The dataset construction reproduces almost exactly (92,858 / 59.5%), and on the one clean case that was runnable (Qwen3-4B-Instruct, quality–conciseness, $w_{\text{qual}} = 0.8$) RACO's endpoint strictly dominates both DPO-LW endpoints on both axes, consistent with the paper's improves both objectives language. But the single reproducible quality–faithfulness point runs against the dominates the endpoints claim (RACO trades higher quality for substantially lower faithfulness), the BeaverTails pipeline is not runnable from the release, Table 2 could not be reproduced (hosted judge quota + missing checkpoints), and no plotting scripts ship. The method appears sound; the manuscript needs presentation repairs and the release needs scaffolding.

Strengths

• Conceptual contribution. Reframing multi-objective preference alignment as a gradient-conflict problem, and identifying that a linear-aggregation baseline like DPO-LW cannot in general find a direction improving all objectives simultaneously, is a clean and useful reframing of the multi-objective RLHF literature.
• Diagnostic ablation. Figure 5's decomposition of $p_1, p_2$ against $m_1, m_2$ concretely shows vanilla CAGrad shifting mass toward the less-preferred objective and the cap constraining it — a rare case where an ablation actually visualises the mechanism it claims. The mechanism reproduces in the veritas run: the vanilla-CAGrad correction weight on the down-weighted objective ($p_2 = 0.25$) is capped to $0.10$ under RACO, matching the story.
• Theoretical scaffolding. Establishing that the clipped update still converges to a Pareto-critical point of a user-weighted objective (as opposed to CAGrad's averaged objective) is a modest but meaningful refinement of Liu et al. (2021), and the two-objective one-step-decrease sharpening is a nice complementary result.
• Empirical breadth by task and family. Three trade-offs (quality–conciseness, quality–faithfulness, harmlessness–helpfulness) across two model families per task, on both Base and Instruct checkpoints, is a reasonable multi-condition demonstration.
• Honest ablation on the correction radius $c$. The paper acknowledges that $c$ is set differently across model families and shows the trajectories are only mildly sensitive within a range, which is more informative than a single hero setting.
• Reasonable safety-content handling. The redaction policy, the explicit content warnings, and the appendix placement of sensitive case studies are proportionate; the checkpoint-release policy (safe subset only) is prudent even if it has reproducibility costs.
• Dataset statistic verifiable. The reported preprocessed size (92,858) and the 60% fully conflicting fraction both reproduce (train + val = 92,858; 59.5% of pairs have opposed quality / conciseness winners) once the correct conversion pipeline is used, giving the reader an unambiguous anchor for §4.1.

Weaknesses

• Missing display equations obstruct verification. The per-objective DPO-style loss $L_i$ (promised by a colon following the corresponding DPO-style losses under objective $i$:) never appears; the Theorem 3.1 rate chain is garbled and cannot be reconstructed with confidence; Theorem 3.2's Define and Then, for each t, blocks are empty; Lemma B.3 cuts off before its descent inequality; the Appendix B.1 closed form has unpopulated quadratic coefficients; and reduces to the original CAGrad if we choose cliff-hangs. These are load-bearing displays.
• Strictness-condition sign error in Theorem 3.2. The theorem prints p^{(t)} = w as one of the strictness conditions, but the elementwise cap is a no-op exactly when $p^{(t)} \le w$ componentwise, so clipping is active means $p^{(t)} \ne w$. The very next sentence asserts strict improvement whenever clipping is active, so as printed the theorem is a self-contradiction and the same sign propagates through Lemma B.5 and Theorem B.6.
• Prose vs. algorithm mismatch on clipping. The prose describes the coefficient being clipped to its corresponding preference weight $w_i$, which reads as a hard assignment $p_i := w_i$. Algorithm 1 line 6 is an elementwise cap $\tilde p_i \leftarrow \min\{p_i, w_i\}$. These are different operations; a reader implementing from the prose would silently apply the wrong one.
• Sub-simplex mixture is not acknowledged. After clipping, $\tilde p$ sums to strictly less than 1 whenever any $p_i > w_i$ (i.e. whenever clipping is active), so $\tilde G_p$ is a scaled, not convex, mixture. The subsequent normalization in line 8 rescales, but the framing does not upweight any objective beyond its assigned importance hides that the total correction mass has shrunk.
• Misplaced gradient-clipping citations. Merity et al. (2018), Gehring et al. (2017) and Peters et al. (2018) are all about gradient-norm clipping, not an elementwise cap on convex-combination coefficients. Invoking that literature invites the reader to expect stabilization guarantees the paper does not provide.
• $\rho_t$ range inconsistent with its definition. Lemma B.3 defines $\rho_t := \langle g_0^{(t)}, u_t\rangle$ and asserts $\rho_t \in [-1, 1]$, but Cauchy–Schwarz gives $|\rho_t| \le \|g_0^{(t)}\|$. The intended object is the normalized alignment $\langle g_0^{(t)}/\|g_0^{(t)}\|, u_t\rangle$; the missing normalization should be stated explicitly for the $1 + c^2 + 2c\rho_t \ge 0$ bound to be internally consistent.
• Length-confounded margins. The margins $m_{\text{qual}}, m_{\text{conc}}$ use raw sequence log-probabilities. Sequence log-probs are length-dependent and here one objective is by construction length; any shift toward shorter outputs mechanically raises $m_{\text{conc}}$ irrespective of preference behavior. This confound is not acknowledged when Figure 2 is interpreted as improves both objectives.
• Conciseness label is a length shortcut. Because shorter wins is a deterministic function of the pair, the 60% fully conflicting statistic is essentially about how often the quality-preferred summary happens to be longer, not evidence that the underlying preferences are conflicted. The second objective carries no independent human signal.
• Empirical scope only $m = 2$. Every reported experiment uses two objectives; the acceleration theorem is also stated only for $m = 2$. The framework is claimed to handle general $m$, but no experiment exercises $m \ge 3$ — where the cap's sub-simplex behavior would matter most and where several coordinates can be simultaneously capped.
• Multiple LLM families oversold. The abstract advertises Qwen 3, Llama 3, and Gemma 3, but no task uses all three: summarization uses Qwen3 and Llama3, safety uses Qwen3 and Gemma3. The scope statement should be qualified.
• Universal-dominance claim overstated. In all cases, RACO attains the outermost frontier is a strict-dominance-at-every-swept-weight claim; the ablation itself concedes that clipping barely triggers at balanced weights, so uniform dominance across the sweep is inconsistent with the paper's own qualitative story.
• Narrow baseline set with a partly circular exclusion. Only DPO-LW and AMoPO are compared, and the exclusion of the other offline / conditioned-policy methods discussed in the related work rests on not fitting our taxonomy. Adding at least one steerable/conditioned baseline would keep the handles conflicts uniqueness of Table 1 from being definitionally reserved.
• Single-run, no variance. Every reported number is one training run; there is no seed sweep, no error bar, and no statistical test on the win-rate differences in Table 2. Several endpoints in Fig. 2c / Fig. 3 are visibly close between methods, and several Table 2 entries are close to 50%.
• Compute overhead of the inner CAGrad problem not reported. RACO requires per-objective backward passes and an inner correction solve each step, whereas DPO-LW is a single weighted backward. The comparison is presented as-is with no wall-clock or per-step timing — the reader cannot see whether the observed frontier gains come at a small or large training-time cost.
• Win-rate reasoning reads as a non-sequitur. The paragraph explains RACO's more moderate WR at higher harmlessness weight by appealing to a harmlessness-priority judge — the naive reading predicts the opposite. The intended argument (baselines maximize harmlessness by sacrificing helpfulness, which the judge sometimes still penalizes) should be spelled out.
• Editorial artifacts in Limitations. The paragraph contains we will state this explicitly and We will add these points explicitly to the limitations section — first-person promises to revise left in the manuscript alongside the statements themselves.
• Base-model list conflates Instruct and Base checkpoints. The footnote list introduced as base models mixes instruction-tuned checkpoints (Qwen3-4B-Instruct-2507, gemma-3-4B-it, Llama3.1-8B-Instruct) with pretrained bases (Qwen3-4B-Base, gemma-3-4B-pt) — a distinction the paper otherwise draws carefully.
• Table 2 rendering is unusable for verification. Header cells are broken across columns (Gemma3-4B-Instruct split into G and emma3-4B-Instruct; DPO LW AMoPO D as a single cell), and the ordering of the six numbers per column (weights × baselines) is not annotated.

Reproducibility & code

The released repository (training loop in trl/scripts/train_raco.py and trl/trl/trainer/dpo_trainer.py; sweep driver in script/run_and_eval.sh; scoring in eval/new_score.py and eval/new_score_beaver.py) implements the RACO / CAGrad-Clip algorithm end-to-end and exposes the toggle via --raco_clip_lambda alongside DPO-LW and AMoPO flags. On the veritas evidence, the code does run and the algorithmic behavior it exhibits is consistent with the paper's story, but the coverage of the manuscript's headline claims by shipped artifacts is only partial and the release is currently runnable-but-fragile rather than turnkey.

• Dataset construction reproduces. The preprocessed 2-objective Reddit dataset totals 92,858 (train 90,858 + val 2,000) and 59.5% of pairs are fully conflicting, matching the paper (92,858 and 60%). This is the single unambiguous quantitative match.
• Quality–conciseness endpoint reproduces the improves both objectives behavior on the one measured case. For Qwen3-4B-Instruct at $w_{\text{qual}} = 0.8$, RACO (quality = 1.458, length = 93.5 words) Pareto-dominates both DPO-LW points on both axes, consistent with the paper's language.
• Quality–faithfulness does not exhibit the dominates the endpoints behavior. The single measured point (Qwen3-4B-Instruct, $w_{\text{qual}} = 0.8$) has RACO at quality = 1.538 / faithfulness = 0.395 versus DPO-LW at quality = 1.382 / faithfulness = 0.626 — a genuine trade-off, not a domination. This qualifies the Figure 3 narrative.
• Table 2 win rates not attempted. The BeaverTails LLM-as-judge (GPT-5.1) evaluation exited with an HTTP 429 quota error, and it also depended on BeaverTails checkpoints from a step that was never built. No cell of Table 2 could be reproduced.
• BeaverTails path is not runnable from the release. No PKU-SafeRLHF / BeaverTails preference-data preparation script is shipped (only the Reddit conversion scripts are). The base-model runs in Figure 4 additionally require an SFT pass on the BeaverTails Q&A subset that exists only as a prose description in Appendix C.3 — no SFT script, template, or config is shipped.
• No shipped script produces the paper's figures or Table 2. Per-run margins are logged but Figures 2(a-c), 3, 4, 5, 6(a-b) and the Table 2 win-rate matrix must be hand-plotted. Not blocking, but it materially raises the reproduction friction and prevents structural verification of the frontier figures.
• Checkpoint release excludes exactly the cases that motivate Examples 3–4. The safe subset only policy is prudent, but the qualitative case studies depend on AMoPO / DPO-LW checkpoints at $w_{\text{help}} = 0.8$ producing unsafe outputs (explicit-slang glossary; sabotage endorsement) — exactly the outputs a safe-subset filter would remove. The release should say which (method, weight, model) triples are shipped.
• DPO $\beta$ silently defaults to TRL's 0.1, not the paper's setting. train_raco.py neither exposes nor sets beta, so training inherits TRL's default $\beta = 0.1$ rather than the paper's $\beta = 0.2$ (and 0.4 for AMoPO). Reproducing the paper's numbers therefore requires patching the trainer to add --beta — a genuine fidelity gap.
• README's dataset-loading recipe is wrong. The documented load_dataset call references a nonexistent validation split and TRL-schema columns the dataset does not carry, and omits a required conversion step (convert_moa_jsonl_to_trl_raco.py). The correct pipeline exists in the repo, but a naive follower would fail.
• No fixed seed / no seed sweep. Neither run_and_eval.sh nor train_raco.py pin a random seed. Trajectories in Fig. 2(a-b), 5, 6(b) and endpoints across the sweep are inherently non-deterministic, and no per-seed variance is reported.
• Hosted-API judge is a drift and quota risk. Table 2 depends on GPT-5.1 through a hosted endpoint; without a pinned snapshot ID the numbers will drift as OpenAI rolls the alias forward, and without funded API access an independent reproducer cannot regenerate the table at all.
• Remark B.7 has no standalone reproducer. The tabular-softmax toy that motivates the acceleration claim ($\rho = -0.46$ vs. $\tilde\rho = 0.98$; 100-iteration loss table) is fully specified in prose but not exposed as a runnable script. The $m = 2$ solver in dpo_trainer.py is not callable outside the training loop.

Recommended Changes

Essential

• Restore all missing display equations. Typeset the per-objective loss $L_i$ (after the corresponding DPO-style losses under objective $i$:), the Theorem 3.1 rate chain, the Define / Then, for each t, blocks in Theorem 3.2, the descent inequality in Lemma B.3, the quadratic coefficients $q_0, q_1, q_2$ in Appendix B.1, and the if we choose reduction line before Theorem B.4. Without these, key claims (per-objective loss form, headline rate, acceleration inequality, closed-form solver, reduction to Liu et al. 2021) cannot be verified.
• Fix the strictness condition in Theorem 3.2. Replace p^{(t)} = w with p^{(t)} \ne w (or the equivalent at least one coordinate is strictly capped) so the condition matches whenever clipping is active; propagate the same fix through Lemma B.5 and Theorem B.6.
• Reconcile the clipping prose with Algorithm 1. Rewrite the sentence about clipping each coefficient to its corresponding preference weight to describe an elementwise cap $\tilde p_i \leftarrow \min\{p_i, w_i\}$, and add a one-line acknowledgement that $\tilde p$ no longer sums to 1 and how line-8 normalization handles that.
• State $\rho_t$ as a normalized alignment. Redefine $\rho_t := \langle g_0^{(t)} / \|g_0^{(t)}\|, u_t\rangle$ so the range $[-1, 1]$ and the descent bound $1 + c^2 + 2c\rho_t \ge 0$ are consistent.
• Address the length confound in the margins. Either normalize log-probabilities by response length, use the DPO log-ratio $\log \pi_\theta - \log \pi_{\mathrm{ref}}$, or report a length-controlled variant of $m_{\text{qual}}, m_{\text{conc}}$ — otherwise the Figure 2 evidence for improves both objectives is partially mechanical.
• Report seed variance for headline plots and Table 2. Rerun at least three seeds for the two-model quality-conciseness sweep and the four-configuration safety sweep, and report shaded confidence intervals on the margins and standard errors on the win rates.
• Ship the BeaverTails pipeline the paper depends on. Add a PKU-SafeRLHF → RACO two-objective preference-conversion script, and the SFT script / config / template used for the base-model runs and the PA baseline, so the entire safety-alignment path is reproducible from the release.
• Expose the DPO --beta flag in train_raco.py and document the paper's per-method values. Currently the code silently uses TRL's $\beta = 0.1$ default instead of the paper's setting; state the values used per (method, task).

Suggested

• Ship plotting / aggregation scripts for Figures 2, 3, 4, 5, 6 and Table 2 so a reproducer can regenerate the paper's figures directly from the logged metrics.
• Add an $m \ge 3$ ablation on Reddit summary (quality / conciseness / faithfulness — the code path is already implemented) so the generality claim beyond two objectives is empirically supported.
• Soften the quality–faithfulness endpoint dominance language in the paragraph around Figure 3, reporting numerical Pareto coordinates and acknowledging cases where the frontier is a trade-off rather than a domination (as the reproduced point at $w_{\text{qual}} = 0.8$ shows).
• Add at least one steerable / conditioned-policy baseline (e.g. Wang et al. 2024a, Yang et al. 2024, Guo et al. 2024) so the handles conflicts uniqueness of Table 1 is empirically anchored rather than definitionally reserved.
• Report training-time overhead relative to DPO-LW / AMoPO so the frontier improvements can be read against their computational cost.
• Replace or drop the gradient-norm clipping citations (Merity et al. 2018; Gehring et al. 2017; Peters et al. 2018) in the sentence introducing the coefficient cap, since they refer to a fundamentally different operation.
• Qualify the model-family scope statement. Rewrite the abstract's multiple LLM families (Qwen 3, Llama 3, Gemma 3) to note that families are alternated across tasks rather than jointly evaluated per task.
• Soften the universal-dominance phrasing. Replace In all cases, RACO attains the outermost frontier with a statement about the envelope of swept weights, most pronounced at the extremes.
• Rewrite the win-rate explanation. Explicitly separate the judge's harmlessness priority from RACO's helpfulness gain at high $w_{\text{harm}}$, so the more moderate WR outcome does not read as a non-sequitur.
• Publish the AMoPO and DPO-LW checkpoints used in Examples 3–4 or explicitly acknowledge which are withheld under the safe-subset policy, so the qualitative case studies are independently verifiable.
• Pin the GPT-5.1 API model snapshot (or add an offline judge alternative such as the Beaver reward + cost combination) so Table 2 can be recomputed identically or at least locally.
• Fix the README dataset-loading recipe to reflect the actual conversion script and document the tie-breaking rule for the conciseness label when $|y^a| = |y^b|$.
• Ship a standalone Remark B.7 reproducer that computes the (Initial / GD / CAGrad / CAGrad-Clip) loss table and the $\rho, \tilde\rho$ values from the specified inputs, so the illustrative example is independently checkable.
• Retitle the base models footnote list as, e.g., model checkpoints used across all experiments so Instruct vs. Base is preserved.
• Clean up the Limitations paragraph by executing the promises: state the limitation directly and remove sentences of the form we will state this explicitly.
• Re-render Table 2 with an unambiguous $(w_{\text{help}}, \text{baseline})$ key per cell and a fixed header row that does not split across columns.