VALUEFLOW: Toward Pluralistic and Steerable Value-based Alignment in Large Language Models

Summary

VALUEFLOW proposes an end-to-end pipeline for value-based alignment of LLMs that combines a hierarchical value embedding space (HIVES) unifying four value theories (Schwartz's Basic Values, Moral Foundations Theory, Ross's prima facie duties, and Vasak's three generations of rights), a Value Intensity Database (VIDB) of ranked, LLM-scored texts, and a ranking-based evaluator that estimates a response's intensity by placing it against VIDB anchors under a Plackett–Luce model. The conceptual move is worthwhile: it reframes value alignment as controlled expression at a graded intensity rather than presence/absence, and it treats reliable intensity measurement as a first-class problem — arguing convincingly (Table 1) that direct LLM rating suffers from cross-model variance and prompt sensitivity, and that pairwise ranking recovers a more stable latent scale. The empirical study is genuinely broad: ten frontier models, four theories, thirty-two values, prompt- and activation-based steering, plus multi-value composition, refusal analysis, and multilingual and Buddhist-ethics extensions. Downstream, HIVES-based profiling improves OpinionQA group prediction on average, giving the framework a concrete personalization use case.

The main conceptual limitations are that the "intensity" being steered collapses to a four-level ordinal target ($-2,-1,+1,+2$) even though VIDB is continuous, so the reported dose–response is inherently coarse; that the judge model (Gemma-3-27B-it) sits inside the evaluated model roster, creating a self-evaluation coupling; and that several headline generalizations (five-value dominance, three-regime steerability clustering, Profile > Default on OpinionQA) are asserted more strongly than the presented evidence — a small permutation subset, no uncertainty bands, and per-attribute regressions in Table 3 — supports. Reproducibility is also uneven: the OpinionQA Default/Profile pipeline is largely re-derivable from the release, but VIDB, HIVES weights, the rating baselines that anchor Section 3.2, and every extension experiment are missing from the shipped code.

Strengths

• Conceptual contribution. Reframing steerability as calibrated intensity control along value axes — rather than presence/absence or direction alone — is a genuinely new angle on pluralistic alignment, and separating the extraction/steering/evaluation stack cleanly is a useful organizing move.
• Motivating diagnosis for ranking over rating. The instability analysis in Section 3.2 and Table 1 (variance 12.6 → 2.1, sign-flip 48% → 29%, plus ValueNet gains) is a compact, well-designed case for why direct scalar rating by LLM judges is inadequate for intensity work, and the ranking-based alternative follows naturally.
• Breadth of empirical study. Ten frontier models × 32 values × two prompt regimes × four theories, plus multi-value composition, activation-vector baselines, refusal, multi-turn, multilingual, and a non-Western value system — the coverage of the audit is unusually thorough for this literature.
• Head-line human-evaluation numbers. Where human annotations are directly compared (VIDB score reliability MAD = 1.4, pairwise consistency 85.3%, $\pm$1-window accuracy 86.7%), the ranking-based evaluator visibly beats plausible rating baselines — a stronger validation than most value-evaluation papers offer.
• Practical downstream demonstration. Profile-based demographic steering on OpinionQA improves accuracy against both Default and Modular Pluralism on most cells and reaches double-digit gains on some (e.g., Phi-4 Religion 44.5% → 57.4%), grounding the framework in a concrete personalization use case.
• Interpretability of the extension protocol. Appendix F sketches a plausible, lightweight recipe for extending the pipeline to non-English corpora and non-Western value systems, which supports the paper's pluralism framing beyond WEIRD samples.

Weaknesses

• Judge–model coupling. Gemma-3-27B-it is chosen as the default judge because it has the least position bias (Appendix C.3), yet it is also one of the ten models whose steerability is measured, and it ends up in the 'strongly steerable' regime. Section 6 does not report a leave-one-out or alternative-judge check on Gemma-3's numbers, so the reader cannot separate genuine steerability from self-consistent scoring.
• Under-powered inference and no uncertainty bands. The regime clusters (weakly / moderately / strongly steerable), the per-value regimes (hard-to-steer / polarity-asymmetric / bi-directional), and the compositional 'dominance' law are drawn from 500 prompts spread across 32 values × 4 intensities with no error bars, no per-seed variance, and only qualitative visual separation on the shift figures. The three-regime taxonomy is the empirical core of the paper and needs quantified uncertainty.
• Only five of 60 possible five-value permutations tested. The compositional-dominance claim ('the +2 target dominates') is generalized from 5 hand-picked permutations of the multiset $\{2,1,1,-1,-2\}$ out of 60 arrangements. The selection procedure is not described, so a reader cannot tell whether the pattern was observed or engineered.
• OpinionQA description overstates the pattern. Table 3 shows Profile can lose to Default on several attribute × model cells (Phi-4 Region $-3.1$, GLM-4 Region $-3.8$, GLM-4 Education $-2.6$, and others). The text summary 'profile-based steering consistently improves accuracy over both baselines across most dimensions' understates these regressions.
• 'Ambiguous pair' interpretation runs against the direction of the reported gap. Inconsistent (disagreement) cases show a larger predicted intensity gap than consistent cases (6.44 vs 2.80). Standard usage associates 'ambiguous' with small gaps; the sign of the statistic argues instead that the evaluator over-separates precisely where humans disagree.
• Windowed-fidelity numbers appear internally inconsistent. Given 60.8% exact match, 86.7% $\pm 1$ accuracy, and $\ge 2$-window deviation for the remaining 13.3%, the mean absolute deviation is at least 0.525, larger than the reported 0.46 — unless 'mean deviation' is signed, which would make it a poor accuracy proxy.
• Coarse operational intensity vs continuous framing. The paper's headline is calibrated intensity control, but every steerability experiment uses a four-level ordinal target ($-2,-1,+1,+2$) despite VIDB living on a $[-10,10]$ scale, and the evaluator's asymmetric clamp (Algorithm 4 allows the raw score just below the minimum anchor but caps it at the maximum) can systematically compress reported $+\Delta$ relative to $-\Delta$.
• Central losses and metric formulas are missing from the manuscript. The Stage 1 hierarchical contrastive loss ('The objective is:'), the Stage 2 'weighted sum', and the correctness predicate / label-affinity target / directional-vector definition in the HIVES metric section all have no equation printed. The two-stage weighting coefficients are only introduced numerically in an appendix. This affects understandability and reproducibility even after code release.
• Failed steers presented alongside successful ones without commentary. Boxes 9 and 10 include $-2$ steers that make prosocial choices (Qwen3 donates, gpt-oss opens with 'You'd put the charity first') and Claude-4 refusing to infer values at all. If these are counted as steered outputs when aggregating $\Delta$, they inflate the reported control; the paper does not report an alternative aggregation that excludes such failures.
• Terminology overloading and priority claim. 'Intensity' is used both for the categorical stance labels at the seed stage ($-1,+1$) and for the continuous PL-derived quantity. The framing 'the first framework that unifies extraction, evaluation, and steering' is stronger than the related-work paragraph, which credits each component to prior work, warrants.
• Minor citation/technical issues. Ross's prima facie duties are from 1930 (The Right and the Good), not 1939; ValueEval/Valuenet are called PVQ datasets in Section B.1 but SVT in Table 5; direction back-off is undefined for the single-level Duties hierarchy; Stage 1/2 precision drops from bfloat16 to float16 without justification; Table 13 shows Conformity's Orig.$(-)$=+4.7 (wrong sign for negative steering) without discussion.

Reproducibility & code

(veritas mode: code was read, not executed. Coverage below reflects what the released repository does and does not include; no numbers were re-derived.)

• VIDB is not shipped. The evaluator (evaluation/evaluate_response_intensity.py:estimate_scores) is present and implements Plackett–Luce with fixed anchors plus isotonic calibration, but it consumes anchor CSVs (data/final_ratings/*_flag_aggregated.csv) that are not in the release; the HuggingFace VIDB link is marked TODO. Because every steerability score, the 1.4 MAD, the 60–79% win rates, and every per-value shift in Section 6/7 flow through VIDB, the release cannot regenerate any of the headline evaluation numbers as-is.
• Rating baselines have no implementation. Table 1 (rating vs ranking instability and ValueNet agreement) and Table 4 (Ours vs four rating-based LLMs) require a rating-based scoring harness for Qwen3-32B / Phi-4 / Gemma-3-27B / Mistral-3.1, a prompt-paraphrase generator, and ValueNet ground truth. None of these are present. This is the entire empirical motivation for the ranking-based approach, so the gap is load-bearing.
• HIVES artifacts. Stage-1/Stage-2 training scripts are in the release but the training CSVs, the trained HIVES-2 checkpoint, the UniVar baseline, the orthogonality metric, and the Figure 5 plotting code are not, so the HIVES-vs-baseline comparison is not re-derivable end-to-end. Cross-theory anchors (274) and the 158/142/107 user-friendly instance inventory are described in prose but no anchor-construction pipeline or artifact is present; Stage 2 training consumes a selected_anchor column with no upstream generator.
• OpinionQA (Table 3) is the most reproducible headline. scripts/run_opinionqa_final.sh and profling/generate_response_opinionqa.py implement Default vs Profile for Qwen3-32B / Phi-4 / GLM-4 with matching group configuration, and 22 group profile JSONs are shipped. However, the Modular Pluralism baseline is not reimplemented; construct_value_profile.py assumes upstream per-attribute intensity CSVs that are absent (so profiles cannot be regenerated); and the paired $t$-test ($t\approx 16{-}20$, $p<10^{-60}$) and temperature sweep $T\in[0.0,0.8]$ have no code — the shipped driver hard-codes $T=0$.
• Human evaluation data (Table 4, Table 15) missing. The 2K scalar ratings and 1.5K pairwise/windowed judgments from 25 evaluators, plus the aggregation code for MAD, win rate, pairwise consistency, and windowed accuracy, are not in the release. These are the strongest external checks on VIDB and cannot be recomputed.
• 10-model orchestration and per-regime classification not scripted. Every closed-API model in Figure 4 (Grok-4, Claude-4-Sonnet, GPT-4.1, Gemini-2.5-Flash, gpt-oss) is commented out in the shipped generation script, and the aggregation code that produces the three-regime classification, the per-value curves, and the intensity-anchor vs user-text comparison is not present.
• Auxiliary studies without shipped code. Persona-vector and embedding-injection kernel baselines (Appendix D.5), the 100-turn multi-turn analysis (Appendix D.10), the AR/ZH/KO multilingual extension (Appendix F), the Buddhist-ethics extension, the Sorry-Bench refusal analysis (Appendix D.6), the position-bias judge study (Figure 19), and the PL/BT/Thurstone convergence comparison (Figure 36) all lack released code or data.
• Underspecified counts. The k-means step ($k\approx 500$) to the final 274 anchors involves a ~45% drop that is only explained by 'fewer than five exemplars' plus dedup, with no per-step counts.

Recommended Changes

Essential

• Release VIDB. Publish the 32-value anchor set (or at minimum released text IDs plus scores) so the evaluator, the 1.4 MAD, the 60–79% win rates, and every per-value intensity in Section 6/7 can be re-derived. Resolve the TODO HuggingFace link.
• Ship the rating-baseline harness and ValueNet split. Add the rating-based LLM scoring scripts, the paraphrase set, and the ValueNet ground truth used for Table 1 and Table 4, so the ranking-vs-rating motivational claim (and the head-to-head win rates) can be re-derived from code.
• Restore the missing loss and metric equations. Typeset the Stage 1 hierarchical contrastive loss, the Stage 2 InfoNCE weighted sum with named coefficients, the pairwise-ranking correctness predicate, the label-affinity target, and the directional-vector formula; state the boundary condition for anchors with no positives at a given level.
• Decouple the judge from the model roster. Report a leave-one-out or alternative-judge check for Gemma-3-27B-it's steerability numbers, or exclude Gemma-3 from the steered set. The current setup lets one model rank its own outputs.
• Report uncertainty on the steerability figures. Add per-seed or bootstrap error bars to Figure 4, Figure 6, and Figure 8 so the three-regime clustering and per-value regimes can be evaluated as more than qualitative visual patterns.
• Correct the OpinionQA summary. Report signed per-cell deltas for Table 3 and update the text to acknowledge that Profile can lose to Default on some attribute × model cells (e.g., Phi-4 Region, GLM-4 Region/Education), rather than 'consistently improves'.
• Expand the five-value study. Either sample a substantially larger fraction of the 60 permutations of $\{2,1,1,-1,-2\}$ or explicitly document the selection procedure for the five shown; recast the 'consistent pattern' language accordingly.
• Fix the windowed-fidelity arithmetic. Reconcile 60.8% exact match, 86.7% $\pm 1$ accuracy, and the reported 0.46 mean deviation; define whether the deviation is signed or absolute.
• Release the OpinionQA Modular Pluralism baseline and profile-construction upstream inputs. Add the missing per-attribute intensity CSVs so profiles can be regenerated from scratch, and reimplement the Modular Pluralism comparison; expose the paired $t$-test and temperature sweep code.
• Release the human evaluation data and aggregation code. Ship (de-identified) 2K scalar + 1.5K pairwise/windowed judgments plus the scripts that produce Table 4 and Table 15.

Suggested

• Reinterpret 'ambiguous pairs'. Either provide supporting evidence that disagreement pairs are genuinely ambiguous (e.g., human-side gap, human agreement rate) or rephrase to acknowledge the evaluator over-separates disagreement pairs.
• Mirror the clamp for above-anchor extremes. Add a symmetric branch in Algorithm 4 so responses ranked above every co-occurring anchor can score just above $s_{\max}$, matching the treatment of below-anchor extremes.
• Ship the auxiliary experiment code. Add drivers for the multi-value composition (Table 13, Figure 8), persona-vector and kernel-injection baselines (Appendix D.5), the multi-turn dialogue analysis (Figure 37), multilingual and Buddhist-ethics VIDBs and prompt sets (Appendix F), the Sorry-Bench refusal harness (Appendix D.6), the position-bias judge study (Figure 19), and the PL/BT/Thurstone convergence comparison (Figure 36).
• Reconcile terminology. Rename the seed-stage $\{-1,+1\}$ stance labels so they do not share the word 'intensity' with the continuous PL target; align the SVT-vs-PVQ labelling between Section B.1 and Table 5.
• Soften priority claims. Rephrase 'the first framework that unifies extraction, evaluation, and steering' to a to-our-knowledge claim, and briefly contrast VALUEFLOW with the closest unified prior systems (e.g., FULCRA + Kaleido).
• Discuss steered-response failure modes. Annotate Boxes 9–10 with which examples the authors regard as successful vs failed steers, and report an aggregation that excludes semantic failures to show robustness of the $\Delta$ statistics.
• Fix citation and small technical issues. Update the Ross date to 1930, resolve the direction back-off for single-level Duties, justify or correct the bfloat16 → float16 precision drop between stages, and either correct or explain the Conformity Orig.$(-)$=+4.7 anomaly in Table 13.
• Quantify VIDB macro-value overlap. Report how much overlap the macro-value operational choice introduces (e.g., cross-value score correlation) so readers can gauge how much of the reported intensity variance is genuine.
• Report the 500-to-274 anchor reduction in detail. Give per-step counts for the k-means → filter → dedup → human-review pipeline so the 274 number is reproducible.