Less is Enough: Synthesizing Diverse Data in Feature Space of LLMs

Summary

The paper introduces Feature Activation Coverage (FAC), a data-diversity metric defined in the sparse-autoencoder (SAE) feature space of a target LLM, and FAC Synthesis, a coverage-guided pipeline that (i) trains a Top-K SAE on residual-stream activations, (ii) identifies task-relevant SAE features via an LLM-annotator rubric, (iii) enumerates "missing" features that the current dataset fails to activate, and (iv) generates contrastive-pair prompts that reliably re-activate those missing features. The conceptual move is worth taking seriously: generic text-space diversity metrics (Distinct-n, embedding entropy, gradient-space coverage) do not track downstream utility, whereas a metric grounded in a shared, interpretable representation space might. The argument is backed by a decomposition of generalization error into a distribution-gap term (reduced by SAE-feature coverage) and a sampling-error term (reduced by contrastive two-step generation), and by four downstream tasks (Toxicity Detection, Reward Modeling, Behavior Steering, Instruction Following) where FAC-guided synthesis reportedly matches MAGPIE at 1/150 the data budget on AlpacaEval 2.0. Cross-model transfer and self-improvement variants extend the story.

The main conceptual limitation is that the mechanism claim — that missing-feature coverage causes the gains — is not cleanly isolated from two confounds. First, the task-relevant feature set is defined by an LLM rubric whose criteria mirror the downstream evaluation, making the FAC–performance correlation partially self-referential. Second, Table 12 shows random SAE-feature selection already outperforms informed dense-cluster coverage, suggesting a large share of the benefit comes from the choice of representation space, not from targeting missing features. Downstream evidence is also uneven: the flagship $r = 0.95$ is single-task; behavior-steering conclusions rest on three synthesised examples per behavior; GSM8K / LiveCodeBench improvements are noise-band. The theoretical framing is thoughtful but leaves gaps — most visibly a default $\delta = 0$ that contradicts the $\delta > 0$ regime the derivations require, and a FAC definition that is only equivalent to "fraction of relevant features covered" under an unstated containment condition. An end-to-end execution attempt confirms two-step > one-step FAC directionally and reproduces the toxicity "Ours" cell within tolerance, but the four-task Table 1, the near-monotonic diversity-metric contrast, and the 150× data-efficiency headline could not be independently verified. Overall the direction is promising, but the empirical case for the specific causal claim needs sharpening.

Strengths

• Conceptual contribution. Grounding data diversity in a model's SAE feature space, rather than in surface text or generic embedding space, is a principled move; the derivation cleanly separates the distribution-gap term (Section 5) from the sampling-error term (Section 6), and motivates the contrastive two-step generation as a way to reduce conditional entropy of the synthesised set.
• Empirical breadth. Four downstream tasks spanning classification (Toxicity Detection), preference modeling (Reward Modeling), steering (Sycophancy / Survival Instinct), and open-ended generation (AlpacaEval 2.0) is a wider coverage than most synthetic-data papers attempt, and the observed FAC–performance monotonicity (Figure 4, Table 9) is a genuinely useful correlational finding.
• Interpretability of the intermediate objects. The qualitative feature tables (Tables 16–19) show the annotator can identify semantically coherent features (theft, cheating, sycophantic phrasings), and the annotation pipeline is auditable at feature granularity. An independent end-to-end execution confirms that GPT-4o-mini produces coherent per-feature summaries with correct task-relevance labels, so this transparency is real.
• Ablation coverage. The paper reports sensitivity to SAE dictionary size and Top-K (Table 7), decoding temperature (Table 4), activation threshold (Figure 6), samples-per-feature (Figure 7), and continuous vs binary FAC variants (Table 11). Even where individual results are noisy, the overall envelope of ablations is honest about limitations.
• Two-step > one-step is a robust effect. An independent reproduction confirms that two-step synthesis activates strictly more SAE features than one-step at every activation threshold on toxicity — the direction survives even when the synthesis algorithm has to be re-derived from prose.
• Cross-model transfer setup. Reusing synthetic data across LLaMA / Mistral / Qwen backbones (Section 7.4) is a genuinely useful practical result — every backbone gained AUPRC in independent reproduction — even if the underlying "shared feature space" claim is stronger than the evidence supports.

Weaknesses

• Mechanism vs representation-space confound. In Table 12, SAE + Random (50.61 AUPRC) already substantially exceeds Dense Cluster + Coverage (45.57), so much of the improvement over dense-embedding baselines comes from being in SAE space rather than from targeting missing features. A Dense Cluster + Random control is missing, and this is the ablation that would justify the paper's headline attribution to feature coverage.
• Circular structure in the FAC–performance correlation. The task-relevant feature set $F$ is chosen by an LLM annotator whose rubrics match the downstream evaluation (toxicity rubric for ToxicChat, IF rubric for AlpacaEval). Correlating "coverage of relevance-labelled features" with downstream accuracy therefore has a partly built-in relationship. The paper shows robustness to label-flip noise (Appendix L.2) but never computes the FAC correlation on the full unfiltered feature set, which would be the cleanest control.
• Over-generalization from a single task's correlation. The abstract quotes $r = 0.95$ and $\rho = 0.90$, but Table 8 shows the other three tasks fall to (0.85 / 0.84), (0.88 / 0.80), (0.79 / 0.65), (0.72 / 0.88). Presenting the toxicity number as the summary statistic is misleading; the four-task correlation should be in the main text.
• Fragile behavior-steering evidence. Sycophancy and Survival rely on only 3 synthetic samples per behavior; baseline SCR standard deviations reach $\pm 38$–$42$ percentage points on a bounded metric, and "Ours" reports $\pm 0.00$ std on Survival — a direct artifact of the tiny budget. The $+42$ headline gap in Table 1 is not statistically well-supported.
• Theory–implementation gap on $\delta$. The whole binary-activation surrogate assumes $\delta > 0$, yet the default configuration runs at $\delta = 0.0$. Figure 6 shows AUPRC peaks near $\delta \in [1.0, 2.0]$, so the paper's own data favours a positive threshold; the mismatch should either be reconciled or the default changed.
• Definition of FAC does not clearly enforce coverage. As written, $\mathrm{FAC} = |F(Q_Z)| / |F(P_Z)|$ is a cardinality ratio, not an intersection ratio; without $F(Q_Z) \subseteq F(P_Z)$, this can be inflated by activating irrelevant features. The appendix implicitly assumes containment; the main text should make it explicit.
• Backbone inconsistency for the flagship Instruction Following result. Appendix J fine-tunes Meta-Llama-3-8B (base) for IF, while the rest of the paper — including the SAE — is on LLaMA-3.1-8B-Instruct. Since Figure 1 is the paper's headline visual, this needs a clear statement of which model the SAE features were extracted from and which is being fine-tuned.
• Weak external validity on reasoning tasks. GSM8K gains of $+0.31$ to $+1.51$ pp and LiveCodeBench gains of $0.00$ to $+1.53$ pp (Table 15) are effectively at-noise. The paper honestly flags this in the conclusion but continues to frame FAC as a general diversity metric elsewhere; the scope caveat deserves earlier and more visible placement.
• Anchor-set / training-set provenance overlap. For Toxicity Detection, the anchor is HH-RLHF and the training set is HH-RLHF-helpful-base. FAC is derived from the anchor's feature distribution, then used to seed synthesis for a fine-tune on a same-provenance corpus. This may explain why the toxicity gaps are far larger than on other tasks with more separation between anchor and eval.
• Overreach on "shared feature space" and "weak-to-strong" claims. Cross-model gains show synthetic data remains useful across backbones; they do not directly demonstrate a shared latent space or a causal weak-to-strong effect. Explicit feature-alignment analyses or additional model pairs are needed before the current phrasing is justified.
• Sample-budget commensurability. Table 1 rows compare methods "at the same budget per task", but the per-task budget varies from 3 to 2 000. Most compared baselines (Alpaca, Magpie, Prismatic) do not produce a natural 3-sample subset for the two steering behaviors, so the trimming protocol needs to be surfaced explicitly.
• Small-scale toxicity training regime. The primary benchmark uses ~200 toxic samples; Figure 12 shows results swing widely with training-scale changes. The reported effect sizes may not persist at more realistic scales.
• Instruction-Following flagship is single-seed for baselines. Figure 10 gives three seeds for "Ours" only; every baseline in Table 1 IF is single-seed. Given the tight margin over SynAlign, at least three seeds for the closest competitors would strengthen the claim.
• Minor factual / notational issues. Table 8 expands FAC as "Feature Attribution Consistency"; the RewardBench Reasoning baseline row lists $\pm 70.09$ std on a bounded accuracy metric; the human-verification prose says 100 sampled features, Table 5 lists 200; three of the four annotation system prompts instruct the annotator to be "as subjective as possible", which contradicts the rubric-based objectivity in the Toxicity prompt.

Reproducibility & code

An end-to-end execution attempt (with code re-derivation from the paper where the shipped repo was incomplete) recovered the paper's story at the level of direction and mechanism, but not at the level of magnitude, and left several headline claims unverifiable.

• Data-efficiency headline unmeasured. The 2 000-sample vs 300 000-sample comparison to MAGPIE on AlpacaEval 2.0 was not independently verified. The reproduction's IF arm had to substitute the judge (Mistral-Large in place of GPT-4-Turbo-1106 after OpenRouter budget was exhausted), the FAC instruction-synthesis run stalled during response generation so the 2 000-sample SFT was trained on a generic Alpaca proxy, and MAGPIE was never rerun. The 150× figure is therefore inherited from Xu et al. (2024b) Table 1, not measured here.
• Headline FAC–AUPRC correlation reproduces at $r \approx 0.69$, not $0.95$. With 5 synthesis methods × 3 seeds on a reimplemented toxicity pipeline, Pearson $r \approx 0.63$–$0.70$ and Spearman $\rho \approx 0.3$–$0.4$ across activation thresholds. The qualitative sign contrast (FAC positive, Distinct-1/2 negative at $r \approx -0.80 / -0.73$) is preserved, but the distinguishing "FAC uniquely predicts" story is not: a bigram / n-gram entropy proxy correlated more strongly ($r \approx +0.83$) than FAC in the reproduction.
• Table 1 headline holds for one task, not four. Only the Toxicity "Ours" cell reproduced as a directly comparable value (AUPRC 64.23 ± 1.68 vs paper 62.60 ± 4.41, ranked first among rebuilt comparators). Reward Modeling's "Ours" partially reproduced (Chat 94.97 vs 94.41, Avg 72.72 vs 76.22 within ~5% relative error; Chat-Hard 62.06 vs 72.22, Safety 51.44 vs 75.63, Reasoning 82.40 vs 62.60 outside). Behavior Steering was not exercised. Instruction Following was proxy-only. External baseline rows were declared out of scope in the reproduction and were not rerun.
• Cross-family transfer replicates directionally but not in magnitude. Every backbone gained AUPRC on the LLaMA-derived shared synthetic data (LLaMA $+12.38$, Mistral $+9.96$, Qwen $+3.38$ pp), matching the qualitative claim, but only the LLaMA gap is close to the paper's $+10.15$; Mistral's collapses from $+19.57$ to $+9.96$ and Qwen's from $+16.76$ to $+3.38$, with all absolute AUPRCs sitting 10–23 points above the paper's — consistent with the reconstructed pipeline shifting the baseline.
• Two-step > one-step is the strongest reproduced effect. At every activation threshold on toxicity, two-step FAC beat one-step by $+5$ / $+25$ / $+21$ / $+3$ pp, though absolute FAC magnitudes differ substantially from Table 10 because FAC was normalised over a reconstructed $F$ of size 1067 rather than the paper's exact feature set.
• The released code could not run any headline pipeline as shipped. The toxicity dataset is marked "released after acceptance"; Algorithm 1's Step-1 and Step-2 scripts ship as non-runnable stubs (empty path globals, undefined imports llama_wrapper_qwen / prompt_config_fn, a typo that silently discarded synthetic data in the LoRA trainer); the RM trainer has a hard flash-attn dependency and empty HF_TOKEN; the LLaMA-Factory driver leaves DATASET, DATASET_DIR, OUTPUT_DIR, EXPORT_DIR blank. No synthesised datasets are shipped, no AlpacaEval-2 harness with GPT-4-Turbo judge is included, and the FAC_freq / FAC_magnitude / Dense Cluster + Coverage / noise-injection / GSM8K / LiveCodeBench pipelines have no corresponding code. The scaffold is careful, but the paper's central synthesis contribution has to be re-derived from prose to reproduce anything.

Recommended Changes

Essential

• Correct or precisely state the FAC definition. Rewrite Section 5.2 so that FAC = $|F(P_Z) \cap F(Q_Z)| / |F(P_Z)|$, or prove $F(Q_Z) \subseteq F(P_Z)$ explicitly under the two-step pipeline, so the metric is unambiguously the "fraction of task-relevant features covered" that the abstract advertises.
• Reconcile the theoretical $\delta > 0$ assumption with the default $\delta = 0$ configuration. Either switch the default to a positive threshold in the range where Figure 6 shows peak AUPRC and re-report Table 1 there, or adapt the derivations in Sections 5–6 and Appendices D / F to $\delta \geq 0$.
• Add a Dense Cluster + Random and an unfiltered-FAC control. Report Dense + Random, Dense + Coverage, SAE + Random, SAE + Coverage, plus FAC computed with no rubric filter, so the mechanism attributed to missing-feature coverage is isolated from the representation-space and rubric-selection confounds.
• Report the four-task correlation in the main text and drop "primary driver" phrasing. Move Table 8 into Section 7.2, quote the range or the mean of $r$ / $\rho$ rather than the toxicity outlier, and downgrade "primary driver of performance gains" to a correlational statement (or add a fixed-sample-count intervention that varies feature identity while holding sample count constant).
• Expand behavior-steering evidence. Increase the synthetic budget for Sycophancy and Survival beyond 3 samples per behavior, report per-seed variance for baselines, and add a paired significance test on SCR before quoting the $+24 / +42$ headline gaps in Table 1.
• Ship at least one end-to-end reproducible task. Fill in the empty path placeholders, add the missing llama_wrapper_qwen / prompt_config_fn, fix the silent-discard bug in the toxicity LoRA trainer, and release the four FAC-synthesised datasets so a reader can reproduce one Table 1 column without contacting the authors. Also release the exact annotated $F$ used for Figures 3 and 5 so downstream FAC comparisons normalise identically.
• Clarify the backbone used for Instruction Following. State explicitly which model provides the SAE features, which is the generator, and which is the fine-tuning target for Table 1 IF and Figure 1; if Meta-Llama-3-8B (base) is used, motivate the deviation from the LLaMA-3.1-8B-Instruct convention used elsewhere.
• Ship the AlpacaEval-2.0 driver and the FAC-synthesised 2 000-sample IF dataset. The 150× data-efficiency headline needs the exact judge configuration (GPT-4-Turbo-1106), the 805-prompt list, and the synthesised training data, since Figure 1 currently imports MAGPIE from Xu et al. (2024b) and no reader can rerun the comparison.

Suggested

• Move the reasoning-benchmark caveat into the main text. Given the near-zero GSM8K / LiveCodeBench gains, adjust the abstract's "various tasks" framing and add a one-paragraph scope statement in Section 8 so the current scope is honest by default rather than by appendix.
• Add an anchor-provenance ablation for Toxicity Detection. Rerun with a HelpSteer-only anchor while fine-tuning on HH-RLHF (or vice versa), to test whether the $+23.63$ AUPRC gap is partially explained by anchor–train provenance overlap.
• Soften or substantiate the "shared feature space" and "weak-to-strong" claims. Either present a direct SAE-to-SAE alignment analysis across backbones, or restate the observations as "synthetic data generated with LLaMA features remains useful when fine-tuning Mistral / Qwen" without invoking a shared latent space or a general weak-to-strong effect.
• Expose per-task budgets in Table 1. Add a column showing the 200 / 300 / 3 / 2 000 sample budgets, and describe how each baseline is trimmed to a natural 3-sample subset for Sycophancy and Survival.
• Report per-seed variance for baselines in Instruction Following. Even three seeds for the two closest competitors (SynAlign, Prismatic Synthesis) would firm up the tight WR / LC margins in Table 1 and Figure 1.
• Scale the toxicity experiment. Report at least one run with a substantially larger training pool (e.g., 2 000 or 20 000 toxic samples) so the reader can see whether the FAC advantage survives outside the linear-probe overfitting regime shown in Figure 12.
• Release the missing ablation scripts. Add implementations for FAC_freq, FAC_magnitude, Dense Cluster + Coverage, the Table 6 noise-injection utility, the Figure 1 aggregation, and a minimal GSM8K / LiveCodeBench harness so the appendix results are auditable.
• Fix minor factual and formatting issues. Correct the Table 8 expansion of FAC to "Feature Activation Coverage"; fix the $\pm 70.09$ std in the RewardBench Reasoning baseline row; reconcile the 100 vs 200 audit sample counts in Table 5 and its surrounding prose; and change "be as subjective as possible" to "be as objective as possible" in the helpfulness / sycophancy / survival / IF annotator prompts.