Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers

Summary

This paper argues that a single LatentQA-style decoder — an Activation Oracle (AO) — can be trained once and then applied off-the-shelf to a wide range of activation-interpretation tasks, without the per-task scaffolding that white-box interpretability methods usually require. The authors extend Pan et al. (2024) along two axes: (i) a norm-matched additive injection scheme that lets the AO consume single or multiple activation vectors of mixed provenance (residual stream, difference vectors, SAE features), and (ii) a training mixture that combines system-prompt QA with seven binary-classification tasks and a self-supervised context-prediction task capable of scaling to arbitrary text. Trained AOs are evaluated on four downstream benchmarks — three secret-keeping games from Cywiński et al. (2025) and the emergent-misalignment model-diffing audit from Minder et al. (2025) — plus PersonaQA and held-out classification. The headline claim is that the best AO matches or exceeds white-box baselines on all four tasks and the best overall baseline on three of four, generalizing to fine-tuned models it never trained on. The conceptual move — that a generalist language interface over activations is a competitive substitute for hand-crafted probing pipelines — is significant, both because it lowers the barrier to applying interpretability tools and because the ablations plausibly show that dataset diversity (not just SPQA scale) drives the gains. The main limitations are conceptual and evidentiary rather than technical: the 'general-purpose' framing outruns the fact that every benchmark is a narrow single-attribute fine-tune; the 2/5 'effective ceiling' argument on the misalignment task turns a numerical tie into a positive claim that leans heavily on an assumption contradicted by one of the six model organisms; several central controls (a no-injection AO baseline, per-prompt variance, apples-to-apples yes/no PersonaQA comparison) are missing; and a handful of author-level results in the discussion — Consensus@10 calibration, sycophancy, MMLU-correctness prediction, and the emergent-misalignment pipeline — are not reproducible from the released code alone.

Strengths

• Conceptual contribution. Reframing LatentQA as a generalist question-answering interface — rather than a specialized decoder — is a clean, potentially widely adopted idea, and the norm-matched additive injection makes it plausible that a single trained decoder can handle inputs of very different provenance.
• Ablation design. The Full-vs-truncated-vs-SPQA+Classification comparison (Figure 10) genuinely separates data-diversity from data-quantity effects rather than confounding them, and the 12-combination progression in Figure 6 is a real (if noisy) piece of evidence for the 'diverse mixture helps' claim.
• Honest treatment of PersonaQA brittleness. The paper explicitly reports that PersonaQA fine-tuned models drop to ~55% on yes/no reformulations of the same knowledge (Figures 17–18) and discusses how this complicates the AO comparison rather than hiding it.
• Breadth of coverage. Running the pipeline on four base models — including a closed-weight Anthropic model with a different injection scheme — provides some evidence that the method is not architecture-specific, even if the Claude column is not externally reproducible.
• Downstream head-to-head with real baselines. Comparing to Cywiński et al.'s manually-tuned agentic pipelines and Minder et al.'s ADL on shared prompts (rather than reporting isolated numbers) is exactly the kind of honest comparison this line of work needed.

Weaknesses

• 'General-purpose' claim exceeds the empirical scope. Every downstream benchmark encodes a single behavioral dimension (one taboo word, a gender label, a base64 string, one misalignment domain, one persona attribute). The Discussion acknowledges this ('our benchmarks are simplified and narrow'), but the abstract still markets AOs as general-purpose. There is no compositional evaluation where the AO must simultaneously recover multiple attributes of a rich activation, which is what 'general-purpose' would require.
• Missing no-injection control for the AO itself. The paper never checks how much of the AO's secret-keeping accuracy comes from the injected activations versus from the AO's prior over 'what secrets are LLM-audit prompts usually hiding?'. Running the AO on the same oracle prompt with an all-zeros or randomized injected vector would bound the contribution of the activation channel.
• The 2/5 ceiling argument is over-claimed. Section 4.2 asserts that '2/5 represents the effective ceiling for this evaluation' because 'models only exhibit domain-specific issues'. But Figure 13 shows that the risky-financial Qwen3-8B fine-tune still produces ~18% emergent misalignment, which under the rubric should permit scores > 2/5. The Figure 4 caption hedges ('it is unclear'); the body text does not, and the 'AO matches baseline at ceiling' framing depends on the stronger claim.
• SSC framing contradicts its own setup. The paper claims 'None of these secrets can be recovered from the prompt alone', yet the SSC secret is literally base64-encoded in the target prompt and can be decoded by any capable reader. This is a small logical slip but it undercuts the 'AOs extract information absent from input text' framing on one of the four headline tasks.
• GPT-4.1 post-processing is not neutral. The SSC pipeline runs conversational AO answers through a GPT-4.1 extractor whose prompt includes a 'return the concise predicate phrase' fallback. This is described as 'lightweight' but is doing real work — potentially adding information the AO did not produce. Reporting SSC with and without the extractor (and applying the same treatment to baselines) is necessary for a clean comparison.
• Under-specified statistical framing. Headline percentages come without stated seed counts, per-prompt variance, or per-word variance (for Taboo, 20 words × few prompts each), and the 18% ± 8% emergent-misalignment interval is called 'significant' without saying what the interval represents or whether multiple-comparison correction was applied. On close comparisons — e.g., SSC 59% vs. 52% — the reader cannot judge whether the reported ordering is stable.
• Technical inconsistencies in Section 3 and the appendices. The injection equation is missing after 'v_i:' in Section 3.1; the injection layer is described as both 'after the second transformer layer' and 'in layer 1'; Section 3.2 gives $K, J \sim \text{Uniform}(1, 20)$ but Appendix F.2 uses 1–50 for the same quantities; Table 1 claims to cover 'all experiments' while the Llama batch size is 64; and the persona-attribute count differs between Appendix C (six) and Appendix I (seven). Individually minor, but they compound for a reader trying to reproduce the setup.
• Figure 19 vs. main text. The Figure 19 caption identifies bars as the open-ended PersonaQA metric, but the extracted values (Full Dataset near 0.33–0.37 for one row of models) do not obviously match the body text's summary that open-source AOs achieve ~14% and only Claude achieves ~35%.
• PersonaQA yes/no result is presented without apples-to-apples justification. Reporting that AOs 'outperform the fine-tuned PersonaQA models themselves' on yes/no is striking, but the paper does not confirm that Figures 18 and 20 use the same items or difficulty, and it does not rule out that the AO's persona-name prompt gives it a template advantage the source model does not exploit.
• Prompt selection. Reporting the best of two hand-written oracle prompts (chosen on a validation split) is defensible, but the paper reports no per-prompt variance, so a reader cannot tell whether the AO's SSC gain over the best white-box baseline is within the two-prompt spread.
• Related Work is empty. Section 6's heading appears with no body — the actual related-work paragraphs sit under Section 7 (Discussion and Limitations).

Reproducibility & code

• Emergent-misalignment pipeline is not vendored. plot_em_agent.py reads from an em_runs/ tree produced by Minder et al.'s ADL and talkative-probe agents, but that scaffold, the six target fine-tunes, and the gpt-5-mini grading harness are all outside the repo. As a result, the misalignment leg of the 'match or exceed white-box baselines on all four tasks' claim (Figure 4, AO 2.00 vs ADL 2.03) is not reproducible from the release alone.
• Confidence-calibration numbers require missing instrumentation. The Consensus@10 PR curve on Taboo (Figure 11) and the yes/no logit-margin PR-AUC = 0.754 on OOD classification (Appendix E) both require additions to the shipped code — the taboo eval defaults to do_sample=False, and the classification eval does not save yes/no logit differences. No calibration script is included.
• Sycophancy / MMLU-correctness AUCs unreleased. The AUC = 0.83 sycophancy and AUC = 0.75 MMLU-correctness numbers cited in the Discussion have no dataset, prompt template, or evaluation harness in the repository, even though they carry non-trivial rhetorical weight ('preliminary evidence of broader applicability').
• Layer ablation trees not shipped. The layer-0-vs-layer-1 Llama-3.3-70B comparison (Figure 8), the layer-50-vs-layer-75 secret-keeping comparison (Figure 16), and the 0–75% classification layer sweep (Figure 15) all point to result trees not in the release; regenerating them requires re-training or ~21 additional evaluation passes, and there is code-level ambiguity about whether layer-75 evaluations use a layer-75-trained AO or the same layer-50-trained AO.
• Claude Haiku 3.5 columns are structurally not reproducible. Appendix B.6 documents a different injection scheme, tokenization, and full-parameter fine-tuning on internal infrastructure. Every Claude bar in per-model figures is thus a black-box datapoint; the main text does not flag this at first appearance.
• Baselines are hard-coded rather than recomputed. plot_secret_keeping_results.py embeds the Cywiński et al. baseline constants as literals; a reproducer running the full stack never touches the baseline pipeline. No commit/URL is cited for the METRICS.md revision the numbers come from.
• Data-mixture ablation configs require editing sft.py. The four training variants required for Figure 6 are not exposed as flags; only the 'Full' iteration is uncommented in nl_probes/sft.py. This is a friction point rather than a blocker but raises the risk of silent drift from the paper-run configuration.

Recommended Changes

Essential

• Add a no-injection control for the AO. Report AO accuracy on the same oracle prompts with the injected activations replaced by zeros (or randomized noise) on all four downstream tasks, so the reader can bound how much of the reported accuracy is carried by the activation channel vs. the AO's language prior. This addresses the missing no-injection control weakness above.
• Rework the 2/5 'effective ceiling' argument. Either (i) soften the body text to match Figure 4's hedge and drop the 'near-maximum performance' framing, or (ii) explicitly analyze the risky-financial Qwen3-8B fine-tune (which does show 18% emergent misalignment) and show that AO/ADL still saturate for that model in particular. This addresses the over-claimed ceiling weakness.
• Fix the missing injection equation, layer-index inconsistency, and Section 3.2 vs. Appendix F.2 range mismatch. Restore the equation after 'v_i:', pick a single 0- or 1-based convention for the injection layer, and reconcile $K, J \sim \text{Uniform}(1, 20)$ with the appendix's 1–50 range. These map to the technical inconsistencies weakness.
• Vendor or clearly bridge the emergent-misalignment pipeline. Ship the target fine-tuning script, the misaligned-answer-rate evaluation for the 10 Betley et al. prompts, and either the ADL adapter code or a minimal replacement that reproduces the 2.00 vs 2.03 comparison in Figure 4. This addresses the emergent-misalignment pipeline not vendored reproducibility bullet.
• Release the calibration and preliminary-evidence experiments. Ship the Consensus@10 sampling + PR-AUC script for Taboo, the yes/no logit-margin logging for OOD classification, and the sycophancy / MMLU-correctness datasets and evaluation harness — or downgrade those numbers in the Discussion to 'author-observed pilots'. This covers the confidence-calibration and sycophancy / MMLU AUCs reproducibility bullets.
• Rephrase the SSC framing and the 'general-purpose' framing. Change 'None of these secrets can be recovered from the prompt alone' to reflect that the SSC secret is textually present but obfuscated, and align the abstract's 'general-purpose activation explainer' claim with the empirical scope by either qualifying the phrase or adding one compositional evaluation. Maps to the SSC framing and general-purpose scope weaknesses.

Suggested

• Report per-prompt variance for the two-oracle-prompts protocol. Add a small table showing validation-set accuracy for each of the two prompts on each secret-keeping task, so the reader can see whether the ~7-point SSC gap sits within the prompt-choice spread. Addresses the prompt selection weakness.
• Add error bars and state their definition on Figures 3, 4, 5, 10, and 22. Specify what the ±8% on the emergent-misalignment rate represents, seed/prompt counts per average, and whether multiple-comparison correction was applied. Addresses the under-specified statistical framing weakness.
• Clarify the PersonaQA yes/no comparison. State explicitly that Figures 18 and 20 use the same items, and add an AO-on-base-model control to show the yes/no accuracy is not a template artifact. Addresses the PersonaQA yes/no result weakness.
• Move the related-work paragraphs back under Section 6 and reconcile the persona-attribute count (six vs seven), the Table 1 batch-size caption, and the shuffled-example that retains 'Arabic Pop'. Addresses the empty Related Work and residual technical-inconsistency weaknesses.
• Report SSC accuracy with and without the GPT-4.1 post-processing step, and apply the same normalization to the baselines it is compared against. Addresses the GPT-4.1 SSC post-processing weakness.
• Ship layer-ablation fixture JSONs and expose the training-mixture variants as CLI flags in sft.py. Removes the biggest remaining reproducibility friction for Figures 6, 8, 15, and 16. Addresses the layer ablation trees and training-mixture ablation configs reproducibility bullets.
• Add a one-line note at first mention of Claude Haiku 3.5 results flagging that all Claude columns rely on internal infrastructure and are not externally reproducible. Addresses the Claude columns reproducibility bullet.
• Cite the exact commit of Cywiński et al.'s METRICS.md used for the hard-coded baseline constants, and prefer parsing that file to embedding literals. Addresses the hard-coded baselines reproducibility bullet.