Robust Harmful Features Under Jailbreak Attacks: Mechanistic Evidence from Attention Head Specialization in Large Language Models

Summary

The paper asks a genuinely interesting mechanistic question: when a jailbreak "succeeds," what actually happens inside the model? Rather than treating the refusal representation as a single monolithic direction that either survives or is erased, the authors decompose contributions to a mid-layer refusal direction across attention heads via OV back-projection, then use kernel-density overlap between three input distributions (benign, harmful, successful-attack) to partition heads. The central conceptual move is a functional taxonomy: Adversarially Compromised Heads (ACHs) that are suppressed under attack (concentrated in early layers) versus Safety-Aligned Heads (SAHs) whose activation is sustained or amplified under attack (concentrated in mid-layers). Two follow-on results are meant to close the loop — ablating a small set of ACHs induces jailbreak-like behavior on Llama-3-8B, and a training-free detector that reads standardized activations along discriminative components matches or beats dedicated safety classifiers on aggregate safety-eval benchmarks. The reframing — attacks bypass rather than erase safety representations — is a useful conceptual advance for the jailbreak-mechanism literature. That said, the paper's argumentative structure has real gaps: the back-projection recursion omits the residual-stream identity term; the ACH intervention subtracts a scaled multiple of the very direction whose activation is being scored, so the ablation is a direction-specific perturbation rather than an independent test of the mechanism; the ">95%" saturation framing is only true for Llama-3, and the 70B scaling analysis shows the "8 critical heads" number swelling to ~40 without the paper engaging with what that implies. The SAH-ablation effect (14–18% Mean|Act| reduction) is smaller than the surrounding "substantially weakens" prose implies, and the third head class (harmful-salient heads) that the method actually identifies is quietly absent from the abstract's two-type framing. The detector is a legitimate practical validation, but its components are selected on the same $D_{\text{harmful}} / D_{\text{attack}}$ pool used to characterise the taxonomy, and per-model tuning of $m^*, \tau^*$ substantially blurs the "training-free" positioning. Cross-family generalisation is only asserted; only Llama-family models are studied mechanistically. Together these leave a genuinely worthwhile mechanistic contribution somewhat overclaimed.

Strengths

• Conceptual reframing. Moving from "attacks erase safety" to "attacks selectively suppress and other components sustain robust harmful features" is a productive and testable hypothesis, and the taxonomy operationalises it at head resolution.
• Coupled correlational + causal design. The paper does not stop at distribution-overlap statistics: ACHs earn causal legitimacy via progressive ablation with a random-head control, and the token-attribution analysis pinpoints attack-template tokens as the driver of ACH suppression.
• Threshold and bandwidth sensitivity analysis. Appendix A works hard: overlap thresholds are swept across $[0.1, 0.9]$, KDE bandwidth is swept across $[0.2 \times h_{\text{Scott}}, 2.0 \times h_{\text{Scott}}]$, and Spearman rank correlation of head OVLs stays $> 0.92$ throughout. This substantially raises confidence that the ranking structure is not KDE artifact.
• Training-free practical validation. The detector is not a mechanism-agnostic bolt-on: it exploits the same persistent activations the mechanistic analysis identifies, so a strong detection result is legitimate corroboration of the mechanistic story, not an unrelated add-on.
• Preliminary scaling analysis on 70B. The authors are honest that only 40 filtered pairs are available for 70B and label the results as preliminary; the S-shaped ASR-vs-$k$ curve reappearing at 70B is worth reporting even as an under-powered check.
• Honest limitations paragraph. MLP exclusion, single-direction refusal summary, and small-family model scope are all acknowledged in the closing paragraph and Appendix B, which sets a good tone.

Weaknesses

• Residual-stream identity dropped in back-projection. Section 3.2 writes $r^{(l-1)} = \\sum_h r^{(l,h)}_{\text{in}}$, replacing $r^{(l)}$ entirely with the layer-$l$ head contributions. The forward pass includes a residual skip that should back-project as an identity term ($r^{(l-1)}$ should carry $r^{(l)}$ plus the head contributions). This directly contradicts the "residual connections imply that mid-layer representations accumulate outputs from earlier layers" motivation just above. The entire ACH/SAH taxonomy is built on projections along the back-propagated direction, so this deserves explicit justification or correction.
• Direction-specific ablation is close to tautological. The ACH intervention subtracts $\alpha \cdot r_v^{(l,h)}$ from the head's EOI output — the very direction whose activation is measured. It is not a surprise that this raises ASR; what would be surprising is that a matched-magnitude orthogonal perturbation does not. Without an orthogonal-perturbation control, the ablation establishes that refusal-direction projection at these heads causally affects refusal, not the stronger claim that ACH suppression is the mechanism attacks exploit.
• Random-head control does not match ACH statistics. Random heads are drawn without matching activation magnitude or classifier score. Because ACHs by construction have large attack-induced shifts, the observed 99.5% vs 4.0% gap conflates directional targeting with magnitude targeting. A "top-$N$ non-ACH heads by harmful-input activation magnitude" control would sharpen the specificity claim.
• Scaling breaks the "8 heads" narrative on 70B. Table A10 shows 70B needs ~40 ACH ablations to reach 95% ASR — five times the 8B number. The paper reports this in Appendix E but does not engage with what it implies for the "small number of critical components" framing of the abstract and conclusion.
• ">95% saturating at 99.5%" is only Llama-3. Llama-2 saturates at 83.7% and only reaches 81.6% at $k=8$. The abstract, introduction, and conclusion generalise across both models; the section body reports both numbers honestly, but the top-level framing overstates the Llama-2 result.
• "Substantially weakens" overstates a 14–18% reduction. Mean|Act| drops of 0.222→0.181 (Llama-2) and 0.0062→0.0053 (Llama-3) do not obviously earn "substantially weakened" or "nearly disappears" language, especially when most of the gap from original-harmful is already the attack's own doing, not the ablation's. Either the language should be softened or an effect-size normalisation added.
• Taxonomy is binary in framing but ternary in the method. The abstract promises two head types; Section 3.4 explicitly names a third — "Harmful-Salient Heads" — which is numerically the largest class and is at least as relevant to "persistent activation under attack" as SAHs are. The two-tier framing is never reconciled with the three-tier method.
• Cross-family generalisation is asserted, not tested. All mechanistically analysed models are from the Llama family (Llama-2-7B, Llama-3-8B, Llama-3-70B). The Section 5 detection results span Gemma and Qwen, but that is downstream detection F1, not head-level ACH/SAH pattern replication. Suggesting the taxonomy "may extend beyond a single model family" would require at least one non-Llama mechanistic study.
• Detector borders on circularity. Candidate components $\mathcal{C}$ are selected on the same $D_{\text{harmful}}$/$D_{\text{benign}}$ pool used for mechanistic analysis; $\mu_B, \sigma_B$ come from the same $D_{\text{benign}}$; per-model $m^*, \tau^*$ are tuned by validation-set F1 (with the search grid unspecified). The "no training" positioning is then used to compare against trained guards; that comparison is fair only under a careful attack-family holdout, which the paper does not describe.
• SAH ablation lacks benign-input and refusal-input controls. The experiment only asks whether SAH ablation changes mid-layer activation under attack. It does not ask (i) whether the same ablation harms benign completions (specificity) or (ii) whether it flips refused-harmful requests into compliance (functional necessity). Either would substantially strengthen the "important source of Robust Harmful Features" claim.
• Single refusal direction is used as more than a first-order summary. The paper acknowledges the limitation but never tests it: swapping in a nearby direction within the refusal cone (Wollschläger et al.) or an orthogonal cone basis should be checked to confirm the ACH/SAH sets are not fragile to the choice of $r^{(L^*)}$.
• Additivity of standardized token contributions is asserted, not shown. Section 4.4's "$\\sum_k z_k = z$" identity is proved by invoking $\\sum_k A_{EOI \to k} = 1$, but the standardization subtracts a benign mean before dividing by $\sigma_B$, and how that mean is split across tokens is not shown. The 3σ / 6σ separation claims in Section 4.4 and Appendix E.4 further conflate "gap in standardized-score units" with "between-group effect size."
• Detector aggregation claim overstates parsimony. "$m$ typically much smaller than $|\mathcal{C}|$" is contradicted by Table A5: Gemma-2-9B keeps 91% of $\mathcal{C}$, Llama-3-8B keeps 83%, Qwen-7B keeps 80%.
• Missing MLP treatment for Figure 1. Figure 1's x-axis is "Component Index (Heads + MLP)," and the mid-layer red regions clearly include MLP columns. The paper never defines how MLP contributions to the refusal direction are computed for this figure, even though it is the motivational visualization of the whole paper.
• Minor caption and citation issues. Figure A3's blue/orange harmful-vs-attack caption does not match the bandwidth-only panel structure; "Baroni et al. 2025" and "Bricken et al. 2023" appear in-text but not in the visible reference list; the "~2–3% rate" phrasing brackets neither the 1.8% (Llama-3) nor the 3.8% (Llama-2) it purports to summarise; and Appendix E's "over 97% at full ablation ($k=62$)" mis-labels which ablation level owns which percentage.

Reproducibility & code

The released materials in the paper-code bundle inspected here contain the manuscript but no runnable code, configuration, filtered attack pairs, or evaluation harness. The paper documents its pipeline reasonably well at the algorithmic level (Algorithm 1, Section 3 formulas, Table A5 detector hyperparameters), but several critical operational details are unspecified. The gaps below all trace to the veritas assessments of data_available: false for most headline claims.

• Attack-success oracle is never named. Every ASR number, every $D_{\text{attack}}$ construction step, and the paper's whole notion of "successful attack" hinges on an unnamed classifier (LLM-as-judge? Zou-style keyword match? human?). Different judges have been shown to shift ASR by tens of points on the same completions; without this spec, Table 1, Figure 5, Table A10, and the 176/378/40 pair counts cannot be reproduced.
• Attack pipeline hyperparameters unspecified. The four attack families (GCG, AutoDAN, many-shot, PAIR) each have several sensitivity axes (steps, suffix length, temperature, shot count, target model). None are stated, and neither is the mix over the ~10k candidate pool.
• Filtered attack pairs not released. The 176 (Llama-3-8B), 378 (Llama-2-7B), and 40 (Llama-3-70B) paired examples drive every mechanistic result (KDE fits, OVL classifications, ACH ranking, token attribution). Without these, no downstream count can be reproduced end-to-end.
• $D_{\text{benign}}, D_{\text{harmful}}$ subsets undocumented. Alpaca sample and per-source harmful-instruction selection are not enumerated, but the $(\mu_B, \sigma_B)$ derived from $D_{\text{benign}}$ enters every standardized-activation number the paper reports.
• Token partition rule for $\mathcal{H}$ vs $\mathcal{J}$ undefined. The central token-attribution finding (Figure 7, Figure A8) depends on labelling each source token as "harmful request" or "attack template." The rule is nowhere given; for role-play or PAIR-style rewrites the boundary is genuinely ambiguous.
• VANILLA baseline is a functional but unspecified classifier. Table 2's +81.5% / +70.0% relative improvements are anchored on VANILLA F1s of 0.470 and 0.506, but the mapping from a model completion to a harmful/benign prediction (pattern match? refusal keywords? judge?) is not documented.
• Detector hyperparameter search grid and validation split not stated. Per-model $m^*$ and $\tau^*$ in Table A5 span 38–875 and 0.677–4.992, respectively. This is meaningful per-model tuning; without the search grid and validation split, both reproducibility and the "training-free" framing are hard to audit.
• "10 runs" for ACH ablation is under-defined. ACH identity is deterministic given the classification, so the reader needs to know what varies over the 10 runs (generation seeds? judge samples? subsets of the harmful pool?) to interpret the ±0.3 / ±1.1 CIs and compare them to the random-head ±3.7 / ±11.6 CIs.
• MLP contribution to Figure 1 is un-formalised. The motivational heatmap includes MLP columns, but the method section never defines how MLP output projects onto the refusal direction; this leaves the Figure 1 mid-layer red regions partially un-attributed.
• 70B analysis rests on $n=40$ pairs. KDE fitting per head over 40 samples is noisy in absolute terms (Scott's-rule bandwidth grows with $n^{-1/5}$); no bootstrap CI on the 200/62/83 counts is reported, which makes the 70B classification a preliminary result that the current text sometimes treats as confirmatory.

Recommended Changes

Essential.

• Fix or justify the back-projection recursion. State whether $r^{(l-1)}$ should include the residual-stream identity term (as the forward-pass structure implies), and if not, provide the justification for dropping it. This is the mathematical backbone of the whole ACH/SAH taxonomy.
• Add an orthogonal-perturbation control to the ACH ablation. Show that a matched-$\\ell_2$ perturbation to head $(l,h)$ outputs that is orthogonal to $r_v^{(l,h)}$ does not raise ASR the way ACH ablation does. This is what would upgrade "8 ACHs suffice" from a direction-specific result to a mechanism-specific one.
• Add a magnitude-matched random-head control. Replace or supplement "random heads" with "top-$N$ non-ACH heads by $|z_{\text{harmful}}|$" so the specificity claim distinguishes directional targeting from magnitude targeting.
• Engage with the 8→40 scaling. Explicitly discuss what the 70B "~40 ACHs required for 95% ASR" result implies for the "small number of critical components" framing. Either update the abstract/conclusion to reflect the scaling, or provide the argument for why 40/1728 heads is still the same phenomenon as 8/1024.
• Release attack pairs, success oracle, and code. Publishing the 176/378/40 filtered pair lists, the ASR judge (model, prompt, thresholds), the attack-pipeline hyperparameters, and the classification/detector code would resolve the majority of the reproducibility gaps listed above.
• Document the VANILLA classifier and detector search grid. State how a completion becomes a harmful/benign label under VANILLA, and list the $m$/$\tau$ search grid plus the validation split used to select the per-model $m^*, \tau^*$ in Table A5.
• Name the token-partition rule for $\mathcal{H}$ vs $\mathcal{J}$. Give the exact procedure that assigns each source token to "harmful" or "attack-template" for the Figure 7 and Figure A8 decompositions.
• Reconcile the two-type vs three-type taxonomy. Explicitly discuss the harmful-salient class in the abstract / introduction / contributions, or explain why it is subsumed into the two-tier framing.
• Add a benign-input and a refused-harmful control for SAH ablation. Show whether ablating SAHs (i) harms benign completions and (ii) flips refused-harmful requests into compliance; this would upgrade the "important source" claim from a correlational reading to a causal one.

Suggested.

• Test refusal-direction robustness. Reclassify heads using a direction chosen from within the refusal cone (or two orthogonal basis directions of it) and report Jaccard consistency of the ACH/SAH sets. This addresses the "single direction as first-order summary" limitation directly.
• Soften the "substantially weakens" language. Match the prose to the 14–18% reduction, or add a normalisation that makes the effect appear as large as the language implies.
• Tighten ">95%" framing. State the Llama-3 and Llama-2 numbers separately in the abstract, introduction, and conclusion, or use per-model wording that does not average away the 83.7% saturation on Llama-2.
• Qualify the "$m$ typically much smaller than $|\mathcal{C}|$" claim — three of seven models retain $\\geq 80\\%$ of the candidate set.
• Report per-head bootstrapped OVL confidence intervals at 70B. Given $n=40$, per-head KDE OVL is noisy; a bootstrap band on the 62/83 counts would ground the "preliminary" language quantitatively.
• Formalise (or explicitly acknowledge) the MLP channel in Figure 1. Either compute an MLP contribution to the refusal-direction activation (integrated-gradient projection is a defensible standard) or restrict the heatmap to attention heads and update the caption.
• Add explicit denominators to the "three sigma" / "six sigma" claims. State whether the standard deviation is per-token benign, within-group over heads, or between-group, so effect-size claims are unambiguous.
• Fix Figure A3's caption, add the missing Baroni/Bricken references, correct the "over 97% at $k=62$" $\to$ "~98% at $k=62$" attribution, label the 17 vs 21 8B counts, and replace "~2–3%" with the actual 1.8% / 3.8% rates.
• Add a cross-family mechanistic study. Even one Gemma or Qwen model with the full ACH/SAH pipeline would substantially strengthen the "beyond a single model family" claim.
• Address the potential circularity of the detector benchmark. Report a held-out-attack-family evaluation (e.g., train the taxonomy without WildJailbreak-style attacks, then evaluate the detector on WildJailbreak) so that "strong adversarial robustness" is not confounded with in-distribution component selection.