Orthogonal Concept Erasure for Diffusion Models

Summary

This paper proposes Orthogonal Concept Erasure (OCE), a closed-form editing method that removes designated concepts from text-to-image diffusion models by post-multiplying cross-attention projection matrices with a layer-wise orthogonal transformation P, rather than adding a perturbation ΔW as prior editing-based methods (UCE, RECE, SPEED) do. The conceptual move is clean: because PW preserves the norms of neurons (columns of W) and pairwise inter-neuron angles exactly, the update can rotate concept-carrying directions while — by construction — leaving what the paper calls the intrinsic angular geometry untouched. The vector-wise variant reduces to a classical Procrustes problem with a single SVD; the multi-concept variant lifts the erasure objective to subspace suppression against a projector onto the orthogonal complement of the anchor subspace, and combines it with a vector-wise preservation prior K_0 precomputed once from COCO-30k. Experimentally, OCE reports the best average H_o on CIFAR-10 object erasure, the best target-style CS on three artistic styles, and the strongest reported 10/50/100-celebrity trade-off at 4.3 s. The method is also shown qualitatively to transfer to FLUX.1 dev. Relative to the additive-editing literature this is a genuine reframing — the paper turns concept editing into a Procrustes problem — and the resulting one-line update is attractively simple. The main conceptual limitation is that the geometric motivation leans almost entirely on a single toy demonstration (Sec. 3.1) that uses one concept, one scaling factor, an unspecified rotation angle, and a Case B manipulation that changes two factors at once, so the isolation of C2 from C1 is not clean. The empirical evidence is comparably thin on the statistical side: all headline numbers are single-run point estimates with no error bars, adversarial robustness is only tested on nudity, and the closest orthogonal-projection contemporaries (CURE, AdaVD, SUMA) are not benchmarked. Notational and prose inconsistencies in Sec. 3.2 and Sec. 5.2, together with a garbled harmonic-mean formula, further blur the technical picture.

Strengths

• Conceptual contribution. Reformulating editing-based concept erasure as a Procrustes / orthogonal-transform problem — as opposed to an additive perturbation — is a genuinely useful reframing, and the argument that PW preserves norms and pairwise angles by construction is clean.
• Subspace-level multi-concept objective. Extending vector-wise anchor alignment to a subspace-suppression formulation is a natural response to the conflicting-constraint problem in mass erasure, and the asymmetric erasure-subspace / preservation-vector design (validated by Table 5) is an interesting design choice.
• Efficiency and scale. Erasing 100 celebrities in a single closed-form update on the order of seconds is genuinely useful for deployment, and the reported time (4.3 s on an A100) is directly instrumented in the released Orthogonal_Erase routine.
• Reusable global prior. The offline construction of K_0 from COCO-30k, shared across all downstream edits, is a sensible engineering decision that keeps the per-edit cost low, and Table 6 shows a monotonic improvement with more generic tokens.
• Task and architecture breadth. The paper covers object, artistic style, celebrity, and implicit-NSFW erasure on SD v1.4, and extends the approach qualitatively to a DiT (FLUX.1 dev) architecture.

Weaknesses

• Toy experiment overreach (Sec. 3). The two claims (C1, C2) that motivate the entire orthogonal reformulation rest on one concept ('cat'), one scaling value (α = 0.5), an unspecified 'small' rotation for Case C, and a Case B manipulation that changes both neuron directions and inter-neuron geometry simultaneously. The conclusion that magnitude 'has little effect' and that inter-neuron geometry is 'crucial' therefore relies on an unmatched-perturbation, single-seed, single-concept demonstration in which C2 is not cleanly isolated from C1.
• Notational inconsistencies in Sec. 3.2 and Sec. 5.2. The additive-update block writes cos θ*_i = cos θ_i for a mechanism the prose describes as 'primarily rotating' directions — the equality here contradicts the prose (which would need an inequality, matching the magnitude term). Meanwhile, Sec. 1 and Sec. 5.2 describe the subspace-suppression objective with opposite signs (targets pushed into vs. out of the orthogonal complement of the anchor space), so the intended geometry is not recoverable from the prose alone.
• Garbled harmonic-mean definition. H_o is the paper's headline overall metric, but the printed formula reads as H_o = (1 − Acc_e)^{-2^1} + Acc_s^{-1}, which is not a harmonic mean; the intended 2·[(1 − Acc_e)^{-1} + Acc_s^{-1}]^{-1} (with the 100 − Acc_e convention) is only recoverable by back-computing from the tables.
• Statistical rigor. All headline numbers appear to be single-run point estimates with a fixed seed and no error bars, seed variance, or confidence intervals. Several improvements the paper leans on are within a few points (e.g., 'Person' vs. 'celebrity' anchor H_o in Table 11, style CS gaps in Table 2, and the SPEED / OCE runtime gap of 0.7 s), for which variance would matter.
• Ablation gap on which matrices to edit. Section 4 motivates the method using both W_k and W_v, but the Implementation Details state that only W_k is edited across all tasks, with no ablation over the choice. Given that key baselines edit both, this omission leaves an obvious confound between 'orthogonal vs. additive update' and 'which parameters are being touched.'
• Overstated efficiency claim vs. SPEED. The paper concedes SPEED's runtime excludes preprocessing while OCE's does not, and Table 3 shows a 0.7 s gap at 100 concepts. The prose nonetheless asserts 'clear efficiency advantages'; the argument is not defensible without SPEED's total (preprocessing-inclusive) time, which is not reported.
• Unverifiable and mis-placed 8.32 vs. 4.61 / 0.01% claims. The 'reduces average Acc_e from 8.32 to 4.61' comparison does not identify the SOTA row, and the '0.01% Acc_s drop compared to the original model' is placed under Table 3 (multi-concept), where the actual Acc_s drop is 0.6–3.1 percentage points depending on the erasure scale.
• Adversarial robustness limited to nudity. Table 4 (I2P/MMA/Ring-A-Bell) is entirely nudity-oriented, using a single Nudenet detector. The AT variant's robustness claim does not generalise to celebrities, objects, or styles under adversarial paraphrase.
• Missing engagement with the closest orthogonal-projection baselines. CURE (Biswas et al., 2025) and AdaVD (Wang et al., 2025) — both orthogonal — are dismissed in Appendix B as 'inference-time,' and SUMA (Nguyen et al., 2025), a subspace-mapping erasure approach, is cited but never benchmarked. For a paper whose title is 'Orthogonal Concept Erasure,' a quantitative comparison with these methods is warranted.
• Anchor selection is not operationalised. 'Same high-level category ... noticeable differences' is qualitative, and the anchor for celebrity multi-concept erasure is the single generic word 'celebrity' (Table 13), which is not what the described heuristic prescribes. Given that Table 10 shows anchor choice moves H_o, the selection rule needs a concrete specification.
• FLUX transferability supported mostly qualitatively. Fig. 4 is qualitative and the only quantitative FLUX result (Table 9) is nudity-focused; no CIFAR / style / celebrity numbers or DiT-specific baselines (e.g., EraseAnything) are reported.
• Minor label mismatches. In Appendix D.4 the third hyperparameter is written λ_p and the preservation metric Acc_r, both of which have no counterpart in the tables (λ_r / Acc_s). Tables 1 and 10 report inconsistent H_o averages (97.01 vs. 96.98) for the same underlying (Acc_e, Acc_s) averages.

Reproducibility & code

Based on reading — not executing — the released repository, the code implements the closed-form update faithfully and covers most of the main experiments in scaffolded form, but several coverage gaps and defaults mean out-of-the-box runs will not reproduce the reported tables.

• Shipped hyperparameters and anchors do not match the paper's chosen row. The 100-celebrity script uses ERASE_SCALE=800, PRESERVE_GLOBAL_SCALE=70, PRESERVE_CONCEPT_SCALE=2 and GUIDE_CONCEPTS='tree', whereas Appendix C.1 states λ_e=900, λ_0=50, λ_r=3 with anchor 'celebrity' (Table 13). Similar mismatches appear in the 10-celeb, 50-celeb, object, and style scripts. Because Table 11 shows anchor choice moves H_o, these defaults matter.
• Object erasure has no per-class driver. The object script hard-wires airplane / anchor 'sky' with different λs than the paper; reproducing Tables 1 and 8 requires 10 manual edits and 10 anchor changes not encoded in the code. The evaluator prints per-class CLIP probabilities but does not compute Acc_e, Acc_s, or H_o.
• Style-erasure protocol differs from the paper. The released style generator uses ~30 diverse painting templates rather than the single 'a painting in the style of {style}' prompt described for Table 2, and only Van Gogh is scripted; Picasso and Monet require manual edits.
• Objective-ablation variants (Table 5) not runnable. The erase term is hard-wired to the subspace formulation and preservation to vector-wise; there is no flag or script to switch to variants (1) or (2). build_preserve_subspace is defined but never used downstream.
• K_0 scripts have blocking bugs. The COCO K_0 script has a syntax error near line 120 and uses a sample_size inconsistent with COCO-30k's actual token count; the FLUX K_0 script's __main__ calls an undefined compute_and_save_Cg_flux (defined as compute_and_save_Ce_flux) and raises NameError as shipped. There is also no code path for the 1/3 / 2/3 subsets or the alternative regularizer used for the 'None' row in Table 6.
• Adversarial evaluation and Ours w/ AT not runnable end-to-end. The Ring-A-Bell and MMA-Diffusion prompt sets used in Table 4 are not shipped/fetched, and the adversarial-editing pipeline the paper attributes to RECE's setup is not integrated into the OCE codebase.
• Toy-experiment code (Sec. 3.1) not shipped. No script implements the magnitude-scaling, neuron-wise rotation, or layer-wise rotation cases that underpin C1/C2; a reader cannot reproduce Fig. 2 from the repository.
• FLUX extension covers a single panel of Fig. 4. Only the Elon Musk (celebrity) FLUX demo is scripted; there are no FLUX object, style, or NSFW scripts, and no batch generator.
• Undocumented lamb hyperparameter. All celebrity scripts pass LAMB=10 to oce.py, but this parameter is neither documented in Appendix C nor swept in Table 12.

Recommended Changes

Essential

• Fix the notation errors in Sec. 3.2 and Sec. 5.2. Rewrite the direction relation for additive updates using an inequality (parallel to the magnitude term) so the prose ('primarily rotate directions') and the equation match, and choose one consistent statement — with the correct sign — for what the subspace-suppression objective minimises relative to the orthogonal complement of the anchor subspace.
• Rewrite the harmonic-mean definition of H_o explicitly. State H_o = 2·[(1 − Acc_e)^{-1} + Acc_s^{-1}]^{-1} with the 100 − Acc_e percentage convention next to Table 1, since this is the paper's headline aggregate and cannot currently be read from the prose.
• Report variance across seeds for the headline tables. Rerun Tables 1–3, 5, 6, 10, 11 with at least three seeds and report standard deviations or 95% intervals, so that small deltas (e.g., 'Person' vs. 'celebrity' anchor H_o in Table 11, SPEED / OCE runtime gap) can be judged.
• Broaden and de-confound the Sec. 3 toy experiment. Sweep several α values including small magnitude perturbations, report the Case C rotation angle (or ‖Q − I‖_F), match Case A and Case C in the induced ‖WC − W*C‖, run multiple concepts and seeds, and add a fourth case that perturbs inter-neuron angles while approximately preserving individual directions so C2 can be isolated from C1.
• Ablate over which cross-attention matrices are edited. Report Table 1 / Table 3 numbers for W_k only, W_v only, and W_k + W_v, to separate the effect of the orthogonal update from the effect of which parameters are touched, matching the two-matrix motivation in Sec. 4.
• Add quantitative comparisons against orthogonal / subspace baselines. Include CURE (Biswas et al., 2025), AdaVD (Wang et al., 2025), and SUMA (Nguyen et al., 2025) in Tables 1–4 or explain concretely why each setting is not comparable.
• Report SPEED's total (preprocessing-inclusive) runtime and rephrase the efficiency claim. Include the preprocessing steps SPEED needs and quote the total time; where the resulting gap is small, drop or soften 'clear efficiency advantages.'
• Align the released scripts with the paper's hyperparameters and anchors. Match celeb_100.sh (and the 10/50-celeb, object, style scripts) to the λ_e / λ_0 / λ_r values and anchor concepts stated in Appendix C.1 and Table 13, and remove or document the extra lamb parameter.
• Fix the blocking bugs in the K_0 scripts. Resolve the syntax error in compute_Cg.py, correct the NameError in compute_Cg_flux.py, and set sample_size consistently with COCO-30k's token count.

Suggested

• Add drivers and evaluators that compute the reported metrics directly. Ship a per-class object-erasure driver, a H_o / Acc_e / Acc_s aggregator, and objective-ablation flags for Table 5's three variants (including using build_preserve_subspace in the preservation term).
• Ship the toy-experiment script. A short, self-contained script that reproduces the three panels of Fig. 2 would materially strengthen the Sec. 3 motivation.
• Extend adversarial robustness beyond nudity. Rerun Table 4 for at least one non-nudity category (e.g., celebrity or artistic style under adversarial paraphrase) and report an ASR under a detector appropriate to that category, so the AT robustness claim is not tied to Nudenet alone.
• Extend the FLUX quantitative evaluation. Add object, style, and celebrity numbers for FLUX.1 dev with at least one DiT-specific baseline, and ship the corresponding training scripts for those panels of Fig. 4.
• Operationalise the anchor-selection heuristic. Replace 'some similarity but noticeable differences' with a concrete rule (e.g., a CLIP-text-embedding cosine window), reconcile it with the 'celebrity' anchor used in Table 13, and report H_o sensitivity to perturbations of the rule.
• Fix the local prose issues. Rename 'efficiency' to 'efficacy' in Sec. 6.1's Setup paragraphs; correct λ_p / Acc_r to λ_r / Acc_s in Appendix D.4; reconcile the Table 1 vs. Table 10 H_o averages (97.01 vs. 96.98); replace 'the additional three preprocessing' with the intended noun and, in the SOTA comparison, name the '8.32' row explicitly and move the '0.01% drop' sentence next to Table 1 rather than Table 3.