The Linear Representation Hypothesis: Park, Choe and Veitch (2024), replicated and extended to 2025

People had long noticed that language models seem to store concepts as directions, the way "king minus man plus woman" lands near "queen". Park, Choe and Veitch made that folklore precise. They showed that binary concepts like male⇒female or French⇒Spanish really do live as linear directions in a model's output space, and they proposed a better ruler for that space: a causal inner product, which is the ordinary dot product taken after rescaling the space by the covariance of the model's word vectors. Under this ruler, concepts that can vary independently of each other (causally separable ones) become approximately orthogonal, meaning their directions sit at right angles and do not interfere.

That matters because most interpretability work quietly assumes some inner product, and the paper argues the naive Euclidean one is the wrong choice. But everything was demonstrated on a single 2023 model, LLaMA-2-7B. We re-ran the paper's own code from its published repository, then took the same geometry to a ladder of five open base models spanning 2023 to 2025 and asked whether the finding survives.

The paper's main claims

1. Almost every binary concept tested (26 of 27) has a linear subspace representation: differences between counterfactual word pairs project strongly onto the concept direction, far more than random word pairs do. The sole exception is thing⇒part.
2. Under the causal inner product, causally separable concepts are approximately orthogonal; the Euclidean inner product only partly achieves this.
3. The directions are causally usable: adding $\alpha$ times the male⇒female direction to the representation of "Long live the" flips the top prediction from king to queen by $\alpha = 0.2$, and pushes king out of the top five by $\alpha = 0.3$.
4. A sanity check holds: over the whole vocabulary, projections onto a causally separable pair of directions are uncorrelated, while a non-separable pair (two verb inflections) is clearly correlated.

What we got when we re-ran it

We re-ran the authors' repository end to end on LLaMA-2-7B and scored 1.00: all 11 graded claims matched, with one claim (a Gemma-2B appendix figure the repo ships no code for) out of scope rather than failed. The code needed only six fixes, five of them routine environment drift and one a float32 cast that restores, rather than alters, the authors' original numerical path. A genuinely reproducible paper.

The core object is the whitened word representation

$g = \gamma \, \mathrm{Cov}(\gamma)^{-1/2},$

where $\gamma$ is the model's unembedding matrix (one output vector per word); the causal inner product of two directions is just the Euclidean inner product of their images in $g$-space. Orthogonality is then measured as the mean absolute inner product between the 27 unit-normalized concept directions, off the diagonal, where 0 would be perfectly orthogonal.

Table 1: The paper's headline results against our re-execution of the published archive. Notes: the one concept without a linear subspace is thing⇒part in both columns. The paper states the orthogonality claims through heatmap figures rather than summary numbers, so its column there is qualitative; the italic values are the means our replication computed from the produced concept directions (medians 0.012 and 0.029). The intervention result also matches the paper's stronger form, king leaving the top five by $\alpha=0.3$, and both vocabulary scatters match the paper's Figure 13.

The same analysis on today's models

We then ran the identical geometry on five base models, reusing the paper's word pairs verbatim and changing only the model-loading code: Llama-2-7B (2023), Llama-3.1-8B (2024), Qwen2.5-7B (2024), Qwen3-8B-Base (2025) and OLMo-3-7B (2025). As a gate, recomputing Llama-2-7B in the new pipeline reproduced the replication's 0.046 / 0.069 exactly before we trusted any other model.

The short answer is that the hypothesis survives. On every model the causal inner product is strictly more orthogonal than the Euclidean one, essentially all viable concepts keep their linear subspace (Llama-2's thing⇒part exception disappears in every newer model), the sanity check holds identically (separable $|r| \le 0.03$ against non-separable $r \approx 0.37$ on all five), and the male⇒female intervention still flips king to queen on all five. What changes is the size of the advantage.

Table 2: The paper's orthogonality anchor across the 2023 to 2025 ladder. Notes: means are over the off-diagonal absolute inner products of all 27 concept directions; the ratio is the Euclidean mean divided by the causal mean, so it is the size of the causal advantage; subspace counts use each model's own viable concepts (at least 10 single-token pairs) with a Cohen's d threshold of 1.

The paper's roughly 1.5x to 2.5x orthogonality advantage holds up in the LLaMA lineage (1.52 and 1.50) but drops to about 1.25x for Qwen and OLMo. The advantage is real everywhere, just smaller outside the family the paper studied.

What sets the size of the advantage

A follow-up fit over these measurements asked whether the advantage tracks scale, era or family. The causal off-diagonal itself is essentially flat across the ladder (slope +0.001 per year, $R^2 = 0.06$), and the ratio is uncorrelated with parameter count (correlation −0.09). What moves it is family: the same-era Llama versus Qwen gap in the ratio is about 4.8 times larger than a within-family generation step, and OLMo-3 lands with Qwen, not with its 2025 cohort. So the updated claim reads: the causal inner product keeps concepts approximately orthogonal on every 2023 to 2025 base model, and its advantage over the Euclidean one is set by model family, not by scale or calendar year.

That runs against the paper's own Appendix D.2 conjecture, which predicted the Euclidean inner product would "somewhat work" on LLaMA-2 but fail on newer or different models. We find the opposite direction: the Euclidean metric draws closer to the causal one on the newer non-LLaMA models (OLMo-3's Euclidean off-diagonal, 0.062, is actually lower than Llama-2's 0.070), so the causal advantage is largest exactly where the paper measured it.

Notes

These are the judgment calls and data limits behind our numbers, so you can decide how much weight each result deserves.

• The replication score covers 11 of 12 extracted claims; the twelfth (the Gemma-2B appendix figure) had no code path in the repo and was out of scope, not contradicted. The model-ladder study supersedes it by adding newer models directly.
• The extension applies a viability gate the paper did not (a concept needs at least 10 single-token pairs under each model's tokenizer), which leaves Llama-2 with 22 viable concepts and drops 5 morphological ones. That is why its subspace count reads 21/22 where the paper reads 26/27; the all-27 orthogonality anchor is unaffected and reproduces exactly.
• Five models is five data points. Every fitted trend has a negative leave-one-out $R^2$ and a bootstrap slope interval that includes zero, so the theory update is a descriptive fit, honestly capped, not a scaling law. Era is also confounded with family (2023 is Llama-only, 2025 is Qwen and OLMo only) and with tokenizer coverage.
• The optional measurement probe (forward passes over Wikipedia context pairs) was not run, to stay inside the GPU budget; the optional intervention diagnostic was run and holds on all five models.
• Models load in bfloat16 but all geometry (covariance, eigendecomposition, $g$) is computed in float32, matching the replication's numerical path.
• An independent QA pass verified every number on this page against the on-disk artifacts (verdict PASS). Its one forward-looking note: the ladder tops out at 2025 models, and the family claim rests on three families, so the next update should add a fourth (Gemma-class) family.

Data and model sources for the extension studies

The extension studies ran on September 9, 2026, entirely offline from locally cached model snapshots (verified the same day); no new datasets were collected. In full:

• Meta, Llama 2 7B base (meta-llama/Llama-2-7b-hf, 2023), from a local Hugging Face snapshot.
• Meta, Llama 3.1 8B base (meta-llama/Llama-3.1-8B, 2024), from a local Hugging Face snapshot.
• Alibaba, Qwen2.5 7B base (Qwen/Qwen2.5-7B, 2024), from a local Hugging Face snapshot.
• Alibaba, Qwen3 8B base (Qwen/Qwen3-8B-Base, 2025), from a local Hugging Face snapshot.
• Allen Institute for AI, OLMo 3 7B base (allenai/Olmo-3-1025-7B, 2025), from a local Hugging Face snapshot.
• The paper's own replication package: the 27-concept counterfactual word-pair lists (1,998 pairs) and the paired Wikipedia context files, reused verbatim from the replicated codebase; the only code changes were model loading and a tokenizer-agnostic single-token lookup.

The theory update fetched no new data; it works entirely from the living update's saved measurements.