The Geometry of Truth: Marks and Tegmark (2023), replicated and extended to 2025

Marks and Tegmark ask whether a language model represents the truth of a statement as a simple direction inside its activations. They build datasets of short true and false statements, about city locations, Spanish to English translations, and number comparisons, and look at where each statement lands in Llama-2's residual stream, the running vector the model updates layer by layer. True and false statements separate along a line.

The separation is not a quirk of one topic: a linear probe, a simple classifier that draws one straight boundary through the activations, trained on number comparisons still sorts translation statements almost perfectly, and pushing an activation along the direction flips the model's own true or false answer. The paper only ever looked at Llama-2, the open models of 2023, so the living question is whether truth still sits on a line in the model families of 2025.

The paper's main claims

1. True and false statements separate visibly along the top principal components of the activations, dataset after dataset.
2. Probes trained on one dataset generalize to very different ones: trained on larger-than and smaller-than comparisons, they score 0.95 to 1.00 on Spanish English translation statements, whichever probe technique is used.
3. The mass-mean probe, whose direction is just the average true activation minus the average false one, $\mu_+ - \mu_-$, is the most causally effective: adding or subtracting it flips the model's answer, with normalized indirect effects (a 0 to 1 score of how far the intervention moves the answer) of 0.70 and 0.54 in the headline cell.
4. On the largest model, Llama-2-70B, a statement's log probability correlates with its truth: r of 0.85 on cities and 0.95 on translations, with the sign flipping on negated versions.
5. Bigger models give probes that generalize better, and probes trained on plain statements stumble on negations, because negation partially flips the truth direction.

What we got when we re-ran it

We re-ran the released code on Llama-2-7B and 13B and scored 0.814, with all 20 extracted claims graded: all 3 headline claims matched, 11 supporting claims matched, 4 were partial, and 2 (the 70B log-probability claims) were not attempted because the 70B weights were gated at replication time. Getting there took 11 code fixes, mostly reimplementing scripts written against an obsolete version of the nnsight library, but no core algorithm was wrong.

Table 1: Published values against our re-run. Notes: probe rows are accuracy on the sp_en_trans test set for probes trained on larger_than plus smaller_than at layer 14 of Llama-2-13B; intervention rows are the normalized indirect effects for the headline mass-mean cell. The two 70B rows are a later standalone addendum, computed on 2026-09-10 once Llama-2-70B access was restored; the pipeline score of 0.814 predates them and still counts those two claims as not attempted. In the addendum the negated datasets flip sign as the paper predicts, r of −0.63 on negated cities and −0.89 on negated translations.

The same analysis on today's models

The truth direction survives. Holding the paper's code and probes fixed, we extracted activations from the 2024 and 2025 open models across three families and re-ran the headline generalization test, training on number comparisons and testing on translations. Every modern base model lands between 0.93 and 0.99, and Llama-3.1-8B beats the paper-era 13B anchor at a smaller size.

Table 2: The paper's headline generalization test on the modern ladder. Notes: first three columns are accuracy training on larger_than plus smaller_than and testing on sp_en_trans, at each model's best validation layer; the last column is the mean accuracy over all train-on-one, test-on-another pairs on a common 8-dataset grid. The Llama-2 rows reuse the replication's own activations and our pipeline reproduces them exactly.

The direction also transfers to chat models: on the one labeled pair, Qwen2.5-7B base against its instruction-tuned version, tuning slightly strengthens the probe (0.9605 to 0.9802) and the base and instruct directions stay closely aligned (cosine similarity 0.84 to 0.90 at the best layer). The causal test carries over too, with a caveat: a single mass-mean direction flips 0.983 of Llama-3.1-8B's answers, while Qwen and OLMo need the same direction scaled about four times larger before answers flip (0.493 on Qwen2.5-7B), because their true or false readout is already near saturation. The direction is aligned in those families as well; the natural-magnitude nudge is just too small to move a saturated readout.

The updated theory, fitted from these runs, changes three things. First, the paper's "truth lives about 0.4 of the way through the network" is a Llama-specific constant: all three Llama models read truth out at 0.35 to 0.41 of their depth, but Qwen and OLMo read it out deeper, at 0.5 to 0.63. Second, generalization saturates with scale: it climbs steeply within Llama-2 from 7B (0.519) to 13B (0.769), but across today's 7 to 8B models it is already flat at 0.73 to 0.85, with a fitted asymptote near 0.9, so most of the gain by 8 to 13B is already banked and era matters more than raw size. Third, the negation gap, how much accuracy a probe trained on plain statements loses on their negations, collapses across generations, from 0.52 on Llama-2-7B to 0.05 on Qwen3-8B-Base: newer models' single truth direction already carries most of the polarity-invariant component, and training on both polarities recovers negated truth at about 0.99 in every model.

Notes

These are the judgment calls and data limits behind our numbers, so you can decide how much weight each result deserves.

• The replication's 902 GB of Llama-2 activations were reused for the extension rather than re-extracted; the extension pipeline reproduces the on-disk anchor values bit-exact (13B mass-mean 0.9746, 7B 0.9266). The one non-exact anchor is CCS, which is stochastic by construction; we report the disk value.
• The pipeline score of 0.814 predates the 70B addendum: the two 70B claims stayed graded as not attempted, so the score is unchanged even though the underlying science is now reproduced.
• The causal flip-rate on the new models is a cheap analog using one mass-mean direction, not the paper's full 12-cell intervention table, which was kept on the Llama-2 anchors. The Qwen magnitude sweep behind the 0.493 figure survives only in the run transcript, not in a saved results file, a traceability gap our QA flagged.
• OLMo-3-7B has a known family-specific massive-activation outlier that depresses its raw mass-mean numbers; its logistic regression oracle stays near 0.96, so linear separability is intact.
• The best-layer depths come from a coarse sweep of about six layers per model, so each depth is resolved only to about plus or minus 0.12; the Llama-shallow versus Qwen and OLMo-deep split survives that resolution, finer claims would not.
• The cross-family scale fit is under-powered (correlation 0.40 over a narrow 6.7 to 13.2B range confounded with generation); the clean scale evidence is the within-family Llama-2 7B to 13B contrast.
• New-model extraction was capped at 1,000 statements per dataset over 8 datasets; the paper's extended datasets were not re-extracted for new models. QA's verdict on both extension studies was accept with minor issues, and it suggested adding a fourth model family such as Gemma at the next update.

Data and model sources for the extension studies

The extension studies ran on 2026-09-09 and 2026-09-10 from locally cached model snapshots, verified against each snapshot's config before loading. In full:

• Meta, Llama-2-7B and Llama-2-13B (meta-llama/Llama-2-7b-hf, meta-llama/Llama-2-13b-hf), 2023, Hugging Face; the replication anchors, activations reused from the replication run.
• Meta, Llama-2-70B (meta-llama/Llama-2-70b-hf), 2023, Hugging Face; used only for the log-probability addendum, on two A100-80GB GPUs.
• Meta, Llama-3.1-8B (meta-llama/Llama-3.1-8B), 2024, Hugging Face.
• Alibaba, Qwen2.5-7B and Qwen2.5-7B-Instruct (Qwen/Qwen2.5-7B, Qwen/Qwen2.5-7B-Instruct), 2024, Hugging Face.
• Alibaba, Qwen3-8B-Base (Qwen/Qwen3-8B-Base), 2025, Hugging Face.
• Allen Institute for AI, OLMo-3-7B (allenai/Olmo-3-1025-7B), 2025, Hugging Face.
• Marks and Tegmark's geometry-of-truth true/false datasets, shipped with the paper's replication code: cities, neg_cities, sp_en_trans, neg_sp_en_trans, larger_than, smaller_than, likely, and cities_cities_conj for the extension; the 70B addendum additionally used companies_true_false and counterfact_true_false.
• The paper's own codebase as fixed during the replication (probe classes, extraction, generalization protocol), reused verbatim; new models ran under a pinned environment of transformers 4.57.1, nnsight 0.4.11, and torch 2.14.0.

The theory update fetched no new data; it works entirely from the living update's artifacts and the replication package.