Function Vectors: Todd, Li, Sharma, Mueller, Wallace and Bau (2024), replicated and extended to 2025

Show a language model a few examples like "hot becomes cold, big becomes small" and it catches on and gives the antonym of the next word; that is in-context learning. Todd and coauthors ask where the task itself lives inside the network, and find it in a small set of attention heads, the little parallel units in each layer that decide which earlier words to read from. Averaging the outputs of the ten or so heads that matter most gives a function vector: a compact activation pattern that carries a task like "give the capital of this country" and can be pasted into the model's hidden state to trigger the task with no examples at all, which is what zero-shot means.

To find those heads the paper uses causal mediation: swap a head's averaged output into a corrupted prompt and measure how much that alone repairs the answer. On GPT-J, a 6-billion-parameter model from 2021, inserting the function vector lifts mean zero-shot accuracy across six word tasks from 5.5 to 57.5 percent, and the trick works best when the vector is inserted about a third of the way through the network's layers. We re-ran the paper on its own model, then asked whether any of this survives in the models of 2023 to 2025.

The paper's main claims

1. Averaging the outputs of the top ten or so causally important attention heads over in-context prompts yields a function vector that triggers the task zero-shot: GPT-J's mean top-1 accuracy on six tasks rises from 5.5 to 57.5 percent, and from 39.1 to 90.8 percent on prompts whose example labels have been deliberately scrambled.
2. The same recipe works in other models: GPT-NeoX-20B reaches 57.1 percent zero-shot and Llama-2-70B reaches 83.8.
3. The effect is layer-dependent: it peaks when the vector is inserted around one third of network depth and collapses in late layers.
4. About ten heads suffice, and several of them are induction heads, heads that copy patterns they have already seen in the prompt.
5. The vector is more than its vocabulary readout (a synthetic vector matching its decoded word distribution does much worse), and it keeps working in free-form natural text and across unusual prompt templates.

What we got when we re-ran it

We re-executed the authors' published code on GPT-J from scratch. Of 16 extracted claims, 11 were graded: 3 matched, 8 partial, none contradicted, for a score of 0.625. The remaining 5 (appendix visualizations, vector composition, and results needing then-gated Llama-2 models) were out of the plan's scope and excluded rather than counted as failures. Partial here mostly means the numbers landed close but outside the strict tolerance band, or a claim about a figure was verified from the underlying numbers because the plot itself was never rendered; the paper's story held up throughout.

Table 1: The paper's core intervention on GPT-J. Notes: mean top-1 accuracy in percent over the six core tasks, function vector inserted at layer 9, our runs averaged over three seeds; shuffled-label prompts carry examples with scrambled answers.

Table 2: Zero-shot function-vector accuracy per task, GPT-J at layer 9. Notes: five of six tasks land within about two standard deviations of the published value; english-french replicates far above the paper, a deviation stable across all three seeds (see Notes).

The supporting mechanics reproduced too: the three flagged induction heads reappear in our top-10 head list with prefix-matching scores of 0.545, 0.569 and 0.333 against the paper's 0.49, 0.56 and 0.31, the vocabulary-reconstruction control behaves as published, and accuracy plateaus at roughly ten heads.

The same analysis on today's models

We carried the paper's exact pipeline forward across an era ladder of open models from 2021 to 2025, spanning three families and the industry's shift from multi-head attention, where every head keeps its own memory of the context, to grouped-query attention (GQA), where heads share it to save memory. The comparison uses three of the six tasks (antonym, country-capital, present-past) under a compute cap, with the GPT-J anchor recomputed on the same three tasks.

Table 3: Zero-shot function-vector accuracy, 2021 to 2025. Notes: mean top-1 accuracy on the three-task subset at each model's best insertion layer and at the paper's canonical one-third-depth layer; GPT-NeoX is reused from the replication and covers antonym only; Qwen3-8B's mean is dragged down by a total failure on country-capital (5.1 percent, equal to baseline) while its other two tasks are healthy at 48.4 and 81.4 percent; new models run in bf16, single seed.

The survival answer is yes. In every model tested, summing the mean outputs of a handful of high-effect attention heads still produces a vector that triggers the task from a near-zero baseline, and the newer models are mostly stronger: Llama-2 at 92.2 percent and OLMo-2 at 89.3 against the GPT-J anchor's 65.9. Head localization is intact too; the top heads stay concentrated in the first two thirds of the layers everywhere (0.61 to 1.00 of them).

But the paper's rule for where to insert the vector breaks in the Qwen family. GPT-J, Llama-2 and OLMo-2 peak at 0.26 to 0.37 of depth, right where the paper says. Qwen2.5 and Qwen3 peak at 0.64 and 0.55 of depth, and inserting at the paper's one-third-depth layer collapses their accuracy to about 6 percent while the true peak stays high. And one cell fails outright: Qwen3-8B's country-capital function vector does nothing at any layer, a real failure rather than a bug (see Notes).

The updated theory

A follow-up re-analysis of the ladder asked what actually governs the effect, and the answer reshapes the paper's claims without overturning them.

• How strong the effect is depends on model family and pre-training, not on year or size. Peak accuracy sits in the 45 to 92 percent band with no trend over time (about one point per year, R² of 0.01) and a weakly negative trend with parameter count.
• Where to insert depends on the attention architecture. Across the four MHA models, three families and five years, the peak sits at 0.315 of depth give or take 0.05, confirming the paper's one-third heuristic; the GQA Qwen models sit at roughly 0.59, about twice as deep, so no single depth rule fits the whole ladder.
• How many heads you need also tracks the architecture: 8 to 11 for the MHA models, matching the paper's plateau of about ten, but 25 to 27 for the Qwen models, where the task is spread across most of the heads.
• The induction-head connection fades. The top heads are enriched for induction behavior in every model except OLMo-2, but the enrichment falls from a ratio of 4.2 in GPT-J to around 1.2 to 2.0 in the newer models, and OLMo-2 shows none at all.

In short: the function vector is a real, causal, transferable object across families and the MHA-to-GQA shift, but the paper's tidy universals, one third of depth and ten heads and induction heads, are properties of the 2021-era MHA design rather than of function vectors themselves.

Notes

These are the judgment calls and data limits behind our numbers, so you can decide how much weight each result deserves.

• Our replication first scored 0.5 because the run had been truncated after 2 of its 10 steps; re-run with all steps mandated, the six blocked claims recovered and the score settled at 0.625. The recovery reflects the harness, not the paper.
• Nothing was contradicted. The five uncounted claims were never targeted by the replication plan (appendix figures, vector composition, then-gated Llama-2 models), and most partials are near-misses outside a strict 5 percent band.
• The one odd replication cell is english-french at 70.3 versus the published 44.7, consistent across all three seeds; the direction favors the paper's thesis, but the gap is unexplained.
• The Qwen3-8B country-capital failure was adversarially QA-checked and confirmed real, not a code bug: a shape guard verified the head decomposition on every model, and the same model's antonym and present-past vectors work fine. Why that one task fails on that one model was not probed.
• The replication loaded GPT-J in fp16 and the extension ran new models in bf16, both disclosed budget choices; the accuracy metrics are rank-based and robust to this, but exact magnitudes could shift at the margins.
• Extension limits: three tasks not six, one seed, all base models (the planned instruct rung, Llama-3.1-8B-Instruct, was gated and skipped), and GPT-NeoX enters as a reused antonym-only rung.
• One disclosed bookkeeping wrinkle: a stored summary file records Qwen3's peak-layer fraction as 0.361 by naively averaging in the failed task's spurious layer-0 peak; the reported 0.55 excludes that task, and QA judged the choice correct.

Data and model sources for the extension studies

The extension studies pulled model weights from Hugging Face on September 9 and 10, 2026. In full:

• EleutherAI, GPT-J-6B (Hugging Face model EleutherAI/gpt-j-6b), 2021 release; all GPT-J numbers reused from our replication artifacts rather than recomputed.
• EleutherAI, GPT-NeoX-20B (EleutherAI/gpt-neox-20b), 2022 release; antonym-only results reused from the replication artifacts.
• Meta, Llama 2 7B (meta-llama/Llama-2-7b-hf), 2023 release, loaded from a local copy of the official weights.
• Alibaba Cloud, Qwen2.5-7B base model (Qwen/Qwen2.5-7B), 2024 release, downloaded from Hugging Face.
• Alibaba Cloud, Qwen3-8B base model (Qwen/Qwen3-8B-Base), 2025 release, downloaded from Hugging Face.
• Allen Institute for AI, OLMo-2-7B base model (allenai/OLMo-2-1124-7B), 2025 release, downloaded from Hugging Face.
• In-context task datasets: the six core word-task files (antonym, capitalize, country-capital, english-french, present-past, singular-plural) shipped with the paper's public replication code; the extension used the antonym, country-capital and present-past tasks.

The theory update fetched no new data or models; it works entirely from the living update's artifacts and the replication anchors on disk.