On the Limits of LLM Adaptability: Impact of LLM Pre-Training on Annotation Task Performance

Summary

The paper studies a real and under-examined question: when a practitioner writes a task definition for zero-shot LLM annotation, how much does that definition actually govern the model's behaviour, versus a concept the model has already internalised through pre-training and RLHF? The authors operationalise this with a new metric, Definition-Specific Familiarity (DSF), which is the cosine similarity between the model's own free-text description of a concept and the dataset's official definition, averaged across six sentence encoders. Across 9 instruction-tuned LLMs and 5 toxicity datasets (plus one Jigsaw robustness check and two non-safety datasets for generalisation), they show three main results: (1) DSF is positively partially correlated with zero-shot accuracy after controlling for dataset (+0.41, N=54), while three text-memorisation metrics — ROUGE-L, BERTScore, and embedding cosine similarity — are not; (2) most zero-shot errors resist correction: an aggregate rescue rate of 34.8% under prompted conditions, with high-confidence errors especially sticky, and only 18.7% pooled rescue after a three-turn history-aware sequence; and (3) misaligned definitions shift prediction bias systematically with definition scope, yet self-reported confidence is essentially flat, so calibration cannot flag definition errors. The reframing from contamination / memorisation toward concept alignment is the paper's main conceptual move and is well-motivated relative to the LLM-as-annotator literature. The empirical design is thorough — mixed-effects regressions clustered on message, an extended-model tier, an out-of-domain generalisation appendix, and multiple encoder ablations — and the qualitative pattern is credible. The main limitations are conceptual and identification-level rather than execution-level: DSF is measured from the model's own generation, which entangles it with verbal / instruction-following capability; "domain" effects fully confound with dataset, definition wording, and label prevalence; and several central claims are slightly overstated ("any prompting strategy," "confirms," "however measured"). The released code covers prompts and single-pair DSF but ships no end-to-end inference or aggregation pipeline, so headline numbers are not independently checkable from the release alone.

Strengths

• Conceptual reframing. The shift from "is the test text memorised?" to "does the model's own concept boundary match the definition?" is a genuinely useful move for the LLM-as-annotator literature, and DSF operationalises it in a way that is cheap to run (no labelled data required to measure it).
• Encoder-robustness of DSF. Appendix E.1 reports per-encoder partial correlations and pairwise agreement (Table 21, Table 22) across six sentence encoders spanning MiniLM to OpenAI text-embedding-3-small; the +0.30 to +0.49 range and pairwise r above 0.87 make the DSF signal look encoder-agnostic rather than a lucky embedding choice.
• Non-safety generalisation. Appendix F replicates both the DSF-accuracy direction (partial r = +0.34 on irony + subjectivity, N=18) and decision stickiness (few-shot rescue 45.0% across 3,052 zero-shot errors) outside the toxicity domain, materially strengthening the "internalised priors" framing.
• Multi-turn stickiness protocol. The three-turn history-aware rescue experiment (Appendix D.1) directly rebuts the natural "maybe you just need iterative prompting" objection: pooled rescue only reaches 18.7%, and high-confidence rescue only 8.5%.
• Mixed-effects clustering. Modelling correctness with random intercepts for message and fixed effects for model, condition, and domain (Appendix C) is the right response to the repeated-measures structure of the design; scoping the regressions to the six open-weights core keeps statistical inference tied to a single design tier.
• Lexical-to-semantic memorisation spectrum. Reporting ROUGE-L, BERTScore, and Embedding Similarity as a progression from purely lexical to fully semantic memorisation is a clean and informative robustness check for the RQ1 negative result.
• Explicit limitations paragraph. The paper is upfront about correlational evidence, the inability to isolate training stage, and the difficulty of separating a "stickiness bug" from an intentional safety-training feature.

Weaknesses

• DSF may be confounded with verbal / self-description capability. DSF is computed from the model's own free-text explanation of the concept scored against the dataset definition. A model that writes more coherent, instruction-following prose will produce a higher DSF regardless of whether it has a "better internalised concept," and that same capability plausibly drives annotation accuracy. Controlling for input NLL (text complexity of the classification input) does not address this — NLL is a property of the input, not of the model's generation. Without a control on generation quality (e.g., DSF on a matched "nonsense concept" acting as a placebo, or a paraphrase-DSF held constant), the +0.41 partial correlation is compatible with a plain capability story.
• Domain effects are not identified. The Table 5 / Table 11 mixed model reports "Gaming vs Forum OR = 13.85," "Social Media OR = 5.45," but each domain corresponds to a single dataset. Base rate (42.5% vs 9.6%), definition wording, message length, and dataset all move together with the domain label, so the coefficient does not measure a domain effect at all — it measures a dataset effect. The DSF narrative elsewhere in the paper actually predicts this (broad-definition GameTox is aligned with model priors), so the two threads sit in tension.
• BERTScore encoder is inconsistent between Section 3.1 and Appendix B.2. Section 3.1 states BERTScore uses "DeBERTa-XL contextual token embeddings" (confirmed in Appendix E.3 as microsoft/deberta-xlarge-mnli), but Appendix B.2 asserts that both BERTScore and Embedding Similarity "use the all-MiniLM-L6-v2 encoder — the same encoder as DSF." These are architecturally incompatible; the encoder-choice-controlled framing survives only for Embedding Similarity vs DSF.
• Several universal / causal claims overreach. "Nearly two-thirds of zero-shot errors resist correction through any prompting strategy" (abstract) is a union claim over strategies but 34.8% is an average per condition; "This confirms that DSF captures semantic alignment rather than simple processing fluency" treats an NLL control as decisive; "confirming that the performance ceiling reflects model-internalized priors rather than suboptimal prompt design" treats a single DSPy null result (with cited Llama regressions) as ceiling-establishing; and "text reproduction itself, however measured, does not predict annotation performance" is stronger than a design in which all three metrics share a single continuation procedure can license.
• Narrow/medium/broad definition scope is author-coded and unvalidated. The narrow/medium/broad taxonomy (Table 9) is central to the misalignment findings and the prediction-bias story, but is assigned by the authors on textual reading with no independent rater, IAA, or operational rule.
• No fine-tuned baseline anchors the rescue-rate ceiling. The ~35% rescue rate is framed as a limit of prompt-based steering, but there is no supervised (labelled-data) reference point for what fraction of these errors is fundamentally correctable. Without one, "stickiness" cannot be distinguished from ordinary task difficulty on the same instances.
• Calibration argument relies entirely on verbalized 0-100 confidence. The paper cites prior work showing verbalized confidence is prompt-shaped and unstable, yet the whole RQ3 conclusion rests on a single elicitation template applied uniformly. An alternative elicitation (paraphrase ensemble, or token-probability confidence on the two base models where it is available) would strengthen this.
• Interpretive overreach on Llama-70B. The reading of Table 12's Alignment × Llama-3.1-70B OR = 1.55 as "larger models have stronger priors that are harder to override" is drawn from a single 70B model in the core tier and cannot separate size from base capability, instruction-tuning recipe, or dataset composition.
• Multi-turn pooled rescue is distorted by Llama-3.1-8B's 78.8% refusal attrition. Table 20's Pooled row folds in a model whose numerator/denominator use only ~21 of the sampled 100 errors, and the non-refused subset is not verified to be random.
• Numeric mismatch on Fox News confidence. Section 4.3's "91.2%" for the highest-confidence Fox News hate speech condition does not match the 90.6% shown in Tables 7 and 8; the qualitative claim is fine but the specific number needs reconciliation.

Reproducibility & code

Assessment is based on inspection of the linked repository (paper-code mode; nothing executed) plus the veritas coverage-and-risk assessments per claim. In short: the code that is released is fine as far as it goes, but the pipeline needed to produce any headline number is not shipped.

• No end-to-end pipeline. The repository contains prompts.py (all classification, continuation, and rescue-turn templates) and rq1_familiarity/dsf.py (single-pair DSF). It ships no code to run classification against any model API, parse verbalized confidence, compute rescue rate P(correct | prompted, ZS-wrong), aggregate the Table 3 accuracy/AUC matrix, or fit any of the four mixed-effects models in Appendix C. The rq2_decision_stickiness README states the implementation is "planned for a future release." Every headline value in the paper (34.8% rescue, +0.41 DSF partial r, all Table 5 ORs, Figure 3, Figure 4) would need to be independently reimplemented before any spot-check is possible.
• consensus_dsf() in the code averages five encoders, not the paper's six. rq1_familiarity/dsf.py averages MiniLM + MPNet + BGE-large + E5-large + Instructor-large but drops OpenAI text-embedding-3-small. The paper's consensus DSF (Tables 4, 21) is over all six. A code-only recomputation will therefore not match the +0.41 and +0.74 headline correlations exactly.
• DSPy configuration is undocumented at code level. The DSPy Optimized and DSPy Aligned columns of Table 3 anchor the "performance ceiling reflects model-internalized priors" claim, but no DSPy program is shipped, and Appendix B.1 does not name the optimizer (BootstrapFewShot vs BootstrapFewShotWithRandomSearch vs MIPRO), number of trials, judge, seed, validation-pool sampling, or how many demonstrations end up in the final prompt.
• 1,000-instance subsampling seed is not disclosed. All aggregates depend on the same random 1,000-instance sample per dataset-model pair. Without the seed or the sampled index list, an independent run lands on a different subset with different per-cell accuracies.
• None of the five toxicity datasets are shipped, and per-dataset column mappings and dataset versions are not pinned. Reproducers must fetch Jigsaw (Kaggle), GameTox (NAACL), Twitter Hate, OLID, and Fox News from originals, several of which have restrictive access.
• Refusal-detection heuristic is undocumented. Table 20 reports 78.8% attrition for Llama-3.1-8B in the multi-turn experiment; the rule that flags a response as a refusal (versus an unparseable label) is not shown as code, and the same heuristic presumably affects the primary rescue-rate analysis.
• No mixed-effects fitting code. The four Appendix C models (BinomialBayesMixedGLM formulas, reference categories, random-effects specifications, variational seeds) are described in text but not implemented in a runnable script.
• No figure code for the two central plots. Figures 3 and 4 have no generation script and the paper does not tabulate bin edges, minimum bin counts, or the CI method for the shaded bands, so the visual claims (51.8% peak, 20.8% high-conf rescue) cannot be verified from the release.

Recommended Changes

Essential

• Add a verbal-capability control for DSF. To defend the DSF-is-alignment interpretation, add a placebo elicitation (e.g., a nonsense-concept DSF using the same six encoders, or DSF against a random-permuted definition) and show that only the on-topic DSF predicts accuracy. This addresses the confound raised in Weaknesses under "DSF may be confounded with verbal / self-description capability."
• Reframe the "domain" effects. Replace "Gaming is 14 times easier than Forum" language with dataset-level effects, or add a design that varies domain independently of dataset (e.g., two datasets per domain), so the OR = 13.85 is not read as identifying a domain effect. Ties to "Domain effects are not identified" in Weaknesses.
• Reconcile the BERTScore encoder. Either fix Appendix B.2's claim that BERTScore uses MiniLM to say DeBERTa-XL (consistent with Section 3.1 and E.3), or, if BERTScore was in fact run with a MiniLM-backed variant, document which BERTScore implementation and drop the encoder-choice-controlled framing.
• Soften three universal / causal claims. In the abstract, replace "resist correction through any prompting strategy" with an average-per-condition framing; in Section 4.1, change "This confirms" to "is not attributable to text-complexity fluency"; in Section 4.2, change "confirming" for the DSPy null to "consistent with"; in Appendix E.3, change "text reproduction itself, however measured" to "as measured by three complementary scoring functions over the same continuations."
• Release an end-to-end pipeline. Ship at least (a) a classification / confidence-parsing driver, (b) a rescue-rate script, (c) a consensus-DSF wrapper that includes the OpenAI encoder, (d) a DSPy program with named optimizer + seed + trial count, and (e) a regression script that fits the four Appendix C models from a per-instance parquet. Ties directly to the Reproducibility & code section.
• Disclose the 1,000-instance subsampling seed (or ship the sampled index list per dataset). Small but load-bearing.

Suggested

• Add a fine-tuned or supervised reference for rescue rate. A single labelled-data classifier (even a 100-example probe) per dataset would let readers see whether ~65% of zero-shot errors are truly "unreachable" or merely unreachable via prompting.
• Report per-model variance for the pooled multi-turn rescue. Exclude the Llama-3.1-8B row (78.8% refusal attrition) from the Pooled average, or weight it by non-refused N, and document the refusal-detection rule.
• Show calibration under a second confidence elicitation. A paraphrase-ensemble or a token-probability confidence for the base models (as in Appendix E.2 for Min-K%) would guard against the calibration finding being an artifact of a single verbalized template.
• Validate the narrow / medium / broad scope taxonomy with independent rater agreement, or replace it with an operational rule (e.g., protected-attribute mentions), given how much of RQ3 hinges on it.
• Reconcile the 91.2% vs 90.6% Fox News confidence number in Section 4.3 vs Tables 7-8, and add the two middle-scope conditions to the prediction-bias summary ranges instead of dropping them.
• Report attenuation-of-partial-r under aligned definitions with a bootstrap CI for the difference from +0.41 → +0.25, so the "variance compression" mechanism has an interval estimate rather than a p = 0.085 single-point inference.
• Tighten the non-safety comparison. Frame the 45.0% (non-safety) vs 34.8% (toxicity) rescue rates as a substantive difference (non-safety is easier to rescue) rather than "comparable in magnitude."