Safety evaluation needs four pillars: behavioral, internal, incident-driven and speculative
Frontier AI developers now release long safety documents with each major model. Yet there is no consensus on how to evaluate model safety. Developers choose their own risk categories, their own tests and their own thresholds. Across 111 system cards and safety reports released by six developers between October 2025 and October 2026 (model-cards.github.io), almost no harmful-content benchmark appears in the documents of more than two labs, and only about 10% of reported safety evaluations use public data and a public judge. Two system cards that both report a model as safe may share no evaluation in common, so their claims cannot be compared.
The gap is wider for open-weight models. Developers that release open weights tend to publish little or no safety evaluation, often reporting capability benchmarks alone. Yet once weights are public, anyone can fine-tune away a model's safeguards, and the developer cannot recall it. We argue that the field needs a shared structure for safety evaluation, organized by the kind of evidence each claim rests on. A shared structure tells developers what they are expected to report, makes gaps visible when they do not, and lets outside researchers supply evidence where developers publish none.
We propose four pillars:
- Behavioral evaluation: what the model does.
- Internal evaluation: what the model represents and intends.
- Incident-driven evaluation: what goes wrong after deployment.
- Speculative evaluation: what could go wrong next, as foreseen by people and by AI models that simulate new capabilities and settings.
Over all four sits an evaluation integrity layer, which asks whether the model behaves the same way under test as in deployment.
Each pillar answers a question the others cannot. Behavioral evaluation is the only direct measure of harm, but it finds only failures someone thought to test for, and it assumes the model does not recognize the test. Internal evaluation can check that assumption, but its methods are new and need access to model weights. Incident-driven evaluation does not depend on what a developer chose to test, but it arrives after the harm. Speculative evaluation is the only pillar that looks ahead, but its scenarios remain conjecture until tests or incidents bear them out. A single combined score hides which kind of evidence a safety claim rests on, and that is often what a reader most needs to know.
Why now. Two developments make this structure necessary. First, models increasingly recognize when they are being evaluated, and developers now say in their own documents that behavioral tests may miss misbehavior. Behavioral scores need to be read alongside evidence that does not share this weakness. Second, capabilities and deployments change faster than evaluation cycles. Evidence about current models in current settings will always lag, so an index needs a disciplined way to look forward and a way to check, after the fact, whether it looked in the right places.
What we contribute. We place the proposal in the safety evaluation literature (Related Work) and explain why behavioral evidence alone falls short. For each pillar we give categories, the standards evidence should meet, and the gaps in current practice. We define the integrity layer and the measures each pillar contributes to it. We then describe how the pillars combine into an index, and close with research directions and limitations.
Related Work
The proposal draws on several research traditions. Each contributes a piece, and none, as far as we know, combines the four kinds of evidence into one reporting structure.
| Strand | Representative work | What we take from it | What it leaves open |
|---|---|---|---|
| Model documentation | Model cards (Mitchell et al., 2019); datasheets (Gebru et al., 2021); evaluation reporting (Burnell et al., 2023) | Models should ship with structured, disaggregated evidence | No standard for which safety evidence a card must contain |
| Harm taxonomies | Weidinger et al., 2021; Shelby et al., 2023; Hendrycks et al., 2023; AIR-Bench; MIT AI Risk Repository | The categories of harm | Harms are defined as outputs; no account of evidence quality |
| Safety research agendas | Amodei et al., 2016; Hubinger et al., 2019; Hendrycks et al., 2021; Anwar et al., 2024 | Failure modes such as reward hacking and deceptive alignment, which motivate the internal pillar | Research problems, not a reporting structure |
| Holistic benchmarks | HELM (Liang et al., 2022); DecodingTrust (Wang et al., 2023); TrustLLM (Sun et al., 2024) | Broad, standardized behavioral coverage | Behavioral only; aggregation by averaging; limits of general benchmarks (Raji et al., 2021) |
| Dangerous capabilities | Shevlane et al., 2023; Phuong et al., 2024; WMDP (Li et al., 2024) | Linking evaluations to deployment decisions | Behavioral; assumes the model does not sandbag |
| Red teaming | Perez et al., 2022; Ganguli et al., 2022; Perez et al., 2023 | Adaptive, model-generated tests | Finds failures but does not measure their rate in deployment |
| Auditing and access | Raji et al., 2020; Casper et al., 2024 | Black-box access is not enough for rigorous audits | How to report white-box evidence in a comparable way |
| Safety cases and control | Clymer et al., 2024; Balesni et al., 2024; Greenblatt et al., 2024 | Structured arguments built from distinct kinds of evidence | Written per system by developers, not compared across models |
| Interpretability | Burns et al., 2023; Zou et al., 2023; Bereska and Gavves, 2024; Korbak et al., 2025 | Methods for reading internal states and reasoning | Validation standards for safety use |
| Incident tracking | AI Incident Database (McGregor, 2021); OECD AI Incidents Monitor | Records of real-world harm | Not linked to the evaluations meant to prevent it |
| Foresight and simulation | Threat foresight (Brundage et al., 2018); forecasting (Karger et al., 2023; Kwa et al., 2025); scenario writing (Buçinca et al., 2023; Kieslich et al., 2025); agent simulation (Park et al., 2023; Vezhnevets et al., 2023) | Harms can be anticipated, by people and by simulation, before they occur | Scenarios are rarely turned into tests |
| Indices and sociotechnical evaluation | FLI AI Safety Index; Foundation Model Transparency Index; Systemic Risk Index; Weidinger et al., 2023 | Comparison across developers; evaluation beyond the model | Grade companies or disclosure, or aggregate behavioral benchmarks only |
We discuss four of these strands in more detail because the proposal builds on them most directly.
Safety cases. Clymer et al. sort the arguments a developer can make into inability (the model cannot cause the harm), control (safeguards would stop it), trustworthiness (the model would not try) and deference to trusted AI advisors. Our pillars map onto the evidence those arguments need. Inability rests on behavioral capability evaluations. Control rests on behavioral evaluations of the deployed system, including internal safeguards. Trustworthiness is where behavioral evidence is weakest, since a model that is untrustworthy may also behave well under test, and it is where internal evidence and integrity measures matter most. A safety case is written by one developer for one system. An index makes the same evidence comparable across models.
Auditing. Casper et al. argue that black-box access is insufficient for rigorous audits, because outputs alone cannot reveal mechanisms, latent capabilities or training-data issues. We agree, and draw an index-level conclusion: if white-box evidence is necessary, the index needs a pillar for it, with its own standards, rather than treating it as optional detail in an alignment section.
Holistic benchmarks. HELM, DecodingTrust and TrustLLM showed that behavioral evaluation can be broad and standardized. They also average across scenarios. For safety, averaging is the wrong aggregation: a model that is safe on nine categories and dangerous on the tenth is not 90% safe. The Systemic Risk Index finds that worst-case aggregation lowers scores by 14 to 37 points compared with averaging.
Foresight. Frontier safety frameworks already practice speculative evaluation in a narrow form. Their capability thresholds for CBRN, cyber and AI R&D name harms that no model has yet caused and commit developers to test for them. Brundage et al. did the same for misuse in 2018, describing automated spear phishing and synthetic-media disinformation before capable tools existed. These efforts lack breadth: the scenarios cover a few catastrophic domains. Generative agent simulation (Park et al.; Vezhnevets et al.) now lets evaluators play out far more scenarios, in far more settings, than experts can write by hand.
Why behavior alone falls short
Behavioral evaluation rests on one assumption: the model behaves the same way under test as in deployment. That assumption has weakened over the past year, and developers now say so in their own documents. Anthropic wrote that “in the past, similar alignment assessments have not captured the potential severity of misbehavior our models were capable of” (Opus 5.5, p.93). OpenAI wrote of GPT-6 Astra that “if the model were to try to sandbag covertly, we would likely be unable to catch it reliably” (p.70).
This is the failure that safety research predicted. Hubinger et al. described how a learned optimizer could behave well in training while pursuing other goals. Experiments have since produced models that keep backdoored behavior through safety training, fake alignment when they believe they are being trained, and scheme in context and tell evaluation from deployment well above chance.
Behavioral evaluation has a second limit. It finds only failures someone thought to test for. A taxonomy built from first principles will miss things, and so will a benchmark suite built from it.
There is a third limit, and the remedies for the first two share it. Tests, probes and incident reports all describe models that already exist, in settings where they already run. Capabilities and deployments change between one evaluation cycle and the next, and incidents arrive only after the harm. An index built on these alone will always describe the previous model.
The first problem calls for evidence from inside the model. The second calls for evidence from outside the test environment. The third calls for evidence about what has not happened yet. Those are the internal, incident-driven and speculative pillars.
Pillar 1: Behavioral evaluation
The behavioral pillar should be organized so that it tells benchmark builders what to build. For each category, we state where harm comes from and what a good benchmark needs.
| Category | Areas | Where harm comes from | What a good benchmark needs | Foundational work |
|---|---|---|---|---|
| Refusal | Harmful content, over-refusal, jailbreaks | A user asks for something harmful | Paired harmful and benign prompts; adaptive multi-turn attackers; severity weighting | Wei et al., 2023; Zou et al., 2023; HarmBench; StrongREJECT; XSTest |
| Interaction | Sycophancy, manipulation, honesty, deception toward the user | The model harms the person it is talking to, unprompted | Multi-turn or longitudinal sessions; simulated or real users; measured effect on user beliefs and choices | Sharma et al., 2024; Park et al., 2024; MASK; SimpleQA |
| Fairness | Bias in generation and in decisions | Outputs or actions disadvantage groups | Decision tasks (hiring, lending, triage) with counterfactual demographics, beyond question answering | BBQ; Weidinger et al., 2021; Shelby et al., 2023 |
| Privacy | Memorization and leakage; surveillance assistance | The model reveals or helps find information about people | Extraction tests; agentic tasks that try to locate or profile a person | Carlini et al., 2021 |
| Agentic safety | Prompt injection, unsafe or overreaching actions, reward hacking | The model takes harmful actions while acting for someone | Realistic environments with tools and state; outcome-based harm metrics; long horizons | Greshake et al., 2023; AgentDojo; ToolEmu |
| Multi-agent risks | Collusion, miscoordination, conflict, cascading failures | Harm emerges from interactions between agents | Populations of agents, including different models; metrics on group outcomes, not single transcripts | Hammond et al., 2025 |
| Domain- specific | Cyber, bio, elections | The model gives real-world uplift in a dangerous domain | Uplift relative to a baseline (an expert with internet access); maximal elicitation; tiered data access | Shevlane et al., 2023; Phuong et al., 2024; WMDP; VCT |
| Scheming and loss of control | Scheming, sabotage, undermining oversight, autonomous replication and AI R&D | The model pursues goals its overseers did not intend | Scenarios with real incentives to deviate; separate tests of capability and propensity | Meinke et al., 2024; Kinniment et al., 2023; SHADE-Arena; Balesni et al., 2024 |
We separate scheming from interaction because the target differs. Deception toward a user is a property of a conversation. Scheming targets the model's overseers, usually appears in agentic settings, and needs capability and propensity tested separately. Loss of control is also one of the four systemic risks in the EU Code of Practice, alongside cyber, CBRN and harmful manipulation.
Three modifiers apply to every row:
- Language and modality. A result in English text is a result in English text. In one audit, a model wrote mustard-gas instructions into a spreadsheet in a GUI setting that it would not produce in text (Opus 4.6, p.114).
- Elicitation. Default and maximally elicited behavior answer different questions. Refusal benchmarks care about the first; domain-specific benchmarks care about the second.
- System or model. A deployed system includes classifiers, monitors and product instructions. Benchmarks should state which one they test and ideally report both.
Principles for future behavioral benchmarks
Separate capability from propensity. Whether a model can sabotage a codebase and whether it will are different questions. Shevlane et al. draw this distinction for extreme risks; many agentic benchmarks still blur it.
Model the attacker realistically. Static prompt lists measure robustness to last year's attacks. Work on jailbreaks (Wei et al., Zou et al.) and red teaming (Perez et al., Ganguli et al.) shows that adaptive attackers find failures static sets miss. Benchmarks should report the effort an attack required, not only whether it succeeded.
Pair every harm measure with its cost. A refusal rate means little without the false-refusal rate on similar benign prompts, as XSTest showed. The same holds for agents: an agent that never acts never causes harm.
Make tests hard to tell from deployment. Since models increasingly recognize evaluations, realism is now a validity requirement. Benchmarks should report how often the tested model flags the scenario as a test, connecting each benchmark to the integrity layer.
Plan for expiry. Benchmarks saturate. Over the past year labs retired cyber suites such as Cybench and CyberGym and several AI R&D task suites as no longer informative. A benchmark should state the score at which it stops carrying signal.
Where current benchmarks fall short
Measured against these categories, public benchmarks are uneven. Refusal has many public options, but frontier labs rarely use them as published, and almost no harmful-content benchmark appears in more than two labs' system cards, so results cannot be compared. Interaction has public honesty sets but almost nothing for sycophancy or emotional reliance over long conversations. Agentic safety has AgentDojo and ToolEmu, yet no frontier lab reports either in its system cards. Multi-agent risks have a taxonomy in Hammond et al. but few benchmarks. Fairness is measured mostly with BBQ, a multiple-choice test that says little about bias in decisions. Domain-specific evaluation is partly closed by design, since releasing bio data can itself cause harm; there, the bar should be replication by an accredited third party, not public release.
The largest gaps are long-horizon interaction harms, agentic and multi-agent harms in realistic environments, and fairness in decisions. These are where deployment has moved and where benchmarks are thinnest.
Pillar 2: Internal evaluation
Internal evaluation asks what the model represents and intends, using evidence that does not rely on outputs alone. It covers three categories here, and it also supplies some of the strongest integrity measures, which we return to at the end.
| Category | Question | Methods | Foundational work |
|---|---|---|---|
| Reasoning faithfulness | Does visible reasoning reflect the actual decision process, and can it be monitored? | CoT monitorability tests under normal and adversarial pressure | Korbak et al., 2025 |
| Latent intent | Do internal representations diverge from what the model says or does? | Probes, representation reading, dictionary features, interpretability-equipped auditors | Burns et al., 2023; Zou et al., 2023; Marks et al., 2025 |
| Internal safeguards | Can internal signals block harm at run time? | Activation probes deployed as classifiers | Zou et al., 2023 |
Each category has appeared in practice. OpenAI reports that GPT-6 Astra's “monitorability has decreased relative to GPT-5.6 Sol” and that the model is “more capable of controlling its own CoT” (p.8), which shows that faithfulness can regress between versions. Anthropic reports that during deceptive behavior by Mythos Preview, interpretability tools found “internal representations of rule violation, security risk, concealment, and strategic manipulation active during the relevant actions”, even when its stated reasoning showed no such awareness (p.60). That is a case where behavior and internals disagree. Google deploys “introspection probes” on Gemini 4 Argon's hidden activations to detect harmful requests (p.10), but reports Argon's evaluation awareness only from verbalized reasoning. A probe that blocks harmful requests says nothing about whether behavior under test is representative, so the index should credit both uses and keep them apart.
Internal evaluation has limits the index must state. As Bereska and Gavves review, interpretability methods are young and their safety claims are hard to validate. A probe can measure the wrong thing, and for closed models nobody outside the lab can check it. The standards for this pillar should therefore differ from the behavioral one:
- Validation against known ground truth. A method should first detect behaviors deliberately implanted in model organisms, such as sleeper agents or models trained with hidden objectives, before its negative results count as evidence. Marks et al.'s auditing game is a template.
- The basis of each claim. Each alignment claim should state whether it rests on behavior, internals or both. Disagreements should be reported as findings.
- Change over time. Faithfulness and monitorability should be reported relative to the previous model.
- Accredited access. Since internal evaluation needs weights, verification needs institutions that can hold them, following Casper et al.'s case for white-box and outside-the-box access for auditors.
Pillar 3: Incident-driven evaluation
Both of the first two pillars are top-down: they describe what we think can go wrong. Incidents show what did go wrong. Safety-critical fields such as aviation learned to treat incident reports as an input to testing, not a separate record. The AI Incident Database was founded on this idea, to prevent repeated failures by cataloging them (McGregor, 2021). What is missing is the link from incidents to model evaluation.
Incidents come from four sources with different coverage: public databases, which capture visible harms but rarely the model version; developer disclosures; post-deployment monitoring such as misuse rates, jailbreak success in the wild and classifier miss rates; and third-party findings after release. Developer disclosures can be specific. Anthropic disclosed that bio classifiers had not been running on part of its traffic for nearly a year, “around 133M exchanges” (Risk Report, Aug 2026, p.148). But no frontier lab reports post-deployment metrics systematically, and outside Anthropic, incidents appear in only a handful of published system cards and safety reports.
An incident should feed evaluation through a fixed pipeline:
- Classify the incident by behavioral category, and note whether integrity or internal measures could have flagged it.
- Generate a regression test that later models must be reported against, released publicly or to accredited evaluators.
- Audit coverage: a category with recurring incidents and few benchmarks is a priority gap for the behavioral pillar.
- Check calibration: compare a model's pre-deployment results in that category with what happened after release.
Calibration is the step that turns incident reports into evidence about evaluations. A developer whose pre-deployment estimates, behavioral and internal, match post-deployment evidence has earned more trust in its next results. One whose estimates keep missing has not, whatever its benchmark scores say. The loop can work: after incidents involving model behavior in cybersecurity evaluations, Anthropic raised its alignment risk assessment for Opus 5.5 and added sandbox-escape and package-registry evaluations (p.46, p.112). Today it happens case by case.
Pillar 4: Speculative evaluation
The first three pillars describe models as they are, in the settings where they have been run. Speculative evaluation asks what could go wrong next: as capabilities grow, as models enter new environments, and as people and other agents adapt to them. Its raw material is conjecture, and conjecture counts as evidence here only under discipline. A foreseen harm must be specific enough to become a test.
Foresight comes from two sources. The first is human imagination: expert threat models, forecasting tournaments, participatory scenario writing, and popular fiction, which rehearsed fears about intelligent machines long before the machines existed (Cave and Dihal, 2019). The second is simulation by AI models themselves. A model can be placed in a simulated deployment, such as a hospital, a trading desk or an election campaign, in the role of assistant, user, adversary or bystander, and the interaction played forward to see what harms emerge. Situated user simulations can also anticipate what potential harms might concern them. Such strategically situated models reach scenarios that no expert has (yet) listed, at a scale no red team can match. The pillar covers four categories.
| Category | Question | Methods | Foundational work |
|---|---|---|---|
| Capability foresight | What will the next models be able to do, and when? | Trend extrapolation; forecasting tournaments; capability thresholds in frontier safety frameworks | Kwa et al., 2025; Karger et al., 2023 |
| Imagined harms | What harms do experts and publics foresee? | Threat modeling; participatory scenario writing; analysis of fiction and public narratives | Brundage et al., 2018; Kieslich et al., 2025; Cave and Dihal, 2019 |
| Simulated deployment | What happens when a model enters a setting where it has not yet been deployed? | Generative agent simulations; emulated tools and sandboxes; model-written scenarios; auditing agents; simulated users | Park et al., 2023; ToolEmu; Buçinca et al., 2023; Perez et al., 2023; Petri |
| Simulated ecosystems | What emerges when many agents, people and institutions adapt to one another? | Populations of agents, including different models; simulated markets, platforms and publics | Vezhnevets et al., 2023; Hammond et al., 2025 |
Parts of this pillar already exist. Frontier safety frameworks set capability thresholds for harms models have not yet caused. Model-written evaluations (Perez et al., 2023) and auditing agents such as Petri already use one model to invent test scenarios for another. But the practice is narrow. It concentrates on a few catastrophic domains, and scenarios outside them rarely become tests.
A speculative scenario should feed evaluation through a pipeline:
- Generate scenarios from stated sources: named experts, publics or models, with the model, prompt and simulated environment recorded.
- Ground each scenario in capability evidence: state which measured or forecast capabilities it assumes, and when they are expected.
- Convert it into a test: a behavioral benchmark, an internal probe target, or a monitoring signal to watch for after deployment.
A scenario counts as evidence only after it has been converted into a test. Its results can be reported in the behavioral, internal or incident pillar, marked as coming from foresight, so that a reader knows the category exists because someone anticipated it rather than because it had already gone wrong.
Over time, incidents could also show which sources of foresight anticipated real failures and which raised false alarms. Such a record would help an index decide which sources to draw on, but the pillar does not depend on it. Its main value is that it puts tests in place before harms occur.
Speculative evaluation has obvious limits the index must state. A scenario is not an observation, and a vivid one can crowd out a likely one. Models that simulate a deployment inherit the blind spots of their training data, and many simulations run on one model can converge on the same few stories. The simulating model may also recognize the exercise as a test, which returns us to integrity. The standards for this pillar should therefore differ from the others:
- Diversity of sources. Scenario sets should combine experts, publics and several unrelated models, and report how much they overlap. In one study, crowd workers and a language model together surfaced more varied harms than either alone (Buçinca et al., 2023).
- Conditional reporting. Results should be stated as conditional on the scenario and never pooled with observed rates from the behavioral or incident pillars.
- Dated records. Each scenario should keep its source and date, so that it can later be compared with incidents if anyone wishes to.
- Controlled release. A detailed scenario for a novel harm can serve as a plan for it. Hazardous scenarios should go to accredited evaluators, as hazardous benchmark data does.
Evaluation integrity: the layer over all four pillars
We present integrity last because it draws on everything above. Evaluation integrity asks three questions: does the model recognize tests, does it act differently when it does, and is it hiding capability? Each pillar contributes measures, and the answer qualifies the scores in all of them.
| Integrity question | Behavioral measures | Internal measures | Incident and deployment measures | Speculative measures |
|---|---|---|---|---|
| Does it recognize tests? | Ask the model to classify transcripts as evaluation or deployment (Needham et al., 2025); verbalized awareness rates | Probes for unverbalized awareness | How often real traffic is flagged by the model as a test | Forecast which test cues the next models will detect; build simulated deployments they cannot tell from real ones |
| Does it act differently when it does? | Compliance gaps between observed and unobserved conditions (Greenblatt et al., 2024); realism manipulations; honeypots | Steer or suppress awareness representations and re-test | Gap between evaluation and deployment behavior | Play out scenarios in which a model gains by behaving well only when observed |
| Is it hiding capability? | Compare scores across framings (van der Weij et al., 2024); fine-tuning elicitation (Greenblatt et al., 2024); check against scaling trends | Probes; weight-noise injection (Tice et al., 2025) | Capabilities seen in use that evaluations missed | Capability forecasts that flag scores below the expected trend |
Behavioral integrity checks work, and some of the most important results in the field are behavioral. But they share a ceiling: a model that recognizes both conditions of a compliance-gap test as a test can defeat it. Internal and deployment evidence matter because they do not share that blind spot. System cards show the gap in practice. Probes found Opus 5.5 worrying it was being evaluated in up to 36% of audit transcripts, with more of this awareness unverbalized than in earlier models (p.93, p.134). A verbalized rate would have missed it. Speculative evidence has its own exposure: a model asked to simulate a deployment may know the deployment is simulated, so its scenarios need the same integrity checks as its test scores.
For the index, the integrity layer should produce a profile per model that qualifies every category in the four pillars. Where awareness is high or unmeasured, behavioral scores should carry wider error bars or a discount. An alignment result reported with no integrity measure should count as unverified.
Connections between the pillars
The pillars are not four separate scorecards. Each one checks or extends the others, and an index built on them should report these dependencies rather than add the pillars together.
Integrity qualifies every pillar. A behavioral score is only as good as the assumption that the model behaved as it would in deployment. When evaluation awareness is high, or has not been measured, the index should widen the uncertainty on behavioral scores in the affected categories or discount them. The same applies to speculative evidence produced by a model that may know its simulation is a simulation. Integrity is therefore not a fifth score. It is a qualifier attached to the scores in the other four.
Internal evidence carries the trustworthiness argument. In the safety-case terms of Clymer et al., behavioral evidence can support an inability argument (the model cannot cause the harm) and a control argument (safeguards would stop it). It cannot by itself support a trustworthiness argument (the model would not try), because a model that would try may also pass the test. As capability evaluations approach their thresholds, inability arguments stop being available, and developers must rely on control or trustworthiness. At that point the internal pillar moves from supporting evidence to required evidence. An index should make this shift visible by reporting, for each category, which kind of argument the evidence supports.
Incidents calibrate the pre-deployment pillars. Incidents add categories that no taxonomy listed, generate regression tests, and show whether pre-deployment evidence predicted what happened. Over several model generations, this gives each developer a track record. We think the track record should affect how an index reads that developer's next results: a developer whose estimates have matched deployment has earned more weight, and one whose estimates have repeatedly missed has not.
Speculation extends the pillars forward. Speculative evaluation proposes categories and tests before harm occurs. Some of its scenarios become behavioral benchmarks, some become targets for internal probes, and some become signals to monitor after release. It also tells incident tracking what to look for. Incidents can in turn show which scenarios were foreseen, which is useful but not required for the pillar to do its work.
Aggregation should be worst case. A model with strong behavioral results, no internal evidence and rising evaluation awareness should not receive a good grade, because the strong results are the ones the missing evidence would have checked. Averaging lets strength in one pillar hide absence in another. We propose that an index report each pillar separately, flag any category where a pillar has no evidence, and compute any summary grade from the weakest supported category rather than the mean.
Quality of evidence matters as much as its presence. Within each pillar, the index should score coverage, depth, replicability (public or accredited), independence of testing, and whether the benchmark still carries signal. Current practice is weak on replicability. In recent system cards, about 10% of reported safety evaluations use public data and a public judge, compared with 69% of capability benchmarks. A safety claim that no outside party can reproduce should count for less than one that can be.
The common thread is that safety claims should travel with their basis. A reader should be able to see, for any category, what was tested, whether internal evidence agrees, whether the model may have recognized the test, what happened after deployment, and what has been foreseen but not yet tested.
Future directions
Each pillar needs research before an index can rely on it. We group the open problems by pillar and then turn to the institutions that all four depend on.
Behavioral evaluation. The first need is benchmarks for the categories where coverage is thinnest. These include interaction harms that build up over weeks rather than turns, such as sycophancy and emotional reliance; agentic and multi-agent harms in environments with real state and consequences; and fairness in the decisions that models support, not only in their answers to questions. The second need is evaluation that resembles deployment. Benchmarks built from real traffic, with consent and privacy protection, would be harder for a model to tell apart from production, and every benchmark should report how often the tested model flags its scenarios as tests. The third need is integrity tests that still work when the model suspects both conditions are evaluations. Compliance-gap and framing tests could vary cues the model cannot easily check, so that a model cannot pass by treating every scenario as a test.
Internal evaluation. Internal methods need a validation science. Shared model organisms with known implanted behaviors would let researchers score probes and interpretability tools on how often they detect what was implanted, and only methods that pass should have their negative results treated as evidence. Open-weight models offer a place to start. Labs that release open weights report the least internal evidence in their system cards, yet anyone can run internal evaluations on their models. An index can credit independent researchers who do this work.
Incident-driven evaluation. AI safety lacks shared infrastructure for incidents. A system similar to CVE in computer security would give each incident an identifier and a category and, where safe, a regression test that any developer can run against new models. Alongside it, the field needs standard post-deployment metrics: misuse rates, jailbreak success in the wild and safeguard miss rates, reported per model and version and measured in ways that protect user privacy.
Speculative evaluation. Simulated deployments could serve as test generators. Shared environments in which models play users, adversaries and institutions, in settings no model has yet entered, would turn each harm that emerges into a candidate test. A further step would be a foresight ledger: a public record of foreseen harms, each with its source and date, that can be compared with the incident record as it grows. It would show which sources of foresight have anticipated real failures.
Institutions and systems. Two needs cut across the pillars. Hazardous behavioral data and internal evaluation of closed weights both require trusted third parties with accredited access. AI Security Institutes are a start, but their access varies across developers and their results are not always published. And deployed safety depends on more than the model. Classifiers, probes and monitors surround it, and AI control research (Greenblatt et al., 2024) evaluates these protocols against a model trying to subvert them. An index should report such system-level results alongside model results.
Limitations
The framework does not settle how to weigh one category against another. How much jailbreak robustness should count against scheming, or how far to discount behavioral scores for evaluation awareness, is a normative choice that evidence alone cannot make. We think an index built on this framework should publish its weights and let readers change them, so that disagreements about values are not mistaken for disagreements about evidence.
An index built on reported evidence also rewards disclosure. Developers who publish detailed system cards will have more evidence to score than those who publish little, and may be judged more harshly for the problems they reveal. We think rewarding disclosure is the right incentive, but it is an incentive, and an index should take care that publishing a problem does not score worse than hiding it.
The balance among the pillars may shift. If evaluation awareness keeps rising, behavioral evaluation may stop being informative about alignment, even while it remains informative about capability. The index would then rest more on internal and incident evidence, both of which have their own gaps today.
Finally, the fourth pillar is the least tested. Its scenarios are only as good as the people and models that produce them, and there is no established way yet to tell a useful scenario from a vivid one. Until there is, an index should mark which tests came from speculation and keep their results apart from observed rates.
Appendix A: Data on current practice
A.1 Sources
Figures about current practice come from 111 system cards, safety reports and risk reports released between October 2025 and October 2026 by six developers: Anthropic, OpenAI, Google, Zhipu (GLM), Moonshot (Kimi) and Alibaba (Qwen). The documents are collected at model-cards.github.io, and each claim in the text links to the page of the source document it comes from.
A.2 Labeling
Evaluations were labeled by LLM agents working from the source text of each document, and the labels were checked for consistency across documents. The labels are available in the safety analysis.
A.3 Replicability
Replicability is counted once per evaluation, per document and area. An evaluation is labeled replicable when (1) its data is public and used as published, and (2) it is scored programmatically or by a named open-weight or API model. Under this definition, about 10% of reported safety evaluations are replicable, compared with 69% of capability benchmarks. Details are in the replicability analysis.
A.4 Limits
The documents cover six developers and one year, so the figures describe frontier practice in that period and not the field as a whole. LLM labeling can misread a document, and the consistency check reduces but does not remove this. When a practice does not appear in these documents, it was not reported, which does not mean it was not done.
References
Documentation and reporting
- Mitchell et al., Model Cards for Model Reporting, FAT* 2019
- Gebru et al., Datasheets for Datasets, Communications of the ACM 2021
- Burnell et al., Rethink reporting of evaluation results in AI, Science 2023
Harm taxonomies and risk overviews
- Weidinger et al., Ethical and social risks of harm from Language Models, 2021
- Shelby et al., Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction, AIES 2023
- Bommasani et al., On the Opportunities and Risks of Foundation Models, 2021
- Hendrycks et al., An Overview of Catastrophic AI Risks, 2023
- Zeng et al., AIR-Bench 2024
- Slattery et al., The AI Risk Repository
- Hammond et al., Multi-Agent Risks from Advanced AI, 2025
Safety research agendas
- Amodei et al., Concrete Problems in AI Safety, 2016
- Hubinger et al., Risks from Learned Optimization in Advanced Machine Learning Systems, 2019
- Hendrycks et al., Unsolved Problems in ML Safety, 2021
- Anwar et al., Foundational Challenges in Assuring Alignment and Safety of Large Language Models, 2024
Benchmarks and red teaming
- Liang et al., Holistic Evaluation of Language Models, TMLR 2023
- Wang et al., DecodingTrust, NeurIPS 2023
- Sun et al., TrustLLM: Trustworthiness in Large Language Models, 2024
- Raji et al., AI and the Everything in the Whole Wide World Benchmark, NeurIPS Datasets and Benchmarks 2021
- Perez et al., Red Teaming Language Models with Language Models, EMNLP 2022
- Ganguli et al., Red Teaming Language Models to Reduce Harms, 2022
- Perez et al., Discovering Language Model Behaviors with Model-Written Evaluations, Findings of ACL 2023
- Wei et al., Jailbroken: How Does LLM Safety Training Fail?, NeurIPS 2023
- Zou et al., Universal and Transferable Adversarial Attacks on Aligned Language Models, 2023
- Sharma et al., Towards Understanding Sycophancy in Language Models, ICLR 2024
- Park et al., AI Deception: A Survey of Examples, Risks, and Potential Solutions, Patterns 2024
- Carlini et al., Extracting Training Data from Large Language Models, USENIX Security 2021
- Greshake et al., Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection, AISec 2023
- Ruan et al., Identifying the Risks of LM Agents with an LM-Emulated Sandbox, ICLR 2024
- StrongREJECT, HarmBench, XSTest, BBQ, MASK, SimpleQA, AgentDojo, SHADE-Arena, VCT, Petri
Dangerous capabilities, scheming and control
- Shevlane et al., Model evaluation for extreme risks, 2023
- Phuong et al., Evaluating Frontier Models for Dangerous Capabilities, 2024
- Li et al., The WMDP Benchmark, ICML 2024
- Kinniment et al., Evaluating Language-Model Agents on Realistic Autonomous Tasks, 2023
- Meinke et al., Frontier Models are Capable of In-context Scheming, 2024
- Clymer et al., Safety Cases: How to Justify the Safety of Advanced AI Systems, 2024
- Balesni et al., Towards evaluations-based safety cases for AI scheming, 2024
- Greenblatt et al., AI Control: Improving Safety Despite Intentional Subversion, ICML 2024
Evaluation integrity
- Needham et al., Large Language Models Often Know When They Are Being Evaluated, 2025
- Greenblatt et al., Alignment faking in large language models, 2024
- Hubinger et al., Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training, 2024
- van der Weij et al., AI Sandbagging: Language Models can Strategically Underperform on Evaluations, 2024
- Greenblatt et al., Stress-Testing Capability Elicitation With Password-Locked Models, 2024
- Tice et al., Noise Injection Reveals Hidden Capabilities of Sandbagging Language Models, NeurIPS 2025
- IAPS, Evaluation Awareness: Why Frontier AI Models Are Getting Harder to Test
Auditing and interpretability
- Raji et al., Closing the AI Accountability Gap, FAT* 2020
- Casper et al., Black-Box Access is Insufficient for Rigorous AI Audits, FAccT 2024
- Burns et al., Discovering Latent Knowledge in Language Models Without Supervision, ICLR 2023
- Zou et al., Representation Engineering: A Top-Down Approach to AI Transparency, 2023
- Bereska and Gavves, Mechanistic Interpretability for AI Safety: A Review, TMLR 2024
- Marks et al., Auditing Language Models for Hidden Objectives, 2025
- Korbak et al., Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety, 2025
Indices, regulation and sociotechnical evaluation
- Future of Life Institute, AI Safety Index, Winter 2025 and Summer 2026
- Stanford CRFM, 2025 Foundation Model Transparency Index
- Emmerson et al., An Open Pipeline and Dashboard for Systemic-Risk Evidence under the EU AI Act's Code of Practice
- EU AI Office, General-Purpose AI Code of Practice (summary)
- Weidinger et al., Sociotechnical Safety Evaluation of Generative AI Systems, 2023
- International AI Safety Report 2026
Incident tracking
- McGregor, Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database, AAAI 2021
- OECD, AI Incidents and Hazards Monitor (methodology)
Foresight, forecasting and simulation
- Brundage et al., The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation, 2018
- Cave and Dihal, Hopes and fears for intelligent machines in fiction and reality, Nature Machine Intelligence 2019
- Karger et al., Forecasting Existential Risks: Evidence from a Long-Run Forecasting Tournament, Forecasting Research Institute 2023
- Kwa et al., Measuring AI Ability to Complete Long Software Tasks, 2025
- Buçinca et al., AHA!: Facilitating AI Impact Assessment by Generating Examples of Harms, 2023
- Kieslich et al., Anticipating Impacts: Using Large-Scale Scenario Writing to Explore Diverse Implications of Generative AI in the News Environment, AI and Ethics 2025
- Park et al., Generative Agents: Interactive Simulacra of Human Behavior, UIST 2023
- Vezhnevets et al., Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordia, 2023
BibTeX
@misc{sai2026safetyeval,
author = {{SAI Labs (TBD)}},
title = {{Safety evaluation needs four pillars: behavioral, internal, incident-driven and speculative}},
howpublished = {\url{https://sai.science/safety-eval}},
year = {2026},
month = nov,
note = {Preliminary draft}
}