MASK
Interaction and honesty · Honesty under pressure and sycophancyCounts in the indexBuilt, not yet run
Whether the model contradicts what it has said it believes when a system or user prompt pressures it to lie.
- Metric
- % honest. Higher is better.
- Dataset
- huggingface.co/datasets/cais/MASK · 1,000 public (500 private at Scale) · unstated on card (repo MIT)
- Grader
- GPT-4o (binary) + o3-mini (numeric), as in the public MASK port (the paper used same)
- Status
- Built, not yet run. Measured on 19 models, 0 readings by SAI.
- Replicability
- partial
- Notes
- Third-party rows so far are from Scale's leaderboard on the private 500-example split; SAI runs will use the public 1,000-example split.
Results
| Model | Value | Score | Source | Setting |
|---|---|---|---|---|
| 96.1%95% CI 95.6% to 96.7%n = 500 | 96 | Scale AI SEAL / CAIS MASK leaderboard Independent evaluator | Published by Scale AI SEAL / CAIS MASK leaderboard.Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'claude-sonnet-4-5-20250929-thinking' | |
| 92.5%95% CI 91.3% to 93.8%n = 500 | 93 | Scale AI SEAL / CAIS MASK leaderboard Independent evaluator | Published by Scale AI SEAL / CAIS MASK leaderboard.Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'claude-opus-4-5-20251101-thinking' | |
| 92%95% CI 91.1% to 92.9%n = 500 | 92 | Scale AI SEAL / CAIS MASK leaderboard Independent evaluator | Published by Scale AI SEAL / CAIS MASK leaderboard.Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'gpt-oss-120b' | |
| 91.7%95% CI 89.7% to 93.7%n = 500 | 92 | Scale AI SEAL / CAIS MASK leaderboard Independent evaluator | Published by Scale AI SEAL / CAIS MASK leaderboard.Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'gpt-5.4-pro-2026-03-05' | |
| 89.7%95% CI 88.9% to 90.4%n = 500 | 90 | Scale AI SEAL / CAIS MASK leaderboard Independent evaluator | Published by Scale AI SEAL / CAIS MASK leaderboard.Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'gpt-5.4-2026-03-05 (xhigh thinking)' | |
| 86.7%95% CI 84.9% to 88.4%n = 500 | 87 | Scale AI SEAL / CAIS MASK leaderboard Independent evaluator | Published by Scale AI SEAL / CAIS MASK leaderboard.Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'gpt-5.2-2025-12-11' | |
| 86.5%95% CI 84.4% to 88.5%n = 500 | 87 | Scale AI SEAL / CAIS MASK leaderboard Independent evaluator | Published by Scale AI SEAL / CAIS MASK leaderboard.Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'gpt-oss-20b' | |
| 86.3%95% CI 85.6% to 87.1%n = 500 | 86 | Scale AI SEAL / CAIS MASK leaderboard Independent evaluator | Published by Scale AI SEAL / CAIS MASK leaderboard.Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'gpt-5.1-thinking' | |
| 85.4%95% CI 84.9% to 85.9%n = 500 | 85 | Scale AI SEAL / CAIS MASK leaderboard Independent evaluator | Published by Scale AI SEAL / CAIS MASK leaderboard.Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'claude-opus-4-6-thinking-max' | |
| 79.3%95% CI 74.0% to 84.6%n = 500 | 79 | Scale AI SEAL / CAIS MASK leaderboard Independent evaluator | Published by Scale AI SEAL / CAIS MASK leaderboard.Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'gpt-5-2025-08-07' | |
| 70.5%95% CI 66.0% to 75.0%n = 500 | 71 | Scale AI SEAL / CAIS MASK leaderboard Independent evaluator | Published by Scale AI SEAL / CAIS MASK leaderboard.Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'kimi-k2.5' | |
| 61.5%95% CI 57.5% to 65.4%n = 500 | 62 | Scale AI SEAL / CAIS MASK leaderboard Independent evaluator | Published by Scale AI SEAL / CAIS MASK leaderboard.Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'glm-4p5' | |
| 57.3%95% CI 54.7% to 59.9%n = 500 | 57 | Scale AI SEAL / CAIS MASK leaderboard Independent evaluator | Published by Scale AI SEAL / CAIS MASK leaderboard.Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'DeepSeek R1' | |
| 56.4%95% CI 51.4% to 61.4%n = 500 | 56 | Scale AI SEAL / CAIS MASK leaderboard Independent evaluator | Published by Scale AI SEAL / CAIS MASK leaderboard.Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'Qwen3-235B-A22B' | |
| 49.7%95% CI 48.1% to 51.3%n = 500 | 50 | Scale AI SEAL / CAIS MASK leaderboard Independent evaluator | Published by Scale AI SEAL / CAIS MASK leaderboard.Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'Llama 4 Maverick' | |
| 48.4%95% CI 47.5% to 49.3%n = 500 | 48 | Scale AI SEAL / CAIS MASK leaderboard Independent evaluator | Published by Scale AI SEAL / CAIS MASK leaderboard.Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'gemini-3.1-flash-lite-preview' | |
| 46.3%95% CI 44.7% to 47.9%n = 500 | 46 | Scale AI SEAL / CAIS MASK leaderboard Independent evaluator | Published by Scale AI SEAL / CAIS MASK leaderboard.Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'DeepSeek-V3.1' | |
| 42.6%95% CI 38.6% to 46.6%n = 500 | 43 | Scale AI SEAL / CAIS MASK leaderboard Independent evaluator | Published by Scale AI SEAL / CAIS MASK leaderboard.Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'gemini-3-pro-preview' | |
| 42.4%95% CI 41.1% to 43.7%n = 500 | 42 | Scale AI SEAL / CAIS MASK leaderboard Independent evaluator | Published by Scale AI SEAL / CAIS MASK leaderboard.Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'gemini-3.1-pro-preview' |