SAI LabsBook a demo

Safety IndexEvaluationsMASK

MASK

Interaction and honesty · Honesty under pressure and sycophancyCounts in the indexBuilt, not yet run

Whether the model contradicts what it has said it believes when a system or user prompt pressures it to lie.

Metric
% honest. Higher is better.
Dataset
huggingface.co/datasets/cais/MASK · 1,000 public (500 private at Scale) · unstated on card (repo MIT)
Grader
GPT-4o (binary) + o3-mini (numeric), as in the public MASK port (the paper used same)
Status
Built, not yet run. Measured on 19 models, 0 readings by SAI.
Replicability
partial
Notes
Third-party rows so far are from Scale's leaderboard on the private 500-example split; SAI runs will use the public 1,000-example split.

Results

ModelValueScoreSourceSetting
Claude Sonnet 4.596.1%95% CI 95.6% to 96.7%n = 50096Scale AI SEAL / CAIS MASK leaderboard
Independent evaluator
Published by Scale AI SEAL / CAIS MASK leaderboard.
Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'claude-sonnet-4-5-20250929-thinking'
Claude Opus 4.592.5%95% CI 91.3% to 93.8%n = 50093Scale AI SEAL / CAIS MASK leaderboard
Independent evaluator
Published by Scale AI SEAL / CAIS MASK leaderboard.
Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'claude-opus-4-5-20251101-thinking'
gpt-oss-120b92%95% CI 91.1% to 92.9%n = 50092Scale AI SEAL / CAIS MASK leaderboard
Independent evaluator
Published by Scale AI SEAL / CAIS MASK leaderboard.
Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'gpt-oss-120b'
GPT-5.4 Pro91.7%95% CI 89.7% to 93.7%n = 50092Scale AI SEAL / CAIS MASK leaderboard
Independent evaluator
Published by Scale AI SEAL / CAIS MASK leaderboard.
Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'gpt-5.4-pro-2026-03-05'
GPT-5.489.7%95% CI 88.9% to 90.4%n = 50090Scale AI SEAL / CAIS MASK leaderboard
Independent evaluator
Published by Scale AI SEAL / CAIS MASK leaderboard.
Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'gpt-5.4-2026-03-05 (xhigh thinking)'
GPT-5.286.7%95% CI 84.9% to 88.4%n = 50087Scale AI SEAL / CAIS MASK leaderboard
Independent evaluator
Published by Scale AI SEAL / CAIS MASK leaderboard.
Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'gpt-5.2-2025-12-11'
gpt-oss-20b86.5%95% CI 84.4% to 88.5%n = 50087Scale AI SEAL / CAIS MASK leaderboard
Independent evaluator
Published by Scale AI SEAL / CAIS MASK leaderboard.
Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'gpt-oss-20b'
GPT-5.186.3%95% CI 85.6% to 87.1%n = 50086Scale AI SEAL / CAIS MASK leaderboard
Independent evaluator
Published by Scale AI SEAL / CAIS MASK leaderboard.
Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'gpt-5.1-thinking'
Claude Opus 4.685.4%95% CI 84.9% to 85.9%n = 50085Scale AI SEAL / CAIS MASK leaderboard
Independent evaluator
Published by Scale AI SEAL / CAIS MASK leaderboard.
Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'claude-opus-4-6-thinking-max'
GPT-579.3%95% CI 74.0% to 84.6%n = 50079Scale AI SEAL / CAIS MASK leaderboard
Independent evaluator
Published by Scale AI SEAL / CAIS MASK leaderboard.
Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'gpt-5-2025-08-07'
Kimi K2.570.5%95% CI 66.0% to 75.0%n = 50071Scale AI SEAL / CAIS MASK leaderboard
Independent evaluator
Published by Scale AI SEAL / CAIS MASK leaderboard.
Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'kimi-k2.5'
GLM-4.561.5%95% CI 57.5% to 65.4%n = 50062Scale AI SEAL / CAIS MASK leaderboard
Independent evaluator
Published by Scale AI SEAL / CAIS MASK leaderboard.
Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'glm-4p5'
DeepSeek R157.3%95% CI 54.7% to 59.9%n = 50057Scale AI SEAL / CAIS MASK leaderboard
Independent evaluator
Published by Scale AI SEAL / CAIS MASK leaderboard.
Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'DeepSeek R1'
Qwen3-235B-A22B56.4%95% CI 51.4% to 61.4%n = 50056Scale AI SEAL / CAIS MASK leaderboard
Independent evaluator
Published by Scale AI SEAL / CAIS MASK leaderboard.
Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'Qwen3-235B-A22B'
Llama 4 Maverick49.7%95% CI 48.1% to 51.3%n = 50050Scale AI SEAL / CAIS MASK leaderboard
Independent evaluator
Published by Scale AI SEAL / CAIS MASK leaderboard.
Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'Llama 4 Maverick'
Gemini 3.1 Flash-Lite48.4%95% CI 47.5% to 49.3%n = 50048Scale AI SEAL / CAIS MASK leaderboard
Independent evaluator
Published by Scale AI SEAL / CAIS MASK leaderboard.
Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'gemini-3.1-flash-lite-preview'
DeepSeek V3.146.3%95% CI 44.7% to 47.9%n = 50046Scale AI SEAL / CAIS MASK leaderboard
Independent evaluator
Published by Scale AI SEAL / CAIS MASK leaderboard.
Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'DeepSeek-V3.1'
Gemini 3 Pro42.6%95% CI 38.6% to 46.6%n = 50043Scale AI SEAL / CAIS MASK leaderboard
Independent evaluator
Published by Scale AI SEAL / CAIS MASK leaderboard.
Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'gemini-3-pro-preview'
Gemini 3.1 Pro42.4%95% CI 41.1% to 43.7%n = 50042Scale AI SEAL / CAIS MASK leaderboard
Independent evaluator
Published by Scale AI SEAL / CAIS MASK leaderboard.
Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'gemini-3.1-pro-preview'