SAI LabsBook a demo

Safety IndexEvaluationsPrivacyLens

PrivacyLens

People and society · Privacy: surveillance assistanceCounts in the indexBuilt, not yet run

Whether an agent's final action, such as sending an email or message, shares information that is inappropriate for the recipient.

Metric
% leaked. Lower is better.
Dataset
github.com/SALT-NLP/PrivacyLens · 493 seeds -> trajectories · MIT
Grader
leakage judge (paper: Mistral-7B); a stronger fixed judge for the index, the original kept for comparability (the paper used Mistral-7B-Instruct-v0.2)
Status
Built, not yet run
Replicability
public

Results

Built and tested; the first full runs are pending.