Safety IndexEvaluationsPrivacyLens
PrivacyLens
People and society · Privacy: surveillance assistanceCounts in the indexBuilt, not yet run
Whether an agent's final action, such as sending an email or message, shares information that is inappropriate for the recipient.
- Metric
- % leaked. Lower is better.
- Dataset
- github.com/SALT-NLP/PrivacyLens · 493 seeds -> trajectories · MIT
- Grader
- leakage judge (paper: Mistral-7B); a stronger fixed judge for the index, the original kept for comparability (the paper used Mistral-7B-Instruct-v0.2)
- Status
- Built, not yet run
- Replicability
- public
Results
Built and tested; the first full runs are pending.