Behavioral Health Expert, AI Safety and Model Evaluation

About this Role

What you'll do

• Evaluate conversations between users and AI models across topics such as relationship and family advice, emotional wellbeing, spiritual and metaphysical questions, and unconventional or unfounded beliefs, and assess whether responses are neutral, appropriate and safe.
• Identify patterns of concern, including excessive agreement, taking sides, reinforcing distorted or unfounded beliefs, moralizing, or overstepping into clinical or directive advice, and document them with clear written rationale.
• Develop rubrics, guidelines and reference responses that define a balanced, supportive and appropriately bounded reply, grounded in established practice from counseling and behavioral health.
• Design test scenarios and conversations that probe how models handle sensitive but non crisis topics.
• Collaborate with the lab's researchers and fellow experts to keep evaluation standards consistent, calibrated and well documented.

What you need

• A degree in psychology, counseling, social work, behavioral health, behavioral science, human services or a closely related field, or equivalent professional experience in the mental health field.
• 3+ years of professional experience supporting people in a mental health, counseling or social services setting, for example as a therapist, counselor, clinical social worker, psychologist, psychiatric nurse, case manager, crisis counselor, peer support specialist or mental health advocate. Clinical licensure (for example LMFT, LCSW, LPC, LMHC, PsyD or PhD) is valued but not required.
• Demonstrated ability to remain neutral and nonjudgmental across diverse perspectives, relationships, belief systems and worldviews, and to explain the reasoning behind a professional judgment.
• Working familiarity with concepts such as sycophancy, cognitive distortions, healthy boundaries and client centered approaches such as motivational interviewing.
• Ability to engage reliably for at least 20 hours/week during weekdays.
• Strong written communication skills and the ability to deliver precise, well structured written feedback. Nice to have: a background in AI safety, applied ethics, trust and safety or content policy; experience in couples, family or relationship counseling; familiarity with spiritual care, religious or alternative belief communities, or the psychology of misinformation and conspiracy belief; prior experience evaluating, annotating or red teaming AI systems. You don't need all of these to apply.

Skills & Categories

Frequently Asked Questions

Related Roles