Mathematics Model Prompt Evaluator

Role Overview

We are seeking expert mathematicians to author and verify high-quality open-ended prompts for AI model evaluation. You will craft and review challenging, unambiguous mathematical problems across core subdomains, assessing AI reasoning quality and helping establish rigorous evaluation standards for frontier language models.

Role Overview

You will be assigned one of two task types

Authoring Task

Verification Task

Mathematics Subdomains Covered

Key Responsibilities

Ideal Qualifications

More About the Opportunity

Role Overview We are seeking expert mathematicians to author and verify high-quality open-ended prompts for AI model evaluation. You will craft and review challenging, unambiguous mathematical problems across core subdomains, assessing AI reasoning quality and helping establish rigorous evaluation standards for frontier language models. You will be assigned one of two task types: Authoring Task Create 5 original, open-ended prompts from your assigned subdomain at varying difficulty levels (undergraduate, advanced undergraduate, or graduate/professional). Prompts should require human judgment to evaluate the quality of the AI's response, such as chain-of-thought reasoning or proof construction. Verification Task Review 5 authored prompts for clarity, scope alignment, difficulty accuracy, and uniqueness. Edit prompts and difficulty ratings where needed. Mathematics Subdomains Covered Probability & Statistics, Algebra (incl. Linear Algebra), Ordinary/Partial Differential Equations & Dynamical Systems, Geometry, Graph Theory, Number Theory. Key Responsibilities - Author clear, unambiguous, open-ended mathematical prompts that elicit evaluable AI responses - Verify prompts are within the scope of the assigned subdomain and correctly rated for difficulty - Ensure all 5 prompts in a task are sufficiently distinct from one another with varying difficulty levels - Apply expert judgment to assess the depth and quality of mathematical reasoning required - Edit prompts and difficulty assignments where standards are not met Ideal Qualifications - Master's degree or higher in Mathematics, Applied Mathematics, Statistics, or a closely related field - 2–6 years of professional or research experience in a quantitative field - Strong command of graduate-level mathematical concepts including proof writing, analysis, and formal reasoning - Experience in academic research, mathematical competition design, or quantitative industry roles is a plus - Excellent written English and ability to craft precise, well-scoped technical questions More About the Opportunity - Expected commitment: 10+ hours/week - Asynchronous, fully remote work We consider all qualified applicants without regard to legally protected characteristics and provide reasonable accommodations upon request.