Company
Vals AI
Annual salary
$140k – $185k/yr
Location
San Francisco, CA
Listed
140d ago
- Experience:
- 0 to 4 years
- Workplace:
- 5 days in-office in San Francisco, CA, relocation support provided
- Visa:
- Visa sponsorship available
- Equity:
- Competitive equity
On-site in San Francisco, CA. Visa sponsorship available.
Send us your LinkedIn and CV. If your experience fits, we'll introduce you to the recruiter filling this role.
Apply for a referral →The recruiter emails you before anything happens and may suggest other jobs that suit you better.
About this Role
We are looking for strong engineers to join our team and own the leaderboards that appear on Vals AI.
You will be responsible for testing and benchmarking new models as they are released on tasks in law, tax, coding, finance, and more. You will analyze error modes of models, evaluate their strengths and weaknesses, and work with our communications team to release results.
Our results are used by startups, enterprises, and research labs alike. We work with all the major foundation model labs, some of the largest financial institutions, and hospital systems in the world. Our work has been featured by the Wall Street Journal, Washington Post, and Bloomberg.
We are building the standard for evaluating the ability of LLMs to perform real-world tasks. You will contribute directly to the leaderboards that make this possible.
What you'll do
- Evaluate new LLM model releases across the Vals AI suite of benchmarks
- Work directly with both open-source and closed-source foundation model labs in evaluating model performance
- Use tools like Docent to analyze common failure modes and patterns in model performance
- Work directly with our social media team to post interesting findings and results
- Add new models and maintain integrations in our model library
- Help improve and maintain the infrastructure we use to run benchmarks (agentic and non-agentic).
- Collaborate closely with our research team on the creation of new benchmarks This role follows the rhythm of model releases. Expect intense sprints in the days following a major launch, and calmer stretches in between releases.
Frequently Asked Questions
How do I apply for the Evaluations Engineer (US) role at Vals AI? +
Use the Apply for a referral button on this page to send us your LinkedIn and CV. If your experience fits, we'll introduce you to the recruiter filling this role. They'll email you to check you're interested, then put you forward for this job or others that suit you better. It's free.
What does this Vals AI role pay? +
The listing gives $140k – $185k/yr.
Is this role remote? +
The listing gives the location as San Francisco, CA. 5 days in-office in San Francisco, CA, relocation support provided. Visa sponsorship available.
Related Roles
Browse all startup roles
Salaried roles at startups, AI companies and established businesses, all filled by referral. One application covers every role.
View all startup roles →