AI Training Scenarios - Reviewer (Quality Control)
Mercor • Remote
Education
Any
Type
hourly
Pay Rate (by country)
$80–$105/hr
Listed
102d ago
✅ Applying through this link supports our platform at no cost to you.
This position is hosted on an external talent platform. Please only apply for this position if it fits your skills and interests.
In our Talent Pool?
Apply through this link and we can vouch for you to Mercor. ? We vouch for Talent Pool members who apply through our referral link, when we believe they're a strong match. Not every applicant gets a vouch. Not in the pool yet? Set up your profile first.
Set up your profile →Mercor: our referral track record
We've referred 556 candidates to Mercor roles. 5% (29) were placed.
What We Know About This Role
- Weekly hours
- 40 hrs/week
About this Role
From the Mercor listing
We're looking for highly analytical, detail-oriented professionals to serve as Quality Control Leads, focused on evaluating and auditing complex simulated environments and scenarios designed to train frontier AI agents. As an Auditor/Reviewer, you will be the final line of defense, ensuring that the digital worlds (spanning Gmail, Slack, Drive, etc.) and agentic tasks crafted by our design team meet the highest standards of realism, complexity, and technical accuracy. You will apply strict grading rubrics to assess scenario quality and provide vital feedback to the creators.
This is a W2 employment position with Cincinnatus LLC, with the opportunity to be placed at a leading AI Lab as part of their extended workforce. You will join a team of domain experts and together, you will guide the next generation of frontier AI tools and agents.
- Audit & Evaluate: Review richly detailed personas, digital environments, and tasks created by AI Training Scenario Designers to ensure they are logically sound and properly structured.
- Apply Strict Rubrics: Utilize complex evaluation rubrics to score scenarios objectively, ensuring the tasks effectively challenge an AI agent's ability to reason, filter, and prioritize without being broken or impossible.
- Quality Assurance: Identify logical inconsistencies, factual errors, missing context, and technical flaws (e.g., invalid JSON structures) within the simulated environments.
- Provide Feedback: Deliver clear, constructive, and actionable feedback to the scenario designers to help them refine their worlds and improve their overall output.
- Document Outcomes: Maintain rigorous quality standards by logging your review outcomes, scores, and feedback clearly in Airtable and Crucible.
- Have a proven track record in QA, editing, auditing, or reviewing complex written/technical content.
- Ability to engage for ~40 hours/week
- Excel at learning and rigorously applying comprehensive grading rubrics (prior high scores in rubric-based training or academies is a massive plus).
- Have an "eagle eye" for inconsistencies, edge cases, and logical gaps that others miss.
- Are comfortable giving constructive, structured feedback to peers and creators.
- Can quickly navigate structured tools like JSON editors, Airtable, and AI platforms.
- Editing / QA background — Senior editor, QA tester, technical reviewer, compliance auditor, or peer-reviewer.
- Technical / data comfort — JSON familiarity, Jira/Asana power user, data analyst, SQL/Python exposure. You don't need to write the code from scratch, but you must be able to spot when a JSON structure is broken.
- Process & Systems thinking — Operations managers, technical PMs, or systems designers who understand how complex, multi-stakeholder workflows should logically function in a professional setting.
Requirements
- Must be eligible to work in Remote
- Fluent proficiency in English (Written & Verbal)
- Reliable high-speed internet connection
- Bachelor's degree or equivalent professional experience
- Demonstrated expertise in Accounting
Why This Role
This AI Training Scenarios - Reviewer (Quality Control) role offers a powerful combination of high income ($92.5/hr) and total flexibility. Unlike traditional contracting, you generally set your own hours. It is an ideal "second stream" of income for Accounting professionals who want to stay sharp in their field while gaining exposure to the booming AI industry.
Skills & Categories
Explore other opportunities in related specializations:
Related Jobs
Browse All Jobs from Mercor
Discover more opportunities on Mercor that match your skills and interests.
View All Mercor Jobs →Verified Reviews
Community Reviews
Share your experience with Mercor
Help other candidates make better decisions by leaving a review.
Sign in to leave a reviewLeave your review
Frequently Asked Questions
Is Mercor for freelancers or full-time contractors?
Mercor places you with one client for a defined engagement, like 'Python Tutor for 3 months', rather than having you grab small tasks from a shared queue. Most roles function as steady contract work, not one-off gigs.
Does Mercor's application require an on-camera interview?
Yes, every applicant records a video interview with an AI interviewer that asks questions about your resume. Clients review that recording to judge communication skills before matching, so there's no way to apply without going on camera.
Does it cost money to apply to Mercor?
No, applying and joining Mercor is free. Mercor's revenue comes from a fee it charges the client on top of your hourly rate, not from applicants. Treat any request for payment to join as a red flag.
Is AI training work the same as traditional consulting?
No. Instead of client deliverables, you're given complex scenarios to evaluate: grading the AI's logic, correcting its hallucinations, and supplying expert-level reasoning it doesn't have on its own. The job is closer to teaching than consulting.
Why do these AI training roles pay so much?
Because general knowledge isn't what's being tested. The model already knows the basics; what it needs is expertise on edge cases, the rare, difficult, highly technical judgment calls only a senior professional in the field would make correctly.
What does the day-to-day workload look like for elite-expert AI training roles?
Slow and deep, not fast and repetitive. A single task can take 45-60 minutes of researching citations or verifying complex calculations. Quality is what's being measured here, not throughput.
What does Accounting work look like for a AI Training Scenarios - Reviewer (Quality Control)?
Tasks here are scoped to Accounting, not generic labeling. As a AI Training Scenarios - Reviewer (Quality Control), expect to draw on real domain judgment (evaluating outputs, correcting errors, or providing expert reasoning specific to Accounting) rather than following a one-size-fits-all rubric. If you don't have hands-on Accounting background, this is likely not the right listing to start with.
How many hours per week does this role require?
Based on the listing, this role is scoped at about 40 hours per week. Treat this as a real commitment expectation, not a loose estimate.
What happens when I click Apply on this listing?
You'll be taken to Mercor's external site to complete your application there. This listing links through a referral, but the process is identical to applying directly; the link just routes you correctly. Create an account on their site and follow their onboarding steps.
How soon will I start working after applying to Mercor?
Not immediately. Mercor is a talent marketplace, not a task queue, so applying puts you in a pool of candidates. You start working only once a specific client, like a major AI lab, selects your profile, and that matching process can take weeks.