RLHF and AI Evaluator Jobs: Get Paid to Rate Model Responses

A session opens with a prompt someone sent to a model, followed by two to four responses to it. You read all of them, pick the best one or rank them, score each against a short rubric (did it follow the instruction, is it correct, is it safe, how is the tone), and write a rationale in your own words. Your choices become the preference data that reinforcement learning from human feedback (RLHF) trains the next model version on, which is why the listing title can say annotator, evaluator, AI rater, reviewer or task auditor and describe the same afternoon of work.

The skill being bought is calibration, not machine learning. Nobody in this queue needs to know what a reward model is. What matters is whether your 5 out of 7 means the same thing as the next rater's 5 out of 7, whether you spot that a confident answer invented a citation, and whether your rationale lets a reviewer check your reasoning in thirty seconds. The word rater comes from search-engine quality rating, and the habits transfer: read the guidelines, apply them the same way every time, flag what the guidelines do not cover.

Three tiers show up in the listings on this page. Generalist rating takes any careful reader with strong written English or another required language, and most language-paired roles sit here. Expert rating asks for a credential or a career (clinicians grading medical advice, lawyers grading legal reasoning, engineers grading code) and pays accordingly. Benchmark authoring is the hardest version: writing questions a frontier model cannot yet answer, with a reference solution and a grading key, so a lab can measure progress.

This is an original example written for this page, not a question from any platform. The prompt: a user asks for a three-day packing list for a November work trip to Oslo, carry-on only, with one client dinner.

Response A runs to forty items, includes a beach towel and sunscreen, and never mentions the carry-on limit. Response B is twelve items, grouped by day, reminds the user that liquids over 100 ml will not clear security, and adds a wool layer for the dinner. It also states that Oslo averages 15°C in November, which is wrong by about ten degrees.

The task asks you to pick the better response, score each from 1 to 7 on helpfulness, accuracy and instruction following, and write a rationale. B is the pick: it followed the carry-on constraint and the dinner, and it is usable as written. The 15°C claim costs it on accuracy, and your rationale has to say so, because an unflagged factual error is the thing a reviewer will catch. A good rationale names the specific failure and the specific win; a weak one restates the scores.

Most queues run tasks like this at ten to thirty minutes each, with a quality reviewer re-checking a sample of your work. The free Academy module Your First Annotation Task walks through a full pairwise comparison with the rationale written out.

Three groups get hired from this page. The largest is people who read closely and write clearly in the language the listing names: many roles here are language-paired, which means a native Korean, Danish or Brazilian Portuguese reader with good English is more employable than a monolingual English speaker with the same judgment. The language pages list those roles by language.

The second group is domain experts. A hospitalist grading a model's discharge summary, a tax preparer scoring a workflow, a security engineer auditing a code change: these listings name the credential in the title and pay at the top of the range. The third group is engineers and researchers writing benchmarks and evaluation harnesses, where the work is closer to test design than to rating.

It does not suit people who want to skim. Rating queues track agreement with gold-standard answers and with other raters, and accounts that drift below the threshold stop receiving work without much explanation. If you tend to finish tasks fast and move on, transcription or agentic task work rewards speed more than this does.

The usual sequence is a profile, a qualification test, and a paid trial batch. The qualification test gives you sample prompts and responses with the platform's guidelines and compares your scores with a gold standard, so the question it answers is how calibrated you are, not how much you know. Some platforms add an AI-led video interview before the test, and expert tiers add a credential check or a domain task. Where a listing on this page states its screening step, it is almost always an interview; where it says nothing, assume a test.

Preparation that works: read the guidelines twice before the first sample, score slowly, and write every rationale as if a stranger will audit it. The Academy modules Passing the Screening Process and Reading the Rubric Like a Grader cover the stages and the anatomy of a rubric, and the guide on what rubrics are explains the vocabulary. Interview questions on this site are never quoted from a platform, because candidates sign NDAs.

The pay block at the top of this page is computed from the open listings below it: typical (median) hourly rate, the range most roles pay, and how many posted rates it is based on, updated on the date shown. It is recomputed as listings change, so this page never quotes a range that a listing did not state. The method is on the pay methodology page.

Listed rates on this page currently run $35 to $100/hr for the middle half of roles.

Within that spread, generalist rating sits at the bottom, language-paired rating in the middle, and credentialed expert rating and benchmark authoring at the top. A few listings pay per task rather than per hour; those are excluded from the hourly figures. The notes beside this section say when one platform or our own estimates dominate the numbers. For the ranking across every category, see the highest-paying fields and platforms.

1. Apply to two or three platforms at once. Rating queues fill and empty by project, and one platform rarely gives steady hours. Sort the list below by newest and pick listings whose language and expertise match yours exactly.
2. Do the free walkthrough before the test. Foundations of RLHF (50 minutes) explains where your ratings go, and Your First Annotation Task has you make a real pairwise decision and write the rationale. Both are readable without an account.
3. Write your profile in the listing's words. If the title says "AI Response Evaluator (Native Speaker)", your profile should say native speaker, evaluation, and the language, with any credential spelled out.
4. Take the qualification test rested and unhurried. Agreement with the gold standard decides it, and most people fail by rushing the last third.
5. Treat the first paid batch as the second test. Quality review on the first fifty tasks decides whether you see more work. Advanced Instruction Following covers the failure reviewers look for most: a response that looks compliant and is not.

Which companies are hiring raters right now, with live counts, is kept up to date in this guide. If you would rather have the board filtered to your background and minimum rate, the matcher does it in three questions and no account.

Adjacent kinds of work: red teaming and safety review grades the model on what it should have refused, agentic tasks grade what a model did rather than what it said, and transcription produces the speech data that voice models train on.