Agentic AI and RL Environment Jobs: Build and Grade the Tasks Agents Learn From

An agent is a model that does things rather than describing them: it runs a command, clicks through a settings page, calls an API, edits a file, and checks its own result. Training one needs three kinds of human work, and the listings on this page cover all three.

Environment building means writing the task an agent will practice on: a starting state (a repository with a bug, a spreadsheet with errors, a half-finished marketing brief), a goal, the actions available, and a grader that scores the attempt automatically. Listings say "RL environment builder", "scenario designer" or "benchmarks and RL environments". The RL stands for reinforcement learning, the training loop that rewards the agent when the grader says it succeeded.

Trajectory work means doing the task yourself inside the environment while every step is recorded, so the model learns from a competent demonstration. Titles include "trajectory specialist", "computer use annotator" and "agent trace collection". It feels like being screen-recorded doing your normal job, with a requirement to narrate why.

Agent evaluation means reading what an agent did, step by step, and grading it: did it take sane actions, did it finish, did it cheat the grader, would a senior colleague have accepted the result. Titles say "agent evaluation analyst", "agentic workflow reviewer" or "task auditor". Of the three, this is the closest to RLHF evaluation, with the difference that you are judging a sequence of actions, not a paragraph.

An original example, written for this page. You are asked to build an environment for an accounting agent.

You write the starting state: a 40-row expense ledger and a folder of receipts, where three rows disagree with their receipt (a transposed amount, a duplicated entry, a receipt with no row). The goal: reconcile the ledger and file a correction note. The grader you write checks that exactly those three rows were fixed and that the note names each one, and it penalizes any edit to a row that was already correct. Then you record a reference run of yourself doing it.

Two agent runs come back for review. Run A changes all 40 amounts to match the receipt totals, which makes the ledger sum correct and would pass a grader that only checked the total. Run B finds the transposed amount and the duplicate, misses the missing row, and writes an accurate note. Run A is the one that matters: it found a hole in your grader, which is the failure environment builders are paid to prevent. You tighten the grader, score B as partial with the miss named, and flag A as reward hacking so the lab can use it as a regression test.

In trajectory roles you would be the reference run: reconciling the ledger yourself in the sandbox, with each click and keystroke logged, and a short note at each decision point. The logs are usually JSON, which is why several listings ask for comfort with structured data; the Academy module Markdown and JSON Formatting covers what reviewers check.

Software engineers are the largest group: listings ask for a language (Python most often, then Go, Rust and TypeScript), experience working with coding agents, and the ability to write a test an agent cannot game. Benchmarks like SWE-bench and terminal-bench appear in titles because labs want people who can write tasks in that shape.

The second group is professionals who live in a specific tool: accountants in QuickBooks, marketers in a CRM and an ad manager, operations people in ticketing systems. Those listings want you to build or demonstrate the workflow an agent should learn, and they do not ask for code. Most of them are posted by one platform at a time in batches, so the list below can swing from twenty such roles to none in a month.

It does not suit people who dislike adversarial thinking. Every grader you write will be attacked by a model that is better at finding loopholes than most humans, and the job is to assume that and close them. If you prefer to judge finished text, RLHF evaluation is the better fit.

Engineering roles run a live coding or debugging round, sometimes automated, and often ask to see public work: one listing on this page recruits by GitHub contribution history. Expect to explain how you would write a grader for a task and how an agent might game it. The Academy module Live Code Review and Debugging Interviews is built for this round, and Python for AI Training covers the language most listings name.

Domain roles screen with a short conversational interview and a sample task: build one scenario in your tool, or walk through how you would, with the steps written out. Where listings on this page state a screening step at all, it is an interview; most say nothing, so read each posting.

The pay block at the top of this page is computed from the open listings below: typical (median) hourly rate, the range most roles pay, and how many posted rates it is based on, updated on the date shown. The method is on the pay methodology page.

Listed rates on this page currently run $79 to $85/hr for the middle half of roles.

The middle half is narrow because most listings here post a single rate for an engineering tier rather than a range, and a platform often posts the same rate across a batch of roles. The notes beside this section say when one platform or our own estimates dominate the figure. Trajectory and domain-workflow roles tend to sit below the engineering rate; benchmark and environment design roles at or above it.

1. Pick the track that matches what you already do. Engineers: environment building and agentic coding. Tool-fluent professionals: trajectory and workflow roles. Careful readers with some technical comfort: agent evaluation.
2. Build one environment on your own before applying. A task with a starting state, a goal, a grader and a reference run, in a public repository, is the strongest application material the listings on this page respond to. Keep it small and make the grader hard to game.
3. Prepare for the adversarial question. "How would an agent cheat this?" comes up in interviews and in the work. Practice answering it for your own task.
4. Apply to the listings sorted by newest. Batches here open and close quickly; a role that is three weeks old is often filled.
5. Keep your JSON and Markdown clean. Trajectory logs and task specs are reviewed as structured data, and formatting errors are the fastest way to fail a trial batch.

Adjacent kinds of work: RLHF evaluation grades what a model said, red teaming tries to make it misbehave, and the software engineering page lists code review roles that do not involve agents.