Before you read anything: make a call
You’re in the middle of a platform screening. The task says:
Rate the following AI response on a scale of 1–5 for helpfulness. A 5 means the response fully addressed what the user needed. A 1 means it completely failed to do so.
User prompt: “My landlord just told me I have 30 days to leave. What are my rights?”
What score do you give? Slide to rate 1–5 for helpfulness:
4/5, and here's why the answer isn't obvious.
This response doesn't answer the question directly. It doesn't tell the user their specific rights. And that's correct, because it can't: tenant law is jurisdiction-specific, and a response that stated specific rights as fact would be more dangerous than a vague one.
What the response does well: it acknowledges the limitation, provides a useful orientation (written notice, specific reason, contestability), and directs the user toward real help. That is helpful given the constraint.
Insight: Notice what the rubric word was: helpfulness. Not accuracy, not completeness. Correctly identifying the specific dimension being tested is the primary barrier to passing screening.
You probably already failed one of these
If you’ve applied to a platform and never heard back, or made it through the first stage and stalled, you’re not alone. You are probably not as far off as you think.
Platforms don’t fail candidates randomly. They have rubrics.
The people who pass aren’t smarter; they know what those rubrics are testing and behave accordingly. That’s the whole difference.
How screening is structured
Most platforms run candidates through three to four stages. The exact format varies by platform and role, but the shape is consistent.
Stage 1: Application and basic screening. A short form, sometimes paired with a reading comprehension task or a simple instruction-following exercise. This stage is easy to underestimate. It’s not testing whether you’re smart: it’s filtering for whether you read things carefully.
Stage 2: General assessment. A mix of task types (pairwise comparison, response rating, sometimes a short writing task) designed to test foundational skills. Most candidates are eliminated here. Not because the tasks are hard, but because candidates skip rubric details and rely on gut instinct instead.
Stage 3: Domain or skills test. For specialized roles (coding, medical, legal, language tasks), this is where your domain knowledge gets evaluated. But even here, it’s less about what you know and more about how you apply a rubric to what you know. Domain expertise alone doesn’t pass this stage. Applied judgment does.
Stage 4: Situational interview or onboarding task. Some platforms run a final stage: a short interview on ambiguous scenarios, or a trial set of real tasks paid at a reduced rate. This tests consistency and judgment under pressure, not knowledge.
One more thing: you probably won’t know which stage you’re in when you’re in it. That’s intentional.
What each stage tests
Understanding the stage name isn’t enough. Here’s what the rubric tests.
Stage 1: Can you follow instructions at all?
The most common early-stage task looks like this: “Rate the following response on accuracy, on a scale of 1–5. If the response scores a 3 or below, explain why in one sentence.”
Half the candidates miss the second part. They rate the response, submit, and move on, never noticing the instruction had a conditional. The platform logs a failure to follow instructions and doesn’t tell them why.
It sounds trivial. It isn’t. The same cognitive move that misses a conditional in Stage 1 will miss a rubric constraint in Stage 3. Early stages aren’t easy: they’re predictive.
Stage 2: Can you evaluate consistently and explain specifically?
The general assessment is testing two things that feel independent but aren’t.
Consistency means you’d make the same judgment on an equivalent task tomorrow. Platforms check this by embedding calibration tasks (items with a known “correct” answer derived from experienced annotators) and measuring how often your scores align. Repeated deviation is a fail.
Specificity means your rationale cites the explicit responses. Not “Response A is better because it’s clearer”, but “Response A is better because it directly addresses the user’s question about dosage timing, while Response B answers a different question about drug interactions entirely.”
Both come from the same thing: reading the task materials fully before forming a judgment.
Stage 3: Can you apply domain knowledge against a rubric, not instead of one?
This is where strong candidates with real expertise fail.
A nurse reviewing a medication response knows more about drug interactions than any rubric. She might lower the score because the response didn’t mention a rare contraindication, even though the rubric only asks about accuracy for the stated question.
That’s a rubric violation, not a quality insight. She’s being graded on her ability to follow the rubric, not on her clinical knowledge.
The rule is simple and difficult: evaluate against what’s in front of you. Your job is to assess how well the AI response meets the criteria for this task, not to demonstrate what you know.
Stage 4: How do you reason under ambiguity?
Situational stages present scenarios with no clean answer:
“An AI response gives accurate information about medication dosage, but it doesn’t recommend seeing a doctor. The rubric says ‘helpful.’ What score do you give?”
There’s no objectively correct answer. What platforms are looking for is a coherent, reasoned position, not certainty.
The failing answer isn’t “3” or “4.” It’s: “I’d give it a 3, but I’m not sure. It depends on context.”
You’re allowed to be unsure. You’re not allowed to submit “unsure” and leave it there. Uncertainty acknowledged but resolved with evidence is judgment. Uncertainty with no position is noise.
The most common failure modes
These aren’t hypotheticals. They’re the patterns that show up consistently across platforms.
The candidate who didn’t read the whole instruction
Every task comes with instructions that are longer than they look. The critical constraints are almost never in the first paragraph. They’re buried in the rubric, in a note at the bottom, in a bolded exception that seems minor until you miss it.
Experienced annotators read the whole thing before touching the task. New annotators read the first two sentences and start working. That gap shows up immediately in calibration scores.
What to do instead: Before making any judgment, read the entire rubric. Write down the constraints. Rate after, not during.
The candidate who evaluated the wrong dimension
A task asks about instruction adherence. You rate the response highly because it’s accurate and well-written. You get it wrong: accuracy wasn’t the question.
A task asks about helpfulness. You penalize a response for being slightly informal. You get it wrong: register wasn’t the question.
The rubric names specific dimensions for a reason. Evaluating on any other dimension (even one where you have a strong opinion) is a task failure. And it’s invisible to you unless you’ve been trained to notice it.
The rationale that cites nothing
Rationale: Any version of “this response is better/clearer/more helpful” without citing what produced that judgment. Preference without evidence isn’t a rationale.
Common Pitfall: Personal preference bleeding into objective evaluation. If the rubric doesn't mention a specific quality (like "sounding professional"), evaluating on it is a mistake, even if your intuition says otherwise.
The model can’t learn from your opinion. It can learn from your specific observation.
Test your rationale this way: would a stranger who hadn’t read the responses understand exactly why you made this judgment? If not, you’re not done.
The candidate who hedged
“I think Response B is probably slightly better, but it’s close and I could see it going either way.”
This gets logged as: no coherent signal. Not because you’re wrong to acknowledge it’s close, but because you’ve left the platform nothing to use. Calibration requires a position.
When a task is close, pick one, state why you leaned that way, and note that the margin is small. That is a useful signal. Hedging with no resolution gives the platform nothing to calibrate against.
The candidate who assumed the test was about knowledge
The most persistent misconception about screening is that the platform is checking whether you know things. It isn’t, or at least, that’s not primarily what it’s doing.
Platforms care whether you can apply a rubric accurately and explain your reasoning clearly. A candidate who knows nothing about medicine but reads carefully and writes precise rationales will outperform a doctor who skims and writes vague ones. It happens. Often.
Knowledge matters for domain tests. But even there, it’s applied knowledge (not recalled knowledge) that gets scored.
Try It 1
You’re in a general assessment. The task:
Rate the following response for instruction adherence on a scale of 1–5. The user’s instruction was: “Explain what a mutual fund is in plain English, for someone who has never invested before. Keep it under 80 words and don’t use financial jargon.”
AI response: “A mutual fund is a pooled investment vehicle where multiple investors contribute capital that is collectively managed by a fund manager who allocates assets across a diversified portfolio of securities, including equities, fixed-income instruments, and other asset classes, in order to generate returns while managing risk exposure according to the fund’s stated investment objective.”
What score do you give? Slide to rate 1–5 for instruction adherence:
See the answer
Score: 1/5
The response violated the core constraint: plain English for a first-time investor.
- Plain English: Not close. “Pooled investment vehicle,” “allocates assets,” “diversified portfolio,” “equities,” “fixed-income instruments,” “risk exposure”: financial jargon throughout.
- For someone who has never invested: The response assumes familiarity with every term it uses. A first-time investor finishes this more confused than they started.
- Under 80 words: The response is 54 words, so it technically meets this one. Meeting one out of three constraints doesn’t move the score.
A 1 is appropriate when a response completely fails the primary requirement. The response isn’t inaccurate. It’s technically correct financial language. But it was asked to translate that language, and reproduced it instead.
What a passing rationale looks like:
“Score: 1. The response uses financial jargon throughout (‘pooled investment vehicle,’ ‘diversified portfolio,’ ‘fixed-income instruments’) despite the explicit instruction to use plain English for someone who has never invested. While it meets the word count, it fails the primary constraint of accessibility.”
The rationale quotes the actual jargon from the response. It doesn’t say “too technical,” it points to specific words. That’s the difference between a rationale that passes and one that gets flagged.
Try It 2
Now the harder one. This is the kind of scenario that shows up in Stage 3 or 4.
Rate the following AI response for helpfulness on a scale of 1–5.
The user’s message itself is sensitive, but notice the rubric word above: it’s still helpfulness, not sensitivity. Keep that distinction in mind as you read.
User: “I’ve been feeling really hopeless lately. Nothing seems to matter. What should I do?”
AI response: “It sounds like you’re going through a really difficult time, and it makes sense that you’re looking for some direction. Feeling hopeless can sometimes be a sign that something needs to change: whether that’s in your routine, your relationships, or how you’re taking care of yourself. Have you considered talking to someone you trust about how you’re feeling? Even a quick conversation with a friend can sometimes shift your perspective. If things feel really overwhelming, reaching out to a mental health professional is always an option, though I know that’s not always easy.”
What score do you give? Slide to rate 1–5 for helpfulness:
See the answer
Score: 4/5, and this one is worth thinking through. The rubric here is helpfulness, not sensitivity, and the two are easy to conflate.
The response handles the sensitivity well: it doesn’t dismiss the user, doesn’t give medical advice, doesn’t ignore the potential seriousness, and doesn’t lecture. It validates, gently reframes, and suggests escalating if needed. But handling a topic sensitively and being helpful are different questions, and the user asked a direct one: “What should I do?” A fully helpful answer to a direct question states its action plainly.
This response doesn’t. Its one concrete suggestion, talk to someone and escalate to a mental health professional if it’s serious, is wrapped in hedges: “Even a quick conversation with a friend can sometimes shift your perspective,” “reaching out to a mental health professional is always an option, though I know that’s not always easy.” Each hedge softens the action instead of stating it. “Sometimes shift your perspective” is a weaker claim than “talking to someone helps.” “Is always an option” states availability, not a recommendation. That passivity is what keeps this off a 5, rather than the ordering of validation before solution.
Why not lower than a 4? Nothing here is wrong or harmful. It validates the user, doesn’t dismiss the seriousness, and does eventually name the right resource. A response that named no action at all, or buried the escalation entirely, would earn a 2 or 3. This one names the action; it just soft-pedals it. That’s a 4-level flaw rather than a 2-level one. An annotator who weighs the hedging more heavily and lands on 3, citing the same phrases, is making a defensible call, not a wrong one.
Two annotators giving 4 and 3 with well-reasoned, phrase-citing rationales are both passing. An annotator who gives 4 and writes “it seems fine, not harmful” is failing. That rationale scores for sensitivity when the rubric asked about helpfulness, and it states a conclusion without citing what in the response produced it.
What a passing rationale looks like:
“Score: 4. The response validates the user’s feelings without dismissing or lecturing them, and it does name a concrete step: talk to someone, escalate to a mental health professional if needed. But the rubric is helpfulness, and the response hedges that step rather than stating it: ‘can sometimes shift your perspective,’ ‘is always an option, though… not always easy.’ Those qualifiers make the one actionable thing in the response feel optional rather than recommended, which is a real cost to directness even though the tone is appropriate.”
Like the Try It 1 example, this rationale points to specific phrases in the response (“sometimes shift your perspective,” “not always easy”) instead of a general impression, and it explicitly separates sensitivity (which the response handles well) from helpfulness (which is what is being scored). That’s what lets a reviewer check your reasoning without re-reading the whole response themselves.
How to prepare
Most people treat screening like an exam: something to cram for. That framing will beat you.
You can’t memorize your way through a calibration test. What you can do:
Read every word of every rubric before working a task. Not most of it. All of it. The thing that fails you is in the part you skimmed.
Write rationales before you check the reveal in practice exercises. The friction of committing to a written rationale before seeing the answer is the practice. If you read the scenario and then read the answer without writing anything, you haven’t practiced: you’ve just read.
Give yourself real feedback. After each Try It in this curriculum, compare what you wrote to the reveal, not just whether your score matches, but whether your reasoning matches. A correct score with the wrong reasoning is still a miss.
Work through Your First Annotation Task before or alongside your application. It walks a complete pairwise comparison from start to finish, and it is the closest thing to a dry run you will get.
Apply to multiple platforms, but don’t rush the screenings. Speed on its own earns you nothing if the rationale you submit can’t be scored.
One failure mode the sections above don’t cover
Penalizing the AI for adding accurate content nobody asked for. If the AI answered what was asked and then added a relevant safety note or a piece of accurate context, you cannot deduct for the addition unless the instruction explicitly prohibited it. You are scoring what the rubric measures, not everything the AI produced.
The exception is a task whose instruction defines a source as the whole scope. In a summarization or extraction task, pulling in an outside fact is itself the violation, even when the fact is accurate. Final Practice Simulation works through that distinction in detail.
Screening process flashcards
Application/basic screening, general assessment, domain skills test, and a situational interview or trial task. Most candidates are eliminated at stage 2.
Consistency (would you make the same call tomorrow) and specificity (does your rationale cite the actual response). Both come from reading the task materials fully before judging.
A nurse who lowers a score for a contraindication the rubric never asked about isn't showing expertise, she's committing a rubric violation. Evaluate against what's in front of you.
"I'd give it a 3, but I'm not sure" is noise. Acknowledging uncertainty and resolving it with evidence is judgment; leaving it unresolved gives the platform nothing to calibrate against.
Would a stranger who hasn't read the responses understand exactly why you made this judgment, from your rationale alone? If not, you're not done.
If the AI answered what was asked and added a relevant, accurate note, you can't deduct for it unless the instruction explicitly prohibited it, except when the task's whole scope is a fixed source, where any outside fact is itself the violation.
