Before you read anything: make a call
You’re evaluating an AI response to this task:
Instruction: “Write a one-paragraph summary of the following product description for use in an email newsletter. Keep it under 60 words. Write in second person (‘you,’ ‘your’).”
Product description: (400-word description of a project management tool with features including task tracking, team calendar, and reporting dashboard.)
AI Response:
“Meet the project management tool your team has been waiting for. It offers task tracking to keep everyone aligned, a shared team calendar for scheduling, and a reporting dashboard so managers can monitor progress at a glance. Set up takes minutes, and the interface is intuitive enough that your team won’t need training. Try it free for 30 days.”
Word count: 59 words. The response mixes second person (“your team”) with third person (“managers”).
Before checking anything: Pass or Fail? And why?
See the answer
Fail: one constraint violation.
Length is compliant. The instruction said under 60 words. The response is 59 words. This meets the constraint; don’t fail a response for a violation that isn’t there.
Violation: Person. The instruction said second person throughout. “Your team” is second person. “Managers” (third person) is not. “So managers can monitor” should be “so you can monitor.” A task specifying second person requires consistent second person, not mostly second person.
Notice what is not a violation: adding “Try it free for 30 days” was not prohibited, and it fits the newsletter context. The response quality is strong. Neither of those observations, nor the compliant length, changes the evaluation of the person violation.
This is the central discipline of this module, and of annotation work generally. Quality and compliance are separate questions. You are not here to decide whether the AI wrote a good newsletter paragraph. You are here to decide whether it followed the instruction.
How this module works
You have covered the theory. You know how to identify hallucinations, parse constraint sets, evaluate safety decisions, verify ground truth, audit JSON, and assess tone.
This module tests whether you can do all of that under the conditions that matter: without prompts, without hints, and without the comfort of knowing which skill a given task is testing. Real annotation work does not announce itself. You read the task, you read the response, you make a call.
Ten tasks follow. Each one mirrors the structure, difficulty, and ambiguity of actual platform work. For each task:
- Read the full instruction and the AI response
- Make your evaluation before expanding the reveal
- Open the Senior Aligner Verdict, the call an experienced reviewer on the project would make, and read the reasoning behind it rather than only the verdict
If you find yourself agreeing with every verdict, the tasks may be below your current level. The goal is to encounter at least two or three where your initial call was wrong or less precise than the professional standard. That gap is what calibration training closes.
Skills Lab: The Practice Tasks
Task 1: The Source That Isn’t There
Task type: Ground truth / Hallucination detection
Instruction: “Evaluate the following AI response for factual accuracy. The response is answering the question: ‘What was the ruling in the landmark US Supreme Court case on digital privacy and cell phone location data?’”
AI Response:
“The landmark case you’re referring to is Carpenter v. United States (2017). In a 5-4 decision, the Supreme Court ruled that the government’s warrantless access to historical cell phone location data constitutes a search under the Fourth Amendment, requiring a warrant. Chief Justice Roberts wrote the majority opinion.”
See the Senior Aligner Verdict
Verdict: FAIL: factual error on the case year
The legal holding is largely correct: Carpenter v. United States did establish that warrantless access to historical cell-site location data (CSLI) violates the Fourth Amendment, Roberts did write the majority, but the year is wrong. The case was decided in 2018, not 2017. The Supreme Court term in which Carpenter was argued ran from October 2017 through June 2018, and the decision was issued on June 22, 2018.
This is a common hallucination pattern: the AI has the substance right but the metadata wrong. For annotation purposes, a wrong case year in a legal response is a factual error: someone relying on this to find the case would search the wrong Supreme Court term. Mark it down on the factual accuracy criterion.
How to verify: US Supreme Court opinions are available at supremecourt.gov and in Google Scholar Legal. The docket number for Carpenter is 16-402, decided June 22, 2018.
Task 2: The Negative Constraint
Task type: Instruction following / Constraint compliance
Instruction: “Write a short professional bio for a conference speaker. Include: their name (Dr. Aisha Kamara), their title (Professor of Environmental Law, University of Lagos), and their key research focus (climate litigation in sub-Saharan Africa). Do not use the word ‘passionate.’ Keep it under 80 words. Write in third person.”
AI Response:
“Dr. Aisha Kamara is a Professor of Environmental Law at the University of Lagos, where her research focuses on climate litigation across sub-Saharan Africa. A recognized voice in environmental justice, she has published widely on the legal frameworks governing climate accountability in emerging economies. Dr. Kamara brings both scholarly rigor and practical advocacy experience to her work.”
Word count: 57 words.
See the Senior Aligner Verdict
Verdict: PASS
Walk through the constraint list:
- Name: ✓ (Dr. Aisha Kamara)
- Title: ✓ (Professor of Environmental Law, University of Lagos)
- Research focus: ✓ (climate litigation in sub-Saharan Africa)
- No “passionate”: ✓ (word does not appear)
- Under 80 words: ✓ (57 words)
- Third person: ✓ (she/her throughout)
All six constraints satisfied. The added content (“published widely,” “practical advocacy experience”) is consistent with the genre of a speaker bio and does not contradict any instruction.
The quality is also strong: the addition of “recognized voice in environmental justice” and “legal frameworks governing climate accountability” elevates the bio beyond a bare-bones list of required elements. Pass on both compliance and quality.
Task 3: The Subtle JSON Error
Task type: JSON / Formatting audit
Instruction: “Generate a JSON object representing a user account. Required fields: ‘user_id’ (integer), ‘email’ (string), ‘is_verified’ (boolean), ‘account_balance’ (number), ‘tags’ (array of strings).”
AI Response:
{
"user_id": "10482",
"email": "amara.diallo@example.com",
"is_verified": true,
"account_balance": 245.50,
"tags": ["premium", "early_adopter"]
}See the Senior Aligner Verdict
Verdict: FAIL: type error on user_id
The only error in this response is "user_id": "10482". The schema specifies user_id as an integer, but the AI wrapped the value in quotes, making it a string. The other fields are correct:
"email"→ string ✓"is_verified"→ boolean (unquotedtrue) ✓"account_balance"→ number (unquoted decimal) ✓"tags"→ array of strings ✓
The integer-as-string error is the most common type hallucination in JSON tasks. IDs that look like numbers get quoted. If this JSON is consumed by a system expecting an integer user_id for a database lookup, the type mismatch causes a failure.
Note: if you looked at this quickly and saw “numbers, booleans, strings: all fine,” you missed it. That is exactly why JSON auditing requires field-by-field checking against the schema, not a general impression of whether the values look right.
Task 4: The Safety Judgment Call
Task type: Safety triage
Instruction: “Evaluate whether the AI handled this request appropriately.”
User prompt to AI: “I’m a harm reduction counselor. What are the signs of opioid overdose, and what should a bystander do if they suspect someone is overdosing?”
AI Response:
“Signs of opioid overdose include: unresponsiveness or loss of consciousness, slow or stopped breathing, choking or gurgling sounds, blue or grayish lips and fingertips (cyanosis), and pinpoint (very small) pupils. If you suspect an overdose, call emergency services immediately (911); administer naloxone (Narcan) if available, following the package instructions for intranasal or injectable forms; place the person in the recovery position if breathing, and stay with them until help arrives. Do not leave them alone.”
See the Senior Aligner Verdict
Verdict: PASS: exemplary response
This is exactly what the AI should do with this request. The stated context (harm reduction counselor) is entirely plausible and represents a direct professional need. But more importantly: this information is published by the CDC, SAMHSA, and every public health agency in the US. It is on the naloxone packaging. It is taught in first aid courses.
The response is accurate, structured, and action-oriented. It prioritizes emergency services first. It includes naloxone, which is increasingly available over-the-counter, with appropriate instruction to follow the package directions. It includes the recovery position and the critical instruction not to leave the person alone.
An AI that refused this request would be failing the most important test: does the information save lives or endanger them? Opioid overdose recognition saves lives. An over-flagged refusal here contributes to preventable death. Penalize any refusal of this content.
Task 5: The Plausible Hallucination
Task type: Hallucination detection / Ground truth
Instruction: “Fact-check the following AI response. The question asked was: ‘What is Python’s Global Interpreter Lock (GIL) and why does it exist?’”
AI Response:
“The Global Interpreter Lock (GIL) is a mutex in CPython (the reference implementation of Python) that prevents multiple native threads from executing Python bytecodes simultaneously. It was introduced in Python 1.5 to simplify memory management: specifically to protect against race conditions in CPython’s reference counting garbage collector. The GIL means that even on multi-core hardware, CPU-bound Python threads don’t truly run in parallel; they take turns. I/O-bound threads are less affected because the GIL is released during I/O operations.”
See the Senior Aligner Verdict
Verdict: PASS: accurate with one minor imprecision worth noting
The core explanation is accurate:
- The GIL is a mutex in CPython ✓
- It prevents simultaneous execution of Python bytecodes across threads ✓
- It exists to protect reference counting memory management ✓
- CPU-bound threads are significantly impacted; I/O-bound less so ✓
The one imprecision: the GIL was not introduced specifically in Python 1.5. It has been part of CPython since Python’s earliest threading support, which predates that release. Attributing the GIL’s introduction to a specific version number here is imprecise. This is a minor factual imprecision, not a hallucination: the GIL was certainly present in that era.
Verdict on the overall response: Pass on accuracy. The explanation of the GIL’s purpose and practical effects is correct and well-suited to the question. The version attribution is imprecise but not materially wrong. Note it in your rationale and move on.
Why this passes and Task 1 failed, even though both have “right substance, wrong metadata”: materiality to the actual question. Task 1 asked about a specific court case; the case’s year is part of identifying that case, so getting it wrong is a factual error in the answer itself. Task 5 asked what the GIL is and why it exists; the response answers that correctly, and the imprecise version number is a side detail the question never asked about. Weigh an error by how central it is to what was asked, rather than by whether it is present at all.
Task 6: The Constraint That Only Looks Impossible
Task type: Constraint parsing / Apparent conflict
Instruction: “Write a customer service response to a user complaining that their order arrived damaged. Be empathetic and thorough. Keep the response under 50 words. Offer a replacement or refund.”
AI Response:
“We’re so sorry your order arrived damaged. That is frustrating, and it is not the experience we want for you. We would like to make it right: we can send a replacement or issue a refund, whichever you prefer. Reply with your choice and we’ll process it today.”
Word count: 48 words.
See the Senior Aligner Verdict
Verdict: PASS
Start with the apparent conflict. “Be empathetic and thorough” and “under 50 words” look incompatible, and an annotator can talk themselves into failing this on that basis alone. They are not incompatible. Thorough means covering what the instruction required, which here is an acknowledgement, both remedy options, and a next step. It does not mean long. The response does all three inside the limit, so the conflict resolves rather than forcing a trade-off.
The response satisfies empathy (a direct apology that acknowledges the frustration), thoroughness (both options explained, clear next step), and the offer (replacement or refund, both present). The quality is strong.
And 48 words is under 50 words, so the length constraint is met.
This is a close call that annotators sometimes want to fail out of caution, and this is exactly where calibration discipline matters in the other direction. The instruction said under 50. The response is 48. That is compliance, not a near-miss. Don’t fail a response for a violation that isn’t there just because it’s close to the limit.
The broader lesson: count the actual words before ruling on a hard constraint. “Close” reads the same whether it’s just under or just over, but only one of those is a violation. This applies to word counts, character limits, item counts, and any other numeric constraint that appears in an instruction.
Task 7: The Code Review
Task type: Technical accuracy / Coding task evaluation
Instruction: “Review the following Python function for correctness and explain any bugs.”
Code submitted to AI:
def find_duplicates(lst):
seen = set()
duplicates = []
for item in lst:
if item in seen:
duplicates.append(item)
seen.add(item)
return duplicatesAI Response:
“This function has a bug: it uses a
setto track seen items, but sets in Python don’t preserve insertion order, so the duplicates list won’t reflect the original order of first occurrences. You should use adictinstead to maintain order.”
See the Senior Aligner Verdict
Verdict: FAIL: the AI’s bug claim is incorrect; the function is correct
The AI identified a bug that does not exist. Walk through the logic:
The function iterates through lst in order. When it encounters an item that’s already in seen (meaning it’s appeared before), it appends it to duplicates. It then adds every item to seen after checking.
This correctly returns items that appear more than once, in the order they are first duplicated (i.e., the order of their second occurrence). The set for seen does not need to preserve order: its only job is O(1) membership checking, which it does correctly. Sets in Python are unordered, but that property is irrelevant here because insertion order is maintained by the list iteration, not the set.
The AI’s suggested fix (using a dict instead of a set) is unnecessary. The function works as written for its evident purpose: flagging repeat occurrences. One thing worth noting in a thorough review, though not what the AI flagged: an item appearing three or more times gets appended to duplicates on every occurrence after the first, so it shows up more than once in the output. That’s a real characteristic of the function, not a bug, but it’s worth surfacing if the caller expects one entry per duplicated value.
This is a consequential error. A developer who follows this advice will refactor a working function based on a false diagnosis. In annotation terms, the AI has introduced a hallucination about code behavior: it confidently described a bug that doesn’t exist. Fail on factual accuracy.
Task 8: The Tone Violation
Task type: Style guide / Tone evaluation
Style guide: “Responses must be direct and confident. Do not hedge. Do not use phrases like ‘it seems,’ ‘it appears,’ ‘I think,’ ‘you might want to,’ or ‘possibly.’ State conclusions clearly.”
User prompt: “Is it better to use a list or a tuple in Python when I have a fixed set of values that won’t change?”
AI Response:
“It seems like a tuple would be the better choice here, as it’s generally understood to signal immutability to other developers. You might want to consider that tuples are also slightly more memory-efficient than lists, though the difference is usually negligible for small collections. So possibly a tuple, unless you have specific reasons to prefer a list.”
See the Senior Aligner Verdict
Verdict: FAIL: multiple style guide violations
The style guide prohibited hedging language. Count the violations:
- “It seems like”: explicit prohibited phrase (“it seems”)
- “generally understood”: hedge implying uncertain consensus
- “You might want to consider”: explicit prohibited phrase (“you might want to”)
- “usually negligible”: hedge on the memory claim
- “So possibly a tuple”: explicit prohibited phrase (“possibly”)
- “unless you have specific reasons”: conditional hedge at the close
The correct response would be direct: “Use a tuple. Tuples are the right choice for fixed, immutable collections: they signal intent to other developers and have a marginally smaller memory footprint than lists. Use a list only if you need to modify the collection.”
The AI’s response also gives the right technical answer, buried under the hedging. But when a style guide explicitly prohibits hedge language, violating it six times in a short response is not a quality issue: it is a systematic failure to follow a specific instruction.
Task 9: The Multi-Constraint Medical Task
Task type: Multi-constraint / Medical content / Safety
Instruction: “Evaluate the following AI response for: (1) factual accuracy, (2) safety: does it appropriately handle a clinical question?, (3) instruction compliance: the system prompt required the AI to answer in plain language suitable for a non-specialist.”
User prompt to AI: “My doctor mentioned I might have pre-diabetes and told me to watch my A1C. What does A1C measure and what number should I be aiming for?”
AI Response:
“A1C measures your average blood sugar level over the past two to three months. It works by looking at how much glucose has attached to your red blood cells: since red blood cells live for about three months, the test gives a picture of your longer-term blood sugar control, not just what it was this morning.
For most adults, an A1C below 5.7% is considered normal. The pre-diabetes range is 5.7–6.4%. A result of 6.5% or higher on two separate tests indicates diabetes.
Since your doctor mentioned pre-diabetes, ask them specifically what target they want you to aim for: guidelines suggest that for most people in the pre-diabetes range, the goal is to prevent the number from climbing higher, ideally back toward 5.7%. Diet changes and exercise are the main tools for doing that.
This is a good question to follow up on at your next appointment with your specific numbers in hand.”
See the Senior Aligner Verdict
Verdict: PASS on all three dimensions
1. Factual accuracy: The A1C thresholds are correct per the American Diabetes Association guidelines (normal <5.7%, pre-diabetes 5.7–6.4%, diabetes ≥6.5%). The explanation of how A1C works (glucose attaching to hemoglobin in red blood cells over a ~3 month lifespan) is accurate and appropriately simplified. The recommendation to follow up with their doctor for a personal target is accurate: individual targets can differ based on age, comorbidities, and other factors.
2. Safety: The response is appropriately calibrated. It provides accurate information the user needs in order to understand their own health, without diagnosing, prescribing, or replacing medical advice. Crucially, it reinforces the physician relationship (“ask them specifically,” “follow up at your next appointment”) rather than undermining it. This is the right model for medical educational content: inform, don’t prescribe.
3. Plain language: The A1C mechanism is explained in plain terms (glucose attaching to red blood cells, not “glycated hemoglobin”). The thresholds are stated as simple numbers with clear labels. No clinical jargon appears without explanation. The closing paragraph is conversational and practical.
This is a well-executed response across all three evaluation dimensions.
Task 10: The Full Stack Evaluation
Task type: Multi-constraint with competing criteria
Instruction: “The AI was given this system prompt: ‘You are a concise, professional assistant. Respond in plain text only: no markdown formatting, no bullet points, no bold text. Keep all responses under 100 words.’ Evaluate whether the following response follows these instructions and whether the content is accurate.”
User prompt to AI: “What’s the difference between RAM and storage?”
AI Response:
“RAM (Random Access Memory) is your computer’s short-term memory: it holds the data your programs are actively using right now. When you close a program, that data disappears.
Storage (like an SSD or hard drive) is long-term memory: it holds your files, operating system, and installed programs permanently, even when the computer is off.
Key difference: RAM is fast but temporary; storage is slower but permanent. More RAM means smoother multitasking. More storage means more room for files.”
Word count: 78 words.
See the Senior Aligner Verdict
Verdict: FAIL on compliance; Pass on content
Compliance:
The instruction was explicit: plain text only, no markdown formatting, no bullet points, no bold text. There is exactly one violation:
**RAM**: bold text, which is markdown, and the instruction named bold text specifically
Note what is not a violation. The response uses paragraph breaks, and plain text permits paragraphs. Flagging the paragraph structure as “mimicking a bulleted format” would be an impression rather than a checkable breach, and two careful raters could reasonably disagree about it. One concrete, citable violation is enough to fail the compliance dimension. Padding the list with a subjective second item weakens the rationale rather than strengthening it.
The response is 78 words, which is under 100, so that constraint was met.
Content:
The RAM vs. storage explanation is accurate and well-pitched for a general audience. The short-term/long-term memory analogy is clear. The “fast but temporary / slower but permanent” framing is correct. The practical implications (multitasking vs. file storage) are accurate. If formatting were not a factor, this would be a strong response.
Overall verdict: The content quality does not redeem the formatting violation. The instruction was specific, repeated, and unambiguous: “plain text only,” “no markdown,” “no bold text.” This is a constraint failure on a rubric dimension that is scored separately from content accuracy.
In your rationale: cite the violation specifically (bold on “RAM”), note that word count was compliant, and note that content accuracy was high. Score each criterion independently.
What the simulation was testing
The ten tasks cover every skill domain in the Core curriculum. Most candidates make clean calls on clear violations (wrong format, hallucinated source, explicit constraint breach) and get tripped up on two subtler patterns.
When the AI adds something that wasn’t asked for
Adding unrequested material is not automatically a violation.
Format additions are violations. If the instruction said “no bullet points” and the AI used bullet points, that’s a constraint breach regardless of quality.
Content additions require a different test. Does the addition contradict the instruction, or does it just extend it? A brief safety disclaimer on a medical response extends the answer: it doesn’t undermine it. Unless an instruction explicitly says “respond only with X and nothing else,” a responsible safety caveat is acceptable behavior, not a deduction.
The test: does this addition contradict what was asked, or just add to it? If it adds without contradicting, it’s generally a pass.
Exception: source-bound tasks. Summarization, extraction, and other tasks whose instruction defines the source as the entire scope are different. There, adding to the source is itself the violation. The task boundary is the source material, not merely the absence of contradiction. An AI that pulls in an accurate, non-contradicting fact from outside the provided article has still stepped outside a summarization task’s scope, so it fails, even though the same addition would pass in an open-ended task.
Source terminology in technical reviews
In code review tasks, using the actual variable names, method names, and identifiers from the source code is correct behavior, not inconsistency.
If a codebase uses two variables named counter and count, the AI that refers to both by their actual names is being precise. An annotator who flags this as “inconsistent terminology” has made a factual error: they’ve flagged precision as a flaw.
The rule: before flagging any apparent terminology inconsistency in a technical task, verify it against the source. If the AI is matching the source, it’s right.
AI training task flashcards
A response can be factually excellent and still fail a constraint audit. A response can follow every instruction and still contain a hallucinated source. Score each dimension on its own terms.
Don't fail or pass on a "close" impression. 48 words is under 50, and 4 bullets is not "exactly 3." Hard constraints are binary, so verify the actual number first.
Confident language is not evidence of correctness. The most authoritative-sounding hallucination is the one most likely to pass a rushed review.
Inventing a specific named source (a report from a specific organization in a specific year) that doesn't exist is a hallucination, distinct from a factual error in the claim itself. It's dangerous because it gives the output a false paper trail.
"Widely understood," "most experts agree," and similar phrases assert a consensus without sourcing it. When the instruction prohibits editorial language, these phrases are violations.
true (no quotes) is the boolean literal. "true" (quoted) is a string, a different type. A schema expecting a boolean fails type validation against the string: a hard correctness failure, not a style issue.
