Before you read anything, try this first
A screening question gives you two versions of the same investment outcome:
Option A: You are guaranteed to keep 1,000 fund. Option B: You have a 90% chance to keep the full $1,000, and a 10% chance to keep nothing.
Both options have the same expected value: $900. Most people pick Option A anyway.
Which explanation is the exam testing?
The first one. The two options have equal expected value. The exam isn't checking whether you can do the arithmetic. It's checking whether you recognize the pattern: a guaranteed outcome is overweighted relative to a probabilistically equivalent one.
That pattern has a name: the certainty effect. Once you can name it, the "trick" stops being a trick. It's just a labeled concept with a predictable shape.
Insight: These questions rarely ask you to compute anything. They ask you to identify which named bias or design flaw is being illustrated. Compute the numbers, notice they're equal or irrelevant, then look for the pattern.
Why a data-labeling job asks you this
It looks out of place. You applied to rate AI responses or write training data, and the screening exam is asking you to explain why people misjudge probabilities.
It fits once you see what the job is. Evaluating an AI response for quality often means judging whether the response itself reasons well about uncertainty, tradeoffs, and cause and effect. A rubric that says “penalize overconfident claims” or “flag responses that treat a correlation as a cause” assumes you already know what overconfidence and correlation-versus-causation look like. The exam checks that baseline before the job hands you a rubric that leans on it.
The two clusters that come up most: judgment and decision-making (how people systematically misjudge probability, value, and risk) and basic experimental design (what separates a claim you can trust from one you can’t). Neither requires an economics or statistics background. Both are learnable from a short list of named patterns.
These biases describe predictable ways people misjudge a situation
Each of these has one job: describe a specific, predictable way that people misjudge a situation. Screening questions present a scenario and ask you to name the pattern or predict the biased outcome.
Certainty effect. A guaranteed outcome is weighted more heavily than a probabilistically equivalent uncertain one. You just saw this above. It also runs in reverse: a guaranteed loss is judged worse than a larger probable loss with the same expected value, which is why people take bad gambles to avoid a certain, smaller loss.
Availability heuristic. People judge how likely or common something is by how easily examples come to mind, not by actual frequency. A vivid, memorable event (a plane crash reported for a week) makes a rare risk feel common. A common but unremarkable event (a car accident) feels less risky because it doesn’t come to mind as easily. If a scenario has someone overestimating a risk right after hearing a dramatic story about it, that’s availability.
Anchoring. An initial number, even an arbitrary one, pulls subsequent estimates toward it. If a scenario shows someone’s price estimate shifting after being shown an unrelated high or low number first, that’s anchoring.
Loss aversion. Losing something is felt more strongly than gaining the equivalent amount. A person who wouldn’t pay 20 once they own it. Loss aversion explains decisions that look inconsistent (valuing the same object differently depending on whether you already have it) once you know what to look for.
Sunk cost reasoning. Continuing an effort because of what’s already been spent, rather than what remains to be gained. A scenario describing someone finishing a bad project because they’ve “already put so much into it” is describing sunk cost, not commitment.
Base rate neglect. Ignoring how common something is overall in favor of specific-sounding details. A scenario with a rare disease, a mostly-accurate test, and a question asking for the real probability of having the disease given a positive result is testing this. The specific test result feels more informative than it is once you account for how rare the disease is.
Try It: name the pattern
For each scenario, name the concept being illustrated.
Scenario 1: After reading several news stories about a rare cyberattack, a manager insists the company needs to spend most of its security budget defending against that specific attack, even though far more common and costly incidents (phishing, weak passwords) get far less budget.
Scenario 2: A team has spent eight months and a large budget building a feature. User testing shows nobody wants it. The team decides to keep building it anyway because of how much has already gone into it.
Scenario 3: Someone is offered a single coin flip: heads, they gain 100. The expected value of the flip is 100 downside is an amount they can comfortably afford to lose. They turn it down anyway.
See answer
Scenario 1 → Availability heuristic. The rare attack is vivid and recently reinforced by news coverage, so it feels more likely and more urgent than the statistically bigger risks that don’t come with a memorable story attached.
Scenario 2 → Sunk cost reasoning. The decision to keep building is justified by past investment (time, budget already spent) rather than the actual evidence in front of them (nobody wants it). What’s already spent is gone either way; it shouldn’t factor into what to do next.
Scenario 3 → Loss aversion. The potential gain (100), the odds are even, and the downside is affordable, so the bet is favorable. Declining it only makes sense if losses are weighted more heavily than equivalent gains. Someone who valued a dollar gained the same as a dollar lost would take this bet every time.
The move in every case is the same: identify the numbers or facts that make the decision look reasonable on the surface, then check whether it’s inconsistent with what a purely rational calculation would produce. The gap between the two is the bias.
Experimental design separates a real claim from one that looks real
The second cluster tests whether you can spot the difference between evidence that supports a claim and evidence that only looks like it does. You don’t need to design a study. You need to recognize when one is missing a piece.
Control versus treatment. A study needs a group that receives the intervention (treatment) and a comparable group that doesn’t (control), so the outcome can be attributed to the intervention rather than to something else that changed anyway. A before-and-after measurement on one group, with no control group, can’t rule out the possibility that the change would have happened regardless.
Causal versus observational. An experiment that randomly assigns subjects to conditions can support a causal claim. A study that simply observes an existing pattern (people who do X tend to have more of Y) cannot, no matter how strong the pattern is, because the people who happened to do X might differ from the rest in ways that also explain Y.
Confounding variables. A third factor that influences both the supposed cause and the supposed effect, creating a correlation between them that has nothing to do with one causing the other. Ice cream sales and drowning rates both rise in summer. Ice cream doesn’t cause drowning. Warm weather drives both.
Sample size and selection. A result from five people, or from a group that wasn’t randomly chosen, doesn’t generalize the way a large, representative sample does. A scenario describing a strong-sounding result from a tiny or self-selected group is testing whether you notice the sample problem before the conclusion.
Try It: evaluate the claim
A company reports: “Employees who use our productivity app report 30% higher job satisfaction than employees who don’t. Our app clearly improves job satisfaction.”
What’s the flaw, and what would need to be true for the causal claim to hold up?
See answer
The flaw is observational data being read as a causal claim. Employees who chose to use the app were not randomly assigned to use it. They opted in. People who already care more about their own productivity and satisfaction may be more likely to seek out a tool like this in the first place. That’s a plausible confounding variable: some underlying trait or circumstance driving both app adoption and satisfaction, rather than the app driving satisfaction on its own.
For the causal claim to hold up, the company would need something closer to a controlled experiment: randomly assign a group of otherwise similar employees to either use the app or not, hold other conditions constant, and compare satisfaction afterward. Random assignment is what rules out the alternative explanation, that some pre-existing difference between app users and non-users, not the app, produced the gap.
A passing answer names the specific issue (self-selection or confounding, not random assignment) rather than a general “correlation isn’t causation.” Naming the mechanism is what the exam is checking for.
The one thing that trips people up
The instinct on these questions is to do the math and stop. The numbers are usually either equal, irrelevant, or a distraction from the actual thing being tested.
Work through it in this order: read the scenario, do any arithmetic quickly to confirm whether the options are equivalent, then ask what named concept the setup is built to illustrate. The concept, not the number, is the answer the exam wants.
If a question doesn’t give you multiple choice and instead asks you to explain a decision in your own words, name the concept explicitly in your answer. “The certainty effect explains why the guaranteed option was chosen despite equal expected value” reads as a stronger answer than a description of the scenario without the label attached.
Behavioral economics flashcards
- The move: confirm the numbers are equal or irrelevant, then name the specific bias or design flaw the scenario is built to illustrate.
- Four experimental design checks: is there a control group, is the evidence causal or just observational, is there a confounding variable, and is the sample large and representative enough to generalize.
A guaranteed outcome is weighted more heavily than a probabilistically equivalent uncertain one, in both directions: people overvalue a sure gain and overreact to avoid a certain loss.
People judge how likely something is by how easily examples come to mind, not by actual frequency. A vivid, memorable event makes a rare risk feel common.
An initial number, even an arbitrary one, pulls subsequent estimates toward it.
Losing something is felt more strongly than gaining the equivalent amount, which is why someone who wouldn't pay a given price for an item still refuses to sell it for that same price once they own it.
Continuing an effort because of what's already been spent, rather than what remains to be gained.
Ignoring how common something is overall in favor of specific-sounding details, like treating a positive test result as more informative than it is without accounting for how rare the underlying condition is.
