Natural experiments

Overview

Natural experiments represent a critical reasoning pattern in LSAT Logical Reasoning sections, particularly within questions involving causation and explanation. Unlike controlled laboratory experiments where researchers manipulate variables deliberately, natural experiments occur when real-world circumstances create conditions that allow researchers to observe causal relationships without direct intervention. For instance, when a city implements a new traffic policy while a neighboring city does not, researchers can compare outcomes between the two cities to infer the policy's effects. Understanding this concept is essential for LSAT success because the test frequently presents arguments that rely on observational data from naturally occurring situations to establish causal claims.

The LSAT tests natural experiments across multiple question types, including Strengthen, Weaken, Flaw, Assumption, and Evaluate questions. Test-makers favor this topic because it requires students to distinguish between genuine causal evidence and mere correlation, assess whether comparison groups are truly analogous, and identify confounding variables that might undermine causal conclusions. Students who master natural experiments gain a significant advantage in analyzing real-world reasoning patterns that appear throughout the Logical Reasoning sections.

Within the broader framework of LSAT natural experiments and causal reasoning, this topic connects intimately with other logical reasoning concepts including correlation versus causation, alternative explanations, necessary and sufficient conditions, and the evaluation of evidence quality. Natural experiments serve as a bridge between pure logical structure and practical reasoning about empirical claims, making them indispensable for achieving a competitive LSAT score.

Learning Objectives

• [ ] Identify how Natural experiments appears in LSAT questions
• [ ] Explain the reasoning pattern behind Natural experiments
• [ ] Apply Natural experiments to solve LSAT-style problems accurately
• [ ] Distinguish between valid and invalid natural experiment comparisons
• [ ] Recognize confounding variables that undermine natural experiment conclusions
• [ ] Evaluate whether comparison groups in natural experiments are sufficiently similar
• [ ] Construct strengthening and weakening scenarios for natural experiment arguments

Prerequisites

• Basic understanding of causation: Natural experiments fundamentally concern causal relationships, so students must distinguish causes from effects and recognize that correlation does not automatically establish causation.
• Familiarity with argument structure: Students should identify premises, conclusions, and the logical connections between them, as natural experiment arguments follow standard argument patterns.
• Knowledge of comparison reasoning: Natural experiments rely on comparing two or more groups, requiring students to evaluate whether comparisons are fair and relevant.
• Understanding of evidence evaluation: Students must assess whether evidence adequately supports a conclusion, a skill central to analyzing natural experiment arguments.

Why This Topic Matters

Natural experiments appear with remarkable frequency on the LSAT, showing up in approximately 15-20% of Logical Reasoning questions across recent administrations. This high frequency reflects the real-world importance of observational reasoning—most causal claims in law, policy, and everyday life derive from natural experiments rather than controlled laboratory studies. Lawyers regularly encounter arguments based on comparing jurisdictions with different laws, analyzing outcomes before and after policy changes, or evaluating evidence from naturally occurring differences between groups.

On the LSAT, natural experiments most commonly appear in:

• Weaken questions: Where students must identify factors that undermine the comparison between groups
• Strengthen questions: Where students must provide evidence that the comparison groups are truly analogous
• Flaw questions: Where the argument incorrectly assumes that compared groups are similar in all relevant respects
• Assumption questions: Where the argument depends on unstated premises about group similarity
• Evaluate questions: Where students must determine what additional information would help assess the argument's validity

The practical significance extends beyond test performance. Legal reasoning frequently involves natural experiments: comparing crime rates before and after sentencing reforms, analyzing accident rates in jurisdictions with different traffic laws, or evaluating the effects of regulatory changes by comparing affected and unaffected industries. Mastering this reasoning pattern develops critical thinking skills essential for law school and legal practice.

Core Concepts

Definition and Structure of Natural Experiments

A natural experiment occurs when circumstances beyond researchers' control create conditions resembling an experimental setup, allowing observation of causal relationships through comparison. Unlike controlled experiments where researchers randomly assign subjects to treatment and control groups, natural experiments exploit existing variation in real-world conditions. The fundamental structure involves:

1. Treatment group: A population exposed to the condition or factor of interest
2. Control group: A comparable population not exposed to that condition
3. Outcome measurement: Observed differences in results between the groups
4. Causal inference: A conclusion that the condition caused the observed difference

For example, if Country A bans a certain pesticide while neighboring Country B does not, and researchers subsequently observe different crop yields, this constitutes a natural experiment. The key assumption—often unstated and vulnerable to attack on the LSAT—is that the two countries are similar in all other relevant respects.

The Logic of Causal Inference in Natural Experiments

Natural experiment arguments follow this reasoning pattern:

Premise 1: Group X was exposed to condition C
Premise 2: Group Y was not exposed to condition C
Premise 3: Groups X and Y differ in outcome O
Conclusion: Condition C caused the difference in outcome O

This reasoning depends critically on an implicit assumption: that Groups X and Y are comparable in all other factors that might affect outcome O. The LSAT exploits this vulnerability by presenting answer choices that introduce confounding variables—alternative factors that differ between the groups and could explain the observed outcome without invoking the proposed cause.

Types of Natural Experiment Comparisons

Confounding Variables and Alternative Explanations

A confounding variable is a factor that:

1. Differs between the comparison groups
2. Could plausibly affect the outcome of interest
3. Provides an alternative explanation for observed differences

On the LSAT, identifying or eliminating confounding variables is the most common way to strengthen or weaken natural experiment arguments. Consider this example:

Argument: "Hospital A implemented a new surgical protocol and subsequently had fewer post-operative infections than Hospital B, which did not implement the protocol. Therefore, the new protocol reduces infection rates."

Potential confounding variables:

• Hospital A may serve healthier patients overall
• Hospital B may be located in an area with more antibiotic-resistant bacteria
• Hospital A may have newer facilities with better sanitation
• The hospitals may differ in staff training or experience levels

Establishing Group Comparability

For a natural experiment to support a causal conclusion validly, the comparison groups must be relevantly similar—that is, similar in all factors that could affect the outcome except for the condition being studied. The LSAT tests this concept by:

Strengthen approaches:

• Showing that groups are similar in potentially relevant factors
• Eliminating alternative explanations
• Providing evidence that the proposed cause preceded the effect
• Demonstrating that the relationship holds across multiple comparisons

Weaken approaches:

• Revealing differences between groups in relevant factors
• Introducing alternative explanations for the observed outcome
• Showing that the effect preceded the proposed cause
• Demonstrating that similar outcomes occur without the proposed cause

The Temporal Dimension

Natural experiments involving before-and-after comparisons face a specific vulnerability: temporal confounding. When comparing the same group at different times, any factor that changed between the measurements could explain observed differences. The LSAT frequently exploits this by presenting arguments like:

"After the city installed speed cameras, traffic accidents decreased by 30%. Therefore, speed cameras reduce accidents."

This argument assumes no other relevant factors changed during the same period—an assumption vulnerable to counterexamples like improved road conditions, changes in traffic volume, new traffic laws, or seasonal variations in driving conditions.

Sample Size and Representativeness

While the LSAT rarely emphasizes statistical concepts explicitly, natural experiment arguments sometimes depend on whether observed differences are meaningful rather than coincidental. Arguments may be weakened by revealing that:

• The sample size is too small to draw reliable conclusions
• The comparison groups are not representative of broader populations
• The time period observed is too brief to establish patterns
• Unusual circumstances during the observation period make results atypical

Concept Relationships

Natural experiments connect to broader causal reasoning concepts in a hierarchical structure:

Causation and Explanation (parent concept) → includes → Natural Experiments (current topic) → relies on → Comparison Reasoning → requires → Group Similarity Assessment → vulnerable to → Confounding Variables → leads to → Alternative Explanations

The relationship to prerequisite knowledge flows as follows:

• Basic causation understanding provides the foundation for recognizing that natural experiments attempt to establish causal claims
• Argument structure knowledge enables identification of premises (group comparisons) and conclusions (causal claims)
• Evidence evaluation skills allow assessment of whether the comparison adequately supports the causal inference

Natural experiments also connect forward to related topics:

• Correlation vs. Causation: Natural experiments represent one method of moving from observed correlation to causal inference
• Necessary and Sufficient Conditions: Natural experiments may attempt to establish that a condition is necessary or sufficient for an outcome
• Statistical Reasoning: Some natural experiment arguments incorporate statistical claims about the strength or significance of observed differences

The internal relationships among concepts within natural experiments follow this logic chain:

1. Natural experiment structure defines the basic comparison framework
2. Group comparability determines whether the comparison is valid
3. Confounding variables represent threats to comparability
4. Alternative explanations emerge from confounding variables
5. Strengthening/weakening strategies target group comparability and confounding variables

High-Yield Facts

⭐ Natural experiments compare groups that differ in one condition to infer causal relationships without researcher manipulation of variables.

⭐ The critical assumption in natural experiment arguments is that comparison groups are similar in all relevant respects except the condition being studied.

⭐ Confounding variables—factors that differ between groups and could affect the outcome—provide alternative explanations that weaken natural experiment arguments.

⭐ Temporal comparisons (before/after) are vulnerable to the objection that other factors changed during the same time period.

⭐ Geographic comparisons are vulnerable to the objection that locations differ in other relevant characteristics beyond the condition being studied.

• Natural experiment arguments can be strengthened by evidence that comparison groups are similar in potentially relevant factors.

• Natural experiment arguments can be weakened by revealing differences between groups that could explain observed outcomes.

• The LSAT frequently tests natural experiments in Weaken, Strengthen, Flaw, and Assumption question types.

• Valid natural experiment reasoning requires that the proposed cause preceded the observed effect temporally.

• Multiple natural experiments showing the same relationship across different contexts strengthen causal claims.

• Small sample sizes or brief observation periods can undermine natural experiment conclusions.

• Natural experiments differ from controlled experiments in that researchers do not randomly assign subjects to groups.

Common Misconceptions

Misconception: Natural experiments are inherently weaker than all other forms of evidence and can never establish causation convincingly.

Correction: While natural experiments have vulnerabilities, they can provide strong causal evidence when comparison groups are demonstrably similar in relevant respects and confounding variables are adequately addressed. The LSAT tests the ability to evaluate natural experiments on their merits, not to dismiss them categorically.

Misconception: Any difference between comparison groups automatically invalidates a natural experiment argument.

Correction: Only differences in relevant factors—those that could plausibly affect the outcome—undermine natural experiment reasoning. Irrelevant differences (e.g., the groups prefer different sports teams) do not weaken causal inferences about unrelated outcomes (e.g., health effects of a dietary change).

Misconception: Strengthening a natural experiment argument requires proving that comparison groups are identical in every possible way.

Correction: Strengthening requires showing similarity in relevant respects—factors that could affect the outcome. Complete identity is impossible and unnecessary; relevant similarity is sufficient.

Misconception: If a natural experiment shows correlation, causation is automatically established.

Correction: Natural experiments provide evidence for causation only when comparison groups are adequately similar and confounding variables are ruled out. The comparison structure alone does not eliminate the correlation-causation gap; group comparability does.

Misconception: Temporal natural experiments (before/after comparisons) are always stronger than geographic comparisons because they involve the same population.

Correction: Temporal comparisons face unique vulnerabilities from other factors that changed over time. Neither comparison type is inherently superior; each must be evaluated based on whether relevant confounding variables are adequately addressed.

Misconception: Natural experiment arguments always explicitly state that they are making causal claims.

Correction: LSAT arguments often imply causal relationships through language like "resulted in," "led to," "because of," or "the reason for," without using the word "cause" explicitly. Recognizing implicit causal claims is essential for identifying natural experiment reasoning patterns.

Worked Examples

Example 1: Weakening a Natural Experiment Argument

Stimulus: "Residents of Town A, which has fluoridated water, have significantly fewer dental cavities than residents of Town B, which does not fluoridate its water. This demonstrates that water fluoridation reduces cavity rates."

Question: Which of the following, if true, most weakens the argument?

Analysis:

Step 1: Identify the argument structure

• Treatment group: Town A (fluoridated water)
• Control group: Town B (non-fluoridated water)
• Observed outcome: Town A has fewer cavities
• Conclusion: Fluoridation causes the difference

Step 2: Identify the critical assumption

The argument assumes Town A and Town B are similar in all other factors that could affect cavity rates.

Step 3: Evaluate answer choices (hypothetical)

(A) Town A residents consume significantly less sugar than Town B residents.

• This weakens the argument strongly. Sugar consumption affects cavity rates, so this difference provides an alternative explanation for the observed outcome. The lower cavity rate might result from lower sugar consumption rather than fluoridation.

(B) Some residents of Town A prefer bottled water to tap water.

• This does not significantly weaken the argument. If some Town A residents avoid fluoridated water, this would make the observed difference more impressive, not less.

(C) Town B has a larger population than Town A.

• Population size is not clearly relevant to cavity rates per capita. This does not introduce a confounding variable.

(D) Water fluoridation has been practiced in various countries for decades.

• This provides background information but does not address whether Towns A and B are comparable. It does not weaken the specific argument about these two towns.

(E) Dental cavities can be prevented through regular brushing and flossing.

• While true, this does not establish that Towns A and B differ in brushing/flossing habits. Without evidence of a difference between the towns, this does not weaken the argument.

Answer: (A)

Connection to learning objectives: This example demonstrates how to identify natural experiment structure, recognize the implicit assumption about group comparability, and apply the concept of confounding variables to weaken the argument.

Example 2: Strengthening a Natural Experiment Argument

Stimulus: "After City X implemented a ban on plastic bags, the amount of plastic waste in the city's waterways decreased by 40% over the following year. This shows that the plastic bag ban effectively reduced plastic pollution in waterways."

Question: Which of the following, if true, most strengthens the argument?

Analysis:

Step 1: Identify the argument structure

• This is a temporal natural experiment (before/after comparison)
• Treatment: Plastic bag ban implementation
• Observed outcome: 40% decrease in plastic waste in waterways
• Conclusion: The ban caused the decrease

Step 2: Identify vulnerabilities

Temporal comparisons are vulnerable to other factors that changed during the same period. The argument assumes no other relevant factors changed that could explain the decrease in plastic waste.

Step 3: Evaluate answer choices (hypothetical)

(A) No other significant changes in waste management or environmental policies occurred in City X during that year.

• This strengthens the argument strongly. By eliminating alternative explanations (other policy changes that could have reduced plastic waste), this makes the bag ban the most plausible explanation for the observed decrease.

(B) Plastic bags constitute a significant portion of plastic waste found in waterways.

• This makes the outcome more plausible but does not address whether other factors could explain the decrease. It does not eliminate alternative explanations.

(C) Several other cities have also implemented plastic bag bans.

• This provides context but does not address whether City X's specific decrease resulted from its ban rather than other factors.

(D) The amount of plastic waste in City X's waterways had been increasing for five years before the ban.

• This provides background but does not eliminate alternative explanations for the decrease following the ban.

(E) Residents of City X expressed support for environmental protection measures.

• Public opinion does not address whether factors other than the bag ban could explain the observed decrease in plastic waste.

Answer: (A)

Connection to learning objectives: This example illustrates how to strengthen natural experiment arguments by eliminating alternative explanations and addressing the vulnerability of temporal comparisons to confounding variables that change over time.

Exam Strategy

Recognizing Natural Experiment Questions

Watch for these trigger phrases that signal natural experiment reasoning:

• "After [change], [outcome] occurred..."
• "In [Location A] where [condition exists], [outcome] differs from [Location B] where [condition does not exist]..."
• "Compared to [Group X], [Group Y] shows..."
• "Since implementing [policy], [jurisdiction] has experienced..."
• "[Group] that [has characteristic] shows [outcome] unlike [group] that [lacks characteristic]..."

Systematic Approach to Natural Experiment Questions

Step 1: Identify the comparison structure (15 seconds)

• What are the two groups or time periods being compared?
• What condition differs between them?
• What outcome is being observed?

Step 2: Identify the causal claim (10 seconds)

• What does the argument conclude caused the observed difference?
• Is the causal claim explicit or implied?

Step 3: Identify the critical assumption (10 seconds)

• The groups/time periods are similar in all other relevant respects
• No confounding variables explain the outcome

Step 4: Predict answer characteristics (15 seconds)

• For Weaken: Look for relevant differences between groups or alternative explanations
• For Strengthen: Look for evidence of group similarity or elimination of alternatives
• For Flaw: Look for descriptions of assuming groups are comparable without justification
• For Assumption: Look for statements that groups are similar in specific relevant respects

Step 5: Evaluate answer choices (60 seconds)

• Eliminate answers addressing irrelevant differences
• Eliminate answers that strengthen when you need to weaken (or vice versa)
• Select the answer that most directly addresses group comparability or confounding variables

Process of Elimination Tips

Eliminate answers that:

• Introduce differences between groups that are clearly irrelevant to the outcome
• Provide background information without addressing group comparability
• Discuss the general validity of the type of reasoning without addressing the specific comparison
• Strengthen the argument when the question asks to weaken (or vice versa)
• Address only one group rather than the comparison between groups

Favor answers that:

• Introduce or eliminate relevant confounding variables
• Directly address whether groups are similar in factors that could affect the outcome
• Provide alternative explanations for observed differences (for Weaken questions)
• Rule out alternative explanations (for Strengthen questions)
• Identify specific relevant factors that the argument overlooks

Time Allocation

For natural experiment questions:

• Reading and initial analysis: 30-40 seconds
• Prediction: 15-20 seconds
• Answer choice evaluation: 50-70 seconds
• Total target time: 90-130 seconds

Natural experiment questions often require careful consideration of whether factors are relevant, so allocate slightly more time than for simpler logical reasoning questions. However, avoid getting trapped in overthinking—focus on clearly relevant factors rather than speculating about remote possibilities.

Memory Techniques

The COMPARE Acronym

Use COMPARE to remember key elements of natural experiment analysis:

• Comparison groups: Identify what is being compared
• Outcome: What difference is observed?
• Match: Are groups similar in relevant respects?
• Preceding: Did the proposed cause come before the effect?
• Alternatives: What other explanations exist?
• Relevance: Are differences between groups relevant to the outcome?
• Evidence: What would strengthen or weaken the comparison?

Visualization Strategy

Picture natural experiments as a balance scale:

• Each side represents one comparison group
• The scale tips based on the observed outcome
• Confounding variables are hidden weights on one side that could explain the tipping without invoking the proposed cause
• Strengthening the argument means removing hidden weights (showing groups are balanced)
• Weakening the argument means revealing hidden weights (showing groups differ in relevant ways)

The "Twin Test" Mental Model

Ask: "If these groups/time periods were twins, what differences between them could explain the outcome besides the proposed cause?"

This mental model helps identify confounding variables by focusing attention on relevant differences. Just as identical twins who differ in one outcome likely differ in some relevant experience or exposure, comparison groups that differ in outcomes likely differ in some relevant factor—the question is whether that factor is the one the argument proposes or an alternative.

Summary

Natural experiments represent a fundamental reasoning pattern on the LSAT, appearing in approximately 15-20% of Logical Reasoning questions. These arguments attempt to establish causal relationships by comparing groups or time periods that differ in one condition and observing differences in outcomes. The critical vulnerability in natural experiment reasoning is the assumption that comparison groups are similar in all relevant respects except the condition being studied. LSAT questions exploit this vulnerability by introducing confounding variables—alternative factors that differ between groups and could explain observed outcomes. Success with natural experiment questions requires systematically identifying the comparison structure, recognizing implicit assumptions about group comparability, and evaluating whether answer choices introduce or eliminate relevant confounding variables. Strengthening natural experiment arguments involves providing evidence that groups are similar in potentially relevant factors or eliminating alternative explanations, while weakening involves revealing relevant differences or introducing plausible alternative causes. Mastering this topic provides a significant advantage across multiple question types including Weaken, Strengthen, Flaw, Assumption, and Evaluate questions.

Key Takeaways

• Natural experiments compare groups or time periods that differ in one condition to infer causal relationships without researcher manipulation
• The critical assumption—that comparison groups are similar in all relevant respects—is the primary vulnerability in these arguments
• Confounding variables (relevant factors that differ between groups) provide alternative explanations that weaken natural experiment conclusions
• Temporal comparisons are vulnerable to other factors changing over time; geographic comparisons are vulnerable to location differences
• Strengthening requires evidence of group similarity or elimination of alternatives; weakening requires revealing relevant differences or introducing alternative explanations
• Only relevant differences—factors that could plausibly affect the outcome—impact the validity of natural experiment reasoning
• Systematic identification of comparison structure, causal claims, and critical assumptions enables efficient and accurate analysis of natural experiment questions

Related Topics

Correlation vs. Causation: Natural experiments represent one method of attempting to establish causation from observed correlations. Mastering natural experiments provides foundation for understanding when correlational evidence does and does not support causal inferences.

Alternative Explanations: The concept of confounding variables in natural experiments connects directly to the broader topic of alternative explanations in causal reasoning. Understanding how to generate and evaluate alternative explanations is essential for both natural experiment questions and other causal reasoning patterns.

Necessary and Sufficient Conditions: Natural experiments sometimes attempt to establish that a condition is necessary or sufficient for an outcome. Understanding these logical relationships enhances the ability to evaluate whether natural experiment evidence adequately supports such claims.

Statistical Reasoning: Some advanced natural experiment arguments incorporate statistical concepts like sample size, representativeness, and significance. Building on natural experiment mastery enables progression to more sophisticated statistical reasoning questions.

Controlled Experiments: Understanding natural experiments provides contrast with controlled experiments, where researchers manipulate variables and randomly assign subjects. This comparison illuminates the specific vulnerabilities of natural experiments and the reasoning patterns that address them.

Practice CTA

Now that you have mastered the core concepts of natural experiments, you are prepared to tackle LSAT practice questions with confidence. Apply the COMPARE framework systematically, focus on identifying confounding variables and group comparability issues, and practice distinguishing relevant from irrelevant differences between comparison groups. The practice questions and flashcards will reinforce these concepts and develop the pattern recognition skills essential for rapid, accurate performance on test day. Remember: natural experiments appear frequently on the LSAT, so time invested in mastering this topic yields substantial score improvements. Approach each practice question as an opportunity to refine your systematic analysis process, and you will see consistent improvement in both accuracy and speed.