Summary of the Previous Session
In the previous session, we discussed the foundational principles of experimental design, focusing on randomization, replication, and control as tools to address biases such as selection bias and omitted variable bias. We highlighted how these principles ensure internal validity by isolating causal effects. Additionally, we examined the challenges of scalability and external validity, including issues like representativeness, voltage effects, and heterogeneous treatment effects, which complicate the generalization of findings to broader contexts.
Ecological Validity
Definition: Ecological validity refers to the extent to which the results of an experiment can be generalized to real-world settings. It assesses how closely the experimental conditions, tasks, and participant behaviors mirror those found in the natural environment. High ecological validity ensures that the findings of an experiment are applicable and relevant outside the controlled experimental setup.
Example from the Real World
Consider a psychological study on memory conducted in a laboratory where participants are asked to memorize and recall lists of random words. While the study might yield accurate data on memory processes in controlled settings, its ecological validity is questionable because memorizing lists of random words is far removed from how people use memory in daily life (e.g., remembering grocery lists or recalling important conversations).
A more ecologically valid approach would involve observing participants’ memory in a realistic context, such as recalling items they had seen on a simulated online shopping website or during a guided tour.
Example in a Business Context
In a business experiment, suppose a company tests the effectiveness of a new email marketing strategy in a simulated environment where participants are asked to "imagine" they are potential customers and respond to promotional emails. While the results might suggest the strategy is effective, the study lacks ecological validity because participants are aware they are part of an experiment and may behave differently.
A more ecologically valid approach would involve testing the email marketing strategy in a real-world setting by sending actual promotional emails to a segment of the company’s customer base and tracking their click-through and purchase behaviors. This real-world test would provide insights into how the strategy performs under natural conditions, accounting for factors like competing emails, time constraints, and real customer motivations.
The Interplay between Internal, External, and Ecological Validity
| Validity Type | Focus | Goal | Example |
|---|---|---|---|
| Internal | Ensures the observed effects are due to the treatment or intervention, not confounding variables. | Ensure the study is methodologically sound and free of biases (e.g., selection bias, omitted variable bias). | In an RCT for a new drug, ensuring differences in health outcomes are due to the drug and not other factors. |
| External | Determines whether the study’s findings can generalize to other populations, settings, or times. | Apply findings beyond the specific conditions of the experiment. | A drug trial conducted with young adults might lack generalizability to elderly populations. |
| Ecological | Assesses how well the experimental conditions represent real-world scenarios. | Ensure the study mirrors the context in which the phenomenon naturally occurs. | Testing an advertising strategy in a controlled lab environment might not reflect real-world market behavior. |
The Spectrum of Experiment Types based on List (2011)
Laboratory Experiments
Explanation: Conducted in a highly controlled environment where participants are aware they are part of a study. Researchers manipulate variables to establish causal relationships while minimizing external influences.
Example: Testing the impact of time pressure on decision-making by asking business students to complete investment allocation tasks in a lab setting with varying time constraints.
Artefactual Field Experiments
Explanation: Similar to laboratory experiments but conducted outside the lab with participants from a relevant population. The setting is still controlled, but the participants are more representative.
Example: Investigating how professional managers allocate budgets under uncertainty by inviting them to participate in a structured simulation.
Framed Field Experiments
Explanation: Conducted in a natural environment with participants who are unaware of being studied, but they are informed about the purpose or context of the experiment. The environment reflects real-world conditions but with some framing by the researchers.
Example: Studying the effect of incentive structures on employee productivity by informing workers in a firm about a temporary bonus scheme and observing how their output changes.
Natural Field Experiments
Explanation: Participants are unaware they are part of an experiment, and the study occurs in their natural environment without framing or artificial interventions. This approach closely mirrors real-world conditions.
Example: Testing the effect of algorithmic changes on consumer behavior by adjusting the recommendation engine of an e-commerce platform for certain users without their knowledge and observing purchasing patterns.
| Experiment Type | Internal Validity | External Validity | Ecological Validity |
|---|---|---|---|
| Laboratory Experiments | High | Low | Low |
| Artefactual Field Experiments | Moderate | Moderate | Moderate |
| Framed Field Experiments | Moderate to High | High | Moderate to High |
| Natural Field Experiments | Low to Moderate | High | High |
Online Platforms in Experiments (Based on Fréchette et al., 2022)
Online platforms like Amazon Mechanical Turk (MTurk) and Prolific have revolutionized behavioral and experimental economics by providing accessible and cost-effective ways to conduct experiments. These platforms bring unique advantages and challenges:
Advantages
Increased Participant Diversity: Online platforms allow researchers to access a global pool of participants with diverse demographics. This enhances external validity by ensuring the findings are more representative of broader populations.
Example: A researcher can test the impact of framing effects on purchasing decisions with participants from different age groups, educational backgrounds, and geographic regions.
Lower Costs and Faster Recruitment: Online platforms significantly reduce the time and financial resources required for recruitment and data collection compared to traditional methods.
Example: Running an experiment to assess the effects of ad targeting can be done at a fraction of the cost on MTurk compared to organizing in-person focus groups.
Role of Online Platforms in Enhancing Validity
Internal Validity: Online platforms allow for randomization, a key method for reducing confounding factors, thereby improving internal validity.
Example: Randomly assigning participants to different marketing messages and tracking their engagement removes systematic differences between groups.
External Validity: Platforms like MTurk enable access to diverse and globally distributed samples, improving the representativeness of findings and their generalizability.
Example: A study testing app usability using participants from various countries and demographics enhances the relevance of results.
Ecological Validity: Online experiments conducted on platforms where participants use real-world interfaces (e.g., e-commerce websites or digital ads) can closely mimic natural behaviors.
Example: Observing ad click-through rates in a simulated online shopping environment mirrors real-world consumer decision-making.
Challenges of Online Platforms
Noise and Attention: Participants on online platforms may multitask or engage in minimal-effort behaviors, reducing the quality of data and potentially lowering internal validity.
Mitigation: Employ attention checks or incentivize high-quality participation.
Learning Effects: Frequent MTurk participants might recognize experimental setups, altering their responses and lowering ecological validity.
Mitigation: Use naive participants or design experiments to reduce predictability.
Sampling Bias: While online platforms are diverse, they may not fully represent specific target populations, limiting external validity in certain cases.
Mitigation: Stratified sampling or targeting specific demographics on the platform.
A Simple Experiment
Objective: To investigate how the label (Created by AI vs. Created by Human) affects the perceived quality of an image, where all images are actually generated by AI.
Key Details:
Images: All are AI-generated.
Labels: Randomly assign each image one of two labels:
Created by AI
Created by Human
Response Scale: Participants will rate the quality of the image on a Likert scale (e.g., 1 to 7, where 1 = Very Poor and 7 = Excellent).
Implementation Instructions in Qualtrics
This guide provides step-by-step instructions to set up the AI vs. Human-labeled image experiment using Qualtrics:
Create a New Project:
Log in to Qualtrics and create a new survey project.
Name the project (e.g., “AI vs. Human-Labeled Image Experiment”).
Set Up Random Assignment:
In the Survey Flow section, add a Randomizer block.
Configure the Randomizer to assign participants evenly to one of two conditions:
Condition 1: Images labeled as “AI-created.”
Condition 2: Images labeled as “Human-created.”
Under each condition, create blocks to display the images and questions specific to that label.
Embed the Images:
In each block (Condition 1 and Condition 2), upload or embed the AI-generated images. Use the image upload feature or paste links to hosted images.
Add captions for each image, such as “Created by AI” or “Created by a Human,” depending on the condition.
Control the Number of Images per Respondent:
Single Image per Respondent: Limit the number of questions to one per block and use the Randomizer in the Survey Flow to show only one block to each participant.
Multiple Images per Respondent: Add multiple questions (one for each image) within the assigned block.
Add Survey Questions:
After each image, add evaluation questions such as:
“How creative is this image?” (e.g., Likert scale: 1 = Not at all creative, 5 = Very creative).
“How likely do you think this image was created by a human?”
Use Skip Logic or Display Logic to ensure questions are displayed appropriately based on the random assignment.
Insert Attention Checks:
Include an attention check question within the survey, such as:
“What label was displayed for the image you just evaluated?” (e.g., options: “AI-created,” “Human-created”).
Use Qualtrics’ Validation features to ensure responses meet quality standards.
Preview and Test the Survey:
Use the Preview feature to test the survey flow.
Ensure randomization, image display, and question routing work as intended.
Set Up Survey Distribution:
Generate a survey link from the Distributions tab in Qualtrics.
Set survey options to limit one response per participant.
Recruit Participants on MTurk:
(Only if you have IRB approval), Post the Qualtrics survey link on Amazon Mechanical Turk (MTurk).
Specify participant criteria.
Collect and Export Data:
Monitor responses in the Data & Analysis tab.
Export the data for statistical analysis.
Advantages and Disadvantages of Including Multiple Images per Respondent vs. One Image per Respondent
Single Image per Respondent
Advantages:
Minimized Learning Effects: By showing only one image, respondents are less likely to detect patterns or anticipate the experiment’s purpose, preserving the ecological validity of their responses.
Example: If respondents see multiple images labeled as "AI-created," they might suspect that the study is testing perceptions of AI-generated content, potentially biasing their evaluation.
Focused Attention: Respondents can dedicate their full attention to evaluating one image, reducing noise and enhancing the internal validity of their responses.
Simplified Analysis: Each respondent provides a single independent data point, simplifying statistical analyses and avoiding within-subject correlations that might complicate interpretation.
Disadvantages:
Larger Sample Size Needed: Since each respondent contributes only one data point, achieving statistical power requires a larger number of participants, increasing the cost of the study.
Limited Context for Comparison: Respondents may evaluate the single image without a reference point, potentially increasing variability in responses.
Example: One respondent’s "excellent" rating may differ significantly from another’s due to a lack of comparative context.
Multiple Images per Respondent
Advantages:
Cost Efficiency: Each respondent provides multiple data points, reducing the number of participants needed to achieve statistical power. This approach can lower costs, especially on platforms like MTurk.
Comparative Context: Respondents can compare images, which may improve the reliability of their ratings by reducing subjective variability.
Example: A respondent might rate the second image more accurately after evaluating the first, as it provides a reference point for their judgment.
Disadvantages:
Increased Risk of Learning Effects: Respondents might notice experimental patterns, such as consistent labeling of "AI-created," which could bias their subsequent responses and lower ecological validity.
Fatigue Effects: Evaluating multiple images can lead to respondent fatigue, increasing noise in later responses and potentially reducing internal validity.
Within-Subject Correlation: Responses from the same participant are not independent, complicating statistical analysis and requiring more advanced modeling techniques (e.g., mixed-effects models).
Required readings
Fréchette, G. R., Sarnoff, K., and Yariv, L. (2022). Experimental Economics: Past and Future. Annual Review of Economics, 14(1), 777–794. https://doi.org/10.1146/annurev-economics-081621-124424
List, J. A. (2011). Why Economists Should Conduct Field Experiments and 14 Tips for Pulling One Off. Journal of Economic Perspectives, 25(3), 3–16. https://doi.org/10.1257/jep.25.3.3
Yaraghi, N., West, D. M., Gopal, R. D., and Ramesh, R. (2020). (How) Did Attack Advertisements Increase Affordable Care Act Enrollments?. PLOS ONE, 15(2), e0228185. https://doi.org/10.1371/journal.pone.0228185