IQMR 2025

Integrating Qualitative and Experimental Methods (Modules 17, 21, 25)

Monday, June 23; Tuesday, June 24, Wednesday, June 25

Chris Carter (University of Virginia), Guadalupe Tuñón (Princeton University), Tesalia Rizzo Reyes (UC Merced), Charles Crabtree (Dartmouth College)

In this module sequence, we introduce natural and randomized experiments and discuss their strengths and limitations through a survey of recent examples from political science and economics. We introduce a common framework for understanding and assessing natural and randomized experiments based on the credibility of causal and statistical assumptions. We discuss tools for developing and accessing experimental designs, such as instrumental variable analysis, sampling principles, power analysis, data collection do’s and don’ts as well as a variety of robustness tests. We then discuss how to bolster the credibility of natural and randomized experiments in the design stage. We will focus on the role of “ex-ante” approaches to improve the quality and transparency of research designs, such as the use of pre-analysis plans. The module incorporates applied research and practical advice, especially on how to conduct fieldwork, collect data, and analyze the logistics and ethics surrounding experiments. We end the module by evaluating the promise and obstacles to the use of multi-method research in the analysis of natural and randomized experiments. We discuss how qualitative methods can help address some of the criticisms of experiments, as well as how experiments can bolster the inferences drawn from qualitative evidence.

Participants may enter the module sequence after it has begun.

There is online content associated with this module that students should view (and complete the assignments for) prior to June 23

Integrating Qualitative and Experimental Methods (I) (M17, June 23)

8:45am - 10:15am – Qualitative approaches to natural experiments

Carter and Tuñón

We emphasize the crucial role of qualitative methods in analyzing natural experiments. Through illustrative examples, we demonstrate how qualitative evidence enhances the credibility of causal assumptions and aids in interpreting quantitative results. We also explore how qualitative methods address common criticisms of natural experiments and how natural experiments, in turn, strengthen inferences drawn from qualitative evidence.

Required Reading:

Suggested Reading:

1:30pm - 3:00pm – Multi-method approaches to Instrumental Variables (IV) analysis

Carter and Tuñón

In this session, we discuss the role of causal and statistical assumptions in the analysis of natural experiments. We then delve deeper into one of the most widely used types of natural experiments—instrumental variables (IV)— to illustrate the plausibility of these assumptions in a variety of applications. We discuss qualitative methods that can be used to discover, strengthen, and assess the validity of instrumental variables.

Required readings:

Suggested Readings:

3:30pm - 5:00pm – Multi-method approaches to Regression-Discontinuity (RD) Designs

Carter and Tuñón

In this session we focus on a second broad category of natural experiments, regression-discontinuity (RD) designs. We provide an overview of core assumptions, discussing the role of qualitative and quantitative methods in understanding the process that determines treatment assignment. We discuss recent applications of RD designs and how they might be bolstered by additional qualitative evidence.

Required readings:

Suggested Readings:

Integrating Qualitative and Experimental Methods (II) (M21 June 24)

8:45am - 10:15am – Designing natural experiments: Applications and Placebo Tests

Carter and Tuñón

We offer a review of the core approaches to natural experiments, working through a guided review of a problem set (to be completed before class). We then discuss how students might identify and evaluate natural experiments in their own research, as well as when evaluating research conducted by others.

In lieu of required readings, students should complete the assigned problem set for this session.

Suggested readings:

1:30pm - 3:00pm – Nuts and Bolts of Implementing Randomized Experiments: Practical Considerations and Mixed-Methods Approaches

Rizzo

This session offers a review of the fundamental components of experimental design, including selection bias, randomization, and sampling strategies. We will also introduce different causal estimands, estimation strategies, and common threats to causal inference. This section assumes prior exposure to basic causal inference concepts and is intended as a refresher.

In the second half, we explore the comparative advantages of field experiments over other empirical strategies. Particular attention will be given to ongoing debates over external validity and the robustness of experimental findings. We conclude by discussing how qualitative methods can enhance experimental design—particularly in the development of treatments and the identification of mechanisms (a topic we will return to more in depth in the next session).

Required Readings – External Validity

Required Readings – Integrating Qualitative Methods

Suggested Reading:

3:30pm - 5:00pm – Nuts and Bolts of Implementing Randomized Experiments: Practical Considerations and Mixed-Methods Approaches

Rizzo

This session focuses on the practicalities of implementing field experiments. In the first half, we examine strategies for data collection, including survey design, enumerator training, quality control, and ensuring treatment compliance. Emphasis is placed on using qualitative methods—such as interviews, focus groups, and participant observation—to limit implementation issues and develop locally grounded treatments and minimize measurement error.

In the second half, we discuss how to collaborate effectively with implementation partners (e.g., NGOs, survey firms, government partners, etc.). We will especially focus on how qualitative insights from fieldwork can help anticipate logistical challenges, strengthen enumerator training, and improve collaboration with relevant actors

Required Readings:

Integrating Qualitative and Experimental Methods (III) (M25 June 25)

8:45am - 10:15am – Discussion of Recent Field Experiments: Ethics, Research Transparency, and the Role of Randomized Experiments in Social Science

Rizzo

In this session, we will examine key ethical considerations in conducting field experiments in political science through the lens of recent studies. Topics will include the ethics of randomizing (and withholding) treatments, potential interference, power dynamics in the field, and ensuring the safety of respondents and field staff. We will also explore recent debates on positionality in research, particularly when working with vulnerable populations, and how researcher identity and power relations shape fieldwork dynamics.

Building on these ethical discussions, we will consider best practices in research transparency, including pre-registration, ethical reporting, and replication, to ensure rigor and accountability in experimental research.

The case studies will not be discussed in detail. Read them enough to be familiar with these examples’ general idea, design, and implementation.

Required Readings - Case Studies for Class Discussion (Skim them, focus on the research designs, treatments, and implementation).

Required Readings – Ethics in Field Experiments and Fieldwork

Suggested Reading:

Suggested Additional Cases:

1:30pm - 3:00pm – Advanced Topics in Experiments: Ecological validity

Crabtree

In this session, we examine ecological validity in survey experiments, focusing on how well experimental findings generalize to real-world settings. We discuss how qualitative methods can help assess external validity, uncover contextual factors that shape responses, and improve study design. We review recent experimental work, explore ecological limitations, and outline strategies and concrete recommendations for strengthening ecological validity. To ground understanding of this issue, we’ll work through a toy research design that examines race-based discrimination.

Required readings:

Suggested readings:

3:30pm - 5:00pm – Advanced Topics in Experiments: Information Equivalence

Crabtree

In this session, we examine information equivalence in survey experiments, focusing on how experimental treatments convey meaning to respondents. We discuss challenges in ensuring treatments are understood as intended and explore how qualitative methods can identify variation in interpretation. We review recent experimental work, explore issues related to information equivalence, and outline strategies and concrete recommendations for designing treatments with high levels of construct validity. To ground understanding of this issue, we’ll work through a toy research design that examines gender-based discrimination.

Required readings:

Suggested readings: