Can Variation in Subgroups' Average Treatment Effects Explain Treatment Effect Heterogeneity? Evidence from a Social Experiment

The extent to which survey experiments conducted with nonrepresentative convenience samples are generalizable to target populations depends critically on the degree of treatment effect heterogeneity. Recent inquiries have found a strong correspondence between sample average treatment effects estimated in nationally representative experiments and in replication studies conducted with convenience samples. We consider here two possible explanations: low levels of effect heterogeneity or high levels of effect heterogeneity that are unrelated to selection into the convenience sample. We analyze subgroup conditional average treatment effects using 27 original–replication study pairs (encompassing 101,745 individual survey responses) to assess the extent to which subgroup effect estimates generalize. While there are exceptions, the overwhelming pattern that emerges is one of treatment effect homogeneity, providing a partial explanation for strong correspondence across both unconditional and conditional average treatment effect estimates.

Download Full-text

Covariance Regression Models for Studying Treatment Effect Heterogeneity Across One or More Outcomes: Understanding How Treatments Shape Inequality

Sociological Methods & Research ◽

10.1177/0049124119882449 ◽

2019 ◽

pp. 004912411988244

Author(s):

Deirdre Bloome ◽

Daniel Schrage

Keyword(s):

Treatment Effect ◽

Regression Models ◽

Treatment Effects ◽

Well Being ◽

Heterogeneous Treatment Effects ◽

Multiple Dimensions ◽

Treatment Effect Heterogeneity ◽

Distributional Consequences ◽

Average Treatment Effects ◽

Effect Heterogeneity

Causal analyses typically focus on average treatment effects. Yet for substantive research on topics like inequality, interest extends to treatments’ distributional consequences. When individuals differ in their responses to treatment, three types of inequality may result. Treatment may shape inequalities between subgroups defined by pretreatment covariates, it may induce more inequality in one subgroup than another, or it may polarize people across multiple dimensions of well-being. We introduce a model, called a covariance regression, that captures all three types of inequality via the means, variances, and correlations between multiple outcomes. The model can test for heterogeneous treatment effects, quantify the heterogeneity, and explain its structure using covariates. Finding that a treatment creates inequalities could drive theoretical refinement and inform policy decisions (targeting groups where payoffs will be most predictable). We illustrate the utility of covariance regressions by analyzing the effects of sharing information about income inequality on redistributive preferences.

Download Full-text

The Effect of Extension Services and Credit on Agricultural Production in Bolivia, Peru, and Colombia

10.18235/0003404 ◽

2021 ◽

Author(s):

Mateus C. R. Neves ◽

Felipe De Figueiredo Silva ◽

Carlos Otávio Freitas

Keyword(s):

Travel Time ◽

Treatment Effect ◽

Agricultural Production ◽

Treatment Effects ◽

Geographic Information ◽

Average Treatment Effect ◽

Extension Services ◽

Average Treatment ◽

Andean Countries ◽

Average Treatment Effects

In this paper we estimate the average treatment effect from access to extension services and credit on agricultural production in selected Andean countries (Bolivia, Peru, and Colombia). More specifically, we want to identify the effect of accessibility, here represented as travel time to the nearest area with 1,500 or more inhabitants per square kilometer or at least 50,000 inhabitants, on the likelihood of accessing extension and credit. To estimate the treatment effect and identify the effect of accessibility on these variables, we use data from the Colombian and Bolivian Agricultural Censuses of 2013 and 2014, respectively; a national agricultural survey from 2017 for Peru; and geographic information on travel time. We find that the average treatment effect for extension is higher compared to that of credit for farms in Bolivia and Peru, and lower for Colombia. The average treatment effects of extension and credit for Peruvian farms are $2,387.45 and $3,583.42 respectively. The average treatment effect for extension and credit are $941.92 and $668.69, respectively, while in Colombia are $1,365.98 and $1,192.51, respectively. We also find that accessibility and the likelihood of accessing these services are nonlinearly related. Results indicate that higher likelihood is associated with lower travel time, especially in the analysis of credit.

Download Full-text

Estimating Treatment Effects in the Presence of Noncompliance and Nonresponse: The Generalized Endogenous Treatment Model

Political Analysis ◽

10.1093/pan/mpr005 ◽

2011 ◽

Vol 19 (2) ◽

pp. 205-226 ◽

Cited By ~ 8

Author(s):

Kevin M. Esterling ◽

Michael A. Neblo ◽

David M. J. Lazer

Keyword(s):

Treatment Effect ◽

Instrumental Variable ◽

Treatment Effects ◽

Mean Squared Error ◽

Principal Stratification ◽

Squared Error ◽

Treatment Effect Heterogeneity ◽

Causal Treatment ◽

Effect Heterogeneity ◽

And Control

If ignored, noncompliance with a treatment or nonresponse on outcome measures can bias estimates of treatment effects in a randomized experiment. To identify and estimate causal treatment effects in the case where compliance and response depend on unobservables, we propose the parametric generalized endogenous treatment (GET) model. GET incorporates behavioral responses within an experiment to measure each subject's latent compliance type and identifies causal effects via principal stratification. Using simulation methods and an application to field experimental data, we show GET has a dramatically lower mean squared error for treatment effect estimates than existing approaches to principal stratification that impute, rather than measure, compliance type. In addition, we show that GET allows one to relax and test the instrumental variable exclusion restriction assumption, to test for the presence of treatment effect heterogeneity across a range of compliance types, and to test for treatment ignorability when treatment and control samples are balanced on observable covariates.

Download Full-text

Extending the I-squared statistic to describe treatment effect heterogeneity in cluster, multi-centre randomized trials and individual patient data meta-analysis

Statistical Methods in Medical Research ◽

10.1177/0962280220948550 ◽

2020 ◽

pp. 096228022094855

Author(s):

Karla Hemming ◽

James P Hughes ◽

Joanne E McKenzie ◽

Andrew B Forbes

Keyword(s):

Randomized Trial ◽

Treatment Effect ◽

Treatment Effects ◽

Randomized Trials ◽

Individual Patient Data ◽

Meta Analysis ◽

Cluster Randomized Trial ◽

Patient Data ◽

Treatment Effect Heterogeneity ◽

Effect Heterogeneity

Treatment effect heterogeneity is commonly investigated in meta-analyses to identify if treatment effects vary across studies. When conducting an aggregate level data meta-analysis it is common to describe the magnitude of any treatment effect heterogeneity using the I-squared statistic, which is an intuitive and easily understood concept. The effect of a treatment might also vary across clusters in a cluster randomized trial, or across centres in multi-centre randomized trial, and it can be of interest to explore this at the analysis stage. In cross-over trials and other randomized designs, in which clusters or centres are exposed to both treatment and control conditions, this treatment effect heterogeneity can be identified. Here we derive and evaluate a comparable I-squared measure to describe the magnitude of heterogeneity in treatment effects across clusters or centres in randomized trials. We further show how this methodology can be used to estimate treatment effect heterogeneity in an individual patient data meta-analysis.

Download Full-text

On the treatment effect heterogeneity of antidepressants in major depression. A Bayesian meta-analysis

10.1101/2020.02.20.19015677 ◽

2020 ◽

Author(s):

Constantin Volkmann ◽

Alexander Volkmann ◽

Christian A. Müller

Keyword(s):

Major Depression ◽

Random Effects ◽

Treatment Effect ◽

Clinical Relevance ◽

Meta Analysis ◽

Placebo Response ◽

Average Treatment Effect ◽

Average Treatment ◽

Treatment Effect Heterogeneity ◽

Effect Heterogeneity

AbstractBackgroundThe average treatment effect of antidepressants in major depression was found to be about 2 points on the 17-item Hamilton Depression Rating Scale, which lies below clinical relevance. Here, we searched for evidence of a relevant treatment effect heterogeneity that could justify the usage of antidepressants despite their low average treatment effect.MethodsBayesian meta-analysis of 169 randomized, controlled trials including 58,687 patients. We considered the effect sizes log variability ratio (lnVR) and log coefficient of variation ratio (lnCVR) to analyze the difference in variability of active and placebo response. We used Bayesian random-effects meta-analyses (REMA) for lnVR and lnCVR and fitted a random-effects meta-regression (REMR) model to estimate the treatment effect variability between antidepressants and placebo.ResultsThe variability ratio was found to be very close to 1 in the best fitting models (REMR: 95% HPD [0.98, 1.02], REMA: 95% HPD [1.00, 1.02]). The between-study variance τ2 under the REMA was found to be low (95% HPD [0.00, 0.00]). Simulations showed that a large treatment effect heterogeneity is only compatible with the data if a strong correlation between placebo response and individual treatment effect is assumed.ConclusionsThe published data from RCTs on antidepressants for the treatment of major depression is compatible with a near-constant treatment effect. Although it is impossible to rule out a substantial treatment effect heterogeneity, its existence seems rather unlikely. Since the average treatment effect of antidepressants falls short of clinical relevance, the current prescribing practice should be re-evaluated.

Download Full-text

A Unified Survey of Treatment Effect Heterogeneity Modelling and Uplift Modelling

ACM Computing Surveys ◽

10.1145/3466818 ◽

2022 ◽

Vol 54 (8) ◽

pp. 1-36

Author(s):

Weijia Zhang ◽

Jiuyong Li ◽

Lin Liu

Keyword(s):

Treatment Effect ◽

Treatment Effects ◽

Causal Effect ◽

Personalised Medicine ◽

Online Marketing ◽

Heterogeneous Treatment Effects ◽

Source Codes ◽

Outcome Framework ◽

Treatment Effect Heterogeneity ◽

Effect Heterogeneity

A central question in many fields of scientific research is to determine how an outcome is affected by an action, i.e., to estimate the causal effect or treatment effect of an action. In recent years, in areas such as personalised healthcare, sociology, and online marketing, a need has emerged to estimate heterogeneous treatment effects with respect to individuals of different characteristics. To meet this need, two major approaches have been taken: treatment effect heterogeneity modelling and uplifting modelling. Researchers and practitioners in different communities have developed algorithms based on these approaches to estimate the heterogeneous treatment effects. In this article, we present a unified view of these two seemingly disconnected yet closely related approaches under the potential outcome framework. We provide a structured survey of existing methods following either of the two approaches, emphasising their inherent connections and using unified notation to facilitate comparisons. We also review the main applications of the surveyed methods in personalised marketing, personalised medicine, and sociology. Finally, we summarise and discuss the available software packages and source codes in terms of their coverage of different methods and applicability to different datasets, and we provide general guidelines for method selection.

Download Full-text