scholarly journals Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk

2012 ◽  
Vol 20 (3) ◽  
pp. 351-368 ◽  
Author(s):  
Adam J. Berinsky ◽  
Gregory A. Huber ◽  
Gabriel S. Lenz
Keyword(s):  
Labor Markets ◽  
Field Experiments ◽  
Low Cost ◽  
Mechanical Turk ◽  
Subject Recruitment ◽  
Trade Offs ◽  
Online Labor Markets ◽  
Online Labor ◽  
Internal And External Validity ◽  
The U.S

We examine the trade-offs associated with using Amazon.com's Mechanical Turk (MTurk) interface for subject recruitment. We first describe MTurk and its promise as a vehicle for performing low-cost and easy-to-field experiments. We then assess the internal and external validity of experiments performed using MTurk, employing a framework that can be used to evaluate other subject pools. We first investigate the characteristics of samples drawn from the MTurk population. We show that respondents recruited in this manner are often more representative of the U.S. population than in-person convenience samples—the modal sample in published experimental political science—but less representative than subjects in Internet-based panels or national probability samples. Finally, we replicate important published experimental work using MTurk samples.

Values of Participants in Behavioral Accounting Research: A Comparison of the M-Turk Population to a Nationally Representative Sample

2018 ◽  
Vol 31 (1) ◽  
pp. 97-117 ◽  
Author(s):  
William D. Brink ◽  
Lorraine S. Lee ◽  
Jonathan S. Pyzoha
Keyword(s):  
Labor Markets ◽  
Data Availability ◽  
World Values Survey ◽  
Behavioral Accounting ◽  
The World ◽  
Online Labor Markets ◽  
Nationally Representative ◽  
Online Labor ◽  
Amazon's Mechanical Turk ◽  
The U.S

ABSTRACT The external validity of conclusions from behavioral accounting experiments is in part dependent upon the representativeness of the sample compared to the population of interest. Researchers are beginning to leverage the availability of workers via online labor markets, such as Amazon's Mechanical Turk (M-Turk), as proxies for the general population (e.g., investors, jurors, and taxpayers). Using over 200 values-based items from the World Values Survey (WVS), the purpose of the current study is to explore whether U.S. M-Turk workers' values are similar to those of the U.S. population. Results show for the majority of items collected, M-Turk participants' values are significantly different from the WVS participants (e.g., values related to trust, ethics, religious beliefs, and politics). We present select items and themes representing values shown to influence judgments in prior research and discuss how those values may affect inferences of behavioral accounting researchers. Data Availability: Data are available from the authors upon request.

More for Less: Adaptive Labeling Payment for Online Labor Markets

2016 ◽  
Author(s):  
Tomer Geva ◽  
Harel Lustiger ◽  
Maytal Saar-Tsechansky
Keyword(s):  
Labor Markets ◽  
Online Labor Markets ◽  
Online Labor

Scoundrels or Stars? Theory and Evidence on the Quality of Workers in Online Labor Markets

2016 ◽  
Vol 92 (1) ◽  
pp. 93-114 ◽  
Author(s):  
Anne M. Farrell ◽  
Jonathan H. Grenier ◽  
Justin Leiby
Keyword(s):  
Labor Markets ◽  
Data Availability ◽  
Accounting Research ◽  
Research Participants ◽  
Large Numbers ◽  
Research Designs ◽  
Online Labor Markets ◽  
Online Labor ◽  
Low Pay

ABSTRACT Online labor markets allow rapid recruitment of large numbers of workers for very low pay. Although online workers are often used as research participants, there is little evidence that they are motivated to make costly choices to forgo wealth or leisure that are often central to addressing accounting research questions. Thus, we investigate the validity of using online workers as a proxy for non-experts when accounting research designs use more demanding tasks than these workers typically complete. Three experiments examine the costly choices of online workers relative to student research participants. We find that online workers are at least as willing as students to make costly choices, even at significantly lower wages. We also find that online workers are sensitive to performance-based wages, which are just as effective in inducing high effort as high fixed wages. We discuss implications of our results for conducting accounting research with online workers. Data Availability: Contact the authors.

scholarly journals Monopsony in Online Labor Markets

2018 ◽  
Author(s):  
Arindrajit Dube ◽  
Jeff Jacobs ◽  
Suresh Naidu ◽  
Siddharth Suri
Keyword(s):  
Labor Markets ◽  
Online Labor Markets ◽  
Online Labor

Just DM Me (Politely): Direct Messaging, Politeness, and Hiring Outcomes in Online Labor Markets

2021 ◽  
Author(s):  
Yili Hong ◽  
Jing Peng ◽  
Gordon Burtch ◽  
Ni Huang
Keyword(s):  
Labor Markets ◽  
Communication Channel ◽  
Message Content ◽  
Job Fit ◽  
Beneficial Effects ◽  
Online Labor Markets ◽  
Lower Status ◽  
Prospective Employer ◽  
Online Labor

This study examines the role of text-based direct messaging systems in online labor markets, which provide a communication channel between workers and employers, adding a personal touch to the exchange of online labor. We propose the effect of workers’ use of the direct messaging system on employers’ hiring decisions and conceptualize the information role of direct messaging. To empirically evaluate the information role of the direct messaging system, we leverage data on the direct messaging activities between workers and employers across more than 470,000 job applications on a leading online labor market. We report evidence that direct messaging with a prospective employer increases a worker’s probability of being hired by 8.9%. However, the degree to which workers benefit from direct messaging is heterogeneous, and the effect amplifies for workers approaching employers from a position of disadvantage (lacking tenure or fit with the job) and attenuates as more workers attempt to message the same prospective employer. The effects also depend on message content. In particular, we find that the benefits of direct messaging for workers depend a great deal on the politeness of the workers, and this “politeness effect” depends on several contextual factors. The beneficial effects are amplified for lower-status workers (i.e., workers lacking tenure and job fit) and workers who share a common language with the employer. At the same time, the beneficial effects weaken in the presence of typographical errors. These findings provide important insights into when and what to message to achieve favorable hiring outcomes in online employment settings.

Sign in / Sign up

Export Citation Format

Share Document