scholarly journals Empirical methodology for crowdsourcing ground truth

Semantic Web ◽  
2020 ◽  
pp. 1-19
Author(s):  
Anca Dumitrache ◽  
Oana Inel ◽  
Benjamin Timmermans ◽  
Carlos Ortiz ◽  
Robert-Jan Sips ◽  
...  
Keyword(s):  
Majority Vote ◽  
Relation Extraction ◽  
Ground Truth ◽  
Extraction Methods ◽  
Event Extraction ◽  
Ground Truth Data ◽  
Usual Practice ◽  
Event Identification ◽  
News Event

The process of gathering ground truth data through human annotation is a major bottleneck in the use of information extraction methods for populating the Semantic Web. Crowdsourcing-based approaches are gaining popularity in the attempt to solve the issues related to volume of data and lack of annotators. Typically these practices use inter-annotator agreement as a measure of quality. However, in many domains, such as event detection, there is ambiguity in the data, as well as a multitude of perspectives of the information examples. We present an empirically derived methodology for efficiently gathering of ground truth data in a diverse set of use cases covering a variety of domains and annotation tasks. Central to our approach is the use of CrowdTruth metrics that capture inter-annotator disagreement. We show that measuring disagreement is essential for acquiring a high quality ground truth. We achieve this by comparing the quality of the data aggregated with CrowdTruth metrics with majority vote, over a set of diverse crowdsourcing tasks: Medical Relation Extraction, Twitter Event Identification, News Event Extraction and Sound Interpretation. We also show that an increased number of crowd workers leads to growth and stabilization in the quality of annotations, going against the usual practice of employing a small number of annotators.

Tablet-Based Traffic Counting Application Designed to Minimize Human Error

2013 ◽  
Vol 2339 (1) ◽  
pp. 39-46 ◽  
Author(s):  
Christopher Toth ◽  
Wonho Suh ◽  
Vetri Elango ◽  
Ramik Sadana ◽  
Angshuman Guin ◽  
...  
Keyword(s):  
Count Data ◽  
Transportation Planning ◽  
Human Error ◽  
Ground Truth ◽  
Model Verification ◽  
Freeway Traffic ◽  
Ground Truth Data ◽  
Traffic Count ◽  
System Verification

Basic traffic counts are among the key elements in transportation planning and forecasting. As emerging data collection technologies proliferate, the availability of traffic count data will expand by orders of magnitude. However, availability of data does not always guarantee data accuracy, and it is essential that observed data are compared with ground truth data. Little research or guidance is available that ensures the quality of ground truth data with which the count results of automated technologies can be compared. To address the issue of ground truth data based on manual counts, a manual traffic counting application was developed for an Android tablet. Unlike other manual count applications, this application allows data collectors to replay and toggle through the video in supervisory mode to review and correct counts made in the first pass. For system verification, the review function of the application was used to count and recount freeway traffic in videos from the Atlanta, Georgia, metropolitan area. Initial counts and reviewed counts were compared, and improvements in count accuracy were assessed. The results indicated the benefit of the review process and suggested that this application could minimize human error and provide more accurate ground truth traffic count data for use in transportation planning applications and for model verification.

A Total Error Approach for Validating Event Data

2021 ◽  
pp. 000276422110216
Author(s):  
Scott Althaus ◽  
Buddy Peyton ◽  
Dan Shalmon
Keyword(s):  
Gold Standard ◽  
Total Error ◽  
Ground Truth ◽  
Event Extraction ◽  
Event Data ◽  
Actual Distribution ◽  
Ground Truth Data ◽  
Total Event ◽  
Conflict Research

Understanding how useful any particular set of event data might be for conflict research requires appropriate methods for assessing validity when ground truth data about the population of interest do not exist. We argue that a total error framework can provide better leverage on these critical questions than previous methods have been able to deliver. We first define a total event data error approach for identifying 19 types of error that can affect the validity of event data. We then address the challenge of applying a total error framework when authoritative ground truth about the actual distribution of relevant events is lacking. We argue that carefully constructed gold standard datasets can effectively benchmark validity problems even in the absence of ground truth data about event populations. To illustrate the limitations of conventional strategies for validating event data, we present a case study of Boko Haram activity in Nigeria over a 3-month offensive in 2015 that compares events generated by six prominent event extraction pipelines—ACLED, SCAD, ICEWS, GDELT, PETRARCH, and the Cline Center’s SPEED project. We conclude that conventional ways of assessing validity in event data using only published datasets offer little insight into potential sources of error or bias. Finally, we illustrate the benefits of validating event data using a total error approach by showing how the gold standard approach used to validate SPEED data offers a clear and robust method for detecting and evaluating the severity of temporal errors in event data.

Collective annotation patterns in learning from crowds

2020 ◽  
Vol 24 ◽  
pp. 63-86
Author(s):  
Francisco Mena ◽  
Ricardo Ñanculef ◽  
Carlos Valle
Keyword(s):  
Machine Learning ◽  
Large Scale ◽  
Ground Truth ◽  
Experimental Results ◽  
Ground Truth Data ◽  
Satisfactory Performance ◽  
Machine Learning Applications ◽  
Data Points ◽  
Confusion Matrices

The lack of annotated data is one of the major barriers facing machine learning applications today. Learning from crowds, i.e. collecting ground-truth data from multiple inexpensive annotators, has become a common method to cope with this issue. It has been recently shown that modeling the varying quality of the annotations obtained in this way, is fundamental to obtain satisfactory performance in tasks where inexpert annotators may represent the majority but not the most trusted group. Unfortunately, existing techniques represent annotation patterns for each annotator individually, making the models difficult to estimate in large-scale scenarios. In this paper, we present two models to address these problems. Both methods are based on the hypothesis that it is possible to learn collective annotation patterns by introducing confusion matrices that involve groups of data point annotations or annotators. The first approach clusters data points with a common annotation pattern, regardless the annotators from which the labels have been obtained. Implicitly, this method attributes annotation mistakes to the complexity of the data itself and not to the variable behavior of the annotators. The second approach explicitly maps annotators to latent groups that are collectively parametrized to learn a common annotation pattern. Our experimental results show that, compared with other methods for learning from crowds, both methods have advantages in scenarios with a large number of annotators and a small number of annotations per annotator.

scholarly journals DeepSatData: Building large scale datasets of satellite images for training machine learning models

2021 ◽  
Author(s):  
Michael Tarasiou
Keyword(s):  
Machine Learning ◽  
Large Scale ◽  
Ground Truth ◽  
Semantic Segmentation ◽  
Point Of View ◽  
Learning Models ◽  
Ground Truth Data ◽  
Machine Learning Models ◽  
Sentinel 2

This paper presents DeepSatData a pipeline for automatically generating satellite imagery datasets for training machine learning models. We also discuss design considerations with emphasis on dense classification tasks, e.g. semantic segmentation. The implementation presented makes use of freely available Sentinel-2 data which allows the generation of large scale datasets required for training deep neural networks (DNN). We discuss issues faced from the point of view of DNN training and evaluation such as checking the quality of ground truth data and comment on the scalability of the approach.

scholarly journals Crowdsourcing Image Analysis for Plant Phenomics to Generate Ground Truth Data for Machine Learning

2018 ◽  
Author(s):  
Naihui Zhou ◽  
Zachary D Siegel ◽  
Scott Zarecor ◽  
Nigel Lee ◽  
Darwin A Campbell ◽  
...  
Keyword(s):  
Machine Learning ◽  
Image Analysis ◽  
Best Practices ◽  
Ground Truth ◽  
Training Data ◽  
Quality Data ◽  
High Quality ◽  
Ground Truth Data ◽  
Plant Phenomics

AbstractThe accuracy of machine learning tasks critically depends on high quality ground truth data. Therefore, in many cases, producing good ground truth data typically involves trained professionals; however, this can be costly in time, effort, and money. Here we explore the use of crowdsourcing to generate a large number of training data of good quality. We explore an image analysis task involving the segmentation of corn tassels from images taken in a field setting. We investigate the accuracy, speed and other quality metrics when this task is performed by students for academic credit, Amazon MTurk workers, and Master Amazon MTurk workers. We conclude that the Amazon MTurk and Master Mturk workers perform significantly better than the for-credit students, but with no significant difference between the two MTurk worker types. Furthermore, the quality of the segmentation produced by Amazon MTurk workers rivals that of an expert worker. We provide best practices to assess the quality of ground truth data, and to compare data quality produced by different sources. We conclude that properly managed crowdsourcing can be used to establish large volumes of viable ground truth data at a low cost and high quality, especially in the context of high throughput plant phenotyping. We also provide several metrics for assessing the quality of the generated datasets.Author SummaryFood security is a growing global concern. Farmers, plant breeders, and geneticists are hastening to address the challenges presented to agriculture by climate change, dwindling arable land, and population growth. Scientists in the field of plant phenomics are using satellite and drone images to understand how crops respond to a changing environment and to combine genetics and environmental measures to maximize crop growth efficiency. However, the terabytes of image data require new computational methods to extract useful information. Machine learning algorithms are effective in recognizing select parts of images, butthey require high quality data curated by people to train them, a process that can be laborious and costly. We examined how well crowdsourcing works in providing training data for plant phenomics, specifically, segmenting a corn tassel – the male flower of the corn plant – from the often-cluttered images of a cornfield. We provided images to students, and to Amazon MTurkers, the latter being an on-demand workforce brokered by Amazon.com and paid on a task-by-task basis. We report on best practices in crowdsourcing image labeling for phenomics, and compare the different groups on measures such as fatigue and accuracy over time. We find that crowdsourcing is a good way of generating quality labeled data, rivaling that of experts.

scholarly journals RIGOROUS STRIP ADJUSTMENT OF AIRBORNE LASERSCANNING DATA BASED ON THE ICP ALGORITHM

2015 ◽  
Vol II-3/W5 ◽  
pp. 73-80 ◽  
Author(s):  
P. Glira ◽  
N. Pfeifer ◽  
C. Briese ◽  
C. Ressl
Keyword(s):  
Measurement Errors ◽  
Laser Scanning ◽  
Ground Truth ◽  
Point Clouds ◽  
Redundant Information ◽  
Ground Truth Data ◽  
Icp Algorithm ◽  
Airborne Laser ◽  
Fully Automatic

Airborne Laser Scanning (ALS) is an efficient method for the acquisition of dense and accurate point clouds over extended areas. To ensure a gapless coverage of the area, point clouds are collected strip wise with a considerable overlap. The redundant information contained in these overlap areas can be used, together with ground-truth data, to re-calibrate the ALS system and to compensate for systematic measurement errors. This process, usually denoted as <i>strip adjustment</i>, leads to an improved georeferencing of the ALS strips, or in other words, to a higher data quality of the acquired point clouds. We present a fully automatic strip adjustment method that (a) uses the original scanner and trajectory measurements, (b) performs an on-the-job calibration of the entire ALS multisensor system, and (c) corrects the trajectory errors individually for each strip. Like in the Iterative Closest Point (ICP) algorithm, correspondences are established iteratively and directly between points of overlapping ALS strips (avoiding a time-consuming segmentation and/or interpolation of the point clouds). The suitability of the method for large amounts of data is demonstrated on the basis of an ALS block consisting of 103 strips.

scholarly journals DeepSatData: Building large scale datasets of satellite images for training machine learning models

2021 ◽  
Author(s):  
Michael Tarasiou
Keyword(s):  
Machine Learning ◽  
Large Scale ◽  
Ground Truth ◽  
Semantic Segmentation ◽  
Point Of View ◽  
Learning Models ◽  
Ground Truth Data ◽  
Machine Learning Models ◽  
Sentinel 2

This paper presents DeepSatData a pipeline for automatically generating satellite imagery datasets for training machine learning models. We also discuss design considerations with emphasis on dense classification tasks, e.g. semantic segmentation. The implementation presented makes use of freely available Sentinel-2 data which allows the generation of large scale datasets required for training deep neural networks (DNN). We discuss issues faced from the point of view of DNN training and evaluation such as checking the quality of ground truth data and comment on the scalability of the approach.

scholarly journals Predicting Venue Popularity Using Crowd-Sourced and Passive Sensor Data

Smart Cities ◽  
2020 ◽  
Vol 3 (3) ◽  
pp. 818-841
Author(s):  
Stanislav Timokhin ◽  
Mohammad Sadrani ◽  
Constantinos Antoniou
Keyword(s):  
Auxiliary Information ◽  
Ground Truth ◽  
Spatiotemporal Data ◽  
Sensor Data ◽  
Mobility Patterns ◽  
Ground Truth Data ◽  
Dependent Variables ◽  
Passive Sensor ◽  
Location Based Social Networks

Efficient and reliable mobility pattern identification is essential for transport planning research. In order to infer mobility patterns, however, a large amount of spatiotemporal data is needed, which is not always available. Hence, location-based social networks (LBSNs) have received considerable attention as a potential data provider. The aim of this study is to investigate the possibility of using several different auxiliary information sources for venue popularity modeling and provide an alternative venue popularity measuring approach. Initially, data from widely used services, such as Google Maps, Yelp and OpenStreetMap (OSM), are used to model venue popularity. To estimate hourly venue occupancy, two different classes of model are used, including linear regression with lasso regularization and gradient boosted regression (GBR). The predictions are made based on venue-related parameters (e.g., rating, comments) and locational properties (e.g., stores, hotels, attractions). Results show that the prediction can be improved using GBR with a logarithmic transformation of the dependent variables. To investigate the quality of social media-based models by obtaining WiFi-based ground truth data, a microcontroller setup is developed to measure the actual number of people attending venues using WiFi presence detection, demonstrating that the similarity between the results of WiFi data collection and Google “Popular Times” is relatively promising.

scholarly journals Evaluation measure for group-based record linkage

2020 ◽  
Vol 4 (1) ◽  
Author(s):  
Charini Nanayakkara ◽  
Peter Christen ◽  
Thilina Ranbaduge ◽  
Eilidh Garrett
Keyword(s):  
Record Linkage ◽  
Evaluation Method ◽  
Ground Truth ◽  
Ground Truth Data ◽  
Evaluation Measures ◽  
Individual Record ◽  
Clustering Quality ◽  
Linkage Evaluation ◽  
Traditional Evaluation

Introduction The robustness of record linkage evaluation measures is of high importance since linkage techniques are assessed based on these. However, minimal research has been conducted to evaluate the suitability of existing evaluation measures in the context of linking groups of records. Linkage quality is generally evaluated based on traditional measures such as precision and recall. As we show, these traditional evaluation measures are not suitable for evaluating groups of linked records because they evaluate the quality of individual record pairs rather than the quality of records grouped into clusters. Objectives We highlight the shortcomings of traditional evaluation measures and then propose a novel method to evaluate clustering quality in the context of group-based record linkage. Methods The proposed linkage evaluation method assesses how well individual records have been allocated into predicted groups/clusters with respect to ground-truth data. We first identify the best representative predicted cluster for each ground-truth cluster and, based on the resulting mapping, each record in a ground-truth cluster is assigned to one of seven categories. These categories reflect how well the linkage technique assigned records into groups. Results We empirically evaluate our proposed method using real-world data and show that it better reflects the quality of clusters generated by three group-based record linkage techniques. We also show that traditional measures such as precision and recall can produce ambiguous results whereas our method does not. Conclusions The proposed evaluation method provides unambiguous results regarding the assessed group-based record linkage approaches. The method comprises of seven categories which reflect how each record was predicted, providing more detailed information about the quality of the linkage result. This will help to make better-informed decisions about which linkage technique is best suited for a given linkage application.

scholarly journals Improving the Quality of Satellite Imagery Based on Ground-Truth Data from Rain Gauge Stations

2018 ◽  
Vol 10 (3) ◽  
pp. 398 ◽  
Author(s):  
Ana Militino ◽  
M. Ugarte ◽  
Unai Pérez-Goya
Keyword(s):  
Satellite Imagery ◽  
Ground Truth ◽  
Rain Gauge ◽  
Ground Truth Data
Sign in / Sign up

Export Citation Format

Share Document