scholarly journals Towards automatic generation of relevance judgments for a test collection

2016 ◽  
Author(s):  
Mireille Makary ◽  
Michael Oakes ◽  
Fadi Yamout
Keyword(s):  
Automatic Generation ◽  
Test Collection ◽  
Relevance Judgments

scholarly journals Creation of Reliable Relevance Judgments in Information Retrieval Systems Evaluation Experimentation through Crowdsourcing: A Review

2014 ◽  
Vol 2014 ◽  
pp. 1-13 ◽  
Author(s):  
Parnia Samimi ◽  
Sri Devi Ravana
Keyword(s):  
Information Retrieval ◽  
Low Cost ◽  
Cost Evaluation ◽  
Test Collection ◽  
Relevance Judgment ◽  
Research Fields ◽  
Retrieval Systems ◽  
Relevance Judgments ◽  
Information Retrieval Systems ◽  
Novel Method

Test collection is used to evaluate the information retrieval systems in laboratory-based evaluation experimentation. In a classic setting, generating relevance judgments involves human assessors and is a costly and time consuming task. Researchers and practitioners are still being challenged in performing reliable and low-cost evaluation of retrieval systems. Crowdsourcing as a novel method of data acquisition is broadly used in many research fields. It has been proven that crowdsourcing is an inexpensive and quick solution as well as a reliable alternative for creating relevance judgments. One of the crowdsourcing applications in IR is to judge relevancy of query document pair. In order to have a successful crowdsourcing experiment, the relevance judgment tasks should be designed precisely to emphasize quality control. This paper is intended to explore different factors that have an influence on the accuracy of relevance judgments accomplished by workers and how to intensify the reliability of judgments in crowdsourcing experiment.

The Role of Visual Distractors in the Simon Effect

2017 ◽  
Vol 64 (6) ◽  
pp. 387-397 ◽  
Author(s):  
Luisa Lugli ◽  
Stefania D’Ascenzo ◽  
Roberto Nicoletti ◽  
Carlo Umiltà
Keyword(s):  
Simon Effect ◽  
Simon Task ◽  
Automatic Generation ◽  
Spatial Code ◽  
Attentional Model ◽  
Previous Evidence ◽  
Spatial Stimulus ◽  
Attentional Resources ◽  
Code Changes

Abstract. The Simon effect lies on the automatic generation of a stimulus spatial code, which, however, is not relevant for performing the task. Results typically show faster performance when stimulus and response locations correspond, rather than when they do not. Considering reaction time distributions, two types of Simon effect have been individuated, which are thought to depend on different mechanisms: visuomotor activation versus cognitive translation of spatial codes. The present study aimed to investigate whether the presence of a distractor, which affects the allocation of attentional resources and, thus, the time needed to generate the spatial code, changes the nature of the Simon effect. In four experiments, we manipulated the presence and the characteristics of the distractor. Findings extend previous evidence regarding the distinction between visuomotor activation and cognitive translation of spatial stimulus codes in a Simon task. They are discussed with reference to the attentional model of the Simon effect.

A Method for Reliability Assessment of Spatial Plate Structures Through Automatic Generation of Ultimate Collapse Modes

1991 ◽  
Vol 1991 (170) ◽  
pp. 483-491 ◽  
Author(s):  
Hiroo Okada ◽  
Yoshisada Murotsu ◽  
Keiji Ueyama ◽  
Minoru Harada ◽  
Kazuya Kondo
Keyword(s):  
Reliability Assessment ◽  
Automatic Generation ◽  
Plate Structures
Sign in / Sign up

Export Citation Format

Share Document