Test Collection Based Evaluation of Information Retrieval Systems

2010 ◽  
Vol 4 (4) ◽  
pp. 247-375 ◽  
Author(s):  
Mark Sanderson
Keyword(s):  
Information Retrieval ◽  
Test Collection ◽  
Retrieval Systems ◽  
Information Retrieval Systems

Document-based approach to improve the accuracy of pairwise comparison in evaluating information retrieval systems

2015 ◽  
Vol 67 (4) ◽  
pp. 408-421
Author(s):  
Sri Devi Ravana ◽  
MASUMEH SADAT TAHERI ◽  
Prabha Rajagopal
Keyword(s):  
Information Retrieval ◽  
Pairwise Comparison ◽  
Current Method ◽  
Statistical Testing ◽  
Test Collection ◽  
Content Type ◽  
Retrieval Systems ◽  
Information Retrieval Systems ◽  
The Mean ◽  
The Difference

Purpose – The purpose of this paper is to propose a method to have more accurate results in comparing performance of the paired information retrieval (IR) systems with reference to the current method, which is based on the mean effectiveness scores of the systems across a set of identified topics/queries. Design/methodology/approach – Based on the proposed approach, instead of the classic method of using a set of topic scores, the documents level scores are considered as the evaluation unit. These document scores are the defined document’s weight, which play the role of the mean average precision (MAP) score of the systems as a significance test’s statics. The experiments were conducted using the TREC 9 Web track collection. Findings – The p-values generated through the two types of significance tests, namely the Student’s t-test and Mann-Whitney show that by using the document level scores as an evaluation unit, the difference between IR systems is more significant compared with utilizing topic scores. Originality/value – Utilizing a suitable test collection is a primary prerequisite for IR systems comparative evaluation. However, in addition to reusable test collections, having an accurate statistical testing is a necessity for these evaluations. The findings of this study will assist IR researchers to evaluate their retrieval systems and algorithms more accurately.

scholarly journals Creation of Reliable Relevance Judgments in Information Retrieval Systems Evaluation Experimentation through Crowdsourcing: A Review

2014 ◽  
Vol 2014 ◽  
pp. 1-13 ◽  
Author(s):  
Parnia Samimi ◽  
Sri Devi Ravana
Keyword(s):  
Information Retrieval ◽  
Low Cost ◽  
Cost Evaluation ◽  
Test Collection ◽  
Relevance Judgment ◽  
Research Fields ◽  
Retrieval Systems ◽  
Relevance Judgments ◽  
Information Retrieval Systems ◽  
Novel Method

Test collection is used to evaluate the information retrieval systems in laboratory-based evaluation experimentation. In a classic setting, generating relevance judgments involves human assessors and is a costly and time consuming task. Researchers and practitioners are still being challenged in performing reliable and low-cost evaluation of retrieval systems. Crowdsourcing as a novel method of data acquisition is broadly used in many research fields. It has been proven that crowdsourcing is an inexpensive and quick solution as well as a reliable alternative for creating relevance judgments. One of the crowdsourcing applications in IR is to judge relevancy of query document pair. In order to have a successful crowdsourcing experiment, the relevance judgment tasks should be designed precisely to emphasize quality control. This paper is intended to explore different factors that have an influence on the accuracy of relevance judgments accomplished by workers and how to intensify the reliability of judgments in crowdsourcing experiment.

An evaluation of some conflation algorithms for information retrieval

1981 ◽  
Vol 3 (4) ◽  
pp. 177-183 ◽  
Author(s):  
Martin Lennon ◽  
David S. Peirce ◽  
Brian D. Tarry ◽  
Peter Willett
Keyword(s):  
Information Retrieval ◽  
Test Collection ◽  
Retrieval Systems ◽  
Information Retrieval Systems

The characteristics of conflation algorithms are discussed and examples given of some algorithms which have been used for information retrieval systems. Comparative experiments with a range of keyword dictionaries and with the Cranfield document test collection suggest that there is relatively little difference in the performance of the algorithms despite the widely disparate means by which they have been developed and by which they operate.

Relevance Predictability in Information Retrieval Systems

1967 ◽  
Vol 06 (02) ◽  
pp. 45-51 ◽  
Author(s):  
A. Kent ◽  
J. Belzer ◽  
M. Kuhfeerst ◽  
E. D. Dym ◽  
D. L. Shirey ◽  
...  
Keyword(s):  
Information Retrieval ◽  
Experimental Conditions ◽  
Treatment Groups ◽  
Retrieval Systems ◽  
Significant Difference ◽  
Information Retrieval Systems ◽  
High Predictability ◽  
Intermediate Response ◽  
Quantitative Indicators ◽  
Level Of Processing

An experiment is described which attempts to derive quantitative indicators regarding the potential relevance predictability of the intermediate stimuli used to represent documents in information retrieval systems. In effect, since the decision to peruse an entire document is often predicated upon the examination of one »level of processing« of the document (e.g., the citation and/or abstract), it became interesting to analyze the properties of what constitutes »relevance«. However, prior to such an analysis, an even more elementary step had to be made, namely, to determine what portions of a document should be examined.An evaluation of the ability of intermediate response products (IRPs), functioning as cues to the information content of full documents, to predict the relevance determination that would be subsequently made on these documents by motivated users of information retrieval systems, was made under controlled experimental conditions. The hypothesis that there might be other intermediate response products (selected extracts from the document, i.e., first paragraph, last paragraph, and the combination of first and last paragraph), that would be as representative of the full document as the traditional IRPs (citation and abstract) was tested systematically. The results showed that:1. there is no significant difference among the several IRP treatment groups on the number of cue evaluations of relevancy which match the subsequent user relevancy decision on the document;2. first and last paragraph combinations have consistently predicted relevancy to a higher degree than the other IRPs;3. abstracts were undistinguished as predictors; and4. the apparent high predictability rating for citations was not substantive.Some of these results are quite different than would be expected from previous work with unmotivated subjects.

Performance of eight information retrieval systems in the Spanish web

2005 ◽  
Vol 14 (5) ◽  
pp. 335-346
Author(s):  
Por Carlos Benito Amat ◽  
Por Carlos Benito Amat
Keyword(s):  
Information Retrieval ◽  
Retrieval Systems ◽  
Information Retrieval Systems

A Dialectical Approach to Search Engine Evaluation

Libri ◽  
2020 ◽  
Vol 70 (3) ◽  
pp. 227-237
Author(s):  
Mahdi Zeynali-Tazehkandi ◽  
Mohsen Nowkarizi
Keyword(s):  
Information Retrieval ◽  
Information Science ◽  
Library And Information Science ◽  
Related Literature ◽  
Philosophical Foundations ◽  
Dialectical Approach ◽  
Retrieval Systems ◽  
The World ◽  
Information Retrieval Systems ◽  
Oriented Approach

AbstractEvaluation of information retrieval systems is a fundamental topic in Library and Information Science. The aim of this paper is to connect the system-oriented and the user-oriented approaches to relevant philosophical schools. By reviewing the related literature, it was found that the evaluation of information retrieval systems is successful if it benefits from both system-oriented and user-oriented approaches (composite). The system-oriented approach is rooted in Parmenides’ philosophy of stability (immovable) which Plato accepts and attributes to the world of forms; the user-oriented approach is rooted in Heraclitus’ flux philosophy (motion) which Plato defers and attributes to the tangible world. Thus, using Plato’s theory is a comprehensive approach for recognizing the concept of relevance. The theoretical and philosophical foundations determine the type of research methods and techniques. Therefore, Plato’s dialectical method is an appropriate composite method for evaluating information retrieval systems.

Sign in / Sign up

Export Citation Format

Share Document