Beyond the bag-of-words paradigm to enhance information retrieval applications

2011 ◽  
Author(s):  
Paolo Ferragina
Keyword(s):  
Information Retrieval ◽  
Bag Of Words

scholarly journals To Phrase or Not to Phrase – Impact of User versus System Term Dependence upon Retrieval

2018 ◽  
Vol 2 (1) ◽  
pp. 1-14 ◽  
Author(s):  
Christina Lioma ◽  
Birger Larsen ◽  
Peter Ingwersen
Keyword(s):  
Information Retrieval ◽  
Search Engine ◽  
Small Subset ◽  
Bag Of Words ◽  
Versus System ◽  
Algorithmic Approaches ◽  
Performance Gains

Abstract When submitting queries to information retrieval (IR) systems, users often have the option of specifying which, if any, of the query terms are heavily dependent on each other and should be treated as a fixed phrase, for instance by placing them between quotes.In addition to such cases where users specify term dependence, automatic ways also exist for IR systems to detect dependent terms in queries. Most IR systems use both user and algorithmic approaches. It is not however clear whether and to what extent user-defined term dependence agrees with algorithmic estimates of term dependence, nor which of the two may fetch higher performance gains. Simply put, is it better to trust users or the system to detect term dependence in queries? To answer this question, we experiment with 101 crowdsourced search engine users and 334 queries (52 train and 282 test TREC queries) and we record 10 assessments per query. We find that (i) user assessments of term dependence differ significantly from algorithmic assessments of term dependence (their overlap is approximately 30%); (ii) there is little agreement among users about term dependence in queries, and this disagreement increases as queries become longer; (iii) the potential retrieval gain that can be fetched by treating term dependence (both user- and system-defined) over a bag of words baseline is reserved to a small subset (approximately 8%) of the queries, and is much higher for low-depth than deep precision measures. Points (ii) and (iii) constitute novel insights into term dependence.

scholarly journals A Structure-Driven Method for Information Retrieval-Based Software Change Impact Analysis

2018 ◽  
Vol 2018 ◽  
pp. 1-16
Author(s):  
Yun He ◽  
Tong Li ◽  
Wei Wang ◽  
Wei Lan ◽  
Xiang Li
Keyword(s):  
Information Retrieval ◽  
Impact Analysis ◽  
Contextual Information ◽  
Source Code ◽  
Word Embedding ◽  
Bag Of Words ◽  
Change Impact Analysis ◽  
Software Change ◽  
Change Impact ◽  
Single Method

An important application of information retrieval technology is software change impact analysis. Existing information retrieval-based change impact analysis methods select a single method to transform the source code corpus into vectors in a process known as indexing. The single method is chosen from two primary methods, known as the bag-of-words and word embedding models, each having their specific advantages and disadvantages. The bag-of-words model records every word in the source code but ignores contextual information in the corpus. The word embedding model records the contextual information but loses detail for individual words. To address this problem, we propose a structure-driven method for information retrieval-based change impact analysis (named SDM-CIA). SDM-CIA integrates the bag-of-words and word embedding models based on the software’s structure. Our experiments using a standard benchmark shows that when compared with the existing methods, SDM-CIA improves on precision performance, recall performance, F-score performance, and MRR performance by an average of 3.65%, 3.82%, 3.6%, and 10.28%, respectively. Our experiments confirm the effectiveness of SDM-CIA.

scholarly journals An efficient voice based information retrieval using bag of words based indexing

2018 ◽  
Vol 7 (3.3) ◽  
pp. 622
Author(s):  
R Uma ◽  
B Latha
Keyword(s):  
Data Mining ◽  
Information Retrieval ◽  
Retrieval System ◽  
Bag Of Words ◽  
Text Input ◽  
Indexing Method ◽  
Three Stages ◽  
Physically Challenged ◽  
Index Table ◽  
Infor Mation

Data mining is one of the leading and drastically growing researches nowadays. One of the main areas in data mining is Information Retrieval (IR). Information retrieval is a broad job and it is finding information without any structured nature. Infor-mation retrieval retrieves the user required information from a large collection of data. The existing approaches yet to improve the accuracy in terms of relevant accuracy. In this paper, it is motivated to provide an Information Retrieval System (IRS) where it can retrieve information with high relevancy. The proposed IRS is specially designed for physically challenged people like blind people where the input and the output taken/given is voice. The functionality of proposed IRS consists of three stages such as: (i) Voice to Text input, (II). Pattern Matching, and (III). Text to Voice output.In order to improve the accuracy and relevancy the proposed IRS uses an indexing method called Bag of Words (BOW). BOW is like an index-table which can be referred to store, compare and retrieve the information speedily and accurately. Index-table utilization in IRS improves the accuracy with minimized computational complexity. The proposed IRS is simulated in DOTNET software and the results are compared with the existing system results in order to evaluate the performance.  

scholarly journals Fuzzy Information Retrieval Based on Continuous Bag-of-Words Model

Symmetry ◽  
2020 ◽  
Vol 12 (2) ◽  
pp. 225
Author(s):  
Dong Qiu ◽  
Haihuan Jiang ◽  
Shuqiao Chen
Keyword(s):  
Information Retrieval ◽  
Large Scale ◽  
Query Language ◽  
Word Embedding ◽  
Bag Of Words ◽  
Fuzzy Information ◽  
Large Scale Data ◽  
Harmonic Average ◽  
Scale Data ◽  
Fuzzy Information Retrieval

In this paper, we study the feasibility of performing fuzzy information retrieval by word embedding. We propose a fuzzy information retrieval approach to capture the relationships between words and query language, which combines some techniques of deep learning and fuzzy set theory. We try to leverage large scale data and the continuous-bag-of words model to find the relevant feature of words and obtain word embedding. To enhance retrieval effectiveness, we measure the relativity among words by word embedding, with the property of symmetry. Experimental results show that the recall ratio, precision ratio, and harmonic average of two ratios of the proposed method outperforms the ones of the traditional methods.

scholarly journals Using Semantics for Granularities of Tokenization

2018 ◽  
Vol 44 (3) ◽  
pp. 483-524 ◽  
Author(s):  
Martin Riedl ◽  
Chris Biemann
Keyword(s):  
Information Retrieval ◽  
Data Sets ◽  
Bag Of Words ◽  
Semantic Model ◽  
Single Word ◽  
Evaluation Data ◽  
Pos Tagging ◽  
Distributional Information ◽  
Supervised Methods ◽  
Set Up

Depending on downstream applications, it is advisable to extend the notion of tokenization from low-level character-based token boundary detection to identification of meaningful and useful language units. This entails both identifying units composed of several single words that form a several single words that form a, as well as splitting single-word compounds into their meaningful parts. In this article, we introduce unsupervised and knowledge-free methods for these two tasks. The main novelty of our research is based on the fact that methods are primarily based on distributional similarity, of which we use two flavors: a sparse count-based and a dense neural-based distributional semantic model. First, we introduce DRUID, which is a method for detecting MWEs. The evaluation on MWE-annotated data sets in two languages and newly extracted evaluation data sets for 32 languages shows that DRUID compares favorably over previous methods not utilizing distributional information. Second, we present SECOS, an algorithm for decompounding close compounds. In an evaluation of four dedicated decompounding data sets across four languages and on data sets extracted from Wiktionary for 14 languages, we demonstrate the superiority of our approach over unsupervised baselines, sometimes even matching the performance of previous language-specific and supervised methods. In a final experiment, we show how both decompounding and MWE information can be used in information retrieval. Here, we obtain the best results when combining word information with MWEs and the compound parts in a bag-of-words retrieval set-up. Overall, our methodology paves the way to automatic detection of lexical units beyond standard tokenization techniques without language-specific preprocessing steps such as POS tagging.

PEMBANGUNAN LINK PENELUSURAN KEBUTUHAN FUNGSIONAL DAN METHOD PADA KODE SUMBER DENGAN METODE PENGAMBILAN INFORMASI

JURNAL ELTEK ◽  
2018 ◽  
Vol 16 (2) ◽  
pp. 151
Author(s):  
Djoko Pramono ◽  
Nanang Yudi Setiawan
Keyword(s):  
Information Retrieval ◽  
Latent Dirichlet Allocation ◽  
Bag Of Words ◽  
Dirichlet Allocation

Link penelusuran antara dokumen kebutuhan dan kode sumber sangatmembantu dalam proses pengembangan dan pemeliharaan perangkatlunak. Dalam proses pemeliharaan perangkat lunak, pengembangmelakukan perubahan pada kode sumber tetapi sering tidakmemperbarui dokumen yang menyertainya. Adanya link penelusuranantara dokumen kebutuhan dengan kode sumber diharapkanmeningkatkan kecepatan menemukan bagian kode sumber yang perludiubah ketika ada perubahan kebutuhan.Dalam penelitian ini dilakukan evaluasi terhadap dua metodepengambilan informasi (information retrieval/IR) yaitu LSA(LatentSemantic Analysis) dan LDA(Latent Dirichlet Allocation) untukmenemukan link antara kebutuhan fungsional dan method dalam kodesumber program. LSA merupakan sebuah metode yang memanfaatkanmodel statistik matematis untuk menganalisa struktur semantik suatuteks. LDA adalah model probabilistik generatif untuk sekelompok datadiskrit seperti corpus. Langkah pertama adalah membentuk kumpulankata(bag of words) dari method dalam kode sumber dan dokumenkebutuhan fungsional. Langkah berikutnya adalah menghitung nilaikedekatan semantic menggunakan cosine similarity.Pengujian dilakukan menggunakan kedua metode pada dataset yangterdiri dari kebutuhan fungsional dan kode sumber ITrust dan GanttProject. Kemudian dihitung nilai precision dan recall. Nilai f-measuresebesar 0,26 diperoleh pada recall sebesar 0,23 dan precission 0,305pada pengujian menggunakan metode LDA. Metode LDA memberikanhasil lebih baik daripada LSA namun nilai precision dan recall yangdihasilkan dari kedua metode tersebut masih rendah . 

Sign in / Sign up

Export Citation Format

Share Document