Semantics in High-Dimensional Space

Geometric models are used for modelling meaning in various semantic-space models. They are seductive in their simplicity and their imaginative qualities, and for that reason, their metaphorical power risks leading our intuitions astray: human intuition works well in a three-dimensional world but is overwhelmed by higher dimensionalities. This note is intended to warn about some practical pitfalls of using high-dimensional geometric representation as a knowledge representation and a memory model—challenges that can be met by informed design of the representation and its application.

Graphical vs. Spatial Models of Distributional Semantics

Semantic space models based on distributional information and semantic network (graphical) models are two of the most popular models of semantic representation. Both types of models succeed at modeling or explaining various tasks. Both types of models also have limitations. Spatial models have difficulties representing indirect semantic relations, while graphical models have lacked a theoretical account for the construction of their semantic network. In this article, we develop the Distributional Graph Model. The new model resembles semantic space models in the way that it is a representation of semantic memory obtained from statistical learning on a linguistic corpus. But like other graphical models, it is able to capture indirect semantic relatedness as well. Using an artificial language specifically designed to test different types of syntagmatic and paradigmatic relationships, we show that the Distributional Graph Model demonstrates the benefits of both graphical and spatial distributional models.

Word Space Creator: A Visual Tool for Semantic Space Visualization

Semantic spaces are used as a representation of language, capturing the meaning between linguistic units. These spaces are often built in large corpora requiring advanced equipment, specialized computational skills, and considerable effort. This project note will introduce and demonstrate the use of an accessible Shiny graphical interface allowing users to create semantic space models easily. Shiny is an R package in which one can program interactive web applications in R for others to interact with data or analyses. The advantage to Shiny applications is that naïve users can explore data without understanding the programming, and open sharing of code with the application can aid in learning the programming for one’s own use in their research. Within the application, users will be able to load popular semantic spaces or their own corpus for semantic space creation utilizing their preferred modeling technique, including LSA and TOPICS. A variety of user-friendly graphical tools, such as n-nearest neighbors or topic weighted graph, will further aid data visualization of the semantic network. Additionally, the application provides the calculation of cosine or simple co-occurrence, among other popular-relatedness values. This tool is intended for researchers who may not be programming-savvy, or as a teaching extension for psycholinguistics courses.

Encoding Sequential Information in Semantic Space Models: Comparing Holographic Reduced Representation and Random Permutation

Circular convolution and random permutation have each been proposed as neurally plausible binding operators capable of encoding sequential information in semantic memory. We perform several controlled comparisons of circular convolution and random permutation as means of encoding paired associates as well as encoding sequential information. Random permutations outperformed convolution with respect to the number of paired associates that can be reliably stored in a single memory trace. Performance was equal on semantic tasks when using a small corpus, but random permutations were ultimately capable of achieving superior performance due to their higher scalability to large corpora. Finally, “noisy” permutations in which units are mapped to other units arbitrarily (no one-to-one mapping) perform nearly as well as true permutations. These findings increase the neurological plausibility of random permutations and highlight their utility in vector space models of semantics.

Lexical acquisition and semantic space models: Learning the semantics of unknown words

In recent studies it has been shown that syntax-based semantic space models outperform models in which the context is represented as a bag-of-words in several semantic analysis tasks. This has been generally attributed to the fact that syntax-based models employ corpora that are syntactically annotated by a parser and a computational grammar. However, if the corpora processed contain words which are unknown to the parser and the grammar, a syntax-based model may lose its advantage since the syntactic properties of such words are unavailable. On the other hand, bag-of-words models do not face this issue since they operate on raw, non-annotated corpora and are thus more robust. In this paper, we compare the performance of syntax-based and bag-of-words models when applied to the task of learning the semantics of unknown words. In our experiments, unknown words are considered the words which are not known to the Alpino parser and grammar of Dutch. In our study, the semantics of an unknown word is defined by finding its most similar word incornetto, a Dutch lexico-semantic hierarchy. We show that for unknown words the syntax-based model performs worse than the bag-of-words approach. Furthermore, we show that if we first learn the syntactic properties of unknown words by an appropriate lexical acquisition method, then in fact the syntax-based model does outperform the bag-of-words approach. The conclusion we draw is that, for words unknown to a given grammar, a bag-of-words model is more robust than a syntax-based model. However, the combination of lexical acquisition and syntax-based semantic models is best suited for learning the semantics of unknown words.

Automatic discovery of word semantic relations using paraphrase alignment and distributional lexical semantics analysis

Thesauri, which list the most salient semantic relations between words, have mostly been compiled manually. Therefore, the inclusion of an entry depends on the subjective decision of the lexicographer. As a consequence, those resources are usually incomplete. In this paper, we propose an unsupervised methodology to automatically discover pairs of semantically related words by highlighting their local environment and evaluating their semantic similarity in local and global semantic spaces. This proposal differs from all other research presented so far as it tries to take the best of two different methodologies, i.e. semantic space models and information extraction models. In particular, it can be applied to extract close semantic relations, it limits the search space to few, highly probable options and it is unsupervised.