scholarly journals Evolutionary Neural Architecture Search (NAS) Using Chromosome Non-Disjunction for Korean Grammaticality Tasks

2020 ◽  
Vol 10 (10) ◽  
pp. 3457
Author(s):  
Kang-moon Park ◽  
Donghoon Shin ◽  
Yongsuk Yoo
Keyword(s):  
Word Order ◽  
Language Model ◽  
Future Research ◽  
Linguistic Data ◽  
Neural Architecture ◽  
Grammaticality Judgment ◽  
Great Complexity

In this paper, we apply the neural architecture search (NAS) method to Korean grammaticality judgment tasks. Since the word order of a language is the final result of complex syntactic operations, a successful neural architecture search in linguistic data suggests that NAS can automate language model designing. Although NAS application to language has been suggested in the literature, we add a novel dataset that contains Korean-specific linguistic operations, which adds great complexity in the patterns. The result of the experiment suggests that NAS provides an architecture for the language. Interestingly, NAS has suggested an unprecedented structure that would not be designed manually. Research on the final topology of the architecture is the topic of our future research.

scholarly journals Developing Language-Specific Models Using a Neural Architecture Search

2021 ◽  
Vol 11 (21) ◽  
pp. 10324
Author(s):  
YongSuk Yoo ◽  
Kang-moon Park
Keyword(s):  
Word Order ◽  
Language Modeling ◽  
Future Research ◽  
Neural Architecture ◽  
Grammaticality Judgment ◽  
Korean And English

This paper applies the neural architecture search (NAS) method to Korean and English grammaticality judgment tasks. Based on the previous research, which only discusses the application of NAS on a Korean dataset, we extend the method to English grammatical tasks and compare the resulting two architectures from Korean and English. Since complex syntactic operations exist beneath the word order that is computed, the two different resulting architectures out of the automated NAS language modeling provide an interesting testbed for future research. To the extent of our knowledge, the methodology adopted here has not been tested in the literature. Crucially, the resulting structure of the NAS application shows an unexpected design for human experts. Furthermore, NAS has generated different models for Korean and English, which have different syntactic operations.

A Comprehensive Survey of Neural Architecture Search

2021 ◽  
Vol 54 (4) ◽  
pp. 1-34
Author(s):  
Pengzhen Ren ◽  
Yun Xiao ◽  
Xiaojun Chang ◽  
Po-yao Huang ◽  
Zhihui Li ◽  
...  
Keyword(s):  
Research Work ◽  
Search Space ◽  
Feature Representation ◽  
Future Research ◽  
Related Research ◽  
Neural Architecture ◽  
Comprehensive Survey ◽  
Future Research Directions ◽  
Intuitive Idea ◽  
New Perspective

Deep learning has made substantial breakthroughs in many fields due to its powerful automatic representation capabilities. It has been proven that neural architecture design is crucial to the feature representation of data and the final performance. However, the design of the neural architecture heavily relies on the researchers’ prior knowledge and experience. And due to the limitations of humans’ inherent knowledge, it is difficult for people to jump out of their original thinking paradigm and design an optimal model. Therefore, an intuitive idea would be to reduce human intervention as much as possible and let the algorithm automatically design the neural architecture. Neural Architecture Search ( NAS ) is just such a revolutionary algorithm, and the related research work is complicated and rich. Therefore, a comprehensive and systematic survey on the NAS is essential. Previously related surveys have begun to classify existing work mainly based on the key components of NAS: search space, search strategy, and evaluation strategy. While this classification method is more intuitive, it is difficult for readers to grasp the challenges and the landmark work involved. Therefore, in this survey, we provide a new perspective: beginning with an overview of the characteristics of the earliest NAS algorithms, summarizing the problems in these early NAS algorithms, and then providing solutions for subsequent related research work. In addition, we conduct a detailed and comprehensive analysis, comparison, and summary of these works. Finally, we provide some possible future research directions.

scholarly journals Pipelined language model construction for Polish speech recognition

2013 ◽  
Vol 23 (3) ◽  
pp. 649-668 ◽  
Author(s):  
Jerzy Sas ◽  
Andrzej Żołnierek
Keyword(s):  
Speech Recognition ◽  
Word Order ◽  
Language Model ◽  
Short Word ◽  
Initial Model ◽  
Practical Applications ◽  
Second Stage ◽  
Polish Language ◽  
Consistent Method ◽  
Lm Smoothing

Abstract The aim of works described in this article is to elaborate and experimentally evaluate a consistent method of Language Model (LM) construction for the sake of Polish speech recognition. In the proposed method we tried to take into account the features and specific problems experienced in practical applications of speech recognition in the Polish language, reach inflection, a loose word order and the tendency for short word deletion. The LM is created in five stages. Each successive stage takes the model prepared at the previous stage and modifies or extends it so as to improve its properties. At the first stage, typical methods of LM smoothing are used to create the initial model. Four most frequently used methods of LM construction are here. At the second stage the model is extended in order to take into account words indirectly co-occurring in the corpus. At the next stage, LM modifications are aimed at reduction of short word deletion errors, which occur frequently in Polish speech recognition. The fourth stage extends the model by insertion of words that were not observed in the corpus. Finally the model is modified so as to assure highly accurate recognition of very important utterances. The performance of the methods applied is tested in four language domains.

scholarly journals BioALBERT: A Simple and Effective Pre-trained Language Model for Biomedical Named Entity Recognition

2020 ◽  
Author(s):  
Usman Naseem ◽  
Matloob Khushi ◽  
Vinay Reddy ◽  
Sakthivel Rajendran ◽  
Imran Razzak ◽  
...  
Keyword(s):  
State Of The Art ◽  
Language Model ◽  
Named Entity Recognition ◽  
Training Data ◽  
Entity Recognition ◽  
Future Research ◽  
Named Entity ◽  
Domain Specific ◽  
Context Dependent ◽  
Biomedical Named Entity Recognition

Abstract Background: In recent years, with the growing amount of biomedical documents, coupled with advancement in natural language processing algorithms, the research on biomedical named entity recognition (BioNER) has increased exponentially. However, BioNER research is challenging as NER in the biomedical domain are: (i) often restricted due to limited amount of training data, (ii) an entity can refer to multiple types and concepts depending on its context and, (iii) heavy reliance on acronyms that are sub-domain specific. Existing BioNER approaches often neglect these issues and directly adopt the state-of-the-art (SOTA) models trained in general corpora which often yields unsatisfactory results. Results: We propose biomedical ALBERT (A Lite Bidirectional Encoder Representations from Transformers for Biomedical Text Mining) - bioALBERT - an effective domain-specific pre-trained language model trained on huge biomedical corpus designed to capture biomedical context-dependent NER. We adopted self-supervised loss function used in ALBERT that targets on modelling inter-sentence coherence to better learn context-dependent representations and incorporated parameter reduction strategies to minimise memory usage and enhance the training time in BioNER. In our experiments, BioALBERT outperformed comparative SOTA BioNER models on eight biomedical NER benchmark datasets with four different entity types. The performance is increased for; (i) disease type corpora by 7.47% (NCBI-disease) and 10.63% (BC5CDR-disease); (ii) drug-chem type corpora by 4.61% (BC5CDR-Chem) and 3.89 (BC4CHEMD); (iii) gene-protein type corpora by 12.25% (BC2GM) and 6.42% (JNLPBA); and (iv) Species type corpora by 6.19% (LINNAEUS) and 23.71% (Species-800) is observed which leads to a state-of-the-art results. Conclusions: The performance of proposed model on four different biomedical entity types shows that our model is robust and generalizable in recognizing biomedical entities in text. We trained four different variants of BioALBERT models which are available for the research community to be used in future research.

On-line grammaticality judgments in French children and adults: a crosslinguistic perspective

2004 ◽  
Vol 31 (3) ◽  
pp. 713-737 ◽  
Author(s):  
MICHELE KAIL
Keyword(s):  
Word Order ◽  
Position Effect ◽  
Age Groups ◽  
Age Group ◽  
Grammaticality Judgment ◽  
Grammaticality Judgments ◽  
On Line ◽  
Verbal Agreement ◽  
And Gender ◽  
New Processing

This study examined the on-line processing of French sentences in a grammaticality judgment experiment. Three age groups of French children (mean age: 6;8, 8;6 and 10;10 years) and a group of adults were asked to detect grammatical violations as quickly as possible. Three factors were studied: the violation type: agreement violations (number and gender) vs. word order violations; the violation position: early vs. late in the sentence; the target type of the violations: intra vs. interphrasal. An example of an early interphrasal verbal agreement violation follows: ‘Chaque semaine la voisine remplissent le frigo après avoir fait les courses au marché’ (Every week the neighbour fill the fridge after shopping at the market). The main developmental results were the following: not surprisingly, children were always slower than adults in the detection of grammatical violations. At each age level, morphological violations were more rapidly detected than word order violations. Each age group was faster at judging sentences with later occurring violations and the position effect was especially strong in the youngest groups. Finally, intraphrasal violations were more rapidly detected than interphrasal ones, this effect being observed only in the oldest groups (i.e. 10;10 years and adults). The results were compared to previous on-line data obtained in modern Greek (Kail & Diakogiorgi, 1998) showing strong similarities, even though Greek is a very rich morphological language. These results are discussed within the framework of the Competition Model, outlining the necessity to incorporate new processing constraints into the model.

How similar are adult second language learners and Spanish heritage speakers? Spanish clitics and word order

2009 ◽  
Vol 31 (1) ◽  
pp. 167-207 ◽  
Author(s):  
SILVINA MONTRUL
Keyword(s):  
Second Language ◽  
Word Order ◽  
Second Language Learners ◽  
Judgment Task ◽  
Heritage Speakers ◽  
Oral Production ◽  
Grammaticality Judgment ◽  
L2 Learners ◽  
Family Language ◽  
Spanish Heritage Speakers

ABSTRACTRecent studies of heritage speakers, many of whom possess incomplete knowledge of their family language, suggest that these speakers may be linguistically superior to second language (L2) learners only in phonology but not in morphosyntax. This study reexamines this claim by focusing on knowledge of clitic pronouns and word order in 24 L2 learners and 24 Spanish heritage speakers. Results of an oral production task, a written grammaticality judgment task, and a speeded comprehension task showed that, overall, heritage speakers seem to possess more nativelike knowledge of Spanish than their L2 counterparts. Implications for theories that stress the role of age and experience in L2 ultimate attainment and for the field of heritage language acquisition and teaching are discussed.

scholarly journals English and Russian Genitive Alternations: A Study in Construction Typology

2020 ◽  
Vol 24 (1) ◽  
pp. 96-116
Author(s):  
Sergei Monakhov
Keyword(s):  
Word Order ◽  
Structural Similarity ◽  
Construction Grammar ◽  
Future Research ◽  
Underlying Structure ◽  
Scientific Analysis ◽  
The World ◽  
The Comparative Study ◽  
Genitive Case

There is little doubt that one of the most important areas of future research within the framework of Construction Grammar will be the comparative study of constructions in different languages of the world. One significant gain that modern Construction Grammar can make thanks to the cross-linguistic perspective is finding a clue to some contradictory cases of construction alternation. The aim of the present paper is to communicate the results of a case study of two pairs of alternating constructions in English and Russian: s-genitive (SG) and of-genitive (OG) in English and noun + noun in genitive case (NNG) and relative adjective derived from noun + noun (ANG) in Russian. It is evident that the long years of elaborate scientific analysis have not yielded any universally accepted view on the problem of English genitive alternation. There are at least five different accounts of this problem: the hypotheses of the animacy hierarchy, given-new hierarchy, topic-focus hierarchy, end-weight principle, and two semantically distinct constructions. We hypothesised that in this case the comparison of the distribution of two English and two Russian genitives could be insightful. The analysis presupposed two consecutive steps. First, we established an inter-language comparability of two pairs of constructions in English and Russian. Second, we tested the similarity of intra-language distribution of each pair of constructions from the perspective of the animacy hierarchy. For these two purposes, two types of corpora were used: (1) a translation corpus consisting of original texts in one language and their translations into one or more languages; and (2) national corpora consisting of original texts in two respective languages. It was established that in both languages, the choice between members of an alternating pair is governed by the rules of animacy hierarchisation. Additionally, it was possible to disprove the idea that the animacy hierarchy is necessarily based on the linearisation hierarchy. Two Russian constructions are typologically aligned with their English counterparts, not on the grounds of the linear order of head and modifier but on the grounds of structural similarity. The English SG and Russian NNG construction are diametrically opposed in terms of word order. However, they reveal the same underlying structure of the inflectional genitive as contrasted with the analytical genitive of the Russian ANG and the English OG. These findings speak strongly in favour of the animacy hierarchy account of English genitive alternation.

scholarly journals Constructing an XML database of linguistics data

2010 ◽  
Vol 6 (1) ◽  
Author(s):  
J H Kroeze ◽  
T JD Bothma ◽  
M C Matthee
Keyword(s):  
Programming Language ◽  
Old Testament ◽  
Word Order ◽  
Ad Hoc ◽  
Hebrew Text ◽  
Visual Basic ◽  
Free Text ◽  
Xml Database ◽  
Linguistic Data

A language-oriented, multi-dimensional database of the linguistic characteristics of the Hebrew text of the Old Testament can enable researchers to do ad hoc queries. XML is a suitable technology to transform free text into a database. A clause’s word order can be kept intact while other features such as syntactic and semantic functions can be marked as elements or attributes. The elements or attributes from the XML “database” can be accessed and proces sed by a 4th generation programming language, such as Visual Basic. XML is explored as an option to build an exploitable database of linguistic data by representing inherently multi-dimensional data, including syntactic and semantic analyses of free text.

scholarly journals Automated Machine Learning on Graphs: A Survey

2021 ◽  
Author(s):  
Ziwei Zhang ◽  
Xin Wang ◽  
Wenwu Zhu
Keyword(s):  
Machine Learning ◽  
Learning Algorithm ◽  
Research Community ◽  
Future Research ◽  
Research Directions ◽  
Vast Number ◽  
Neural Architecture ◽  
Future Research Directions ◽  
Automated Machine Learning ◽  
Optimal Machine

Machine learning on graphs has been extensively studied in both academic and industry. However, as the literature on graph learning booms with a vast number of emerging methods and techniques, it becomes increasingly difficult to manually design the optimal machine learning algorithm for different graph-related tasks. To solve this critical challenge, automated machine learning (AutoML) on graphs which combines the strength of graph machine learning and AutoML together, is gaining attention from the research community. Therefore, we comprehensively survey AutoML on graphs in this paper, primarily focusing on hyper-parameter optimization (HPO) and neural architecture search (NAS) for graph machine learning. We further overview libraries related to automated graph machine learning and in-depth discuss AutoGL, the first dedicated open-source library for AutoML on graphs. In the end, we share our insights on future research directions for automated graph machine learning. This paper is the first systematic and comprehensive review of automated machine learning on graphs to the best of our knowledge.

Sign in / Sign up

Export Citation Format

Share Document