Cross-lingual language modeling for low-resource speech recognition

2012 ◽  
Author(s):  
Ping Xu
Keyword(s):  
Speech Recognition ◽  
Language Modeling ◽  
Low Resource ◽  
Cross Lingual

scholarly journals Cross-Lingual Word Embeddings for Low-Resource Language Modeling

2017 ◽  
Author(s):  
Oliver Adams ◽  
Adam Makarucha ◽  
Graham Neubig ◽  
Steven Bird ◽  
Trevor Cohn
Keyword(s):  
Language Modeling ◽  
Word Embeddings ◽  
Low Resource ◽  
Cross Lingual

scholarly journals Exploring the Data Efficiency of Cross-Lingual Post-Training in Pretrained Language Models

2021 ◽  
Vol 11 (5) ◽  
pp. 1974 ◽  
Author(s):  
Chanhee Lee ◽  
Kisu Yang ◽  
Taesun Whang ◽  
Chanjun Park ◽  
Andrew Matteson ◽  
...  
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Language Model ◽  
Language Modeling ◽  
Language Models ◽  
Low Resource ◽  
High Resource ◽  
Cross Lingual ◽  
Data Efficiency

Language model pretraining is an effective method for improving the performance of downstream natural language processing tasks. Even though language modeling is unsupervised and thus collecting data for it is relatively less expensive, it is still a challenging process for languages with limited resources. This results in great technological disparity between high- and low-resource languages for numerous downstream natural language processing tasks. In this paper, we aim to make this technology more accessible by enabling data efficient training of pretrained language models. It is achieved by formulating language modeling of low-resource languages as a domain adaptation task using transformer-based language models pretrained on corpora of high-resource languages. Our novel cross-lingual post-training approach selectively reuses parameters of the language model trained on a high-resource language and post-trains them while learning language-specific parameters in the low-resource language. We also propose implicit translation layers that can learn linguistic differences between languages at a sequence level. To evaluate our method, we post-train a RoBERTa model pretrained in English and conduct a case study for the Korean language. Quantitative results from intrinsic and extrinsic evaluations show that our method outperforms several massively multilingual and monolingual pretrained language models in most settings and improves the data efficiency by a factor of up to 32 compared to monolingual training.

scholarly journals Exploiting Adapters for Cross-lingual Low-resource Speech Recognition

2021 ◽  
pp. 1-1
Author(s):  
Wenxin Hou ◽  
Han Zhu ◽  
Yidong Wang ◽  
Jindong Wang ◽  
Tao Qin ◽  
...  
Keyword(s):  
Speech Recognition ◽  
Low Resource ◽  
Cross Lingual
Sign in / Sign up

Export Citation Format

Share Document