Cross-lingual transfer learning during supervised training in low resource scenarios

2015 ◽  
Author(s):  
Amit Das ◽  
Mark Hasegawa-Johnson
Keyword(s):  
Transfer Learning ◽  
Low Resource ◽  
Supervised Training ◽  
Cross Lingual

scholarly journals Text-to-speech system for low-resource language using cross-lingual transfer learning and data augmentation

2021 ◽  
Vol 2021 (1) ◽  
Author(s):  
Zolzaya Byambadorj ◽  
Ryota Nishimura ◽  
Altangerel Ayush ◽  
Kengo Ohta ◽  
Norihide Kitaoka
Keyword(s):  
Transfer Learning ◽  
Data Augmentation ◽  
Prediction Models ◽  
Target Language ◽  
Text To Speech ◽  
Paired Data ◽  
Low Resource ◽  
Language Data ◽  
Single Speaker ◽  
Cross Lingual

AbstractDeep learning techniques are currently being applied in automated text-to-speech (TTS) systems, resulting in significant improvements in performance. However, these methods require large amounts of text-speech paired data for model training, and collecting this data is costly. Therefore, in this paper, we propose a single-speaker TTS system containing both a spectrogram prediction network and a neural vocoder for the target language, using only 30 min of target language text-speech paired data for training. We evaluate three approaches for training the spectrogram prediction models of our TTS system, which produce mel-spectrograms from the input phoneme sequence: (1) cross-lingual transfer learning, (2) data augmentation, and (3) a combination of the previous two methods. In the cross-lingual transfer learning method, we used two high-resource language datasets, English (24 h) and Japanese (10 h). We also used 30 min of target language data for training in all three approaches, and for generating the augmented data used for training in methods 2 and 3. We found that using both cross-lingual transfer learning and augmented data during training resulted in the most natural synthesized target speech output. We also compare single-speaker and multi-speaker training methods, using sequential and simultaneous training, respectively. The multi-speaker models were found to be more effective for constructing a single-speaker, low-resource TTS model. In addition, we trained two Parallel WaveGAN (PWG) neural vocoders, one using 13 h of our augmented data with 30 min of target language data and one using the entire 12 h of the original target language dataset. Our subjective AB preference test indicated that the neural vocoder trained with augmented data achieved almost the same perceived speech quality as the vocoder trained with the entire target language dataset. Overall, we found that our proposed TTS system consisting of a spectrogram prediction network and a PWG neural vocoder was able to achieve reasonable performance using only 30 min of target language training data. We also found that by using 3 h of target language data, for training the model and for generating augmented data, our proposed TTS model was able to achieve performance very similar to that of the baseline model, which was trained with 12 h of target language data.

scholarly journals Multilingual Offensive Language Identification for Low-resource Languages

2022 ◽  
Vol 21 (1) ◽  
pp. 1-13
Author(s):  
Tharindu Ranasinghe ◽  
Marcos Zampieri
Keyword(s):  
Transfer Learning ◽  
Hate Speech ◽  
Training And Development ◽  
Language Identification ◽  
Shared Task ◽  
Low Resource ◽  
Government Organizations ◽  
Cross Lingual ◽  
Offensive Language ◽  
Clear Majority

Offensive content is pervasive in social media and a reason for concern to companies and government organizations. Several studies have been recently published investigating methods to detect the various forms of such content (e.g., hate speech, cyberbullying, and cyberaggression). The clear majority of these studies deal with English partially because most annotated datasets available contain English data. In this article, we take advantage of available English datasets by applying cross-lingual contextual word embeddings and transfer learning to make predictions in low-resource languages. We project predictions on comparable data in Arabic, Bengali, Danish, Greek, Hindi, Spanish, and Turkish. We report results of 0.8415 F1 macro for Bengali in TRAC-2 shared task [23], 0.8532 F1 macro for Danish and 0.8701 F1 macro for Greek in OffensEval 2020 [58], 0.8568 F1 macro for Hindi in HASOC 2019 shared task [27], and 0.7513 F1 macro for Spanish in in SemEval-2019 Task 5 (HatEval) [7], showing that our approach compares favorably to the best systems submitted to recent shared tasks on these three languages. Additionally, we report competitive performance on Arabic and Turkish using the training and development sets of OffensEval 2020 shared task. The results for all languages confirm the robustness of cross-lingual contextual embeddings and transfer learning for this task.

scholarly journals Machine Translation in Low-Resource Languages by an Adversarial Neural Network

2021 ◽  
Vol 11 (22) ◽  
pp. 10860
Author(s):  
Mengtao Sun ◽  
Hao Wang ◽  
Mark Pasquine ◽  
Ibrahim A. Hameed
Keyword(s):  
Machine Translation ◽  
Transfer Learning ◽  
Grammatical Structure ◽  
Neural Machine Translation ◽  
Low Resource ◽  
High Resource ◽  
Learning Techniques ◽  
Good Potential ◽  
Target Languages ◽  
Cross Lingual

Existing Sequence-to-Sequence (Seq2Seq) Neural Machine Translation (NMT) shows strong capability with High-Resource Languages (HRLs). However, this approach poses serious challenges when processing Low-Resource Languages (LRLs), because the model expression is limited by the training scale of parallel sentence pairs. This study utilizes adversary and transfer learning techniques to mitigate the lack of sentence pairs in LRL corpora. We propose a new Low resource, Adversarial, Cross-lingual (LAC) model for NMT. In terms of the adversary technique, LAC model consists of a generator and discriminator. The generator is a Seq2Seq model that produces the translations from source to target languages, while the discriminator measures the gap between machine and human translations. In addition, we introduce transfer learning on LAC model to help capture the features in rare resources because some languages share the same subject-verb-object grammatical structure. Rather than using the entire pretrained LAC model, we separately utilize the pretrained generator and discriminator. The pretrained discriminator exhibited better performance in all experiments. Experimental results demonstrate that the LAC model achieves higher Bilingual Evaluation Understudy (BLEU) scores and has good potential to augment LRL translations.

scholarly journals Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation

2020 ◽  
Vol 34 (05) ◽  
pp. 8854-8861 ◽  
Author(s):  
Aditya Siddhant ◽  
Melvin Johnson ◽  
Henry Tsai ◽  
Naveen Ari ◽  
Jason Riesa ◽  
...  
Keyword(s):  
Machine Translation ◽  
Transfer Learning ◽  
Single Model ◽  
Neural Machine Translation ◽  
Low Resource ◽  
Sequence Labeling ◽  
Transfer Capability ◽  
Learning Scenarios ◽  
The Cross ◽  
Cross Lingual

The recently proposed massively multilingual neural machine translation (NMT) system has been shown to be capable of translating over 100 languages to and from English within a single model (Aharoni, Johnson, and Firat 2019). Its improved translation performance on low resource languages hints at potential cross-lingual transfer capability for downstream tasks. In this paper, we evaluate the cross-lingual effectiveness of representations from the encoder of a massively multilingual NMT model on 5 downstream classification and sequence labeling tasks covering a diverse set of over 50 languages. We compare against a strong baseline, multilingual BERT (mBERT) (Devlin et al. 2018), in different cross-lingual transfer learning scenarios and show gains in zero-shot transfer in 4 out of these 5 tasks.

Sign in / Sign up

Export Citation Format

Share Document