scholarly journals Analyzing Information Leakage of Updates to Natural Language Models

2020 ◽  
Author(s):  
Santiago Zanella-Béguelin ◽  
Lukas Wutschitz ◽  
Shruti Tople ◽  
Victor Rühle ◽  
Andrew Paverd ◽  
...  
Keyword(s):  
Natural Language ◽  
Information Leakage ◽  
Language Models

scholarly journals Learning the language of viral evolution and escape

Science ◽  
2021 ◽  
Vol 371 (6526) ◽  
pp. 284-288 ◽  
Author(s):  
Brian Hie ◽  
Ellen D. Zhong ◽  
Bonnie Berger ◽  
Bryan Bryson
Keyword(s):  
Immune System ◽  
Natural Language ◽  
Vaccine Development ◽  
Sequence Data ◽  
Viral Evolution ◽  
Machine Learning Algorithms ◽  
Language Models ◽  
Viral Escape ◽  
Human Immune System ◽  
Influenza Hemagglutinin

The ability for viruses to mutate and evade the human immune system and cause infection, called viral escape, remains an obstacle to antiviral and vaccine development. Understanding the complex rules that govern escape could inform therapeutic design. We modeled viral escape with machine learning algorithms originally developed for human natural language. We identified escape mutations as those that preserve viral infectivity but cause a virus to look different to the immune system, akin to word changes that preserve a sentence’s grammaticality but change its meaning. With this approach, language models of influenza hemagglutinin, HIV-1 envelope glycoprotein (HIV Env), and severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) Spike viral proteins can accurately predict structural escape patterns using sequence data alone. Our study represents a promising conceptual bridge between natural language and viral evolution.

scholarly journals Pre-Trained Transformer-Based Language Models for Sundanese

2021 ◽  
Author(s):  
Wilson Wongso ◽  
Henry Lucky ◽  
Derwin Suhartono
Keyword(s):  
Natural Language ◽  
Text Classification ◽  
Training Data ◽  
Language Models ◽  
Classification Task ◽  
Language Understanding ◽  
Training Corpus ◽  
Low Resource ◽  
Corpus Size ◽  
Fine Tune

Abstract The Sundanese language has over 32 million speakers worldwide, but the language has reaped little to no benefits from the recent advances in natural language understanding. Like other low-resource languages, the only alternative is to fine-tune existing multilingual models. In this paper, we pre-trained three monolingual Transformer-based language models on Sundanese data. When evaluated on a downstream text classification task, we found that most of our monolingual models outperformed larger multilingual models despite the smaller overall pre-training data. In the subsequent analyses, our models benefited strongly from the Sundanese pre-training corpus size and do not exhibit socially biased behavior. We released our models for other researchers and practitioners to use.

scholarly journals Viable Threat on News Reading: Generating Biased News Using Natural Language Models

2020 ◽  
Author(s):  
Saurabh Gupta ◽  
Hong Huy Nguyen ◽  
Junichi Yamagishi ◽  
Isao Echizen
Keyword(s):  
Natural Language ◽  
Language Models

scholarly journals Text: An R-package for Analyzing and Visualizing Human Language Using Natural Language Processing and Deep Learning

2021 ◽  
Author(s):  
Oscar Nils Erik Kjell ◽  
H. Andrew Schwartz ◽  
Salvatore Giorgi
Keyword(s):  
Deep Learning ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Rating Scale ◽  
State Of The Art ◽  
R Package ◽  
Language Models ◽  
Categorical Variables ◽  
Human Language

The language that individuals use for expressing themselves contains rich psychological information. Recent significant advances in Natural Language Processing (NLP) and Deep Learning (DL), namely transformers, have resulted in large performance gains in tasks related to understanding natural language such as machine translation. However, these state-of-the-art methods have not yet been made easily accessible for psychology researchers, nor designed to be optimal for human-level analyses. This tutorial introduces text (www.r-text.org), a new R-package for analyzing and visualizing human language using transformers, the latest techniques from NLP and DL. Text is both a modular solution for accessing state-of-the-art language models and an end-to-end solution catered for human-level analyses. Hence, text provides user-friendly functions tailored to test hypotheses in social sciences for both relatively small and large datasets. This tutorial describes useful methods for analyzing text, providing functions with reliable defaults that can be used off-the-shelf as well as providing a framework for the advanced users to build on for novel techniques and analysis pipelines. The reader learns about six methods: 1) textEmbed: to transform text to traditional or modern transformer-based word embeddings (i.e., numeric representations of words); 2) textTrain: to examine the relationships between text and numeric/categorical variables; 3) textSimilarity and 4) textSimilarityTest: to computing semantic similarity scores between texts and significance test the difference in meaning between two sets of texts; and 5) textProjection and 6) textProjectionPlot: to examine and visualize text within the embedding space according to latent or specified construct dimensions (e.g., low to high rating scale scores).

scholarly journals Measuring and Improving Consistency in Pretrained Language Models

2021 ◽  
Vol 9 ◽  
pp. 1012-1031
Author(s):  
Yanai Elazar ◽  
Nora Kassner ◽  
Shauli Ravfogel ◽  
Abhilasha Ravichander ◽  
Eduard Hovy ◽  
...  
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Desirable Property ◽  
Language Models ◽  
Factual Knowledge ◽  
High Quality ◽  
Model Consistency

Abstract Consistency of a model—that is, the invariance of its behavior under meaning-preserving alternations in its input—is a highly desirable property in natural language processing. In this paper we study the question: Are Pretrained Language Models (PLMs) consistent with respect to factual knowledge? To this end, we create ParaRel🤘, a high-quality resource of cloze-style query English paraphrases. It contains a total of 328 paraphrases for 38 relations. Using ParaRel🤘, we show that the consistency of all PLMs we experiment with is poor— though with high variance between relations. Our analysis of the representational spaces of PLMs suggests that they have a poor structure and are currently not suitable for representing knowledge robustly. Finally, we propose a method for improving model consistency and experimentally demonstrate its effectiveness.1

scholarly journals Automatic Mixed-Precision Quantization Search of BERT

2021 ◽  
Author(s):  
Changsheng Zhao ◽  
Ting Hua ◽  
Yilin Shen ◽  
Qian Lou ◽  
Hongxia Jin
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Language Models ◽  
Model Compression ◽  
Mixed Precision ◽  
Knowledge Distillation ◽  
Model Size ◽  
Orthogonal Methods ◽  
Weight Pruning

Pre-trained language models such as BERT have shown remarkable effectiveness in various natural language processing tasks. However, these models usually contain millions of parameters, which prevent them from the practical deployment on resource-constrained devices. Knowledge distillation, Weight pruning, and Quantization are known to be the main directions in model compression. However, compact models obtained through knowledge distillation may suffer from significant accuracy drop even for a relatively small compression ratio. On the other hand, there are only a few attempts based on quantization designed for natural language processing tasks, and they usually require manual setting on hyper-parameters. In this paper, we proposed an automatic mixed-precision quantization framework designed for BERT that can conduct quantization and pruning simultaneously. Specifically, our proposed method leverages Differentiable Neural Architecture Search to assign scale and precision for parameters in each sub-group automatically, and at the same pruning out redundant groups of parameters. Extensive evaluations on BERT downstream tasks reveal that our proposed method beats baselines by providing the same performance with much smaller model size. We also show the possibility of obtaining the extremely light-weight model by combining our solution with orthogonal methods such as DistilBERT.

Database Tuning using Natural Language Processing

2021 ◽  
Vol 50 (3) ◽  
pp. 27-28
Author(s):  
Immanuel Trummer
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Training Data ◽  
Language Models ◽  
Learning Approaches ◽  
Training Samples ◽  
Starting Point ◽  
Training Cost ◽  
Transformer Model

Introduction. We have seen significant advances in the state of the art in natural language processing (NLP) over the past few years [20]. These advances have been driven by new neural network architectures, in particular the Transformer model [19], as well as the successful application of transfer learning approaches to NLP [13]. Typically, training for specific NLP tasks starts from large language models that have been pre-trained on generic tasks (e.g., predicting obfuscated words in text [5]) for which large amounts of training data are available. Using such models as a starting point reduces task-specific training cost as well as the number of required training samples by orders of magnitude [7]. These advances motivate new use cases for NLP methods in the context of databases.

Nominalizations in Requirements Engineering Natural Language Models

2018 ◽  
pp. 5127-5135 ◽  
Author(s):  
Claudia S. Litvak ◽  
Graciela Dora Susana Hadad ◽  
Jorge Horacio Doorn
Keyword(s):  
Software Development ◽  
Natural Language ◽  
Requirements Engineering ◽  
Language Models ◽  
Application Domain ◽  
Engineering Process ◽  
Usual Practice ◽  
Precise Meaning ◽  
Requirements Engineering Process ◽  
Nominal Form

It is a usual practice to use natural language in any document intended for clients and users in the requirements engineering process of a software development. This facilitates the comprehension of the requirements engineer's proposals to clients and users. However, natural language introduces some drawbacks, such as ambiguity and incompleteness, which attempt against a good comprehension of those documents. Glossaries help by reducing ambiguity, though they introduce their own linguistic weaknesses. The nominalization of verbs is one of them. There are sometimes appreciable differences between using a verb form or its nominal form, while in other cases they may be synonyms. Therefore, the requirements engineer must be aware of the precise meaning of each term used in the application domain, in order to correctly define them and properly use them in every document. In this chapter, guidelines about treatment of verb nominalization are given when constructing a specific glossary, called Language Extended Lexicon.

HIDING CRITICAL INFORMATION WHEN TRAINING LANGUAGE MODELS

2021 ◽  
pp. 15-18
Author(s):  
A. Evtushenko
Keyword(s):  
Natural Language ◽  
Language Processing ◽  
Text Processing ◽  
Language Model ◽  
Personal Data ◽  
Language Models ◽  
Training Dataset ◽  
Critical Information ◽  
Research Company ◽  
Learning Language

Machine learning language models are combinations of algorithms and neural networks designed for text processing composed in natural language (Natural Language Processing, NLP).  In 2020, the largest language model from the artificial intelligence research company OpenAI, GPT-3, was released, the maximum number of parameters of which reaches 175 billion. The parameterization of the model increased by more than 100 times made it possible to improve the quality of generated texts to a level that is hard to distinguish from human-written texts. It is noteworthy that this model was trained on a training dataset mainly collected from open sources on the Internet, the volume of which is estimated at 570 GB.  This article discusses the problem of memorizing critical information, in particular, personal data of individual, at the stage of training large language models (GPT-2/3 and derivatives), and also describes an algorithmic approach to solving this problem, which consists in additional preprocessing training dataset and refinement of the model inference in the context of generating pseudo-personal data and embedding into the results of work on the tasks of summarization, text generation, formation of answers to questions and others from the field of seq2seq.

Sign in / Sign up

Export Citation Format

Share Document