scholarly journals Mapping ESG Trends by Distant Supervision of Neural Language Models

2020 ◽  
Vol 2 (4) ◽  
pp. 453-468 ◽  
Author(s):  
Natraj Raman ◽  
Grace Bang ◽  
Armineh Nourbakhsh
Keyword(s):  
Business Strategy ◽  
Corporate Sustainability ◽  
Language Model ◽  
Semantic Knowledge ◽  
Language Models ◽  
Classification Model ◽  
Investment Strategies ◽  
Business Decisions ◽  
Distant Supervision ◽  
Business Operations

The integration of Environmental, Social and Governance (ESG) considerations into business decisions and investment strategies have accelerated over the past few years. It is important to quantify the extent to which ESG-related conversations are carried out by companies so that their impact on business operations can be objectively assessed. However, profiling ESG language is challenging due to its multi-faceted nature and the lack of supervised datasets. This research study aims to detect historical trends in ESG discussions by analyzing the transcripts of corporate earning calls. The proposed solution exploits recent advances in neural language modeling to understand the linguistic structure in ESG discourse. In detail, firstly we develop a classification model that categorizes the relevance of a text sentence to ESG. A pre-trained language model is fine-tuned on a small corporate sustainability reports dataset for this purpose. The semantic knowledge encoded in this classification model is then leveraged by applying it to the sentences in the conference transcripts using a novel distant-supervision approach. Extensive empirical evaluations against various pretraining techniques demonstrate the efficacy of the proposed transfer learning framework. Our analysis indicates that in the last 5 years, nearly 15% of the discussions during earnings calls pertained to ESG, implying that ESG factors are integral to business strategy.

A study of Turkish emotion classification with pretrained language models

2021 ◽  
pp. 016555152098550
Author(s):  
Alaettin Uçan ◽  
Murat Dörterler ◽  
Ebru Akçapınar Sezer
Keyword(s):  
Machine Learning ◽  
Language Model ◽  
Experimental Studies ◽  
Classification Performance ◽  
Research Field ◽  
Language Models ◽  
Classification Model ◽  
Data Sets ◽  
Emotion Classification ◽  
Model Approach

Emotion classification is a research field that aims to detect the emotions in a text using machine learning methods. In traditional machine learning (TML) methods, feature engineering processes cause the loss of some meaningful information, and classification performance is negatively affected. In addition, the success of modelling using deep learning (DL) approaches depends on the sample size. More samples are needed for Turkish due to the unique characteristics of the language. However, emotion classification data sets in Turkish are quite limited. In this study, the pretrained language model approach was used to create a stronger emotion classification model for Turkish. Well-known pretrained language models were fine-tuned for this purpose. The performances of these fine-tuned models for Turkish emotion classification were comprehensively compared with the performances of TML and DL methods in experimental studies. The proposed approach provides state-of-the-art performance for Turkish emotion classification.

scholarly journals Transformer-based deep neural network language models for Alzheimer’s disease detection from targeted speech

2020 ◽  
Author(s):  
Alireza Roshanzamir ◽  
Hamid Aghajan ◽  
Mahdieh Soleymani Baghshah
Keyword(s):  
Alzheimer’S Disease ◽  
Alzheimer's Disease ◽  
Natural Language ◽  
Language Processing ◽  
Language Model ◽  
Large Datasets ◽  
Language Models ◽  
Classification Model ◽  
Picture Description ◽  
Network Language

Abstract Background: We developed transformer-based deep learning models based on natural language processing for early diagnosis of Alzheimer’s disease from the picture description test.Methods: The lack of large datasets poses the most important limitation for using complex models that do not require feature engineering. Transformer-based pre-trained deep language models have recently made a large leap in NLP research and application. These models are pre-trained on available large datasets to understand natural language texts appropriately, and are shown to subsequently perform well on classification tasks with small training sets. The overall classification model is a simple classifier on top of the pre-trained deep language model.Results: The models are evaluated on picture description test transcripts of the Pitt corpus, which contains data of 170 AD patients with 257 interviews and 99 healthy controls with 243 interviews. The large bidirectional encoder representations from transformers (BERTLarge) embedding with logistic regression classifier achieves classification accuracy of 88.08%, which improves thestate-of-the-art by 2.48%.Conclusions: Using pre-trained language models can improve AD prediction. This not only solves the problem of lack of sufficiently large datasets, but also reduces the need for expert-defined features.

scholarly journals Language Models Application in Sentiment Attitude Extraction Task

2021 ◽  
Vol 33 (3) ◽  
pp. 199-222
Author(s):  
Nicolay Leonidovich Rusnachenko
Keyword(s):  
Mass Media ◽  
Language Model ◽  
Training Data ◽  
Language Models ◽  
Negative Effects ◽  
Named Entities ◽  
Distant Supervision ◽  
Lexical Resource ◽  
Attitude Extraction ◽  
Over The Top

Large text can convey various forms of sentiment information including the author’s position, positive or negative effects of some events, attitudes of mentioned entities towards to each other. In this paper, we experiment with BERT based language models for extracting sentiment attitudes between named entities. Given a mass media article and list of mentioned named entities, the task is to ex tract positive or negative attitudes between them. Efficiency of language model methods depends on the amount of training data. To enrich training data, we adopt distant supervision method, which provide automatic annotation of unlabeled texts using an additional lexical resource. The proposed approach is subdivided into two stages FRAME-BASED: (1) sentiment pairs list completion (PAIR-BASED), (2) document annotations using PAIR-BASED and FRAME-BASED factors. Being applied towards a large news collection, the method generates RuAttitudes2017 automatically annotated collection. We evaluate the approach on RuSentRel-1.0, consisted of mass media articles written in Russian. Adopting RuAttitudes2017 in the training process results in 10-13% quality improvement by F1-measure over supervised learning and by 25% over the top neural network based model results.

scholarly journals Text Classification Model Enhanced by Unlabeled Data for LaTeX Formula

2021 ◽  
Vol 11 (22) ◽  
pp. 10536
Author(s):  
Hua Cheng ◽  
Renjie Yu ◽  
Yixin Tang ◽  
Yiquan Fang ◽  
Tao Cheng
Keyword(s):  
Text Classification ◽  
Language Model ◽  
Unlabeled Data ◽  
Language Models ◽  
Classification Model ◽  
Specific Domain ◽  
Proper Nouns ◽  
Teacher Student ◽  
Generic Language ◽  
Model Training

Generic language models pretrained on large unspecific domains are currently the foundation of NLP. Labeled data are limited in most model training due to the cost of manual annotation, especially in domains including massive Proper Nouns such as mathematics and biology, where it affects the accuracy and robustness of model prediction. However, directly applying a generic language model on a specific domain does not work well. This paper introduces a BERT-based text classification model enhanced by unlabeled data (UL-BERT) in the LaTeX formula domain. A two-stage Pretraining model based on BERT(TP-BERT) is pretrained by unlabeled data in the LaTeX formula domain. A double-prediction pseudo-labeling (DPP) method is introduced to obtain high confidence pseudo-labels for unlabeled data by self-training. Moreover, a multi-rounds teacher–student model training approach is proposed for UL-BERT model training with few labeled data and more unlabeled data with pseudo-labels. Experiments on the classification of the LaTex formula domain show that the classification accuracies have been significantly improved by UL-BERT where the F1 score has been mostly enhanced by 2.76%, and lower resources are needed in model training. It is concluded that our method may be applicable to other specific domains with enormous unlabeled data and limited labelled data.

scholarly journals Corporate Sustainability Strategies and Decision Support Methods: A Bibliometric Analysis

2020 ◽  
Vol 12 (2) ◽  
pp. 521 ◽  
Author(s):  
Fotis Kitsios ◽  
Maria Kamariotou ◽  
Michael A. Talias
Keyword(s):  
Strategic Management ◽  
Bibliometric Analysis ◽  
Business Strategy ◽  
Corporate Sustainability ◽  
Management Decision ◽  
Sustainability Strategies ◽  
Strategic Approach ◽  
Management Decision Making ◽  
Number Of Publications ◽  
Support Methods

Sustainability is becoming an increasing issue for decision-makers and scholars worldwide and many managers understand the significance of the strategic approach of corporate sustainability. However, they face difficulties in aligning sustainable development and strategic management as well as to implement it in practice. Thus, the purpose of this paper is to conduct a bibliometric analysis exploring the integration of strategic management, decision-making and corporate sustainability, providing a framework of interrelated issues according to the current literature in this area. 72 peer-reviewed papers were analyzed based on Webster’s and Watson’s (2002) methodology. The results of this review revealed that the number of publications in this domain has increased in the last decade, and there is a need to foster research (especially empirical) in this field because managers should find out ways to implement, in action, corporate sustainability strategies and integrate their action plans with their business strategy. This review concludes with a framework that includes the most commonly addressed issues of this topic and provides opportunities and challenges for further research.

INTEGRATION OF n-GRAM LANGUAGE MODELS IN MULTIPLE CLASSIFIER SYSTEMS FOR OFFLINE HANDWRITTEN TEXT LINE RECOGNITION

2008 ◽  
Vol 22 (07) ◽  
pp. 1301-1321 ◽  
Author(s):  
ROMAN BERTOLAMI ◽  
HORST BUNKE
Keyword(s):  
Language Model ◽  
Language Models ◽  
Combination Method ◽  
Text Line ◽  
Multiple Classifier Systems ◽  
Classifier Systems ◽  
Handwritten Text ◽  
Handwritten Text Recognition ◽  
Multiple Classifier ◽  
N Gram

Current multiple classifier systems for unconstrained handwritten text recognition do not provide a straightforward way to utilize language model information. In this paper, we describe a generic method to integrate a statistical n-gram language model into the combination of multiple offline handwritten text line recognizers. The proposed method first builds a word transition network and then rescores this network with an n-gram language model. Experimental evaluation conducted on a large dataset of offline handwritten text lines shows that the proposed approach improves the recognition accuracy over a reference system as well as over the original combination method that does not include a language model.

scholarly journals Automated Source Code Generation and Auto-Completion Using Deep Learning: Comparing and Discussing Current Language Model-Related Approaches

AI ◽  
2021 ◽  
Vol 2 (1) ◽  
pp. 1-16
Author(s):  
Juan Cruz-Benito ◽  
Sanjay Vishwakarma ◽  
Francisco Martin-Fernandez ◽  
Ismael Faro
Keyword(s):  
Deep Learning ◽  
Learning Community ◽  
Programming Languages ◽  
Language Processing ◽  
Code Generation ◽  
Language Model ◽  
Language Models ◽  
Stochastic Gradient Descent ◽  
Network Architectures ◽  
Learning Architectures

In recent years, the use of deep learning in language models has gained much attention. Some research projects claim that they can generate text that can be interpreted as human writing, enabling new possibilities in many application areas. Among the different areas related to language processing, one of the most notable in applying this type of modeling is programming languages. For years, the machine learning community has been researching this software engineering area, pursuing goals like applying different approaches to auto-complete, generate, fix, or evaluate code programmed by humans. Considering the increasing popularity of the deep learning-enabled language models approach, we found a lack of empirical papers that compare different deep learning architectures to create and use language models based on programming code. This paper compares different neural network architectures like Average Stochastic Gradient Descent (ASGD) Weight-Dropped LSTMs (AWD-LSTMs), AWD-Quasi-Recurrent Neural Networks (QRNNs), and Transformer while using transfer learning and different forms of tokenization to see how they behave in building language models using a Python dataset for code generation and filling mask tasks. Considering the results, we discuss each approach’s different strengths and weaknesses and what gaps we found to evaluate the language models or to apply them in a real programming context.

HATIMU AISYAH KARYA ZURINAH HASSAN MENERUSI PERSPEKTIF ELAINE SHOWALTER MODEL BAHASA

2020 ◽  
Vol 8 (2) ◽  
pp. 54-62
Author(s):  
NUR ZALIKHA MAT RADZI ◽  
NASIRIN ABDILLAH ◽  
DAENG HALIZA DAENG JAMAL
Keyword(s):  
Language Model ◽  
Southeast Asian ◽  
Language Models ◽  
Symbolic Language ◽  
Female Characters ◽  
Literary Works ◽  
The Past ◽  
Historical Practices

Hatimu Aisyah karya Sasterawan Negara ke-13 iaitu - Zurinah Hassan, yang juga penerima Anugerah Hadiah Penulis Asia Tenggara (SEA Write Award) pada tahun 2004. Rentetan kejayaan beliau, telah menjadi tumpuan para pengkaji untuk meneliti aspek mengenai pengarangan wanita. Hatimu Aisyah merupakan novel pertama dihasilkan oleh Zurinah Hassan yang menekankan mengenai amalan adat resam zaman terdahulu sehingga ditelan arus pemodenan zaman. Novel Hatimu Aisyah mengetengahkan gambaran wanita yang mengutamakan adat dalam konteks perjalanan hidup bermasyarakat. Kajian terhadap karya Zurinah Hassan ini, bersandarkan kepada Model Bahasa Gagasan Elaine Showalter dari perspektif ginokritik untuk melihat watak-watak wanita. Antara Perbincangan dalam kajian ini adalah berfokuskan kepada simbolik bahasa dan bahasa sebagai ekspresi kesedaran wanita. Hasil dapatan keseluruhan kajian menunjukkan bahawa Zurinah Hassan menggunakan bahasa yang bersesuaian dengan gagasan bahasa daripada Elaine Showalter tetapi agak kurang menyerlah. Hal ini disebabkan keterbatasan penggunaan bahasa selaras dengan sosiobudaya masyarakat Melayu. Penemuan kajian ini dalam model bahasa wanita dapat dilihat menerusi simbolik bahasa dan bahasa sebagai ekspresi kesedaran wanita. Hasil manfaat dan kepentingan diperolehi masa hadapan dapat dilihat bahawa golongan wanita menzahirkan protes dan kritikan menerusi corak penulisan karya mereka meskipun masih dalam keadaan terkawal.   Hatimu Aisyah the 13th National literary works, namely-Zurinah Hassan, who is also the recipient of the Southeast Asian Writer award (SEA Write Award) in 2004. His success string has been the focus of researchers to examine the aspects of women's writings. Hatimu Aisyah is the first novel to be produced by Zurinah Hassan that emphasizes on the historical practices of the past, having swallowed the current modernization of the day. The Hatimu Aisyah Novel highlights the portrayal of women who are customcentric in the context of the communities life. Studies on Zurinah Hassan's work are based on the language Model of Elaine Showalter from the perspective of Ginokritik to see the female characters. Among the discussions in this study are focused on symbolic language and language as a expression of women's awareness. The overall findings of the study showed that Zurinah Hassan used a language that fits the language idea of Elaine Showalter but was somewhat less striking. This is due to the limitations of usage in line with the Malay social. The findings of this study in female language models can be seen through the symbolic language and language in the expression of women's awareness. The results of the benefits and interests gained future can be seen that women are in their protest and criticism through their work writing patterns despite being controlled.

Astrid

2020 ◽  
Vol 14 (4) ◽  
pp. 471-484
Author(s):  
Suraj Shetiya ◽  
Saravanan Thirumuruganathan ◽  
Nick Koudas ◽  
Gautam Das
Keyword(s):  
Deep Learning ◽  
Objective Function ◽  
Pattern Matching ◽  
Language Processing ◽  
Language Model ◽  
Language Models ◽  
Selectivity Estimation ◽  
Statistical Correlations ◽  
Benchmark Datasets ◽  
Traditional Approaches

Accurate selectivity estimation for string predicates is a long-standing research challenge in databases. Supporting pattern matching on strings (such as prefix, substring, and suffix) makes this problem much more challenging, thereby necessitating a dedicated study. Traditional approaches often build pruned summary data structures such as tries followed by selectivity estimation using statistical correlations. However, this produces insufficiently accurate cardinality estimates resulting in the selection of sub-optimal plans by the query optimizer. Recently proposed deep learning based approaches leverage techniques from natural language processing such as embeddings to encode the strings and use it to train a model. While this is an improvement over traditional approaches, there is a large scope for improvement. We propose Astrid, a framework for string selectivity estimation that synthesizes ideas from traditional and deep learning based approaches. We make two complementary contributions. First, we propose an embedding algorithm that is query-type (prefix, substring, and suffix) and selectivity aware. Consider three strings 'ab', 'abc' and 'abd' whose prefix frequencies are 1000, 800 and 100 respectively. Our approach would ensure that the embedding for 'ab' is closer to 'abc' than 'abd'. Second, we describe how neural language models could be used for selectivity estimation. While they work well for prefix queries, their performance for substring queries is sub-optimal. We modify the objective function of the neural language model so that it could be used for estimating selectivities of pattern matching queries. We also propose a novel and efficient algorithm for optimizing the new objective function. We conduct extensive experiments over benchmark datasets and show that our proposed approaches achieve state-of-the-art results.

scholarly journals Chord-aware automatic music transcription based on hierarchical Bayesian integration of acoustic and language models

2018 ◽  
Vol 7 ◽  
Author(s):  
Yuta Ojima ◽  
Eita Nakamura ◽  
Katsutoshi Itoyama ◽  
Kazuyoshi Yoshii
Keyword(s):  
Latent Variables ◽  
Language Model ◽  
Language Models ◽  
Sequential Dependency ◽  
Acoustic Model ◽  
Hierarchical Bayesian ◽  
Generative Process ◽  
Music Transcription ◽  
Automatic Music Transcription ◽  
Music Audio

This paper describes automatic music transcription with chord estimation for music audio signals. We focus on the fact that concurrent structures of musical notes such as chords form the basis of harmony and are considered for music composition. Since chords and musical notes are deeply linked with each other, we propose joint pitch and chord estimation based on a Bayesian hierarchical model that consists of an acoustic model representing the generative process of a spectrogram and a language model representing the generative process of a piano roll. The acoustic model is formulated as a variant of non-negative matrix factorization that has binary variables indicating a piano roll. The language model is formulated as a hidden Markov model that has chord labels as the latent variables and emits a piano roll. The sequential dependency of a piano roll can be represented in the language model. Both models are integrated through a piano roll in a hierarchical Bayesian manner. All the latent variables and parameters are estimated using Gibbs sampling. The experimental results showed the great potential of the proposed method for unified music transcription and grammar induction.

Sign in / Sign up

Export Citation Format

Share Document