scholarly journals Deep Learning Based on Hierarchical Self-Attention for Finance Distress Prediction Incorporating Text

2021 ◽  
Vol 2021 ◽  
pp. 1-11
Author(s):  
Sumei Ruan ◽  
Xusheng Sun ◽  
Ruanxingchen Yao ◽  
Wei Li
Keyword(s):  
Financial Distress ◽  
Financial Risk ◽  
Large Scale ◽  
Early Stage ◽  
Fine Tuning ◽  
Original Text ◽  
Word Level ◽  
Sentence Level ◽  
Proposed Model ◽  
Red Flag

To detect comprehensive clues and provide more accurate forecasting in the early stage of financial distress, in addition to financial indicators, digitalization of lengthy but indispensable textual disclosure, such as Management Discussion and Analysis (MD&A), has been emphasized by researchers. However, most studies divide the long text into words and count words to treat the text as word count vectors, bringing massive invalid information but ignoring meaningful contexts. Aiming to efficiently represent the text of large size, an end-to-end neural networks model based on hierarchical self-attention is proposed in this study after the state-of-the-art pretrained model is introduced for text embedding including contexts. The proposed model has two notable characteristics. First, the hierarchical self-attention only affords the essential content with high weights in word-level and sentence-level and automatically neglects lots of information that has no business with risk prediction, which is suitable for extracting effective parts of the large-scale text. Second, after fine-tuning, the word embedding adapts the specific contexts of samples and conveys the original text expression more accurately without excessive manual operations. Experiments confirm that the addition of text improves the accuracy of financial distress forecasting and the proposed model outperforms benchmark models better at AUC and F2-score. For visualization, the elements in the weight matrix of hierarchical self-attention act as scalers to estimate the importance of each word and sentence. In this way, the “red-flag” statement that implies financial risk is figured out and highlighted in the original text, providing effective references for decision-makers.

scholarly journals Antonym-Synonym Classification Based on New Sub-Space Embeddings

2019 ◽  
Vol 33 ◽  
pp. 6204-6211
Author(s):  
Muhammad Asif Ali ◽  
Yifang Sun ◽  
Xiaoling Zhou ◽  
Wei Wang ◽  
Xiang Zhao
Keyword(s):  
Large Scale ◽  
Semantic Information ◽  
Relation Extraction ◽  
Specific Information ◽  
Lexical Semantic ◽  
Word Level ◽  
Novel Approach ◽  
Proposed Model ◽  
Low Performance ◽  
And Performance

Distinguishing antonyms from synonyms is a key challenge for many NLP applications focused on the lexical-semantic relation extraction. Existing solutions relying on large-scale corpora yield low performance because of huge contextual overlap of antonym and synonym pairs. We propose a novel approach entirely based on pre-trained embeddings. We hypothesize that the pre-trained embeddings comprehend a blend of lexical-semantic information and we may distill the task-specific information using Distiller, a model proposed in this paper. Later, a classifier is trained based on features constructed from the distilled sub-spaces along with some word level features to distinguish antonyms from synonyms. Experimental results show that the proposed model outperforms existing research on antonym synonym distinction in both speed and performance.

scholarly journals Distant Supervision for Relation Extraction with Sentence Selection and Interaction Representation

2021 ◽  
Vol 2021 ◽  
pp. 1-16
Author(s):  
Tiantian Chen ◽  
Nianbin Wang ◽  
Hongbin Wang ◽  
Haomin Zhan
Keyword(s):  
Large Scale ◽  
Semantic Information ◽  
State Of The Art ◽  
Relation Extraction ◽  
Semantic Features ◽  
Distant Supervision ◽  
Word Level ◽  
Proposed Model ◽  
Relation Prediction ◽  
Better Than

Distant supervision (DS) has been widely used for relation extraction (RE), which automatically generates large-scale labeled data. However, there is a wrong labeling problem, which affects the performance of RE. Besides, the existing method suffers from the lack of useful semantic features for some positive training instances. To address the above problems, we propose a novel RE model with sentence selection and interaction representation for distantly supervised RE. First, we propose a pattern method based on the relation trigger words as a sentence selector to filter out noisy sentences to alleviate the wrong labeling problem. After clean instances are obtained, we propose the interaction representation using the word-level attention mechanism-based entity pairs to dynamically increase the weights of the words related to entity pairs, which can provide more useful semantic information for relation prediction. The proposed model outperforms the strongest baseline by 2.61 in F1-score on a widely used dataset, which proves that our model performs significantly better than the state-of-the-art RE systems.

scholarly journals Dual-View Variational Autoencoders for Semi-Supervised Text Matching

2019 ◽  
Author(s):  
Zhongbin Xie ◽  
Shuai Ma
Keyword(s):  
Question Answering ◽  
Fundamental Problem ◽  
Sentence Pair ◽  
Community Question Answering ◽  
Word Level ◽  
Sentence Level ◽  
Proposed Model ◽  
Variational Autoencoder ◽  
Matching Models ◽  
Text Matching

Semantically matching two text sequences (usually two sentences) is a fundamental problem in NLP. Most previous methods either encode each of the two sentences into a vector representation (sentence-level embedding) or leverage word-level interaction features between the two sentences. In this study, we propose to take the sentence-level embedding features and the word-level interaction features as two distinct views of a sentence pair, and unify them with a framework of Variational Autoencoders such that the sentence pair is matched in a semi-supervised manner. The proposed model is referred to as Dual-View Variational AutoEncoder (DV-VAE), where the optimization of the variational lower bound can be interpreted as an implicit Co-Training mechanism for two matching models over distinct views. Experiments on SNLI, Quora and a Community Question Answering dataset demonstrate the superiority of our DV-VAE over several strong semi-supervised and supervised text matching models.

Chronological Ordering Based on Context Overlap Detection

2012 ◽  
Vol 2 (4) ◽  
pp. 31-44
Author(s):  
Mohamed H. Haggag ◽  
Bassma M. Othman
Keyword(s):  
Language Processing ◽  
Linguistic Knowledge ◽  
Syntactic Analysis ◽  
Text Generation ◽  
Context Processing ◽  
Domain Specific ◽  
Word Level ◽  
Sentence Level ◽  
Proposed Model ◽  
Chronological Sequence

Context processing plays an important role in different Natural Language Processing applications. Sentence ordering is one of critical tasks in text generation. Following the same order of sentences in the row sources of text is not necessarily to be applied for the resulted text. Accordingly, a need for chronological sentence ordering is of high importance in this regard. Some researches followed linguistic syntactic analysis and others used statistical approaches. This paper proposes a new model for sentence ordering based on sematic analysis. Word level semantics forms a seed to sentence level sematic relations. The model introduces a clustering technique based on sentences senses relatedness. Following to this, sentences are chronologically ordered through two main steps; overlap detection and chronological cause-effect rules. Overlap detection drills down into each cluster to step through its sentences in chronological sequence. Cause-effect rules forms the linguistic knowledge controlling sentences relations. Evaluation of the proposed algorithm showed the capability of the proposed model to process size free texts, non-domain specific and open to extend the cause-effect rules for specific ordering needs.

scholarly journals From Words to Sentences: A Progressive Learning Approach for Zero-resource Machine Translation with Visual Pivots

2019 ◽  
Author(s):  
Shizhe Chen ◽  
Qin Jin ◽  
Jianlong Fu
Keyword(s):  
Machine Translation ◽  
Large Scale ◽  
Learning Approach ◽  
Model Learning ◽  
Neural Machine Translation ◽  
Parallel Corpora ◽  
Translation Model ◽  
Word Level ◽  
Sentence Level ◽  
Progressive Learning

The neural machine translation model has suffered from the lack of large-scale parallel corpora. In contrast, we humans can learn multi-lingual translations even without parallel texts by referring our languages to the external world. To mimic such human learning behavior, we employ images as pivots to enable zero-resource translation learning. However, a picture tells a thousand words, which makes multi-lingual sentences pivoted by the same image noisy as mutual translations and thus hinders the translation model learning. In this work, we propose a progressive learning approach for image-pivoted zero-resource machine translation. Since words are less diverse when grounded in the image, we first learn word-level translation with image pivots, and then progress to learn the sentence-level translation by utilizing the learned word translation to suppress noises in image-pivoted multi-lingual sentences. Experimental results on two widely used image-pivot translation datasets, IAPR-TC12 and Multi30k, show that the proposed approach significantly outperforms other state-of-the-art methods.

scholarly journals Exploring Bilingual Parallel Corpora for Syntactically Controllable Paraphrase Generation

2020 ◽  
Author(s):  
Mingtong Liu ◽  
Erguang Yang ◽  
Deyi Xiong ◽  
Yujie Zhang ◽  
Chen Sheng ◽  
...  
Keyword(s):  
Language Processing ◽  
Latent Variable ◽  
Large Scale ◽  
Fine Tuning ◽  
Parallel Corpora ◽  
Shared Content ◽  
Proposed Model ◽  
Parallel Data ◽  
Cross Lingual ◽  
Paraphrase Generation

Paraphrase generation is of great importance to many downstream tasks in natural language processing. Recent efforts have focused on generating paraphrases in specific syntactic forms, which, generally, heavily relies on manually annotated paraphrase data that is not easily available for many languages and domains. In this paper, we propose a novel end-to-end framework to leverage existing large-scale bilingual parallel corpora to generate paraphrases under the control of syntactic exemplars. In order to train one model over the two languages of parallel corpora, we embed sentences of them into the same content and style spaces with shared content and style encoders using cross-lingual word embeddings. We propose an adversarial discriminator to disentangle the content and style space, and employ a latent variable to model the syntactic style of a given exemplar in order to guide the two decoders for generation. Additionally, we introduce cycle and masking learning schemes to efficiently train the model. Experiments and analyses demonstrate that the proposed model trained only on bilingual parallel data is capable of generating diverse paraphrases with desirable syntactic styles. Fine-tuning the trained model on a small paraphrase corpus makes it substantially outperform state-of-the-art paraphrase generation models trained on a larger paraphrase dataset.

Target-Templated de novo Design of Macrocyclic D-/L-Peptides: Inhibitors of the PD-1/PD-L1 Interaction

2020 ◽  
Author(s):  
Salvador Guardiola ◽  
Monica Varese ◽  
Xavier Roig ◽  
Jesús Garcia ◽  
Ernest Giralt
Keyword(s):  
Protein Interactions ◽  
Cyclic Peptides ◽  
General Framework ◽  
Large Scale ◽  
De Novo ◽  
Inhibitory Effect ◽  
Original Text ◽  
Protein Protein Interactions ◽  
Retraction Notice ◽  
Pharmaceutical Properties

<p>NOTE: This preprint has been retracted by consensus from all authors. See the retraction notice in place above; the original text can be found under "Version 1", accessible from the version selector above.</p><p><br></p><p>------------------------------------------------------------------------</p><p><br></p><p>Peptides, together with antibodies, are among the most potent biochemical tools to modulate challenging protein-protein interactions. However, current structure-based methods are largely limited to natural peptides and are not suitable for designing target-specific binders with improved pharmaceutical properties, such as macrocyclic peptides. Here we report a general framework that leverages the computational power of Rosetta for large-scale backbone sampling and energy scoring, followed by side-chain composition, to design heterochiral cyclic peptides that bind to a protein surface of interest. To showcase the applicability of our approach, we identified two peptides (PD-<i>i</i>3 and PD-<i>i</i>6) that target PD-1, a key immune checkpoint, and work as protein ligand decoys. A comprehensive biophysical evaluation confirmed their binding mechanism to PD-1 and their inhibitory effect on the PD-1/PD-L1 interaction. Finally, elucidation of their solution structures by NMR served as validation of our <i>de novo </i>design approach. We anticipate that our results will provide a general framework for designing target-specific drug-like peptides.<i></i></p>

A Target-Based Method for Designing Heterochiral Cyclic Peptide Binders: De Novo Inhibitors of the PD-1/PD-L1 Interaction

2020 ◽  
Author(s):  
Salvador Guardiola ◽  
Monica Varese ◽  
Xavier Roig ◽  
Jesús Garcia ◽  
Ernest Giralt
Keyword(s):  
Protein Interactions ◽  
Cyclic Peptides ◽  
General Framework ◽  
Large Scale ◽  
Cyclic Peptide ◽  
De Novo ◽  
Inhibitory Effect ◽  
Original Text ◽  
Retraction Notice ◽  
Pharmaceutical Properties

<p>NOTE: This preprint has been retracted by consensus from all authors. See the retraction notice in place above; the original text can be found under "Version 1", accessible from the version selector above.</p><p><br></p><p>------------------------------------------------------------------------</p><p><br></p><p>Peptides, together with antibodies, are among the most potent biochemical tools to modulate challenging protein-protein interactions. However, current structure-based methods are largely limited to natural peptides and are not suitable for designing target-specific binders with improved pharmaceutical properties, such as macrocyclic peptides. Here we report a general framework that leverages the computational power of Rosetta for large-scale backbone sampling and energy scoring, followed by side-chain composition, to design heterochiral cyclic peptides that bind to a protein surface of interest. To showcase the applicability of our approach, we identified two peptides (PD-<i>i</i>3 and PD-<i>i</i>6) that target PD-1, a key immune checkpoint, and work as protein ligand decoys. A comprehensive biophysical evaluation confirmed their binding mechanism to PD-1 and their inhibitory effect on the PD-1/PD-L1 interaction. Finally, elucidation of their solution structures by NMR served as validation of our <i>de novo </i>design approach. We anticipate that our results will provide a general framework for designing target-specific drug-like peptides.<i></i></p>

Target-Templated de novo Design of Macrocyclic D-/L-Peptides: Inhibitors of the PD-1/PD-L1 Interaction

2020 ◽  
Author(s):  
Salvador Guardiola ◽  
Monica Varese ◽  
Xavier Roig ◽  
Jesús Garcia ◽  
Ernest Giralt
Keyword(s):  
Protein Interactions ◽  
Cyclic Peptides ◽  
General Framework ◽  
Large Scale ◽  
De Novo ◽  
Inhibitory Effect ◽  
Original Text ◽  
Protein Protein Interactions ◽  
Retraction Notice ◽  
Pharmaceutical Properties

<p>NOTE: This preprint has been retracted by consensus from all authors. See the retraction notice in place above; the original text can be found under "Version 1", accessible from the version selector above.</p><p><br></p><p>------------------------------------------------------------------------</p><p><br></p><p>Peptides, together with antibodies, are among the most potent biochemical tools to modulate challenging protein-protein interactions. However, current structure-based methods are largely limited to natural peptides and are not suitable for designing target-specific binders with improved pharmaceutical properties, such as macrocyclic peptides. Here we report a general framework that leverages the computational power of Rosetta for large-scale backbone sampling and energy scoring, followed by side-chain composition, to design heterochiral cyclic peptides that bind to a protein surface of interest. To showcase the applicability of our approach, we identified two peptides (PD-<i>i</i>3 and PD-<i>i</i>6) that target PD-1, a key immune checkpoint, and work as protein ligand decoys. A comprehensive biophysical evaluation confirmed their binding mechanism to PD-1 and their inhibitory effect on the PD-1/PD-L1 interaction. Finally, elucidation of their solution structures by NMR served as validation of our <i>de novo </i>design approach. We anticipate that our results will provide a general framework for designing target-specific drug-like peptides.<i></i></p>

Model and Method for Contributor’s Quality Assessment in Community Image Tagging Systems

2018 ◽  
pp. 45-51
Author(s):  
A. V. Ponomarev
Keyword(s):  
Large Scale ◽  
Wide Spectrum ◽  
Preference Relation ◽  
Pairwise Comparison ◽  
Ground Truth ◽  
Comparison Method ◽  
Characteristic Matrix ◽  
Image Tagging ◽  
Proposed Model

Introduction: Large-scale human-computer systems involving people of various skills and motivation into the information processing process are currently used in a wide spectrum of applications. An acute problem in such systems is assessing the expected quality of each contributor; for example, in order to penalize incompetent or inaccurate ones and to promote diligent ones.Purpose: To develop a method of assessing the expected contributor’s quality in community tagging systems. This method should only use generally unreliable and incomplete information provided by contributors (with ground truth tags unknown).Results:A mathematical model is proposed for community image tagging (including the model of a contributor), along with a method of assessing the expected contributor’s quality. The method is based on comparing tag sets provided by different contributors for the same images, being a modification of pairwise comparison method with preference relation replaced by a special domination characteristic. Expected contributors’ quality is evaluated as a positive eigenvector of a pairwise domination characteristic matrix. Community tagging simulation has confirmed that the proposed method allows you to adequately estimate the expected quality of community tagging system contributors (provided that the contributors' behavior fits the proposed model).Practical relevance: The obtained results can be used in the development of systems based on coordinated efforts of community (primarily, community tagging systems). 

Sign in / Sign up

Export Citation Format

Share Document