scholarly journals Entity-Centric Fully Connected GCN for Relation Classification

2021 ◽  
Vol 11 (4) ◽  
pp. 1377
Author(s):  
Jun Long ◽  
Ye Wang ◽  
Xiangxiang Wei ◽  
Zhen Ding ◽  
Qianqian Qi ◽  
...  
Keyword(s):  
Language Processing ◽  
Knowledge Graph ◽  
Data Sets ◽  
Semantic Features ◽  
Convolutional Network ◽  
Aggregate Information ◽  
Dependency Tree ◽  
Relation Classification ◽  
The Cost ◽  
Fully Connected

Relation classification is an important task in the field of natural language processing, and it is one of the important steps in constructing a knowledge graph, which can greatly reduce the cost of constructing a knowledge graph. The Graph Convolutional Network (GCN) is an effective model for accurate relation classification, which models the dependency tree of textual instances to extract the semantic features of relation mentions. Previous GCN based methods treat each node equally. However, the contribution of different words to express a certain relation is different, especially the entity mentions in the sentence. In this paper, a novel GCN based relation classifier is propose, which treats the entity nodes as two global nodes in the dependency tree. These two global nodes directly connect with other nodes, which can aggregate information from the whole tree with only one convolutional layer. In this way, the method can not only simplify the complexity of the model, but also generate expressive relation representation. Experimental results on two widely used data sets, SemEval-2010 Task 8 and TACRED, show that our model outperforms all the compared baselines in this paper, which illustrates that the model can effectively utilize the dependencies between nodes and improve the performance of relation classification.

scholarly journals Low-Rank Deep Convolutional Neural Network for Multitask Learning

2019 ◽  
Vol 2019 ◽  
pp. 1-10
Author(s):  
Fang Su ◽  
Hai-Yang Shang ◽  
Jing-Yan Wang
Keyword(s):  
Language Processing ◽  
Back Propagation ◽  
Multitask Learning ◽  
Low Rank ◽  
Learning Problem ◽  
Convolutional Network ◽  
Deep Network ◽  
Benchmark Datasets ◽  
Deep Cnn ◽  
Fully Connected

In this paper, we propose a novel multitask learning method based on the deep convolutional network. The proposed deep network has four convolutional layers, three max-pooling layers, and two parallel fully connected layers. To adjust the deep network to multitask learning problem, we propose to learn a low-rank deep network so that the relation among different tasks can be explored. We proposed to minimize the number of independent parameter rows of one fully connected layer to explore the relations among different tasks, which is measured by the nuclear norm of the parameter of one fully connected layer, and seek a low-rank parameter matrix. Meanwhile, we also propose to regularize another fully connected layer by sparsity penalty so that the useful features learned by the lower layers can be selected. The learning problem is solved by an iterative algorithm based on gradient descent and back-propagation algorithms. The proposed algorithm is evaluated over benchmark datasets of multiple face attribute prediction, multitask natural language processing, and joint economics index predictions. The evaluation results show the advantage of the low-rank deep CNN model over multitask problems.

scholarly journals Sentence Compression Using BERT and Graph Convolutional Networks

2021 ◽  
Vol 11 (21) ◽  
pp. 9910
Author(s):  
Yo-Han Park ◽  
Gyong-Ho Lee ◽  
Yong-Seok Choi ◽  
Kong-Joo Lee
Keyword(s):  
Language Processing ◽  
Large Scale ◽  
Convolutional Network ◽  
Convolutional Networks ◽  
Compression Model ◽  
Sentence Compression ◽  
Dependency Tree ◽  
Proposed Model ◽  
Syntactic Information ◽  
Input Sentence

Sentence compression is a natural language-processing task that produces a short paraphrase of an input sentence by deleting words from the input sentence while ensuring grammatical correctness and preserving meaningful core information. This study introduces a graph convolutional network (GCN) into a sentence compression task to encode syntactic information, such as dependency trees. As we upgrade the GCN to activate a directed edge, the compression model with the GCN layers can distinguish between parent and child nodes in a dependency tree when aggregating adjacent nodes. Furthermore, by increasing the number of GCN layers, the model can gradually collect high-order information of a dependency tree when propagating node information through the layers. We implement a sentence compression model for Korean and English, respectively. This model consists of three components: pre-trained BERT model, GCN layers, and a scoring layer. The scoring layer can determine whether a word should remain in a compressed sentence by relying on the word vector containing contextual and syntactic information encoded by BERT and GCN layers. To train and evaluate the proposed model, we used the Google sentence compression dataset for English and a Korean sentence compression corpus containing about 140,000 sentence pairs for Korean. The experimental results demonstrate that the proposed model achieves state-of-the-art performance for English. To the best of our knowledge, this sentence compression model based on the deep learning model trained with a large-scale corpus is the first attempt for Korean.

scholarly journals An Approach to Knowledge Base Completion by a Committee-Based Knowledge Graph Embedding

2020 ◽  
Vol 10 (8) ◽  
pp. 2651
Author(s):  
Su Jeong Choi ◽  
Hyun-Je Song ◽  
Seong-Bae Park
Keyword(s):  
Knowledge Base ◽  
Language Processing ◽  
Graph Embedding ◽  
Knowledge Bases ◽  
Knowledge Graph ◽  
Data Sets ◽  
Complete Knowledge ◽  
Proposed Model ◽  
Ranking Task ◽  
Low Dimensional

Knowledge bases such as Freebase, YAGO, DBPedia, and Nell contain a number of facts with various entities and relations. Since they store many facts, they are regarded as core resources for many natural language processing tasks. Nevertheless, they are not normally complete and have many missing facts. Such missing facts keep them from being used in diverse applications in spite of their usefulness. Therefore, it is significant to complete knowledge bases. Knowledge graph embedding is one of the promising approaches to completing a knowledge base and thus many variants of knowledge graph embedding have been proposed. It maps all entities and relations in knowledge base onto a low dimensional vector space. Then, candidate facts that are plausible in the space are determined as missing facts. However, any single knowledge graph embedding is insufficient to complete a knowledge base. As a solution to this problem, this paper defines knowledge base completion as a ranking task and proposes a committee-based knowledge graph embedding model for improving the performance of knowledge base completion. Since each knowledge graph embedding has its own idiosyncrasy, we make up a committee of various knowledge graph embeddings to reflect various perspectives. After ranking all candidate facts according to their plausibility computed by the committee, the top-k facts are chosen as missing facts. Our experimental results on two data sets show that the proposed model achieves higher performance than any single knowledge graph embedding and shows robust performances regardless of k. These results prove that the proposed model considers various perspectives in measuring the plausibility of candidate facts.

End-to-end Relation-Enhanced Learnable Graph Self-attention Network for Knowledge Graphs Embedding

2021 ◽  
Author(s):  
Shengchen Jiang ◽  
Hongbin Wang ◽  
Xiang Hou
Keyword(s):  
Large Scale ◽  
Structural Characteristics ◽  
Graph Embedding ◽  
Knowledge Graph ◽  
Data Sets ◽  
Relevance Ranking ◽  
Convolutional Network ◽  
Attention Network ◽  
Knowledge Graphs ◽  
End To End

Abstract The existing methods ignore the adverse effect of knowledge graph incompleteness on knowledge graph embedding. In addition, the complexity and large-scale of knowledge information hinder knowledge graph embedding performance of the classic graph convolutional network. In this paper, we analyzed the structural characteristics of knowledge graph and the imbalance of knowledge information. Complex knowledge information requires that the model should have better learnability, rather than linearly weighted qualitative constraints, so the method of end-to-end relation-enhanced learnable graph self-attention network for knowledge graphs embedding is proposed. Firstly, we construct the relation-enhanced adjacency matrix to consider the incompleteness of the knowledge graph. Secondly, the graph self-attention network is employed to obtain the global encoding and relevance ranking of entity node information. Thirdly, we propose the concept of convolutional knowledge subgraph, it is constructed according to the entity relevance ranking. Finally, we improve the training effect of the convKB model by changing the construction of negative samples to obtain a better reliability score in the decoder. The experimental results based on the data sets FB15k-237 and WN18RR show that the proposed method facilitates more comprehensive representation of knowledge information than the existing methods, in terms of Hits@10 and MRR.

scholarly journals Relation Extraction with Convolutional Network over Learnable Syntax-Transport Graph

2020 ◽  
Vol 34 (05) ◽  
pp. 8928-8935
Author(s):  
Kai Sun ◽  
Richong Zhang ◽  
Yongyi Mao ◽  
Samuel Mensah ◽  
Xudong Liu
Keyword(s):  
Relation Extraction ◽  
Weighted Graph ◽  
Relevant Information ◽  
Irrelevant Information ◽  
Classification Task ◽  
Convolutional Network ◽  
Structure Information ◽  
Convolutional Networks ◽  
Dependency Tree ◽  
Relation Classification

A large majority of approaches have been proposed to leverage the dependency tree in the relation classification task. Recent works have focused on pruning irrelevant information from the dependency tree. The state-of-the-art Attention Guided Graph Convolutional Networks (AGGCNs) transforms the dependency tree into a weighted-graph to distinguish the relevance of nodes and edges for relation classification. However, in their approach, the graph is fully connected, which destroys the structure information of the original dependency tree. How to effectively make use of relevant information while ignoring irrelevant information from the dependency trees remains a challenge in the relation classification task. In this work, we learn to transform the dependency tree into a weighted graph by considering the syntax dependencies of the connected nodes and persisting the structure of the original dependency tree. We refer to this graph as a syntax-transport graph. We further propose a learnable syntax-transport attention graph convolutional network (LST-AGCN) which operates on the syntax-transport graph directly to distill the final representation which is sufficient for classification. Experiments on Semeval-2010 Task 8 and Tacred show our approach outperforms previous methods.

scholarly journals Exploiting Syntactic and Semantic Information for Textual Similarity Estimation

2021 ◽  
Vol 2021 ◽  
pp. 1-12
Author(s):  
Jiajia Luo ◽  
Hongtao Shan ◽  
Gaoyu Zhang ◽  
George Yuan ◽  
Shuyi Zhang ◽  
...  
Keyword(s):  
Language Processing ◽  
Semantic Information ◽  
Similarity Judgment ◽  
Weight Vector ◽  
Semantic Features ◽  
Similarity Estimation ◽  
Dependency Tree ◽  
Structure Tree ◽  
New Type ◽  
The Common

The textual similarity task, which measures the similarity between two text pieces, has recently received much attention in the natural language processing (NLP) domain. However, due to the vagueness and diversity of language expression, only considering semantic or syntactic features, respectively, may cause the loss of critical textual knowledge. This paper proposes a new type of structure tree for sentence representation, which exploits both syntactic (structural) and semantic information known as the weight vector dependency tree (WVD-tree). WVD-tree comprises structure trees with syntactic information along with word vectors representing semantic information of the sentences. Further, Gaussian attention weight is proposed for better capturing important semantic features of sentences. Meanwhile, we design an enhanced tree kernel to calculate the common parts between two structures for similarity judgment. Finally, WVD-tree is tested on widely used semantic textual similarity tasks. The experimental results prove that WVD-tree can effectively improve the accuracy of sentence similarity judgments.

scholarly journals Research on Automatic Question Answering of Generative Knowledge Graph Based on Pointer Network

Information ◽  
2021 ◽  
Vol 12 (3) ◽  
pp. 136
Author(s):  
Shuang Liu ◽  
Nannan Tan ◽  
Yaqian Ge ◽  
Niko Lukač
Keyword(s):  
Knowledge Base ◽  
Word Frequency ◽  
Language Processing ◽  
Question Answering ◽  
Language Model ◽  
Word List ◽  
Superior Performance ◽  
Knowledge Graph ◽  
Semantic Features ◽  
Question Answering Systems

Question-answering systems based on knowledge graphs are extremely challenging tasks in the field of natural language processing. Most of the existing Chinese Knowledge Base Question Answering(KBQA) can only return the knowledge stored in the knowledge base by extractive methods. Nevertheless, this processing does not conform to the reading habits and cannot solve the Out-of-vocabulary(OOV) problem. In this paper, a new generative question answering method based on knowledge graph is proposed, including three parts of knowledge vocabulary construction, data pre-processing, and answer generation. In the word list construction, BiLSTM-CRF is used to identify the entity in the source text, finding the triples contained in the entity, counting the word frequency, and constructing it. In the part of data pre-processing, a pre-trained language model BERT combining word frequency semantic features is adopted to obtain word vectors. In the answer generation part, one combination of a vocabulary constructed by the knowledge graph and a pointer generator network(PGN) is proposed to point to the corresponding entity for generating answer. The experimental results show that the proposed method can achieve superior performance on WebQA datasets than other methods.

scholarly journals Relation Classification for Bleeding Events from Electronic Health Records: Exploration of Deep Learning Systems (Preprint)

2021 ◽  
Author(s):  
Avijit Mitra ◽  
Bhanu Pratap Singh Rawat ◽  
David D McManus ◽  
Hong Yu
Keyword(s):  
Deep Learning ◽  
Electronic Health Records ◽  
Language Processing ◽  
Bleeding Event ◽  
Learning Systems ◽  
Bleeding Events ◽  
Convolutional Network ◽  
Health Records ◽  
Electronic Health ◽  
Relation Classification

BACKGROUND Accurate detection of bleeding events from electronic health records (EHR) is crucial for identifying and characterizing different common and serious medical problems. To extract such information from EHRs, it is essential to identify the relations between bleeding events and related clinical entities (e.g., bleeding anatomic sites, lab tests). With the advent of natural language processing (NLP) and deep learning (DL) based techniques, many studies have focused on their applicability for various clinical applications. However, there has been no prior work that utilized deep learning to extract relations between bleeding events and relevant entities. OBJECTIVE In this study, we aim to evaluate multiple deep learning systems on a novel EHR dataset for bleeding event related relation classification. METHODS We first expert-annotated a new dataset of 1283 de-identified EHR notes for bleeding events and their attributes. On this dataset, we evaluated three state-of-the-art deep learning architectures, namely, convolutional neural network (CNN), graph convolutional network with attention (AGGCN) and BERT-based models (BioBERT, Bio+Clinical BERT and EhrBERT) for bleeding event relation classification task. RESULTS Our experiments show that the BERT-based models significantly outperformed CNN and AGGCN. Specifically, BioBERT achieved a macro F1 score of 0.842, outperforming both AGGCN (macro F1 score, 0.828) and CNN (macro F1 score, 0.763) by 1.4% (P<.001) and 7.9% (P<.001) respectively. CONCLUSIONS In this comprehensive study, we explored and compared different DL systems to classify relations between bleeding events and other medical concepts. On our corpus, BERT-based models outperformed other deep learning models for identifying the relations of bleeding related entities. BERT-based models were benefited from their pre-trained contextualized word representation and the use of target entity representation over traditional sequence representation.

scholarly journals Shall I Work with Them? A Knowledge Graph-Based Approach for Predicting Future Research Collaborations

Entropy ◽  
2021 ◽  
Vol 23 (6) ◽  
pp. 664
Author(s):  
Nikos Kanakaris ◽  
Nikolaos Giarelis ◽  
Ilias Siachos ◽  
Nikos Karacapilidis
Keyword(s):  
Language Processing ◽  
Scientific Knowledge ◽  
Link Prediction ◽  
Performance Metrics ◽  
Future Research ◽  
Knowledge Graph ◽  
Prediction Problem ◽  
Textual Information ◽  
Research Collaborations ◽  
Processing Techniques

We consider the prediction of future research collaborations as a link prediction problem applied on a scientific knowledge graph. To the best of our knowledge, this is the first work on the prediction of future research collaborations that combines structural and textual information of a scientific knowledge graph through a purposeful integration of graph algorithms and natural language processing techniques. Our work: (i) investigates whether the integration of unstructured textual data into a single knowledge graph affects the performance of a link prediction model, (ii) studies the effect of previously proposed graph kernels based approaches on the performance of an ML model, as far as the link prediction problem is concerned, and (iii) proposes a three-phase pipeline that enables the exploitation of structural and textual information, as well as of pre-trained word embeddings. We benchmark the proposed approach against classical link prediction algorithms using accuracy, recall, and precision as our performance metrics. Finally, we empirically test our approach through various feature combinations with respect to the link prediction problem. Our experimentations with the new COVID-19 Open Research Dataset demonstrate a significant improvement of the abovementioned performance metrics in the prediction of future research collaborations.

Learning emotional word embeddings for sentiment analysis

2021 ◽  
pp. 1-13
Author(s):  
Qingtian Zeng ◽  
Xishi Zhao ◽  
Xiaohui Hu ◽  
Hua Duan ◽  
Zhongying Zhao ◽  
...  
Keyword(s):  
Sentiment Analysis ◽  
Language Processing ◽  
State Of The Art ◽  
Research Problem ◽  
Emotional Word ◽  
Classification Model ◽  
Data Sets ◽  
Word Embeddings ◽  
Real World Data ◽  
Text Documents

Word embeddings have been successfully applied in many natural language processing tasks due to its their effectiveness. However, the state-of-the-art algorithms for learning word representations from large amounts of text documents ignore emotional information, which is a significant research problem that must be addressed. To solve the above problem, we propose an emotional word embedding (EWE) model for sentiment analysis in this paper. This method first applies pre-trained word vectors to represent document features using two different linear weighting methods. Then, the resulting document vectors are input to a classification model and used to train a text sentiment classifier, which is based on a neural network. In this way, the emotional polarity of the text is propagated into the word vectors. The experimental results on three kinds of real-world data sets demonstrate that the proposed EWE model achieves superior performances on text sentiment prediction, text similarity calculation, and word emotional expression tasks compared to other state-of-the-art models.

Sign in / Sign up

Export Citation Format

Share Document