scholarly journals A Transformer-Based Hierarchical Variational Autoencoder Combined Hidden Markov Model for Long Text Generation

Entropy ◽  
2021 ◽  
Vol 23 (10) ◽  
pp. 1277
Author(s):  
Kun Zhao ◽  
Hongwei Ding ◽  
Kai Ye ◽  
Xiaohui Cui
Keyword(s):  
Markov Model ◽  
Hidden Markov Model ◽  
Latent Variables ◽  
Latent Variable ◽  
Hidden Markov ◽  
Variable Structure ◽  
Text Generation ◽  
Short Text ◽  
Variational Autoencoder ◽  
Generation Network

The Variational AutoEncoder (VAE) has made significant progress in text generation, but it focused on short text (always a sentence). Long texts consist of multiple sentences. There is a particular relationship between each sentence, especially between the latent variables that control the generation of the sentences. The relationships between these latent variables help in generating continuous and logically connected long texts. There exist very few studies on the relationships between these latent variables. We proposed a method for combining the Transformer-Based Hierarchical Variational Autoencoder and Hidden Markov Model (HT-HVAE) to learn multiple hierarchical latent variables and their relationships. This application improves long text generation. We use a hierarchical Transformer encoder to encode the long texts in order to obtain better hierarchical information of the long text. HT-HVAE’s generation network uses HMM to learn the relationship between latent variables. We also proposed a method for calculating the perplexity for the multiple hierarchical latent variable structure. The experimental results show that our model is more effective in the dataset with strong logic, alleviates the notorious posterior collapse problem, and generates more continuous and logically connected long text.

scholarly journals Using Hidden Markov Modeling to Decompose Human-Written Summaries

2002 ◽  
Vol 28 (4) ◽  
pp. 527-543 ◽  
Author(s):  
Hongyan Jing
Keyword(s):  
Markov Model ◽  
Hidden Markov Model ◽  
Hidden Markov ◽  
Model Solution ◽  
Original Text ◽  
Markov Modeling ◽  
Text Generation ◽  
Decomposition Problem ◽  
Hidden Markov Modeling

Professional summarizers often reuse original documents to generate summaries. The task of summary sentence decomposition is to deduce whether a summary sentence is constructed by reusing the original text and to identify reused phrases. Specifically, the decomposition program needs to answer three questions for a given summary sentence: (1) Is this summary sentence constructed by reusing the text in the original document? (2) If so, what phrases in the sentence come from the original document? and (3) From where in the document do the phrases come? Solving the decomposition problem can lead to better text generation techniques for summarization. Decomposition can also provide large training and testing corpora for extraction-based summarizers. We propose a hidden Markov model solution to the decomposition problem. Evaluations show that the proposed algorithm performs well.

scholarly journals A Hidden Markov Model to Address Measurement Errors in Ordinal Response Scale and Non-Decreasing Process

Mathematics ◽  
2020 ◽  
Vol 8 (4) ◽  
pp. 622 ◽  
Author(s):  
Lizbeth Naranjo ◽  
Luz Judith R. Esparza ◽  
Carlos J. Pérez
Keyword(s):  
Markov Model ◽  
Hidden Markov Model ◽  
Latent Variables ◽  
Measurement Errors ◽  
Hidden Markov ◽  
Response Scale ◽  
Ordinal Response ◽  
Continuous State Space ◽  
Continuous State ◽  
Categorical Response

A Bayesian approach was developed, tested, and applied to model ordinal response data in monotone non-decreasing processes with measurement errors. An inhomogeneous hidden Markov model with continuous state-space was considered to incorporate measurement errors in the categorical response at the same time that the non-decreasing patterns were kept. The computational difficulties were avoided by including latent variables that allowed implementing an efficient Markov chain Monte Carlo method. A simulation-based analysis was carried out to validate the approach, whereas the proposed approach was applied to analyze aortic aneurysm progression data.

scholarly journals RaptGen: A variational autoencoder with profile hidden Markov model for generative aptamer discovery

2021 ◽  
Author(s):  
Natsuki Iwano ◽  
Tatsuo Adachi ◽  
Kazuteru Aoki ◽  
Yoshikazu Nakamura ◽  
Michiaki Hamada
Keyword(s):  
Markov Model ◽  
Hidden Markov Model ◽  
High Throughput Sequencing ◽  
Sequence Data ◽  
Hidden Markov ◽  
Bayesian Optimization ◽  
Sequencing Data ◽  
Latent Space ◽  
Variational Autoencoder ◽  
Profile Hidden Markov Model

AbstractNucleic acid aptamers are generated by an in vitro molecular evolution method known as systematic evolution of ligands by exponential enrichment (SELEX). A variety of candidates is limited by actual sequencing data from an experiment. Here, we developed RaptGen, which is a variational autoencoder for in silico aptamer generation. RaptGen exploits a profile hidden Markov model decoder to represent motif sequences effectively. We showed that RaptGen embedded simulation sequence data into low-dimension latent space dependent on motif information. We also performed sequence embedding using two independent SELEX datasets. RaptGen successfully generated aptamers from the latent space even though they were not included in high-throughput sequencing. RaptGen could also generate a truncated aptamer with a short learning model. We demonstrated that RaptGen could be applied to activity-guided aptamer generation according to Bayesian optimization. We concluded that a generative method by RaptGen and latent representation are useful for aptamer discovery. Codes are available at https://github.com/hmdlab/raptgen.

Hidden Markov Model Variational Autoencoder for Acoustic Unit Discovery

2017 ◽  
Author(s):  
Janek Ebbers ◽  
Jahn Heymann ◽  
Lukas Drude ◽  
Thomas Glarner ◽  
Reinhold Haeb-Umbach ◽  
...  
Keyword(s):  
Markov Model ◽  
Hidden Markov Model ◽  
Hidden Markov ◽  
Variational Autoencoder

Full Bayesian Hidden Markov Model Variational Autoencoder for Acoustic Unit Discovery

2018 ◽  
Author(s):  
Thomas Glarner ◽  
Patrick Hanebrink ◽  
Janek Ebbers ◽  
Reinhold Haeb-Umbach
Keyword(s):  
Markov Model ◽  
Hidden Markov Model ◽  
Hidden Markov ◽  
Variational Autoencoder

scholarly journals Accident Prediction System Based on Hidden Markov Model for Vehicular Ad-Hoc Network in Urban Environments

Information ◽  
2018 ◽  
Vol 9 (12) ◽  
pp. 311 ◽  
Author(s):  
Nyothiri Aung ◽  
Weidong Zhang ◽  
Sahraoui Dhelim ◽  
Yibo Ai
Keyword(s):  
Markov Model ◽  
Hidden Markov Model ◽  
Risk Prediction ◽  
Latent Variable ◽  
Ad Hoc ◽  
Hidden Markov ◽  
Urban Environments ◽  
Prediction System ◽  
Accident Prediction ◽  
Driving Risk

With the emergence of autonomous vehicles and internet of vehicles (IoV), future roads of smart cities will have a combination of autonomous and automated vehicles with regular vehicles that require human operators. To ensure the safety of the road commuters in such a network, it is imperative to enhance the performance of Advanced Driver Assistance Systems (ADAS). Real-time driving risk prediction is a fundamental part of an ADAS. Many driving risk prediction systems have been proposed. However, most of them are based only on vehicle’s velocity. But in most of the accident scenarios, other factors are also involved, such as weather conditions or driver fatigue. In this paper, we proposed an accident prediction system for Vehicular ad hoc networks (VANETs) in urban environments, in which we considered the crash risk as a latent variable that can be observed using multi-observation such as velocity, weather condition, risk location, nearby vehicles density and driver fatigue. A Hidden Markov Model (HMM) was used to model the correlation between these observations and the latent variable. Simulation results showed that the proposed system has a better performance in terms of sensitivity and precision compared to state of the art single factor schemes.

scholarly journals A Method of Combining Hidden Markov Model and Convolutional Neural Network for the 5G RCS Message Filtering

2021 ◽  
Vol 11 (14) ◽  
pp. 6350
Author(s):  
Bibu Gao ◽  
Wenqiang Zhang
Keyword(s):  
Neural Network ◽  
Feature Extraction ◽  
Markov Model ◽  
Hidden Markov Model ◽  
Convolutional Neural Network ◽  
Hidden Markov ◽  
Short Text ◽  
Feature Extraction Algorithm ◽  
Extraction Algorithm ◽  
Message Service

As one of the 5G applications, rich communication suite (RCS), known as the next generation of Short Message Service (SMS), contains multimedia and interactive information for a better user experience. Meanwhile, the RCS industry worries that spammers may migrate their spamming misdeeds to RCS messages, the complexity of which challenges the filtering technology because each of them contains hundreds of fields with various types of data, such as texts, images and videos. Among the data, the hundreds of fields of text data contain the main content, which is adequate and more efficient for combating spam. This paper first discusses the text fields, which possibly contain spam information, then use the hidden Markov model (HMM) to weight the fields and finally use convolutional neural network (CNN) to classify the RCS messages. In the HMM step, the text fields are treated differently. The short texts of these fields are represented as feature weight sequences extracted by a feature extraction algorithm based on a probability density function. Then, the proposed HMM learns the weight sequence and produces a proper weight for each short text. Other text fields with fewer words are also weighted by the feature extraction algorithm. In the CNN step, all these feature weights first construct the RCS message matrix. The matrices of the training RCS messages are used as the CNN model inputs for learning and the matrices of testing messages are used as the trained CNN model inputs for RCS message property prediction. Four optimization technologies are introduced into the CNN classification process. Promising experiment results are achieved on the real industrial data.

Analysis of demand in steel and iron industry – latent variables model

2020 ◽  
Vol 66 (4) ◽  
pp. 247-269
Author(s):  
Magdalena Barska
Keyword(s):  
Markov Model ◽  
Hidden Markov Model ◽  
Latent Variables ◽  
Hidden Markov ◽  
Second Order ◽  
Model Parameters ◽  
Iron Industry ◽  
Economic Cycles ◽  
Multiple Factors ◽  
Analysis Of Time Series

Demand in the steel and iron industry is influenced by multiple factors. Not all of them can be identified and measured. The paper presents the results of the analysis of the levels of demand achieved by a selected enterprise from this sector in the years 2010–2014. The aim of the study is to build a hidden Markov model which would reflect the turning points of this demand, thus making it possible to forecast its future levels. The model’s forecasting properties and stability have been examined. A simulation has been carried out that involved generating a high number of series for selected model parameters and checking their properties. This demonstrated that a three-state second order hidden Markov model was most relevant to the purpose of the study. Thanks to the model’s application, it was possible to describe states which could potentially shape the demand. Moreover, taking the transition state into consideration allowed spotting the signal about the upcoming replacement of the growth phase with the decline phase, and vice versa. The presented second order hidden Markov model can serve as an alternative to the traditional methods of the analysis of time series. The forecast generated by the model informs about the shaping of a trend in demand and serves as an indication of the shifts in economic cycles.

Sign in / Sign up

Export Citation Format

Share Document