scholarly journals Extended Variational Message Passing for Automated Approximate Bayesian Inference

Entropy ◽  
2021 ◽  
Vol 23 (7) ◽  
pp. 815
Author(s):  
Semih Akbayrak ◽  
Ivan Bocharov ◽  
Bert de Vries
Keyword(s):  
Bayesian Inference ◽  
Probabilistic Model ◽  
Message Passing ◽  
Probabilistic Models ◽  
Hidden Variables ◽  
Generative Models ◽  
Wide Range ◽  
Complex Models ◽  
Algorithmic Framework ◽  
Approximate Bayesian

Variational Message Passing (VMP) provides an automatable and efficient algorithmic framework for approximating Bayesian inference in factorized probabilistic models that consist of conjugate exponential family distributions. The automation of Bayesian inference tasks is very important since many data processing problems can be formulated as inference tasks on a generative probabilistic model. However, accurate generative models may also contain deterministic and possibly nonlinear variable mappings and non-conjugate factor pairs that complicate the automatic execution of the VMP algorithm. In this paper, we show that executing VMP in complex models relies on the ability to compute the expectations of the statistics of hidden variables. We extend the applicability of VMP by approximating the required expectation quantities in appropriate cases by importance sampling and Laplace approximation. As a result, the proposed Extended VMP (EVMP) approach supports automated efficient inference for a very wide range of probabilistic model specifications. We implemented EVMP in the Julia language in the probabilistic programming package ForneyLab.jl and show by a number of examples that EVMP renders an almost universal inference engine for factorized probabilistic models.

scholarly journals Model-based kernel sum rule: kernel Bayesian inference with probabilistic models

2020 ◽  
Vol 109 (5) ◽  
pp. 939-972
Author(s):  
Yu Nishiyama ◽  
Motonobu Kanagawa ◽  
Arthur Gretton ◽  
Kenji Fukumizu
Keyword(s):  
Bayesian Inference ◽  
Probabilistic Model ◽  
Probabilistic Models ◽  
State Transition ◽  
Hybrid Approach ◽  
Transition Process ◽  
The State ◽  
Nonparametric Inference ◽  
Sum Rule ◽  
Model Based

AbstractKernel Bayesian inference is a principled approach to nonparametric inference in probabilistic graphical models, where probabilistic relationships between variables are learned from data in a nonparametric manner. Various algorithms of kernel Bayesian inference have been developed by combining kernelized basic probabilistic operations such as the kernel sum rule and kernel Bayes’ rule. However, the current framework is fully nonparametric, and it does not allow a user to flexibly combine nonparametric and model-based inferences. This is inefficient when there are good probabilistic models (or simulation models) available for some parts of a graphical model; this is in particular true in scientific fields where “models” are the central topic of study. Our contribution in this paper is to introduce a novel approach, termed the model-based kernel sum rule (Mb-KSR), to combine a probabilistic model and kernel Bayesian inference. By combining the Mb-KSR with the existing kernelized probabilistic rules, one can develop various algorithms for hybrid (i.e., nonparametric and model-based) inferences. As an illustrative example, we consider Bayesian filtering in a state space model, where typically there exists an accurate probabilistic model for the state transition process. We propose a novel filtering method that combines model-based inference for the state transition process and data-driven, nonparametric inference for the observation generating process. We empirically validate our approach with synthetic and real-data experiments, the latter being the problem of vision-based mobile robot localization in robotics, which illustrates the effectiveness of the proposed hybrid approach.

scholarly journals Active Inference, Curiosity and Insight

2017 ◽  
Vol 29 (10) ◽  
pp. 2633-2683 ◽  
Author(s):  
Karl J. Friston ◽  
Marco Lin ◽  
Christopher D. Frith ◽  
Giovanni Pezzulo ◽  
J. Allan Hobson ◽  
...  
Keyword(s):  
Bayesian Inference ◽  
Dual Problem ◽  
Structure Learning ◽  
Rule Learning ◽  
Generative Models ◽  
Current Trends ◽  
The World ◽  
Abstract Rule ◽  
Approximate Bayesian ◽  
Approximate Bayesian Inference

This article offers a formal account of curiosity and insight in terms of active (Bayesian) inference. It deals with the dual problem of inferring states of the world and learning its statistical structure. In contrast to current trends in machine learning (e.g., deep learning), we focus on how people attain insight and understanding using just a handful of observations, which are solicited through curious behavior. We use simulations of abstract rule learning and approximate Bayesian inference to show that minimizing (expected) variational free energy leads to active sampling of novel contingencies. This epistemic behavior closes explanatory gaps in generative models of the world, thereby reducing uncertainty and satisfying curiosity. We then move from epistemic learning to model selection or structure learning to show how abductive processes emerge when agents test plausible hypotheses about symmetries (i.e., invariances or rules) in their generative models. The ensuing Bayesian model reduction evinces mechanisms associated with sleep and has all the hallmarks of “aha” moments. This formulation moves toward a computational account of consciousness in the pre-Cartesian sense of sharable knowledge (i.e., con: “together”; scire: “to know”).

Bayesian Fundamentalism or Enlightenment? On the explanatory status and theoretical contributions of Bayesian models of cognition

2011 ◽  
Vol 34 (4) ◽  
pp. 169-188 ◽  
Author(s):  
Matt Jones ◽  
Bradley C. Love
Keyword(s):  
Bayesian Inference ◽  
Probabilistic Models ◽  
Bayesian Models ◽  
Expressive Power ◽  
Psychological Theory ◽  
Rational Approach ◽  
Potential Contribution ◽  
Level Data ◽  
Wide Range ◽  
Neurological Processes

AbstractThe prominence of Bayesian modeling of cognition has increased recently largely because of mathematical advances in specifying and deriving predictions from complex probabilistic models. Much of this research aims to demonstrate that cognitive behavior can be explained from rational principles alone, without recourse to psychological or neurological processes and representations. We note commonalities between this rational approach and other movements in psychology – namely, Behaviorism and evolutionary psychology – that set aside mechanistic explanations or make use of optimality assumptions. Through these comparisons, we identify a number of challenges that limit the rational program's potential contribution to psychological theory. Specifically, rational Bayesian models are significantly unconstrained, both because they are uninformed by a wide range of process-level data and because their assumptions about the environment are generally not grounded in empirical measurement. The psychological implications of most Bayesian models are also unclear. Bayesian inference itself is conceptually trivial, but strong assumptions are often embedded in the hypothesis sets and the approximation algorithms used to derive model predictions, without a clear delineation between psychological commitments and implementational details. Comparing multiple Bayesian models of the same task is rare, as is the realization that many Bayesian models recapitulate existing (mechanistic level) theories. Despite the expressive power of current Bayesian models, we argue they must be developed in conjunction with mechanistic considerations to offer substantive explanations of cognition. We lay out several means for such an integration, which take into account the representations on which Bayesian inference operates, as well as the algorithms and heuristics that carry it out. We argue this unification will better facilitate lasting contributions to psychological theory, avoiding the pitfalls that have plagued previous theoretical movements.

scholarly journals Viterbi training in PRISM

2014 ◽  
Vol 15 (2) ◽  
pp. 147-168 ◽  
Author(s):  
TAISUKE SATO ◽  
KEIICHI KUBOTA
Keyword(s):  
Probabilistic Models ◽  
Hidden Variables ◽  
Specific Model ◽  
Generative Models ◽  
Superior Performance ◽  
Probabilistic Logic ◽  
Good Prediction ◽  
Good Prediction Performance ◽  
Probabilistic Grammars ◽  
Inference Methods

AbstractVT (Viterbi training), or hard expectation maximization (EM), is an efficient way of parameter learning for probabilistic models with hidden variables. Given an observation y, it searches for a state of hidden variables x that maximizes p(x,y | θ) by coordinate ascent on parameters θ and x. In this paper we introduce VT to PRogramming In Statistical Modeling (PRISM), a logic-based probabilistic modeling system for generative models. VT improves PRISM in three ways. First, VT in PRISM converges faster than EM in PRISM due to VT's termination condition. Second, parameters learned by VT often show good prediction performance compared with those learned by EM. We conducted two parsing experiments with probabilistic grammars while learning parameters by a variety of inference methods, i.e. VT, EM, MAP and VB. The result is that VT achieved the best parsing accuracy among them in both experiments. Also, we conducted a similar experiment for classification tasks where a hidden variable is not a prediction target unlike probabilistic grammars. We found that in such a case VT does not necessarily yield superior performance. Third, since VT always deals with a single probability of a single explanation, Viterbi explanation, the exclusiveness condition imposed on PRISM programs is no more required if we learn parameters by VT. Last but not least, we can say that as VT in PRISM is general and applicable to any PRISM program, it largely reduces the need for the user to develop a specific VT algorithm for a specific model. Furthermore, since VT in PRISM can be used just by setting a PRISM flag appropriately, it makes VT easily accessible to (probabilistic) logic programmers.

scholarly journals Scalable probabilistic matrix factorization for single-cell RNA-seq analysis

2018 ◽  
Author(s):  
Pedro F. Ferreira ◽  
Alexandra M. Carvalho ◽  
Susana Vinga
Keyword(s):  
Gene Expression ◽  
Dimensionality Reduction ◽  
Single Cell ◽  
Matrix Factorization ◽  
Latent Variables ◽  
Probabilistic Models ◽  
Negative Binomial ◽  
Hidden Variables ◽  
Generative Models ◽  
Rna Seq

Motivation: The gene expression profile of a cell dictates its function in molecular processes, and can be used to probe its health status. This represents a step forward in the deep characterization of diseases such as cancer and may lead to breakthroughs in their treatment. The technology used to measure the gene expression of isolated cells, single-cell RNA-seq (scRNA-seq), has emerged in the last decade as a key enabler of this progress. However, the use of existing methods for dimensionality reduction, clustering and differential expression is limited by the specificities of the data obtained from scRNA-seq experiments, where technical factors may confound analyses of the true biological signal and contribute to spurious results. To overcome this issue, a possible approach is designing probabilistic generative models of the data with hidden variables encoding different underlying processes. Results: We propose two novel probabilistic models for scRNA-seq data: modified probabilistic count matrix factorization (m-pCMF) and Bayesian zero-inflated negative binomial factorization (ZINBayes). These build upon previous models in the literature while leveraging scalable Bayesian inference via variational methods. We show that the proposed methods are competitive with the state-of-the-art models for robust dimensionality reduction in modern data sets, and improve upon the current best Bayesian model for small numbers of cells. The results show that building probabilistic models of latent variables which encode domain knowledge and using variational inference constitute a promising approach to analyse scRNA-seq data in a scalable way. Availability: m-pCMF and ZINBayes are publicly available as Python packages at https://github.com/pedrofale/, along with the code to reproduce all the results. Contact: [email protected]

scholarly journals Comparing two sequential Monte Carlo samplers for exact and approximate Bayesian inference on biological models

2017 ◽  
Vol 14 (134) ◽  
pp. 20170340 ◽  
Author(s):  
Aidan C. Daly ◽  
Jonathan Cooper ◽  
David J. Gavaghan ◽  
Chris Holmes
Keyword(s):  
Bayesian Inference ◽  
Bayesian Methods ◽  
Sequential Monte Carlo ◽  
Model Parameters ◽  
Biological Models ◽  
Exact Inference ◽  
Modelling Studies ◽  
Approximate Bayesian ◽  
Approximate Bayesian Inference ◽  
Abc Methods

Bayesian methods are advantageous for biological modelling studies due to their ability to quantify and characterize posterior variability in model parameters. When Bayesian methods cannot be applied, due either to non-determinism in the model or limitations on system observability, approximate Bayesian computation (ABC) methods can be used to similar effect, despite producing inflated estimates of the true posterior variance. Owing to generally differing application domains, there are few studies comparing Bayesian and ABC methods, and thus there is little understanding of the properties and magnitude of this uncertainty inflation. To address this problem, we present two popular strategies for ABC sampling that we have adapted to perform exact Bayesian inference, and compare them on several model problems. We find that one sampler was impractical for exact inference due to its sensitivity to a key normalizing constant, and additionally highlight sensitivities of both samplers to various algorithmic parameters and model conditions. We conclude with a study of the O'Hara–Rudy cardiac action potential model to quantify the uncertainty amplification resulting from employing ABC using a set of clinically relevant biomarkers. We hope that this work serves to guide the implementation and comparative assessment of Bayesian and ABC sampling techniques in biological models.

scholarly journals Modelling with star-shaped distributions

2020 ◽  
Vol 8 (1) ◽  
pp. 45-69
Author(s):  
Eckhard Liebscher ◽  
Wolf-Dieter Richter
Keyword(s):  
Probabilistic Models ◽  
Arbitrary Dimension ◽  
Probability Density Functions ◽  
Density Functions ◽  
Wide Range ◽  
Gaussian Density ◽  
Spherical Distributions ◽  
Data Points ◽  
Non Gaussian ◽  
General Method

AbstractWe prove and describe in great detail a general method for constructing a wide range of multivariate probability density functions. We introduce probabilistic models for a large variety of clouds of multivariate data points. In the present paper, the focus is on star-shaped distributions of an arbitrary dimension, where in case of spherical distributions dependence is modeled by a non-Gaussian density generating function.

scholarly journals Syllabification of the Divine Comedy

2021 ◽  
Vol 14 (3) ◽  
pp. 1-26
Author(s):  
Andrea Asperti ◽  
Stefano Dal Bianco
Keyword(s):  
Text Processing ◽  
Solution Space ◽  
Generative Models ◽  
Divine Comedy ◽  
Point Of View ◽  
Computer Assisted ◽  
Wide Range ◽  
The Divine Comedy ◽  
Left And Right ◽  
Assisted Analysis

We provide a syllabification algorithm for the Divine Comedy using techniques from probabilistic and constraint programming. We particularly focus on the synalephe , addressed in terms of the "propensity" of a word to take part in a synalephe with adjacent words. We jointly provide an online vocabulary containing, for each word, information about its syllabification, the location of the tonic accent, and the aforementioned synalephe propensity, on the left and right sides. The algorithm is intrinsically nondeterministic, producing different possible syllabifications for each verse, with different likelihoods; metric constraints relative to accents on the 10th, 4th, and 6th syllables are used to further reduce the solution space. The most likely syllabification is hence returned as output. We believe that this work could be a major milestone for a lot of different investigations. From the point of view of digital humanities it opens new perspectives on computer-assisted analysis of digital sources, comprising automated detection of anomalous and problematic cases, metric clustering of verses and their categorization, or more foundational investigations addressing, e.g., the phonetic roles of consonants and vowels. From the point of view of text processing and deep learning, information about syllabification and the location of accents opens a wide range of exciting perspectives, from the possibility of automatic learning syllabification of words and verses to the improvement of generative models, aware of metric issues, and more respectful of the expected musicality.

Sign in / Sign up

Export Citation Format

Share Document