scholarly journals Belief propagation for networks with loops

2021 ◽  
Vol 7 (17) ◽  
pp. eabf1211
Author(s):  
Alec Kirkley ◽  
George T. Cantwell ◽  
M. E. J. Newman
Keyword(s):  
Graphical Models ◽  
Message Passing ◽  
Probabilistic Models ◽  
Belief Propagation ◽  
Probability Distributions ◽  
Fast Calculation ◽  
Potential Applications ◽  
The Common ◽  
Bayesian Graphical Models ◽  
Standing Problem

Belief propagation is a widely used message passing method for the solution of probabilistic models on networks such as epidemic models, spin models, and Bayesian graphical models, but it suffers from the serious shortcoming that it works poorly in the common case of networks that contain short loops. Here, we provide a solution to this long-standing problem, deriving a belief propagation method that allows for fast calculation of probability distributions in systems with short loops, potentially with high density, as well as giving expressions for the entropy and partition function, which are notoriously difficult quantities to compute. Using the Ising model as an example, we show that our approach gives excellent results on both real and synthetic networks, improving substantially on standard message passing methods. We also discuss potential applications of our method to a variety of other problems.

scholarly journals A Spectral Approach to Analysing Belief Propagation for 3-Colouring

2009 ◽  
Vol 18 (6) ◽  
pp. 881-912 ◽  
Author(s):  
AMIN COJA-OGHLAN ◽  
ELCHANAN MOSSEL ◽  
DAN VILENCHIK
Keyword(s):  
Graphical Models ◽  
Message Passing ◽  
Graphical Model ◽  
Belief Propagation ◽  
Graph Colouring ◽  
Rigorous Result ◽  
Theoretical Understanding ◽  
Message Passing Algorithm ◽  
Survey Propagation ◽  
Graphical Structure

Belief propagation (BP) is a message-passing algorithm that computes the exact marginal distributions at every vertex of a graphical model without cycles. While BP is designed to work correctly on trees, it is routinely applied to general graphical models that may contain cycles, in which case neither convergence, nor correctness in the case of convergence is guaranteed. Nonetheless, BP has gained popularity as it seems to remain effective in many cases of interest, even when the underlying graph is ‘far’ from being a tree. However, the theoretical understanding of BP (and its new relative survey propagation) when applied to CSPs is poor.Contributing to the rigorous understanding of BP, in this paper we relate the convergence of BP to spectral properties of the graph. This encompasses a result for random graphs with a ‘planted’ solution; thus, we obtain the first rigorous result on BP for graph colouring in the case of a complex graphical structure (as opposed to trees). In particular, the analysis shows how belief propagation breaks the symmetry between the 3! possible permutations of the colour classes.

scholarly journals Linear Response Algorithms for Approximate Inference in Graphical Models

2004 ◽  
Vol 16 (1) ◽  
pp. 197-221 ◽  
Author(s):  
Max Welling ◽  
Yee Whye Teh
Keyword(s):  
Graphical Models ◽  
Random Fields ◽  
Linear Response ◽  
Belief Propagation ◽  
Probability Distributions ◽  
Gaussian Random Fields ◽  
Marginal Probability ◽  
Joint Distributions ◽  
Propagation Algorithm ◽  
New Algorithms

Belief propagation (BP) on cyclic graphs is an efficient algorithm for computing approximate marginal probability distributions over single nodes and neighboring nodes in the graph. However, it does not prescribe a way to compute joint distributions over pairs of distant nodes in the graph. In this article, we propose two new algorithms for approximating these pairwise probabilities, based on the linear response theorem. The first is a propagation algorithm that is shown to converge if BP converges to a stable fixed point. The second algorithm is based on matrix inversion. Applying these ideas to gaussian random fields, we derive a propagation algorithm for computing the inverse of a matrix.

scholarly journals Cutset Bayesian Networks: A New Representation for Learning Rao-Blackwellised Graphical Models

2019 ◽  
Author(s):  
Tahrima Rahman ◽  
Shasha Jin ◽  
Vibhav Gogate
Keyword(s):  
Bayesian Networks ◽  
Graphical Models ◽  
Prediction Accuracy ◽  
Probabilistic Models ◽  
Probability Distributions ◽  
Main Idea ◽  
Exact Inference ◽  
Trade Off ◽  
Probability Estimates ◽  
Prediction Time

Recently there has been growing interest in learning probabilistic models that admit poly-time inference called tractable probabilistic models from data. Although they generalize poorly as compared to intractable models, they often yield more accurate estimates at prediction time. In this paper, we seek to further explore this trade-off between generalization performance and inference accuracy by proposing a novel, partially tractable representation called cutset Bayesian networks (CBNs). The main idea in CBNs is to partition the variables into two subsets X and Y, learn a (intractable) Bayesian network that represents P(X) and a tractable conditional model that represents P(Y|X). The hope is that the intractable model will help improve generalization while the tractable model, by leveraging Rao-Blackwellised sampling which combines exact inference and sampling, will help improve the prediction accuracy. To compactly model P(Y|X), we introduce a novel tractable representation called conditional cutset networks (CCNs) in which all conditional probability distributions are represented using calibrated classifiers—classifiers which typically yield higher quality probability estimates than conventional classifiers. We show via a rigorous experimental evaluation that CBNs and CCNs yield more accurate posterior estimates than their tractable as well as intractable counterparts.

scholarly journals Learning General Latent-Variable Graphical Models with Predictive Belief Propagation

2020 ◽  
Vol 34 (04) ◽  
pp. 6118-6126
Author(s):  
Borui Wang ◽  
Geoffrey Gordon
Keyword(s):  
Em Algorithm ◽  
Graphical Models ◽  
Message Passing ◽  
Latent Variable ◽  
Belief Propagation ◽  
Learning Problems ◽  
Local Optima ◽  
Learning Framework ◽  
The Em Algorithm ◽  
Spectral Algorithm

Learning general latent-variable probabilistic graphical models is a key theoretical challenge in machine learning and artificial intelligence. All previous methods, including the EM algorithm and the spectral algorithms, face severe limitations that largely restrict their applicability and affect their performance. In order to overcome these limitations, in this paper we introduce a novel formulation of message-passing inference over junction trees named predictive belief propagation, and propose a new learning and inference algorithm for general latent-variable graphical models based on this formulation. Our proposed algorithm reduces the hard parameter learning problem into a sequence of supervised learning problems, and unifies the learning of different kinds of latent graphical models into a single learning framework, which is local-optima-free and statistically consistent. We then give a proof of the correctness of our algorithm and show in experiments on both synthetic and real datasets that our algorithm significantly outperforms both the EM algorithm and the spectral algorithm while also being orders of magnitude faster to compute.

An Accelerated Cue Combination Principle Accounts for Multi-cue Depth Perception

2020 ◽  
Vol 3 (1) ◽  
pp. 10501-1-10501-9
Author(s):  
Christopher W. Tyler
Keyword(s):  
Belief Propagation ◽  
Probability Distributions ◽  
Three Dimensional ◽  
Dimensional Structure ◽  
Fusion Model ◽  
Cue Combination ◽  
Depth Estimate ◽  
Appropriate Behavior ◽  
Depth Cues ◽  
Bayesian Averaging

Abstract For the visual world in which we operate, the core issue is to conceptualize how its three-dimensional structure is encoded through the neural computation of multiple depth cues and their integration to a unitary depth structure. One approach to this issue is the full Bayesian model of scene understanding, but this is shown to require selection from the implausibly large number of possible scenes. An alternative approach is to propagate the implied depth structure solution for the scene through the “belief propagation” algorithm on general probability distributions. However, a more efficient model of local slant propagation is developed as an alternative.The overall depth percept must be derived from the combination of all available depth cues, but a simple linear summation rule across, say, a dozen different depth cues, would massively overestimate the perceived depth in the scene in cases where each cue alone provides a close-to-veridical depth estimate. On the other hand, a Bayesian averaging or “modified weak fusion” model for depth cue combination does not provide for the observed enhancement of perceived depth from weak depth cues. Thus, the current models do not account for the empirical properties of perceived depth from multiple depth cues.The present analysis shows that these problems can be addressed by an asymptotic, or hyperbolic Minkowski, approach to cue combination. With appropriate parameters, this first-order rule gives strong summation for a few depth cues, but the effect of an increasing number of cues beyond that remains too weak to account for the available degree of perceived depth magnitude. Finally, an accelerated asymptotic rule is proposed to match the empirical strength of perceived depth as measured, with appropriate behavior for any number of depth cues.

scholarly journals Bayesian graphical models for modern biological applications

2021 ◽  
Author(s):  
Yang Ni ◽  
Veerabhadran Baladandayuthapani ◽  
Marina Vannucci ◽  
Francesco C. Stingo
Keyword(s):  
Graphical Models ◽  
Cancer Genomics ◽  
Graph Structure ◽  
Biological Processes ◽  
Limited Sample ◽  
Large Networks ◽  
Bayesian Approaches ◽  
Complex Sampling ◽  
Complex Dependence ◽  
Bayesian Graphical Models

AbstractGraphical models are powerful tools that are regularly used to investigate complex dependence structures in high-throughput biomedical datasets. They allow for holistic, systems-level view of the various biological processes, for intuitive and rigorous understanding and interpretations. In the context of large networks, Bayesian approaches are particularly suitable because it encourages sparsity of the graphs, incorporate prior information, and most importantly account for uncertainty in the graph structure. These features are particularly important in applications with limited sample size, including genomics and imaging studies. In this paper, we review several recently developed techniques for the analysis of large networks under non-standard settings, including but not limited to, multiple graphs for data observed from multiple related subgroups, graphical regression approaches used for the analysis of networks that change with covariates, and other complex sampling and structural settings. We also illustrate the practical utility of some of these methods using examples in cancer genomics and neuroimaging.

scholarly journals Cooperative Localization for Mobile Networks: A Distributed Belief Propagation–Mean Field Message Passing Algorithm

2016 ◽  
Vol 23 (6) ◽  
pp. 828-832 ◽  
Author(s):  
Burak Cakmak ◽  
Daniel N. Urup ◽  
Florian Meyer ◽  
Troels Pedersen ◽  
Bernard H. Fleury ◽  
...  
Keyword(s):  
Mobile Networks ◽  
Message Passing ◽  
Belief Propagation ◽  
Mean Field ◽  
Cooperative Localization ◽  
Message Passing Algorithm

scholarly journals Beyond Trees: Analysis and Convergence of Belief Propagation in Graphs with Multiple Cycles

2020 ◽  
Vol 34 (05) ◽  
pp. 7333-7340
Author(s):  
Roie Zivan ◽  
Omer Lev ◽  
Rotem Galiki
Keyword(s):  
Graphical Models ◽  
Belief Propagation ◽  
Optimization Problems ◽  
Optimal Solution ◽  
Damping Factor ◽  
Constraint Optimization ◽  
Single Cycle ◽  
Distributed Constraint Optimization ◽  
Multiple Cycles ◽  
Constraint Optimization Problems

Belief propagation, an algorithm for solving problems represented by graphical models, has long been known to converge to the optimal solution when the graph is a tree. When the graph representing the problem includes a single cycle, the algorithm either converges to the optimal solution or performs periodic oscillations. While the conditions that trigger these two behaviors have been established, the question regarding the convergence and divergence of the algorithm on graphs that include more than one cycle is still open.Focusing on Max-sum, the version of belief propagation for solving distributed constraint optimization problems (DCOPs), we extend the theory on the behavior of belief propagation in general – and Max-sum specifically – when solving problems represented by graphs with multiple cycles. This includes: 1) Generalizing the results obtained for graphs with a single cycle to graphs with multiple cycles, by using backtrack cost trees (BCT). 2) Proving that when the algorithm is applied to adjacent symmetric cycles, the use of a large enough damping factor guarantees convergence to the optimal solution.

scholarly journals Constraint Satisfaction by Survey Propagation

2005 ◽  
Author(s):  
Alfredo Braunstein ◽  
Marc Mézard
Keyword(s):  
Constraint Satisfaction ◽  
Message Passing ◽  
Statistical Physics ◽  
Belief Propagation ◽  
Solution Space ◽  
Error Correcting Codes ◽  
Constraint Satisfaction Problems ◽  
Probabilistic Methods ◽  
Survey Propagation ◽  
Algorithmic Strategy

Methods and analyses from statistical physics are of use not only in studying the performance of algorithms, but also in developing efficient algorithms. Here, we consider survey propagation (SP), a new approach for solving typical instances of random constraint satisfaction problems. SP has proven successful in solving random k-satisfiability (k -SAT) and random graph q-coloring (q-COL) in the “hard SAT” region of parameter space [79, 395, 397, 412], relatively close to the SAT/UNSAT phase transition discussed in the previous chapter. In this chapter we discuss the SP equations, and suggest a theoretical framework for the method [429] that applies to a wide class of discrete constraint satisfaction problems. We propose a way of deriving the equations that sheds light on the capabilities of the algorithm, and illustrates the differences with other well-known iterative probabilistic methods. Our approach takes into account the clustered structure of the solution space described in chapter 3, and involves adding an additional “joker” value that variables can be assigned. Within clusters, a variable can be frozen to some value, meaning that the variable always takes the same value for all solutions (satisfying assignments) within the cluster. Alternatively, it can be unfrozen, meaning that it fluctuates from solution to solution within the cluster. As we will discuss, the SP equations manage to describe the fluctuations by assigning joker values to unfrozen variables. The overall algorithmic strategy is iterative and decomposable in two elementary steps. The first step is to evaluate the marginal probabilities of frozen variables using the SP message-passing procedure. The second step, or decimation step, is to use this information to fix the values of some variables and simplify the problem. The notion of message passing will be illustrated throughout the chapter by comparing it with a simpler procedure known as belief propagation (mentioned in ch. 3 in the context of error correcting codes) in which no assumptions are made about the structure of the solution space. The chapter is organized as follows. In section 2 we provide the general formalism, defining constraint satisfaction problems as well as the key concepts of factor graphs and cavities, using the concrete examples of satisfiability and graph coloring.

Sign in / Sign up

Export Citation Format

Share Document