scholarly journals On a Generalization of the Jensen–Shannon Divergence and the Jensen–Shannon Centroid

Entropy ◽  
2020 ◽  
Vol 22 (2) ◽  
pp. 221 ◽  
Author(s):  
Frank Nielsen
Keyword(s):  
Iterative Algorithm ◽  
Bregman Divergences ◽  
Leibler Divergence ◽  
Probability Densities ◽  
Jensen Shannon Divergence

The Jensen–Shannon divergence is a renown bounded symmetrization of the Kullback–Leibler divergence which does not require probability densities to have matching supports. In this paper, we introduce a vector-skew generalization of the scalar α -Jensen–Bregman divergences and derive thereof the vector-skew α -Jensen–Shannon divergences. We prove that the vector-skew α -Jensen–Shannon divergences are f-divergences and study the properties of these novel divergences. Finally, we report an iterative algorithm to numerically compute the Jensen–Shannon-type centroids for a set of probability densities belonging to a mixture family: This includes the case of the Jensen–Shannon centroid of a set of categorical distributions or normalized histograms.

scholarly journals On the Jensen–Shannon Symmetrization of Distances Relying on Abstract Means

Entropy ◽  
2019 ◽  
Vol 21 (5) ◽  
pp. 485 ◽  
Author(s):  
Frank Nielsen
Keyword(s):  
Closed Form ◽  
Exponential Family ◽  
Geometric Mean ◽  
Mixture Distribution ◽  
Exponential Families ◽  
Leibler Divergence ◽  
Probability Densities ◽  
The Mean ◽  
Jensen Shannon Divergence ◽  
Closed Form Formula

The Jensen–Shannon divergence is a renowned bounded symmetrization of the unbounded Kullback–Leibler divergence which measures the total Kullback–Leibler divergence to the average mixture distribution. However, the Jensen–Shannon divergence between Gaussian distributions is not available in closed form. To bypass this problem, we present a generalization of the Jensen–Shannon (JS) divergence using abstract means which yields closed-form expressions when the mean is chosen according to the parametric family of distributions. More generally, we define the JS-symmetrizations of any distance using parameter mixtures derived from abstract means. In particular, we first show that the geometric mean is well-suited for exponential families, and report two closed-form formula for (i) the geometric Jensen–Shannon divergence between probability densities of the same exponential family; and (ii) the geometric JS-symmetrization of the reverse Kullback–Leibler divergence between probability densities of the same exponential family. As a second illustrating example, we show that the harmonic mean is well-suited for the scale Cauchy distributions, and report a closed-form formula for the harmonic Jensen–Shannon divergence between scale Cauchy distributions. Applications to clustering with respect to these novel Jensen–Shannon divergences are touched upon.

Vector Quantization by Minimizing Kullback-Leibler Divergence between the Class Label Distributions over Quantization Input and Output

2014 ◽  
Vol 1006-1007 ◽  
pp. 764-767
Author(s):  
Hao Xiang Wang ◽  
Shan Yue ◽  
Yang Li
Keyword(s):  
Objective Function ◽  
Iterative Algorithm ◽  
Vector Quantization ◽  
The Novel ◽  
Classification Problems ◽  
Class Label ◽  
Bag Of Features ◽  
Input And Output ◽  
Leibler Divergence ◽  
Novel Method

This paper proposes a new method for vector quantization by minimizing the Divergence of Kullback-Leibler between the class label distributions over the quantization inputs, which are original vectors, and the output, which is the quantization subsets of the vector set. In this way, the vector quantization output can keep as much information of the class label as possible. An objective function is constructed and we developed an iterative algorithm to minimize it as well. The novel method is evaluated on bag-of-features based image classification problems.

scholarly journals minicore: Fast scRNA-seq clustering with various distances

2021 ◽  
Author(s):  
Daniel N. Baker ◽  
Nathan Dyjack ◽  
Vladimir Braverman ◽  
Stephanie C. Hicks ◽  
Ben Langmead
Keyword(s):  
Open Source ◽  
Count Data ◽  
Probability Distributions ◽  
Expression Profiles ◽  
Distance Measures ◽  
Bhattacharyya Distance ◽  
Link Type ◽  
Leibler Divergence ◽  
Careful Handling ◽  
Jensen Shannon Divergence

AbstractSingle-cell RNA-sequencing (scRNA-seq) analyses typically begin by clustering a gene-by-cell expression matrix to empirically define groups of cells with similar expression profiles. We describe new methods and a new open source library, minicore, for efficient k-means++ center finding and k-means clustering of scRNA-seq data. Minicore works with sparse count data, as it emerges from typical scRNA-seq experiments, as well as with dense data from after dimensionality reduction. Minicore’s novel vectorized weighted reservoir sampling algorithm allows it to find initial k-means++ centers for a 4-million cell dataset in 1.5 minutes using 20 threads. Minicore can cluster using Euclidean distance, but also supports a wider class of measures like Jensen-Shannon Divergence, Kullback-Leibler Divergence, and the Bhattacharyya distance, which can be directly applied to count data and probability distributions.Further, minicore produces lower-cost centerings more efficiently than scikit-learn for scRNA-seq datasets with millions of cells. With careful handling of priors, minicore implements these distance measures with only minor (<2-fold) speed differences among all distances. We show that a minicore pipeline consisting of k-means++, localsearch++ and minibatch k-means can cluster a 4-million cell dataset in minutes, using less than 10GiB of RAM. This memory-efficiency enables atlas-scale clustering on laptops and other commodity hardware. Finally, we report findings on which distance measures give clusterings that are most consistent with known cell type labels.AvailabilityThe open source library is at https://github.com/dnbaker/minicore. Code used for experiments is at https://github.com/dnbaker/minicore-experiments.

scholarly journals Fluctuation–response inequality out of equilibrium

2020 ◽  
Vol 117 (12) ◽  
pp. 6430-6436 ◽  
Author(s):  
Andreas Dechant ◽  
Shin-ichi Sasa
Keyword(s):  
Steady State ◽  
Uncertainty Relation ◽  
Stochastic Dynamics ◽  
Direct Consequence ◽  
Transport Processes ◽  
Unperturbed System ◽  
Mean Velocity ◽  
Leibler Divergence ◽  
Probability Densities ◽  
Local Mean

We present an approach to response around arbitrary out-of-equilibrium states in the form of a fluctuation–response inequality (FRI). We study the response of an observable to a perturbation of the underlying stochastic dynamics. We find that the magnitude of the response is bounded from above by the fluctuations of the observable in the unperturbed system and the Kullback–Leibler divergence between the probability densities describing the perturbed and the unperturbed system. This establishes a connection between linear response and concepts of information theory. We show that in many physical situations, the relative entropy may be expressed in terms of physical observables. As a direct consequence of this FRI, we show that for steady-state particle transport, the differential mobility is bounded by the diffusivity. For a “virtual” perturbation proportional to the local mean velocity, we recover the thermodynamic uncertainty relation (TUR) for steady-state transport processes. Finally, we use the FRI to derive a generalization of the uncertainty relation to arbitrary dynamics, which involves higher-order cumulants of the observable. We provide an explicit example, in which the TUR is violated but its generalization is satisfied with equality.

Information geometry of divergence functions

2010 ◽  
Vol 58 (1) ◽  
pp. 183-195 ◽  
Author(s):  
S. Amari ◽  
A. Cichocki
Keyword(s):  
Probability Distributions ◽  
Flat Space ◽  
Information Geometry ◽  
Dual Pair ◽  
Bregman Divergence ◽  
Bregman Divergences ◽  
Leibler Divergence ◽  
Fisher Information Metric ◽  
Information Metric ◽  
Dually Flat Space

Information geometry of divergence functionsMeasures of divergence between two points play a key role in many engineering problems. One such measure is a distance function, but there are many important measures which do not satisfy the properties of the distance. The Bregman divergence, Kullback-Leibler divergence andf-divergence are such measures. In the present article, we study the differential-geometrical structure of a manifold induced by a divergence function. It consists of a Riemannian metric, and a pair of dually coupled affine connections, which are studied in information geometry. The class of Bregman divergences are characterized by a dually flat structure, which is originated from the Legendre duality. A dually flat space admits a generalized Pythagorean theorem. The class off-divergences, defined on a manifold of probability distributions, is characterized by information monotonicity, and the Kullback-Leibler divergence belongs to the intersection of both classes. Thef-divergence always gives the α-geometry, which consists of the Fisher information metric and a dual pair of ±α-connections. The α-divergence is a special class off-divergences. This is unique, sitting at the intersection of thef-divergence and Bregman divergence classes in a manifold of positive measures. The geometry derived from the Tsallisq-entropy and related divergences are also addressed.

Constructing common height maps with various entropy-based similarity metrics and utilizing layering method for heterogeneous robot teams

2020 ◽  
Vol 47 (6) ◽  
pp. 889-902
Author(s):  
Mehmet Caner Akay ◽  
Hakan Temeltaş
Keyword(s):  
Global Positioning System ◽  
Shannon Entropy ◽  
Similarity Metrics ◽  
Positioning System ◽  
Content Type ◽  
Robot Teams ◽  
Novel Approach ◽  
Leibler Divergence ◽  
Global Positioning ◽  
Jensen Shannon Divergence

Purpose Heterogeneous teams consisting of unmanned ground vehicles and unmanned aerial vehicles are being used for different types of missions such as surveillance, tracking and exploration. Exploration missions with heterogeneous robot teams (HeRTs) should acquire a common map for understanding the surroundings better. The purpose of this paper is to provide a unique approach with cooperative use of agents that provides a well-detailed observation over the environment where challenging details and complex structures are involved. Also, this method is suitable for real-time applications and autonomous path planning for exploration. Design/methodology/approach Lidar odometry and mapping and various similarity metrics such as Shannon entropy, Kullback–Leibler divergence, Jeffrey divergence, K divergence, Topsoe divergence, Jensen–Shannon divergence and Jensen divergence are used to construct a common height map of the environment. Furthermore, the authors presented the layering method that provides more accuracy and a better understanding of the common map. Findings In summary, with the experiments, the authors observed features located beneath the trees or the roofed top areas and above them without any need for global positioning system signal. Additionally, a more effective common map that enables planning trajectories for both vehicles is obtained with the determined similarity metric and the layering method. Originality/value In this study, the authors present a unique solution that implements various entropy-based similarity metrics with the aim of constructing common maps of the environment with HeRTs. To create common maps, Shannon entropy–based similarity metrics can be used, as it is the only one that holds the chain rule of conditional probability precisely. Seven distinct similarity metrics are compared, and the most effective one is chosen for getting a more comprehensive and valid common map. Moreover, different from all the studies in literature, the layering method is used to compute the similarities of each local map obtained by a HeRT. This method also provides the accuracy of the merged common map, as robots’ sight of view prevents the same observations of the environment in features such as a roofed area or trees. This novel approach can also be used in global positioning system-denied and closed environments. The results are verified with experiments.

scholarly journals IT-SVO: Improved Semi-Direct Monocular Visual Odometry Combined with JS Divergence in Restricted Mobile Devices

Sensors ◽  
2021 ◽  
Vol 21 (6) ◽  
pp. 2025
Author(s):  
Chang Liu ◽  
Jin Zhao ◽  
Nianyi Sun ◽  
Qingrong Yang ◽  
Leilei Wang
Keyword(s):  
Information Theory ◽  
Mobile Devices ◽  
Mobile Robotics ◽  
A Priori ◽  
Computational Simulation ◽  
Visual Odometry ◽  
Kl Divergence ◽  
Wide Range ◽  
Leibler Divergence ◽  
Jensen Shannon Divergence

Simultaneous localization and mapping (SLAM) has a wide range for applications in mobile robotics. Lightweight and inexpensive vision sensors have been widely used for localization in GPS-denied or weak GPS environments. Mobile robots not only estimate their pose, but also correct their position according to the environment, so a proper mathematical model is required to obtain the state of robots in their circumstances. Usually, filter-based SLAM/VO regards the model as a Gaussian distribution in the mapping thread, which deals with the complicated relationship between mean and covariance. The covariance in SLAM or VO represents the uncertainty of map points. Therefore, the methods, such as probability theory and information theory play a significant role in estimating the uncertainty. In this paper, we combine information theory with classical visual odometry (SVO) and take Jensen-Shannon divergence (JS divergence) instead of Kullback-Leibler divergence (KL divergence) to estimate the uncertainty of depth. A more suitable methodology for SVO is that explores to improve the accuracy and robustness of mobile devices in unknown environments. Meanwhile, this paper aims to efficiently utilize small portability for location and provide a priori knowledge of the latter application scenario. Therefore, combined with SVO, JS divergence is implemented, which has been realized. It not only has the property of accurate distinction of outliers, but also converges the inliers quickly. Simultaneously, the results show, under the same computational simulation, that SVO combined with JS divergence can more accurately locate its state in the environment than the combination with KL divergence.

Jensen–Shannon Divergence Based on Horizontal Visibility Graph for Complex Time Series

2020 ◽  
pp. 2150013
Author(s):  
Yi Yin ◽  
Wenjing Wang ◽  
Qiang Li ◽  
Zunsong Ren ◽  
Pengjian Shang
Keyword(s):  
Time Series ◽  
Financial Time Series ◽  
Visibility Graph ◽  
Horizontal Visibility ◽  
Stock Indices ◽  
Leibler Divergence ◽  
Time Irreversibility ◽  
Jensen Shannon Divergence ◽  
Time Series Irreversibility ◽  
Horizontal Visibility Graph

In this paper, we propose Jensen–Shannon divergence (JSD) based on horizontal visibility graph (HVG) to measure the time series irreversibility for both stationary and non-stationary series efficiently. Numerical simulations are first conducted to show the validity of the proposed method and then empirical applications to the financial time series and traffic time series are investigated. It can be found that JSD shows better robustness than Kullback–Leibler divergence (KLD) on quantifying time series irreversibility and correctly distinguishes the different type of simulated series. For the empirical analysis, JSD based on HVG is able to detect the significant time irreversibility of stock indices and reveal the relationship between different stock indices. JSD results show the time irreversibility of speed time series for different detectors and present better accuracy and robustness than KLD. The hierarchical clustering based on their behavior of time irreversibility obtained by JSD classifies the detectors into four groups.

scholarly journals Comparison of beta diversity measures in clustering the high-dimensional microbial data

PLoS ONE ◽  
2021 ◽  
Vol 16 (2) ◽  
pp. e0246893
Author(s):  
Biyuan Chen ◽  
Xueyi He ◽  
Bangquan Pan ◽  
Xiaobing Zou ◽  
Na You
Keyword(s):  
Beta Diversity ◽  
Compositional Data ◽  
Vital Role ◽  
High Dimensional ◽  
Simulation Experiments ◽  
Diversity Measures ◽  
Leibler Divergence ◽  
Set Up ◽  
Jensen Shannon Divergence

The heterogeneity of disease is a major concern in medical research and is commonly characterized as subtypes with different pathogeneses exhibiting distinct prognoses and treatment effects. The classification of a population into homogeneous subgroups is challenging, especially for complex diseases. Recent studies show that gut microbiome compositions play a vital role in disease development, and it is of great interest to cluster patients according to their microbial profiles. There are a variety of beta diversity measures to quantify the dissimilarity between the compositions of different samples for clustering. However, using different beta diversity measures results in different clusters, and it is difficult to make a choice among them. Considering microbial compositions from 16S rRNA sequencing, which are presented as a high-dimensional vector with a large proportion of extremely small or even zero-valued elements, we set up three simulation experiments to mimic the microbial compositional data and evaluate the performance of different beta diversity measures in clustering. It is shown that the Kullback-Leibler divergence-based beta diversity, including the Jensen-Shannon divergence and its square root, and the hypersphere-based beta diversity, including the Bhattacharyya and Hellinger, can capture compositional changes in low-abundance elements more efficiently and can work stably. Their performance on two real datasets demonstrates the validity of the simulation experiments.

scholarly journals Advanced information criterion for environmental data quality assurance

2012 ◽  
Vol 8 (1) ◽  
pp. 99-104 ◽  
Author(s):  
A. Düsterhus ◽  
A. Hense
Keyword(s):  
Mean Shift ◽  
Real Data ◽  
Information Criterion ◽  
Environmental Data ◽  
Testing Time ◽  
Mean Square ◽  
Density Distributions ◽  
Leibler Divergence ◽  
Probability Density Distributions ◽  
Jensen Shannon Divergence

Abstract. A new method for testing time series of environmental data for internal inconsistencies is presented. The method divides the dataset into several disjunct blocks. By means of a comparison of the blocks' estimated probability density distributions, each block is compared with the others. In order to judge the differences, four different measures are used and compared: Kullback-Leibler Divergence, Jensen-Shannon Divergence, Earth Mover's Distance and the Root Mean Square. By looking at the resulting patterns, conclusions on possible inconsistencies in the data can be drawn. This paper shows some sensitivitiy tests and gives an example for an application to real data. Furthermore, it is shown, in which cases of errors (shift in mean, shift in variance and rounding), which measure performs best.

Sign in / Sign up

Export Citation Format

Share Document