scholarly journals On the α-q-Mutual Information and the α-q-Capacities

Entropy ◽  
2021 ◽  
Vol 23 (6) ◽  
pp. 702
Author(s):  
Velimir Ilić ◽  
Ivan Djordjević
Keyword(s):  
Information Theory ◽  
Mutual Information ◽  
Information Transfer ◽  
Communication Channels ◽  
Detection Error ◽  
Maximum Likelihood Detection ◽  
Information Measures ◽  
Special Cases ◽  
Previous Definition ◽  
Two Parameter

The measures of information transfer which correspond to non-additive entropies have intensively been studied in previous decades. The majority of the work includes the ones belonging to the Sharma–Mittal entropy class, such as the Rényi, the Tsallis, the Landsberg–Vedral and the Gaussian entropies. All of the considerations follow the same approach, mimicking some of the various and mutually equivalent definitions of Shannon information measures, and the information transfer is quantified by an appropriately defined measure of mutual information, while the maximal information transfer is considered as a generalized channel capacity. However, all of the previous approaches fail to satisfy at least one of the ineluctable properties which a measure of (maximal) information transfer should satisfy, leading to counterintuitive conclusions and predicting nonphysical behavior even in the case of very simple communication channels. This paper fills the gap by proposing two parameter measures named the α-q-mutual information and the α-q-capacity. In addition to standard Shannon approaches, special cases of these measures include the α-mutual information and the α-capacity, which are well established in the information theory literature as measures of additive Rényi information transfer, while the cases of the Tsallis, the Landsberg–Vedral and the Gaussian entropies can also be accessed by special choices of the parameters α and q. It is shown that, unlike the previous definition, the α-q-mutual information and the α-q-capacity satisfy the set of properties, which are stated as axioms, by which they reduce to zero in the case of totally destructive channels and to the (maximal) input Sharma–Mittal entropy in the case of perfect transmission, which is consistent with the maximum likelihood detection error. In addition, they are non-negative and less than or equal to the input and the output Sharma–Mittal entropies, in general. Thus, unlike the previous approaches, the proposed (maximal) information transfer measures do not manifest nonphysical behaviors such as sub-capacitance or super-capacitance, which could qualify them as appropriate measures of the Sharma–Mittal information transfer.

Information Measures

2015 ◽  
pp. 129-142
Keyword(s):  
Information Theory ◽  
Mutual Information ◽  
Order Logic ◽  
Unified Framework ◽  
Higher Order Logic ◽  
Absolutely Continuous ◽  
Information Measures ◽  
Finite Spaces ◽  
Nikodym Derivative ◽  
Lebesgue Integration

This chapter presents a higher-order-logic formalization of the main concepts of information theory (Cover & Thomas, 1991), such as the Shannon entropy and mutual information, using the formalization of the foundational theories of measure, Lebesgue integration, and probability. The main results of the chapter include the formalizations of the Radon-Nikodym derivative and the Kullback-Leibler (KL) divergence (Coble, 2010). The latter provides a unified framework based on which most of the commonly used measures of information can be defined. The chapter then provides the general definitions that are valid for both discrete and continuous cases and then proves the corresponding reduced expressions where the measures considered are absolutely continuous over finite spaces.

EEG TRANSFER ENTROPY TRACKS CHANGES IN INFORMATION TRANSFER ON THE ONSET OF VISION

2012 ◽  
Vol 17 ◽  
pp. 9-18 ◽  
Author(s):  
M. D. MADULARA ◽  
P. A. B. FRANCISCO ◽  
S. NAWANG ◽  
D. C. AROGANCIA ◽  
C. J. CELLUCCI ◽  
...  
Keyword(s):  
Mutual Information ◽  
Information Transfer ◽  
Occipital Lobe ◽  
Transfer Entropy ◽  
The Other ◽  
Information Measures ◽  
Eyes Closed ◽  
Eyes Open ◽  
Almost All ◽  
Nonlinear Correlations

We investigate the pairwise mutual information and transfer entropy of ten-channel, free-running electroencephalographs measured from thirteen subjects under two behavioral conditions: eyes open resting and eyes closed resting. Mutual information measures nonlinear correlations; transfer entropy determines the directionality of information transfer. For all channel pairs, mutual information is generally lower with eyes open compared to eyes closed indicating that EEG signals at different scalp sites become more dissimilar as the visual system is engaged. On the other hand, transfer entropy increases on average by almost two-fold when the eyes are opened. The largest one-way transfer entropies are to and from the Oz site consistent with the involvement of the occipital lobe in vision. The largest net transfer entropies are from F3 and F4 to almost all the other scalp sites.

scholarly journals Alternative Entropy Measures and Generalized Khinchin–Shannon Inequalities

Entropy ◽  
2021 ◽  
Vol 23 (12) ◽  
pp. 1618
Author(s):  
Rubem P. Mondaini ◽  
Simão C. de Albuquerque Neto
Keyword(s):  
Information Theory ◽  
Algebraic Structure ◽  
Physical Systems ◽  
Special Cases ◽  
Entropy Measures ◽  
The Rich ◽  
Two Parameter

The Khinchin–Shannon generalized inequalities for entropy measures in Information Theory, are a paradigm which can be used to test the Synergy of the distributions of probabilities of occurrence in physical systems. The rich algebraic structure associated with the introduction of escort probabilities seems to be essential for deriving these inequalities for the two-parameter Sharma–Mittal set of entropy measures. We also emphasize the derivation of these inequalities for the special cases of one-parameter Havrda–Charvat’s, Rényi’s and Landsberg–Vedral’s entropy measures.

scholarly journals Informeasure: an R/Bioconductor package to quantify nonlinear dependence between variables in biological networks from an information theory perspective

2021 ◽  
Author(s):  
CHU PAN
Keyword(s):  
Information Theory ◽  
Mutual Information ◽  
Biological Networks ◽  
Regulatory Networks ◽  
Partial Information ◽  
R Package ◽  
Nonlinear Dependence ◽  
Bioconductor Package ◽  
Information Measures ◽  
Biological Regulatory Networks

Using information measures to infer biological regulatory networks can observe nonlinear relationship between variables, but it is computationally challenging and there is currently no convenient tool available. We here describe an information theory R package named Informeasure that devotes to quantifying nonlinear dependence between variables in biological regulatory networks from an information theory perspective. This package compiles most of the information measures currently available: mutual information, conditional mutual information, interaction information, partial information decomposition and part mutual information. The first estimator is used to infer bivariate networks while the last four estimators are dedicated to analysis of trivariate networks. The base installation of this turn-key package allows users to approach these information measures out of the box. Informeasure is implemented in R program and is available as an R/Bioconductor package at https://bioconductor.org/packages/Informeasure.

scholarly journals A Path-Based Partial Information Decomposition

Entropy ◽  
2020 ◽  
Vol 22 (9) ◽  
pp. 952
Author(s):  
David Sigtermans
Keyword(s):  
Information Theory ◽  
Mutual Information ◽  
Graphical Model ◽  
Partial Information ◽  
Communication Channels ◽  
Negative Information ◽  
Information Measure ◽  
Probability Mass ◽  
Information Components ◽  
Mass Functions

Based on the conceptual basis of information theory, we propose a novel mutual information measure—‘path-based mutual information’. This information measure results from the representation of a set of random variables as a probabilistic graphical model. The edges in this graph are modeled as discrete memoryless communication channels, that is, the underlying data is ergodic, stationary, and the Markov condition is assumed to be applicable. The associated multilinear stochastic maps, tensors, transform source probability mass functions into destination probability mass functions. This allows for an exact expression of the resulting tensor of a cascade of discrete memoryless communication channels in terms of the tensors of the constituting communication channels in the paths. The resulting path-based information measure gives rise to intuitive, non-negative, and additive path-based information components—redundant, unique, and synergistic information—as proposed by Williams and Beer. The path-based redundancy satisfies the axioms postulated by Williams and Beer, the identity axiom postulated by Harder, and the left monotonicity axiom postulated Bertschinger. The ordering relations between redundancies of different joint collections of sources, as captured in the redundancy lattices of Williams and Beer, follow from the data processing inequality. Although negative information components can arise, we speculate that these either result from unobserved variables, or from adding additional sources that are statistically independent from all other sources to a system containing only non-negative information components. This path-based approach illustrates that information theory provides the concepts and measures for a partial information decomposition.

scholarly journals Empirical Estimation of Information Measures: A Literature Guide

Entropy ◽  
2019 ◽  
Vol 21 (8) ◽  
pp. 720 ◽  
Author(s):  
Sergio Verdú
Keyword(s):  
Machine Learning ◽  
Information Theory ◽  
Mutual Information ◽  
Relative Entropy ◽  
Data Science ◽  
Differential Entropy ◽  
Empirical Estimation ◽  
Central Importance ◽  
Information Measures ◽  
Related Information

We give a brief survey of the literature on the empirical estimation of entropy, differential entropy, relative entropy, mutual information and related information measures. While those quantities are of central importance in information theory, universal algorithms for their estimation are increasingly important in data science, machine learning, biology, neuroscience, economics, language, and other experimental sciences.

Quantifying Stimulus Discriminability: A Comparison of Information Theory and Ideal Observer Analysis

2005 ◽  
Vol 17 (4) ◽  
pp. 741-778 ◽  
Author(s):  
Eric E. Thomson ◽  
William B. Kristan
Keyword(s):  
Information Theory ◽  
Mutual Information ◽  
Conditional Entropy ◽  
Ideal Observer ◽  
Upper And Lower Bounds ◽  
Specific Information ◽  
Information Theoretic ◽  
Information Measures ◽  
Improved Performance ◽  
Ideal Observer Analysis

Performance in sensory discrimination tasks is commonly quantified using either information theory or ideal observer analysis. These two quantitative frameworks are often assumed to be equivalent. For example, higher mutual information is said to correspond to improved performance of an ideal observer in a stimulus estimation task. To the contrary, drawing on and extending previous results, we show that five information-theoretic quantities (entropy, response-conditional entropy, specific information, equivocation, and mutual information) violate this assumption. More positively, we show how these information measures can be used to calculate upper and lower bounds on ideal observer performance, and vice versa. The results show that the mathematical resources of ideal observer analysis are preferable to information theory for evaluating performance in a stimulus discrimination task. We also discuss the applicability of information theory to questions that ideal observer analysis cannot address.

Sign in / Sign up

Export Citation Format

Share Document