Estimation of Mutual Information and Conditional Entropy for Surveillance Optimization

2013 ◽  
Author(s):  
Duc H. Le ◽  
Albert C. Reynolds
Keyword(s):  
Mutual Information ◽  
Conditional Entropy

scholarly journals Information in the Nonstationary Case

2009 ◽  
Vol 21 (3) ◽  
pp. 688-703 ◽  
Author(s):  
Vincent Q. Vu ◽  
Bin Yu ◽  
Robert E. Kass
Keyword(s):  
Mutual Information ◽  
Conditional Distribution ◽  
Direct Method ◽  
Conditional Entropy ◽  
Difficult Problem ◽  
Response Distribution ◽  
Direct Estimate ◽  
Estimation Strategy ◽  
The Difference

Information estimates such as the direct method of Strong, Koberle, de Ruyter van Steveninck, and Bialek (1998) sidestep the difficult problem of estimating the joint distribution of response and stimulus by instead estimating the difference between the marginal and conditional entropies of the response. While this is an effective estimation strategy, it tempts the practitioner to ignore the role of the stimulus and the meaning of mutual information. We show here that as the number of trials increases indefinitely, the direct (or plug-in) estimate of marginal entropy converges (with probability 1) to the entropy of the time-averaged conditional distribution of the response, and the direct estimate of the conditional entropy converges to the time-averaged entropy of the conditional distribution of the response. Under joint stationarity and ergodicity of the response and stimulus, the difference of these quantities converges to the mutual information. When the stimulus is deterministic or nonstationary the direct estimate of information no longer estimates mutual information, which is no longer meaningful, but it remains a measure of variability of the response distribution across time.

Information Flow in Chaotic Symbolic Dynamics for Finite and Infinitesimal Resolution

1997 ◽  
Vol 07 (01) ◽  
pp. 97-105 ◽  
Author(s):  
Gustavo Deco ◽  
Christian Schittenkopf ◽  
Bernd Schürmann
Keyword(s):  
Mutual Information ◽  
Information Flow ◽  
Symbolic Dynamics ◽  
Initial Conditions ◽  
Deterministic Chaos ◽  
Conditional Entropy ◽  
Information Loss ◽  
Statistical Correlations ◽  
The Future ◽  
One Step

We introduce an information-theory-based concept for the characterization of the information flow in chaotic systems in the framework of symbolic dynamics for finite and infinitesimal measurement resolutions. The information flow characterizes the loss of information about the initial conditions, i.e. the decay of statistical correlations (i.e. nonlinear and non-Gaussian) between the entire past and a point p steps into the future as a function of p. In the case where the partition generating the symbolic dynamics is finite, the information loss is measured by the mutual information that measures the statistical correlations between the entire past and a point p steps into the future. When the partition used is a generator and only one step ahead is observed (p = 1), our definition includes the Kolmogorov–Sinai entropy concept. The profiles in p of the mutual information describe the short- and long-range forecasting possibilities for the given partition resolution. For chaos it is more relevant to study the information loss for the case of infinitesimal partitions which characterizes the intrinsic behavior of the dynamics on an extremely fine scale. Due to the divergence of the mutual information for infinitesimal partitions, the "intrinsic" information flow is characterized by the conditional entropy which generalizes the Kolmogorov–Sinai entropy for the case of observing the uncertainty more than one step into the future. The intrinsic information flow offers an instrument for characterizing deterministic chaos by the transmission of information from the past to the future.

ENTROPIES OF FUZZY INDISCERNIBILITY RELATION AND ITS OPERATIONS

2004 ◽  
Vol 12 (05) ◽  
pp. 575-589 ◽  
Author(s):  
QINGHUA HU ◽  
DAREN YU
Keyword(s):  
Information Theory ◽  
Mutual Information ◽  
Relative Entropy ◽  
Rough Set ◽  
Data Reduction ◽  
Conditional Entropy ◽  
Point Of View ◽  
Joint Entropy ◽  
Fuzzy Rough Set ◽  
Numeric Data

Yager's entropy was proposed to compute the information of fuzzy indiscernibility relation. In this paper we present a novel interpretation of Yager's entropy in discernibility power of a relation point of view. Then some basic definitions in Shannon's information theory are generalized based on Yager's entropy. We introduce joint entropy, conditional entropy, mutual information and relative entropy to compute the information changes for fuzzy indiscerniblity relation operations. Conditional entropy and relative conditional entropy are proposed to measure the information increment, which is interpreted as the significance of an attribute in fuzzy rough set model. As an application, we redefine independency of an attribute set, reduct, relative reduct in fuzzy rough set model based on Yager's entropy. Some experimental results show the proposed approach is suitable for fuzzy and numeric data reduction.

Estimation of Mutual Information and Conditional Entropy for Surveillance Optimization

SPE Journal ◽  
2014 ◽  
Vol 19 (04) ◽  
pp. 648-661 ◽  
Author(s):  
Duc H. Le ◽  
Albert C. Reynolds
Keyword(s):  
Information Theory ◽  
Mutual Information ◽  
History Matching ◽  
Conditional Entropy ◽  
Uncertainty Reduction ◽  
Expected Value ◽  
Data Vector ◽  
Significant Research ◽  
Matching Procedure ◽  
2D And 3D

Summary Given a suite of potential surveillance operations, we define surveillance optimization as the problem of choosing the operation that gives the minimum expected value of P90 minus P10 (i.e., P90 – P10) of a specified reservoir variable J (e.g., cumulative oil production) that will be obtained by conditioning J to the observed data. Two questions can be posed: (1) Which surveillance operation is expected to provide the greatest uncertainty reduction in J? and (2) What is the expected value of the reduction in uncertainty that would be achieved if we were to undertake each surveillance operation to collect the associated data and then history match the data obtained? In this work, we extend and apply a conceptual idea that we recently proposed for surveillance optimization to 2D and 3D waterflooding problems. Our method is based on information theory in which the mutual information between J and the random observed data vector Dobs is estimated by use of an ensemble of prior reservoir models. This mutual information reflects the strength of the relationship between J and the potential observed data and provides a qualitative answer to Question 1. Question 2 is answered by calculating the conditional entropy of J to generate an approximation of the expected value of the reduction in (P90 – P10) of J. The reliability of our method depends on obtaining a good estimate of the mutual information. We consider several ways to estimate the mutual information and suggest how a good estimate can be chosen. We validate the results of our proposed method with an exhaustive history-matching procedure. The methodology provides an approximate way to decide the data that should be collected to maximize the uncertainty reduction in a specified reservoir variable and to estimate the reduction in uncertainty that could be obtained. We expect this paper will stimulate significant research on the application of information theory and lead to practical methods and workflows for surveillance optimization.

scholarly journals Conditional Rényi Divergences and Horse Betting

Entropy ◽  
2020 ◽  
Vol 22 (3) ◽  
pp. 316 ◽  
Author(s):  
Cédric Bleuler ◽  
Amos Lapidoth ◽  
Christoph Pfister
Keyword(s):  
Mutual Information ◽  
Data Processing ◽  
Utility Function ◽  
Side Information ◽  
Conditional Entropy ◽  
Utility Functions ◽  
Power Mean ◽  
Rényi Divergence ◽  
Dependence Measures ◽  
Measure Of Dependence

Motivated by a horse betting problem, a new conditional Rényi divergence is introduced. It is compared with the conditional Rényi divergences that appear in the definitions of the dependence measures by Csiszár and Sibson, and the properties of all three are studied with emphasis on their behavior under data processing. In the same way that Csiszár’s and Sibson’s conditional divergence lead to the respective dependence measures, so does the new conditional divergence lead to the Lapidoth–Pfister mutual information. Moreover, the new conditional divergence is also related to the Arimoto–Rényi conditional entropy and to Arimoto’s measure of dependence. In the second part of the paper, the horse betting problem is analyzed where, instead of Kelly’s expected log-wealth criterion, a more general family of power-mean utility functions is considered. The key role in the analysis is played by the Rényi divergence, and in the setting where the gambler has access to side information, the new conditional Rényi divergence is key. The setting with side information also provides another operational meaning to the Lapidoth–Pfister mutual information. Finally, a universal strategy for independent and identically distributed races is presented that—without knowing the winning probabilities or the parameter of the utility function—asymptotically maximizes the gambler’s utility function.

Quantifying Stimulus Discriminability: A Comparison of Information Theory and Ideal Observer Analysis

2005 ◽  
Vol 17 (4) ◽  
pp. 741-778 ◽  
Author(s):  
Eric E. Thomson ◽  
William B. Kristan
Keyword(s):  
Information Theory ◽  
Mutual Information ◽  
Conditional Entropy ◽  
Ideal Observer ◽  
Upper And Lower Bounds ◽  
Specific Information ◽  
Information Theoretic ◽  
Information Measures ◽  
Improved Performance ◽  
Ideal Observer Analysis

Performance in sensory discrimination tasks is commonly quantified using either information theory or ideal observer analysis. These two quantitative frameworks are often assumed to be equivalent. For example, higher mutual information is said to correspond to improved performance of an ideal observer in a stimulus estimation task. To the contrary, drawing on and extending previous results, we show that five information-theoretic quantities (entropy, response-conditional entropy, specific information, equivocation, and mutual information) violate this assumption. More positively, we show how these information measures can be used to calculate upper and lower bounds on ideal observer performance, and vice versa. The results show that the mathematical resources of ideal observer analysis are preferable to information theory for evaluating performance in a stimulus discrimination task. We also discuss the applicability of information theory to questions that ideal observer analysis cannot address.

Sign in / Sign up

Export Citation Format

Share Document