scholarly journals Using the Semantic Information G Measure to Explain and Extend Rate-Distortion Functions and Maximum Entropy Distributions

Entropy ◽  
2021 ◽  
Vol 23 (8) ◽  
pp. 1050
Author(s):  
Chenguang Lu
Keyword(s):  
Machine Learning ◽  
Mutual Information ◽  
Data Compression ◽  
Maximum Entropy ◽  
Semantic Information ◽  
Rate Distortion ◽  
Distortion Function ◽  
Partition Functions ◽  
Statistical Probability ◽  
Distortion Functions

In the rate-distortion function and the Maximum Entropy (ME) method, Minimum Mutual Information (MMI) distributions and ME distributions are expressed by Bayes-like formulas, including Negative Exponential Functions (NEFs) and partition functions. Why do these non-probability functions exist in Bayes-like formulas? On the other hand, the rate-distortion function has three disadvantages: (1) the distortion function is subjectively defined; (2) the definition of the distortion function between instances and labels is often difficult; (3) it cannot be used for data compression according to the labels’ semantic meanings. The author has proposed using the semantic information G measure with both statistical probability and logical probability before. We can now explain NEFs as truth functions, partition functions as logical probabilities, Bayes-like formulas as semantic Bayes’ formulas, MMI as Semantic Mutual Information (SMI), and ME as extreme ME minus SMI. In overcoming the above disadvantages, this paper sets up the relationship between truth functions and distortion functions, obtains truth functions from samples by machine learning, and constructs constraint conditions with truth functions to extend rate-distortion functions. Two examples are used to help readers understand the MMI iteration and to support the theoretical results. Using truth functions and the semantic information G measure, we can combine machine learning and data compression, including semantic compression. We need further studies to explore general data compression and recovery, according to the semantic meaning.

scholarly journals Semantic Information G Theory and Logical Bayesian Inference for Machine Learning

Information ◽  
2019 ◽  
Vol 10 (8) ◽  
pp. 261 ◽  
Author(s):  
Lu
Keyword(s):  
Machine Learning ◽  
Bayesian Inference ◽  
Mutual Information ◽  
Em Algorithm ◽  
Semantic Information ◽  
Feature Space ◽  
Iteration Algorithm ◽  
Membership Functions ◽  
Multilabel Learning ◽  
Learning Functions

An important problem in machine learning is that, when using more than two labels, it is very difficult to construct and optimize a group of learning functions that are still useful when the prior distribution of instances is changed. To resolve this problem, semantic information G theory, Logical Bayesian Inference (LBI), and a group of Channel Matching (CM) algorithms are combined to form a systematic solution. A semantic channel in G theory consists of a group of truth functions or membership functions. In comparison with the likelihood functions, Bayesian posteriors, and Logistic functions that are typically used in popular methods, membership functions are more convenient to use, providing learning functions that do not suffer the above problem. In Logical Bayesian Inference (LBI), every label is independently learned. For multilabel learning, we can directly obtain a group of optimized membership functions from a large enough sample with labels, without preparing different samples for different labels. Furthermore, a group of Channel Matching (CM) algorithms are developed for machine learning. For the Maximum Mutual Information (MMI) classification of three classes with Gaussian distributions in a two-dimensional feature space,only 2–3 iterations are required for the mutual information between three classes and three labels to surpass 99% of the MMI for most initial partitions For mixture models, the Expectation-Maximization (EM) algorithm is improved to form the CM-EM algorithm, which can outperform the EM algorithm when the mixture ratios are imbalanced, or when local convergence exists. The CM iteration algorithm needs to combine with neural networks for MMI classification in high-dimensional feature spaces. LBI needs further investigation for the unification of statistics and logic.

scholarly journals Research a New Tradeoff Method Between Privacy and Utility in Differential privacy

2021 ◽  
Author(s):  
panjun sun
Keyword(s):  
Mutual Information ◽  
Privacy Protection ◽  
Differential Privacy ◽  
Rate Distortion ◽  
Information Privacy ◽  
Distortion Function ◽  
Approximate Algorithm ◽  
Data Utility ◽  
Rate Distortion Theory ◽  
Privacy Leakage

Abstract The solution of the contradiction between privacy protection and data utility is a research hotspot in the field of privacy protection. Aiming at the problem of tradeoff between privacy and utility in the scenario of differential privacy offline data release, the optimal differential privacy mechanism is studied by using the rate distortion theory. Firstly, based on Shannon communication theory, the noise channel model of differential privacy is abstracted, and the mutual information and the distortion function is used to measure the privacy and utility of data publishing, and the optimization model based on rate distortion theory is constructed. Secondly, considering the influence of associated auxiliary background knowledge on mutual information privacy leakage, a mutual information privacy measure based on joint events is proposed, and a minimum privacy leakage model is proposed by modifying the rate distortion function. Finally, aiming at the difficulty in solving the Lagrange multiplier method, an approximate algorithm for solving the mutual information privacy optimization channel mechanism is proposed based on the alternating iterative method. The effectiveness of the proposed iterative approximation method is verified by experimental simulation. At the same time, the experimental results show that the proposed method reduces the mutual information privacy leakage under the condition of limited distortion, and improves the data utility under the same privacy tolerance

scholarly journals Calculating the Number of Cluster Heads Based on the Rate-Distortion Function in Wireless Sensor Networks

2014 ◽  
Vol 2014 ◽  
pp. 1-7
Author(s):  
Mingxin Yang ◽  
Jingsha He ◽  
Yuqiang Zhang
Keyword(s):  
Wireless Sensor Networks ◽  
Energy Consumption ◽  
Sensor Networks ◽  
Data Fusion ◽  
Rate Distortion ◽  
Distortion Function ◽  
Sensor Nodes ◽  
Wireless Sensor ◽  
Wireless Sensor Nodes ◽  
Cluster Heads

Due to limited resources in wireless sensor nodes, energy efficiency is considered as one of the primary constraints in the design of the topology of wireless sensor networks (WSNs). Since data that are collected by wireless sensor nodes exhibit the characteristics of temporal association, data fusion has also become a very important means of reducing network traffic as well as eliminating data redundancy as far as data transmission is concerned. Another reason for data fusion is that, in many applications, only some of the data that are collected can meet the requirements of the sink node. In this paper, we propose a method to calculate the number of cluster heads or data aggregators during data fusion based on the rate-distortion function. In our discussion, we will first establish an energy consumption model and then describe a method for calculating the number of cluster heads from the point of view of reducing energy consumption. We will also show through theoretical analysis and experimentation that the network topology design based on the rate-distortion function is indeed more energy-efficient.

Sign in / Sign up

Export Citation Format

Share Document