Examining the Causal Structures of Deep Neural Networks Using Information Theory

Deep Neural Networks (DNNs) are often examined at the level of their response to input, such as analyzing the mutual information between nodes and data sets. Yet DNNs can also be examined at the level of causation, exploring “what does what” within the layers of the network itself. Historically, analyzing the causal structure of DNNs has received less attention than understanding their responses to input. Yet definitionally, generalizability must be a function of a DNN’s causal structure as it reflects how the DNN responds to unseen or even not-yet-defined future inputs. Here, we introduce a suite of metrics based on information theory to quantify and track changes in the causal structure of DNNs during training. Specifically, we introduce the effective information (EI) of a feedforward DNN, which is the mutual information between layer input and output following a maximum-entropy perturbation. The EI can be used to assess the degree of causal influence nodes and edges have over their downstream targets in each layer. We show that the EI can be further decomposed in order to examine the sensitivity of a layer (measured by how well edges transmit perturbations) and the degeneracy of a layer (measured by how edge overlap interferes with transmission), along with estimates of the amount of integrated information of a layer. Together, these properties define where each layer lies in the “causal plane”, which can be used to visualize how layer connectivity becomes more sensitive or degenerate over time, and how integration changes during training, revealing how the layer-by-layer causal structure differentiates. These results may help in understanding the generalization capabilities of DNNs and provide foundational tools for making DNNs both more generalizable and more explainable.

Theory and application of reverberated direct and indirect noise

The generation of a temperature disturbance in a flow is accompanied by the production of acoustic waves (direct noise) and of an entropy perturbation. If this entropy perturbation is accelerated or decelerated (as is the case through a nozzle or flow restriction), additional acoustic waves are generated (indirect noise). Several studies have demonstrated this mechanism in controlled conditions by using entropy wave generators, in which entropy waves are generated and convected through a nozzle, leading to direct and indirect noise. An analytical analysis of the direct and indirect noise produced by the generation and acceleration of entropy waves in a reflective environment is presented. The effect of reverberation (repeated acoustic reflections) on low-frequency perturbations (characteristic of entropy wave generators) is determined analytically. These results are then implemented in a set of limit cases, showing the limit behaviours of such systems. The analytical model is applied to the case of the Cambridge entropy wave generator experiment, in which entropy waves are generated by an electric heater and accelerated through a subsonic orifice plate. Due to the clear time separation of direct and indirect noise in the experimental results, direct and indirect noise transfer functions can be extracted from the experimental data for the first time and compared directly with existing theoretical models. The backward-propagating indirect noise generated at an orifice plate is shown to be significantly higher than predicted by existing theoretical models for isentropic nozzles.

Magnetohydrodynamic wave mixing in shear flows: Hamiltonian equations and wave action

Magnetohydrodynamic wave interactions in a linear shear flow are investigated using the Lagrangian fluid displacement ξ and entropy perturbation Δ S, in which a spatial Fourier solution is obtained in the frame of the background shear flow (Kelvin's method). The equations reduce to three coupled oscillator equations, with time-dependent coefficients and with source terms proportional to the entropy perturbation. In the absence of entropy perturbations, the system admits a wave action conservation integral consisting of positive and negative energy waves. Variational and Hamiltonian forms of the equations are obtained. Examples of wave amplification phenomena and sharp resonant-type wave interactions are obtained. Implications for the interaction of magnetohydrodynamic waves in the shear flow between fast, polar coronal-hole solar wind and slow, streamer belt solar wind are discussed.

The Role of Convection in Reducing Nonadiabaticity and Mode Coupling in Cepheids

The effect of convection on the strength of coupling is examined. It is found that in Cepheid models the inclusion of convection smooths the sharp peak of entropy perturbation in the ionization region and reduces significantly the coupling.