Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach

A spike-timing-dependent Hebbian mechanism governs the plasticity of recurrent excitatory synapses in the neocortex: synapses that are activated a few milliseconds before a postsynaptic spike are potentiated, while those that are activated a few milliseconds after are depressed. We show that such a mechanism can implement a form of temporal difference learning for prediction of input sequences. Using a biophysical model of a cortical neuron, we show that a temporal difference rule used in conjunction with dendritic backpropagating action potentials reproduces the temporally asymmetric window of Hebbian plasticity observed physiologically. Furthermore, the size and shape of the window vary with the distance of the synapse from the soma. Using a simple example, we show how a spike-timing-based temporal difference learning rule can allow a network of neocortical neurons to predict an input a few milliseconds before the input's expected arrival.

Download Full-text

Inverse streamflow routing

Hydrology and Earth System Sciences ◽

10.5194/hess-17-4577-2013 ◽

2013 ◽

Vol 17 (11) ◽

pp. 4577-4588 ◽

Cited By ~ 18

Author(s):

M. Pan ◽

E. F. Wood

Keyword(s):

Water Budget ◽

Temporal Dynamics ◽

Water Cycle ◽

Ohio River ◽

Temporal Difference ◽

Time And Space ◽

Ohio River Basin ◽

Budget Analysis ◽

Spatially Distributed ◽

Terrestrial Water

Abstract. The process whereby the spatially distributed runoff (generated through saturation/infiltration excesses, subsurface flow, etc.) travels over the hillslope and river network and becomes streamflow is generally referred to as "routing". In short, routing is a runoff-to-streamflow process, and the streamflow in rivers is the response to runoff integrated in both time and space. Here we develop a methodology to invert the routing process, i.e., to derive the spatially distributed runoff from streamflow (e.g., measured at gauge stations) by inverting an arbitrary linear routing model using fixed interval smoothing. We refer to this streamflow-to-runoff process as "inverse routing". Inversion experiments are performed using both synthetically generated and real streamflow measurements over the Ohio River basin. Results show that inverse routing can effectively reproduce the spatial field of runoff and its temporal dynamics from sufficiently dense gauge measurements, and the inversion performance can also be strongly affected by low gauge density and poor data quality. The runoff field is the only component in the terrestrial water budget that cannot be directly measured, and all previous studies used streamflow measurements in its place. Consequently, such studies are limited to scales where the spatial and temporal difference between the two can be ignored. Inverse routing provides a more sophisticated tool than traditional methods to bridge this gap and infer fine-scale (in both time and space) details of runoff from aggregated measurements. Improved handling of this final gap in terrestrial water budget analysis may potentially help us to use space-borne altimetry-based surface water measurements for cross-validating, cross-correcting, and assimilation with other space-borne water cycle observations.

Download Full-text