scholarly journals Kernel Taylor-Based Value Function Approximation for Continuous-State Markov Decision Processes

2020 ◽  
Author(s):  
Junhong Xu ◽  
Kai Yin ◽  
Lantao Liu
Keyword(s):  
Markov Decision Processes ◽  
Function Approximation ◽  
Value Function ◽  
Decision Processes ◽  
Value Function Approximation ◽  
Continuous State ◽  
Markov Decision

scholarly journals Scalable Methods for Computing State Similarity in Deterministic Markov Decision Processes

2020 ◽  
Vol 34 (06) ◽  
pp. 10069-10076
Author(s):  
Pablo Samuel Castro
Keyword(s):  
Markov Decision Processes ◽  
Decision Processes ◽  
Behavioral Equivalence ◽  
Continuous State ◽  
Markov Decision ◽  
Scalable Methods ◽  
New Algorithms ◽  
Optimal Behaviour

We present new algorithms for computing and approximating bisimulation metrics in Markov Decision Processes (MDPs). Bisimulation metrics are an elegant formalism that capture behavioral equivalence between states and provide strong theoretical guarantees on differences in optimal behaviour. Unfortunately, their computation is expensive and requires a tabular representation of the states, which has thus far rendered them impractical for large problems. In this paper we present a new version of the metric that is tied to a behavior policy in an MDP, along with an analysis of its theoretical properties. We then present two new algorithms for approximating bisimulation metrics in large, deterministic MDPs. The first does so via sampling and is guaranteed to converge to the true metric. The second is a differentiable loss which allows us to learn an approximation even for continuous state MDPs, which prior to this work had not been possible.

scholarly journals An analysis of transient Markov decision processes

2006 ◽  
Vol 43 (3) ◽  
pp. 603-621 ◽  
Author(s):  
Huw W. James ◽  
E. J. Collins
Keyword(s):  
Markov Decision Processes ◽  
Value Function ◽  
Decision Processes ◽  
Optimal Value Function ◽  
Natural Form ◽  
Convergence Results ◽  
Optimal Value ◽  
Finite State ◽  
Markov Decision ◽  
Bounded Below

This paper is concerned with the analysis of Markov decision processes in which a natural form of termination ensures that the expected future costs are bounded, at least under some policies. Whereas most previous analyses have restricted attention to the case where the set of states is finite, this paper analyses the case where the set of states is not necessarily finite or even countable. It is shown that all the existence, uniqueness, and convergence results of the finite-state case hold when the set of states is a general Borel space, provided we make the additional assumption that the optimal value function is bounded below. We give a sufficient condition for the optimal value function to be bounded below which holds, in particular, if the set of states is countable.

scholarly journals Kalman Temporal Differences

2010 ◽  
Vol 39 ◽  
pp. 483-532 ◽  
Author(s):  
M. Geist ◽  
O. Pietquin
Keyword(s):  
Markov Decision Processes ◽  
Function Approximation ◽  
Approximation Scheme ◽  
State Of The Art ◽  
Decision Processes ◽  
Temporal Differences ◽  
Special Cases ◽  
Markov Decision ◽  
Biased Estimates ◽  
Q Function

Because reinforcement learning suffers from a lack of scalability, online value (and Q-) function approximation has received increasing interest this last decade. This contribution introduces a novel approximation scheme, namely the Kalman Temporal Differences (KTD) framework, that exhibits the following features: sample-efficiency, non-linear approximation, non-stationarity handling and uncertainty management. A first KTD-based algorithm is provided for deterministic Markov Decision Processes (MDP) which produces biased estimates in the case of stochastic transitions. Than the eXtended KTD framework (XKTD), solving stochastic MDP, is described. Convergence is analyzed for special cases for both deterministic and stochastic transitions. Related algorithms are experimented on classical benchmarks. They compare favorably to the state of the art while exhibiting the announced features.

scholarly journals Task Scoping for Efficient Planning in Open Worlds (Student Abstract)

2020 ◽  
Vol 34 (10) ◽  
pp. 13845-13846
Author(s):  
Nishanth Kumar ◽  
Michael Fishman ◽  
Natasha Danas ◽  
Stefanie Tellex ◽  
Michael Littman ◽  
...  
Keyword(s):  
Markov Decision Processes ◽  
Value Function ◽  
Decision Processes ◽  
Initial State ◽  
Open World ◽  
Optimal Value ◽  
Markov Decision ◽  
Efficient Planning ◽  
Action Spaces ◽  
Action Variables

We propose an abstraction method for open-world environments expressed as Factored Markov Decision Processes (FMDPs) with very large state and action spaces. Our method prunes state and action variables that are irrelevant to the optimal value function on the state subspace the agent would visit when following any optimal policy from the initial state. This method thus enables tractable fast planning within large open-world FMDPs.

Sign in / Sign up

Export Citation Format

Share Document