Multi-agent temporal-difference learning with linear function approximation: Weak convergence under time-varying network topologies

2016 ◽  
Author(s):  
Milos S. Stankovic ◽  
Srdjan S. Stankovic
Keyword(s):  
Weak Convergence ◽  
Linear Function ◽  
Function Approximation ◽  
Time Varying ◽  
Temporal Difference ◽  
Temporal Difference Learning ◽  
Linear Function Approximation ◽  
Network Topologies ◽  
Multi Agent

scholarly journals A Finite Time Analysis of Temporal Difference Learning with Linear Function Approximation

2021 ◽  
Author(s):  
Jalaj Bhandari ◽  
Daniel Russo ◽  
Raghav Singal
Keyword(s):  
Linear Function ◽  
Finite Time ◽  
Function Approximation ◽  
Gradient Descent ◽  
Convergence Rates ◽  
Temporal Difference ◽  
Temporal Difference Learning ◽  
Convergence Results ◽  
Linear Function Approximation ◽  
Markov Reward

Temporal difference learning (TD) is a simple iterative algorithm widely used for policy evaluation in Markov reward processes. Bhandari et al. prove finite time convergence rates for TD learning with linear function approximation. The analysis follows using a key insight that establishes rigorous connections between TD updates and those of online gradient descent. In a model where observations are corrupted by i.i.d. noise, convergence results for TD follow by essentially mirroring the analysis for online gradient descent. Using an information-theoretic technique, the authors also provide results for the case when TD is applied to a single Markovian data stream where the algorithm’s updates can be severely biased. Their analysis seamlessly extends to the study of TD learning with eligibility traces and Q-learning for high-dimensional optimal stopping problems.

Sign in / Sign up

Export Citation Format

Share Document