Title: Experimental analysis of eligibility traces strategies in temporal difference learning

Authors: Jinsong Leng, Lakhmi Jain, Colin Fyfe

Addresses: School of Electrical and Information Engineering, Knowledge Based Intelligent Engineering Systems Centre, University of South Australia, Mawson Lakes SA 5095, Australia. ' School of Electrical and Information Engineering, Knowledge Based Intelligent Engineering Systems Centre, University of South Australia, Mawson Lakes SA 5095, Australia. ' Applied Computational Intelligence Research Unit, University of the West of Scotland, 1 Westerfield, High Calside, PA2 6BY, Paisley, Scotland

Abstract: Temporal difference (TD) learning is a model-free reinforcement learning technique, which adopts an infinite horizon discount model and uses an incremental learning technique for dynamic programming. The state value function is updated in terms of sample episodes. Utilising eligibility traces is a key mechanism in enhancing the rate of convergence. TD(λ) represents the use of eligibility traces by introducing the parameter λ. However, the underlying mechanism of eligibility traces with an approximation function has not been well understood, either from theoretical point of view or from practical point of view. The TD(λ) method has been proved to be convergent with local tabular state representation. Unfortunately, proving convergence of TD(λ) with function approximation is still an important open theoretical question. This paper aims to investigate the convergence and the effects of different eligibility traces. In this paper, we adopt Sarsa(λ) learning control algorithm with a large, stochastic and dynamic simulation environment called SoccerBots. The state value function is represented by a linear approximation function known as tile coding. The performance metrics generated from the simulation system can be used to analyse the mechanism of eligibility traces.

Keywords: agents; temporal difference learning; eligibility traces; decision making; reinforcement learning; infinite horizon discount; incremental learning; dynamic simulation; soccer games; SoccerBots.

DOI: 10.1504/IJKESDP.2009.021982

International Journal of Knowledge Engineering and Soft Data Paradigms, 2009 Vol.1 No.1, pp.26 - 39

Published online: 15 Dec 2008 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article