Experimental analysis of eligibility traces strategies in temporal difference learning
by Jinsong Leng, Lakhmi Jain, Colin Fyfe
International Journal of Knowledge Engineering and Soft Data Paradigms (IJKESDP), Vol. 1, No. 1, 2009

Abstract: Temporal difference (TD) learning is a model-free reinforcement learning technique, which adopts an infinite horizon discount model and uses an incremental learning technique for dynamic programming. The state value function is updated in terms of sample episodes. Utilising eligibility traces is a key mechanism in enhancing the rate of convergence. TD(λ) represents the use of eligibility traces by introducing the parameter λ. However, the underlying mechanism of eligibility traces with an approximation function has not been well understood, either from theoretical point of view or from practical point of view. The TD(λ) method has been proved to be convergent with local tabular state representation. Unfortunately, proving convergence of TD(λ) with function approximation is still an important open theoretical question. This paper aims to investigate the convergence and the effects of different eligibility traces. In this paper, we adopt Sarsa(λ) learning control algorithm with a large, stochastic and dynamic simulation environment called SoccerBots. The state value function is represented by a linear approximation function known as tile coding. The performance metrics generated from the simulation system can be used to analyse the mechanism of eligibility traces.

Online publication date: Mon, 15-Dec-2008

The full text of this article is only available to individual subscribers or to users at subscribing institutions.

 
Existing subscribers:
Go to Inderscience Online Journals to access the Full Text of this article.

Pay per view:
If you are not a subscriber and you just want to read the full contents of this article, buy online access here.

Complimentary Subscribers, Editors or Members of the Editorial Board of the International Journal of Knowledge Engineering and Soft Data Paradigms (IJKESDP):
Login with your Inderscience username and password:

Username:        Password:          Forgotten your password?

Want to subscribe?
A subscription gives you complete access to all articles in the current issue, as well as to all articles in the previous three years (where applicable). See our Orders page to subscribe.

If you still need assistance, please email subs@inderscience.com