Open AccessProceedings Article
On-line Reinforcement Learning Using Incremental Kernel-Based Stochastic Factorization
Doina Precup,Joelle Pineau,Andre Barreto +2 more
- 03 Dec 2012
- Vol. 25, pp 1484-1492
TL;DR: Theoretical results are presented showing that KBSF can approximate the value function that would be computed by conventional kernel-based learning with arbitrary precision, and the effectiveness of the proposed algorithm in the challenging three-pole balancing task is empirically demonstrated.
read more
Abstract: Kernel-based stochastic factorization (KBSF) is an algorithm for solving reinforcement learning tasks with continuous state spaces which builds a Markov decision process (MDP) based on a set of sample transitions. What sets KBSF apart from other kernel-based approaches is the fact that the size of its MDP is independent of the number of transitions, which makes it possible to control the trade-off between the quality of the resulting approximation and the associated computational cost. However, KBSF's memory usage grows linearly with the number of transitions, precluding its application in scenarios where a large amount of data must be processed. In this paper we show that it is possible to construct KBSF's MDP in a fully incremental way, thus freeing the space complexity of this algorithm from its dependence on the number of sample transitions. The incremental version of KBSF is able to process an arbitrary amount of data, which results in a model-based reinforcement learning algorithm that can be used to solve continuous MDPs in both off-line and on-line regimes. We present theoretical results showing that KBSF can approximate the value function that would be computed by conventional kernel-based learning with arbitrary precision. We empirically demonstrate the effectiveness of the proposed algorithm in the challenging three-pole balancing task, in which the ability to process a large number of transitions is crucial for success.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
•Proceedings Article
Incremental learning algorithms and applications
Alexander Gepperth,Barbara Hammer +1 more
- 01 Jan 2016
TL;DR: The concept of incremental learning is formalised, particular challenges which arise in this setting are discussed, and an overview about popular approaches, its theoretical foundations, and applications which emerged in the last years are given.
309
•Journal Article
Efficient non-linear control through neuroevolution
TL;DR: In this article, a novel neuroevolution method called CoSyNE that evolves networks at the level of weights was introduced for the pole-balancing problem, which was tested in difficult versions of the pole balancing problem.
137
•Journal Article
Regularized policy iteration with nonparametric function spaces
TL;DR: This work analyzes the statistical properties of REG-LSPI and provides an upper bound on the policy evaluation error and the performance loss of the policy returned by this method, the first work that provides such a strong guarantee for a nonparametric approximate policy iteration algorithm.
•Proceedings Article
Reinforcement Learning using Kernel-Based Stochastic Factorization
Andre Barreto,Doina Precup,Joelle Pineau +2 more
- 12 Dec 2011
TL;DR: A novel algorithm is introduced to improve the scalability of kernel-based reinforcement-learning by resorting to a special decomposition of a transition matrix, called stochastic factorization, to fix the size of the approximator while at the same time incorporating all the information contained in the data.
Practical kernel-based reinforcement learning
TL;DR: An algorithm that turns KBRL into a practical reinforcement learning tool that significantly outperforms other state-of-the-art reinforcement learning algorithms on the tasks studied and derive upper bounds for the distance between the value functions computed by KBRL and KBSF using the same data.
References
•Book
Reinforcement Learning: An Introduction
Richard S. Sutton,Andrew G. Barto +1 more
- 01 Jan 1988
TL;DR: This book provides a clear and simple account of the key ideas and algorithms of reinforcement learning, which ranges from the history of the field's intellectual foundations to the most recent developments and applications.
•Book
Markov Decision Processes: Discrete Stochastic Dynamic Programming
Martin L. Puterman
- 15 Apr 1994
TL;DR: Puterman as discussed by the authors provides a uniquely up-to-date, unified, and rigorous treatment of the theoretical, computational, and applied research on Markov decision process models, focusing primarily on infinite horizon discrete time models and models with discrete time spaces while also examining models with arbitrary state spaces, finite horizon models, and continuous time discrete state models.
12.3K
Markov Decision Processes
P. Whittle,M. L. Puterman +1 more
TL;DR: Markov Decision Processes covers recent research advances in such areas as countable state space models with average reward criterion, constrained models, and models with risk sensitive optimality criteria, and explores several topics that have received little or no attention in other books.
•Book
Introduction to Reinforcement Learning
Richard S. Sutton,Andrew G. Barto +1 more
- 01 Mar 1998
TL;DR: In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning.
7.7K
Neuronlike adaptive elements that can solve difficult learning control problems
Andrew G. Barto,Richard S. Sutton,Charles W. Anderson +2 more
- 01 Sep 1983
TL;DR: In this article, a system consisting of two neuron-like adaptive elements can solve a difficult learning control problem, where the task is to balance a pole that is hinged to a movable cart by applying forces to the cart base.
3.4K
Related Papers (5)
Richard S. Sutton,Andrew G. Barto +1 more
- 01 Jan 1988
Farbod Farshidian,Jonas Buchli +1 more
- 01 Jan 2014