Proceedings Article10.1109/ADPRL.2009.4927540
Eigenfunction approximation methods for linearly-solvable optimal control problems
Emanuel Todorov
- 15 May 2009
- pp 161-168
TL;DR: A general class of nonlinear stochastic optimal control problems which can be reduced to computing the principal eigenfunction of a linear operator is identified and function approximation methods exploiting this inherent linearity are developed.
read more
Abstract: We have identified a general class of nonlinear stochastic optimal control problems which can be reduced to computing the principal eigenfunction of a linear operator. Here we develop function approximation methods exploiting this inherent linearity. First we discretize the time axis in a novel way, yielding an integral operator that approximates not only our control problems but also more general elliptic PDEs. The eigenfunction problem is then approximated with a finite-dimensional eigenvector problem - by discretizing the state space, or by projecting on a set of adaptive bases evaluated at a set of collocation states. Solving the resulting eigenvector problem is faster than applying policy or value iteration. The bases are adapted via Levenberg-Marquardt minimization with guaranteed convergence. The collocation set can also be adapted so as to focus the approximation on a region of interest. Numerical results on test problems are provided.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
General duality between optimal control and estimation
Emanuel Todorov
- 01 Dec 2008
TL;DR: This work obtains a more natural form of LQG duality by replacing the Kalman-Bucy filter with the information filter and generalizes this result to non-linear stochastic systems, discrete stochastics systems, and deterministic systems.
•Proceedings Article
Inverse Optimal Control with Linearly-Solvable MDPs
Krishnamurthy Dvijotham,Emanuel Todorov +1 more
- 21 Jun 2010
TL;DR: Unlike all prior IRL algorithms which assume pre-existing features, this work study feature adaptation and shows that such adaptation is essential in continuous state spaces.
Infinite-Horizon Model Predictive Control for Periodic Tasks with Contacts
Tom Erez,Yuval Tassa,Emanuel Todorov +2 more
- 27 Jun 2011
TL;DR: This paper uses offline optimization to find the limit-cycle solution of an infinite-horizon average-cost optimal-control task, and compute a local quadratic approximation of the Value function around this limit cycle that is used as the terminal cost of an online MPC.
99
Merging in Congested Freeway Traffic Using Multipolicy Decision Making and Passive Actor-Critic Learning
Tomoki Nishi,Prashant Doshi,Danil V. Prokhorov +2 more
- 20 Mar 2019
TL;DR: In this article, a method for freeway merge based on multipolicy decision making coupled with a reinforcement learning technique called passive actor-critic (pAC) is presented, which learns with less knowledge of the system and without active exploration.
45
Freeway Merging in Congested Traffic based on Multipolicy Decision Making with Passive Actor Critic.
TL;DR: A method for the freeway merging based on multi-policy decision making with a reinforcement learning method called pAC, which learns with less knowledge of the system and without active exploration to achieve 92% success rate to merge into a freeway, which is comparable to human decision making.
40
References
•Proceedings Article
Linearly-solvable Markov decision problems
Emanuel Todorov
- 04 Dec 2006
TL;DR: A class of MPDs which greatly simplify Reinforcement Learning, which have discrete state spaces and continuous control spaces and enable efficient approximations to traditional MDPs.
General duality between optimal control and estimation
Emanuel Todorov
- 01 Dec 2008
TL;DR: This work obtains a more natural form of LQG duality by replacing the Kalman-Bucy filter with the information filter and generalizes this result to non-linear stochastic systems, discrete stochastics systems, and deterministic systems.
Numerical Methods for Stochastic Control Problems in Continuous Time (Harold J. Kushner and Paul G. Dupuis)
TL;DR: Numerical methods for stochastic control problems in continuous time to help people facing with some harmful virus inside their desktop computer to read a good book with a cup of coffee in the afternoon.
138
A Variational Approach to Nonlinear Estimation
Sanjoy K. Mitter,Nigel J. Newton +1 more
TL;DR: Regular conditional versions of the forward and inverse Bayes formula are shown to have dual variational characterizations involving the minimization of apparent information and the maximization of compatible information, according to which Bayes' formula and its inverse are optimal information processors.
Linear theory for control of nonlinear stochastic systems.
TL;DR: The role of noise and the issue of efficient computation in stochastic optimal control problems are addressed and a class of nonlinear control problems that can be formulated as a path integral and where the noise plays the role of temperature is considered.