formulation in latent space, supports long-term prediction of image sequences
and exhibits strong performance on a variety of complex control problems.,
archivePrefix = arXiv, arxivId = 1506.07365, author = Watter, Manuel and
Springenberg, Jost Tobias and Boedecker, Joschka and Riedmiller, Martin,
booktitle = Advances in Neural Information Processing Systems (NIPS), eprint =
1506.07365, file = ::;::, issn = 10495258, pages = 2728–2736, title = Embed to
Control: A Locally Linear Latent Dynamics Model for Control from Raw Images,
url = https://arxiv.org/pdf/1506.07365.pdf http://arxiv.org/abs/1506.07365,
year = 2015 @inproceedingslange2010deep, abstract = This paper discusses the
effectiveness of deep auto-encoding neural nets in visual reinforcement
learning (RL) tasks. We describe a new algorithm and give results on
succesfully learning policies directly on synthesized and real images without
a predefined image processing. Fur-thermore, we present a thorough evaluation
of the learned feature spaces., author = Lange, Sascha and Riedmiller, Martin
A, booktitle = European Symposium on Artificial Neural Networks (ESANN), file
= ::, organization = Citeseer, title = Deep learning of visual control
policies., url =
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.226.6898&rep=rep1&type=pdf,
year = 2010 @articlepopov17stacking, abstract = Deep learning and
reinforcement learning methods have recently been used to solve a variety of
problems in continuous control domains. An obvious application of these
techniques is dexterous manipulation tasks in robotics which are difficult to
solve using traditional control theory or hand-engineered approaches. One
example of such a task is to grasp an object and precisely stack it on
another. Solving this difficult and practically relevant problem in the real
world is an important long-term goal for the field of robotics. Here we take a
step towards this goal by examining the problem in simulation and providing
models and techniques aimed at solving it. We introduce two extensions to the
Deep Deterministic Policy Gradient algorithm (DDPG), a model-free Q-learning
based method, which make it significantly more data-efficient and scalable.
Our results show that by making extensive use of off-policy data and replay,
it is possible to find control policies that robustly grasp objects and stack
them. Further, our results hint that it may soon be feasible to train
successful stacking policies by collecting interactions on real robots.,
archivePrefix = arXiv, arxivId = 1704.03073, author = Popov, Ivaylo and Heess,
Nicolas and Lillicrap, Timothy and Hafner, Roland and Barth-maron, Gabriel and
Vecerik, Matej and Lampe, Thomas and Tassa, Yuval and Erez, Tom and
Riedmiller, Martin and Deepmind, Martin Riedmiller, eprint = 1704.03073, file
= ::, journal = CoRR, keywords = popov2017stacking, title = Data-efficient
Deep Reinforcement Learning for Dexterous Manipulation, url =
https://arxiv.org/pdf/1704.03073.pdf, volume = abs/1704.0, year = 2017
@inproceedingsrauber2017hindsight, abstract = Goal-conditional policies allow
reinforcement learning agents to pursue specific goals during different
episodes. In addition to their potential to generalize desired behavior to
unseen goals, such policies may also help in defining options for arbi-trary
subgoals, enabling higher-level planning. While trying to achieve a specific
goal, an agent may also be able to exploit information about the degree to
which it has achieved alternative goals. Reinforcement learning agents have
only recently been endowed with such capacity for hindsight, which is highly
valuable in environ-ments with sparse rewards. In this paper, we show how
hindsight can be introduced to likelihood-ratio policy gradient methods,
generalizing this capacity to an entire class of highly successful algorithms.
Our preliminary experiments suggest that hindsight may increase the sample
efficiency of policy gradient methods., author = Rauber, Paulo and Mutz,
Filipe and Schmidhuber, Juergen Jürgen, booktitle = CoRR, file = ::, title =
Hindsight policy gradients, url = https://arxiv.org/pdf/1711.06006.pdf, volume
= abs/1711.0, year = 2017 @inproceedingslillicrap2015continuous, abstract = We
adapt the ideas underlying the success of Deep Q-Learning to the continuous
action domain. We present an actor-critic, model-free algorithm based on the
de-terministic policy gradient that can operate over continuous action spaces.
Using the same learning algorithm, network architecture and hyper-parameters,
our al-gorithm robustly solves more than 20 simulated physics tasks, including
classic problems such as cartpole swing-up, dexterous manipulation, legged
locomotion and car driving. Our algorithm is able to find policies whose
performance is com-petitive with those found by a planning algorithm with full
access to the dynamics of the domain and its derivatives. We further
demonstrate that for many of the tasks the algorithm can learn policies ” end-
to-end ” : directly from raw pixel in-puts., archivePrefix = arXiv, arxivId =
cs/9605103, author = Lillicrap, Timothy P and Hunt, Jonathan J and Pritzel,
Alexander and Heess, Nicolas and Erez, Tom and Tassa, Yuval and Silver, David
and Wierstra, Daan, booktitle = International Conference on Learning
Representations (ICLR), doi = 10.1613/jair.301, eprint = 9605103, file =
::;:Users/ashvin/code/kindlize/pdfs/deeprl/continuous-control-
Lillicrap16.pdf:pdf, isbn = 0-7803-3213-X, issn = 10769757, pmid = 17255001,
primaryClass = cs, title = Continuous control with deep reinforcement
learning, url = https://arxiv.org/pdf/1509.02971.pdf, year = 2016
@inproceedingskober2008mp, abstract = Many motor skills in humanoid robotics
can be learned using parametrized motor primitives as done in imitation
learning. However, most interesting motor learn-ing problems are high-
dimensional reinforcement learning problems often beyond the reach of current
methods. In this paper, we extend previous work on policy learning from the
immediate reward case to episodic reinforcement learning. We show that this
results in a general, common framework also connected to pol-icy gradient
methods and yielding a novel algorithm for policy learning that is
particularly well-suited for dynamic motor primitives. The resulting algorithm
is an EM-inspired algorithm applicable to complex motor learning tasks. We
compare this algorithm to several well-known parametrized policy search
methods and show that it outperforms them. We apply it in the context of motor
learning and show that it can learn a complex Ball-in-a-Cup task using a real
Barrett WAM TM robot arm., author = Kober, Jens and Peter, J., booktitle =
Advances in Neural Information Processing Systems (NIPS), doi =
10.1007/978-3-319-03194-1_4, file = ::, isbn = 1099401052236, issn = 1610742X,
pages = 83–117, title = Policy search for motor primitives in robotics, url =
http://papers.nips.cc/paper/3545-policy-search-for-motor-primitives-in-
robotics.pdf, volume = 97, year = 2008 @inproceedingsdeisenroth2011pilco,
abstract = In this paper, we introduce pilco, a practi-cal, data-efficient
model-based policy search method. Pilco reduces model bias, one of the key
problems of model-based reinforce-ment learning, in a principled way. By
learn-ing a probabilistic dynamics model and ex-plicitly incorporating model
uncertainty into long-term planning, pilco can cope with very little data and
facilitates learning from scratch in only a few trials. Policy evaluation is
performed in closed form using state-of-the-art approximate inference.
Furthermore, policy gradients are computed analytically for policy
improvement. We report unprece-dented learning efficiency on challenging and
high-dimensional control tasks., author = Deisenroth, Marc Peter and
Rasmussen, Carl Edward, booktitle = International Conference on Machine
Learning (ICML), file = ::, pages = 465–472, title = PILCO: A model-based and
data-efficient approach to policy search, url =
http://mlg.eng.cam.ac.uk/pub/pdf/DeiRas11.pdf, year = 2011
@inproceedingsgu2016naf, abstract = Model-free reinforcement learning has been
successfully applied to a range of challenging problems, and has recently been
extended to handle large neural network policies and value functions. However,
the sample complexity of model-free algorithms, particularly when using high-
dimensional function approximators, tends to limit their applicability to
physical systems. In this paper, we explore algorithms and representations to
reduce the sample complexity of deep reinforcement learning for continuous
control tasks. We propose two complementary techniques for improving the
efficiency of such algorithms. First, we derive a continuous variant of the
Q-learning algorithm, which we call normalized adantage functions (NAF), as an
alternative to the more commonly used policy gradient and actor-critic
methods. NAF representation allows us to apply Q-learning with experience
replay to continuous tasks, and substantially improves performance on a set of
simulated robotic control tasks. To further improve the efficiency of our
approach, we explore the use of learned models for accelerating model-free
reinforcement learning. We show that iteratively refitted local linear models
are especially effective for this, and demonstrate substantially faster
learning on domains where such models are applicable., archivePrefix = arXiv,
arxivId = 1603.00748, author = Gu, Shixiang and Lillicrap, Timothy and
Sutskever, Ilya and Levine, Sergey, booktitle = International Conference on
Machine Learning (ICML), doi = 10.3390/robotics2030122, eprint = 1603.00748,
file = :Users/ashvin/code/kindlize/pdfs/deeprl/continuous-deepq-qlearning-
with-model-based-acceleration-Gu16.pdf:pdf;::, isbn = 3405062780, issn = null,
pmid = 21487784, title = Continuous Deep Q-Learning with Model-based
Acceleration, url = https://arxiv.org/pdf/1603.00748.pdf
http://arxiv.org/abs/1603.00748, year = 2016 @inproceedingsoh2015action,
abstract = Motivated by vision-based reinforcement learning (RL) problems, in
particular Atari games from the recent benchmark Aracade Learning Environment
(ALE), we consider spatio-temporal prediction problems where future
(image-)frames are dependent on control variables or actions as well as
previous frames. While not composed of natural scenes, frames in Atari games
are high-dimensional in size, can involve tens of objects with one or more
objects being controlled by the actions directly and many other objects being
influenced indirectly, can involve entry and departure of objects, and can
involve deep partial observability. We propose and evaluate two deep neural
network architectures that consist of encoding, action-conditional
transformation, and decoding layers based on convolutional neural networks and
recurrent neural networks. Experimental results show that the pro-posed
architectures are able to generate visually-realistic frames that are also
use-ful for control over approximately 100-step action-conditional futures in
some games. To the best of our knowledge, this paper is the first to make and
evaluate long-term predictions on high-dimensional video conditioned by
control inputs., author = Oh, Junhyuk and Guo, Xiaoxiao and Lee, Honglak and
Lewis, Richard and Singh, Satinder, booktitle = Advances in Neural Information
Processing Systems (NIPS), file = ::, title = Action-Conditional Video
Prediction using Deep Networks in Atari Games, url =
https://arxiv.org/pdf/1507.08750v1.pdf, year = 2015 @articlehenderson2017deep,
author = Henderson, Peter and Islam, Riashat and Bachman, Philip and Pineau,
Joelle and Precup, Doina and Meger, David, journal = arXiv preprint
arXiv:1709.06560, title = Deep reinforcement learning that matters, year =
2017 @inproceedingsinoue1985hand, author = Inoue, H and Inaba, M, booktitle =
Robotics Research: The First International Symposium, title = Hand-eye
coordination in rope handling, volume = 1, year = 1985 @bookwinograd72shrdlr,
author = Winograd, Terry, publisher = Academic Press, title = Understanding
Natural Language, year = 1972 @articlebilliards, author = Fragkiadaki,
Katerina and Agrawal, Pulkit and Levine, Sergey and Malik, Jitendra, journal =
International Conference on Learning Representations (ICLR), title = Learning
Visual Predictive Models of Physics for Playing Billiards, year = 2016
@articleburgess2018understanding, author = Burgess, Christopher P and Higgins,
Irina and Pal, Arka and Matthey, Loic and Watters, Nick and Desjardins,
Guillaume and Lerchner, Alexander, journal = arXiv preprint arXiv:1804.03599,
title = Understanding disentangling in $β$-VAE, year = 2018
@articlemachado2018eigenoption, author = Machado, Marlos C and Rosenbaum,
Clemens and Guo, Xiaoxiao and Liu, Miao and Tesauro, Gerald and Campbell,
Murray, journal = International Conference on Learning Representations (ICLR),
title = Eigenoption Discovery through the Deep Successor Representation, year
= 2018 @inproceedingskopicki2011learning, author = Kopicki, Marek and Zurek,
Sebastian and Stolkin, Rustam and Mörwald, Thomas and Wyatt, Jeremy, booktitle
= IEEE International Conference on Robotics and Automation (ICRA),
organization = IEEE, pages = 5722–5729, title = Learning to predict how rigid
objects behave under simple manipulation, year = 2011
@articlekuniyoshi1994learning, author = Kuniyoshi, Yasuo and Inaba, Masayuki
and Inoue, Hirochika, journal = IEEE transactions on robotics and automation,
number = 6, pages = 799–822, publisher = IEEE, title = Learning by watching:
Extracting reusable task knowledge from visual observation of human
performance, volume = 10, year = 1994 @articlejaderberg2016auxiliary, author =
Jaderberg, Max and Mnih, Volodymyr and Czarnecki, Wojciech Marian and Schaul,
Tom and Leibo, Joel Z and Silver, David and Kavukcuoglu, Koray, journal =
International Conference on Learning Representations (ICLR), title =
Reinforcement learning with unsupervised auxiliary tasks, year = 2017
@articlehansen2001completely, author = Hansen, Nikolaus and Ostermeier,
Andreas, journal = Evolutionary computation, number = 2, pages = 159–195,
publisher = MIT Press, title = Completely derandomized self-adaptation in
evolution strategies, volume = 9, year = 2001 @articlepinto2016curious, author
= Pinto, Lerrel and Gandhi, Dhiraj and Han, Yuanfeng and Park, Yong-Lae and
Gupta, Abhinav, journal = European Conference on Computer Vision (ECCV), title
= The Curious Robot: Learning Visual Representations via Physical
Interactions, year = 2016 @inproceedingslau2011automatic, author = Lau,
Manfred and Mitani, Jun and Igarashi, Takeo, booktitle = IEEE International
Conference on Robotics and Automation (ICRA), organization = IEEE, pages =
3733–3738, title = Automatic learning of pushing strategy for delivery of
irregular-shaped objects, year = 2011 @articlekingma2014adam, author = Kingma,
Diederik and Ba, Jimmy, journal = International Conference on Learning
Representations (ICLR), title = Adam: A method for stochastic optimization,
year = 2015 @articleponomarenko2015image, author = Ponomarenko, Nikolay and
Jin, Lina and Ieremeiev, Oleg and Lukin, Vladimir and Egiazarian, Karen and
Astola, Jaakko and Vozel, Benoit and Chehdi, Kacem and Carli, Marco and
Battisti, Federica and Others, journal = Signal Processing: Image
Communication, pages = 57–77, publisher = Elsevier, title = Image database
TID2013: Peculiarities, results and perspectives, volume = 30, year = 2015
@articleschmidhuber1992learning, author = Schmidhuber, Jürgen, journal =
Neural Computation, number = 6, pages = 863–879, publisher = MIT Press, title
= Learning factorial codes by predictability minimization, volume = 4, year =
1992 @inproceedingsabbeel2004apprenticeship, author = Abbeel, Pieter and Ng,
Andrew Y, booktitle = International Conference on Machine Learning (ICML),
pages = 1, title = Apprenticeship learning via inverse reinforcement learning,
year = 2004 @articleaksoy2011learning, author = Aksoy, Eren Erdal and Abramov,
Alexey and Dörr, Johannes and Ning, Kejun and Dellen, Babette and Wörgötter,
Florentin, journal = The International Journal of Robotics Research, pages =
0278364911410459, publisher = Sage Publications, title = Learning the
semantics of object–action relations by observation, year = 2011
@inproceedingssaha2006motion, author = Saha, Mitul and Isto, Pekka, booktitle
= IEEE International Conference on Robotics and Automation (ICRA),
organization = IEEE, pages = 2478–2484, title = Motion planning for robotic
manipulation of deformable linear objects, year = 2006 @articleha2018world,
author = Ha, David and Schmidhuber, Jürgen, journal = arXiv preprint
arXiv:1803.10122, title = World Models, year = 2018
@inproceedingsDBLP:conf/bmvc/KyriazisOA11, author = Kyriazis, Nikolaos and
Oikonomidis, Iason and Argyros, Antonis A, booktitle = British Machine Vision
Conference, {BMVC} 2011, Dundee, UK, August 29 - September 2, 2011.
Proceedings, doi = 10.5244/C.25.43, pages = 1–11, title = Binding Computer
Vision to Physics Based Simulation: The Case Study of a Bouncing Ball, url =
http://dx.doi.org/10.5244/C.25.43, year = 2011 @articlelee2013syntactic,
author = Lee, Kyuhwa and Su, Yanyu and Kim, Tae-Kyun and Demiris, Yiannis,
journal = Robotics and Autonomous Systems, number = 12, pages = 1323–1334,
publisher = Elsevier, title = A syntactic approach to robot imitation learning
using probabilistic activity grammars, volume = 61, year = 2013
@inproceedingskatz2008manipulating, author = Katz, Dov and Brock, Oliver,
booktitle = IEEE International Conference on Robotics and Automation (ICRA),
organization = IEEE, pages = 272–277, title = Manipulating articulated objects
with interactive perception, year = 2008 @articleflorensa2017stochastic,
author = Florensa, Carlos and Duan, Yan and Abbeel, Pieter, journal = arXiv
preprint arXiv:1704.03012, title = Stochastic neural networks for hierarchical
reinforcement learning, year = 2017 @articlemccloskey1983intuitive, author =
McCloskey, Michael, journal = Scientific american, number = 4, pages =
122–130, title = Intuitive physics, volume = 248, year = 1983
@inproceedingschi2022ropes, title=Iterative Residual Policy for Goal-
Conditioned Dynamic Manipulation of Deformable Objects, author=Chi, Cheng and
Burchfiel, Ben and Cousineau, Eric and Feng, Siyuan and Song, Shuran,
booktitle=RSS, year=2022, @articlepinto2015supersizing, author = Pinto, Lerrel
and Gupta, Abhinav, journal = IEEE International Conference on Robotics and
Automation (ICRA), title = Supersizing Self-supervision: Learning to Grasp
from 50K Tries and 700 Robot Hours, year = 2016
@articlepinto2017pushbygrasping, author = Pinto, Lerrel and Gupta, Abhinav,
journal = IEEE International Conference on Robotics and Automation (ICRA),
title = Learning to Push by Grasping: Using multiple tasks for effective
learning, year = 2017 @inproceedingsschulman2013warping, author = Schulman,
John and Gupta, Ankush and Venkatesan, Sibi and Tayson-Frederick, Mallory and
Abbeel, Pieter, booktitle = IEEE/RSJ International Conference on Intelligent
Robots and Systems (IROS), title = A Case Study of Trajectory Transfer Through
Non-Rigid Registration for a Simplified Suturing Scenario, year = 2013
@articlemericcli2015push, author = Meriçli, Tekin and Veloso, Manuela and
Ak$\backslash$in, H Levent, journal = Autonomous Robots, number = 3, pages =
317–329, publisher = Springer, title = Push-manipulation of complex passive
mobile objects using experimentally acquired motion models, volume = 38, year
= 2015 @inproceedingskietzmann2009neuro, author = Kietzmann, Tim C and
Riedmiller, Martin, booktitle = International Conference on Machine Learning
(ICML), organization = IEEE, pages = 311–316, title = The neuro slot car
racer: Reinforcement learning in a real world setting, year = 2009
@inproceedingschui2000tpsrpm, author = Chui, Haili and Rangarajan, Anand,
booktitle = Conference on Computer Vision and Pattern Recognition (CVPR),
title = A new algorithm for non-rigid point matching, year = 2000
@inproceedingstodorov2003unsupervised, author = Todorov, Emanuel and
Ghahramani, Zoubin, booktitle = Engineering in Medicine and Biology Society,
2003. Proceedings of the 25th Annual International Conference of the IEEE,
organization = IEEE, pages = 1750–1753, title = Unsupervised learning of
sensory-motor primitives, volume = 2, year = 2003
@articlezhang2018unreasonable, author = Zhang, Richard and Isola, Phillip and
Efros, Alexei A and Shechtman, Eli and Wang, Oliver, journal = Conference on
Computer Vision and Pattern Recognition (CVPR), title = The Unreasonable
Effectiveness of Deep Features as a Perceptual Metric, year = 2018

@articledesjardins2012disentangling, author = Desjardins, Guillaume and
Courville, Aaron and Bengio, Yoshua, journal = CoRR, title = Disentangling
factors of variation via generative entangling, volume = abs/1210.5, year =
2012 @bookcrowell2012introduction, author = Crowell, Richard H and Fox, Ralph
Hartzler, publisher = Springer Science & Business Media, title = Introduction
to knot theory, volume = 57, year = 2012 @articlelerer2016learning, author =
Lerer, Adam and Gross, Sam and Fergus, Rob, journal = International Conference
on Machine Learning (ICML), title = Learning Physical Intuition of Block
Towers by Example, year = 2016 @articlecheung2014discovering, author = Cheung,
Brian and Livezey, Jesse A and Bansal, Arjun K and Olshausen, Bruno A, journal
= arXiv preprint arXiv:1412.6583, title = Discovering hidden factors of
variation in deep networks, year = 2014
@inproceedingsschulman2013generalization, author = Schulman, John and Ho,
Jonathan and Lee, Cameron and Abbeel, Pieter, booktitle = Proceedings of the
16th International Symposium on Robotics Research (ISRR), title =
Generalization in robotic manipulation through the use of non-rigid
registration, year = 2013 @articlewolpert1995internal, author = Wolpert,
Daniel M and Ghahramani, Zoubin and Jordan, Michael I, journal = Science-AAAS-
Weekly Paper Edition, number = 5232, pages = 1880–1882, publisher = New York,
NY:[sn] 1880-, title = An internal model for sensorimotor integration, volume
= 269, year = 1995 @inproceedingskrizhevsky2012imagenet, author = Krizhevsky,
Alex and Sutskever, Ilya and Hinton, Geoffrey E, booktitle = NeurIPS, pages =
1097–1105, title = Imagenet classification with deep convolutional neural
networks, year = 2012 @articlehuh2016makes, author = Huh, Minyoung and
Agrawal, Pulkit and Efros, Alexei A, journal = arXiv preprint
arXiv:1608.08614, title = What makes ImageNet good for transfer learning?,
year = 2016 @articledogar2012planning, author = Dogar, Mehmet R and Srinivasa,
Siddhartha S, journal = Autonomous Robots, number = 3, pages = 217–236,
publisher = Springer, title = A planning framework for non-prehensile
manipulation under clutter and uncertainty, volume = 33, year = 2012
@articlelevine2017grasping, author = Levine, Sergey and Pastor, Peter and
Krizhevsky, Alex and Quillen, Deirdre, journal = International Journal of
Robotics Research, title = Learning Hand-Eye Coordination for Robotic Grasping
with Deep Learning and Large-Scale Data Collection, year = 2017
@articlehiggins2017darla, author = Higgins, Irina and Pal, Arka and Rusu,
Andrei A and Matthey, Loic and Burgess, Christopher P and Pritzel, Alexander
and Botvinick, Matthew and Blundell, Charles and Lerchner, Alexander, journal
= International Conference on Machine Learning (ICML), title = Darla:
Improving zero-shot transfer in reinforcement learning, year = 2017
@articlefujimoto2018td3, author = Fujimoto, Scott and van Hoof, Herke and
Meger, David, journal = ICML, title = Addressing Function Approximation Error
in Actor-Critic Methods, year = 2018 @articlemayer2008system, author = Mayer,
Hermann and Gomez, Faustino and Wierstra, Daan and Nagy, Istvan and Knoll,
Alois and Schmidhuber, Jürgen, journal = Advanced Robotics, number = 13-14,
pages = 1521–1537, publisher = Taylor & Francis, title = A system for robotic
heart surgery that learns to tie knots using recurrent neural networks, volume
= 22, year = 2008 @inproceedingshamrick2011internal, author = Hamrick, Jessica
and Battaglia, Peter and Tenenbaum, Joshua B, booktitle = Proceedings of the
33rd annual conference of the cognitive science society, organization =
Cognitive Science Society Austin, TX, pages = 1545–1550, title = Internal
physics models guide probabilistic judgments about object dynamics, year =
2011 @articlemayne2014model, abstract = Abstract This paper recalls a few past
achievements in Model Predictive Control, gives an overview of some current
developments and suggests a few avenues for future research. , author = Mayne,
David Q, doi = http://dx.doi.org/10.1016/j.automatica.2014.10.128, issn =
0005-1098, journal = Automatica, keywords = Model predictive control, number =
12, pages = 2967–2986, title = Model predictive control: Recent developments
and future promise, url =
http://www.sciencedirect.com/science/article/pii/S0005109814005160, volume =
50, year = 2014 @articlemnih2015human, author = Mnih, Volodymyr and
Kavukcuoglu, Koray and Silver, David and Rusu, Andrei A and Veness, Joel and
Bellemare, Marc G and Graves, Alex and Riedmiller, Martin and Fidjeland,
Andreas K and Ostrovski, Georg and Others, journal = Nature, number = 7540,
pages = 529–533, publisher = Nature Publishing Group, title = Human-level
control through deep reinforcement learning, volume = 518, year = 2015
@articlemottaghi2015newtonian, author = Mottaghi, Roozbeh and Bagherinezhad,
Hessam and Rastegari, Mohammad and Farhadi, Ali, journal = arXiv preprint
arXiv:1511.04048, title = Newtonian Image Understanding: Unfolding the
Dynamics of Objects in Static Images, year = 2015
@inproceedingsstilman2007manipulation, author = Stilman, Mike and Schamburek,
Jan-Ullrich and Kuffner, James and Asfour, Tamim, booktitle = IEEE
International Conference on Robotics and Automation (ICRA), organization =
IEEE, pages = 3327–3332, title = Manipulation planning among movable
obstacles, year = 2007 @articlehiggins2016beta, author = Higgins, Irina and
Matthey, Loic and Pal, Arka and Burgess, Christopher and Glorot, Xavier and
Botvinick, Matthew and Mohamed, Shakir and Lerchner, Alexander, journal =
International Conference on Learning Representations (ICLR), title = $$-VAE:
Learning basic visual concepts with a constrained variational framework, year
= 2017 @articleharuno2001mosaic, author = Haruno, Masahiko and Wolpert, David
H and Kawato, Mitsuo, journal = Neural computation, number = 10, pages =
2201–2220, publisher = MIT Press, title = Mosaic model for sensorimotor
learning and control, volume = 13, year = 2001 @inproceedingslenz2015deepMPC,
author = Lenz, Ian and Knepper, Ross and Saxena, Ashutosh, booktitle =
Robotics: Science and Systems (RSS), title = DeepMPC: Learning Deep Latent
Features for Model Predictive Control, year = 2015 @articlehopcroft1991case,
author = Hopcroft, John E and Kearney, Joseph K and Krafft, Dean B, journal =
The International Journal of Robotics Research, number = 1, pages = 41–50,
publisher = Sage Publications, title = A case study of flexible object
manipulation, volume = 10, year = 1991 @articlerusu2016sim, author = Rusu,
Andrei A and Vecerik, Matej and Rothörl, Thomas and Heess, Nicolas and
Pascanu, Razvan and Hadsell, Raia, journal = Conference on Robot Learning
(CoRL), title = Sim-to-real robot learning from pixels with progressive nets,
year = 2017 @techreportwinograd1971procedures, author = Winograd, Terry,
institution = DTIC Document, title = Procedures as a representation for data
in a computer program for understanding natural language, year = 1971
@inproceedings5459407, author = Brubaker, Marcus A and Sigal, L and Fleet, D
J, booktitle = Conference on Computer Vision and Pattern Recognition (CVPR),
doi = 10.1109/ICCV.2009.5459407, issn = 1550-5499, keywords = Biological
system modeling,Geometry,Gravity,Humans, month = sep, pages = 2389–2396, title
= Estimating contact dynamics, year = 2009 @articleeysenbach2018diayn, author
= Eysenbach, Benjamin and Gupta, Abhishek and Ibarz, Julian and Levine,
Sergey, journal = arXiv preprint arXiv:1802.06070, title = Diversity is All
You Need: Learning Skills without a Reward Function, year = 2018
@inproceedingsagarwal1997nonholonomic, author = Agarwal, Pankaj K and Latombe,
Jean-Claude and Motwani, Rajeev and Raghavan, Prabhakar, booktitle = IEEE
International Conference on Robotics and Automation, organization = Citeseer,
pages = 3124–3129, title = Nonholonomic path planning for pushing a disk among
obstacles, year = 1997 @inproceedingsconf/eccv/BhatSP02, author = Bhat, Kiran
S and Seitz, Steven M and Popovic, Jovan, booktitle = European Conference on
Computer Vision (ECCV), editor = Heyden, Anders and Sparr, Gunnar and Nielsen,
Mads and Johansen, Peter, isbn = 3-540-43745-2, keywords = dblp, pages =
551–565, publisher = Springer, series = Lecture Notes in Computer Science,
title = Computing the Physical Parameters of Rigid-Body Motion from Video.,
url = http://dblp.uni-trier.de/db/conf/eccv/eccv2002-1.html#BhatSP02, volume =
2350, year = 2002 @articlewahlstrom2015from, author = Wahlström, Niklas and
Schön, Thomas B and Deisenroth, Marc Peter, journal = CoRR, title = From
Pixels to Torques: Policy Learning with Deep Dynamical Models, volume =
abs/1502.0, year = 2015 @inproceedingswu2015galileo, author = Wu, Jiajun and
Yildirim, Ilker and Lim, Joseph J and Freeman, Bill and Tenenbaum, Josh,
booktitle = Advances in Neural Information Processing Systems (NIPS), pages =
127–135, title = Galileo: Perceiving Physical Object Properties by Integrating
a Physics Engine with Deep Learning, year = 2015 @articleargall2009survey,
author = Argall, Brenna D and Chernova, Sonia and Veloso, Manuela and
Browning, Brett, journal = Robotics and autonomous systems, number = 5, pages
= 469–483, publisher = Elsevier, title = A survey of robot learning from
demonstration, volume = 57, year = 2009 @articleshelhamer2016loss, author =
Shelhamer, Evan and Mahmoudieh, Parsa and Argus, Max and Darrell, Trevor,
journal = arXiv preprint arXiv:1612.07307, title = Loss is its own reward:
Self-supervision for reinforcement learning, year = 2016
@inproceedingsyamakawa2007one, author = Yamakawa, Yuji and Namiki, Akio and
Ishikawa, Masatoshi and Shimojo, Makoto, booktitle = IEEE/RSJ International
Conference on Intelligent Robots and Systems (IROS), organization = IEEE,
pages = 703–708, title = One-handed knotting of a flexible rope with a high-
speed multifingered hand having tactile sensors, year = 2007
@articlemaaten2008visualizing, author = van der Maaten, Laurens and Hinton,
Geoffrey, journal = Journal of Machine Learning Research (JMLR), number = Nov,
pages = 2579–2605, title = Visualizing data using t-SNE, volume = 9, year =
2008 @phdthesisbell2010flexible, author = Bell, Matthew, school = Dartmouth
College, Hanover, New Hampshire, title = Flexible object manipulation, year =
2010 @inproceedingsbellemare2016unifying, author = Bellemare, Marc and
Srinivasan, Sriram and Ostrovski, Georg and Schaul, Tom and Saxton, David and
Munos, Remi, booktitle = Advances in Neural Information Processing Systems
(NIPS), pages = 1471–1479, title = Unifying count-based exploration and
intrinsic motivation, year = 2016 @inproceedingsmaitin2010cloth, author =
Maitin-Shepard, Jeremy and Cusumano-Towner, Marco and Lei, Jinna and Abbeel,
Pieter, booktitle = IEEE International Conference on Robotics and Automation
(ICRA), organization = IEEE, pages = 2308–2315, title = Cloth grasp point
detection based on multiple-view geometric cues with application to robotic
towel folding, year = 2010 @inproceedingsmiller2011parametrized, author =
Miller, Stephen and Fritz, Mario and Darrell, Trevor and Abbeel, Pieter,
booktitle = IEEE International Conference on Robotics and Automation (ICRA),
organization = IEEE, pages = 4861–4868, title = Parametrized shape models for
clothing, year = 2011 @inproceedingssutton2011horde, abstract = Maintaining
accurate world knowledge in a complex and changing environment is a perennial
problem for robots and other artificial intelligence systems. Our architecture
for addressing this problem, called Horde, consists of a large number of
independent reinforcement learning sub-agents, or demons. Each demon is
responsible for answering a sin-gle predictive or goal-oriented question about
the world, thereby contributing in a factored, modular way to the sys-tem’s
overall knowledge. The questions are in the form of a value function, but each
demon has its own policy, re-ward function, termination function, and
terminal-reward function unrelated to those of the base problem. Learning
proceeds in parallel by all demons simultaneously so as to extract the maximal
training information from whatever ac-tions are taken by the system as a
whole. Gradient-based temporal-difference learning methods are used to learn
ef-ficiently and reliably with function approximation in this off-policy
setting. Horde runs in constant time and memory per time step, and is thus
suitable for learning online in real-time applications such as robotics. We
present results using Horde on a multi-sensored mobile robot to successfully
learn goal-oriented behaviors and long-term predictions from off-policy
experience. Horde is a significant incremental step towards a real-time
architecture for efficient learning of gen-eral knowledge from unsupervised
sensorimotor interaction., author = Sutton, Richard S and Modayil, Joseph and
Delp, Michael and Degris, Thomas and Pilarski, Patrick M and White, Adam and
Precup, Doina, booktitle = International Conference on Autonomous Agents and
Multiagent Systems (AAMAS), file = ::, keywords = Categories
and,Descriptors,Subject, pages = 761–768, title = Horde: A Scalable Real-time
Architecture for Learning Knowledge from Unsupervised Sensorimotor
Interaction, url = https://www.cs.swarthmore.edu/
meeden/DevelopmentalRobotics/horde1.pdf, volume = 10, year = 2011
@inproceedingsziebart2008maxent, abstract = Recent research has shown the
benefit of framing problems of imitation learning as solutions to Markov
Decision Problems. This approach reduces learning to the problem of recovering
a utility function that makes the behavior induced by a near-optimal policy
closely mimic demonstrated behavior. In this work, we develop a probabilistic
approach based on the principle of maximum entropy. Our approach provides a
well-defined, globally normalized distribution over decision sequences, while
providing the same performance guarantees as existing methods. We develop our
technique in the context of modeling real-world navigation and driving
behaviors where collected data is inherently noisy and imperfect. Our
probabilistic approach enables modeling of route preferences as well as a
powerful new approach to inferring destinations and routes based on partial
trajectories. Copyright 2008., archivePrefix = arXiv, arxivId =
arXiv:1507.04888v2, author = Ziebart, Brian D and Maas, Andrew and Bagnell, J
Andrew and Dey, Anind K, booktitle = AAAI Conference on Artificial
Intelligence, eprint = arXiv:1507.04888v2, file = ::, isbn = 9781577353683
(ISBN), issn = 10450823, keywords = Artificial intelligence,Bionics,Driving
behaviors,Existing methods,Imitation learnings,Inverse problems,Maximum
entropies,New approaches,Optimal policies,Partial trajectories,Performance
guarantees,Principle of Maximum entropies,Probabilistic
approaches,Probability,Probability distributions,Reinforcement,Reinforcement
learning,Route preferences,Utility functions, pages = 1433–1438, title =
Maximum Entropy Inverse Reinforcement Learning., url =
https://www.aaai.org/Papers/AAAI/2008/AAAI08-227.pdf
http://www.scopus.com/inward/record.url?eid=2-s2.0-57749097473&partnerID=40%5Cnhttp://www.aaai.org/Papers/AAAI/2008/AAAI08-227.pdf,
year = 2008 @articleGu2016b, abstract = Reinforcement learning holds the
promise of enabling autonomous robots to learn large repertoires of behavioral
skills with minimal human intervention. However, robotic applications of
reinforcement learning often compromise the autonomy of the learning process
in favor of achieving training times that are practical for real physical
systems. This typically involves introducing hand-engineered policy
representations and human-supplied demonstrations. Deep reinforcement learning
alleviates this limitation by training general-purpose neural network
policies, but applications of direct deep reinforcement learning algorithms
have so far been restricted to simulated settings and relatively simple tasks,
due to their apparent high sample complexity. In this paper, we demonstrate
that a recent deep reinforcement learning algorithm based on off-policy
training of deep Q-functions can scale to complex 3D manipulation tasks and
can learn deep neural network policies efficiently enough to train on real
physical robots. We demonstrate that the training times can be further reduced
by parallelizing the algorithm across multiple robots which pool their policy
updates asynchronously. Our experimental evaluation shows that our method can
learn a variety of 3D manipulation skills in simulation and a complex door
opening skill on real robots without any prior demonstrations or manually
designed representations., archivePrefix = arXiv, arxivId = 1610.00633, author
= Gu, Shixiang and Holly, Ethan and Lillicrap, Timothy and Levine, Sergey, doi
= 10.1038/nature20101, eprint = 1610.00633, file = ::, isbn = 0896-6273, issn
= 0028-0836, journal = IEEE International Conference on Robotics and
Automation (ICRA), pmid = 26774160, title = Deep Reinforcement Learning for
Robotic Manipulation with Asynchronous Off-Policy Updates, url =
https://arxiv.org/pdf/1610.00633.pdf http://arxiv.org/abs/1610.00633, year =
2017 @articleDeisenroth2011, abstract = âOver the last years, there has been
substantial progress in robust manipulation in unstructured environments. The
long-term goal of our work is to get away from precise, but very expensive
robotic systems and to develop affordable, potentially imprecise, self-
adaptive manipulator systems that can interactively perform tasks such as
playing with children. In this paper, we demonstrate how a low-cost off-the-
shelf robotic system can learn closed-loop policies for a stacking task in
only a handful of trialsâfrom scratch. Our manipulator is inaccurate and
provides no pose feedback. For learning a controller in the work space of a
Kinect-style depth camera, we use a model-based reinforcement learning
technique. Our learning method is data efficient, reduces model bias, and
deals with several noise sources in a principled way during long-term
planning. We present a way of incorporating state-space constraints into the
learning process and analyze the learning gain by exploiting the sequential
structure of the stacking task., author = Deisenroth, Marc Peter and
Rasmussen, Carl Edward and Fox, Dieter, file = ::, journal = Robotics: Science
and Systems (RSS), pages = 57–64, title = Learning to Control a Low-Cost
Manipulator using Data-Efficient Reinforcement Learning, url =
http://www.roboticsproceedings.org/rss07/p08.pdf, volume = VII, year = 2011
@inproceedingsross2011dagger, abstract = Sequential prediction problems such
as imitation learning, where future observations depend on previous
predictions (actions), violate the com-mon i.i.d. assumptions made in
statistical learn-ing. This leads to poor performance in theory and often in
practice. Some recent approaches (Daumé III et al., 2009; Ross and Bagnell,
2010) provide stronger guarantees in this setting, but re-main somewhat
unsatisfactory as they train either non-stationary or stochastic policies and
require a large number of iterations. In this paper, we propose a new
iterative algorithm, which trains a stationary deterministic policy, that can
be seen as a no regret algorithm in an online learning set-ting. We show that
any such no regret algorithm, combined with additional reduction assumptions,
must find a policy with good performance under the distribution of
observations it induces in such sequential settings. We demonstrate that this
new approach outperforms previous approaches on two challenging imitation
learning problems and a benchmark sequence labeling problem., author = Ross,
Stéphane and Gordon, Geoffrey J and Bagnell, J Andrew, booktitle =
International Conference on Artificial Intelligence and Statistics (AISTATS),
file = ::;::, title = A Reduction of Imitation Learning and Structured
Prediction to No-Regret Online Learning, url =
https://arxiv.org/pdf/1011.0686.pdf https://www.cs.cmu.edu/
sross1/publications/Ross-AIStats11-NoRegret.pdf, year = 2011
@articleKaelbling2011, abstract = â In this paper we outline an approach to
the integration of task planning and motion planning that has the following
key properties: It is aggressively hierarchical. It makes choices and commits
to them in a top-down fashion in an attempt to limit the length of plans that
need to be constructed, and thereby exponentially decrease the amount of
search required. Importantly, our approach also limits the need to project the
effect of actions into the far future. It operates on detailed, continuous
geometric representations and partial symbolic descriptions. It does not
require a complete symbolic representation of the input geometry or of the
geometric effect of the task-level operations., author = Kaelbling, Leslie
Pack and Lozano-Perez, Tomas, doi = 10.1109/ICRA.2011.5980391, file = ::, isbn
= 9781612843865, issn = 10504729, journal = IEEE International Conference on
Robotics and Automation, pages = 1470–1477, title = Hierarchical task and
motion planning in the now, url =
http://people.csail.mit.edu/lpk/papers/hpn2.pdf
http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5980391, year =
2011 @inproceedingstodorov12mujoco, abstract = We describe a new physics
engine tailored to model-based control. Multi-joint dynamics are represented
in generalized coordinates and computed via recursive algorithms. Contact
responses are computed via efficient new algorithms we have developed, based
on the modern velocity-stepping approach which avoids the difficulties with
spring-dampers. Models are specified using either a high-level C++ API or an
intuitive XML file format. A built-in compiler transforms the user model into
an optimized data structure used for runtime computation. The engine can
compute both forward and inverse dynamics. The latter are well-definedevenin
the presence of contacts and equality constraints. Themodel can include tendon
wrapping as well as actuator activation states (e.g. pneumatic cylinders or
muscles). To facilitate optimal control applications and in particular
sampling and finite differencing, the dynamics can be evaluated for different
states and controls in parallel. Around 400,000 dynamics evaluations per
second are possible on a 12-core machine, for a 3D homanoid with 18 dofs and 6
active contacts. We have already used the engine in a number of control
applications. It will soon be made publicly available., author = Todorov,
Emanuel and Erez, Tom and Tassa, Yuval, booktitle = IEEE/RSJ International
Conference on Intelligent Robots and Systems (IROS), doi =
10.1109/IROS.2012.6386109, file = ::, isbn = 9781467317375, issn = 21530858,
pages = 5026–5033, title = MuJoCo: A physics engine for model-based control,
url = https://homes.cs.washington.edu/ todorov/papers/TodorovIROS12.pdf, year
= 2012 @inproceedingskim2013apid, abstract = We propose a Learning from
Demonstration (LfD) algorithm which leverages ex-pert data, even if they are
very few or inaccurate. We achieve this by using both expert data, as well as
reinforcement signals gathered through trial-and-error inter-actions with the
environment. The key idea of our approach, Approximate Policy Iteration with
Demonstration (APID), is that expert’s suggestions are used to de-fine linear
constraints which guide the optimization performed by Approximate Policy
Iteration. We prove an upper bound on the Bellman error of the estimate
computed by APID at each iteration. Moreover, we show empirically that APID
outperforms pure Approximate Policy Iteration, a state-of-the-art LfD
algorithm, and supervised learning in a variety of scenarios, including when
very few and/or suboptimal demonstrations are available. Our experiments
include simulations as well as a real robot path-finding task., author = Kim,
Beomjoon and Farahmand, Amir-Massoud and Pineau, Joelle and Precup, Doina,
booktitle = Advances in Neural Information Processing Systems (NIPS), file =
::, title = Learning from Limited Demonstrations, url =
https://papers.nips.cc/paper/4918-learning-from-limited-demonstrations.pdf,
year = 2013 @inproceedingsschaul2015uva, abstract = Value functions are a core
component of rein- forcement learning systems. The main idea is to to
construct a single function approximator V (s; $\theta$) that estimates the
long-term reward from any state s, using parameters $\theta$. In this paper we
introduce universal value function approx- imators (UVFAs) V (s, g; $\theta$)
that generalise not just over states s but also over goals g. We de- velop an
efficient technique for supervised learn- ing of UVFAs, by factoring observed
values into separate embedding vectors for state and goal, and then learning a
mapping from s and g to these factored embedding vectors. We show how this
technique may be incorporated into a re- inforcement learning algorithm that
updates the UVFAsolely from observed rewards. Finally, we demonstrate that a
UVFAcan successfully gener- alise to previously unseen goals. 1., author =
Schaul, Tom and Horgan, Daniel and Gregor, Karol and Silver, David, booktitle
= International Conference on Machine Learning (ICML), file = ::, isbn =
9781510810587, pages = 1312–1320, title = Universal Value Function
Approximators, url = http://proceedings.mlr.press/v37/schaul15.pdf
http://jmlr.org/proceedings/papers/v37/schaul15.html, year = 2015
@inproceedingsfinn2016deep, abstract = Reinforcement learning provides a
powerful and flexible framework for automated acquisition of robotic motion
skills. However, applying reinforcement learning requires a sufficiently
detailed representation of the state, including the configuration of task-
relevant objects. We present an approach that automates state-space
construction by learning a state representation directly from camera images.
Our method uses a deep spatial autoencoder to acquire a set of feature points
that describe the environment for the current task, such as the positions of
objects, and then learns a motion skill with these feature points using an
efficient reinforcement learning method based on local linear models. The
resulting controller reacts continuously to the learned feature points,
allowing the robot to dynamically manipulate objects in the world with closed-
loop control. We demonstrate our method with a PR2 robot on tasks that include
pushing a free-standing toy block, picking up a bag of rice using a spatula,
and hanging a loop of rope on a hook at various positions. In each task, our
method automatically learns to track task-relevant objects and manipulate
their configuration with the robot’s arm., archivePrefix = arXiv, arxivId =
1509.06113, author = Finn, Chelsea and Tan, Xin Yu and Duan, Yan and Darrell,
Trevor and Levine, Sergey and Abbeel, Pieter, booktitle = IEEE International
Conference on Robotics and Automation (ICRA), doi = 10.1109/ICRA.2016.7487173,
eprint = 1509.06113, file = :Users/ashvin/code/kindlize/pdfs/deeprl/spatial-
autoencoders-Finn15.pdf:pdf, isbn = 9781467380263, issn = 10504729,
organization = IEEE, pages = 512–519, title = Deep spatial autoencoders for
visuomotor learning, volume = 2016-June, year = 2016
@inproceedingsschulman2015trpo, abstract = We describe a iterative procedure
for optimizing policies, with guaranteed monotonic improve-ment. By making
several approximations to the theoretically-justified procedure, we develop a
practical algorithm, called Trust Region Policy Optimization (TRPO). This
algorithm is similar to natural policy gradient methods and is effec-tive for
optimizing large nonlinear policies such as neural networks. Our experiments
demon-strate its robust performance on a wide variety of tasks: learning
simulated robotic swimming, hopping, and walking gaits; and playing Atari
games using images of the screen as input. De-spite its approximations that
deviate from the theory, TRPO tends to give monotonic improve-ment, with
little tuning of hyperparameters., archivePrefix = arXiv, arxivId =
1502.05477, author = Schulman, John and Levine, Sergey and Moritz, Philipp and
Jordan, Michael I. and Abbeel, Pieter, booktitle = International Conference on
Machine Learning (ICML), doi = 10.1063/1.4927398, eprint = 1502.05477, file =
::;::, isbn = 0375-9687, issn = 2158-3226, title = Trust Region Policy
Optimization, url = https://arxiv.org/pdf/1502.05477.pdf
http://arxiv.org/abs/1502.05477, year = 2015
@inproceedingskolev2015physically, abstract = â Successful model based
control relies heavily on proper system identification and accurate state
estimation. We present a framework for solving these problems in the context
of robotic control applications. We are particularly interested in robotic
manipulation tasks, which are especially hard due to the non-linear nature of
contact phenomena. We developed a solution that solves both the problems of
estimation and system identification jointly. We show that these two problems
are difficult to solve separately in the presence of discontinuous phenomena
such as contacts. The problem is posed as a joint optimization across both
trajectory and model parameters and solved via Newton’s method. We present
several challenges we encountered while modeling contacts and performing state
estimation and propose solutions within the MuJoCo physics engine. We present
experimental results performed on our manip-ulation system consisting of 3-DOF
Phantom Haptic Devices, turned into finger manipulators. Cross-validation
between dif-ferent datasets, as well as leave-one-out cross-validation show
that our method is robust and is able to accurately explain sensory data.,
author = Kolev, Svetoslav and Todorov, Emanuel, booktitle = IEEE-RAS
International Conference on Humanoid Robots (Humanoids), file = ::,
organization = IEEE, pages = 1036–1043, title = Physically consistent state
estimation and system identification for contacts, url =
http://homes.cs.washington.edu/ todorov/papers/KolevHumanoids15.pdf, year =
2015 @inproceedingspeters2010reps, abstract = Policy search is a successful
approach to reinforcement learning. However, policy improvements often result
in the loss of information. Hence, it has been marred by premature convergence
and implausible solutions. As first suggested in the context of covariant
policy gradients (Bagnell and Schneider 2003), many of these problems may be
addressed by constraining the infor- mation loss. In this paper, we continue
this path of rea- soning and suggest the Relative Entropy Policy Search (REPS)
method. The resulting method differs signif- icantly from previous policy
gradient approaches and yields an exact update step. It works well on typical
reinforcement learning benchmark problems. Introduction, author = Peters, Jan
and Mülling, Katharina and Altün, Yasemin, booktitle = AAAI Conference on
Artificial Intelligence, file = ::, pages = 1607–1612, title = Relative
Entropy Policy Search, url =
https://pdfs.semanticscholar.org/ff47/526838ce85d77a50197a0c5f6ee5095156aa.pdf
http://www-clmc.usc.edu/publications/P/Peters_POTTNCOAIPGAT_2010.pdf, year =
2010 @inproceedingsho2016gail, abstract = Consider learning a policy from
example expert behavior, without interaction with the expert or access to
reinforcement signal. One approach is to recover the expert’s cost function
with inverse reinforcement learning, then extract a policy from that cost
function with reinforcement learning. This approach is indirect and can be
slow. We propose a new general framework for directly extracting a policy from
data, as if it were obtained by reinforcement learning following inverse
reinforcement learning. We show that a certain instantiation of our framework
draws an analogy between imitation learning and generative adversarial
networks, from which we derive a model-free imitation learning algorithm that
obtains signif-icant performance gains over existing model-free methods in
imitating complex behaviors in large, high-dimensional environments., author =
Ho, Jonathan and Ermon, Stefano, booktitle = Advances in Neural Information
Processing Systems (NIPS), file = ::, title = Generative Adversarial Imitation
Learning, url = https://arxiv.org/pdf/1606.03476.pdf, year = 2016
@inproceedingschen2016infogan, abstract = This paper describes InfoGAN, an
information-theoretic extension to the Generative Adversarial Network that is
able to learn disentangled representations in a completely unsupervised
manner. InfoGAN is a generative adversarial network that also maximizes the
mutual information between a small subset of the latent variables and the
observation. We derive a lower bound to the mutual information objective that
can be optimized efficiently, and show that our training procedure can be
interpreted as a variation of the Wake-Sleep algorithm. Specifically, InfoGAN
successfully disentangles writing styles from digit shapes on the MNIST
dataset, pose from lighting of 3D rendered images, and background digits from
the central digit on the SVHN dataset. It also discovers visual concepts that
include hair styles, presence/absence of eyeglasses, and emotions on the
CelebA face dataset. Experiments show that InfoGAN learns interpretable
representations that are competitive with representations learned by existing
fully supervised methods., archivePrefix = arXiv, arxivId = 1606.03657, author
= Chen, Xi and Duan, Yan and Houthooft, Rein and Schulman, John and Sutskever,
Ilya and Abbeel, Pieter, booktitle = Advances in Neural Information Processing
Systems (NIPS), eprint = 1606.03657, file =
:Users/ashvin/code/kindlize/pdfs/deeprl/infogan-Chen16.pdf:pdf, pages =
2172–2180, title = Infogan: Interpretable representation learning by
information maximizing generative adversarial nets, url =
http://arxiv.org/abs/1606.03657, year = 2016 @articlesilver2016alphago,
abstract = The game of Go has long been viewed as the most challenging of
classic games for artificial intelligence owing to its enormous search space
and the difficulty of evaluating board positions and moves. Here we introduce
a new approach to computer Go that uses âvalue networks’ to evaluate board
positions and âpolicy networks’ to select moves. These deep neural networks
are trained by a novel combination of supervised learning from human expert
games, and reinforcement learning from games of self-play. Without any
lookahead search, the neural networks play Go at the level of state-of-the-art
Monte Carlo tree search programs that simulate thousands of random games of
self-play. We also introduce a new search algorithm that combines Monte Carlo
simulation with value and policy networks. Using this search algorithm, our
program AlphaGo achieved a 99.8% winning rate against other Go programs, and
defeated the human European Go champion by 5 games to 0. This is the first
time that a computer program has defeated a human professional player in the
full-sized game of Go, a feat previously thought to be at least a decade
away., archivePrefix = arXiv, arxivId = 1610.00633, author = Silver, David and
Huang, Aja and Maddison, Chris J. and Guez, Arthur and Sifre, Laurent and van
den Driessche, George and Schrittwieser, Julian and Antonoglou, Ioannis and
Panneershelvam, Veda and Lanctot, Marc and Dieleman, Sander and Grewe, Dominik
and Nham, John and Kalchbrenner, Nal and Sutskever, Ilya and Lillicrap,
Timothy and Leach, Madeleine and Kavukcuoglu, Koray and Graepel, Thore and
Hassabis, Demis, doi = 10.1038/nature16961, eprint = 1610.00633, file = ::,
isbn = 1476-4687 (Electronic)$\backslash$r0028-0836 (Linking), issn =
0028-0836, journal = Nature, month = jan, number = 7587, pages = 484–489, pmid
= 26819042, publisher = Nature Publishing Group, a division of Macmillan
Publishers Limited. All Rights Reserved., title = Mastering the game of Go
with deep neural networks and tree search, url =
http://dx.doi.org/10.1038/nature16961 http://10.0.4.14/nature16961
http://www.nature.com/nature/journal/v529/n7587/abs/nature16961.html#supplementary-
information http://airesearch.com/wp-content/uploads/2016/01/deepmind-
mastering-go.pdf, volume = 529, year = 2016
@inproceedingsnakanishi2004bipedlfd, abstract = In this paper, we introduce a
framework for learning biped locomotion using dynamical movement primitives
based on non-linear oscillators. Our ultimate goal is to establish a design
principle of a controller in order to achieve natural human-like locomotion.
We suggest dynamical movement primitives as a central pattern generator (CPG)
of a biped robot, an approach we have previously proposed for learning and
encoding complex human movements. Demonstrated trajectories are learned
through movement primitives by locally weighted regression, and the frequency
of the learned trajectories is adjusted automatically by a novel frequency
adaptation algorithm based on phase resetting and entrainment of coupled
oscillators. Numerical simulations and experimental implementation on a
physical robot demonstrate the effectiveness of the proposed locomotion
controller. 2004 Elsevier B.V. All rights reserved., author = Nakanishi, Jun
and Morimoto, Jun and Endo, Gen and Cheng, Gordon and Schaal, Stefan and
Kawato, Mitsuo, booktitle = Robotics and Autonomous Systems, doi =
10.1016/j.robot.2004.03.003, file = ::, isbn = 0921-8890, issn = 09218890,
keywords = Biped locomotion,Dynamical movement primitives,Frequency
adaptation,Learning from demonstration,Phase resetting, number = 2-3, pages =
79–91, title = Learning from demonstration and adaptation of biped locomotion,
url =
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.77.534&rep=rep1&type=pdf,
volume = 47, year = 2004 @inproceedingsschaal2001augmentation, abstract =
Real-time modeling of complex nonlinear dynamic processes has become
increasingly impor-tant in various areas of robotics and human aug-mentation.
To address such problems, we have been developing special statistical learning
methods that meet the demands of on-line learning, in particular the need for
low computational complexity, rapid learning, and scalability to high-
dimensional spaces. In this paper, we introduce a novel algorithm that
possesses all the necessary properties by combining methods from probabilistic
and nonparametric learning. We demonstrate the applicability of our methods
for three different applications in humanoid robotics, i.e., the on-line
learning of a full-body in-verse dynamics model, an inverse kinematics model,
and imitation learning. The latter application will also introduce a novel
method to shape attractor landscapes of dynamical system by means of statis-
tical learning., author = Schaal, Stefan and Vijayakumar, Sethu and Souza,
Aaron D ’ and Ijspeert, Auke and Nakanishi, Jun, booktitle = International
Symposium on Robotics Research, file = ::, title = Real-Time Statistical
Learning For Robotics and Human Augmentation, url = http://www-clmc.usc.edu
@articleKavraki1996, author = Kavraki, Lydia and Svestka, Petr and Latombe,
J-C and Overmars, Mark, file = ::, journal = IEEE Transactions on Robotics and
Automation, number = 4, pages = 566–580, title = Probabilistic roadmaps for
path planning in high-dimensional configuration spaces, volume = 12, year =
1996 @inproceedingspathak2017curiosity, abstract = In many real-world
scenarios, rewards extrinsic to the agent are extremely sparse, or absent
altogether. In such cases, curiosity can serve as an intrinsic reward signal
to enable the agent to explore its environment and learn skills that might be
useful later in its life. We formulate curiosity as the error in an agent’s
ability to predict the consequence of its own actions in a visual feature
space learned by a self-supervised inverse dynamics model. Our formulation
scales to high-dimensional continuous state spaces like images, bypasses the
difficulties of directly predicting pixels, and, critically, ignores the
aspects of the environment that cannot affect the agent. The proposed approach
is evaluated in two environments: VizDoom and Super Mario Bros. Three broad
settings are investigated: 1) sparse extrinsic reward, where curiosity allows
for far fewer interactions with the environment to reach the goal; 2)
exploration with no extrinsic reward, where curiosity pushes the agent to
explore more efficiently; and 3) generalization to unseen scenarios (e.g. new
levels of the same game) where the knowledge gained from earlier experience
helps the agent explore new places much faster than starting from scratch.
Demo video and code available at https://pathak22.github.io/noreward-rl/,
archivePrefix = arXiv, arxivId = 1705.05363, author = Pathak, Deepak and
Agrawal, Pulkit and Efros, Alexei A. and Darrell, Trevor, booktitle =
International Conference on Machine Learning (ICML), eprint = 1705.05363, file
= ::, organization = IEEE, pages = 488–489, title = Curiosity-Driven
Exploration by Self-Supervised Prediction, year = 2017
@articlesmith2005development, abstract = The embodiment hypothesis is the idea
that intelligence emerges in the interaction of an agent with an environment
and as a result of sensorimotor activity. In this paper we offer six lessons
for developing embodied intelligent agents suggested by research in
developmental psychology. We argue that starting as a baby grounded in a
physical, social and linguistic world is crucial to the development of the
flexible and inventive intelligence that characterizes humankind., author =
Smith, Linda and Gasser, Michael, file = ::, journal = Artificial life,
keywords = cognition,development,embodiment,language,motor control, number =
1-2, pages = 13–29, publisher = MIT Press, title = The development of embodied
cognition: Six lessons from babies, url =
https://www.cogsci.msu.edu/DSS/2010-2011/Smith/6lessons.pdf, volume = 11, year
= 2005 @inproceedingsng2000irl, author = Ng, Andrew and Russell, Stuart,
booktitle = International Conference on Machine Learning (ICML), file = ::,
title = Algorithms for Inverse Reinforcement Learning, url =
http://ai.stanford.edu/ ang/papers/icml00-irl.pdf, year = 2000
@inproceedingssrivastava14tamp, abstract = The need for combined task and
motion planning in robotics is well understood. Solutions to this problem have
typically relied on special purpose, integrated implementations of task
planning and motion planning algorithms. We propose a new approach that uses
off-the-shelf task planners and motion planners and makes no assumptions about
their implementa-tion. Doing so enables our approach to directly build on, and
benefit from, the vast literature and latest advances in task planning and
motion planning. It uses a novel representational abstraction and requires
only that failures in computing a mo-tion plan for a high-level action be
identifiable and expressible in the form of logical predicates at the task
level. We evaluate the approach and illustrate its robustness through a number
of experiments using a state-of-the-art robotics simulator and a PR2 robot.
These experiments show the system accomplishing a diverse set of challenging
tasks such as taking advantage of a tray when laying out a table for dinner
and picking objects from cluttered environments where other objects need to be
re-arranged before the target object can be reached., author = Srivastava,
Siddharth and Fang, Eugene and Riano, Lorenzo and Chitnis, Rohan and Russell,
Stuart and Abbeel, Pieter, booktitle = IEEE International Conference on
Robotics and Automation (ICRA), title = Combined Task and Motion Planning
Through an Extensible Planner-Independent Interface Layer, year = 2014
@articlepeters2003naturalac, abstract = Reinforcement learning offers a
promising framework to take planning for real-world systems towards true
autonomy and versatility. However, apply-ing reinforcement learning to high
dimensional movement systems (such as real-world robots) in the presence of
uncertainty and continuous state-action spaces remains an unsolved problem. In
order to make progress towards solving this issue, we focus on a particular
type of reinforcement learning methods, i.e., policy gradient methods. These
methods are particularly in-teresting to the robotics community as they seem
to scale better to continuous state-action problems and have been successfully
applied on a variety of high-dimensional robots. However, the main
disadvantages of these methods have been the high variance in the gradient
estimate, the very slow convergence, and the dependence on baseline functions.
In this poster, we show how these policy gradients can be improved in respect
to each of these problems. Our approach to policy gradients focuses on the
natural policy gradient instead of the regular policy gradient. Natural policy
gradients for reinforce-ment learning have first been suggested by Kakade [2]
as ’average natural policy gradients’, and subsequently been shown to be the
true natural policy gradient by Bagnell & Schneider [1], and Peters et al.
[3]. As shown by Kakade, natural policy gradients are particularly interesting
due to the fact that they equal the parameters of the compatible function
approximation. We present a general algorithm for estimating the natural
gradient, the Nat-ural Actor-Critic algorithm. This algorithm uses the fact
that the compatible function approximation represents an advantage function
which can be em-1 bedded cleanly into the Bellman equation. It can be used in
two different ways, i.e., in form of general temporal difference learning
where reasonable basis functions for the value function are required in order
to obtain an un-biased gradient, and in form of start-state reinforcement
learning where the gradient is estimated using the property that the sum of
all advantages along a roll-out has to equal the sum of rewards plus a single
value function offset parameter. The later method is guaranteed to yield an
unbiased estimate of the gradient and is well suited for learning and refining
parameterized policies, even in the light of incomplete state information. We
show two examples of the application of the Natural Actor-Critic algorithm,
one where it by far outperforms non-natural policy gradients in the classical
cart-pole balancing system, and one for learning nonlinear dy-namic motor
primitives for humanoid robot control. From our experience, the start-state
Natural Actor-Critic algorithm seems to be one of the most efficient model-
free reinforcement learning techniques and offers a promising route for the
development of reinforcement learning techniques for truly high dimensionally
continuous state-action systems., author = Peters, Jan and Vijayakumar, Sethu
and Schaal, Stefan, file = ::, journal = Neurocomputing, pages = 1180–1190,
title = Natural Actor Critic, url =
http://kyb.tuebingen.mpg.de/fileadmin/user_upload/files/publications/attachments/NIPS-
Workshop-2003-Peters_%5B0%5D.pdf, volume = 71, year = 2008
@inproceedingsboots2014traces, author = Boots, Byron and Byravan, Arunkumar
and Fox, Dieter, booktitle = IEEE International Conference on Robotics and
Automation (ICRA), doi = 10.1109/ICRA.2014.6907443, pages = 4021–4028, title =
Learning predictive models of a depth camera & manipulator from raw execution
traces, url = http://dx.doi.org/10.1109/ICRA.2014.6907443, year = 2014
@inproceedingsflorensa2017resets, archivePrefix = arXiv, arxivId = 1707.05300,
author = Florensa, Carlos and Held, David and Wulfmeier, Markus and Abbeel,
Pieter, booktitle = International Conference on Learning Representations
(ICLR), eprint = 1707.05300, file = ::, keywords = Automatic Curricu-lum
Generation,Reinforcement Learning,Robotic Manipulation, title = Reverse
Curriculum Generation for Reinforcement Learning, url =
https://arxiv.org/pdf/1707.05300.pdf http://arxiv.org/abs/1707.05300, year =
2018 @articlepeters2008baseball, abstract = Autonomous learning is one of the
hallmarks of human and animal behavior, and understanding the principles of
learning will be crucial in order to achieve true autonomy in advanced
machines like humanoid robots. In this paper, we examine learning of complex
motor skills with human-like limbs. While supervised learning can offer useful
tools for bootstrapping behavior, e.g.,??by learning from demonstration, it is
only reinforcement learning that offers a general approach to the final trial-
and-error improvement that is needed by each individual acquiring a skill.
Neither neurobiological nor machine learning studies have, so far, offered
compelling results on how reinforcement learning can be scaled to the high-
dimensional continuous state and action spaces of humans or humanoids. Here,
we combine two recent research developments on learning motor control in order
to achieve this scaling. First, we interpret the idea of modular motor control
by means of motor primitives as a suitable way to generate parameterized
control policies for reinforcement learning. Second, we combine motor
primitives with the theory of stochastic policy gradient learning, which
currently seems to be the only feasible framework for reinforcement learning
for humanoids. We evaluate different policy gradient methods with a focus on
their applicability to parameterized motor primitives. We compare these
algorithms in the context of motor primitive learning, and show that our most
modern algorithm, the Episodic Natural Actor-Critic outperforms previous
algorithms by at least an order of magnitude. We demonstrate the efficiency of
this reinforcement learning method in the application of learning to hit a
baseball with an anthropomorphic robot arm. ?? 2008 Elsevier Ltd. All rights
reserved., archivePrefix = arXiv, arxivId = arXiv:1411.3159v1, author =
Peters, Jan and Schaal, Stefan, doi = 10.1016/j.neunet.2008.02.003, eprint =
arXiv:1411.3159v1, file = ::, isbn = 0893-6080, issn = 08936080, journal =
Neural Networks, keywords = Motor primitives,Motor skills,Natural Actor-
Critic,Natural gradients,Policy gradient methods,Reinforcement learning,
number = 4, pages = 682–697, pmid = 18482830, title = Reinforcement learning
of motor skills with policy gradients, url =
https://pdfs.semanticscholar.org/eb5b/459c8a3e56064158fb3514eeab763486e437.pdf,
volume = 21, year = 2008 @inproceedingskalakrishnan09terraintemplates,
abstract = â We address the problem of foothold selection in robotic legged
locomotion over very rough terrain. The difficulty of the problem we address
here is comparable to that of human rock-climbing, where foot/hand-hold
selection is one of the most critical aspects. Previous work in this domain
typically involves defining a reward function over footholds as a weighted
linear combination of terrain features. However, a significant amount of
effort needs to be spent in designing these features in order to model more
complex decision functions, and hand-tuning their weights is not a trivial
task. We propose the use of terrain templates, which are discretized height
maps of the terrain under a foothold on different length scales, as an
alternative to manually designed features. We describe an algo-rithm that can
simultaneously learn a small set of templates and a foothold ranking function
using these templates, from expert-demonstrated footholds. Using the LittleDog
quadruped robot, we experimentally show that the use of terrain templates can
produce complex ranking functions with higher performance than standard
terrain features, and improved generalization to unseen terrain., author =
Kalakrishnan, Mrinal and Buchli, Jonas and Pastor, Peter and Schaal, Stefan,
booktitle = IEEE/RSJ International Conference on Intelligent Robots and
Systems (IROS), file = ::, title = Learning Locomotion over Rough Terrain
using Terrain Templates, url =
https://pdfs.semanticscholar.org/02db/2c0100ffb02592e8738d0ffcf454224f4b1b.pdf,
year = 2009 @articleschaal97lfd, abstract = By now it is widely accepted that
learning a task from scratch, i.e., without any prior knowledge, is a daunting
undertaking. Humans, however, rarely at- tempt to learn from scratch. They
extract initial biases as well as strategies how to approach a learning
problem from instructions and/or demonstrations of other humans. For learning
control, this paper investigates how learning from demonstration can be
applied in the context of reinforcement learning. We consider priming the
Q-function, the value function, the policy, and the model of the task dynamics
as possible areas where demonstrations can speed up learning. In general
nonlinear learning problems, only model-based rein- forcement learning shows
significant speed-up after a demonstration, while in the special case of
linear quadratic regulator (LQR) problems, all methods profit from the
demonstration. In an implementation of pole balancing on a complex
anthropomorphic robot arm, we demonstrate that, when facing the complexities
of real signal processing, model-based reinforcement learning offers the most
robustness for LQR problems. Using the suggested methods, the robot learns
pole balancing in just a single trial after a 30 second long demonstration of
the human instructor., author = Schaal, Stefan, doi =
10.1016/j.robot.2004.03.001, file = ::, isbn = 1558604863, issn = 1049-5258,
journal = Advances in Neural Information Processing Systems (NIPS), number =
9, pages = 1040–1046, pmid = 11540378, title = Robot learning from
demonstration, url = http://www.cc.gatech.edulfac
http://wwwiaim.ira.uka.de/users/rogalla/WebOrdnerMaterial/ml-
robotlearning.pdf, year = 1997 @inproceedingsfinn16guidedcostlearning,
abstract = Reinforcement learning can acquire complex be-haviors from high-
level specifications. However, defining a cost function that can be optimized
effectively and encodes the correct task is chal-lenging in practice. We
explore how inverse op-timal control (IOC) can be used to learn behav-iors
from demonstrations, with applications to torque control of high-dimensional
robotic sys-tems. Our method addresses two key challenges in inverse optimal
control: first, the need for in-formative features and effective
regularization to impose structure on the cost, and second, the dif-ficulty of
learning the cost function under un-known dynamics for high-dimensional
continu-ous systems. To address the former challenge, we present an algorithm
capable of learning ar-bitrary nonlinear cost functions, such as neural
networks, without meticulous feature engineer-ing. To address the latter
challenge, we formu-late an efficient sample-based approximation for MaxEnt
IOC. We evaluate our method on a series of simulated tasks and real-world
robotic manip-ulation problems, demonstrating substantial im-provement over
prior methods both in terms of task complexity and sample efficiency., author
= Finn, Chelsea and Levine, Sergey and Abbeel, Pieter, booktitle =
International Conference on Machine Learning (ICML), file = ::, title = Guided
Cost Learning: Deep Inverse Optimal Control via Policy Optimization, url =
https://arxiv.org/pdf/1603.00448.pdf, year = 2016
@inproceedingsbahdanau14attention, abstract = Neural machine translation is a
recently proposed approach to machine transla-tion. Unlike the traditional
statistical machine translation, the neural machine translation aims at
building a single neural network that can be jointly tuned to maximize the
translation performance. The models proposed recently for neu-ral machine
translation often belong to a family of encoderâdecoders and encode a source
sentence into a fixed-length vector from which a decoder generates a
translation. In this paper, we conjecture that the use of a fixed-length
vector is a bottleneck in improving the performance of this basic
encoderâdecoder architec-ture, and propose to extend this by allowing a
model to automatically (soft-)search for parts of a source sentence that are
relevant to predicting a target word, without having to form these parts as a
hard segment explicitly. With this new approach, we achieve a translation
performance comparable to the existing state-of-the-art phrase-based system on
the task of English-to-French translation. Furthermore, qualitative analysis
reveals that the (soft-)alignments found by the model agree well with our
intuition., author = Bahdanau, Dzmitry and Cho, Kyunghyun and Bengio, Yoshua,
booktitle = International Conference on Learning Representations (ICLR), file
= ::, title = Neural Machine Translation by Jointly Learning to Align and
Translate, url = https://arxiv.org/pdf/1409.0473.pdf, year = 2015
@inproceedingsduan2017oneshotimitation, abstract = Imitation learning has been
commonly applied to solve different tasks in isolation. This usually requires
either careful feature engineering, or a significant number of samples. This
is far from what we desire: ideally, robots should be able to learn from very
few demonstrations of any given task, and instantly generalize to new
situations of the same task, without requiring task-specific engineering. In
this paper, we propose a meta-learning framework for achieving such
capability, which we call one-shot imitation learning. Specifically, we
consider the setting where there is a very large set of tasks, and each task
has many instantiations. For example, a task could be to stack all blocks on a
table into a single tower, another task could be to place all blocks on a
table into two-block towers, etc. In each case, different instances of the
task would consist of different sets of blocks with different initial states.
At training time, our algorithm is presented with pairs of demonstrations for
a subset of all tasks. A neural net is trained that takes as input one
demonstration and the current state (which initially is the initial state of
the other demonstration of the pair), and outputs an action with the goal that
the resulting sequence of states and actions matches as closely as possible
with the second demonstration. At test time, a demonstration of a single
instance of a new task is presented, and the neural net is expected to perform
well on new instances of this new task. The use of soft attention allows the
model to generalize to conditions and tasks unseen in the training data. We
anticipate that by training this model on a much greater variety of tasks and
settings, we will obtain a general system that can turn any demonstrations
into robust policies that can accomplish an overwhelming variety of tasks.
Videos available at https://bit.ly/one-shot-imitation ., archivePrefix =
arXiv, arxivId = 1703.07326, author = Duan, Yan and Andrychowicz, Marcin and
Stadie, Bradly C. and Ho, Jonathan and Schneider, Jonas and Sutskever, Ilya
and Abbeel, Pieter and Zaremba, Wojciech, booktitle = Advances in Neural
Information Processing Systems (NIPS), eprint = 1703.07326, file = ::, title =
One-Shot Imitation Learning, year = 2017 @inproceedingspong2018tdm, author =
Pong, Vitchyr and Gu, Shixiang and Dalal, Murtaza and Levine, Sergey,
booktitle = International Conference on Learning Representations (ICLR), title
= Temporal Difference Models: Model-Free Deep RL For Model-Based Control, year
= 2018 @inproceedingsfinn2016visualforesight, abstract = A key challenge in
scaling up robot learning to many skills and environments is removing the need
for human supervision, so that robots can collect their own data and improve
their own performance without being limited by the cost of requesting human
feedback. Model-based reinforcement learning holds the promise of enabling an
agent to learn to predict the effects of its actions, which could provide
flexible predictive models for a wide range of tasks and environments, without
detailed human supervision. We develop a method for combining deep action-
conditioned video prediction models with model-predictive control that uses
entirely unlabeled training data. Our approach does not require a calibrated
camera, an instrumented training set-up, nor precise sensing and actuation.
Our results show that our method enables a real robot to perform nonprehensile
manipulation – pushing objects – and can handle novel objects not seen during
training., archivePrefix = arXiv, arxivId = 1610.00696, author = Finn, Chelsea
and Levine, Sergey, booktitle = Advances in Neural Information Processing
Systems (NIPS), eprint = 1610.00696, file = ::;::, title = Deep Visual
Foresight for Planning Robot Motion, url =
https://arxiv.org/pdf/1610.00696.pdf http://arxiv.org/abs/1610.00696, year =
2016 @inproceedingsijspeert2002attractor, abstract = Many control problems
take place in continuous state-action spaces,$\backslash$n$\backslash$ne.g.,
as in manipulator robotics, where the control objective is
of-$\backslash$n$\backslash$nten deï¿œned as ï¿œnding a desired trajectory
that reaches a particular$\backslash$n$\backslash$ngoal state. While
reinforcement learning oï¿œers a theoretical
frame-$\backslash$n$\backslash$nwork to learn such control policies from
scratch, its applicability$\backslash$nto$\backslash$n$\backslash$nhigher
dimensional continuous state-action spaces remains
rather$\backslash$n$\backslash$nlimited to date. Instead of learning from
scratch, in this paper we$\backslash$n$\backslash$nsuggest to learn a desired
complex control policy by transforming$\backslash$n$\backslash$nan existing
simple canonical control policy. For this purpose,
we$\backslash$n$\backslash$nrepresent canonical policies in terms of
diï¿œerential equations with$\backslash$n$\backslash$nwell-deï¿œned attractor
properties. By nonlinearly transforming the$\backslash$n$\backslash$ncanonical
attractor dynamics using techniques from
nonparametric$\backslash$n$\backslash$nregression, almost arbitrary new
nonlinear policies can be gener-$\backslash$n$\backslash$nated without losing
the stability properties of the canonical sys-$\backslash$n$\backslash$ntem.
We demonstrate our techniques in the context of learning
a$\backslash$n$\backslash$nset of movement skills for a humanoid robot from
demonstrations$\backslash$n$\backslash$nof a human teacher. Policies are
acquired rapidly, and, due to the$\backslash$n$\backslash$nproperties of well
formulated diï¿œerential equations, can be re-
used$\backslash$n$\backslash$nand modiï¿œed on-line under dynamic changes of
the environment.$\backslash$n$\backslash$nThe linear parameterization of
nonparametric regression moreover$\backslash$n$\backslash$nlends itself to
recognize and classify previously learned
movement$\backslash$n$\backslash$nskills. Evaluations in simulations and on an
actual 30 degree-of-$\backslash$n$\backslash$nfreedom humanoid robot exemplify
the feasibility and robustness$\backslash$n$\backslash$nof our approach.,
author = Ijspeert, Auke Jan and Nakanishi, Jun and Schaal, Stefan, booktitle =
Advances in Neural Information Processing Systems (NIPS), file = ::, isbn =
1049-5258, issn = 10495258, keywords = reinforcement, pages = 1547–1554, title
= Learning Attractor Landscapes for Learning Motor Primitives, url =
https://papers.nips.cc/paper/2140-learning-attractor-landscapes-for-learning-
motor-primitives.pdf, year = 2002 @inproceedingspathak2018zeroshot, abstract =
The current dominant paradigm for imitation learning relies on strong
supervision of expert actions to learn both what and how to imitate. We pursue
an alternative paradigm wherein an agent first explores the world without any
expert supervision and then distills its experience into a goal-conditioned
skill policy with a novel forward consistency loss. In our framework, the role
of the expert is only to communicate the goals (i.e., what to imitate) during
inference. The learned policy is then employed to mimic the expert (i.e., how
to imitate) after seeing just a sequence of images demonstrating the desired
task. Our method is ” zero-shot ” in the sense that the agent never has access
to expert actions during training or for the task demonstration at inference.
We evaluate our zero-shot imitator in two real-world settings: complex rope
manipulation with a Baxter robot and navigation in previously unseen office
environments with a TurtleBot. Through further experiments in VizDoom
simulation, we provide evidence that better mechanisms for exploration lead to
learning a more capable policy which in turn improves end task performance.
Videos, models, and more details are available at
https://pathak22.github.io/zeroshot-imitation/., author = Pathak, Deepak and
Mahmoudieh, Parsa and Luo, Guanghao and Agrawal, Pulkit and Chen, Dian and
Shentu, Yide and Shelhamer, Evan and Malik, Jitendra and Efros, Alexei A and
Darrell, Trevor, booktitle = International Conference on Learning
Representations (ICLR), file = ::, title = Zero-Shot Visual Imitation, url =
https://arxiv.org/pdf/1804.08606.pdf, year = 2018
@inproceedingspomerleau1989alvinn, abstract = ALVINN (Autonomous Land Vehicle
In a Neural Network) is a 3-layer back-propagation network designed for the
task of road following. Currently ALVINN takes images from a camera and a
laser range finder as input and produces as output the direction the vehicle
should travel in order to follow the road. Training has been conducted using
simulated road images. Successful tests on the Carnegie Mellon autonomous
navigation test vehicle indicate that the network can effectively follow real
roads under certain field conditions. The representation developed to perform
the task differs dramatically when the network is trained under various
conditions, suggesting the possibility of a novel adaptive autonomous
navigation system capable of tailoring its processing to the conditions at
hand., author = Pomerleau, Dean A, booktitle = Advances in Neural Information
Processing Systems (NIPS), file = ::, isbn = 1-558-60015-9, pages = 305–313,
title = Alvinn: An autonomous land vehicle in a neural network, url =
http://repository.cmu.edu/compsci, year = 1989
@inproceedingssun17deeplyaggrevated, abstract = Researchers have demonstrated
state-of-the-art performance in sequential decision making prob-lems (e.g.,
robotics control, sequential predic-tion) with deep neural network models. One
often has access to near-optimal oracles that achieve good performance on the
task during training. We demonstrate that AggreVaTeD â a policy gradient
extension of the Imitation Learn-ing (IL) approach of (Ross & Bagnell, 2014)
â can leverage such an oracle to achieve faster and better solutions with
less training data than a less-informed Reinforcement Learning (RL) technique.
Using both feedforward and recur-rent neural predictors, we present stochastic
gra-dient procedures on a sequential prediction task, dependency-parsing from
raw image data, as well as on various high dimensional robotics con-trol
problems. We also provide a comprehen-sive theoretical study of IL that
demonstrates we can expect up to exponentially lower sample complexity for
learning with AggreVaTeD than with RL algorithms, which backs our empirical
findings. Our results and theory indicate that the proposed approach can
achieve superior per-formance with respect to the oracle when the demonstrator
is sub-optimal., author = Sun, Wen and Venkatraman, Arun and Gordon, Geoffrey
J and Boots, Byron and Edu, Bboots@cc Gatech and Bagnell, J Andrew, booktitle
= International Conference on Machine Learning (ICML), file = ::, title =
Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential
Prediction, url = https://arxiv.org/pdf/1703.01030.pdf, year = 2017
@articlewakamatsu2006knotting, author = Wakamatsu, Hidefumi and Arai, Eiji and
Hirai, Shinichi, journal = The International Journal of Robotics Research,
number = 4, pages = 371–395, publisher = SAGE Publications, title =
Knotting/unknotting manipulation of deformable linear objects, volume = 25,
year = 2006 @articleRoss2014, abstract = Recent work has demonstrated that
problems– particularly imitation learning and structured prediction– where a
learner’s predictions influence the input-distribution it is tested on can be
naturally addressed by an interactive approach and analyzed using no-regret
online learning. These approaches to imitation learning, however, neither
require nor benefit from information about the cost of actions. We extend
existing results in two directions: first, we develop an interactive imitation
learning approach that leverages cost information; second, we extend the
technique to address reinforcement learning. The results provide theoretical
support to the commonly observed successes of online approximate policy
iteration. Our approach suggests a broad new family of algorithms and provides
a unifying view of existing techniques for imitation and reinforcement
learning., archivePrefix = arXiv, arxivId = 1406.5979, author = Ross, Stéphane
Stephane and Bagnell, J Andrew, eprint = 1406.5979, file = ::, journal = CoRR,
keywords = (), title = Reinforcement and Imitation Learning via Interactive
No-Regret Learning, url = https://arxiv.org/pdf/1406.5979.pdf
http://arxiv.org/abs/1406.5979, volume = abs/1406.5, year = 2014
@articlevecerik17ddpgfd, abstract = We propose a general and model-free
approach for Reinforcement Learning (RL) on real robotics with sparse rewards.
We build upon the Deep Deterministic Policy Gradient (DDPG) algorithm to use
demonstrations. Both demonstrations and actual interactions are used to fill a
replay buffer and the sam-pling ratio between demonstrations and transitions
is automatically tuned via a prioritized replay mechanism. Typically,
carefully engineered shaping rewards are required to enable the agents to
efficiently explore on high dimensional control problems such as robotics.
They are also required for model-based acceleration methods relying on local
solvers such as iLQG (e.g. Guided Policy Search and Nor-malized Advantage
Function). The demonstrations replace the need for carefully engineered
rewards, and reduce the exploration problem encountered by classical RL
approaches in these domains. Demonstrations are collected by a robot kines-
thetically force-controlled by a human demonstrator. Results on four simulated
insertion tasks show that DDPG from demonstrations out-performs DDPG, and does
not require engineered rewards. Finally, we demonstrate the method on a real
robotics task consisting of inserting a clip (flexible object) into a rigid
object., author = Večerík, Matej and Hester, Todd and Scholz, Jonathan and
Wang, Fumin and Pietquin, Olivier and Piot, Bilal and Heess, Nicolas and
Rothörl, Thomas and Lampe, Thomas and Riedmiller, Martin, file = ::, journal =
CoRR, keywords = Apprenticeship,Demonstrations,Learning,Robot, title =
Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems
with Sparse Rewards, url = https://arxiv.org/pdf/1707.08817.pdf, volume =
abs/1707.0, year = 2017 @inproceedingsreed2014learning, author = Reed, Scott
and Sohn, Kihyuk and Zhang, Yuting and Lee, Honglak, booktitle = International
Conference on Machine Learning (ICML), pages = 1431–1439, title = Learning to
disentangle factors of variation with manifold interaction, year = 2014
@inproceedingsyang2015robot, author = Yang, Yezhou and Li, Yi and Fermüller,
Cornelia and Aloimonos, Yiannis, booktitle = AAAI Conference on Artificial
Intelligence, pages = 3686–3693, title = Robot Learning Manipulation Action
Plans by” Watching” Unconstrained Videos from the World Wide Web., year = 2015
@articlemichotte1963perception, author = Michotte, Albert, publisher = Basic
Books, title = The perception of causality., year = 1963
@articlelevy2017hierarchical, author = Levy, Andrew and Platt, Robert and
Saenko, Kate, journal = arXiv preprint arXiv:1712.00948, title = Hierarchical
Actor-Critic, year = 2017 @inproceedingsmorita2003knot, author = Morita,
Takuma and Takamatsu, Jun and Ogawara, Koichi and Kimura, Hiroshi and Ikeuchi,
Katsushi, booktitle = IEEE International Conference on Robotics and Automation
(ICRA), organization = IEEE, pages = 3887–3892, title = Knot planning from
observation, volume = 3, year = 2003 @booklavalle2006planning, author =
LaValle, Steven M, publisher = Cambridge university press, title = Planning
algorithms, year = 2006 @inproceedingskakade2002approximatelyoptimal, abstract
= Abstract In order to solve realistic reinforcement learning problems, it is
critical that approximate algorithms be used. In this paper, we present the
conservative policy iteration algorithm which finds an” approximately” optimal
policy, given access to a restart … $\backslash$n, author = Kakade, Sham and
Langford, John, booktitle = International Conference on Machine Learning
(ICML), file = ::, isbn = 1-55860-873-7, pages = 267–274, title =
Approximately Optimal Approximate Reinforcement Learning, url =
http://www.cs.cmu.edu/afs/cs/Web/People/jcl/papers/aoarl/Final.pdf, year =
2002 @inproceedingshosu2016humancheckpoint, abstract = This paper introduces a
novel method for learning how to play the most difficult Atari 2600 games from
the Arcade Learn-ing Environment using deep reinforcement learning. The
proposed method, called human checkpoint replay, consists in using check-
points sampled from human gameplay as starting points for the learn-ing
process. This is meant to compensate for the difficulties of current
exploration strategies, such as $\epsilon$-greedy, to find successful control
policies in games with sparse rewards. Like other deep reinforce-ment learning
architectures, our model uses a convolutional neural network that receives
only raw pixel inputs to estimate the state value function. We tested our
method on Montezuma’s Revenge and Pri-vate Eye, two of the most challenging
games from the Atari platform. The results we obtained show a substantial
improvement compared to previous learning approaches, as well as over a random
player. We also propose a method for training deep reinforcement learning
agents using human gameplay experience, which we call human ex-perience
replay., author = Hosu, Ionel-Alexandru and Rebedea, Traian, booktitle =
Workshop on Evaluating General Purpose AI, file = ::, title = Playing Atari
Games with Deep Reinforcement Learning and Human Checkpoint Replay, url =
https://arxiv.org/pdf/1607.05077.pdf, year = 2016 @articleDragan2017, abstract
= Robots interacting with the physical world plan with models of physics. We
advocate that robots interacting with people need to plan with models of
cognition. This writeup summarizes the insights we have gained in integrating
computational cognitive models of people into robotics planning and control.
It starts from a general game-theoretic formulation of interaction, and
analyzes how different approximations result in different useful coordination
behaviors for the robot during its interaction with people., archivePrefix =
arXiv, arxivId = 1705.04226, author = Dragan, Anca D., eprint = 1705.04226,
file = ::, month = may, title = Robot Planning with Mathematical Models of
Human State and Action, url = https://arxiv.org/pdf/1705.04226.pdf
http://arxiv.org/abs/1705.04226, year = 2017 @inproceedingsmnih2013atari,
abstract = We present the first deep learning model to successfully learn
control policies directly from high-dimensional sensory input using
reinforcement learning. The model is a convolutional neural network, trained
with a variant of Q-learning, whose input is raw pixels and whose output is a
value function estimating future rewards. We apply our method to seven Atari
2600 games from the Arcade Learning Environment, with no adjustment of the
architecture or learning algorithm. We find that it outperforms all previous
approaches on six of the games and surpasses a human expert on three of them.,
archivePrefix = arXiv, arxivId = 1312.5602, author = Mnih, Volodymyr and
Kavukcuoglu, Koray and Silver, David and Graves, Alex and Antonoglou, Ioannis
and Wierstra, Daan and Riedmiller, Martin, booktitle = NIPS Workshop on Deep
Learning, doi = 10.1038/nature14236, eprint = 1312.5602, file = ::;::;::, isbn
= 1476-4687 (Electronic) 0028-0836 (Linking), issn = 0028-0836, pages = 1–9,
pmid = 25719670, title = Playing Atari with Deep Reinforcement Learning, url =
https://arxiv.org/pdf/1312.5602.pdf http://arxiv.org/abs/1312.5602
https://www.cs.toronto.edu/ vmnih/docs/dqn.pdf, year = 2013
@inproceedingsrajeswaran2017simplicity, abstract = This work shows that
policies with simple lin-ear and RBF parameterizations can be trained to solve
a variety of continuous control tasks, in-cluding the OpenAI gym benchmarks.
The per-formance of these trained policies are competi-tive with state of the
art results, obtained with more elaborate parameterizations such as fully
connected neural networks. Furthermore, exist-ing training and testing
scenarios are shown to be very limited and prone to over-fitting, thus giving
rise to only trajectory centric policies. Training with a diverse initial
state distribution is shown to produce more global policies with better gen-
eralization. This allows for interactive control scenarios where the system
recovers from large on-line perturbations; as shown in the video., author =
Rajeswaran, Aravind and Lowrey, Kendall and Todorov, Emanuel and Kakade, Sham,
booktitle = Advances in Neural Information Processing Systems (NIPS), file =
::, title = Towards Generalization and Simplicity in Continuous Control, url =
https://arxiv.org/pdf/1703.02660.pdf https://homes.cs.washington.edu/
todorov/papers/RajeswaranICML17.pdf, year = 2017
@inproceedingsnair2018demonstrations, abstract = Exploration in environments
with sparse rewards has been a persistent problem in reinforcement learning
(RL). Many tasks are natural to specify with a sparse reward, and manually
shaping a reward function can result in suboptimal performance. However,
finding a non-zero reward is exponen-tially more difficult with increasing
task horizon or action dimensionality. This puts many real-world tasks out of
practical reach of RL methods. In this work, we use demonstrations to overcome
the exploration problem and successfully learn to perform long-horizon, multi-
step robotics tasks with continuous control such as stacking blocks with a
robot arm. Our method, which builds on top of Deep Deterministic Policy
Gradients and Hindsight Experience Replay, provides an order of magnitude of
speedup over RL on simulated robotics tasks. It is simple to implement and
makes only the additional assumption that we can collect a small set of
demonstrations. Furthermore, our method is able to solve tasks not solvable by
either RL or behavior cloning alone, and often ends up outperforming the
demonstrator policy., author = Nair, Ashvin and Mcgrew, Bob and Andrychowicz,
Marcin and Zaremba, Wojciech and Abbeel, Pieter, booktitle = IEEE
International Conference on Robotics and Automation (ICRA), file = ::, title =
Overcoming Exploration in Reinforcement Learning with Demonstrations, url =
https://arxiv.org/pdf/1709.10089.pdf, year = 2018
@inproceedingsvezhnevets2017fun, abstract = We introduce FeUdal Networks
(FuNs): a novel architecture for hierarchical reinforcement learning. Our
approach is inspired by the feudal reinforcement learning proposal of Dayan
and Hinton, and gains power and efficacy by decoupling end-to-end learning
across multiple levels – allowing it to utilise different resolutions of time.
Our framework employs a Manager module and a Worker module. The Manager
operates at a lower temporal resolution and sets abstract goals which are
conveyed to and enacted by the Worker. The Worker generates primitive actions
at every tick of the environment. The decoupled structure of FuN conveys
several benefits – in addition to facilitating very long timescale credit
assignment it also encourages the emergence of sub-policies associated with
different goals set by the Manager. These properties allow FuN to dramatically
outperform a strong baseline agent on tasks that involve long-term credit
assignment or memorisation. We demonstrate the performance of our proposed
system on a range of tasks from the ATARI suite and also from a 3D DeepMind
Lab environment., archivePrefix = arXiv, arxivId = 1703.01161, author =
Vezhnevets, Alexander Sasha and Osindero, Simon and Schaul, Tom and Heess,
Nicolas and Jaderberg, Max and Silver, David and Kavukcuoglu, Koray, booktitle
= International Conference on Machine Learning (ICML), eprint = 1703.01161,
file = ::;::;::, title = FeUdal Networks for Hierarchical Reinforcement
Learning, url = https://arxiv.org/pdf/1703.01161.pdf
http://arxiv.org/abs/1703.01161, year = 2017 @inproceedingswang2016duelingdqn,
abstract = In recent years there have been many successes of using deep
representations in reinforcement learning. Still, many of these applications
use conventional architectures, such as convolutional networks, LSTMs, or
auto-encoders. In this pa-per, we present a new neural network architec-ture
for model-free reinforcement learning. Our dueling network represents two
separate estima-tors: one for the state value function and one for the state-
dependent action advantage function. The main benefit of this factoring is to
general-ize learning across actions without imposing any change to the
underlying reinforcement learning algorithm. Our results show that this
architec-ture leads to better policy evaluation in the pres-ence of many
similar-valued actions. Moreover, the dueling architecture enables our RL
agent to outperform the state-of-the-art on the Atari 2600 domain., author =
Wang, Ziyu and Schaul, Tom and Hessel, Matteo and Com, Mtthss@google and Van
Hasselt, Hado and Lanctot, Marc, booktitle = International Conference on
Machine Learning (ICML), file = ::, title = Dueling Network Architectures for
Deep Reinforcement Learning, url = https://arxiv.org/pdf/1511.06581.pdf, year
= 2016 @inproceedingssilver2014dpg, abstract = In this paper we consider
deterministic policy gradient algorithms for reinforcement learning with
continuous actions. The deterministic pol-icy gradient has a particularly
appealing form: it is the expected gradient of the action-value func-tion.
This simple form means that the deter-ministic policy gradient can be
estimated much more efficiently than the usual stochastic pol-icy gradient. To
ensure adequate exploration, we introduce an off-policy actor-critic algorithm
that learns a deterministic target policy from an exploratory behaviour
policy. We demonstrate that deterministic policy gradient algorithms can
significantly outperform their stochastic counter-parts in high-dimensional
action spaces., author = Silver, David and Heess, Nicolas and Degris, Thomas
and Wierstra, Daan and Riedmiller, Martin, booktitle = International
Conference on Machine Learning (ICML), file = ::, title = Deterministic Policy
Gradient Algorithms, url = http://proceedings.mlr.press/v32/silver14.pdf, year
= 2014 @inproceedingsliu2018imitation, author = Liu, Yuxuan and Gupta,
Abhishek and Abbeel, Pieter and Levine, Sergey, booktitle = IEEE International
Conference on Robotics and Automation (ICRA), file = ::, title = Imitation
from Observation: Learning to Imitate Behaviors from Raw Video via Context
Translation, url = https://arxiv.org/pdf/1707.03374.pdf, year = 2018
@inproceedingsGroshev, abstract = We consider the problem of learning for
planning, where knowledge acquired while planning is reused to plan faster in
new problem instances. For robotic tasks, among others, plan execution can be
captured as a sequence of visual images. For such domains, we propose to use
deep neural networks in learning for planning, based on learning a reactive
policy that imitates execution traces produced by a planner. We investigate
architectural properties of deep networks that are suitable for learning long-
horizon planning behavior, and explore how to learn, in addition to the
policy, a heuristic function that can be used with classical planners or
search algorithms such as A * . Our results on the challenging Sokoban domain
show that, with a suitable network design, complex decision making policies
and powerful heuristic functions can be learned through imitation. Videos
available at https://sites.google.com/site/learn2plannips/., author = Groshev,
Edward and Tamar, Aviv and Srivastava, Siddharth and Abbeel, Pieter, file =
::, title = Learning Generalized Reactive Policies using Deep Neural Networks,
url = https://arxiv.org/pdf/1708.07280.pdf @inproceedingsfu2017ex2, abstract =
Deep reinforcement learning algorithms have been shown to learn complex tasks
using highly general policy classes. However, sparse reward problems remain a
significant challenge. Exploration methods based on novelty detection have
been particularly successful in such settings but typically require generative
or predictive models of the observations, which can be difficult to train when
the observations are very high-dimensional and complex, as in the case of raw
images. We propose a novelty detection algorithm for exploration that is based
entirely on discriminatively trained exemplar models, where classifiers are
trained to discriminate each visited state against all others. Intuitively,
novel states are easier to distinguish against other states seen during
training. We show that this kind of discriminative modeling corresponds to
implicit density estimation, and that it can be combined with count-based
exploration to produce competitive results on a range of popular benchmark
tasks, including state-of-the-art results on challenging egocentric
observations in the vizDoom benchmark., author = Fu, Justin and Co-Reyes, John
D and Levine, Sergey, booktitle = Advances in Neural Information Processing
Systems (NIPS), file = ::, title = EX 2 : Exploration with Exemplar Models for
Deep Reinforcement Learning, url = https://arxiv.org/pdf/1703.01260.pdf, year
= 2017 @booksutton1998rl, author = Sutton, Richard S and Barto, Andrew G, file
= ::, title = Reinforcement Learning: An Introduction, url =
http://incompleteideas.net/sutton/book/bookdraft2016sep.pdf
https://webdocs.cs.ualberta.ca/ sutton/book/bookdraft2016sep.pdf, year = 1998
@inproceedingsheess2015svg, abstract = We present a unified framework for
learning continuous control policies using backpropagation. It supports
stochastic control by treating stochasticity in the Bellman equation as a
deterministic function of exogenous noise. The product is a spectrum of
general policy gradient algorithms that range from model-free methods with
value functions to model-based methods without value functions. We use learned
models but only require observations from the environment in-stead of
observations from model-predicted trajectories, minimizing the impact of
compounded model errors. We apply these algorithms first to a toy stochastic
control problem and then to several physics-based control problems in
simulation. One of these variants, SVG(1), shows the effectiveness of learning
models, value functions, and policies simultaneously in continuous domains.,
author = Heess, Nicolas and Wayne, Greg and Silver, David and Lillicrap,
Timothy and Tassa, Yuval and Erez, Tom and Deepmind, Google, booktitle =
Advances in Neural Information Processing Systems (NIPS), file = ::, title =
Learning Continuous Control Policies by Stochastic Value Gradients, url =
https://arxiv.org/pdf/1510.09142.pdf, year = 2015
@inproceedingsjang2017grasping, abstract = We consider the task of semantic
robotic grasping, in which a robot picks up an object of a user-specified
class using only monocular images. Inspired by the two-stream hypothesis of
visual reasoning, we present a semantic grasping framework that learns object
detection, classification, and grasp planning in an end-to-end fashion. A ”
ventral stream ” recognizes object class while a ” dorsal stream ”
simultaneously interprets the geometric relationships necessary to exe-cute
successful grasps. We leverage the autonomous data collection capabilities of
robots to obtain a large self-supervised dataset for training the dorsal
stream, and use semi-supervised label propagation to train the ventral stream
with only a modest amount of human supervision. We experimentally show that
our approach improves upon grasping systems whose components are not learned
end-to-end, including a baseline method that uses bounding box detection.
Furthermore, we show that jointly training our model with auxiliary data
consisting of non-semantic grasping data, as well as semantically labeled
images without grasp actions, has the potential to substantially improve
semantic grasping performance 1 ., author = Jang, Eric and Brain, Google and
Vijayanarasimhan, Sudheendra and Pastor, Peter and Ibarz, Julian and Levine,
Sergey, booktitle = Conference on Robot Learning (CoRL), file = ::;::,
keywords = deep learning,semantic grasping, title = End-to-End Learning of
Semantic Grasping, url = https://arxiv.org/pdf/1707.01932.pdf, year = 2017
@inproceedingswu2017acktr, abstract = In this work, we propose to apply trust
region optimization to deep reinforce-ment learning using a recently proposed
Kronecker-factored approximation to the curvature. We extend the framework of
natural policy gradient and propose to optimize both the actor and the critic
using Kronecker-factored approximate curvature (K-FAC) with trust region;
hence we call our method Actor Critic using Kronecker-Factored Trust Region
(ACKTR). To the best of our knowledge, this is the first scalable trust region
natural gradient method for actor-critic methods. It is also a method that
learns non-trivial tasks in continuous control as well as discrete control
policies directly from raw pixel inputs. We tested our approach across
discrete domains in Atari games as well as continuous domains in the Mu-JoCo
environment. With the proposed methods, we are able to achieve higher rewards
and a 2-to 3-fold improvement in sample efficiency on average, compared to
previous state-of-the-art on-policy actor-critic methods. Code is available at
https://github.com/openai/baselines., author = Wu, Yuhuai and Mansimov, Elman
and Liao, Shun and Grosse, Roger and Ba, Jimmy, booktitle = Advances in Neural
Information Processing Systems (NIPS), file = ::;::, title = Scalable trust-
region method for deep reinforcement learning using Kronecker-factored
approximation, url = https://arxiv.org/pdf/1708.05144.pdf, year = 2017
@inproceedingsmishra2017icml, abstract = We introduce a method for learning
the dynamics of complex nonlinear systems based on deep gen-erative models
over temporal segments of states and actions. Unlike dynamics models that
oper-ate over individual discrete timesteps, we learn the distribution over
future state trajectories con-ditioned on past state, past action, and planned
future action trajectories, as well as a latent prior over action
trajectories. Our approach is based on convolutional autoregressive models and
vari-ational autoencoders. It makes stable and accu-rate predictions over long
horizons for complex, stochastic systems, effectively expressing uncer-tainty
and modeling the effects of collisions, sen-sory noise, and action delays. The
learned dy-namics model and action prior can be used for end-to-end, fully
differentiable trajectory opti-mization and model-based policy optimization,
which we use to evaluate the performance and sample-efficiency of our method.,
author = Mishra, Nikhil and Abbeel, Pieter and Mordatch, Igor, booktitle =
International Conference on Machine Learning (ICML), file = ::, title =
Prediction and Control with Temporal Segment Models, url =
https://arxiv.org/pdf/1703.04070.pdf, year = 2017
@inproceedingshaarnoja2017sql, abstract = We propose a method for learning
expressive energy-based policies for continuous states and actions, which has
been feasible only in tabular domains before. We apply our method to learn-ing
maximum entropy policies, resulting into a new algorithm, called soft
Q-learning, that ex-presses the optimal policy via a Boltzmann dis-tribution.
We use the recently proposed amor-tized Stein variational gradient descent to
learn a stochastic sampling network that approximates samples from this
distribution. The benefits of the proposed algorithm include improved explo-
ration and compositionality that allows transfer-ring skills between tasks,
which we confirm in simulated experiments with swimming and walk-ing robots.
We also draw a connection to actor-critic methods, which can be viewed
perform-ing approximate inference on the corresponding energy-based model.,
author = Haarnoja, Tuomas and Tang, Haoran and Abbeel, Pieter and Levine,
Sergey, booktitle = International Conference on Machine Learning (ICML), file
= ::;::, title = Reinforcement Learning with Deep Energy-Based Policies, url =
https://arxiv.org/pdf/1702.08165.pdf, year = 2017
@inproceedingschristiano2017humanpreferences, abstract = For sophisticated
reinforcement learning (RL) systems to interact usefully with real-world
environments, we need to communicate complex goals to these systems. In this
work, we explore goals defined in terms of (non-expert) human preferences
between pairs of trajectory segments. We show that this approach can
effectively solve complex RL tasks without access to the reward function,
including Atari games and simulated robot locomotion, while providing feedback
on less than one percent of our agent’s interactions with the environment.
This reduces the cost of human oversight far enough that it can be practically
applied to state-of-the-art RL systems. To demonstrate the flexibility of our
approach, we show that we can successfully train complex novel behaviors with
about an hour of human time. These behaviors and environments are considerably
more complex than any that have been previously learned from human feedback.,
archivePrefix = arXiv, arxivId = 1706.03741, author = Christiano, Paul F and
Leike, Jan and Brown, Tom B and Martic, Miljan and Legg, Shane and Amodei,
Dario, booktitle = Advances in Neural Information Processing Systems (NIPS),
eprint = 1706.03741, file = ::;::, title = Deep reinforcement learning from
human preferences, url = https://arxiv.org/pdf/1706.03741.pdf
http://arxiv.org/abs/1706.03741, year = 2017 @bookszepesvari2010, abstract =
Reinforcement learning is a learning paradigm concerned with learning to
control a system so as to maximize a numerical performance measure that
expresses a long-term objective.What distin- guishes reinforcement learning
from supervised learning is that only partial feedback is given to the learner
about the learner’s predictions. Further, the predictions may have long term
effects through influencing the future state of the controlled system. Thus,
time plays a special role. The goal in reinforcement learning is to develop
efficient learning algorithms, as well as to understand the al- gorithms’
merits and limitations. Reinforcement learning is of great interest because of
the large number of practical applications that it can be used to address,
ranging from problems in artificial intelligence to operations research or
control engineering. In this book,we focus on those algorithms of
reinforcement learning that build on the powerful theory of dynamic
programming.We give a fairly comprehensive catalog of learning problems,
describe the core ideas, note a large number of state of the art algorithms,
followed by the discussion of their theoretical properties and limitations.,
author = Szepesvári, Csaba, booktitle = Synthesis Lectures on Artificial
Intelligence and Machine Learning, doi = 10.2200/S00268ED1V01Y201005AIM009,
file = ::, isbn = 9781608454921, issn = 1939-4608, number = 1, pages = 1–103,
title = Algorithms for Reinforcement Learning, url =
https://sites.ualberta.ca/ szepesva/papers/RLAlgsInMDPs.pdf, volume = 4, year
= 2010 @inproceedingsgoodfellow2014gan, abstract = We propose a new framework
for estimating generative models via an adversar-ial process, in which we
simultaneously train two models: a generative model G that captures the data
distribution, and a discriminative model D that estimates the probability that
a sample came from the training data rather than G. The train-ing procedure
for G is to maximize the probability of D making a mistake. This framework
corresponds to a minimax two-player game. In the space of arbitrary functions
G and D, a unique solution exists, with G recovering the training data
distribution and D equal to 1 2 everywhere. In the case where G and D are
defined by multilayer perceptrons, the entire system can be trained with
backpropagation. There is no need for any Markov chains or unrolled
approximate inference net-works during either training or generation of
samples. Experiments demonstrate the potential of the framework through
qualitative and quantitative evaluation of the generated samples., author =
Goodfellow, Ian J and Pouget-Abadie, Jean and Mirza, Mehdi and Xu, Bing and
Warde-Farley, David and Ozair, Sherjil and Courville, Aaron and Bengio,
Yoshua, booktitle = Advances in Neural Information Processing Systems (NIPS),
file = :Users/ashvin/Library/Application Support/Mendeley
Desktop/Downloaded/Goodfellow et al. - Unknown - Generative Adversarial
Nets.pdf:pdf;:Users/ashvin/code/kindlize/pdfs/deeprl/gans-
Goodfellow14.pdf:pdf, title = Generative Adversarial Nets, url =
https://arxiv.org/pdf/1406.2661.pdf, year = 2014 @articleSunderhauf2018,
abstract = The application of deep learning in robotics leads to very specific
problems and research questions that are typically not addressed by the
computer vision and machine learning communities. In this paper we discuss a
number of robotics-specific learning, reasoning, and embodiment challenges for
deep learning. We explain the need for better evaluation metrics, highlight
the importance and unique challenges for deep robotic learning in simulation,
and explore the spectrum between purely data-driven and model-driven
approaches. We hope this paper provides a motivating overview of important
research directions to overcome the current limitations, and help fulfill the
promising potentials of deep learning in robotics., archivePrefix = arXiv,
arxivId = 1804.06557, author = Sünderhauf, Niko and Brock, Oliver and
Scheirer, Walter and Hadsell, Raia and Fox, Dieter and Leitner, Jürgen and
Upcroft, Ben and Abbeel, Pieter and Burgard, Wolfram and Milford, Michael and
Corke, Peter, doi = 10.1177/ToBeAssigned, eprint = 1804.06557, file = ::, isbn
= 0037549716666, issn = 1756-8277, journal = International Journal of Robotics
Research, number = 4-5, pages = 405–420, title = The Limits and Potentials of
Deep Learning for Robotics, url = http://arxiv.org/abs/1804.06557, volume =
37, year = 2018 @articleKeramati, abstract = Humans learn to play video games
significantly faster than state-of-the-art rein-forcement learning (RL)
algorithms. Inspired by this, we introduce strategic object oriented
reinforcement learning (SOORL) to learn simple dynamics model through
automatic model selection and perform efficient planning with strategic
exploration. We compare different exploration strategies in a model-based
setting in which exact planning is impossible. Additionally, we test our
approach on perhaps the hardest Atari game Pitfall! and achieve significantly
improved exploration and performance over prior methods., archivePrefix =
arXiv, arxivId = 1806.00175, author = Keramati, Ramtin and Whang, Jay and Cho,
Patrick and Brunskill, Emma, eprint = 1806.00175, file = ::, pages = 1–10,
title = Strategic Object Oriented Reinforcement Learning
@inproceedingsTodorov2018, abstract = â We develop a general control
framework where a low-level optimizer is built into the robot dynamics. This
optimizer together with the robot constitute a goal directed dynamical system,
controlled on a higher level. The high level command is a cost function. It
can encode desired accelerations, end-effector poses, center of pressure, and
other intuitive features that have been studied before. Unlike the currently
popular quadratic programming framework, which comes with performance
guarantees at the expense of modeling flexibility, the optimization problem we
solve at each time step is non-convex and non-smooth. Nevertheless, by
exploiting the unique properties of the soft-constraint physics model we have
recently developed, we are able to design an efficient solver for goal
directed dynamics. It is only two times slower than the forward dynamics
solver, and is much faster than real time. The simulation results reveal that
complex movements can be generated via greedy optimization of simple costs.
This new computational infrastructure can facilitate teleoperation, feature-
based control, deep learning of control policies, and trajectory optimization.
It will become a standard feature in future releases of the MuJoCo simulator.,
author = Todorov, Emanuel, booktitle = IEEE International Conference on
Robotics and Automation (ICRA), file = ::;::, isbn = 9781538630808, title =
Goal Directed Dynamics, url = https://homes.cs.washington.edu/
todorov/papers/TodorovICRA18.pdf, year = 2018 @articleJohnson, author =
Johnson, Matthew James and Duvenaud, David and Wiltschko, Alexander B and
Datta, Sandeep R and Adams, Ryan P, file = ::;::;::, title = Composing
graphical models with neural networks for structured representations and fast
inference @inproceedingsArjovsky2017, author = Arjovsky, Martin and Chintala,
Soumith and Bottou, Léon, booktitle = International Conference on Machine
Learning (ICML), file = ::, title = Wasserstein GAN, url =
https://arxiv.org/pdf/1701.07875.pdf, year = 2017 @articleKalakrishnan2011,
abstract = We present a control architecture for fast quadruped locomotion
over rough terrain. We approach the problem by decomposing it into many sub-
systems, in which we apply state-of-the-art learning, planning, optimization,
and control techniques to achieve robust, fast locomotion. Unique features of
our control strategy include: (1) a system that learns optimal foothold
choices from expert demonstration using terrain templates, (2) a body
trajectory optimizer based on the Zero-Moment Point (ZMP) stability criterion,
and (3) a floating-base inverse dynamics controller that, in conjunction with
force control, allows for robust, compliant locomotion over unperceived
obstacles. We evaluate the performance of our controller by testing it on the
LittleDog quadruped robot, over a wide variety of rough terrains of varying
difficulty levels. The terrain that the robot was tested on includes rocks,
logs, steps, barriers, and gaps, with obstacle sizes up to the leg length of
the robot. We demonstrate the generalization ability of this controller by
presenting results from testing performed by an independent external test team
on terrain that has never been shown to us. 2011 The Author(s)., author =
Kalakrishnan, Mrinal and Buchli, Jonas and Pastor, Peter and Mistry, Michael
and Schaal, Stefan, doi = 10.1177/0278364910388677, file = ::;::, isbn =
0278-3649, issn = 0278-3649, journal = The International Journal of Robotics
Research, keywords = floating base inverse,locomotion planning and
control,quadruped locomotion,template learning,zmp optimization, number = 2,
pages = 236–258, title = Learning, planning, and control for quadruped
locomotion over challenging terrain, url =
https://www.researchgate.net/profile/Michael_Mistry/publication/220122611_Learning_planning_and_control_for_quadruped_locomotion_over_challenging_terrain/links/02bfe50ca950d456c2000000.pdf,
volume = 30, year = 2011 @inproceedingsLambert2018, abstract = We consider the
problems of learning forward models that map state to high-dimensional images
and inverse models that map high-dimensional images to state in robotics.
Specifically, we present a perceptual model for generating video frames from
state with deep networks, and provide a framework for its use in tracking and
prediction tasks. We show that our proposed model greatly outperforms standard
deconvolutional methods and GANs for image generation, producing clear, photo-
realistic images. We also develop a convolutional neural network model for
state estimation and compare the result to an Extended Kalman Filter to
estimate robot trajectories. We validate all models on a real robotic system.,
archivePrefix = arXiv, arxivId = 1710.11311, author = Lambert, Alexander and
Shaban, Amirreza and Raj, Amit and Liu, Zhen and Boots, Byron, booktitle =
IEEE International Conference on Robotics and Automation (ICRA), eprint =
1710.11311, file = ::, isbn = 9781538630808, title = Deep Forward and Inverse
Perceptual Models for Tracking and Prediction, url =
http://arxiv.org/abs/1710.11311, year = 2018 @articleWayne2018, abstract =
Animals execute goal-directed behaviours despite the limited range and scope
of their sensors. To cope, they explore environments and store memories
maintaining estimates of important information that is not presently
available. Recently, progress has been made with artificial intelligence (AI)
agents that learn to perform tasks from sensory input, even at a human level,
by merging reinforcement learning (RL) algorithms with deep neural networks,
and the excitement surrounding these results has led to the pursuit of related
ideas as explanations of non-human animal learning. However, we demonstrate
that contemporary RL algorithms struggle to solve simple tasks when enough
information is concealed from the sensors of the agent, a property called
”partial observability”. An obvious requirement for handling partially
observed tasks is access to extensive memory, but we show memory is not
enough; it is critical that the right information be stored in the right
format. We develop a model, the Memory, RL, and Inference Network (MERLIN), in
which memory formation is guided by a process of predictive modeling. MERLIN
facilitates the solution of tasks in 3D virtual reality environments for which
partial observability is severe and memories must be maintained over long
durations. Our model demonstrates a single learning agent architecture that
can solve canonical behavioural tasks in psychology and neurobiology without
strong simplifying assumptions about the dimensionality of sensory input or
the duration of experiences., archivePrefix = arXiv, arxivId = 1803.10760,
author = Wayne, Greg and Hung, Chia-Chun and Amos, David and Mirza, Mehdi and
Ahuja, Arun and Grabska-Barwinska, Agnieszka and Rae, Jack and Mirowski, Piotr
and Leibo, Joel Z. and Santoro, Adam and Gemici, Mevlana and Reynolds, Malcolm
and Harley, Tim and Abramson, Josh and Mohamed, Shakir and Rezende, Danilo and
Saxton, David and Cain, Adam and Hillier, Chloe and Silver, David and
Kavukcuoglu, Koray and Botvinick, Matt and Hassabis, Demis and Lillicrap,
Timothy, eprint = 1803.10760, file = ::, title = Unsupervised Predictive
Memory in a Goal-Directed Agent, url = http://arxiv.org/abs/1803.10760, year =
2018 @articleMirowski2018, abstract = Navigating through unstructured
environments is a basic capability of intelligent creatures, and thus is of
fundamental interest in the study and development of artificial intelligence.
Long-range navigation is a complex cognitive task that relies on developing an
internal representation of space, grounded by recognisable landmarks and
robust visual processing, that can simultaneously support continuous self-
localisation (”I am here”) and a representation of the goal (”I am going
there”). Building upon recent research that applies deep reinforcement