Speeding Up Incremental Learning Using Data Efficient Guided Exploration

Murtaza Hazara, Ville Kyrki

Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review

2 Citations (Scopus)
210 Downloads (Pure)


To cope with varying conditions, motor primitives (MPs) must support generalization over task parameters to avoid learning separate primitives for each situation. In this regard, deterministic and probabilistic models have been proposed for generalizing MPs to new task parameters, thus providing limited generalization. Although generalization of MPs using probabilistic models has been studied, it is not clear how such generalizable models can be learned efficiently. Reinforcement learning can be more efficient when the exploration process is tuned with data uncertainty, thus reducing unnecessary exploration in a data-efficient way. We propose an empirical Bayes method to predict uncertainty and utilize it for guiding the exploration process of an incremental learning framework. The online incremental learning framework uses a single human demonstration for constructing a database of MPs. The main ingredients of the proposed framework are a global parametric model (GPDMP) for generalizing MPs for new situations, a model-free policy search agent for optimizing the failed predicted MPs, model selection for controlling the complexity of GPDMP, and empirical Bayes for extracting the uncertainty of MPs prediction. Experiments with a ball-in-a-cup task demonstrate that the global GPDMP model generalizes significantly better than linear models and Locally Weighted Regression especially in terms of extrapolation capability. Furthermore, the model selection has successfully identified the required complexity of GPDMP even with few training samples while satisfying the Occam Razor’s prinicple. Above all, the uncertainty predicted by the proposed empirical Bayes approach successfully guided the exploration process of the model-free policy search. The experiments indicated statistically significant improvement of learning speed over covariance matrix adaptation (CMA) with a significance of p = 0.002.
Original languageEnglish
Title of host publicationProceedings of the 2018 IEEE International Conference on Robotics and Automation, ICRA 2018
Number of pages8
ISBN (Electronic)978-1-5386-3081-5
ISBN (Print)978-1-5386-3082-2
Publication statusPublished - 2018
MoE publication typeA4 Article in a conference publication
EventIEEE International Conference on Robotics and Automation - Brisbane, Australia
Duration: 21 May 201825 May 2018

Publication series

NameIEEE International Conference on Robotics and Automation
ISSN (Print)2152-4092
ISSN (Electronic)2379-9552


ConferenceIEEE International Conference on Robotics and Automation
Abbreviated titleICRA


  • guided exploration
  • exploration
  • Reinforcement Learning
  • incremental learning
  • learning from demonstration
  • Robotics

Fingerprint Dive into the research topics of 'Speeding Up Incremental Learning Using Data Efficient Guided Exploration'. Together they form a unique fingerprint.

Cite this