Integrating Multi-Purpose Natural Language Understanding, Robot's Memory, and Symbolic Planning for Task Execution in Humanoid Robots

Research output: Contribution to journalArticle


  • Mirko Wächter
  • Ekaterina Ovchinnikova
  • Valerij Wittenbeck
  • Peter Kaiser
  • Sandor Szedmak

  • Wail Mustafa
  • Dirk Kraft
  • Norbert Krüger
  • Justus Piater
  • Tamim Asfour

Research units

  • University of Southern Denmark
  • University of Innsbruck
  • Karlsruhe Institute of Technology


We propose an approach for instructing a robot using natural language to solve complex tasks in a dynamic environment. In this study, we elaborate on a framework that allows a humanoid robot to understand natural language, derive symbolic representations of its sensorimotor experience, generate complex plans according to the current world state, and monitor plan execution. The presented development supports replacing missing objects and suggesting possible object locations. It is a realization of the concept of structural bootstrapping developed in the context of the European project Xperience. The framework is implemented within the robot development environment ArmarX. We evaluate the framework on the humanoid robot ARMAR-III in the context of two experiments: a demonstration of the real execution of a complex task in the kitchen environment on ARMAR-III and an experiment with untrained users in a simulation environment.


Original languageEnglish
Pages (from-to)148-165
JournalRobotics and Autonomous Systems
Publication statusPublished - 2018
MoE publication typeA1 Journal article-refereed

    Research areas

  • Structural bootstrapping, Natural language understanding, Planning, Task execution, Object replacement, Humanoid robotics

ID: 16059365