Advances in Optimizing Physically Simulated Movements

Amin Babadi

Research output: ThesisDoctoral ThesisCollection of Articles


A common and challenging problem in robotics and video games is how to synthesize movements using optimal control of physically simulated environments. Thanks to the deep learning techniques, the performance of movement optimization has experienced major growth in the past decade. However, the methods still suffer from high sample complexity and require extensive reference datasets. These are among the main reasons why these methods are not yet common in real-life applications such as robotics and games. In this dissertation, methods and algorithms are developed for analyzing and solving movement optimization problems. The developed approaches do not require any reference datasets and are applicable to different characters (e.g., humanoids, quadrupeds) and tasks (e.g., locomotion, getting up after falling). The dissertation focuses on trajectory optimization and reinforcement learning (RL), which are two main classes of algorithms for movement optimization. Although both these classes are based on learning through evaluating randomly sampled actions, they use different approaches for modeling and exploring the search space. This dissertation draws inspiration from both classes to reach a better understanding and mitigating the complexities of movement optimization. This dissertation contributes by combining ideas from Covariance Matrix Adaptation Evolution Strategy (CMA-ES)—a classic (trajectory) optimization algorithm—and Proximal Policy Optimization (PPO), to propose a novel RL algorithm that mitigates the premature convergence problem of PPO. It is also explored how trajectory optimization can be used for synthesizing high-quality animation movements that can be used as reference data for RL. Furthermore, the dissertation contributes novel visualizations of movement optimization landscapes, advancing the understanding of different optimization schemes and problem modifications. Finally, motivated by the findings of the visualization study, a novel method is developed for implementing task-agnostic action spaces that can improve the efficiency of both trajectory optimization and RL.
Translated title of the contributionAdvances in Optimizing Physically Simulated Movements
Original languageEnglish
QualificationDoctor's degree
Awarding Institution
  • Aalto University
  • Hämäläinen, Perttu, Supervising Professor
Print ISBNs978-952-64-0873-6
Electronic ISBNs978-952-64-0874-3
Publication statusPublished - 2022
MoE publication typeG5 Doctoral dissertation (article)


  • movement optimization
  • trajectory optimization
  • policy optimization
  • hierarchical reinforcement learning


Dive into the research topics of 'Advances in Optimizing Physically Simulated Movements'. Together they form a unique fingerprint.

Cite this