Advances in Optimizing Physically Simulated Movements

A common and challenging problem in robotics and video games is how to synthesize movements using optimal control of physically simulated environments. Thanks to the deep learning techniques, the performance of movement optimization has experienced major growth in the past decade. However, the methods still suffer from high sample complexity and require extensive reference datasets. These are among the main reasons why these methods are not yet common in real-life applications such as robotics and games. In this dissertation, methods and algorithms are developed for analyzing and solving movement optimization problems. The developed approaches do not require any reference datasets and are applicable to different characters (e.g., humanoids, quadrupeds) and tasks (e.g., locomotion, getting up after falling). The dissertation focuses on trajectory optimization and reinforcement learning (RL), which are two main classes of algorithms for movement optimization. Although both these classes are based on learning through evaluating randomly sampled actions, they use different approaches for modeling and exploring the search space. This dissertation draws inspiration from both classes to reach a better understanding and mitigating the complexities of movement optimization. This dissertation contributes by combining ideas from Covariance Matrix Adaptation Evolution Strategy (CMA-ES)—a classic (trajectory) optimization algorithm—and Proximal Policy Optimization (PPO), to propose a novel RL algorithm that mitigates the premature convergence problem of PPO. It is also explored how trajectory optimization can be used for synthesizing high-quality animation movements that can be used as reference data for RL. Furthermore, the dissertation contributes novel visualizations of movement optimization landscapes, advancing the understanding of different optimization schemes and problem modifications. Finally, motivated by the findings of the visualization study, a novel method is developed for implementing task-agnostic action spaces that can improve the efficiency of both trajectory optimization and RL.