Pantomime: Mid-Air Gesture Recognition with Sparse Millimeter-Wave Radar Point Clouds

Sameera Palipana, Dariush Salami, Luis A. Leiva, Stephan Sigg

Research output: Contribution to journalArticleScientificpeer-review

1 Citation (Scopus)

Abstract

We introduce Pantomime, a novel mid-air gesture recognition system exploiting spatiooral properties of millimeter-wave radio frequency (RF) signals. Pantomime is positioned in a unique region of the RF landscape: mid-resolution mid-range high-frequency sensing, which makes it ideal for motion gesture interaction. We configure a commercial frequency-modulated continuous-wave radar device to promote spatial information over the temporal resolution by means of sparse 3D point clouds and contribute a deep learning architecture that directly consumes the point cloud, enabling real-time performance with low computational demands. Pantomime achieves 95% accuracy and 99% AUC in a challenging set of 21 gestures articulated by 41 participants in two indoor environments, outperforming four state-of-the-art 3D point cloud recognizers. We further analyze the effect of the environment in 5 different indoor environments, the effect of articulation speed, angle, and the distance of the person up to 5m. We have publicly made available the collected mmWave gesture dataset consisting of nearly 22,000 gesture instances along with our radar sensor configuration, trained models, and source code for reproducibility. We conclude that pantomime is resilient to various input conditions and that it may enable novel applications in industrial, vehicular, and smart home scenarios.

Original languageEnglish
Article number3448110
Number of pages27
JournalProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Volume5
Issue number1
DOIs
Publication statusPublished - 29 Mar 2021
MoE publication typeA1 Journal article-refereed

Keywords

  • deep learning
  • gesture recognition
  • radar sensing

Fingerprint

Dive into the research topics of 'Pantomime: Mid-Air Gesture Recognition with Sparse Millimeter-Wave Radar Point Clouds'. Together they form a unique fingerprint.

Cite this