Spatio-temporal Relation Modeling for Few-shot Action Recognition

Anirudh Thatipelli, Sanath Narayan, Salman Khan, Rao Muhammad Anwer, Fahad Shahbaz Khan, Bernard Ghanem

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingsScientificpeer-review

46 Citations (Scopus)


We propose a novel few-shot action recognition framework, STRM, which enhances class-specific feature discriminability while simultaneously learning higher-order temporal representations. The focus of our approach is a novel spatio-temporal enrichment module that aggregates spatial and temporal contexts with dedicated local patch-level and global frame-level feature enrichment sub-modules. Local patch-level enrichment captures the appearance-based characteristics of actions. On the other hand, global frame-level enrichment explicitly encodes the broad temporal context, thereby capturing the relevant object features over time. The resulting spatio-temporally enriched representations are then utilized to learn the relational matching between query and support action sub-sequences. We further introduce a query-class similarity classifier on the patch-level enriched features to enhance class-specific feature discriminability by reinforcing the feature learning at different stages in the proposed framework. Experiments are performed on four few-shot action recognition benchmarks: Kinetics, SSv2, HMDB51 and UCF101. Our extensive ablation study reveals the benefits of the proposed contributions. Furthermore, our approach sets a new state-of-the-art on all four benchmarks. On the challenging SSv2 benchmark, our approach achieves an absolute gain of 3.5% in classification accuracy, as compared to the best existing method in the literature.

Original languageEnglish
Title of host publicationProceedings - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022
Number of pages10
ISBN (Electronic)978-1-6654-6946-3
Publication statusPublished - 2022
MoE publication typeA4 Conference publication
EventIEEE Conference on Computer Vision and Pattern Recognition - New Orleans, United States
Duration: 18 Jun 202224 Jun 2022

Publication series

NameIEEE Computer Society Conference on Computer Vision and Pattern Recognition
ISSN (Print)1063-6919
ISSN (Electronic)2575-7075


ConferenceIEEE Conference on Computer Vision and Pattern Recognition
Abbreviated titleCVPR
Country/TerritoryUnited States
CityNew Orleans


  • Action and event recognition
  • categorization
  • Recognition: detection
  • retrieval
  • Transfer/low-shot/long-tail learning


Dive into the research topics of 'Spatio-temporal Relation Modeling for Few-shot Action Recognition'. Together they form a unique fingerprint.

Cite this