Class-Agnostic Object Detection with Multi-modal Transformer

Muhammad Maaz*, Hanoona Rasheed, Salman Khan, Fahad Shahbaz Khan, Rao Muhammad Anwer, Ming Hsuan Yang

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingsScientificpeer-review

18 Citations (Scopus)

Abstract

What constitutes an object? This has been a long-standing question in computer vision. Towards this goal, numerous learning-free and learning-based approaches have been developed to score objectness. However, they generally do not scale well across new domains and novel objects. In this paper, we advocate that existing methods lack a top-down supervision signal governed by human-understandable semantics. For the first time in literature, we demonstrate that Multi-modal Vision Transformers (MViT) trained with aligned image-text pairs can effectively bridge this gap. Our extensive experiments across various domains and novel objects show the state-of-the-art performance of MViTs to localize generic objects in images. Based on the observation that existing MViTs do not include multi-scale feature processing and usually require longer training schedules, we develop an efficient MViT architecture using multi-scale deformable attention and late vision-language fusion. We show the significance of MViT proposals in a diverse range of applications including open-world object detection, salient and camouflage object detection, supervised and self-supervised detection tasks. Further, MViTs can adaptively generate proposals given a specific language query and thus offer enhanced interactability. Code: https://git.io/J1HPY.

Original languageEnglish
Title of host publicationComputer Vision – ECCV 2022 - 17th European Conference, Proceedings
EditorsShai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, Tal Hassner
PublisherSpringer
Pages512-531
Number of pages20
ISBN (Print)978-3-031-20079-3
DOIs
Publication statusPublished - 2022
MoE publication typeA4 Conference publication
EventEuropean Conference on Computer Vision - Tel Aviv, Israel
Duration: 23 Oct 202227 Oct 2022
Conference number: 17
https://eccv2022.ecva.net

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
PublisherSpringer
Volume13670 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

ConferenceEuropean Conference on Computer Vision
Abbreviated titleECCV
Country/TerritoryIsrael
CityTel Aviv
Period23/10/202227/10/2022
Internet address

Keywords

  • Class-agnostic
  • Object detection
  • Vision transformers

Fingerprint

Dive into the research topics of 'Class-Agnostic Object Detection with Multi-modal Transformer'. Together they form a unique fingerprint.

Cite this