InDepth: Real-time Depth Inpainting for Mobile Augmented Reality

Yunfan Zhang, Tim Scargill, Ashutosh Vaishnav, Gopika Premsankar, Mario Di Francesco, Maria Gorlatova

Research output: Contribution to journalArticleScientificpeer-review

14 Citations (Scopus)
213 Downloads (Pure)

Abstract

Mobile Augmented Reality (AR) demands realistic rendering of virtual content that seamlessly blends into the physical environment. For this reason, AR headsets and recent smartphones are increasingly equipped with Time-of-Flight (ToF) cameras to acquire depth maps of a scene in real-time. ToF cameras are cheap and fast, however, they suffer from several issues that affect the quality of depth data, ultimately hampering their use for mobile AR. Among them, scale errors of virtual objects - appearing much bigger or smaller than what they should be - are particularly noticeable and unpleasant. This article specifically addresses these challenges by proposing InDepth, a real-time depth inpainting system based on edge computing. InDepth employs a novel deep neural network (DNN) architecture to improve the accuracy of depth maps obtained from ToF cameras. The DNN fills holes and corrects artifacts in the depth maps with high accuracy and eight times lower inference time than the state of the art. An extensive performance evaluation in real settings shows that InDepth reduces the mean absolute error by a factor of four with respect to ARCore DepthLab. Finally, a user study reveals that InDepth is effective in rendering correctly-scaled virtual objects, outperforming DepthLab.

Original languageEnglish
Article number37
Pages (from-to)1-25
Number of pages25
JournalProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Volume6
Issue number1
DOIs
Publication statusPublished - Mar 2022
MoE publication typeA1 Journal article-refereed

Keywords

  • Augmented reality
  • depth inpainting
  • depth sensing
  • edge computing
  • user experience

Fingerprint

Dive into the research topics of 'InDepth: Real-time Depth Inpainting for Mobile Augmented Reality'. Together they form a unique fingerprint.

Cite this