Unsupervised Workflow Extraction from First-person Video of Mechanical Assembly

Recently, Augmented Reality (AR) applications have proved to help improve the efficiency in accomplishing assembly tasks. However, due to the lack of approaches to automatic workflow extraction, the existing AR-based assembly assistance applications require manual authoring, which hampers scalability. Moreover, most of these applications only support information visualization and video documentation. To tackle the challenge of scalability and to enable more intelligent functionalities, such as real-time quality control, we propose in this paper a novel solution for unsupervised workflow extraction from first-person video of mechanical assembly, without any pre-labeled training data or pre-trained classifiers. Our solution automatically discovers a sequence of working steps and the meaningful operations in each step from the input video, and describes the extracted workflow information with semantics. Preliminary evaluation demonstrates the feasibility of our solution and highlights the technical challenges.