Using semantic features to improve large-scale visual concept detection

Currently there are many multimedia benchmarks and databases available with a predefined set of concepts for which detectors can be formed or are even already available. One can use these background concepts to form semantic concept vectors for each image or video in the database by concatenating the concept prediction outputs. In this paper we investigate the use of such semantic concept features for detecting novel concepts in two large-scale experiments: the TRECVID 2012 evaluation with 800 hours of video data, and MIRFLICKR with 1 million images. We show that the detection performance can improve significantly over using visual features only. In some applications, computationally expensive kernel classifiers cannot be used in the detection phase, and our experiments show a consistent significant improvement using fast linear classifiers when we replace visual features with the semantic concept feature. We also propose a Self-Organising Map-based method which affords fast training-free detection and intuitive visualisation properties.