Comparing and combining unimodal methods for multimodal recognition

Satoru Ishikawa, Jorma Laaksonen

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingsScientificpeer-review


Multimodal recognition has recently become more attractive and common method in multimedia information retrieval. In many cases it shows better recognition results than using only unimodal methods. Most of current multimodal recognition methods still depend on unimodal recognition results. Therefore, in order to get better recognition performance, it is important to choose suitable features and classification models for each unimodal recognition task. In this paper, we research several unimodal recognition methods, features for them and their combination techniques, in the application setup of concept detection in image-text data. For image features, we use GoogLeNet deep convolutional neural network (DCNN) activation features and semantic concept vectors. For text features, we use simple binary vectors for tags and word2vec vectors. As the concept detection model, we apply the Multimodal Deep Boltzmann Machine (DBM) model and the Support Vector Machine (SVM) with the linear homogeneous kernel map and the non-linear radial basis function (RBF) kernel. The experimental results with the MIRFLICKR-1M data set show that the Multimodal DBM or the non-linear SVM approaches produce equally good results within the margins of statitistical variation.

Original languageEnglish
Title of host publication2016 14th International Workshop on Content-Based Multimedia Indexing, CBMI 2016
ISBN (Electronic)9781467386951
Publication statusPublished - 27 Jun 2016
MoE publication typeA4 Conference publication
EventInternational Workshop on Content-Based Multimedia Indexing - Bucharest, Romania
Duration: 15 Jun 201617 Jun 2016
Conference number: 14


WorkshopInternational Workshop on Content-Based Multimedia Indexing
Abbreviated titleCBMI


Dive into the research topics of 'Comparing and combining unimodal methods for multimodal recognition'. Together they form a unique fingerprint.

Cite this