Image pseudo tag generation with Deep Boltzmann machine anc topic-concept similarity map

Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review

Researchers

Research units

Abstract

General purpose search engines are used for searching not only plain text but also multimedia information. In multimodal search, it is common to use multiple queries to find the demanded information in the different media modalities. In most cases, however, it is hard to prepare such multimodal search queries. In addition, the semantic connection between the individual modalities is often weak or totally lacking in such multimodal search. Hence, single modality searching makes it hard to find the searched for information in the multimodal domain. In this paper we improve the Deep Boltzmann Machine applied to multimodal search by using GoogLeNet deep convolutional neural network and semantic concept features. We also propose a supervised method to produce a similarity map between hidden topics in text documents and the visual concepts in corresponding images, and an unsupervised method which uses the hidden topics in the documents as pseudo labels. The model can be used to infer and generate pseudo tags for untagged input query images in order to complement an image-only query to a multimodal one. The classification results with pseudo tag inputs show in our experiments improvement compared to the original tag inputs.

Details

Original languageEnglish
Title of host publication2017 International Joint Conference on Neural Networks, IJCNN 2017 - Proceedings
Publication statusPublished - 30 Jun 2017
MoE publication typeA4 Article in a conference publication
EventInternational Joint Conference on Neural Networks - Anchorage, United States
Duration: 14 May 201719 May 2017

Conference

ConferenceInternational Joint Conference on Neural Networks
Abbreviated titleIJCNN
CountryUnited States
CityAnchorage
Period14/05/201719/05/2017

    Research areas

  • Visualization, Semantics, Search engines, Automobiles, Multimedia communication, Resource management

ID: 15875392