Image pseudo tag generation with Deep Boltzmann machine anc topic-concept similarity map

Satoru Ishikawa, Jorma Laaksonen, Juha Karhunen

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingsScientificpeer-review


General purpose search engines are used for searching not only plain text but also multimedia information. In multimodal search, it is common to use multiple queries to find the demanded information in the different media modalities. In most cases, however, it is hard to prepare such multimodal search queries. In addition, the semantic connection between the individual modalities is often weak or totally lacking in such multimodal search. Hence, single modality searching makes it hard to find the searched for information in the multimodal domain. In this paper we improve the Deep Boltzmann Machine applied to multimodal search by using GoogLeNet deep convolutional neural network and semantic concept features. We also propose a supervised method to produce a similarity map between hidden topics in text documents and the visual concepts in corresponding images, and an unsupervised method which uses the hidden topics in the documents as pseudo labels. The model can be used to infer and generate pseudo tags for untagged input query images in order to complement an image-only query to a multimodal one. The classification results with pseudo tag inputs show in our experiments improvement compared to the original tag inputs.

Original languageEnglish
Title of host publication2017 International Joint Conference on Neural Networks, IJCNN 2017 - Proceedings
Number of pages8
ISBN (Electronic)9781509061815
Publication statusPublished - 30 Jun 2017
MoE publication typeA4 Conference publication
EventInternational Joint Conference on Neural Networks - Anchorage, United States
Duration: 14 May 201719 May 2017


ConferenceInternational Joint Conference on Neural Networks
Abbreviated titleIJCNN
Country/TerritoryUnited States


  • Visualization
  • Semantics
  • Search engines
  • Automobiles
  • Multimedia communication
  • Resource management


Dive into the research topics of 'Image pseudo tag generation with Deep Boltzmann machine anc topic-concept similarity map'. Together they form a unique fingerprint.

Cite this