Learning Cross-Aligned Latent Embeddings for Zero-Shot Cross-Modal Retrieval

Zero-Shot Cross-Modal Retrieval (ZS-CMR) is an emerging research hotspot that aims to retrieve data of new classes across different modality data. It is challenging for not only the heterogeneous distributions across different modalities, but also the inconsistent semantics across seen and unseen classes. A handful of recently proposed methods typically borrow the idea from zero-shot learning, i.e., exploiting word embeddings of class labels (i.e., class-embeddings) as common semantic space, and using generative adversarial network (GAN) to capture the underlying multimodal data structures, as well as strengthen relations between input data and semantic space to generalize across seen and unseen classes. In this paper, we propose a novel method termed Learning Cross-Aligned Latent Embeddings (LCALE) as an alternative to these GAN based methods for ZS-CMR. Unlike using the class-embeddings as the semantic space, our method seeks for a shared low-dimensional latent space of input multimodal features and class-embeddings by modality-specific variational autoencoders. Notably, we align the distributions learned from multimodal input features and from class-embeddings to construct latent embeddings that contain the essential cross-modal correlation associated with unseen classes. Effective cross-reconstruction and cross-alignment criterions are further developed to preserve class-discriminative information in latent space, which benefits the efficiency for retrieval and enable the knowledge transfer to unseen classes. We evaluate our model using four benchmark datasets on image-text retrieval tasks and one large-scale dataset on image-sketch retrieval tasks. The experimental results show that our method establishes the new state-of-the-art performance for both tasks on all datasets.

Sketch-Based 3D Model Retrieval Using Attributes

With the rapid growth of available 3D models on the Internet, how to retrieve 3D models based on hand-drawn sketch retrieval are becoming increasingly important. This article proposes a new sketch-based 3D model retrieval approach. This approach is different from current methods that make use of low-level visual features to capture the search intention of users. The proposed method uses two kinds of semantic attributes, including pre-defined attributes and latent attributes. Specifically, pre-defined attributes are defined manually which can provide prior knowledge about different sketch categories and latent-attributes are more discriminative which can differentiate sketch categories at a finer level. Therefore, these semantic attributes can provide a more descriptive and discriminative meaningful representation than low-level feature descriptors. The experiment results demonstrate that this proposed method can achieve superior performance over previously proposed sketch-based 3D model retrieval methods.