1 Introduction

Over the last decade there has been a massive explosion of multimedia content on the web. More and more people upload pictures tagged with texts to the Internet. Articles describing news and technologies include a lot of pictures and texts. The presence of massive multimodal data on the Internet brings a growing demand for cross-modal retrieval, such as using a text query to search for images, and in turn using an image query to search for texts.

Many unsupervised methods [7, 8, 14, 26, 34] have been proposed for cross-modal retrieval. These methods focus on modeling intra-pair correlation between different views. However the inter-pair correlation which corresponds to the semantic consistency of retrieved results is ignored. To address this problem, several methods [16, 22, 23, 25, 41, 46] are proposed to regularize the latent space with the help of the supervised semantic information. Semantic correlation matching(SCM) [23] is a very typical method to conduct supervised cross-modal retrieval. The learning procedure of SCM can be divided into two learning stages (Fig. 1a): in the first learning stage, SCM uses 2-view canonical correlation analysis(CCA) to project image view and text view into a common latent space. In the second learning stage, SCM extracts high-level semantic representations of image view and text view based on the features in the latent space. Similarity measured in the semantic space is used for cross-modal retrieval. However, the supervised semantic is only considered in the second stage of SCM. This drawback of SCM is limited by the fact that traditional CCA is a 2-view method for building correlation. To maximize the value of supervised semantics, we propose to conduct supervised learning in both stages, as shown in Fig. 1b.

Fig. 1
figure 1

Frameworks of SCM and our method

Full size image

Deep learning models have achieved great success in representation learning recently. Deep models use a cascade of multiple layers of nonlinear processing units for feature extraction and possess significantly greater representation power than traditional shallow models. However, deep learning methods are prone to over-fitting because of the added layers of abstraction, which allow them to model rare dependencies in the training data. To address this problem, several methods [6, 37, 43] are proposed to learn better features with multiple supervisory signals. Multiple supervisory signals are beneficial for enhancing the discriminative power of the deeply learned features, i.e., center loss is added to reduce the intra-class features variations in face recognition. What’s more, these supervisory signals can be viewed as the regularization term for each other, thus overcome overfitting. Inspired by this fact, we propose to combine correlation learning with representation learning in the first stage. In the second stage, the deep network named C2Net is trained with the joint supervision of contrastive loss [13] and center loss [43].

1.1 Related work

Many approaches have been proposed to develop solutions to cross-modal retrieval tasks. We classify these approaches into two categories: conventional shallow models and deep models.

Shallow models

Grangier et al. [12] propose a passive-aggressive model, which is the first attempt to address the problem of ranking images retrieved by text queries. Rasiwasia et al. [23] propose correlation matching to map the features of images and texts into a common latent space using CCA. Complete introduction and recent extensions about CCA can be found in [35, 36]. Based on CCA, various variants [2, 5, 27, 48] are proposed to model the multi-modal correlations. Gong et al. [11] expand two-view CCA to three-view CCA by incorporating a third view that captures high-level image semantics, represented either by a single category or multiple non-mutually-exclusive concepts. Wu et al. [45] formalize the retrieval task as a ranking problem(Bi-CMSRM) similar to [12] and try to learn a common latent space for images and texts as CCA. The latent space embedding of Bi-CMSRM is discriminatively learned by the structural large margin learning for optimization with certain ranking criteria (mean average precision) directly. Other algorithms are also proposed to deal with cross-modal retrieval problems, such as partial least square (PLS) [24], Bilinear Model(BLM) [18, 21, 38] and etc.

Deep models

Srivastava et al. [34] propose to learn a generative model of the joint space of image and text inputs using Deep Belief Network(DBN), which consists of multiple stacked restricted boltzmann machine(RBM [28]). Gaussian RBM [42] and replicated softmax RBM [15] are used to model the real-valued feature vectors for image and the discrete sparse word count vectors for text, respectively. Based on DBN, Feng et al. [6] propose to learn the latent space of image and text inputs by correspondence autoencoder(Corr-AE). Corr-AE defines a novel optimal objective, which minimizes a linear combination of representation learning errors for each modality and correlation learning error between hidden representations of two modalities. Moreover, we notice several deep learning methods [7, 8, 29, 44] for learning a joint embedding space of image and text very recently. Andrew et al. [1] build a deep architecture (DCCA)to learn complex nonlinear transformations of two views of data such that the resulting representations are highly linearly correlated. DCCA can be viewed as a nonlinear extension of traditional linear CCA. Though the representation power improves, DCCA is easy to over-fitting, especially when the datasets are not big enough. Peng et al. [22] propose the cross-media multiple deep network (CMDN) to exploit the complex cross-media correlation by hierarchical learning. Huang et al. [16] propose cross-modal deep metric learning with multi-task regularization (CDMLMR). Recently, we also notice several impressive methods [9, 10, 30,31,32,33, 39, 40, 47] for cross-modal retrieval and image analysis.

1.2 Contributions

The main contributions of this work are summarized as follows:

  1. 1)

    Inspired by SCM, we propose a two-stage deep learning method for supervised cross-modal retrieval. We expand CCA from 2-view to 3-view to conduct supervised learning in both stages, which could maximize the value of supervised semantics.

  2. 2)

    Multiple supervisory signals are successfully combined in both stages to learn better feature representations, and meanwhile, overcome the overfitting.

  3. 3)

    The promising results on two public datasets demonstrate the effectiveness of the proposed algorithm.

2 Methods

In this section we describe details of 3-view DCCA in the first learning stage and details of C2Net in the second learning stage.

2.1 3-view DCCA

The first learning stage of SCM is unsupervised due to the fact that traditional CCA is a 2-view method for building correlation between image view and text view. To reinforce the value of supervised semantics, we expand CCA from 2-view to 3-view and maximize the correlation among 3 views simultaneously. To learn nonlinear correlation between these 3 views, we embed 3-view CCA into a deep architecture. Meanwhile, in order to overcome over-fitting of deep network, we add the reconstruct loss of each view into the loss function.

In this section, we first give the difference between 2-view CCA and 3-view CCA. Next, we introduce the architecture of 3-view DCCA and the formulation details.

2.1.1 3-view CCA

Practical models for cross-modal retrieval tasks should meet two requirements. First, the top one result should be accurate. It is a big challenge because image features are noisy and text features are ambiguous. Second, the top n results should be relevant to the query. In other words, image and text features which are coherent in semantics should be close to each other in the latent space for cross-modal retrieval. 2-view CCA (Fig. 2a) which only maximize the correlation between image and corresponding text does not meet the second requirement. To address this problem, We use the 3rd semantic view to unite all the relevant paired samples (Fig. 2b).

Fig. 2
figure 2

a Paired points in text and image spaces are mapped into a common latent space independently in 2-view CCA. b Mapped points from relevant pairs are clustered in the latent space according to their semantic information in 3-view CCA

Full size image

2.1.2 Model architecture

As shown in Fig. 3, 3-view DCCA includes three subnets(feedforward neural network), each of them corresponds to different view(image view, text view and semantic view) of data. The semantic view, is added to regularize the common latent space of text view and image view. Each subnet includes four kinds of layers: input layer, hidden layer, output layer and reconstruct layer. The representations learned from the output layer are furnished to the second learning stage.

Fig. 3
figure 3

The architecture of 3view DCCA

Full size image

2.1.3 Formulation

Assuming that we have n training images and each image is associated with a k1-dimensional visual feature vector, a k2-dimensional text feature vector, and a k3-dimensional semantic feature vector. The respective vectors are stacked as rows in matrices X1 ∈Rn∗k1, X2 ∈Rn∗k2, and X3 ∈Rn∗k3. Traditional two-view CCA tries to find matrices W1 ∈Rk1∗k and W2 ∈Rk2∗k where text and image are projected into a k-dimensional latent space so that the correlation between text and image could be maximized. This procedure is equal to minimizing the distance between text and image in the latent space. The loss function for two-view CCA is given by:

$$ L_{cca(1,2)}=-corr(X_{1}*W_{1},X_{2}*W_{2}) $$
(1)

Therefore, the loss function for our three-view CCA model includes the correlation loss of every two-view:

$$ L_{cca(1,2,3)}=L_{cca(1,2)}+L_{cca(2,3)}+L_{cca(3,1)} $$
(2)
$$ L_{cca(i,j)}=-corr(X_{i}*W_{i},X_{j}*W_{j}) $$
(3)

To handle complex transformations, we define three nonlinear functions f1(X1,Θ1), f2(X2,Θ2),f3(X3,Θ3) and update the equations above:

$$ L_{cca(i,j)}=-corr(f_{i}(X_{i},{\Theta}_{i}),f_{j}(X_{j},{\Theta}_{j})) $$
(4)

In our deep model, Ci = fi(Xi,Θi) denotes the representation of the output layer in the ith subnet. Meanwhile, in order to overcome over-fitting, we add the reconstruction loss and regularization penalty:

$$\begin{array}{@{}rcl@{}} Loss &=& L_{cca(1,2,3)}+\alpha\cdot\left( L_{1,1}+L_{2,2}+L_{3,3}\right)\\ &&+~\lambda\left( \|{\Theta}_{1}\|^{2}_{F}+\|{\Theta}_{2}\|^{2}_{F}+\|{\Theta}_{3}\|^{2}_{F}\right) \end{array} $$
(5)
$$L_{i,i}=\left\|X_{i}-\hat{X_{i}}\right\|^{2}_{F}\ i = 1,...,3 $$

where, \(\hat {X_{i}}\) is the reconstruction data from Xi. The parameters Θ1,Θ2,Θ3 are trained to optimize this quantity using gradient-based optimization. In the output layer, the gradient of corr(Ci,Cj)with respect to Ci is calculated following the solution in [1]. Let \(\overline {C}_{i}=C_{i}-\frac {1}{n}C_{i}\) be the centered data matrix and define \( \hat {\sum }_{12}{ = \frac {1}{n-1}}\overline {C}_{1}\overline {C}_{2}^{\prime }\), \(\hat {\sum }_{11}{=\frac {1}{n-1}}(\overline {C}_{1}\overline {C}_{1}^{\prime }+r_{1}I)\) for regularization constant r1, and \(\hat {\sum }_{22}{=\frac {1}{n-1}}(\overline {C}_{2}\overline {C}_{2}^{\prime }+r_{2}I)\) for regularization constant r2. The total correlation corr(C1,C2) is the sum of the singular values of the matrix \(T=\hat {\sum }_{11}^{-\frac {1}{2}}\hat {\sum }_{12}\hat {\sum }_{22}^{-\frac {1}{2}}\). If the singular value decomposition of T is T = UDV′, then

$$\begin{array}{@{}rcl@{}} \frac{\partial corr(C_{1},C_{2})}{\partial C_{1}}&=&\frac{1}{n-1}\left( -\hat{\sum}_{11}^{-\frac{1}{2}}UDU^{\prime}\hat{\sum}_{11}^{-\frac{1}{2}}\overline{C}_{1} +~\hat{\sum}_{11}^{-\frac{1}{2}}UV^{\prime}\hat{\sum}_{22}^{-\frac{1}{2}}\overline{C}_{2}\right) \end{array} $$
(6)

The gradient of Li,i with respect to \(\hat {X_{i}}\) in the reconstruct layer is:

$$\begin{array}{@{}rcl@{}} \frac{\partial L_{i,i}}{\partial \hat{X_{i}}}&=&\frac{\partial \left\|X_{i}-\hat{X_{i}}\right\|_{F}^{2}}{\partial \hat{X_{i}}}\\ &=&\frac {\partial Tr\left.\left( X_{i}-\hat{X_{i}}\right)\left( X_{i}-\hat{X_{i}}\right)^{\prime}\right)}{\partial \hat{X_{i}}}\\ &=&2\left( \hat {X_{i}}-X_{i}\right) \end{array} $$
(7)

The loss function is minimized by using L-BFGS method, which is particularly suitable for the optimization of a large number of variables. Any mistake in the computation of gradient or loss will lead to the failure of line search in L-BFGS. This is very helpful for developers.

2.2 C2Net

To further improve the cross-modal retrieval performance, we build a fully-convolutional network to learn better representations. In this section, we describe details of our C2Net.

2.2.1 Model architecture

An illustration of the C2Net structure is shown in Fig. 4. Our network includes 3 fully-convolutional layers followed by batch normalization layers, taking the representations obtained from 3-view DCCA as the input. With the help of batch normalization layers, our model is able to speed up the training from a higher learning rate, without ill side effects. Similar pairs and dissimilar pairs are obtained in the pair-merging layer. Features are L2 −normalized before calculating the contrastive loss.

Fig. 4
figure 4

The architecture of C2Net

Full size image

2.2.2 Formulation

Contrastive loss [13] is proposed to learn an invariant mapping for dimension reduction. Similar points are mapped to nearby points on the output manifold and dissimilar points to distant points. In cross-modal retrieval, paired text and image are similar if they are from the same semantic class. The contrastive loss [13] is defined to penalize large distance between intra-class pairs and distance smaller than a margin between inter-class pairs:

$$ L_{con}=\frac{1}{2}\sum\limits_{i = 1}^{m/2}s_{i} {d_{i}^{2}}+(1-s_{i}) max(\gamma-d_{i},0)^{2} $$
(8)

where di = ||x2i− 1 − x2i||2, γ is the margin, m is the batch size, and binary si specifies whether x2i− 1 and x2i belong to the same class. The gradient of Lcon with respect to x2i− 1 is computed as:

$$ \frac{\partial L_{con}}{\partial x_{2i-1}}=\left\{\begin{array}{lr} x_{2i-1}-x_{2i} \qquad &s_{i}= 1 \\ -\frac{\gamma-d_{i}}{d_{i}}(x_{2i-1}-x_{2i})\qquad \qquad &s_{i}= 0,\gamma-d_{i}\textgreater 0 \\ 0 \qquad &s_{i}= 0,\gamma-d_{i}\leq 0 \end{array} \right. $$
(9)

Center loss is proposed to minimize the intra-class distances of deep features and has been verified to be able to effectively enhance the discriminative power of features. Namely, the learned features are not only separable but also discriminative in terms of the compactness of intra-class features. The center loss function is defined as:

$$ L_{cen}=\frac{1}{2}\sum\limits_{i = 1}^{m}||x_{i}-c_{y_{i}}||_{2}^{2} $$
(10)

where \(c_{y_{i}}\) denotes the center of the yi −th class, and m is the batch size. The gradient of Lcen with respect to xi is:

$$ \frac{\partial L_{cen}}{\partial x_{i}}= x_{i}-c_{y_{i}} $$
(11)

\(c_{y_{i}}\) changes along with the changes of deep features. The error term of cj is defined as:

$$ {\Delta} c_{j}=\frac{{\sum}_{i = 1}^{m}\delta\{y_{i}=j\}(c_{j}-x_{i})}{\epsilon+{\sum}_{i = 1}^{m}\delta\{y_{i}=j\}} $$
(12)

where 𝜖 is added to cope with the absence of samples belonging to the j −th class, and δ{condition} is defined as:

$$ \delta\{condition\}= \left\{\begin{array}{ll} 1\qquad \qquad &if \ \ condition \\ 0 \qquad &otherwise \end{array} \right. $$
(13)

3 Experiments and results

3.1 Data sets

Wikipedia contains 2866 text-image pairs belonging to 10 semantic categories. We randomly split the data set into three subsets: 2,173 pairs for training, 231 pairs for validation and the last 462 pairs for testing. Each image is represented by three descriptors, including 1000-D pyramid histogram of dense SIFT, 512-D Gist, and 784-D MPEG-7. And each text is represented by 3000-D bag of high-frequency words.

NUS-WIDE-10k. This data set is a subset of NUS-WIDE [3], which contains about 269,648 images with tag annotations from 81 categories. Because some categories are scare, we only choose 10 most common categories. We have 8000 image-text pairs for training, 1000 for parameter validation, and 1000 for test. For image representation, six types of low-level features are extracted from these images, including 64-D color histogram, 144-D color correlogram, 73-D edge direction histogram, 128-D wavelet texture, 225-D block-wise color moments and 500-D bag of words based on SIFT descriptors. For text representation, we use 1000-D bag of words.

3.2 Evaluation metric

We use mean average precision (mAP) as the evaluation criterion. Given one query and top-R retrieved data, the average precision is defined as:

$$ \frac{1}{N}\sum\limits_{i = 1}^{R}prec(i)*rel(i) $$
(14)

where, N is the number of the relevant documents in the retrieved set, prec(i) is the percentage of the relevant text documents (images) in the top-i text documents (images). rel(i) is an indicator function, when the i-th result is relevant to the query it equals 1, otherwise 0.

3.3 Implement details

For 3-view DCCA, feature whitening and dimension reduction are applied. For Wikipedia(NUS-WIDE-10k), image features and text features are reduced to 128-D(256-D). We perform grid search for the number of units in the hidden layers with the setting 128, 256, 384, 512, 768, 1024. In the output layer, the number of units is set to 9 for both data sets. The reconstruct layer has the same number of units as the input layer. After setting up the network architecture, the parameters of each layer are pretrained with a Denoising AutoEncoder(DAE), which reconstructs the original input from a distorted status, to ensure the representations of the input are robust to small irrelevant changes. For the tolerance of L-BFGS algorithm, we use the default setting of DCCA. The implementation of 3-view DCCA is built on Galen Andrew’s DCCA library [1].

As for C2Net, the implementation is built based on Caffe [17]. Features obtained from 3-view DCCA are stored in hdf5 format. C2Net consists of 3 fully-convolutional layers with 256 filters of size 1 × 1, followed by batch normalization layers. The network is trained with a base learning rate 0.1 by stochastic gradient descent with 0.9 momentum, and the weight decay parameter is 0.005. C2Net is trained with batch size of 1024.

Experiments for our method are conducted on a computer which has Intel i7 4.0 GHZ 8 processors, 8 GB RAM,6GB Nvidia Gefore GTX TITAN GPU, Intel MKL 11.2 and Windows7. The training time increases with the number of training instances increasing. The training time of our method on NUS-WIDE-10k is less than 20 minutes.

3.4 Performance comparison

In this subsection, we compare our approach with several typical cross-modal retrieval methods, including both supervised and unsupervised algorithms. For all baselines listed below, two cross-modal tasks are investigated: text-to-image retrieval and image-to-text retrieval.

CCA :

[14]: CCA learns a common latent space by maximizing the correlation between original features of image view and text view.

PCA-CCA :

: CCA is performed on features reduced by PCA.

KCCA :

[4]: KCCA is a nonlinear extension of CCA. And we choose Gaussian kernel RBF as the kernel function of KCCA.

DCCA :

[1]: DCCA builds two separate deep networks to learn nonlinear correlation between text and image. Input features of text view and image view are whitened and reduced. This preprocessing is the same for all DCCA-based methods.

CFA :

[19]: CFA proposes to learn orthogonal transformation for each modal by minimizing the pair-wise distances in the latent space.

Multimodal DBN :

[34]: Multimodal DBN learns a shared representation for image view and text view with a separate two-layer DBN.

Bimodal AE :

[20]: Bimodal AE connects image features and text features with a joint layer followed by two intra-modal autoencoders.

Corr-AE :

[6]: Corr-AE builds two separate uni-modal autoencoders and try to learn the correlation between two modalities by correlating hidden representations of two autoencoders. Corr-AE performs grid search for the number of hidden units of each layer with the setting 32, 64, 128, 256, 512, 1024. The number of units for all hidden layers in Corr-AE is restricted to the same. The code with parameter specifications of Multimodal DBN, Bimodal AE and Corr-AE is available online.Footnote 1

DCCA-PHS :

[26]: DCCA-PHS, our previous method, proposes to extract semantic features from text features with hypergraph learning. We embed image view, text view and unsupervised semantic into a progressive framework to learn complex nonlinear correlation. The numbers of layers and numbers of nodes per layer in DCCA-PHS are described in the experiment section of [26].

3-view CCA :

: For 3-view CCA, an Eigenvector implementation is available at http://www.unc.edu/~yunchao/crossmodal.htm. We implement 3-view CCA based on DCCA. It’s clear that 3-view CCA can be viewed as a single layer DCCA with linear activation function.

SCM :

[23]: SCM combines correlation matching(CM), an unsupervised method which models cross-modal correlation, and semantic matching(SM), an supervised technique that relies on semantic representation. Our method is inspired by SCM. The source code of CCA and SCM is available at http://www.svcl.ucsd.edu/projects/crossmodal/.

JRL :

[46]: JRL simultaneously learns linear projections for different modalities with semi-supervised regularization and sparse regularization.

CMDN :

[22]: CMDN proposes to learn intra-modal representations and inter-modal representations with Stacked AutoEncoders(SAE) and Multimodal DBN [34] respectively. Both representations are combined to generate the shared representations with Bimodal Autoencoder [20] in a stacked style. The implementation of CMDN is based on deepnet.Footnote 2

CDMLMR :

[16]: CDMLMR integrates quadruplet ranking loss and semi-supervised contrastive loss for modeling cross-modal semantic similarity in a unified multi-task learning architecture.

Table 1 summarizes the mAP@50 scores of different methods for cross-modal retrieval. PCA-CCA outperforms CCA on both data sets. Dimension reduction is an effective way to boost the performance of CCA. So we apply dimension reduction on DCCA and 3-view DCCA. The performance of DCCA is worse than PCA-CCA due to over-fitting. 3-view CCA outperforms CCA, which indicates that the semantic information is very useful for improving the semantic consistency in the latent space. Overall, supervised methods achieve better performance than unsupervised ones. Our method can be viewed as a nonlinear extension of SCM. It benefits from the better representation power of deep models. CDMLMR achieves state-of-the-art performance in terms of mAP@50. However, CDMLMR is trained with the help of data from the test set in the semi-supervised learning stage as mentioned in [16], which is not fair for other methods. So in the following section, we mainly compare our method with another state-of-the-art method, CMDN. Although our algorithm and CMDN build two cascaded networks, the network structures are quite different: CMDN is optimized with multiple reconstruct loss, L2 loss and softmax loss, while our method is trained by multiple reconstruct loss, CCA loss, contrastive loss and center loss. The reconstruct loss in representation learning is very beneficial for correlation learning in cross-modal retrieval. CCA loss and L2 loss are common losses for building correlation. In our experiments, CCA loss could achieve the same performance as L2 loss with much fewer dimensions. We have also tested single softmax loss, but did not achieve satisfactory results. The advantages of CMDN are the combination of two representations in the first stage and the stacked architecture in the second stage.

Table 1 mAP@50 results of different methods for image-to-text(Img2Txt) and text-to-image(Txt2Img) retrieval
Full size table

Figure 5 presents a more detailed analysis of the retrieval performance, in the form of 11-point interpolated precision-recall(PR) curves. These PR curves show that our method attains higher precision at lower levels of recall, which indicate that the top retrieved results of our method are more relevant to the queries. However, our method fails to obtain higher precision at all levels of recall. Therefore, better networks would be explored in the next work.

Fig. 5
figure 5

The PR curves on Wikipedia and NUS-WIDE-10k

Full size image

3.5 Effectiveness of reconstruct loss in the first stage

We use t-SNE to visualize the image and text representations, which are learned by 3-view DCCA when α= 0 and α= 1 in Fig. 6. When α is set to 0, the effect of the reconstruct loss is ignored. As discussed above, the similarity metric we use for retrieval is normalized correlation(NC) [23] which is not suitable for t-SNE. So we apply L2 normalization on the features in the latent space. In the normalized space, NC distance is equal to Euclidean distance. As shown in Fig. 6a and (b), 3-view DCCA gets a good latent space when α is set to 0, since a lot of relevant pairs are clustered. However, the results on the test set (c) and (d) are opposite. The result on NUS-WIDE-10k is the same as that on Wikipedia. Like most deep learning methods, 3-view DCCA easily gets into over-fitting without the reconstruct loss. The reconstruct loss back-propagated from the last layer can be viewed as a regularization term for the correlation loss in the output layer and prevents overfitting to some extent.

Fig. 6
figure 6

Visualizations of the representations in the output layer learned by 3-view DCCA when α= 0 and α= 1. The points in the same color belong to the same category

Full size image

3.6 Effectiveness of joint loss in the second stage

In this subsection we demonstrate the effectiveness of the combination of contrastive loss and center loss. We check the performance of our method in 3 cases: (1) C1Net-con, the center loss layer is removed. (2) C1Net-cen, the contrastive loss layer and relevant normalization layers are removed. (3) C2Net, i.e., the proposed method. Figure 7 shows the mAP@50 scores achieved per category by above cases. C2Net is better than C1Net-con and C1Net-cen for most categories on both data sets. The contributions of contrastive loss and center loss are complementary and the best performance is achieved when the two are combined.

Fig. 7
figure 7

Average per-class mAP@50 of image-to-text retrieval and text-to-image retrieval for each category

Full size image

4 Conclusions and future work

In this paper, we propose a two-stage deep learning method for supervised cross-modal retrieval. We expand CCA from 2-view to 3-view to conduct supervised learning in both stages, which could maximize the value of semantics. Multiple supervisory signals are successfully combined in both stages to overcome over-fitting and learn better feature representations as well. Experiments on two public data sets show that our method is competitive with state-of-the-art performance. In the future, we will build an end-to-end network instead of cascaded two-stage network so as to easily adjust network structures and finetune parameters. Meanwhile, better network architecture will be explored to improve cross-modal retrieval performance.