# UIGR: Unified Interactive Garment Retrieval

Xiao Han1,2 Sen He1,2 Li Zhang3 Yi-Zhe Song1,2 Tao Xiang1,2
1CVSSP  University of Surrey
2iFlyTek-Surrey Joint Research Centre on Artificial Intelligence
3School of Data Science  Fudan University
{xiao.han, sen.he, y.song<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Interactive garment retrieval (IGR) aims to retrieve a target garment image
based on a reference garment image along with user feedback on what to change
on the reference garment. Two IGR tasks have been studied extensively: text-
guided garment retrieval (TGR) and visually compatible garment retrieval
(VCR). The user feedback for the former indicates what semantic attributes to
change with the garment category preserved, while the category is the only
thing to be changed explicitly for the latter, with an implicit requirement on
style preservation. Despite the similarity between these two tasks and the
practical need for an efficient system tackling both, they have never been
unified and modeled jointly. In this paper, we propose a Unified Interactive
Garment Retrieval (UIGR) framework to unify TGR and VCR. To this end, we first
contribute a large-scale benchmark suited for both problems. We further
propose a strong baseline architecture to integrate TGR and VCR in one model.
Extensive experiments suggest that unifying two tasks in one framework is not
only more efficient by requiring a single model only, it also leads to better
performance. Code and datasets are available at GitHub.

##  1 Introduction

In computer vision, there is a long line of research on understanding garment
image content [31, 51, 47, 18, 2, 8]. Among them, Interactive Garment
Retrieval (IGR) [56, 49, 51] is most relevant to the garment search problem.
IGR aims to retrieve a target garment image based on a reference garment image
along with user feedback on what to change on the reference garment. It
enables a shopper to find exactly what she/he wants because it allows for the
fine-tuning of search results through user feedback.

Figure 1: An example IGR scenario where both TGR and VCR can take place. Given
a reference garment, users may search for a garment of the same category with
some attribute changes (TGR), or visually compatible garments of different
categories (VCR).

Two IGR tasks, namely Text-Guided garment Retrieval (TGR) [49, 32] and
Visually Compatible garment Retrieval (VCR) [30, 21] have been studied so far
(see Figure 1). TGR (dialog 1 & 3 of Figure 1) retrieves garments in the same
category as the reference garment. The feedback is in the form of either
synthetic sentence [14, 49] or natural language [51], indicating the intended
attribute changes from the reference to the target garment. In contrast, the
feedback for VCR (dialog 2 of Figure 1) typically only indicates category
change explicitly, in the form of an indicator rather than text [30, 21].
Nevertheless, as the retrieval is constrained to only visually compatible
items, implicit feedback is to preserve the style so that the reference and
target look lovely when worn together.

Despite being two instantiations of IGR, TGR and VCR have never been studied
together in a unified framework. Indeed, they are evaluated on completely
different sets of benchmarks. The developed methods also seem pretty
different. TGR is usually done by first compositing the reference garment with
the interaction signal together and then retrieving the garments similar to
the composited query [49, 7, 29, 44]. Since different garments are compatible
along multiple dimensions, such as color, pattern, and material, previous
works in VCR typically learn subspace embeddings to capture different notions
of similarity and aim to learn a joint embedding space where compatible
garments of different categories are close [30, 21].

In this paper, for the first time, we propose Unified Interactive Garment
Retrieval (UIGR) to unify the two tasks in a single framework. We argue that
there are two benefits for doing so: (1) As shown in Figure 1, it is common to
have both tasks incurred in the same shopping session. It is thus more
efficient to build one rather than two separate models to tackle both tasks.
(2) Due to the similarity in format (_i.e._ , both are IGR tasks), having a
single multi-task framework makes it possible for both tasks to benefit from
each other when trained jointly end-to-end. However, unifying the two tasks is
challenging, with two main obstacles to overcome: the lack of benchmarks and
the discrepancy in the two types of user feedback.

To this end, we try to solve these problems with two main contributions: (1)
We establish a novel benchmark for the study of this unified problem by re-
purposing Fashionpedia [24], where prompt engineering is adopted to generate
user feedback from fine-grained attributes. (2) We introduce a multi-task
model jointly learning two tasks, which unifies TGR and VCR in a single
framework and serves as a strong baseline for UIGR. Experiments demonstrate
that unifying the two tasks in a single model is not only possible but also
yields better overall performance, compared with modeling them separately
using two models.

##  2 Related work

Text-guided garment retrieval. TGR is a special type of image retrieval
problem with multimodal compositional queries [49, 32, 4]. In general, the
user feedback used to guide the searching process can be attributes [56, 16,
1], synthetic sentences [14, 49], and natural language (free text) [51, 53].
Different TGR models proposed so far differ primarily in the design of their
compositors. A compositor plays a fundamental role to integrate the textual
information with the imagery modality. TGR compositors have been proposed
based on various techniques, such as gating mechanism [49], hierarchical
attention [7, 23, 12, 20], graph neural network [54, 44], joint learning [6,
27, 44, 52, 55], ensemble learning [50], style-content modification [29, 5]
and vision & language pre-training [32].

Figure 2: Overview of the dataset collection process. The whole pipeline is
based on the image and corresponding high-quality annotations from
Fashionpedia [24]. (1) We firstly construct an image pool by cropping each
garment using its ground truth mask. (2) To construct TGR triplets, we select
a pair of images with the same category and high similarity. Then the user
feedback is generated by filling relative attributes in the blank of prompt
templates. (3) For VCR triplets construction, the image pair is selected
according to whether both images are from the same outfit. We generate this
kind of user feedback by mentioning the categories of both reference and
target images.

Visually compatible garment retrieval. Predicting fashion compatibility is to
determine whether two garments of different categories match well
aesthetically. On this basis, the recommendation can be done either as fill-
in-the-blank [17] at item level or as personalized outfit recommendation [36,
35] at outfit level. In addition to being a set, an outfit can also be
represented as a sequence [17], or a graph [10].

Instead of computing the compatibility in a single space, most approaches [48,
47, 45, 30, 21, 26] explore learning subspace embeddings to capture different
notions of compatibility. [48, 47] learn many conditional subspaces, each for
a pair of categories. [45] learns several subspaces conditioned on the
features from both the reference garment and the target garment. However, this
kind of method is not suitable for large-scale retrieval where exhaustive
comparison is prohibitive. [30, 21] concatenate one-hot labels of the
reference and target category to represent the interaction signal to meet the
setting of large-scale retrieval.

Fashion datasets. Over the past few years, many fashion datasets have been
proposed for multiple applications [8], such as detection [31, 33, 13],
retrieval [31, 13, 38], attribute recognition [31, 16], popularity learning
[34, 37, 2] and synthesis [18, 25]. The most related datasets to our work are
[14, 16, 51] for TGR and [47, 30, 43] for VCR. Besides not being suitable for
the unified setting, previous TGR and VCR benchmarks have some other problems,
which will be explained in next Section.

##  3 New benchmark for IGR

Next, we describe the data collection process and provide an in-depth analysis
of UIGR. The overall data collection procedure is illustrated in Figure 2. The
basic statistics is summarized in Table 1 and 2 111“Triplet” in this article
refers to one piece of data, _i.e._ , two images and one sentence, rather than
anchor, positive and negative sample pair..

###  3.1 Image and attribute collection

We collect UIGR garment images based on the original images, garment bounding
boxes, garment segmentation masks, and fine-grained attributes from
Fashionpedia [24] with a series of pre-processing 222More details about pre-
processing steps are listed in Supp. Mat..

Figure 3: Typical bad triplets in FashionIQ [51].

Figure 4: Triplet examples in UIGR TGR subset.

Figure 5: Triplet examples in UIGR VCR subset.

###  3.2 Image pair selection

TGR subset. Previous benchmarks [16, 51] select image pairs by comparing the
similarity of text information, _e.g._ , image titles or attribute labels. As
shown in Figure 3, this selection strategy often leads to weakly related image
pairs with drastically different visual appearances. The user feedback thus
cannot accurately describe all the changes necessary to align the image pairs
because there are too many changes needed. We thus take a different strategy:
using image similarity instead of text similarity for pair selection.
Specifically, we use a DenseNet [22, 27] pre-trained on DeepFashion [31] to
get image feature vectors. Next, for each image, we calculate the cosine
similarity between it and all images of the same category in the image pool
and only consider the top three most similar matches.

VCR subset. We select all garments coming from the same outfit in a
bidirectional way to construct image pairs, which is a standard procedure
adopted in previous VCR benchmarks [47, 30].

Split | # Images | # Outfits | # Triplets
---|---|---|---
TGR | VCR
Train | 76,685 | 29,321 | 210,189 | 190,150
Validation | 25,181 | 9,688 | 68,847 | 61,776
Test | 25,434 | 9,814 | 69,639 | 62,418

Table 1: Dataset statistics of UIGR.

Dataset | # Triplets | # Categories | Caption length
---|---|---|---
Shoes [3, 14] | 10k | 1 | 5.22 words
Fashion200K [16, 49] | 172k | 5 | 4.00 words
FashionIQ [51] | 18k | 3 | 5.36 words
Our TGR | 381k | 27 | 6.33 words
Dataset | # Outfits | # Categories | Interaction signal
Polyvore retrieval [30] | 17k | 16 | One-hot labels
Our VCR | 49k | 27 | Text

Table 2: Comparisons with other related datasets.

###  3.3 User feedback generation

Because the scale of UIGR is more than twenty times that of FashionIQ,
manually annotating each image pair with fine-grained user feedback is
laborious and costly. To this end, we adopt prompt engineering to
automatically generate the user feedback based on the relative attributes
between two garments. Following the setting of FashionIQ, we generate two
sentences for each image pair.

TGR subset. We manually summarize tens of cloze prompt templates from
FashionIQ captions. These templates include several single phrases, such as
“has {V} {A}” and “change {A} to {V}”, where {V} and {A} hold the blank for
one attribute name and its value. The templates of multiple phrases are based
on the combination of single phrases. Finally, the relative attributes between
two images are filled in the blanks of the randomly selected prompt template.

VCR subset. To unify the VCR task with TGR, they need to have the same user
feedback format, _i.e._ , sentences describing the intended changes to the
reference garment. One obvious choice is to use the prompt engineering
technique to generate sentences describing only the category changes for VCR.
However, this fails to capture the implicit user feedback when it comes to
VCR. That is, the style of the target garment needs to be consistent with that
of the reference.

To this end, we first calculate the correlation matrix of all attributes
between any two kinds of garments. When constructing VCR triplets, we will
predict the most likely target attributes based on the existing attributes of
the reference image. Next, we will randomly mention one attribute in the
predicted attributes using the attribute correlation matrix when generating
user feedback.

Different from the TGR subset, we manually design several prompt templates for
VCR, such as “search a {TV} {TC} that matches this {RC} best” and “for this
{RC}, find a visually compatible {TV} {TC}”, where {TV}, {TC} and {RC} stand
for the target attribute value, target category and reference category,
respectively.

###  3.4 Dataset analysis

The examples of our collected TGR triplets are depicted in Figure 4. Compared
with those from FashionIQ in Figure 3, our triplets seems more reasonable. In
particular, although all relative captions in FashionIQ are annotated via a
crowdsourcing platform, many captions are too ambiguous to describe the exact
search direction. Since we select image pairs based on the image similarity to
avoid significant visual changes, the subsequently generated user feedback is
more accurate and fine-grained.

Figure 5 shows the examples of VCR subset Compared with one-hot labels for
user feedback, sentences are more flexible and scalable to integrate more
fine-grained information from users. Further, the VCR task now has the same
setting as the TGR, making unification possible.

##  4 Experiments

Figure 6: Proposed multi-task architecture for UIGR.

Although there are different implementations for the compositors of VCR and
TGR, they share the same goal: preserving unmentioned visual appearance
aspects of the reference and changing only those mentioned in the interaction
signal/feedback. Our multi-task model unifies the two tasks based on the same
goal. However, to accommodate the major difference in the change directions of
the two tasks, namely whether the category is preserved or changed, we use
different compositors. As shown in Figure 10, two branches are used for
separately learning two composition processes with shared image and signal
encoders. Considering that the features needed to be modified for the two
branches are not the same, we use two projection modules to project image
features to two latent spaces ahead of the composition process. We also
jointly learn a classifier to distinguish different user feedback. With it,
our model can automatically determine which branch should be selected to do
composition during inference, thus allowing the real-world application
scenario depicted in Figure 1 to be supported by one model.

| TGR Results | VCR Results | Mean
---|---|---|---
Comp. Metrics | R@10 | R@50 | mAP | R@10 | R@50 | mAP | R@K | mAP
| I | 38.98 | 72.08 | 14.29 | 71.03 | 86.83 | 46.82 | 67.23 | 30.56
CSA [30] | U | 36.90 | 70.57 | 13.37 | 70.46 | 86.88 | 46.47 | 66.20 | 29.92
| I | 46.27 | 77.57 | 19.78 | 69.30 | 85.88 | 46.15 | 69.76 | 32.97
TIRG [49] | U | 45.06 | 76.75 | 18.91 | 72.11 | 88.42 | 48.54 | 70.59 | 33.73
| I | 43.27 | 75.10 | 18.06 | 62.99 | 81.97 | 40.40 | 65.83 | 29.23
VAL [7] | U | 40.19 | 71.78 | 17.28 | 67.95 | 86.24 | 44.72 | 66.54 | 31.00
| I | 40.24 | 72.15 | 17.31 | 64.10 | 83.18 | 41.36 | 64.92 | 29.34
CoSMo [29] | U | 40.96 | 72.40 | 17.62 | 68.64 | 86.41 | 45.05 | 67.10 | 31.36
| I | 48.23 | 78.79 | 19.98 | 69.28 | 86.26 | 46.10 | 70.64 | 33.04
RTIC [44] | U | 46.75 | 77.80 | 19.26 | 74.18 | 89.56 | 50.78 | 72.07 | 35.02

Table 3: The evaluation results for the proposed unified (U) model with five
different compositors on UIGR test split. For each compositor, the compared
model (I) is the combination of two models independently trained on TGR and
VCR.

We compare our multi-task model with previous methods where TGR and VCR are
studied independently 333We put implementation details, hyperparameter
settings, evaluation protocols, ablation study and qualitative results in
Supp. Mat.. The main experiment results are reported in Table 3. We can draw
the following conclusions from the results: (1) Overall, our proposed multi-
task model achieves comparable and even better performance (1.18 mAP increase
on average) compared with the combination of two separately trained models.
The best result (the last row) is achieved by our multi-task model with RTIC
[44] as the compositor. (2) In most cases (4 out of 5), our model achieves
significantly better performance than an independently trained model on the
VCR task. It suggests that text is more suitable than one-hot labels as the
user feedback for VCR. With the user feedback in the same modality of TGR, VCR
can learn useful information from TGR in our unified model. (3) Although our
model has a slight performance drop on the TGR task, its performance is still
competitive against an independently trained model on the TGR subset (_e.g._ ,
only 0.58 mAP drop for TGR but 2.95 mAP gain for VCR on average).

In summary, the experiment results demonstrate that VCR and TGR can be unified
and implemented in a single model through our proposed framework. It is more
efficient by having one model only and more effective with improved overall
performance over the two tasks.

##  5 Conclusion

We have proposed a unified setting for TGR and VCR with a new large-scale
benchmark and a baseline multi-task architecture, in which we use text as the
unified user feedback format for both TGR and VCR. We conducted experiments to
show that the proposed baseline model has competitive or even better
performance than previous methods, and it is also more efficient to use one
model instead of two.

##
UIGR: Unified Interactive Garment Retrieval
–
Supplementary Material

##  Appendix A Additional information on UIGR dataset

Our dataset is built upon Fashionpedia [24], which is a large-scale dataset
for garment segmentation and fine-grained attribute localization. Fashionpedia
provides an ontology built by fashion experts containing 27 garment categories
and 19 garment parts. It provides not only fine-grained attributes but also
implicit visual compatibility relationships for all garments in an outfit. All
alternatives [16, 30] cannot meet all these conditions at the same time.

Image pre-processing. We want an IGR model to focus on the garment to be
refined by the user feedback. The background and other garment items in a
given image are thus distractions and should be removed. To this end, a series
of pre-processing steps are introduced: (1) We use a salient object detection
model [41, 57] to remove the background, which is an easy task given the
typical clean background in fashion catalog images. (2) When there are
multiple garments with the same category in one image (_e.g._ , shoes and
gloves), if they do not overlap, we only keep the one with the largest pixel
area; (3) We delete the masks of garment parts (_e.g._ , sleeves and pockets)
but merge their attributes into the garments they belong to; (4) We delete the
garments that have low-resolution or extreme aspect ratio; (5) If there are
pixels of other garments in the bounding box, we mask these excess pixels with
gray color. Finally, we cropped each garment with its attributes from the
original image to construct a substantial image pool.

TGR
---
search another item with a similar style
there are no changes between two images
change {$A$} to {$V$}
has {$V$} {$A$}
is {$V$}
change {$A$} to {$V_{1}$} and {$V_{2}$}
change {$A_{1}$} to {$V_{1}$} and change {$A_{2}$} to {$V_{2}$}
has {$V_{1}$} and {$V_{2}$} {$A_{2}$}
has {$V_{1}$} {$A_{1}$} and {$V_{2}$} {$A_{2}$}
is {$V_{1}$} and with {$V_{2}$} {$A_{2}$}
is {$V_{1}$} and {$V_{2}$}
VCR
search a {$TC$} that matches this {$RC$} best
retrieve a {$TC$} having a similar style with current {$RC$}
for this {$RC$}, find a visually compatible {$TC$}
replace this {$RC$} with a {$TC$} that has a consistent style
search a {$TV$} {$TC$} that matches this {$RC$} best
retrieve a {$TV$} {$TC$} having a similar style with current {$RC$}
for this {$RC$}, find a visually compatible {$TC$} with {$TV$} {$TA$}
replace this {$RC$} with a {$TC$} that has {$TV$} {$TA$}

Table 4:  All prompts for user feedback generation of UIGR. {$V$} and {$A$}
hold the blank for one attribute name and its value. {$TV$}, {$TC$} and {$RC$}
stand for the target attribute value, target category and reference category,
respectively. Which kind of TGR prompt to choose depends on how many related
attributes (0, 1 or 2) need to be mentioned. Which kind of VCR prompt to
choose depends on whether the target attributes need to be mentioned.

Prompt engineering. We list all used prompts for user feedback generation in
Table 4. Our prompts simulate a variety of syntax structures: single phrases,
compositional phrases, and propositional phrases.

Figure 7: More triplet examples in UIGR TGR subset.

Figure 8: More triplet examples in UIGR VCR subset.

More triplet examples. We present more triplet examples of UIGR in Figure 7
and Figure 8. As discussed in the main paper, the TGR triplets we collected
successfully follow the assumption that there could not be too many visual
changes between the reference garment and the target garment. Our TGR triplets
thus are much higher quality than those in FashionIQ [51] with less ambiguity.
Besides, our TGR subset contains 27 different garments, far more than
FashionIQ, which only has three categories (top tee, shirt and dress).

Thanks to the flexibility of text, our VCR subset includes more meaningful
information compared to concatenated one-hot labels [30]. Now each user
feedback sentence states category changes and intended attributes based on the
statistics of attribute co-occurrence between compatible garment items, which
is more in line with reality. With the help of such kind of VCR triplets, the
potential user can specific the search direction through mentioning some
specific target attributes. Most importantly, now the VCR has the unified
setting with TGR.

##  Appendix B Additional information on the multi-task baseline model

Given a reference garment image $g^{r}$ and an interaction signal (user
feedback) $s$, the ultimate goal of interactive retrieval is to search the
gallery for another garment image $g^{t}$ that best matches the modification
mentioned in $s$. Regardless of whether the user wants to modify the
attributes or the category of the reference garment, the interaction signal is
in the same textual format. TGR and VCR can thus be modeled in the same
framework.

We will first briefly introduce how previous works study these two tasks
separately and then describe our unified solution based on multi-task
learning.

###  B.1 Preliminary method

In Figure 9, an existing pipeline for interactive retrieval typically consists
of three components: image encoder $\mathcal{E}^{I}$, interaction signal
encoder $\mathcal{E}^{S}$ and compositor $\mathcal{C}$.

Firstly, both reference image and target image are fed into the image encoder
to obtain representations in the feature space:
$\noindent\textbf{g}^{r}=\mathcal{E}^{I}\left(g^{r}\right),\noindent\textbf{g}^{t}=\mathcal{E}^{I}\left(g^{t}\right),$
where $\mathcal{E}^{I}$ is usually instantiated by a CNN pre-trained on
ImageNet [11] and a linear projection layer [49, 44].

In the meantime, the interaction signal is processed by the signal encoder to
get the signal representation:
$\noindent\textbf{s}=\mathcal{E}^{S}\left(s\right),$ where the interaction
signal is represented by the concatenation of reference category $c^{r}$ and
target category $c^{t}$ for VCR [30] or by user feedback $t$ for TGR [49].

Finally, the most important step is to incorporate the interaction signal’s
feature into reference image’s feature via a compositor:
$\noindent\textbf{x}=\mathcal{C}\left(\noindent\textbf{g}^{r},\noindent\textbf{s}\right).$
For VCR, this compositor is always instantiated by a conditional similarity
module [30, 21] to learn different sub-spaces with different notions. For TGR,
this compositor works globally [49, 44] or locally [7, 29] to modify the
feature map of reference image.

The goal of this pipeline is to make the composed query x as close as possible
to the target $\noindent\textbf{g}^{t}$ in a shared feature space. A widely
used objective function is the batch-based classification loss (BBC) [49],
which assumes the same form as the InfoNCE loss [39]:

$\mathcal{L}_{bbc}=\frac{1}{B}\sum_{i=1}^{B}-\log\frac{\exp\left[\kappa\left(\noindent\textbf{x}_{i},\noindent\textbf{g}^{t}_{i}\right)/\tau\right]}{\sum_{j=1}^{B}\exp\left[\kappa\left(\noindent\textbf{x}_{i},\noindent\textbf{g}^{t}_{j}\right)/\tau\right]},$
(1)

where $\kappa(\cdot,\cdot)$ and $\tau$ are cosine distance metric and tuneable
temperature, respectively. In this loss, each example is contrasted with a set
of other negatives. It thus achieves better discriminative learning and faster
convergence.

During inference, the features of all gallery images will be calculated in
advance by image encoder. For each composed query, its cosine similarity with
all gallery features will be obtained. Finally, an identity list is sorted
according to the cosine similarity as the retrieval result sequence.

Figure 9: Previous architecture for TGR/VCR. Figure 10: Proposed multi-task
architecture for UIGR.

###  B.2 Proposed multi-task framework

Although there are different implementations for the compositors of VCR and
TGR, they share the same goal: preserving unmentioned visual appearance
aspects of the reference and changing only those mentioned in the interaction
signal/feedback. Our multi-task model unifies the two tasks based on the same
goal. However, to accommodate the major difference in the change directions of
the two tasks, namely whether the category is preserved or changed, we use
different compositors. As shown in Figure 10, two branches are used for
separately learning two composition processes with shared image and signal
encoders.

More specifically, we use a quintuplet
$\left\\{g^{r},s_{v},s_{t},g^{t}_{v},g^{t}_{t}\right\\}$ containing reference
garment, VCR signal, TGR signal, VCR target garment, and TGR target garment as
the input for training. These three garment images will be fed into a shared
image encoder $\mathcal{E}^{I}$ to get respective features
$\noindent\textbf{g}^{r}$, $\noindent\textbf{g}^{t}_{v}$ and
$\noindent\textbf{g}^{t}_{t}$. Similarly, two signals will get their features
$\noindent\textbf{s}_{v}$ and $\noindent\textbf{s}_{t}$ via a shared signal
encoder $\mathcal{E}^{S}$.

Considering that the features needed to be modified for the two branches are
not the same, we use two projection modules $\mathcal{P}_{t}$ and
$\mathcal{P}_{v}$ to project image features to two latent spaces ahead of the
composition process. Exactly how the projection module is realized depends on
what compositor is employed here. Specifically, for the compositor who
directly modifies the feature map [7, 29], we implement the projection module
with a lightweight CNN; for the compositor working globally [49, 44], we use a
linear projection layer following the global average pooling instead.

After choosing a compositor architecture from a existing method (_e.g._ , [49,
44, 7, 29]), we need two compositors $\mathcal{C}_{t}$ and $\mathcal{C}_{v}$
of the same architecture but without shared weights, to separately learn two
composition processes for the two tasks. For each branch, the compositor
serves for incorporating signal feature into the projected image feature of
reference garment:

$\noindent\textbf{x}_{v}=\mathcal{C}_{v}\left(\mathcal{P}_{v}\left(\noindent\textbf{g}^{r}\right),\noindent\textbf{s}_{v}\right),\quad\noindent\textbf{x}_{t}=\mathcal{C}_{t}\left(\mathcal{P}_{t}\left(\noindent\textbf{g}^{r}\right),\noindent\textbf{s}_{t}\right).$
(2)

For both branches, two BBC losses $\mathcal{L}_{bbc}^{v}$ and
$\mathcal{L}_{bbc}^{t}$ will be calculated independently according to Equation
1.

We also jointly learn a classifier to distinguish different user feedback.
Specifically, we simply choose the branch with a higher score predicted by the
classifier, _i.e._ , hard selection, which is empirically found to be the most
effective design. We instantiate this branch classifier with an MLP
$\mathcal{M}$ and optimize it via cross-entropy loss (CE):

$\displaystyle\mathcal{L}_{ce}=\frac{1}{B}\sum_{i=1}^{B}-\log\frac{\exp\left[\mathcal{M}_{0}\left(\noindent\textbf{s}_{vi}\right)\right]}{\exp\left[\mathcal{M}_{0}\left(\noindent\textbf{s}_{vi}\right)\right]+\exp\left[\mathcal{M}_{1}\left(\noindent\textbf{s}_{vi}\right)\right]}$
(3)
$\displaystyle+\frac{1}{B}\sum_{i=1}^{B}-\log\frac{\exp\left[\mathcal{M}_{1}\left(\noindent\textbf{s}_{ti}\right)\right]}{\exp\left[\mathcal{M}_{0}\left(\noindent\textbf{s}_{ti}\right)\right]+\exp\left[\mathcal{M}_{1}\left(\noindent\textbf{s}_{ti}\right)\right]}.$

Our model is end-to-end optimized by the overall objective function, which is
the direct summation of two BBC losses and one CE loss:

$\mathcal{L}=\mathcal{L}_{bbc}^{v}+\mathcal{L}_{bbc}^{t}+\mathcal{L}_{ce}.$
(4)

Arch. | TGR Results | VCR Results | Mean
---|---|---|---
R@10 | R@50 | mAP | R@10 | R@50 | mAP | R@K | mAP
I | 46.27 | 77.57 | 19.78 | 69.30 | 85.88 | 46.15 | 69.76 | 32.97
U+SC | 43.97 | 76.22 | 17.67 | 71.18 | 87.89 | 46.89 | 69.82 | 32.28
U+SP | 42.74 | 75.04 | 17.84 | 69.67 | 87.51 | 45.60 | 68.74 | 31.72
U+SC+SP | 43.94 | 75.76 | 18.30 | 68.83 | 87.26 | 44.42 | 68.95 | 31.36
U | 45.10 | 76.84 | 18.94 | 72.15 | 88.61 | 48.49 | 70.68 | 33.72

Table 5: Ablation study on the proposed multi-task model. SC: sharing
compositor across two branches; SP: sharing projection module across two
branches.

User Feedback | Attribute Augmented | R@10 | R@50 | mAP
---|---|---|---|---
One-hot |  | 69.30 | 85.88 | 46.15
One-hot | ✓ | 70.98 | 87.16 | 47.80
Text |  | 70.77 | 86.88 | 47.51
Text | ✓ | 72.65 | 88.64 | 49.06

Table 6: Experiment results of attribute argumented models (with one-hot
labels or text as the user feedback) on VCR subset.

##  Appendix C Additional information on experiments

Implementation details. We realize the image encoder and signal encoder by
utilizing ResNet50 [19] and Bi-GRU [9]. The ResNet50 is pre-trained on
ImageNet [11] and the word embeddings of Bi-GRU are initialized by CLIP text
encoder [42, 15]. To demonstrate the universality of our multi-task
architecture, we instantiate the compositor with recent representative methods
[30, 49, 7, 29, 44]. For the projection module, we adopt two different
architectures (convolution layer with 512 output channels or linear layer with
512 output dimensions) according to whether the compositor is used to modify
the feature map or the global feature.

Hyper-parameters setting. We use random horizontally flip and random crop as
image data augmentation methods. All images are resized to $224\times 224$.
The batch size and temperature in the $\mathcal{L}_{bbc}$ are 64 and 0.0625,
respectively. Our model is trained with Adam optimizer [28] for 40 epochs with
an initial learning rate $2\times 10^{-4}$, which is decayed by a factor 0.1
at the $15^{th}$ and $25^{th}$ epoch, respectively. We also linearly increase
the learning rate from $2\times 10^{-5}$ to $2\times 10^{-4}$ at the first 5
epochs. All experiments are conducted on one Tesla V100 GPU (32GB memory) with
Pytorch [40].

Evaluation metrics. We adopt the standard evaluation metric for retrieval,
_i.e._ , Recall@K, denoted as R@K for short. To circumvent the problem of
false negatives [32], we follow FashionIQ [51] to set K as larger values (10
and 50). In addition, we also report the mean Average Precision (mAP) 444For
each query, mAP is calculated with top 50 results. for a comprehensive
evaluation.

Evaluation protocols. Since we are integrating VCR into TGR, we want the model
has the ability to distinguish different categories. Consequently, we lead a
more difficult evaluation protocol than FashionIQ. Unlike FashionIQ, which
evaluates three categories separately, category labels are not available for
our evaluation protocol. That is, all images in the gallery will calculate a
similarity with the composed query.

##  Appendix D More quantitative results

###  D.1 Ablation study

We examine the design of each component in our proposed model. The critical
problem we are going to explore is whether compositor and projection module
can be shared between TGR and VCR. In all experiments, we remove the branch
classifier and use TIRG [49] as the compositor.

As shown in Table 5, sharing both projection module and compositor leads to a
performance drop. In addition, a shared projection module alone leads to a
more considerable performance drop than a shared compositor. This result
demonstrates that projecting the features of reference garments into different
latent spaces is vital for this multi-task framework. To unify VCR and TGR in
a single model, the projection module and compositor thus cannot be shared
because different tasks need different embedding features.

###  D.2 Attribute augmented VCR model

In addition to helping to unify VCR and TGR, we believe that mentioning target
attributes is a more general way for VCR, even for models that use one-hot
labels as user feedback. To demonstrate that, we conduct a small experiment by
concatenating the one-hot label of the target attribute behind that of the
reference category and target category.

Comp. | Training Dataset | Dress | Shirt | Top Tee | Mean
---|---|---|---|---|---
R@10 | R@50 | mAP | R@10 | R@50 | mAP | R@10 | R@50 | mAP | R@K | mAP
TIRG[49] | UIGR | 7.59 | 19.98 | 3.25 | 7.90 | 18.99 | 3.25 | 8.77 | 23.56 | 3.91 | 14.47 | 3.47
FashionIQ | 23.65 | 49.93 | 11.89 | 21.98 | 46.61 | 9.31 | 27.84 | 55.07 | 12.53 | 37.51 | 11.24
UIGR + FashionIQ | 26.97 | 53.64 | 12.65 | 22.87 | 46.07 | 10.29 | 29.58 | 57.73 | 13.80 | 39.48 | 12.25
VAL[7] | UIGR | 6.05 | 18.20 | 2.78 | 7.31 | 17.76 | 2.85 | 7.50 | 20.04 | 3.05 | 12.81 | 2.89
FashionIQ | 19.09 | 44.57 | 9.02 | 16.68 | 37.93 | 7.21 | 20.45 | 46.76 | 8.88 | 30.91 | 8.37
UIGR + FashionIQ | 26.43 | 52.66 | 13.02 | 20.36 | 43.52 | 9.54 | 25.85 | 53.14 | 12.21 | 36.99 | 11.59
CoSMo[29] | UIGR | 7.14 | 18.80 | 3.23 | 6.04 | 17.52 | 2.73 | 7.45 | 20.96 | 3.22 | 12.99 | 3.06
FashionIQ | 20.87 | 46.80 | 9.35 | 18.30 | 40.92 | 8.00 | 22.95 | 50.33 | 10.36 | 33.36 | 9.24
UIGR + FashionIQ | 23.50 | 49.48 | 10.42 | 17.96 | 41.76 | 8.22 | 25.14 | 52.58 | 11.68 | 35.07 | 10.11
RTIC[44] | UIGR | 8.13 | 21.32 | 3.53 | 7.85 | 20.31 | 3.32 | 9.43 | 23.56 | 4.12 | 15.10 | 3.66
FashionIQ | 25.93 | 51.76 | 12.00 | 22.37 | 46.57 | 9.91 | 27.84 | 56.65 | 13.10 | 38.52 | 11.00
UIGR + FashionIQ | 28.01 | 53.74 | 13.58 | 24.04 | 47.64 | 11.36 | 31.67 | 57.78 | 14.92 | 40.48 | 13.29

Table 7: The cross-domain (UIGR-TGR $\rightarrow$ FashionIQ [51]) evaluation
results. All results are reported on the three subsets of FashionIQ.

As shown in Table 6, we can conclude that one-hot labels also benefit from
mentioning target attributes, but text modality can integrate this kind of
attribute information into user feedback better.

###  D.3 Cross-domain evaluation

To demonstrate the universality of our generated user feedback, we conduct
cross-domain evaluation. Precisely, we compare the results of the same model
with 3 different strategies: (1) trained on UIGR-TGR, tested on FashionIQ
(zero-shot); trained and tested on FahsionIQ (fully supervised); (3) trained
on UIGR-TGR and FashionIQ, and then tested on FashionIQ (transfer learning).
As shown in Table 7, we can draw several conclusions: (1) Even under the zero-
shot setting, every method achieves reasonable performance; (2) With the
transferred knowledge from UIGR-TGR, every model has a substantial performance
gain (2.93 R@K and 1.85 mAP increase on average). In general, although our
user feedback is generated, its generalization ability is sufficient to help
the model achieve good performance on the manually annotated dataset.

##  Appendix E More qualitative results

###  E.1 Visualizations of retrieval results

To better understand the retrieval process of our unified interactive garment
retrieval, we visualize some retrieval results in Figure Figure 11 and Figure
12. It shows that given a sentence, our model captures both concrete and
abstract semantics, including fine-grained attributes and various garment
categories. Besides, many failure cases are also provided in Figure 13 and
Figure 14 to better understand our model’s performance. Even for the failure
cases, our model also provides very reasonable predictions.

###  E.2 Visualizations of learned latent spaces

To gain insights into the latent spaces learned by our model, we provide t-SNE
[46] visualizations for features processed by projection modules in two
branches. Figure 15(a) and 15(b) illustrate the latent space learned in TGR
and VCR branch, respectively. Both of them demonstrate that our model can
learn meaningful latent spaces, where the clusters contain garments with
similar appearances. Specifically, the latent space of the TGR branch mainly
focuses on the semantic visual similarity among garments, demonstrating that
our TGR branch is superior in learning visual attributes. Nevertheless, the
latent space of the VCR branch does not have a clear boundary as those in the
TGR branch. It seems to pay more attention to the common features of different
categories, demonstrating its ability to measure visual compatibility across
categories.

(a) is scoop neck and change color to mustard (b) is a crop top and has round
neck neckline type (c) is plunging neckline and with zip up opening type (d)
has flap type pocket and change color to maroon

(e) is applique and with loose fit silhouette

Figure 11: Retrieval results of our multi-task model on TGR subset.Yellow:
reference garment; Red: target garment (ground truth); Blue: other retrieved
garments.

(a) search a brown shoe that matches this jacket best (b) retrieve a hat
having a similar style with current skirt (c) retrieve a yellow shoe having a
similar style with current skirt (d) search an above the hip length jacket
that matches this shoe best

(e) retrieve a zip up skirt having a similar style with current top

Figure 12: Retrieval results of our model on VCR subset.

(a) is a sheath dress and has printed textile techniques (b) is napoleon lapel
and no special manufacturing technique (c) is a leggings and has curved fit
silhouette (d) is a tank top and is short length

(e) change non-textile material to plastic and change decorations to ruffle

Figure 13: Failure cases of our multi-task model on TGR subset.

(a) replace this shoe with a jacket that has lining textile techniques (b)
retrieve a blazer jacket having a similar style with current belt (c) for this
shoe, find a visually compatible tights with black color (d) retrieve a hat
having a similar style with current bag

(e) retrieve a gem dress having a similar style with current shoe

Figure 14: Failure cases of our multi-task model on VCR subset.

(a) Visualized latent space of TGR branch.

(b) Visualized latent space of VCR branch.

Figure 15: t-SNE visualizations for two latent spaces learned via our multi-
task model. Best zoom in and view in color.

## References

  * [1] Kenan E Ak, Ashraf A Kassim, Joo Hwee Lim, and Jo Yew Tham.  Learning attribute representations with localization for flexible fashion search.  In CVPR, 2018.
  * [2] Ziad Al-Halah, Rainer Stiefelhagen, and Kristen Grauman.  Fashion forward: Forecasting visual style in fashion.  In ICCV, 2017.
  * [3] Tamara L Berg, Alexander C Berg, and Jonathan Shih.  Automatic attribute discovery and characterization from noisy web data.  In ECCV, 2010.
  * [4] Soravit Changpinyo, Jordi Pont-Tuset, Vittorio Ferrari, and Radu Soricut.  Telling the what while pointing to the where: Multimodal queries for image retrieval.  In ICCV, 2021.
  * [5] Pranit Chawla, Surgan Jandial, Pinkesh Badjatiya, Ayush Chopra, Mausoom Sarkar, and Balaji Krishnamurthy.  Leveraging style and content features for text conditioned image retrieval.  In CVPR workshops, 2021.
  * [6] Yanbei Chen and Loris Bazzani.  Learning joint visual semantic matching embeddings for language-guided retrieval.  In ECCV, 2020.
  * [7] Yanbei Chen, Shaogang Gong, and Loris Bazzani.  Image search with text feedback by visiolinguistic attention learning.  In CVPR, 2020.
  * [8] Wen-Huang Cheng, Sijie Song, Chieh-Yun Chen, Shintami Chusnul Hidayati, and Jiaying Liu.  Fashion meets computer vision: A survey.  ACM CSUR, 2021.
  * [9] Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  Learning phrase representations using RNN encoder–decoder for statistical machine translation.  In EMNLP, 2014.
  * [10] Guillem Cucurull, Perouz Taslakian, and David Vazquez.  Context-aware visual compatibility prediction.  In CVPR, 2019.
  * [11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li.  Imagenet: A large-scale hierarchical image database.  In CVPR, 2009.
  * [12] Eric Dodds, Jack Culpepper, Simao Herdade, Yang Zhang, and Kofi Boakye.  Modality-agnostic attention fusion for visual search with text feedback.  arXiv preprint arXiv:2007.00145, 2020.
  * [13] Yuying Ge, Ruimao Zhang, Xiaogang Wang, Xiaoou Tang, and Ping Luo.  Deepfashion2: A versatile benchmark for detection, pose estimation, segmentation and re-identification of clothing images.  In CVPR, 2019.
  * [14] Xiaoxiao Guo, Hui Wu, Yu Cheng, Steven Rennie, Gerald Tesauro, and Rogério Schmidt Feris.  Dialog-based interactive image retrieval.  In NeurIPS, 2018.
  * [15] Xiao Han, Sen He, Li Zhang, and Tao Xiang.  Text-based person search with limited data.  In BMVC, 2021.
  * [16] Xintong Han, Zuxuan Wu, Phoenix X Huang, Xiao Zhang, Menglong Zhu, Yuan Li, Yang Zhao, and Larry S Davis.  Automatic spatially-aware fashion concept discovery.  In ICCV, 2017.
  * [17] Xintong Han, Zuxuan Wu, Yu-Gang Jiang, and Larry S Davis.  Learning fashion compatibility with bidirectional lstms.  In ACM MM, 2017.
  * [18] Xintong Han, Zuxuan Wu, Zhe Wu, Ruichi Yu, and Larry S Davis.  Viton: An image-based virtual try-on network.  In CVPR, 2018.
  * [19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [20] Mehrdad Hosseinzadeh and Yang Wang.  Composed query image retrieval using locally bounded features.  In CVPR, 2020.
  * [21] Yuxin Hou, Eleonora Vig, Michael Donoser, and Loris Bazzani.  Learning attribute-driven disentangled representations for interactive fashion retrieval.  In ICCV, 2021.
  * [22] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In CVPR, 2017.
  * [23] Surgan Jandial, Ayush Chopra, Pinkesh Badjatiya, Pranit Chawla, Mausoom Sarkar, and Balaji Krishnamurthy.  Trace: Transform aggregate and compose visiolinguistic representations for image search with text feedback.  arXiv preprint arXiv:2009.01485, 2020.
  * [24] Menglin Jia, Mengyun Shi, Mikhail Sirotenko, Yin Cui, Claire Cardie, Bharath Hariharan, Hartwig Adam, and Serge Belongie.  Fashionpedia: Ontology, segmentation, and an attribute localization dataset.  In ECCV, 2020.
  * [25] Wentao Jiang, Si Liu, Chen Gao, Jie Cao, Ran He, Jiashi Feng, and Shuicheng Yan.  Psgan: Pose and expression robust spatial-aware gan for customizable makeup transfer.  In CVPR, 2020.
  * [26] Donghyun Kim, Kuniaki Saito, Samarth Mishra, Stan Sclaroff, Kate Saenko, and Bryan A Plummer.  Self-supervised visual attribute learning for fashion compatibility.  In ICCV workshops, 2021.
  * [27] Jongseok Kim, Youngjae Yu, Hoeseong Kim, and Gunhee Kim.  Dual compositional learning in interactive image retrieval.  In AAAI, 2021.
  * [28] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2015.
  * [29] Seungmin Lee, Dongwan Kim, and Bohyung Han.  Cosmo: Content-style modulation for image retrieval with text feedback.  In CVPR, 2021.
  * [30] Yen-Liang Lin, Son Tran, and Larry S Davis.  Fashion outfit complementary item retrieval.  In CVPR, 2020.
  * [31] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang.  Deepfashion: Powering robust clothes recognition and retrieval with rich annotations.  In CVPR, 2016.
  * [32] Zheyuan Liu, Cristian Rodriguez-Opazo, Damien Teney, and Stephen Gould.  Image retrieval on real-life images with pre-trained vision-and-language models.  In ICCV, 2021.
  * [33] Ziwei Liu, Sijie Yan, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Fashion landmark detection in the wild.  In ECCV, 2016.
  * [34] Ling Lo, Chia-Lin Liu, Rong-An Lin, Bo Wu, Hong-Han Shuai, and Wen-Huang Cheng.  Dressing for attention: Outfit based fashion popularity prediction.  In ICIP, 2019.
  * [35] Zhi Lu, Yang Hu, Yan Chen, and Bing Zeng.  Personalized outfit recommendation with learnable anchors.  In CVPR, 2021.
  * [36] Zhi Lu, Yang Hu, Yunchao Jiang, Yan Chen, and Bing Zeng.  Learning binary code for personalized fashion recommendation.  In CVPR, 2019.
  * [37] Yunshan Ma, Xun Yang, Lizi Liao, Yixin Cao, and Tat-Seng Chua.  Who, where, and what to wear? extracting fashion knowledge from social media.  In ACM MM, 2019.
  * [38] Zhe Ma, Jianfeng Dong, Zhongzi Long, Yao Zhang, Yuan He, Hui Xue, and Shouling Ji.  Fine-grained fashion similarity learning by attribute-specific embedding network.  In AAAI, 2020.
  * [39] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv preprint arXiv:1807.03748, 2018.
  * [40] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  In NeurIPS, 2019.
  * [41] Xuebin Qin, Zichen Zhang, Chenyang Huang, Chao Gao, Masood Dehghan, and Martin Jagersand.  Basnet: Boundary-aware salient object detection.  In CVPR, 2019.
  * [42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [43] Ambareesh Revanur, Vijay Kumar, and Deepthi Sharma.  Semi-supervised visual representation learning for fashion compatibility.  In ACM RecSys, 2021.
  * [44] Minchul Shin, Yoonjae Cho, Byungsoo Ko, and Geonmo Gu.  Rtic: Residual learning for text and image composition using graph convolutional network.  arXiv preprint arXiv:2104.03015, 2021.
  * [45] Reuben Tan, Mariya I Vasileva, Kate Saenko, and Bryan A. Plummer.  Learning similarity conditions without explicit supervision.  In ICCV, 2019.
  * [46] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  JMLR, 2008.
  * [47] Mariya I Vasileva, Bryan A Plummer, Krishna Dusad, Shreya Rajpal, Ranjitha Kumar, and David Forsyth.  Learning type-aware embeddings for fashion compatibility.  In ECCV, 2018.
  * [48] Andreas Veit, Serge Belongie, and Theofanis Karaletsos.  Conditional similarity networks.  In CVPR, 2017.
  * [49] Nam Vo, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, and James Hays.  Composing Text and Image for Image Retrieval - an Empirical Odyssey.  In CVPR, 2019.
  * [50] Haokun Wen, Xuemeng Song, Xin Yang, Yibing Zhan, and Liqiang Nie.  Comprehensive linguistic-visual composition network for image retrieval.  In SIGIR, 2021.
  * [51] Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Rogerio Feris.  Fashion iq: A new dataset towards retrieving images by natural language feedback.  In CVPR, 2021.
  * [52] Yuchen Yang, Min Wang, Wengang Zhou, and Houqiang Li.  Cross-modal joint prediction and alignment for composed query image retrieval.  In ACM MM, 2021.
  * [53] Yifei Yuan and Wai Lam.  Conversational fashion image retrieval via multiturn natural language feedback.  In SIGIR, 2021.
  * [54] Feifei Zhang, Mingliang Xu, Qirong Mao, and Changsheng Xu.  Joint attribute manipulation and modality alignment learning for composing text and image to image retrieval.  In ACM MM, 2020.
  * [55] Gangjian Zhang, Shikui Wei, Huaxin Pang, and Yao Zhao.  Heterogeneous feature fusion and cross-modal alignment for composed image retrieval.  In ACM MM, 2021.
  * [56] Bo Zhao, Jiashi Feng, Xiao Wu, and Shuicheng Yan.  Memory-augmented attribute manipulation networks for interactive fashion search.  In CVPR, 2017.
  * [57] Mingchen Zhuge, Dehong Gao, Deng-Ping Fan, Linbo Jin, Ben Chen, Haoming Zhou, Minghui Qiu, and Ling Shao.  Kaleido-bert: Vision-language pre-training on fashion domain.  In CVPR, 2021.