# Aligning Source Visual and Target Language Domains for Unpaired Video
Captioning

Fenglin Liu, Xian Wu, Chenyu You, Shen Ge, Yuexian Zou, and Xu Sun Fenglin Liu
and Yuexian Zou are with ADSPLAB, School of ECE, Peking University. Yuexian
Zou also with the Peng Cheng Laboratory, Shenzhen, China. E-mail:
{fenglinliu98<EMAIL_ADDRESS>Xu Sun is with MOE Key Laboratory of
Computational Linguistics, School of EECS, Peking University, China. E-mail:
<EMAIL_ADDRESS>Xian Wu and Shen Ge are with Tencent, China.
E-mail: {kevinxwu<EMAIL_ADDRESS>Chenyu You is with Department of
Electrical Engineering, Yale University, USA. E-mail<EMAIL_ADDRESS>Manuscript received 12 April 2021; revised 23 August 2021; accepted 16
November 2021. Date of publication 2 December 2021(Corresponding authors: Xu
Sun, Yuexian Zou and Xian Wu.)Recommended for acceptance by K. Saenko.Digital
Object Identifier no. 10.1109/TPAMI.2021.3132229

###### Abstract

Training supervised video captioning model requires coupled video-caption
pairs. However, for many targeted languages, sufficient paired data are not
available. To this end, we introduce the unpaired video captioning task aiming
to train models without coupled video-caption pairs in target language. To
solve the task, a natural choice is to employ a two-step pipeline system:
first utilizing video-to-pivot captioning model to generate captions in pivot
language and then utilizing pivot-to-target translation model to translate the
pivot captions to the target language. However, in such a pipeline system, 1)
visual information cannot reach the translation model, generating visual
irrelevant target captions; 2) the errors in the generated pivot captions will
be propagated to the translation model, resulting in disfluent target
captions. To address these problems, we propose the Unpaired Video Captioning
with Visual Injection system (UVC-VI). UVC-VI first introduces the Visual
Injection Module (VIM), which aligns source visual and target language domains
to inject the source visual information into the target language domain.
Meanwhile, VIM directly connects the encoder of the video-to-pivot model and
the decoder of the pivot-to-target model, allowing end-to-end inference by
completely skipping the generation of pivot captions. To enhance the cross-
modality injection of the VIM, UVC-VI further introduces a pluggable video
encoder, i.e., Multimodal Collaborative Encoder (MCE). The experiments show
that UVC-VI outperforms pipeline systems and exceeds several supervised
systems. Furthermore, equipping existing supervised systems with our MCE can
achieve 4% and 7% relative margins on the CIDEr scores to current state-of-
the-art models on the benchmark MSVD and MSR-VTT datasets, respectively.

###### Index Terms:

Video Captioning, Unpaired Video Captioning, Pipeline System, Pseudo
Supervised Training, Adversarial Training.

##  1 Introduction

Video captioning targets to understand the visual content of given videos and
generate corresponding descriptive sentences. Video captioning has a wide
range of applications, such as video retrieval [1], human-robot interaction
[2] and visually impaired people aiding [3]. Due to its broad usage scenarios,
video captioning has received extensive research interests. Among existing
approaches, the encoder-decoder based systems [4, 5, 6, 7] have achieved great
success in advancing the state-of-the-art.

Currently, training a supervised video captioning model requires large volume
of video and caption pairs. However, for many targeted languages (i.e., Non-
English), sufficient video-caption pairs are not available. For example, to
generate video captions in Chinese, it’s a necessity to collect video-Chinese
caption pairs which is both costly and time-consuming. To this end, we
introduce the problem of unpaired video captioning in which the video-caption
pairs in target language are not available, which has practical value for non-
English languages.

As shown in the left sub-figure of Figure 1, to build video captioning model
without paired data, which has not been well studied yet, an intuitive
approach is to build a pivot language based, two-step pipeline system. In
implementation, given an input video, the pipeline system firstly generates
the captions in pivot language by a video-to-pivot video captioning model;
then translates the pivot captions to the target language by a pivot-to-target
translation model. In this way, the pipeline system can leverage existing
resources, e.g., video-pivot paired dataset (denoted as $D_{1}$), pivot-target
paired dataset (denoted as $D_{2}$), as well as pre-trained video captioning
and translation models. Take video Chinese captioning for example, the
pipeline system uses English as a pivot language, and trains a video-to-pivot
captioning model on video-English pairs [8] and a pivot-to-target translation
model on English-Chinese pairs (e.g. WMT En-Zh (http://statmt.org/wmt17/)).
Sending the video through the captioning model and subsequently the
translation model, a video caption in Chinese can be generated.

The pipeline systems have achieved great success in many tasks, such as
machine listening comprehension [9], spoken question answering [10] and
unpaired machine translation [11]. However, when applying to video captioning,
the training of the captioning model and the translation model are rather
separated compared to the conventional end-to-end systems. As shown in Figure
1, on one hand, visual information in the video-to-pivot captioning model is
unable to reach the pivot-to-target translation model, resulting in the lack
of visual details in generated target captions, bringing in visual irrelevancy
errors; On the other hand, the errors in the pivot captions generated by the
video-to-pivot captioning model cannot be corrected by the pivot-to-target
translation model [12, 13], resulting in disfluent target captions. Moreover,
the optimization over the pivot-to-target translation model can not be back-
propagated to the video-to-pivot captioning model.

To address above problems of the pipeline system, we propose the Unpaired
Video Captioning with Visual Injection system (UVC-VI) which includes two
major components: Visual Injection Module (VIM) and Multimodal Collaborative
Encoder (MCE). As shown in the right sub-figure of Figure 1, UVC-VI uses VIM
and MCE to directly bridge the encoder of the captioning model and the decoder
of the translation model. As a result, 1) the source visual information can be
injected into the target language domain, which addresses the visual
irrelevancy problems; 2) UVC-VI no longer generates captions in pivot
language, which addresses the disfluency problems by allowing end-to-end
inference. Therefore, our UVC-VI can solve the problems in the pipeline
systems, and generate more detailed, accurate and fluent captions in target
language.

For clarity, we first introduce the VIM, followed by the MCE. The input of VIM
is the visual embedding, which is the output of video encoder and has rich
source visual information. Next, VIM uses the proposed pseudo supervised
training and adversarial training to project the visual embedding to textual
embedding, which injects the source visual information into the target
language domain. The textual embedding, i.e., projected visual embedding, is
directly fed into the decoder part of the translation model to generate
captions in target language. We further propose the MCE to enhance the cross-
modality projection of the VIM. In detail, MCE first accepts original visual
embedding from the video encoder and incorporates textual
concepts111Motivation: Textual concepts contain a set of words describing
object (e.g., cat), attribute (e.g., small) and relationship (e.g., standing)
of videos [14]. Therefore, textual concepts provide a more semantic
representation of visual information and thus help shorten the gap between
visual and the language domains. [14] into them. The transformed textual-
enriched visual embedding helps to shorten the modality gap between source
visual and target language domains for VIM, and in turn enhance the cross-
modality projection of the VIM.

In the absence of video-caption pairs in the target language domain (denoted
as $D_{3}$), we train our approach using existing resources, i.e., video-pivot
paired dataset $D_{1}$ and pivot-target paired dataset $D_{2}$.222The two
paired datasets, i.e., $D_{1}$ and $D_{2}$, can have no overlap. To learn the
parameters of VIM and MCE, we introduce two training approaches: pseudo
supervised training and adversarial training. Take video Chinese captioning
for example, given video-English pairs $D_{1}$ and English-Chinese pairs
$D_{2}$, 1) for the pseudo supervised training, we feed video and English
caption pairs from $D_{1}$ into the encoder of the video-to-English captioning
model and the encoder of English-to-Chinese translation model, to generate
visual embedding ${{V}}$ and textual embedding ${{T}_{1}}$. The
${{V}}$-${{T}_{1}}$ pairs are used as pseudo paired data to train the VIM; 2)
for the adversarial training, we further use $D_{2}$ by feeding the English
sentence from $D_{2}$ into the encoder of English-to-Chinese translation model
to generate textual embedding ${{T}_{2}}$. Since there are no pairs of $V$ and
$T_{2}$ , we adopt the adversarial training [15, 16] to exploit ${{V}}$ and
${{T}_{2}}$ to further train our VIM. In this manner, once the VIM are
trained, we can project (i.e., inject) the source visual information to the
target language domain directly. Besides, as shown in the right sub-figure of
Figure 1, the encoder of the video-to-English captioning model and the decoder
of the English-to-Chinese translation model can be concatenated by VIM to form
an end-to-end video Chinese captioning model. The decoder of the captioning
model and the encoder of the translation model are simply dropped away, so the
pivot video captions in English are no longer generated. In this way, our UVC-
VI is capable of generating desirable and fluent captions without the training
on the pairs of video and target caption $D_{3}$. The experiments and analyses
on two benchmark datasets, i.e., MSVD [17] and MSR-VTT [8], prove our
arguments and verify the effectiveness of our proposed approach. Moreover, we
extend our UVC-VI to image captioning task, obtaining positive experimental
results.

Overall, our main contributions are as follows:

  * •

In this work, we introduce the problem of unpaired video captioning where the
video-caption pairs in target language are not available, while all existing
studies need sufficient video-caption pairs in target language.

  * •

We make the first attempt to conduct unpaired video captioning. In particular,
we propose the Unpaired Video Captioning with Visual Injection system (UVC-
VI), which consists of the Visual Injection Module (VIM) and the Multimodal
Collaborative Encoder (MCE), to address the visual irrelevancy and disfluency
problems in the pipeline systems.

  * •

The experiments show that the UVC-VI outperforms both pipeline systems and
several supervised systems over all metrics. In addition to automatic metrics,
we also conduct human evaluations on three different target languages, i.e.,
Chinese, French and German, to verify our arguments and prove the advantage of
our proposed UVC-VI from the perspective of user experience.

  * •

It is worth noting that the MCE can be easily integrated into conventional
supervised video captioning systems as an individual video encoder module.
Equipping MCE helps these systems to achieve new state-of-the-art results on
MSVD and MSR-VTT benchmark datasets.

##  2 Related Work

###  2.1 Conventional Supervised Video Captioning

In recent years, there is a surge of research interests in video captioning.
There are a large number of encoder-decoder based neural models proposed for
video captioning [4, 19, 20, 21, 22, 7, 23, 6]. State-of-the-art approaches
[20, 24, 25, 21, 7] introduce a video encoder to encode the video and a
language decoder, e.g., LSTM [26] and Transformer [27, 28, 6], to generate
coherent captions with the attention mechanism [29, 30]. However, all of
existing video captioning models are trained on large-scale video-caption
pairs, while collecting paired video-caption data for training is expensive
and time-consuming. In contrast to the existing models, we utilize the
pipeline system to relax the reliance on the paired dataset for video
captioning.

###  2.2 Pipeline Systems

Currently, pipeline systems have been proved to be effective in various
applications: 1) Machine Listening Comprehension [9] and Spoken Question
Answering [10] which include an automatic speech recognition (ASR) model [31]
followed by either a machine comprehension model [32] or an information
retrieval model [10]; 2) Medical Image Classification [33] which includes a
image segmentation model [34] followed by a classification model; 3) Unpaired
Machine Translation [11, 35, 36] which stacks a source-to-pivot translation
model followed by a pivot-to-target translation model. Specifically, in image
captioning, [37] first adopts a multi-label classification framework to
predict the visual attributes, and then employs an RNN model to generate
captions. Although these pipeline systems achieve success in multiple tasks,
stacking multiple models can cause error accumulation problems. For example,
in machine listening comprehension, [38] found that ASR errors severely impair
the machine comprehension system. Similarly, we find that the pipeline systems
for unpaired video captioning will bring the visual irrelevancy and disfluency
errors.

###  2.3 Unpaired Image Captioning

Although the unpaired video captioning has not been well studied yet, the
unpaired image captioning has been explored recently [40, 41, 42, 43, 44, 45,
46, 47]. Specifically, unpaired image captioning, which is also known as
unpaired image-to-sentence translation, is essentially similar to the problem
of unpaired sentence-to-sentence translation [11, 35] and unpaired image-to-
image translation [16]. Typically, the source sentence/image and the target
sentence/image are mapped into a common latent space, in which
sentences/images with the same semantic/visual meaning are well aligned so
that unpaired translation can be performed. Nevertheless, due to the great
disparities between the vision and the language domains, unpaired image
captioning is considerably more challenging.

To train unpaired image captioning models, existing approaches can be
classified into two categories: unsupervised approaches and pipeline
approaches. For the unsupervised approaches, [43, 47] and [44, 46] take the
scene graph and textual concepts to align the vision and the language domains,
respectively, in an adversarial way. Although these works are totally
unsupervised, the existing resources like large-scale paired datasets and pre-
trained models are not fully utilized. For the pipeline approaches, [42] first
adopts a pivot-based model to generate captions in a pivot language (Chinese)
and then translate them into the target language (English). In particular,
they improve pipeline approaches by adapting 1) the encoder of the translation
model to the decoder of the captioning model; 2) the decoder of the
translation model to the decoder of a pre-trained auto-encoder. Since only
text embedding parameters are optimized and more processing steps are
appended, the drawbacks of pipeline approaches, i.e., visual irrelevancy and
disfluency errors, are not fully addressed. Besides, [40] and [41] also
introduce the translation model in unpaired image captioning, but they did not
follow the pipeline approach. Instead they translate the captions in pivot
language to the target language, and use the translated captions to train the
image to target language captioning model. Since the translation model is
fixed in advance, the translation error could constantly affect the captioning
performance.

Although unpaired image captioning has been explored, the unpaired video
captioning is relatively more challenging, because videos involve scenes that
are volatile and likely to change. Besides, there are three source modalities
(i.e., image, motion and audio) and the temporal dynamics information should
be captured to understand the video efficiently, while image captioning only
use image as input. In our work, we focus on making full use of existing
resources, e.g., video-pivot and pivot-target paired datasets and pre-trained
models, to perform the unpaired video captioning.

Figure 1: Illustration of the pipeline system (left) and our UVC-VI system
(right) under the video-to-Chinese captioning scenario. The target caption
generated by pipeline system suffers from visual irrelevancy (see words
highlighted red) and disfluency (see words highlighted blue) errors. The words
highlighted purple denote the detailed visual element that is included in the
caption. For better understanding, we add the English translation below the
Chinese captions in brackets.

##  3 Approach

We first formulate the conventional supervised video captioning problems;
Then, we describe the pipeline system and the proposed UVC-VI for unpaired
video captioning.

###  3.1 Problem Formulation of Video Captioning

Given a video ${X}$, the goal is to generate a descriptive target sentence
${Y}=\\{y_{1},y_{2},\dots,y_{T}\\}$. The state-of-the-art systems [4, 20, 24,
21] normally include a video encoder and a target sentence decoder, which can
be formulated as:

$\displaystyle\text{Video Encoder}:X\to V;\ \ \text{Target Decoder}:V\to Y.$
(1)

The video encoder aims to extract the visual embedding ${{V}}$ of input video.
We use the set
${{V}}=[{{V}}_{\text{T}};{{V}}_{\text{I}};{{V}}_{\text{M}};{{V}}_{\text{A}}]$
to denote the visual embedding of video.
${{V}}_{\text{I}}\in\mathbb{R}^{N\times d_{i}}$ denotes the image features
extracted by CNNs [49]; ${{V}}_{\text{M}}\in\mathbb{R}^{N\times d_{m}}$ and
${{V}}_{\text{A}}\in\mathbb{R}^{N\times d_{a}}$ denote the motion and audio
features in video which can be extracted by 3D CNN [50] and Bag-of-Audio-Words
[51], respectively. In this paper, we further follow [14] to extract a set of
textual concepts ${{V}}_{\text{T}}\in\mathbb{R}^{N\times d_{c}}$ from the
input video, which describes the object (e.g., table), attribute (e.g.,
wooden) and relationship (e.g., sitting) of the video. The embedding vectors
of these textual concepts are then taken as the ${{V}}_{\text{T}}$. More
details of how to generate ${{V}}_{\text{T}}$ can be found in [14]. In
implementation, all extracted features are projected to the same dimension of
$d_{h}$, constituting the visual embedding
${{V}}=[{{V}}_{\text{T}};{{V}}_{\text{I}};{{V}}_{\text{M}};{{V}}_{\text{A}}]\in\mathbb{R}^{4N\times
d_{h}}$.

The language decoder, e.g., LSTM [26] and Transformer [27], is used to predict
the descriptive target sentence ${Y}$ from ${{V}}$. Given the ground truth
sentence for the source video, we can simply train an encoder-decoder system
by minimizing supervised training loss, e.g., cross-entropy loss.

Here, we introduce three datasets: 1) the video-pivot paired dataset $D_{1}$ ;
2) the pivot-target paired dataset $D_{2}$ ; 3) the video-target paired
dataset $D_{3}$. For example, in the context of generating Chinese captions
for input videos, $D_{1}$ refers to pairs of videos and their corresponding
captions in English; $D_{2}$ refers to pairs of English sentences and their
translations in Chinese; $D_{3}$ refers to pairs of videos and their
corresponding captions in Chinese.333The English captions in $D_{1}$ and the
English sentences in $D_{2}$ are separate sets and can have no overlap. To
train the video Chinese captioning model in Eq. (1), all existing methods need
the dataset in the form of $D_{3}$. Although many datasets of video-English
caption pairs ($D_{1}$) have been released [8, 52, 53, 17], in terms of non-
English languages, only a very few datasets ($D_{3}$) [54] are available. As a
result, insufficient paired data poses a great challenge for building video
captioning models for non-English languages.

###  3.2 Pipeline System for Unpaired Video Captioning

A simple yet effective solution is to build a pipeline system, assembling a
video-to-pivot captioning model and a pivot-to-target translation model
together. The trainings are conducted independently on different datasets,
i.e., a video-pivot paired dataset $D_{1}$ for training the video-to-pivot
video captioning model, as well as a pivot-target paired dataset $D_{2}$ for
training the pivot-to-target machine translation model. As a result, there is
no need for a video-target paired dataset $D_{3}$. As shown in the left sub-
figure of Figure 1, the pipeline system can be formulated as:

$\displaystyle\text{Video Encoder}:X\to V;\quad\text{Pivot Decoder}:V\to P;$
(2) $\displaystyle\text{Pivot Encoder}:P\to T;\quad\text{Target Decoder}:T\to
Y,$

where ${P}$ denotes the pivot sentence generated by the video-to-pivot
captioning model and ${{T}}$ denotes the textual embedding in the target
language. As we can see, the visual embedding $V$, which contains the rich
visual information, are separated from the translating process by the
intermediate pivot caption, leading to possible visual irrelevancy errors.
Moreover, compared with Eq. (1), two more steps are added to Eq. (2), which
increases the chances of error generation and amplification, especially the
generation process [12, 13], leading to possible disfluency errors.

TABLE I: Training datasets needed for different approaches. $D_{1}$, $D_{2}$ and $D_{3}$ represent the video-pivot paired dataset, pivot-target paired dataset and video-target paired dataset, respectively. There is no overlap between $D_{1}$, $D_{2}$ and $D_{3}$. As we can see, our approach aims to leverage existing resources, e.g., video-pivot and pivot-target paired datasets, to perform the unpaired video captioning, i.e., generate fluent and desirable captions in target language without the training on video-target paired dataset $D_{3}$. As a result, our approach has practical value for non-English languages, especially for the low-resource target language applications, e.g., French and German, in which the pairs of video and target caption are not available. Methods | $D_{1}$ | $D_{2}$ | $D_{3}$
---|---|---|---
Conventional Supervised System. i.e., Eq. (1) | $\times$ | $\times$ | $\surd$
Pipeline System, i.e., Eq. (2) | $\surd$ | $\surd$ | $\times$
Proposed UVC-VI System, i.e., Eq. (LABEL:eq:uvc-vi) | $\surd$ | $\surd$ | $\times$

###  3.3 Unpaired Video Captioning with Visual Injection

To address the issues in the pipeline systems, we propose the UVC-VI system,
which introduces the Multimodal Collaborative Encoder (MCE) and Visual
Injection Module (VIM) to connect the encoder of the video-to-pivot captioning
model and the decoder of the pivot-to-target translation model. As shown in
the Figure 1, our proposed UVC-VI can be formulated as:

$\displaystyle\text{Video Encoder + MCE}:X\to{{V}};$ (3) VIM
$\displaystyle:{{V}}\to{{T}};\quad\text{Target Decoder}:{{T}}\to{Y}.$

Compared to Eq. (2), in Eq. (LABEL:eq:uvc-vi), we make three updates: 1)
injecting the source visual information into the target language; 2) replacing
two steps, i.e., Pivot Decoder and Pivot Encoder, in Eq. (2) with the proposed
MCE and VIM; 3) skip the generation of pivot captions ${P}$. Specifically, to
efficiently inject the source visual information into the target language,
i.e., conduct the cross-modality projection, VIM should devote on aligning
source visual and target language domains without the training on the pairs of
video and target caption. Then, the MCE is proposed to enhance the cross-
modality projection of the VIM. As a result, 1) the source visual information
can be injected into the target language domain, which addresses the visual
irrelevancy errors; 2) UVC-VI no longer generates captions in pivot language,
allowing end-to-end inference, which addresses the disfluency errors.

For clarity, we first introduce the VIM in Section 3.3.1, then introduce the
MCE in Section 3.3.2. As shown in Eq. (LABEL:eq:uvc-vi), VIM first receives
visual embedding $V$, and then projects the visual embedding $V$ to textual
embedding $T$, i.e., inject the source visual information into the target
language domain. Therefore, the input of VIM is ${V}$, the output of VIM is
${T}$. Due to the lack of video-target paired dataset $D_{3}$, i.e., $X$-$Y$
pairs, we can not train our approach by minimizing supervised training loss as
like in existing works. Instead, we propose the pseudo supervised training and
adversarial training to train the cross-modality projection of VIM. In
implementations, as shown in Table I, we make full use of the video-pivot
paired dataset $D_{1}$ and the pivot-target paired dataset $D_{2}$. Firstly,
we feed the video and pivot caption pairs from $D_{1}$ into the encoder of the
video-to-pivot captioning model and the encoder of pivot-to-target translation
model, respectively, to acquire visual embedding ${{V}}$ and textual embedding
${{T}_{1}}$. The ${{V}}$-${{T}_{1}}$ pairs can be used as pseudo paired data
in pseudo supervised training to train the VIM. Secondly, we further feed the
pivot sentence from $D_{2}$ into the encoder of pivot-to-target translation
model to acquire the textual embedding ${{T}_{2}}$. Since the two datasets,
i.e., $D_{1}$ and $D_{2}$, can have no overlap, there are no pairs of ${{V}}$
and ${{T}_{2}}$. To exploit the $V$ and ${{T}_{2}}$ to improve the
performance, we introduce the adversarial training [15] to further train the
proposed VIM.

####  3.3.1 Visual Injection Module (VIM)

As expected, the VIM aims to inject the visual information into the target
language domain by aligning the source visual and target language domains. In
other words, the VIM devotes on projecting the visual embedding ${V}$ of video
domain to the textual embedding ${{T}}$ of target language domain without the
need for video-target pairs. To achieve this, we empirically find that a
multilayer perceptron (MLP) [55] can efficiently conduct the projection,
therefore, we directly apply the MLP to implement the VIM:

${{T}}\approx\text{VIM}(V)=\max\left(0,V{W}_{\text{f}}+{b}_{\text{f}}\right){W}_{\text{ff}}+{b}_{\text{ff}},$
(4)

where ${W}_{\text{f}}\in\mathbb{R}^{d_{h}\times 2d_{h}}$ and
${W}_{\text{{ff}}}\in\mathbb{R}^{2d_{h}\times d_{h}}$ are linear
transformation matrices; ${b}_{\text{f}}$ and ${b}_{\text{ff}}$ are the bias
terms.

Parameter Optimization As mentioned above, in order to estimate the parameters
of VIM, we propose two training methods, pseudo supervised training and
adversarial training. The pseudo supervised training method relies on the
video-pivot pairs from $D_{1}$ for training and the adversarial training
method further relies on the pivot-target pairs from $D_{2}$ for training. It
is worth noting that neither method requires the pairs of video and caption in
target language.

$\bullet$ Pseudo Supervised Training In order to effectively make use of
$D_{1}$, we propose the pseudo supervised training. In implementation, we can
generate a large number of the pairs between ${{V}}$ and ${{T_{1}}}$ by
running the encoder part of video-to-pivot captioning model and the encoder
part of pivot-to-target translation model on the video-pivot pairs from
$D_{1}$. In detail, we first extract the visual embedding ${{V}}$ by inputting
the video from $D_{1}$ into the encoder of the captioning model; Then we
extract the textual embedding ${{T_{1}}}$ by inputting the pivot sentence from
$D_{1}$ into the encoder of the translation model. In this way, we can acquire
the coupled ${{V}}$-${{T_{1}}}$ pairs. Since UVC-VI aims to project the visual
embedding ${{V}}$ to the textual embedding ${{T_{1}}}$, we use the L1 norm
between $\text{VIM}(V)$ and $T_{1}$ as the training loss:

$\displaystyle{L}_{\text{pseudo}}({{V}},{{T_{1}}})=\mathbb{E}_{({{V}},{{T_{1}}})\sim
D_{1}}\left[\|\text{VIM}\left(V\right)-{{T_{1}}}\|_{1}\right].$ (5)

$\bullet$ Adversarial Training In order to effectively make use of the pivot-
target paired dataset $D_{2}$, we further extract the textual embedding
$T_{2}$ by inputting the pivot sentence from $D_{2}$ into the encoder of the
translation model. Based on the visual embedding $V$ from $D_{1}$ and the
textual embedding $T_{2}$ from $D_{2}$, since the two datasets, i.e., $D_{1}$
and $D_{2}$, can have no overlap, there are no pairs of $V$ and $T_{2}$, so we
introduce the adversarial training [16, 15] to train our approach. In
implementation, we introduce a discriminator D to distinguish between the
$\text{VIM}(V)$ and the $T_{2}$. Then we adopt the adversarial loss in Eq.
(6):

$\displaystyle{L}_{\text{adv}}({{V}},{{T_{2}}})=$
$\displaystyle\mathbb{E}_{{{V}}\sim
D_{1}}\left[\log\left(1-\text{D}(\text{VIM}({{V}}))\right)\right]$ (6)
$\displaystyle+\mathbb{E}_{{{T_{2}}}\sim
D_{2}}\left[\log\text{D}({{T_{2}}})\right].$

With the adversarial loss, the VIM devotes on projecting the visual embedding
$V$ of video domain to target language domain $\text{VIM}(V)$, while D devotes
on distinguishing between projected visual embedding $\text{VIM}(V)$ and
textual embedding $T_{2}$. In other words, through the adversarial training,
we could align the latent space of video domain and target language domain
without the requirement for video-target pairs.

Inspired by [16], we further introduce a cycle strategy to enhance such
alignment. For the cycle strategy, a Textual Injection Module (TIM), which
shares the same structure as VIM and is defined as:

$V\approx\text{TIM}(T)=\max\left(0,T{W}^{\prime}_{\text{f}}+{b}^{\prime}_{\text{f}}\right){W}^{\prime}_{\text{ff}}+{b}^{\prime}_{\text{ff}},$
(7)

and a new discriminator ${\text{D}^{\prime}}$ are introduced, where the former
devotes on injecting textual embedding ${{T_{2}}}$ to the visual domain and
the latter devotes on discriminating between $\text{TIM}(T_{2})$ and the $V$.
Similarly as Eq. (6), there is an adversarial loss
${{L}^{\prime}}_{\text{adv}}(T_{2},V)$ to train the TIM and
${\text{D}^{\prime}}$:

$\displaystyle{{L}^{\prime}}_{\text{adv}}({{T_{2}}},{{V}})=$
$\displaystyle\mathbb{E}_{{{T_{2}}}\sim
D_{2}}\left[\log\left(1-{\text{D}^{\prime}}(\text{TIM}({{T_{2}}}))\right)\right]$
(8) $\displaystyle+\mathbb{E}_{{{V}}\sim
D_{1}}\left[\log{\text{D}^{\prime}}(V))\right].$

The cycle consistency loss is further introduced to regularize the adversarial
training [16]:

$\displaystyle{L}_{\text{cyc}}({{V}},{{T_{2}}})=$
$\displaystyle\mathbb{E}_{{{V}}\sim
D_{1}}\left[\|\text{TIM}(\text{VIM}(V)))-V)\|_{1}\right]+$ (9)
$\displaystyle\mathbb{E}_{{{T_{2}}}\sim
D_{2}}\left[\|\text{VIM}(\text{TIM}({{T_{2}}}))-{{T_{2}}}\|_{1}\right].$

With above equations, the proposed UVC-VI is able to maintain the backward
cycle consistency, i.e., $V\to\text{VIM}(V)\to\text{TIM}(\text{VIM}(V))\approx
V$.

Overall, combining the pseudo supervised training loss and adversarial
training loss, the full training objective is defined as:

$\displaystyle{L}_{\text{full}}(V,$ $\displaystyle\
T_{1},T_{2})={L}_{\text{pseudo}}(V,T_{1})\ +$
$\displaystyle\alpha\left({L}_{\text{adv}}({{V}},T_{2})+{L}^{\prime}_{\text{adv}}(T_{2},V)+\beta{L}_{\text{cyc}}(V,T_{2})\right),$
(10)

where $\alpha=0.1$ and $\beta=10$ are the hyper-parameters that control the
regularization.

Through the introduced Eq. (3.3.1), we are able to estimate the parameters of
the VIM. As a result, the VIM can align the source visual and target language
domains, resulting in efficiently projecting the visual embedding of source
video domain to the textual embedding of target language domain without the
requirement for video-target pairs.

####  3.3.2 Multimodal Collaborative Encoder (MCE)

In this section, we further introduce the Multimodal Collaborative Encoder
(MCE) to enhance the cross-modality projection of the Visual Injection Module
(VIM). Since the visual embedding will be projected to the language domain and
will be used for textual caption generation, based on the attention mechanism
[27], we propose the MCE to transform the original visual embedding into the
textual-enriched visual embedding.

In implementation, as stated in Section 3.1, the ${{V}}_{\text{T}}$ denotes
the set of textual concepts extracted from the video. According to the
attention theorem444The attention mechanism computes the association weights
between different features. Thus, the attention mechanism allows probabilistic
many-to-many relations instead of monotonic relations, as in [27, 56, 57]., if
we take textual features ${{V}}_{\text{T}}$ as the query, and take the
image/motion/audio features as the key and value, the image/motion/audio
features will be associated with the textual features. Thus, through combining
the textual features with the associated image/motion/audio features, we can
acquire the textual-enriched image/motion/audio embedding, i.e., textual-
enriched visual embedding. At last, we further introduce a gate mechanism [58]
to calibrate the contributions of each type of features. The MCE is defined
as:

${V}^{\prime}=\text{MCE}(V)=\sum_{i\in\\{\text{I},\text{M},\text{A}\\}}{\gamma}_{i}\odot\text{Attention}\left({{V}}_{\text{T}},{{V}}_{i}\right),$
(11)

where $V^{\prime}$ denotes the text-enriched visual embedding and the
${\gamma}_{i}$ and $\text{Attention}\left(\cdot,\cdot\right)$ are defined as
follows:

$\displaystyle{\gamma}_{i}=$
$\displaystyle\sigma\left(\left[{{V}}_{\text{T}};\text{Attention}\left({{V}}_{\text{T}},{{V}}_{i}\right)\right]W_{i}\right),$
(12) Attention
$\displaystyle(x,y)=\text{softmax}\left(xW_{\text{q}}\left(yW_{\text{k}}\right)^{\top}\right)yW_{\text{v}}.$

In above equations, $\odot$ stands for the element-wise multiplication;
$\sigma$ represents the sigmoid activation; and $[\cdot;\cdot]$ denotes the
concatenation operation. $W_{i}\in\mathbb{R}^{2d_{h}\times d_{h}}$,
$W_{\text{q}}\in\mathbb{R}^{d_{h}\times d_{h}}$,
$W_{\text{k}}\in\mathbb{R}^{d_{h}\times d_{h}}$ and
$W_{\text{v}}\in\mathbb{R}^{d_{h}\times d_{h}}$ are learnable parameters. The
proposed MCE is then followed by dropout [59], shortcut connection [60] and
layer normalization [61].

The obtained textual-enriched visual embedding ${V}^{\prime}$ can shorten the
modality gap between visual and textual modalities for VIM. In our UVC-VI,
during the training of VIM, i.e., Eq. (3.3.1), we directly replace the $V$
with ${V}^{\prime}$. Meanwhile, the MCE can be integrated into other existing
supervised video captioning models to improve their performance (see Section
5.2 and Table VI).

TABLE II: Performance of automatic evaluations on the MSVD and MSR-VTT video captioning datasets under the unpaired setting. The I, M and A denote image, motion and audio features, respectively. Higher value denotes better performance in all columns. The Underlined numbers denote the best results of unpaired methods, where coupled video and target captions are not available; The Bold numbers denote the best results across all approaches, both supervised and unpaired. ‡ denotes our own implementation. As we can see, both our UVC-VI (I+M+A) and UVC-VI (I+M) outperform the pipeline systems, even surpassing several supervised video captioning models in terms of CIDEr.  Types | Methods | Features |  Dataset: MSVD [17] |  Dataset: MSR-VTT [8]
---|---|---|---|---
BLEU-4 | METEOR | ROUGE-L | CIDEr | BLEU-4 | METEOR | ROUGE-L | CIDEr
Setting: Conventional Supervised Video Captioning
Supervised Systems (Section 3.1) |  AF${}_{\text{(ICCV2017)}}$ [62] | I+M+A | - | - | - | - | 39.7 | 25.5 | - | 40.0
MA-LSTM${}_{\text{(ACMMM2017)}}$ [19] | I+M+A | - | - | - | - | 36.5 | 26.5 | 59.8 | 41.0
Two-stream${}_{\text{(TPAMI2020)}}$ [63] | I+M | 54.3 | 33.5 | - | 72.8 | 39.7 | 27.0 | - | 42.1
PickNet${}_{\text{(ECCV2018)}}$ [64] | I | 52.3 | 33.3 | 69.6 | 76.5 | 39.4 | 27.3 | 59.7 | 42.3
RecNet${}_{\text{(TPAMI2020)}}$ [65] | I | 52.3 | 34.1 | 69.8 | 80.3 | 39.1 | 26.6 | 59.3 | 42.7
TDConvED${}_{\text{(AAAI2019)}}$ [21] | I | 53.3 | 33.8 | - | 76.4 | 39.5 | 27.5 | - | 42.8
POS-CG${}_{\text{(ICCV2019)}}$ [66] | I+M | - | - | - | - | 38.3 | 26.8 | 60.1 | 43.4
STAT${}_{\text{(TMM2020)}}$ [67] | I+M | 52.0 | 33.3 | - | 73.8 | 39.3 | 27.1 | - | 43.8
GRU-EVE${}_{\text{(CVPR2019)}}$ [22] | I+M | 47.9 | 35.0 | 71.5 | 78.1 | 38.3 | 28.4 | 60.7 | 48.1
SAAT${}_{\text{(CVPR2020)}}$ [5] | I+M | 46.5 | 33.5 | 69.4 | 81.0 | 40.5 | 28.2 | 60.9 | 49.1
SGN${}_{\text{(AAAI2021)}}$ [7] | I+M | 52.8 | 35.5 | 72.9 | 94.3 | 40.8 | 28.3 | 60.8 | 49.5
MGSA${}_{\text{(AAAI2019)}}$ [24] | I+M+A | - | - | - | - | 45.4 | 28.6 | - | 50.1
Setting: Unpaired Video Captioning
Pipeline Systems (Section 3.2) |  MA-LSTM [19] \+ Google Translator [68]‡ | I+M | 43.1 | 29.5 | 65.8 | 53.3 | 31.5 | 23.9 | 54.2 | 30.6
SAAT [5] \+ Google Translator [68]‡ | I+M | 46.4 | 30.2 | 66.4 | 61.1 | 35.2 | 25.8 | 57.2 | 37.8
SGN [7] \+ Google Translator [68]‡ | I+M | 50.7 | 32.6 | 69.2 | 72.9 | 38.0 | 26.7 | 57.1 | 39.6
Base Model | I+M(+A) | 47.2 | 31.8 | 67.6 | 68.9 | 34.7 | 25.1 | 55.8 | 36.3
Proposed (Section 3.3) | UVC-VI | I+M | 49.6 | 34.7 | 70.3 | 83.4 | 37.2 | 26.8 | 57.7 | 43.7
I+M+A | - | - | - | - | 38.9 | 27.8 | 59.5 | 44.5

##  4 Experiments

In this section, we firstly describe the implementation details, which consist
of model settings and training details. Then we present the evaluation of our
proposed approach on the benchmark datasets.

###  4.1 Implementation Details

####  4.1.1 Model Settings

For the visual embedding
${{V}}=[{{V}}_{\text{T}};{{V}}_{\text{I}};{{V}}_{\text{M}};{{V}}_{\text{A}}]\in\mathbb{R}^{4N\times
d_{h}}$, which includes the textual features ${{V}}_{\text{T}}$, image
features ${{V}}_{\text{I}}$, motion features ${{V}}_{\text{M}}$ and audio
features ${{V}}_{\text{A}}$, given a video, $N=8$ key frames are uniformly
sampled to extract image features $V_{\text{I}}$. Considering both the past
and the future contexts, we take each key frame as the center to generate
corresponding motion features $V_{\text{M}}$ and audio features
$V_{\text{A}}$. For the image features $V_{\text{I}}$, which are good at
illustrating the shapes, the colors and the relationships of the items in the
image, we adopt the Inception-ResNet-V2 [49] pre-trained on the ImageNet [69]
to extract the 1536-D CNN-based image features
$V_{\text{I}}\in\mathbb{R}^{8\times 1536}$. The motion features
$V_{\text{M}}\in\mathbb{R}^{8\times 4096}$ are important for capturing the
actions and temporal interactions, and are usually given by the output of fc6
layer in the C3D network [50] pre-trained on Sports-1M dataset [70], where the
dimension of extracted features is 4096. For the audio features, which are
helpful for distinguishing events, the Bag-of-Audio-Words (BoAW) [51], Fisher
Vector [71] and VGGish [72] are introduced to extract the 256-D, 260-D and
128-D audio features, respectively. We use the concatenation of the three
extracted audio features as the final audio features
$V_{\text{A}}\in\mathbb{R}^{8\times 644}$ in our implementation. For the
textual features $V_{\text{T}}$, we extract textual concepts using the concept
extractor proposed by [14]. At last, the dimension of the extracted textual,
image, motion and audio features, will all be projected to $d_{h}=512$.

####  4.1.2 Training Details

To train the UVC-VI, we first use a video captioning paired dataset to train
the video-to-pivot captioning model as well as the MCE, and then employ a
machine translation dataset to train the pivot-to-target translation model;
Next, we freeze the parameters of the captioning and translation models, and
use the proposed pseudo supervised training and adversarial training (see Eq.
(3.3.1)) to further train the VIM. At the inference phase, we employ Eq.
(LABEL:eq:uvc-vi) to generate the target sentence in an end-to-end inference
manner. To implement the pivot-to-target machine translation model and the
pivot language decoder of the captioning model, we directly employ the
Transformer-BASE model [27], which achieves great success in machine
translation. Accordingly, the model size $d_{h}$ is set to 512. For the MCE,
we adopt the multi-head attention [27], where the number of heads is set to 8.
According to the performance on the validation set, we set the $\alpha=0.1$
and $\beta=10$.

For the discriminators, following [74], the embeddings will be first projected
into 1-D, making the dimension to be 1. Specifically, in the Adversarial
Training stage, we apply the same techniques in [16] to stabilize the
adversarial training of VIM and discriminator D, e.g., we train the VIM by
minimizing: $\mathbb{E}_{V\sim
D_{1}}\left[(\text{D}(\text{VIM}(V))-1)^{2}\right]$ and the D by minimizing:
$\mathbb{E}_{T_{2}\sim D_{2}}\left[(\text{D}(T_{2})-1)^{2}\right]$ \+
$\mathbb{E}_{V\sim D_{1}}\left[\text{D}(\text{VIM}(V))^{2}\right]$. Moreover,
to further stabilize the Adversarial Training procedure, we first use our
proposed Pseudo Supervised Training to pre-train our approach for 50 epochs to
initialize proper parameter weights. Then, we incorporate the Adversarial
Training [16] to further train the model for 100 epochs. We use Adam [75] for
parameter optimization. The learning rate is set to 2e-4. Meanwhile, we also
use momentum of 0.8 and weight decay of 0.999.

For the pivot-to-target translation model, we adopt the Transformer-BASE model
[27, 76]. Following common practice [77, 27], we use the fairseq [76] for our
implementation of pivot-to-target machine translation model.

###  4.2 Comparing Methods

####  4.2.1 Conventional Supervised Systems

We first compare our approach with several conventional supervised methods,
i.e., AF [62], MA-LSTM [19], Two-stream [63], PickNet [64], RecNet [65],
TDConvED [21], POS-CG [66], STAT [67], GRU-EVE [22], SAAT [5], SGN [7] and
MGSA [24]. These supervised methods follow the common encoder-decoder
architecture, trained on large-scale pairs of video and target caption. The
results of these methods are copied from their papers except POS-CG, whose
results are copied from [5].

####  4.2.2 Pipeline Systems

For fair comparison, we investigate the ablative structure of our UVC-VI,
i.e., Base Model. As shown in the left plot of Figure 1), we directly assemble
a video-to-pivot captioning model and a pivot-to-target translation model
together. Base Model shares the same structures with the UVC-VI (see Section
4.1.2), i.e., the pivot-to-target machine translation model and the pivot
language decoder of the video-to-pivot captioning model are implemented by
Transformer-BASE [27]. Meanwhile, the Base Model exploits the image+motion and
image+motion+audio features on MSVD and MSR-VTT datasets, respectively.
Besides, to improve the evaluation significantly, we further implement three
conventional supervised video captioning models equipped with the Google
Translator555Our preliminary experiments show that using Google Translator
could achieve better performance than the Transformer-BASE from fairseq used
in our approach. [68], acquiring three pipeline systems: 1) MA-LSTM [19] \+
Google Translator [68], 2) SAAT [5] \+ Google Translator [68] and 3) SGN [7]
\+ Google Translator [68].

TABLE III: Performance of human evaluation for comparing “UVC-VI” with “Base Model” and “SGN [7] \+ Google Translator [68]”, under three language application scenarios, i.e., French, German and Chinese, where the video-caption pairs in target language are not available. Different from automatic metrics in Table II, the human evaluation results verify the advantage of UVC-VI from the perspective of user experience and prove the effectiveness of UVC-VI in mitigating the visual irrelevancy and disfluency problems in pipeline systems.  Metrics | Languages | Setting: Unpaired Video Captioning
---|---|---
“Base Model” | Tie (%) | “UVC-VI” |  “SGN [7] \+ Google Translator [68]”‡ | Tie (%) | “UVC-VI”
wins (%) | wins (%) | wins (%) | wins (%)
Fluency | French | 26.0 | 45.5 | 28.5 | 29.5 | 36.5 | 34.0
German | 31.0 | 32.0 | 37.0 | 32.5 | 33.0 | 34.5
Chinese | 28.0 | 35.5 | 36.5 | 32.0 | 29.5 | 38.5
Visual Relevance | French | 15.5 | 25.5 | 59.0 | 17.5 | 37.5 | 45.0
German | 24.5 | 26.5 | 49.0 | 26.5 | 27.0 | 46.5
Chinese | 19.5 | 39.0 | 41.5 | 20.5 | 26.5 | 53.0

###  4.3 Automatic Evaluation

####  4.3.1 Datasets

To conduct the automatic evaluation, we verify the effectiveness of our
approach on the benchmark MSVD [17] and MSR-VTT [8] video English captioning
datasets. In particular, for the MSR-VTT dataset, it contains 10,000 video
clips, and each video is paired with 20 annotated sentences. We use the
official splits to report our results [8]. Thus, there are 6513, 497 and 2990
video clips in training, validation and test set, respectively. For MSVD, it
contains 1,970 video clips and roughly 80,000 English sentences. We follow the
split settings in [20], resulting in 1,200, 100 and 670 videos for the
training set, validation set and test set, respectively. It is worth noting
that we focus on the unpaired video captioning, where the video-caption pairs
in target language are not available, the video-caption pairs in the training
sets of MSVD and MSR-VTT are discarded and are not used in our training stage.

In implementation, to train our UVC-VI, we take Chinese as the pivot language
and train the video-to-pivot captioning model on VATEX video Chinese
captioning dataset [54], which contains 41,250 video clips, and each video
clip contains 10 human-annotated Chinese captions. We train the pivot-to-
target translation model on the WMT17 Chinese-English
dataset666http://statmt.org/wmt17/. We follow their official settings to pre-
process the WMT17 and VATEX datasets. In our experiment, we further employ the
target captions from training sets to train the decoder of the pivot-to-target
model by reconstructing the target captions in an auto-encoding pipeline.

####  4.3.2 Metrics

We report the results using the standard automatic evaluation toolkit [78],
which includes the widely-used metrics CIDEr [79], ROUGE-L [80], METEOR [81]
and BLEU [82]. CIDEr, which is based on n-gram matching, incorporating the
consensus of a reference set for an example. ROUGE-L is proposed for automatic
evaluation of the extracted text summarization. METEOR and BLEU are originally
designed for machine translation evaluation. Among them, the CIDEr is
specifically designed for evaluating captioning systems and will be the main
considered metric.

####  4.3.3 Results

As mentioned in Section 4.2.2, in comparable settings, i.e., unpaired video
captioning, to improve the evaluation significantly, we further implement
three representative video captioning models equipped with the Google
Translator [68], resulting in three pipeline systems: “MA-LSTM + Google
Translator”, “SAAT + Google Translator” and “SGN + Google Translator”. The
performance of the three implemented pipeline systems and our baseline method
on the test set of MSVD and MSR-VTT datasets are shown in Table II. As we can
see, both of them are significantly lower than our proposed UVC-VI across most
metrics, which demonstrates the effectiveness of our approach. The reason is
that our UVC-VI can simplify the entire workflow and enable visual information
to directly reach the translation model. As a result, our UVC-VI can address
the visual irrelevancy and disfluency errors in the pipeline systems.

More encouragingly, our UVC-VI (I+M) and UVC-VI (I+M+A) can surpass several
supervised video captioning models, which consist of some recently supervised
models, i.e., Two-stream${}_{\text{(TPAMI2020)}}$ [63],
RecNet${}_{\text{(TPAMI2020)}}$ [65], POS-CG${}_{\text{(ICCV2019)}}$ [66],
STAT${}_{\text{(TMM2020)}}$ [67], GRU-EVE${}_{\text{(CVPR2019)}}$ [22] and
SAAT${}_{\text{(CVPR2020)}}$ [5]. It is worth noting that the POS-
CG${}_{\text{(ICCV2019)}}$ [66], GRU-EVE${}_{\text{(CVPR2019)}}$ [22] and
SAAT${}_{\text{(CVPR2020)}}$ [5] adopt the same features, i.e., Inception-
ResNet-V2 [49] for image features and C3D [50] for motion features, with our
UVC-VI (I+M), the superiority of our approach using same features further
proves our arguments and corroborates the effectiveness of our approach, which
can relax the reliance on the paired dataset for video captioning.

The experimental results show that our approach is able to generate fluent and
desirable video captions without the training on the pairs of video and target
caption, which could have the potential to promote the application of video
captioning for various low-resource language applications.

###  4.4 Human Evaluation

####  4.4.1 Datasets

In this section, we further conduct the experiments on three language
application scenarios, i.e., French, German and Chinese. In implementation, we
take English as the pivot language and train the video-to-pivot captioning
model on the MSR-VTT video English captioning dataset [8]. For different
language application scenarios, we train the pivot-to-target translation model
on WMT translation datasets. Specifically, we adopt the WMT14 English-French
dataset for French, WMT14 English-German dataset for
German777http://statmt.org/wmt14/, and WMT17 English-Chinese dataset for
Chinese888http://statmt.org/wmt17/. Specifically, WMT14 English-German, WMT14
English-French and WMT17 English-Chinese consist of 4.5M, 36M and 24M sentence
pairs, respectively.

TABLE IV: Quantitative Analysis of our approach which is performed on the test set of MSR-VTT video captioning dataset [8] under the unpaired setting. The Bold numbers denote the best results across all methods.  Methods | ${L}_{\text{pseudo}}$ | ${L}_{\text{adv}}$ | ${L}^{\prime}_{\text{adv}}$ | ${L}_{\text{cyc}}$ | BLEU-4 | METEOR | CIDEr
---|---|---|---|---|---|---|---
Setting: Unpaired Video Captioning
Base Model |  |  |  |  | 34.7 | 25.1 | 36.3
w/ VIM | $\surd$ |  |  |  | 36.8 | 26.6 | 40.7
w/ VIM | $\surd$ | $\surd$ |  |  | 38.2 | 27.2 | 42.0
w/ VIM | $\surd$ | $\surd$ | $\surd$ | $\surd$ | 38.5 | 27.4 | 42.3
|  w/ VIM + MCE
---
(UVC-VI)
$\surd$ | $\surd$ | $\surd$ | $\surd$ | 38.9 | 27.8 | 44.5
Figure 2: Visualization of (a) the raw visual embedding $V$ and (b) projected
visual embedding $\text{VIM}(V)$ in 2D space by t-SNE [83] (see Eq. (4)). For
comparison, we show the textual embedding $T$ in the target language domain.
We plot the scatter diagrams for 1,000 samples. As we can see, our VIM can
align the latent space of $V$ and $T$ effectively, i.e., successfully project
the visual embedding of source video domain into the target language domain.

####  4.4.2 Metrics

Due to the lack of video-caption pairs in the target language domain, to prove
our arguments and evaluate the effectiveness of our approach in mitigating the
visual irrelevancy and disfluency problems on the target language, for each
target language, we randomly selected 200 videos from the MSR-VTT dataset, and
recruit 10 annotators with sufficient language skills (5 proficient in French,
4 in German and 1 in Chinese), to conduct the human evaluation. Each annotator
is required to evaluate the perceptual quality, including the visual relevance
and fluency, of the generated captions in the target language and compare the
performance of UVC-VI with the baselines. They have no idea which model the
captions are produced by. The results are calculated by the picking
percentages (%).

####  4.4.3 Results

To obtain the results of human evaluation, we select the “Base Model” and the
“SGN [7] \+ Google Translator [68]”. The results in Table III show that the
captions generated by our UVC-VI are more favored by annotators than the two
selected baseline approaches on all metrics across all language application
scenarios. It verifies our arguments and proves the effectiveness of our
approach in mitigating the visual irrelevancy and disfluency problems in
pipeline systems. In particular, our approach achieves better improvements in
terms of the visual relevance than the fluency, which indicates that the
visual relevance problem is the more critical in the pipeline systems. The
reason is that, in the pipeline systems, the visual information cannot reach
the translation model at all, which further proves the effectiveness of our
approach.

From the results of automatic and human evaluations, we can see that our UVC-
VI can well-perform the unpaired video captioning, regardless of the
downstream language application scenarios. Especially, our approach can
outperform several supervised video captioning models. As a result, since our
model dose not rely on any video-target paired data, our work could improve
the practicality of video captioning in real-world applications, such as video
retrieval [1] and visually impaired people aiding [3], especially for non-
English languages applications.

##  5 Analysis

In this section, to better understand our methods, we conduct the analysis to
investigate the contribution of each component in our proposed model, i.e.,
Visual Injection Module (VIM) and Multimodal Collaborative Encoder (MCE).

###  5.1 Effect of Visual Injection Module (VIM)

In this section, we analyze the effect of VIM, including the proposed pseudo
supervised training and adversarial training. From Table IV, we can see that
the ${L}_{\text{pseudo}}$ provides a solid basis for VIM to inject the visual
information into language domain. Since adversarial training can further make
use of the pivot-target paired dataset, the introduction of ${L}_{\text{adv}}$
further improves the performance. By introducing the cycle strategy (i.e.,
${L}^{\prime}_{\text{adv}}$ \+ ${L}_{\text{cyc}}$), the performance improves
as well. The reason may be that the cycle strategy enforces ${L}_{\text{adv}}$
and ${L}^{\prime}_{\text{adv}}$ to collaborate with each other [16].

To better understand our approach, we further report the breakdown of SPICE
F-scores [84] in Table V. As we can see, comparing “w/ VIM” and “Base Model”
in SPICE F-score, the former significantly achieves better Attributes and
Color scores, this is due to the enriched visual details contained in the
visual embedding, which are injected into target language domain by our VIM.

To further verify the VIM indeed aligns the source visual and target language
domains, following [43] and [46], we adopt the t-SNE [83] to visualize the
features in Figure 2, which shows that the VIM actually injects the visual
information into the latent space of target language domain, resulting in a
blended distribution.

TABLE V: We further report the SPICE F-scores [84] on the test set of MSR-VTT dataset for a better understanding of the differences of the target captions generated by different methods. Methods | SPICE
---|---
All | Objects | Attributes | Relations | Color
Setting: Unpaired Video Captioning
Base Model | 6.1 | 13.0 | 1.9 | 2.3 | 0.1
w/ VIM | 6.5 | 13.3 | 2.4 | 2.4 | 0.6
|  w/ VIM + MCE
---
(UVC-VI)
6.7 | 13.9 | 2.5 | 2.4 | 0.7
TABLE VI: Analysis of the Multimodal Collaborative Encoder (MCE). As a pluggable video encoder, MCE can be easily integrated into existing video captioning models. Therefore, we further equip our MCE with two representative models, i.e., Attention-based LSTM model (Attention-LSTM) [85] and Transformer model [27]. We perform the analysis on the MSVD and MSR-VTT datasets under the paired setting, i.e., conventional supervised video captioning. Symbol ‡ is defined similarly. The Bold numbers denote the best results across all methods. As we can see, our MCE can successfully boost the baselines, with the most improvement up to relatively 7% and 11% for MSVD and MSR-VTT in terms of CIDEr, which is specifically designed for evaluating captioning systems, respectively. More encouragingly, our Transformer w/ MCE model surpasses the previous state-of-the-art models on the MSVD and MSR-VTT, with relatively 4% and 7% margins and in terms of the CIDEr, respectively. Methods |  Dataset: MSVD [17] |  Dataset: MSR-VTT [8]
---|---|---
Features | BLEU-4 | METEOR | ROUGE-L | CIDEr | Features | BLEU-4 | METEOR | ROUGE-L | CIDEr
Setting: Conventional Supervised Video Captioning
Two-stream${}_{\text{(TPAMI2020)}}$ [63] | I+M | 54.3 | 33.5 | - | 72.8 | I+M | 39.7 | 27.0 | - | 42.1
RecNet${}_{\text{(TPAMI2020)}}$ [65] | I | 52.3 | 34.1 | 69.8 | 80.3 | I | 39.1 | 26.6 | 59.3 | 42.7
STAT${}_{\text{(TMM2020)}}$ [67] | I+M | 52.0 | 33.3 | - | 73.8 | I+M | 39.3 | 27.1 | - | 43.8
GRU-EVE${}_{\text{(CVPR2019)}}$ [22] | I+M | 47.9 | 35.0 | 71.5 | 78.1 | I+M | 38.3 | 28.4 | 60.7 | 48.1
SibNet${}_{\text{(TPAMI2020)}}$ [86] | I | 55.7 | 35.5 | 72.6 | 88.8 | I | 41.2 | 27.8 | 60.8 | 48.6
SAAT${}_{\text{(CVPR2020)}}$ [5] | I+M | 46.5 | 33.5 | 69.4 | 81.0 | I+M | 39.9 | 27.7 | 61.2 | 51.0
SGN${}_{\text{(AAAI2021)}}$ [7] | I+M | 52.8 | 35.5 | 72.9 | 94.3 | I+M | 40.8 | 28.3 | 60.8 | 49.5
MGSA${}_{\text{(AAAI2019)}}$ [24] | I+M | 53.4 | 35.0 | - | 86.7 | I+M+A | 45.4 | 28.6 | - | 50.1
Attention-LSTM [85]‡ | I+M | 51.6 | 35.4 | 71.8 | 88.5 | I+M+A | 44.4 | 29.7 | 62.1 | 48.9
w/ MCE | I+M | 52.8 | 35.7 | 72.6 | 93.4 | I+M+A | 44.9 | 30.0 | 63.0 | 52.6
Transformer [27] ‡ | I+M | 49.8 | 35.1 | 72.2 | 91.2 | I+M+A | 43.2 | 28.5 | 61.8 | 49.3
w/ MCE | I+M | 55.1 | 36.2 | 73.3 | 97.7 | I+M+A | 45.9 | 30.7 | 64.3 | 54.6

###  5.2 Effect of Multimodal Collaborative Encoder (MCE)

Table IV shows that the MCE can further promote the performance, which verify
our arguments and prove the MCE can smooth the cross-modality projection of
VIM. To better understand our MCE, we also report the breakdown of SPICE
F-scores [84] in Table V. It shows that the MCE can significantly improve the
Object score. It proves that textual-enriched visual embedding produced by MCE
can efficiently integrate multimodal features according to each object in
textual concepts.

As a video encoder, MCE can be easily integrated into existing supervised
video captioning models. Therefore, we further equip our MCE with two
representative models, i.e., Attention-based LSTM model (Attention-LSTM) [85]
and Transformer model [27]. We evaluate the performance of these models on the
MSVD and MSR-VTT video captioning datasets, under the paired setting, i.e.,
conventional supervised video captioning. Table VI shows that MCE can
successfully boost the baselines, with the most significant improvement up to
relatively 7% and 11% for MSVD and MSR-VTT in terms of CIDEr, respectively.
Moreover, our approach surpasses the previous state-of-the-art models on the
MSVD and MSR-VTT datasets, with relatively 4% and 7% margins on the CIDEr
scores, respectively. The improvements demonstrate the effectiveness and
generalization ability of our proposed MCE.

###  5.3 Examples

In this section, we provide some examples under different language application
scenarios, i.e., English, French, German and Chinese, to show the advantages
of UVC-VI from the perspective of user experience. As mentioned above, in case
of English as the target language, we take Chinese as pivot language; for the
other three languages, we take English as pivot language. As shown in Figure
3, the generated captions by UVC-VI are more fluent than the pipeline system
(i.e., Base Model). Moreover, the proposed UVC-VI includes more objects, such
as beach and jacket in the $1^{\rm st}$ and $2^{\rm nd}$ examples,
respectively, and is good at portraying the attribute, e.g., black in the
$2^{\rm nd}$ example, and describing the details, e.g., boys and girls in the
$1^{\rm st}$ example. The reason is that the introduction of our Visual
Injection Module (VIM) in UVC-VI can enable visual information to directly
reach the target language decoder which lowers the chances of error occurrence
and amplification.

Figure 3: The examples of video captions generated by the Base Model and UVC-
VI for different languages, i.e., French, German, Chinese and English, under
the unpaired setting, where the video-caption pairs in target language are not
available. As we can see, without the training on the pairs of video and
target caption, our approach can generate fluent and desirable video captions
for different languages.

###  5.4 Extension to Unpaired Image Captioning

Considering the fact that the unpaired image captioning has been explored in
the literature [42, 43, 44, 46, 47, 45] (see Section 2.3), we further extend
our UVC-VI to the unpaired image captioning to verify the effectiveness of our
approach. Specifically, following the existing approaches, we also perform the
experiments on the MSCOCO image English captioning dataset [78].

####  5.4.1 Metrics and Datasets

Following common practice [43, 46, 47], we test the performance of image
captioning model with captioning evaluation toolkit [78], which reports the
widely-used automatic evaluation metrics CIDEr [79], ROUGE-L [80], METEOR [81]
and BLEU [82]. For fair comparisons, we conduct the evaluation on the popular
MSCOCO image English captioning dataset [78]. Specifically, we use the
publicly-available splits in [88] for evaluation. There are 5,000 images each
in validation set and test set. During training, following [42], we take
Chinese as the pivot language, learn the image-to-pivot captioning model on
ICC image Chinese captioning dataset [89] and train the pivot-to-target
translation model on the WMT17 Chinese-English dataset.

####  5.4.2 Settings

To extend our UVC-VI to the unpaired image captioning task, we keep the inner
structure of the UVC-VI untouched and directly remove the motion and audio
features which are only available in videos, i.e., we use the set
${V}=[{V}_{\text{T}};{V}_{\text{I}}]$ to denote the visual embedding of image.

For fair comparisons, for the image features ${{V}}_{\text{I}}$, we use the
RCNN-based image features provided by [85]. The dimension of the extracted
RCNN-based image features will all be projected to 512, which is equal to the
dimension of our captioning model $d_{h}=512$. For the textual features
${{V}}_{\text{T}}$, we extract textual concepts using the textual concept
extractor proposed by [90]. The embedding vectors (512-D) of these extracted
textual concepts are then taken as the ${{V}}_{\text{T}}$.

TABLE VII: Performance of our proposed UVC-VI on the test split of MSCOCO image captioning dataset [78] under the unpaired setting, i.e., unpaired image captioning, where the image-caption pairs in target language are not available. The Bold numbers denote the best results across all approaches. As we can see, our UVC-VI achieves the best performance among the existing approaches for unpaired image captioning across all metrics.  Methods |  Dataset: MSCOCO [78]
---|---
BLEU-4 | METEOR | ROUGE-L | CIDEr
Setting: Unpaired Image Captioning
Language Pivoting${}_{\text{(ECCV2018)}}$ [42] | 5.4 | 13.2 | - | 17.7
Adv.+Recon.${}_{\text{(CVPR2019)}}$ [44] | 18.6 | 17.9 | 43.1 | 54.9
Shared Embeddings${}_{\text{(ICCV2019)}}$ [46] | 19.3 | 20.2 | 45.0 | 61.8
Graph-Align${}_{\text{(ICCV2019)}}$ [43] | 21.5 | 20.9 | 47.2 | 69.5
USGAE${}_{\text{(TPAMI2020)}}$ [47] | 17.1 | 19.1 | 43.8 | 55.1
Base Model | 16.7 | 17.8 | 40.8 | 51.7
w/ VIM | 20.5 | 19.9 | 45.7 | 64.1
w/ VIM + MCE (UVC-VI) | 22.0 | 21.4 | 47.6 | 72.3
Figure 4: The examples of image captions generated by the Base Model and our
UVC-VI for different languages, i.e., French, German, Chinese and English,
under the unpaired setting. As we can see, our UVC-VI can be extended to
unpaired image captioning, where the image-caption pairs in target language
are not available, to generate desirable and fluent image captions.

####  5.4.3 Implementation Details

Since the image captioning only accept the image features ${{V}}_{\text{T}}$
and textual concepts ${{V}}_{\text{T}}$, we should modify the Multimodal
Collaborative Encoder (MCE), which is original designed for video captioning,
to extend to the image captioning, while the rest of model structure and
implementation details are the same as as the UVC-VI used for unpaired video
captioning. In implementations, given the multimodal features
${V}=[{V}_{\text{T}};{V}_{\text{I}}]$, the Multimodal Collaborative Encoder
(MCE) for image captioning is defined as:

$V^{\prime}=\text{MCE}(V)={\gamma}\odot\text{Attention}\left({{V}}_{\text{T}},{{V}}_{\text{I}}\right),$
(13)

where ${V}^{\prime}$ denotes the text-enriched visual embedding and the
${\gamma}$ and $\text{Attention}\left(\cdot,\cdot\right)$ are defined as
follows:

$\displaystyle{\gamma}=$
$\displaystyle\sigma\left(\left[{{V}}_{\text{T}};\text{Attention}\left({{V}}_{\text{T}},{{V}}_{\text{I}}\right)\right]W_{\gamma}\right),$
(14) Attention
$\displaystyle(x,y)=\text{softmax}\left(xW_{\text{q}}\left(yW_{\text{k}}\right)^{\top}\right)yW_{\text{v}}.$

where $\odot$ and $\sigma$ denote the element-wise multiplication and the
sigmoid activation, respectively; $[\cdot;\cdot]$ denotes concatenation
operation. $W_{\gamma}\in\mathbb{R}^{2d_{h}\times d_{h}}$,
$W_{\text{q}}\in\mathbb{R}^{d_{h}\times d_{h}}$,
$W_{\text{k}}\in\mathbb{R}^{d_{h}\times d_{h}}$ and
$W_{\text{v}}\in\mathbb{R}^{d_{h}\times d_{h}}$ are learnable parameters.

####  5.4.4 Results

For the existing models for unpaired image captioning [42, 47], where the
image-caption pairs in target language are not available, we directly report
the results from the original papers. The quantitative results are illustrated
in the Table VII, showing that both the MCE and VIM can boost the baseline.
Furthermore, the UVC-VI achieves the best performance among the existing
methods across all metrics. It further proves the effectiveness of our
approach for unpaired visual captioning task.

We further show the image captions generated by the method Base Model and the
method UVC-VI in Figure 4 to intuitively analyze the differences of the
methods. As we can see, the proposed UVC-VI includes more objects, such as
“black suit” in the $2^{\rm nd}$ example, and is good at portraying the
attribute and color, such as “glass table” and “blue umbrella” $1^{\rm st}$
and $2^{\rm nd}$ examples, respectively. Besides, the generated captions by
UVC-VI are more fluent, because the introduction of our Visual Injection
Module in UVC-VI enables visual information to directly reach the target
language decoder and lowers the chances of error generation and amplification.
These examples further prove our arguments and the effectiveness of our
approach.

##  6 Conclusions

In this paper, we make the first attempt to conduct unpaired video captioning
under various low-resource language application scenarios, e.g., French,
German and Chinese, in which the video-caption pairs are not available. To
this end, we present the Unpaired Video Captioning with Visual Injection
system (UVC-VI). Different from the Pipeline System, which first utilizes
video-to-pivot captioning model to generate captions in pivot language and
then utilizes pivot-to-target translation model to translate the pivot
captions to the target language, 1) UVC-VI can inject the source visual
information into the target language domain by aligning the source visual and
target language domains; 2) UVC-VI directly connects the encoder of the video-
to-pivot captioning model and the decoder of the pivot-to-target translation
model, the pivot caption is no longer generated, which allows end-to-end
inference. The experiments on two benchmark datasets prove the effectiveness
of UVC-VI, which outperforms both pipeline systems and several supervised
systems. More encouragingly, the proposed MCE in UVC-VI can be incorporated
into existing video captioning models to boost their performances, up to 4%
and 7% relative margin in terms of CIDEr scores with current state-of-the-art
models on the MSVD and MSR-VTT datasets, respectively. Moreover, we extend our
approach to the unpaired image captioning. The experiments show that UVC-VI
obtains positive experimental results.

It can be interesting to apply UVC-VI to improve other pipeline systems. The
tasks include broad natural language generation tasks such as machine
listening comprehension and spoken question answering, as well as those in
computer vision such as medical image classification.

## Acknowledgments

This paper was partially supported by National Natural Science Foundation of
China (NSFC 62176008). Special acknowledgements are given to AOTO-PKUSZ Joint
Research Center for its support. More importantly, we would like to sincerely
thank all the anonymous reviewers and editors for their constructive comments
and suggestions that substantially improved this paper. Xu Sun, Yuexian Zou
and Xian Wu are the corresponding authors of this paper.

## References

  * [1] Y. Yu, H. Ko, J. Choi, and G. Kim, “End-to-end concept word detection for video captioning, retrieval, and question answering,” in _CVPR_ , 2017\.
  * [2] A. Das, S. Kottur, K. Gupta, A. Singh, D. Yadav, J. M. F. Moura, D. Parikh, and D. Batra, “Visual dialog,” in _CVPR_ , 2017.
  * [3] V. Voykinska, S. Azenkot, S. Wu, and G. Leshed, “How blind people interact with visual content on social networking services,” in _CSCW_ , 2016.
  * [4] S. Venugopalan, H. Xu, J. Donahue, M. Rohrbach, R. J. Mooney, and K. Saenko, “Translating videos to natural language using deep recurrent neural networks,” in _NAACL-HLT_ , 2015.
  * [5] Q. Zheng, C. Wang, and D. Tao, “Syntax-aware action targeting for video captioning,” in _CVPR_ , 2020.
  * [6] F. Liu, X. Ren, X. Wu, B. Yang, S. Ge, and X. Sun, “O2NA: an object-oriented non-autoregressive approach for controllable video captioning,” in _ACL/IJCNLP_ , 2021.
  * [7] H. Ryu, S. Kang, H. Kang, and C. D. Yoo, “Semantic grouping network for video captioning,” in _AAAI_ , 2021.
  * [8] J. Xu, T. Mei, T. Yao, and Y. Rui, “MSR-VTT: A large video description dataset for bridging video and language,” in _CVPR_ , 2016.
  * [9] B. Tseng, S. Shen, H. Lee, and L. Lee, “Towards machine comprehension of spoken content: Initial TOEFL listening comprehension test by machine,” in _INTERSPEECH_ , 2016.
  * [10] S. Shiang, H. Lee, and L. Lee, “Spoken question answering using tree-structured conditional random fields and two-layer random walk,” in _INTERSPEECH_ , 2014.
  * [11] H. Wu and H. Wang, “Pivot language approach for phrase-based statistical machine translation,” in _ACL_ , 2007.
  * [12] S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer, “Scheduled sampling for sequence prediction with recurrent neural networks,” in _NIPS_ , 2015.
  * [13] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: Lessons learned from the 2015 MSCOCO image captioning challenge,” _PAMI_ , 2017\.
  * [14] Y. Pan, T. Yao, H. Li, and T. Mei, “Video captioning with transferred semantic attributes,” in _CVPR_ , 2017.
  * [15] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville, and Y. Bengio, “Generative adversarial nets,” in _NIPS_ , 2014.
  * [16] J. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in _ICCV_ , 2017\.
  * [17] S. Guadarrama, N. Krishnamoorthy, G. Malkarnenkar, S. Venugopalan, R. J. Mooney, T. Darrell, and K. Saenko, “Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition,” in _ICCV_ , 2013.
  * [18] F. Liu, S. Ge, and X. Wu, “Competence-based multimodal curriculum learning for medical report generation,” in _ACL/IJCNLP_ , 2021.
  * [19] J. Xu, T. Yao, Y. Zhang, and T. Mei, “Learning multimodal attention LSTM networks for video captioning,” in _ACM MM_ , 2017.
  * [20] W. Pei, J. Zhang, X. Wang, L. Ke, X. Shen, and Y. Tai, “Memory-attended recurrent network for video captioning,” in _CVPR_ , 2019.
  * [21] J. Chen, Y. Pan, Y. Li, T. Yao, H. Chao, and T. Mei, “Temporal deformable convolutional encoder-decoder networks for video captioning,” in _AAAI_ , 2019.
  * [22] N. Aafaq, N. Akhtar, W. Liu, S. Z. Gilani, and A. Mian, “Spatio-temporal dynamics and semantic attribute enriched visual encoding for video captioning,” in _CVPR_ , 2019.
  * [23] Z. Zhang, Y. Shi, C. Yuan, B. Li, P. Wang, W. Hu, and Z. Zha, “Object relational graph with teacher-recommended learning for video captioning,” in _CVPR_ , 2020.
  * [24] S. Chen and Y. Jiang, “Motion guided spatial attention for video captioning,” in _AAAI_ , 2019.
  * [25] F. Liu, X. Ren, X. Wu, S. Ge, W. Fan, Y. Zou, and X. Sun, “Prophet attention: Predicting attention with future attention,” in _NeurIPS_ , 2020.
  * [26] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural Computation_ , 1997.
  * [27] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in _NIPS_ , 2017.
  * [28] L. Zhou, Y. Zhou, J. J. Corso, R. Socher, and C. Xiong, “End-to-end dense video captioning with masked transformer,” in _CVPR_ , 2018.
  * [29] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in _ICLR_ , 2015.
  * [30] Y. Pan, T. Mei, T. Yao, H. Li, and Y. Rui, “Jointly modeling embedding and translation to bridge video and language,” in _CVPR_ , 2016.
  * [31] D. Yu and J. Li, “Recent progresses in deep learning based acoustic models,” _IEEE CAA J. Autom. Sinica_ , 2017.
  * [32] G. Lai, Q. Xie, H. Liu, Y. Yang, and E. H. Hovy, “RACE: large-scale reading comprehension dataset from examinations,” in _EMNLP_ , 2017.
  * [33] K. C. L. Wong, T. F. Syeda-Mahmood, and M. Moradi, “Building medical image classifiers with very limited data using segmentation networks,” _Medical Image Anal._ , 2018.
  * [34] R. Mehta and J. Sivaswamy, “M-net: A convolutional neural network for deep brain structure segmentation,” in _ISBI_ , 2017.
  * [35] M. Utiyama and H. Isahara, “A comparison of pivot methods for phrase-based statistical machine translation,” in _NAACL-HLT_ , 2007.
  * [36] S. T. Zahabi, S. Bakhshaei, and S. Khadivi, “Using context vectors in improving a machine translation system with bridge language,” in _ACL_ , 2013\.
  * [37] Q. Wu, C. Shen, L. Liu, A. R. Dick, and A. van den Hengel, “What value do explicit high level concepts have in vision to language problems?” in _CVPR_ , 2016.
  * [38] C. Lee, H. Lee, S. Wu, C. Liu, W. Fang, J. Hsu, and B. Tseng, “Machine comprehension of spoken content: TOEFL listening test and spoken squad,” _IEEE ACM Trans. Audio Speech Lang. Process._ , 2019.
  * [39] F. Liu, X. Wu, S. Ge, W. Fan, and Y. Zou, “Federated learning for vision-and-language grounding problems,” in _AAAI_ , 2020.
  * [40] W. Lan, X. Li, and J. Dong, “Fluency-guided cross-lingual image captioning,” in _ACM MM_ , 2017.
  * [41] Y. Song, S. Chen, Y. Zhao, and Q. Jin, “Unpaired cross-lingual image caption generation with self-supervised rewards,” in _ACM MM_ , 2019.
  * [42] J. Gu, S. R. Joty, J. Cai, and G. Wang, “Unpaired image captioning by language pivoting,” in _ECCV_ , 2018.
  * [43] J. Gu, S. R. Joty, J. Cai, H. Zhao, X. Yang, and G. Wang, “Unpaired image captioning via scene graph alignments,” in _ICCV_ , 2019.
  * [44] Y. Feng, L. Ma, W. Liu, and J. Luo, “Unsupervised image captioning,” in _CVPR_ , 2019.
  * [45] F. Liu, M. Gao, T. Zhang, and Y. Zou, “Exploring semantic relationships for image captioning without parallel data,” in _ICDM_ , 2019.
  * [46] I. Laina, C. Rupprecht, and N. Navab, “Towards unsupervised image captioning with shared multimodal embeddings,” in _ICCV_ , 2019.
  * [47] X. Yang, H. Zhang, and J. Cai, “Auto-encoding and distilling scene graphs for image captioning,” _IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI)_ , 2020\.
  * [48] F. Liu, X. Wu, S. Ge, W. Fan, and Y. Zou, “Exploring and distilling posterior and prior knowledge for radiology report generation,” in _CVPR_ , 2021\.
  * [49] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi, “Inception-v4, inception-resnet and the impact of residual connections on learning,” in _AAAI_ , 2017.
  * [50] D. Tran, L. D. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning spatiotemporal features with 3d convolutional networks,” in _ICCV_ , 2015\.
  * [51] S. Pancoast and M. Akbacak, “Softening quantization in bag-of-audio-words,” in _ICASSP_ , 2014.
  * [52] L. Zhou, C. Xu, and J. J. Corso, “Towards automatic learning of procedures from web instructional videos,” in _AAAI_ , 2018.
  * [53] R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. C. Niebles, “Dense-captioning events in videos,” in _ICCV_ , 2017.
  * [54] X. Wang, J. Wu, J. Chen, L. Li, Y. Wang, and W. Y. Wang, “VATEX: A large-scale, high-quality multilingual dataset for video-and-language research,” in _ICCV_ , 2019.
  * [55] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning internal representations by error propagation,” California Univ San Diego La Jolla Inst for Cognitive Science, Tech. Rep., 1985.
  * [56] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y. Bengio, “Show, attend and tell: Neural image caption generation with visual attention,” in _ICML_ , 2015.
  * [57] F. Liu, Y. Liu, X. Ren, X. He, and X. Sun, “Aligning visual regions and textual concepts for semantic-grounded image representations,” in _NeurIPS_ , 2019.
  * [58] M. Cornia, M. Stefanini, L. Baraldi, and R. Cucchiara, “Meshed-memory transformer for image captioning,” in _CVPR_ , 2020.
  * [59] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” _JMLR_ , 2014.
  * [60] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _CVPR_ , 2016.
  * [61] L. J. Ba, R. Kiros, and G. E. Hinton, “Layer normalization,” _arXiv preprint arXiv:1607.06450_ , 2016.
  * [62] C. Hori, T. Hori, T. Lee, Z. Zhang, B. Harsham, J. R. Hershey, T. K. Marks, and K. Sumi, “Attention-based multimodal fusion for video description,” in _ICCV_ , 2017.
  * [63] L. Gao, X. Li, J. Song, and H. T. Shen, “Hierarchical lstms with adaptive attention for visual captioning,” _IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI)_ , 2020.
  * [64] Y. Chen, S. Wang, W. Zhang, and Q. Huang, “Less is more: Picking informative frames for video captioning,” in _ECCV_ , 2018.
  * [65] W. Zhang, B. Wang, L. Ma, and W. Liu, “Reconstruct and represent video contents for captioning via reinforcement learning,” _IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI)_ , 2020.
  * [66] B. Wang, L. Ma, W. Zhang, W. Jiang, J. Wang, and W. Liu, “Controllable video captioning with POS sequence guidance based on gated fusion network,” in _ICCV_ , 2019.
  * [67] C. Yan, Y. Tu, X. Wang, Y. Zhang, X. Hao, Y. Zhang, and Q. Dai, “STAT: spatial-temporal attention mechanism for video captioning,” _IEEE Trans. Multim._ , 2020.
  * [68] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu, L. Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, W. Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, O. Vinyals, G. Corrado, M. Hughes, and J. Dean, “Google’s neural machine translation system: Bridging the gap between human and machine translation,” _arXiv preprint arXiv:1609.08144_ , 2016.
  * [69] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and F. Li, “Imagenet: A large-scale hierarchical image database,” in _CVPR_ , 2009.
  * [70] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and F. Li, “Large-scale video classification with convolutional neural networks,” in _CVPR_ , 2014.
  * [71] J. Sánchez, F. Perronnin, T. Mensink, and J. J. Verbeek, “Image classification with the fisher vector: Theory and practice,” _IJCV_ , 2013\.
  * [72] S. Hershey, S. Chaudhuri, D. P. W. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, M. Slaney, R. J. Weiss, and K. W. Wilson, “CNN architectures for large-scale audio classification,” in _ICASSP_ , 2017.
  * [73] F. Liu, C. Yin, X. Wu, S. Ge, P. Zhang, and X. Sun, “Contrastive attention for automatic chest x-ray report generation,” in _ACL/IJCNLP (Findings)_ , 2021\.
  * [74] L. Yu, W. Zhang, J. Wang, and Y. Yu, “Seqgan: Sequence generative adversarial nets with policy gradient,” in _AAAI_ , 2017.
  * [75] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _ICLR_ , 2014.
  * [76] M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli, “fairseq: A fast, extensible toolkit for sequence modeling,” in _NAACL-HLT (Demonstrations)_ , 2019.
  * [77] F. Wu, A. Fan, A. Baevski, Y. N. Dauphin, and M. Auli, “Pay less attention with lightweight and dynamic convolutions,” in _ICLR_ , 2019.
  * [78] X. Chen, H. Fang, T. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick, “Microsoft COCO captions: Data collection and evaluation server,” _arXiv preprint arXiv:1504.00325_ , 2015.
  * [79] R. Vedantam, C. L. Zitnick, and D. Parikh, “Cider: Consensus-based image description evaluation,” in _CVPR_ , 2015.
  * [80] C.-Y. Lin, “ROUGE: A package for automatic evaluation of summaries,” in _ACL_ , 2004.
  * [81] S. Banerjee and A. Lavie, “METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,” in _ACL_ , 2005.
  * [82] K. Papineni, S. Roukos, T. Ward, and W. Zhu, “BLEU: a Method for automatic evaluation of machine translation,” in _ACL_ , 2002.
  * [83] L. van der Maaten and G. Hinton, “Visualizing data using t-sne,” _JMLR_ , 2008\.
  * [84] P. Anderson, B. Fernando, M. Johnson, and S. Gould, “SPICE: Semantic propositional image caption evaluation,” in _ECCV_ , 2016.
  * [85] P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang, “Bottom-up and top-down attention for image captioning and VQA,” in _CVPR_ , 2018.
  * [86] S. Liu, Z. Ren, and J. Yuan, “Sibnet: Sibling convolutional encoder for video captioning,” _IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI)_ , 2020.
  * [87] D. You, F. Liu, S. Ge, X. Xie, J. Zhang, and X. Wu, “Aligntransformer: Hierarchical alignment of visual regions and disease tags for medical report generation,” in _MICCAI_ , 2021.
  * [88] A. Karpathy and F. Li, “Deep visual-semantic alignments for generating image descriptions,” in _CVPR_ , 2015.
  * [89] J. Wu, H. Zheng, B. Zhao, Y. Li, B. Yan, R. Liang, W. Wang, S. Zhou, G. Lin, Y. Fu, Y. Wang, and Y. Wang, “Large-scale datasets for going deeper in image understanding,” in _ICME_ , 2019.
  * [90] H. Fang, S. Gupta, F. N. Iandola, R. K. Srivastava, L. Deng, P. Dollár, J. Gao, X. He, M. Mitchell, J. C. Platt, C. L. Zitnick, and G. Zweig, “From captions to visual concepts and back,” in _CVPR_ , 2015.

|  Fenglin Liu is a PhD student at the University of Oxford. His research
interests include Natural Language Processing (NLP), especially vision-and-
language, Machine Learning, and their applications to clinical, i.e., Clinical
NLP. He has published papers at top-tier journals and conferences, e.g.,
TPAMI, NeurIPS, CVPR, ACL, EMNLP, NAACL. He has served as a senior program
committee member for IJCAI and was awarded as the Distinguished/Outstanding
Reviewer of CVPR, AAAI, and IJCAI.
---|---
|  Xian Wu is now a Principal Researcher in Tencent. Before joining Tencent,
he worked as a Senior Scientist Manager and a Staff Researcher in Microsoft
and IBM Research. Xian Wu received his PhD degree from Shanghai Jiao Tong
University. His research interests includes Medical AI, Natural Language
Processing and Multi-Modal modeling. Xian Wu has published papers in CVPR,
NeurIPS, ACL, WWW, AAAI, IJCAI etc. He also served as PC member of TKDE, TKDD,
TOIS, TIST, CVPR, ICCV, AAAI etc.
---|---
|  Chenyu You is a Ph.D. student at Yale University. He received the B.S. and
M.Sc. degree from Rensselaer Polytechnic Institute and Stanford University,
respectively. His research interests are broadly in the area of machine
learning theory and algorithms intersecting the fields of computer & medical
vision, natural language processing, and signal processing. He has been
awarded as the Outstanding/Distinguished Reviewer of CVPR, MICCAI, IEEE
Transactions on Medical Imaging (TMI), and Medical Physics.
---|---
|  Shen Ge Shen Ge is a Senior Researcher in Tencent. His research interests
include Natural Language Processing, Deep Learning and Medical AI-based
systems. Previously, he has published several works in AAAI, IJCAI, NeurIPS,
CVPR, ACMMM, TKDE and KAIS, etc. Prior to Tencent, he worked in DAMO Academy
of Alibaba and Microsoft. He received his PhD’s degree from the University of
Hong Kong, in 2012.
---|---
|  Yuexian Zou received her B.Sc. degree from the University of Electronic
Science and Technology in 1985 and Ph.D. degree from the University of Hong
Kong in 2001, respectively. She is currently a Full Professor with Peking
University and the Director of the Advanced Data and Signal Processing
Laboratory in Peking University Shenzhen Graduate School. She was a recipient
of the award Leading Figure for Science and Technology by Shenzhen Municipal
Government in 2009 and now is the adjunct professor in Peng Cheng Laboratory.
She also serves as the Deputy Director of Shenzhen Association of Artificial
Intelligence (SAAI). Since 2010, she has been actively involved in teaching
and research on machine learning and its applications in video and audio
analysis. She conducted more than 20 research projects including NSFC and 863
projects. She has published more than 230 academic papers in famous journals
and flagship conferences, issued eight invention patents, and two of them have
been transferred to a company. She conducts several courses for graduate
students, such as machine learning and pattern recognition, digital signal
processing, and array signal processing. Her research interests mainly in
machine learning for signal processing and scene understanding.
---|---
|  Xu Sun is an associate professor in the Department of Computer Science,
Peking University. His research focuses on natural language processing and
machine learning, especially on structured learning for language, and natural
language generation. He has been senior area chair or area chair of ACL,
EMNLP, etc. He received COLING Best Paper Award in 2018.
---|---