# Learning to Retrieve Videos by Asking Questions

Avinash Madasu  Department of Computer Science
UNC Chapel HillUSA<EMAIL_ADDRESS>,  Junier Oliva  Department of Computer
Science
UNC Chapel HillUSA<EMAIL_ADDRESS>and  Gedas Bertasius  Department of
Computer Science
UNC Chapel HillUSA<EMAIL_ADDRESS>
(2018)

###### Abstract.

The majority of traditional text-to-video retrieval systems operate in static
environments, i.e., there is no interaction between the user and the agent
beyond the initial textual query provided by the user. This can be suboptimal
if the initial query has ambiguities, which would lead to many falsely
retrieved videos. To overcome this limitation, we propose a novel framework
for Video Retrieval using Dialog (ViReD), which enables the user to interact
with an AI agent via multiple rounds of dialog. The key contribution of our
framework is a novel multimodal question generator that learns to ask
questions that maximize the subsequent video retrieval performance. Our
multimodal question generator uses (i) the video candidates retrieved during
the last round of interaction with the user and (ii) the text-based dialog
history documenting all previous interactions, to generate questions that
incorporate both visual and linguistic cues relevant to video retrieval.
Furthermore, to generate maximally informative questions, we propose an
Information-Guided Supervision (IGS), which guides the question generator to
ask questions that would boost subsequent video retrieval accuracy. We
validate the effectiveness of our interactive ViReD framework on the AVSD
dataset, showing that our interactive method performs significantly better
than traditional non-interactive video retrieval systems. Furthermore, we also
demonstrate that our proposed approach also generalizes to the real-world
settings that involve interactions with real humans, thus, demonstrating the
robustness and generality of our framework.

interactive video retrieval, dialog generation, multi-modal learning

††copyright: acmcopyright††journalyear: 2018††doi: XXXXXXX.XXXXXXX††copyright:
none††conference: ; ; ††ccs: Computing methodologies Visual content-based
indexing and retrieval Figure 1. An illustration of our interactive dialog-
based video retrieval framework. The order of conversation between the user
and the agent is illustrated from left to right. The agent has access to a
large video database, which is used for retrieving user-specified videos. For
example, in this case, the user starts an interaction with the agent by asking
it to “Show some cooking videos.” The agent then searches for relevant videos
in the database and returns eight candidate videos. Due to high uncertainty in
the initial query, the agent then asks another follow-up question “Which
cuisine do you prefer?” for which the user responds: “Mediterranean.” As the
number of retrieved video candidates is reduced to four, the agent asks one
final question: “Do you like plant or meat diet?” The user’s response (i.e.,
“plant diet”) then helps the agent to reduce the search space to the final
candidate video, which is then displayed to the user.

##  1\. Introduction

The typical (static) video retrieval framework fetches a limited list of
candidate videos from a large collection of videos according to a user query
(e.g. ‘cooking videos’). However, the specificity of this query will likely be
limited, and the uncertainty among candidate videos based on the user query is
typically opaque (i.e. the user might not know what additional information
will yield better results).

For example, consider the scenario where you are deciding what dish to make
for dinner on a Friday night. Now also suppose that you have access to an
interactive AI agent who can help you with this task by retrieving the videos
of relevant dishes and detailed instructions on how to make those dishes. In
this particular scenario, you might start an interaction with the agent by
asking it to “show some cooking videos” (Figure 1 left-most query). Any
traditional video retrieval model will look for the matching cooking videos
from the database and display them to the user. However, what happens if there
are too many matching videos, most of which don’t satisfy the user’s
_internal_ criteria? A user-friendly video retrieval framework will not
display all such videos to the user and expect them to sift through hundreds
of videos to find the videos that are most relevant to them. Not only this
would be overly time consuming, but it would also hurt the user experience.
Instead, one way to address the uncertainty would be to ask another follow-up
question of the same user: “Which cuisine do you prefer?” This would then
allow the user to provide additional information clarifying some of his/her
preferences (e.g., plant or meat diet, etc.) so that an AI agent can narrow
down its search.

The rise of conversational AI systems, such as chat-bots and voice assistants,
have made the user interaction with a digital agent relatively smooth.
Inspired by this emerging technology, and a huge availability of video data,
we propose ViReD , a framework for Video Retrieval using Dialog. Injecting
dialog into standard text-to-video retrieval systems has two key advantages:
(i) it reduces the uncertainty associated with the initial user text queries,
(ii) it enables the agent to infer user’s custom preferences, thus, making the
AI model more personalized to the user.

Several works (Flickner et al., 1995; Wu et al., 2004; Rui et al., 1998;
Kovashka et al., 2012; Kovashka and Grauman, 2013) explored the idea of
interactive mechanisms in the context of image retrieval. Prior methods in
this area used relevance scores (Rui et al., 1998; Wu et al., 2004) and
attribute comparisons (Kovashka et al., 2012; Kovashka and Grauman, 2013) to
get user feedback for retrieval. Additionally, the recent work of Cai et al.
(Cai et al., 2021) proposed Ask-and-Confirm, a framework that allows the user
to confirm if the proposed object is present or absent in the image. One
downside of these prior approaches is that they typically require many
interaction rounds (e.g., $>5$), which increases user effort, and degrades
user experience. Furthermore, these approaches significantly limit the form of
the user-agent interaction, i.e., the users can only verify the presence or
absence of a particular object/attribute in an image but nothing more. In
contrast, our ViReD framework enables the user to interact with an agent using
free-form questions, which is a natural form of interaction for most humans.
We also note that our interactive framework achieves excellent video retrieval
results with a few (e.g., $2-3$) interaction rounds.

Our key technical contribution is a multimodal question generator optimized
with a novel Information-Guided Supervision (IGS). Unlike text-only question
generators, our question generator operates on (i) the entire textual dialog
history (if any) and (ii) previously retrieved top video candidates, which
allows it to incorporate relevant visual and linguistic cues into the question
generation process. Furthermore, our proposed IGS training objective enables
our model to generate maximally informative questions, thus, leading to higher
text-to-video retrieval accuracy.

We validate our entire interactive framework ViReD on the Audio-Visual Scene
Aware Dialog dataset (AVSD) (Alamri et al., 2019) demonstrating that it
outperforms all non-interactive methods by a substantial margin. Furthermore,
compared to other strong dialog-based baselines, our approach requires fewer
dialog rounds to achieve similar or even better results. We also demonstrate
that our approach generalizes to the real-world scenarios involving
interactions with real humans, thus, indicating its effectiveness and
generality. Lastly, we thoroughly ablate different design choices of our
interactive video retrieval framework to inspire future work in this area.

##  2\. Related Work

###  2.1. Multimodal Conversational Agents

There has been a significant progress in designing multimodal conversational
agents especially in the context of image-based visual dialog (Antol et al.,
2015; Das et al., 2017a; Niu et al., 2019; Das et al., 2017b; Qi et al.,
2020). Das et al.(Das et al., 2017a) proposed the task of visual dialog, in
which an agent interacts with a user to answer questions about the visual
video content. There also exists prior work in co-operative image guessing
between a pair of AI agents (Das et al., 2017b). Furthermore, the recent work
of Niu et al. (Niu et al., 2019) proposes a recursive visual attention scheme
for visual dialog generation. We note that most of these prior approaches
operate in closed-set environments, i.e., selecting questions/answers from a
small set of candidates. In contrast, our model leverages visual and
linguistic cues to generate open-ended questions optimized for video
retrieval.

###  2.2. Video Question Answering

Following standard visual question answering (VQA) methods in images (Teney et
al., 2018; Agrawal et al., 2016, 2018; Mishra et al., 2019), video based
question answering (video QA) aims to answer questions about videos (Zeng et
al., 2017; Lei et al., 2018, 2019; Yang et al., 2020). Compared to visual
question answering in images, video question answering is more challenging
because it requires complex temporal reasoning. Le et al. (Le et al., 2019)
introduced a multi-modal transformer model for video QA to incorporate
representations from different modalities. Additionally, Le et al. (Le et al.,
2020) proposed a bi-directional spatial temporal reasoning model to capture
inter dependencies along spatial and temporal dimensions of videos. Recently,
Lin et al. (Lin et al., 2021) introduced Vx2Text, a multi-modal transformer-
based generative network for video QA. Compared to these prior methods, we aim
to develop a framework for interactive dialog-based video retrieval setting.

###  2.3. Multimodal Video Retrieval

Most of the recent multimodal video retrieval systems are based on deep neural
networks (Dzabraev et al., 2021; Gabeur et al., 2020; Chen et al., 2014;
Mithun et al., 2018; Bain et al., 2021; Fang et al., 2021; Croitoru et al.,
2021). With the advent of transformer-based language models (Radford et al.,
2019; Lewis et al., 2019; Raffel et al., 2019), several methods proposed
transformer architectures for video retrieval (Bain et al., 2021; Fang et al.,
2021; Croitoru et al., 2021). However, these methods focus on static query-
based video retrieval and perform poorly when the textual user queries are
ambiguous. In contrast, our work proposes to use dialog as means to gain
additional information for improving video retrieval results.

###  2.4. Interactive Modeling Techniques

Ishan et al. (Misra et al., 2018) proposed an interactive learning framework
for visual question answering. In this framework, the agent actively interacts
with the oracle to get the information needed to answer visual questions.
Several approaches utilized interactive mechanisms to perform image retrieval
(Cai et al., 2021; Wu et al., 2004; Kovashka et al., 2012; Murrugarra-Llerena
and Kovashka, 2021; Maeoki et al., 2020). Instead, we propose an interactive
dialog-based framework for video retrieval.

Figure 2. Comparison between the traditional (i.e., static) and our
interactive dialog-based video retrieval frameworks. In the traditional set-
up, the user interacts with the agent once by providing a single textual query
to retrieve the desired video. In comparison, our proposed framework leverages
multiple rounds of dialog with the user to improve video retrieval
performance. Specifically, after the initial user query, the first round of
retrieved videos are used to generate a question $q_{t}$, which the user then
answers with an answer $a_{t}$. The generated dialog is added to the dialog
history $H_{t}=\\{H_{t-1},(q_{t},a_{t})\\}$, which is then used as additional
input in the subsequent rounds of interaction.

##  3\. Video Retrieval using Dialog

In this section, we introduce ViReD , our proposed video retrieval framework
using dialog. Formally, given an initial text query $T$ specified by the user,
and the previously generated dialog history $H_{t-1}$, our goal is to retrieve
$k$ most relevant videos $V_{1},V_{2},...,V_{k}$.

Our high-level framework, which is illustrated in Figure 2, consists of three
main components: (i) a multimodal question generator trained with an
information-guided supervision (IGS), (ii) an answer generation oracle, which
can answer any questions about a given video, thus, simulating an interaction
with a human, and (iii) a video retrieval module, which takes as inputs the
initial textual query and any generated dialog history and retrieves relevant
videos from a large video database. We now describe each of these components
in more detail.

Figure 3. Illustration of the proposed question generator. It receives (i) an
initial user-specified textual query, (ii) top-k retrieved candidate videos
(from the previous interaction rounds), and (iii) the entire dialog history as
its inputs. We then use a pretrained caption generator (Vid2Sum (Song et al.,
2021)) to map the videos into text. Afterward, all of the text-based inputs
(including the predicted video captions) are fed into an autoregressive BART
model for a new question generation.

###  3.1. Question Generator

As illustrated in Figure 3, at time $t$, our question generator takes as
inputs (i) the initial text query $T$, (ii) top $k$ retrieved videos at time
$t-1$, and (iii) previously generated dialog history $H_{t-1}$. To eliminate
the need for ad-hoc video-and-text fusion modules (Lin et al., 2021; Li et
al., 2020), we use Vid2Sum video caption model (Song et al., 2021) trained on
the AVSD dataset to predict textual descriptions for each of the top-$k$
previously retrieved videos. Specifically, given a video $V_{i}$, the Vid2Sum
model provides a detailed textual summary of the video content, which we
denote as $S_{i}$. Afterward, the predicted summaries for all $k$ videos
retrieved at timestep $t-1$, denoted as $S_{1},S_{2},...,S_{k}$, are fed into
the question generator along with the initial textual query $T$ and previous
dialog history $H_{t-1}$. More precisely, we concatenate the (i) initial text
query, (ii) the predicted summaries and (iii) the previous generated dialog
history and pass it to an autoregressive BART language model (Lewis et al.,
2019) for generating the next question.

(1) $X_{q}=\textrm{Concat}(T,S_{1},S_{2},..,S_{k},H_{t-1}),$ (2)
$q_{t}=\textrm{BART${}_{q}(X_{q})$}.$

###  3.2. Answer Generation Oracle

The answer generator serves as an oracle that can answer any questions about a
given video. We design our answer generator with a goal of simulating the
presence of a human in an interactive dialog setting. Our goal is to use our
answer generator to answer any open-ended questions posed by our previously
described question generator. This characteristic is highly appealing as it
makes our framework flexible and applicable to many diverse dialog scenarios.
In contrast, the majority of prior methods (Cai et al., 2021) are typically
constrained to a small set of closed-set question/answer pairs, which makes it
difficult to generalize them to diverse real-world dialog scenarios. In our
experimental section 6.4, we also conduct a user-study evaluation
demonstrating that our answer generation oracle effectively replaces a human
answering the questions.

Our answer generator takes a video $V_{i}$ and a question $q_{t}$ as its
inputs. Similar to before, we first use a pretrained video caption model
Vid2Sum (Song et al., 2021) to predict a detailed textual summary $S_{i}$,
which we then use as part of the inputs to the answer generator. Afterward,
the generated summary $S_{i}$ and the question $q_{t}$ are concatenated and
passed to a separate BART answer generation model to generate an answer
$a_{t}$ about the video $V_{i}$.

(3) $X_{a}=\textrm{Concat}(S_{i},q_{t}),$ (4)
$a_{t}=\textrm{BART${}_{a}(X_{a})$}.$

Note that the BART models used for question and answer generation have the
same architecture but that their weights are different (i.e., they are trained
for two different tasks).

With these individual components in place, we can now generate $t$ rounds of
dialog using our previously defined question and answer generators. The whole
dialogue history generated over $t$ rounds can then be written as:

(5) $H_{t}=(\\{q_{1},a_{1}\\},\\{q_{2},a_{2}\\},....,\\{q_{t},a_{t}\\}).$

The generated dialog history $H_{t}$ is then used as an additional input to
the video retrieval framework, which we describe below.

###  3.3. Text-to-Video Retrieval Model

Our video retrieval model (VRM) takes an initial textual query $T$ and
previous dialog history $H_{t}$ and returns a probability distribution
$p\in\mathbb{R}^{N}$ that encodes the (normalized) similarity between each
video $V^{(i)}$ in the database of $N$ videos and the concatenated text query
$[T,H_{t}]$. Formally, we can write this operation as:

(6) $\displaystyle p=\mathrm{VRM}(T,H_{t}),$

where each $p_{i}$ value encodes a probability that the $i^{\mathrm{th}}$
video $V^{(i)}$ is the correct video associated with the concatenated textual
query $[T,H_{t}]$.

Our video retrieval model consists of two main components: (i) a visual
encoder $F(V;\theta_{v})$ with learnable parameters $\theta_{v}$ and (ii) a
textual encoder $G(T,H_{t};\theta_{t})$ with learnable parameters
$\theta_{t}$. During training, we assume access to a manually labeled video
retrieval dataset
$\mathcal{X}=\\{(V^{(1)},T^{(1)},H^{(1)}_{t}),\ldots,(V^{(N)},T^{(N)},H^{(N)}_{t})\\}$,
where $T^{(i)}$ and $H^{(i)}_{t}$ depict textual queries and dialog histories
associated with a video $V^{(i)}$ respectively. As our visual encoder, we use
a video transformer encoder (Bertasius et al., 2021) that computes a visual
representation $f^{(i)}=F(V^{(i)};\theta_{v})$ where
$f^{(i)}\in\mathbb{R}^{d}$. As our textual encoder, we use DistilBERT (Sanh et
al., 2019) to compute a textual representation
$g^{(i)}=G(T^{(i)},H^{(i)}_{t};\theta_{t})$ where $g^{(i)}\in\mathbb{R}^{d}$.
We can jointly train the visual and textual encoders end-to-end by minimizing
the sum of video-to-text and text-to-video matching losses as is done in (Bain
et al., 2021):

(7) $\displaystyle\mathcal{L}_{\mathrm{v2t}}$
$\displaystyle=-\frac{1}{B}\sum_{i=1}^{B}\log\frac{\exp({f}^{(i)}\cdot
g^{(i)})}{\sum_{j=1}^{B}\exp({f}^{(i)}\cdot g^{(j)})},$

(8) $\displaystyle\mathcal{L}_{\mathrm{t2v}}$
$\displaystyle=-\frac{1}{B}\sum_{i=1}^{B}\log\frac{\exp({g}^{(i)}\cdot
f^{(i)})}{\sum_{j=1}^{B}\exp({g}^{(i)}\cdot f^{(j)})}.$

Here, $B$ is the batch size, and $f^{(i)},g^{(j)}$ are the embeddings of the
$i^{th}$ video and $j^{th}$ text embeddings (corresponding to the $j^{th}$
video) respectively. Note that we use the matching text-video pairs in a given
batch as positive samples and all the other pairs as negative samples.

During inference, given an initial user query $T$ and the previous dialog
history $H_{t}$, we extract a textual embedding $g=G(T,H_{t};\theta_{t})$
using our trained textual encoder where $g\in\mathbb{R}^{1\times d}$.
Additionally, we also extract visual embeddings
$f^{(i)}=F(V^{(i)};\theta_{v})$ for every video $V^{(i)}$ where $i=1\ldots N$.
We then stack the resulting visual embeddings $[f^{(1)};\ldots;f^{(N)}]$ into
a single feature matrix $Y\in\mathbb{R}^{N\times d}$. Afterward, the video
retrieval probability distribution $p\in\mathbb{R}^{1\times N}$ is computed as
a normalized dot product between a single textual embedding $g$ and all the
visual embeddings $Y$. This can be written as:
$p=\mathrm{Softmax}(gY^{\top})$. For simplicity, throughout the remainder of
the draft, we denote this whole operation as $p=\mathrm{VRM}(T,H_{t})$.

##  4\. Information-Guided Supervision for Question Generation

Our goal in the above-described question generation step is to generate
questions that will maximize the subsequent video retrieval performance. To do
so, the generator must be able to comprehend: (i) the information it has
already obtained, e.g. through the history of dialogue and initial query; (ii)
its current belief and ambiguity over potential videos that should be
retrieved to the user, e.g. through the current top candidate videos; and
(iii) the potential information gain of posing new questions, e.g. the
anticipated increase in performance that may be had by posing certain
questions.

Although providing currently known information and belief over videos is
straightforward via the dialogue history and top-$k$ candidate videos,
respectively, comprehending (and planning for) future informative questions is
difficult. A major challenge stems from the free-form nature of questions that
may be posed. There is a large multitude of valid next questions to pose.
Explicitly labeling the potential information gain of all valid next questions
shall not scale. One may define the task of posing informative queries as a
Markov decision process (MDP), where the current state contains known
information, actions include possible queries to make, and rewards are based
on the number of queries that were made versus the accuracy of the resulting
predictions (Shim et al., 2018; Li and Oliva, 2021). Previous interactive
image retrieval (Cai et al., 2021; Murrugarra-Llerena and Kovashka, 2021)
approaches have used similar MDPs optimized through reinforcement learning to
train policies that may select next questions from a limited finite list.
However, these reinforcement learning (RL) approaches suffer when the action
space is large (as is the case with open-ended question generation) and when
rewards are sparse (as is the case with accuracy after final prediction) (Li
and Oliva, 2021). Thus, we propose an alternative approach, information-guided
question generation supervision (IGS), that bypasses a difficult RL problem,
by explicitly defining informative targets for the question generated based on
a post-hoc search.

Suppose that for each video $V^{(i)}$, $i\in\\{1,\ldots,N\\}$, we also have
$m$ distinct human-generated questions/answers relevant to the video
$D^{(i)}=\\{D^{(i)}_{1},\ldots,D^{(i)}_{m}\\}$. Typically, such data is
collected independently to any particular video retrieval system; e.g. in the
AVSD (Alamri et al., 2019) dataset, users ask (and answer) multiple questions
about the content of a given video (without any particular goals in mind).
However, these human-generated questions can serve as potential targets for
our question generator. With IGS, we propose to filter through $D^{(i)}$
according to the retrospective performance as follows. During training, we
collect targets for the question generator at each round of dialogue
separately. Let $T^{(i)}$, be an initial textual query corresponding to ground
truth video $V^{(i)}$. Then, also let $S_{t,1}^{(i)},\ldots,S_{t,k}^{(i)}$ be
our predicted text summaries of top-$k$ retrieved candidate videos after the
$t^{\mathrm{th}}$ rounds of dialogue, $H^{(i)}_{t}$ (note that
$H^{(i)}_{0}=\emptyset$). We try appending question/answers $(q,a)$ in
$D^{(i)}$ not in $H^{(i)}_{t}$ to $H^{(i)}_{t}$ and see which remaining
question would most improve retrieval performance. That is, we collect

(9)
$\displaystyle\begin{split}(q^{*(i)}_{t+1},a^{*(i)}_{t+1})&=\underset{(q,a)\in(D^{(i)}\setminus
H^{(i)}_{t})}{\mathrm{argmax}}\left[\,\mathrm{VRM}(T^{(i)},H^{(i)}_{t}\cup\\{(q,a)\\})\,\right]_{i},\\\
\end{split}$

where $\mathrm{VRM}$ is our previously described video retrieval model (see
Sec. 3.3). Note that here,
$\left[\mathrm{VRM}(T^{(i)},H^{(i)}_{t}\cup\\{(q,a)\\})\right]_{i}=p_{i}$,
which depicts our previously defined retrieval probability between the ground
truth video $V^{(i)}$, and the concatenated text query
$T^{(i)},H^{(i)}_{t}\cup\\{(q,a)\\}$. Each of the retrospective best questions
are then set up as a target for the question generator at the
$t+1^{\mathrm{th}}$ round

(10)
$\displaystyle\mathcal{D}_{t+1}=\left\\{\left(\left(T^{(i)},S_{t,1}^{(i)},\ldots,S_{t,k}^{(i)},H^{(i)}_{t}\right),\,q^{*(i)}_{t+1}\right)\right\\}_{i=1}^{N},$

Table 1. Comparison to prior video retrieval models. In the ”Pretrain Data” column, we list external datasets used for pretraining. The ”Dialog” and ”Dialog Rounds” columns depict whether the dialog is used as additional input, and if so how many rounds of it. Based on these results, we observe that our ViReD approach outperforms all baselines, including a strong Frozen-in-Time baseline augmented with 10 rounds of human-generated ground truth dialog. Model | Pretrain Data | Dialog | Dialog Rounds ($\downarrow$) | R@1 ($\uparrow$) | R@5 ($\uparrow$) | R@10 ($\uparrow$) | MedianR ($\downarrow$) | MeanR ($\downarrow$)
---|---|---|---|---|---|---|---|---
LSTM (Maeoki et al., 2020) | ImageNet (Russakovsky et al., 2015) | ✓ | 10 | 4.2 | 13.5 | 22.1 | $-$ | 119
FiT (Bain et al., 2021) | WebVid2M (Bain et al., 2021) | ✗ | $-$ | 5.6 | 18.4 | 27.5 | 25 | 95.4
FiT (Bain et al., 2021) w/ Human Dialog (Maeoki et al., 2020) | WebVid2M (Bain et al., 2021) | ✓ | 10 | 10.8 | 28.9 | 40 | 18 | 58.7
ViReD | WebVid2M (Bain et al., 2021) | ✓ | 3 | 12 | 30.5 | 42.1 | 17 | 69.1

where $\left(T^{(i)},S_{t,1}^{(i)},\ldots,S_{t,k}^{(i)},H^{(i)}_{t}\right)$
are the respective initial query, our predicted text summaries of top-k
previous retrievals, and dialogue history that are inputs to the question
generator, $\mathrm{BART}_{q}$. The target question/answers are appended to
the histories
$H_{t+1}^{(i)}=H_{t}^{(i)}\cup\\{(q^{*(i)}_{t+1},a^{*(i)}_{t+1})\\}$, and the
next round of target questions $\mathcal{D}_{t+2}$ is similarly collected.
Please note that $\mathcal{D}_{t+1}$ depends on $\mathcal{D}_{t}$ since we
consider appending questions to previous histories. That is, at each round we
look for informative questions based on the histories seen at that round.
Jointly, the dataset
$\mathcal{D}_{1}\cup\mathcal{D}_{2}\cup\ldots\cup\mathcal{D}_{M}$ serve as a
_supervised_ dataset to directly train the question generator,
$\mathrm{BART}_{q}$, to generate informative questions.

##  5\. Experiments

###  5.1. Dataset

We test our model on the audio-visual scene aware dialog dataset (AVSD)
(Alamri et al., 2019), which contains ground truth dialog data for every video
in the dataset. Specifically, each video in the AVSD dataset has $10$ rounds
of human-generated questions and answers describing various details related to
the video content (e.g., objects, actions, scenes, people, etc.). Thus, we
believe that this dataset is well suited to our setting. In total, the AVSD
dataset consists of $7,985$ training, $863$ validation, and $1,000$ testing
videos (Maeoki et al., 2020). Throughout our experiments, we use standard
training, validation and test splits.

###  5.2. Implementation Details

####  5.2.1. Question Generator.

We train our question generator using the BART large architecture. We set the
maximum sentence length to 120. During generation, we use the beam search of
size 10. The question generator is trained for 5 epochs with a batch size of
32.

####  5.2.2. Answer Generator.

We also use the BART large architecture to train our answer generator. Note
that the question and answer generators use the same architecture but are
trained with two different objectives, thus, resulting in two distinct models.
The maximum sentence length for answer generation is set to 135. During
generation, we use the beam search of size 8. The model is trained with a
batch size of 32 for 2 epochs.

####  5.2.3. Video Retrieval Model.

We use Frozen-in-Time (FiT) (Bain et al., 2021) codebase to implement our
video retrieval model. Specifically, we fine-tune their provided pretrained
model on the AVSD dataset for 20 epochs with a batch size of 16. Early
stopping is applied if the validation loss doesn’t improve for 10 epochs. We
use AdamW (Loshchilov and Hutter, 2017) optimizer with a learning rate of
$3e^{-5}$.

####  5.2.4. Vid2Sum Captioning Model.

We fine-tune the video paragraph captioning model on the training set of AVSD
for 5 epochs. We use the same hyper parameters as in (Song et al., 2021).
During inference, we use our trained Vid2Sum model to generate textual
summaries for each input video. The generated summary has a maximum length of
25.

###  5.3. Evaluation Metrics.

We measure the video retrieval performance using standard Recall@k
($k={1,5,10}$), and MedianR MeanR evaluation metrics. Recall@k calculates the
percentage of test data for which the ground-truth video is found in the
retrieved $k$ videos (the higher the better). Additionally, the MeanR and
MedianR metrics depict the mean and the median rank of the retrieved ground
truth videos respectively (the lower the better). Unless noted otherwise, all
models are averaged over 3 runs.

###  5.4. Video Retrieval Baselines

LSTM (Maeoki et al., 2020). Maeok et al. (Maeoki et al., 2020) proposed an
LSTM-based model that processes human-generated ground truth dialog for video
retrieval. Unlike this prior approach, our interactive ViReD approach does not
require ground truth dialog data during inference. Instead, during each round
of interaction, our method generates novel open-ended questions that maximize
video retrieval accuracy.

Frozen-in-Time (Bain et al., 2021). We fine-tune the Frozen-in-Time (FiT)
model to retrieve the correct video using the initial textual query $T$ as its
input (without using dialog).

Frozen-in-Time w/ Ground Truth Human Dialog. We fine-tune the Frozen-in-Time
model using the textual query and the full $10$ rounds of human-generated
ground truth dialog history. Unlike our ViReD approach, which uses our
previously introduced question and answer generators to generate dialog, this
Frozen-in-Time w/ Dialog baseline uses $10$ rounds of manually annotated human
dialog history during inference. In this setting, we concatenate $10$ rounds
of ground truth dialog with the initial text query, and use the concatenated
text for video retrieval.

Figure 4. We study the video retrieval performance (R@1) as a function of the number of dialog rounds. Based on these results, we observe that the video retrieval accuracy consistently improves as we consider additional rounds of dialog. We also note that the performance of our interactive framework reaches its peak after 3 rounds of dialog. Table 2. Comparison to the state-of-the-art on the video question answering task on the AVSD dataset. Our answer generator outperforms most prior methods and achieves comparable performance to the state-of-the-art Vx2Text (Lin et al., 2021) system. Model | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | METEOR | ROUGE-L
---|---|---|---|---|---|---
MA-VDS (Hori et al., 2019) | 0.256 | 0.161 | 0.109 | 0.078 | 0.113 | 0.277
QUALIFIER (Ye et al., 2022) | 0.276 | 0.177 | 0.121 | 0.086 | 0.119 | 0.294
Simple (Schwartz et al., 2019) | 0.279 | 0.183 | 0.13 | 0.095 | 0.122 | 0.303
RLM (Li et al., 2021) | 0.289 | 0.198 | 0.145 | 0.11 | 0.14 | 0.337
VX2TEXT (Lin et al., 2021) | 0.311 | 0.217 | 0.16 | 0.123 | 0.148 | 0.35
Ours | 0.308 | 0.215 | 0.158 | 0.121 | 0.149 | 0.351

##  6\. Results and Discussion

###  6.1. Quantitative Video Retrieval Results

In Table 1, we compare our method with the previously described video
retrieval baselines. We summarize our key results below.

####  6.1.1. The Importance of Pretraining

The results in Table 1 indicate that large-scale pretraining provides
significant boost in video retrieval performance. Specifically, we note that
the original Frozen-in-Time (FiT) baseline pretrained on large-scale WebVid2M
(Bain et al., 2021) outperforms the previous state-of-the-art LSTM approach
(Maeoki et al., 2020) by $1.4\%$ according to R@1 even without using any
dialog data.

####  6.1.2. Dialog Effectiveness

Next, we demonstrate that dialog is a highly effective cue for the video
retrieval task. Specifically, we first show that the FiT baseline augmented
with $10$ rounds of human-generated ground truth dialog performs $5.2\%$
better in R@1 than the same FiT baseline that does not use dialog (Table 1).
This is a significant improvement that highlights the importance of additional
information provided by dialog.

####  6.1.3. The Number of Dialog Rounds

Next, we observe that despite using only $3$ rounds of dialog our ViReD
approach outperforms the strong FiT w/ Human Dialog baseline, which uses $10$
rounds of human-generated ground truth dialog. It is worth noting that these
$10$ rounds of dialog were generated in a retrieval-agnostic manner (i.e.,
without any particular goal in mind), which may explain this result.
Nevertheless, this result indicates that a few questions (e.g., $3$) generated
by our model are as informative as $10$ task-agnostic human generated
questions.

In Figure 4, we also plot the R@1 video retrieval accuracy of our model as a
function of the number of the dialog rounds. From this figure, we observe that
the performance of our system consistently improves as we use more dialog
rounds. Furthermore, we note that the performance reaches its peak with $3$
rounds of interactions.

Table 3. To validate the effectiveness of our interactive framework in the real-world setting, we replace our automatic answer generator oracle with several human subjects. Specifically, we randomly select 50 videos from the AVSD dataset, and ask 3 subjects to provide answers to our generated questions. We then use those dialogs to measure video retrieval performance as before. Our results suggest that our framework generalizes effectively to the settings involving real human subjects. The video retrieval is performed only on the subset of 50 selected videos. Method | R@1 ($\uparrow$) | R@5 ($\uparrow$) | R@10 ($\uparrow$) | MedianR ($\downarrow$) | MeanR ($\downarrow$)
---|---|---|---|---|---
Answer Generator | 43.8 | 79.2 | 91.7 | 2.0 | 3.6
Human Subject #1 | 45.2 | 81 | 92.7 | 2.0 | 3.5
Human Subject # | 45.8 | 81.2 | 93 | 2.0 | 3.4
Human Subject # | 45.6 | 81.2 | 92.9 | 2.0 | 3.4

###  6.2. Video Question Answering Results

####  6.2.1. Comparison to the State-of-the-Art

As discussed above, we use our answer generator to simulate human presence in
an interactive dialog setting. To validate the effectiveness of our answer
generator, we evaluate its performance on the video question answering task on
AVSD using the same setup as in Simple (Schwartz et al., 2019), and Vx2Text
(Lin et al., 2021). We present these results in Table 2 where we compare our
answer generation method with the existing video question answering baselines.
Our results indicate that our answer generation model significantly
outperforms many previous methods, including MA-VDS (Hori et al., 2019),
QUALIFIER (Ye et al., 2022), Simple (Schwartz et al., 2019) and RLM (Li et
al., 2021). Furthermore, we note that our answer generator achieves similar
performance as the recent Vx2Text model (Lin et al., 2021). These results
indicate that our answer generator is comparable or even better than the
state-of-the-art video-based question answering approaches.

####  6.2.2. Replacing Our Answer Generator with a Human Subject

To validate whether our interactive framework generalizes to the real-world
setting, we conduct a human study where we replace our proposed answer
generator with several human subjects. To do this, we randomly select 50 test
videos from AVSD, and ask $3$ human subjects to answer questions produced by
our question generator. We then use the answers of each subject along with the
generated questions as input to the video retrieval model (similar to our
previously described setup). In Table 3, we report these results for each of
$3$ human subjects. These results suggest that our interactive framework works
reliably even with real human subjects. Furthermore, we note that compared to
the variant that uses an automatic answer generator, the variant with a human
in the loop performs only slightly better, thus, indicating the robustness of
our automatic answer generation framework. Note that in this case, the video
retrieval is performed only on the subset of 50 selected videos.

Figure 5. We study the effectiveness of using our proposed Information-guided
Supervision (IGS) to train the question generator. As our baseline we train
the question generator to generate questions in a video retrieval-agnostic
fashion, i.e., using the same order as the human annotators did when they
asked those questions. Our results indicate that our proposed IGS training
objective produces superior performance compared to the video retrieval-
agnostic baseline. Figure 6. We study the video retrieval performance in two
settings: (i) when the question generator uses top-k retrieved videos as part
of its inputs, and (ii) when it does not. In this case, k is set to 4. Based
on the results, we observe that including top-k retrieved video candidates as
part of the question generator inputs improves video retrieval accuracy for
all number of dialog rounds. Figure 7. We investigate the video retrieval
performance as a function of the number of retrieved candidate video inputs
that are fed into the question generator. These results indicate that the
video retrieval performance is the best when we use 4 retrieved videos as
inputs to our question generator. Figure 8. Qualitative results of our
interactive video retrieval framework. On the left we illustrate the keyframe
of the ground truth video $V_{gt}$ (i.e., the video that the user wants to
retrieve) and the initial textual query for that video. From left to right, we
visualize the three rounds of our generated dialog history (using our question
generator and the answer generator oracle). Furthermore, under each dialog
box, we also illustrate the rank of the ground truth video $V_{gt}$ among all
videos in the database (i.e., the lower the better). Based on these results,
we observe that each dialog round significantly improves video retrieval
results (as indicated by the lower rank of the ground truth video). These
results indicate the usefulness of dialog cues.

###  6.3. Ablation Studies

Next, we ablate various design choices of our model. Specifically, (i) we
validate the effectiveness of our proposed Information-guided Supervision
(IGS), (ii) the importance of using retrieved candidate videos for question
generation, and (iii) how video retrieval performance changes as we vary the
number of candidate video inputs to the question generator.

####  6.3.1. Effectiveness of IGS

To show the effectiveness of IGS, we compare the performance of our
interactive video retrieval framework when using (i) IGS as a training
objective to the question generator vs. (ii) using a video retrieval-agnostic
objective. Specifically, we note the AVSD dataset has 10 pairs of questions
and answers associated with each video. For the retrieval-agnostic baseline,
we use the original order of the questions (i.e., as they appear in the
dataset) to construct a supervisory signal for the question generator. In
other words, we train our BART question generator to ask questions in the same
order as the original AVSD human annotators did. In contrast, for our IGS-
based objective, we order the questions such that they would maximize the
subsequent video retrieval accuracy at each round of questions/answers.

We illustrate these results in Figure 5 where we plot the video retrieval
accuracy of both methods as a function of the number of dialog rounds. These
results suggest that IGS significantly outperforms the retrieval-agnostic
baseline, thus, validating the effectiveness of our proposed IGS technique.

####  6.3.2. The Importance of Using Retrieved Videos for Question Generation

In Figure 6, we verify the importance of using retrieved candidate videos for
the question generation process. Specifically, we compare the video retrieval
performance (i) when the question generator uses top-$k$ retrieved videos as
part of its inputs and (ii) when it does not. Our results suggest that using
top-$k$ retrieved videos for question generation produces a substantial boost
in video retrieval performance. We use $k=4$ in this experiment.

####  6.3.3. Ablating the Number of Video Inputs for Question Generation

Next, in Figure 7, we study the video retrieval performance as a function of
the number of retrieved video inputs fed to the question generator. These
results indicate that the performance gradually increases with every
additional video candidate input and reaches the peak when using $k=4$
retrieved videos. We also observe that the performance slightly drops if we
set $k$ larger than $4$. We hypothesize that this happens because the input
sequence length to the BART question generator becomes too long, potentially
causing overfitting or other optimization-related issues.

###  6.4. Qualitative Results

In Figure 8, we also illustrate some of our qualitative interactive video
retrieval results. On the left we show the keyframe of the ground truth video
$V_{gt}$ (i.e., the video that the user wants to retrieve) and the initial
textual query for that video. From left to right, we illustrate the three
rounds of questions and answers produced by our question and answer
generators. Additionally, under each question/answer box, we also visualize
the rank of the video $V_{gt}$ among all videos in the database (i.e., the
lower the better, where rank of $1$ implies that the correct video was
retrieved).

Our results indicate a few interesting trends. First, we observe that each
dialog round boosts video retrieval performance, which is indicated by the
lower rank of the ground truth video. Second, we note that our question
generator learns to ask question that produce new pieces of information (i.e.,
information not mentioned in the initial textual query). Lastly, we observe
that the questions asked by our model focus on diverse concepts including
gender, presence of certain objects, human actions, clothes colors, etc. This
highlights the flexibility and generality of our open-ended question
generator.

##  7\. Conclusions

In this work, we introduced ViReD , an interactive framework for video
retrieval using dialog. We demonstrated that dialog provides valuable cues for
video retrieval, thus, leading to significantly better performance compared to
the non-interactive baselines. Furthermore, we also showed that our proposed
(i) multimodal question generator, and (ii) information-guided supervision
techniques provide significant improvements to our model’s performance.

In summary, our method is (i) conceptually simple, (ii) it achieves state-of-
the-art results on the interactive video retrieval task on the AVSD dataset,
and (iii) it can generalize to the real-world settings involving human
subjects. In the future, we will extend our framework to other video-and-
language tasks such as interactive video question answering and interactive
temporal moment localization.

## References

  * (1)
  * Agrawal et al. (2016) Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. 2016\.  Analyzing the behavior of visual question answering models.  _arXiv preprint arXiv:1606.07356_ (2016).
  * Agrawal et al. (2018) Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. 2018.  Don’t just assume; look and answer: Overcoming priors for visual question answering. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 4971–4980.
  * Alamri et al. (2019) Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Anoop Cherian, Irfan Essa, Dhruv Batra, Tim K Marks, Chiori Hori, Peter Anderson, et al. 2019\.  Audio visual scene-aware dialog. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 7558–7567.
  * Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015.  Vqa: Visual question answering. In _Proceedings of the IEEE international conference on computer vision_. 2425–2433.
  * Bain et al. (2021) Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. 2021\.  Frozen in time: A joint video and image encoder for end-to-end retrieval. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 1728–1738.
  * Bertasius et al. (2021) Gedas Bertasius, Heng Wang, and Lorenzo Torresani. 2021\.  Is Space-Time Attention All You Need for Video Understanding?. In _Proceedings of the International Conference on Machine Learning (ICML)_.
  * Cai et al. (2021) Guanyu Cai, Jun Zhang, Xinyang Jiang, Yifei Gong, Lianghua He, Fufu Yu, Pai Peng, Xiaowei Guo, Feiyue Huang, and Xing Sun. 2021\.  Ask&Confirm: Active Detail Enriching for Cross-Modal Retrieval With Partial Query. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 1835–1844.
  * Chen et al. (2014) Huizhong Chen, Matthew Cooper, Dhiraj Joshi, and Bernd Girod. 2014.  Multi-modal language models for lecture video retrieval. In _Proceedings of the 22nd ACM international conference on Multimedia_. 1081–1084.
  * Croitoru et al. (2021) Ioana Croitoru, Simion-Vlad Bogolin, Marius Leordeanu, Hailin Jin, Andrew Zisserman, Samuel Albanie, and Yang Liu. 2021\.  Teachtext: Crossmodal generalized distillation for text-video retrieval. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 11583–11593.
  * Das et al. (2017a) Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. 2017a.  Visual dialog. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 326–335.
  * Das et al. (2017b) Abhishek Das, Satwik Kottur, José MF Moura, Stefan Lee, and Dhruv Batra. 2017b.  Learning cooperative visual dialog agents with deep reinforcement learning. In _Proceedings of the IEEE international conference on computer vision_. 2951–2960.
  * Dzabraev et al. (2021) Maksim Dzabraev, Maksim Kalashnikov, Stepan Komkov, and Aleksandr Petiushko. 2021.  Mdmmt: Multidomain multimodal transformer for video retrieval. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 3354–3363.
  * Fang et al. (2021) Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen. 2021\.  Clip2video: Mastering video-text retrieval via image clip.  _arXiv preprint arXiv:2106.11097_ (2021).
  * Flickner et al. (1995) Myron Flickner, Harpreet Sawhney, Wayne Niblack, Jonathan Ashley, Qian Huang, Byron Dom, Monika Gorkani, Jim Hafner, Denis Lee, Dragutin Petkovic, et al. 1995\.  Query by image and video content: The QBIC system.  _computer_ 28, 9 (1995), 23–32.
  * Gabeur et al. (2020) Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. 2020.  Multi-modal transformer for video retrieval. In _European Conference on Computer Vision_. Springer, 214–229.
  * Hori et al. (2019) Chiori Hori, Huda Alamri, Jue Wang, Gordon Wichern, Takaaki Hori, Anoop Cherian, Tim K Marks, Vincent Cartillier, Raphael Gontijo Lopes, Abhishek Das, et al. 2019\.  End-to-end audio visual scene-aware dialog using multimodal attention-based video features. In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2352–2356.
  * Kovashka and Grauman (2013) Adriana Kovashka and Kristen Grauman. 2013.  Attribute pivots for guiding relevance feedback in image search. In _Proceedings of the IEEE International Conference on Computer Vision_. 297–304.
  * Kovashka et al. (2012) Adriana Kovashka, Devi Parikh, and Kristen Grauman. 2012\.  Whittlesearch: Image search with relative attribute feedback. In _2012 IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 2973–2980.
  * Le et al. (2019) Hung Le, Doyen Sahoo, Nancy F Chen, and Steven CH Hoi. 2019\.  Multimodal transformer networks for end-to-end video-grounded dialogue systems.  _arXiv preprint arXiv:1907.01166_ (2019).
  * Le et al. (2020) Hung Le, Doyen Sahoo, Nancy F Chen, and Steven CH Hoi. 2020\.  BiST: Bi-directional spatio-temporal reasoning for video-grounded dialogues.  _arXiv preprint arXiv:2010.10095_ (2020).
  * Lei et al. (2018) Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. 2018\.  Tvqa: Localized, compositional video question answering.  _arXiv preprint arXiv:1809.01696_ (2018).
  * Lei et al. (2019) Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal. 2019\.  Tvqa+: Spatio-temporal grounding for video question answering.  _arXiv preprint arXiv:1904.11574_ (2019).
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019\.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_ (2019).
  * Li et al. (2020) Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. 2020\.  Hero: Hierarchical encoder for video+ language omni-representation pre-training.  _arXiv preprint arXiv:2005.00200_ (2020).
  * Li and Oliva (2021) Yang Li and Junier Oliva. 2021.  Active feature acquisition with generative surrogate models. In _International Conference on Machine Learning_. PMLR, 6450–6459.
  * Li et al. (2021) Zekang Li, Zongjia Li, Jinchao Zhang, Yang Feng, and Jie Zhou. 2021.  Bridging Text and Video: A Universal Multimodal Transformer for Audio-Visual Scene-Aware Dialog.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 29 (2021), 2476–2483.
  * Lin et al. (2021) Xudong Lin, Gedas Bertasius, Jue Wang, Shih-Fu Chang, Devi Parikh, and Lorenzo Torresani. 2021\.  Vx2text: End-to-end learning of video-based text generation from multimodal inputs. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 7005–7015.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ (2017).
  * Maeoki et al. (2020) Sho Maeoki, Kohei Uehara, and Tatsuya Harada. 2020.  Interactive video retrieval with dialog. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_. 952–953.
  * Mishra et al. (2019) Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. 2019.  Ocr-vqa: Visual question answering by reading text in images. In _2019 International Conference on Document Analysis and Recognition (ICDAR)_. IEEE, 947–952.
  * Misra et al. (2018) Ishan Misra, Ross Girshick, Rob Fergus, Martial Hebert, Abhinav Gupta, and Laurens Van Der Maaten. 2018.  Learning by asking questions. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 11–20.
  * Mithun et al. (2018) Niluthpol Chowdhury Mithun, Juncheng Li, Florian Metze, and Amit K Roy-Chowdhury. 2018.  Learning joint embedding with multimodal cues for cross-modal video-text retrieval. In _Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval_. 19–27.
  * Murrugarra-Llerena and Kovashka (2021) Nils Murrugarra-Llerena and Adriana Kovashka. 2021.  Image retrieval with mixed initiative and multimodal feedback.  _Computer Vision and Image Understanding_ 207 (2021), 103204.
  * Niu et al. (2019) Yulei Niu, Hanwang Zhang, Manli Zhang, Jianhong Zhang, Zhiwu Lu, and Ji-Rong Wen. 2019\.  Recursive visual attention in visual dialog. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 6679–6688.
  * Qi et al. (2020) Jiaxin Qi, Yulei Niu, Jianqiang Huang, and Hanwang Zhang. 2020\.  Two causal principles for improving visual dialog. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 10860–10869.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019\.  Language models are unsupervised multitask learners.  _OpenAI blog_ 1, 8 (2019), 9.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_ (2019).
  * Rui et al. (1998) Yong Rui, Thomas S Huang, Michael Ortega, and Sharad Mehrotra. 1998\.  Relevance feedback: A power tool for interactive content-based image retrieval.  _IEEE Transactions on circuits and systems for video technology_ 8, 5 (1998), 644–655.
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015\.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ 115, 3 (2015), 211–252.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  _ArXiv_ abs/1910.01108 (2019).
  * Schwartz et al. (2019) Idan Schwartz, Alexander G Schwing, and Tamir Hazan. 2019\.  A simple baseline for audio-visual scene-aware dialog. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 12548–12558.
  * Shim et al. (2018) Hajin Shim, Sung Ju Hwang, and Eunho Yang. 2018.  Joint active feature acquisition and classification with variable-size set encoding. In _Advances in neural information processing systems_. 1368–1378.
  * Song et al. (2021) Yuqing Song, Shizhe Chen, and Qin Jin. 2021.  Towards Diverse Paragraph Captioning for Untrimmed Videos. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 11245–11254.
  * Teney et al. (2018) Damien Teney, Peter Anderson, Xiaodong He, and Anton Van Den Hengel. 2018.  Tips and tricks for visual question answering: Learnings from the 2017 challenge. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 4223–4232.
  * Wu et al. (2004) Hong Wu, Hanqing Lu, and Songde Ma. 2004.  Willhunter: interactive image retrieval with multilevel relevance. In _Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004._ , Vol. 2. IEEE, 1009–1012.
  * Yang et al. (2020) Zekun Yang, Noa Garcia, Chenhui Chu, Mayu Otani, Yuta Nakashima, and Haruo Takemura. 2020\.  Bert representations for video question answering. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 1556–1565.
  * Ye et al. (2022) Muchao Ye, Quanzeng You, and Fenglong Ma. 2022.  QUALIFIER: Question-Guided Self-Attentive Multimodal Fusion Network for Audio Visual Scene-Aware Dialog. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 248–256.
  * Zeng et al. (2017) Kuo-Hao Zeng, Tseng-Hung Chen, Ching-Yao Chuang, Yuan-Hong Liao, Juan Carlos Niebles, and Min Sun. 2017\.  Leveraging video descriptions to learn video question answering. In _Thirty-First AAAI Conference on Artificial Intelligence_.


*captionUnknown document class (or package)

# Scalable Human-Machine Point Cloud Compression

Mateen Ulhaq and Ivan V. Bajić  School of Engineering Science
Simon Fraser University
Burnaby, BC, Canada
<EMAIL_ADDRESS>School of Engineering Science
Simon Fraser University
Burnaby, BC, Canada
<EMAIL_ADDRESS>
###### Abstract

Due to the limited computational capabilities of edge devices, deep learning
inference can be quite expensive. One remedy is to compress and transmit point
cloud data over the network for server-side processing. Unfortunately, this
approach can be sensitive to network factors, including available bitrate.
Luckily, the bitrate requirements can be reduced without sacrificing inference
accuracy by using a machine task-specialized codec. In this paper, we present
a scalable codec for point-cloud data that is specialized for the machine task
of classification, while also providing a mechanism for human viewing. In the
proposed scalable codec, the “base” bitstream supports the machine task, and
an “enhancement” bitstream may be used for better input reconstruction
performance for human viewing. We base our architecture on PointNet++, and
test its efficacy on the ModelNet40 dataset. We show significant improvements
over prior non-specialized codecs.

###### Index Terms:

deep learning, point cloud compression, coding for machines, scalable coding,
classification

##  I Introduction

Point clouds representing 3D visual data are increasingly being used in many
applications, including augmented reality, robotics, and autonomous driving.
Advances in deep learning have led to the development of deep models for
performing machine vision tasks on point cloud data, including classification,
object detection, and segmentation. However, most deep learning models are
computationally expensive. This poses a challenge for computationally-limited
edge devices that want to perform machine vision tasks, and yet are limited in
size, energy consumption, cost, and other factors.

One option for performing a machine task on the edge device is to limit the
complexity of the model. Unfortunately, this usually comes at the cost of
model accuracy. Another option is to transmit the input data to a server for
machine analysis. In this approach, the edge device compresses the input data
prior to transmission, often using a codec designed to reconstruct the input
for human viewing. However, such _input reconstruction_ codecs are not
optimized for machine analysis. Thus, they often spend large amounts of bits
on encoding information that is not relevant to the machine task. This results
in a worse rate-accuracy trade-off than is possible with a more specialized
codec. In situations where a low rate is desired — for instance, in areas of
poor network connectivity, or congested networks — using a non-specialized
codec may result in excessively high machine task latency [1].

In order to improve the rate-accuracy trade-off, we may instead use a codec
that is specialized for the machine task. Such codecs often perform part of
the machine task on the edge device itself. In this hybrid approach, the edge
device simultaneously compresses the input and performs part of the machine
task. This allows the model to discard unnecessary information, thus reducing
the rate, resulting in a system that is more robust to changing network
conditions, and may reduce system latency over a certain range of available
bitrates [2].

In [3], a novel codec for point cloud classification was proposed. This
learned codec, based on PointNet [4], compresses the input point cloud into a
highly compressed representation that is intended solely for machine analysis,
in this case classification. This codec was shown to achieve a significantly
better rate-accuracy trade-off in comparison with alternative methods using
standard codecs that are not specialized for machine analysis. This was
achieved by removing not only statistical redundancy, but also task-irrelevant
information, during the compression process.

While the codec in [3] achieves a good rate-accuracy trade-off for point cloud
classification, it is not suitable for other purposes. Most applications
involving automated machine-based analysis are expected to run the machine
task continuously, but may occasionally require human verification or review.
Hence, it is important to develop codecs that are able to support both tasks —
machine vision and human viewing — efficiently. In this paper, we present such
a scalable codec, the first in the point cloud literature, which supports
point cloud classification, while also providing a mechanism for human
viewing. Our code is available online.111
https://github.com/multimedialabsfu/learned-point-cloud-compression-for-
classification

##  II Related work

Point cloud classification is among the most researched point cloud analysis
tasks. Related classification models accept different input formats, including
point lists (e.g., PointNet [4] and PointNet++ [5]), 3D voxel grids (e.g.,
VoxNet [6]), and Octrees (e.g., OctNet [7]). Since our work builds on PointNet
and PointNet++, we assume the input format is a point list.

Conventional handcrafted point cloud codecs include Draco [8] and G-PCC [9]
(implemented as TMC13 [10]). More recently, the research focus has shifted
towards learned codecs, following the seminal work of Ballé _et al._ [11] who
proposed a variational autoencoder (VAE) based architecture for image
compression. In their architecture, the input ${\boldsymbol{x}}$ is first
transformed into a latent representation ${\boldsymbol{y}}$, which is then
quantized and entropy-coded using a learned entropy model. Such an
architecture can be trained end-to-end using the loss

$\mathcal{L}=R+\lambda\cdot D({\boldsymbol{x}},{\boldsymbol{\hat{x}}}),$ (1)

where $D({\boldsymbol{x}},{\boldsymbol{\hat{x}}})$ is the distortion measure
between the input ${\boldsymbol{x}}$ and decoded ${\boldsymbol{\hat{x}}}$, and
$R$ is the estimate of the entropy of ${\boldsymbol{\hat{y}}}$. This mechanism
has been successful in various fields of learned compression, including point
cloud compression [12, 13, 14, 15, 16].

Specialized compression for machine tasks — often referred to as _coding for
machines_ — has been explored for images [17] and video [18] and, recently,
for point clouds [3]. Moreover, scalable multi-task coding approaches [19, 20]
have shown that one can perform a machine vision task at a fairly low bitrate,
while enabling other tasks, such as input reconstruction for human viewing,
with an additional enhancement bitstream.

While quality-scalable point cloud coding has been studied before [21], this
paper presents the first scalable multi-task point cloud codec: the base
bitstream supports point cloud classification, while the enhancement bitstream
allows point cloud reconstruction. Unlike our earlier work [3], which
presented a classification-optimized codec based on PointNet [4], our scalable
codec is based on PointNet++ [5], which is a hierarchical extension of
PointNet.

##  III Proposed codec

(a) Input compression.

(b) Machine task compression, as used in e.g. [3].

(c) Scalable multi-task compression.

Figure 1: High-level comparison of codec architectures. Figure 2: Proposed
codec architecture. Figure 3: Proposed codec architecture (details).

###  III-A Preliminaries

In fig. 1(a), we show an abstract representation of an _input compression
codec_. The input point cloud ${\boldsymbol{x}}$ is encoded and decoded as
${\boldsymbol{\hat{x}}}$ by any desired point cloud codec, including non-
learned codecs such as G-PCC [9]. Then, the reconstructed point cloud
${\boldsymbol{\hat{x}}}$ is fed into a classification model (e.g., PointNet)
in order to obtain the class prediction ${\boldsymbol{\hat{t}}}$. This
approach provides a baseline for comparison with our proposed codec.

In fig. 1(b), we show an abstract representation of a _machine task codec_ ,
as was explored by [3] for point cloud classification. Using the same
terminology as in [11], $g_{a}$ refers to the _analysis transform_ , and
$g_{s}$ refers to the _synthesis transform_. In this codec, the input point
cloud ${\boldsymbol{x}}$ is first encoded into a latent representation
${\boldsymbol{y}}=g_{a}({\boldsymbol{x}})$. This is then quantized as
${\boldsymbol{\hat{y}}}=Q({\boldsymbol{y}})$, and then losslessly compressed
using a learned entropy model. For instance, in [3], a fully-factorized
entropy model was used. The reconstructed latent representation
${\boldsymbol{\hat{y}}}$ may then be used to predict the classes
${\boldsymbol{\hat{t}}}=g_{s,t}({\boldsymbol{\hat{y}}})$.

###  III-B Scalable human-machine compression codec

In fig. 1(c), we show a high-level representation of a _scalable multi-task
codec_. Following the principle of _latent space scalability_ [20], the
scalable multi-task codec splits the latent space into two parts,
$[{\boldsymbol{\hat{y}}}_{1},{\boldsymbol{\hat{y}}}_{2}]=\operatorname{split}({\boldsymbol{\hat{y}}})$,
along the channel dimension. The first part ${\boldsymbol{\hat{y}}}_{1}$ is
called the “base” layer and the second part ${\boldsymbol{\hat{y}}}_{2}$ is
called the “enhancement” layer. The base layer is used for the machine task
(i.e., classification) to predict the class
${\boldsymbol{\hat{t}}}=g_{s,t}({\boldsymbol{\hat{y}}}_{1})$. Both base and
enhancement layers are used for input reconstruction:
${\boldsymbol{\hat{x}}}=g_{s,x}(\operatorname{concat}[\operatorname{detach}({\boldsymbol{\hat{y}}}_{1}),{\boldsymbol{\hat{y}}}_{2}])$.
This approach allows for scalability: the enhancement bitstream only needs to
be computed and transmitted when human viewing is desired. Note that the
$\operatorname{detach}$ operation is used to disable gradient propagation of
$D({\boldsymbol{x}},{\boldsymbol{\hat{x}}})$ backwards through
${\boldsymbol{\hat{y}}}_{1}$. This improves the specialization of
${\boldsymbol{\hat{y}}}_{1}$ towards the machine task; otherwise,
${\boldsymbol{\hat{y}}}_{1}$ may end up with some enhancement-layer
information [22]. Two separate bitstreams (base and enhancement) are produced,
whose total rate is
$R_{{\boldsymbol{\hat{y}}}}=R_{{\boldsymbol{\hat{y}}}_{1}}+R_{{\boldsymbol{\hat{y}}}_{2}}$.

###  III-C Proposed architecture

Our complete proposed codec architecture, based on PointNet++ [5], is shown in
fig. 2, and the details of each block are shown in fig. 3. Our codec is
provided in “full” and “lite” configurations, as detailed in table I. The
input point cloud of $P$ points is represented as a matrix
${\boldsymbol{x}}\in\mathbb{R}^{3\times P}$ of $(x,y,z)$ coordinates.
${\boldsymbol{u}}$ is an additional set of attribute features (e.g., normals,
color, etc.) that optionally may also be compressed. At the beginning of the
encoder, ${\boldsymbol{x}}$ and ${\boldsymbol{u}}$ are concatenated along the
channel dimension so that
${\boldsymbol{u}}^{(0)}=\operatorname{concat}[{\boldsymbol{x}},{\boldsymbol{u}}]$.
This ensures that the same encoding capabilities are available for both. We
feed ${\boldsymbol{x}}^{(0)}={\boldsymbol{x}}$ and ${\boldsymbol{u}}^{(0)}$
into a sequence of downsampling blocks. Each $i$-th downsampling block takes
in ${\boldsymbol{x}}^{(i-1)}$ and ${\boldsymbol{u}}^{(i-1)}$, and outputs and
a smaller set of centroids ${\boldsymbol{x}}^{(i)}$ and features
${\boldsymbol{u}}^{(i)}$, along with features ${\boldsymbol{u_{g}}}^{(i-1)}$
that are grouped by the centroids. The final ${\boldsymbol{u}}^{(3)}$ is
compressed in a multi-task scalable manner, from which
${\boldsymbol{\hat{u}}}^{(3)}$ is derived. Each ${\boldsymbol{u_{g}}}^{(i-1)}$
is compressed using a standard transform-encoder-decoder-transform compression
pipeline to obtain ${\boldsymbol{\hat{u}_{g}}}^{(i-1)}$. A sequence of
upsampling blocks is then applied, where each $i$-th upsampling block takes in
${\boldsymbol{\hat{u}}}^{(i)}$. For $i>0$, the output of the $i$-th upsampling
block is concatenated with ${\boldsymbol{\hat{u}_{g}}}^{(i-1)}$ to give
${\boldsymbol{\hat{u}}}^{(i-1)}$. For $i=0$, the output is
${\boldsymbol{\hat{x}}}$.

####  III-C1 Downsampling

Each downsampling block is a PointNet++ “set abstraction” layer (see [5] for
more details), with minor modifications. We used single-scale grouping (SSG)
of points for our proposed codec, though it may likely be improved with multi-
scale grouping (MSG) or multi-resolution grouping (MRG) as described in [5].
The “set abstraction” layer takes as input a (subsampled) point cloud
${\boldsymbol{x}}^{(i-1)}$ of shape $3\times P^{(i-1)}$ and features
${\boldsymbol{u}}^{(i-1)}$ of shape $D^{(i-1)}\times P^{(i-1)}$ containing
information about each corresponding point. Using farthest point sampling
(FPS), a set of $P^{(i)}$ centroids ${\boldsymbol{x}}^{(i)}$ of shape $3\times
P^{(i)}$ is selected from ${\boldsymbol{x}}^{(i-1)}$. Then, the points in
${\boldsymbol{x}}^{(i-1)}$ are grouped into $P^{(i)}$ groups of $S^{(i)}$
points each. Then for each centroid, a ball query222Note that due to the ball
query, the same point may be assigned to multiple centroids. Also, centroids
are not always present within their own group of points. Nonetheless, the ball
query has the benefit of scale-invariant grouping. [5] is performed to find
the first $S^{(i)}$ points that are within a radius of $R^{(i)}$ from the
given centroid. The relative positions of each group of points is then
computed with respect to its associated centroid, resulting in the residuals
$r^{(i)}=\operatorname{ball\\_query}({\boldsymbol{x}}^{(i-1)},{\boldsymbol{x}}^{(i)},R^{(i)})-\operatorname{repeat}({\boldsymbol{x}}^{(i)},S^{(i)})$
of shape $3\times P^{(i)}\times S^{(i)}$. Each point in
${\boldsymbol{r}}^{(i)}$ is concatenated with its respective feature vector in
${\boldsymbol{u}}^{(i-1)}$, resulting in a grouped feature tensor
${\boldsymbol{u_{g}}}^{(i-1)}$ of shape $(D^{(i-1)}+3)\times P^{(i)}\times
S^{(i)}$. Then, ${\boldsymbol{u_{g}}}^{(i-1)}$ is fed into a miniature group-
wise “PointNet encoder” block consisting of a shared multi-layer perceptron
(MLP) with a max pooling operation at the end. That is, the same PointNet
encoder is applied to each group of features independently and identically.
This results in ${\boldsymbol{u}}^{(i)}$ of shape $D^{(i)}\times P^{(i)}$.
Three downsampling blocks are used in our proposed codec. The last
downsampling block groups all the remaining points into a single group, i.e.,
$P^{(3)}=1$.

####  III-C2 Feature compression

At each level $i<3$, a latent representation
${\boldsymbol{y}}^{(i)}={h_{a}}^{(i)}({\boldsymbol{u_{g}}}^{(i)})$ of shape
$M^{(i)}\times P^{(i)}$ is computed, then quantized as
${\boldsymbol{\hat{y}}}^{(i)}=Q({\boldsymbol{y}}^{(i)})$, and then entropy-
coded using a learned entropy model. The resulting bitstream is transmitted
and then decoded as ${\boldsymbol{\hat{y}}}^{(i)}$. From this, a set of
grouped features
${\boldsymbol{\hat{u}_{g}}}^{(i)}=h_{s}^{\,(i)}({\boldsymbol{\hat{y}}}^{(i)})$
is computed for usage during upsampling. (Note that
${\boldsymbol{\hat{u}_{g}}}^{(i)}$ and ${\boldsymbol{u_{g}}}^{(i)}$ may
inhabit entirely different feature spaces, and are unrelated.)

The final feature vector ${\boldsymbol{u}}^{(3)}$ is fed into the scalable
multi-task compression pipeline described in section III-B. The latent
representation ${\boldsymbol{y}}^{(3)}={h_{a}}^{(3)}({\boldsymbol{u}}^{(3)})$
is computed, then quantized as
${\boldsymbol{\hat{y}}}^{(3)}=Q({\boldsymbol{y}}^{(3)})$, and then entropy-
coded to generate a base bitstream for ${\boldsymbol{\hat{y}}}_{1}^{(3)}$ and
an enhancement bitstream for ${\boldsymbol{\hat{y}}}_{2}^{(3)}$. The base
bitstream is decoded as ${\boldsymbol{\hat{y}}}_{1}^{(3)}$, and then fed into
a decoder-side task backend to obtain the class prediction
${\boldsymbol{\hat{t}}}=g_{s,t}^{(3)}({\boldsymbol{\hat{y}}}_{1}^{(3)})$. The
enhancement bitstream is decoded as ${\boldsymbol{\hat{y}}}_{2}^{(3)}$, and
then its concatenation with ${\boldsymbol{\hat{y}}}_{1}^{(3)}$ (detached) is
fed into a decoder-side transform to obtain
${\boldsymbol{\hat{u}_{g}}}^{(3)}=h_{s}^{(3)}(\operatorname{concat}[\operatorname{detach}({\boldsymbol{\hat{y}}}_{1}^{(3)}),{\boldsymbol{\hat{y}}}_{2}^{(3)}])$.

Note that we split only ${\boldsymbol{y}}^{(3)}$ into
${\boldsymbol{y}}^{(3)}_{1}$ and ${\boldsymbol{y}}^{(3)}_{2}$. This is because
${\boldsymbol{y}}^{(3)}_{1}$ suffices for the classification task, while
${\boldsymbol{y}}^{(3)}_{2}$ and ${\boldsymbol{y}}^{(j)}$ for $j<3$ are only
needed for input reconstruction. For brevity, in the experiments we only test
input reconstruction with all enhancement layers, but the proposed
architecture also allows for quality- or density-scalable input
reconstruction.

####  III-C3 Upsampling

An upsampling block takes in $E^{(i+1)}+D^{(i)}+3\cdot(1-\delta_{i,3})$
channels and outputs $E^{(i)}$ channels, where $\delta_{i,j}$ is the Kronecker
delta ($\delta_{i,j}=1$ if $i=j$, $0$ otherwise). Each upsampling block
consists of a simple two-layer point-wise MLP. For upsampling blocks at levels
deeper than $i>0$, batch normalizations are also present. Furthermore, each
such block is followed by a reshape-transpose-reshape combination that
converts the resulting shape $(E^{(i)}\cdot S^{(i)})\times P^{(i)}$ into the
output shape $E^{(i)}\times(S^{(i)}\cdot P^{(i)})$. To ensure compatibility of
shapes among various components during reconstruction, we enforce the
constraint $P^{(i-1)}=P^{(i)}\cdot S^{(i)}$.

TABLE I: Architecture hyperparameter configurations Codec | $i$ | $P^{(i)}$ | $S^{(i)}$ | $R^{(i)}$ | $D^{(i)}$ | $E^{(i)}$ | $M^{(i)}$
---|---|---|---|---|---|---|---
full | 0 | 1024 |  |  | 3 | 3 | 0
| 1 | 256 | 4 | 0.2 | 128 | 64 | 0
| 2 | 64 | 4 | 0.4 | 192 | 32 | 64
| 3 | 1 | 64 |  | 256 | 16 | (48, 16)
lite | 0 | 1024 |  |  | 3 | 3 | 0
| 1 | 256 | 4 | 0.2 | 32 | 32 | 0
| 2 | 64 | 4 | 0.4 | 48 | 16 | 16
| 3 | 1 | 64 |  | 64 | 8 | (48, 16)

##  IV Experiments

We trained our models on the ModelNet40 [23] dataset, sampling $P=1024$ points
per object, and reconstructed the same number of points. Our implementation
uses the PyTorch, CompressAI [24], and CompressAI Trainer [25] libraries.

(a) RA for base task.

(b) RA for base task (zoomed in).

(c) RD for reconstruction task.

Figure 4: Rate-accuracy (RA) and rate-distortion (RD) curves on the ModelNet40
dataset, with rate units of bits per point (bpp) scaled for 1024 points.

The loss function used for training was

$\mathcal{L}=\sum_{i=0}^{3}R_{{\boldsymbol{\hat{y}}}^{(i)}}+\lambda_{x}D({\boldsymbol{x}},{\boldsymbol{\hat{x}}})+\lambda_{t}D({\boldsymbol{t}},{\boldsymbol{\hat{t}}}).$
(2)

The rate estimate $R_{{\boldsymbol{\hat{y}}}^{(i)}}$ for each level is the
negative log likelihood outputted by the respective entropy models. The
distortion $D({\boldsymbol{t}},{\boldsymbol{\hat{t}}})$ is the cross-entropy
between the one-hot encoded labels ${\boldsymbol{t}}$ and the softmax of the
model’s prediction ${\boldsymbol{\hat{t}}}$. The distortion
$D({\boldsymbol{x}},{\boldsymbol{\hat{x}}})$ is the Chamfer distance between
the input and reconstructed point clouds. We trained different models to
operate at different base/enhancement rate points by varying the
hyperparameters $\lambda_{t}\in\\{2^{-7},2^{-6},\ldots,2^{0}\\}$ and
$\lambda_{x}\in[1,8000]$. These values are reported for rates in units of bits
per point (bpp), i.e., the rates are divided by $P=1024$. To improve stability
during training, we disabled the first two levels by setting $M^{(0)}=0$ and
$M^{(1)}=0$. We also set $M^{(2)}=64$ and $M^{(3)}=64$, and used a
partitioning ratio of $0.75$ for the last layer, i.e., $M^{(3)}_{1}=48$ (base)
and $M^{(3)}_{2}=16$ (enhancement).

The input compression codecs were evaluated using the same methodology for
varying $P$ and input scaling as in [3]. Since codecs like TMC13 [10] and
OctAttention [15] are tailored to point clouds with a large number of points,
the coding overhead they produce may be significant when applied to smaller
point clouds such as those used in classification. To improve the fairness
when measuring the rate, we ignored such overhead within the bitstream when
possible.

##  V Results

figs. 4(b) and 4(a) show the rate-accuracy curves for the base task of point-
cloud classification. Only the rate of the base bitstream is measured for the
base task. Scalable codec curves are grouped by their worst-case
reconstruction ability measured in terms of Chamfer distance [26]. Our
proposed codec achieves better classification performance than all existing
codecs, including [3] (denoted “Ulhaq2023”), which in turn is better than
input compression codecs (“ICC”) that use a point-cloud codec (TMC13 [10] and
OctAttention [15]) followed by a PointNet/PointNet++ classifier. Only the best
ICC curves are shown. Note that “$P{=}{}^{*}$” indicates the best results
attained when varying $P\in\\{8,16,\ldots,1024\\}$.

There is a large gap in base task performance between task-specialized codecs
(the proposed one and [3]) and non-specialized codecs because, in addition to
statistical redundancy, the former codecs remove a significant amount of task-
irrelevant information. Meanwhile, the proposed codec outperforms [3] due to
its reliance on the more powerful PointNet++, rather than PointNet.

fig. 4(c) show the rate-distortion curves for the reconstruction task in terms
of the rate versus the Chamfer distance. For the proposed codec, the rate
includes both the base and enhancement bitstreams. The figures show that our
proposed codec is also capable of achieving better reconstruction quality than
conventional codecs at low rates, below 1.6 bpp. As the rate increases, the
conventional codecs eventually catch up in terms of reconstruction quality.

##  VI Conclusion

In this paper, we proposed a scalable multi-task compression codec for point
clouds. Our proposed codec produces a base bitstream that supports point cloud
classification and an enhancement bitstream that, together with the base
bitstream, supports reconstruction of the point cloud for human viewing. The
proposed codec showed a significant improvement in rate-accuracy performance
for the base task over existing codecs, while also achieving competitive
performance against conventional codecs on input reconstruction at reasonably
low rates. In the future, we hope that this work will inspire further research
into scalable multi-task compression codecs for point clouds on more complex
real-world datasets, and for other tasks such as semantic segmentation.

## References

  * [1] N. Shlezinger and I. V. Bajić, “Collaborative inference for AI-empowered IoT devices,” _IEEE Internet of Things Magazine_ , vol. 5, no. 4, pp. 92–98, Dec. 2022.
  * [2] I. V. Bajić, W. Lin, and Y. Tian, “Collaborative intelligence: Challenges and opportunities,” in _Proc. IEEE ICASSP_ , 2021, pp. 8493–8497.
  * [3] M. Ulhaq and I. V. Bajić, “Learned point cloud compression for classification,” in _Proc. IEEE MMSP_ , 2023.
  * [4] C. Qi, H. Su, K. Mo, and L. J. Guibas, “PointNet: Deep learning on point sets for 3D classification and segmentation,” in _Proc. IEEE/CVF CVPR_ , 2017, pp. 77–85.
  * [5] C. Qi, L. Yi, H. Su, and L. J. Guibas, “PointNet++: Deep hierarchical feature learning on point sets in a metric space,” in _Proc. NIPS_ , 2017.
  * [6] D. Maturana and S. A. Scherer, “VoxNet: A 3D convolutional neural network for real-time object recognition,” in _Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS)_ , 2015, pp. 922–928.
  * [7] G. Riegler, A. O. Ulusoy, and A. Geiger, “OctNet: Learning deep 3D representations at high resolutions,” in _Proc. IEEE/CVF CVPR_ , 2017, pp. 6620–6629.
  * [8] Google, “Draco: 3D data compression,” 2017. [Online]. Available: https://google.github.io/draco/
  * [9] K. Mammou, P. A. Chou, D. Flynn, M. Krivokuća, O. Nakagami, and T. Sugio, “G-PCC codec description v2,” 2019, ISO/IEC JTC1/SC29/WG11 N18189.
  * [10] D. Flynn and K. Mammou, “MPEG-PCC-TMC13,” 2021. [Online]. Available: https://github.com/MPEGGroup/mpeg-pcc-tmc13
  * [11] J. Ballé, D. C. Minnen, S. Singh, S. J. Hwang, and N. Johnston, “Variational image compression with a scale hyperprior,” _arXiv preprint arXiv:1802.01436_ , 2018.
  * [12] M. Quach, G. Valenzise, and F. Dufaux, “Learning convolutional transforms for lossy point cloud geometry compression,” in _Proc. IEEE ICIP_ , 2019, pp. 4320–4324.
  * [13] W. Yan, Y. shao, S. Liu, T. H. Li, Z. Li, and G. Li, “Deep autoencoder-based lossy geometry compression for point clouds,” _arXiv preprint arXiv:1905.03691_ , 2019.
  * [14] Y. He, X. Ren, D. Tang, Y. Zhang, X. Xue, and Y. Fu, “Density-preserving deep point cloud compression,” _arXiv preprint arXiv:2204.12684v1_ , 2022.
  * [15] C. Fu, G. Li, R. Song, W. Gao, and S. Liu, “OctAttention: Octree-based large-scale contexts model for point cloud compression,” in _Proc. AAAI_ , vol. 36, no. 1, Jun. 2022, pp. 625–633.
  * [16] K.-S. You, P. Gao, and Q. T. Li, “IPDAE: Improved patch-based deep autoencoder for lossy point cloud geometry compression,” in _Proc. 1st Int. Workshop on Advances in Point Cloud Compression, Processing and Analysis_ , 2022.
  * [17] N. Le, H. Zhang, F. Cricri, R. Ghaznavi-Youvalari, and E. Rahtu, “Image coding for machines: an end-to-end learned approach,” in _Proc. IEEE ICASSP_ , 2021, pp. 1590–1594.
  * [18] L.-Y. Duan, J. Liu, W. Yang, T. Huang, and W. Gao, “Video Coding for Machines: A paradigm of collaborative compression and intelligent analytics,” _IEEE Trans. Image Process._ , vol. 29, pp. 8680–8695, 2020.
  * [19] Y. Hu, S. Yang, W. Yang, L.-Y. Duan, and J. Liu, “Towards coding for human and machine vision: A scalable image coding approach,” in _Proc. IEEE ICME_ , 2020, pp. 1–6.
  * [20] H. Choi and I. V. Bajić, “Scalable image coding for humans and machines,” _IEEE Trans. Image Process._ , vol. 31, pp. 2739–2754, 2022.
  * [21] A. F. R. Guarda, N. M. M. Rodrigues, and F. Pereira, “Point cloud geometry scalable coding with a single end-to-end deep learning model,” in _Proc. IEEE ICIP_ , 2020, pp. 3354–3358.
  * [22] Y. Foroutan, A. Harell, A. de Andrade, and I. V. Bajić, “Base layer efficiency in scalable human-machine coding,” in _Proc. IEEE ICIP_ , 2023, pp. 3299–3303.
  * [23] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao, “3D ShapeNets: A deep representation for volumetric shapes,” in _Proc. IEEE/CVF CVPR_ , 2015, pp. 1912–1920.
  * [24] J. Bégaint, F. Racapé, S. Feltman, and A. Pushparaja, “CompressAI: a PyTorch library and evaluation platform for end-to-end compression research,” _arXiv preprint arXiv:2011.03029_ , 2020.
  * [25] M. Ulhaq and F. Racapé, “CompressAI Trainer,” 2022. [Online]. Available: https://github.com/InterDigitalInc/CompressAI-Trainer
  * [26] H. Blum, “A transformation for extracting new descriptors of shape,” in _Models for the Perception of Speech and Visual Form_ , W. Wathen-Dunn, Ed. Cambridge, MA: MIT Press, 1967, pp. 362–380.