# MINOTAUR: Multi-task Video Grounding From Multimodal Queries

Raghav Goyal∗1,2 Effrosyni Mavroudi4 Xitong Yang4 Sainbayar Sukhbaatar4
Leonid Sigal1,2,3 Matt Feiszli4 Lorenzo Torresani4 Du Tran4

1University of British Columbia 2Vector Institute for AI 3CIFAR AI Chair
4 Meta AI

###### Abstract

Video understanding tasks take many forms, from action detection to visual
query localization and spatio-temporal grounding of sentences. These tasks
differ in the type of inputs (only video, or video-query pair where query is
an image region or sentence) and outputs (temporal segments or spatio-temporal
tubes). However, at their core they require the same fundamental understanding
of the video, i.e., the actors and objects in it, their actions and
interactions. So far these tasks have been tackled in isolation with
individual, highly specialized architectures and pre-extracted features, which
do not exploit the interplay between tasks. In contrast, in this paper, we
present a single, unified model for tackling query-based video understanding
in long-form videos. In particular, our model can address all three tasks of
the Ego4D Episodic Memory benchmark which entail queries of three different
forms: given an egocentric video and a visual, textual or activity query, the
goal is to determine when and where the answer can be seen within the video.
Our model design is inspired by recent query-based approaches to spatio-
temporal grounding, and contains modality-specific query encoders and task-
specific sliding window inference that allow multi-task training with diverse
input modalities and different structured outputs. We exhaustively analyze
relationships among the tasks and illustrate that cross-task learning leads to
improved performance on each individual task, as well as the ability to
generalize to unseen tasks, such as zero-shot spatial localization of language
queries.

††* Algorithmic work was done during an internship at Meta AI.†† Application
of the model to some down-stream tasks was conducted†† at UBC / Vector in
collaboration with Meta AI partners.†† Correspondence<EMAIL_ADDRESS>
Figure 1: Overview. Our unified approach tackles the three episodic memory
tasks with a single model architecture, MINOTAUR, and leverage the
complementary information across different tasks (depicted with shaded
circles).

##  1 Introduction

Fine-grained video understanding has emerged as a crucial capability for the
development of augmented reality (AR) and robotics applications. To achieve
this level of video comprehension, an agent (_e.g_., virtual assistant) must
be equipped with the ability to identify and reason about events and objects
captured in videos, tackling a range of vision tasks such as activity
detection [6, 69, 102], object retrieval [20],and (spatio-)temporal grounding
[2, 9, 18, 32]. However, current research on video understanding has focused
primarily on training individual, highly specialized models using task-
specific annotated datasets [6, 18, 21, 30, 64, 98], without considering the
synergies between multiple tasks and the model’s generalization to novel
tasks.

In this paper, we seek an alternative approach – building a _unified_
framework for multiple fine-grained video understanding tasks. Specifically,
we aim to investigate two key research questions: (i) Is it feasible to design
a single architecture that can seamlessly handle multiple tasks with
multimodal inputs in an end-to-end manner? (ii) To what extent can this
unified framework leverage the complementary information present between
different tasks to achieve superior performance and generalization?

Intuitively, the learning process of a model can be facilitated by capturing
the underlying relationships between different tasks. For example, a model
trained to answer natural language queries [20] (_e.g_., “Where did I put the
spanner?”) can benefit from the learning of answering visual queries (_e.g_.,
searching the latest occurrence of “spanner”) and moment queries (_e.g_.,
“repair small equipment (mower, trimmer, etc.)”), as illustrated in Figure 1.
We show similar empirical findings with our unified framework in Section 4.1.
Moreover, a model trained across multiple tasks has been observed to exhibit
superior generalization ability [48, 47] (including to novel tasks as we show
in Section 4.1) due to the fact that the model has access to diverse data
annotated for different purposes and is not overfitted to a single objective.
Our goals in this paper are synergetic with those of video foundation models
[8, 39], but are also distinct in that our focus is not on learning a model
(using pretex tasks) that can easily adapt to various target tasks, but rather
one joint model that is applicable and can solve variety of target video
understanding tasks directly.

While the idea of building a unified framework for multiple tasks is
conceptually straightforward, it does come with a number of core technical
challenges. The first challenge is the inconsistent input (visual,
lingual)/output (spatial, temporal) format and output distribution across
different tasks. Taking the three episodic memory tasks [20] as an example,
the input of natural language query (NLQ) is free-form language and the output
spans $<2$ seconds, while the input of visual query (VQ) is an image crop and
the output of moment query (MQ) typically spans $>1$ minute. The problem
becomes more severe when localizing a short answer in the long-form videos (8
minutes) in an end-to-end manner. In addition, effective training of models
that capture multimodal information across tasks without compromising per-task
performance remains an open challenge [49, 83].

In light of this, we present MINOTAUR, a unified multi-task model that is
designed to accept long videos and multi-modal queries as input, and generate
diverse structured outputs. The key idea is to cast diverse tasks as special
cases of (spatio-)temporal grounding of queries, which allows them to be
streamlined towards predicting the spatial and/or temporal response locations
of the queries within videos. MINOTAUR builds upon the Transformer encoder-
decoder architecture [48, 89, 27], and task-specificity is baked into the
input/output processing modules such as modality-specific query embeddings and
prediction heads. To address the challenges arising from inconsistent output
distribution in long videos, we design a sliding-window based training and
inference regime, combined with a foreground frame prediction module capable
of generalizing across different tasks. Unlike prior work on long video
understanding that relies on pre-extracted visual features [94, 99, 39], our
model is fully end-to-end learnable and can be trained with partially
annotated data, i.e., without requiring annotations for all tasks per video.
We empirically illustrate that our joint model is capable of transferring and
utilizing information across tasks (see Table 1) improving performance by as
much as 18% on NLQ task compared to the single-task counterpart. We also show
that our model can generalize and produce meaningful results for zero-shot
spatio-temporal grounding, without training on the task.

Contributions. The contributions and novelty of this work is both
methodological and technical. Foremost, we propose MINOTAUR, a unified,
Transformer-based model for grounding multi-modal queries in long-form videos.
To the best of our knowledge, this is the first work that trains and evaluates
a single model for tasks ranging from spatio-temporal grounding to activity
detection. We illustrate that this architecture is able to learn and leverage
information across varied tasks as well as enable novel zero-shot task in
inference. In service of this model, we propose efficient, multi-task training
and inference strategies, which enable training with fixed duration, partially
annotated video segments, and testing on variable-length, long-form videos. We
train our model in both single-task and multi-task settings and evaluate it on
the three tasks of the Ego4D Episodic Memory benchmark [20]. Our results show
that our unified model is competitive with architectures designed and
optimized for each individual task, as well as allows zero-shot spatio-
temporal grounding (not shown elsewhere).

##  2 Related Work

Our work is related to prior works on fine-grained video understanding
(_e.g_., action detection, video grounding), multi-task learning and video-
language modeling.

Video grounding. Grounding natural language query in a reference video has
been an active research area in recent years. The task can be categorized to
temporal grounding [9, 10, 23, 42, 52, 66, 81, 82, 96, 71, 26] which aims to
localizing temporal segments in a video, and spatio-temporal grounding [28,
56, 77, 80, 84] which further requires localizing spatio-temporal tubes.

One direction for temporal grounding is the proposal-based approach [2, 18,
11, 86, 95], where proposals are first generated and then ranked according to
query. The proposal-free approach[19, 92, 95, 32], on the other hand, predicts
the start and end positions directly given the query input. For spatio-
temporal grounding, prior works approach the task from a frame-level
perspective [58, 67, 70, 84, 12, 56] where predictions are made on individual
frames and linked to form action tubes, or tubelet-level perspective [28, 25,
34, 91, 100, 37] where they directly treat tubelet as a unit. Recent works
[89, 101], highlight the effectiveness of Transformer-based approaches, where
in particular, TubeDETR [89] proposed a spatio-temporal grounding system for a
given language query. We also use a Transformer-based approach, extending
TubeDETR [89] to include task-specific components to enable multi-task
learning.

Temporal action detection. The task aims to determine the semantic label and
the temporal interval of every action instance in an untrimmed video [6, 69,
41, 85, 99]. Similar to temporal grounding, prior works either adopt a two-
stage approach which proposes and classifies action proposals using graph
neural networks [3, 88, 93, 99] or Transformers [76, 72], or a single-stage
approach that localizes actions in one step [40, 38, 90]. Recent single-stage
models leverage the Transformer-based framework and achieves the state-of-the-
art results [94, 44]. Our work also follows a single-stage approach where a
dedicated foreground head is introduced to guide the detection in long videos.

It is also noteworthy that most temporal localization or grounding methods
rely on pre-extracted visual features to model long videos and, in doing so,
are limited in ability to holistically understand videos. In this work, we
propose an end-to-end learning framework that tackles all the above tasks in a
unified manner.

Multi-task learning. Multi-task learning has been applied to vision [5, 51,
73, 97], language [63, 14, 43, 50] and robotics [57, 78]. Of particular
interest are works that explore multi-modal multi-task learning [54, 61, 48].
In particular, Lu et al. [48] trains on $12$ image-understanding tasks with
task-specific prediction heads. In our work, we also use task-specific
components for encoding query and decoding outputs, but focus on video
understanding.

Vision and Language. Recently, vision-language approaches [46, 75, 74, 36, 29,
62, 35] using contrastive or masked-data-modeling pre-training losses have
proven to be effective for downstream tasks, which are further extended to
videos [33], and in particular, EgoVLP [39] and InternVideo [8] extends the
video-language pre-training to egocentric domain. However, our work
specifically investigates a general spatio-temporal approach for videos.
Another line of work looks at unified models for structured tasks in vision
[29, 22, 1, 27, 47, 103]. Our approach follows the same line of work and
proposes a unified framework for general spatio-temporal video understanding.

##  3 Approach

Figure 2: Overview of the model Given a video $\mathcal{V}$ and task-query
$\mathcal{Q}_{\nu}$, our model encodes them using a Visual Backbone and
Modality-specific Query Encoder. The obtained video and query features,
$\mathbf{v}$ and $\mathbf{q}$, are encoded by Video-Query Encoder and decoded
using Space-time Decoder by refining Modality-specific time embeddings
$\\{\mathbf{e}_{t}\\}_{t=1}^{T}$. Finally, the prediction heads predict
foreground, start/end time logit, and bounding box for every frame.

Our goal is to design a _unified_ framework that addresses various fine-
grained video understanding tasks and leverages the complementary information
they provide. Specifically, we focus on tackling the challenging episodic
memory tasks [20] in this paper, which involves diverse types of queries that
ask about a user’s past experience.

###  3.1 Problem Formulation

The episodic memory tasks [20] include three different types of queries: (1)
Visual Query (VQ2D) which asks for the most recent spatio-temporal location of
the query object in a video; (2) Moment Query (MQ) which requires localizing
all temporal segments of the query moment; (3) Natural Language Query (NLQ)
which requires retrieving the temporal moments in a video that correspond to
the input language question. Due to their distinct input/output format and
distribution, prior work typically approaches these tasks separately with
specialized models (_e.g_., object-based siamese networks [20, 87] for VQ2D,
action detection models [94, 99] for MQ and temporal grounding networks [95,
32] for NLQ). In contrast, our key idea is to unify all the three episodic
memory tasks to a _multi-task, multi-modal video grounding_ problem.

Formally, given a reference video $\mathcal{V}$ and a query
$\mathcal{Q}_{\nu}$ where $\nu\in\\{\text{VQ2D},\text{NLQ},\text{MQ}\\}$, our
goal is to identify a response track $\mathcal{R}_{\nu}$ represented as a
spatio-temporal tube or a temporal segment. Specifically,

  * •

VQ2D. With $\mathcal{Q}_{\nu}$ as a visual crop of an object at a query frame,
$\mathcal{R}_{\nu}=\\{r_{s},r_{s+1},...,r_{e}\\}$ is a spatio-temporal tube
tracking the object of interest, where $s$ and $e$ are start and end frame
indices respectively, and $r_{i}$ is a bounding box
$(x,y,w,h)\in\mathbb{R}^{4}$ in the $i^{th}$ frame. If there are multiple
occurrences of the object, the most recent occurrence before the query frame
is retrieved.

  * •

MQ. With $\mathcal{Q}_{\nu}$ being one of the activities from a pre-defined
taxonomy (described in language), the response track consists of all the
instances of query activity in the video,
$\mathcal{R}_{\nu}=\\{(s_{n},e_{n})\\}_{n=1}^{N}$, where $N$ is the total
number of instances, $s_{n}$ and $e_{n}$ are start and end frame indices for
the $n^{th}$ instance.

  * •

NLQ. With $\mathcal{Q}_{\nu}$ as a text question, $\mathcal{R}_{\nu}=(s,e)$ is
a temporal segment with $s$ and $e$ as start and end frame indices,
respectively.

###  3.2 MINOTAUR

To jointly solve these three episodic memory tasks with a unified model, we
introduce MINOTAUR, a Transformer-based architecture, illustrated in Figure 2.
Our model receives an egocentric video and a query (visual, textual or
categorical) as inputs, and encodes them with a _Visual Backbone_ and a
_Modality-Specific Query Encoder_ , respectively. After obtaining the video
and query features, they are fused with a _Video-Query Encoder_ that captures
multi-modal interactions. Then, the context-aware video-query features are
decoded using a _Space-Time Decoder_ , which models long-range temporal
interactions. The decoded features are fed to _prediction heads_ to generate a
bounding box prediction, as well as predictions for the start, end, and
foreground probabilities at each frame index. Lastly, these predictions are
processed by _task-specific output processing_ modules, which convert them to
response track $\mathcal{R}_{\nu}$ for each task.

We base our architecture on TubeDETR [89], a state-of-the-art architecture for
spatio-temporal grounding of natural language queries. Our model generalizes
TubeDETR to handle multiple input query modalities, together with task-
specific outputs in a long-form video, where the outputs can take the form of
a single or multiple instances of spatio-temporal or temporal tubes. We
describe TubeDETR [89] below and our task-specific components in Section
3.2.1.

Visual backbone. We start with encoding $T$ frames of video $\mathcal{V}$
using a 2D-CNN backbone and a feed-forward layer. We add 2D positional
encoding to the features and flatten them to give
$\mathbf{v}\in\mathbb{R}^{T\times HW\times d}$, where $T$ is the number of
frames, $H$ and $W$ are height and width of feature map and $d$ is the hidden
size of the model.

Video-Query encoder. Given input query features
$\mathbf{q}\in\mathbb{R}^{L\times d}$ from our Modality-specific Query Encoder
(described later in Section 3.2.1), where $d$ is the hidden size of the model
and $L$ is the size of the query features, we replicate $\mathbf{q}$ across
$T$ frames, concatenate with video features $\mathbf{v}$, and forward it
through a $N_{e}$-layer transformer encoder to give
$Enc(\mathbf{v},\mathbf{q})\in\mathbb{R}^{T\times(HW+L)\times d}$. The encoder
applies transformer layers to each frame-query feature independently and
effectively fuses query information with every frame feature. Note that for
computational reasons, the video-query encoder can be applied to downsampled
video-query features (sampled at regular intervals with stride $k$) and the
output can be temporally replicated or upsampled by the factor of $k$ to match
the original $T$ number of frames.

Space-time decoder. The decoder takes as input a sequence of $T$ learned,
modality-specific time embeddings
$\\{\mathbf{e}_{t}\\}_{t=1}^{T}\in\mathbb{R}^{d}$ (described later in Section
3.2.1). The decoder then refines the time embeddings for each frame $t$ by
cross-attending to video-query encoder features $Enc(\mathbf{v},\mathbf{q})$
in order to predict whether the answer to the query is visible at each frame
$t$. Following TubeDETR [89], decoding is factorized over time and space for
efficiency by having $N_{d}$-blocks of temporal self-attention and frame-wise
cross-attention layers, interleaved with feed-forward and normalization
layers. The temporal self-attention layer allows time embeddings to attend to
each other, thereby facilitating temporal interactions across the video. The
frame-wise cross-attention layer performs cross-attention on each frame
separately, where for an $t^{th}$ frame, the corresponding time embedding
$\mathbf{e}_{t}$ cross-attends to its video-query encoder feature
$Enc(\mathbf{v},\mathbf{q})_{i}\in\mathbb{R}^{(HW+L)\times d}$. In effect, the
decoder accounts for information in both temporal and spatial dimensions to
produce refined time embeddings. See [89] for more details. Finally, the
refined time embeddings $\\{\hat{\mathbf{e}}_{t}\\}_{t=1}^{T}$ are used for
making predictions for every frame in the video.

####  3.2.1 Modality-specific Encoders and Inference

In this section, we describe the novel components of our architecture,
inference procedures and losses that enable us to train and employ a single,
unified model for all tasks.

Modality-specific query encoder. Depending on the task, the query
$\mathcal{Q}_{\nu}$ can be a text question in the case of MQ and NLQ, or a
visual crop in the form of image in the case of VQ2D. To handle different
query modalities, we introduce a modality-specific query encoder. Language
queries are embedded using a RoBERTa language model and mapped to hidden size
$d$ of the model using a feed-forward layer,
$\mathbf{q}_{text}\in\mathbb{R}^{L^{\prime}\times d}$, where $L^{\prime}$ is
the number of tokens. On the other hand, visual queries are encoded using the
visual backbone and a feed-forward layer. The resulting features are flattened
along spatial dimensions to give
$\mathbf{q}_{visual}\in\mathbb{R}^{H^{\prime}W^{\prime}\times d}$, where
$H^{\prime}$, $W^{\prime}$ are spatial dimensions of the feature map.
Formally, the query feature can be written as
$\mathbf{q}\in\mathbb{R}^{L\times d}$, where
$L\in\\{L^{\prime},H^{\prime}W^{\prime}\\}$.

Modality-specific time embeddings. As described earlier, the Space-time
decoder takes as input a sequence of time embeddings
$\\{\mathbf{e}_{t}\\}_{t=1}^{T}$ and refines them using the encoded video-
query features. We obtain the above modality-specific time embeddings by
replicating a learned, modality-specific encoding vector ($\mathbf{e}^{text}$
or $\mathbf{e}^{visual}$) across time $T$ and summing it with a sinusoidal
time-encoding.

Prediction heads. Similar to TubeDETR [89], we use MLPs to predict for each
frame $t$: (a) the relative coordinates for the bounding box
$\hat{\mathbf{b}}_{t}\in[0,1]^{4}$ related to the query, (b) the score
$\hat{\tau}_{s,t}$ that frame $t$ is the start of an answer to the query, and
(c) the score $\hat{\tau}_{e,t}$ that frame $t$ is the end of an answer. To
better handle long-form videos, where multiple segments can be relevant to the
query, we also output a confidence score $\hat{f}_{t}$ for each frame being
relevant to the query. In summary, our model outputs 7-dimensional vector,
$[\hat{\mathbf{b}},\hat{\boldsymbol{\tau}}_{s},\hat{\boldsymbol{\tau}}_{e},\hat{\mathbf{f}}]\in\mathbb{R}^{T\times
7}$, as predictions for every frame in the video (shown in Fig 3 (a)).

Task-specific inference. Due to the limitations of GPU memory our model cannot
process all frames of a long-form video at once. To handle that, we adopt a
sliding window approach, where we train using video segments of fixed duration
of $w$ frames, and perform inference by aggregating model outputs in a sliding
window fashion. These outputs then need to be post-processed to generate task-
appropriate response tracks. To predict a spatio-temporal tube, we form
probabilities for start and end time, $\hat{\mathbf{p}}_{s}$ and
$\hat{\mathbf{p}}_{e}$ by taking softmax over the logits
$\hat{\boldsymbol{\tau}}_{s}$ and $\hat{\boldsymbol{\tau}}_{e}$ respectively.
We infer start and end frame indices, $\hat{s}$ and $\hat{e}$, by choosing
argmax over the distributions such that $\hat{e}>\hat{s}$. The spatial
predictions are formed by selecting bounding boxes in the predicted start/end
range, resulting in space-time tube
$\\{\hat{\mathbf{b}}_{t}\\}_{t=\hat{s}}^{\hat{e}}$.

Figure 3: Task-specific inference. During inference, we identify all possible
peaks from the time-series of foreground scores $\hat{f}$, and for generating
VQ2D predictions (a), we retrieve the most recent occurrence of an object by
forming predictions around the latest peak, while for MQ and NLQ (b), we use
all the peaks for generating temporal segments and rank them according to
their respective heights.

Inference: VQ2D. The task requires finding the most recent occurrence of a
query object. Therefore, we use foreground scores
$\hat{\mathbf{f}}\in\mathbb{R}^{T}$ to identify a set of peaks in the scores
which would indicate high probability of occurrence. From the set of peaks, we
select the most recent one as our candidate peak as shown in Fig 3(a). We
sample a window of size $w$ frames around the peak, and predict start and end
frame indices, $\hat{s}$ and $\hat{e}$, by only considering the predictions
within the window, _i.e_., outside the selected window, the start and end time
probabilities, $\hat{p}_{s}$ and $\hat{p}_{e}$, are zero.

Inference: NLQ + MQ. Since the two tasks require multiple predictions, we
generate predictions using all identified peaks, and sort the predictions
using the scores of the peaks. In addition NLQ and MQ don’t require bounding
boxes, so we only predict time intervals (or frame indices).

|  | VQ2D | NLQ | MQ |
---|---|---|---|---|---
|  | $\text{tAP}_{25}$ | $\text{stAP}_{25}$ | rec$\%$ | Succ | tIoU=0.3 | tIoU=0.5 | tIoU=0.3 | tIoU=0.5 |  # params (# models)
|  | R@1 | R@5 | R@1 | R@5 | R@1x | R@5x | R@1x | R@5x
1 | Single-Task | 0.41 | 0.21 | 29.4 | 61.2 | 6.53 | 17.22 | 3.61 | 9.58 | 33.64 | 55.38 | 23.86 | 39.48 | 432M (3)
2 | All-Tasks (AT) | 0.41 | 0.19 | 26.4 | 60.2 | 7.74 | 21.14 | 4.78 | 12.60 | 34.82 | 56.68 | 25.58 | 42.30 | 186M (1)
3 | AT $\ \ \rightarrow$ Tasks | 0.42 | 0.22 | 29.4 | 61.1 | 7.69 | 21.17 | 5.21 | 12.98 | 33.82 | 56.61 | 25.21 | 42.83 | 432M (3)

Table 1: Multi-task learning on validation set. We observe that, 1) our All-
Tasks (AT) model (row 2) trained using multi-task learning outperforms
individual Single-Task (row 1) models on $9$ out of $12$ metrics across the
three tasks, and 2) further fine-tuning the All-Tasks (AT) model to individual
tasks generally boost the performance (row 1 v/s row 3).

Training. During training, we randomly sample a window of $w$ frames (out of
total $T$ frames) around a ground-truth temporal segment that answers the
query (if there are multiple ground-truth segments we randomly pick one). In
the case that the ground-truth segment is longer than $w$, we randomly place
the window covering a part of the ground-truth.

Losses. Our model is trained using _spatial_ losses, which refine the bounding
box predictions, and _temporal losses_ , which refine the start/end/foreground
frame predictions). In particular, given ground-truth bounding boxes
$\mathbf{b}\in[0,1]^{4\times(e-s+1)}$ for every frame between the ground-truth
start frame $s$, and end frame $e$, the spatial loss computes
$\mathcal{L}_{1}$ and generalized IoU ($\mathcal{L}_{gIoU}$) [7] loss between
$b$ and $\hat{b}$ as:

$\mathcal{L}_{spatial}=\lambda_{\mathcal{L}_{1}}\mathcal{L}_{\mathcal{L}_{1}}(\hat{\mathbf{b}},\mathbf{b})+\lambda_{gIoU}\mathcal{L}_{gIoU}(\hat{\mathbf{b}},\mathbf{b}).$
(1)

For temporal losses, we define target start and end time distributions
($\mathbf{p}_{s},\mathbf{p}_{e}\in[0,1]^{w}$) using standard Normal
($\mathcal{N}(.,1)$) centered at the ground-truth start and end frames,
respectively. For predicted start and end time logits
($\hat{\boldsymbol{\tau}_{s}},\hat{\boldsymbol{\tau}_{e}}\in\mathbb{R}^{w}$),
we normalize them across $w$ frames using softmax to get corresponding
probabilities ($\hat{\mathbf{p}}_{s},\hat{\mathbf{p}}_{e}\in[0,1]^{w}$), and
compute KL-divergence ($\mathcal{L}_{KL}$) between target distribution
$\mathbf{p}_{s}$ (or $\mathbf{p}_{e}$) and predicted distribution
$\hat{\mathbf{p}}_{s}$ (or $\hat{\mathbf{p}}_{e}$). In addition, we also
regress foreground scores, $\hat{\mathbf{f}}\in[0,1]^{w}$, which indicate the
likelihood of a frame to be a part of the ground-truth. We form our target as
$\mathbf{f}\in\\{0,1\\}^{w}$, such that $\mathbf{p}$ is 1 within the start and
end frame indices ($s$ and $e$), and 0 otherwise. We use positive-weighted
binary cross-entropy loss ($\mathcal{L}_{BCE}$) between $\mathbf{f}$ and
$\hat{\mathbf{f}}$ to compute foreground loss. Lastly, we have guided
attention loss $\mathcal{L}_{att}(A)$ which promotes cross-attention weights
in space-time decoder to be diagonal-like and have greater values in their
corresponding temporal boundaries (see [89]).

$\begin{split}\mathcal{L}_{temporal}=\lambda_{KL}&\left[\mathcal{L}_{KL}(\hat{\mathbf{p}}_{s},\mathbf{p}_{s})+\mathcal{L}_{KL}(\hat{\mathbf{p}}_{e},\mathbf{p}_{e})\right]\\\
&+\lambda_{f}\mathcal{L}_{BCE}(\hat{\mathbf{f}},\mathbf{f})+\lambda_{att}\mathcal{L}_{att}(A)\end{split}$
(2)

Depending on the type of ground-truth annotations that are available for each
task, the model is trained with different loss components. Since, NLQ and MQ
only have temporal annotations, we employ only the _temporal_ loss terms:
$\mathcal{L}_{NLQ}=\mathcal{L}_{MQ}=\mathcal{L}_{temporal}$. However, for VQ2D
where we have ground-truth spatio-temporal tubes available, we apply all loss
terms: $\mathcal{L}_{VQ2D}=\mathcal{L}_{spatial}+\mathcal{L}_{temporal}$.

Multi-Task Learning. Accommodating the three Episodic Memory tasks into a
single, unified architecture, has the potential for mutual beneficial cross-
transfer of knowledge among the tasks. For joint-training we use Round-Robin
Sampling [48] which samples batches from tasks one-by-one in a cyclical
manner, effectively training the model with equal proportion of individual
tasks even though the dataset sizes of the tasks could be different. The total
loss in multi-task learning is the sum of the individual tasks’ losses.

##  4 Experiments

##### Datasets.

Ego4D is a large-scale egocentric video dataset with benchmarks that evaluate
first-person visual understanding [20]. We experiment with the three episodic
memory tasks. VQ2D consists of roughly $22k$ video-query pairs. The mean
length of spatio-temporal tubes is approx. $15$ frames (3 secs). NLQ consists
of $19.2k$ video-query pairs where language queries are based on $13$ query
templates. The mean and mode of temporal tube length in train set is approx.
$54$ ($11$ secs) and $6$ ($1.2$ sec) frames respectively. MQ has $22.2k$
action instances across $2.5k$ videos with $110$ pre-definined activity
classes. The mean and mode of temporal tube length in train set is approx.
$220$ ($44$ secs) and $7$ ($1.4$ sec) frames respectively. All three tasks are
split into train:val:test sets with 3:1:1 ratio.

Evaluation metrics. For each task, we use the evaluation metrics defined in
Ego4D [20]. For VQ2D we report _temporal AP_ ($\text{tAP}_{25}$) and _spatio-
temporal AP_ ($\text{stAP}_{25}$) which measure the average-precision of
predicted temporal and spatio-temporal extent of a tube with the ground-truth
at IoU $=0.25$. We also have _Recovery_ ($\text{rec}\%$) which calculates $\%$
of frames in predicted tube where bounding box has at least $0.5$ IoU with the
ground-truth, and _Success_ (Succ) which measures whether prediction has any
overlap with ground-truth as $\%$ of samples where prediction has at least
$0.05$ IoU with the ground-truth. NLQ is evaluated using _recall@k, tIoU=m_ ,
with $k\in\\{1,5\\}$ and $m\in\\{0.3,0.5\\}$, which measures the percentage of
samples where at least one of the top-$k$ predictions has temporal IoU of at
least $m$ with the ground-truth. For MQ, we have _recall@kx, tIoU=m_ , with
$k\in\\{1,5\\}$ and $m\in\\{0.3,0.5\\}$, which measures $\%$ of predicted
instances of class $x$ that have at least one prediction with tIoU greater
than $m$ in the top-$k$ results.

Implementation details. Following TubeDETR [89], we use a ResNet-101 [24] as
our visual backbone and RoBERTa [45] as the language model. We pre-process
videos by resizing them with a shorter side size of 320, and sampling them at
5 fps. During training, we sample a window of length $w$, where $w$ is $200$,
$400$ and $400$ frames, decoded at $5$, $1$ and $1$ fps, and downsampled with
stride $1$, $5$ and $5$ for VQ2D, NLQ and MQ, respectively. The rest of our
hyperparameters are adopted from TubeDETR [89]:
$N_{e}=N_{d}=6,d=256,\lambda_{\mathcal{L}_{1}}=5,\lambda_{\mathcal{L}_{gIoU}}=2,\lambda_{KL}=10,\lambda_{att}=1,\lambda_{f}=2$.
We initialize our weights from MDETR [29] which is pretrained on Flickr30K
[59], MS COCO [13] and Visual Genome [31]. We train our models on 16 GPUs with
an effective batch size of $16$ videos for 25 epochs with a drop in learning
rate at every $10^{th}$ epoch by a factor of 10. The final model is selected
based on best performance on a small subset of validation data.

###  4.1 Main Results

##### Single-task v/s multi-task.

We train our model individually on the three tasks to setup a baseline for
multi-task learning, Single-Task, see row 1 of Table 1. We use the same model
architecture except in the case of VQ2D where text encoder is not needed. In
addition, we train a single model on all the three tasks, All-Tasks (AT),
shown in row 2. For $9$ out of $12$ metrics across the three tasks, we obtain
improvements in multi-task results over Single-task (row 1 v/s row 2).
Moreover, the total number of models required for all three tasks drops from
$3$ to $1$, leading to $2.3\times$ fewer parameters overall. We analyze
additional pair-wise trained task combinations and transfer among tasks in
Suppl Mat.

We highlight that amount of data per task and the architecture is the same for
Single-task and All-Tasks (AT) models; meaning, improvements are purely based
on effective multi-task training and complementary information learned across
tasks. All-Task (AT) model results in largest improvements, over the Single-
Task counterpart, on NLQ task owning to significant synergies with VQ2D. In
particular, since VQ2D focuses on inference of objects and places, the
improvements in NLQ are largest on questions of those categories as well (see
Table 2). Meanwhile, improvements on people questions over Single-Task model
are negligible.

|  | R@$5$, tIoU=$0.5$
---|---|---
Category | Template |  | NLQ
---
only
All-Tasks |  | Gain
---
(in %)
Objects | Where is object X before / after event Y? | 6.21 | 7.30 | +17.50
Where is object X? | 10.29 | 13.42 | +30.43
What did I put in X? | 5.43 | 7.67 | +41.18
How many X’s? (quantity) | 17.67 | 23.67 | +33.96
What X did I Y? | 9.94 | 13.78 | +38.71
In what location did I see object X? | 10.24 | 11.95 | +16.67
What X is Y? | 10.13 | 12.42 | +22.58
State of an object | 11.31 | 22.02 | +94.74
Where is my object X? | 6.49 | 11.69 | +80.00
Place | Where did I put X? | 5.43 | 7.67 | +41.18
People | Who did I interact with when I did activity X? | 12.75 | 11.76 | -7.69
Who did I talk to in location X? | 15.66 | 16.87 | +7.69
When did I interact with person with role X? | 4.00 | 4.00 | 0.00

Table 2: All-Tasks v/s Single-Task on NLQ. Performance on NLQ across $13$
question types. We observe that All-Tasks model brings larger improvement for
object-centric questions compared to Single-Task (“NLQ only”).

We also experiment with fine-tuning All-Tasks (AT) model to individual tasks
with the idea that AT model may allow downstream tasks to take advantage from
multi-task joint pre-training [48]. Shown in row 3, for $11$ out of $12$
metrics across the three tasks, we observe comparable or better performance
than Single-Task (row 2). The improvements of fine-tuning (row 3) over the
All-Tasks (AT) model (row 3) are modest. This highlights that our multi-task
learning is effective and unified model is itself competitive.

Figure 4: Spatio-temporal predictions for NLQ with the _All-Tasks (AT)_ model.
Predictions are shown with yellow bounding boxes, while ground-truth temporal
segments are denoted with green borders. Although the model has not be trained
with ground-truth spatio-temporal tubes for language queries, multi-task
training results in some meaningful results for zero-shot spatio-temporal
grounding.

| Spatio-temporal | Temporal
---|---|---
Model |  spatial branch | stIoU=0.3 |  mean stIoU |  mean tIoU
R@1 | R@5
NLQ-only | N/A | - | - | - | 5.35
MINOTAUR (All-Tasks) | random boxes | 0 $\pm$ 0 | 0 $\pm$ 0 | 0.40 $\pm$ 0.04 | 8.35
| random centered
---
boxes
0 $\pm$ 0 | 0.47 $\pm$ 0.38 | 1.25 $\pm$ 0.03
All-Tasks | 2.33 | 4.65 | 2.27

Table 3: Zero-shot spatio-temporal grounding on NLQ. We evaluate spatio-
temporal predictions on NLQ by annotating a subset of validation videos
($=130$), and comparing it with random baselines. We observe non-trivial
performance despite the model not trained on (language query, spatio-temporal
target).

|  | VQ2D | NLQ | MQ
---|---|---|---|---
|  |  |  |  |  | tIoU=0.3 | tIoU=0.5 | tIoU=0.5
| Methods | $\text{tAP}_{25}$ | $\text{stAP}_{25}$ | rec$\%$ | Succ | R@1 | R@5 | R@1 | R@5 | R@1x
1 | Siam-RCNN [20] | 0.21 | 0.13 | 34.0 | 41.6 | - | - | - | - | -
2 | VSLNet [20, 95] | - | - | - | - | 5.47 | 11.21 | 2.80 | 6.57 | -
3 | VSGN [20, 99] | - | - | - | - | - | - | - | - | 24.25
4 | Xu et al. [87] | 0.26 | 0.18 | 43.2 | 48.1 | - | - | - | - | -
5 | EgoVLP: VSLNet [39] | - | - | - | - | 4.87 | 8.67 | 2.50 | 4.97 | -
6 | EgoVLP: VSGN [39] | - | - | - | - | - | - | - | - | 19.74
7 | InternVideo [8] | - | - | - | - | - | - | - | - | 27.71
8 | ActionFormer [53] | - | - | - | - | - | - | - | - | 24.25
9 | Ours: All-Tasks (AT) | 0.41 | 0.19 | 26.5 | 60.6 | 7.47 | 19.61 | 4.85 | 12.09 | 24.62
10 | Ours: AT $\ \rightarrow\ $ Tasks | 0.42 | 0.21 | 29.0 | 61.4 | 8.07 | 20.63 | 5.52 | 12.94 | 24.44

Table 4: Comparison with existing works on test set. We observe, 1) our
approach outperforms Ego4D [20] baselines (row 1 \- 3) on $8$ out of $9$
metrics across the three tasks, 2) our approach performs comparably to SOTA
using task-specific data (without pretraining on additional $3.85$M Ego4D
video-text corpus). For MQ, only R@1x at tIoU=$0.5$ is reported on test set.

MQ | tIoU=0.5
---|---
R@1x | R@5x
MINOTAUR (All-Tasks model) | 25.58 | 42.30
Input | w/o text encoder | 18.22 | 34.54
| w/o modality-specific time embedding | 24.39 | 42.04
Train | w/o round-robin sampling | 21.97 | 39.64
| w/o fine-tuning text encoder | 18.71 | 31.40
Inf. | w/o multi-scale inference | 20.86 | 38.48
w/o overlapping windows | 20.01 | 37.31

Table 5: Ablation study on MQ. We ablate individual components in MINOTAUR by
removing each independently.

Synergy across different tasks. Another benefit of training a unified model on
multiple tasks is the ability to transfer learned concepts across the tasks.
Specifically, for NLQ, even though the model was trained on (language query,
temporal target) pairs, we can turn on the spatial prediction branch of the
model, and make spatial predictions on language query, despite the fact that
the spatial branch was trained only on the (visual query, spatio-temporal
target) pairs in VQ2D. We call this setup zero-shot spatio-temporal grounding
where a unified model can transfer learned concepts across the modalities.

We benchmark the performance of spatial predictions by manually annotating a
subset of NLQ data with spatial bounding boxes. Table 3 shows the results
comparing Single-Task model on NLQ (“NLQ only”) and All-Tasks (AT) trained on
all tasks. We added two random baselines for spatial predictions - random
boxes: which produces random boxes for each frame, and random centered boxes:
which produces center-biased random boxes which are centered for each frame.
We observe that, 1) our All-Tasks model outperforms Single-Task “NLQ only”
model on the temporal benchmark (similar conclusion as in Table 1), and 2)
spatio-temporal predictions from All-Tasks model outperforms the two random
baselines (repeated $5$ times). This shows that our All-Tasks model produces
non-trivial zero-shot spatio-temporal localization on task for which it wasn’t
trained on. We also show a few examples of spatio-temporal predictions on NLQ
task using All-Tasks model in Figure 4.

###  4.2 Ablations

We ablate different design choices using our All-Tasks (AT) model on MQ in
Table 5. More ablation study results on VQ and NLQ are available in the Suppl
Mat. 1) w/o text encoder: instead of training a language encoder for query
“When did I do $<$activity c$>$?” , we experiment with one-hot encodings for
each moment class and embed them using 1-layered MLP, 2) w/o modality-specific
time-embedding: instead of having separate time embeddings for visual and text
queries ($\mathbf{e}^{visual},\mathbf{e}^{text}$), we report results on using
a single time embedding, 3) w/o round-robin sampling: instead of uniformly
sampling over three tasks during training, we show results for simply
concatenating the datasets, 4) w/o fine-tuning text encoder: , we report
results on freezing the text encoder instead of jointly fine-tuning it with
the query objectives, 5) w/o multi-scale inference: instead of accumulating
inference results at different scales (fps: $0.2,0.5,1,1.66,5$), we show the
results with a single scale (fps: $5$), 6) w/o overlapping windows: increasing
the stride between adjacent sliding windows to remove overlapping during
inference. We observe significant drop in performance across all tested
ablations, indicating the effectiveness of our technical designs in achieving
a unified multi-task model.

We also study the impact of the foreground head (detailed in Section 3.2.1)
for generating predictions on long videos. Specifically for VQ2D, we leverage
the foreground scores $\hat{\mathbf{f}}$ over a video to identify the most
recent occurrence of a query object. This is in contrast to TubeDETR [89]
which makes predictions on whole-video without accounting for this bias. Our
results show that using foreground predictions improves the baseline from 0.25
(0.15) to 0.41 (0.21) for tAP25 (stAP25).

###  4.3 Comparison with Existing Works

We compare our approach with prior works in Table 4. Note that we only report
the methods trained on task-specific data (without additional large-scale
pretraining on $3.85$M video-text Ego4D corpus) due to space limit and leave
the others to the Suppl Mat. We first observe that compared to baselines
reported in Ego4D paper [20] (rows 1 \- 3), our All-Tasks model (row 9)
outperforms on 8 out of 9 metrics across the three tasks. Second, compared to
state-of-the-art on VQ2D task: Xu et al. [87] (row 4), our All-Tasks model
(row 9) outperforms on $3$ out of $4$ metrics on VQ2D. Third, compared to
other approaches which use different combinations of backbone architectures
and pre-training data where EgoVLP [39] uses Frozen [4] backbone with CC3M
[68] and WebVid2M [4] pre-training, InternVideo [8] uses VideoMAE [79]
backbone with Kinetics-700 [30] pre-training, and ActionFormer [94] uses
SlowFast [17] backbone with Kinetics-400 [30] pre-training; our All-Tasks (AT)
model outperforms on NLQ (row 5 v/s 9), and outperforms $2$ out of $3$ related
works on MQ (rows 6 \- 8 v/s row 9), despite the fact that these approaches
use dedicated architectures designed specifically for each individual task.

##  5 Conclusion

We present a unified approach for grounding multi-modal queries in long-videos
with different degrees of spatio-temporal outputs. To achieve that, we propose
a multi-task training and inference strategies that enables learning on data
with varying degree of spatio-temporal annotations. Finally, we observe
effectiveness of our approach over task-specific single architectures in terms
of generalization and transfer of learned abilities such as zero-shot spatio-
temporal grounding.

## Acknowledgement

This work was initiated as an internship project at Meta AI and has been to a
large part enabled and made possible by the computational resources provided
by Meta. In addition, this work was funded, in part, by the Vector Institute
for AI, Canada CIFAR AI Chair, NSERC CRC and an NSERC DG. Hardware resources,
in part, were provided by the Province of Ontario, the Government of Canada
through CIFAR, and companies sponsoring the Vector
Institute111www.vectorinstitute.ai/#partners. Finally, we would like to thank
Mennatullah Siam for valuable discussions.

## References

  * [1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  arXiv preprint arXiv:2204.14198, 2022.
  * [2] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell.  Localizing moments in video with natural language.  In Proceedings of the IEEE international conference on computer vision, pages 5803–5812, 2017.
  * [3] Yueran Bai, Yingying Wang, Yunhai Tong, Yang Yang, Qiyue Liu, and Junhui Liu.  Boundary content graph neural network for temporal action proposal generation.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVIII 16, pages 121–137. Springer, 2020.
  * [4] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  Frozen in time: A joint video and image encoder for end-to-end retrieval.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1728–1738, 2021.
  * [5] Felix JS Bragman, Ryutaro Tanno, Sebastien Ourselin, Daniel C Alexander, and Jorge Cardoso.  Stochastic filter groups for multi-task cnns: Learning specialist and generalist convolution kernels.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1385–1394, 2019.
  * [6] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles.  Activitynet: A large-scale video benchmark for human activity understanding.  In Proceedings of the ieee conference on computer vision and pattern recognition, pages 961–970, 2015.
  * [7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European conference on computer vision, pages 213–229. Springer, 2020.
  * [8] Guo Chen, Sen Xing, Zhe Chen, Yi Wang, Kunchang Li, Yizhuo Li, Yi Liu, Jiahao Wang, Yin-Dong Zheng, Bingkun Huang, et al.  Internvideo-ego4d: A pack of champion solutions to ego4d challenges.  arXiv preprint arXiv:2211.09529, 2022.
  * [9] Jingyuan Chen, Xinpeng Chen, Lin Ma, Zequn Jie, and Tat-Seng Chua.  Temporally grounding natural sentence in video.  In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 162–171, 2018.
  * [10] Jingyuan Chen, Lin Ma, Xinpeng Chen, Zequn Jie, and Jiebo Luo.  Localizing natural language in videos.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8175–8182, 2019.
  * [11] Shaoxiang Chen and Yu-Gang Jiang.  Semantic proposal for activity localization in videos via sentence query.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8199–8206, 2019.
  * [12] Shoufa Chen, Peize Sun, Enze Xie, Chongjian Ge, Jiannan Wu, Lan Ma, Jiajun Shen, and Ping Luo.  Watch only once: An end-to-end video action detection framework.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8178–8187, 2021.
  * [13] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco captions: Data collection and evaluation server.  arXiv preprint arXiv:1504.00325, 2015.
  * [14] Ronan Collobert and Jason Weston.  A unified architecture for natural language processing: Deep neural networks with multitask learning.  In Proceedings of the 25th international conference on Machine learning, pages 160–167, 2008.
  * [15] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al.  Scaling egocentric vision: The epic-kitchens dataset.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 720–736, 2018.
  * [16] Ahmad Darkhalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan Kar, Richard Higgins, Sanja Fidler, David Fouhey, and Dima Damen.  Epic-kitchens visor benchmark: Video segmentations and object relations.  In Proceedings of the Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, 2022.
  * [17] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202–6211, 2019.
  * [18] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia.  Tall: Temporal activity localization via language query.  In Proceedings of the IEEE international conference on computer vision, pages 5267–5275, 2017.
  * [19] Soham Ghosh, Anuva Agarwal, Zarana Parekh, and Alexander Hauptmann.  Excl: Extractive clip localization using natural language descriptions.  arXiv preprint arXiv:1904.02755, 2019.
  * [20] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al.  Ego4d: Around the world in 3,000 hours of egocentric video.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995–19012, 2022.
  * [21] Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al.  Ava: A video dataset of spatio-temporally localized atomic visual actions.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6047–6056, 2018.
  * [22] Tanmay Gupta, Amita Kamath, Aniruddha Kembhavi, and Derek Hoiem.  Towards general purpose vision systems: An end-to-end task-agnostic vision-language architecture.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16399–16409, 2022.
  * [23] Dongliang He, Xiang Zhao, Jizhou Huang, Fu Li, Xiao Liu, and Shilei Wen.  Read, watch, and move: Reinforcement learning for temporally grounding natural language descriptions in videos.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8393–8400, 2019.
  * [24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [25] Rui Hou, Chen Chen, and Mubarak Shah.  Tube convolutional neural network (t-cnn) for action detection in videos.  In Proceedings of the IEEE international conference on computer vision, pages 5822–5831, 2017.
  * [26] Zhijian Hou, Wanjun Zhong, Lei Ji, Difei Gao, Kun Yan, Wing-Kwong Chan, Chong-Wah Ngo, Zheng Shou, and Nan Duan.  Cone: An efficient coarse-to-fine alignment framework for long video temporal grounding.  arXiv preprint arXiv:2209.10918, 2022.
  * [27] Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al.  Perceiver io: A general architecture for structured inputs & outputs.  arXiv preprint arXiv:2107.14795, 2021.
  * [28] Mihir Jain, Jan Van Gemert, Hervé Jégou, Patrick Bouthemy, and Cees GM Snoek.  Action localization with tubelets from motion.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 740–747, 2014.
  * [29] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion.  Mdetr-modulated detection for end-to-end multi-modal understanding.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780–1790, 2021.
  * [30] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al.  The kinetics human action video dataset.  arXiv preprint arXiv:1705.06950, 2017.
  * [31] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  International journal of computer vision, 123(1):32–73, 2017.
  * [32] Jie Lei, Tamara L Berg, and Mohit Bansal.  Detecting moments and highlights in videos via natural language queries.  Advances in Neural Information Processing Systems, 34:11846–11858, 2021.
  * [33] Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven CH Hoi.  Align and prompt: Video-and-language pre-training with entity prompts.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4953–4963, 2022.
  * [34] Dong Li, Zhaofan Qiu, Qi Dai, Ting Yao, and Tao Mei.  Recurrent tubelet proposal and recognition networks for action detection.  In Proceedings of the European conference on computer vision (ECCV), pages 303–318, 2018.
  * [35] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In International Conference on Machine Learning, pages 12888–12900. PMLR, 2022.
  * [36] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In European Conference on Computer Vision, pages 121–137. Springer, 2020.
  * [37] Yixuan Li, Zixu Wang, Limin Wang, and Gangshan Wu.  Actions as moving points.  In European Conference on Computer Vision, pages 68–84. Springer, 2020.
  * [38] Chuming Lin, Chengming Xu, Donghao Luo, Yabiao Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Yanwei Fu.  Learning salient boundary feature for anchor-free temporal action localization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3320–3329, 2021.
  * [39] Kevin Qinghong Lin, Alex Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Zhongcong Xu, Difei Gao, Rongcheng Tu, Wenzhe Zhao, Weijie Kong, et al.  Egocentric video-language pretraining.  arXiv preprint arXiv:2206.01670, 2022.
  * [40] Tianwei Lin, Xu Zhao, and Zheng Shou.  Single shot temporal action detection.  In Proceedings of the 25th ACM international conference on Multimedia, pages 988–996, 2017.
  * [41] Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang.  Bsn: Boundary sensitive network for temporal action proposal generation.  In Proceedings of the European conference on computer vision (ECCV), pages 3–19, 2018.
  * [42] Zhijie Lin, Zhou Zhao, Zhu Zhang, Qi Wang, and Huasheng Liu.  Weakly-supervised video moment retrieval via semantic completion network.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11539–11546, 2020.
  * [43] Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao.  Multi-task deep neural networks for natural language understanding.  arXiv preprint arXiv:1901.11504, 2019.
  * [44] Xiaolong Liu, Qimeng Wang, Yao Hu, Xu Tang, Shiwei Zhang, Song Bai, and Xiang Bai.  End-to-end temporal action detection with transformer.  IEEE Transactions on Image Processing, 31:5427–5441, 2022.
  * [45] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * [46] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  Advances in neural information processing systems, 32, 2019.
  * [47] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi.  Unified-io: A unified model for vision, language, and multi-modal tasks.  arXiv preprint arXiv:2206.08916, 2022.
  * [48] Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee.  12-in-1: Multi-task vision and language representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10437–10446, 2020.
  * [49] Aakarsh Malhotra, Mayank Vatsa, and Richa Singh.  Dropped scheduled task: Mitigating negative transfer in multi-task learning using dynamic task dropping.  In Transactions on Machine Learning Research, 2022.
  * [50] Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher.  The natural language decathlon: Multitask learning as question answering.  arXiv preprint arXiv:1806.08730, 2018.
  * [51] Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert.  Cross-stitch networks for multi-task learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3994–4003, 2016.
  * [52] Niluthpol Chowdhury Mithun, Sujoy Paul, and Amit K Roy-Chowdhury.  Weakly supervised video moment retrieval from text queries.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11592–11601, 2019.
  * [53] Fangzhou Mu, Sicheng Mo, Gillian Wang, and Yin Li.  Where a strong backbone meets strong features – actionformer for ego4d moment queries challenge.  arXiv e-prints, 2022.
  * [54] Duy-Kien Nguyen and Takayuki Okatani.  Multi-task learning of hierarchical vision-language representation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10492–10501, 2019.
  * [55] Seoung Wug Oh, Joon-Young Lee, Ning Xu, and Seon Joo Kim.  Video object segmentation using space-time memory networks.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9226–9235, 2019.
  * [56] Junting Pan, Siyu Chen, Mike Zheng Shou, Yu Liu, Jing Shao, and Hongsheng Li.  Actor-context-actor relation network for spatio-temporal action localization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 464–474, 2021.
  * [57] Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov.  Actor-mimic: Deep multitask and transfer reinforcement learning.  arXiv preprint arXiv:1511.06342, 2015.
  * [58] Xiaojiang Peng and Cordelia Schmid.  Multi-region two-stream r-cnn for action detection.  In European conference on computer vision, pages 744–759. Springer, 2016.
  * [59] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  In Proceedings of the IEEE international conference on computer vision, pages 2641–2649, 2015.
  * [60] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool.  The 2017 davis challenge on video object segmentation.  arXiv preprint arXiv:1704.00675, 2017.
  * [61] Subhojeet Pramanik, Priyanka Agrawal, and Aman Hussain.  Omninet: A unified architecture for multi-modal multi-task learning.  arXiv preprint arXiv:1907.07804, 2019.
  * [62] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [63] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  The Journal of Machine Learning Research, 21(1):5485–5551, 2020\.
  * [64] Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Manfred Pinkal.  Grounding action descriptions in videos.  Transactions of the Association for Computational Linguistics, 1:25–36, 2013.
  * [65] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28, 2015.
  * [66] Cristian Rodriguez, Edison Marrese-Taylor, Fatemeh Sadat Saleh, Hongdong Li, and Stephen Gould.  Proposal-free temporal moment localization of a natural-language query in video using guided attention.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2464–2473, 2020.
  * [67] Suman Saha, Gurkirt Singh, Michael Sapienza, Philip HS Torr, and Fabio Cuzzolin.  Deep learning for detecting multiple space-time action tubes in videos.  arXiv preprint arXiv:1608.01529, 2016.
  * [68] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565, 2018.
  * [69] Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta.  Hollywood in homes: Crowdsourcing data collection for activity understanding.  In European Conference on Computer Vision, pages 510–526. Springer, 2016.
  * [70] Gurkirt Singh, Suman Saha, Michael Sapienza, Philip HS Torr, and Fabio Cuzzolin.  Online real-time multiple spatiotemporal action localisation and prediction.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3637–3646, 2017.
  * [71] Mattia Soldan, Mengmeng Xu, Sisi Qu, Jesper Tegner, and Bernard Ghanem.  Vlg-net: Video-language graph matching network for video grounding.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3224–3234, 2021.
  * [72] Deepak Sridhar, Niamul Quader, Srikanth Muralidharan, Yaoxin Li, Peng Dai, and Juwei Lu.  Class semantics-based attention for action detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13739–13748, 2021.
  * [73] Gjorgji Strezoski, Nanne van Noord, and Marcel Worring.  Many task learning with task routing.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1375–1384, 2019.
  * [74] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  Vl-bert: Pre-training of generic visual-linguistic representations.  arXiv preprint arXiv:1908.08530, 2019.
  * [75] Hao Tan and Mohit Bansal.  Lxmert: Learning cross-modality encoder representations from transformers.  arXiv preprint arXiv:1908.07490, 2019.
  * [76] Jing Tan, Jiaqi Tang, Limin Wang, and Gangshan Wu.  Relaxed transformer decoders for direct action proposal generation.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 13526–13535, 2021.
  * [77] Jiajun Tang, Jin Xia, Xinzhi Mu, Bo Pang, and Cewu Lu.  Asynchronous interaction aggregation for action detection.  In European Conference on Computer Vision, pages 71–87. Springer, 2020.
  * [78] Yee Teh, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu.  Distral: Robust multitask reinforcement learning.  Advances in neural information processing systems, 30, 2017.
  * [79] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang.  VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training.  In Advances in Neural Information Processing Systems, 2022.
  * [80] Du Tran and Junsong Yuan.  Max-margin structured output regression for spatio-temporal action localization.  Advances in neural information processing systems, 25, 2012.
  * [81] Jingwen Wang, Lin Ma, and Wenhao Jiang.  Temporally grounding language queries in videos by contextual boundary-aware prediction.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12168–12175, 2020.
  * [82] Weining Wang, Yan Huang, and Liang Wang.  Language-driven temporal activity localization: A semantic matching reinforcement learning model.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 334–343, 2019.
  * [83] Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime Carbonell.  Characterizing and avoiding negative transfer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
  * [84] Philippe Weinzaepfel, Zaid Harchaoui, and Cordelia Schmid.  Learning to track for spatio-temporal action localization.  In Proceedings of the IEEE international conference on computer vision, pages 3164–3172, 2015.
  * [85] Huijuan Xu, Abir Das, and Kate Saenko.  R-c3d: Region convolutional 3d network for temporal activity detection.  In Proceedings of the IEEE international conference on computer vision, pages 5783–5792, 2017.
  * [86] Huijuan Xu, Kun He, Bryan A Plummer, Leonid Sigal, Stan Sclaroff, and Kate Saenko.  Multilevel language and vision integration for text-to-clip retrieval.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9062–9069, 2019.
  * [87] Mengmeng Xu, Yanghao Li, Cheng-Yang Fu, Bernard Ghanem, Tao Xiang, and Juan-Manuel Perez-Rua.  Where is my wallet? modeling object proposal sets for egocentric visual query localization.  arXiv preprint arXiv:2211.10528, 2022.
  * [88] Mengmeng Xu, Chen Zhao, David S Rojas, Ali Thabet, and Bernard Ghanem.  G-tad: Sub-graph localization for temporal action detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10156–10165, 2020.
  * [89] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid.  Tubedetr: Spatio-temporal video grounding with transformers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16442–16453, 2022.
  * [90] Le Yang, Houwen Peng, Dingwen Zhang, Jianlong Fu, and Junwei Han.  Revisiting anchor mechanisms for temporal action localization.  IEEE Transactions on Image Processing, 29:8535–8548, 2020.
  * [91] Xitong Yang, Xiaodong Yang, Ming-Yu Liu, Fanyi Xiao, Larry S Davis, and Jan Kautz.  Step: Spatio-temporal progressive learning for video action detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 264–272, 2019.
  * [92] Yitian Yuan, Tao Mei, and Wenwu Zhu.  To find where you talk: Temporal sentence localization in video with attention based location regression.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9159–9166, 2019.
  * [93] Runhao Zeng, Wenbing Huang, Mingkui Tan, Yu Rong, Peilin Zhao, Junzhou Huang, and Chuang Gan.  Graph convolutional networks for temporal action localization.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 7094–7103, 2019.
  * [94] Chen-Lin Zhang, Jianxin Wu, and Yin Li.  Actionformer: Localizing moments of actions with transformers.  In European Conference on Computer Vision, pages 492–510. Springer, 2022.
  * [95] Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou.  Span-based localizing network for natural language video localization.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6543–6554, Online, July 2020. Association for Computational Linguistics.
  * [96] Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo.  Learning 2d temporal adjacent networks for moment localization with natural language.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12870–12877, 2020.
  * [97] Tianzhu Zhang, Bernard Ghanem, Si Liu, and Narendra Ahuja.  Robust visual tracking via structured multi-task sparse learning.  International journal of computer vision, 101(2):367–383, 2013\.
  * [98] Zhu Zhang, Zhou Zhao, Yang Zhao, Qi Wang, Huasheng Liu, and Lianli Gao.  Where does it exist: Spatio-temporal video grounding for multi-form sentences.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10668–10677, 2020.
  * [99] Chen Zhao, Ali K Thabet, and Bernard Ghanem.  Video self-stitching graph network for temporal action localization.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13658–13667, 2021.
  * [100] Jiaojiao Zhao and Cees GM Snoek.  Dance with flow: Two-in-one stream action detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9935–9944, 2019.
  * [101] Jiaojiao Zhao, Yanyi Zhang, Xinyu Li, Hao Chen, Bing Shuai, Mingze Xu, Chunhui Liu, Kaustav Kundu, Yuanjun Xiong, Davide Modolo, et al.  Tuber: Tubelet transformer for video action detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13598–13607, 2022.
  * [102] Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin.  Temporal action detection with structured segment networks.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2914–2923, 2017.
  * [103] Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hongsheng Li, Xiaohua Wang, and Jifeng Dai.  Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16804–16815, 2022.

The supplementary is structured in the following way:

  * •

Section A details complete comparison to SOTA

  * •

Section B discusses pair-wise joint and transfer learning results

  * •

Section C discusses details of zero-shot spatio-temporal grounding on NLQ

  * •

Section D presents results on a novel task: Semi-Supervised Video Object
Segmentation

  * •

Section E discusses implementation details

  * •

Section F presents additional ablation studies

  * •

Section G shows qualitative results on VQ2D

  * •

Section H shows qualitative results on NLQ

##  Appendix A Comparison to Existing Work

|  |  | VQ2D | NLQ | MQ
---|---|---|---|---|---
|  |  |  |  |  |  | tIoU=0.3 | tIoU=0.5 | tIoU=0.5
|  | Methods | $\text{tAP}_{25}$ | $\text{stAP}_{25}$ | rec$\%$ | Succ | R@1 | R@5 | R@1 | R@5 | R@1x
| 1 | Siam-RCNN [20] | 0.21 | 0.13 | 34.0 | 41.6 | - | - | - | - | -
without additional large-scale pre-training | 2 | VSLNet [20, 95] | - | - | - | - | 5.47 | 11.21 | 2.80 | 6.57 | -
3 | VSGN [20, 99] | - | - | - | - | - | - | - | - | 24.25
4 | Xu et al. [87] | 0.26 | 0.18 | 43.2 | 48.1 | - | - | - | - | -
5 | EgoVLP: VSLNet [39] | - | - | - | - | 4.87 | 8.67 | 2.50 | 4.97 | -
6 | EgoVLP: VSGN [39] | - | - | - | - | - | - | - | - | 19.74
7 | InternVideo [8] | - | - | - | - | - | - | - | - | 27.71
8 | ActionFormer [53] | - | - | - | - | - | - | - | - | 24.25
9 | Ours: All-Tasks (AT) | 0.41 | 0.19 | 26.5 | 60.6 | 7.47 | 19.61 | 4.85 | 12.09 | 24.62
10 | Ours: AT $\ \rightarrow\ $ Tasks | 0.42 | 0.21 | 29.0 | 61.4 | 8.07 | 20.63 | 5.52 | 12.94 | 24.44
with additional large-scale pre-training | 11 | EgoVLP: VSLNet [39] | - | - | - | - | 10.46 | 16.76 | 6.24 | 11.29 | -
12 | EgoVLP: VSGN [39] | - | - | - | - | - | - | - | - | 28.03
13 | InternVideo [8] | - | - | - | - | 16.45 | 22.95 | 10.06 | 16.10 | -
14 | InternVideo [8] | - | - | - | - | - | - | - | - | 41.13
15 | ActionFormer [53] | - | - | - | - | - | - | - | - | 42.54

Table 6: Comparison with existing works on test set with and without
additional large scale pre-training. In the main paper (Table 4), we presented
apples-to-apples comparison between MINOTAUR and related works using the same
task-only data (i.e., VQ2D, NLQ and MQ) without additional large-scale pre-
training. Here, we show results on related approaches that leverage additional
large-scale pre-training on $3.85$M Ego4D video-text corpus (EgoCLIP) to
obtain better performance. Although, pre-training with additional data has the
potential to benefit our model too, but it’s not the focus of this paper.

As discussed in Section 4.3 and Table 4 of the main paper, we reported methods
trained only on the task-specific data, i.e., VQ2D, NLQ and MQ tasks of Ego4D
[20] for fair comparison. We did this to differentiate between approaches that
use 1.) only task-specific data, such as our own, and 2.) task-specific data
with additional large-scale pre-training on 3.85M video-text Ego4D corpus,
where the methods in the latter case are positioned to leverage additional
data and perform better. In this section, we provide details of the methods
that perform large-scale pre-training and present results in Table 6 for
methods with and without additional large-scale pre-training.

Additional large-scale pre-training. EgoVLP [39] proposes to use EgoCLIP
dataset which is chosen from the broad Ego4D [20] data and contains $3.85$M
video-text pairs covering egocentric daily human activities, with the idea
that pre-training on this additional data would benefit task-specific
benchmarks. Indeed we observe notable improvements in performance (rows 11 \-
15), however, we note that our approach is trained only on the task-specific
data (VQ2D, NLQ and MQ), which is orders of magnitude smaller than EgoCLIP
($60$K v/s $3.85$M). Pre-training with additional data also has the potential
to benefit our model, but it’s not the focus of this paper.

|  |  | VQ2D | NLQ | MQ |
---|---|---|---|---|---|---
|  |  | $\text{tAP}_{25}$ | $\text{stAP}_{25}$ | rec$\%$ | Succ | tIoU=0.3 | tIoU=0.5 | tIoU=0.3 | tIoU=0.5 |  | # params
---
(# models)
|  |  | R@1 | R@5 | R@1 | R@5 | R@1x | R@5x | R@1x | R@5x |
Joint | 1 | Ego4D [20] | 0.20 | 0.12 | 32.2 | 39.8 | 5.45 | 10.74 | 3.12 | 6.63 | 33.45 | 58.43 | 25.16 | 46.18 |
2 | Single-Task | 0.41 | 0.21 | 29.4 | 61.2 | 6.53 | 17.22 | 3.61 | 9.58 | 33.64 | 55.38 | 23.86 | 39.48 | 432M (3)
3 | MQ + NLQ | - | - | - | - | 7.72 | 19.80 | 4.75 | 11.82 | 32.68 | 54.16 | 24.27 | 41.41 | 185M (1)
4 | MQ + VQ2D | 0.41 | 0.20 | 27.7 | 60.2 | - | - | - | - | 33.87 | 56.35 | 25.47 | 42.32 | 186M (1)
5 | NLQ + VQ2D | 0.40 | 0.19 | 27.7 | 60.0 | 6.17 | 16.86 | 3.67 | 10.12 | - | - | - | - | 186M (1)
6 | All-Tasks (AT) | 0.41 | 0.19 | 26.4 | 60.2 | 7.74 | 21.14 | 4.78 | 12.60 | 34.82 | 56.68 | 25.58 | 42.30 | 186M (1)
Transfer | 7 | NLQ $\ \,\,\rightarrow$ MQ | - | - | - | - | - | - | - | - | 32.70 | 55.17 | 23.67 | 40.78 | 185M (1)
8 | VQ2D $\rightarrow$ MQ | - | - | - | - | - | - | - | - | 35.75 | 56.17 | 26.05 | 41.78
9 | MQ $\quad\rightarrow$ NLQ | - | - | - | - | 6.94 | 19.93 | 4.13 | 11.72 | - | - | - | - | 185M (1)
10 | VQ2D $\rightarrow$ NLQ | - | - | - | - | 6.69 | 17.73 | 4.26 | 10.66 | - | - | - | -
11 | MQ $\ \,\rightarrow$ VQ2D | 0.40 | 0.21 | 29.1 | 60.4 | - | - | - | - | - | - | - | - | 61M (1)
12 | NLQ $\rightarrow$ VQ2D | 0.42 | 0.20 | 29.2 | 61.1 | - | - | - | - | - | - | - | -
13 | AT $\ \ \rightarrow$ Tasks | 0.42 | 0.22 | 29.4 | 61.1 | 7.69 | 21.17 | 5.21 | 12.98 | 33.82 | 56.61 | 25.21 | 42.83 | 432M (3)

Table 7: Pair-wise joint and transfer learning results on validation set. We
explore pair-wise relations between the tasks and found better synergy between
MQ and VQ2D compared to MQ and NLQ, by observing both joint (row 3 vs 4) and
transfer (row 7 vs 8) results. And similarly, we found better synergy between
NLQ and MQ compared to NLQ and VQ2D using both joint (row 3 vs 5) and transfer
(row 9 vs 10) results. We also observe negative synergy between NLQ and VQ2D
(row 5) where joint-training leads both the tasks to underperform w.r.t.
Single-Task (row 2).

##  Appendix B Pair-wise joint training and transfer results.

In the main paper, we presented joint-training results (All-Tasks model) and
compare it with Single-Task models in Section 4.1 and Table 1. Here, we
present an exhaustive analysis of synergies among the tasks by exploring all
pair-wise joint and transfer learning results in Table 7.

Pair-wise joint-training. We explore all pair-wise combinations among the
three tasks, shown in rows 3 \- 5. Compared to the Single-Task models on the
same data in row 2, we observe performance improvements for $2$ out of the $3$
tasks, namely, for NLQ and MQ; and for VQ2D the performance stays competitive
to the Single-Task (row 1) and decreases slightly w.r.t. spatio-temporal
performance. Furthermore, we found performance of NLQ in the case of MQ + NLQ
(row 3) to be higher than NLQ + VQ2D (row 5), which indicates greater synergy
between NLQ and MQ tasks. On the other hand, we found performance of MQ in the
case of MQ + VQ2D (row 4) to be higher than MQ + NLQ (row 3), which indicates
greater synergy between MQ and VQ2D. We also found negative transfer in the
case of NLQ + VQ2D (row 5), where both the tasks’ performances reduce w.r.t.
Single-Task (row 2).

Moreover, the total number of models required for all three tasks drops to $2$
from $3$, since we can pick any two of the three pair-wise models, and the
total number of parameters drops to approx $372$M ($2\times 186$) from $432$M
which is a factor of $1.16\times$ drop.

Pair-wise transfer-training. We also explore pair-wise transfer between the
tasks where a model trained on one task is used as an initialization for the
other task, shown in rows 7 \- 12. We observe that the transfer outperforms
Single-Task performance in $4$ out of $6$ cases. For MQ (rows 7 \- 8), we
observe better transfer performance from VQ2D compared to NLQ, which also
confirms better synergy results between MQ and VQ2D from the above joint-
training results. One possible explanation is that the queries in MQ are from
a set of $110$ activity templates in the form of “When did I do activity c?”,
so its reliance on text encoder is low and sees more gain in performance from
the visual task than its language counterpart. And for NLQ (rows 9 \- 10), we
observe better transfer performance from MQ compared to VQ2D, which also
confirms the negative transfer between NLQ and VQ2D in the above joint-
training results. For VQ2D (rows 11 \- 12), we don’t observe significant
differences w.r.t. Single-Task performance (row 2).

##  Appendix C Zero-shot spatio-temporal grounding

In the main paper (Section 4.1), we presented evaluation of zero-shot spatio-
temporal prediction capability of All-Tasks model on NLQ task. Since, the NLQ
data is annotated only with (language query, temporal target) pairs, we
labelled spatial bounding-boxes for a subset of validation videos from NLQ, to
serve as the benchmark for evaluating spatio-temporal predictions. We discuss
the details of annotation below.

Selection of videos. Given that we have $13$ question templates in the NLQ
task, we uniformly sub-sample validation videos from each template. We
randomly choose $10$ videos per template, amounting to $130$ videos in total
(out of total $3.8$K validation data videos).

Category | Template |  | Num videos
---
selected
| Num frames
---
annotated
| Total duration
---
annotated (in secs)
Objects | Where is object X before / after event Y? | 10 | 127 | 25.4
Where is object X? | 10 | 121 | 24.2
What did I put in X? | 10 | 155 | 31
How many X’s? (quantity) | 10 | 157 | 31.4
What X did I Y? | 10 | 119 | 23.8
In what location did I see object X? | 10 | 140 | 28
What X is Y? | 10 | 152 | 30.4
State of an object | 10 | 149 | 29.8
Where is my object X? | 10 | 149 | 29.8
Place | Where did I put X? | 10 | 96 | 19.2
People | Who did I interact with when I did activity X? | 10 | 100 | 20
Who did I talk to in location X? | 10 | 142 | 28.4
When did I interact with person with role X? | 10 | 150 | 30
(mean $\pm$ std) | (10 $\pm$ 0) | (135.15 $\pm$ 19.85) | (27.03 $\pm$ 3.97)
Total | 130 | 1757 | 351.4

Table 8: Selection of videos for spatial-annotation for NLQ task. We uniformly
sample $10$ videos across $13$ question templates and annotate spatial
bounding-boxes for each ground-truth extent. In total, we annotated bounding-
boxes for $1757$ frames or $351.4$ secs of video footage where the answers to
NLQ queries can be found.

Spatial annotation. Shown in Table 8, we provide breakdown of the number of
frames and duration of spatial bounding-boxes annotated per question template.
We note that each question template is appropriately represented in the chosen
subset, and there isn’t any significant skewness in the data, where the mean
and standard deviation of durations are $27.03$ and $3.97$ secs respectively.
Overall, we annotated $1757$ frames in total at $5$ fps, which amounts to
$351.4$ secs of spatially annotated video content, with the help of Make
Sense222https://www.makesense.ai/ web application.

| Spatio-temporal | Temporal
---|---|---
Model |  spatial branch | stIoU=0.3 |  mean stIoU | tIoU=0.3 |  mean tIoU
‘ | R@1 | R@5 | R@1 | R@5
NLQ-only | N/A | - | - | - | 9.30 | 18.60 | 5.35
MINOTAUR (All-Tasks) | random boxes | 0 $\pm$ 0 | 0 $\pm$ 0 | 0.40 $\pm$ 0.04 | 11.63 | 20.93 | 8.35
| random centered
---
boxes
0 $\pm$ 0 | 0.47 $\pm$ 0.38 | 1.25 $\pm$ 0.03
All-Tasks | 2.33 | 4.65 | 2.27

Table 9: Zero-shot spatio-temporal grounding on NLQ. In addition to the Table
3 in the main paper, we report recall numbers on temporal metrics. We evaluate
spatio-temporal predictions on NLQ by annotating a subset of validation videos
($=130$), and comparing it with random baselines. We observe non-trivial
performance despite the model not trained on (language query, spatio-temporal
target).

Evaluation. We use the standard evaluation protocol for NLQ, and extended it
to account for spatial predictions as well, thereby extending the temporal-
only metrics to spatio-temporal metrics. The complete table is shown in Table
9 where we also provide Recall scores for temporal-only evaluation. We note
that despite evaluating on a small subset of validation videos, we observe
trends consistent with the full validation set shown in Table 1 of the main
paper, where the temporal-only performance of All-Tasks model is better than
Single-Task model.

Qualitative predictions. In Figure 5, we show a handful of example spatial
predictions and ground-truth on the validation subset of NLQ task. We note
that to show zero-shot spatial prediction capability of the model, we plot
spatial predictions on the frames belonging to ground-truth regardless of
temporal prediction.

Figure 5: Zero-shot Spatio-temporal predictions for NLQ with the _All-Tasks
(AT)_ model. We show spatio-temporal predictions on a small subset of NLQ val
set where the ground-truth bounding-boxes are annotated separately (not
provided with the dataset). Predictions and ground-truth bounding boxes are
shown with yellow and green borders respectively. Although the model has not
been trained with ground-truth spatio-temporal tubes for language queries,
multi-task training results in some meaningful results for zero-shot spatio-
temporal grounding. The last two rows show failure cases.

##  Appendix D Semi-supervised Video Object Segmentation (Semi-VOS)

In this section, we experiment with an additional benchmark task in egocentric
domain to test extensibility of MINOTAUR to additional video structured
prediction tasks, and generalizability of the features learned by All-Tasks
model on egocentric data domain. We choose Semi-Supervised Video Object
Segmentation (Semi-VOS) task for this purpose.

Task specification. Given a video sequence and a ground-truth object mask in
the first frame of the sequence, the Semi-VOS task requires segmenting the
ground-truth object mask in the following frames of the sequence [60]. We
selected VISOR [16] as our benchmark dataset, as it is based on egocentric
data [15], which is consistent with the data domain of Ego4D [20].

Approach. We follow the proposed architecture with an additional prediction
head in the form of a segmentation head provided by DETR [7]. The only
difference from prior prediction heads for foreground, start and end time and
spatial bounding box is that the segmentation head uses multi-scale image
features to generate segmentation masks, and it readily extends using our
MINOTAUR framework.

Evaluation metrics. Following standard protocols in DAVIS [60, 16], we use
Jaccard Index/Intersection over Union ($\mathcal{J}$) and Boundary F-Measure
($\mathcal{F}$) metrics.

Method | Pretraining |  $\mathcal{J}$&$\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$
---|---|---|---|---
STM | None | 62.8 | 60.6 | 64.9
COCO | 75.8 | 73.6 | 78.0
MINOTAUR | ImageNet | 16.34 | 12.11 | 20.58
MDETR | 70.95 | 69.05 | 72.86
| Ego4D
---
(All-Tasks)
73.15 | 71.32 | 74.98
Table 10: Semi-Supervised Video Object Segmentation results on VISOR [16]
validation set. We compare with the STM [55] baseline reported in VISOR [16].
We observe that, 1) our method (All-Tasks) performs comparably to STM [55]
which is pre-trained on additional segmentation data (from COCO), and 2) our
Ego4D (All-Tasks) initialization outperforms MDETR [29] initialization, which
shows the generic and expressive nature of our features for other tasks at
least in egocentric domain.

Comparison with baselines. We report results in Table 10. First, we compare
with Space-Time Memory Networks (STM) [55] baseline reported in VISOR [16]. We
observe that our method (All-Tasks) performs comparably to STM [55] which is
pre-trained on COCO segmentation masks, while our method has not been pre-
trained on any segmentation data. Second, our Ego4D (All-Tasks) initialization
outperforms MDETR [29] initialization, which shows the generic and expressive
nature of our features for other tasks at least in egocentric domain.

Figure 6: Additional results on Semi-Supervised Video Object Segmentation
(Semi-VOS) task on VISOR [16] dataset. Given a visual crop of an object mask
as query (shown in left), the Semi-VOS task requires outputs as segmentation
masks of the object in the video sequence (shown as Ground-truth). We train
MINOTAUR with an additional segmentation head [7] on the task with 1.) MDETR
[29] and 2.) All-Tasks initializations, and found the latter to perform better
(with quantitative results shown in Table 10).

Example predictions. We show a couple of example predictions along with the
ground-truth in Figure 6, where we compare models trained with MDETR [29] and
All-Tasks initialization.

##  Appendix E Implementation details

Identification of peaks. With reference to Task-specific inference in the main
paper (Section 3.1 and Figure 3), during inference, we choose peaks by simple
thresholding over the foreground scores $\hat{\mathbf{f}}\in\mathbb{R}^{T}$,
where $T$ is the number of frames in a video. Following Siam-RCNN [20], we
first apply a median filter, medfilt
333https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.medfilt.html,
with kernel size of $5$. For VQ2D, we select the peaks with scores above $0.5$
threshold and pick the latest peak among them to form predictions. For MQ and
NLQ, since we need to retrieve multiple predictions, we select all the peaks
above $0.1$ threshold (maximum $1000$ in total), and rank them according to
their scores.

Forming predictions from peaks. As mentioned in the main paper (Section 3.1),
upon selection of the peak(s), we form predictions by sampling a window size
of $w$ frames centered around every peak, and predict start and end frame,
$\hat{s}$ and $\hat{e}$, by considering only the predictions within the
window. We do this by masking out logits $\hat{\boldsymbol{\tau}}_{s}$ and
$\hat{\boldsymbol{\tau}}_{e}$ outside the window. We then form start and end
probabilities, $\hat{\mathbf{p}}_{s},\hat{\mathbf{p}}_{e}\in\mathbb{R}^{w}$
specific to the window by taking softmax, and picking $\hat{s},\hat{e}$ by
taking argmax over the joint probability such that $\hat{e}>\hat{s}$.

For VQ2D, we choose a window of size $70$ frames centered around the chosen
peak to form prediction. For NLQ and MQ, we perform _test-time augmentation_
by accumulating predictions at different frames-per-second (fps) to capture
longer extents. For NLQ, we accumulate at fps $\\{1,5\\}$, and, for MQ, we
accumulate at fps: $\\{0.2,0.5,1,1.66,5\\}$, which we empirically found to
perform best for validation set. We use NMS [65] with the threshold $0.4$ to
disambiguate repeated or heavily overlapping entries.

Optimization details. In addition to the implementation details discussed in
the main paper (Section 4), we use AdamW as the optimizer with learning rate
for visual backbone as $1e^{-5}$, text encoder $5e^{-5}$ and the remaining
components $5e^{-5}$. We train models for $25$ epochs with learning rate drop
by a factor of $10$ at every $10^{th}$ epoch.

##  Appendix F Additional ablations

Strategy | VQ2D | NLQ | MQ
---|---|---|---
tAP25 | stAP25 | rec | Succ | tIoU=0.3 | tIoU=0.5 | tIoU=0.3 | tIoU=0.5
R@1 | R@5 | R@1 | R@5 | R@1x | R@5x | R@1x | R@5x
Concat | 0.41 | 0.20 | 27.1 | 59.3 | 6.20 | 17.86 | 3.98 | 10.92 | 31.54 | 53.51 | 21.97 | 39.64
Round-Robin | 0.41 | 0.19 | 26.4 | 60.2 | 7.56 | 20.06 | 4.62 | 11.80 | 33.28 | 54.93 | 24.09 | 40.27
Table 11: Sampling strategy for multi-task learning (Round Robin v/s Concat).
We train our All-Tasks (AT) approach using different sampling strategies and
evaluate on validation set. We found that Round-Robin performs better than
Concat on $10$ out of $12$ metrics across the three tasks.

Batch-level sampling: Round Robin v/s Concat. In multi-task training, we can
combine different tasks, by either concatenating the datasets together
(_Concat_), or we can sample batches in _Round-Robin_ fashion such that each
dataset is sampled equally even though the individual dataset sizes are
different. As shown in Table 11, we found that _Round-Robin_ performs better
on $10$ out of $12$ metrics across the three tasks.

MQ: fps 5 v/s fps 1. As mentioned earlier that we train using fps 5 for VQ2D
and fps 1 for NLQ and MQ. We show results for fps 5 on MQ in Table 12. We
found fps 1 to perform better, especially for MQ, since lower fps enables
capturing longer temporal extents during training, and out of the three tasks
MQ has the longest ground truth temporal extents, followed by NLQ and then
VQ2D.

| Methods | MQ
---|---|---
| tIoU=0.3 | tIoU=0.5
| R@1x | R@5x | R@1x | R@5x
Single-Task | fps 5 | 17.55 | 40.82 | 12.12 | 28.49
fps 1 | 33.64 | 55.38 | 23.86 | 39.48

Table 12: MQ: fps 5 v/s fps 1. We obtain better performance on MQ task with
lower fps, since lower fps allows capturing longer temporal context, and this
works well for MQ since out of the three tasks, MQ has the longest ground-
truth extents.

Step size of sliding windows. During evaluation, we accumulate predictions on
long videos in a sliding-window fashion, where a window of size $w$ frames is
used to gather predictions with step size of $k_{step}$ frames. In case of
overlap between consecutive windows (or $k_{step}<w$), we average the
predictions,
$[\hat{\mathbf{b}},\hat{\boldsymbol{\tau}}_{s},\hat{\boldsymbol{\tau}}_{e},\hat{\mathbf{f}}]\in\mathbb{R}^{T\times
7}$, across overlapping frames. We show how different step size $k_{step}$
affects performance in Table 13. In general, accumulating predictions with
lower step size leads to better performance.

|  Step size $k_{step}$ | MQ
---|---|---
| tIoU=0.3 | tIoU=0.5
| R@1x | R@5x | R@1x | R@5x
All-Tasks | 200 | 30.49 | 51.93 | 20.01 | 37.31
100 | 33.28 | 54.93 | 24.09 | 40.27
50 | 34.82 | 56.68 | 25.58 | 42.49

Table 13: Effect of step size $k_{step}$ during evaluation on MQ. We observe
that with more overlap between sliding windows (or lower step size
$k_{step}$), we obtain better performance.

##  Appendix G Additional Qualitative Results on VQ2D

Please find the qualitative results in Figure 7.

Figure 7: Qualitative results on VQ2D. Given a visual crop of an object, the
task requires localizing the most recent occurrence of the object. For each
example, we show query crop, plot of foreground score ($\mathbf{\hat{f}}$) (in
yellow) and ground-truth extent (in green), and frames with predicted (in
blue) and ground-truth bounding boxes (in green). In (a), with a video of
relatively short duration ($248$ frames), the model is able to correctly
predict the object which is towards the beginning of the video with ground-
truth frames: $[20,34]$ and predicted frames: $[19,37]$, which can also be
seen in the foreground score plot. In (b), the video is of longer duration
($1223$ frames), and the model is able to correctly identify the object with
ground-truth frames: $[1191,1213]$ and predicted frames: $[1189,1215]$.
However, note the appearance of the object before the ground-truth in frame
$1134$ which can also be seen as a peak in the foreground score plot, but the
model made prediction using the latest peak. Lastly, (c) shows a failure case
($909$ frames), where the model incorrectly identifies a “red looking” object
as the prediction.

##  Appendix H Additional Qualitative Results on NLQ

Please find the qualitative results in Figure 8.

Figure 8: Qualitative results on NLQ. Given a language query, the task
requires temporally localizing the extent of the video where the answer can be
found. For each example, we show language query, plot of foreground score
($\mathbf{\hat{f}}$) (in yellow) and ground-truth extent (in green), and
top-$k$ predictions; and frames with predicted bounding boxes (in yellow) and
ground-truth temporal extent (in green). We note that the model is able to
predict spatial bounding boxes, athough the training data for (language query,
spatio-temporal target) was not provided. In (a), we show that the top-$1st$
prediction (frames: $[1049,1063]$) and ground-truth (frames: $[1050,1085]$)
extents overlap with each other. We also show top-$2nd$ (frames: $[544,571]$)
and top-$3rd$ (frames: $[1835,1840]$) predictions as well. In (b), we show
that the top-$5th$ prediction (frames: $[1731,1760]$) and ground-truth
(frames: $[1735,1762]$) extents overlap with each other. We also show
top-$1st$ (frames: $[2000,2059]$), top-$2nd$ (frames: $[1877,2059]$), and
top-$3rd$ (frames: $[1874,1900]$) predictions as well. Notice that we have
overlapping predictions owing to the nature of the tasks and multi-scale
predictions.