# FrameExit: Conditional Early Exiting for Efficient Video Recognition

Amir Ghodrati Babak Ehteshami Bejnordi11footnotemark: 1 Amirhossein Habibian
Qualcomm AI Research
{ghodrati, behtesha<EMAIL_ADDRESS>Equal contribution Qualcomm AI
Research is an initiative of Qualcomm Technologies, Inc

###### Abstract

In this paper, we propose a conditional early exiting framework for efficient
video recognition. While existing works focus on selecting a subset of salient
frames to reduce the computation costs, we propose to use a simple sampling
strategy combined with conditional early exiting to enable efficient
recognition. Our model automatically learns to process fewer frames for
simpler videos and more frames for complex ones. To achieve this, we employ a
cascade of gating modules to automatically determine the earliest point in
processing where an inference is sufficiently reliable. We generate on-the-fly
supervision signals to the gates to provide a dynamic trade-off between
accuracy and computational cost. Our proposed model outperforms competing
methods on three large-scale video benchmarks. In particular, on
ActivityNet1.3 and mini-kinetics, we outperform the state-of-the-art efficient
video recognition methods with $1.3\times$ and $2.1\times$ less GFLOPs,
respectively. Additionally, our method sets a new state of the art for
efficient video understanding on the HVU benchmark.

##  1 Introduction

With the massive growth in the generation of video content comes an increasing
demand for efficient and scalable action or event recognition in videos. The
astounding performance of deep neural networks for action recognition [5, 47,
62, 12, 46, 60] are obtained by densely applying 2D [60, 35, 62, 13] or 3D
[44, 5, 18, 12] models over video frames. Despite demonstrating top-notch
performance in recognizing complex and corner case actions, the high data
volumes, compute demands, and latency requirements, limit the application of
the state-of-the-art video recognition models on resource-constrained devices.

Figure 1: Efficient video recognition by early exiting. Our proposed method
adjusts the amount of computation to the difficulty of the input, allowing for
significant reduction of computational costs. Videos are adopted from [22, 24,
25].

Extensive studies have been conducted to remedy this issue by designing
efficient and light-weight architectures [39, 11, 47, 40, 57, 63, 46, 32, 35,
10]. These models have a static computational graph and treat all the videos
equally regardless of how complex or easy they are for recognition and hence
yield sub-optimal results. A large body of research has been focusing on
selecting a subset of salient frames to efficiently process the video
conditioned on the input [59, 56, 61, 8, 15, 33]. Current methods for frame
selection rely on learning a policy function to determine what action should
be taken on the selected frame (e.g. process by a heavy recognition model
[55], process at a specific spatial resolution [37], etc.). Most of these
methods either rely on the assumption that salient frames for the sampler
network are also salient for the recognition network [33, 15] or require
carefully selected reward functions in case of using policy gradient methods
[59, 56, 53]. Moreover, the sampler network may create an additional
computational overhead to the model.

An alternative promising direction to reduce the computational complexity of
analyzing video content is conditional compute using early exiting. Early
exiting has recently been explored for image classification tasks by inserting
a cascade of intermediate classifiers throughout the network [34, 43, 58, 28].
In this line of work, the model adjusts the amount of computation to the
difficulty of the input and allows for significant reduction of computational
requirements. Inspired by that, we design an efficient video recognition model
that performs automatic early exiting by adjusting the computational budget on
a per-video basis. Our motivation is that a few frames are sufficient for
classifying “easy” samples, and only some “hard” samples need temporally
detailed information (see Figure 1).

In this paper, we propose FrameExit, a conditional early exiting framework
with learned gating units that decide to stop the computation when an
inference is sufficiently reliable. FrameExit has $T$ classifiers accompanied
by their associated gates that are attached at different time steps to allow
early exiting. The gates are learned in a self-supervised fashion to control
the trade-off between model accuracy and total computation costs. We use the
recognition loss as a proxy to generate on-the-fly pseudo-labels to train the
gates.

Additionally, our early exiting mechanism combined with a simple,
deterministic sampling strategy obviates the need for complex sampling policy
functions and yet achieves excellent recognition performance. Finally, we
propose an accumulated feature pooling module to generate video
representations that enable more reliable predictions by the model. Our
contributions are as follows:

  * •

We propose a method that employs a simple, deterministic frame sampling
strategy, combined with an accumulated feature pooling module to obtain
accurate action recognition results.

  * •

We propose a conditional early exiting framework for efficient video
recognition. We use a cascade of gating modules to determine when to stop
further processing of the video. The gates adapt the amount of computation to
the difficulty of an input video, leading to significant reductions in
computational costs.

  * •

We show state-of-the-art performance on three large-scale datasets. In all
cases, we greatly improve the inference efficiency at a better or comparable
recognition accuracy. In particular, on the HVU [7] dataset, we report
$5\times$ reduction in computation costs while improving the recognition
accuracy upon the state-of-the-art methods.

##  2 Related work

Efficient Video Recognition: There are two lines of approaches for efficient
video recognition. The first focuses on proposing new lightweight video
architectures. This can be obtained by decomposing 3D filters into separate 2D
spatial and 1D temporal filters [47, 40, 57], extending efficient 2D
architectures to their 3D counterparts [46, 32], using shifting operations
[35, 9], or exploring neural architecture search for videos [39, 11]. Our
method is agnostic to the network architecture and can be complementary to
these types of methods.

The second line of works focus on saving compute by selecting salient
frames/clips using a policy function parametrized with a neural network (CNN
or RNN). This is commonly done by training agents to find which frame to
observe next [59, 56, 61, 8], ranking frames based on saliency [33], gating
frames [29], skipping RNN states [4], or by using audio to select relevant
frames [15, 33]. LiteEval [55] proposes binary gates for selecting coarse or
fine features. At each time step, LiteEval computes coarse features with a
lightweight CNN to determine whether to examine the current frame more
carefully using a heavy CNN. Meng  _et al_. [37] propose to adaptively select
the input resolution, on a per-frame basis to further balance the accuracy vs
compute. However, using policy networks in these methods may come with
additional memory and compute overhead. Moreover, optimization is sub-optimal
if policy and recognition networks are not trained jointly [33, 15], or
requires carefully selected reward functions if policy gradient methods are
used [56, 59, 53]. In contrast, our method relies on a single recognition
network and does not require complex RL optimization for frame sampling. We
formulate the problem in the early-exiting framework and show that a simple
sampling strategy, if combined with a proper exiting function, leads to
excellent recognition performance.

Figure 2: The overview of FrameExit. Given a video, at each time step $t$, we
sample a frame from the video using the deterministic policy function
$\mathbf{\pi}$. Each frame is independently represented by the feature
extraction network $\Phi$ and is aggregated to features of previous time steps
using the accumulated feature pooling module (for $t>1$). The gating modules
($g_{t}$) are trained to allow the network to automatically determine the
earliest exiting point based on the inferred complexity of the input video.
The architecture of the gating module is illustrated in the bottom right
corner of the figure. Note that $g_{1}$ only receives $z_{1}$ as input. The
video is adopted from [24].

Conditional compute via early exiting: Conditional computation in neural
networks aims to dynamically allocate the components of the model (e.g.
layers, sub-networks, etc.) on a per-input basis. ConvNet-AIG [49], SkipNet
[52], and BlockDrop [54] exploit the robustness of Residual Networks (ResNets)
to layer dropping and activate or deactivate full residual blocks conditioned
on the input, to save compute. GaterNet [6], dynamic channel pruned networks
[16], batch-shaped channel-gated networks [2], and Conditional channel gated
networks for continual learning [1] perform gating in a more fine-grained
fashion by turning individual channels of a layer on or off conditioned on the
input. Other works focus on adaptive spatial attention for faster inference
[14, 50, 51]. Concurrent to our work, AdaFuse [38] dynamically fuses channels
from current and history feature maps to reduce the computation. A major
research direction in conditional compute is early exiting. Prior works have
mainly used early exiting for image classification. To adapt computational
costs to the difficulty of the input, Deeply-Supervised Nets [34] and
BranchyNet [43] propose architectures that are composed of a cascade of
intermediate classifiers. This allows simpler examples to exit early via an
intermediate classifier while more difficult samples proceed deeper in the
network for more accurate predictions. Multi-scale dense networks [28] and
adaptive resolution networks [58] focus on spatial redundancy of input samples
and use a multi-scale dense connection architecture for early exiting.

Our model draws inspiration from these works, but rather than having
intermediate classifiers operating on intermediate network features, we focus
on early exiting over the temporal dimension for video recognition.

##  3 FrameExit

Given a set of videos and their labels
$\\{\mathbf{v}_{i},\mathbf{y}_{i}\\}_{i=1}^{D}$, we aim to classify each video
by processing the minimum number of frames. Figure 2 illustrates the overall
architecture of our proposed model. Our model consists of i) a frame sampling
policy $\mathbf{\pi}$, ii) a feature extraction network $\Phi$, iii) an
accumulated feature pooling module, and iv) $T$ classifiers $\mathbf{f}_{t}$
and their associated exiting gates $\mathbf{g}_{t}$, where $T$ is the number
of input frames. Given an input video, we extract a partial clip
$\mathbf{x}_{1:t}$ by sampling $t$ frames, one at a time, from the video based
on the sampling policy $\mathbf{\pi}$:

$\mathbf{x}_{1:t}=[\mathbf{x}_{1:t-1};\mathbf{x}_{t}],\quad\mathbf{x}_{t}=\mathbf{v}_{\mathbf{\pi}(t)},$
(1)

where $\mathbf{x}_{1:t-1}$ denotes a partial clip of length $t-1$ and
$\mathbf{x}_{t}$ is a single video frame. We use the feature extraction
network $\Phi$ to generate independent representation for each frame
$\mathbf{x}_{t}$. These representations are then aggregated using the
accumulated feature pooling module. The resulting clip level representation,
$\mathbf{z}_{t}$, is then passed to the classifier $\mathbf{f}_{t}$ and its
associated early exiting gate $\mathbf{g}_{t}$.

Starting from a single-frame clip, we incrementally add more temporal details
at each time step until one of the gates predicts the halt signal. Each gate
$\mathbf{g}_{t}:(\mathbf{z}_{t-1},\mathbf{z}_{t})\rightarrow\\{0,1\\}$ is a
binary function indicating whether the network has reached a desired
confidence level to exit. A gate receives the aggregated features
$\mathbf{z}_{t}$ and $\mathbf{z}_{t-1}$ as input. This allows the gate to make
a more informed decisions by considering the agreement between temporal
features. For example, a highly confident incorrect exiting prediction solely
based on the current frame representation $\mathbf{z}_{t}$ could potentially
be mitigated if there is a significant disagreement with the representation
$\mathbf{z}_{t-1}$. Note that the first gating module in the network only
receives $\mathbf{z}_{1}$ as input. In the end, the final video label will be
predicted as:

$\mathbf{y}=\mathbf{f}_{t}(\mathbf{z}_{t}),\quad\text{where}~{}t=\operatorname*{arg\,min}_{t}\\{t~{}|~{}\mathbf{g}_{t}=1\\}$
(2)

where $t$ is the earliest frame that meets the gating condition. Note that if
none of the gates predict the halt signal, the last classifier will classify
the example. In the following sections, we describe our choice for frame
sampling policy, accumulated feature pooling module, and our gates for early
exiting.

Frame Sampling Policy. We sample $T$ frames, one at a time, using the policy
function $\pi(t)$. While in most existing works $\mathbf{\pi}$ is
parameterized with a light-weight network and is trained using policy gradient
methods [41] or Gumbel reparametrization [36], we use a simple, deterministic,
and parameter-free function and show that it performs as well as sophisticated
frame selection models. Our sampling function $\mathbf{\pi}$ follows a coarse-
to-fine principle for sampling in temporal dimension. It starts sampling from
a coarse temporal scale and gradually samples more frames to add finer details
to the temporal structure. Specifically, we sample the first frames from the
middle, beginning, and end of the video, respectively, and then repeatedly
sample frames from the halves (check Appendix for more details). Compared to
sampling sequentially in time, this strategy allows the model to have access
to a broader time horizon at each timestamp while mimicking the behaviour of
RL-based approaches that jumps forward and backward to seek future informative
frames and re-examine past information.

Feature extraction network. Our feature extraction network
$\Phi(\mathbf{x}_{i};\theta_{\Phi})$ is a 2D image representation model,
parametrized by $\theta_{\Phi}$, that extracts features for input frame
$\mathbf{x}_{i}$. We use ResNet-50 [19], EfficientNet-b3 [42], and X3D-S [11]
in our experiments.

Accumulated Feature Pooling. This step aims to create a representation for the
entire clip $\mathbf{x}_{1:t}$. A major design criteria for our accumulated
feature pooling module is efficiency. To limit the computation costs to only
the newly sampled frame, the clip representation is incrementally updated.
Specifically, given the sampled frame $\mathbf{x}_{t}$ and features
$\mathbf{z}_{t-1}$, we represent a video clip as:

$\mathbf{z}_{t}=\Psi(\mathbf{z}_{t-1},\Phi(\mathbf{x}_{i};\theta_{\Phi})),$
(3)

where $\Psi$ is a temporal aggregation function that can be implemented by
statistical pooling methods such as average/max pooling, LSTM [20, 17], or
self-attention [48].

Early Exiting. While processing the entire frames of a video is
computationally expensive, processing a single frame may also restrict the
network’s ability to recognize an action. Our conditional early exiting model
has $T$ classifiers accompanied by their associated early exiting gates that
are attached at different time steps to allow early exiting. Each classifier
$\mathbf{f}_{t}$ receives the clip representation $\mathbf{z}_{t}$ as input
and makes a prediction about the label of the video. During training, we
optimize the parameters of the feature extractor network and the classifiers
using the following loss function:

$\mathcal{L}_{cls}=\frac{1}{T}\sum_{t=0}^{t=T}\ell_{cls}(\mathbf{f}_{t}(\mathbf{z}_{t};\theta_{f}),\mathbf{y})$
(4)

In our experiments, we use the standard cross-entropy loss for single-label
video datasets and binary cross-entropy loss for the multi-label video
datasets.

We parameterize each exiting gate $\mathbf{g}_{t}$ as a multi-layer
perceptron, predicting whether the partially observed clip $\mathbf{x}_{1:t}$
is sufficient to accurately classify the entire video. The gates have a very
light design to avoid any significant computational overhead. As mentioned
earlier, each gate
$\mathbf{g}_{t}(\mathbf{z}_{t-1},\mathbf{z}_{t})\rightarrow\\{0,1\\}$ receives
as input the aggregated representations $\mathbf{z}_{t}$ and
$\mathbf{z}_{t-1}$ (See Figure 2 \- bottom right). Each of these
representations are first passed to two layers of MLP with 64 neurons
independently (shared weights between the two streams). The resulting features
are then concatenated and linearly projected and fed to a sigmoid function.
Note that $g_{1}$ only receives $\mathbf{z}_{1}$ as input.

During training, gates may naturally learn to postpone exiting so that the
last classifier always generates the model output because that may tend to
maximize accuracy at the expense of additional processing. However, this sort
of training largely defeats the purpose of the early exiting architecture. To
overcome this problem we regularize the gates such that they are enforced to
early exit. The parameters of the gates $\theta_{g}$ are learned in a self-
supervised way by minimizing the binary cross-entropy between the predicted
gating output and pseudo labels $\mathbf{y}^{g}_{t}$:

$\mathcal{L}_{gate}=\frac{1}{T}\sum_{t=0}^{t=T}\operatorname{BCE}(\mathbf{g}_{t}(\mathbf{z}_{t-1},\mathbf{z}_{t};\theta_{g}),\mathbf{y}^{\mathbf{g}}_{t}),$
(5)

We define the pseudo labels for the gate $\mathbf{g}_{t}$ based on the
classification loss:

$\mathbf{y}^{g}_{t}=\begin{cases}1&\text{$\ell_{cls}(\mathbf{f}_{t}(\mathbf{z}_{t}),\mathbf{y})\leq\epsilon_{t}$}\\\
0&\text{else}\\\ \end{cases}$ (6)

where $\epsilon_{t}$ determines the minimum loss required to exit through
$\mathbf{f}_{t}$. A label $1$ indicates the exiting signal while label $0$
indicates that the network should proceed with processing further frames.
Using this loss, we enforce the gate $\mathbf{g}_{t}$ to exit when the
classifier $\mathbf{f}_{t}$ generates a sufficiently reliable prediction with
a low loss. Provided that the early stage classifiers observe very limited
number of frames, we only want to enable exiting when the classifier is highly
confident about the prediction, i.e. $\mathbf{f}_{t}$ has a very low loss.
Hence, it is preferred to use smaller $\epsilon_{t}$ for these classifiers. On
the other hand, late stage classifiers mostly deal with difficult videos with
high loss. Therefore, as we proceed to later stage classifiers, we increase
$\epsilon_{t}$ to enable early exiting. In our experiments, we define
$\epsilon_{t}=\beta\exp(\frac{t}{2})$, where $\beta$ is a hyper-parameter that
controls the trade-off between model accuracy and total computation costs. The
higher the $\beta$, the more computational saving we obtain.

The final objective for training our model is given as:

$\mathcal{L}=\mathbb{E}_{(\mathbf{v},\mathbf{y})\sim
D_{train}}\left[\mathcal{L}_{cls}+\mathcal{L}_{gate}\right]$ (7)

Note that we use equal weights for the classification and gating loss terms in
all of our experiments.

##  4 Experiments

Figure 3: Watermelon visualization of FrameExit predictions over time. The
exiting statistics of our method over time for $\beta=1e^{-4}$ (top) and
$\beta=1e^{-6}$ (bottom) on the validation set of ActivityNet. The area of a
circle at time step $t$ represents the percentage of samples that exited
through classifier $t$. Easier examples exit earlier from the network with
higher accuracy while only hard examples reach to late stage classifiers
leading to increased miss-classifications. The illustration is inspired from
[56].

We conduct extensive experiments to investigate the efficacy of our proposed
method using three large-scale datasets on two video understanding tasks,
namely action recognition and holistic video understanding, as described in
Section 4.1. Our experiments demonstrate that FrameExit outperforms the state
of the art while significantly reducing computational costs as discussed in
Section 4.2. Finally, we present an ablation study of our design choices in
Section 4.3

###  4.1 Experimental setup

Datasets. We conduct experiments on three large-scale datasets:
ActivityNet-v1.3 [3], Mini-Kinetics [30], and HVU [7]. ActivityNet-v1.3 [3] is
a long-range action recognition dataset consisting of $200$ classes with
$10,024$ videos for training and $4,926$ videos for validation. The average
length of the videos in this dataset is $167$ seconds. Mini-Kinetics [30] is a
short-range action recognition dataset, provided by [37], that contains $200$
classes from Kinetics dataset [30] with $121,215$ training and $9,867$ testing
videos. The average duration of videos is $10$ seconds. HVU [7] is a large-
scale, multi-label dataset for holistic video understanding with $3142$
classes of actions, objects, scenes, attributes, events and concepts. It
contains $476k$ training and $31k$ validation videos. The average duration of
videos in this dataset is $8.5$ seconds.

Evaluation metrics. Following the literature, we use mean average precision
(mAP) and top-1 accuracy to evaluate accuracy for multi-class (Mini-Kinetics)
and multi-label (ActivityNet and HVU) classification respectively. We measure
the computational cost as giga floating point operations (GFLOPs). As a
reference, ResNet-50 and EfficientNet-b3 have $4.12$ and $1.8$ GFLOPs
respectively for a single input image of size $224\times 224$. Moreover, the
X3D-S architecture has $1.96$ GFLOPs for an input size of $13\times 160\times
160$. As different baseline methods use different number of frames for
recognition, we report per video GFLOPs in all of our experiments.

Figure 4: Illustration of the exiting performance. The matrix entry $(i,j)$
illustrates the accuracy of the classifier $C_{j}$, if it were to classify the
video data $D_{i}$ that originally early exited by the model through $C_{i}$.
Interestingly, if a video prematurely exits earlier than what was decided, we
significantly lose accuracy while if it exits later than the model’s decision,
no significant gain is obtained.

Implementation details. We use ResNet-50 [19] and EfficientNet-b3 [42]
hereafter referred to as ResNet and EfficientNet, as well as X3D-S [11] as our
backbone networks. Both backbones are pretrained on ImageNet. We remove the
last classification layers from the backbone networks, and replace it with a
fully-connected layer with $4096$ neurons. We set the number of input frames,
$T$, to $10$. We use max-pooling as the accumulated feature pooling function,
$\Psi$ in Eq. 3, for all the experiments unless otherwise stated. We train the
model in two stages: we first train the backbone network and classifiers, then
we learn the parameters of the gating modules. We use Adam optimizer [31] with
a learning rate of $1e^{-4}$ for both training stages. The first training step
runs for $35$ epochs whilst dropping the learning rate after $16$, and $30$
epochs with a factor of $0.1$. The second training step runs for $10$ epochs
and the learning rate is dropped after epochs $5$ and $8$. We set the hyper-
parameter $\beta$ ranging from $1e^{-6}$ to $1e^{-2}$ to generate varying
trade-off points between accuracy and computational costs.

###  4.2 Results

We first analyze the behaviour of our conditional early exiting method. Then
we compare FrameExit with the state of the art in action recognition and
holistic video understanding. This section is concluded by reporting several
qualitative results.

Conditional early exiting. We analyze the effectiveness of FrameExit in
adjusting the amount of computations per video based on its content. Figure 3
illustrates predictions and the number of frames processed by two FrameExit
models trained with different values of $\beta=1e^{-4}$ and $\beta=1e^{-6}$.
As expected, a higher $\beta$ encourages more video samples to exit early from
the network (top row) compared to a lower $\beta$ (bottom row). A general
observation is that the more we proceed to classifiers in later stages, the
more inaccurate the predictions become. This may sound counter-intuitive,
because if we were to have a model without early exiting, late-stage
classifiers produce the most accurate predictions. However, the trend shown in
Figure 3 is highly desirable, because it shows that the easier examples have
already exited from the network while only hard examples reach to late stage
classifiers.

This observation is more clear in Figure 4. The matrix entry $(i,j)$
illustrates the accuracy of the classifier $C_{j}$, if it were to classify the
video data $D_{i}$ that originally early exited by the model through $C_{i}$.
The diagonal entries $(i,j=i)$ in the matrix represent the actual early-
exiting results obtained by the model. In an ideal early exiting scenario, for
each row $i$, it is desired to have a significantly lower performance for
$C_{j}$ compared to $C_{i}$ when $j<i$ and a similar accuracy to $C_{i}$ when
$j>i$. A similar pattern is observed in Figure 4. As can be seen, if a video
prematurely exits earlier than what was decided by the model, we lose accuracy
significantly while if it exits later than the model’s decision, no
significant gain in accuracy is obtained. FrameExit learns to exit from the
optimal classifier to balance computational costs and model accuracy.

Figure 5 illustrates average precision of all categories in the ActivityNet
dataset as well as the average number of frames required for each category to
confidently exit from the network. As can be seen, the classes with strong
visual cues (such as riding bumper car, playing accordion, playing pool, etc.)
appear to be simpler and therefore exit in the early stages with a high
accuracy. In contrast, actions which involve more complex interactions and
require a series of frames to unveil (such as Javelin throw and trimming
branches) take more frames to be recognized.

Figure 5: AP vs number of processed frames per category in the ActivityNet
dataset. Categories with object/scene cues exit in the early stages with high
accuracy while complex actions require more frames for recognition.

Comparison to state of the art: Action recognition.

Table 1: Comparison to state of the art on action recognition. FrameExit
outperforms competing methods in terms of accuracy and efficiency using
ResNet, EfficientNet, and X3D-S backbones. Results of other methods are
adopted from [37]. ∗(IA$|$IA) denotes the variant of the ListenToLook method
that additionally uses audio for sampling and recognition.

| ActivityNet | Mini-kinetics
---|---|---
mAP (%) | GFLOPs | Top-1 (%) | GFLOPs
ResNet |  |  |  |
AdaFrame [56] | 71.5 | 79.0 | - | -
LiteEval [55] | 72.7 | 95.1 | 61.0 | 99.0
ListenToLook [15] | 72.3 | 81.4 | - | -
ListenToLook (IA$|$IA)${}^{\mbox{*}}$ [15] | 75.6 | 37.5 | - | -
SCSampler [33] | 72.9 | 41.9 | 70.8 | 41.9
AR-Net [37] | 73.8 | 33.5 | 71.7 | 32.0
Ours (w/o exit) | 77.3 | 41.2 | 73.3 | 41.2
Ours (FrameExit) | 76.1 | 26.1 | 72.8 | 19.7
EfficientNet |  |  |  |
AR-Net [37] | 79.7 | 15.3 | 74.8 | 16.3
Ours (w/o exit) | 81.1 | 18.0 | 75.9 | 18.0
Ours (FrameExit) | 80.0 | 11.4 | 75.3 | 7.8
X3D-S |  |  |  |
Ours (w/o exit) | 87.4 | 19.6 | - | -
Ours (FrameExit) | 86.0 | 9.8 | - | -

We compare our method with an extensive list of recent works on efficient
video recognition: AR-Net [37], AdaFrame [56], LiteEval [55], SCSampler [33],
MARL [53], and ListenToLook [15]. AR-Net uses MobileNet-V2 as the sampler
network and adaptively chooses a ResNet architecture with varying depths as
the recognition network. The method is additionally evaluated with variants of
EfficientNet [42] as the recognition network. AdaFrame and LiteEval both use
MobileNet-V2 as the policy network and ResNet-101 as the recognition network.
SCSampler uses MobileNet-V2 as the sampler network and ResNet-50 as the
recognition network. MARL uses a ResNet-101 as the recognition network
combined with a fully connected layer as a policy network. ListenToLook uses
MobileNet-V2 as the sampler network and ResNet-101 as the recognition network.
Finally, ListenToLook (IA$|$IA) uses two ResNet-18 for audio and visual
modality respectively as the sampler. The same architecture is used for
recognition network.

While the competing methods use two networks for sampling and recognizing,
FrameExit uses a single network for efficient video recognition. The results
for ActivityNet and Mini-Kinetics are shown in Table 1. Our method with a
ResNet backbone outperforms all other approaches by obtaining an improved
accuracy while using $1.3\times$– $5\times$ less GFLOPs. The gain in accuracy
is mainly attributed to our accumulated feature pooling module, while the gain
in efficiency is attributed to the proposed sampling policy and gating modules
for conditional early exiting (see Section 4.3 for detailed analyses).
Compared to our model without early exiting, denoted as “Ours (w/o exit)”,
FrameExit achieves a comparable accuracy with a significant reduction in
GFLOPs (see Appendix for wall-clock timing of FrameExit). In addition, Figure
6 presents accuracy vs. computations trade-off curves on ActivityNet for
various methods. Note that except ListenToLook (IA—IA) [15] that uses audio
and visual features for sampling and recognition, other methods rely solely on
visual features. As shown, FrameExit achieves the same top-performance as
other methods at a much lower computational cost.

To verify that the performance of our model is not limited to one
architecture, we conduct similar experiments with EfficientNet and X3D
backbones. Using EfficientNet-b3, FrameExit obtains $3.9\%$ further absolute
gain in mAP on ActivityNet and $2.5\%$ on Mini-Kinetics, while consuming
$2.3\times$ and $2.5\times$ less compute. In particular, on ActivityNet and
mini-kinetics, we outperform AR-Net [37], the leading method among
competitors, with $1.3\times$ and $2.1\times$ less GFLOPs, respectively. When
using the highly efficient X3D-S as our backbone, at a similar computational
cost, FrameExit achieves 86.0% which is 6.0% higher than the best 2D model.
This demonstrates the superiority of our method for efficient video
understanding using both 2D and 3D backbones.

Figure 6: Accuracy vs. efficiency curves on ActivityNet. FrameExit performs
similar to or better than other methods at a much lower computational cost.
Note that L2L (IA$|$ IA) denotes the variant of the ListenToLook method that
additionally uses audio for sampling and recognition.

Comparison to state of the art: Holistic understanding. We evaluate our method
for the task of holistic video understanding, where the goal is to recognize
various semantic aspects including objects, scenes, and actions within a
video. To this end, we conduct experiments on the large-scale, multi-label HVU
dataset [7]. we use average pooling in our accumulated feature pooling module,
$\Psi$ in Eq. 3, as it outperforms max pooling. We compare our method with the
commonly-used uniform and random sampling baselines. These baselines sample
$K$ frames (uniformly/randomly) and average frame-level predictions to
generate a final video-level prediction. We additionally, compare our method
with 3D-ResNet18 [45] and mixed 2D/3D ResNet [7] models. As shown in Table 2,
our method consistently outperforms other methods in terms of accuracy while
saving significant computation. In particular, we report two variants of
FrameExit with the ResNet backbone by changing accuracy-efficiency trade-off
parameter $\beta$. FrameExit trained with $\beta=1e^{-3}$ only uses $8.6$
GFLOPs to obtain $45.7\%$ mAP, which is highly suitable for low-budget
requirements. FrameExit trained with $\beta=1e^{-5}$ uses $18.7$ GFLOPs on
average to obtain $49.2\%$ mAP, which is suitable for high-accuracy
requirements. Interestingly, FrameExit outperforms 3D models [45, 7] in this
task. Given that HVU requires recognizing static 2D concepts such as scenes
($8\%$ of the class labels) and objects ($55\%$ of the class labels), 3D-CNN
methods are not necessarily the optimal choice for this task. The most budget-
friendly trained model is FrameExit with the EfficientNet backbone that uses
only $5.7$ GFLOPs and achieves a mAP of $46.1$.

This experiment implies that our method can be used in a wider range of
efficient video recognition tasks. Our results set a new state of the art on
the HVU dataset for future research.

Table 2: Comparison to state of the art on holistic video understand. FrameExit significantly improves mAP on HVU dataset while saving compute. ∗indicates GFLOPs for one input clip.  | mAP (%) | GFLOPs
---|---|---
2D/3D ResNet |  |
Uniform-10 | $44.7$ | $41.2$
Random-10 | $43.6$ | $41.2$
3D-ResNet18 [45] | $35.4$ | $38.6^{*}$
HATNet [7] | $39.6$ | $41.8^{*}$
FrameExit ($\beta=1e^{-3}$) | $45.7$ | ${\bf 8.6}$
FrameExit ($\beta=1e^{-5}$) | ${\bf 49.2}$ | $18.7$
EfficientNet |  |
FrameExit ($\beta=1e^{-3}$) | $\bf{47.7}$ | $11.7$
FrameExit ($\beta=1e^{-2}$) | $46.1$ | ${\bf 5.7}$

###  4.3 Ablation study

In this section we inspect different aspects of FrameExit. For all ablations,
we follow the same training procedure explained in Section 4 and use the
Activitynet-v1.3 dataset.

Impact of policy function. To validate the impact of our proposed
deterministic policy function for frame sampling, we make comparisons with two
commonly used sampling approaches namely “Sequential” and “Random”, both
during training and evaluation. For “Sequential”, we keep original frame order
while in “Random”, we randomly sample frames. Results for all combinations are
shown in Table 3. Using the original frame ordering during training and
testing results in a lower mAP and a higher GFLOPs. We speculate that during
training, earlier classifiers receive less informative gradients due to not
observing a holistic view of the entire video (row 1 vs. row 2). Similarly,
the model consumes higher GFLOPs because it needs to observe more frames to
infer a confident prediction (row 1 vs. row 3). Random sampling both during
training and testing improves the results because early stage classifiers have
a higher chance of obtaining more useful information reflecting the entire
video time-span. The best result, however, is obtained by our simple
deterministic sampling policy. This is because jumping forward and backward
enabled by our frame sampling policy effectively improves the chance of
picking up informative frames for recognition.

Table 3: Impact of sampling policy. we report the results over 5 runs.

| Train-random | Test-random | mAP (%) | GFLOPS
---|---|---|---|---
(1) | ✗ | ✗ | 73.7 | 27.6
(2) | ✓ | ✗ | 74.3 $\pm$ 0.9 | 26.7 $\pm$ 0.33
(3) | ✗ | ✓ | 74.4 $\pm$ 0.02 | 27.0 $\pm$ 0.19
(4) | ✓ | ✓ | 74.4 $\pm$ 0.34 | 25.0 $\pm$ 0.37
FrameExit | - | - | 76.1 | 26.1

Impact of accumulated feature pooling.

Table 4: Impact of accumulated feature pooling. Feature pooling | mAP (%) | GFLOPs
---|---|---
✓ | 76.1 | 26.1
✗ | 67.8 | 27.6

We evaluate the impact of accumulated feature pooling module by comparing the
performance of FrameExit with and without feature pooling as reported in Table
4. For the FrameExit without feature pooling, we train each classifier and its
associated gating module only on the currently sampled frame. We then average
frame-level predictions made by all classifiers up until the exiting
classifier to obtain the video-level prediction. Therefore, the major
difference in the design of these two settings relates to the use of the
pooling operation in feature space or in the prediction space. The results in
Table 4 demonstrates that pooling over features is much more effective than
over output predictions.

Number of input frames. We train FrameExit with various number of input frames
$T=\\{4,8,10,16\\}$. During inference, the gates decide when to stop
processing. As shown in Table 5, the performance of the model increases as the
number of input frames increases, but up to a certain limit. This is a
reasonable observation, as certain actions/videos may require more frames to
be recognized. However, the reason why increasing the number of frames after a
limit does not further improve the performance could be mainly attributed to
the limited capacity of 2D convolutional networks in leveraging temporal
information. As a result, to keep the balance between accuracy and efficiency,
we set $T=10$ in our experiments.

Table 5: Impact of number of frames on FrameExit.  | $T=4$ | $T=8$ | $T=10$ | $T=16$
---|---|---|---|---
mAP (%) | $66.2$ | $74.3$ | $76.1$ | $76.1$
GFLOPs | $13.0$ | $22.4$ | $26.1$ | $35.1$

Adaptive vs fixed exiting. To show the merits of adaptive early exiting versus
a fixed budget exiting, we conduct an ablation in two settings. In the first
setting, we use our conditional early exiting model and in the second setting,
we assume a fixed number of frames is processed for each test video. Table 6
shows that our conditional early exiting method consistently outperforms fixed
exiting.

Table 6: Impact of adaptive exiting. ∗for each column, results are reported over an average no. of processed frames.  No. processed frames | $3$ | $4$ | $6$ | $7$
---|---|---|---|---
Fixed budget | $60.8$ | $67.5$ | $74.0$ | $75.2$
FrameExit* | $67.5$ | $70.1$ | $75.8$ | $76.4$

##  5 Conclusions

In this paper, we presented FrameExit, a conditional early exiting method for
efficient video recognition. Our proposed method uses gating modules, trained
to allow the network to automatically determine the earliest exiting point
based on the inferred complexity of the input video. To enable gates to make
reliable decisions we use an effective video representation obtained using
accumulated feature pooling. We showed that our early exiting mechanism
combined with a simple, deterministic sampling strategy obviates the need for
complex sampling policy techniques. Our proposed method is model-agnostic and
can be used with various network architectures. Comprehensive experiments show
the power of our method in balancing accuracy versus efficiency in action
recognition as well as holistic video understanding tasks.

#### Acknowledgements

We thank Fatih Porikli, Michael Hofmann, Haitam Ben Yahia, Arash Behboodi and
Ilia Karmanov for their feedback and discussions.

## References

  * [1] Davide Abati, Jakub Tomczak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi.  Conditional channel gated networks for task-aware continual learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3931–3940, 2020.
  * [2] Babak Ehteshami Bejnordi, Tijmen Blankevoort, and Max Welling.  Batch-shaping for learning conditional channel gated networks.  In International Conference on Learning Representations, 2019.
  * [3] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles.  Activitynet: A large-scale video benchmark for human activity understanding.  In Proceedings of the ieee conference on computer vision and pattern recognition, pages 961–970, 2015.
  * [4] Víctor Campos, Brendan Jou, Xavier Giró-i Nieto, Jordi Torres, and Shih-Fu Chang.  Skip rnn: Learning to skip state updates in recurrent neural networks.  arXiv preprint arXiv:1708.06834, 2017.
  * [5] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In CVPR, 2017.
  * [6] Zhourong Chen, Yang Li, Samy Bengio, and Si Si.  Gaternet: Dynamic filter selection in convolutional neural network via a dedicated global gating network.  arXiv preprint arXiv:1811.11205, 2018.
  * [7] Ali Diba, Mohsen Fayyaz, Vivek Sharma, Manohar Paluri, Jürgen Gall, Rainer Stiefelhagen, and Luc Van Gool.  Large scale holistic video understanding.  In European Conference on Computer Vision, pages 593–610. Springer, 2020.
  * [8] Hehe Fan, Zhongwen Xu, Linchao Zhu, Chenggang Yan, Jianjun Ge, and Yi Yang.  Watching a small portion could be as good as watching all: Towards efficient video classification.  In IJCAI International Joint Conference on Artificial Intelligence, 2018.
  * [9] Linxi Fan, Shyamal Buch, Guanzhi Wang, Ryan Cao, Yuke Zhu, Juan Carlos Niebles, and Li Fei-Fei.  Rubiksnet: Learnable 3d-shift for efficient video action recognition.  In European Conference on Computer Vision, pages 505–521. Springer, 2020.
  * [10] Quanfu Fan, Chun-Fu Chen, Hilde Kuehne, Marco Pistoia, and David Cox.  More is less: Learning efficient video representations by big-little network and depthwise temporal aggregation.  arXiv preprint arXiv:1912.00869, 2019.
  * [11] Christoph Feichtenhofer.  X3d: Expanding architectures for efficient video recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 203–213, 2020.
  * [12] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  In Proceedings of the IEEE international conference on computer vision, pages 6202–6211, 2019.
  * [13] Basura Fernando, Efstratios Gavves, José Oramas, Amir Ghodrati, and Tinne Tuytelaars.  Rank pooling for action recognition.  IEEE transactions on pattern analysis and machine intelligence, 39(4):773–787, 2016.
  * [14] Michael Figurnov, Maxwell D Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry Vetrov, and Ruslan Salakhutdinov.  Spatially adaptive computation time for residual networks.  In CVPR, 2017.
  * [15] Ruohan Gao, Tae-Hyun Oh, Kristen Grauman, and Lorenzo Torresani.  Listen to look: Action recognition by previewing audio.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10457–10467, 2020.
  * [16] Xitong Gao, Yiren Zhao, Lukasz Dudziak, Robert Mullins, and Cheng-Zhong Xu.  Dynamic channel pruning: Feature boosting and suppression.  arXiv preprint arxiv:1810.05331, 2018.
  * [17] Amir Ghodrati, Efstratios Gavves, and Cees GM Snoek.  Video time: Properties, encoders and evaluation.  arXiv preprint arXiv:1807.06980, 2018.
  * [18] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh.  Learning spatio-temporal features with 3d residual networks for action recognition.  In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 3154–3160, 2017.
  * [19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
  * [20] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural computation, 9(8):1735–1780, 1997.
  * [21] ORBEA ALMA M30 CARBON · 2014 / BICIMAG by bicimag is licensed under CC BY.
  * [22] Sophie and Greg playing acro yoga by Gregology is licensed under CC BY.
  * [23] Kilby Plays the Harmonica - Sort of by aplusjimages is licensed under CC BY.
  * [24] SPORK! Exclusive: Pathways Waveland Bowling by SPORK! NFP is licensed under CC BY.
  * [25] Canada’s National Ballet School: Training in the Professional Ballet Program by Canada’s National Ballet School is licensed under CC BY.
  * [26] Batlle of The Year: Dau Truong Breakdance TV Spot by Galaxy Studio is licensed under CC BY.
  * [27] Show de tango COOPRUDEA by Cooperativa de Profesores Cooprudea is licensed under CC BY.
  * [28] Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Weinberger.  Multi-scale dense networks for resource efficient image classification.  In International Conference on Learning Representations, 2018.
  * [29] Noureldien Hussein, Mihir Jain, and Babak Ehteshami Bejnordi.  Timegate: Conditional gating of segments in long-range activities.  arXiv preprint arXiv:2004.01808, 2020.
  * [30] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al.  The kinetics human action video dataset.  arXiv preprint arXiv:1705.06950, 2017.
  * [31] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2014.
  * [32] Okan Köpüklü, Neslihan Kose, Ahmet Gunduz, and Gerhard Rigoll.  Resource efficient 3d convolutional neural networks.  In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pages 1910–1919. IEEE, 2019.
  * [33] Bruno Korbar, Du Tran, and Lorenzo Torresani.  Scsampler: Sampling salient clips from video for efficient action recognition.  In Proceedings of the IEEE International Conference on Computer Vision, pages 6232–6242, 2019.
  * [34] Chen-Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu.  Deeply-supervised nets.  In Artificial intelligence and statistics, pages 562–570, 2015\.
  * [35] Ji Lin, Chuang Gan, and Song Han.  Tsm: Temporal shift module for efficient video understanding.  In Proceedings of the IEEE International Conference on Computer Vision, pages 7083–7093, 2019.
  * [36] Chris J Maddison, Andriy Mnih, and Yee Whye Teh.  The concrete distribution: A continuous relaxation of discrete random variables.  In ICLR, 2017.
  * [37] Yue Meng, Chung-Ching Lin, Rameswar Panda, Prasanna Sattigeri, Leonid Karlinsky, Aude Oliva, Kate Saenko, and Rogerio Feris.  Ar-net: Adaptive frame resolution for efficient action recognition.  arXiv preprint arXiv:2007.15796, 2020.
  * [38] Yue Meng, Rameswar Panda, Chung-Ching Lin, Prasanna Sattigeri, Leonid Karlinsky, Kate Saenko, Aude Oliva, and Rogerio Feris.  Adafuse: Adaptive temporal fusion network for efficient action recognition.  In International Conference on Learning Representations, 2021.
  * [39] AJ Piergiovanni, Anelia Angelova, and Michael S Ryoo.  Tiny video networks.  arXiv preprint arXiv:1910.06961, 2019.
  * [40] Zhaofan Qiu, Ting Yao, and Tao Mei.  Learning spatio-temporal representation with pseudo-3d residual networks.  In proceedings of the IEEE International Conference on Computer Vision, pages 5533–5541, 2017.
  * [41] Richard S Sutton and Andrew G Barto.  Reinforcement learning: An introduction.  MIT press, 2018.
  * [42] Mingxing Tan and Quoc V Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  arXiv preprint arXiv:1905.11946, 2019.
  * [43] S. Teerapittayanon, B. McDanel, and H. T. Kung.  Branchynet: Fast inference via early exiting from deep neural networks.  In 2016 23rd International Conference on Pattern Recognition (ICPR), pages 2464–2469, Dec 2016.
  * [44] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri.  Learning spatiotemporal features with 3d convolutional networks.  In Proceedings of the IEEE international conference on computer vision, pages 4489–4497, 2015.
  * [45] Du Tran, Jamie Ray, Zheng Shou, Shih-Fu Chang, and Manohar Paluri.  Convnet architecture search for spatiotemporal feature learning.  arXiv preprint arXiv:1708.05038, 2017.
  * [46] Du Tran, Heng Wang, Lorenzo Torresani, and Matt Feiszli.  Video classification with channel-separated convolutional networks.  In Proceedings of the IEEE International Conference on Computer Vision, pages 5552–5561, 2019.
  * [47] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri.  A closer look at spatiotemporal convolutions for action recognition.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 6450–6459, 2018.
  * [48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [49] Andreas Veit and Serge Belongie.  Convolutional networks with adaptive inference graphs.  In European Conference on Computer Vision (ECCV), 2018.
  * [50] Thomas Verelst and Tinne Tuytelaars.  Dynamic convolutions: Exploiting spatial sparsity for faster inference.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2320–2329, 2020.
  * [51] Longguang Wang, Xiaoyu Dong, Yingqian Wang, Xinyi Ying, Zaiping Lin, Wei An, and Yulan Guo.  Learning sparse masks for efficient image super-resolution.  arXiv preprint arXiv:2006.09603, 2020.
  * [52] Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E. Gonzalez.  Skipnet: Learning dynamic routing in convolutional networks.  In Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XIII, pages 420–436, 2018.
  * [53] Wenhao Wu, Dongliang He, Xiao Tan, Shifeng Chen, and Shilei Wen.  Multi-agent reinforcement learning based frame sampling for effective untrimmed video recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6222–6231, 2019.
  * [54] Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris.  Blockdrop: Dynamic inference paths in residual networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8817–8826, 2018.
  * [55] Zuxuan Wu, Caiming Xiong, Yu-Gang Jiang, and Larry S Davis.  Liteeval: A coarse-to-fine framework for resource efficient video recognition.  In Advances in Neural Information Processing Systems, pages 7780–7789, 2019.
  * [56] Zuxuan Wu, Caiming Xiong, Chih-Yao Ma, Richard Socher, and Larry S Davis.  Adaframe: Adaptive frame selection for fast video recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1278–1287, 2019.
  * [57] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy.  Rethinking spatiotemporal feature learning for video understanding.  arXiv preprint arXiv:1712.04851, 1(2):5, 2017.
  * [58] Le Yang, Yizeng Han, Xi Chen, Shiji Song, Jifeng Dai, and Gao Huang.  Resolution adaptive networks for efficient inference.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2369–2378, 2020.
  * [59] Serena Yeung, Olga Russakovsky, Greg Mori, and Li Fei-Fei.  End-to-end learning of action detection from frame glimpses in videos.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2678–2687, 2016.
  * [60] Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici.  Beyond short snippets: Deep networks for video classification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4694–4702, 2015.
  * [61] Yin-Dong Zheng, Zhaoyang Liu, Tong Lu, and Limin Wang.  Dynamic sampling networks for efficient action recognition in videos.  IEEE Transactions on Image Processing, 29:7970–7983, 2020.
  * [62] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba.  Temporal relational reasoning in videos.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 803–818, 2018.
  * [63] Mohammadreza Zolfaghari, Kamaljeet Singh, and Thomas Brox.  Eco: Efficient convolutional network for online video understanding.  In Proceedings of the European conference on computer vision (ECCV), pages 695–712, 2018.

## Appendix

### A. Algorithmic description of the policy

In this section, we present an algorithmic description of our policy function.
Our policy function follows a coarse-to-fine principle for sampling frames. It
starts sampling from a coarse temporal scale and gradually samples more frames
to add finer details to the temporal structure. Specifically, we sample the
first frames from the middle, beginning, and end of the video, respectively,
and then repeatedly sample frames from the halves as shown in Algorithm 1.

Input: N (number of frames in the video)

Output: S (a sequence of sampled frame indices)

S = [floor((N+1)/2), 1, N] ;

q = 2 ;

while  _len( S) $<$ N_ do

interval = floor(linspace(1, N, q+1));

for  _i=1 : len(interval)-1_ do

a = interval[i];

b = interval[i+1];

ind = floor((a+b)/2);

if  _ind is not in S_ then

S.append(ind);

end if

end for

q = q * 2;

end while

Algorithm 1 Policy function

### B. Wall-clock time inference

The wall-clock speedup of FrameExit is highly correlated with FLOPS. This is
because the major computational cost comes from the number of frames being
processed by the backbone. For example, using an Nvidia GeForce GTX 1080Ti
GPU, the inference time of FrameExit with a ResNet50 backbone at computational
costs of 41.2, 26.1, and 12.3 GFLOPs are 15.8, 10.9, and 4.8 ms, respectively.

### C. Qualitative results

Figure 7 presents several video clips and the number of frames required to
process by FrameExit before exiting. As shown, a few frames are sufficient for
FrameExit to classify a video if the observed frames are discriminative
enough. As the complexity of video scene increases (such as obscured objects
and cluttered background) FrameExit needs more temporal information to reach a
confident prediction.

Figure 7: Qualitative results. Each row contains $10$ frames of a video,
sampled according to the policy function. Our method observes only the green
box to recognize an action. As the content becomes more complex, FrameExit
needs more frames to make a confident prediction. Zoom in for higher quality.
Videos are adopted from [21, 22, 23, 26, 27].


# ColorFoil: Investigating Color Blindness in Large Vision and Language Models

Ahnaf Mozib Samin
University of Malta
Msida, Malta
<EMAIL_ADDRESS>Firoz Ahmed
Shahjalal University of Science and Technology
Sylhet, Bangladesh
<EMAIL_ADDRESS>Md. Mushtaq Shahriyar Rafee
Metropolitan University
Sylhet, Bangladesh
<EMAIL_ADDRESS>
###### Abstract

With the utilization of Transformer architecture, large Vision and Language
(V&L) models have shown promising performance in even zero-shot settings.
Several studies, however, indicate a lack of robustness of the models when
dealing with complex linguistics and visual attributes. In this work, we
introduce a novel V&L benchmark - ColorFoil, by creating color-related foils
to assess the models’ perception ability to detect colors like red, white,
green, etc. We evaluate seven state-of-the-art V&L models including CLIP,
ViLT, GroupViT, and BridgeTower, etc. in a zero-shot setting and present
intriguing findings from the V&L models. The experimental evaluation indicates
that ViLT and BridgeTower demonstrate much better color perception
capabilities compared to CLIP and its variants and GroupViT. Moreover, CLIP-
based models and GroupViT struggle to distinguish colors that are visually
distinct to humans with normal color perception ability.

##  1 Introduction

Vision and language models (V&L) have exhibited improved performance for many
V&L tasks in recent years [15, 24, 4, 13, 20, 7]. Thus, the current paradigm
has now been shifting towards zero-shot learning, where models are evaluated
without fine-tuning for specific tasks [20]. Large-scale V&L models, in
particular, show promise for task-independent zero-shot evaluation [20].

Several studies have been conducted to perform comprehensive evaluations of
V&L models on a variety of tasks to identify their strengths and weaknesses
[2, 10, 9, 22, 1]. For instance, the VALSE evaluation benchmark has been
proposed to assess the state-of-art V&L models for challenging linguistic
constructs [17]. Therefore, five distinct tasks, including existence,
plurality, counting, relations, actions, and coreference, have been
introduced. In this benchmark, foils are generated from the existing V&L
datasets for each of the tasks. A foil is referred to as a distractor or
slightly incorrect example that is passed along with the correct example to
the V&L model to assess the model’s ability to correctly distinguish them [22,
17]. Although the existing V&L benchmarks like VALSE help the community to
test the capabilities of V&L models, there is still much work to be done to
evaluate the robustness and generalizability of the models on numerous other
tasks. It remains unknown how well the large V&L models can perceive colors
from the visual content.

Color perception requires a human-like understanding of visual content. Thus,
by evaluating the V&L models on color attributes, we can determine how closely
the large V&L models perceive colors to humans. A V&L model can be biased
towards detecting particular colors and perform poorly with others. Therefore,
it is essential to investigate it in order to improve the explainability and
interpretability of the models. By assessing the V&L models with their color-
perception ability, we can ensure robustness in real-life applications.

In this study, we aim to shed light on the following research question: how
well can the state-of-the-art large-scale V&L models perceive color-related
attributes, such as red, green, yellow, etc.? Our contributions are mainly
twofold:

  * •

We introduce a novel V&L benchmark ColorFoil by creating foils from the MS
COCO and Flickr30k datasets [14, 19] to investigate how well the models
perceive and identify the color-related attributes.

  * •

We perform a comparison between seven of the state-of-the-art V&L models
including CLIP [20], ViLT [12], ViT [6] and BridgeTower[29] using our
benchmark.

The outline of this paper is as follows. We provide a background study in
Section 2. In Section 3, we describe the process of constructing ColorFoil
from the MS COCO dataset. Experiment setup is provided in Section 4. We report
the results and discuss them in Section 5. In Section 6, we discuss the
limitations of our work. Ethical considerations are provided in 7. A
conclusion and future scope is presented in Section 8.

##  2 Background

V&L Models The current state-of-art models are first pre-trained in a self-
supervised way with a multi-task learning objective. The learning objectives
can be predicting the masked texts or masked region in the images, determining
whether or not the image and text corresponds, etc. The text and image input
features can be concatenated together and passed to a Transformer encoder.
This approach is known as single stream. Alternatively, the text and image
inputs can be separately encoded to two different Transformers and then
additional layers to merge them into multi-modal features.

CLIP Contrastive Language-Image Pre-training (CLIP) is a V&L model that is
pre-trained with 400M image-text pairs with a contrastive objective [20]. The
model jointly trains a text encoder and an image encoder to maximize the
cosine similarity of the image-text embeddings of real pairing while
minimizing the cosine similarity of the embeddings of the incorrect pairings
within a multi-modal embedding space. Each of the encoders are based on
transformers. CLIP demonstrates the ability to perform zero-shot visual
classification, object detection, and image generation tasks.

ViLT Vision-and-Language Transformer (ViLT) is pre-trained using a Transformer
with more than 4M images with two objectives such as image text matching and
masked language modeling [12]. The text embedding and the image features are
concatenated into a sequence and then fed into the transformer. Thus, ViLT is
a single stream model. ViLT achieves competitive or better performance than
other V&L models on downstream tasks while being 10 times faster due to
simpler processing of visual inputs.

BridgeTower There is a visual encoder, a textual encoder and a cross-modal
encoder with multiple lightweight bridge layers in the BridgeTower
architecture [29]. The top layers of the unimodal encoders and each layer of
the cross-modal encoder are connected with the bridge layers, thus enabling
extensive interactions at each layer of the cross-modal encoder. Each of
visual, textual and cross-modal encoders is transformer-based encoders. The
model is pre-trained with 4M images with two common objectives: masked
language modeling and image text matching. The model is found to outperform in
all downstream V&L tasks with negligible additional computational cost.

ViT A Vision Transformer (ViT) is designed for image classification tasks,
adapting the Transformer architecture from natural language processing [6]. It
divides an image into fixed-size patches, linearly embeds each patch, and
treats these embeddings as sequences akin to word tokens in text. Using self-
attention mechanisms, the ViT captures global image context more effectively
than convolutional networks, allowing for superior performance on large-scale
image datasets. ViTs leverage transfer learning and pretraining for enhanced
accuracy and efficiency.

GroupViT (Group Vision Transformer) is a variant of the Vision Transformer
designed to improve efficiency and scalability in image classification tasks
[28]. It enhances the standard ViT by introducing a group-wise processing
mechanism, where the input image is divided into smaller groups of patches.
Each group is processed independently through parallel self-attention layers,
reducing computational complexity. The results from these groups are then
aggregated to form a cohesive representation. GroupViT aims to retain the
global context modeling capabilities of ViTs while optimizing resource usage,
making it more suitable for large-scale and real-time applications.

Related Work Several V&L tasks include visual question answering [9], visual
reasoning [25], image retrieval [19], etc. Foiling is an approach that
slightly edits the original captions to evaluate the robustness of the V&L
models [22]. Similar to our work, Shekhar et al. [22] foiled the MS COCO
dataset, and constructed the FOIL-COCO dataset. However, their work did not
focus on the perception of colors of the V&L models. Following the work of
Shekhar et al. [22], several studies have been performed that evaluated the
V&L models [23, 8, 3, 18, 21].

##  3 Construction of the ColorFoil benchmark

The ColorFoil benchmark is automatically derived from the MS COCO (Microsoft
Common Objects in Context) and Flickr30k dataset, which serves as a resource
for studying image understanding, object recognition, image captioning, and
visual question-answering tasks [14, 19]. In the MS COCO dataset, textual
annotations are provided solely for the train and validation (val) sets. To
construct the ColorFoil, we obtain the images and annotations from the 2017 MS
COCO validation set, resulting in a total of 5,000 image-text pairs. Among
these instances, each of 2,511 pairs includes at least one word related to
color. For Flickr30k dataset, we use the standard val and test sets to prepare
the ColorFoil benchmark.

Our aim is to foil only the color name from the textual input, leaving the
original image and the rest of the text input as it is. For example, given a
caption like A blue bus driving down a street past a park. We foil the color-
related word, resulting in a modified sentence like - A brown bus driving down
a street past a park. If there are multiple color attributes in a caption, we
foil all of them.

Figure 1: Examples from the ColorFoil benchmark where color-related attributes
in the original captions have been modified to different colors.

We utilize the webcolors 1.3 python package to determine whether a substring
within a caption corresponds to a color [26]. This package encompasses a total
of 147 colors. Our filtering process involves excluding captions that lack
color names and selecting solely those containing at least one color name.

When replacing the original color name with a foiled alternative, we consider
the most widely used colors. The chosen target colors for foiling consist of
"blue", "black", "red", "pink", "yellow", "grey", "orange", "white", "green",
and "brown." So, rather than utilizing the complete list of 147 colors from
the webcolors package, we opt for a narrower selection of common colors for
foiling. This decision is based on the fact that numerous colors in the
package have limited practical usage (e.g. medium blue, mint cream, etc.). The
target color for foiling is selected randomly from the 10 common colors. If
the original color in the caption is one of the common colors, we randomly
select any other common color for foiling except for the one found in the
caption.

After excluding four instances of two-dimensional grayscale images due to
compatibility issues with certain models, our resulting dataset comprises
2,507 pairs of RGB images along with their captions and foils from MSCOCO and
2500 pairs of RGB image-caption pairs from Flickr30k. To ensure data
integrity, we conduct manual validation on a significant number of image-text
pairs randomly selected from the benchmark and find no anomalies. Examples of
original captions and corresponding foils are illustrated in Figure 1.

##  4 Experimental Setup

We pass the original caption, foil as well as the corresponding image to a V&L
model. The model provides the logits for each of the caption and foil
corresponding to the image. We take the softmax of the logits. Our hypothesis
is that a model with a well-perceivable ability to distinguish colors is
supposed to provide a higher probability for the original caption and a lower
probability for the foil.

We evaluate all the models in a zero-shot setting. We utilize the HuggingFace
transformer library to load the models [27]. These models are chosen due to
the fact that they represent different architectural variants. CLIP has a text
encoder and an image encoder, which are jointly trained with a contrastive
loss. ViLT is a single-stream model. BridgeTower contains multiple bridge
layers that connect the uni-modal encoders with the cross-modal encoder.

The evaluation metric employed in our study is accuracy and F1-score, which
are widely used in similar contexts. To elaborate, if the model accurately
identifies the foil in comparison to the original caption, the accuracy of
that particular example is incremented.

Table 1: Experiment results. We evaluate seven of the state-of-the-art V&L models on the MS COCO and Flickr30k subsets from ColorFoil. Accuracy (%) and F1-scores are reported. We conduct three experiments in which the models are presented different number of foils (modified caption) along with the original caption. The V&L models tend to struggle in challenging conditions with more foils. BridgeTower and ViLT outperform other V&L models including CLIP and its variants and GroupViT by a large margin. Models | 1 Foil | 2 Foils | 4 Foils
---|---|---|---
| MSCOCO | Flickr30k | MSCOCO | Flickr30k | MSCOCO | Flickr30k
| Accuracy | F1 | Accuracy | F1 | Accuracy | F1 | Accuracy | F1 | Accuracy | F1 | Accuracy | F1
ALIGN [11] | 86.03 | 0.93 | 87.70 | 0.93 | 79.47 | 0.89 | 81.43 | 0.90 | 71.03 | 0.83 | 74.57 | 0.85
AltCLIP [5] | 83.41 | 0.92 | 82.69 | 0.91 | 77.18 | 0.87 | 73.68 | 0.85 | 69.08 | 0.82 | 64.39 | 0.78
BridgeTower [29] | 97.26 | 0.99 | 96.83 | 0.98 | 95.71 | 0.98 | 94.46 | 0.97 | 92.61 | 0.96 | 90.81 | 0.95
CLIP [20] | 84.37 | 0.92 | 85.24 | 0.92 | 76.82 | 0.87 | 76.26 | 0.87 | 67.37 | 0.81 | 68.33 | 0.81
CLIPSeg [16] | 83.05 | 0.91 | 82.01 | 0.90 | 74.00 | 0.85 | 72.97 | 0.84 | 64.56 | 0.78 | 63.07 | 0.77
GroupViT [28] | 82.73 | 0.91 | 81.64 | 0.90 | 73.10 | 0.84 | 71.77 | 0.84 | 63.80 | 0.78 | 62.12 | 0.77
ViLT [12] | 95.53 | 0.98 | 94.29 | 0.97 | 92.83 | 0.96 | 91.85 | 0.96 | 88.74 | 0.94 | 87.38 | 0.93
|  |  |  |  |  |  |  |  |  |  |  |
Table 2: Examples of caption-foil pairs for which the CLIP model wrongly choose the foils instead of the captions. Type | Example
---|---
caption | A surfer is riding a wave in light blue water.
foil | A surfer is riding a wave in light brown water.
caption | A man in black jersey pitching in baseball game.
foil | A man in red jersey pitching in baseball game.
caption | A red traffic light at night next to a Christmas Tree.
foil | A white traffic light at night next to a Christmas Tree.

##  5 Results and Discussion

Table 2 shows the performance of different V&L models evaluated on the
ColorFoil. All the models achieve much higher accuracy compared to a baseline
random classifier with a 50% accuracy. CLIP obtains 83.1% accuracy while ViLT
and BridgeTower get substantially higher accuracy of 95.6% and 97.2%,
respectively on the 1-Foil experiment. It is worthwhile to mention that CLIP
is pre-trained with 400M images, although this model is outperformed by both
ViLT and BridgeTower pre-trained with only 4M images. BridgeTower
architecture, which contains multiple bridges to make connections between the
uni-modal encoders and the cross-modal encoder, achieves the highest accuracy.

The relatively poor performance of CLIP is also evident in its variants,
including AltCLIP and CLIPSeg. While the ALIGN model outperforms CLIP, it
still lags behind BridgeTower and ViLT. GroupViT, similar to CLIP, struggles
to achieve high performance. This performance trend is consistent across both
MSCOCO and Flickr30k datasets, reinforcing our observations. When presented
with more foils alongside the original caption, the models exhibit performance
degradation. Nonetheless, BridgeTower and ViLT maintain strong performance
even under these challenging conditions with more foils.

We present several pairings of captions and foils for which the CLIP model
incorrectly assigns higher probabilities to the foils (See Table 2). These
examples demonstrate that the CLIP model is unable to distinguish between
blue-brown, black-red, and red-white pairs, despite the fact that they are
visually distinct to most humans.

##  6 Limitations

We consider the 10 most common colors for our foils. However, our choice of
common colors is subjective and there might be other frequently used colors
that are not present in our foils.

##  7 Ethical Considerations

Training V&L models using images and corresponding texts that may contain
gender bias, private data, or harmful content presents challenges in manual
detection. To address this, we utilize the widely recognized MS COCO and
Flickr30k datasets to create the ColorFoil benchmark, as it provides a
reliable foundation [14, 19].

Ensuring reproducibility is a crucial aspect of scientific research. To foster
open research practices, we have made our code publicly accessible, allowing
others to reproduce and verify our findings.111Code is available at
https://github.com/samin9796/ColorFoil

##  8 Conclusion and Future Work

In this work, we introduce a novel benchmark, ColorFoil, derived from the MS
COCO and Flickr30k datasets, to assess the perception ability of the cutting-
edge V&L models to detect colors. To this end, we foil the colors from the
original captions and feed both caption and foil along with the corresponding
image to the model to observe whether it can provide a higher probability for
the caption or not. Seven state-of-the-art V&L models, including CLIP, ViLT,
ViT, and BridgeTower, have been benchmarked using the ColorFoil. While all
models outperform a random classifier, ViLT and BridgeTower are much more
capable to perceive colors compared to CLIP and ViT. This intriguing finding
is seen using both MS COCO and Flickr30k datasets, which strengthens our
analysis.

As part of our future work, we would like to evaluate the robustness of V&L
models on additional tasks by constructing foils that swap gender (man ->
woman), size (small -> large), emotions (smiling -> crying), and sentence
negation (playing football -> not playing football), etc.

## References

  * Agarwal et al. [2020] Vedika Agarwal, Rakshith Shetty, and Mario Fritz.  Towards causal vqa: Revealing and reducing spurious correlations by invariant and covariant semantic editing.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9690–9698, 2020.
  * Agrawal et al. [2016] Aishwarya Agrawal, Dhruv Batra, and Devi Parikh.  Analyzing the behavior of visual question answering models.  _arXiv preprint arXiv:1606.07356_ , 2016.
  * Bitton et al. [2021] Yonatan Bitton, Gabriel Stanovsky, Roy Schwartz, and Michael Elhadad.  Automatic generation of contrast sets from scene graphs: Probing the compositional consistency of gqa.  _arXiv preprint arXiv:2103.09591_ , 2021.
  * Chen et al. [2020] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  Uniter: Universal image-text representation learning.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX_ , pages 104–120. Springer, 2020.
  * Chen et al. [2023] Zhongzhi Chen, Guang Liu, Bo-Wen Zhang, Qinghong Yang, and Ledell Wu.  Altclip: Altering the language encoder in clip for extended language capabilities.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 8666–8682, 2023.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_ , 2020.
  * Dou et al. [2022] Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, et al.  An empirical study of training end-to-end vision-and-language transformers.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18166–18176, 2022.
  * Gokhale et al. [2020] Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang.  Mutant: A training paradigm for out-of-distribution generalization in visual question answering.  _arXiv preprint arXiv:2009.08566_ , 2020.
  * Goyal et al. [2017] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6904–6913, 2017.
  * Jabri et al. [2016] Allan Jabri, Armand Joulin, and Laurens Van Der Maaten.  Revisiting visual question answering baselines.  In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14_ , pages 727–739. Springer, 2016.
  * Jia et al. [2021] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International conference on machine learning_ , pages 4904–4916. PMLR, 2021.
  * Kim et al. [2021] Wonjae Kim, Bokyung Son, and Ildoo Kim.  Vilt: Vision-and-language transformer without convolution or region supervision.  In _International Conference on Machine Learning_ , pages 5583–5594. PMLR, 2021.
  * Li et al. [2020] Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang.  Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 11336–11344, 2020.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Lu et al. [2019] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  _Advances in neural information processing systems_ , 32, 2019.
  * Lüddecke and Ecker [2022] Timo Lüddecke and Alexander Ecker.  Image segmentation using text and image prompts.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 7086–7096, 2022.
  * Parcalabescu et al. [2021a] Letitia Parcalabescu, Michele Cafagna, Lilitta Muradjan, Anette Frank, Iacer Calixto, and Albert Gatt.  Valse: A task-independent benchmark for vision and language models centered on linguistic phenomena.  _arXiv preprint arXiv:2112.07566_ , 2021a.
  * Parcalabescu et al. [2021b] Letitia Parcalabescu, Albert Gatt, Anette Frank, and Iacer Calixto.  Seeing past words: Testing the cross-modal capabilities of pretrained V&L models on counting tasks.  In _Proceedings of the 1st Workshop on Multimodal Semantic Representations (MMSR)_ , pages 32–44, Groningen, Netherlands (Online), 2021b. Association for Computational Linguistics.
  * Plummer et al. [2015] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2641–2649, 2015.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Rosenberg et al. [2021] Daniel Rosenberg, Itai Gat, Amir Feder, and Roi Reichart.  Are vqa systems rad? measuring robustness to augmented data with focused interventions.  _arXiv preprint arXiv:2106.04484_ , 2021.
  * Shekhar et al. [2017] Ravi Shekhar, Sandro Pezzelle, Yauhen Klimovich, Aurélie Herbelot, Moin Nabi, Enver Sangineto, and Raffaella Bernardi.  Foil it! find one mismatch between image and language caption.  _arXiv preprint arXiv:1705.01359_ , 2017.
  * Shekhar et al. [2019] Ravi Shekhar, Ece Takmaz, Raquel Fernández, and Raffaella Bernardi.  Evaluating the representational hub of language and vision models.  _arXiv preprint arXiv:1904.06038_ , 2019.
  * Su et al. [2019] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  Vl-bert: Pre-training of generic visual-linguistic representations.  _arXiv preprint arXiv:1908.08530_ , 2019.
  * Suhr et al. [2018] Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi.  A corpus for reasoning about natural language grounded in photographs.  _arXiv preprint arXiv:1811.00491_ , 2018.
  * Webcolors [2023] Webcolors.  Python package index - webcolors 1.3, 2023.
  * Wolf et al. [2019] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_ , 2019.
  * Xu et al. [2022a] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang.  Groupvit: Semantic segmentation emerges from text supervision.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18134–18144, 2022a.
  * Xu et al. [2022b] Xiao Xu, Chenfei Wu, Shachar Rosenman, Vasudev Lal, and Nan Duan.  Bridge-tower: Building bridges between encoders in vision-language representation learning.  _arXiv preprint arXiv:2206.08657_ , 2022b.