(accv) Package accv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

# Exploring Scalability of Self-Training for Open-Vocabulary Temporal Action
Localization

Jeongseok Hyun1 Su Ho Han1 Hyolim Kang1
Joon-Young Lee2 Seon Joo Kim1
1Yonsei University 2Adobe Research

###### Abstract

The vocabulary size in temporal action localization (TAL) is constrained by
the scarcity of large-scale annotated datasets. To address this, recent works
incorporate powerful pre-trained vision-language models (VLMs), such as CLIP,
to perform open-vocabulary TAL (OV-TAL). However, unlike VLMs trained on
extensive image/video-text pairs, existing OV-TAL methods still rely on small,
fully labeled TAL datasets for training an action localizer. In this paper, we
explore the scalability of self-training with unlabeled YouTube videos for OV-
TAL. Our self-training approach consists of two stages. First, a class-
agnostic action localizer is trained on a human-labeled TAL dataset and used
to generate pseudo-labels for unlabeled videos. Second, the large-scale
pseudo-labeled dataset is combined with the human-labeled dataset to train the
localizer. Extensive experiments demonstrate that leveraging web-scale videos
in self-training significantly enhances the generalizability of an action
localizer. Additionally, we highlighted issues with existing OV-TAL evaluation
schemes and proposed a new evaluation protocol. Code is released at
https://github.com/HYUNJS/STOV-TAL.

###### Keywords:

Open-Vocabulary learning Temporal action localization Self-Training

##  1 Introduction

Temporal action localization (TAL) involves localizing and classifying action
instances in long videos. However, annotating large TAL datasets with
extensive action vocabularies is costly, limiting models to small
vocabularies. Developing TAL models that cover any target actions has been a
long-standing goal.

As relevant research directions, Open-set TAL [1] aims to localize all actions
by assigning base classes and marking novel actions as unknown, and Open-world
TAL [50] extends open-set TAL by integrating continual learning, where the
model is updated with annotations of novel actions after the initial training
phase. However, both settings are not suitable to achieve the goal since they
cannot classify unseen actions. In contrast, recent works [49, 29] adopted a
zero-shot (ZS) learning approach, leveraging semantic information from
language models [27, 10] to classify unseen actions. However, as the main goal
in ZS learning is to recognize concepts not seen during training [42], it
strictly restricts the use of samples of unseen action classes for training.
This stringent restriction hinders scaling TAL models with large-scale web
data.

Different from ZS learning, Zareian _et al_. [47] proposed open-vocabulary
(OV) learning for object detection that alleviates the restriction of training
data. Specifically, OV learning leverages image-text pairs for training, where
the classes used for testing may or may not be present in the training data.
This relaxed constraint allows vision-language models (VLMs) like CLIP [33],
which are pre-trained on extensive image-caption pairs, to demonstrate robust
zero-shot performance across various tasks, including open vocabulary temporal
action localization (OV-TAL).

Figure 1:  (Left) Our two-stage pipeline starts by training a class-agnostic
action localizer on a human-labeled (HL) TAL dataset. Then, it generates
pseudo-labels (PL) for unlabeled videos. Due to high annotation costs, HL is
limited in volume and category, whereas PL is free from them. The combined
dataset is then used to train the localizer, enhancing its generalization
across diverse action categories and video domains. (Right) Two types of data
sources are explored: in-domain, including videos from novel categories of the
target benchmark, and open-domain, including random web videos. The
scalability of our self-training approach is demonstrated by increasing mAPs.

Existing OV-TAL methods [15, 28, 32] utilize a VLM for classification.
Nevertheless, their class-agnostic action localizers continue to rely on
small, fully labeled TAL datasets, even within the OV learning framework. This
approach overlooks the vast potential of numerous web videos for enhancing
action localization. To address this, we explore self-training with additional
unlabeled videos to improve the generalization capability of action
localization. Although the class-agnostic action localizer can already detect
the novel actions [20], we found that self-training with large-scale web video
can improve cross-category generalization capabilities. Our training pipeline
is illustrated in Fig. 1 (Left).

We study two types of data sources for self-training: in-domain and open-
domain. In-domain data is acquired by splitting the target benchmark,
indicating that it includes videos from novel categories of the target
benchmark. On the other hand, Open-domain data includes random web videos.
Although in-domain data is considered gold data, it is confined to the
benchmark. In contrast, open-domain data is a scalable source as more videos
can be scraped from the web. Fig. 1 (Right) shows the results on the
FineAction dataset [23], which covers various video domains. Using 100k random
videos, we surpass the performance of in-domain data for both base and novel
classes. This proves the effectiveness of self-training with web videos. More
scaling results are presented in Tab. 4.3.

Figure 2:  Changes in model generalization ability when training on a human-
labeled (HL) dataset of base categories. (Left) Training an action localizer
on HL significantly reduces performance on novel categories; this issue is
mitigated through our self-training approach. (Right) Partial tuning of VLM
(CLIP [33]) initially increases accuracy for novel categories but leads to
decreased accuracy as base category accuracy saturates. In contrast, ViFi-CLIP
[34], fully tuned on a large video dataset, achieves higher accuracy. However,
partially tuning ViFi-CLIP on a small-scale TAL dataset (thumos14 [13])
diminishes its novel category accuracy, indicating a loss of generalization
ability.

We also explore changes in generalization ability when the action localizer
and classifier are trained on a human-labeled TAL dataset of base categories.
Following EffPrompt [15], we partially tune a VLM by adding a transformer
layer on the visual encoder and learnable prompts on the text encoder. Fig. 2
shows that the performance of both localizer and classifier decreases in novel
actions while base actions performance saturates. When the action localizer is
trained with our self-training approach, such loss of generalization ability
can be prevented. ViFi-CLIP, which is already fine-tuned on large video-text
data, does not benefit from partial tuning on a TAL dataset and is instead
negatively affected by it.

Furthermore, the results indicate that the model with optimal performance in
novel actions is under-fitted to base actions. In other words, a meticulous
training strategy [52, 26] is required to prevent overfitting to base actions.
However, existing evaluation schemes do not assess base actions but only novel
actions that are randomly sampled, leading to potentially misleading
interpretations. For the proper evaluation, we propose new OV-TAL benchmarks
with two features. First, we introduce the generalized zero-shot setting,
where the target categories are the union of base and novel, and report the
accuracy of all, base, and novel actions, respectively. Base and novel actions
are divided using Kinetics dataset [7] Second, we introduce cross-dataset
evaluation that rigorously tests the generalization capability for OV-TAL
across domains.

Our contributions can be summarized as follows:

  * •

We show the effectiveness and scalability of exploiting unlabeled videos with
self-training to acquire cross-category generalization ability for OV-TAL.

  * •

We introduce new benchmarks for rigorous evaluation of OV-TAL: generalized
zero-shot setting with Kinetics based category split and cross-dataset
generalization.

##  2 Related Works

###  2.1 Beyond Closed-Vocabulary TAL

####  2.1.1 Pre-VLM Era.

Enlarging the vocabulary size of TAL models has been a longstanding goal,
studied even before the advent of VLMs. AherNet [24] exploits the combination
of TAL and action classification datasets to expand the vocabulary size of TAL
through adversarial learning, but its vocabulary size is still limited to the
training dataset. Aiming to generalize to unseen actions, Zhang _et al_. [49]
explored zero-shot learning in TAL by leveraging the language model embeddings
[27] of seen and unseen actions. OpenTAL [1] tackles the open-set setting that
localizes and classifies unseen actions as the unknown category on top of seen
actions based on the learned evidential uncertainty.

####  2.1.2 Post-VLM Era.

Recently, OV-TAL has been explored based on a VLM [33]. The decoupled
framework consists of a class-agnostic action localizer and an open-vocabulary
action classifier is studied in [35, 15, 16]. Ju  _et al_. [15] adapted CLIP
[33] for the video domain with extra learnable modules: learnable prompts and
transformer encoder for temporal modeling. In the follow-up [16], the prompt
tuning method is further studied, such as LLM-generated and visually
conditioned prompting, and the use of optical flow with VLM is introduced.
Rathod  _et al_. [35] explored the performance using different VLMs [14, 33,
31] and modalities including optical flow and audio. STALE [28] is the one-
stage framework while UnLoc [46] unifies other video localization tasks, such
as action segmentation and moment retrieval.

####  2.1.3 Our Work.

We explore the scalability of self-training with web videos for OV-TAL,
especially aiming to enhance the generalization ability of “action
localization”.

###  2.2 Vision-Language Models

####  2.2.1 Image VLMs.

CLIP [33] shows remarkable performance in zero-shot image recognition, and the
generalization ability is demonstrated across diverse domains. This is
attributed to the utilization of large-scale image-text datasets scraped from
the web. For example, CLIP [33] and OpenCLIP [8] are trained on 400M and 2B
image-text pairs. Such scalability offers benefits over manually annotated
datasets, such as ImageNet-21k [9] with 14M images.

####  2.2.2 Video VLMs.

Compared to image VLMs, video VLMs are under-explored due to a lack of large-
scale video-caption datasets. Initially, image VLMs are adapted to the video
domain with prompt tuning and extra transformer layers to capture the temporal
relationship [30, 15]. ViFi-CLIP [34] shows that simply fine-tuning CLIP in
video snippets without temporal modeling improves the performance. Open-VCLIP
[41] prevents the loss of generalization during video fine-tuning of CLIP
through the regularization method. MAXI [22] explores the use of LLMs [18, 4]
to generate video-caption pairs in the Kinetics videos [7] while ViCLIP [39]
is trained on InternVid [39] dataset composed of 234M pairs.

####  2.2.3 Our Work.

We adopt ViFi-CLIP [34] as the frozen VLM in the proposed framework while the
recently proposed ones can also be adopted.

##  3 Methodology

To encompass largely available unlabeled untrimmed video to TAL, we adopt the
open-vocabulary learning setting into TAL. As a solution, we propose the
framework, Self-Training for Open-Vocabulary Temporal Action Localization,
namely, STOV-TAL. This section provides a rigorous definition of this setting
with the evaluation protocol. Then, we describe the straightforward yet
effective architecture and the training method that improves the
generalization ability of action localization.

Figure 3:  Comparison of ZS-TAL and OV-TAL training. In ZS-TAL, a training
dataset is confined to $\mathcal{C}_{train}$ which is strictly separated from
$\mathcal{C}_{test}$. OV-TAL allows the use of videos without TAL labels even
if the videos may contain actions of $\mathcal{C}_{test}$.

###  3.1 Problem Definition of OV-TAL

Input for OV-TAL consists of an untrimmed video, $\mathcal{V}$, and $C$ number
of target action categories, $\mathcal{C}=\\{c_{i}\\}_{i=1}^{C}$, which are
unbounded and can be flexibly manipulated by users. Its output consists of $M$
action instances, $\mathbf{\Psi}=\\{(t_{s},t_{e},c,s)_{i}\\}_{i=1}^{M}$, where
$t_{s}$, $t_{e}$, $c$, and $s$ denote the start and end times of the action,
the action category, and the confidence score, respectively.

Regarding the training data utilization, the OV-TAL setting offers a more
flexible approach, as depicted in Fig. 3. In ZS-TAL, training data is limited
to videos with human-annotated TAL labels corresponding to
$\mathcal{C}_{train}$ which is a disjoint set following
$\mathcal{C}_{train}\cap\mathcal{C}_{test}=\varnothing$. However, the OV-TAL
setting relaxes this strict restriction of training data. Specifically, we do
not limit the use of data sources which may include actions of
$\mathcal{C}_{test}$ unless the TAL annotations are provided together. This
allows for the incorporation of abundant unlabeled, random videos from the
web. As a result, training the action localizer is no longer limited to the
small-scale TAL datasets; instead, it involves exploiting data with diverse
categories and domains. Therefore, the OV-TAL setting is more suitable for
exploring generalization ability.

###  3.2 Evaluation Protocol for OV-TAL

Existing ZS-TAL benchmarks [15] face two issues: (1) they do not evaluate the
seen categories, but only unseen categories, and (2) they rely on random
category splits that do not consider the category frequency. To address these,
we introduce the generalized ZS evaluation setting with the Kinetics-based [7]
category split on the proposed OV-TAL benchmarks. The proposed benchmarks are
crucial at this stage for advancing OV-TAL further.

####  3.2.1 Generalized Zero-Shot Evaluation.

As shown by Fig. 3, the benchmark dataset is disjointly divided. In the ZS-TAL
benchmark, the model is trained on the training videos of
$\mathcal{C}_{train}$ and is evaluated on the validation videos of
$\mathcal{C}_{test}$, _i.e_., $\mathcal{C}_{target}=\mathcal{C}_{test}$. In
this way, setting the target categories by either $\mathcal{C}_{train}$ or
$\mathcal{C}_{test}$ is referred to the constrained zero-shot evaluation
protocol. As pointed out by Xian  _et al_. [44], the constrained setting
provides strong prior knowledge and makes the problem easier by not
considering $\mathcal{C}_{train}$, which are used for optimizing the model. On
the contrary, the generalized zero-shot evaluation protocol includes all
categories in the benchmark for evaluation, _i.e_.,
$\mathcal{C}_{target}=\mathcal{C}_{train}\cup\mathcal{C}_{test}$. In addition
to the accuracy of all categories, we also measure the accuracy of
$\mathcal{C}_{train}$ and $\mathcal{C}_{test}$ under this generalized
protocol.

####  3.2.2 Category Split.

Existing ZS-TAL benchmarks are based on the splits whose categories are
randomly chosen [49, 15]. Ju  _et al_. [15] proposed ten random splits with
ratios, _e.g_., 50%–50% for $\mathcal{C}_{train}$ and $\mathcal{C}_{test}$,
and the final accuracy is computed by averaging the results of each split. In
our Kinetics-based split, we regard actions covered in Kinetics as common
categories and use them as base categories ($\mathcal{C}_{B}$); those not
covered are considered rare actions and treated as novel categories
($\mathcal{C}_{N}$). Two reasons support this criterion: (1) The action list
in K400 is thoughtfully selected to encompass actions that are commonly
encountered in daily life. This selection criterion allows us to simulate a
challenging scenario where categories of varying frequency - frequent for
training and infrequent for testing - are utilized. (2) K400 is a common
choice for pre-training video backbones, and detecting actions not present in
K400 can demonstrate a more rigorous test of category generalization ability.

To automatically detect the category overlap, we utilize NLTK [2] to perform
stemming and lemmatization on the text of action categories and verify whether
the words of action category in the benchmark are present in the set of K400
actions. After this process, we derive the sets of
$(|\mathcal{C}_{B}|,|\mathcal{C}_{N}|)$ as $(90,110)$, $(12,8)$, and $(56,50)$
for ActivityNet v1.3, THUMOS14, and FineAction, respectively.

Figure 4:  Architecture. Features $\mathbf{F}_{\mathcal{V}}$ and
$\mathbf{F}_{\mathcal{T}}$ are generated by VLM video and text encoders from
video frames and action names, respectively The action localizer detects
class-agnostic action instances, and their features
($\mathbf{F}_{\mathcal{A}}$) are extracted using RoI-Align. Cosine
similarities between $\mathbf{F}_{\mathcal{A}}$ and $\mathbf{F}_{\mathcal{T}}$
are computed to assign the top-scoring category to each action instance. The
category score ($s_{c}$) is averaged with its actionness score ($s_{a}$) to
obtain the final confidence score ($s$).

###  3.3 STOV-TAL Architecture

We decouple the localization and classification of actions into two components
in the architecture: (1) class-agnostic action localizer; (2) open-vocabulary
action classifier based on the video VLM [34]. Fig. 4 illustrates the detailed
pipeline. Note that this architecture supports various CLIP-like VLMs, which
typically include separate uni-modal encoders for video and text, represented
as the video encoder (${\Phi}_{\mathcal{V}}$) and the text encoder
(${\Phi}_{\mathcal{T}}$).

####  3.3.1 Video Feature Extraction with VLM.

Frames are sampled from the input video using the overlapping sliding window
approach, commonly used in the TAL literature [48]. This takes consecutive
frames of window size and slides a window with the stride size. As a result, a
long untrimmed video is divided into multiple overlapping video snippets,
represented as
$\mathcal{V}=\\{\mathcal{V}_{i}\\}_{i=1}^{S}\in\mathbb{R}^{S\times T\times
H\times W\times 3}$, where $S$ is the number of snippets and $T$ is the window
size. For each snippet, the VLM’s video encoder outputs the feature vector,
${\Phi}_{\mathcal{V}}(\mathcal{V}_{i})\in\mathbb{R}^{D}$, where $D$ is the
dimension of the feature. By processing the entire video, we obtain the video
snippet features,
$\mathbf{F}_{\mathcal{V}}={\Phi}_{\mathcal{V}}(\mathcal{V})\in\mathbb{R}^{S\times
D}$.

####  3.3.2 Detecting Every Action.

In this stage, the primary goal is to detect every action instance with its
actionness score $s_{a}$, rather than predicting its action category. To
achieve this goal, we employ the action localizer that produces class-agnostic
action instances, represented as
$\mathbf{\Psi}_{agn}=\\{(t_{s},t_{e},\varnothing,s_{a})_{i}\\}_{i=1}^{M}$.
Here, $\varnothing$ indicates the absence of the action category. Our
framework offers flexibility in selecting the architecture of the action
localizer, provided that the output format is compatible. It accommodates both
two-stage methods [20, 45] and one-stage methods [19, 48] when trained in a
class-agnostic manner. The class-agnostic action localizer is improved with
stronger generalization ability through the proposed self-training process,
detailed in Sec. 3.4.

To make the paper self-contained, we provide a concise overview of the adopted
action localizer [48]. This localizer includes a convolutional projection
layer, a multi-scale transformer encoder, and parallel convolutional decoders
in sequence, as described in the action localizer block of Fig. 4. The first
two modules update the video snippet features to consider both temporal
locality and long-term contexts. Temporal downsampling operations are
integrated into the encoder to extract multi-scale features for capturing the
actions at different temporal scales due to the varying granularity of
actions. Two parallel convolutional decoders are used to predict the
actionness score and the offset for start and end time in every snippet across
the temporal axis at each scale.

####  3.3.3 Classifying Every Action.

We use an open-vocabulary action classifier for assigning the most appropriate
action category ($c$) among the input actions to the previously detected
action instances ($\mathbf{\Psi}_{agn}$), resulting in the class-aware action
instances ($\mathbf{\Psi}_{cls}$). Following CLIP [33], the input actions are
tokenized by the lower-cased byte pair encoding [36] with the start and end
tokens. The tokenized actions are encoded through the VLM’s text encoder as
the text features, $\mathbf{F}_{\mathcal{T}}\in\mathbb{R}^{C\times D}$, where
$C$ is the number of input action categories. The visual features of each
action instance are extracted from the VLM’s video snippet features by the
RoI-Align operation [12] with its start and end times. Then, we obtain
$\mathbf{F}_{\mathcal{A}}\in\mathbb{R}^{M\times D}$, where $M$ is the number
of instances.

Based on these features, we compute the cosine similarity between
$\mathbf{F}_{\mathcal{A}}$ and $\mathbf{F}_{\mathcal{T}}$, which is a matrix
of $\mathbb{R}^{M\times C}$. It is normalized by the softmax operation in the
axis of the category after scaling it by the temperature parameter. This
matrix value indicates each instance’s class confidence scores ($s_{c}$).
While we can choose top-$K$ action categories for each instance, we only
choose the top-scoring action category as the default setting. Finally, the
actionness score, $s_{a}$, and the category confidence score, $s_{c}$, of each
action instance are fused by the geometric mean to compute the final
confidence score, $s$. The output from this stage is class-aware action
instances, represented as
$\mathbf{\Psi}_{cls}=\\{(t_{s},t_{e},c,s)_{i}\\}_{i=1}^{M}$. As the post-
processing, softNMS [3] is applied to remove the duplicates and obtain the
final action instances.

###  3.4 Learning to Detect Every Action

A class-agnostic action localizer has demonstrated the ability to localize
actions that were not seen during training [20, 15]. Leveraging its cross-
category generalization ability, our architecture can detect and classify any
input actions without requiring specialized training methods. Nonetheless, its
generalization ability remains limited, particularly for unseen domains
(_i.e_., cross-dataset evaluation). This is attributed to the small scale of
the TAL datasets that lack diverse action categories or domains. Inspired by
the semi-supervised learning commonly employed in data-scarce scenarios, we
aim to enhance the action localizer’s generalization ability through the self-
training method [17, 43]. As illustrated in Fig. 1, self-training comprises
two stages: (1) Training a model on a labeled dataset; and (2) Training the
model on an unlabeled dataset, using pseudo-labels generated by the previous
model.

####  3.4.1 Learning from Labeled Videos.

In the first stage, the action localizer is trained using the human-labeled
TAL dataset. Since we employ the class-agnostic action localizer, the
classification is about distinguishing the foreground or background of actions
in a video. As described in Sec 3.3, the output from the two decoders,
$\mathbf{\Psi}_{agn}$, includes the start and end times of the action and the
actionness score. These are trained with the DIoU loss [51] and Focal loss
[21], respectively.

####  3.4.2 Learning from Unlabeled Videos.

In the second stage, we exploit additional unlabeled videos for training. To
obtain pseudo-labels of these videos, the action localizer trained in the
first stage is employed to generate the class-agnostic action instances. Note
that the raw output includes many duplicated and noisy instances. We use the
NMS [3] operation and the fixed threshold value on the actionness score to
remove them. Then, this pseudo-dataset is combined with the first stage’s
labeled dataset to form the joint dataset. Our pseudo-label selection process
requires finding the proper threshold value. In this regard, the sensitivity
analysis of pseudo-labels in Sec 4.4 supports the robustness of this process.
Although advanced pseudo-labeling methods, such as adaptive thresholding [40],
can be adopted, this paper focuses on exploring the generalization ability of
action localization using the straightforward self-training method.

The process of selecting videos is crucial for self-training, where we explore
two distinct settings: (1) Using in-domain (ID) data, and (2) Using open-
domain (OD) data. In the first setting, we employ the training set of the
target benchmark without labels. For instance, videos containing novel actions
are chosen for self-training in the cross-category evaluation. However, this
video sampling policy is geared towards the evaluation dataset. Utilizing
videos from the same dataset ensures in-domain, creating a favorable
environment for achieving higher accuracy. Also, the scaling-up effect with a
larger volume of data is constrained by a fixed and small number of videos. On
the other hand, the second setting utilizes random YouTube videos whose video
IDs are sourced from K600 [6]. In contrast to the Kinetics dataset, where
videos are trimmed to focus on the specific labeled action category, we use
untrimmed videos.

The training objectives used in the first stage are also used in this stage.
When the scale of the unlabeled dataset is significantly larger than that of
the labeled dataset and their distributions differ, the action localizer may
struggle to learn robust knowledge from the labeled dataset. Thus, we adopt
the Mean Teacher framework [37] and initialize the teacher model with the
action localizer trained in the first stage. In this teacher-student
framework, the teacher model is used for inference and is updated with the
student model in the manner of exponential moving average as formulated in Eq.
1:

$\theta_{iter}^{\prime}=(1-\lambda)\theta_{iter}+\lambda{\theta}_{iter-1}^{\prime},$
(1)

where $\theta^{\prime}$ and $\theta$ represent the parameters of the teacher
and student models. $\lambda$ is the smoothing coefficient, and $iter$
indicates the training iteration step.

##  4 Experiments

###  4.1 Datasets

We use the following datasets for experiments. ActivityNet v1.3 (ANET) [5]
consists of 20,000 videos that cover 200 action categories. Since its target
categories are about activity or event, only $1$ – $2$ of long action
instances exist in a video. THUMOS14 (TH14) [13] contains 413 videos with 20
sports-related action categories. Here, about 15 action instances exist per
video on average. FineAction(FinAct) [23] includes 16,732 videos, encompassing
106 different action classes. On average, each video in this dataset contains
6 action instances. In contrast to THUMOS14, FineAction covers various
domains, including not only sports but also actions related to household,
socializing, and personal care activities. Therefore, FineAction would better
evaluate the generalization capability of an action localizer.

For self-training data, we use the video ID provided in Kinetics-600 [6] and
scrape the full video from YouTube. After filtering out very long videos
exceeding 2.5 hours, about 332k videos are collected. Then, we build its
subsets of 5k, 10k, 50k, 100k, and 200k videos, where a larger set encompasses
a smaller set.

###  4.2 Evaluation Metrics

We use the standard mean average precision (mAP) which is widely used to
evaluate TAL methods. For ZS-TAL benchmarks, we follow the conventions [15,
28] that report the mAPs at different temporal intersection over union (tIoU)
thresholds and their average: $[0.3:0.1:0.7]$ and $[0.5:0.05:0.95]$ tIoU
values for THUMOS14 [13] and ActivityNet v1.3 [5]. For OV-TAL benchmarks, we
report the mAP of all, base, and novel categories defined in each dataset at a
tIoU of 0.5. These metrics are denoted by $\text{mAP}^{50}_{A}$,
$\text{mAP}^{50}_{B}$, and $\text{mAP}^{50}_{N}$, respectively.

Across the experiments, we report the mAPs of the framework trained in full-
shot (FS), without self-training (w/o ST), and with self-training (ST)
settings. With self-training, we present the results of using in-domain and
open-domain data, denoted by (ID-ST) and (OD-ST), respectively.

###  4.3 Implementation Details

For the video VLM, we adopt ViFi-CLIP [34] which is based on ViT-Base/16 [11]
visual encoder and is pre-trained as video-text pairs using the K400 dataset
[7]. In the pre-processing step of inputs, we interpolate videos into 30 fps
and resize frames to 256 pixels on the shorter side, followed by a center crop
of 224. Regarding video feature extraction, we use the window size of 16
frames and stride size of 4 frames. For the textual prompt input, we directly
use the action category name without any prompt engineering. We attach more
configurations for training and inference, such as training epoch, learning
rate, and NMS, in the supplementary material.

Table 1:  Scalability with varying volume of videos for self-training.

Extra Videos ($\\#$) | Generalized | Constrained
---|---|---
mAP${}_{A}^{50}$ | mAP${}_{B}^{50}$ | mAP${}_{N}^{50}$ | mAP${}_{B}^{50}$ | mAP${}_{N}^{50}$
0k (w/o ST) | 15.0 | 22.6 | 6.4 | 23.3 | 7.3
1.5k (ID-ST) | 15.7 | 22.8 | 7.8 | 23.4 | 9.4
1-6 5k (OD-ST) | 15.6 | 22.3 | 8.0 | 23.1 | 9.7
10k (OD-ST) | 15.6 | 23.0 | 7.4 | 23.7 | 8.8
50k (OD-ST) | 16.3 | 23.8 | 7.8 | 24.6 | 9.2
100k (OD-ST) | 16.8 | 24.4 | 8.2 | 25.2 | 9.4
200k (OD-ST) | 17.0 | 24.6 |  8.5 | 25.4 | 9.9
332k (OD-ST) |  17.2 |  25.1 | 8.4 |  25.8 | 9.5
Full-Shot (FS) | 18.8 | 23.5 | 13.5 | 24.5 | 16.0

Table 2: Ablation study of VLMs as backbone.

VLM | Generalized | Constrained
---|---|---
$\text{mAP}^{50}_{A}$ | $\text{mAP}^{50}_{B}$ | $\text{mAP}^{50}_{N}$ | $\text{mAP}^{50}_{B}$ | $\text{mAP}^{50}_{N}$
CLIP-B [33] | 10.3 | 15.2 | 4.8 | 16.0 | 5.7
ViCLIP-B [39] | 11.6 | 16.2 | 6.5 | 17.6 | 7.6
ViCLIP-L [39] | 12.9 | 18.7 | 6.5 | 19.9 | 7.9
ViFi-CLIP-B [34] | 15.0 | 22.6 | 6.4 | 23.3 | 7.3

Table 3: Ablation study of score fusion methods.

Score Fusion | Generalized | Constrained
---|---|---
$\text{mAP}^{50}_{A}$ | $\text{mAP}^{50}_{B}$ | $\text{mAP}^{50}_{N}$ | $\text{mAP}^{50}_{B}$ | $\text{mAP}^{50}_{N}$
Geometric Mean | 15.0 | 22.6 | 6.4 | 23.3 | 7.3
Arithmetic Mean | 13.5 | 20.5 | 5.6 | 21.4 | 6.7
Actionness Score ($s_{a}$) | 13.9 | 21.7 | 5.2 | 22.4 | 6.0
Category Score ($s_{c}$) | 1.0 | 1.4 | 0.5 | 1.3 | 0.5

###  4.4 Main Results

We perform extensive experiments on the FineAction dataset from the OV-TAL
benchmark to evaluate the effectiveness of our proposed method.

Scalability Evaluation. Tab. 4.3 shows that self-training using open-domain
data achieves increasing mAP${}_{B}^{50}$ and mAP${}_{N}^{50}$ with larger
volume and outperforms the results of using in-domain data. We achieve these
noteworthy results without a specially designed loss function or pseudo-
labeling strategy. This demonstrates the high potential of self-training for
OV-TAL with large-scale web videos.

The categories of scraped videos most likely belong to $\mathcal{C}_{B}$ since
Kinetics dataset [7] is used for sourcing videos for self-training and
deciding $\mathcal{C}_{B}$. Thus, the improvement in $\text{mAP}_{B}$ is
consistent, compared to fluctuating $\text{mAP}_{N}$.

VLM Backbone. Tab. 3 presents the results of using different VLMs for STOV-
TAL. Both ViCLIP and ViFi-CLIP outperform CLIP, benefiting from fine-tuning on
large-scale video-text data. ViCLIP shows higher mAPN, while ViFi-CLIP
exhibits mAPB since it is trained on Kinetics data, whereas ViCLIP is trained
on newly crawled YouTube videos.

Score Fusion. Tab. 3 studies the score fusion method used in the inference
time. The mAP significantly decreases without relying on the actionness score
since the background category is not modeled in our frozen VLM. The arithmetic
mean operation shows the lower mAPs than only using the $s_{a}$ since some
high $s_{c}$ may dominate the final score. The different scale between $s_{a}$
and $s_{c}$ is caused by the different normalization operations, _e.g_.,
sigmoid and softmax functions. On the other hand, the geometric mean operation
alleviates this problem and leads to optimal performance.

Figure 5:  Sensitivity analysis of pseudo-labels in self-training. Figure 6:
Quality analysis of pseudo-labels. Correlation between the actionness of
pseudo-labels and their tIoU between GT instances is measured on THUMOS14 and
FineAction. The left graphs show the average of tIoU values in each interval
of actionness (0.1). The right graphs show the scatter plot.

Robustness of Self-training. One of the important factors in self-training is
whether it is robust against the threshold values used for pseudo-labels. As
shown in Fig. 5, the self-training method can achieve consistent performance
when we filter out the noisy action instances with a low confidence score.

Quality of Pseudo-labels. Fig. 6 shows the evident positive correlation
between the actionness score and tIoU with GT action instances in the pseudo-
labels. On the right side, we can observe that low-confident predictions are
densely located at low tIoU values. Thus, filtering out pseudo-labels with
very low actionness scores is sufficient to achieve decent performance.

###  4.5 Results of OV-TAL Benchmark

We present the comparison with the previous method using OpenTAL [1] with
ViFi-CLIP for classification due to the absence of reproducible OV-TAL methods
[28, 15]. The details of using OpenTAL for OV-TAL are attached in supp.

Table 4:  Evaluation of cross-category OV-TAL benchmark. For (w/o ST), the
model is trained on $\mathcal{C}_{B}$ of target benchmark (TH14/FinAct). For
(ID-ST), training videos of $\mathcal{C}_{N}$ of the target benchmark are
used. For (OD-ST), web-scraped videos are utilized. For reference of upper-
bound, full-shot results are shown in gray. † indicates our reproduced
results.

Methods | Backbone | THUMOS14 | FineAction
---|---|---|---
Generalized | Constrained | Generalized | Constrained
mAP${}_{A}^{50}$ | mAP${}_{B}^{50}$ | mAP${}_{N}^{50}$ | mAP${}_{B}^{50}$ | mAP${}_{N}^{50}$ | mAP${}_{A}^{50}$ | mAP${}_{B}^{50}$ | mAP${}_{N}^{50}$ | mAP${}_{B}^{50}$ | mAP${}_{N}^{50}$
OpenTAL† [1] | I3D [7] | 30.2 | 42.7 | 11.3 | 42.6 | 11.9 | 6.2 | 10.4 | 1.4 | 11.0 | 1.8
STOV-TAL (w/o ST) | ViFi-CLIP-B [34] | 45.7 | 66.0 | 15.3 | 65.9 | 16.2 | 15.0 | 22.6 | 6.4 | 23.3 | 7.3
STOV-TAL (ID-ST) | ViFi-CLIP-B [34] | 46.6 | 65.8 | 17.9 | 65.8 | 18.2 | 15.7 | 22.8 | 7.8 | 23.4 | 9.4
STOV-TAL (OD-ST) | ViFi-CLIP-B [34] | 44.2 | 62.6 | 16.7 | 62.5 | 17.0 | 17.2 | 25.1 | 8.4 | 25.8 | 9.5
STOV-TAL (FS) | ViFi-CLIP-B [34] | 50.2 | 65.8 | 26.8 | 66.0 | 27.5 | 18.8 | 23.5 | 13.5 | 24.5 | 16.0

Table 5:  Evaluation of cross-dataset OV-TAL benchmark. For (w/o ST), the
model is trained on $\mathcal{C}_{B}$ of TH14. For (ID-ST), all training
videos of FinAct, including $\mathcal{C}_{B}$ and $\mathcal{C}_{N}$, are used
as the source data for self-training. For (OD-ST), web-scraped videos are
exploited. † indicates our reproduced results.

Methods | Backbone | THUMOS14 | FineAction
---|---|---|---
Generalized | Constrained | Generalized | Constrained
mAP${}_{A}^{50}$ | mAP${}_{B}^{50}$ | mAP${}_{N}^{50}$ | mAP${}_{B}^{50}$ | mAP${}_{N}^{50}$ | mAP${}_{A}^{50}$ | mAP${}_{B}^{50}$ | mAP${}_{N}^{50}$ | mAP${}_{B}^{50}$ | mAP${}_{N}^{50}$
OpenTAL† [1] | I3D [7] | 30.2 | 42.7 | 11.3 | 42.6 | 11.9 | 3.7 | 5.7 | 1.4 | 5.1 | 1.7
STOV-TAL (w/o ST) | ViFi-CLIP-B [34] | 45.7 | 66.0 | 15.3 | 65.9 | 16.2 | 6.3 | 9.5 | 2.7 | 9.7 | 3.6
STOV-TAL (ID-ST) | ViFi-CLIP-B [34] | 44.2 | 64.4 | 13.9 | 64.5 | 14.5 | 7.4 | 11.0 | 3.4 | 11.2 | 4.4
STOV-TAL (OD-ST) | ViFi-CLIP-B [34] | 44.2 | 62.6 | 16.7 | 62.5 | 17.0 | 6.9 | 10.1 | 3.2 | 10.3 | 4.0

Cross-category Evaluation. Tab. 4 shows our results in generalized and
constrained ZS evaluation. The mAPs of unseen classes ($\mathcal{C}_{N}$) are
lower than those in Tab. 6. This reflects the difficulty of detecting rare
actions that less frequently occur in training data.

Self-training the model with in-domain data improves mAP${}_{N}^{50}$ in both
THUMOS14 and FineAction. Using open-domain data also enhances the gain of
mAP${}_{N}^{50}$, indicating that our self-training method significantly
boosts the model’s cross-category generalization ability Although self-
training with open-domain data results in a lower mAP${}_{B}^{50}$, this is
justified by cross-dataset evaluation.

Cross-dataset Evaluation. Tab. 5 presents the results of cross-dataset
generalization. Under both ID-ST and OD-ST settings, we can observe the
improvement of all categories in FineAction, compared to the results without
self-training. This demonstrates the enhanced cross-dataset generalization
ability from the self-training with web-scraped data.

However, dropped mAPs of TH14 under ID-ST indicate that the model is fitted to
the target dataset. In contrast, with open-domain data, mAPN increases,
supporting its potential of achieving generalization ability in diverse
domains.

Table 6:  Evaluation of ZS-TAL benchmark. The results are based on RGB only
without optical flow. In each setting, the best for each metric is bolded.
Full-shot results are shown in gray for reference of upper-bound. † indicates
our reproduced results. The values not provided are filled by “-”.

Evaluation Setting | Methods | Backbone | THUMOS14 [13] | ActivityNet v1.3 [5]
---|---|---|---|---
0.3 | 0.4 | 0.5 | 0.6 | 0.7 | Avg. | 0.5 | 0.75 | 0.95 | Avg.
Full-shot 100% Seen 0% Unseen | ActionFormer† [48] | ViFi-CLIP-B [34] | 72.8 | 67.4 | 57.3 | 45.2 | 29.7 | 54.5 | 49.5 | 31.2 | 4.3 | 30.9
EffPrompt [15] | CLIP-B [33] | 50.8 | 44.1 | 35.8 | 25.7 | 15.7 | 34.5 | 44.0 | 27.0 | 5.1 | 27.3
STALE [28] | CLIP-B [33] | 60.6 | 53.2 | 44.6 | 36.8 | 26.7 | 44.4 | 54.3 | 34.0 | 7.7 | 34.3
UnLoc [46] | CLIP-B [33] | - | - | - | - | - | - | 54.6 | - | - | -
UnLoc [46] | CLIP-L [33] | - | - | - | - | - | - | 59.3 | - | - | -
STOV-TAL (FS) | CLIP-B [33] | 47.5 | 41.7 | 33.0 | 24.7 | 15.4 | 32.5 | 37.5 | 23.1 | 1.8 | 22.7
STOV-TAL (FS) | ViFi-CLIP-B [34] | 65.3 | 60.6 | 50.2 | 39.0 | 26.0 | 48.2 | 43.7 | 26.8 | 2.0 | 26.4
Zero-shot 75% Seen 25% Unseen | EffPrompt [15] | CLIP-B [33] | 39.7 | 31.6 | 23.0 | 14.9 | 7.5 | 23.3 | 37.6 | 22.9 | 3.8 | 23.1
STALE [28] | CLIP-B [33] | 40.5 | 32.3 | 23.5 | 15.3 | 7.6 | 23.8 | 38.2 | 25.2 | 6.0 | 24.9
UnLoc [46] | CLIP-B [33] | - | - | - | - | - | - | 40.2 | - | - | -
UnLoc [46] | CLIP-L [33] | - | - | - | - | - | - | 48.8 | - | - | -
Ju  _et al_. [16] | CLIP-B [33] | 46.3 | 39.0 | 29.5 | 18.3 | 8.7 | 28.4 | 42.0 | 25.8 | 3.2 | 25.9
STOV-TAL (w/o ST) | CLIP-B [33] | 47.8 | 39.1 | 28.4 | 17.6 | 9.1 | 28.4 | 47.0 | 28.1 | 1.6 | 27.9
STOV-TAL (w/o ST) | ViFi-CLIP-B [34] | 56.7 | 47.2 | 34.3 | 22.8 | 11.3 | 34.5 | 51.7 | 30.9 | 1.8 | 30.5
STOV-TAL (ID-ST) | ViFi-CLIP-B [34] | 59.5 | 50.2 | 37.5 | 24.6 | 12.5 | 36.9 | 52.0 | 30.6 | 1.2 | 30.1
STOV-TAL (OD-ST) | ViFi-CLIP-B [34] | 58.2 | 48.2 | 35.1 | 23.0 | 11.8 | 35.2 | - | - | - | -
| STOV-TAL (FS) | ViFi-CLIP-B [34] | 67.5 | 60.8 | 47.7 | 34.8 | 21.8 | 46.5 | 52.6 | 31.5 | 2.3 | 31.3
Zero-shot 50% Seen 50% Unseen | EffPrompt [15] | CLIP-B [33] | 37.2 | 29.6 | 21.6 | 14.0 | 7.2 | 21.9 | 32.0 | 19.3 | 2.9 | 19.6
STALE [28] | CLIP-B [33] | 38.3 | 30.7 | 21.2 | 13.8 | 7.0 | 22.2 | 32.1 | 20.7 | 5.9 | 20.5
UnLoc [46] | CLIP-B [33] | - | - | - | - | - | - | 36.9 | - | - | -
UnLoc [46] | CLIP-L [33] | - | - | - | - | - | - | 43.7 | - | - | -
Ju  _et al_. [16] | CLIP-B [33] | 42.3 | 34.7 | 25.8 | 16.2 | 7.5 | 25.3 | 34.3 | 20.8 | 3.0 | 21.0
STOV-TAL (w/o ST) | CLIP-B [33] | 44.2 | 35.7 | 25.7 | 16.5 | 8.0 | 26.0 | 42.1 | 25.0 | 1.3 | 24.8
STOV-TAL (w/o ST) | ViFi-CLIP-B [34] | 53.4 | 43.1 | 31.1 | 19.7 | 9.8 | 31.5 | 48.1 | 28.4 | 1.3 | 28.0
STOV-TAL (ID-ST) | ViFi-CLIP-B [34] | 56.3 | 46.1 | 34.4 | 21.9 | 11.3 | 34.0 | 48.4 | 28.7 | 0.8 | 27.9
STOV-TAL (OD-ST) | ViFi-CLIP-B [34] | 54.3 | 43.7 | 32.5 | 21.4 | 10.6 | 32.5 | - | - | - | -
| STOV-TAL (FS) | ViFi-CLIP-B [34] | 68.2 | 62.5 | 50.7 | 38.2 | 24.6 | 48.8 | 49.4 | 29.9 | 2.2 | 29.6

###  4.6 Results of ZS-TAL Benchmark

Tab. 6 presents the results of existing methods and ours in the ZS-TAL
benchmark. First, we validate the effectiveness of our decoupled architecture,
STOV-TAL. In both full-shot and zero-shot evaluation, a large improvement is
shown by replacing the VLM backbone from CLIP [33] to ViFi-CLIP [34]. Based on
this result, we expect higher mAPs by adopting the advanced video-language
models, supporting the expandability of our framework. While mAPs of STOV-TAL
are lower than those of ActionFormer in full-shot evaluation, ActionFormer
cannot perform zero-shot inference, and our VLM-based classifier is not fine-
tuned on the TAL dataset.

In addition, the decoupled architecture benefits from adopting the state-of-
the-art action localizer. This is supported by the results from the zero-shot
evaluation, comparing STOV-TAL (w/o ST) and the others using the same VLM
backbone (CLIP-B). Our VLM-based action classifier is not fine-tuned on the
benchmark, compared to the others adopting the prompt-tuning paradigm [53] to
fine-tune and improve action classification accuracy. Thus, our high mAP is
attributed to the enhanced action localizer. Note that the others show higher
mAP in the full-shot results, but these are due to the fine-tuned classifier.

In THUMOS14, significant improvements are achieved from self-training with in-
domain data. For example, with a tIoU at 0.5, mAP increases as 31.4
$\rightarrow$ 34.4 and 34.3 $\rightarrow$ 37.5 on 50%–50% and 75%–25% zero-
shot evaluation, respectively. In THUMOS14, using open-domain data results in
smaller improvements as the evaluation split of the dataset is confined to the
relatively narrow sports domain.

Self-training shows only marginal improvement on the ActivityNet dataset. This
is attributed to the characteristics of ActivityNet which lacks the diversity
of action duration. In zero-shot evaluation, the mAP of ZS nearly matches that
of FS under the same VLM; for instance, in the 75%–25% split, the mAP of ZS
and FS, with a tIoU at 0.5, are 51.7 and 52.6, respectively, showing a gap of
only 0.9. This suggests that ActivityNet does not demand cross-category
generalization in action localization, resulting in limited gains from self-
training.

##  5 Conclusion

In this study, we investigated the scalability of action localization in the
OV-TAL setting. Through the utilization of large-scale web data, we
demonstrated significant improvements in the generalizability of class-
agnostic action localizers. Coupled with a rigorous evaluation protocol we
proposed, we aim for our work to serve as a robust benchmark for future
research in this field.

## References

  * [1] Bao, W., Yu, Q., Kong, Y.: Opental: Towards open set temporal action localization. In: CVPR. pp. 2979–2989 (2022)
  * [2] Bird, S., Klein, E., Loper, E.: Natural language processing with Python: analyzing text with the natural language toolkit. " O’Reilly Media, Inc." (2009)
  * [3] Bodla, N., Singh, B., Chellappa, R., Davis, L.S.: Soft-nms–improving object detection with one line of code. In: ICCV. pp. 5561–5569 (2017)
  * [4] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. NeurIPS 33, 1877–1901 (2020)
  * [5] Caba Heilbron, F., Escorcia, V., Ghanem, B., Carlos Niebles, J.: Activitynet: A large-scale video benchmark for human activity understanding. In: CVPR. pp. 961–970 (2015)
  * [6] Carreira, J., Noland, E., Banki-Horvath, A., Hillier, C., Zisserman, A.: A short note about kinetics-600. arXiv preprint arXiv:1808.01340 (2018)
  * [7] Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics dataset. In: CVPR. pp. 6299–6308 (2017)
  * [8] Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., Jitsev, J.: Reproducible scaling laws for contrastive language-image learning. In: CVPR. pp. 2818–2829 (2023)
  * [9] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: CVPR. pp. 248–255. IEEE (2009)
  * [10] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: NACCL. pp. 4171–4186 (2019)
  * [11] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: ICLR (2021), https://openreview.net/forum?id=YicbFdNTTy
  * [12] He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: ICCV. pp. 2961–2969 (2017)
  * [13] Idrees, H., Zamir, A.R., Jiang, Y.G., Gorban, A., Laptev, I., Sukthankar, R., Shah, M.: The thumos challenge on action recognition for videos “in the wild”. CVIU 155, 1–23 (2017)
  * [14] Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q., Sung, Y.H., Li, Z., Duerig, T.: Scaling up visual and vision-language representation learning with noisy text supervision. In: ICML. pp. 4904–4916. PMLR (2021)
  * [15] Ju, C., Han, T., Zheng, K., Zhang, Y., Xie, W.: Prompting visual-language models for efficient video understanding. In: ECCV. pp. 105–124. Springer (2022)
  * [16] Ju, C., Li, Z., Zhao, P., Zhang, Y., Zhang, X., Tian, Q., Wang, Y., Xie, W.: Multi-modal prompting for low-shot temporal action localization. arXiv preprint arXiv:2303.11732 (2023)
  * [17] Li, H., Wu, Z., Shrivastava, A., Davis, L.S.: Rethinking pseudo labels for semi-supervised object detection. In: AAAI. vol. 36, pp. 1314–1322 (2022)
  * [18] Li, J., Li, D., Xiong, C., Hoi, S.: Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In: ICML. pp. 12888–12900. PMLR (2022)
  * [19] Lin, C., Xu, C., Luo, D., Wang, Y., Tai, Y., Wang, C., Li, J., Huang, F., Fu, Y.: Learning salient boundary feature for anchor-free temporal action localization. In: CVPR. pp. 3320–3329 (2021)
  * [20] Lin, T., Liu, X., Li, X., Ding, E., Wen, S.: Bmn: Boundary-matching network for temporal action proposal generation. In: ICCV. pp. 3889–3898 (2019)
  * [21] Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: ICCV. pp. 2980–2988 (2017)
  * [22] Lin, W., Karlinsky, L., Shvetsova, N., Possegger, H., Kozinski, M., Panda, R., Feris, R., Kuehne, H., Bischof, H.: Match, expand and improve: Unsupervised finetuning for zero-shot action recognition with language knowledge. In: ICCV. pp. 2851–2862 (2023)
  * [23] Liu, Y., Wang, L., Wang, Y., Ma, X., Qiao, Y.: Fineaction: A fine-grained video dataset for temporal action localization. IEEE TIP 31, 6937–6950 (2022)
  * [24] Long, F., Yao, T., Qiu, Z., Tian, X., Luo, J., Mei, T.: Learning to localize actions from moments. In: ECCV. pp. 137–154. Springer (2020)
  * [25] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: ICLR (2019)
  * [26] Ma, C., Liu, Y., Deng, J., Xie, L., Dong, W., Xu, C.: Understanding and mitigating overfitting in prompt tuning for vision-language models. IEEE TCSVT (2023)
  * [27] Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. NeurIPS 26 (2013)
  * [28] Nag, S., Zhu, X., Song, Y.Z., Xiang, T.: Zero-shot temporal action detection via vision-language prompting. In: ECCV. pp. 681–697. Springer (2022)
  * [29] Nag, S., Goldstein, O., Roy-Chowdhury, A.K.: Semantics guided contrastive learning of transformers for zero-shot temporal activity detection. In: WACV. pp. 6243–6253 (2023)
  * [30] Ni, B., Peng, H., Chen, M., Zhang, S., Meng, G., Fu, J., Xiang, S., Ling, H.: Expanding language-image pretrained models for general video recognition. In: ECCV. Springer (2022)
  * [31] Pham, H., Dai, Z., Ghiasi, G., Kawaguchi, K., Liu, H., Yu, A.W., Yu, J., Chen, Y.T., Luong, M.T., Wu, Y., Tan, M., Le, Q.V.: Combined scaling for zero-shot transfer learning. Neurocomputing 555, 126658 (2023)
  * [32] Phan, T., Vo, K., Le, D., Doretto, G., Adjeroh, D., Le, N.: Zeetad: Adapting pretrained vision-language model for zero-shot end-to-end temporal action detection. In: WACV. pp. 7046–7055 (2024)
  * [33] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML. vol. 139, pp. 8748–8763. PMLR (2021)
  * [34] Rasheed, H., Khattak, M.U., Maaz, M., Khan, S., Khan, F.S.: Fine-tuned clip models are efficient video learners. In: CVPR. pp. 6545–6554 (2023)
  * [35] Rathod, V., Seybold, B., Vijayanarasimhan, S., Myers, A., Gu, X., Birodkar, V., Ross, D.A.: Open-vocabulary temporal action detection with off-the-shelf image-text features. arXiv preprint arXiv:2212.10596 (2022)
  * [36] Sennrich, R., Haddow, B., Birch, A.: Neural machine translation of rare words with subword units. In: ACL. vol. 1, pp. 1715–1725 (2016)
  * [37] Tarvainen, A., Valpola, H.: Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. NeurIPS 30 (2017)
  * [38] Wang, L., Xiong, Y., Lin, D., Van Gool, L.: Untrimmednets for weakly supervised action recognition and detection. In: CVPR. pp. 4325–4334 (2017)
  * [39] Wang, Y., He, Y., Li, Y., Li, K., Yu, J., Ma, X., Chen, X., Wang, Y., Luo, P., Liu, Z., et al.: Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.06942 (2023)
  * [40] Wang, Y., Chen, H., Heng, Q., Hou, W., Fan, Y., Wu, Z., Wang, J., Savvides, M., Shinozaki, T., Raj, B., Schiele, B., Xie, X.: Freematch: Self-adaptive thresholding for semi-supervised learning. In: ICLR (2023)
  * [41] Weng, Z., Yang, X., Li, A., Wu, Z., Jiang, Y.G.: Open-VCLIP: Transforming CLIP to an open-vocabulary video model via interpolated weight optimization. In: ICML. vol. 202, pp. 36978–36989. PMLR (2023)
  * [42] Wu, J., Li, X., Xu, S., Yuan, H., Ding, H., Yang, Y., Li, X., Zhang, J., Tong, Y., Jiang, X., et al.: Towards open vocabulary learning: A survey. IEEE TPAMI (2024)
  * [43] Xia, K., Wang, L., Zhou, S., Hua, G., Tang, W.: Learning from noisy pseudo labels for semi-supervised temporal action localization. In: ICCV. pp. 10160–10169 (October 2023)
  * [44] Xian, Y., Lampert, C.H., Schiele, B., Akata, Z.: Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly. IEEE TPAMI 41(9), 2251–2265 (2018)
  * [45] Xu, M., Zhao, C., Rojas, D.S., Thabet, A., Ghanem, B.: G-tad: Sub-graph localization for temporal action detection. In: CVPR. pp. 10156–10165 (2020)
  * [46] Yan, S., Xiong, X., Nagrani, A., Arnab, A., Wang, Z., Ge, W., Ross, D., Schmid, C.: Unloc: A unified framework for video localization tasks. In: ICCV. pp. 13623–13633 (2023)
  * [47] Zareian, A., Rosa, K.D., Hu, D.H., Chang, S.F.: Open-vocabulary object detection using captions. In: CVPR. pp. 14393–14402 (2021)
  * [48] Zhang, C.L., Wu, J., Li, Y.: Actionformer: Localizing moments of actions with transformers. In: ECCV. pp. 492–510. Springer (2022)
  * [49] Zhang, L., Chang, X., Liu, J., Luo, M., Wang, S., Ge, Z., Hauptmann, A.: Zstad: Zero-shot temporal activity detection. In: CVPR. pp. 879–888 (2020)
  * [50] Zhang, Y., Zhang, X.Y., Shi, H.: Ow-tal: learning unknown human activities for open-world temporal action localization. Pattern Recognition 133, 109027 (2023)
  * [51] Zheng, Z., Wang, P., Liu, W., Li, J., Ye, R., Ren, D.: Distance-iou loss: Faster and better learning for bounding box regression. In: AAAI. vol. 34, pp. 12993–13000 (2020)
  * [52] Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Conditional prompt learning for vision-language models. In: CVPR. pp. 16816–16825 (2022)
  * [53] Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Learning to prompt for vision-language models. IJCV 130(9), 2337–2348 (2022)

## Appendix

In this section, we provide additional experimental results and details not
presented in the main paper.

##  Appendix 0.A Additional Implementation Details

We primarily adopt the hyperparameters from ActionFormer [48] and ViFi-CLIP
[34] since our architecture is based on them. As addressed in Sec. 4.1, the
ActivityNet [5] dataset differs significantly from the THUMOS14 [13] and
FineAction [23] datasets. It consists of only 1 – 2 long action instances,
limiting the evaluation of the capability in action localization. Accordingly,
we adjust hyperparameters for the ActivityNet dataset, following the TAL
literature.

###  0.A.1 Video Feature Extraction Details

Here, we detail the video feature extraction process using VLMs. As stated in
Sec. 4.3, we extract the video snippet features ($\mathbf{F}_{\mathcal{V}}$)
using the conventional sliding window manner with a window size of 16 frames
and a stride size of 4 frames, after interpolating videos into 30 fps. For
ActivityNet, we interpolate $\mathbf{F}_{\mathcal{V}}$ to a fixed length
following the widely used convention [20, 45, 19, 48]. Specifically, each
video is interpolated to a length of 192 feature vectors, as employed in
ActionFormer [48]. For THUMOS14 and FineAction, we retain the video snippet
features in their original length. For the experiments with ViCLIP [39], we
interpolate its learned temporal embedding from 8 to 16 to use the same window
size for video feature extraction. Note that such details are often absent in
existing OV-TAL methods [15, 28], making it challenging to reproduce their
results. We will release the extracted features to encourage the OV-TAL
community.

###  0.A.2 Inference Details

Compared to ActionFormer [48], we use the class-agnostic action localizer and
assign the action classes from the VLM. As a result of this change, we delay
the Soft-NMS operation until after the action classes are assigned, rather
than immediately following the output of the action localizer. On the other
hand, when we compute pseudo labels on unlabeled videos, we directly perform
Soft-NMS on the class-agnostic action instances, based on its actionness score
($s_{a}$). We employ the same Soft-NMS configurations for both cases, with
slight variations based on the dataset. We choose the top 100 scoring action
instances for ActivityNet and 200 for THUMOS14 and FineAction, applying a
minimum confidence score threshold of 0.001 and a tIoU threshold of 0.1.

###  0.A.3 Selection of Previous Methods for OV-TAL

Including the results of existing OV-TAL methods [15, 28, 32] in our proposed
OV-TAL benchmarks would be a good practice to ensure a sufficient comparison.
However, due to the difficulty of reproducing their results111In Issue 1 and
Issue 2, the author mentioned that the method exploits UNet [38] as the
external action classifier which is trained on all action classes in the
dataset., we instead choose OpenTAL [1], which focuses on localizing actions
unseen during training through uncertainty modeling. We train it on the base
categories defined in our OV-TAL benchmark and use it as the class-agnostic
action localizer in our decoupled architecture. Although OpenTAL utilizes the
I3D [7] backbone for action localization, we assign action classes to output
action instances using ViFi-CLIP [34], as same as our model. Thus, comparing
its mAPs with ours solely evaluates the capability of action localization.

###  0.A.4 Training Details

AdamW [25] is chosen as the optimizer, coupled with a scheduler that linearly
warms up the learning rate (lr) to its maximum value and decays to the minimum
value ($1e^{-8}$) following a cosine function. Tab. 7 presents the
hyperparameter values for each dataset. For the OD-ST experiments on THUMOS14,
we use the subset of 100k videos. We empirically found the threshold values
for obtaining pseudo labels. 0.2, 0.05, and 0.4 are used for ActivityNet,
THUMOS14, and FineAction, respectively.

Table 7:  Training hyperparameters values.

Dataset | Stage | max lr | warm-up epoch | main epoch | batch size
---|---|---|---|---|---
ActivityNet | First | $1e^{-5}$ | 5 | 10 | 16
Second | $1e^{-5}$ | 5 | 5 | 16
THUMOS14 | First | $1e^{-4}$ | 5 | 30 | 2
Second (ID-ST) | $1e^{-4}$ | 5 | 10 | 2
FineAction | First | $1e^{-5}$ | 5 | 10 | 4
Second (ID-ST) | $1e^{-5}$ | 5 | 5 | 4
THUMOS14 | Second (OD-ST) | $1e^{-5}$ | 2 | 2 | 4
FineAction

##  Appendix 0.B Additional Experimental Results

Table 8:  Evaluation of cross-category OV-TAL benchmark. For reference of
upper-bound, full-shot results are shown in gray. † indicates our reproduced
results.

Methods | Backbone | ActivityNet v1.3
---|---|---
Generalized | Constrained
mAP${}_{A}^{50}$ | mAP${}_{B}^{50}$ | mAP${}_{N}^{50}$ | mAP${}_{B}^{50}$ | mAP${}_{N}^{50}$
OpenTAL† [1] | I3D [7] | 32.7 | 36.4 | 29.7 | 39.1 | 34.1
STOV-TAL (w/o ST) | ViFi-CLIP-B [34] | 42.9 | 47.5 | 39.1 | 51.3 | 44.1
STOV-TAL (ID-ST) | ViFi-CLIP-B [34] | 43.1 | 47.3 | 39.7 | 51.1 | 44.6
STOV-TAL (FS) | ViFi-CLIP-B [34] | 43.7 | 47.9 | 40.3 | 52.0 | 45.3

###  0.B.1 ActivityNet v1.3 Results for OV-TAL Benchmark

The main paper presents the cross-category OV-TAL benchmark results of the
THUMOS14 [13] and FineAction [23] datasets in Tab. 4. Here, we show the
results of the ActivityNet v1.3 [5] in Tab. 8. As discussed in Sec. 4.6,
ActivityNet v1.3 is not a proper dataset for evaluating the generalization
capability in action localization, supported by the zero-shot performance (w/o
ST) of action localization on par with that of full-shot in Tab. 6. In the
proposed OV-TAL benchmark, we observe a similar trend. For instance, in the
constrained setting, mAP${}^{50}_{N}$ of w/o ST and FS are 44.1 and 45.3,
respectively. Based on these results, we decided not to include the
ActivityNet dataset in the OV-TAL benchmark since there is only a small
potential for improvement in cross-category generalization ability.