# Polarity Loss for Zero-shot Object Detection

Shafin Rahman, Salman Khan and Nick Barnes S. Rahman is with the Research
School of Engineering, The Australian National University, Canberra, ACT 2601,
Australia, and also with Data61, Commonwealth Scientific and Industrial
Research Organization, Canberra, ACT 2601, Australia (e-mail:
shafin.rahman@anu.edu.au).S. Khan is with the Research School of Engineering,
The Australian National University, Canberra, ACT 2601, Australia, and also
with the Inception Institute of Artificial Intelligence, Abu Dhabi 0000,
UAE.N. Barnes is with the Research School of Engineering, The Australian
National University, Canberra, ACT 2601, Australia

###### Abstract

Conventional object detection models require large amounts of training data.
In comparison, humans can recognize previously unseen objects by merely
knowing their semantic description. To mimic similar behaviour, zero-shot
object detection aims to recognize and localize ‘ _unseen_ ’ object instances
by using only their semantic information. The model is first trained to learn
the relationships between visual and semantic domains for _seen_ objects,
later transferring the acquired knowledge to totally _unseen_ objects. This
setting gives rise to the need for correct alignment between visual and
semantic concepts, so that the _unseen_ objects can be identified using only
their semantic attributes. In this paper, we propose a novel loss function
called _‘Polarity loss’_ , that promotes correct visual-semantic alignment for
an improved zero-shot object detection. On one hand, it refines the noisy
semantic embeddings via metric learning on a _‘Semantic vocabulary’_ of
related concepts to establish a better synergy between visual and semantic
domains. On the other hand, it explicitly maximizes the gap between positive
and negative predictions to achieve better discrimination between seen, unseen
and background objects. Our approach is inspired by embodiment theories in
cognitive science, that claim human semantic understanding to be grounded in
past experiences (seen objects), related linguistic concepts (word vocabulary)
and visual perception (seen/unseen object images). We conduct extensive
evaluations on MS-COCO and Pascal VOC datasets, showing significant
improvements over state of the art. Our code and evaluation protocols
available at: https://github.com/salman-h-khan/PL-ZSD_Release

###### Index Terms:

Zero-shot object detection, Zero-shot learning, Deep neural networks, Object
detection, Loss function.

##  1 Introduction

Existing deep learning models generally perform well in a fully supervised
setting with large amounts of annotated data available for training. Since
large-scale supervision is expensive and cumbersome to obtain in many real-
world scenarios, the investigation of learning under reduced supervision is an
important research problem. In this pursuit, human learning provides a
remarkable motivation since humans can learn with very limited supervision.
Inspired by human learning, zero-shot learning (ZSL) aims to reason about
objects that have never been seen before using only their semantics. A
successful ZSL system can help pave the way for life-long learning machines
that intelligently discover new objects and incrementally enhance their
knowledge.

Traditional ZSL literature only focuses on ‘ _recognizing_ ’ unseen objects.
Since real-world objects only appear as a part of a complete scene, the newly
introduced zero-shot object detection (ZSD) task [1, 2] considers a more
practical setting where the goal is to simultaneously ‘ _locate and recognize_
’ unseen objects. This task offers new challenges for the existing object
detection frameworks, the most important being the accurate alignment between
visual and semantic concepts. If a sound alignment between the two
heterogeneous domains is achieved, the unseen objects can be detected at
inference using only their semantic representations. Further, a correct
alignment can help the detector in differentiating between the background and
the previously unseen objects in order to successfully detect the unseen
objects during inference.

In this work, we propose a single-stage object detection pipeline underpinned
by a novel objective function named Polarity loss. Our approach focuses on
learning the complex interplay between visual and semantic domains such that
the unseen objects (alongside the seen ones) can be accurately detected and
localized. To this end, the proposed loss simultaneously achieves two key
objectives during end-to-end training process. First, it learns a new
vocabulary metric that improves the semantic embeddings to better align them
with the visual concepts. Such a dynamic adaptation of semantic
representations is important since the original unsupervised semantic
embeddings (e.g., word2vec) are noisy and thereby complicate the visual-
semantic alignment. Secondly, it directly maximizes the margin between
positive and negative detections to encourage the model to make confident
predictions. This constraint not only improves visual-semantic alignment
(through maximally separating class predictions) but also helps distinguish
between background and unseen classes.

Our ZSD approach is distinct from existing state-of-the-art [2, 3, 1, 4, 5]
due to its direct emphasis on achieving better visual-semantic alignment. For
example, [1, 2, 3, 4] use fixed semantics derived from unsupervised learning
methods that can induce noise in the embedding space. In order to overcome
noise in the semantic space, [5] resorts to using detailed semantic
descriptions of object classes instead of single vectors corresponding to
class names. Further, the lack of explicit margins between positive and
negative predictions leads to confusion between background and unseen classes
[2, 4, 3]. In summary, our main contributions are:

  * •

An end-to-end single-shot ZSD framework based on a novel loss function called
‘ _Polarity loss_ ’ to address object-background imbalance and achieve maximal
separation between positive and negative predictions.

  * •

Using an external vocabulary of words, our approach learns to associate
semantic concepts with both seen and unseen objects. This helps to resolve
confusion between unseen classes and background, and to appropriately reshape
the noisy word embeddings.

  * •

A new seen-unseen split on the MS-COCO dataset that respects practical
considerations such as diversity and rarity among unseen classes.

  * •

Extensive experiments on the old and new splits for MS-COCO and Pascal VOC
which give absolute gains of $9.3$ and $7.6$ in mAP over [2] and [3],
respectively.

A preliminary version of this work appeared in [6], where the contribution is
limited to a single formulation of the polarity loss. Moreover, the
experiments only consider word2vec as word embedding. In the current paper, we
extended our work as follows: (1) An alternate formulation of Polarity loss
(Sec. 3.2). (2) Motivation of the proposed new split based on MSCOCO dataset
for zero-shot detection (Sec. 5). (3) A detailed discussion on related works
in the literature (Sec. 2). (4) More experiments and ablation studies e.g.,
with multiple semantic embeddings including Word2vec, GloVe and FastText, a
validation study on hyper-parameters and studying the impact of changing
overlap threshold (Sec. 5).

##  2 Related work

Zero-shot learning (ZSL): The earliest efforts were based on manually
annotated attributes as a mid-level semantic representation [7]. This approach
resulted in a decent performance on fine-grained recognition tasks but to
eliminate strong attribute supervision; researchers start exploring
unsupervised word-vector based techniques [8, 9]. Independent of the source of
semantic information, a typical ZSL method needs to map both visual and
semantic features to a common space to properly align information available in
both domains. This can be achieved in three ways: (a) transform the image
feature to semantic feature space [10], (b) map the semantic feature to image
feature space [11, 12] or, (c) map both image or semantic features to a common
intermediate latent space [13, 14]. To apply ZSL in practice, a few notable
problem settings are: (1) transductive ZSL [15, 16]: making use of unlabeled
unseen data during training, (2) generalized ZSL [17, 10]: classifying seen
and unseen classes together, (3) domain adaptation [18, 19]: learning a
projection function to adapt unseen target to seen source domain, (4) class-
attribute association [20, 21]: relating unsupervised semantics to human
recognizable attributes. In this paper, our focus is not only recognition but
also simultaneous localization of unseen objects which is a significantly
complex and challenging problem.

Figure 1: (_Top Left_) Traditional ZSD approaches align visual features (solid
dots) to their corresponding semantics (e.g., boat, airplane) without
considering the related semantic concepts (black text). It results in a
fragile description of an unseen class (train) and causes confusion with
background and seen classes (bottom left). (_Top Right_) Our approach
automatically attends to related semantics from an external vocabulary and
reshapes the semantic embedding so that visual features are well-aligned with
related semantics. Moreover, it maximizes the inter-class separation that
avoids confusion between unseen and background (bottom right).

Object detection: End-to-end trainable deep learning models have set new
performance records on object detection benchmarks. The most popular deep
architectures can be categorized into double stage networks (e.g., FasterRCNN
[22], RFCN [23]) and single stage networks (e.g., SSD[24], YOLO[25]).
Generally, double-stage detectors achieve high accuracy, while single-stage
detectors work faster. To further improve performance, recent object detectors
introduce novel concepts such as feature pyramid network (FPN) [26, 27]
instead of region proposal network (RPN), focal loss [28] instead of
traditional cross-entropy loss, non-rectangular region selection [29] instead
of rectangular bounding boxes, designing backbone architectures [30] instead
of ImageNet pre-trained networks (e.g., VGG[31]/ResNet[32]). In this paper, we
attempt to extend object detection to the next level: detecting unseen objects
which are not observed during training.

Zero-shot object detection (ZSD): The ZSL literature is predominated by
classification approaches that focus on single [33, 34, 7, 10, 11, 12] or
multi-label [35, 36] recognition. The extension of conventional ZSL approaches
to zero-shot object localization/detection is relatively less investigated.
Among previous attempts, Li _et al._ [37] learned to segment attribute
locations which can locate unseen classes. [38] used a shared shape space to
segment novel objects that look like seen objects. These approaches are useful
for classification but not extendable to ZSD. [39, 40] proposed novel object
localization based on natural language description. Few other methods located
unseen objects with weak image-level labels [36, 41]. However, none of them
perform ZSD. Very recently, [1, 4, 2, 3, 5] investigated the ZSD problem.
These methods can detect unseen objects using box annotations of seen objects.
Among them, [2] proposed a feature based approach where object proposals are
generated by edge-box [42], and [1, 4, 3] modified the object detection
frameworks [22, 25] to adapt ZSD settings. In another work, [5] use textual
description of seen/unseen classes to obtain semantic representations that are
used for zero-shot object detection. Here, we propose a new loss formulation
that can greatly benefit single stage zero-shot object detectors.

Generalized Zero-shot object detection (GZSD): This setting aims to detect
both seen and unseen objects during inference. The key difference from
generalized zero-shot recognition is that multiple objects can co-exist in a
single image, thereby posing a challenge for the detector. In contrast, in the
recognition setting either a seen or an unseen label is assigned since a
single category is assumed per image. Only a handful of previous works in the
literature reported GZSD results [2, 43, 44]. [44] and [43] reported GZSD
results while addressing transductive zero-shot detection and unseen object
captioning problems, respectively. In the current work, we compare GZSD
results with [2] who evaluate on GZSD problem.

##  3 Polarity Loss

We first introduce the proposed Polarity Loss that builds upon Focal loss [28]
for generic object detection. Focal loss only promotes correct prediction,
whereas a sound ZSL system should also learn to minimize projections on
representative vectors for negative classes. Our proposed loss jointly
maximizes projection on correct classes and minimizes the alignment with
incorrect ones (Sec. 3.2). Furthermore, it allows reshaping the noisy class
semantics using a metric learning approach (Sec. 3.3). This approach
effectively allows distinction between background _vs._ unseen classes and
promotes better overall alignment between visual and semantic concepts. Below
in Sec. 3.1, we provide a brief background followed by a description of
proposed loss.

###  3.1 Balanced Cross-Entroy vs. Focal loss

Consider a binary classification task where $y\in\\{0,1\\}$ denotes the
ground-truth class and $p\in[0,1]$ is the prediction probability for the
positive class (i.e., $y=1$). The standard binary cross-entropy (CE)
formulation gives:

$\displaystyle\text{CE}(p,y)=-\alpha_{t}\log p_{t},\quad
p_{t}=\begin{cases}p,&\text{if $y=1$,}\\\ 1-p,&\text{otherwise}.\end{cases}$
(1)

where, $\alpha$ is a loss hyper-parameter representing inverse class frequency
and the definition of $\alpha_{t}$ is analogous to $p_{t}$. In the object
detection case, the object vs. background ratio is significantly high (e.g.,
$10^{-3}$). Using a weight factor $\alpha$ is a traditional way to address
this strong imbalance. However, being independent of the model’s prediction,
this approach treats both well-classified (easy) and poorly-classified (hard)
cases equally. It favors easily classified examples to dominate the gradient
and fails to differentiate between easy and hard examples. To address this
problem, Lin _et al._ [28] proposed ‘Focal loss’ (FL):

$\displaystyle\text{FL}(p,y)=-\alpha_{t}(1-p_{t})^{\gamma}\log p_{t},$ (2)

where, $\gamma\in[0,5]$ is a loss hyper-parameter that dictates the slope of
cross entropy loss (a large value denotes higher slope). The term
$(1-p_{t})^{\gamma}$ enforces a high and low penalty for hard and easy
examples respectively. In this way, FL simultaneously addresses object vs.
background imbalance and easy vs. hard examples difference during training.

Shortcomings: In zero-shot learning, it is highly important to align visual
features with semantic word vectors. This alignment requires the training
procedure to (1) push visual features close to their ground-truth embedding
vector and (2) push them away from all negative class vectors. FL can only
perform (1) but cannot enforce (2) during the training of ZSD. Therefore,
although FL is well-suited for traditional seen object detection, but not for
the ZSD scenario.

###  3.2 Max-margin Formulation

To address the above-mentioned shortcomings, we propose a margin maximizing
loss formulation that is particularly suitable for ZSD. This formulation is
generalizable and can work with loss functions other than Eqs. 1 and 2.
However, for the sake of comparison with the best model, we base our analysis
on the state of the art FL.

####  3.2.1 Objective Function

Multi-class Loss: Consider that a given training set
$\\{\mathbf{x},\mathbf{y}\\}_{i}$ contains $N$ examples belonging to $C$
object classes plus an additional background class. For the multi-label
prediction case, the problem is treated as a sum of individual binary cross-
entropy losses where each output neuron decides whether a sample belongs to a
particular object class or not. Assume,
$\mathbf{y}=\\{y^{i}\in\\{0,1\\}\\}\in\mathbb{R}^{C}$ and
$\mathbf{p}=\\{p^{i}\in[0,1]\\}\in\mathbb{R}^{C}$ denotes the ground-truth
label and prediction vectors respectively, and the background class is denoted
by $\mathbf{y}=\mathbf{0}\in\mathbb{R}^{C}$. Then, the FL for a single box
proposal is:

$\displaystyle\mathscr{L}=\sum_{i}-\alpha^{i}_{t}(1-p_{t}^{i})^{\gamma}\log
p^{i}_{t}.$ (3)

Polarity Loss: Suppose, for a given bounding box feature containing an
$\ell^{th}$ object class, $p^{\ell}$ represents the prediction value for the
ground-truth object class, _i.e._ , $y^{\ell}=1$, see Table I. Note that
$p^{\ell}=0$ for the background class (where $y^{i}=0;\forall i$). Ideally, we
would like to maximize the predictions for ground-truth classes and
simultaneously minimize prediction scores for all other classes. We propose to
explicitly maximize the margin between predictions for positive and negative
classes to improve the visual-semantic alignment for ZSD (see Fig. 3). This
leads to a new loss function that we term as ‘Polarity Loss’ (PL):

$\displaystyle\mathscr{L}_{PL}=\sum_{i}f_{p}(p^{i}-p^{\ell})\text{FL}(p^{i},y^{i}),$
(4)

where, $f_{p}$ is a monotonic penalty function. For any prediction, $p^{i}$
where ${\ell}{\neq}i$, the difference $p^{i}{-}p^{\ell}$ represents the
disparity between the true class prediction and the prediction for the
negative class. The loss function enforces a large negative margin to push
predictions $p^{i}$ and $p^{\ell}$ further apart. Thus, for an object anchor
case, the above objective enforces $p^{\ell}{>}p^{i}$, while for background
case $0{>}p^{i}$ _i.e._ , all $p^{i}$’s are pushed towards zero (since
$p^{\ell}{=}0$).

Our Penalty Function: $f_{p}$ should necessarily be a ‘monotonically
increasing’ function. It offers a small penalty if the gap $p^{i}{-}p^{\ell}$
is low and a large penalty if the gap is high. This constraint enforces that
$p^{i}<p^{\ell}$. In this paper, we implement $f_{p}$ with a $\beta$
parameterized sigmoid function:

$\displaystyle f_{p}(p^{i}-p^{\ell})=\frac{1}{1+\exp(-\beta(p^{i}-p^{\ell}))}$
(5)

For the case when $p^{i}{=}p^{\ell}$, the FL part guides the loss because
$f_{p}$ becomes a constant. We choose a sigmoid form for $f_{p}$ because the
difference $(p^{i}{-}p^{\ell})\in[-1,1]$ and $f_{p}$ can be bounded by
$[0,1]$, similar to $\alpha_{t}$ or the $(1{-}p_{t})$ factor of FL. Note that,
it is not compulsory to stick with this particular choice of $f_{p}$. We also
test a softplus-based function for $f_{p}$ in the next subsection.

Final Objective: The final form of the loss is:

$\displaystyle\mathscr{L}_{PL}(\mathbf{p},\mathbf{y})=\sum_{i}\frac{-\alpha^{i}_{t}(1-p^{i}_{t})^{\gamma}\log
p^{i}_{t}}{1+\exp(-\beta(p^{i}-p^{\ell}))},\text{ where,}$ $\displaystyle
p^{i}_{t}=\begin{cases}p^{i},&\text{if $y^{i}=1$}\\\
1-p^{i},&\text{otherwise}\end{cases}\qquad p^{\ell}=p^{i}\llbracket
y^{i}=1\rrbracket,$ (6)

where, $\llbracket\cdot\rrbracket$ denotes the Iverson bracket. Later in Sec.
3.3, we describe our vocabulary-based metric learning approach to obtain
$\mathbf{p}$ in the above Eq. 3.2.1.

(a) Object case: $p^{\ell}=.8$

$p^{i}$ | .1 | .8 | .9
---|---|---|---
$y^{i}$ | 0 | 1 | 0
$p_{t}^{i}$ | .9 | .8 | .1
$p^{i}-p^{\ell}$ | -.7 | 0 | .1
loss | L | L | H

(b) Background case: $p^{\ell}=0$

$p^{i}$ | .1 | .8 | .9
---|---|---|---
$y^{i}$ | 0 | 0 | 0
$p_{t}^{i}$ | .9 | .2 | .1
$p^{i}-p^{\ell}$ | .1 | .8 | .9
loss | L | H | H

TABLE I: A toy example. Intermediate computations for Polarity Loss are shown.
Low (L) values are shown in green while High (H) values are shown in red. A
mismatch between ($p^{i}$ and $y^{i}$) + a close match between ($y^{i}$ and
$y^{\ell}$) results in a high loss. Figure 2: Plots of multi-class loss
(_left_) and penalty function (_right_).

####  3.2.2 Analysis and Insights

A Toy Example: We explain the proposed loss with a toy example in Table I and
Fig. 2. When an anchor box belongs to an ‘ _object_ ’ and $p_{t}^{i}\geq.5$
(high) then $p^{i}{-}p^{\ell}\leq 0$ (low). From Fig. 2, both a multi-class
loss and the penalty function find low loss which eventually calculates a low
loss. Similarly, when $p_{t}^{i}<.5$ (low), $p^{i}{-}p^{\ell}>0$ (high), which
evaluates to a high loss. When an anchor belongs to ‘ _background_ ’,
$p^{i}{-}p^{\ell}\geq 0$ and a high $p^{i}$ results in a high value for both
multi-class loss and the penalty function and vice versa. In this way, the
penalty function always supports multi-class loss based on the disparity
between the current prediction and ground-truth class’s prediction.

Figure 3: tSNE plot of visual features from 8 unseen classes projected onto
semantic space using (a) FL & (b) PL. FL pushes visual features close to their
ground-truth. Thus, intra-class distances are minimized but inter-class
distances are not considered. This works well for seen class separation, but
is not optimal for unseen classes because inter-class distances must be
increased to ensure unseen class separability. Our PL ensures this requisite.

Polarity Loss Properties: The PL has two intriguing properties. (a) Word-
vectors alignment: For ZSL, generally visual features are projected onto the
semantic word vectors. A high projection score indicates proper alignment with
a word-vector. The overall goal of training is to achieve good alignment
between the visual feature and its corresponding word-vector and an inverse
alignment with all other word-vectors. In our proposed loss,
$\text{FL}(\cdot)$ and $f_{p}$ perform the direct and inverse alignment
respectively. Fig. 3 shows visual features before and after this alignment.
(b) Class imbalance: The penalty function $f_{p}$ follows a trend similar to
$\alpha_{t}$ and $(1{-}p_{t})^{\gamma}$. It means that $f_{p}$ assigns a low
penalty to well-classified/easy examples and a high penalty to poorly-
performed/hard cases. It greatly helps in tackling class imbalance for single
stage detectors where negative boxes heavily outnumber positive detections.

Alternative formulation of polarity loss: We have used a sigmoid based penalty
function to implement $f_{p}(p^{i}-p^{\ell})$ in our proposed polarity loss.
Now, we present an alternative implementation of the penalty function based on
the softplus function. In Fig. 4, we illustrate the shapes of both sigmoid and
softplus based penalty functions. Both the functions increase penalty when
$p^{i}-p^{\ell}$ moves from $-1$ to $1$. However, the softplus is relatively
smoother than sigmoid. Also, softplus has a flexibility to assign a penalty
$>1$ for any poorly classified examples whereas sigmoid can penalize at most
1. The formulation for the softplus based penalty function is as follows,

$\displaystyle
f_{p}(p^{i}-p^{\ell})=\log\Big{(}1+\text{e}^{\beta^{\prime}(p^{i}-p^{\ell})}\Big{)}$
(7)

where, $\beta^{\prime}$ is the loss hyper-parameter. The final polarity loss
with the softplus based penalty function is the following:

$\displaystyle\mathscr{L}_{PL}$
$\displaystyle(\mathbf{p},\mathbf{y})=\sum_{i}-\alpha^{i}_{t}(1-p^{i}_{t})^{\gamma}\log\Big{(}1+\text{e}^{\beta^{\prime}(p^{i}-p^{\ell})}\Big{)}\log
p^{i}_{t},\text{ }$ $\displaystyle p^{i}_{t}=\begin{cases}p^{i},&\text{if
$y^{i}=1$}\\\ 1-p^{i},&\text{otherwise}\end{cases}\quad
p^{\ell}=p^{i}\llbracket y^{i}=1\rrbracket,$ (8)

Figure 4: Visualization of sigmoid (left) vs. softplus (right) based penalty
functions.

###  3.3 Vocabulary Metric Learning

Apart from proper visual-semantic alignment and class imbalance, a significant
challenge for ZSD is the inherent noise in the semantic space. In this paper,
we propose a new ‘vocabulary metric learning’ approach to improve the quality
of word vectors for ZSL tasks. For brevity of expression, we restrict our
discussion to the case of classification probability prediction or bounding
box regression for a single anchor. For the classification case, metric
learning is considered as a part of Polarity loss in Eq. 3.2.1. Suppose, the
visual feature of that anchor, $\mathbf{a}$ is $\phi(\mathbf{a})=\mathbf{f}$,
where $\phi$ represents the detector network. The total number of seen classes
is $\mathrm{S}$ and a matrix $W_{s}\in\mathbb{R}^{\mathrm{S}\times d}$ denotes
all the $d$-dimensional word vectors of $\mathrm{S}$ seen classes arranged
row-wise. The detector network $\phi$ is augmented with FC layers towards the
head to transform the visual feature $\mathbf{f}$ to have the same dimension
as the word vectors, i.e., $\mathbf{f}\in\mathbb{R}^{d}$. In Fig. 5, we
describe several ways to learn the alignment function between visual features
and semantic information. We elaborate these further below.

####  3.3.1 Learning with Word-vectors

For the traditional detection case, shown in Fig. 5(a), the visual features
$\mathbf{f}$ are transformed with a learnable FC layer
$W_{d}\in\mathbb{R}^{\mathrm{S}\times d}$, followed by a sigmoid/softmax
activation ($\sigma$) to calculate $\mathrm{S}$ prediction probabilities,
$\mathbf{p}_{d}=\sigma(W_{d}\mathbf{f})$. This approach works well for
traditional object detection, but it is not suitable for the zero-shot setting
as the transformation $W_{d}$ cannot work with unseen object classes.

A simple extension of the traditional detection framework to the zero-shot
setting is possible by replacing trainable weights of the FC layers, $W_{d}$,
by the non-trainable seen word vectors $W_{s}$ (Fig. 5(b)). Keeping this layer
frozen, we allow projection of the visual feature $\mathbf{f}$ to the word
embedding space to calculate prediction scores $\mathbf{p}_{s}$:

$\displaystyle\mathbf{p}_{s}=\sigma(W_{s}\mathbf{f})$ (9)

This projection aligns visual features with the word vector of the
corresponding true class. The intuition is that rather than directly learning
a prediction score from visual features (in Fig 5(a)), it is better to learn a
correspondence between the visual features with word vectors before the
prediction.

Figure 5: (a) Traditional basic approach with learnable $W_{d}$, (b) Inserting
word vectors as a fixed embedding $W_{s}$, (c) learnable word vectors with
vocabulary metric $\delta(W_{s}MD)$.

Challenges with Basic Approach:  Although the configuration described in Fig.
5(b) delivers a basic solution to zero-shot detection, it suffers from several
limitations. (1) Fixed Representations: With a fixed embedding $W_{s}$, the
network cannot update the semantic representations and has limited flexibility
to properly align visual and semantic domains. (2) Limited word embeddings:
The word embedding space is usually learned using billions of words from
unannotated texts which results in noisy word embeddings. Understanding the
semantic space with only $\mathrm{S}$ word vectors is therefore unstable and
insufficient to model visual-semantic relationships. (3) Unseen-background
confusion: In ZSD, one common problem is that the model confuses unseen
objects with background since it has not seen any visual instances of unseen
classes [2].

Figure 6: 2D tSNE [45] embedding of word2vec: (a) before (b) after
modification based on vocabulary metric with our loss. Word-vectors are more
evenly distributed in (b) than (a). Also, visually similar classes for
example, apple/banana/orange/broccoli, cell phone/remote/laptop/mouse and
handbag/backpack/umbrella are embedded more closely in (b) than (a). Super-
category annotations are used for visualization only, not during our training.
Figure 7: Network architecture for ZSD. The green colored layer implements Eq.
9 (Our-PL-word) or 10 (Our-PL-vocab).

####  3.3.2 Learning with vocabulary metric

To address the above gaps, we propose to learn a more expressive and flexible
semantic domain representation. Such a representation can lead to a better
alignment between visual features and word vectors. Precisely, we propose a
vocabulary metric method summarized in Fig. 5(c) that takes advantage of the
word vectors of a pre-defined vocabulary, $D\in\mathbb{R}^{v\times d}$ (where
$v$ is the number of words in the vocabulary). By relating the given class
semantics with dictionary atoms, the proposed approach provides an inherent
mechanism to update class-semantics optimally for ZSD. Now, we calculate the
prediction score as follows:

$\displaystyle\mathbf{p}_{v}=\sigma(\delta(W_{s}MD)\mathbf{f})$ (10)

Here, $M\in\mathbb{R}^{d\times v}$ represents the learnable parameters which
connect seen word vectors with the vocabulary and $\delta(.)$ is a $\tanh$
activation function. $M$ can be interpreted as learned attention over the
dictionary. With such an attention, the network can understand the semantic
space better and learn a rich representation because it considers more
linguistic examples (vocabulary words) inside the semantic space.
Simultaneously, it helps the network to update the word embedding space for
better alignment with visual features. Further, it reduces unseen-background
confusion since the network can relate visual features more accurately with a
diverse set of linguistic concepts. We visualize word vectors before and after
the update in Fig. 6 (a) and (b) respectively.

Here, we emphasize that the previous attempts to use such an external
vocabulary have their respective limitations. For example, [20] considered a
limited set of attributes while [21] used several disjoint training stages.
These approaches are therefore not end-to-end trainable. Further, they only
investigate the recognition problem.

Regression branch with semantics: Eq. 10 allows our network to predict seen
class probabilities at the classification branch directly using semantic
information from the vocabulary metric. Similarly, we also apply such
semantics in the regression branch with some additional trainable FC layers.
In our experiments, we show that adding semantics in this manner leads to
further improvement in ZSD. It shows that the predicted regression box can
benefit from the semantic information that improves the overall performance.

##  4 Architecture details

Single-stage Detector: Our proposed ZSD framework is specially designed to
work with single-stage detectors. The primary motivation is the direct
connection between anchor classification and localization that ensures a
strong feedback for both tasks. For this study, we choose a recent unified
single architecture, RetinaNet [28] to implement our proposed method.
RetinaNet is the best detector known for its high speed (on par with single-
stage detectors) and high accuracy (outperforming two-stage detectors). In
Fig. 7, we illustrate the overall architecture of the model. In addition to a
novel loss formulation, we also perform modifications to the RetinaNet
architecture to link visual features (from ResNet50 [32]) with semantic
information. To adapt this network to ZSL setting, we perform simple
modifications in both classification and box regression subnets to consider
word-vectors (with vocabulary metric) during training (see Fig. 7).

RetinaNet has one backbone network called Feature Pyramid Network (FPN) [26]
and two task-specific subnetwork branches for classification and box
regression. FPN extracts rich, multi-scale features for different anchor boxes
from an image to detect objects at different scales. For each pyramid level,
we use anchors at {1:2,1:1,2:1} aspect ratios with sizes
$\\{2^{0},2^{1/3},2^{2/3}\\}$ totaling to $A{=}9$ anchors per level, covering
an area of $32^{2}$ to $512^{2}$ pixels. The classification and box-regression
subnetworks attempt to predict the one-hot target ground-truth vector of size
$\mathrm{S}$ and box parameters of size four respectively. We consider an
anchor box as an object if it gets an intersection-over-union (IoU) ratio
$>0.5$ with a ground-truth bounding box.

Modifications to RetinaNet: Suppose, a feature map at a given pyramid level
has $C$ channels. For the classification subnet, we first apply four conv
layers with $C$ filters of size $3\times 3$, followed by ReLU, similar to
RetinaNet. Afterward, we apply a $3\times 3$ conv layer with $d\times A$
filters to convert visual features to the dimension of word vectors, $d$.
Next, we apply a custom layer which projects image features onto the word
vectors. We also apply a sigmoid activation function to the output of the
projection. This custom layer may have fixed parameters like Fig. 3(b) or
trainable parameters like 3(c) of the main paper with vocabulary metric. These
operations are formulated as: $\mathbf{p}_{s}=\sigma(W_{s}\mathbf{f})$ or
$\mathbf{p}_{v}=\sigma(\delta(W_{s}MD)\mathbf{f})$ depending on the
implementation of the custom layer. Similarly, for the box-regression branch,
we attach another $3\times 3$ convolution layer with $C$ filters and ReLU non-
linearity, followed by $3\times 3$ convolution with $d$ filters and the custom
layer to get the projection response. Finally, another convolution with $4A$
to predict a relative offset between the anchor and ground-truth box. In this
way, the box-prediction branch gets semantic information of word-vectors to
predict offsets for regression. Note that, similar to [28], the classification
and regression branches do not share any parameters, however, they have a
similar structure.

Training: We train the classification subnet branch with our proposed loss
defined in Eq. 3.2.1. Similar to [28], to address the imbalance between hard
and easy examples, we normalize the total classification loss (calculated from
$\sim$100k anchors) by the total number of object/positive anchor boxes rather
than the total number of anchors. We use standard smooth $L_{1}$ loss for the
box-regression subnet branch. The total loss is the sum of the loss of both
branches.

Figure 8: Instances of each class in the MS-COCO dataset (except class
‘person’). Tall bars are clipped for better visualization. The class bars with
black border are selected as unseen classes. We choose 20% rarest classes from
each superclass as unseen.

Inference: For seen object detection, a simple forward pass predicts both
confidence scores and bounding boxes in the classification and box-regression
subnetworks respectively. Note that we only consider a fixed number (e.g.,
100) of boxes from RPN having confidence greater than 0.05 for inference.
Moreover, we apply Non-Maximum Suppression (NMS) with a threshold of 0.5 to
obtain final detections. We select the final detections that satisfy a seen
score-threshold ($t_{s}$). To detect unseen objects, we use the following
equation, followed by an unseen score-thresholding with a relatively lower
value ($t_{u}<t_{s}$)111Empirically, we found $t_{s}{=}0.3$ and $t_{u}{=}0.1$
generally work well. $t_{u}$ is kept smaller to counter classifier bias
towards unseen classes due to the lack of visual examples during training. :

$\displaystyle\mathbf{p}_{u}=W_{u}W_{s}^{T}\sigma(\delta(W_{s}MD)\mathbf{f})$
(11)

where, $W_{u}\in\mathbb{R}^{\mathrm{U}\times d}$ contains unseen class word
vectors. For generalized zero-shot object detection (GZSD), we simply consider
all detected seen and unseen objects together. In our experiments, we report
performances for traditional seen, zero-shot unseen detection and GZSD. One
can notice that our architecture predicts a bounding box for every anchor
which is independent of seen classes. It enables the network to predict
bounding boxes dedicated to unseen objects. Previous attempts like [1] detect
seen objects first and then attempt to classify those detections to unseen
objects based on semantic similarity. By contrast, our model allows detection
of unseen bounding boxes that are different to those seen.

Reduced description of unseen: All seen semantics vectors are not necessary to
describe an unseen objects [10]. Thus, we only consider the top $T$
predictions, $\mathbf{p}_{v}^{\prime}\in\mathbb{R}^{T}$ from
$\sigma(\delta(W_{s}MD)\mathbf{f})$ and the corresponding seen word vectors,
$W_{s}^{\prime}\in\mathbb{R}^{\mathrm{T}\times d}$ to predict unseen scores.
For the reduced case, $\mathbf{p}_{u}^{\prime}=W_{u}W_{s}^{\prime
T}\mathbf{p}_{v}^{\prime}.$ In the experiments, we vary the number of the
closest seen $T$ from 5 to $\mathrm{S}$ and find that a relatively small value
of $T$ (e.g., 5) performs better than using all available $T=\mathrm{S}$ seen
word vectors.

Method | Seen / |  | GZSD
---|---|---|---
| Unseen | ZSD | Seen | Unseen | HM
Split in [2] ($\downarrow$) |  | (mAP/RE) | (mAP/RE) | (mAP/RE) | (mAP/RE)
SB [2] | 48/17 | 0.70/24.39 | - | - | -
DSES [2] | 48/17 | 0.54/27.19 | -/15.02 | -/15.32 | -/15.17
ZSD-Textual [5] | 48/17 | -/34.3 | -/- | -/- | -/-
Baseline | 48/17 | 6.99/18.65 | 40.46/43.69 | 2.88/17.89 | 5.38/25.38
Ours | 48/17 | 10.01/43.56 | 35.92/38.24 | 4.12/26.32 | 7.39/31.18
Proposed Split ($\downarrow$) | mAP/RE | mAP/RE | mAP/RE | mAP/RE
Baseline | 65/15 | 8.48/20.44 | 36.96/40.09 | 8.66/20.45 | 14.03/27.08
Ours | 65/15 | 12.40/37.72 | 34.07/36.38 | 12.40/37.16 | 18.18/36.76

Figure 9: (left) Overall performance on MS-COCO. Hyper-parameters are set on
the validation set: $\beta{=}5$, IoU${=}0.5$. mAP = mean average precision and
RE = recall (@100). The top part shows results on [2] split and the lower part
shows results on our proposed split. Ours achieves best performance in terms
of mAP on unseen classes. (Right) Qualitative examples of ZSD (top row) and
GZSD (bottom row). Pink and yellow box represent unseen and seen detections
respectively. More qualitative results are presented in Fig. 14.

##  5 Experiments

Datasets: We evaluate our method with MS-COCO($2014$) [46] and Pascal VOC
($2007/12$) [47]. With $80$ object classes, MS-COCO includes $82783$ training
and $40504$ validation images. For the ZSD task, only unseen class performance
is of interest. As the test data labels are not known, the ZSD evaluation is
done on a subset of validation data. MS-COCO($2014$) has more validation
images than any later versions which motivates us to use it. For Pascal VOC,
we use the train set of $2007$ and $2012$ for training and use validation+test
set of $2007$ for testing.

Issues with existing MS-COCO split: Recently, [2] proposed a split of
seen/unseen classes for MS-COCO (2014). It considers $73,774$ training images
from $48$ seen classes and $6608$ test images from $17$ unseen classes. The
split criteria were the cluster embedding of class semantics and synset
WordNet hierarchy [48]. We identify two practical drawbacks of this split: (1)
Because all $63$ classes are not used as seen, this split does not take full
advantage of training images/annotations, (2) Because of choosing unseen
classes based on wordvector clustering it cannot guarantee the desired diverse
nature of the unseen set. For example, this split does not choose any classes
from ‘ _outdoor_ ’ super-category of MS-COCO.

Proposed seen/unseen split on MS-COCO: To address these issues, we propose a
more realistic split of MS-COCO for ZSD. Following the practical consideration
of unseen classes discussed in [1] i.e. rarity and diverseness, we follow the
following steps: (1) We sort classes of each superclass in ascending order
based on the total number of instances in the training set. (2) For each
superclass, we pick $20\%$ rare classes as unseen which results in $15$ unseen
and $65$ seen classes. Note that the superclass information is only used to
create a diverse seen/unseen split, and never used during training. (3) Being
zero-shot, we remove all the images from the training set where at least one
unseen class appears to create a training set of $62300$ images. (4) For
testing ZSD, we select $10098$ images from the validation set where at least
one instance of an unseen class is present. The total number of unseen
bounding boxes is $16388$. We use both seen and unseen annotation together for
this set to perform GZSD. (5) We prepare another list of $38096$ images from
the validation set where at least one occurrence of the seen instance is
present to test traditional detection performance on seen classes.

In Fig. 8, we present all 80 MS-COCO classes in sorted order across each
super-category based on the number of instance/bounding boxes inside the
training set. Choosing 20% low-instance classes from each super-category
ensures the rarity and diverseness for the chosen unseen classes. In this
paper, we report results on both our and [2] settings.

Method |  Seen |  Unseen |  aeroplane |  bicycle |  bird |  boat |  bottle |  bus |  cat |  chair |  cow |  d.table |  horse |  motrobike |  person |  p.plant |  sheep |  tvmonitor |  car |  dog |  sofa |  train
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Demirel _et al._[3] | 57.9 | 54.5 | 68.0 | 72.0 | 74.0 | 48.0 | 41.0 | 61.0 | 48.0 | 25.0 | 48.0 | 73.0 | 75.0 | 71.0 | 73.0 | 33.0 | 59.0 | 57.0 | 55.0 | 82.0 | 55.0 | 26.0
Ours | 63.5 | 62.1 | 74.4 | 71.2 | 67.0 | 50.1 | 50.8 | 67.6 | 84.7 | 44.8 | 68.6 | 39.6 | 74.9 | 76.0 | 79.5 | 39.6 | 61.6 | 66.1 | 63.7 | 87.2 | 53.2 | 44.1

TABLE II: mAP scores of Pascal VOC’07. Italic classes are unseen.

Pascal VOC Split: For Pascal VOC 2007/12 [47], we follow the settings of [3].
We use $16$ seen and $4$ unseen classes from total $20$ classes. We utilize
$2072$ and $3909$ train images from Pascal VOC $2007$ and $2012$ respectively
after ignoring images containing any instance of unseen classes. For testing,
we use $1402$ val+test images from Pascal VOC 2007 where any unseen class
appears at least once.

Vocabulary: We choose vocabulary atoms from $5018$ Flickr tags in NUS-WIDE
[49]. We only remove MS-COCO class names and tags that have no word vectors.
This vocabulary covers a wide variety of objects, attributes, scene types,
actions, and visual concepts.

Semantic embedding: For MS-COCO classes and vocabulary words, we use
$\ell_{2}$ normalized $300$ dimensional unsupervised word2vec [8], GloVe [9]
and FastText [50] vectors obtained from billions of words from unannotated
texts like Wikipedia. For Pascal VOC [47] classes, we use average 64 dimension
binary per-instance attribute annotation of all training images from aPY
dataset [51]. Unless mentioned otherwise, we use word2vec in our experiments.

Evaluation metric: Being an object detection problem, we evaluate using mean
average precision (mAP) at a particular IoU. Unless mentioned otherwise, we
use IoU$=0.5$. Notably, [3] use the Recall measure for evaluations, however
since recall based evaluation does not penalize a method for the wrongly
predicted bounding boxes, we only recommend mAP based evaluation for ZSD. To
evaluate GZSD, we report the harmonic mean (HM) of mAP and recall [2, 34].

Implementation details: We implement FPN with a basic ResNet50 [32]. All
images are rescaled to make their smallest side $800$px. We train the FL-basic
method using the original RetinaNet architecture with only training images of
seen classes so that the pre-trained network does not get influenced by unseen
instances. Then, to train Our-FL-word, we use the pre-trained weights to
initialize the common layers of our framework. We initialize all other
uncommon layers with a uniform random distribution. Similarly, we train Our-
PL-word and Our-FL-vocab upon the training of Our-FL-word. Finally, we train
Our-PL-vocab using the pre-trained network of Our-FL-vocab. We train each
network for $500$k iterations keeping a single image in the minibatch. The
only exception while training with our proposed loss is to train for $100$k
iterations instead of $500$k. Each training time varies from $72$ to $96$
hours using a single Tesla P$100$ GPU. For optimization, we use Adam optimizer
with learning rate $10^{-5}$, $\beta_{1}=0.9$ and $\beta_{2}=0.999$. We
implement this framework with Keras library.

$\beta$($\rightarrow$) | 5 | 10 | 15 | 20 | 25 | 30
---|---|---|---|---|---|---
mAP | 48.6 | 47.9 | 47.8 | 47.6 | 47.5 | 47.9

TABLE III: Validation study on traditional detection.

Validation strategy: $\alpha,\gamma$ and $\beta$ are the hyper-parameters of
the proposed polarity loss. Among them, $\alpha,\gamma$ are also present in
the focal loss. Therefore, we choose $\alpha=0.25,\gamma=2.0$ as recommended
by the original RetinaNet paper [28]. $\beta$ is the only new hyper-parameter
introduced in the polarity loss. We conduct a validation experiment on seen
classes to perform the traditional detection task. In Table III, we tested
$\beta=\\{5,10,15,20,30\\}$ and picked $\beta=5$ for our loss as it performed
the best among all the considered values.

###  5.1 Quantitative Results

Compared Methods: We rigorously evaluate our proposed ZSD method on both [2]
split ($48/17$) and our new ($65/15$) split of MS-COCO. We provide a brief
description of all compared methods: (a) SB [2]: This method extracts pre-
trained Inception-ResNet-v2 features from Edge-Box object proposals. It
applies a standard max-margin loss to align visual features to semantic
embeddings via linear projections. (b) DSES [2]: In addition to SB, DSES
augments extra bounding boxes other than MSCOCO objects. As [2] reported
recall performances, we also report recall results (in addition to mAP) to
compare with this method. (c) Baseline: This method trains an exact RetinaNet
model. Thus, it does not use any word vectors during training. To extend this
approach to perform ZSD, we apply this formula to calculate unseen scores:
$\mathbf{p}_{u}^{\prime}{=}W_{u}W_{s}^{\prime T}\mathbf{p}_{d}^{\prime}$ where
$\mathbf{p}_{d}^{\prime}$ represents top T seen prediction scores for the
reduced description of unseen. (d) Ours: This method is our final proposal
using vocabulary and polarity loss (Fig. 7).

Overall Results: Fig. 9 presents overall performance on ZSD and GZSD tasks
across different comparison methods with two different seen/unseen split of
MS-COCO. In addition to mAP, we also report recall (RE) to compare with [2].
With 48/17 settings, our method (and baseline) beats [2] (SB and DSES) in both
the ZSD and GZSD by a significantly large margin. Similarly, in 65/15 split,
we outperform our baseline by a margin 3.92 mAP (12.40 vs. 8.48) in ZSD task
and 4.15 harmonic-mAP in GZSD task (18.18 vs. 14.03). This improvement is the
result of end-to-end learning, the inclusion of the vocabulary metric to
update word vectors and the proposed loss in our method. We report results on
GloVe and FastText word vectors in the Table IV

$\mathbf{\gamma}$ | $\mathbf{\alpha}$ |  | GZSD
---|---|---|---
ZSD | Seen | Unseen | HM
0 | 1 | 6.6 | 31.9 | 6.6 | 10.9
0 | .75 | 2.7 | 27.4 | 2.7 | 4.9
0.1 | .75 | 5.4 | 27.9 | 5.4 | 9.0
0.2 | .75 | 7.3 | 31.4 | 7.3 | 11.8
0.5 | .50 | 8.4 | 30.6 | 8.4 | 13.1
1.0 | .25 | 11.6 | 31.3 | 11.6 | 16.9
2.0 | .25 | 12.6 | 33.0 | 12.6 | 18.3
5.0 | .25 | 9.1 | 33.6 | 9.1 | 14.3

Figure 10: Parameter sensitivity analysis: _(Left)_ Varying $\alpha$ and
$\gamma$ with a fixed $\beta{=}20$. _(Right-a)_ Impact of varying $\beta$,
_(Right-b)_ varying $\alpha$ with $\gamma{=}0$ to see the behavior of our loss
with only balanced CE. Note that the actual hyper-parameters choice is made on
a validation set.

Hyper-parameter Sensitivity Analysis: We study the sensitivity of our model to
loss hyper-parameters $\gamma,\alpha$ and $\beta$. First, we vary
$\gamma\in[0,5]$ and $\alpha\in[.25,1]$ keeping $\beta{=}20$. In Fig. 10
(_left_), we report mAP using different parameter settings for ZSD and GZSD.
Our model works best with $\alpha{=}.25$ and $\gamma{=}2.0$ which are also the
recommended values in FL. We also vary $\beta$ from $1$-$30$ to see its effect
on ZSD in Fig. 10 (_Right-a_). This parameter controls the steepness of the
penalty function $f_{p}$ in Eq. 5. Notably $\beta{=}20$ provides correct
steepness to estimate a penalty for incorrect predictions. Our loss can also
work reasonably well with balanced CE (i.e., without FL when $\gamma{=}0$). We
show this in Fig. 10(_Right-b_). With a low $\alpha$ of $0.05$, our method can
achieve around $10\%$ mAP. It shows that our penalty function can effectively
balance object/background and easy/hard cases.

Ablation Studies: In Fig. 11(Left), we report results on different variants of
our method. Our-FL-word: This method is based on the architecture in Fig. 5(b)
and trained with focal loss. It uses static word vectors during training. But,
it cannot update vectors based on visual features. Our-PL-word: Same
architecture as of Our-FL-word but training is done with our proposed polarity
loss. Our-PL-vocab: The method uses our proposed framework in Fig. 7 with
vocabulary metric learning in the custom layer and is learned with polarity
loss (without vocabulary metric learning). Our observations: (1) Our-FL-word
works better than Baseline for ZSD and GZSD because the former uses word
vectors during training whereas the later does not adopt semantics. By
contrast, in GZSD-seen detection cases, Baseline outperforms Our-FL-word
because the use of unsupervised semantics (word vectors) during training in
Our-FL-word introduces noise in the network which degrades the seen mAP. (2)
From Our-FL-word to Our-PL-word unseen mAP improves because of the proposed
loss which increases inter-class and reduces intra-class differences. It
brings better visual-semantic alignment than FL (Fig. 3). (3) Our-PL-vocab
further improves the ZSD performance. Here, the vocabulary metric helps the
word vectors to update based on visual similarity and allows features to align
better with semantics.

Method | ZSD | GZSD
---|---|---
Seen | Unseen | HM
Baseline | 8.48 | 36.96 | 8.66 | 14.03
Our-FL-word | 10.80 | 37.56 | 10.80 | 16.77
Our-PL-word | 12.02 | 33.28 | 12.02 | 17.66
Our-PL-vocab* | 12.62 | 32.99 | 12.62 | 18.26

Figure 11: Ablation studies with $\beta=20$: _(Left)_ Comparison of different
variant of our approach, best method denoted with $*$. _(Right)_ Impact of
word-vectors in the regression branch.

Semantics in Box-regression Subnet: Our framework can be trained without
semantics in the box-regression subnet. In Fig. 11 (Right), we compare
performance with and without using word vectors in the regression branch using
FL and our loss. We observe that using word vectors in regression branch helps
to improve the performance of ZSD.

Alternative formulation: In the first row of Table V, we report performance of
Our-PL-vocab using this alternative polarity loss with $\beta^{\prime}=5$ and
word2vec as semantic information. With this alternative formulation, we
achieve a performance quite close to that of sigmoid-based polarity loss.

Method | Seen/Unseen | Word | ZSD | GZSD
---|---|---|---|---
Vector | seen | unseen | HM
| (mAP) | (mAP) | (mAP) | (mAP)
Our-FL-vocab | 48/17 | ftx | 5.68 | 34.32 | 2.23 | 4.19
Our-PL-vocab | 48/17 | ftx | 6.99 | 35.13 | 2.73 | 5.07
Our-FL-vocab | 65/15 | glo | 10.36 | 36.69 | 10.33 | 16.12
Our-PL-vocab | 65/15 | glo | 11.55 | 36.79 | 11.53 | 17.56
Our-FL-vocab | 65/15 | w2v | 12.04 | 37.31 | 12.05 | 18.22
Our-PL-vocab | 65/15 | w2v | 12.62 | 32.99 | 12.62 | 18.26
TABLE IV: More results on ZSD with different word-vectors (GloVe, FasText and Word2Vec). Method | Penalty Function | Seen/Unseen | Word | ZSD | GZSD
---|---|---|---|---|---
Vector | seen | unseen | HM
| (mAP) | (mAP) | (mAP) | (mAP)
Our-PL-vocab | softplus | 65/15 | w2v | 12.17 | 32.12 | 12.18 | 17.66
Our-PL-vocab | sigmoid | 65/15 | w2v | 12.62 | 32.99 | 12.62 | 18.26
TABLE V: Comparison of ZSD performance with softplus and sigmoid based penalty
functions.

Choice of Semantic Representation: In addition to word2vec as semantic word
vectors reported in the main paper, we also experiment with GloVe [9] and
FastText [50] as word vectors. We report those performances in Table IV. We
notice that Glove (glo) and FastText (ftx) achieve respectable performance,
although they do not work as well as word2vec. However, in all cases, Our-PL-
vocab beats Our-FL-vocab on ZSD in both cases.

Figure 12: (a) Impact of selecting close seen (b) Impact of IoU.

Varying T and IoU: In Sec. 5 of the main paper, we discussed the reduced
description of an unseen class based on closely related seen classes. We
experiment with the behavior of our model by varying a different number of
close seen classes. One can notice in Fig. 12(a), a smaller number of close
seen classes (e.g., 5) results in relatively better performance than using
more seen classes (e.g., 10$-$65) during ZSD prediction. This behavior is
related with the average number of classes per superclass (which is 7.2 for
MS-COCO excluding ‘person’) because dissimilar classes from a different
superclass may not contribute towards describing a particular unseen class.
Thus, we use only five close seen classes to describe an unseen in all our
experiments. In Fig. 12(b), we report the impact of choosing a different IoU
ratio (from $0.2$ to $0.6$) in ZSD. As expected, lower IoUs result in better
performance than higher ones. As practiced in object detection literature, we
use IoU$=0.5$ for all other experiments in this paper.

Traditional detection: We report traditional detection performance of
different versions of our framework in Fig. 13. As a general observation, it
is clear that making detection explicitly dependent on semantic information
hurts the detector’s performance on ‘seen’ classes (traditional detection).
This is consistent with the common belief that training directly on the
desired output space in an end-to-end supervised setting achieves a better
performance [52, 53]. Consistently, we notice that FL-basic achieves the best
performance because it is free from the noisy word vectors. Our-FL-word
performs relatively worse than FL-basic because of using noise word vectors as
class semantics inside the network. Then, word vectors of vocabulary texts
further reduce the performance in Our-FL-vocab. Our proposed loss (Our-PL-word
and Our-PL-vocab cases) aligns visual features to noisy word vectors better
than FL which is valuable for zero-shot learning but slightly degrades the
seen performance. Similarly, we notice that while modifying the word
embeddings, the vocabulary metric focuses more on proper visual-semantic
alignment that is very helpful for ZSD but performs lower for the
seen/traditional detection setting.

Figure 13: Traditional detection performance on 65 seen classes of MSCOCO.
Figure 14: Qualitative results of ZSD (two top rows) and GZSD (two bottom
rows). Pink and yellow boxes represent unseen and seen detections
respectively.

Pascal VOC results: To compare with YOLO-based ZSD Demirel _et al._[3], we
adopt their exact settings with Pascal VOC 2007 and 2012. Note that, their
approach used attribute vectors as semantics from [51]. As such attribute
vectors are not available for our vocabulary list, we compare this approach
with only using fixed attribute vectors inside our network. Our method beats
Demirel _et al._[3] by a large margin ($57.9$ vs $63.5$ on traditional
detection, $54.5$ vs $62.1$ on unseen detection).

##  6 Conclusion

In this paper, we propose an end-to-end trainable framework for ZSD based on a
new loss formulation. Our proposed polarity loss promotes correct alignment
between visual and semantic domains via max-margin constraints and by
dynamically refining the noisy semantic representations. Further, it penalizes
an example considering background vs. object imbalance, easy vs. hard cases
and inter-class vs. intra-class relations. We qualitatively demonstrate that
in the learned semantic embedding space, word vectors become well-distributed
and visually similar classes reside close together. We propose a realistic
seen-unseen split on the MS-COCO dataset to evaluate ZSD methods. In our
experiments, we have outperformed several recent state-of-the-art methods on
both ZSD and GZSD tasks across the MS-COCO and Pascal VOC 2007 datasets.

## References

  * [1] S. Rahman, S. Khan, and F. Porikli, “Zero-shot object detection: Learning to simultaneously recognize and localize novel concepts,” in Asian Conference on Computer Vision (ACCV), December 2018.
  * [2] A. Bansal, K. Sikka, G. Sharma, R. Chellappa, and A. Divakaran, “Zero-shot object detection,” in The European Conference on Computer Vision (ECCV), September 2018.
  * [3] B. Demirel, R. G. Cinbis, and N. Ikizler-Cinbis, “Zero-shot object detection by hybrid region embedding,” in British Machine Vision Conference (BMVC), Sep. 2018.
  * [4] P. Zhu, H. Wang, T. Bolukbasi, and V. Saligrama, “Zero-shot detection,” arXiv preprint arXiv:1803.07113, 2018.
  * [5] Z. Li, L. Yao, X. Zhang, X. Wang, S. Kanhere, and H. Zhang, “Zero-shot object detection with textual descriptions,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 8690–8697, Jul. 2019.
  * [6] S. Rahman, S. Khan, and N. Barnes, “Improved visual-semantic alignment for zero-shot object detection,” in Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), February 2020.
  * [7] C. H. Lampert, H. Nickisch, and S. Harmeling, “Attribute-based classification for zero-shot visual object categorization,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 36, pp. 453–465, March 2014\.
  * [8] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in NIPS (C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, eds.), pp. 3111–3119, Curran Associates, Inc., 2013.
  * [9] J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in EMNLP, pp. 1532–1543, 2014.
  * [10] S. Rahman, S. Khan, and F. Porikli, “A unified approach for conventional zero-shot, generalized zero-shot, and few-shot learning,” IEEE Transactions on Image Processing, vol. 27, pp. 5652–5667, Nov 2018.
  * [11] L. Zhang, T. Xiang, and S. Gong, “Learning a deep embedding model for zero-shot learning,” in CVPR, July 2017.
  * [12] E. Kodirov, T. Xiang, and S. Gong, “Semantic autoencoder for zero-shot learning,” in CVPR, July 2017.
  * [13] Y. Xian, Z. Akata, G. Sharma, Q. Nguyen, M. Hein, and B. Schiele, “Latent embeddings for zero-shot classification,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [14] Y. Yu, Z. Ji, J. Guo, and Z. Zhang, “Zero-shot learning via latent space encoding,” IEEE transactions on cybernetics, no. 99, pp. 1–12, 2018.
  * [15] J. Song, C. Shen, Y. Yang, Y. Liu, and M. Song, “Transductive unbiased embedding for zero-shot learning,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [16] Y. Yu, Z. Ji, X. Li, J. Guo, Z. Zhang, H. Ling, and F. Wu, “Transductive zero-shot learning with a self-training dictionary approach,” IEEE Transactions on Cybernetics, vol. 48, pp. 2908–2919, Oct 2018.
  * [17] R. Felix, V. B. G. Kumar, I. Reid, and G. Carneiro, “Multi-modal cycle-consistent generalized zero-shot learning,” in The European Conference on Computer Vision (ECCV), September 2018.
  * [18] E. Kodirov, T. Xiang, Z. Fu, and S. Gong, “Unsupervised domain adaptation for zero-shot learning,” in The IEEE International Conference on Computer Vision (ICCV), December 2015.
  * [19] A. Zhao, M. Ding, J. Guan, Z. Lu, T. Xiang, and J.-R. Wen, “Domain-invariant projection learning for zero-shot recognition,” in Advances in Neural Information Processing Systems 31 (S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, eds.), pp. 1019–1030, Curran Associates, Inc., 2018.
  * [20] Z. Al-Halah, M. Tapaswi, and R. Stiefelhagen, “Recovering the missing link: Predicting class-attribute associations for unsupervised zero-shot learning,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [21] Z. Al-Halah and R. Stiefelhagen, “Automatic discovery, association estimation and learning of semantic attributes for a thousand categories,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [22] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, pp. 1137–1149, June 2017\.
  * [23] K. H. J. S. Jifeng Dai, Yi Li, “R-FCN: Object detection via region-based fully convolutional networks,” arXiv preprint arXiv:1605.06409, 2016.
  * [24] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, SSD: Single Shot MultiBox Detector, pp. 21–37.  Cham: Springer International Publishing, 2016.
  * [25] J. Redmon and A. Farhadi, “Yolo9000: Better, faster, stronger,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017\.
  * [26] T.-Y. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [27] T. Kong, F. Sun, C. Tan, H. Liu, and W. Huang, “Deep feature pyramid reconfiguration for object detection,” in The European Conference on Computer Vision (ECCV), September 2018.
  * [28] T. Lin, P. Goyal, R. Girshick, K. He, and P. Dollar, “Focal loss for dense object detection,” IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1–1, 2018.
  * [29] H. Xu, X. Lv, X. Wang, Z. Ren, N. Bodla, and R. Chellappa, “Deep regionlets for object detection,” in The European Conference on Computer Vision (ECCV), September 2018.
  * [30] Z. Li, C. Peng, G. Yu, X. Zhang, Y. Deng, and J. Sun, “Detnet: Design backbone for object detection,” in The European Conference on Computer Vision (ECCV), September 2018.
  * [31] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
  * [32] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” vol. 2016-January, pp. 770–778, 2016.  cited By 107.
  * [33] Y. Fu, T. Xiang, Y. Jiang, X. Xue, L. Sigal, and S. Gong, “Recent advances in zero-shot recognition: Toward data-efficient understanding of visual content,” IEEE Signal Processing Magazine, vol. 35, pp. 112–125, Jan 2018\.
  * [34] Y. Xian, C. H. Lampert, B. Schiele, and Z. Akata, “Zero-shot learning - a comprehensive evaluation of the good, the bad and the ugly,” IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1–1, 2018.
  * [35] C.-W. Lee, W. Fang, C.-K. Yeh, and Y.-C. Frank Wang, “Multi-label zero-shot learning with structured knowledge graphs,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [36] S. Rahman and S. Khan, “Deep multiple instance learning for zero-shot image tagging,” in Asian Conference on Computer Vision (ACCV), December 2018\.
  * [37] Z. Li, E. Gavves, T. Mensink, and C. G. Snoek, “Attributes make sense on segmented objects,” in European Conference on Computer Vision, pp. 350–365, Springer, 2014.
  * [38] S. Jetley, M. Sapienza, S. Golodetz, and P. H. Torr, “Straight to shapes: Real-time detection of encoded shapes,” arXiv preprint arXiv:1611.07932, 2016.
  * [39] R. Hu, H. Xu, M. Rohrbach, J. Feng, K. Saenko, and T. Darrell, “Natural language object retrieval,” in CVPR, pp. 4555–4564, 2016.
  * [40] Z. Li, R. Tao, E. Gavves, C. Snoek, and A. Smeulders, “Tracking by natural language specification,” in CVPR, pp. 6495–6503, 2017.
  * [41] Z. Ren, H. Jin, Z. Lin, C. Fang, and A. Yuille, “Multiple instance visual-semantic embedding,” in BMVC, 2017.
  * [42] C. L. Zitnick and P. Dollár, “Edge boxes: Locating object proposals from edges,” in ECCV (D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, eds.), (Cham), pp. 391–405, Springer International Publishing, 2014.
  * [43] B. Demirel, R. G. Cinbis, and N. Ikizler-Cinbis, “Image captioning with unseen objects,” in British Machine Vision Conference (BMVC), Sep. 2019.
  * [44] S. Rahman, S. Khan, and N. Barnes, “Transductive learning for zero-shot object detection,” in The IEEE International Conference on Computer Vision (ICCV), October 2019.
  * [45] L. Van Der Maaten, “Accelerating t-sne using tree-based algorithms.,” Journal of machine learning research, vol. 15, no. 1, pp. 3221–3245, 2014.
  * [46] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in ECCV, pp. 740–755, Springer, 2014.
  * [47] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” International journal of computer vision, vol. 88, no. 2, pp. 303–338, 2010.
  * [48] G. A. Miller, “Wordnet: a lexical database for english,” Communications of the ACM, vol. 38, no. 11, pp. 39–41, 1995.
  * [49] T.-S. Chua, J. Tang, R. Hong, H. Li, Z. Luo, and Y. Zheng, “Nus-wide: A real-world web image database from national university of singapore,” in Proceedings of the ACM International Conference on Image and Video Retrieval, CIVR ’09, (New York, NY, USA), pp. 48:1–48:9, ACM, 2009.
  * [50] A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov, “Bag of tricks for efficient text classification,” vol. 2, pp. 427–431, 2017.
  * [51] A. Farhadi, I. Endres, D. Hoiem, and D. Forsyth, “Describing objects by their attributes,” in CVPR, pp. 1778–1785, IEEE, 2009.
  * [52] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, et al., “End to end learning for self-driving cars,” arXiv preprint arXiv:1604.07316, 2016.
  * [53] J. Schmidhuber, “Deep learning in neural networks: An overview,” Neural networks, vol. 61, pp. 85–117, 2015.


c\in\mathcal{C}^{\prime}\\}=\bigwedge\\{\Diamondblack_{g}c\mid
c\in\mathcal{C}\\}$.

###### Proof.

1\. For the first part, it is enough to show that, if $k\in
K(\mathbb{A}^{\delta})$ and $k\leq\blacksquare_{f}(\bigvee\mathcal{O})$, then
$k\leq\blacksquare_{f}(o_{k})$ for some $o_{k}\in\mathcal{O}$. The assumption
$k\leq\blacksquare_{f}(\bigvee\mathcal{O})$ can be rewritten as
$\Diamond_{f}k\leq\bigvee\mathcal{O}$, which together with
$f(\bot)\leq\bigvee\mathcal{O}$ yields
$f(\bot)\vee\Diamond_{f}k\leq\bigvee\mathcal{O}$. By Lemma 4.1, this
inequality can be rewritten as $f(k)\leq\bigvee\mathcal{O}$. From $f$ being
closed Esakia and $f(a)\in K(\mathbb{B}^{\delta})$ for every $a\in\mathbb{A}$,
it follows that $f(k)\in K(\mathbb{B}^{\delta})$. By compactness,
$f(k)\leq\bigvee_{i=1}^{n}o_{i}$ for some $o_{1},\ldots,o_{n}\in\mathcal{O}$.
Since $\mathcal{O}$ is upward-directed, $o_{k}\geq\bigvee_{i=1}^{n}o_{i}$ for
some $o_{k}\in\mathcal{O}$. Then $f(\bot)\vee\Diamond_{f}k=f(k)\leq o_{k}$,
which yields $\Diamond_{f}k\leq o_{k}$, which by adjunction can be rewritten
as $k\leq\blacksquare_{f}(o_{k})$ as required.

As to the second part of the statement, notice that the assumption
$f(\bot)\leq\bigvee\mathcal{O}$ is too weak to imply that $f(\bot)\leq o$ for
each $o\in\mathcal{O}$, and hence we cannot conclude, by way of Lemma 38, that
$\blacksquare_{f}o\in O(\mathbb{A}^{\delta})$ for every $o\in\mathcal{O}$.
However, let

$\mathcal{O}^{\prime}:=\\{o\in\mathcal{O}\mid o\geq o_{k}\mbox{ for some }k\in
K(\mathbb{A}^{\delta})\mbox{ s.t.
}k\leq\blacksquare_{f}(\bigvee\mathcal{O})\\}.$

Clearly, by construction we have that $\mathcal{O}^{\prime}$ is upward-
directed and $\bigvee\mathcal{O}^{\prime}=\bigvee\mathcal{O}$, and moreover
for each $o\in\mathcal{O}^{\prime}$, we have that $f(\bot)\leq o_{k}\leq o$
for some $k\in K(\mathbb{A}^{\delta})$ s.t.
$k\leq\blacksquare_{f}(\bigvee\mathcal{O})$. Hence, by Lemma 38,
$\blacksquare_{f}o\in O(\mathbb{A}^{\delta})$ for each
$o\in\mathcal{O}^{\prime}$. Moreover, the monotonicity of $\blacksquare_{f}$
and the previous part of the statement imply that
$\bigvee\\{\blacksquare_{f}o\mid
o\in\mathcal{O}^{\prime}\\}=\blacksquare_{f}(\bigvee\mathcal{O})=\bigvee\\{\blacksquare_{f}o\mid
o\in\mathcal{O}\\}$.

2\. is order-dual. ∎

###  9.2 Intersection lemmas for $\mathcal{L}_{\mathrm{LE}}^{+}$-formulas

Recall that
$\blacksquare_{f},\Diamondblack_{g},\blacktriangleleft_{f},\blacktriangleright_{g}:\mathbb{A}^{\delta}\to\mathbb{A}^{\delta}$
respectively denote the adjoints of the normalization maps $\Diamond_{f}$ for
any $f\in\mathcal{F}_{r}$ with $\varepsilon_{f}=1$, $\Box_{g}$ for any
$g\in\mathcal{G}_{r}$ with $\varepsilon_{g}=1$, ${\lhd}_{f}$ for any
$f\in\mathcal{F}_{r}$ with $\varepsilon_{f}=\partial$, ${\rhd}_{g}$ for any
$g\in\mathcal{G}_{r}$ with $\varepsilon_{g}=\partial$. Then Lemma 38
immediately implies the following facts, which will be needed for the
soundness of the topological Ackermann rule:

######  Lemma 40.

  1. 1.

If $f\in\mathcal{F}_{r}$ with $\varepsilon_{f}=1$, and $o\in
O(\mathbb{A}^{\delta})$ s.t. $f(\bot)\leq o$, then
$\blacksquare_{f}(o)=\bigvee\\{a\ |\ a\leq\blacksquare_{f}(o)\\}\in
O(\mathbb{A}^{\delta})$.

  2. 2.

If $g\in\mathcal{G}_{r}$ with $\varepsilon_{g}=1$, and $k\in
K(\mathbb{A}^{\delta})$ s.t. $g(\top)\geq k$, then
$\Diamondblack_{g}(k)=\bigwedge\\{a\ |\ a\geq\Diamondblack_{g}(k)\\}\in
K(\mathbb{A}^{\delta})$.

  3. 3.

If $f\in\mathcal{F}_{r}$ with $\varepsilon_{f}=\partial$, and $o\in
O(\mathbb{A}^{\delta})$ s.t. $f(\top)\leq o$, then
$\blacktriangleleft_{f}(o)=\bigwedge\\{a\ |\
a\geq\blacktriangleleft_{f}(o)\\}\in K(\mathbb{A}^{\delta})$.

  4. 4.

If $g\in\mathcal{G}_{r}$ with $\varepsilon_{g}=\partial$, and $k\in
K(\mathbb{A}^{\delta})$ s.t. $g(\bot)\geq k$, then
$\blacktriangleright_{g}(k)=\bigvee\\{a\ |\
a\leq\blacktriangleright_{g}(k)\\}\in O(\mathbb{A}^{\delta})$.

In what follows, we work under the assumption that the values of all
parameters (propositional variables, nominals and conominals) occurring in the
term functions mentioned in the statements of propositions and lemmas are
given by admissible assignments.

######  Lemma 41.

Let $\varphi(p)$ be syntactically closed, $\psi(p)$ syntactically open (cf.
Definition 31), $c\in K(\mathbb{A}^{\delta})$ and $o\in
O(\mathbb{A}^{\delta})$.

  1. 1.

If $\varphi(p)$ is positive in $p$, $\psi(p)$ is negative in $p$, and for all
$f\in\mathcal{F}_{r}$ and $g\in\mathcal{G}_{r}$,

    1. (a)

if $\varepsilon_{f}=1$, then $f(\bot)\leq\psi^{\prime\mathbb{A}^{\delta}}(c)$
for any subformula $\blacksquare_{f}\psi^{\prime}(p)$ in $\varphi(p)$ and
$\psi(p)$;

    2. (b)

if $\varepsilon_{g}=1$, then $g(\top)\geq\psi^{\prime\mathbb{A}^{\delta}}(c)$
for any subformula $\Diamondblack_{g}\psi^{\prime}(p)$ in $\varphi(p)$ and
$\psi(p)$;

    3. (c)

if $\varepsilon_{f}=\partial$, then
$f(\top)\leq\psi^{\prime\mathbb{A}^{\delta}}(c)$ for any subformula
$\blacktriangleleft_{f}\\!\\!\psi^{\prime}(p)$ in $\varphi(p)$ and $\psi(p)$;

    4. (d)

if $\varepsilon_{g}=\partial$, then
$g(\bot)\geq\psi^{\prime\mathbb{A}^{\delta}}(c)$ for any subformula
$\blacktriangleright_{g}\\!\\!\psi^{\prime}(p)$ in $\varphi(p)$ and $\psi(p)$;

then $\varphi(c)\in K(\mathbb{A}^{\delta})$ and $\psi(c)\in
O(\mathbb{A}^{\delta})$ for each $c\in K(\mathbb{A}^{\delta})$.

  2. 2.

If $\varphi(p)$ is negative in $p$, $\psi(p)$ is positive in $p$, and for all
$f\in\mathcal{F}_{r}$ and $g\in\mathcal{G}_{r}$,

    1. (a)

if $\varepsilon_{f}=1$, then $f(\bot)\leq\psi^{\prime\mathbb{A}^{\delta}}(o)$
for any subformula $\blacksquare_{f}\psi^{\prime}(p)$ in $\varphi(p)$ and
$\psi(p)$;

    2. (b)

if $\varepsilon_{g}=1$, then $g(\top)\geq\psi^{\prime\mathbb{A}^{\delta}}(o)$
for any subformula $\Diamondblack_{g}\psi^{\prime}(p)$ in $\varphi(p)$ and
$\psi(p)$;

    3. (c)

if $\varepsilon_{f}=\partial$, then
$f(\top)\leq\psi^{\prime\mathbb{A}^{\delta}}(o)$ for any subformula
$\blacktriangleleft_{f}\\!\\!\psi^{\prime}(p)$ in $\varphi(p)$ and $\psi(p)$;

    4. (d)

if $\varepsilon_{g}=\partial$, then
$g(\bot)\geq\psi^{\prime\mathbb{A}^{\delta}}(o)$ for any subformula
$\blacktriangleright_{g}\\!\\!\psi^{\prime}(p)$ in $\varphi(p)$ and $\psi(p)$;

then $\varphi(o)\in K(\mathbb{A}^{\delta})$ and $\psi(o)\in
O(\mathbb{A}^{\delta})$ for each $o\in O(\mathbb{A}^{\delta})$.

###### Proof.

1\. The proof proceeds by simultaneous induction on $\varphi$ and $\psi$. It
is easy to see that $\varphi$ cannot be $\mathbf{m}$, and the outermost
connective of $\varphi$ cannot be $f^{\sharp}_{i}$ for any
$f\in\mathcal{F}_{n}$ with $\varepsilon_{f}(i)=1$, or $g_{j}^{\flat}$ for any
$g\in\mathcal{G}_{n}$ with $\varepsilon_{g}(j)=\partial$, or
$\blacksquare_{f}$ for any $f\in\mathcal{F}_{r}$ with $\varepsilon_{f}=1$, or
$\blacktriangleright_{g}$ for any $g\in\mathcal{G}_{r}$ with
$\varepsilon_{g}=\partial$. Similarly, $\psi$ cannot be $\mathbf{i}$, and the
outermost connective of $\psi$ cannot be $g_{j}^{\flat}$ for any
$g\in\mathcal{G}_{n}$ with $\varepsilon_{g}(j)=1$, or $f^{\sharp}_{i}$ for any
$f\in\mathcal{F}_{n}$ with $\varepsilon_{f}(i)=\partial$, or
$\Diamondblack_{g}$ for any $g\in\mathcal{G}_{r}$ with $\varepsilon_{g}=1$, or
$\blacktriangleleft_{f}$ for any $f\in\mathcal{F}_{r}$ with
$\varepsilon_{f}=\partial$.

The basic cases, that is, $\varphi=\perp,\top,p,q,\mathbf{i}$ and
$\psi=\perp,\top,p,q,\mathbf{m}$ are straightforward.

Assume that $\varphi(p)=\Diamondblack_{g}\varphi^{\prime}(p)$. Since
$\varphi(p)$ is positive in $p$, the subformula $\varphi^{\prime}(p)$ is
syntactically closed and positive in $p$, and assumptions 1(a)-1(d) hold also
for $\varphi^{\prime}(p)$. Hence, by inductive hypothesis,
$\varphi^{\prime}(c)\in K(\mathbb{A}^{\delta})$ for any $c\in
K(\mathbb{A}^{\delta})$. In particular, assumption 1(b) implies that
$g(\top)\geq\varphi^{\prime}(c)$. Hence, by Lemma 40,
$\Diamondblack_{g}\varphi^{\prime}(c)\in K(\mathbb{A}^{\delta})$, as required.
The case in which $\varphi(p)$ is negative in $p$ is argued order-dually.

The cases in which
$\varphi(p)=\blacksquare_{f}\varphi^{\prime}(p),\blacktriangleleft_{f}\\!\\!\varphi^{\prime}(p),\blacktriangleright_{g}\\!\\!\varphi^{\prime}(p)$
are similar to the one above.

The cases of the remaining connectives are treated as in [5, Lemma 11.9] and
the corresponding proofs are omitted. Of course, in [5] nominals and co-
nominals are evaluated to completely join- and meet-irreducibles,
respectively. However, the only property of completely join- and meet-
irreducibles used there is the fact that they are, respectively, closed and
open, and therefore the proof is directly transferable to the present setting.
∎

######  Lemma 42 (Intersection lemma).

Let $\varphi(p)$ be syntactically closed, $\psi(p)$ syntactically open,
$\mathcal{C}\subseteq K(\mathbb{A}^{\delta})$ downward-directed,
$\mathcal{O}\subseteq O(\mathbb{A}^{\delta})$ upward-directed. Then

  1. 1.

if $\varphi(p)$ is positive in $p$, $\psi(p)$ is negative in $p$, and for all
$f\in\mathcal{F}_{r}$ and $g\in\mathcal{G}_{r}$,

    1. (a)

if $\varepsilon_{f}=1$, then
$f(\bot)\leq\psi^{\prime\mathbb{A}^{\delta}}(\bigwedge\mathcal{C})$ for any
subformula $\blacksquare_{f}\psi^{\prime}(p)$ in $\varphi(p)$ and $\psi(p)$,

    2. (b)

if $\varepsilon_{g}=1$, then
$g(\top)\geq\psi^{\prime\mathbb{A}^{\delta}}(\bigwedge\mathcal{C})$ for any
subformula $\Diamondblack_{g}\psi^{\prime}(p)$ in $\varphi(p)$ and $\psi(p)$,

    3. (c)

if $\varepsilon_{f}=\partial$, then
$f(\top)\leq\psi^{\prime\mathbb{A}^{\delta}}(\bigwedge\mathcal{C})$ for any
subformula $\blacktriangleleft_{f}\\!\\!\psi^{\prime}(p)$ in $\varphi(p)$ and
$\psi(p)$,

    4. (d)

if $\varepsilon_{g}=\partial$, then
$g(\bot)\geq\psi^{\prime\mathbb{A}^{\delta}}(\bigwedge\mathcal{C})$ for any
subformula $\blacktriangleright_{g}\\!\\!\psi^{\prime}(p)$ in $\varphi(p)$ and
$\psi(p)$,

then

    1. (a)

$\varphi^{\mathbb{A}^{\delta}}(\bigwedge\mathcal{C})=\bigwedge\\{\varphi^{\mathbb{A}^{\delta}}(c):c\in\mathcal{C}^{\prime}\\}$
for some down-directed subcollection
$\mathcal{C}^{\prime}\subseteq\mathcal{C}$ such that
$\varphi^{\mathbb{A}^{\delta}}(c)\in K(\mathbb{A}^{\delta})$ for each
$c\in\mathcal{C}^{\prime}$.

    2. (b)

$\psi^{\mathbb{A}^{\delta}}(\bigwedge\mathcal{C})=\bigvee\\{\psi^{\mathbb{A}^{\delta}}(c):c\in\mathcal{C}^{\prime}\\}$
for some down-directed subcollection
$\mathcal{C}^{\prime}\subseteq\mathcal{C}$ such that
$\psi^{\mathbb{A}^{\delta}}(c)\in O(\mathbb{A}^{\delta})$ for each
$c\in\mathcal{C}^{\prime}$.

  2. 2.

If $\varphi(p)$ is negative in $p$, $\psi(p)$ is positive in $p$, and for all
$f\in\mathcal{F}_{r}$ and $g\in\mathcal{G}_{r}$,

    1. (a)

if $\varepsilon_{f}=1$, then
$f(\bot)\leq\psi^{\prime\mathbb{A}^{\delta}}(\bigvee\mathcal{O})$ for any
subformula $\blacksquare_{f}\psi^{\prime}(p)$ in $\varphi(p)$ and $\psi(p)$,

    2. (b)

if $\varepsilon_{g}=1$, then
$g(\top)\geq\psi^{\prime\mathbb{A}^{\delta}}(\bigvee\mathcal{O})$ for any
subformula $\Diamondblack_{g}\psi^{\prime}(p)$ in $\varphi(p)$ and $\psi(p)$,

    3. (c)

if $\varepsilon_{f}=\partial$, then
$f(\top)\leq\psi^{\prime\mathbb{A}^{\delta}}(\bigvee\mathcal{O})$ for any
subformula $\blacktriangleleft_{f}\\!\\!\psi^{\prime}(p)$ in $\varphi(p)$ and
$\psi(p)$,

    4. (d)

if $\varepsilon_{g}=\partial$, then
$g(\bot)\geq\psi^{\prime\mathbb{A}^{\delta}}(\bigvee\mathcal{O})$ for any
subformula $\blacktriangleright_{g}\\!\\!\psi^{\prime}(p)$ in $\varphi(p)$ and
$\psi(p)$,

then

    1. (a)

$\varphi^{\mathbb{A}^{\delta}}(\bigvee\mathcal{O})=\bigwedge\\{\varphi^{\mathbb{A}^{\delta}}(o):o\in\mathcal{O}^{\prime}\\}$
for some up-directed subcollection $\mathcal{O}^{\prime}\subseteq\mathcal{O}$
such that $\varphi^{\mathbb{A}^{\delta}}(o)\in K(\mathbb{A}^{\delta})$ for
each $o\in\mathcal{O}^{\prime}$.

    2. (b)

$\psi^{\mathbb{A}^{\delta}}(\bigvee\mathcal{O})=\bigvee\\{\psi^{\mathbb{A}^{\delta}}(o):o\in\mathcal{O}^{\prime}\\}$
for some up-directed subcollection $\mathcal{O}^{\prime}\subseteq\mathcal{O}$
such that $\psi^{\mathbb{A}^{\delta}}(o)\in O(\mathbb{A}^{\delta})$ for each
$o\in\mathcal{O}^{\prime}$.

###### Proof.

The proof proceeds by simultaneous induction on $\varphi$ and $\psi$. It is
easy to see that $\varphi$ cannot be $\mathbf{m}$, and the outermost
connective of $\varphi$ cannot be $f^{\sharp}_{i}$ for any
$f\in\mathcal{F}_{n}$ with $\varepsilon_{f}(i)=1$, or $g_{j}^{\flat}$ for any
$g\in\mathcal{G}_{n}$ with $\varepsilon_{g}(j)=\partial$, or
$\blacksquare_{f}$ for any $f\in\mathcal{F}_{r}$ with $\varepsilon_{f}=1$, or
$\blacktriangleright_{g}$ for any $g\in\mathcal{G}_{r}$ with
$\varepsilon_{g}=\partial$. Similarly, $\psi$ cannot be $\mathbf{i}$, and the
outermost connective of $\psi$ cannot be $g_{j}^{\flat}$ for any
$g\in\mathcal{G}_{n}$ with $\varepsilon_{g}(j)=1$, or $f^{\sharp}_{i}$ for any
$f\in\mathcal{F}_{n}$ with $\varepsilon_{f}(i)=\partial$, or
$\Diamondblack_{g}$ for any $g\in\mathcal{G}_{r}$ with $\varepsilon_{g}=1$, or
$\blacktriangleleft_{f}$ for any $f\in\mathcal{F}_{r}$ with
$\varepsilon_{f}=\partial$.

The basic cases in which $\varphi=\perp,\top,p,q,\mathbf{i}$ and
$\psi=\perp,\top,p,q,\mathbf{m}$ are straightforward.

Assume that $\varphi(p)=\Diamondblack_{g}\varphi^{\prime}(p)$ for some
$g\in\mathcal{G}_{r}$ with $\varepsilon_{g}=1$. Since $\varphi(p)$ is positive
in $p$, the subformula $\varphi^{\prime}(p)$ is syntactically closed and
positive in $p$, and assumptions 1(a)-1(d) hold also for
$\varphi^{\prime}(p)$. Hence, by inductive hypothesis,
$\varphi^{\prime}(\bigwedge\mathcal{C})=\bigwedge\\{\varphi^{\prime}(c)\mid
c\in\mathcal{C}^{\prime\prime}\\}$ for some down-directed subcollection
$\mathcal{C}^{\prime\prime}\subseteq\mathcal{C}$ such that
$\varphi^{\prime}(c)\in K(\mathbb{A}^{\delta})$ for each
$c\in\mathcal{C}^{\prime\prime}$. In particular, assumption 1(b) implies that
$g(\top)\geq\varphi^{\prime}(\bigwedge\mathcal{C})=\bigwedge\\{\varphi^{\prime}(c)\mid
c\in\mathcal{C}^{\prime\prime}\\}$. Notice that $\varphi^{\prime}(p)$ being
positive in $p$ and $\mathcal{C}^{\prime\prime}$ being down-directed imply
that $\\{\varphi^{\prime}(c)\mid c\in\mathcal{C}^{\prime\prime}\\}$ is down-
directed. Hence, by Proposition 39 applied to $\\{\varphi^{\prime}(c)\mid
c\in\mathcal{C}^{\prime\prime}\\}$, we get that
$\Diamondblack_{g}\varphi^{\prime}(\bigwedge\mathcal{C})=\Diamondblack_{g}(\bigwedge\\{\varphi^{\prime}(c)\mid
c\in\mathcal{C}^{\prime\prime}\\})=\bigwedge\\{\Diamondblack_{g}\varphi^{\prime}(c)\mid
c\in\mathcal{C}^{\prime\prime}\\}$. Moreover, there exists some down-directed
subcollection $\mathcal{C}^{\prime}\subseteq\mathcal{C}^{\prime\prime}$ such
that $\Diamondblack_{g}\varphi^{\prime}(c)\in K(\mathbb{A}^{\delta})$ for each
$c\in\mathcal{C}^{\prime}$ and
$\bigwedge\\{\Diamondblack_{g}\varphi^{\prime}(c)\mid
c\in\mathcal{C}^{\prime}\\}=\bigwedge\\{\Diamondblack_{g}\varphi^{\prime}(c)\mid
c\in\mathcal{C}^{\prime\prime}\\}$. This gives us
$\Diamondblack_{g}\varphi^{\prime}(\bigwedge\mathcal{C})=\bigwedge\\{\Diamondblack_{g}\varphi^{\prime}(c)\mid
c\in\mathcal{C}^{\prime}\\}$, as required. The case in which $\varphi(p)$ is
negative in $p$ is argued order-dually.

The cases in which
$\varphi(p)=\blacksquare_{g}\varphi^{\prime}(p),\blacktriangleleft_{f}\\!\\!\varphi^{\prime}(p),\blacktriangleright_{g}\\!\\!\varphi^{\prime}(p)$
are similar to the one above.

The cases of the remaining connectives are treated as in [5, Lemma 11.10] and
the corresponding proofs are omitted. ∎

###  9.3 Topological Ackermann Lemmas for $\mathcal{L}_{\mathrm{LE}}^{+}$

######  Definition 43.

A system $S$ of $\mathcal{L}_{\mathrm{LE}}^{+}$-inequalities is topologically
adequate when the following conditions hold for all $f\in\mathcal{F}_{r}$ and
$g\in\mathcal{G}_{r}$:

  1. 1.

if $\varepsilon_{f}=1$ and $\varphi\leq\blacksquare_{f}\psi$ is in $S$, then
$f(\bot)\leq\psi$ is in $S$;

  2. 2.

if $\varepsilon_{g}=1$ and $\Diamondblack_{g}\varphi\leq\psi$ is in $S$, then
$g(\top)\geq\varphi$ is in $S$;

  3. 3.

if $\varepsilon_{f}=\partial$ and
$\blacktriangleleft_{f}\\!\\!\varphi\leq\psi$ is in $S$, then
$f(\top)\leq\varphi$ is in $S$;

  4. 4.

if $\varepsilon_{g}=\partial$ and
$\varphi\leq\blacktriangleright_{g}\\!\\!\psi$ is in $S$, then
$g(\bot)\geq\psi$ is in $S$.

######  Proposition 44 (Right-handed Topological Ackermann Lemma).

Let $S$ be a topologically adequate system of
$\mathcal{L}_{\mathrm{LE}}^{+}$-inequalities which is the union of the
following disjoint subsets:

  * –

$S_{1}$ consists only of inequalities in which $p$ does not occur;

  * –

$S_{2}$ consists of inequalities of the type $\alpha\leq p$, where $\alpha$ is
syntactically closed and $p$ does not occur in $\alpha$;

  * –

$S_{3}$ consists of inequalities of the type $\beta(p)\leq\gamma(p)$ where
$\beta(p)$ is syntactically closed and positive in $p$, and $\gamma(p)$ be
syntactically open and negative in $p$,

Then the following are equivalent:

  1. 1.

$\beta^{\mathbb{A}^{\delta}}(\bigvee\alpha^{\mathbb{A}^{\delta}})\leq\gamma^{\mathbb{A}^{\delta}}(\bigvee\alpha^{\mathbb{A}^{\delta}})$
for all inequalities in $S_{3}$, where $\bigvee\alpha$ abbreviates
$\bigvee\\{\alpha\mid\alpha\leq p\in S_{2}\\}$;

  2. 2.

There exists $a_{0}\in\mathbb{A}$ such that
$\bigvee\alpha^{\mathbb{A}^{\delta}}\leq a_{0}$ and
$\beta^{\mathbb{A}^{\delta}}(a_{0})\leq\gamma^{\mathbb{A}^{\delta}}(a_{0})$
for all inequalities in $S_{3}$.

###### Proof.

$(\Leftarrow)$ By the monotonicity of $\beta_{i}(p)$ and antitonicity of
$\gamma_{i}(p)$ in $p$ for $1\leq i\leq n$, together with
$\alpha^{\mathbb{A}^{\delta}}\leq a_{0}$, we have that
$\beta_{i}^{\mathbb{A}^{\delta}}(\alpha^{\mathbb{A}^{\delta}})\leq\beta_{i}^{\mathbb{A}^{\delta}}(a_{0})\leq\gamma_{i}^{\mathbb{A}^{\delta}}(a_{0})\leq\gamma_{i}^{\mathbb{A}^{\delta}}(\alpha^{\mathbb{A}^{\delta}})$.

$(\Rightarrow)$ Since the quasi-inequality is topologically adequate, by Lemma
42.1, $\alpha^{\mathbb{A}^{\delta}}\in K(\mathbb{A}^{\delta})$.

Hence,
$\alpha^{\mathbb{A}^{\delta}}=\bigwedge\\{a\in\mathbb{A}:\alpha^{\mathbb{A}^{\delta}}\leq
a\\}$, making it the meet of a downward-directed set of clopen elements.
Therefore, we can rewrite each inequality in $S_{3}$ as

$\beta^{\mathbb{A}^{\delta}}(\bigwedge\\{a\in\mathbb{A}:\alpha^{\mathbb{A}^{\delta}}\leq
a\\})\leq\gamma^{\mathbb{A}^{\delta}}(\bigwedge\\{a\in\mathbb{A}:\alpha^{\mathbb{A}^{\delta}}\leq
a\\}).$

Since $\beta$ is syntactically closed and positive in $p$, $\gamma$ is
syntactically open and negative in $p$, again by topological adequacy, we can
apply Lemma 42 and get that

$\bigwedge\\{\beta^{\mathbb{A}^{\delta}}(a):a\in\mathcal{A}_{1}\\}\leq\bigvee\\{\gamma_{i}^{\mathbb{A}^{\delta}}(b):b\in\mathcal{A}_{2}\\}$

for some
$\mathcal{A}_{1},\mathcal{A}_{2}\subseteq\\{a\in\mathbb{A}:\alpha^{\mathbb{A}^{\delta}}\leq
a\\}$ such that $\beta^{\mathbb{A}^{\delta}}(a)\in K(\mathbb{A}^{\delta})$ for
each $a\in\mathcal{A}_{1}$, and $\gamma^{\mathbb{A}^{\delta}}(b)\in
O(\mathbb{A}^{\delta})$ for each $b\in\mathcal{A}_{2}$. By compactness,

$\bigwedge\\{\beta_{i}^{\mathbb{A}^{\delta}}(a):a\in\mathcal{A}^{\prime}_{1}\\}\leq\bigvee\\{\gamma_{i}^{\mathbb{A}^{\delta}}(b):b\in\mathcal{A}^{\prime}_{2}\\}$

for some finite subsets $\mathcal{A}^{\prime}_{1}\subseteq\mathcal{A}_{1}$ and
$\mathcal{A}^{\prime}_{2}\subseteq\mathcal{A}_{2}$. Then let
$a^{*}=\bigwedge\\{\bigwedge\mathcal{A}^{\prime}_{1}\wedge\bigwedge\mathcal{A}^{\prime}_{2}\mid\beta\leq\gamma\in
S_{3}\\}.$ Clearly, $a^{*}\in\mathbb{A}$, and
$\alpha^{\mathbb{A}^{\delta}}\leq a^{*}$. By the monotonicity of $\beta(p)$
and the antitonicity of $\gamma(p)$ in $p$ for each $\beta\leq\gamma$ in
$S_{3}$, we have
$\beta^{\mathbb{A}^{\delta}}(a^{*})\leq\beta^{\mathbb{A}^{\delta}}(a)$ and
$\gamma_{i}^{\mathbb{A}^{\delta}}(b)\leq\gamma_{i}^{\mathbb{A}^{\delta}}(a^{*})$
for all $a\in\mathcal{A}^{\prime}_{1}$ and all $b\in\mathcal{A}^{\prime}_{2}$.
Therefore,

$\beta_{i}^{\mathbb{A}^{\delta}}(a^{*})\leq\bigwedge\\{\beta_{i}^{\mathbb{A}^{\delta}}(a):a\in\mathcal{A}^{\prime}_{1}\\}\leq\bigvee\\{\gamma_{i}^{\mathbb{A}^{\delta}}(b):b\in\mathcal{A}^{\prime}_{2}\\}\leq\gamma_{i}^{\mathbb{A}^{\delta}}(a^{*})$

for each $\beta\leq\gamma$ in $S_{3}$. ∎

######  Proposition 45 (Left-handed Topological Ackermann Lemma).

Let $S$ be a topologically adequate system of
$\mathcal{L}_{\mathrm{LE}}^{+}$-inequalities which is the union of the
following disjoint subsets:

  * –

$S_{1}$ consists only of inequalities in which $p$ does not occur;

  * –

$S_{2}$ consists of inequalities of the type $p\leq\alpha$, where $\alpha$ is
syntactically open and $p$ does not occur in $\alpha$;

  * –

$S_{3}$ consists of inequalities of the type $\beta(p)\leq\gamma(p)$ where
$\beta(p)$ is syntactically closed and negative in $p$, and $\gamma(p)$ be
syntactically open and positive in $p$,

Then the following are equivalent:

  1. 1.

$\beta^{\mathbb{A}^{\delta}}(\bigwedge\alpha^{\mathbb{A}^{\delta}})\leq\gamma^{\mathbb{A}^{\delta}}(\bigwedge\alpha^{\mathbb{A}^{\delta}})$
for all inequalities in $S_{3}$, where $\bigwedge\alpha$ abbreviates
$\bigwedge\\{\alpha\mid p\leq\alpha\in S_{2}\\}$;

  2. 2.

There exists $a_{0}\in\mathbb{A}$ such that
$a_{0}\leq\bigwedge\alpha^{\mathbb{A}^{\delta}}$ and
$\beta^{\mathbb{A}^{\delta}}(a_{0})\leq\gamma^{\mathbb{A}^{\delta}}(a_{0})$
for all inequalities in $S_{3}$.

###### Proof.

The proof is similar to the proof of the right-handed Ackermann lemma and is
omitted. ∎

## References

  * [1] W. Conradie and A. Craig.  Canonicity results for mu-calculi: An algorithmic approach.  Journal of Logic and Computation, forthcoming.
  * [2] W. Conradie, Y. Fomatati, A. Palmigiano, and S. Sourabh.  Algorithmic correspondence for intuitionistic modal mu-calculus.  Theoretical Computer Science, 564:30–62, 2015.
  * [3] W. Conradie, S. Ghilardi, and A. Palmigiano.  Unified correspondence.  In A. Baltag and S. Smets, editors, Johan van Benthem on Logic and Information Dynamics, volume 5 of Outstanding Contributions to Logic, pages 933–975. Springer International Publishing, 2014.
  * [4] W. Conradie, V. Goranko, and D. Vakarelov.  Algorithmic correspondence and completeness in modal logic. I. The core algorithm SQEMA.  Logical Methods in Computer Science, 2006.
  * [5] W. Conradie and A. Palmigiano.  Algorithmic correspondence and canonicity for distributive modal logic.  Annals of Pure and Applied Logic, 163(3):338 – 376, 2012.
  * [6] W. Conradie and A. Palmigiano.  Algorithmic correspondence and canonicity for non-distributive logics.  Journal of Logic and Computation, forthcoming.
  * [7] W. Conradie, A. Palmigiano, S. Sourabh, and Z. Zhao.  Canonicity and relativized canonicity via pseudo-correspondence: an application of ALBA.  Submitted.
  * [8] W. Conradie, A. Palmigiano, and Z. Zhao.  Sahlqvist via translation.  Submitted.
  * [9] W. Conradie and C. Robinson.  On Sahlqvist theory for hybrid logic.  Journal of Logic and Computation, 2015.
  * [10] B. Davey and H. Priestley.  Introduction to Lattices and Order.  Cambridge University Press, 2002.
  * [11] J. M. Dunn, M. Gehrke, and A. Palmigiano.  Canonical extensions and relational completeness of some substructural logics.  Journal Symbolic Logic, 70(3):713–740, 2005.
  * [12] S. Frittella, A. Palmigiano, and L. Santocanale.  Dual characterizations for finite lattices via correspondence theory for monotone modal logic.  Journal of Logic and Computation, forthcoming.
  * [13] M. Gehrke and J. Harding.  Bounded lattice expansions.  Journal of Algebra, 238(1):345–371, 2001.
  * [14] S. Ghilardi and G. Meloni.  Constructive canonicity in non-classical logics.  Annals of Pure and Applied Logic, 86(1):1–32, 1997.
  * [15] V. Goranko and D. Vakarelov.  Elementary canonical formulae: Extending Sahlqvist’s theorem.  Annals of Pure and Applied Logic, 141(1-2):180–217, 2006.
  * [16] G. Greco, M. Ma, A. Palmigiano, A. Tzimoulis, and Z. Zhao.  Unified correspondence as a proof-theoretic tool.  Journal of Logic and Computation, forthcoming.
  * [17] B. Jónsson.  On the canonicity of Sahlqvist identities.  Studia Logica, 53:473–491, 1994.
  * [18] B. Jónsson and A. Tarski.  Boolean algebras with operators.  American Journal of Mathematics, 74:127–162, 1952.
  * [19] M. Ma and Z. Zhao.  Unified correspondence and proof theory for strict implication.  Journal of Logic and Computation, forthcoming.
  * [20] W. Morton.  Canonical extensions of posets.  Algebra universalis, 72(2):167–200, 2014.
  * [21] A. Palmigiano, S. Sourabh, and Z. Zhao.  Jónsson-style canonicity for ALBA-inequalities.  Journal of Logic and Computation, 2015.
  * [22] A. Palmigiano, S. Sourabh, and Z. Zhao.  Sahlqvist theory for impossible worlds.  Journal of Logic and Computation, forthcoming.
  * [23] G. Sambin and V. Vaccaro.  A new proof of Sahlqvist’s theorem on modal definability and completeness.  Journal of Symbolic Logic, 54(3):992–999, 1989.
  * [24] T. Suzuki.  Canonicity results of substructural and lattice-based logics.  The Review of Symbolic Logic, 4(01):1–42, 2011.
  * [25] T. Suzuki.  A Sahlqvist theorem for substructural logic.  The Review of Symbolic Logic, 6(02):229–253, 2013.