# What an event is not: unravelling the identity of events in
quantum theory and gravity

Anne-Catherine de la Hamette∗,† University of Vienna, Faculty of Physics,
Vienna Doctoral School in Physics, and Vienna Center for Quantum Science and
Technology (VCQ), Boltzmanngasse 5, A-1090 Vienna, Austria  Institute for
Quantum Optics and Quantum Information (IQOQI), Austrian Academy of Sciences,
Boltzmanngasse 3, A-1090 Vienna, Austria  Viktoria Kabel† University of
Vienna, Faculty of Physics, Vienna Doctoral School in Physics, and Vienna
Center for Quantum Science and Technology (VCQ), Boltzmanngasse 5, A-1090
Vienna, Austria  Institute for Quantum Optics and Quantum Information (IQOQI),
Austrian Academy of Sciences, Boltzmanngasse 3, A-1090 Vienna, Austria  Časlav
Brukner  University of Vienna, Faculty of Physics, Vienna Doctoral School in
Physics, and Vienna Center for Quantum Science and Technology (VCQ),
Boltzmanngasse 5, A-1090 Vienna, Austria  Institute for Quantum Optics and
Quantum Information (IQOQI), Austrian Academy of Sciences, Boltzmanngasse 3,
A-1090 Vienna, Austria

###### Abstract

We explore the notion of events at the intersection between quantum physics
and gravity, inspired by recent research on superpositions of semiclassical
spacetimes. By going through various experiments and thought experiments –
from a decaying atom, to the double-slit experiment, to the quantum switch –
we analyse which properties can and cannot be used to define events in such
non-classical contexts. Our findings suggest an operational, context-dependent
definition of events which emphasises that their properties can be accessed
without destroying or altering observed phenomena. We discuss the implications
of this understanding of events for indefinite causal order as well as the
non-absoluteness of events in the Wigner’s friend thought experiment. These
findings provide a first step for developing a notion of event in quantum
spacetime.

Essay written for the Gravity Research Foundation 2024 Awards for Essays on
Gravitation.

Submitted on March 29th, 2024

∗Corresponding author<EMAIL_ADDRESS>
†These authors contributed equally to this work.

Imagine yourself as an experimentalist capable of achieving the seemingly
unachievable: placing planet Earth in a superposition of locations. Arguably,
such a mass configuration would source a gravitational field in quantum
superposition [1, 2, 3]. More concretely, we would expect to find the
gravitational field in a quantum state peaked around two semiclassical
configurations, one in which the source is on the left and one in which it is
on the right [4]. Now picture this: amidst this extraordinary experimental
setup, two satellites orbit in opposite directions. Due to a slight oversight
in orbital calibration, they collide, unleashing a catastrophic explosion. How
would one describe such an event in a non-classical spacetime? Does it happen
at the same spacetime point across the superposition, i.e. is it localised in
spacetime or not? This question is complicated by the fact that general
relativity is a diffeomorphism-invariant theory. In this context, what we mean
by an event happening at the same or different points across the superposition
becomes ambiguous. We cannot simply “label” the spacetime points in each of
the branches and identify those that carry the same label. The diffeomorphism
invariance of general relativity provides us with the freedom to arbitrarily
reshuffle the labels, that is, allows to freely change the coordinates. In
particular, invoking the linearity of quantum theory, there is nothing that
prevents us from changing the coordinates independently in each branch of the
superposition. As a result, how we identify spacetime points across the
branches of the superposition – what constitutes the same or different
spacetime points – is not absolute [5]. Thus, asking whether an event happens
at the same or different spacetime points across the superposition does not
have an absolute answer. However, if the localisation of the event does not
provide an inherent description thereof, this prompts the question which
properties do. Is it still the same event if the satellites collide at
different velocities in the two branches? What if the collision occurs at a
different angle or with a different force? What if the impact in one branch is
not sufficient to cause an explosion in the first place? Is it, then, still
the same event?

In this essay, we thus want to examine the question of which properties enable
us to identify or distinguish events across different branches of a
superposition. In fact, as we will see below, this question not only arises in
the context of non-classical spacetimes but also of simpler quantum mechanical
scenarios. Formulating a good notion of “event” is also a pertinent question
in the literature on quantum foundations, as it plays an important role in
several interpretations of quantum mechanics [6, 7, 8, 9, 10], in the field of
causal inference and, by extension to the quantum realm, in quantum causal
models [11, 12, 13] and indefinite causal order [14, 15, 16, 17, 18, 19, 20,
21, 22]. What constitutes an event varies significantly across these different
contexts. Given the multitude of different notions of event, we will not fix a
definition at this point but rather consider several examples illustrating
what an event is _not_. Specifically, we will identify which properties may be
problematic for defining an event in the quantum context. In the course of
this, we will encounter a varied spectrum of events, ranging from the
application of an operation, to a coincidence of systems, to a change in a
system’s state. By collecting and combining these clues, we will eventually
arrive at an _operational_ notion of event that depends on the measurement
context and the phenomenon under consideration. More concretely, we will argue
that a property of an event should be regarded as inherent if one can, in
principle and without destroying the phenomenon of interest, access the
property.

The Atomic Decay Clue. As a first example, consider the decay of an atom of,
for example, Radium-226. This event is characterised by the transition from
Radium-226 to Radon-222 and an alpha-particle. If the system is isolated and
no observation takes place, it will end up in a superposition of having and
not having decayed. Consequently, there is a non-vanishing probability for the
decay to occur at any given time $t_{\ast}$. Should we therefore say that
there is an event “the atom decays at time $t_{\ast}$”? While one might first
be inclined to do so, this clashes with the intuition that the atom really
only decays once – albeit, in a superposition of different times. This can be
taken as evidence that a more sensible notion of event in this context is
given by “the atom decays (at some point in time)”. We can make this intuition
more precise. While “the atom decays at time $t_{\ast}$” can become an actual
event if measured by detecting the decay at time $t_{\ast}$ (e.g. by
activating a Geiger counter at time $t_{\ast}$), in this case the superposed
state is replaced by an updated state after the measurement. If, on the other
hand, we wanted to verify that the state of the atom is indeed in a
superposition of different decay times (e.g. through an appropriate
interference experiment), operationally accessing the decay time would disturb
the verification procedure. This leads to a coarse-grained notion of event
that captures the common features of the fine-grained events “the atom decays
at time $t_{\ast}$” while excluding any reference to non-accessible
attributes. Moreover, unlike “the atom decays at time $t_{\ast}$”, this notion
does not lead to an infinite number of events and thus, if included in the
theory’s ontology, avoids an excessive number of distinct entities. Overall,
we see that, depending on the measurement context, the time at which the event
happens may or may not be regarded as a defining property thereof.

The Double-Slit Clue. As a second example, let us return to the beginning of
everyone’s first lecture on quantum mechanics and consider the double-slit
experiment [23]: A coherent beam of photons impinges on two closely spaced
slits, producing an interference pattern on a screen placed behind the slits.
The behaviour of a single photon is commonly described as passing through the
double-slit in a superposition of the left and the right slit, respectively.
Does this passing through the slits constitute one single or two separate
events? Again, we are faced with the question of which properties to take into
account when discriminating between the events. Clearly, most people would
agree that the photon only passes through the double-slit once – albeit in a
superposition of spatial trajectories. Nevertheless, it also seems valid to
consider the event “the photon passes through the left slit” as distinct from
the event “the photon passes through the right slit”. However, from an
operational perspective, such an assertion seems problematic, for it would
require being able to _measure_ , in principle, which of the two slits the
photon actually passes through. Any such measurement would inevitably disrupt
the interference pattern. Thus, it is impossible to operationally distinguish
the above two events without destroying the phenomenon we were studying in the
first place. If instead one characterises the event simply as “the photon
passes through the double-slit”, without specifying _which_ slit it went
through, the relevant properties of this coarse-grained event _can_ be
accessed operationally without altering the interference pattern. For example,
one could perform a delayed-choice quantum-erasure experiment [24] to confirm
that the photon went through _a_ slit, without any effects on the pattern
created on the screen.

In keeping with the theme of this essay, let us now consider, as a
gedankenexperiment, a gravitational analogue of the original double-slit
setup. Take a massive object in superposition of two locations, left and
right, and let a photon travel freely in the vicinity of this gravitational
source. Taking the gravitational field to be in superposition, the photon’s
trajectory will branch into two paths, entangled with the location of the
massive object [2]. At its closest distance to the respective location of the
massive object, the photon is reflected by a mirror and sent on two converging
paths. If we now recombine the mass adiabatically, we can interfere the photon
with itself and produce an interference pattern, as in the optical analogue.
Let us take a closer look at the event “the photon impinges on a mirror”.
Since there are two different mirrors, it may seem natural to define two
distinct events, “the photon impinges on the left mirror” and “the photon
impinges on the right mirror”, in superposition. However, just as in the
optical case, we have no operational access to the property distinguishing the
two events. If we could measure _which_ mirror the photon hits, at least in
principle, we would acquire which-path information and thus destroy the
interference. What we can measure is _that_ the photon hits one of the
mirrors, i.e. the event “the photon impinges on a mirror”. Thus, from an
operational standpoint and within the context of this phenomenon, it seems
that the latter notion of event is more appropriate. One can, of course,
uphold the position that nevertheless the events are distinct. However, one
then has to accept that they are distinguished by a property that cannot be
operationally accessed within the context of the phenomenon that is being
described. If the latter is changed, other measurements may become available,
turning the spatial location into an operationally accessible quantity.
Overall, we conclude that the spatial location at which the event happens may
or may not be regarded as a defining property thereof, depending on the
context.

The Quantum Switch Clue. As a third clue, we want to consider an example from
the intersection of quantum foundations with quantum gravity: indefinite
causal order (ICO) [14, 15, 16, 17, 18, 19, 20, 21, 22]. This provides a broad
playground to discuss several potential properties of events and is, in fact,
an instance for which the definition of event has important repercussions.

The study of ICO is too broad a field to fully do it justice in the context of
this essay. In the following, we thus restrict to a few concrete
implementations of ICO, most notably the quantum switch in its various
incarnations [15, 16, 25, 26]. In its simplest form, the quantum switch
represents a process in which two operations, $A$ and $B$, are implemented in
a quantum-controlled superposition of orders, i.e. $A$ before $B$ and $B$
before $A$. In general, $A$ and $B$ are arbitrary completely positive maps
(the most general quantum operations); we can, however, for the purpose of
this essay, think of them as unitaries applied to a target system. Given an
initial state
$\ket{\psi_{0}}=\frac{1}{\sqrt{2}}(\ket{0}_{C}+\ket{1}_{C})\ket{\phi}_{T}$ of
the control and target system, the quantum switch realises the state
$\ket{\psi_{1}}=\frac{1}{\sqrt{2}}(\ket{0}_{C}\otimes
BA\ket{\phi}_{T}+\ket{1}_{C}\otimes AB\ket{\phi}_{T})$. There are various
different implementations of the quantum switch. The first is known as the
_optical quantum switch_ and has been realised in the laboratory [25, 27, 28].
It involves a photon as the target system travelling in a superposition of
paths through two different waveplates, thereby implementing the unitaries $A$
and $B$ respectively in a quantum-controlled superposition of orders. At the
end, the photon paths are re-interfered and ICO is verified using a causal
witness measurement. Fig. 1(a) represents this protocol in a spacetime
diagram. One can clearly see that operation $A$ is applied to the target
system in a delocalised manner, that is, in a superposition of two different
spacetime points. The same considerations apply to operation $B$. One may be
tempted to say that there are four different events involved in the optical
quantum switch. Importantly, these events are characterised by the application
of a specific operation at a specific _spacetime point_. While this view of
the optical switch as a 4-event process is endorsed by a number of people [18,
19, 21], others believe that there are only two events involved in this
process [29, 25, 17, 22].111The two sides of this debate are referred to as
the “spatio-temporalist” vs. the “dynamicist” in [21]. Vilasini and Renner
also analyse this debate, distinguishing between more coarse-grained and fine-
grained processes [20]. In this coarse-grained view, events are characterised
solely by the application of operation $A$ and $B$, respectively. While they
happen in a (space)time-delocalised manner, each operation is still only being
applied once – just as the particle in the double-slit experiment only passes
through the double-slit once. Moreover, just as one cannot access which slit
the photon passes through, one cannot operationally access the spacetime
locations of the events without destroying the interference, supporting the
coarse-grained view.222While protocols to measure the number of spacetime
points in various implementations of the quantum switch have been proposed
[18], we believe that the measurement procedure that does not decohere the
target system and destroy the interference pattern of the quantum switch
cannot distinguish between two and four involved spacetime points with
certainty in a single run of the experiment, as is claimed in [18]. In
particular, one can construct scenarios that involve four spacetime points for
which their protocol would imply two, and vice-versa, in some runs of the
experiment. In short, the spacetime location is not regarded as a relevant
property of the event in this view. The question of how many events occur in
the quantum switch has concrete consequences for the significance of the
performed experiments. This is because causal relations are defined _between
events_. If there are four different events, the optical quantum switch merely
implements a superposition of two definite causal orders between different
pairs of events. If, on the other hand, there are only two events $E_{A}$ and
$E_{B}$, we truly have a superposition of opposite causal orders, i.e. $E_{A}$
before $E_{B}$ and $E_{B}$ before $E_{A}$. As a result, there is an ongoing
debate as to whether the optical quantum switch is merely a simulation or a
true realisation of ICO.

(a) Optical quantum switch.

(b) Gravitational quantum switch.

Figure 1: _Different implementations of the quantum switch._ Alice and Bob
apply their operations $A$ and $B$ in a superposition of causal orders. While
the indefinite causal order in (a) is created through a superposition of paths
(represented by the orange, solid and the grey, dotted lines respectively), in
(b) it is due to a superposition of the gravitational source (saturated vs.
grey Earth), which leads to a superposition of lightcones (orange, solid vs.
grey, dotted lines respectively).

There is less disagreement when it comes to the _gravitational_ quantum switch
[26]. In this thought experiment, a large massive object is coherently
superposed in two locations, sourcing a gravitational field in a superposition
of two semiclassical configurations. A probe particle is sent along a
superposition of trajectories, entangled with the position of the massive
object, and passes through the laboratories of two agents, Alice and Bob, once
in each of the branches. The setup is tuned such that, upon the passing of the
target system through each laboratory, the local clock of the respective agent
shows proper time $\tau_{\ast}$. Alice and Bob each apply their operations,
$A$ and $B$, at precisely this time. Due to the superposition of gravitational
fields, this happens in a superposition of opposite orders, thus implementing
the quantum switch. Since in this setup it can be ensured that the target
system only passes through each laboratory once, it is generally agreed that
each operation is only applied once, resulting in a total of two events.
Therefore, the gravitational switch is widely considered a proper realisation
of ICO. When depicted in a spacetime diagram (see Fig. 1(b)), however, one
might come to believe that this scenario involves four events, after all. This
is because, relative to the time measured by a far away observer, the probe
passes through Alice’s and Bob’s laboratory in a superposition of spacetime
coordinates. The assertion that there are four events not only contradicts the
previously established intuition that the system really passes through each
laboratory only once, but is problematic for a deeper reason. Since general
relativity is a diffeomorphism-invariant theory, coordinate positions do not
have intrinsic meaning of their own. By applying different coordinate
transformations in each of the branches, one can always localise Alice’s
application of her operation to a single spacetime point, and similarly for
Bob. In other words, one can always map a 4-point switch into a 2-point switch
by choosing an appropriate quantum coordinate system [22]. In addition to the
previous discussion on being able to operationally access any inherent
property without destroying the phenomenon of interest, we take this to imply
that the spacetime location of an event should not be regarded as an inherent
property, since it is a frame-dependent quantity [5].

Lastly, let us consider a variation of the quantum switch, which introduces a
further dimension to our discourse on the notion of event. Returning to the
abstract setup of the switch with two time-delocalised operations $A$ and $B$,
let us now assume that, in the first branch, an operation $A_{1}$ is applied
at time $t_{1}$ while a different operation $A_{2}$ is applied at time $t_{2}$
in the second (and similarly for $B_{1}$ and $B_{2}$). This modification of
the switch is referred to as the _routed quantum switch_ in [21]. It presents
us with a more intricate problem: can we still consider the event of applying
operation $A_{1}$ as the same event as applying operation $A_{2}$? At first
sight, answering this in the affirmative would seem like we have gone too far.
For surely, the application of two different operations must instantiate two
different events! Yet, if we take seriously the operational account of an
event that has become clear throughout the previous examples, this intuition
must be questioned. Just as we cannot access the time at which the atom
decayed, or which slit the photon passed through, or the exact spacetime
location of the events in the quantum switch, without destroying the
phenomenon of interest, the same limitation applies to the routed quantum
switch. Here, accessing which operation is applied to the photon would also
destroy the interference. This is true even if it can be operationally
verified that operation $A_{1}$ is applied at time $t_{1}$ and operation
$A_{2}$ at time $t_{2}$. Since the photon arrives in a superposition of times
in each lab, there is nevertheless no information about which of the two
operations will be applied to the photon. This raises the critical question:
How do we then define an event within the context of the routed switch? Having
removed this much structure from the notion of event, what remains? The answer
depends on the concrete implementation of the experiment and which attributes
are still accessible. In an optical implementation, the unitary operations can
be applied to the target system using waveplates, which are now taken to
rotate in time such that different unitaries are applied at different times.
In this case, an event would be characterised by the photon passing through
one particular waveplate, say, that marked with label $A$ or $B$, but not by
the specific rotation angle of the plate. In a more interventionist picture,
it would be the fact that a particular agent applies _an_ operation on the
photon or that an operation is applied to the photon in a particular
laboratory. Importantly, whatever property we use to define and distinguish
events in the quantum context, we should ensure that we can, in principle,
access this property without destroying the phenomenon of interest.

Putting the Pieces Together. This also applies to the context of spacetimes in
superposition, which we considered in the introduction. Recall the thought
experiment in which two satellites collide in a superposition of spacetimes.
Suppose we want to conduct an interference experiment to experimentally verify
the superposition. By making use of our extensive experimental capabilities,
we re-interfere both the gravitational source and the satellite remnants and
measure the resulting interference pattern. Let us now revisit the questions
raised at the start: is it still the _same_ event if the satellites collide at
different spacetime points, velocities, angles, or forces? As for the
spacetime location of the events, we have seen that it is a frame-dependent
property and thus depends on the quantum coordinate system that one chooses
[5]. Moreover, just like measuring other potential properties such as
velocity, angle, or force that differ across the branches of the
superposition, measuring explicitly the concrete spacetime location of the
event (relative to the experimental setup) would allow us to acquire which-
path information and thus destroy the interference pattern. Even when there is
an explosion in one branch and a non-explosive crash in the other, we have to
take seriously that we cannot operationally access the property of whether the
explosion occurred without disturbing the recombination process.333Of course,
given our current technological capabilities, decoherence would be
unavoidable; however, if one maintains complete control over all emitted
photons to prevent any which-branch information from being transmitted to the
environment, e.g. by enclosing the setup within a sealed capsule, one could,
in principle, recombine the branches. Like the claim that the application of
two different operations $A_{1}$ and $A_{2}$ might, in some contexts,
constitute the _same_ event, this might seem like a radical statement.
However, we believe that it is a direct consequence of our operational
understanding of the notion of event.

More precisely, the ingredients that went into our construction of this notion
are the following: (i) the _Identity of Indiscernibles_ [30, 31], that is, the
metaphysical stance that if two entities are identical in all of their
properties, one should not treat them as distinct, and (ii) what one might
call _$\text{Operationalism}^{+}$_. By operationalism, we mean the positivist
stance that one should limit oneself to only those properties that are
operationally accessible, that is, can be measured or observed.
$\text{Operationalism}^{+}$ takes this further by restricting to quantities
that can be measured or observed, in principle, _within the given context_ ,
that is, without disrupting the phenomenon of interest.444While the term
$\text{Operationalism}^{+}$ is not used in any of these works, we believe that
this stance is similar to the views purported in [32, 33, 34, 35, 36]. While
this might seem like a drastic extension of operationalism, we believe it
naturally takes into account that, in quantum theory, a measurement is a
constitutive element of the phenomenon under investigation – measuring one
observable can affect the empirical outcomes of the measurement of another.

The resulting notion of event has pertinent implications for several areas in
the foundations of physics. Firstly, events play an important role as the
relata of causal relations [37, 38] and thereby in the fields of (quantum)
causality and (quantum) causal modelling [11, 12, 13]. Thus, what we consider
to be distinct events has significant implications for the causal structure.
This became evident in the context of the quantum switch, where one obtains
definite or indefinite causal order, depending on how one identifies the
different causal relata across the superposition [21]. We have seen that the
spacetime location of an event cannot be accessed operationally, since any
such measurement would decohere the quantum switch. Thus, we conclude that the
location in spacetime is not to be regarded as an inherent property and the
quantum switch represents a two-event process with ICO. More generally, as
shown in [20], causal processes have more or less fine-grained
representations. The stance of the authors of [20] can be read as follows: if
a fine-grained representation with acyclic causal structure (i.e. a causal
order that can be represented by a graph without loops) on a fixed spacetime
background exists, then this process ought to be characterised as having
definite causal order. In this essay, we have challenged this view by arguing
that there are setups in which a coarse-grained notion of event is more
appropriate than its fine-grained counterpart, on the grounds that it is
impossible to operationally access the fine-grained structure.

Moreover, understanding in detail what an event is, and is not, may help to
clarify the conceptual underpinnings of relational quantum mechanics (RQM),
for it relies heavily, at least in some formulations [39, 40, 41], on the
notion of an interaction or event. In particular, the ontology suggested in
recent work is a “flash-ontology” of sparse events or interactions [42, 40,
41]. However, the notion of an event as arising in an interaction whereby the
systems involved take on definite values relative to one another [41, p. 11]
is quite vague. We thus believe that our analysis of which properties can or
cannot constitute an event can help to fill this gap. While RQM, where events
are a fundamental part of the ontology, is a particularly relevant arena for
the notion of event, it also plays a considerable role in other
interpretations. For instance, the question of what constitutes different
events is intimately connected with the notion of branching in many-worlds
interpretations [43]. In particular, depending on “when” or under what
conditions branching occurs, one would presumably end up with different
numbers of distinct events. It would be an interesting question for future
research to investigate whether there is a reading of the many-worlds
interpretation that is consistent with our operationalist notion. Finally, our
understanding of event appears to align well with recent works on the non-
absoluteness of events in quantum theory [7, 8, 44, 45, 46, 47, 48, 9, 10].
There, events are not absolute but relative to observers, measurement
contexts, (sets of) systems, or “bubbles”. While we believe that there is an
overlap in spirit between our understanding of events and these works, they
differ in what concretely events are relative to.

The observation that events (or measurement outcomes or facts) are relative to
certain “bubbles” also plays an important role in attempts to resolve the
Wigner’s friend paradox and extensions thereof [49, 8, 50, 51, 52]. With our
notion of events, we can make this observation more tangible. Since the latter
depends on the measurement context, Wigner and the Friend should give
different accounts of the events involved in this scenario. Relative to the
Friend’s measurement setup, who measures a spin-$1/2$ particle in her
laboratory, it is possible to access her outcome – the spin of the particle –
without disturbing her setup. Relative to Wigner’s setup, in which the
coherence of the entangled state of the Friend and the system is verified
through an interference experiment, however, the property of the spin-$1/2$
particle is not accessible. This is because measuring the spin would disturb
the interference pattern. Thus, the spin of the particle should not be taken
as an inherent property of the event, relative to Wigner’s experimental
context or “bubble”. For Wigner, it is not meaningful to differentiate between
events “spin up” or “spin down” – just as it is not meaningful to talk about
“the photon passing through the right slit” and “the photon passing through
the left slit” in the double-slit experiment. Similarly, it is meaningless for
Wigner to distinguish between “Friend sees spin up” and “Friend sees spin
down”. An event that _can_ be defined meaningfully relative to Wigner’s setup
is that the Friend has measured _an_ event. This has been shown to be possible
without decohering the entangled state of Friend and the system [53] and thus
without destroying Wigner’s experimental setup.

We thus see that questioning the conceptual foundations of basic notions, such
as what constitutes an event, helps to shed light on various problems in the
foundations of physics and suggests ways to understand the underlying
ontology. We believe that this is particularly true in the context of quantum
gravity, where, as we have seen, the simple notion of an event as a spacetime
point, will no longer be sufficient. In fact, even when considering events as
physical coincidences, such as the crash of the satellites, there remains an
ambiguity in which properties are inherent to the event. Thus, it is important
to clarify how this notion can be generalised to non-classical spacetimes,
such as semiclassical spacetimes in superposition. This is particularly true
if one takes events to be part of the fundamental ontology of the theory.
However, we believe that even an understanding of an effective notion of
events that can be applied to these regimes delivers foundational insights
that can guide us in the construction of a future theory of quantum gravity.

###### Acknowledgements.

We thank Jeremy Butterfield for inspiring discussions and for his comments on
an earlier version of this essay. We also thank Nikola Paunkovic, V. Vilasini,
and Marko Vojinovic for helpful discussions. This research was funded in whole
or in part by the Austrian Science Fund (FWF) [10.55776/F71] and
[10.55776/COE1]. For open access purposes, the author has applied a CC BY
public copyright license to any author accepted manuscript version arising
from this submission. Funded by the European Union - NextGenerationEU. VK
acknowledges support through a DOC Fellowship of the Austrian Academy of
Sciences. This publication was made possible through the financial support of
the ID 62312 grant from the John Templeton Foundation, as part of The Quantum
Information Structure of Spacetime (QISS) Project (qiss.fr). The opinions
expressed in this publication are those of the authors and do not necessarily
reflect the views of the John Templeton Foundation.

## References

  * DeWitt and Rickles [2011] C. M. DeWitt and D. Rickles,  _The Role of Gravitation in Physics: Report from the 1957 Chapel Hill Conference_, ASTIA document (Ed. Open Access, 2011).
  * de la Hamette  _et al._ [2023] A.-C. de la Hamette, V. Kabel, E. Castro-Ruiz, and Č. Brukner, Quantum reference frames for an indefinite metric, Communications Physics 6, 231 (2023).
  * Aspelmeyer [2022] M. Aspelmeyer, When Zeh Meets Feynman: How to Avoid the Appearance of a Classical World in Gravity Experiments, in  _From Quantum to Classical: Essays in Honour of H.-Dieter Zeh_, edited by C. Kiefer (Springer International Publishing, Cham, 2022) pp. 85–95.
  * Christodoulou and Rovelli [2019] M. Christodoulou and C. Rovelli, On the possibility of laboratory evidence for quantum superposition of geometries, Physics Letters B 792, 64–68 (2019).
  * Kabel  _et al._ [2024] V. Kabel, A.-C. de la Hamette, L. Apadula, C. Cepollaro, H. Gomes, J. Butterfield, and Č. Brukner, Identification is pointless: Quantum reference frames, localisation of events, and the quantum hole argument (2024), arXiv:2402.10267 [quant-ph] .
  * Wallace [2012] D. Wallace,  _The Emergent Multiverse: Quantum Theory According to the Everett Interpretation_ (Oxford University Press, Oxford, GB, 2012).
  * Fuchs and Schack [2013] C. A. Fuchs and R. Schack, Quantum-Bayesian coherence, Rev. Mod. Phys. 85, 1693–1715 (2013).
  * Brukner [2017] Č. Brukner, On the quantum measurement problem, in  _Quantum [Un]Speakables II: Half a Century of Bell’s Theorem_, edited by R. Bertlmann and A. Zeilinger (Springer International Publishing, Cham, 2017) pp. 95–117.
  * Adlam [2024] E. Adlam, What Does ‘(Non)-absoluteness of Observed Events’ Mean?, Foundations of Physics 54, 13 (2024).
  * Ormrod and Barrett [2024] N. Ormrod and J. Barrett, Quantum influences and event relativity (2024), arXiv:2401.18005 [quant-ph] .
  * Costa and Shrapnel [2016] F. Costa and S. Shrapnel, Quantum causal modelling, New Journal of Physics 18, 063032 (2016).
  * Barrett  _et al._ [2020] J. Barrett, R. Lorenz, and O. Oreshkov, Quantum causal models (2020), arXiv:1906.10726 [quant-ph] .
  * Barrett  _et al._ [2021] J. Barrett, R. Lorenz, and O. Oreshkov, Cyclic quantum causal models, Nature Communications 12, 885 (2021).
  * Hardy [2005] L. Hardy, Probability theories with dynamic causal structure: A new framework for quantum gravity (2005), arXiv:gr-qc/0509120 [gr-qc] .
  * Chiribella  _et al._ [2013] G. Chiribella, G. M. D’Ariano, P. Perinotti, and B. Valiron, Quantum computations without definite causal structure, Phys. Rev. A 88, 022318 (2013).
  * Oreshkov  _et al._ [2012] O. Oreshkov, F. Costa, and Č. Brukner, Quantum correlations with no causal order, Nature Communications 3, 1092 (2012).
  * Oreshkov [2019] O. Oreshkov, Time-delocalized quantum subsystems and operations: on the existence of processes with indefinite causal structure in quantum mechanics, Quantum 3, 206 (2019).
  * Paunković and Vojinović [2020] N. Paunković and M. Vojinović, Causal orders, quantum circuits and spacetime: distinguishing between definite and superposed causal orders, Quantum 4, 275 (2020).
  * Faleiro  _et al._ [2023] R. Faleiro, N. Paunkovic, and M. Vojinovic, Operational interpretation of the vacuum and process matrices for identical particles, Quantum 7, 986 (2023).
  * Vilasini and Renner [2022] V. Vilasini and R. Renner, Embedding cyclic causal structures in acyclic spacetimes: no-go results for process matrices (2022), arXiv:2203.11245 [quant-ph] .
  * Ormrod  _et al._ [2023] N. Ormrod, A. Vanrietvelde, and J. Barrett, Causal structure in the presence of sectorial constraints, with application to the quantum switch, Quantum 7, 1028 (2023).
  * de la Hamette  _et al._ [2022] A.-C. de la Hamette, V. Kabel, M. Christodoulou, and Č. Brukner, Quantum diffeomorphisms cannot make indefinite causal order definite (2022), arXiv:2211.15685 [quant-ph] .
  * Feynman  _et al._ [2011] R. Feynman, R. Leighton, and M. Sands,  _The Feynman Lectures on Physics, Vol. III: The New Millennium Edition: Quantum Mechanics_, The Feynman Lectures on Physics (Basic Books, 2011).
  * Kim  _et al._ [2000] Y.-H. Kim, R. Yu, S. P. Kulik, Y. Shih, and M. O. Scully, Delayed “choice” quantum eraser, Phys. Rev. Lett. 84, 1–5 (2000).
  * Procopio  _et al._ [2015] L. M. Procopio, A. Moqanaki, M. Araújo, F. Costa, I. Alonso Calafell, E. G. Dowd, D. R. Hamel, L. A. Rozema, Č. Brukner, and P. Walther, Experimental superposition of orders of quantum gates, Nature Communications 6, 7913 (2015).
  * Zych  _et al._ [2019] M. Zych, F. Costa, I. Pikovski, and Č. Brukner, Bell’s theorem for temporal order, Nature Communications 10, 3772 (2019).
  * Rubino  _et al._ [2017] G. Rubino, L. A. Rozema, A. Feix, M. Araújo, J. M. Zeuner, L. M. Procopio, Č. Brukner, and P. Walther, Experimental verification of an indefinite causal order, Science Advances 3, e1602589 (2017).
  * Goswami  _et al._ [2018] K. Goswami, C. Giarmatzi, M. Kewming, F. Costa, C. Branciard, J. Romero, and A. G. White, Indefinite causal order in a quantum switch, Phys. Rev. Lett. 121, 090503 (2018).
  * Araújo  _et al._ [2014] M. Araújo, F. Costa, and Č. Brukner, Computational advantage from quantum-controlled ordering of gates, Phys. Rev. Lett. 113, 250402 (2014).
  * Leibniz and Loemker [1956] G. W. Leibniz and L. E. Loemker,  _Philosophical Papers and Letters_ , edited by L. E. Loemker (University of Chicago Press, Chicago,, 1956).
  * Forrest [2020] P. Forrest, The Identity of Indiscernibles, in  _The Stanford Encyclopedia of Philosophy_ , edited by E. N. Zalta (Metaphysics Research Lab, Stanford University, 2020) Winter 2020 ed.
  * Bohr [1931] N. Bohr,  _Atomtheorie und Naturbeschreibung_ (Springer Berlin, Heidelberg, 1931).
  * Bub [2017] J. Bub, Why Bohr was (Mostly) Right (2017), arXiv:1711.01604 [quant-ph] .
  * Healey [2017] R. Healey,  _The Quantum Revolution in Philosophy_ (Oxford University Press, 2017).
  * Janas  _et al._ [2022] M. Janas, M. E. Cuffaro, and M. Janssen,  _Understanding quantum raffles: quantum mechanics on an informational approach : structure and interpretation_ (Springer Cham, Switzerland, 2022).
  * Cuffaro [2023] M. E. Cuffaro, The Measurement Problem Is a Feature, Not a Bug–Schematising the Observer and the Concept of an Open System on an Informational, or (Neo-)Bohrian, Approach, Entropy 25, 10.3390/e25101410 (2023).
  * Lewis [1986] D. Lewis,  _On the Plurality of Worlds_ (Wiley-Blackwell, Malden, Mass., 1986).
  * Pearl [2009] J. Pearl,  _Causality_ , 2nd ed. (Cambridge University Press, 2009).
  * Rovelli [1996] C. Rovelli, Relational quantum mechanics, International Journal of Theoretical Physics 35, 1637–1678 (1996).
  * Laudisa and Rovelli [2021] F. Laudisa and C. Rovelli, Relational Quantum Mechanics, in  _The Stanford Encyclopedia of Philosophy_ , edited by E. N. Zalta (Metaphysics Research Lab, Stanford University, 2021) Winter 2021 ed.
  * Adlam and Rovelli [2023] E. Adlam and C. Rovelli, Information is physical: Cross-perspective links in relational quantum mechanics, Philosophy of Physics 1, 10.31389/pop.8 (2023).
  * Rovelli [2018] C. Rovelli, Space is blue and birds fly through it, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 376, 20170312 (2018), https://royalsocietypublishing.org/doi/pdf/10.1098/rsta.2017.0312 .
  * Vaidman [2021] L. Vaidman, Many-Worlds Interpretation of Quantum Mechanics, in  _The Stanford Encyclopedia of Philosophy_ , edited by E. N. Zalta (Metaphysics Research Lab, Stanford University, 2021) Fall 2021 ed.
  * Bong  _et al._ [2020] K.-W. Bong, A. Utreras-Alarcón, F. Ghafari, Y.-C. Liang, N. Tischler, E. G. Cavalcanti, G. J. Pryde, and H. M. Wiseman, A strong no-go theorem on the Wigner’s friend paradox, Nature Physics 16, 1199–1205 (2020).
  * Di Biagio and Rovelli [2021] A. Di Biagio and C. Rovelli, Stable Facts, Relative Fact, Foundations of Physics 51, 30 (2021).
  * Moreno  _et al._ [2022] G. Moreno, R. Nery, C. Duarte, and R. Chaves, Events in quantum mechanics are maximally non-absolute, Quantum 6, 785 (2022).
  * Brukner [2022] Č. Brukner, Wigner’s friend and relational objectivity, Nature Reviews Physics 4, 628–630 (2022).
  * Ormrod and Barrett [2022] N. Ormrod and J. Barrett, A no-go theorem for absolute observed events without inequalities or modal logic (2022), arXiv:2209.03940 [quant-ph] .
  * Wigner [1995] E. P. Wigner, Remarks on the mind-body question, in  _Philosophical Reflections and Syntheses_, edited by J. Mehra (Springer Berlin Heidelberg, Berlin, Heidelberg, 1995) pp. 247–260.
  * Brukner [2018] Č. Brukner, A no-go theorem for observer-independent facts, Entropy 20, 350 (2018).
  * Frauchiger and Renner [2018] D. Frauchiger and R. Renner, Quantum theory cannot consistently describe the use of itself, Nature Communications 9, 3711 (2018).
  * Cavalcanti [2021] E. G. Cavalcanti, The View from a Wigner Bubble, Foundations of Physics 51, 39 (2021).
  * Deutsch [1985] D. Deutsch, Quantum theory as a universal physical theory, International Journal of Theoretical Physics 24, 1–41 (1985).


# More Photos are All You Need: Semi-Supervised Learning for
Fine-Grained Sketch Based Image Retrieval

Ayan Kumar Bhunia1 Pinaki Nath Chowdhury1,2 Aneeshan Sain1,2 Yongxin Yang1,2
Tao Xiang1,2 Yi-Zhe Song1,2
1 SketchX  CVSSP  University of Surrey  United Kingdom
2 iFlyTek-Surrey Joint Research Centre on Artificial Intelligence.
{a.bhunia, p.chowdhury, a.sain, yongxin.yang, t.xiang<EMAIL_ADDRESS>
###### Abstract

A fundamental challenge faced by existing Fine-Grained Sketch-Based Image
Retrieval (FG-SBIR) models is the data scarcity – model performances are
largely bottlenecked by the lack of sketch-photo pairs. Whilst the number of
photos can be easily scaled, each corresponding sketch still needs to be
individually produced. In this paper, we aim to mitigate such an upper-bound
on sketch data, and study whether unlabelled photos alone (of which they are
many) can be cultivated for performances gain. In particular, we introduce a
novel semi-supervised framework for cross-modal retrieval that can
additionally leverage large-scale unlabelled photos to account for data
scarcity. At the centre of our semi-supervision design is a sequential photo-
to-sketch generation model that aims to generate paired sketches for
unlabelled photos. Importantly, we further introduce a discriminator guided
mechanism to guide against unfaithful generation, together with a distillation
loss based regularizer to provide tolerance against noisy training samples.
Last but not least, we treat generation and retrieval as two conjugate
problems, where a joint learning procedure is devised for each module to
mutually benefit from each other. Extensive experiments show that our semi-
supervised model yields significant performance boost over the state-of-the-
art supervised alternatives, as well as existing methods that can exploit
unlabelled photos for FG-SBIR.

##  1 Introduction

With the ever rising popularity of touch screen devices, sketch-based image
retrieval (SBIR) has witnessed significant interest within the vision
community [5, 36, 37, 54, 11, 49]. Despite starting as a category-level
retrieval problem [9, 10, 3, 12], the fine-grained nature of sketches stirred
current research focus more towards fine-grained SBIR (FG-SBIR) [5, 30] –
which aims to retrieve a _particular_ photo based on a query sketch at an
intra-category basis.

Figure 1: Our proposed method additionally leverages large scale photos
without any manually labelled paired sketches to improve FG-SBIR performance.
Moreover, we show that the two conjugate process, _photo-to-sketch_ generation
and _fine-grained SBIR_ , could improve each other by joint training.

Recent FG-SBIR works [52, 28, 5, 30] predominately rely on fully-supervised
triplet loss-based deep networks to yield retrieval performances of practical
value. The underlying assumption is largely inline with the progression of
supervised photo-only models – that one can always (relatively easily) obtain
additional labelled training data to sustain desired performance gains. This
assumption however does not hold for FG-SBIR – sketch-photo pairs can not be
easily scaled as per their photo-only counterparts. That is, instead of
crawling and then labelling photos, the corresponding sketch for any given
photo will need to be separately drawn by hand. As a result, current FG-SBIR
datasets still remain in their thousands (6.7K for QMUL-ShoeV2 [52, 43], and
2K for QMUL-ChairV2 [43]), while photo-datasets [34] are available in
millions. This data scarcity problem has consequently resulted in very recent
attempts that aim at designing generalisable and zero-shot models [28], yet
performances of these models remain far away from fully-supervised
alternatives.

In this paper, we face the music and make the bold assumption that there will
hardly be sufficiently large sketch-photo pairs to train a good model.
Instead, we test the hypothesis that – freely-available unlabelled photo data
would help to mitigate the performance gap imposed by the lack of specifically
collected photo-sketch paired data. Our utmost contribution is therefore a
semi-supervised FG-SBIR framework where unlabelled photo data (i.e., photos
without matching sketches) are used alongside photo-sketch pairs for model
training. We differ significantly to conventional semi-supervised
classification methods [40, 31] – other than learning pseudo photo labels via
a learnable classifier, our “label” for a photo is in the form of a visual
sketch which needs to be generated rather than classified. Thus, at the core
of our design is a sequential photo-to-sketch generation model that outputs
pseudo sketches for unlabelled photos. The hope is therefore that such pseudo
sketch-photo pairs could augment the training of a retrieval model.

Naively cascading a generator with a retrieval model however would not work.
This is mainly because off-the-shelf photo-to-sketch generation models [41, 7]
could sometimes generate unfaithful sketches that may not resemble their
corresponding photos, especially when it comes to fine-grained visual
features. The downstream retrieval model would then have no way of knowing
which pseudo sketch and photo pairs are worth training with, ultimately
resulting in performance degradation. This leads to an important design
consideration of ours – we advocate that there is positive complementarity
between generation and retrieval that can be explored via _joint learning_
(Figure 1). The intuition is simple – pseudo sketches automatically generated
from unlabelled photos can help to semi-supervise a better retrieval model,
and vice versa that better retrieval model can feed back to the generator in
producing more faithful sketch-photo pairs.

The key therefore lies with how such positive exchange cycle can be
facilitated between the generator and retrieval model. To this end, novelty
lies in the components introduced in both generator and retrieval model
designs, and in how they are jointly trained. _First_ , we formulate a novel
sequential photo-to-sketch generator with spatial resolution preservation and
a cross-modal 2D-attention mechanism. _Second_ , a discriminator is formulated
in the retrieval model, to quantify the reliability of generated pseudo photo-
sketch pairs. Reliability scores are then used for instance-wise weighting of
triplet-loss values upon updating the retrieval model. A consistency loss (via
distillation) is further introduced to simultaneously suppress the noisy
training signal coming from pseudo photo-sketch pairs. _Third_ , to enable
exchange from retrieval to generation, we rely on the following intuition –
good synthetic pairs would trigger a low value on the resulting triplet loss
and a higher output of the discriminator. Feeding these training signals back
to the generator would however involve passing through a non-differentiable
rasterization operation (Figure 2). We thus employ a policy-gradient [44]
based reinforcement learning scheme that feeds back these signals as rewards.

In summary, our contributions are: (a) For the first time, we propose to solve
the data scarcity problem in FG-SBIR by adopting semi-supervised approach that
additionally leverages large scale unlabelled photos to improve retrieval
accuracy. (b) To this end, we couple sequential sketch generation process with
fine-grained SBIR model in a joint learning framework based on reinforcement
learning. (c) We further propose a novel photo-to-sketch generator and
introduce a discriminator guided _instance weighting_ along with _consistency
loss_ to retrieval model training with noisy synthetic photo-sketch pairs. (d)
Extensive experiments validate the efficacy of our approach for overcoming
data scarcity in FG-SBIR (Figure 4) – we can already reach performances at par
with prior arts with just a fraction ($\approx$60$\%$) of the training pairs,
and obtain state-of-the-art performances on both QMUL-Shoe and QMUL-Chair with
the same training data (by $\approx$6$\%$ and $\approx$7$\%$ respectively).

##  2 Related Works

Fine-Grained SBIR: Yu et al.[52] introduced the first deep FG-SBIR model which
employed a deep triplet network to learn a common embedding space for photo
and sketch. Subsequent works have aimed at improving this via attention
mechanisms with higher order retrieval loss [43], joint discriminative-
generative learning with cross-modal image generation [29], text tags [42],
and cross-modal hierarchical co-attention [35]. Cross-category generalisation
[28] and on-the-fly retrieval setup [5] are more recent additions to existing
FG-SBIR literature. These fully supervised methods suffer from the data
scarcity, which we aim to address.

Handling _Data-Scarcity_ for FG-SBIR: Earlier works have tried resolving the
lack of instance-level photo-sketch paired data, by using edge-maps for
training [32] or synthetic sketch stroke deformation [53, 52] for data
augmentation. Umar et al.[33, 26] leveraged reinforcement-learning (RL) in an
attempt to augment sketches from edge-maps under the assumption that real
sketch-strokes are a subset of edge-maps, which however is negated by the
highly abstracted nature of real sketches. Very recently, mixed-modality
jigsaw solving [30] has been used as a pre-training task for FG-SBIR to
exploit additional photo images and their edge maps for cross-modal matching.
Its efficacy remains limited however as edge-maps are not sketches.

Photo-to-Sketch Generation: A plausible solution to data scarcity is
synthesising sketches for unlabelled photos to form pseudo photo-sketch pairs.
Existing photo-to-sketch generation methods can be classified into two types:
the first employs image-to-image translation [25], which however merely works
as a contour detection paradigm, thus failing to model the hierarchically
abstracted nature of human-drawn sketch. The second follows the seminal work
of Sketch-RNN [16], and generates sequential sketch-coordinates given a photo,
thus mimicking subjective human sketching style. The basic design [7],
involving a CNN encoder and RNN decoder, has been further augmented with both
self-domain and two way cross-modal reconstruction losses [41]. Following this
path, we improve sequential sketch-generative process with a 2D attention
mechanism to better exploit the spatial-layout of objects in photos.

Semi-supervised Learning: Our learning framework is semi-supervised in the
sense that the majority of training data are unlabelled photos without their
paired sketches. This is thus very different from most existing semi-
supervised learning methods which are designed for classification rather than
cross-modal retrieval. This means that these methods, based on either pseudo-
labelling [24, 27, 40, 31] or consistency regularisation [2, 1], offer little
insight into how our problem can be solved. In contrast, prior works on semi-
supervised cross-modal learning such as image captioning [6, 23] are more
relevant. However, we uniquely address a cross-modal instance-level retrieval
problem, and train the generator jointly with the retrieval model, rather than
merely providing model pre-training.

Figure 2: Our framework: a FG-SBIR model ($\mathcal{F}$) leverages large scale
unlabelled photos using a _sequential_ photo-to-sketch generation model
($\mathcal{G}$) along with labelled pairs. Discriminator ($\mathrm{D_{C}}$)
guided instance-wise weighting and distillation loss are used to guard against
the noisy generated data. Simultaneously, $\mathcal{G}$ learns by taking
reward from $\mathcal{F}$ and $\mathrm{D_{C}}$ via policy gradient (over both
labelled and unlabelled) together with supervised VAE loss over labelled data.
Note rasterization (vector to raster format) is a non-differentiable
operation.

##  3 Methodology

Overview: For semi-supervised fine-grained SBIR, we consider having access to
a limited amount of labelled photo-sketch pairs
$\mathrm{\mathcal{D}_{L}=\\{({p_{L}^{i}};{s_{p,L}^{i}})\\}_{i}^{N_{L}}}$ and a
much bigger set of unlabelled photos
$\mathrm{\mathcal{D}_{U}=\\{{p_{U}^{i}}\\}_{i}^{N_{U}}}$, where $N_{U}\gg
N_{L}$. The key objective is to improve retrieval performance using both
$\mathcal{D}_{L}$ and unlabelled photos $\mathcal{D}_{U}$ (having no
corresponding sketches). More specifically, our framework consists of two
models learned jointly: a FG-SBIR model, and a photo-to-sketch generation
model. The retrieval model tries to learn an embedding function
$\mathcal{F(\cdot)}:\mathbb{R}^{H\times W\times 3}\rightarrow\mathbb{R}^{d}$
mapping any rasterized sketch or photo having height $H$ and width $W$ to a
$d$-dimensional feature.

Instead of image-to-image translation [21, 25], sketch generation process
needs to be designed by sequential sketch-coordinate decoding [16] in order to
model the hierarchical abstract nature of sketch. In particular, the FG-SBIR
model requires rasterized sketch-images to obtain the sketch embedding, as
performance can collapse on using sketch-coordinate instead [5, 35]. Thus, the
generator learns a function $\mathcal{G}(\cdot):\mathbb{R}^{H\times W\times
3}\rightarrow\mathbb{R}^{T\times 2}$, mapping a photo to equivalent sequential
sketch-coordinate points
$\mathrm{s_{c}=\\{(x_{1},y_{1}),(x_{2},y_{2}),\cdots,(x_{T},y_{T})\\}}$, where
$T$ is the number of points. Note that in order to feed the generated sketches
to the feature embedding network $\mathcal{F}$ of the retrieval model, a
_rasterization_ (sketch-image redrawing from coordinates) operation is
required which is denoted as $s_{p}=\phi(s_{c}):\mathbb{R}^{T\times
2}\rightarrow\mathbb{R}^{H\times W\times 3}$. Finally, we can create
synthesised photo-sketch pairs
$\mathrm{\mathcal{D}_{U}^{\prime}=\\{({p_{U}^{i}};{s_{p,U}^{i}})\\}_{i}^{N_{U}}}$
for unlabelled photos to train the retrieval model. Once trained, only
$\mathcal{F(\cdot)}$ is used for retrieval during inference, while
$\mathcal{G(\cdot)}$ augments pseudo/synthetic photo-sketch pairs for
training.

###  3.1 Photo to Sketch Generation Model

The existing sequential photo-to-sketch generation models [41, 7] comprise a
convolutional image encoder, followed by an LSTM decoder. This however has two
major limitations: Firstly, it reduces the photo representation to a latent
vector, leading to significant spatial information loss. Secondly, one fixed
global representation is given as input at every time step of the LSTM
decoding. To overcome these limitations two novel designs are introduced: (a)
keeping the spatial feature map while ignoring global average pooling; (b)
looking back at the specific part of photo which it _draws_. Overall, it
consists of three major components, a CNN encoding the photo, a 2-D attention
module, and an LSTM decoder generating the coordinates sequentially.

Given a photo $p$, let the extracted convolutional feature map be
$\mathcal{B}\in\mathbb{R}^{h^{\prime}\times w^{\prime}\times c}$ where
$h^{\prime}$, $w^{\prime}$ and $c$ denotes the height, width and number of
channels, respectively. Next, we perform a global average pooling on
$\mathcal{B}$ to obtain a vector of size $\mathbb{R}^{c}$, and project it as
two vectors $\mu$ and $\sigma$, each having size $\mathbb{R}^{N_{z}}$. The
global embedding of photo is obtained through a reparameterization trick as
$z=\mu+\sigma\odot\mathcal{N}(0,1)$. The initial hidden state $h_{0}$ (and
optional cell state $c_{0}$) of decoder RNN is initialised as
$[h_{0};c_{0}]=\tanh(W_{z}z+b_{z})$.

Instead of predicting the absolute coordinates $\\{(x_{i},y_{i})\\}_{i}^{T}$,
we model every point as 5 element vectors $(\Delta x,\Delta
y,p_{1},p_{2},p_{3})$ where $\Delta x$ and $\Delta y$ represents the off-set
distances [16] in the $x$ and $y$ directions from the previous point. The last
three elements represent a binary one-hot vector of three pen-state
situations: pen touching the paper, pen being lifted and end of drawing. Each
offset-position ($\Delta x,\Delta y$) is modelled using a Gaussian mixture
model (GMM) with $M=20$ bivariate normal distributions [16] given by:

$p(\Delta x,\Delta y)=\sum_{j=1}^{M}\Pi_{j}\mathcal{N}(\Delta x,\Delta
y\mid\lambda_{j});\;\sum_{j=1}^{M}\Pi_{j}=1.$ (1)

Each M bivariate normal distribution has _five_ parameters
$\lambda=\\{\mu_{x},\mu_{y},\sigma_{x},\sigma_{x},\rho_{xy}\\}$ with mean
($\mu_{x},\mu_{y}$), standard deviation ($\sigma_{x},\sigma_{y}$) and
correlation ($\rho_{xy}$). The mixture weights of the GMM is modelled by a
categorical distribution of size $\mathbb{R}^{M}$. Thus every time step’s
output $y_{t}$ modelled is of size $\mathbb{R}^{5M+M+3}$, which includes 3
logits for pen-state. At time step $t$, a recurrent decoder network updates
its state $s_{t}=(h_{t},c_{t})$ as follows:
$\mathrm{s_{t}=RNN(s_{t-1};[g_{t},P_{t-1}])}$ where $g_{t}$ is the glimpse
vector encoding the information from specific relevant parts of the feature
map $\mathcal{B}$ to predict $y_{t}$; $P_{t-1}$ is the last predicted point
(start-token $P_{0}=\\{0,0,1,0,0\\}$), $[\cdot]$ signifies a concatenation
operation. The glimpse/context vector is obtained by 2D attention as follows:

$\displaystyle\begin{cases}J=\tanh(W_{\mathcal{B}}\circledast\mathcal{B}+W_{S}h_{t-1});\\\
\alpha_{i,j}=\mathrm{softmax}(W_{a}^{T}J_{i,j})\\\
g_{t}=\sum_{i,j}\alpha_{i,j}\cdot\mathcal{B}_{i,j};\;i=[1,..h^{\prime}],\;j=[1,..w^{\prime}]\end{cases}$
(2)

where $W_{B}$, $W_{S}$, $W_{a}$ are the learnable weights. Calculating the
attention weight $\alpha_{i,j}$ at every spatial position $(i,j)$, we employ a
convolution operation “$\circledast$” with $3\times 3$ kernel
$W_{\mathcal{B}}$ to consider the neighbourhood information in the 2D
attention module, and $g_{t}$ is obtained by weighted summation operation at
the end. A fully-connected layer over every hidden state outputs
$y_{t}=W_{y}h_{t}+b_{y}$ , where $y_{t}\in\mathbb{R}^{6M+3}$. We refer to [16]
for more details. Like the standard VAE, our generator $\mathcal{G}$ is
trained from the weighted summation of a reconstruction loss
($\mathrm{L^{R}_{\mathcal{G}}}$) and a KL-divergence loss
($\mathrm{L^{kl}_{\mathcal{G}}}$) with unit normal distribution as follows:

$\mathrm{L_{\mathcal{G}}^{vae}=L^{R}_{\mathcal{G}}+\omega_{kl}L^{kl}_{\mathcal{G}}},$
(3)

where $L^{R}_{\mathcal{G}}$ is composed of the negative log-likelihood loss of
the offsets $\Delta z=(\Delta x,\Delta y)$ and the pen states
$(p_{1},p_{2},p_{3})$:
$\mathrm{L^{R}_{\mathcal{G}}}=-\frac{1}{T}\Big{[}\sum_{i=1}^{T}\log p(\Delta
z_{i}\mid\lambda_{i})+\hat{p}_{i}\log(p_{i})\Big{]}$.

###  3.2 Baseline FG-SBIR Model

For the discriminative retrieval module $\mathcal{F(\cdot)}$, we use the
state-of-the-art Siamese network [43, 10, 5] (multi-branch with weight-
sharing) with soft spatial attention [47] to focus on salient parts of the
feature map. Concretely, given a photo or rasterized sketch image $I$, we use
a pre-trained InceptionV3 model [45] to extract feature map
$F^{\prime}=f_{\theta}(I)$. This is followed by a residual connection between
backbone feature and attention normalised feature to give
$F=F^{\prime}+F^{\prime}\cdot f_{attn}(F^{\prime})$, upon which global average
pooling is performed to obtain final feature representation of size
$\mathbb{R}^{d}$; and $f_{attn}$ is modelled using 1x1 convolution with
softmax across the spatial dimensions. For training, the distance to a sketch
anchor (a) from a negative photo (n), denoted as
$\beta^{-}=\left\|\mathcal{F}(a)-\mathcal{F}(n)\right\|_{2}$ should increase
while that from the positive photo (p),
$\beta^{+}=\left\|\mathcal{F}(a)-\mathcal{F}(p)\right\|_{2}$ should decrease.
This is brought about by the triplet loss with a margin $\mu>0$ as a
hyperparameter:

$\mathrm{L_{\mathcal{F}}^{trip}=max\\{0,\mu+\beta^{+}-\beta^{-}\\}}.\vspace{-0.1cm}$
(4)

###  3.3 Semi-Supervised Framework for FG-SBIR

Overview: Firstly, we train the photo-to-sketch generation model and
discriminative fine-grained FG-SBIR model independently using the labelled
training set $\mathrm{\mathcal{D}_{L}}$. Thereafter, through our semi-
supervised learning framework, $\mathcal{F}(\cdot)$ starts exploiting the
unlabelled photos to improve its retrieval performance, while enhancing sketch
generation quality of $\mathcal{G}(\cdot)$ by using $\mathcal{F}(\cdot)$ as a
critic to provide training signal to the sketch-generation model
$\mathcal{G}(\cdot)$ using both unlabelled and labelled photos simultaneously.
Hence, both $\mathcal{G}(\cdot)$ and $\mathcal{F}(\cdot)$ can now improve
itself with the help of each other and by exploiting unlabelled photos (see
Figure 2).

Certainty Score for Synthetic Photo-Sketch Pair: The generated photo-sketch
pairs $\mathcal{D}_{U}^{\prime}$ of unlabelled photos are sometimes noisy
compared to real labelled photo-sketch pairs $\mathcal{D}_{L}$. This is mainly
due to large possible output space [41] of sketch drawing even with respect to
a particular photo, as well as difficulties in predicting the sketch ending
token [16] in the sequential decoding process. Every synthetic photo-sketch
instance pair needs to be handled individually based on their quality, thus
requiring a specific _certainty score_ – signifying the reliability of
synthetic photo-sketch pair to train the retrieval model. Existing semi-
supervised classification approach usually considers the probability
distribution over classes to filter out noisy samples based on a predefined
threshold [48], top-K selection [40], or uses entropy-based instance-wise
weighting [20] to deal with noisy synthetic labels. A new solution is thus
needed to not just measure the quality of generated sketch itself, but to
quantify how the generated sketch matches with the particular photo input, in
order to help training the retrieval model.

Inspired by the generative adversarial network [8] where the sigmoid
normalised output of discriminator shows the probability of being a real vs
fake input sample, we use the _discriminator’s confidence_ to quantify the
quality of synthetic photo-sketch pairs. Specifically, the discriminator
$\mathrm{D_{C}}$ learns to classify between real photo sketch pairs and
generated pseudo photo-sketch pairs (concatenated across channels). Thus, the
learning objective for $\mathrm{D_{C}}$ is

$\mathrm{L_{D_{C}}=-\mathbb{E}_{({p_{L};\,{s_{p,L}}})\sim\mathcal{D}_{L}}\big{[}\log\mathrm{D_{C}}\big{(}p_{L},s_{p,L}\big{)}\big{]}}\\\\[-2.0pt]
\mathrm{-\mathbb{E}_{({p_{U};\,{s_{p,U}}})\sim\mathcal{D}_{U}^{\prime}}\big{[}\log\big{(}1-\mathrm{D_{C}}\big{(}p_{U},s_{p,U}\big{)}\big{)}\big{]}}.$
(5)

This objective is computed via a binary cross-entropy loss using label $1$ for
real pairs, and $0$ for synthetic ones. Thus, the discriminator’s output
$\mathrm{\mathrm{D_{C}}(p_{U},s_{p,U})\in[0,1]}$ signifies the extent to which
the synthetic photo-sketch pairs match with the distribution of real labelled
photo-sketch pairs. Therefore, values closer to $1$ indicate better quality
synthetic photo-sketch pairs.

Tolerance against Noisy Pseudo-Labelled Data: To further avoid over-fitting to
noisy synthetic photo-sketch pairs, we introduce a consistency loss with
respect to a pre-trained (on labelled dataset) retrieval model as _weak
teacher_ [13]. More specifically, once the baseline FG-SBIR model is trained
from labelled data, we keep a copy as $\mathcal{F}^{T}$ with weights frozen.
As $\mathcal{F}^{T}$ has been trained from real clean photo-sketch pairs only,
we expect that the feature embedding vector obtained from it would act as an
additional supervision via distillation [18] to regularise the main FG-SBIR
model ($\mathcal{F}$) which is to be trained from both labelled and synthetic
photo-sketch pairs in a semi-supervised manner. This distillation process is
expected to improve the tolerance against noisy information of synthetic data.
Compared to cross-entropy loss [18] used in distillation of classification
network, a naive choice to design distillation for feature embedding network
is to minimise the distance between learnable student’s embedding and
teacher’s embedding of a particular photo or sketch image individually. We
term it _absolute teacher_. However, instead of considering the actual
embedding, we hypothesise that the _relative distance_ between paired photo
and sketch, minimising which is the major purpose of embedding network, could
be a better knowledge to be distilled. We term this as _relative teacher_.
Thus, given a photo-sketch image pair ($p,s_{p}$) and $d(\cdot,\cdot)$ being a
${l_{2}}$ distance function, the consistency loss for learnable student
$\mathcal{F}$ with respect to pre-trained teacher $\mathcal{F}^{T}$ becomes as
follows:

$\mathrm{L}_{\mathcal{F}}^{KD}=\big{\|}d\big{(}\mathcal{F}^{T}(p),\mathcal{F}^{T}(s_{p})\big{)}-d\big{(}\mathcal{F}(p),\mathcal{F}(s_{p})\big{)}\big{\|}_{2}.$
(6)

###  3.4 Joint Training

Optimising FG-SBIR Model: We train the fine-grained SBIR model $\mathcal{F}$
using triplet loss over labelled photo-sketch pairs $\mathcal{D}_{L}$,
_instance weighted triplet loss_ over generated pseudo photo-sketch pairs
$\mathcal{D}_{U}$, and pre-trained teacher based consistency loss over both
$\mathcal{D}_{L}$ and $\mathcal{D}_{U}$. Given sampled data
$\mathcal{D}_{L}^{i}=\\{p_{L}^{i},s_{p,L}^{i}\\}\sim\mathcal{D}_{L}$ and
$\mathcal{D^{\prime}}_{U}^{j}=\\{p_{U}^{j},s_{p,U}^{j}\\}\sim\mathcal{D}_{U}^{\prime}$
(on _same_ ratio), the instance-wise weight is calculated as
$\mathrm{\omega_{j}=D_{C}(\mathcal{D^{\prime}}_{U}^{j})}$. The overall semi-
supervised loss to train the retrieval model becomes:

$\mathrm{L}_{\mathcal{F}}^{all}=L_{\mathcal{F}}^{trip}(\mathcal{D}_{L}^{i})+\omega_{j}\cdot
L_{\mathcal{F}}^{trip}(\mathcal{D^{\prime}}_{U}^{j})+\lambda_{kd}\cdot
L_{\mathcal{F}}^{KD}(\mathcal{D}_{L}^{i},\mathcal{D^{\prime}}_{U}^{j})$

(7)

Optimising Photo-to-Sketch Model: Besides the fully-supervised VAE loss
$L_{\mathcal{G}}^{R}$, during joint training, the photo-to-sketch generation
model is also learned considering $\mathcal{F}$ and $\mathrm{D_{C}}$ as
critics. In particular, if the generated sketch from $\mathcal{G}$ correctly
depicts the corresponding input photo, the triplet loss for that generated
photo-sketch pair from retrieval model would be low, signifying a better
photo-sketch matching and generated sketch quality. Similarly, the higher the
discriminator’s output, the better the quality of generated photo-sketch
pairs. However, these training signals from $\mathcal{F}$ and $\mathrm{D_{C}}$
cannot be directly back-propagated to $\mathcal{G}$, as there exists a non-
differentiable _rasterization_ operation $s_{p}$ before feeding the sketch-
image to both retrieval model and discriminator. Hence, we employ
reinforcement learning based on policy-gradient [44] with REINFORCE [46]
deployed to estimate gradients with respect to parameters
$\theta_{\mathcal{G}}$ of $\mathcal{G}$ given some _reward_. As $\mathcal{G}$
aims to lower this triplet loss value $L_{\mathcal{F}}^{trip}$ (Eqn. 4), the
reward should be negative of $L_{\mathcal{F}}^{trip}$ that needs to be
maximised. Similarly, the discriminator’s output quantifying the goodness of
photo-sketch pairs needs to be maximised. Thus the weighted joint reward is:

$\mathrm{\mathrm{R}_{\mathcal{G}}=-\lambda_{r1}\cdot
L_{\mathcal{F}}^{trip}(\mathcal{D}^{i})+\lambda_{r2}\cdot
D_{C}(\mathcal{D}^{i})}\vspace{-0.1cm}$ (8)

where $\mathcal{D}^{i}\sim\mathcal{D}_{L}\cup\mathcal{D}_{U}$. This reward
could be computed for both labelled and unlabelled data as it does not need
any ground-truth sketch-coordinates unlike the $L_{\mathcal{G}}^{vae}$ loss
(Eqn. 3). Thus two types of gradients are computed to update the parameter
$\theta_{\mathcal{G}}$, one using policy gradient [44] based on joint-reward
guided by the retrieval model and the discriminator, and the other using back-
propagation over only the labelled photos:

${\begin{split}&\nabla_{\theta_{\mathcal{G}}}L(\theta_{\mathcal{G}})=\underbrace{\nabla_{\theta_{\mathcal{G}}}L_{\mathcal{G}}^{vae}(\theta_{\mathcal{G}})}_{\text{over
only labelled
data}}\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\mathrm{(9)}\\\\[-13.0pt]
&\mathrm{-}\lambda_{\mathcal{G}}\sum_{i=1}^{T}\underbrace{\mathbb{E}_{\begin{subarray}{c}p_{i}\sim
p(q_{i})\\\ \Delta z_{i}\sim p(\Delta
z_{i}\mid\lambda_{i})\end{subarray}}\nabla_{\theta_{\mathcal{G}}}\Big{(}\log
p(\Delta z_{i}\mid\lambda_{i})+\log p(p_{i})\Big{)}\cdot
R_{\mathcal{G}}}_{\text{over both labelled and unlabelled data (via policy
gradient)}}\end{split}}$

In our experiments, we only update the final, fully-connected layer of sketch-
decoder (with weights $W_{y},b_{y}$ predicting 6M+3 outputs at every time
step), at times using policy gradient, keeping rest of the parameters of
$\mathcal{G}$ fixed. We use a single global reward for the whole sketch-
coordinate sequence, instead of local reward at every time step, that would
otherwise need costly Monte Carlo roll-outs [51]. Note that in our design,
$\mathcal{G}$ and $\mathrm{D_{C}}$ are connected in a GAN-like fashion [15]
having adversarial objective. Moreover, the retrieval and generative models
are trained alternatively improving each other over time (Algorithm 1).

Algorithm 1 Training of Semi-Supervised FG-SBIR

1:Input: Labelled photo-sketch pairs $\mathcal{D}_{L}$ and Unlabelled photos
$\mathcal{D}_{U}$.

2:Initialise hyper params: $k_{r}$, $k_{g}$.

3:Pre-training: $\mathcal{G}$ and $\mathcal{F}$ from $\mathcal{D}_{L}$ (using
$\mathrm{L_{\mathcal{G}}^{vae}}$ & $\mathrm{L_{\mathcal{F}}^{trip}}$).

4:while not done training do

5: for $k_{r}$ steps do

6: Sample data $\mathcal{D}_{L}^{i}\sim\mathcal{D}_{L}$ and
$\mathcal{D}_{U}^{j}\sim\mathcal{D}_{U}$.

7: Get synthetic paired images $\mathcal{D^{\prime}}_{U}^{j}$ using
$\mathcal{G(\cdot)}$.

8: Train $\mathcal{F}$ using
$\\{\mathcal{D}_{L}^{i},\mathcal{D^{\prime}}_{U}^{j}\\}$ $\triangleright$ Eqn.
7

9: Train $D_{C}$ using
$\\{\mathcal{D}_{L}^{i},\mathcal{D^{\prime}}_{U}^{j}\\}$ $\triangleright$ Eqn.
5

10: end for

11: for $k_{g}$ steps do

12: Sample data $\mathcal{D}_{L}^{i}\sim\mathcal{D}_{L}$ and
$\mathcal{D}_{U}^{j}\sim\mathcal{D}_{U}$.

13: Get reward $R_{\mathcal{G}}$ using $\mathcal{F}$ and $D_{C}$.

14: Train $\mathcal{G}$ using $\\{\mathcal{D}_{L}^{i},\mathcal{D}_{U}^{j}\\}$
$\triangleright$ Eqn. 3.4

15: end for

16:end while

17:Output: Optimised models $\mathcal{F}$, $\mathcal{G}$ and $\mathrm{D_{C}}$.

##  4 Experiments

Datasets: Two publicly available datasets, QMUL-Shoe-V2 [28, 33, 41, 5] and
QMUL-Chair-V2 [5, 41] are used, which contain stroke-level coordinate
information of sketches in addition to instance-wise paired sketch-photo
labels, thus enabling us to train both retrieval and sketch-generative models.
Out of the 6,730 sketches and 2,000 photos in Shoe-V2, 6,051 and 1,800 for
training respectively, and the rest are for testing [5, 41]. The splits [5,
41] for Chair-V2 dataset are 1,275/725 sketches and 300/100 photos for
training/testing respectively. In addition to these labelled training data, we
further use all 50,025 UT-Zap50K images [50] as unlabelled photos for shoe
retrieval, and 7,800 unlabelled chair photos [30] are collected from shopping
websites, including IKEA, Amazon and Taobao. Data, code, and models will be
released soon.
Implementation Details: Firstly, for sketch-generation, we use ImageNet pre-
trained VGG-16 as encoder, excluding any global average pooling operation. We
keep the dimension ($N_{z}$) of $z$ as 128, the hidden state of the decoder
LSTM as 512, the embedding dimension of the 2D-attention module as 256
respectively. We set the max sequence length to $100$, and the generative
model is trained with a batch size of 64 with $\omega_{kl}=1$ using the pre-
training strategy from [41]. Secondly, the retrieval model (ImageNet pre-
trained Inception-V3 [45] ) is trained with a batch-size of 16 with a margin
value of 0.3. Finally, after completing individual training from labelled
data, we start _joint training_ (Section 3.4) by additionally exploiting
unlabelled data using $k_{g}$ and $k_{r}$ as 5. Architecture of
$\mathrm{D_{C}}$ is from [21]. We set $\lambda_{kd}=0.1$, $\lambda_{r1}=1$,
$\lambda_{r2}=1$, and $\lambda_{\mathcal{G}}=10$ respectively. All images are
resized to $256\times 256$, with rasterization from sketch-coordinate
involving a window of same size having centre scaling as well. We use Adam
optimiser for both the generation and retrieval models with a learning rate of
$0.0001$.

Evaluation Metric: (a) FG-SBIR: Following existing FG-SBIR works [52, 30], we
use $\mathrm{Acc@q}$, i.e. percentage of sketches having true-paired photo
appearing in the top-q list. (b) Sketch Generation: Following [41, 4], sketch-
generation is quantified from three perspectives (i) _Recognition:_ Using a
ResNet-50 classifier trained on 250-classes from TU-Berlin sketch dataset, a
generated sketch getting recognised as the same class as that of corresponding
photo signifies category-level realism. (ii) _Retrieval:_ 111Note: Retrieval
accuracy is used to quantify both FG-SBIR and sketch generation performance.
Please refer to [41] for more details. To judge whether the generated sketch
has object-instance specific agreement, we check the retrieval accuracy
$\mathrm{Acc@q}$ via a pre-trained FG-SBIR model using the generated sketches
to retrieve corresponding photos of the testing set. (iii) _Generation:_
Following a recent sketch generation work [4], we further calculate FID-score
[17] using a pre-trained sketch-classifier that captures both the quality and
diversity of generated data compared to real human sketches.

###  4.1 Competitors

Sketch Generation: Sketch Generation could be approached in two following
ways: (a) Image-to-image translation pipeline: Pix2Pix [21] could be adapted
to perform cross-modal translation in the image space. PhotoSketch [25]
extends further to handle the one-to-many possible nature of photo-conditioned
sketch image generation problem, by calculating a mean loss over multiple
sketches corresponding to a particular photo. (b) Image-to-sequence generation
pipeline: Pix2Seq [7] is the ablated version of our model having a
convolutional encoder and LSTM decoder, without involving 2D-attention. L2S
[41] is an extension over [7] that uses two-way cross domain translation with
self-domain reconstruction for better regularisation. Ours-G is a _supervised_
model with 2D-attention, trained independently from labelled data only. Ours-
G-full is our final sketch-generative model involving joint-training to learn
from both labelled and unlabelled data.

Fine-Grained SBIR: We compare with three groups of competitors. (a) _state-of-
the-art:_ SN-Triplet [52] employs triplet ranking loss with Sketch-a-Net as
its baseline feature extractor. SN-HOLEF [42] is an extension over [52]
employing spatial attention along with higher order ranking loss. SN-RL [5] is
a very recent work employing reinforcement learning based fine-tuning for on-
the-fly retrieval. As early retrieval is not our objective, we cite result at
sketch-completion point. (b) _Exploiting unlabelled photos:_ There has been no
prior work addressing semi-supervised learning for FG-SBIR, and model designed
for category-level retrieval [22] does not fit here. We thus adopt a few works
that could be used to leverage unlabelled photos. Edgemap-Pretrain [32] is a
naive-approach to use edge-maps of unlabelled photos to pre-train the
retrieval model. While edge-maps hardly have any similarity to real free-hand
sketches, they could be converted to better pseudo-sketches using the work
[33] that learns how to abstract sketches based on subset-stroke selection. We
term it as Edge2Sketch [33]. Recently, Jigsaw-Pretrain [30] used jigsaw
solving over the mixed patches between a particular photo (unlabelled) and its
edge-map, as a pre-text task for self-supervised learning (SSL) to improve FG-
SBIR performance. Furthermore, we term our self-implemented _supervised_ FG-
SBIR model trained only on labelled data as Ours-F. Ours-F-Full is our final
retrieval model employing joint training over both labelled and unlabelled
photos. We also replace our 2D-attention based sketch-generation process by
baseline sketch-generative model Pix2Pix [21] and L2S [41], and term them as
Ours-F-Pix2Pix and Ours-F-L2S respectively. Finally, we design a naive semi-
supervised FG-SBIR baseline (Vanilla-SSL-F), where we _blindly_ (without
instance-weighting and distillation) use the generated sketch to additionally
train the retrieval model.

Table 1: Quantitative results of photo-to-sketch generation Chair-V2 |  Recognition ($\uparrow$) |  Retrieval($\uparrow$) | FID Score($\downarrow$)
---|---|---|---
$\mathrm{Acc.@1}$ | Acc.@10 | Acc.@1 | Acc.@10
Pix2Pix [21] | 4.5% | 12.1% | 2.4% | 16.2% | 33.4
PhotoSketch [25] | 7.1% | 14.3% | 4.2% | 17.9% | 25.7
Pix2Seq [7] | 5.4% | 52.1% | 4.0% | 31.8% | 14.5
L2S [41] | 12.3% | 53.8% | 8.3% | 36.7% | 12.7
Ours-G (only labelled data) | 15.2% | 56.9% | 13.4% | 40.7% | 10.1
Ours-G-Full | 16.4% | 58.6% | 14.9% | 42.6% | 8.9
Shoe-V2 |  Recognition($\uparrow$) |  Retrieval($\uparrow$) | FID Score ($\downarrow$)
Acc.@1 | Acc.@10 | Acc.@1 | Acc.@10
Pix2Pix [21] | 6.2% | 14.5% | 1.8% | 8.4% | 31.7%
PhotoSketch [25] | 8.9% | 17.3% | 3.4% | 10.2% | 24.3%
Pix2Seq [7] | 51.3% | 86.6% | 5.1% | 25.8% | 11.3
L2S [41] | 53.7% | 89.7% | 6.2% | 28.6% | 10.7
Ours-G (only labelled data) | 56.3% | 91.9% | 9.7% | 33.6% | 9.5
Ours-G-Full | 58.1% | 93.4% | 12.3% | 35.4% | 8.3

###  4.2 Performance Analysis

Photo-to-Sketch Generation: From Table 1, we observe: (i) _Pix2Pix_ and
_PhotoSketch_ based on cross-modal translation in pixel space perform poorly.
They fail to capture the abstraction in human sketching style, where
distribution gap with real sketches is reflected in their significantly poor
FID scores. (ii) _Pix2Seq_ and _L2S_ outputs vector sketches by sequentially
predicting sketch coordinates, thus possessing higher similarity towards human
sketches. They however, still lag behind our ablated version _Ours-G_ in
scores. As both of them reduce the spatial dimension of the convolutional
feature-map to a global context vector, spatial information is significantly
compromised, with the decoder receiving little guidance from the vector on
exact drawing content. In contrast, we retain the spatial dimension of
feature-map, and employ 2D-attention to focus on that specific part of the
photo it draws at any time step. (iii) _L2S_ is a notably close competitor to
ours in terms of recognition accuracy, but better information passage between
every time step of decoder and convolutional encoder delivers much better
sketches with fine-grained details (reflected by retrieval accuracy).
Furthermore, our final model Ours-G-Full employs joint training with a
retrieval model to additionally exploit the unlabelled photos, improving
sketch generation performance (retrieval Acc@1) from $9.7\%$ to $12.3\%$ by
$2.6\%$ over our baseline _Ours-G_ on Shoe-V2, thus justifying the benefits of
our semi-supervised learning. Some qualitative results are shown in Figure 3.
Blue denotes a supervised baseline, while red is Ours-G(F)-Full.

Table 2: Quantitative results of fine-grained SBIR Methods | Chair-V2 | Shoe-V2
---|---|---
Acc.@1 | Acc.@10 | Acc.@1 | Acc.@10
SN-Triplet [52] | 47.4% | 84.3% | 28.7% | 71.6%
SN-HOLEF [42] | 50.7% | 86.3% | 31.2% | 74.6%
SN-RL [5] | 51.2% | 86.9% | 30.8% | 74.2%
Edgemap-Pretrain [32] | 53.9% | 87.7% | 33.8% | 80.9%
Edge2Sketch-Pretrain [33] | 54.3% | 88.2% | 34.2% | 81.2%
Jigsaw-Pretrain [30] | 56.1% | 88.7% | 36.5% | 85.9%
Ours-F (only labelled data) | 53.3% | 87.5% | 33.4% | 80.7%
Vanilla-SSL-F | 49.6% | 85.6% | 30.6% | 74.3%
Ours-F-Pix2Pix | 53.2% | 87.5% | 33.2% | 80.1%
Ours-F-L2S | 57.6% | 89.4% | 36.6% | 84.7%
Ours-F-Full | 60.2% | 90.8% | 39.1% | 87.5%

Fine-grained SBIR: From Table 2, we observe: (i) Our baseline retrieval model
is noticeably better than _SN-Triplet_ , and lies at par with recent state-of-
the-art FG-SBIR baselines like _SN-HOLEF_ and _SN-RL_. (ii) With regards to
exploiting unlabelled photos, _Edgemap-Pretrain_ offers marginal improvement
while using it on top of our baseline with ImageNet pretrained weights.
Aligning with the intuition, while edge-maps are further augmented with
_Edge2Sketch_ by a subset of stroke selection to model the abstracted nature
of sketch over edge-maps, it increases retrieval performance by a reasonable
margin. In context of using edge-maps for pre-training, _Jigsaw-Pretrain_
provides maximum benefits, but still lags behind our final model _Ours-F-
Full_. (iii) While edge-map does not posses sketch abstraction knowledge of
human sketching style, our approach of using a sequential photo-to-sketch
generation model to generate synthetic photo-sketch pairs for unlabelled photo
encodes better knowledge to enhance generalisation. However, it is noteworthy
that _Vanilla-SSL-F_ blindly using synthetic sketch-photo pairs yields
performance lower than the supervised one due to overfitting on noisy
information. Overall, for fine-grained SBIR, due to our proposed semi-
supervised learning, the retrieval accuracy $\mathrm{Acc@1}$ of _Ours-F-Full_
increases from $33.4\%$ to $39.1\%$ by a margin of $5.7\%$ over our baseline
_Ours-F_ on Shoe-V2. Moreover, replacing our photo-to-sketch generation model
by _L2S_ and _Pix2Pix_ reduces the same by $2.5\%$ and $5.9\%$ respectively,
thus justifying the importance of our sketch generative model with 2D
attention. (iv) Note that policy-gradient based RL scheme could be avoided by
using Pix2Pix for sketch generation, and gradient can directly be back-
propagated from retrieval to generative model. However, that is still found to
be inferior to ours.

Figure 3: Qualitative results on our photo-to-sketch generation process, where
sketch is shown with attention-map at progressive instances.

###  4.3 Ablation Study

A thorough ablative study on Shoe-V2 dataset verifies contributions of
individual design components in Table 3. _[i]_ Instance weighting for
retrieval: To simply judge the contribution of discriminator
($\mathrm{D_{C}}$) guided instance weighting we remove it, and adapt the
framework accordingly. Consequently $\mathrm{Acc@1}$ retrieval performance
significantly drops to $36.8\%$ with a decrease of $2.3\%$ on Shoe-V2. Due to
sigmoid normalisation [8], the output of $\mathrm{D_{C}}$ falls in [0,1]. We
quantify it as 10 discrete levels with a step size $0.1$. We calculate the
average ranking percentile (ARP) of synthetic sketch-photo pairs from testing
set which fall under the same discrete level, and plot it against $10$
different levels. From Figure 4 (a), it is evident that the synthetic sketch-
photo pairs having higher discriminator score (towards $1$) tend to have much
better ARP [5] values (i.e better quality), while those with lesser ARP values
are assigned with lesser (towards $0$) certainty score by the discriminator.
This observation is consistent with our assumption that $\mathrm{D_{C}}$
should quantify quality of synthetic sketch-photo pairs for instance-wise
weighting. _[ii]_ Distillation based noise tolerance for retrieval: Removing
knowledge distillation based regularisation, which additionally tries to
provide tolerance against noisy synthetic sketch-photo pairs, $\mathrm{Acc@1}$
is decreased by $1.8\%$ to $37.3\%$ on Shoe-V2 dataset. Our _relative teacher_
based distillation process (Section 3.3) for retrieval network surpasses
_absolute teacher_ alternative by a margin of $0.9\%$ ($\mathrm{Acc@1}$) on
Shoe-V2, thus confirming its usefulness. _[iii]_ 2D-attention for sketch-
generation: The use of 2D attention significantly improves the sketch
generation performance, providing better fine-grained agreement with the input
photo. While we employ a 3x3 convolutional kernel to aggregate neighbourhood
information, using an 1D attention that treats feature maps as 1D sequence,
the retrieval accuracy $\mathrm{Acc@1}$ of generated sketches on Shoe-V2 drops
to $8.1\%$ by margin of $4.2\%$. We conjecture that 2D-spatial attention has
higher efficiency in generating fine-grained sequential sketches from input
photo than two-way translation based regularisation as done in _L2S_ [41].

Figure 4: (a): Consistency of discriminator’s certainty score. (b): Varying
training data size for FG-SBIR - Semi-Supervised Learning (SSL) vs Supervised-
Learning (SL).

_[iv]_ Significance of joint-training: (a) A direct way of judging efficiency
of joint training is employing separately trained photo-to-sketch generation
model to augment synthetic sketch-photo pairs, and using them _blindly_ to
train the retrieval model along with labelled data _without_ instance
weighting or teacher-regularisation. This however lags behind the baseline
(supervised) fine-grained SBIR model by $2.8\%$ (i.e. $30.6\%$), as the model
over-fits to the noisy information present in synthetic sketch-photo data.
This confirms that naively using sketch generation does not help at all. (b)
The retrieval accuracy ($\mathrm{Acc@1}$) from sketch generation performance
improves by $1.4\%$ with an additional policy-gradient based training taking
reward (Eqn. 8) from the retrieval model and the discriminator
$\mathrm{D_{C}}$ as critic. Individually, they help to improve by $0.9\%$ and
$0.8\%$, respectively under the same metric. (c) Furthermore, we compute the
performance of our semi-supervised framework at varying training data size for
both processes in Figure 4 (b). We notice a significant overhead compared to
our supervised baseline model for each dataset individually.

Table 3: Ablative study on Shoe-V2: Instance Weighting (IW), Teacher Regularisation (TR), Attention (AT), Joint-Training (JT). IW | TR | AT | JT | Fine-Grained SBIR | Sketch Generation
---|---|---|---|---|---
Acc.@1 | Acc.@10 | Recognition | Retrieval
Acc.@1 | Acc.@1
✓ | ✓ | ✓ | ✓ | 39.1% | 87.5% | 58.1% | 12.3%
✗ | ✓ | ✓ | ✓ | 36.8% | 85.4% | 57.3% | 11.2%
✓ | ✗ | ✓ | ✓ | 37.3% | 86.1% | 57.8% | 12.1%
✓ | ✓ | ✗ | ✓ | 37.6% | 86.1% | 51.3% | 5.1%
✓ | ✓ | ✓ | ✗ | 37.9% | 86.6% | 56.3% | 9.7%
✗ | ✗ | ✗ | ✗ | 31.1% | 75.4% | 51.3% | 5.1%

##  5 Conclusion

We have proposed a semi-supervised fine-grained sketch-based image retrieval
framework to solve the data scarcity problem. To this end, we proposed to
treat sequential photo-to-sketch generation and fine-grained sketch-based
image retrieval as two conjugate problems along with various regularizers to
address the intricate issues of reliability and tolerance to noisy synthetic
sketch-photo pairs. This leads to substantial improvement on existing
baselines in sparse data-scenarios for FG-SBIR.

## References

  * [1] David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel.  Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring.  In ICLR, 2020.
  * [2] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel.  Mixmatch: A holistic approach to semi-supervised learning.  In NeurIPS, 2019.
  * [3] Ayan Kumar Bhunia, Pinaki Nath Chowdhury, Yongxin Yang, Timothy Hospedales, Tao Xiang, and Yi-Zhe Song.  Vectorization and rasterization: Self-supervised learning for sketch and handwriting.  In CVPR, 2021.
  * [4] Ayan Kumar Bhunia, Ayan Das, Umar Riaz Muhammad, Yongxin Yang, Timothy M. Hospedales, Tao Xiang, Yulia Gryaditskaya, and Yi-Zhe Song.  Pixelor: A competitive sketching ai agent. so you think you can beat me?  In Siggraph Asia, 2020.
  * [5] Ayan Kumar Bhunia, Yongxin Yang, Timothy M Hospedales, Tao Xiang, and Yi-Zhe Song.  Sketch less for more: On-the-fly fine-grained sketch based image retrieval.  In CVPR, 2020.
  * [6] Wenhu Chen, Aurelien Lucchi, and Thomas Hofmann.  A semi-supervised framework for image captioning.  arXiv preprint arXiv:1611.05321, 2016.
  * [7] Yajing Chen, Shikui Tu, Yuqi Yi, and Lei Xu.  Sketch-pix2seq: a model to generate sketches of multiple categories.  arXiv:1709.04121, 2017.
  * [8] LI Chongxuan, Taufik Xu, Jun Zhu, and Bo Zhang.  Triple generative adversarial nets.  In NeurIPS, 2017.
  * [9] John Collomosse, Tu Bui, and Hailin Jin.  Livesketch: Query perturbations for guided sketch-based visual search.  In CVPR, 2019.
  * [10] Sounak Dey, Pau Riba, Anjan Dutta, Josep Llados, and Yi-Zhe Song.  Doodle to search: Practical zero-shot sketch-based image retrieval.  In CVPR, 2019.
  * [11] Anjan Dutta and Zeynep Akata.  Semantically tied paired cycle consistency for zero-shot sketch-based image retrieval.  In CVPR, 2019.
  * [12] Titir Dutta, Anurag Singh, and Soma Biswas.  Adaptive margin diversity regularizer for handling data imbalance in zero-shot sbir.  In ECCV, 2020.
  * [13] Tommaso Furlanello, Zachary C Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar.  Born again neural networks.  In ICML, 2018.
  * [14] Junlong Gao, Shiqi Wang, Shanshe Wang, Siwei Ma, and Wen Gao.  Self-critical n-step training for image captioning.  In CVPR, 2019.
  * [15] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In NeurIPS, 2014.
  * [16] David Ha and Douglas Eck.  A neural representation of sketch drawings.  In ICLR, 2018.
  * [17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In NeurIPS, 2017.
  * [18] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  In NeurIPS Workshop, 2015.
  * [19] Minyoung Huh, Shao-Hua Sun, and Ning Zhang.  Feedback adversarial learning: Spatial feedback for improving generative adversarial networks.  In CVPR, 2019.
  * [20] Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum.  Label propagation for deep semi-supervised learning.  In CVPR, 2019.
  * [21] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In CVPR, 2017.
  * [22] Young Kyun Jang and Nam Ik Cho.  Generalized product quantization network for semi-supervised image retrieval.  In CVPR, 2020.
  * [23] Dong-Jin Kim, Jinsoo Choi, Tae-Hyun Oh, and In So Kweon.  Image captioning with very scarce supervised data: Adversarial semi-supervised learning approach.  In EMNLP, 2019.
  * [24] Dong-Hyun Lee.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.  In ICML, 2013.
  * [25] Mengtian Li, Zhe Lin, Radomir Mech, Ersin Yumer, and Deva Ramanan.  Photo-sketching: Inferring contour drawings from images.  In WACV, 2019.
  * [26] Umar Riaz Muhammad, Yongxin Yang, Timothy M Hospedales, Tao Xiang, and Yi-Zhe Song.  Goal-driven sequential data abstraction.  In ICCV, 2019.
  * [27] Rafael Müller, Simon Kornblith, and Geoffrey E Hinton.  When does label smoothing help?  In NeurIPS, 2019.
  * [28] Kaiyue Pang, Ke Li, Yongxin Yang, Honggang Zhang, Timothy M Hospedales, Tao Xiang, and Yi-Zhe Song.  Generalising fine-grained sketch-based image retrieval.  In CVPR, 2019.
  * [29] Kaiyue Pang, Yi-Zhe Song, Tony Xiang, and Timothy M Hospedales.  Cross-domain generative learning for fine-grained sketch-based image retrieval.  In BMVC, 2017.
  * [30] Kaiyue Pang, Yongxin Yang, Timothy M Hospedales, Tao Xiang, and Yi-Zhe Song.  Solving mixed-modal jigsaw puzzle for fine-grained sketch-based image retrieval.  In CVPR, 2020.
  * [31] Hieu Pham, Qizhe Xie, Zihang Dai, and Quoc V Le.  Meta pseudo labels.  arXiv preprint arXiv:2003.10580, 2020.
  * [32] Filip Radenovic, Giorgos Tolias, and Ondrej Chum.  Deep shape matching.  In ECCV, 2018.
  * [33] Umar Riaz Muhammad, Yongxin Yang, Yi-Zhe Song, Tao Xiang, and Timothy M Hospedales.  Learning deep sketch abstraction.  In CVPR, 2018.
  * [34] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  IJCV, 2015.
  * [35] Aneeshan Sain, Ayan Kumar Bhunia, Yongxin Yang, Tao Xiang, and Yi-Zhe Song.  Cross-modal hierarchical modelling for fine-grained sketch based image retrieval.  In BMVC, 2020.
  * [36] Aneeshan Sain, Ayan Kumar Bhunia, Yongxin Yang, Tao Xiang, and Yi-Zhe Song.  Stylemeup: Towards style-agnostic sketch-based image retrieval.  In CVPR, 2021.
  * [37] Leo Sampaio Ferraz Ribeiro, Tu Bui, John Collomosse, and Moacir Ponti.  Sketchformer: Transformer-based representation for sketched structure.  In CVPR, 2020.
  * [38] Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays.  The sketchy database: learning to retrieve badly drawn bunnies.  ACM TOG, 2016.
  * [39] Firas Shama, Roey Mechrez, Alon Shoshan, and Lihi Zelnik-Manor.  Adversarial feedback loop.  In ICCV, 2019.
  * [40] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  arXiv preprint arXiv:2001.07685, 2020.
  * [41] Jifei Song, Kaiyue Pang, Yi-Zhe Song, Tao Xiang, and Timothy M Hospedales.  Learning to sketch with shortcut cycle consistency.  In CVPR, 2018.
  * [42] Jifei Song, Yi-Zhe Song, Tony Xiang, and Timothy M Hospedales.  Fine-grained image retrieval: the text/sketch input dilemma.  In BMVC, 2017.
  * [43] Jifei Song, Qian Yu, Yi-Zhe Song, Tao Xiang, and Timothy M Hospedales.  Deep spatial-semantic attention for fine-grained sketch-based image retrieval.  In ICCV, 2017.
  * [44] Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour.  Policy gradient methods for reinforcement learning with function approximation.  In NeurIPS, 2000.
  * [45] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In CVPR, 2016.
  * [46] Ronald J Williams.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  Machine Learning, 1992.
  * [47] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio.  Show, attend and tell: Neural image caption generation with visual attention.  In ICML, 2015.
  * [48] I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan.  Billion-scale semi-supervised learning for image classification.  arXiv preprint arXiv:1905.00546, 2019.
  * [49] Sasi Kiran Yelamarthi, Shiva Krishna Reddy, Ashish Mishra, and Anurag Mittal.  A zero-shot framework for sketch based image retrieval.  In ECCV, 2018.
  * [50] Aron Yu and Kristen Grauman.  Fine-grained visual comparisons with local learning.  In CVPR, 2014.
  * [51] Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu.  Seqgan: Sequence generative adversarial nets with policy gradient.  In AAAI, 2017.
  * [52] Qian Yu, Feng Liu, Yi-Zhe Song, Tao Xiang, Timothy M Hospedales, and Chen-Change Loy.  Sketch me that shoe.  In CVPR, 2016.
  * [53] Qian Yu, Yongxin Yang, Feng Liu, Yi-Zhe Song, Tao Xiang, and Timothy M Hospedales.  Sketch-a-net: A deep neural network that beats humans.  IJCV, 2017.
  * [54] Jingyi Zhang, Fumin Shen, Li Liu, Fan Zhu, Mengyang Yu, Ling Shao, Heng Tao Shen, and Luc Van Gool.  Generative domain-migration hashing for sketch-to-image retrieval.  In ECCV, 2018.

Supplementary material for
More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch
Based Image Retrieval

##  Appendix A Necessity for rasterization

This is common practice in the FG-SBIR literature. Rasterized sketch-images
tend to have better spatial encoding than coordinate-based alternatives [28,
5]. This is also verified in our work – removing rasterization and using
sketch-coordinate retrieval reduces acc@1 to only 7.6% on Shoe-V2.

##  Appendix B Motivation behind using discriminator for certainty score:

We are mostly inspired by recent image generation works [39, 19] that use the
discriminator scores to iteratively improve generation quality. We also did an
ablative study to investigate further (see L768-785 and Fig. 4(a)). We found
that synthetic sketch-photo pairs having higher discriminator score, tend to
have much better quality, and vice-versa. We will add some qualitative
examples to further illustrate this correlation in supplementary materials.
Defining a hard threshold (optimised) to eliminate bad generated sketches is
an option – new experiments show acc@1 lags by around 2% compared to ours on
Shoe-V2.

##  Appendix C More details on experimental setup and analysis:

(i) Our self-designed baselines use the same backbone network, while joint-
training is employed for Ours-F-Pix2Pix, Ours-F-L2S and Ours-F-Full.

(ii) SOTA data-augmentation strategies are already adopted by existing FG-SBIR
works [28, 35]. However, they fail to capture the significant style variations
that exist in real human sketches. In fact we already compare with [52] which
employed such sketch specific augmentation strategies, and it is found to be
much inferior to our semi-supervised framework (see Table. 2).

(iii) Optimising the final layer (using Eq. 9 in our case) is a very common
practice during fine-tuning with RL, and is heavily adopted by the image-
captioning literature [14], and very recently by on-the-fly FG-SBIR [5].

(iv) Edge-map hardly resembles the highly abstracted and subjective nature of
amateur human sketches. For example, sketches do not follow the perfect edge
boundary unlike edge-maps, thus model trained on pseudo-sketches via
edge2sketch [33] falls short to generalise to real human sketches.

(v) Acc@1 without using RL scheme for Ours-F-Pix2Pix is $34.14\%$.

(vi) In future, our photo-to-sketch generation model could further be
evaluated on Sketchy [38], however, it seems to be comparatively difficult
than that of QMUL-ShoeV2 due to more noisy background.