# Geometric Additivity of Modular Commutator for Multipartite Entanglement

Sung-Min Park  Department of Physics, Korea Advanced Institute of Science and
Technology, Daejeon 34141, Korea  Isaac H. Kim corresponding author
<EMAIL_ADDRESS>Department of Computer Science, University of California,
Davis, CA 95616, USA  Eun-Gook Moon corresponding author<EMAIL_ADDRESS>Department of Physics, Korea Advanced Institute of Science and Technology,
Daejeon 34141, Korea

###### Abstract

A recent surge of research in many-body quantum entanglement has uncovered
intriguing properties of quantum many-body systems. A prime example is the
modular commutator, which can extract a topological invariant from a single
wave function. Here, we unveil novel geometric properties of many-body
entanglement via a modular commutator of two-dimensional gapped quantum many-
body systems. We obtain the geometric additivity of a modular commutator,
indicating that modular commutator for a multipartite system may be an integer
multiple of the one for tripartite systems. Using our additivity formula, we
also derive a curious identity for the modular commutators involving
disconnected intervals in a certain class of conformal field theories. We
further illustrate this geometric additivity for both bulk and edge subsystems
using numerical calculations of the Haldane and $\pi$-flux models.

#### Introduction:

Two-dimensional gapped quantum systems exhibit intriguing many-body
entanglement phenomena [1, 2]. A remarkable aspect of these systems is the
bulk-edge correspondence, which dictates that the effective field theory at
the boundary determines the universal properties of the bulk. A prime example
is the quantum Hall effect in two spatial dimensions [3], where a nonzero bulk
Chern number indicates the presence of gapless edge modes [4, 5]. These
gapless edge modes can be characterized by the chiral central charge
($c_{-}$), which appears in the zero-temperature limit of the thermal Hall
conductivity [6, 7, 5]. The bulk-edge correspondence suggests that the bulk
ground state wave function may capture the chiral central charge, although how
to extract it had remained puzzling.

Recently, Refs. [8, 9, 10, 11, 12] demonstrated that the modular commutator of
the ground state can capture the chiral central charge. Let us recall the
definition of a modular commutator

$\displaystyle J(A,B,C)$ $\displaystyle\equiv$ $\displaystyle
i\,\mathrm{Tr}(\rho_{ABC}[K_{AB},K_{BC}]),$

where $K_{X}\equiv-\ln\rho_{X}$ is the modular Hamiltonian of the reduced
density matrix $\rho_{X}$ on region $X$. For the ground state of a two-
dimensional gapped system, $|\Psi\rangle$, the modular commutator gives the
chiral central charge [8, 9, 10, 11, 12]:

$J(A,B,C)=\frac{\pi}{3}c_{-}.$ (1)

A tripartition $ABC$ with a complete tri-junction is illustrated in Fig. 1(a).
This result can be viewed as a part of an ongoing research program that aims
to study universal properties of the underlying quantum many-body system via
multipartite entanglement [13, 14, 15, 16, 17, 18, 19]. These recent
developments call for further studies to explore the multipartite entanglement
properties of many-body quantum systems.

In this paper, we extend the applicability of the modular commutator to more
general geometries. Our work unveils a new universal geometric identity for
the modular commutator. In particular, we find the curious identity that
involves disconnected intervals in conformal field theories (CFTs) and
singular regions in the bulk. The geometric aspects of the modular commutator
of multipartite entanglement are illustrated by employing the area law of
entanglement entropy [20, 21] and numerical calculations of lattice models.

Figure 1: Exemplary partitions in two-dimensional systems. (a) Tripartite bulk
disk $ABC$ and edge annulus $XYZ$. The shaded region represents the physical
edge of the system. (b) Multipartite bulk partition with the subregions
$U,V,W$. Two tri-junctions of ($U$, $V$, $W$) consist of two set of ‘balls’,
$\mathfrak{B}_{1}=\mathfrak{b}_{1}\cup\mathfrak{d}_{1}$ and
$\mathfrak{B}_{2}=\mathfrak{b}_{2}\cup\mathfrak{d}_{2}$.

#### Geometric Additivity:

Originally, the modular commutator [8, 9] was defined over a region
partitioned into three disks, all meeting at a single tri-junction. We
consider a more general partition that goes beyond this original setup. An
example of such a system is shown in Fig. 1(b), which features two tri-
junctions. We find that the modular commutator for such subsystems can be
expressed as:

$\displaystyle
J(U,V,W)=\sum_{i}J(U_{\mathfrak{B}_{i}}\\!,V_{\mathfrak{B}_{i}}\\!,W_{\mathfrak{B}_{i}})+J(U_{\mathfrak{r}},V_{\mathfrak{r}},W_{\mathfrak{r}}),$
(2)

where an index $i$ specifies a tri-junction of a subregion $UVW$. At each tri-
junction, $V_{\mathfrak{B}_{i}}=V\cap\mathfrak{B}_{i}$ is for the intersection
between a subregion $V$ and a ball $\mathfrak{B}_{i}$. For an each ball
$\mathfrak{B}_{i}$, a smaller ball $\mathfrak{b}_{i}$ and an associated
annulus region $\mathfrak{d}_{i}$ are introduced to zoom in a tri-junction,
$\mathfrak{B}_{i}=\mathfrak{b}_{i}\cup\mathfrak{d}_{i}$. A relative complement
region of all the tri-junctions is defined as
$V_{\mathfrak{r}}=V\setminus(\cup_{i}\mathfrak{b}_{i})$, which contributes to
the residual term, $J(U_{\mathfrak{r}},V_{\mathfrak{r}},W_{\mathfrak{r}})$. It
then follows that
$J(U_{\mathfrak{B}_{i}},V_{\mathfrak{B}_{i}},W_{\mathfrak{B}_{i}})$ is
quantized as in Eqn. (1) for a complete junction, while an incomplete junction
gives a non-quantized value. See Supplemental Materials (SM) for more details.

The modular commutator satisfies _geometric additivity_ if the residual term
vanishes. We prove that the residual term indeed vanishes for invertible
states by using properties of quantum Markov chains and the area law of the
entanglement entropy [20, 21]. For non-invertible states (e.g., states that
can host anyons), it is an open question whether the geometric additivity
holds in general. We provide a conjecture from which the additivity would
follow, though the proof of it remains open; see SM for more details.

Figure 2: Bulk and edge pizza partitions and corresponding multipartite
subsystems. (a),(c) The bulk pizza partitions of $ABC$. (b),(d) The
corresponding multipartite subsystems of (a) and (c), respectively. (e),(g)
The edge pizza partitions of $XYZ$. (f),(h) The corresponding multipartite
subsystems of (e) and (g), respectively.

#### Applications:

One of the main applications of geometric additivity is an exact calculation
of the modular commutator for new types of subsystems. As an example, we
consider a partition in Fig. 2(a), which we call as the pizza partition. Let
us first consider a bulk system $CDA$ in Fig. 2(b) with the complementary
property $K_{X}|\Psi\rangle=K_{\bar{X}}|\Psi\rangle$. The modular commutator
for $CDA$ becomes

$\displaystyle J(C,D,A)=$ $\displaystyle
i\langle\Psi|[K_{ABE},K_{BCE}]|\Psi\rangle$ $\displaystyle=$ $\displaystyle
i\langle\Psi|[K_{AB},K_{BC}]|\Psi\rangle=J(A,B,C),$ (3)

where in the second line, we use the fact that the state on the two
sufficiently distant regions is a product state; for example,
$K_{EAB}=K_{E}+K_{AB}$. Thus, we can use $J(C,D,A)$ to determine $J(A,B,C)$.
Using the additivity formula we obtain

$J(A,B,C)=2\times\frac{\pi}{3}c_{-}$ (4)

for Fig. 1(a). In essence, each complete tri-junction of $J(C,D,A)$
contributes to the modular commutator (by $\frac{\pi}{3}c_{-}$), yielding a
result that is twice as large as the one for the tripartition in Fig. 1(a).
Similarly, one can apply the additivity formula to the pizza partition in Fig.
2(c), and show $J(A,B,C)=0$. This is because the contributions from the two
tri-junctions cancel each other out.

The additivity formula may also be applied to the physical edge of the system.
The similar use of complementary property gives $J(X,Y,Z)=J(Z,W,X)$, and one
can apply the additivity to the complement region. For example, for the pizza
partition in Fig. 2(e), we find

$\displaystyle J(X,Y,Z)=-2\times\frac{\pi}{3}c_{-}.$ (5)

We note that the modular Hamiltonians involved in the calculation of
$J(X,Y,Z)$ are associated with disconnected intervals (e.g., $XY$ and $YZ$).
This is very different from the setup considered in Ref. [10], which only
involved intervals. While the entanglement Hamiltonian for an interval is
local [22], the entanglement Hamiltonian for disconnected intervals is not
[23]. Our result shows that, in spite of this nonlocality, its modular
commutator result in a substantially simpler form. However, our derivation
only applies only to the edge theory of invertible states. It is currently
unclear if Eqn. (5) holds for any 1+1D CFTs.

Figure 3: The geometric integer $n$ for the various bulk and edge pizza
partitions.

The geometric additivity can be further applied to various pizza partitions as
in Fig. 3. It is straightforward to show that the modular commutators of a
pizza partition yield

$J(A,B,C)=-J(X,Y,Z)=\frac{\pi}{3}\big{(}c_{-}\times n\big{)},\quad
n\in\mathbb{Z},$ (6)

where the arrangement of the bulk and edge pizza partition, $ABC$ and $XYZ$,
are topologically equivalent. The chiral central charge $c_{-}$ is determined
by a ground state $|\Psi\rangle$, and the integer $n$ depends on the
arrangement near the tri-junction. We refer to it as the geometric integer in
this work. We conjecture that Eqn. (6) holds for most two-dimensional gapped
systems, although our proof is limited to invertible states.

Lastly, we discuss the application of the additivity formula for incomplete
junctions. An incomplete tri-junction generally exhibits non-quantized values
dependent on microscopic details [11]. Yet, we find that the modular
commutator with an incomplete junction has intriguing complementary
properties. Namely, for a disk-like region $ABCD$, the sum of two modular
commutators with incomplete junctions yields the following complementary
relation:

$\includegraphics[width=35.56593pt,valign={c}]{ID.pdf}\;\;\;\;\;\;J(A,B,C)+J(B,C,D)=\frac{\pi}{3}c_{-},$
(7)

where the subregion $A,B,C,D$ meet at a point. This is an immediate
consequence of geometric additivity. Interestingly, this identity holds even
for non-invertible states, i.e., topologically ordered states that can host
anyons; see the SM. Note that this complementary relation can be viewed as a
bulk analog of the CFT identity recently discovered in Refs. [10, 24, 11]. A
particular application of this identity arises when the system exhibits proper
spatial symmetry near its tri-junction. This allows us to extract the chiral
central charge within a smaller system size, yielding a half-quantized value:
$J(A,B,C)_{\sigma}=J(B,C,D)_{\sigma}=\frac{1}{2}\times\frac{\pi}{3}c_{-}$.

Figure 4: Schematic description of the Haldane model (a) Upside-down triangles
($\bigtriangledown$) and triangles ($\bigtriangleup$) represent sublattices
$a$ and $b$, respectively. See the main text for the information on hopping
parameters. (b) The phase diagram of the Haldane model. In the chern insulator
phase at $-1<\tilde{\mu}<1$, chiral central charge is $c_{-}=1$, otherwise
zero.

#### Numerical calculation:

We numerically verify the geometric additivity for certain lattice models. We
consider the Haldane model on the honeycomb lattice [25], whose Hamiltonian
consists of three parts:

$H_{H}=H_{0}(\mu)+H_{1}(t_{1})+H_{2}(t_{2};\phi).$ (8)

The first term contains the on-site energy terms with strength $\mu$. We set
the onsite energy to be $\mu$ for $a$ sublattice and $-\mu$ for $b$
sublattice:
$H_{0}(\mu)=\mu\sum_{r}(c^{\dagger}_{r,a}c_{r,a}-c^{\dagger}_{r,b}c_{r,b})$.
The second term $H_{1}$ includes the nearest-neighbor hopping terms with
amplitude $t_{1}$, which are represented as black solid links in Fig. 4(a):
$H_{1}(t_{1})=t_{1}\sum_{\langle j,k\rangle}c^{\dagger}_{j}c_{k}$. The last
term has the next nearest neighbor hopping terms with complex amplitude
$t_{2}$ with phase $\phi$, which breaks the time-reversal symmetry:
$H_{2}(t_{2};\phi)=|t_{2}|\sum_{\langle\langle
j,k\rangle\rangle}e^{-i\phi\nu_{jk}}c^{\dagger}_{j}c_{k}$. Here, $\nu_{jk}$
are $\pm 1$ and depends on the arrow’s direction. The dashed arrows in Fig.
4(a) denote the next nearest hopping. The hopping in the direction of the
arrow accumulates the flux $\phi$. Setting a tuning parameter
$\tilde{\mu}=\mu/(3\sqrt{3}t_{2})$ with $t_{1}=t_{2}=1$, we choose a one-
dimensional path along $\phi=\pi/2$ where two topological phase transitions
are present as in Fig. 4(b). It is well known that the Haldane model has the
chiral central charge $c_{-}=1$ for $-1<\tilde{\mu}<1$ and $c_{-}=0$ for
$\tilde{\mu}>1$ and $\tilde{\mu}<-1$.

To evaluate its modular commutator, we divide the bulk lattice system into six
sectors for a pizza partition. For example, a lattice realization of the bulk
$ABC$ is shown in Fig. 4(a). Note that the total number of lattice points in
the pizza partition is $867$, which makes each sector large enough except at
the critical points at $\tilde{\mu}=\pm 1$.

The exact ground state of the Haldane model, $|\psi_{H}\rangle$, is readily
obtained with a given lattice, and its modular commutator is numerically
evaluated,

$\displaystyle
n_{H}\equiv\frac{3}{\pi}J(A,B,C)=i\frac{3}{\pi}\langle\psi_{H}|[K_{AB},K_{BC}]|\psi_{H}\rangle.$
(9)

The numerical values of $n_{H}$ for the bulk pizza partitions, Fig. 5(a)-(d),
are illustrated in Fig. 5(i) while the ones of
$n_{H}=-\frac{3}{\pi}i\langle\psi_{H}|[K_{XY},K_{YZ}]|\psi_{H}\rangle$ for the
edge pizza partitions, Fig. 5(e)-(f), are illustrated in Fig. 5(j). We
emphasize that the numerical calculations of $n_{H}$ are precisely matched
with the results of the geometric additivity away from the critical points
($\tilde{\mu}=\pm 1$).

Figure 5: Numerical evaluation of the geometric integer ($n_{H}$) for the
pizza partitions. (a)-(h) The lattice realizations of the pizza partitions. We
fix the linear system size $L$, bulk subsystem size $r$, and the edge width
$w$: $L=24$, $r=17$, and $w=13$. (i) Plot of $n_{H}$ for the bulk pizza
partitions with the tuning parameter $\tilde{\mu}$. (j) Plot of $n_{H}$ for
the edge pizza partitions with the tuning parameter $\tilde{\mu}$.

To check the validity of our results, we also introduce small disorder to our
numerical calculations by adding the Anderson term [26],
$\sum_{j}V_{j}c^{\dagger}_{j}c_{j}$, where $V_{j}$ is sampled from a uniform
distribution within the range $[-W/2,W/2]$. We choose $W$ as 10% of the bulk
energy gap, and no differences in the numerical calculations are found. We
also perform similar calculations for the $\pi$-flux model on the square
lattice and obtain qualitatively similar results for the higher geometric
integers $n_{\pi}=3,4$. These results are presented in the SM.

#### Discussion and conclusion:

Intuitively, the geometric integer $n$ can be understood in terms of the
modular current [5, 8, 9]. For any two regions $L$ and $R$, this is defined as
$f(L,R)=\sum_{v\in L}\sum_{u\in R}\tilde{f}_{vu}$, where
$\tilde{f}_{vu}=i\langle[\tilde{K}_{v},\tilde{K}_{u}]\rangle$ and
$\tilde{K}_{v}$ is the local modular Hamiltonian such that $K_{X}=\sum_{v\in
X}\tilde{K}_{v}$. It is well understood that non-trivial modular current flows
along the boundaries of nearby regions [8, 9].

We can apply this intuition to the bulk pizza partitions, reproducing the
results we have shown rigorously. Due to the locality of the bulk modular
Hamiltonian, the modular commutator may be rewritten as

$J(A,B,C)=\sum_{v\in AB}\sum_{u\in BC}\tilde{f}_{vu}.$ (10)

Thus, the modular current can be evaluated as a sum of current flowing from a
subregion of $AB$ to another subregion of $BC$. For instance, consider the
partition of Fig. 6(a), where the subregion is written as
$ABC=A_{1}B_{1}C_{1}A_{2}B_{2}C_{2}$. The modular current has contributions
from the four boundaries, $(A_{1},B_{1})$, $(B_{1},C_{1})$, $(A_{2},B_{2})$,
$(B_{2},C_{2})$. Thus, the geometric integer is twice as large as the one of
Fig. 6(e). We can apply a similar reasoning to the other pizza partitions,
which yields results consistent with our analysis based on the geometric
additivity.

Figure 6: The modular current interpretation for the pizza partitions. The
non-trivial modular current of $J(A,B,C)$ flows from the subregions of $AB$
into the subregions $BC$ in the vicinity of their boundaries. (a)-(d) The
pizza partitions of Fig. 2. (e) The pizza partition of the typical tri-
junction as in Fig. 1(a).

To conclude, we posit a new geometric property of many-body quantum
entanglement in two-dimensional gapped quantum many-body systems. This is the
geometric additivity of the modular commutator, which indicates that modular
commutator for subsystems more general than the one considered in Ref. [8, 9,
14] can be decomposed into a sum of modular commutators over simplified
subsystems (e.g., involving balls or intervals). Numerical calculations of the
Haldane model corroborate the geometric additivity, both in the bulk and at
the edge.

We remark that a recent work [27] reported spurious contributions of modular
commutators. In our numerical calculations, we have not observed such
contributions, which is consistent with the previous work on the chiral
central charge of free fermion models [12]. In future works, it would be
desirable to develop a different method for extracting the chiral central
charge which is free of such problems. One interesting question is whether the
additivity can be proved rigorously for free fermion models.

Another important question is whether the geometric additivity holds for non-
invertible states. We note that the additivity can be proved for a certain
(non-vacuum) reduced density matrix that contains anyons; see the SM. However,
how to relate this result to the state of interest, i.e., the vacuum-reduced
density matrix, is unclear at the moment.

#### Acknowledgement:

We thank K. Hwang, J. McGreevy, X. Li, T.-C. Lin, and B. Shi for helpful
discussions. IK acknowledges support from NSF under award number PHY-2337931.
S.-M.P. and E.-G.M. were supported by 2021R1A2C4001847, 2022M3H4A1A04074153,
National Measurement Standard Services and Technical Services for SME funded
by Korea Research Institute of Standards and Science (KRISS – 2024 –
GP2024-0015) and the Nano & Material Technology Development Program through
the National Research Foundation of Korea(NRF) funded by Ministry of Science
and ICT(RS-2023-00281839).

## References

  * Sachdev [2023] S. Sachdev,  _Quantum Phases of Matter_ (Cambridge University Press, 2023).
  * Wen [2004] X.-G. Wen,  _Quantum field theory of many-body systems: from the origin of sound to an origin of light and electrons_ (OUP Oxford, 2004).
  * Klitzing  _et al._ [1980] K. v. Klitzing, G. Dorda, and M. Pepper, New method for high-accuracy determination of the fine-structure constant based on quantized hall resistance, Phys. Rev. Lett. 45, 494 (1980).
  * Hatsugai [1993] Y. Hatsugai, Chern number and edge states in the integer quantum hall effect, Phys. Rev. Lett. 71, 3697 (1993).
  * Kitaev [2006] A. Kitaev, Anyons in an exactly solved model and beyond, Annals of Physics 321, 2 (2006), january Special Issue.
  * Kane and Fisher [1997] C. L. Kane and M. P. A. Fisher, Quantized thermal transport in the fractional quantum hall effect, Phys. Rev. B 55, 15832 (1997).
  * Read and Green [2000] N. Read and D. Green, Paired states of fermions in two dimensions with breaking of parity and time-reversal symmetries and the fractional quantum hall effect, Phys. Rev. B 61, 10267 (2000).
  * Kim  _et al._ [2022a] I. H. Kim, B. Shi, K. Kato, and V. V. Albert, Chiral central charge from a single bulk wave function, Phys. Rev. Lett. 128, 176402 (2022a).
  * Kim  _et al._ [2022b] I. H. Kim, B. Shi, K. Kato, and V. V. Albert, Modular commutator in gapped quantum many-body systems, Phys. Rev. B 106, 075147 (2022b).
  * Zou  _et al._ [2022] Y. Zou, B. Shi, J. Sorce, I. T. Lim, and I. H. Kim, Modular commutators in conformal field theory, Phys. Rev. Lett. 129, 260402 (2022).
  * Fan [2022] R. Fan, From entanglement generated dynamics to the gravitational anomaly and chiral central charge, Phys. Rev. Lett. 129, 260403 (2022).
  * Fan  _et al._ [2023a] R. Fan, P. Zhang, and Y. Gu, Generalized real-space Chern number formula and entanglement hamiltonian, SciPost Phys. 15, 249 (2023a).
  * Dehghani  _et al._ [2021] H. Dehghani, Z.-P. Cian, M. Hafezi, and M. Barkeshli, Extraction of the many-body chern number from a single wave function, Phys. Rev. B 103, 075102 (2021).
  * Zou  _et al._ [2021] Y. Zou, K. Siva, T. Soejima, R. S. K. Mong, and M. P. Zaletel, Universal tripartite entanglement in one-dimensional many-body systems, Phys. Rev. Lett. 126, 120501 (2021).
  * Siva  _et al._ [2022] K. Siva, Y. Zou, T. Soejima, R. S. K. Mong, and M. P. Zaletel, Universal tripartite entanglement signature of ungappable edge states, Phys. Rev. B 106, L041107 (2022).
  * Fan  _et al._ [2023b] R. Fan, R. Sahay, and A. Vishwanath, Extracting the quantum hall conductance from a single bulk wave function, Phys. Rev. Lett. 131, 186301 (2023b).
  * Kobayashi  _et al._ [2024a] R. Kobayashi, T. Wang, T. Soejima, R. S. K. Mong, and S. Ryu, Extracting higher central charge from a single wave function, Phys. Rev. Lett. 132, 016602 (2024a).
  * Kobayashi  _et al._ [2024b] R. Kobayashi, T. Wang, T. Soejima, R. S. Mong, and S. Ryu, Higher hall conductivity from a single wave function: Obstructions to symmetry-preserving gapped edge of (2+1) d topological order, arXiv preprint arXiv:2404.10814 (2024b).
  * Liu [2024] S. Liu, Anyon quantum dimensions from an arbitrary ground state wave function, Nature Communications 15, 5134 (2024).
  * Kitaev and Preskill [2006] A. Kitaev and J. Preskill, Topological entanglement entropy, Phys. Rev. Lett. 96, 110404 (2006).
  * Levin and Wen [2006] M. Levin and X.-G. Wen, Detecting topological order in a ground state wave function, Phys. Rev. Lett. 96, 110405 (2006).
  * Cardy and Tonni [2016] J. Cardy and E. Tonni, Entanglement hamiltonians in two-dimensional conformal field theory, Journal of Statistical Mechanics: Theory and Experiment 2016, 123103 (2016).
  * Arias  _et al._ [2018] R. E. Arias, H. Casini, M. Huerta, and D. Pontello, Entropy and modular hamiltonian for a free chiral scalar in two intervals, Phys. Rev. D 98, 125008 (2018).
  * Kim  _et al._ [2024] I. H. Kim, X. Li, T.-C. Lin, J. McGreevy, and B. Shi, Conformal geometry from entanglement, arXiv preprint arXiv:2404.03725 (2024).
  * Haldane [1988] F. D. M. Haldane, Model for a quantum hall effect without landau levels: Condensed-matter realization of the” parity anomaly”, Physical review letters 61, 2015 (1988).
  * Anderson [1958] P. W. Anderson, Absence of diffusion in certain random lattices, Phys. Rev. 109, 1492 (1958).
  * Gass and Levin [2024] J. Gass and M. Levin, Many-body systems with spurious modular commutators, arXiv preprint arXiv:2405.15892 (2024).
  * Shi  _et al._ [2020] B. Shi, K. Kato, and I. H. Kim, Fusion rules from entanglement, Annals of Physics 418, 168164 (2020).
  * Note [1] Intuitively, it might be convenient to view this state as a ground state of a gapped Hamiltonian, though we do not make use of that fact.
  * Petz [2003] D. Petz, Monotonicity of quantum relative entropy revisited, Reviews in Mathematical Physics 15, 79 (2003).
  * Peschel and Eisler [2009] I. Peschel and V. Eisler, Reduced density matrices and entanglement entropy in free lattice models, Journal of physics a: mathematical and theoretical 42, 504003 (2009).
  * Note [2] Note that the subsystem $CDA$ is not a true two-hole disk, and its information convex set is not well-defined due to its bowtie shape ($\bowtie$). However, conditional independence remains true because of the strong subadditivity.

###### Contents

  1. A Geometric additivity of a modular commutator
    1. A.1 Setup: area law of the entanglement entropy
    2. A.2 Derivation: geometric additivity formula
  2. B Geometric additivity of modular commutator for the incomplete disk
    1. B.1 Summary of the results
    2. B.2 Numerical Calculations and Remarks
  3. C Additional numerical calculations for $\pi$-flux model
    1. C.1 Model Hamiltonian
    2. C.2 Numerical results
  4. D Absence of the residual term in the invertible states and remarks on the non-invertible states
    1. D.1 Proof: Absence of the residual term in the invertible bulk
    2. D.2 Discussion: non-invertible bulk

The Supplementary Material is organized as follows. Section A provides the
basic setup and a derivation of the geometric additivity formula. We provide
another type of additivity formula for the incomplete disk in Section B.
Section C presents results of numerical calculations on the $\pi$-flux model,
including incomplete disk and $n\geq 3$ disconnected edge intervals. In
Section D, we discuss the modular commutator of the pizza partition for
invertible states and also provide remarks for the non-invertible states.

##  Appendix A Geometric additivity of a modular commutator

In this Section, we derive the geometric additivity formula using the area law
of entanglement entropy [20, 21], or equivalently, the entanglement bootstrap
axiom A1 [28].

###  A.1 Setup: area law of the entanglement entropy

Figure 7: Schematic descriptions on the two entropic axioms. (a) A gapped
quantum many-body system $\Lambda$, and the bulk ‘ball’ regions $\mathfrak{B}$
lying in the bulk. The solid lines at the edge represent the physical
boundary, and the dashed lines represent the entanglement cut. (b) Two
entropic combinations under area law. Subregions $B$, $C$, and $D$ are
sufficiently larger than the bulk correlation length.

The entanglement bootstrap [28] starts with two axioms on the entanglement
entropy of the bulk ball $\mathfrak{B}$ [Fig. 7(b)]:

$\displaystyle\mathbf{A0}\\!:\;$ $\displaystyle\;\Delta(B,C)_{\sigma}=0,$
(11a) $\displaystyle\mathbf{A1}\\!:\;$
$\displaystyle\;\Delta(B,C,D)_{\sigma}=0,$ (11b)

where the subscripts $\sigma$ of the parentheses is a state satisfying the
axioms 111Intuitively, it might be convenient to view this state as a ground
state of a gapped Hamiltonian, though we do not make use of that fact. and
$\Delta$s are linear combinations of entanglement entropies:

$\displaystyle\Delta(B,C)_{\rho}$ $\displaystyle:=\\!$
$\displaystyle(S_{BC}+S_{C}-S_{B})_{\rho}\geq 0,$ (12a)
$\displaystyle\Delta(B,C,D)_{\rho}$ $\displaystyle:=\\!$
$\displaystyle(S_{BC}+S_{CD}\\!-S_{B}\\!-S_{D})_{\rho}\\!\geq\\!0.$ (12b)

Here, the subscript represents the relevant subsystem. For instance,
$(S_{BC}+\ldots)_{\rho}$ is a shorthand notation for $S(\rho_{BC})+\ldots$,
where $S(\rho_{BC})=-\text{Tr}(\rho_{BC}\ln\rho_{BC})$ is the von Neumann
entropy. Note that two entropic combinations are always non-negative by the
Araki-Lieb inequality and strong subadditivity (SSA).

An immediate consequence of Eqns. (11) is that for any subsystem
$A\subset\Lambda\setminus\mathfrak{B}$, $\sigma_{ABC}$ forms a quantum Markov
chain (with respect to the choice of $B$ and $C$ appearing in Eqns. (12)). In
other words, the conditional mutual information of a tripartite state
$\sigma_{ABC}$ — defined as
$I(A:C|B)_{\rho}:=(S_{AB}+S_{BC}-S_{B}-S_{ABC})_{\rho}.$ — vanishes:

$\displaystyle I(A:C|B)_{\sigma}=0.$ (13)

Moreover, the modular Hamiltonian ($K_{X}\equiv-\ln\rho_{X}$) for a quantum
Markov states $\sigma_{ABC}$ can be decomposed into the following form [30]:

$I(A:C|B)=0\Longleftrightarrow K_{ABC}=K_{AB}+K_{BC}-K_{B}.$ (14)

By repeatedly applying the same argument, one can often decompose the modular
Hamiltonian as a linear combination of terms that act on increasingly smaller
subsystems [9].

###  A.2 Derivation: geometric additivity formula

In this subsection, we prove the geometric additivity of the modular
commutator,

$J(U,V,W)\\!=\\!\sum_{i}J(U_{\mathfrak{B}_{i}}\\!,V_{\mathfrak{B}_{i}}\\!,W_{\mathfrak{B}_{i}})+J(U_{\mathfrak{r}},V_{\mathfrak{r}},W_{\mathfrak{r}}),$
(15)

using entanglement bootstrap axiom A1 [Eqn. (11b)]. Throughout this
subsection, we will use the notation summarized in Table. 1 and suppress the
notation for $\sigma$ unless it is unclear.

For concreteness, we consider an exemplary multipartite region $U$, $V$, and
$W$ [Fig. 8(a)]. This will be our guiding example for proving the additivity
formula. We note that these specific regions are chosen only for pedagogical
purposes. The proof itself can be generalized straightforwardly to other
choices of subsystems. This is because our approach is to sequentially excise
a ball that includes a tri-junction one by one; for any choice of subsystems,
one can simply repeat the same procedure, arriving at the additivity formula
[Eqn. (15)].

$\mathfrak{b}_{j}$ | a ball centered at $j$-th tri-junction
---|---
$\mathfrak{d}_{j}$ | an annulus surrounding $\mathfrak{b}_{j}$, $\mathfrak{d}_{j}=\mathfrak{d}_{1,j}\cup\mathfrak{d}_{2,j}$
$\mathfrak{B}_{j}$ | Union of $\mathfrak{b}_{j}$ and $\mathfrak{d}_{j}$, $\mathfrak{B}_{j}=\mathfrak{b}_{j}\cup\mathfrak{d}_{j}$
$\mathfrak{r}_{j}$ | The relative complement of $\mathfrak{b}_{j}$, e.g. $\mathfrak{r}_{j}=\Lambda/\mathfrak{b}_{j}$
$\mathfrak{R}_{j}$ | The relative complement of $\mathfrak{B}_{j}$, e.g. $\mathfrak{R}_{j}=\Lambda/\mathfrak{B}_{j}$
Table 1: Summary of notations (See also Fig. 8(b))

The following are the key observations. First, we can apply axiom A1 to a ball
containing the tri-junction and thus decompose the modular Hamiltonian of $UV$
and $VW$ near the tri-junction [Eqn. (14)]. For instance, consider the $i$-th
tri-junction of the subregion $VW$ and apply the axiom A1. Without loss of
generality, we can decompose the modular Hamiltonian of $VW$ near the tri-
junction, thanks to Eqn. (16):

$I((VW)_{\mathfrak{b}_{i}}:(VW)_{\mathfrak{d}_{2,i}\mathfrak{R}_{i}}|(VW)_{\mathfrak{d}_{1,i}})=0,$
(16)

where $\mathfrak{d}_{1,i}$ is the inner annulus surrounding
$\mathfrak{b}_{i}$, and $\mathfrak{d}_{2,i}$ is the outer annulus surrounding
$\mathfrak{d}_{1,i}$ [Fig. 8(b)]. The subscripts of $VW$ denote the
intersections. The second observation is that applying this decomposition
yields contributions from additional tri-junctions, which exactly cancel each
other out; see the two green-shaded regions in Fig. 8(b).

Figure 8:  (a) Examplary multi-partition $UVW$ in $UVW$ in two-dimensional
quantum many-body system $\Lambda$. (b) Schematic description of the $i$’th
ball region, $\mathfrak{B}_{i}=\mathfrak{b}_{i}\cup\mathfrak{d}_{i}$. The
black and green shaded region are tri-junctions in Eqn. (17).

We now elaborate on these observations in more detail, focusing on a tri-
junction of $UVW$. In what follows, we will obtain a succinct expression [Eqn.
(20)] for $J(U,V,W)$. We start by rewriting the modular commutator as

$\displaystyle J(U,V,W)$
$\displaystyle=i\langle[K_{UV},K_{(VW)_{\mathfrak{bd}_{1}}}]\rangle$
$\displaystyle-i\langle[K_{UV},K_{(VW)_{\mathfrak{d}_{1}}}]\rangle$ (17)
$\displaystyle+i\langle[K_{UV},K_{(VW)_{\mathfrak{r}}}]\rangle.$

All three terms are generally nonzero since they all have tri-junctions [green
shaded regions in Fig. 8(b)]. However, we can judiciously cancel out some of
those contributions using the quantum Markov chain, such as Eqn. (16).

Let us compute the first line of Eqn. (17).

$\displaystyle\langle[K_{UV},K_{(VW)_{\mathfrak{bd}_{1}}}]\rangle-$
$\displaystyle\langle[K_{UV},K_{(VW)_{\mathfrak{d}_{1}}}]\rangle$
$\displaystyle=$
$\displaystyle\langle[K_{(UV)_{\mathfrak{B}}},K_{(VW)_{\mathfrak{bd}_{1}}}$
$\displaystyle-K_{(VW)_{\mathfrak{d}_{1}}}]\rangle$ $\displaystyle=$
$\displaystyle\langle[K_{(UV)_{\mathfrak{B}}},K_{(VW)_{\mathfrak{B}}}]\rangle$
$\displaystyle-\langle[K_{(UV)_{\mathfrak{B}}},K_{(VW)_{\mathfrak{d}}}]\rangle,$
(18)

where in the second line of Eqn. (18) we reduce the region $UV$ into its
restriction onto the ball $\mathfrak{B}$, by using an identity
$\langle[K_{UV},K_{(VW)_{\mathfrak{bd}_{1}}}]\rangle=\langle[K_{(UV)_{\mathfrak{B}}},K_{(VW)_{\mathfrak{bd}_{1}}}]\rangle$.
Using a similar identity, we extend the region $\mathfrak{d}_{1}$ of $VW$ to
$\mathfrak{d}=\mathfrak{d}_{1}\cup\mathfrak{d}_{2}$ in the third line.

The second line of Eqn. (17) can be also calculated similarly:

$\displaystyle\langle[K_{UV},K_{(VW)_{\mathfrak{r}}}]\rangle=$
$\displaystyle\langle[K_{(UV)_{\mathfrak{bd}_{1}}}-K_{(UV)_{\mathfrak{d}_{1}}},K_{(VW)_{\mathfrak{r}}}]\rangle$
$\displaystyle+\langle[K_{(UV)_{\mathfrak{r}}},K_{(VW)_{\mathfrak{r}}}]\rangle$
$\displaystyle=$
$\displaystyle\langle[K_{(UV)_{\mathfrak{B}}}-K_{(UV)_{\mathfrak{d}}},K_{(VW)_{\mathfrak{d}}}]\rangle$
(19)
$\displaystyle+\langle[K_{(UV)_{\mathfrak{r}}},K_{(VW)_{\mathfrak{r}}}]\rangle,$

where in the first line, we used a decomposition of $K_{UV}$. For the first
term of the second equality, we first reduce the region $(VW)_{\mathfrak{r}}$
to $(VW)_{\mathfrak{d}}$ and then extend the $(UV)_{\mathfrak{bd}_{1}}$ to
$(UV)_{\mathfrak{B}}$.

Now that we have computed both lines of Eqn. (17), let us collect those terms
together. By combining Eqn. (18) and Eqn. (19), we obtain the following
decomposition for the $i$’th tri-junction:

$\displaystyle J(U,V,W)=$ $\displaystyle
J(U_{\mathfrak{B}_{i}},W_{\mathfrak{B}_{i}},W_{\mathfrak{B}_{i}})+J(U_{\mathfrak{r}_{i}},V_{\mathfrak{r}_{i}},W_{\mathfrak{r}_{i}})$
$\displaystyle-J(U_{\mathfrak{d}_{i}},V_{\mathfrak{d}_{i}},W_{\mathfrak{d}_{i}}).$
(20)

The only difference between Eqn. (20) and the additivity formula is [Eqn.
(15)] is the presence of an additional term
$J(U_{\mathfrak{d}_{i}},V_{\mathfrak{d}_{i}},W_{\mathfrak{d}_{i}})$.

We note that this additional term is zero:

$J(U_{\mathfrak{d}_{i}},V_{\mathfrak{d}_{i}},W_{\mathfrak{d}_{i}})=0.$ (21)

This is because the two modular Hamiltonians over $(UV)_{\mathfrak{d}_{i}}$
and $(VW)_{\mathfrak{d}_{i}}$ can be written as a linear combination of local
terms, whose resulting modular commutators are zero. (The modular commutators
are zero due to the quantum Markov chain structure.) Thus, we conclude Eqn.
(21). In particular, we obtain

$J(U,V,W)\\!=\\!J(U_{\mathfrak{B}_{i}}\\!,W_{\mathfrak{B}_{i}}\\!,W_{\mathfrak{B}_{i}})\\!+\\!J(U_{\mathfrak{r}_{i}}\\!,V_{\mathfrak{r}_{i}}\\!,W_{\mathfrak{r}_{i}}).$
(22)

This procedure can be repeatedly applied for every tri-junction of $J(U,V,W)$,
yielding the additivity formula:

$J(U,V,W)\\!=\\!J(U_{\mathfrak{r}}\\!,V_{\mathfrak{r}}\\!,W_{\mathfrak{r}})\\!+\\!\sum_{i}J(U_{\mathfrak{B}_{i}}\\!,V_{\mathfrak{B}_{i}}\\!,W_{\mathfrak{B}_{i}}),$

where $\mathfrak{r}=\cup_{i}\mathfrak{r}_{i}$.

Below, we provide some remarks. First, Eqn. (15) of the modular commutator
holds even for incomplete tri-junctions because the observation used in the
proof remains valid for incomplete junctions. Second, $J(U,V,W)$ is invariant
under smooth bulk deformations away from the tri-junction. Here, smooth bulk
deformation means deformations of subsystems within the modular commutator
that do not add or remove the tri-junction. Finally, the modular commutator
for the bulk residual term vanishes

$J(U_{\mathfrak{r}},V_{\mathfrak{r}},W_{\mathfrak{r}})=0,$

when the subsystems $(UV)_{\mathfrak{r}}$ and $(VW)_{\mathfrak{r}}$ are disk-
shaped regions. This follows from the observation in Ref. [9] that the modular
Hamiltonian for a disk-shaped bulk region can be expressed as a sum of local
modular Hamiltonians. Therefore, for the disk-like regions, the modular
commutator $J(U_{\mathfrak{r}},V_{\mathfrak{r}},W_{\mathfrak{r}})$ can be
expressed as a sum of local modular commutators. Each of these commutators is
a quantum Markov chain and always vanishes.

##  Appendix B Geometric additivity of modular commutator for the incomplete
disk

In this Section, we discuss another intriguing application of the additivity
formula [Eqn. (15)], the incomplete disk [11]. Let us consider a disk $ABCD$
shown in Fig. 9(a). Unlike previous cases, subregion $A,B,C$ does not fully
surround its tri-junction. We also call such tripartitions $ABC$ as the
incomplete disk.

###  B.1 Summary of the results

Figure 9: Schematic descriptions for incomplete tri-junction. (a) Bulk disk
$ABCD$ and edge annulus $XYZW$. (b) Partitions $(A,B,C,D,E,F)$ for the proof
of Eqn. (23).

For incomplete disks $ABC$ and $BCD$, we find another type of additivity for
modular commutators:

$J(A,B,C)+J(B,C,D)=\frac{\pi}{3}c_{-}.$ (23)

As an immediate consequence, four modular commutators of incomplete disk
$ABCD$ have the following relations: $J(A,B,C)=J(C,D,A)$ and
$J(B,C,D)=J(D,A,B)$.

To prove it, let us consider partitions $BCDE$ illustrated in Fig. 9(b). We
first switch the modular Hamiltonian of the subsystem $CDE$ to $ABF$ using the
complementary property, $K_{CDE}|\Psi\rangle=K_{ABF}|\Psi\rangle$,

$\displaystyle J(DE,C,B)=J(AF,B,C)=J(A,B,C).$ (24)

where in the second equality, we use $K_{ABF}=K_{AB}+K_{F}$. Then, we apply
the additivity formula to $J(DE,C,B)$, resulting in

$\displaystyle J(DE,C,B)=$ $\displaystyle
J(E_{\mathfrak{B}},C_{\mathfrak{B}},B_{\mathfrak{B}})+J$
$\displaystyle(D_{\mathfrak{B}},C_{\mathfrak{B}},B_{\mathfrak{B}})$ (25)
$\displaystyle+J((DE)_{\mathfrak{r}},C_{\mathfrak{r}},B_{\mathfrak{r}}),$

where two terms in the first line are nonvanishing terms, which are evaluated
as follows

$\displaystyle J(E_{\mathfrak{B}},C_{\mathfrak{B}},B_{\mathfrak{B}})=$
$\displaystyle\frac{\pi}{3}c_{-},$ (26a) $\displaystyle
J(D_{\mathfrak{B}},C_{\mathfrak{B}},B_{\mathfrak{B}})=$ $\displaystyle
J(D,C,B).$ (26b)

The residual term in Eqn. (25) has no triple point, and the modular
Hamiltonian $K_{(BC)_{\mathfrak{r}}}$ is local, but $K_{(CDE)_{\mathfrak{r}}}$
is non-local. However, it can be easily verified

$J((DE)_{\mathfrak{r}},C_{\mathfrak{r}},B_{\mathfrak{r}})=0,$ (27)

once we appropriately decompose $K_{(BC)_{\mathfrak{r}}}$ into smaller three
chunks by Eqn. (14).

In conclusion, we obtain the geometric additivity of a modular commutator for
the incomplete disk,

$J(A,B,C)+J(B,C,D)=\frac{\pi}{3}c_{-}.$ (28)

It states that the sum of two incomplete modular commutators is
$\frac{\pi}{3}c_{-}$ if the union of tripartitions forms a complete disk.

Figure 10: Numerical evaluation of modular commutator for the incomplete
disks. (a)-(d) The lattice realization of the incomplete disks. We fix the
linear size of the system $L$, and the subsystem size $r$: $L=22$ and $r=15$.
(e) Plot of $n_{H}\equiv\frac{3}{\pi}J$ of the Haldane model and the sum for
the incomplete disks with tuning parameter $\tilde{\mu}$. (f) Plot of
conditional mutual information (CMI) $I(A:C|B)$ for the incomplete disk with
tuning parameter $\tilde{\mu}$.

###  B.2 Numerical Calculations and Remarks

We now test the formula (23) for the Haldane model discussed in the main text.
We choose two cases of disks $ABCD$, which are illustrated in Fig. 10(a-b) and
(c-d). For each case, the modular commutators are computed in the parameter
range of $0.5<\tilde{\mu}<1.25$, as shown in Fig. 10(e). In both cases, the
sum of two modular commutators of the incomplete disks yields
$\frac{\pi}{3}c_{-}$, which is consistent with our arguments.

Below, we provide some remarks. The first remark is about the second case in
Fig. 10. In the Chern insulator phase, (a) and (b) have different values.
However, (c) and (d) are nearly identical despite the fact that two incomplete
disks ($ABC$ and $BCD$) are different. It indicates that we can extract an
exact chiral central charge from an incomplete disk alone, yielding $\pi
c_{-}/6$. This is due to the symmetry of the Haldane model, which enforces the
following relation:

$J(A,B,C)=J(B,C,D)=J(C,D,A)=J(D,A,B).$

Specifically, if the partition of incomplete disks $ABCD$ is given as below,

and the system has a proper spatial symmetry, the two modular commutators
$J(A,B,C)$ and $J(D,A,B)$ may be equal, resulting in
$J(A,B,C)=J(B,C,D)=\frac{1}{2}\times\frac{\pi}{3}c_{-}$. This observation may
be useful for extracting the chiral central charge more efficiently in
numerical studies.

The second remark is about the incomplete tri-junction and the conditional
mutual information. It is well-known that if a tripartite state $\rho_{ABC}$
is a quantum Markov chain, then the modular commutator vanishes,

$I(A:C|B)_{\rho}=0\Longrightarrow J(A,B,C)_{\rho}=0.$ (29)

Meanwhile, the tri-junction of an incomplete disk $ABC$, which makes a non-
smooth boundary, provides a non-vanishing value of the conditional mutual
information. Without the tri-junction, the conditional mutual information
(CMI) of the incomplete disk becomes zero because of the area law, and the
modular commutator is zero. We present the conditional mutual information of
incomplete disks in Fig. 10(f), which are nonzero in the chiral topological
phase and decrease in the trivial phase.

Lastly, one can view Eqn. (23) as a bulk analog of an identity that is
satisfied by the edge modular commutator [10, 24, 11]. For instance, the
modular commutators at the contiguous edge intervals $XYZ$ in Fig. 9(a) also
satisfy the similar relation under exchanging $XYZ$ to $YZW$:
$J(X,Y,Z)=\frac{\pi}{3}c_{-}-J(Y,Z,W)$. A similar observation was made in Ref.
[24].

##  Appendix C Additional numerical calculations for $\pi$-flux model

Figure 11: Schematic description of $\pi$-flux model. (a) Each rectangle
denotes the unit cell. See the main text for the details of hopping
parameters. (b) Phase diagram of the $\pi$-flux model along
$\tau=\textrm{Im}(t_{2})$-line. In topological phases, the chiral central
charge is $c_{-}=1$ at $0<\tau<2$, and $c_{-}=-1$ at $-2<\tau<0$.

This Section provides additional numerical calculations involving the
$\pi$-flux model, which is a free fermion model. Since the ground state of the
free fermion systems is the Gaussian state, the reduced density matrix and
modular commutators are completely determined by their two-point correlation
functions [31].

###  C.1 Model Hamiltonian

Figure 12: Numerical evaluation of the geometric integer ($n\geq 3$) for the
edge pizza partition. (a)-(b) The lattice realization of the edge pizza
portions with $n=3,4$, respectively. We fix the system size $L_{x}=47$,
$L_{y}=46$, and the $\ell_{x}=27$, $\ell_{y}=26$. (c) Plot of $n_{\pi}$ for
the edge pizza partition with the tuning parameter $\tau$.

The $\pi$-flux models are defined on the square lattice, consisting of two
sublattices, $a$ and $b$. We use $j=(r,s)$ to label the unit cell $r$ and the
two sublattices $s=a,b$. The $\pi$-flux model is described by the following
Hamiltonian:

$H_{\pi}=H_{1}(t_{1})+H_{2}(t_{2};\phi)$ (30)

The first part concerns the nearest hopping Hamiltonian with amplitude
$t_{1}$, $H_{1}(t_{1})=t_{1}\sum_{\langle
j,k\rangle}\eta_{jk}c^{\dagger}_{j}c_{k}$, which is illustrated as black
links. We use black dashed links for hopping between two $b$ sublattices,
$\eta_{jk}=-1$, and black solid links for all the rest of hopping,
$\eta_{jk}=1$. The second part includes the next nearest hopping terms with
complex amplitude $t_{2}$,
$H(t_{2};\phi)=\tau\sum_{r}(ic^{\dagger}_{r+\hat{x},b}c_{r,a}+\text{h.c})$,
which are depicted as arrows in Fig. 11(a). Here, we choose the amplitude to
be purely imaginary $t_{2}=i\tau$.

In the $\pi$-flux model, we keep $t_{1}$ fixed at 1 and vary $\tau$. The model
exhibits two chiral topological and trivial phases along the $\tau$-parametric
path. The topological phase at $0<\tau<2$ has a chiral central charge of
$c_{-}=1$, while the other topological phase at $-2<\tau<0$ has opposite
chiralities with a chiral central charge of $c_{-}=-1$. Outside of these
ranges, the $\pi$-flux model is trivial.

###  C.2 Numerical results

We first consider the pizza partitions in the $\pi$-flux model with the
geometric integer $(n\geq 3)$. Since the model is defined on a lattice,
verifying higher values of $n$ in the bulk subsystem is not straightforward.
For this reason, we only provide the numerical evaluations of the geometric
integer for the edge pizza partitions, $n_{\pi}\equiv-\frac{3}{\pi}J(X,Y,Z)$
($n\geq 3$). We select $XYZ$ as shown in Fig. 12(a)-(b). Particularly for
$n=3$, we choose the lengths of each interval, $X$, $Y$, and $Z$, not equal.
Numerical estimations of the geometric integers $n_{\pi}$ are presented in
Fig. 12(c). The geometric integers well-converge for a large energy bulk gap
($1<\tau<1.5$). As $\tau$ approaches the critical point, the values of the
modular commutator start to break down due to the decreasing energy gap. This
is because to be well-converged to the integer, each interval demands a larger
size than the bulk correlation length.

Figure 13: Numerical evaluation of the modular commutator for the incomplete
disk $ABCD$. (a) Incomplete disk $ABCD$ in $\pi$-flux model. We fix the disk
size $(\ell_{x},\ell_{y})=(15,14)$ and the system size
$(L_{x},L_{y})=(47,46)$. (c) Plot of modular commutators and the sum for the
incomplete disks with the tuning parameter $\tau$.

Next, we consider the $\pi$-flux model shown in Fig. 13. We choose a partition
$ABCD$ of the $\pi$-flux model as depicted in Fig. 13(a). We compute the
modular commutators at $1<\tau<2.5$ as shown in Fig. 13(b). Two modular
commutators of the incomplete disks exhibit variations as the parameter $\tau$
changes. However, the sum always equals $J(A,BC,D),$ in both chiral
topological and trivial phases. This is consistent with our discussion.

##  Appendix D Absence of the residual term in the invertible states and
remarks on the non-invertible states

In this Section, we prove the residual term of the additivity formula is
absent for the invertible bulk. We also discuss the additivity for the non-
invertible system.

###  D.1 Proof: Absence of the residual term in the invertible bulk

The main issue in evaluating the modular commutator of the pizza partition is
that it has a non-trivial topology. For example, the subsystem of the bulk
pizza partition has a bowtie shape ($\bowtie$); it is non-trivial whether it
is a connected one region or disconnected two regions. These issues become
subtle when the system is topologically ordered. However, based on the
observations below, this issue can be circumvented for the invertible bulk.

Figure 14: Multipartite regions in a two-dimensional system. If the bulk is
invertible, the conditional mutual information $I(A:C|B)$ vanishes for any
smooth bulk deformation of $ABC$. (a) $A,E$ are disconnected edge region at
physical boundary, and $BCD$ is bulk region. (b) A multipartite region
$ABCC^{\prime}D$ in the bulk.

For concreteness, let us consider a two-dimensional gapped system $ABCDE$ on a
disk as shown in Fig. 14(a). The entanglement entropies of these subsystems
are related as

$I(A:C|B)+I(E:C|D)=\Delta(B,C,D),$ (31)

where $\Delta(B,C,D)=S_{BC}+S_{CD}-S_{B}-S_{D}$ and we use $S_{AB}=S_{CDE}$
and $S_{ABC}=S_{DE}$ for pure state. The bulk subsystem $BCD$ is similar to
the partition for the axiom A1 [Eqn. (11b)], but the subsystem $B$ and $D$ are
the union of the disconnected two regions. In this case, $\Delta(B,C,D)$ is no
longer zero; rather, it gives the topological entanglement entropy
($\gamma_{\text{top}}$) [20, 21]: $\Delta(B,C,D)=2\gamma_{\text{top}}$. Note
that the topological entanglement entropy contribution in $\Delta$ depends on
the number of the disconnected components of $B$ (or $D$). For instance, if
the $B$ and $D$ are the union of $m$ disconnected region, then
$\Delta(B,C,D)=2(m-1)\gamma_{\text{top}}$.

A key observation of Eqn. (31) is that if the bulk is invertible
($\gamma_{\text{top}}=0$), the quantum state on the subsystem $ABC$ is the
quantum Markov state. Thus the modular Hamiltonian of $ABC$ is local [30]:

$K_{ABC}=K_{AB}+K_{BC}-K_{B}.$ (32)

This observation can be further generalized. For example, if the bulk is
invertible, we have $I(A:C|B)=0$ for the partition $ABC$ shown in the inset of
Fig. 14(a), where the edge region $A$ and the bulk region $B$ consist of three
disconnected regions.

One can also obtain similar results for the bulk multipartition
$ABCC^{\prime}D$ in Fig. 14(b). We find

$I(A:C|B)+I(A:C^{\prime}|D)=\Delta(B,CC^{\prime},D).$ (33)

Here, $\Delta(B,CC^{\prime},D)=2\gamma_{\text{top}}$ and we use the fact that
the state in sufficiently distant regions is a product state, for example,
$S_{AD}=S_{BCC^{\prime}}+S_{E}$. It is clear that when the bulk is invertible
($\gamma_{\text{top}}=0$), the modular Hamiltonian becomes local.

Below, we prove the absence of the residual term. While we focus on the
examples in Fig. 15, the proof for the other cases is straightforward as well.

Let us first consider the edge residual term
$J(Z_{\mathfrak{r}},W_{\mathfrak{r}},X_{\mathfrak{r}})$ and let
$W_{\mathfrak{r}}=W_{1}W_{2}W^{\prime}$ as shown in Fig. 15(a). From Eqn.
(32), we have

$\displaystyle
K_{(ZW)_{\mathfrak{r}}}=K_{Z_{\mathfrak{r}}W_{2}}+K_{W_{\mathfrak{r}}}-K_{W_{2}}$
$\displaystyle
K_{(XW)_{\mathfrak{r}}}=K_{X_{\mathfrak{r}}W_{1}}+K_{W_{\mathfrak{r}}}-K_{W_{1}}$

and, by plugging into $J(Z_{\mathfrak{r}},W_{\mathfrak{r}},X_{\mathfrak{r}})$,
we obtain

$\displaystyle J(Z_{\mathfrak{r}},W_{\mathfrak{r}},X_{\mathfrak{r}})=$
$\displaystyle
i\langle[K_{(ZW)_{\mathfrak{r}}},K_{(WX)_{\mathfrak{r}}}]\rangle$
$\displaystyle=$ $\displaystyle
J(Z_{\mathfrak{r}},W_{2},W_{\mathfrak{r}}\setminus
W_{2})+J(W_{\mathfrak{r}}\setminus W_{1},W_{1},X_{\mathfrak{r}})$
$\displaystyle=$ $\displaystyle 0.$

In the second and last line, we use the two properties of the modular
commutator in Ref. [9]. In the second line, we use that if any of the subsets
of the modular commutator is empty, the modular commutator vanishes:

$J(\varnothing,B,C)=J(A,\varnothing,C)=J(A,B,\varnothing)=0.$ (34)

In the last line, we use that if the underlying state is a quantum Markov
state, the modular commutator vanishes:

$I(A:C|B)=0\quad\Longrightarrow\quad J(A,B,C)=0.$ (35)

Figure 15: Partitions of the residual term,
$J(Z_{\mathfrak{r}},W_{\mathfrak{r}},X_{\mathfrak{r}})$ and
$J(C_{\mathfrak{r}},D_{\mathfrak{r}},A_{\mathfrak{r}})$.

We can also prove the absence of the bulk residual term similarly. Consider
the bulk residual term $J(C_{\mathfrak{r}},D_{\mathfrak{r}},A_{\mathfrak{r}})$
in Fig. 15(b) and let $D_{\mathfrak{r}}=D_{1}D_{2}D^{\prime}$. Then, we have

$\displaystyle J(C_{\mathfrak{r}},D_{\mathfrak{r}},A_{\mathfrak{r}})=$
$\displaystyle
i\langle[K_{(CD)_{\mathfrak{r}}},K_{(DA)_{\mathfrak{r}}}]\rangle$
$\displaystyle=$ $\displaystyle
J(C_{\mathfrak{r}},D_{2},D_{\mathfrak{r}}\setminus
D_{2})+J(D_{\mathfrak{r}}\setminus D_{1},D_{1},A_{\mathfrak{r}})$
$\displaystyle=$ $\displaystyle 0,$

where the properties of the modular commutator and the decomposition of the
modular Hamiltonian are used:

$\displaystyle K_{(CD)_{\mathfrak{r}}}=$ $\displaystyle
K_{C_{\mathfrak{r}}D_{2}}+K_{D_{\mathfrak{r}}}-K_{D_{2}}$ $\displaystyle
K_{(AD)_{\mathfrak{r}}}=$ $\displaystyle
K_{A_{\mathfrak{r}}D_{1}}+K_{D_{\mathfrak{r}}}-K_{D_{1}}.$

We emphasize that this argument for the invertible bulk is applicable quite
generally. Therefore, the additivity of the modular commutator manifests as
long as the bulk is invertible and the subsystem is sufficiently larger than
the bulk correlation length.

###  D.2 Discussion: non-invertible bulk

In a topologically ordered system, there is in fact a state for which the
additivity of the modular commutator does hold. In the entanglement bootstrap
program [28], certain _merged states_ can be such states. A merged state is a
maximum entropy state locally consistent with the ground state (or vacuum
state, $\sigma$). In anyon theory, a merged state can be considered as a
maximum entropy mixture containing anyons.

Figure 16: The modular commutator of the pizza partitions in the non-
invertible system. (a) An ensemble of the states containing anyons
($a,\bar{a}$), where $\bar{a}$ is anti-charge of topological charge $a$.
Anyons are located in the subsystem $B$. (b) An maximum entropy ensemble of
anyons ($a,b,\bar{c}$) that fuse into vacuum, where $a$ and $b$ fuse into $c$.
The state $\tilde{\tau}$ satisfies $I(ZX:W\setminus
W^{\prime}|W^{\prime})_{\tilde{\tau}}=0$.

For example, consider a merged state on the bulk subsystem $CDA$ as shown in
Fig. 16(a). A merged state of our interest, say $\tau$, is a maximum entropy
state such that

$I(CA:D\setminus D^{\prime}|D^{\prime})_{\tau}=0.$ (36)

In entanglement bootstrap, such a state can be generated by merging the states
on disk and annulus, whose resulting subsystem is the two-hole disk 222 Note
that the subsystem $CDA$ is not a true two-hole disk, and its information
convex set is not well-defined due to its bowtie shape ($\bowtie$). However,
conditional independence remains true because of the strong subadditivity. .
In the context of anyon theory, an ensemble on $CDA$ that has anyons
($a,\bar{a}$) in the holes ($B$) with probability
$p_{a\bar{a}}^{1}=d^{2}_{a}/\mathcal{D}^{2}$ satisfies such conditional
independence. Here, $d_{a}$ is quantum dimension of anyon $a$ and
$\mathcal{D}=\sqrt{\sum_{a\in\mathcal{C}}d_{a}^{2}}$ is total quantum
dimension over anyon superselection sector $\mathcal{C}$.

For this merged state ($\tau$), the modular commutator $J(C,D,A)_{\tau}$ is

$\displaystyle J(C,D,A)_{\tau}=$ $\displaystyle J(C,D^{\prime},A)_{\sigma}$
$\displaystyle=$ $\displaystyle
2\times\frac{\pi}{3}c_{-}+J(C_{\mathfrak{r}},D^{\prime}_{\mathfrak{r}},A_{\mathrm{r}})_{\sigma},$
(37)

where we use the conditional independence of the maximum entropy state,
$I(C:D\setminus D^{\prime}|D^{\prime})_{\tau}=0$ and $I(A:D\setminus
D^{\prime}|D^{\prime})_{\tau}=0$, in the first line and the additivity formula
in the second line. The residual term vanishes
$J(C_{\mathfrak{r}},D^{\prime}_{\mathfrak{r}},A_{\mathrm{r}})_{\sigma}=0$
because the subsystems are disk-like region. Thus, we have
$J(C,D,A)_{\tau}=2\times\frac{\pi}{3}c_{-}$.

Similarly, the additivity of the modular commutator manifests for a maximum
entropy state, say $\tilde{\tau}$, on the edge pizza partition. For example,
consider the subsystem $ZWX$ on a disk as shown in Fig. 16(b). If the
underlying state is a maximum entropy state such that $I(ZX:W\setminus
W^{\prime}|W^{\prime})_{\tilde{\tau}}=0$, we have

$J(Z,W,X)_{\tilde{\tau}}=J(Z,W^{\prime},X)_{\sigma}=-3\times\frac{\pi}{3}c_{-}.$
(38)

Note that the maximum entropy state $\tilde{\tau}$ can also be expressed as an
ensemble containing anyons [Fig. 16(b)].

Therefore, if one proves that the modular commutator is invariant even in the
presence of anyons, then the geometric additivity of the modular commutator is
also proven

$\displaystyle J(C,D,A)_{\tau}\overset{?}{=}$ $\displaystyle
J(C,D,A)_{\sigma}$ (39a) $\displaystyle J(Z,W,X)_{\tilde{\tau}}\overset{?}{=}$
$\displaystyle J(Z,W,X)_{\sigma}$ (39b)

by using the identity, $J(C,D,A)_{\sigma}=J(A,B,C)_{\sigma}$ and
$J(Z,W,X)_{\sigma}=J(X,Y,Z)_{\sigma}$. Thus, examining whether the modular
commutator on the _pizza partition_ remains invariant in the presence of
anyons may be one of the strategies for proving the additivity of the modular
commutator, even in non-invertible systems. We leave it as future work.


11institutetext: Department of Computer Science and Engineering,
Southern University of Science and Technology, Shenzhen, China 11email:
<EMAIL_ADDRESS>NVIDIA 33institutetext: Technical
University of Munich, Germany

# Deep Class-specific Affinity-Guided Convolutional Network for Multimodal
Unpaired Image Segmentation

Jingkun Chen 11 Wenqi Li 22 Hongwei Li 33 Jianguo Zhang 11

###### Abstract

Multi-modal medical image segmentation plays an essential role in clinical
diagnosis. It remains challenging as the input modalities are often not well-
aligned spatially. Existing learning-based methods mainly consider sharing
trainable layers across modalities and minimizing visual feature
discrepancies. While the problem is often formulated as joint supervised
feature learning, multiple-scale features and class-specific representation
have not yet been explored. In this paper, we propose an affinity-guided fully
convolutional network for multimodal image segmentation. To learn effective
representations, we design class-specific affinity matrices to encode the
knowledge of hierarchical feature reasoning, together with the shared
convolutional layers to ensure the cross-modality generalization. Our affinity
matrix does not depend on spatial alignments of the visual features and thus
allows us to train with unpaired, multimodal inputs. We extensively evaluated
our method on two public multimodal benchmark datasets and outperform state-
of-the-art methods.

###### Keywords:

Segmentation Class-specific Affinity Feature Transfer.

##  1 Introduction

Medical image segmentation is a key step in clinical diagnosis and treatment.
Fully convolutional networks [9, 1, 7] have been established as powerful tools
for the segmentation tasks. Benefiting from the learning capability of these
models, researchers start to address more challenging and critical problems
such as learning from multiple imaging modalities. This is an essential task
because different modalities provide complementary information and joint
analysis can provide valuable insights in clinical practice.

Multi-modal learning is inherently challenging for two reasons: 1) supervised
feature learning is often modality-dependent; features learned from a single
modality can not easily be combined with those from other modalities; 2) joint
learning often requires images from different modalities being spatially well-
aligned and paired; obtaining such training data is itself a costly task and
often infeasible. Fig. 1 shows sample slices from cardiac scans in different
modalities. It can be observed that although they all reveal parts of the
heart anatomy, their visual appearances vary. Segmentation networks are often
sensitive to such discrepancies, which has become a major obstacle for model
generalization across modalities.

Figure 1: Left to right: slices of MR (left two) and CT (right two) cardiac
scans.

Spatial misalignment is another issue. Existing image registration methods are
often infeasible, as the spatial correspondences among modalities can be
highly complex and finding a good similarity measurement is non-trivial.

To mitigate these issues, joint learning with unpaired data is emerging as a
promising direction [11, 10, 3]. MultiResUNet [4] has been proposed to improve
upon U-Net in multimodal medical image analysis. In brain image segmentation,
Nie et al. [8] trained networks independently for single modalities and then
fused the high-layer outputs for final segmentation. Yang et al. [12] used
disentangled representations to achieve CT and MR adaptation. Existing methods
didn’t take into account class-specific information, even though the features
obtained by supervised training are highly correlated with the tasks (Fig. 2).

Our assumption is that, with the same network architecture, the underlying
anatomical features should be extracted in a similar manner across modalities.
At the same time, each network instance should have modality-specific modules
to tolerate the imaging domain gaps. With this assumption, to facilitate
effective joint feature learning, we adopt an FCN for all modalities, where
the convolutional kernels are shared, while the modality-specific batch
feature normalizations remain local to each modality. More importantly, we
extract class-specific affinity measurements at multiple feature scales, and
minimize an affinity loss during training. Different from cross-modal feature
consistency loss, our design ensures that the networks extract modality
independent features in a similar hierarchical manner. Intuitively, this could
be interpreted as “high-order” feature extraction consistency compared with
the feature map consistency loss. We show that this is a more appropriate
joint model regularizer that effectively guides the anatomical feature
extractions.

Figure 2: Visualization of feature maps of one cardiac slice in four classes.
Left to right: the background, RV, myo and LV. The higher brightness part
indicates the region for the corresponding class; it could be observed that
class-specific feature representation is brighter than the other parts, which
is highly correlated with the ground truth mask in this class.

In summary, our main contributions are: 1) we propose a novel unpaired
multimodal segmentation framework, which explicitly extracts the modality-
agnostic knowledge; 2) we introduce a joint learning strategy and a class-
specific affinity matrix to guide the training, which is capable of distilling
between-layer relationships at multiple scales; 3) we extensively evaluated
our method on two public multimodal benchmark datasets, and the proposed
method outperforms the state-of-the-art multimodal segmentation approaches.

##  2 Methodology

This section details the proposed joint training for segmentation tasks.

###  2.1 Multimodal Learning

We adopt an FCN as the backbone of our framework. Without any loss of
generality, we present our framework in the case of training with two imaging
modalities. The overall architecture is illustrated in Fig. 3. The training of
the system operates on random unpaired samples from both modalities, the same
set of convolutional layers of the network are updated, while the batch
normalization layers are initialized and updated individually for each
modality.

Figure 3: Architecture of the proposed class-specific affinity guided learning
for multimodal segmentation (CSA denotes class-specific affinity). The overall
structure of the proposed method contains two streams of FCNs with shared
layers. For the detail of the CSA module, please see Fig. 5.

###  2.2 Modality-specific Batch Normalization

Using two independent sets of parameters for joint training leads to large
models and thus tends to overfit. Karani et al. [5] showed that using a
domain-specific batch normalization is effective in addressing domain gaps
issue while keeping the model compact. Here we employ the same technique for
modality-specific feature extraction. Specifically, the batch normalization
layer matches the first and the second moment of the distributions of feature
x:

$x^{*}=\gamma\frac{x-E(x)}{\sqrt{Var(x)+\epsilon}}+\beta$ (1)

where $\gamma$, $\beta$ are trainable parameters and are modality-dependent in
our design. $\epsilon$ is a very small positive number to avoid dividing by
zero.

###  2.3 Class-specific Affinity

It has been shown that feature maps in a network could reflect the saliency of
the class of interest in a multi-class setting [6] in a recent study by Levine
et al. Such a saliency map could give a robust interpretation of the reasoning
process of model predictions. Motivated by this study, in the multi-modal
segmentation network, since all the modalities share the same tasks (e.g.,
multi-class heart region segmentation), we hypothesized that the reasoning
process of model-specific channels should be similar and its feature map
should reflect class-specific saliency (i.e., interpretation of the class of
interest). As shown in Fig. 2, ideally, the region of interest in a learned
feature map should be salient and aligned well with its class-label.
Therefore, for a learned feature map $F(l)$ of layer $l$ and a given class
$c$, we introduce the class-specific feature map $F^{c}(l)$ defined as

$F^{c}(l)=F(l)\odot M^{c}$ (2)

where $M^{c}$ denotes the ground truth mask of size $(h,w)$ for class $c$
(reshape to the size of feature map if necessary), and $\odot$ represents
Hadamard product.
Suppose that $F^{c}_{m}(l)$ and $F^{c}_{n}(k)$ are the $m$-th and $n$-th
class-specific feature maps from layer $l$ and $k$ respectively, we measure
their relationships using an affinity defined by their cosine similarity,
i.e.,

$a^{c}_{m,n}=\frac{1}{S_{c}}*cos(F^{c}_{m}(l),F^{c}_{n}(k))$ (3)

Where $S_{c}$ is the size of the region of interest in $M^{c}$. Such a
normalization is to ensure that the affinity is invariant to the size of the
saliency region. Suppose that layer $l$ and layer $k$ have $M$ and $N$ number
of class-specific feature maps, we construct the between-layer affinity matrix
$A^{c}_{m,n}$, where the entry at $(m,n)$ is $a^{c}_{m,n}$. The size of
$A^{c}$ is $M$ by $N$. Since the affinity is computed based class-specific on
feature map, we term this as the class-specific affinity (CSA) matrix.

Figure 4: The proposed class-specific affinity guided learning layer. The
dotted box on the left shows three convolutional layers for feature
extraction. The dotted box in the middle shows the affinity computation by
incorporating feature maps at multiple scales as well as the multi-class
ground truth segmentation map.

Fig. 4 shows our design of a class-specific affinity (CSA) layer. Our CSA
could be computed for each of the modality, based on which we build the CSA
module. It is worth noting that our design is based on a class-specific
feature map, and independent to the choice of modalities; therefore, our
network does not require inputs spatial alignment.

###  2.4 CSA Module

Suppose that we have two modalities using the same network architecture for
joint learning. We compute $A^{c,1}$ for modality-1 (e.g., CT) and $A^{c,2}$
modality-2 (e.g MR) across layer $l$ and $k$. The knowledge encoded by CSA for
a specific class $c$ could be transferred by enforcing the consistency of CSA
between the two modalities using an L2 norm. We then aggregate all of the
consistencies for all of the classes to formulate a consistency loss function
as below:

Figure 5: The CSA guided multi-modal knowledge sharing. The CSA Layer computes
affinity from ”Mod-1” and ”Mod-2” respectively (according to Sec. 2.3). The
affinity loss between two modalities is computed according to Sec. 2.4.

$L_{CSA}=\frac{1}{C}\sum_{c}^{C}\left\|\frac{1}{P}\sum_{i=1}^{P}(A_{i}^{c,1}-A_{i}^{c,2})\right\|_{2}^{2}$
(4)

where $C$ is the number of classes. $P$ is the total number of entries in
$A^{c}$, i.e., $MN$. Normalizing by $P$ is to ensure that the consistency is
invariant to the number of feature channels.

When minimizing the CSA loss, the affinity consistency between the two
modalities could be maximized thus ensuring joint learning. For the
segmentation loss, we use a Dice coefficient loss $L_{dice}$ to ensure good
segmentation at region level, and a $L_{ce}$ cross-entropy loss for pixel-
level segmentation. Taking all of the three losses together, the final loss of
our multi-modal learning framework for two modalities 1 and 2 is defined as:

$L=\alpha(L_{dice}^{1}+L_{dice}^{2})+\beta(L_{ce}^{1}+L_{ce}^{2})+\lambda
L_{CSA}$ (5)

Where $L_{CSA}$ is the CSA transfer loss, $\alpha$,$\beta$,$\lambda$ are the
system parameters to weight the loss components.

##  3 Experiments

Datasets. We evaluated our multimodal learning method on two public datasets:
MS-CMRSeg 2019 [13] [15] and MM-WHS [14]. MS-CMRSeg 2019 contains 45 patients
who had previously suffered from cardiomyopathy. Each patient has images in
three cardiac MR modalities (LGE, T2, and bSSFP) with the three segmentation
classes: left ventricles, myocardium, and right ventricles. We use LGE and
bSSFP in our experiments. For each slice, we crop the image and get region of
interests with a fixed bounding box (224$\times$224), enclosing all the
annotated regions. we randomly divided the dataset into 80% for training and
20% for testing according to the patients. MM-WHS contains the MR and CT
images whole heart from the upper abdomen to the aortic arch. The segmentation
task includes four structures: left ventricular myocardium (LVM), left atrium
(LAC), left ventricle (LVC), and ascending aorta (AA). We crop each slice with
a region with a 256 $\times$ 256-pixel bounding box in the coronal plane and
randomly divided the dataset into training (80%) and test sets (20%). For
preprocessing, we use _z-score_ normalization [2] to calibrate the intensity
of each 2D slice from all modalities in both datasets.

Implementation. Our architecture consists of nine convolutional operation
groups, one deconvolutional group and one softmax layer. Each group contains
two to four shared convolutional layers and domain-specific batch
normalization layers. We implemented the proposed method with Python-based on
Tensorflow 1.14.0 library using Nvidia Quadro RTX GPU (24G). We optimize our
network with Adam optimizer with a batch size of 8. The learning rate is
initialized to 1$\times$ $10^{-4}$ and decayed by 5% per 1000 iterations.
Besides, we incorporated dropout layers (drop rate of 0.75) into the network
to validate the performance. In the training stage, we used three loss
functions. Empirically, $\alpha$, $\beta$ and $\lambda$ are set to 1, 1 and
0.5 respectively. Our assumption is that the higher layer features are too
closely related to the ground truth mask, so the affinity features of the
migration process are not obvious, so we choose the feature maps from the
intermediate layer. We believe that the information between remote feature
maps is difficult to express completely with affinity feature maps, so we used
the feature maps from the layers relatively close to each other in our
experiment.

Comparison and Analysis. We designed the following six experimental settings
(single training of separate modalities (Single), Unpaired Multi-modal
Segmentation via Knowledge Distillation (UMMKD)[3], Joint training of two
modalities in shared BN model(Joint), Modality-specific batch normalization
only (MSBN), Affinity-guided learning (Affinity), and CSA guided learning
(ours)). For all settings, the network architecture and datasets are fixed so
that different methods can be compared fairly. In terms of quantitative
performance measurements, we adopt the volume Dice score and surface Hausdorff
distance as listed in Table 1, 2 and 3.

Table 1: Average Dice score (%) and Surface Hausdorff distance(mm) on LGE and bSSFP, the highest performance in each class is highlighted. Method | LGE_Dice | bSSFP_Dice | LGE_Dist. | bSSFP_Dist.
---|---|---|---|---
LV | myo | RV | LV | myo | RV | LV | myo | RV | LV | myo | RV
Single[7] | 90.17 | 80.31 | 86.51 | 92.89 | 85.11 | 88.76 | 6.63 | 3.61 | 25.10 | 47.80 | 2.45 | 11.58
UMMKD[3] | 90.41 | 80.48 | 86.99 | 93.38 | 85.69 | 89.91 | 5.00 | 3.46 | 8.66 | 3.00 | 2.24 | 5.39
Joint | 90.22 | 80.91 | 86.61 | 92.74 | 85.19 | 89.12 | 4.00 | 6.08 | 5.12 | 3.16 | 2.45 | 5.00
MSBN | 90.23 | 80.81 | 86.08 | 93.46 | 85.57 | 89.91 | 5.75 | 3.61 | 5.48 | 3.00 | 2.24 | 5.10
Affinity | 90.65 | 81.18 | 87.27 | 93.54 | 85.93 | 89.93 | 5.00 | 3.16 | 9.72 | 3.00 | 2.24 | 5.10
Ours | 91.89 | 83.39 | 87.66 | 93.48 | 85.72 | 90.64 | 4.12 | 3.00 | 5.00 | 3.00 | 2.24 | 4.12

1) Results on MS-CMRSeg 2019: Table 1 lists the results of three classes
cardiac segmentation. As can be observed, compared with individual training
and the other multi-modal learning, our methods showed a significant
performance gain. The overall _average_ Dice score of three classes on two
modalities increased to 87.65% and 89.95% which is much higher than single
training of separate modalities (85.66% and 88.92%) and UMMKD (85.96% and
89.66%), and the _average_ Hausdorff distance of three classes on LGE modality
of UMMKD decreased from 5.71mm to 4.04mm. This indicates that class-based
cross-modal knowledge transfer is effective. We have conducted the Wilcoxon
signed-rank test for the improvements of our method over UMMKD on the results
based on the patient-level predictions (the p-value is 0.010 for LGE, and
0.048 for bSSFPP), indicating that improvements are statistically significant.
It is observed that the affinity guided learning also achieved the better
result on the two modalities, especially on the bSSFP modality, but it is
still lower than the CSA module, as the CSA module learned to share the class-
based semantic knowledge.

2) Results on MM-WHS: The proposed method was also used to segment six classes
cardiac MRI and CT, and achieves promising segmentation performance on this
relatively large dataset (Table 2 and 3), with an _average_ Dice score of
79.27% for LVM, 87.46% for LAC, and 81.42% for AA on CT modality; 88.89% for
LVM, 92.53% for LVC, and 96.07% for AA. Hausdorff distance of our method on
the two modalities is also lower in most of the classes. The differences when
comparing with the models trained in MS-CMRSeg 2019 were the number of samples
for training and the weight of the CSA loss increased to 0.5. Overall, it can
be seen that the proposed methods (ours) outperforms both the single modality
model and the multi-modal method, which confirms their effectiveness. We
tested the CSA learning between the more distant layers and the closer layers,
and we found that the performance within the closer layers is better. The CSA
knowledge between the closer layers can be better learned and also easier to
be migrated.

Table 2: Average Dice score (%) on MRI and CT. Method | CT | MRI
---|---|---
LVM | LAC | LVC | AA | LVM | LAC | LVC | AA
Single[7] | 78.17 | 85.84 | 93.07 | 81.08 | 87.56 | 90.29 | 91.66 | 94.26
UMMKD[3] | 78.73 | 83.47 | 93.29 | 81.41 | 87.89 | 91.68 | 91.88 | 95.32
Joint | 77.92 | 83.96 | 93.51 | 80.08 | 84.03 | 88.41 | 90.92 | 94.67
MSBN | 78.82 | 86.07 | 94.40 | 80.57 | 87.57 | 92.30 | 91.88 | 96.02
Affinity | 78.63 | 87.13 | 94.35 | 79.49 | 85.31 | 91.06 | 91.84 | 92.24
Ours | 79.27 | 87.46 | 94.22 | 81.42 | 88.89 | 91.18 | 92.53 | 96.07
Table 3: Surface Hausdorff distance(mm) on CT and MRI. Method | CT | MRI
---|---|---
LVM | LAC | LVC | AA | LVM | LAC | LVC | AA
Single[7] | 5.00 | 10.63 | 5.00 | 13.38 | 8.50 | 15.03 | 4.47 | 6.00
UMMKD[3] | 5.83 | 14.40 | 6.01 | 13.97 | 10.05 | 8.94 | 4.47 | 3.61
Joint | 5.39 | 14.32 | 4.24 | 13.30 | 6.40 | 40.17 | 9.84 | 4.47
MSBN | 5.00 | 10.63 | 4.24 | 13.93 | 4.12 | 9.06 | 4.47 | 3.00
Affinity | 6.00 | 10.63 | 4.89 | 15.13 | 10.05 | 12.08 | 5.00 | 67.09
Ours | 5.39 | 10.30 | 4.24 | 15.80 | 4.12 | 9.49 | 4.12 | 2.83

Visualization. Fig. 6 shows the predicted masks from the six methods. It could
be seen that our methods improve the performance of only using single-modal
method and the other multi-modal method, especially for the LGE modality.
These observations are consistent with those shown in the Table 1, 2 and 3.

Figure 6: Segmentation maps on CT (top) and MRI (bottom). Left to right:
Ground Truth mask, Single, UMMKD, Joint, MSBN, Affinity and CSA.

##  4 Conclusion

We propose a new framework for unpaired multi-modal segmentation, and
introduce class-specific affinity measurements to regularize the jointly model
training. We have derived the formulations and experimented with spatially 2D
feature maps. As future work, the same concepts could be extended to spatially
3D cases. The results based on the proposed class-specific affinity loss are
encouraging. Further quantitative analysis of the feature maps and
investigating model interpretability is also an interesting future direction.

## References

  * [1] Badrinarayanan, V., Kendall, A., Cipolla, R.: Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE transactions on pattern analysis and machine intelligence 39(12), 2481–2495 (2017)
  * [2] Chen, J., Li, H., Zhang, J., Menze, B.: Adversarial convolutional networks with weak domain-transfer for multi-sequence cardiac mr images segmentation. In: International Workshop on Statistical Atlases and Computational Models of the Heart. pp. 317–325. Springer (2019)
  * [3] Dou, Q., Liu, Q., Heng, P.A., Glocker, B.: Unpaired multi-modal segmentation via knowledge distillation. TMI (2020)
  * [4] Ibtehaz, N., Rahman, M.S.: Multiresunet: Rethinking the u-net architecture for multimodal biomedical image segmentation. Neural Networks 121, 74–87 (2020)
  * [5] Karani, N., Chaitanya, K., Baumgartner, C., Konukoglu, E.: A lifelong learning approach to brain mr segmentation across scanners and protocols. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 476–484. Springer (2018)
  * [6] Levine, A., Singla, S., Feizi, S.: Certifiably robust interpretation in deep learning. arXiv preprint arXiv:1905.12105 (2019)
  * [7] Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3431–3440 (2015)
  * [8] Nie, D., Wang, L., Gao, Y., Shen, D.: Fully convolutional networks for multi-modality isointense infant brain image segmentation. In: 2016 IEEE 13Th international symposium on biomedical imaging (ISBI). pp. 1342–1345. IEEE (2016)
  * [9] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. pp. 234–241. Springer (2015)
  * [10] Valindria, V.V., Pawlowski, N., Rajchl, M., Lavdas, I., Aboagye, E.O., Rockall, A.G., Rueckert, D., Glocker, B.: Multi-modal learning from unpaired images: Application to multi-organ segmentation in ct and mri. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). pp. 547–556. IEEE (2018)
  * [11] Wolterink, J.M., Dinkla, A.M., Savenije, M.H., Seevinck, P.R., van den Berg, C.A., Išgum, I.: Deep mr to ct synthesis using unpaired data. In: International workshop on simulation and synthesis in medical imaging. pp. 14–23. Springer (2017)
  * [12] Yang, J., Dvornek, N.C., Zhang, F., Chapiro, J., Lin, M., Duncan, J.S.: Unsupervised domain adaptation via disentangled representations: Application to cross-modality liver segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 255–263. Springer (2019)
  * [13] Zhuang, X.: Multivariate mixture model for cardiac segmentation from multi-sequence mri. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 581–588. Springer (2016)
  * [14] Zhuang, X., Li, L., Payer, C., Štern, D., Urschler, M., Heinrich, M.P., Oster, J., Wang, C., Smedby, Ö., Bian, C., et al.: Evaluation of algorithms for multi-modality whole heart segmentation: an open-access grand challenge. Medical image analysis 58, 101537 (2019)
  * [15] Zhuang, X., Xu, J., Luo, X., Chen, C., Ouyang, C., Rueckert, D., Campello, V.M., Lekadir, K., Vesal, S., RaviKumar, N., et al.: Cardiac segmentation on late gadolinium enhancement mri: A benchmark study from multi-sequence cardiac mr segmentation challenge. arXiv preprint arXiv:2006.12434 (2020)