# Multi-state Data Storage in a Two-dimensional Stripy Antiferromagnet
Implemented by Magnetoelectric Effect

Pingfan Gu1,2 Cong Wang3 Dan Su4 Zehao Dong1 Qiuyuan Wang1 Zheng Han5,6
Kenji Watanabe7 Takashi Taniguchi8 Wei Ji3,⋆ Young Sun9,⋆ and Yu Ye1,2,10,⋆

1 State Key Laboratory for Mesoscopic Physics and Frontiers Science Center for
Nano-optoelectronics, School of Physics, Peking University, Beijing, 100871,
China
2 Collaborative Innovation Center of Quantum Matter, Beijing 100871, China
3 Department of Physics and Beijing Key Laboratory of Optoelectronic
Functional Materials and Micro-Nano Devices, Renmin University of China,
Beijing 100872, China
4 Beijing National Laboratory for Condensed Matter Physics, Institute of
Physics, Beijing 100190, China
5 State Key Laboratory of Quantum Optics and Quantum Optics Devices, Institute
of Optoelectronics, Shanxi University, Taiyuan 030006, China
6 Collaborative Innovation Center of Extreme Optics, Shanxi University,
Taiyuan 030006, China
7 Research Center for Functional Materials, National Institute for Materials
Science, Tsukuba 305-0044, Japan
8 International Center for Materials Nanoarchitectonics, National Institute
for Materials Science, Tsukuba 305-0044, Japan
9 Center of Quantum Materials and Devices, and Department of Applied Physics,
Chongqing University, Chongqing 400044, China
10 Yangtze Delta Institute of Optoelectronics, Peking University, Nantong
226010 Jiangsu, China
$\star$ Email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

> ## Abstract
>
> A promising approach to the next generation of low-power, functional, and
> energy-efficient electronics relies on novel materials with coupled magnetic
> and electric degrees of freedom. In particular, stripy antiferromagnets
> often exhibit broken crystal and magnetic symmetries, which may bring about
> the magnetoelectric (ME) effect and enable the manipulation of intriguing
> properties and functionalities by electrical means. The demand for expanding
> the boundaries of data storage and processing technologies has led to the
> development of spintronics toward two-dimensional (2D) platforms. This work
> reports the ME effect in the 2D stripy antiferromagnetic insulator CrOCl
> down to a single layer. By measuring the tunneling resistance of CrOCl on
> the parameter space of temperature, magnetic field, and applied voltage, we
> verified the ME coupling down to the 2D limit and unraveled its mechanism.
> Utilizing the multi-stable states and ME coupling at magnetic phase
> transitions, we realize multi-state data storage in the tunneling devices.
> Our work not only advances the fundamental understanding of spin-charge
> coupling, but also demonstrates the great potential of 2D antiferromagnetic
> materials to deliver devices and circuits beyond the traditional binary
> operations.

## Introduction

The field of spintronics concerns the processing of digital information, where
an external stimulus, preferably an electrical stimulus, is applied to control
the spin order in a magnetic system that acts as a "0" or "1" digital bit. As
a more common magnetic ground state than ferromagnetism, antiferromagnetism
has received increasing attention in recent years due to its promising
prospect for spintronic devices, such as their robustness against external
perturbations, no stray fields, and ultrafast dynamics [1]. More intriguingly,
antiferromagnets often manifest complicated spin configurations and phase
transitions, opening up the possibility to implement new data storage logic
superior to conventional binary algorithms. However, the negligible net
magnetization also makes it difficult to read out or electrically manipulate
the antiferromagnetic order, which is desired for information technology. As a
result, formidable efforts have been devoted to switching antiferromagnets
through spin torques from exchange bias[2], spin-orbit torque[3] and spin-
galvanic effect[4], etc.

Apart from the above-mentioned "external" approaches, the "internal" approach
to antiferromagnetic spintronics rests upon materials with coupled degrees of
freedom [5, 6, 7] such as magnetic moment, electric polarization and strain,
which are often referred to as the magnetoelastic or magnetoelectric
materials[8]. However, as the transition metal $d$ electrons are supposed to
repel the tendency for off-center ferroelectric distortion[9], the coexistence
of magnetic moment and electric polarization is hard to achieve. The coupled
energy terms, as well as the electric polarization, require materials with low
crystal and magnetic symmetry. Improper magnetic ferroelectrics[10, 11] where
the ferroelectricity originated from spin-order-driven inversion symmetry
breaking are considered to be an ideal platform to realize the mutual clamping
of the order parameters of ferroelectricity and antiferromagnetism[12, 13,
14]. Based on the spin frustration and spin-orbit coupling theories, three
spin-structure-induced electric polarization mechanisms have been established,
namely the exchange striction model[15], the inverse Dzyaloshinskii-Moriya
interaction model[16] and the p-d hybridization model[17].

To further explore ME-based fundamental physics and develop practical
applications for information devices, it is inevitable to extend the research
to the two-dimensional (2D) limit[18, 19, 20, 21, 22, 23]. Recently,
multiferroicity resulting from inverse Dzyaloshinskii-Moriya interaction[24,
25] and p-d hybridization[26] have been evidenced in van der Waals materials.
Nevertheless, the direct coupling between magnetic moment and electric
polarization in the 2D limit has not been reported. The air-stable van der
Waals insulator CrOCl, with a stripy antiferromagnetic ground state and thus
both broken rotational symmetry and translation symmetry perpendicular to the
stripes, is a potential candidate to exhibit the intrinsic 2D ME effect. We
have verified the magnetic phase transitions and magnetoelastic coupling
effect of CrOCl in the 2D limit[27]. Additionally, exotic quantum Hall
effects[28] and insulator phases[29] were observed in the graphene/CrOCl
heterostructures. In this work, we report the intrinsic ME effect in CrOCl
down to the 2D limit. By means of dielectric measurements combined with first-
principles calculations, we find that the wave symmetry of stripy
antiferromagnetism induces antiferroelectric dipoles below the Néel
temperature. The antiferroelectric dipoles persist to the 2D limit, as
evidenced by the $I-V$ hysteresis loops and magnetoresistance of CrOCl
tunneling devices. We also realize direct manipulation of resistance states
through electric and magnetic excitations, demonstrating a data storage device
with continuously adjustable outputs. The metastable states in the first-order
phase transition resulting from the ME coupling break through the barriers of
write-in and read-out information in antiferromagnets and hopefully can open
an effective route for multi-state memories with high storage density,
nonvolatility, low energy consumption, and small memory cell.

## Results and Discussion

Fig. 1: Magnetic and dielectric properties of bulk CrOCl. (a), Crystal
structure of CrOCl above the Néel temperature with the $Pmmn$ space group in a
unit cell. (b), Crystal structure of CrOCl below the Néel temperature with the
$P2_{1}/m$ space group in a unit cell. The structural parameter $\alpha$ that
defines the crystal symmetry is labeled in the unit cell. The vectors on each
atom represent the atomic displacements (magnified 100 times), while the
dashed boxes and open arrows represent the calculated antiferroelectric
polarizations of different Cl-Cr-O chains. (c), Temperature dependence of
relative permittivity $\epsilon_{r}$ and magnetic moment of bulk CrOCl
crystal. (d), $\Delta\epsilon_{r}/\epsilon_{r}$(0 T) versus out-of-plane
magnetic field and the corresponding $M-H$ curve at 2 K.  Fig. 2: $I-V$
hysteresis of CrOCl tunneling devices. (a), Illustration of the tunneling
device. (b), $I-V$ curves of a few-layer CrOCl ($\sim$10 nm) tunneling device
at 20 K (above the Néel temperature) and 2 K (below the Néel temperature).
(c), 2D map of current polarization $\rho$ as a function of temperature and
bias voltage. The green dashed line marks the Néel temperature of exfoliated
CrOCl. (d), $\rho$ versus temperature at different voltages extracted from
(b). (e), Resistance versus temperature of CrOCl tunneling device at 5 V in
different sweeping processes. (f), Tunneling resistance versus out-of-plane
magnetic field at $-$5 V in different sweeping processes. (g), Reproducible
manipulation of the resistance states at $-$0.8 V by alternately changing the
sweeping direction. Data in (a-f) were obtained in the $\sim$10 nm CrOCl
tunneling device, while (g) was obtained in a single-layer CrOCl tunneling
device in parallel with a 10 M$\Omega$ protection resistor.  Fig. 3: Operation
principles of multi-state data storage. (a),$I-B$ curve of the CrOCl tunneling
device at 2 K. The pink curve shows the differential current $\delta
I=I(B)|_{\rm{up}}-I(B)|_{\rm{down}}$. The critical field where $\delta I$
reaches a maximum ($B_{0}$) and the hysteresis closes ($B_{s}$) are annotated
by black arrows. (b), Tunneling current over time after several electric
excitations. The dashed lines represent the current values at 3.33 T in
$B_{\rm{down}}$ and $B_{\rm{up}}$ sweeping cycles extracted from (a). (c),
$I-B$ curves with different magnetic sweep ranges. (d), 2D map of the
tunneling current after designed electric and magnetic excitations. (e,f),
Tunneling current (e) after alternating electric and magnetic excitations. The
corresponding magnetic and electric fields versus time are plotted in (f).

To understand the intrinsic ME effect, we start with the energy coupling terms
for the ground state of CrOCl. As reported by several neutron scattering
results [30, 31, 32] and confirmed by DFT calculations[27], CrOCl transforms
into the $\uparrow\uparrow\downarrow\downarrow$ stripy AFM ground state at
$\sim$14 K. The magnetic frustration drives the crystal from the orthorhombic
space group $Pmmn$ (Fig.1a) to the monoclinic space group $P2_{1}/m$ (Fig. 1b)
[33, 31, 32]. Despite the monoclinic distortion, the space inversion operation
$P$ preserves in the $\uparrow\uparrow\downarrow\downarrow$ ground state due
to the special zigzag atom-chain structure of CrOCl. The two sublayers of a
single van der Waals layer are spatially inverse to each other, thus
preventing net electric polarization from appearing. However, based on the
symmetry operations of a wave order, we can obtain the free energy in the
form[34, 35]:

$\displaystyle\mathcal{F}=$
$\displaystyle\frac{1}{2}r_{a}|A_{\mathbf{k}}|^{2}+\frac{1}{2}r_{s}|\mathbf{S_{q}}|^{2}$
(1)
$\displaystyle+\lambda_{1}\left[(\mathbf{S_{q}}\cdot\mathbf{S_{q}})A_{\mathbf{k}}^{*}+\text{
c.c. }\right]+2\lambda_{2}|\mathbf{S_{q}}|^{2}|A_{\mathbf{k}}|^{2}$

where $\mathbf{S_{q}}$ is the amplitude of spin density modulation with the
wave vector of $\mathbf{q}$ and $A_{\mathbf{k}}$ is the corresponding lattice
modulation with the wave vector of $\mathbf{k}$. $r_{a}$, $r_{s}$,
$\lambda_{1}$ and $\lambda_{2}$ are constant parameters of the material.
Minimizing the free energy, we obtain the lattice modulation in a collinear
spin configuration:

$A_{\mathbf{k}}=\frac{\lambda_{1}}{r_{a}}\mathbf{S_{q}}\cdot\mathbf{S_{q}}+\text{c.c.}$
(2)

The lattice modulation, consequently, can only be present with $\mathbf{k}=\pm
2\mathbf{q}$. In other words, a long-range magnetic wave order with a period
of 4$b$ induces a strain wave with half of its period, 2$b$. The displacements
of the atoms in CrOCl are resolved by neutron scattering[32]. Our density
functional theory (DFT) calculations also confirm the displacements of atoms
in the $\uparrow\uparrow\downarrow\downarrow$ ground state with a period of
2$b$, as shown in Fig. 1b.

Based on the above phenomena, we can regard the
$\uparrow\uparrow\downarrow\downarrow$ stripy AFM phase as an
antiferroelectric order consisting of the electric dipoles of the distorted
Cl-Cr-O chains away from the inversion center (dashed boxes in Fig. 1b).
Therefore, ordered effective dipoles are formed as indicated by the open
arrows in Fig. 1b, while the electric dipole of each atom is provided in
Supplementary Information Fig. S1. Both the contributions of ions and
electrons to polarization are taken into account by the Born effective charge
method (see Methods). The relative permittivity $\epsilon_{r}$ of the bulk
CrOCl crystal increases sharply at $\sim$14 K (the $\epsilon_{r}-T$ curve in
Fig. 1c) with the decreasing temperature, in perfect agreement with the Néel
temperature where the net moment drops (the $M-T$ curve in Fig. 1c),
evidencing the structural phase transition to be a product of ME coupling.
More importantly, the layer thickness experiences a sudden expansion at the
Néel temperature[31], which tends to result in a decrease in the measured
capacitance as $C=\frac{\epsilon_{r}\epsilon_{0}S}{d}$ (see Methods). As a
result, the increase in $\epsilon_{r}$ should be attributed to the appearance
of additional antiferroelectric dipoles. According to the previous report[27],
CrOCl transforms into the collinear
$\uparrow\uparrow\uparrow\downarrow\downarrow$ phase under an external out-of-
plane field of $\sim$4.5 T (the $M-H$ curve in Fig. 1d). Our calculations show
that in the $\uparrow\uparrow\uparrow\downarrow\downarrow$ phase, the crystal
relaxes to an orthorhombic structure (see Supplementary Information Table S1),
in which the atomic displacements and the electric dipoles show a period of
5$b$. Under an external electric field of 0.07 V/Å (close to the experimental
values of the tunneling measurements below), the
$\uparrow\uparrow\downarrow\downarrow$ phase produces a net polarization of
$\rm{116.5\ \upmu C/m^{2}}$, which is eight times larger than that of the
$\uparrow\uparrow\uparrow\downarrow\downarrow$ phase ($\rm{14.42\ \upmu
C/m^{2}}$). This suggests that the electric dipoles of the
$\uparrow\uparrow\downarrow\downarrow$ phase can be tilted by the external
field, while the $\uparrow\uparrow\uparrow\downarrow\downarrow$ phase is
rather structural stable and can hardly be adjusted by the external field.
Correspondingly, the measured $\epsilon_{r}$ undergoes a decrease upon the
magnetic phase transition to the
$\uparrow\uparrow\uparrow\downarrow\downarrow$ phase (Fig. 1d). The magnetic
order-induced electric dipoles were further verified by the pyroelectric
measurements, where the small net polarization could arise from uncompensated
antiferroelectric domains (see Supplementary Information Fig. S2).

Now that we have demonstrated the existence of spin-induced electric dipoles
in bulk CrOCl, from a macroscopic perspective, we would expect the
interactions between the electric and magnetic degrees of freedom and
consequently the manipulation of the spin order by the electric field, or vice
versa. To achieve a large electric field in ultra-compact nanodevices, we
fabricated vertical tunneling devices based on single- to few-layer CrOCl
flakes (Fig. 2a). Cross-structured few-layer graphite stripes are used to
contact the CrOCl tunneling layer, and the whole device is encapsulated by
hexagonal boron nitride (h-BN). The electric field in CrOCl can reach
$\sim$0.1 V/Å, where the spin-induced electric dipoles are expected to be
polarized and have a substantial impact on the tunneling current.

The existence of the antiferroelectric dipoles can be verified by the
hysteresis of the $I-V$ curves. As shown in Fig. 2b, the tunneling currents in
the sweep-up and the sweep-down processes deviate significantly from each
other under the same applied voltage at a temperature of 2 K, manifesting as
an obvious clockwise hysteresis. In sharp contrast, the $I-V$ curve at 20 K
(above the Néel temperature) shows no sign of hysteresis. The electric
hysteresis can be further viewed in the 2D map of temperature and bias voltage
(Fig. 2c), where the current polarization $\rho$ is defined as
$\rho=(R_{\rm{up}}-R_{\rm{down}})/(R_{\rm{up}}+R_{\rm{down}})$. Apart from the
hysteresis loop induced by extrinsic charge traps at low voltages[36], $\rho$
exhibits non-zero values below $\sim$12 K, perfectly coincides with the Néel
temperature of exfoliated CrOCl[27]. Clockwise hysteresis results in a
negative $\rho$ under positive bias and a positive $\rho$ under negative bias.
The absolute value of $\rho$ decreases with increasing temperature and
eventually becomes zero at the Néel temperature (Fig. 2c and Fig. 2d),
implying the disappearance of the interplay between charge and spin order.
From the $R-T$ curves of different sweeping processes at 5 V (Fig. 2e), it can
be seen that in the sweep-up process, the resistance drops significantly
(entering a lower resistance state) below the Néel temperature, while in the
sweep-down process, the resistance increases (entering a higher resistance
state). The above phenomena indicate that the configuration of the dipole
order dominates the measured tunneling resistance below the Néel temperature.
The resistance at $-5$ V exhibits completely antisymmetric behavior, as shown
in Supplementary Information Fig. S3.

As mentioned earlier, in the absence of adjustable electric dipoles, the
$\uparrow\uparrow\uparrow\downarrow\downarrow$ phase under an external
magnetic field is rather structurally stable. The resistances under different
electrical sweeping processes show completely different magnetoresistance
behaviors at the $\uparrow\uparrow\downarrow\downarrow$ phase (see details in
Supplementary Information Fig. S6 and S7) and converge at the transition field
(Fig. 2f), which is in perfect agreement with our theory. The hysteresis of
the $I-V$ curve in the $\uparrow\uparrow\downarrow\downarrow$ state can be
observed in all CrOCl devices of varying thicknesses down to monolayer.
Suspending the sweeping process at a specific voltage, the resistance value
relaxes slowly with a time constant of several hours (see Supplementary
Information Fig. S5), which is highly reproducible as shown in a single-layer
CrOCl device (Fig. 2g). Such splitting of resistances between different
electric sweep directions is direct evidence of ME coupling in 2D CrOCl, as
the emergence of antiferromagnetic order leads to additional resistance
states. The ramping electric field acts as an electric excitation to tilt the
electric dipoles, while the interactions between spin and charge tend to relax
the system to the antiferroelectric ground state. Hysteresis and relaxation
behavior are discussed in detail in Supplementary Information Fig. S4 and S5.
In addition to the $I-V$ hysteresis, the structural modulation of the
$\uparrow\uparrow\downarrow\downarrow$ phase under the electric field also
leads to different magnetoresistance behavior with varying bias voltage, as
shown in Supplementary Information Fig. S8 and S9.

Switching the tunneling resistance between different metastable states
resulting from ME coupling by sweeping the electric field provides us with
design principles for magnetoelectronic devices. We have demonstrated that the
magnetic transitions in CrOCl under external fields are first-order
transitions with large hysteresis loops[27]. Therefore, we can expect that the
electric field can adjust the tunneling resistance to any meta-stable states
inside the magnetic hysteresis loop by tilting the electric dipoles. From the
$I-B$ curve of the CrOCl tunneling device at 5.5 V (Fig. 3a), the hysteresis
closes at $B_{s}$=5.63 T, where CrOCl is believed to transform to the
$\uparrow\uparrow\uparrow\downarrow\downarrow$ state. Subtracting the current
of the $B$-up curve from that of the $B$-down curve, we obtain the maximum
hysteresis occurs at $B_{0}$=3.33 T (Fig. 3a). In other words, the spin
configuration of CrOCl at 3.33 T is close to the
$\uparrow\uparrow\downarrow\downarrow$ AFM ground state (low current) in the
$B$-up cycle, but remains in the
$\uparrow\uparrow\uparrow\downarrow\downarrow$ state (high current) in the
$B$-down cycle. The maximum value of $\delta I$ is $\sim$2 nA, 10% of the
tunneling current. Similar to the operation in Fig. 2, we apply electric
excitations by sweeping the bias voltage up and back to 5.5 V. After the
electric excitation, the tunneling current changes to an intermediate state
following a slight relaxation and stabilizes at the state for several hours
without any sign of change. The current value of the intermediate state is
determined by the electric excitation, specifically, by the sweeping rate and
the peak voltage. By applying different electric excitations by design, we can
switch the tunneling current to an arbitrary value between the highest
($\uparrow\uparrow\uparrow\downarrow\downarrow$) and lowest
($\uparrow\uparrow\downarrow\downarrow$) current values, as demonstrated in
Fig. 3b. The noise fluctuation of each state is less than 0.02 nA, which means
that a single tunneling junction can store at least a decimal number if the
difference between distinguishable adjacent states is set to be an order of
magnitude larger than the noise fluctuation. After sweeping the magnetic field
across $B_{s}$ and back to $B_{0}$, the junction can be reset to the
$\uparrow\uparrow\uparrow\downarrow\downarrow$ high current state, which means
the erasing of the stored information.

Likewise, to expand the capabilities of the device, the magnetic field can
also serve as another degree of freedom to tune the tunneling resistance.
Sweeping the magnetic field to a value below $B_{s}$ and back to $B_{0}$ also
adjusts the tunneling current to an intermediate state, as shown in Fig. 3c.
Accordingly, we can also adjust the tunneling current to different values with
varying magnetic excitations, and set the current back to the
$\uparrow\uparrow\downarrow\downarrow$ low current state with a large electric
excitation. Repeating the previous electric excitations following different
magnetic excitations, we obtain a new set of current values. In this way, we
constructed a 5$\times$5 2D list of different resistance states by electric
and magnetic excitations (Fig. 3d). The opposite dependence of the current on
electric and magnetic excitations strongly evidences the ME coupling in this
system, as in the classical electrodynamics picture, the magnetic field flips
the electron spins and the electric field reverses the electric dipoles. The
combination of spin and charge order determines the tunneling resistance of
the device, which can be manipulated both by electric and magnetic
excitations. Furthermore, it is worth mentioning that similar multi-sate data
storage operation principles are also applicable at zero-field and to single-
layer CrOCl devices (see Supplementary Information Fig. S10 and S11). We
finally verified the repeatability of device operation as shown in Fig. 3e-f.
The device is first set to the $\uparrow\uparrow\downarrow\downarrow$ AFM
ground state ($\sim$ 19.4 nA) by a sufficiently large electric excitation of
$\delta V=$2 V, and then reset to the
$\uparrow\uparrow\uparrow\downarrow\downarrow$ state ($\sim$21.4 nA) by the
magnetic stimulation of $\delta B=$1.67 T. The writing and erasing operations
were repeated seven times, and the high and low resistance states showed
perfect stability.

In summary, we have demonstrated the ME effect in the 2D stripy
antiferromagnet CrOCl. By sweeping the bias voltage, we achieved electrically
driven magnetic transitions of the AFM states. In particular, the tunneling
resistance can be set to arbitrary values via electric and magnetic
excitations, which has never been reported in other 2D systems. Thanks to the
stability of CrOCl, direct coupling between electric and magnetic degrees of
freedom can be observed in all devices of varying thicknesses down to
monolayer. Furthermore, the special ME coupling term gives rise to successive
metastable states that are rather stable and hardly degenerate over time. The
multi-state data storage realized in CrOCl may serve as a new paradigm with
the potential to impact information technology, such as analog data storage
and computation in an array of tunneling devices, stepping beyond Von Neumann
architecture and enabling neuromorphic computing with low power consumption.
Furthermore, the ME coupling in CrOCl may give rise to more unexplored
fantastic properties, highlighting the characteristics of 2D antiferromagnetic
materials and their promising potential in fundamental research and spintronic
applications.

## Methods

Crystal synthesis and characterizations. The mixture of powdered
$\rm{CrCl_{3}}$ and $\rm{Cr_{2}O_{3}}$ with a molar ratio of 1:1 and a total
mass of 1.5 g was sealed in an evacuated quartz ampule. The ampule was then
placed in a two-zone furnace where the source and sink temperatures for growth
were set to 940 °C and 800 °C, respectively, and kept for two weeks.
Subsequently, the furnace was slowly cooled to room temperature, and high-
quality CrOCl crystals were obtained. Magnetization measurements were
performed by standard modules of a Quantum Design PPMS.

Dielectric and pyroelectric measurements. Both pyroelectric measurements and
dielectric measurements were performed on a TeslatronPT System, Oxford
Instruments. Silver epoxy was painted on opposite sides of the sample as
electrodes. For pyroelectric measurements, an electrometer (Keithley 6517B)
was used as a DC voltage source and ammeter. When the temperature stabilized
at 60 K, an external electric field was applied to the sample. The sample was
then cooled from 60 K to 2 K under different electric fields. After the
temperature stabilized at 2 K, the electric field was removed and the
temperature was increased to 60 K at a rate of 5 K/min. During the heating
process, the change of pyroelectric current with temperature was collected.
The polarization can be obtained by integrating the pyroelectric current with
time. Dielectric measurements were made using a capacitive bridge meter
(AH2700A). From 2 K to 50 K, the change in relative permittivity with
temperature was measured by heating at the rate of 2 K/min. Magneto-dielectric
effect at 2 K was measured at a magnetic field sweep rate of 15 Oe/s from 0 to
8 T. All test frequencies are 20 kHz.

Device fabrication. Few-layer graphite, h-BN (10-30 nm), and CrOCl flakes were
obtained by the scotch tape exfoliation method under ambient conditions. The
heterostructures were then assembled with a conventional tear-and-stack
technique based on polypropylene carbonate(PPC)/polydimethylsiloxane(PDMS)
polymer stacks placed on glass slides. Once encapsulated, the devices were
annealed in a high vacuum with a mixed gas flow of $\rm{H_{2}}$ and Ar to
remove residual PPC. Metal contact of Cr/Au (5/25 nm ) electrodes were then
defined using electron-beam lithography, reactive ion etching (in plasma of
the $\rm{CHF_{3}/O_{2}}$ mixture), electron beam evaporation, and lift-off
processes.

Electrical transport measurements. Transport measurements were performed in a
$\rm{Heliox^{3}}$ He insert system equipped with a 14 T superconducting
magnet. The lowest temperature of the system is 1.6 K. To measure the $I-V$
characteristics of the tunnel barrier and the magnetoresistance, a Keithley
2636B source meter was used to apply a bias voltage and a standard two-probe
module was used to measure the tunneling current. To obtain intrinsic signals
and at the same time exclude the possibility of the Joule heating effect, the
tunneling current is limited to $\sim$50 nA, so the total power in a junction
of $\sim$1 ${\upmu}\rm{m^{2}}$ is merely $\sim$0.3 $\upmu$W.

DFT calculations. Our DFT calculations were performed using the generalized
gradient approximation for the exchange-correlation potential, the projector
augmented wave method[37], and a plane-wave basis set implemented in the
Vienna ab-initio simulation package (VASP)[38]. Dispersion correction was made
at the van der Waals density functional (vdW-DF) level[39], with the optB86b
functional for the exchange potential[40], and which was proved to be accurate
in describing the structural properties of layered materials[41] and was
adopted for structure related calculations. The shape and volume of each
supercell and all atomic positions were fully relaxed until the residual force
per atom was less than $1\times 10^{-3}$ eV⋅Å-1 in our calculations. In VASP
calculations, the kinetic energy cut-off for the plane-wave basis set was set
to be 700 eV for geometric and electronic structure calculations. A $k$-mesh
of $10\times 14\times 4$ was adopted to sample the first Brillouin zone of the
conventional unit cell of CrOCl bulk. The $U$ and $J$ values of the on-site
Coulomb interaction of the Cr $d$ orbitals are 3.0 eV and 1.0 eV,
respectively, as revealed by a linear response method[42] and comparison with
the experimental results[27]. These values are comparable to those adopted in
modeling CrSCl[43] and $\rm{CrI_{3}}$[44]. The Born effective charges were
calculated using density functional perturbation theory[45]. The dipole moment
of each atom is calculated by $P_{i}=Z_{i}^{*}\cdot\mu_{i}$, where $P_{i}$ is
the dipole moment of the ion $i$ in one unit cell, $Z_{i}$ is the Born
effective charge tensors and $\mu_{i}$ is the atomic displacement. For
calculations of single-layer CrOCl, a sufficiently large vacuum layer over 20
Å along the out-of-plane direction was adopted to eliminate the interaction
among monolayer. The out-of-plane electric polarization of single-layer CrOCl
under the external electric field is well defined in terms of the classical
model due to the presence of a vacuum layer, which is calculated by
integrating electron density times $z$-coordinate over the supercell.

## Data Availability

The data that support the findings of this study will be available at an open-
access repository with a doi link, when accepted for publishing.

## Code Availability

The codes that support the findings of this study will be available at an
open-access repository with a doi link, when accepted for publishing.

## Acknowledgements

This work was supported by the National Key R&D Program of China (Grants No.
2018YFA0306900), and Beijing Natural Science Foundation (Grant No. JQ21018).
Y.S. acknowledge support from the National Natural Science Foundation of China
(Grant No. 51725104). W.J. acknowledge support from the Ministry of Science
and Technology (MOST) of China (Grant No. 2018YFE0202700), and the National
Natural Science Foundation of China (Grants No. 61761166009, No. 11974422 and
No. 12104504). C.W. was supported by the China Postdoctoral Science Foundation
(2021M693479). Calculations were performed at the Physics Lab of High-
Performance Computing of Renmin University of China, Shanghai Supercomputer
Center. K.W. and T.T. acknowledge support from the Elemental Strategy
Initiative conducted by the MEXT, Japan (Grant NO. JPMXP0112101001), JSPS
KAKENHI (Grant Nos. 19H05790, 20H00354 and 21H05233) and the A3 Foresight by
JSPS.

## Authors contributions

Y.Y. and P.G. conceived the project. P.G. synthesized the CrOCl crystals and
fabricated the devices. P.G. conducted the transport measurements with the
help of Z.D., Q.W. and Z.H. C.W. conducted the DFT calculations under the
supervision of W.J. D.S. conducted the dielectric and pyroelectric measurement
under the supervision of Y.S. K.W. and T.T. grew the h-BN bulk crystals. P.G.
and Y.Y. drafted the manuscript. All authors discussed the results and
contributed to the manuscript.

## Competing interests

The authors declare no competing interests.

## References

  * [1] Jungwirth, T., Marti, X., Wadley, P. & Wunderlich, J.  Antiferromagnetic spintronics.  _Nat. Nanotechnol._ 11, 231–241 (2016).
  * [2] Fukami, S., Zhang, C., DuttaGupta, S., Kurenkov, A. & Ohno, H.  Magnetization switching by spin–orbit torque in an antiferromagnet–ferromagnet bilayer system.  _Nat. Mater._ 15, 535–541 (2016).
  * [3] Tsai, H. _et al._ Electrical manipulation of a topological antiferromagnetic state.  _Nature_ 580, 608–613 (2020).
  * [4] Wadley, P. _et al._ Electrical switching of an antiferromagnet.  _Science_ 351, 587–590 (2016).
  * [5] Matsukura, F., Tokura, Y. & Ohno, H.  Control of magnetism by electric fields.  _Nat. Nanotechnol._ 10, 209–220 (2015).
  * [6] Eerenstein, W., Mathur, N. & Scott, J. F.  Multiferroic and magnetoelectric materials.  _Nature_ 442, 759–765 (2006).
  * [7] Spaldin, N. A. & Ramesh, R.  Advances in magnetoelectric multiferroics.  _Nat. Mater._ 18, 203–212 (2019).
  * [8] Yang, J.-C., He, Q., Yu, P. & Chu, Y.-H.  $\rm{BiFeO_{3}}$ thin films: a playground for exploring electric-field control of multifunctionalities.  _Annu. Rev. Mater. Res._ 45, 249–275 (2015).
  * [9] Hill, N. A.  Why are there so few magnetic ferroelectrics?  _J. Phys. Chem. B_ 104, 6694–6709 (2000).
  * [10] Cheong, S.-W. & Mostovoy, M.  Multiferroics: a magnetic twist for ferroelectricity.  _Nat. Mater._ 6, 13–20 (2007).
  * [11] Tokura, Y., Seki, S. & Nagaosa, N.  Multiferroics of spin origin.  _Rep. Prog. Phys._ 77, 076501 (2014).
  * [12] Fiebig, M., Lottermoser, T., Fröhlich, D., Goltsev, A. V. & Pisarev, R. V.  Observation of coupled magnetic and electric domains.  _Nature_ 419, 818–820 (2002).
  * [13] Lottermoser, T. _et al._ Magnetic phase control by an electric field.  _Nature_ 430, 541–544 (2004).
  * [14] Lorenz, B., Litvinchuk, A. P., Gospodinov, M. M. & Chu, C.  Field-induced reentrant novel phase and a ferroelectric-magnetic order coupling in $\rm{HoMnO_{3}}$.  _Phys. Rev. Lett._ 92, 087204 (2004).
  * [15] Sergienko, I. A., Şen, C. & Dagotto, E.  Ferroelectricity in the magnetic e-phase of orthorhombic perovskites.  _Phys. Rev. Lett._ 97, 227204 (2006).
  * [16] Katsura, H., Nagaosa, N. & Balatsky, A. V.  Spin current and magnetoelectric effect in noncollinear magnets.  _Phys. Rev. Lett._ 95, 057205 (2005).
  * [17] Jia, C., Onoda, S., Nagaosa, N. & Han, J. H.  Bond electronic polarization induced by spin.  _Phys. Rev. B_ 74, 224444 (2006).
  * [18] Lin, X., Yang, W., Wang, K. L. & Zhao, W.  Two-dimensional spintronics for low-power electronics.  _Nat. Electron._ 2, 274–283 (2019).
  * [19] Sierra, J. F., Fabian, J., Kawakami, R. K., Roche, S. & Valenzuela, S. O.  Van der waals heterostructures for spintronics and opto-spintronics.  _Nat. Nanotechnol._ 16, 856–868 (2021).
  * [20] Huang, B. _et al._ Electrical control of 2D magnetism in bilayer $\rm{CrI_{3}}$.  _Nat. Nanotechnol._ 13, 544–548 (2018).
  * [21] Jiang, S., Shan, J. & Mak, K. F.  Electric-field switching of two-dimensional van der waals magnets.  _Nat. Mater._ 17, 406–410 (2018).
  * [22] Sharpe, A. L. _et al._ Emergent ferromagnetism near three-quarters filling in twisted bilayer graphene.  _Science_ 365, 605–608 (2019).
  * [23] He, W.-Y., Goldhaber-Gordon, D. & Law, K. T.  Giant orbital magnetoelectric effect and current-induced magnetization switching in twisted bilayer graphene.  _Nat. Commun._ 11, 1–8 (2020).
  * [24] Ju, H. _et al._ Possible persistence of multiferroic order down to bilayer limit of van der waals material $\rm{NiI_{2}}$.  _Nano Lett._ 21, 5126–5132 (2021).
  * [25] Song, Q. _et al._ Evidence for a single-layer van der waals multiferroic.  _Nature_ 602, 601–605 (2022).
  * [26] Park, C. B. _et al._ Observation of spin-induced ferroelectricity in a layered van der waals antiferromagnet $\rm{CuCrP_{2}S_{6}}$.  _Adv. Electron. Mater._ 2101072 (2022).
  * [27] Gu, P. _et al._ Magnetic phase transitions and magnetoelastic coupling in a two-dimensional stripy antiferromagnet.  _Nano Lett._ (2022).
  * [28] Wang, Y. _et al._ Flavoured quantum hall phase in graphene/crocl heterostructures.  _arXiv preprint arXiv:2110.02899_ (2021).
  * [29] Yang, K. _et al._ Realization of graphene logics in an exciton-enhanced insulating phase.  _arXiv preprint arXiv:2110.02921_ (2021).
  * [30] Noerlund Christensen, A., Johansson, T. & Quézel, S.  Preparation and Magnetic Properties of CrOCl.  _Chem. Informationsdienst_ 6, 1171–1174 (1975).
  * [31] Reuvekamp, P. G.  _Investigation into the magnetic and the structural properties of two low-dimensional antiferromagnets $\rm{TiPO_{4}}$ and CrOCl_.  Ph.D. thesis, University of Stuttgart (2014).
  * [32] Zhang, J.  _Temperature dependent magnetic order in FeOCl type compounds_.  Ph.D. thesis, University of Bayreuth (2014).
  * [33] Angelkort, J., Wölfel, A., Schönleber, A., van Smaalen, S. & Kremer, R. K.  Observation of strong magnetoelastic coupling in a first-order phase transition of crocl.  _Phys. Rev. B_ 80, 144416 (2009).
  * [34] Balatsky, A., Basov, D. & Zhu, J.-X.  Induction of charge density waves by spin density waves in iron-based superconductors.  _Phys. Rev. B_ 82, 144522 (2010).
  * [35] Zachar, O., Kivelson, S. & Emery, V.  Landau theory of stripe phases in cuprates and nickelates.  _Phys. Rev. B_ 57, 1422 (1998).
  * [36] Kaushik, N. _et al._ Reversible hysteresis inversion in $\rm{MoS_{2}}$ field effect transistors.  _npj 2D Mater. Appl._ 1, 1–9 (2017).
  * [37] Kresse, G. & Joubert, D.  From ultrasoft pseudopotentials to the projector augmented-wave method.  _Phys. Rev. B_ 59, 1758 (1999).
  * [38] Kresse, G. & Furthmüller, J.  Efficiency of ab-initio total energy calculations for metals and semiconductors using a plane-wave basis set.  _Comput. Mater. Sci._ 6, 15–50 (1996).
  * [39] Dion, M., Rydberg, H., Schröder, E., Langreth, D. C. & Lundqvist, B. I.  Van der waals density functional for general geometries.  _Phys. Rev. Lett._ 92, 246401 (2004).
  * [40] Klimeš, J., Bowler, D. R. & Michaelides, A.  Van der waals density functionals applied to solids.  _Phys. Rev. B_ 83, 195131 (2011).
  * [41] Qiao, J. _et al._ Few-layer tellurium: one-dimensional-like layered elementary semiconductor with striking physical properties.  _Sci. Bull._ 63, 159–168 (2018).
  * [42] Cococcioni, M. & De Gironcoli, S.  Linear response approach to the calculation of the effective interaction parameters in the LDA+U method.  _Phys. Rev. B_ 71, 035105 (2005).
  * [43] Wang, C. _et al._ A family of high-temperature ferromagnetic monolayers with locked spin-dichroism-mobility anisotropy: MnNX and CrCX (X= Cl, Br, I; C= S, Se, Te).  _Sci. Bull._ 64, 293–300 (2019).
  * [44] Jiang, P. _et al._ Stacking tunable interlayer magnetism in bilayer $\rm{CrI_{3}}$.  _Phys. Rev. B_ 99, 144401 (2019).
  * [45] Baroni, S., De Gironcoli, S., Dal Corso, A. & Giannozzi, P.  Phonons and related crystal properties from density-functional perturbation theory.  _Rev. Mod. Phys._ 73, 515 (2001).


# LLM-assisted Concept Discovery: Automatically Identifying and Explaining
Neuron Functions

Xuan-Nhat Hoang1 Minh Vu1 My T. Thai1 1University of Florida {hoangx,
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Providing textual concept-based explanations for neurons in deep neural
networks (DNNs) is of importance in understanding how a DNN model works. Prior
works have associated concepts with neurons based on examples of concepts or a
pre-defined set of concepts, thus limiting possible explanations to what the
user expects, especially in discovering new concepts. Furthermore, defining
the set of concepts requires manual work from the user, either by directly
specifying them or collecting examples. To overcome these, we propose to
leverage multimodal large language models for automatic and open-ended concept
discovery. We show that, without a restricted set of pre-defined concepts, our
method gives rise to novel interpretable concepts that are more faithful to
the model’s behavior. To quantify this, we validate each concept by generating
examples and counterexamples and evaluating the neuron’s response on this new
set of images. Collectively, our method can discover concepts and
simultaneously validate them, providing a credible automated tool to explain
deep neural networks.

##  1 Introduction

As large deep neural networks become more prevalent in everyday life, so does
the need to understand their decision-making process to ensure their
trustworthiness. Careful analysis of deep neural networks (DNNs) by looking at
individual neurons and their combinations, has yielded valuable insights on
how DNNs function Cammarata et al. (2020a); Goh et al. (2021); Bricken et al.
(2023). For example, in the GPT-2 model, a specific set of neurons has been
discovered to perform the natural language task of indirect object
identification Wang et al. (2022). In the vision domain, some neurons have
been found to detect curves Cammarata et al. (2020b). However, these analyses
require extensive human labor in examining the neurons and coming up with
possible explanations.

Concept-based explanations seek to characterize the model’s global behavior
via concepts, which are related to how human reasons Yeh et al. (2020). These
concepts serve to provide an intuition on the common characteristics of the
inputs that the network recognizes. They can be more intuitive and require
less time to inspect than individual inputs.

In providing concept-based explanations to neurons in DNNs, we seek to
characterize each neuron by a textual concept that represents a common trait
of the inputs activating this neuron. Moreover, we aim to do this
automatically, reducing the need for human interference in defining the
concepts and validating the generated output. Recently, Large Language Models
(LLMs) have been leveraged to automatically generate explanations for other
language modelsBills et al. (2023); Singh et al. (2023). LLMs are further
utilized to score the resulting explanations, creating a complete framework.
It is natural to pose the question of whether a textual concept-based
explanation method with the same degree of automation can be realized in the
vision domain. In fact, this work demonstrates that, by utilizing LLMs, we can
overcome the limitations of existing works Hernandez et al. (2022); Kalibhat
et al. (2023) in terms of the explainable concepts’ space and the explaining
language. Those limitations come from the fact that the existing explaining
algorithms strongly depend on human annotations or curation of a large image-
text dataset.

In particular, we propose a method to automatically discover concepts for
neural representations. Our method probes the neuron with an image-only
dataset, identifies an interpretable concept among the highly activating
images, and explains it using a Multimodal Large Language Model (MLLM). Our
method is training-free, does not require examples of a concept, and can
recognize concepts in an open-ended manner. Beyond concept discovery, we
propose a procedure to evaluate a proposed concept independently of the
probing dataset. Our score compares the activation of a neuron on examples
possessing the concept and non-examples containing the concept’s co-hyponyms.
This serves to quantify the ability of the neuron in telling the concept from
other semantically similar concepts. Intuitively, given a concept produced by
our concept discovery method, these co-hyponyms represent the concepts one
would test the neuron on to ensure the specificity of the proposed concept,
therefore, our method helps with automating this process. To verify the
effectiveness of our method, we apply our approach in examining various pre-
trained models and demonstrate how improved explanations can help with scaling
up the analysis of neural networks.

The rest of this paper is organized as follows. Section 2 discusses the
related work and some preliminaries. Section 3 describes our concept discovery
method. Section 4 shows how we compute our validation score. We conduct
experiments to show the effectiveness of our method in Section 5, and finally
Section 6 presents our conclusion.

##  2 Preliminary and Related Work

##### Visual neuron-concept association

Network Dissection Bau et al. (2017) proposes to label neurons with textual
concepts by computing the similarity between the output feature map and the
output of a segmentation model. Since then, many methods Mu and Andreas
(2020); Oikarinen and Weng (2023); Bykov et al. (2023) have been developed to
associate concepts with neurons using different criteria. Since they only
focus on matching the concepts, they usually require a pre-defined set of
concepts as input or default to simple alternatives (e.g., 20,000 most common
English words Oikarinen and Weng (2023)). As such, these methods rely on the
user anticipating the possible concepts, or their vocabulary will be severely
limited. Because the association methods only work when comparing different
concepts, in our work, we propose an example-based causality criterion to
evaluate a single concept that can also be used for neuron-concept
association.

##### Automatic vision concept discovery

Unlike concept association, where concepts are provided, concept discovery
needs need to come up with a concept from the vocabulary. FALCON Kalibhat et
al. (2023) addresses the challenge of a limited concept set by extracting
terms from a large image-text dataset like LAION-400M Schuhmann et al. (2021).
While they benefit from the variety of concepts, they can only output words
and phrases present in the dataset, greatly limiting the flexibility of their
output. On the other hand, MILAN Hernandez et al. (2022) is the first
generative method that can generalize to different domains and tasks. However,
the method requires a specific dataset of images with masks annotated
concepts, posing challenges in obtaining such annotated data. Our method
exploits the inherent capabilities of MLLMs and uses them for concept
discovery, eliminating the need for annotated data. As MLLMs improve across
domains, our method will directly inherit those improvements, while previous
works will require additional collection of data.

##### Multimodal large language models

MLLMs are similar to LLMs in the sense that they both accept and yield text,
however, MLLMs can accept an image as additional context. MLLMs can connect
information present in images and text, allowing it to answer questions such
as “What does this image describe?”

Available MLLMs generally belong to one of two categories: open-source MLLMs
and proprietary MLLMs. While open-source MLLMs allow the fine-tuning of MLLMs
for a specific task, proprietary MLLMs are often much larger, more capable,
and easier to integrate, only requiring an external API call. We elect to use
a proprietary MLLM to ensure the highest quality for the task of concept
discovery. We compare with open-source MLLMs in ablation tests in the
supplementary material.

##  3 LLM-assisted Concept Discovery

In this section, we first describe our Concept Discovery problem in Section
3.1. Since it is infeasible to directly solve that problem, we introduce the
LLM-assisted Concept Discovery algorithm that, given a neuron, automatically
generates a concept $c$ that the neuron captures. The algorithm consists of 3
main steps, which are illustrated in Fig. 1. Sect. 3.2 describes the first
step selecting a set of inputs that highly activate the neuron, called the
exemplar representations $E_{f}(\mu)$. To ensure the algorithm returns a
single interpretable concept, the second step (Sect. 3.3) eliminates inputs
with low cosine similarity from $E_{f}(\mu)$ and returns a more filtered set.
Given that, the final step of the algorithm utilizes a MLLM to generate a
concept explaining the examined neuron (Sect. 3.4).

###  3.1 Problem formulation

We consider a neural network $\Phi:\mathcal{X}\rightarrow\mathcal{Y}$, where
$\mathcal{X}$ and $\mathcal{Y}$ denote the input and output domains,
respectively. Given an arbitrary neuron in $\Phi$, we refer to it by its
activation function $f:\mathcal{X}\rightarrow\mathbb{R}$. Given a dataset
$D\subset\mathcal{X}^{*}$ and a concept $c$, we define $D_{c}$ and
$D_{\bar{c}}$ the set of $D$’s elements that contains and does not contain
$c$, respectively:

$\displaystyle D_{c}:=\\{x\in D|\mathcal{O}_{c}(x)=1\\}$ $\displaystyle
D_{\bar{c}}:=\\{x\in D|\mathcal{O}_{c}(x)=0\\}$

where $\mathcal{O}_{c}(x)$ is an oracle determining whether or not the concept
$c$ occurs in $x$, i.e., $\mathcal{O}_{c}(x)=1$ iff $c$ appears in $x$.

Given a neuron or, equivalently, its activation $f$, the goal of this work is
to find a descriptive concept $c$ that the neuron captures: we say a neuron
$f$ captures a concept $c$ if $f$ can differentiate the elements of $D_{c}$
from the elements of $D_{\bar{c}}$. More formally, we aim to generate the
concept that maximizes:

$\displaystyle
c^{*}=\operatorname*{arg\,max}_{c}\textup{Pr}[f(X_{1})>f(X_{2})|X_{1}\in
D_{c},X_{2}\in D_{\bar{c}}]$ (1)

In other words, we aim to find the concept $c$ such that the activation of a
random input $X_{1}$ sampled from $D_{c}$ has a high probability of being
greater than that of an $X_{2}$ sampled from $D_{\bar{c}}$. The intuition for
this objective is, for a solution $c$ of (1), we can expect that $f(x)$ for an
$x\in D_{c}$ is generally greater than $f(x),x\in D_{\bar{c}}$. It implies
that we can directly use $f(x)$ to differentiate $D_{c}$ from $D_{\bar{c}}$,
i.e., $f(x)$ indeed capture the concept $c$.

There are several challenges to directly solve (1). First, there is no trivial
way to search, iterate, and/or optimize over the solution’s space, i.e., the
space of concepts, especially when it might not even be a metric space.
Secondly, even if we can construct a metric space on the set of concepts,
computing the actual value of (1) is not trivial. The reasons are the absence
of the oracle $\mathcal{O}$ in practice and complex behavior $f$, which
prevent efficient computations of the probability.

Figure 1: Overview of our LLM-assisted concept discovery algorithm. Given a
neuron of interest, we generate an exemplar representation from the probing
dataset. Next, we find an interpretable concept by looking for a group of
images with high average CLIP similarity. Finally, we prompt the MLLM with the
combined image for the common concept that the neuron activates on.

###  3.2 Exemplar representation

The first step of the algorithm is to generate a set of exemplar
representations Hernandez et al. (2022), which is given as:

$\displaystyle E_{f}(\mu):=\\{x\in X|f(x)\geq\mu\\}$ (2)

where $\mu$ is a threshold parameter. Intuitively, $E_{f}(\mu)$ is the set of
inputs that highly activate the neuron’s activation, i.e., the inputs whose
activation at the neuron $f$ is not lower than $\mu$.

Although the exemplar representations $E_{f}(\mu)$ may not fully characterize
the neuron, previous studies have demonstrated that meaningful information can
still be extracted from this set alone. For example, Bau et al. (2020, 2019)
utilize it to identify the neurons that trigger certain class predictions or
generate specific objects. Andreas et al. (2018) employ these sets to forecast
output sequence for newly observed inputs. Additionally, Olah et al. (2018)
and Mu and Andreas (2020) use them to pinpoint adversarial vulnerabilities.
Following previous research, the first step of our proposed concept discovery
algorithm also represents the examined neuron as the set of exemplar
representations. In particular, we use a single dataset $D_{probe}$ consisting
only of images. For all neurons we wish to explain in a model, we record the
neuron activations on this dataset and extract the exemplar representation
from the top activating images of this dataset. In the next sections, we
explain how we utilize a MLLM to extract the desired concept from the exemplar
representation.

###  3.3 Filtering the exemplar representation for interpretable concept
identification

Given an exemplar representation, we wish to extract a concept that is common
to this set of images. However, it is natural to pose the question whether
such a concept always exists. Neurons are known to be polysemantic, whereby a
single neuron can respond to multiple unrelated concepts O’Mahony et al.
(2023); Olah et al. (2020). This polysemantic character of a neuron is
expected to appear and influence the exemplar representations, making it
challenging to identify a single concept explaining the neuron. Secondly, when
the neuron exhibits polysemy, the MLLM might fail to generate a meaningful
output, whose reason is discussed later in Section 3.4.

To address those issues, prior to extracting, the second step of our algorithm
involves the selection of a subset of $E_{f}{(\mu)}$ that promotes the
discovery of a single interpretable concept. Specifically, we adopt the
observation of Kalibhat et al. (2023) that members of $E_{f}{(\mu)}$ with high
average cosine similarity are more likely to correspond to a single
interpretable semantic concept. As such, we consider the problem of
determining this subset as a problem of finding a subgraph with minimal
diameter. Let $h$ denote the CLIP vision encoder. We construct a weighted
complete graph $G=(V,E)$ where $V\equiv E_{f}(\mu)$ and a distance function
$d:E\rightarrow\mathbb{R}$ such that $d(u,v)=\left\lVert
h(u)-h(v)\right\rVert_{2}$. The $M$ images to be kept are those in the
following solution:

$\displaystyle S^{*}(G)=\operatorname*{arg\,min}_{S\subset
V,|S|=M}\max\\{d(u,v)|u,v\in S\\}$ (3)

This problem is reducible to the clique decision problem, hence it is NP-
complete. However, since $M$ and $|V|$ are small, the solution can be found in
a short time with binary search and an efficient clique-finding routine. This
subset is then used to prompt the MLLM, as further described in Section 3.4.

###  3.4 MLLM as concept proposer

Directly extracting the concept $c$ from the filtered exemplar representation
is still non-trivial due to the modality gap. This problem can be viewed as a
summarization task, as we wish to concisely describe the common feature of a
set of images with only a short phrase. However, this multi-image
summarization task is less explored compared to its natural language
counterpart, the multi-document summarization, due to the scarcity of related
datasets and models Trieu et al. (2020). MILAN Hernandez et al. (2022) tackled
the initial challenge by designing a specific dataset for neuron explanations
and training the explanation model. However, adopting the method to another
domain requires the collection of new data with specialized labels, which can
be costly and/or even infeasible in many applications. In contrast, our work
leverages the recent advancement of MLLM and utilizes it as a concept
proposer.

We utilize an API-based MLLM (GPT-4V), as it is more capable than open-source
alternatives. Note that, since we have no access to the intermediate
representations of the inputs nor their gradients, the task needs to be fully
specified by the inputs alone. To this end, we pose the problem as a Visual
Question Answering (VQA) task where we give the MLLM images and ask for a
common visual concept. The MLLM that we use can take multiple images, however,
its ability to comprehend dozens of images is not well-studied. Furthermore,
doing so would incur tremendous API costs, reducing the utility of our method.
To make our method adaptable to the majority of MLLMs which can take only one
image and to reduce API costs, our algorithm transforms $E_{f}(\mu)$ into a
singular image while preserving the needed information. This is achieved by
downsampling the images and arranging them in a grid-like pattern to form a
single image, as shown in Figure 1. As shown by previous work Dunlap et al.
(2023), MLLMs are capable of understanding this kind of “condensed” image.

Despite the MLLM’s great capabilities, we observed that it can produce
unwanted outputs. Particularly, it can produce concepts that are too abstract
or unhelpful for the task (for example, “this image features a variety of
objects in different settings).” Since we cannot “train” the model, we try to
guide the model towards desired outputs by putting additional information in
the prompt. For LLMs, the common method is to perform In-context learning,
where examples of inputs and answers are presented in the prompt so the LLM
can “learn” from them. In our case, doing so necessitates multiple image
inputs, which would dramatically raise the API costs. We opt for a simpler
alternative: we collect unwanted outputs during initial trials and use them as
examples of bad answers in the final prompt. This strategy reduces those
unwanted concepts to some extent, while keeping the input length and costs
reasonable.

Another challenge in utilizing MLLMs is the presence of built-in safety
mechanisms. While general-purpose MLLMs possess the necessary general
knowledge for automatic novel concept discovery, they are also designed to
reject certain types of prompts OpenAI (2023). In the case of the MLLM
deployed in our method (GPT-4V), the prompt characteristics leading to model
refusal have not been explicitly provided. However, through our trials, we
observed that GPT-4V tends to refuse instances in which the exemplar
representation does not appear to contain a single identifiable common
concept. Consequently, we hypothesize that the model rejects responses when it
lacks confidence, possibly as a measure to mitigate misinformation. This
motivates us to have an additional processing step before prompting the MLLM,
which is discussed in the section below.

##  4 Concept validation

Beyond concept discovery, in this paper, we propose a score to validate the
discovered concept. Our evaluation takes in a neuron and its proposed concept
$c$, and returns a score which can be used to quantify whether the neuron’s
activation is consistently higher for inputs containing $c$. The overview of
our evaluation is depicted in Fig. 2.

The following sections are organized as follows. In Section 4.1, we formulate
our scoring objective and relate it to our previous goal of concept discovery
in (1). Section 4.2 describes our proposed idea of finding harder non-examples
to better evaluate the proposed concept. Section 4.3 and Section 4.4 discuss
how our evaluation can be realized.

Figure 2: Overview of our concept evaluation. Given a concept, we use Chain-
of-Thought prompting on an LLM to generate its co-hyponyms. Then, we use
another prompt to generate captions for the concept and the co-hyponyms. The
last step utilizes a diffusion model to transform the caption into the sets of
examples and non-examples $E_{c}$ and $E_{\bar{c}}$

###  4.1 Objective formulation

Since the objective of concept discovery is (1), it is intuitive to use the
following score to evaluate our solution:

$\displaystyle s(c)=\textup{Pr}[f(X_{1})>f(X_{2})|X_{1}\in D_{c},X_{2}\in
D_{\bar{c}}]$ (4)

As mentioned previously in Section 3.1, it is non-trivial to compute this
score due to the absence of the oracle $\mathcal{O}$. Previous approach Bykov
et al. (2023) overcomes this issue by relying on the availability of concept
labels along with the dataset, which may not be the case in many scenarios.
Even if we could approximate the oracle $\mathcal{O}$, finding a comprehensive
dataset $D$ such that $D_{c}\neq\emptyset$ for all concept $c$ learned by a
neural network is non-trivial. To address this issue, we propose a generative
approach to obtain sets of examples and non-examples, $E_{c}$ and
$E_{\bar{c}}$, as substitutes for $D_{c}$ and $D_{\bar{c}}$ in the evaluation
(4). Given $E_{c}$ and $E_{\bar{c}}$, we compute the following score:

$\displaystyle s(c)=\frac{\sum_{x_{1}\in E_{c}}\sum_{x_{2}\in
E_{\bar{c}}}1[f(x_{1})>f(x_{2})]}{|E_{c}|\cdot|E_{\bar{c}}|}$ (5)

The intuition is the same as discussed in Section 3.1: when we say a neuron
activates on concept $c$, we expect the value of $f(x),x\in E_{c}$ is higher
than $f(x),x\in E_{\bar{c}}$. This score is closely related to the Area Under
the Receiver Operating Characteristic curve (AUC), a metric used to evaluate
binary classifiers, however, they are different due to our definition of
$E_{c}$ and $E_{\bar{c}}$.

Our validation process consists of four steps. Given a concept $c$, the first
step is to generate the set of co-hyponyms of $c$, whose definition we will
give in the next sub-section. Next, we generate captions of images that
contain $c$ and its co-hyponyms. Then, the captions are used to generate the
sets of examples and non-examples $E_{c}$ and $E_{\bar{c}}$. Finally, those
sets are used to compute the score in Equation (5).

####  4.1.1 Definitions

Before we further describe how we generate $E_{c}$ and $E_{\bar{c}}$, we state
some definitions that we will make use of extensively in the later sections.

##### Hypernym

A hypernym of a concept is another concept that has a broader meaning. For
example, $\mathtt{animal}$ is a hypernym of $\mathtt{dog}$ since a dog is a
type of animal.

##### Hyponym

A hyponym of a concept is another concept that has a more specific meaning.
For example, $\mathtt{dog}$ is a hyponym of $\mathtt{animal}$.

##### Co-hyponym

Co-hyponyms of a concept are concepts that share the same hypernym with that
concept. For example, $\mathtt{pembroke}$ is a co-hyponym of
$\mathtt{golden\;retriever}$ as they both are hyponyms of the concept
$\mathtt{dog}$.

###  4.2 Sampling hard non-examples

Simply sampling random images to obtain the negative examples $E_{\bar{c}}$
will not give us a good evaluation score $s(c)$ as random images might not
contain enough concepts to distinguish the evaluated concept $c$ from other
semantically similar concepts. For example, let’s consider the scenario where
the true concept $c=\mathtt{golden\;retriever}$, the candidate concept
$c^{\prime}=\mathtt{pembroke}$, and the set $E_{\bar{c}}$ contains images of
other non-dog animals. Then, both $s(c)$ and $s(c^{\prime})$ will likely be
high since both concepts will result in a low activation for all samples in
$E_{\bar{c}}$. The reason is that the members of $E_{\bar{c}}$ are both
non-$\mathtt{golden\;retriever}$ and non-$\mathtt{pembroke}$ as they are all
non-dog. As a result, a more careful sampling strategy to obtain $E_{\bar{c}}$
is needed to make a higher score more meaningful.

In particular, we wish to generate $E_{\bar{c}}$ in a way that our score is
high only when the neuron activates on that specific concept. To this end, our
evaluation uses a concept’s co-hyponyms to generate $E_{\bar{c}}$. The
intuition is illustrated in the previous scenario: the neuron not activating
on $\mathtt{pembroke}$ is a stronger evidence for
$c=\mathtt{golden\;retriever}$ than the neuron not activating on random
images, given the similarity between $\mathtt{pembroke}$ and
$\mathtt{golden\;retriever}$ (they are both breeds of dogs).

###  4.3 Finding co-hyponyms with an LLM

For the first step of our validation process, given a concept $c$, we wish to
find a set of its co-hyponyms to serve as strong non-examples for validation.
While the task of finding co-hyponyms suggests the usage of a lexical database
with hyponym relations built in like WordNet Miller (1994), such operations
are only possible for single concepts. For instance, humans can intuitively
think of non-examples of “red sedans” as “blue sedans,” yet such concepts are
absent in WordNet. Instead, we leverage the flexibility of an LLM to generate
co-hyponyms for all concepts. This reduces the need for ad-hoc methods and
post-processing and enhances automation, which suits the goal of our method.

To use an LLM for the task of finding co-hyponyms, we use a two-step process,
inspired by Chain-of-Thought prompting Wei et al. (2022). First, given a
concept $c$, we ask the LLM to find a hypernym of it. Then, in the same
prompt, we ask it to generate concepts that share the same (previously
generated) hypernym with $c$. In this setting, Chain-of-Thought prompting can
improve performance by breaking down the complex problem into two smaller
tasks. It also enables us to partially understand how the LLM come up with the
answer.

There are two common failures the LLM can make. First, it can give a hypernym
that is broader than what we want. For example, when asked for a hypernym of
“red sadans”, the LLM tends to answer “cars”, instead of “sedans of a
particular color”. Second, it might come up with a co-hyponym that is not
entirely exclusive of the original concept. For instance, while it can
correctly come up with the hypernym “shapes” for the concept “lines”, it might
use “rectangles” as a co-hyponym. These behaviors can be attributed to the
inherent limitations of LLMs having a limited understanding of concepts,
compared to humans Shani et al. (2023). Nevertheless, we alleviate these
issues by using examples to trigger in-context learning Dong et al. (2023).

###  4.4 From concepts to images

Now that we have the concept to be evaluated and its co-hyponyms, we wish to
generate a set of examples and non-examples. We utilize a text-to-image
diffusion model for this task. The diffusion model takes in a description of
an image, which we will call a caption, and generates an image matching that
caption. As this model expects a descriptive caption rather than a single
word, we need to first convert the concepts to captions. For each concept, we
want its captions to describe images containing that concept, but not its co-
hyponyms. Likewise, we do not want the captions corresponding to the co-
hypnoyms to unintentionally contain the original concept. To this end, we
generate captions by considering pairs of concept and co-hyponym. For each
pair, we explicitly ask the LLM to generate a pair of captions that contain
one concept but not the other. Finally, the captions are passed to the
diffusion model for image generation.

Method | $D$ | Interpretable concepts
---|---|---
FALCON | ImageNet val | 11/100
Ours | ImageNet val | 99/100
Table 1: Performance when labeling the first 100 neurons of the final layer of
a ResNet50 trained on ImageNet.

Unit Ours ImageNet class 0 people holding fish tench 1 goldfish goldfish 2
sharks great white shark 3 sharks tiger shark 4 sharks hammerhead 5 spotted
creatures electric ray 6 stingrays stingray 7 red combs on chickens cock 8
chickens hen 9 flightless birds with long necks ostrich

Table 2: Descriptions of final layer neurons of a ResNet50 pre-trained on
ImageNet.

##  5 Experiments

We conduct experiments to show the effectiveness of our method, comparing it
to other baselines in terms of explanation quality. Section 5.1 provides
details on our method’s implementation and the experiment setting. Section 5.2
describes our experiment on ImageNet pre-trained models. In Section 5.3, we
apply our method to the popular CLIP model and show the effectiveness of both
concept discovery and concept validation. Due to the page constraint, we put
more experiments, including ablation tests, more qualitative evaluations, a
user study and experiments on different layers in the supplementary materials.
The exact prompt used for the MLLM and the LLM will also be provided in the
supp. mat.

###  5.1 Implementation

In this section, we provide the implementaion details of our method, including
the specific model choice and the parameters used. In the subsequent
experiments, the configurations are as follows unless otherwise noted. The
source code will be made available.

For concept discovery, we use GPT-4V OpenAI (2023) as the MLLM. To extract the
exemplar representations, we let $\mu$ be equal to the $50$-th highest
activation for each neuron, and sample $M=36$ images using (3). The vision
encoder $h$ is CLIP ViT-B/32 Radford et al. (2021).

For concept validation, we utilize LLaMA 2 13B Touvron et al. (2023) as our
LLM, both in co-hyponym generation and caption generation. To turn captions
into images, we use Stable Diffusion XL Turbo Sauer et al. (2023) with
$\texttt{num\\_inference\\_steps}=4$ for enhanced prompt alignment. For each
concept, we generate $5$ co-hyponyms, then, we generate $2$ pairs of captions
for each pair of concept and co-hyponym. Finally, for each caption, we
generate $5$ images, resulting in $|E_{\bar{c}}|=|E_{c}|=5\times 2\times
5=50$.

Unit FALCON Ours ImageNet class 48 iguana,grass,face reptiles Komodo dragon 63
snake,head,carpet snakes Indian cobra 70 people,person,harvestman spiders
harvestman 71 swatch,carpet,iii scorpions scorpion 96 parrot,yellow,head
toucans toucan 98 water,baby,march waterfowl red merganser

Table 3: Comparison of descriptions generated by FALCON and our method on
final layer neurons of ResNet50 pre-trained on ImageNet, on cases where FALCON
produced an explanation.

(a) Ours

(b) MILAN

Figure 3: Count of occurrences of the top-50 most frequent words in the
explanations of the last layer of CLIP ResNet50.

###  5.2 Accuracy of explanations

It is especially challenging to evaluate the accuracy of the generated
explanations as we do not have the groundtruth concept in most cases. A common
approach is evaluating on neurons whose expected behavior is given by the
task, such as the final layer of classification models Oikarinen and Weng
(2023); Bykov et al. (2023). In this experiment, we compute explanations for
the first 100 neurons of the final layer of ResNet50. We use the ImageNet-1K
validation set as $D_{probe}$. We compare our results with FALCON Kalibhat et
al. (2023), which we produce using their official code. Note that we do not
compare with MILAN for this experiments because MILAN relies on thresholding
the feature maps to obtain the exemplar representation, which the fully-
connected layer we are explaining does not possess.

Figure 4: Examples of cases where our method produce a more detailed concept.
All neurons are labeled as “Text” by MILAN.

Tables 1, 2, and 3 demonstrate our results. Note that we were able to
interpret all but one out of the first 100 neurons, while FALCON was only able
to recognize 11. From Table 2, it can be seen that our method produces the
exact ImageNet class, a superclass of it, or a related concept. Table 3
compares our descriptions with FALCON’s top-3 words, in cases where they
produced an explanation. While they were able to generate the exact concept in
some case (unit 70), in general, our explanations are more consistently
correct.

###  5.3 Analyzing CLIP vision encoder

To show that our method can discover more insightful concepts on practical
DNNs, we experiment with explaining the final layer of the backbone of a CLIP-
ResNet50 Radford et al. (2021). To be precise, we generate explanations for
the first 512 of the total 2048 neurons of the last layer (prior to attention
pooling layer) of the backbone. We use a $1$M subset of LAION-400M Schuhmann
et al. (2021) as $D_{probe}$. In this experiment, we set $\mu$ to the $100$-th
highest activation to account for a larger probing dataset. We compare our
results with MILAN Hernandez et al. (2022). For FALCON, their method generally
only explains a small portion of neurons that they deem interpretable.
Furthermore, this ratio depends on the choice of the threshold parameter. In
contrast, our method and MILAN both aim to generate an explanation for an
arbitrary neuron. As such, FALCON requires a different evaluation, which we
provide in the supplementary material.

####  5.3.1 Evaluating explanation variety

Figure 5: Example of concept validation in action. We show the process of
evaluating our proposed concept “nfl team merchandise”, which results in a
score of $0.984$ on the full set of $E_{c}$ and $E_{\bar{c}}$. We italicise
and underline important words in the captions that are directly related to the
concept or the co-hyponym.

We analyze the generated explanations on a corpus level to show that our
method generates more detailed concepts. In particular, we consider all
explanations of each method as a corpus. Figure 3 illustrates the word-level
analysis for each corpus. It counts the number of occurrences of each word in
a corpus and plots the top 50 of them, as well as some other corpus-level
statistics. Overall, we can see that the number of unique words (abbr. vocab)
and the number of words that appear only once (abbr. hapax) are significantly
higher than those of MILAN. This suggests that our method produce more
specific concepts that is tailored to each neuron. Looking at the body of the
distribution, MILAN labels neurons with generic concepts more frequently
(“text”: $175+$ usages versus ours’ $50$, “people”: $50+$ vs. $30-$).

Observing that MILAN labels a significant $168$ out of $512$ neurons as
“text”, we seek to compare those with our own explanations. In contrast with
MILAN, we were able to extract a variety of different concepts. Fig. 4 shows
some examples of the results. In many cases, we discover the specific word on
which the neuron activates. In other cases, we find the neuron activates on
text coming from a particular subject, as in the “video game” neuron. The
results demonstrate that our method can automatically produce more specific
concepts that provide more insights on the functionality of neurons.

####  5.3.2 Explanation scores

We demonstrate how the evaluation methodology proposed in Section 4 works in
practice. First, we show an example in Figure 5 with the concept to be
evaluated, the co-hyponyms, the captions, and the set of examples and non-
examples. Note that this is only a portion of the sets; we generate $100$
images in total for both examples and non-examples combined as described in
Section 5.1. We can observe that the exclusivity of the concept and its co-
hyponyms was successfully transmitted to the captions and then to the images.

We plot the distribution of the score for each neuron in Figure 6. The
distribution suggests that our concepts better represents the chactersistic of
the input that cause the neuron to fire strongly. Examples in Fig. 4 supports
this. We further evaluate our results with more qualitative examples and a
user study, which we provide in the supplementary materials.

Figure 6: Distribution of explanation scores computed by our concept
validation. The validated concepts are from explaining the last layer of CLIP
ResNet50. Note that all cases where we get a zero score are from GPT-4V’s
refusal to answer.

##  6 Conclusion

Previous methods for concept-based neuron explanations suffer from limited
output language and require a curated dataset that defines the concepts. Our
concept discovery method addresses this issue, leading to interpretable
concepts that are more insightful and distinctive. To evaluate the generated
concepts without any groundtruth, our evaluation method provides a meaningful
to help users judge the quality of the result. Future work can seek to use
this score as a signal to create a feedback loop, where the discovery method
improves on its answer automatically.

## References

  * Andreas et al. [2018] Jacob Andreas, Dan Klein, and Sergey Levine.  Learning with Latent Language.  In Marilyn Walker, Heng Ji, and Amanda Stent, editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2166–2179, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
  * Bau et al. [2017] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Network Dissection: Quantifying Interpretability of Deep Visual Representations.  pages 6541–6549, 2017.
  * Bau et al. [2019] David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman, and Antonio Torralba.  GAN Dissection: Visualizing and Understanding Generative Adversarial Networks.  In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
  * Bau et al. [2020] David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba.  Understanding the role of individual units in a deep neural network.  Proceedings of the National Academy of Sciences, 117(48):30071–30078, December 2020.  Publisher: Proceedings of the National Academy of Sciences.
  * Bills et al. [2023] Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders.  Language models can explain neurons in language models, 2023.
  * Bricken et al. [2023] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah.  Towards Monosemanticity: Decomposing Language Models With Dictionary Learning.  Transformer Circuits Thread, 2023.
  * Bykov et al. [2023] Kirill Bykov, Laura Kopf, Shinichi Nakajima, Marius Kloft, and Marina MC Höhne.  Labeling Neural Representations with Inverse Recognition.  In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  * Cammarata et al. [2020a] Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Michael Petrov, Ludwig Schubert, Chelsea Voss, Ben Egan, and Swee Kiat Lim.  Thread: Circuits.  Distill, 2020.
  * Cammarata et al. [2020b] Nick Cammarata, Gabriel Goh, Shan Carter, Ludwig Schubert, Michael Petrov, and Chris Olah.  Curve Detectors.  Distill, 5(6):e00024.003, June 2020.
  * Dong et al. [2023] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui.  A Survey on In-context Learning, June 2023.  arXiv:2301.00234 [cs].
  * Dunlap et al. [2023] Lisa Dunlap, Yuhui Zhang, Xiaohan Wang, Ruiqi Zhong, Trevor Darrell, Jacob Steinhardt, Joseph E. Gonzalez, and Serena Yeung-Levy.  Describing Differences in Image Sets with Natural Language, December 2023.  arXiv:2312.02974 [cs].
  * Goh et al. [2021] Gabriel Goh, Nick Cammarata †, Chelsea Voss †, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah.  Multimodal Neurons in Artificial Neural Networks.  Distill, 6(3):e30, March 2021.
  * Hernandez et al. [2022] Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas.  Natural Language Descriptions of Deep Visual Features.  In International Conference on Learning Representations, 2022.
  * Kalibhat et al. [2023] Neha Kalibhat, Shweta Bhardwaj, C. Bayan Bruss, Hamed Firooz, Maziar Sanjabi, and Soheil Feizi.  Identifying Interpretable Subspaces in Image Representations.  In Proceedings of the 40th International Conference on Machine Learning, pages 15623–15638. PMLR, July 2023.  ISSN: 2640-3498.
  * Miller [1994] George A. Miller.  WordNet: A Lexical Database for English.  In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994, 1994.
  * Mu and Andreas [2020] Jesse Mu and Jacob Andreas.  Compositional explanations of neurons.  In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, pages 17153–17163, Red Hook, NY, USA, December 2020. Curran Associates Inc.
  * Oikarinen and Weng [2023] Tuomas Oikarinen and Tsui-Wei Weng.  CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networks.  In The Eleventh International Conference on Learning Representations, 2023.
  * Olah et al. [2018] Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev.  The Building Blocks of Interpretability.  Distill, 2018.
  * Olah et al. [2020] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter.  Zoom In: An Introduction to Circuits.  Distill, 2020.
  * O’Mahony et al. [2023] Laura O’Mahony, Vincent Andrearczyk, Henning Müller, and Mara Graziani.  Disentangling Neuron Representations with Concept Vectors.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3769–3774, 2023.
  * OpenAI [2023] OpenAI.  GPT-4 Technical Report, December 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning Transferable Visual Models From Natural Language Supervision.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR, 2021.
  * Sauer et al. [2023] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach.  Adversarial Diffusion Distillation, November 2023.  arXiv:2311.17042 [cs].
  * Schuhmann et al. [2021] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs, November 2021.  arXiv:2111.02114 [cs].
  * Shani et al. [2023] Chen Shani, Jilles Vreeken, and Dafna Shahaf.  Towards Concept-Aware Large Language Models.  In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
  * Singh et al. [2023] Chandan Singh, Aliyah R. Hsu, Richard Antonello, Shailee Jain, Alexander G. Huth, Bin Yu, and Jianfeng Gao.  Explaining black box text modules in natural language with language models, May 2023.  arXiv:2305.09863 [cs, q-bio].
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open Foundation and Fine-Tuned Chat Models, July 2023.  arXiv:2307.09288 [cs].
  * Trieu et al. [2020] Nicholas Trieu, Sebastian Goodman, Pradyumna Narayana, Kazoo Sone, and Radu Soricut.  Multi-Image Summarization: Textual Summary from a Set of Cohesive Images, June 2020.  arXiv:2006.08686 [cs].
  * Wang et al. [2022] Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt.  Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small.  September 2022.
  * Wei et al. [2022] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou.  Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.  May 2022.
  * Yeh et al. [2020] Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar.  On Completeness-aware Concept-Based Explanations in Deep Neural Networks.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 20554–20565. Curran Associates, Inc., 2020.


A note on homological systems

Jesús Efrén Pérez Terrazas

###### Abstract

We give an elementary short proof of the known fact that the category
$\mathfrak{F}\left(\Delta\right)$ of $\Delta-$filtered modules, associated to
a given finite homological system $\left(\Delta;\Omega,\leq\right),$ is closed
under direct summands.

Homological systems are generalizations of the standard modules, associated to
quasi-hereditary algebras, and so they have similar features (see [2] and
[1]).

Homological systems in module categories over pre-ordered sets were introduced
in [1]. There they show, see Corollary 3.16, that the associated category of
the $\Delta-$filtered modules is closed under direct summands: if $M$ is a
$\Delta-$filtered module and there is a decomposition $M=M_{1}\oplus M_{2},$
then $M_{1}$ and $M_{2}$ are $\Delta-$filtered modules.

In this note we use a _height map associated to a pre-ordered set_ defined
below, a function determined by the relative position of the elements of the
pre-ordered set, to provide a direct proof of that property.

From now on, $\Lambda$ is an Artin $R-$algebra, where $R$ is an artinian
commutative ring, $\Lambda-$mod is the full subcategory of the finitely
generated left $\Lambda-$modules, and all modules and homomorphisms considered
in this note belong to $\Lambda-$mod.

######  Definition 0.1.

A homological system $\left(\Delta;\Omega,\leq\right)$ consists of the
following:

HS1: A finite pre-ordered set $\left(\Omega,\leq\right):$ $\Omega$ is a finite
not empty set and $\leq$ is a reflexive and transitive relation on $\Omega.$

HS2: The set $\Delta=\left\\{\Delta_{\omega}\right\\}_{\omega\in\Omega},$
where $\Delta_{\omega}$ is indecomposable in $\Lambda-$mod and
$\omega\neq\omega^{\prime}$ implies
$\Delta_{\omega}\ncong\Delta_{\omega^{\prime}}.$

HS3: If ${\rm
Hom}_{\Lambda}\left(\Delta_{\omega},\Delta_{\omega^{\prime}}\right)\neq 0$
then $\omega\leq\omega^{\prime}.$

HS4: If ${\rm
Ext}_{\Lambda}\left(\Delta_{\omega},\Delta_{\omega^{\prime}}\right)\neq 0$
then $\omega\leq\omega^{\prime}$ and it is not true that
$\omega^{\prime}\leq\omega.$

######  Definition 0.2.

Given the set $\Delta=\left\\{\Delta_{\omega}\right\\}_{\omega\in\Omega},$ as
in HS2, denote by $\mathfrak{F}\left(\Delta\right)$ the full subcategory of
$\Lambda-$mod of those $M$ having a $\Delta-$filtration, that is a filtration
of the form

$\\{0\\}=M_{0}\subsetneq M_{1}\subsetneq\cdots\subsetneq M_{t}=M$

such that $M_{i}/M_{i-1}\cong\Delta_{\omega},$ for some $\omega$ that depends
of $i,$ and for each $i\in\left\\{1,2,\ldots,t\right\\}.$ So
$\mathfrak{F}\left(\Delta\right)$ is closed under isomorphisms, extensions and
it contains zero objects. The modules in $\mathfrak{F}\left(\Delta\right)$ are
called $\Delta-$filtered modules.

Given a pre-ordered set $\left(\Omega,\leq\right),$ the relation
$\omega\sim\omega^{\prime}$ if and only if $\omega\leq\omega^{\prime}$ and
$\omega^{\prime}\leq\omega$ is an equivalence relation, and so there is a
canonical surjective order-preserving function
$\pi:\Omega\longrightarrow\Omega/\sim,$ where
$\left(\Omega/\sim,\preceq\right)$ is a poset and $\preceq$ is the order
induced by $\leq.$

######  Definition 0.3.

Let $\left(\Omega^{\prime},\preceq\right)$ be a finite poset. The height
function $h^{\prime}:\Omega^{\prime}\longrightarrow\mathbb{N}$ of the poset is
defined recursively: the elements of height 1 are the minimal elements of the
poset, and the elements of height $n+1$ are the minimal elements of the
induced poset on
$\Omega-\left\\{\omega\in\Omega\;\mid\;h^{\prime}\left(\omega\right)\in\left\\{1,2,\ldots,n\right\\}\right\\}.$
Given a finite pre-ordered set $\left(\Omega,\leq\right),$ the composition
$h:\Omega\stackrel{{\scriptstyle\pi}}{{\longrightarrow}}\Omega/\sim\stackrel{{\scriptstyle
h^{\prime}}}{{\longrightarrow}}\mathbb{N}$ is the height function of the pre-
ordered set.

######  Remark 0.4.

Given a finite pre-ordered set $\left(\Omega,\leq\right)$ and its height
function $h,$ if $\omega\leq\omega^{\prime}$ and $\omega^{\prime}\nleq\omega$
then $h\left(\omega\right)<h\left(\omega^{\prime}\right),$ and if
$h\left(\omega\right)<h\left(\omega^{\prime}\right)$ then
$\omega^{\prime}\nleq\omega.$

So we get, for $\left(\Delta;\Omega,\leq\right)$ a homological system, that if
$h\left(\omega\right)>h\left(\omega^{\prime}\right)$ then ${\rm
Hom}_{\Lambda}\left(\Delta_{\omega},\Delta_{\omega^{\prime}}\right)=0,$ and if
$h\left(\omega\right)\geq h\left(\omega^{\prime}\right)$ then ${\rm
Ext}_{\Lambda}\left(\Delta_{\omega},\Delta_{\omega^{\prime}}\right)=0.$

######  Definition 0.5.

Let $F=\left\\{M_{0}\subsetneq M_{1}\subsetneq\cdots\subsetneq M_{t}\right\\}$
be a $\Delta-$filtration of $M\in\mathfrak{F}\left(\Delta\right).$ We will
denote by $\ell\left(F\right)=t$ the length of $F,$ and by
$\ell_{\omega}\left(F\right)$ the number of factors isomorphic to
$\Delta_{\omega},$ and so
$\ell\left(F\right)=\sum_{\omega\in\Omega}\ell_{\omega}\left(F\right).$

######  Proposition 0.6.

Let $\left(\Delta;\Omega,\leq\right)$ be a homological system,
$M\in\mathfrak{F}\left(\Delta\right)-\\{0\\},$ and $F=\left\\{M_{0}\subsetneq
M_{1}\subsetneq\cdots\subsetneq M_{t}\right\\}$ be a $\Delta-$filtration. Then
there exists a $\Delta-$filtration
$F^{\prime}=\left\\{M^{\prime}_{0}\subsetneq
M^{\prime}_{1}\subsetneq\cdots\subsetneq M^{\prime}_{t}\right\\}$ of $M$ with
$\ell_{\omega}\left(F\right)=\ell_{\omega}\left(F^{\prime}\right),$ for each
$\omega\in\Omega,$ and such that: $1\leq i\leq j\leq t,$
$M^{\prime}_{i}/M^{\prime}_{i-1}\cong\Delta_{\omega}$ and
$M^{\prime}_{j}/M^{\prime}_{j-1}\cong\Delta_{\omega^{\prime}},$ imply
$h\left(\omega\right)\geq h\left(\omega^{\prime}\right).$

Proof: With the notation of the statement, but starting with the filtration
$F,$ let us assume that for a fixed $i$ and $j=i+1$ we have
$h\left(\omega\right)<h\left(\omega^{\prime}\right).$ Then, from HS4, we have
that the exact sequence

$\textstyle{0\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{M_{i}/M_{i-1}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{M_{i+1}/M_{i-1}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{M_{i+1}/M_{i}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{0}$

splits, so there exists $\overline{N}$ submodule of $M_{i+1}/M_{i-1}$ such
that $\overline{N}\cong\Delta_{\omega^{\prime}}$ and
$\left(M_{i+1}/M_{i-1}\right)/\overline{N}\cong\Delta_{\omega}.$

From the third theorem of isomorphism we get for $N,$ the pre-image of
$\overline{N}$ under the canonical epimorphism $p:M_{i+1}\longrightarrow
M_{i+1}/M_{i-1},$ that $F_{1}=\left\\{M_{0}\subsetneq
M_{1}\subsetneq\cdots\subsetneq M_{i-1}\subsetneq N\subsetneq
M_{i+1}\subsetneq\cdots\subsetneq M_{t}\right\\}$ is a $\Delta-$filtration
with $\ell_{\omega}\left(F\right)=\ell_{\omega}\left(F_{1}\right)$ for each
$\omega,$ satisfying the order condition in positions $i$ and $i+1.$

We can repeat this process generating new $\Delta-$filtrations, and in a
finite number of steps to obtain one as in the statement. $\hfill\square$

For the proof of the following proposition, let us recall that given
$L,N\in\Lambda-$Mod, the _trace_ ${\rm tr}_{L}\left(N\right)$ of $L$ on $N$ is
the sum of all the images of the homomorphisms from $L$ to $N.$

######  Proposition 0.7.

Let $\left(\Delta;\Omega,\leq\right)$ be a homological system and
$M\in\mathfrak{F}\left(\Delta\right).$ Let $\left\\{1,2,\ldots,a\right\\}$ be
the image of the height function $h.$ There exists a filtration
$\\{0\\}=W_{a+1}\subseteq W_{a}\subseteq\cdots\subseteq W_{1}=M$ such that

$W_{i}/W_{i+1}\cong\displaystyle\bigoplus_{\begin{subarray}{c}\omega\in\Omega\\\
h\left(\omega\right)=i\end{subarray}}n_{\omega}\Delta_{\omega}$

with $n_{\omega}\in\\{0\\}\cup\mathbb{N}.$ Let us call a such filtration an
$h-$filtration. Also, the numbers $n^{\prime}_{\omega}$ of any other
$h-$filtration are the same.

Proof: The existence of at least one $h-$filtration follows directly from the
Proposition 0.6 and HS4.

Now assume that $\\{0\\}=W^{\prime}_{a+1}\subseteq
W^{\prime}_{a}\subseteq\cdots\subseteq W^{\prime}_{1}=M$ is another
$h-$filtration. By HS3 and the definition of trace we have

$W_{a}={\rm tr}_{W_{a}}\left(W_{a}\right)={\rm tr}_{W_{a}}\left(M\right)={\rm
tr}_{W_{a}}\left(W^{\prime}_{a}\right)=W^{\prime}_{a}$

and so, by the Krull-Schmidt-Remak Theorem, it follows that
$n_{\omega}=n^{\prime}_{\omega}$ for any $\omega\in\Omega$ of height $a.$

We repeat the argument in the quotients $M/W_{a},$ $M/W_{a-1},$ …, $M/W_{2}$
in order to verify the statement. $\hfill\square$

######  Corollary 0.8.

Let $\left(\Delta;\Omega,\leq\right)$ be a homological system.

  1. 1.

Let $M\in\mathfrak{F}\left(\Delta\right)$ and $F$ and $F^{\prime}$ be
$\Delta-$filtrations of $M.$ Then
$\ell_{\omega}\left(F\right)=\ell_{\omega}\left(F^{\prime}\right)$ for each
$\omega\in\Omega.$ It follows that the $\Delta-$length and the number of
$\Delta-$factors are well defined for $M.$

  2. 2.

Let $L,N\in\mathfrak{F}\left(\Delta\right)$ and
$\textstyle{0\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{L\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{M\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{N\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{0}$
be an exact sequence. Then
$\ell_{\omega}\left(M\right)=\ell_{\omega}\left(L\right)+\ell_{\omega}\left(N\right)$
for each $\omega\in\Omega.$

  3. 3.

$\mathfrak{F}\left(\Delta\right)$ is closed under direct summands.

Proof:

1.- If $F$ is a $\Delta-$filtration of $M,$ and $\underline{F}$ is an
$h-$filtration obtained from $F,$ as in the proofs of the propositions 0.6 and
0.7, then $\ell_{\omega}\left(F\right)=n_{\omega}$ for each $\omega,$ where
$n_{\omega}$ are the coefficients associated to $\underline{F}$ as in 0.7.
Then the claim follows by the uniqueness of those coefficients.

2.- It is a direct consequence of the previous item.

3.- Consider an $h-$filtration of $M$ and assume that $M=M_{1}\oplus M_{2}.$

Then, by additivity of the trace, we have $W_{a}={\rm
tr}_{W_{a}}\left(M\right)={\rm tr}_{W_{a}}\left(M_{1}\right)\oplus{\rm
tr}_{W_{a}}\left(M_{2}\right),$ and so ${\rm tr}_{W_{a}}\left(M_{j}\right)$ is
a submodule of $M_{j}$ which is in $\mathfrak{F}\left(\Delta\right),$ for
$j\in\left\\{1,2\right\\}.$

We can repeat this argument for the quotients $M/W_{a}\cong\left(M_{1}/{\rm
tr}_{W_{a}}\left(M_{1}\right)\right)\oplus\left(M_{2}/{\rm
tr}_{W_{a}}\left(M_{2}\right)\right)$ in order to get bigger submodules of
$M_{1}$ and $M_{2}$ that are in $\mathfrak{F}\left(\Delta\right).$

Repeating the procedure we obtain $h-$filtrations of $M_{1}$ and $M_{2},$ so
they belong to $\mathfrak{F}\left(\Delta\right).$ $\hfill\square$

## References

  * [1] O. Mendoza, C. Sáenz, C. Xi. Homological Systems in Module Categories over Pre-ordered Sets. Quart. J. Math. 60, (2009) 75-103.
  * [2] C. M. Ringel. The category of $\Delta-$good modules over a quasi-hereditary algebra has almost split sequences. Math. Z. 208, (1991) 209-223.

E. Pérez

Facultad de Matemáticas

Universidad Autónoma de Yucatán

Mérida, México

<EMAIL_ADDRESS>