# Low temperature ($<$ 200∘C) solution processed tunable flash memory device
without tunneling and blocking layer

Sandip Mondal1,2, V Venkataraman2 1SanDisk India Device Design Center,
Bangalore 560103, India  2Department of Physics, Indian Institute of Science,
Bangalore 560012, India

Intrinsic charge trap capacitive non-volatile flash memories take a
significant share of the semiconductor electronics market today. It is a
challenge to create intrinsic traps in the dielectric layer without high
temperature processing stepsHsinChiangYou2006 . While low temperature
processed memory devices fabricated from polymers have been demonstrated as an
alternativeLee2007 ; Kapetanakis2008 ; Tsai2013 ; Chiu2014a ; Chiu2015 ;
Wei2012 , their performance degrade rapidly after a few cycles of
operationBaeg2014 ; Park2015a ; Kim2010a ; Faber2009 ; Park2010c ; Han2011a ;
Wang2010 . Moreover conventional memory devices need the support of tunneling
and blocking layers since the memory dielectric or polymer is incapable of
preventing memory leakageKim2010b ; Yi2016a ; Lee2009 . The main issue in
designing a memory device is to optimize the leakage current and intrinsic
trap density simultaneouslyLee2011b . Here we report a tunable flash memory
device without tunneling and blocking layer by combining the discovery of high
intrinsic charge traps ($>$1012 cm-2) together with low leakage
current($<$10-7 A.cm-2) in solution derived, inorganic, spin$-$coated
dielectric films which were heated at 200∘C or below. In addition, the memory
storage is tuned systematically upto 96% by controlling the trap density with
increasing heating temperature.

Today’s semiconductor memory technology is dominated by silicon-oxide-nitride-
oxide-silicon (SONOS) non-volatile flash memory which is based on intrinsic
charge traps in silicon-rich silicon nitride films deposited by high
temperature ($\sim$ 780oC) compatible chemical vapor depositionXue2012 ;
Liu2012 . The intrinsic charge traps in silicon-rich silicon nitride films
were first reported in 1967 Hu1967 and the first flash memory device
incorporating silicon nitride charge storage was demonstrated in 1980’s
Junction1989 . However the trap density and distribution are difficult to
control in such materialPark2006 . Traps can be increased by ion bombardment
and plasma-passivationLiu2012 , but the leakage current increases. Alternate
high-k dielectrics such as TiO2, HfO2, ZrO2, etc. are excellent insulators for
transistor applications Avis2011 ; Suzuki2009b ; Dutta2013a ; Kumar2015a , but
do not have the intrinsic charge trapping properties as silicon nitride.
Although solution processed HfO2 has been used to fabricate SONOS type flash
memoryHsinChiangYou2006 , the devices required the support of additional
dielectric layers which were deposited by sophisticated ultrahigh vacuum
techniques with high temperature processing steps to improve the memory
leakage. For most dielectrics, precursor solutions with organic solvents
result in poor leakage current which can be improved to some extent by high
heating temperatureMondal2017 . However the high temperature heating process
lowers leakage current but reduces trap density. Hence, the main challenge is
to simultaneously achieve deep intrinsic charge traps together with very low
leakage current at low processing temperatures. There are a few reports on
solution processed flash memory by using polymer materialsChang2013 ; Shih2015
, but these devices degrade after only few cycles of operation in normal
environmental conditions and they are not capable of working at higher
temperatures.

In the last few years, a novel inorganic, completely carbon free, water
soluble dielectric Aluminium Oxide Phosphate (ALPO), has been successfully
employed as gate dielectric in high performance TFTs that are competitive with
a-Si TFTsKim2011 . Due to its very low leakage current density, it was used
recently as tunneling and blocking layer to fabricate fully solution processed
two terminal capacitive flash memory devicesMondal2016 with CdTe-NP as the
charge storage center. Nevertheless a fully spin$-$coated low temperature
processed (below 200∘C) high$-$performance flash memory device without
tunneling and blocking layers has not been reported so far.

Here we report the discovery of ultra-high number of intrinsic charge traps
($>$1012cm-2) in low temperature solution processed inorganic ALPO dielectric.
At the same time, the leakage current is sufficiently low ($<$ 10-7 A.cm-2)
for flash memory operation without incorporating tunneling or blocking layers.
In addition, the number of traps can be varied with heating temperature, which
is strongly correlated with the oxygen vacancy concentration in the film.
Furthermore, we demonstrate optimized robust high performance fully solution
processed inorganic oxide precursor based flash memory devices without
tunneling and blocking layer, where the processing temperature does not exceed
200∘C. Our devices outperform other similar memory devices reported
earlierLee2007 ; Kapetanakis2008 ; Tsai2013 ; Chiu2014a ; Chiu2015 ; Wei2012 .

Figure 1: Device and deep level charge trapping response:  (a) Schematic of
device architecture. (b) Cross-sectional SEM image of a typical device
(thickness $\sim$139 nm). (c)(top-panel) AFM topography of the surface of the
devices. (bottom-panel) Optical image of an array of devices. (d) $C-V$ traces
of as prepared (AP), heated at 200∘C and 600∘C respectively (also see Section
II, Supporting Information). The heating was done for 1 hr for each sample.
$C-V$ was measured with the up-down DC sweep of $\pm$ 20 V at a rate of 2V
min-1 on the gate (Gate Bias) of the MIS while imposing a small AC with
amplitude and frequency of 100 mV and 100 kHz respectively. (e) Variation of
hysteresis window ($\triangle$VFB) as a function of heating temperature.
(inset) variation of trap density as a function of heating temperature.

A schematic of typical device architecture is depicted in Fig. 1a. ALPO is
deposited by spin-coating a completely inorganic, carbon free and aqueous
precursor solution on low-doped $p$-silicon substrate ($p\sim$4$\times$1015
cm-3). After deposition various substrates are heated at different
temperatures including at low temperature such as 200∘C. The inset of Fig. 1a
shows a schematic of molecular structure of the low temperature (200∘C)
processed ALPO. The thickness of the deposited film is measured by
ellipsometry (see Section I, Supporting Information) and verified by cross-
sectional scanning electron microscope (SEM) image (Fig. 1b). Such prepared
films are found to be atomically smooth showing surface roughness to be
$\sim$0.08 nmMeyers2007a ; Mondal2017 which is determined by the atomic force
microscopy (AFM) with an areal scan over 50$\mu$m$\times$50$\mu$m (top-
panel,Fig. 1c). An aluminum contact is deposited thereafter by thermal
evaporation for the purpose of electrical measurement. The deep level charge
storage in the deposited ALPO is characterized via capacitance-voltage (CV)
measurement. An optical image of device on chip has been shown in the bottom-
panel of Fig. 1c.

Intrinsic trap levels in ALPO cause a hysteretic output of the CV traces
highlighting the memory-like behavior of the devices (Fig. 1d). Further, the
trap density can be varied by heating the samples at different temperatures
which alters the hysteresis-window. For example, an as-prepared sample shows a
memory window of $\sim$12.9 V (left-panel) which corresponds to a trap density
of n $\sim$5.65$\times$1012 cm-2, where as, an ALPO-film heated at 200∘C for
one hour shows a hysteresis window of $\sim$17.5 V which corresponds to the
trap density of n $\sim$ 6.37$\times$1012 cm-2. When the film is heate at
600∘C, the trap density drastically reduces (n $\sim 10^{11}$ cm-2) causing a
significantly low hysteresis window ($\sim 2.2$ V, right-panel). A similar
variation of electronic traps with heating temperature is also observed from
the devices which are made with lower thickness ALPO film, however, there is
no affect due to change in device dimension (see Section III, Supporting
Information). From the sweep direction of the CV curve it is inferred that
gate injection of carriers controls the memory operation. Although
conventional flash memory architecture follows channel injection of
carriersLiu2012 , the interface degrades fast in such devicesWu2008 . Thus,
compared to channel injected devices, gate-injected devices show higher
endurance which is one of the key requirements of memory operationLue2007 .
Because of such advantages, the quest for efficient gate-injected flash memory
devices is ongoingWu2008 ; Lue2007 ; Lue2008 ; YeolYun2014 . A systematic
change in the hysteresis window as a function of heating temperature is
presented in Fig. 1e, where, solid squares are the experimental data, shaded
region is guide to the eye and the width of the shaded region indicate the
standard deviation around their mean vertical position estimated from similar
results. Use of higher heating temperature ($>600^{\circ}$C) causes a dramatic
reduction in trap density, thus reducing the hysteresis window. A 96%
reduction in trap density ($n\sim 2.19\times$1011 cm-2) is observed at an
heating temperature of $\sim$800∘C resulting a memory window of 0.7 V only. In
addition, a negligible degradation in CV hysteresis is obtained from as
prepared devices even after 5 years of storage in ambient conditions (see
Section IV, Supporting Information). This demonstrates that the devices are
very stable for long term application, in spite of the water absorbing
property of ALPOPerkins2016 ; Anderson2015 .

Figure 2: Program-Disturb(PD) Measurement: (a) PD measurement with 20 times
$C-V$ sweep on same device. Inset: $C-V$ curve of first and 20th measurement.
(b) Flatband voltage shift ($\Delta V_{FB}$) as a function of gate sweep
voltage and frequency of sweep voltage. (Inset) Charge density ($n_{ALPO}$)
variation as a function of gate voltage ($V_{G}$). (c) Programming ($P$) and
erasing ($E$) operations of the flash memory device as a function of time
period ($t$) of the input pulse. Insets indicate shape of the pulses used for
program ($P$), erase ($E$) and read operations.

In order to investigate the robustness of intrinsic charge storage property of
ALPO, we demonstrate a series of memory operations on the metal-insulator-
semiconductor structured (MIS) two terminal capacitive devices which were
heated at 200∘C. The memory characterization focuses upon the flat band
voltage shift ($\Delta$VFB) due to the voltage sweep operation (i.e.
erasing($E$) or programming($P$) operation) on the gate terminal of the
devices. A device is programmed ($P$) or erased ($E$) by sweeping the gate
voltage at a slow rate Mondal2016 . Sequential $P$ and $E$ cycles set and
reset the flat band voltage reversibly and is confirmed from the nearly
constant $V_{FB}$ values obtained from multiple $E$-$P$ cycles (Fig. 2a). Two
of the $C$-$V$ traces corresponding to the 1st and 20th $E$-$P$ traces are
shown in the inset of Fig. 2a. Nearly constant values of $V_{FB}$ in $E$ and
$P$ states indicate that $E$-$P$ cycles have negligible effect on the memory
state thus indicating high reliability of the devices.

Frequency and gate bias dependent $\Delta~{}V_{FB}$ changes are shown in Fig.
2b. Flatband voltage window shows insignificant change within the range of
10-1000 kHz frequency of excitation voltage. Such behavior is attributed to
the slow trapping de-trapping phenomena which thus indicates that the states
can be probed by deploying high speed CV measurement. With different gate bias
voltages $\Delta~{}V_{FB}$ increases first and then shows a decrease. Such
behavior appears because of the competition between the trap filling and
charge losses. Initially ($\sim$5-20 V) as the sweep voltage is increased,
larger electric field causes more charge injection into the traps, hence, the
hysteresis width increases. Beyond 20 V, which corresponds to an electric
field of $\sim$1.2 MV.cm-1), the high leakage current (see Section V,
Supporting Information) leads to increased charge loss, thus a reduction in
hysteresis window is observed (Fig. 2b). A hysteresis window of 15 V is
obtained for a sweep voltage range of $\pm$ 15 V which corresponds to an
optimum memory window of 50% of the total sweep-range. The result of 50%
window for intrinsic traps in ALPO exceeds previously reported
valuesLehninger2015c for other dielectrics. It is known from literature that
the high performance operation of flash memory stack should not have leakage
current density more than 10-6 A.cm-2 when operated with an electric field of
1 MV.cm-1 ChiChangWu2010 . The ALPO devices show a leakage current density of
$\sim$4.5$\times$10-8 A.cm-2 only at -1 MV.cm-1 electric field. Such low
leakage current not only meets the criteria for the application of high-
performance flash memory devices, but also outperforms other solution
processed inorganic dielectrics namely Al2O3Avis2011 , HfO2Suzuki2009b ,
ZrO2Dutta2013a , and TiO2Kumar2015a , which show typical leakage currents
$\sim$10-5 A.cm-2, $\sim$10-7 A.cm-2, $\sim$10-2 A.cm-2 and $\sim$10-5 A.cm-2
respectively at 1 MV cm-1. This observation for ALPO also ensures that high
quality flash memory devices can be fabricated without the need of blocking or
tunneling layers.

True program ($P$) and erase ($E$) operations are realized by application of
positive and negative square pulses respectively, Whereas the read operation
is performed by using triangular pulses having shorter time periods (T $\sim$4
$\mu$s). Since high speed voltage sweep does not alter the memory state (Fig.
2b), shorter triangular pulses are expected to probe the $E$ and $P$ states
without disturbing them. By varying the width of the square-pulses, various
memory-windows ($V_{FB}|_{P}-V_{FB}|_{E}$) are obtained and shown in Fig. 2c.
While using a single pulse to program or erase, a pulse width of 500 ms can
set the memory window to be as large as $\sim$9 V. The program/erase speed of
these devices is found to be $\sim$200 ms which is significantly faster
compared to other solution processed flash memory devicesLehninger2015c ;
Park2014 ; Yun2009 ; Bar2015c ; Chen2011 . During programming and erasing, a
maximum charge capturing efficiency of 7.46 % is estimated while operating
with a gate electric field of 2.37 MV/cm (see Section VI, Supporting
Information). This capturing efficiency is higher than the reported values
obtained from other dielectricsZhang2007 .

Figure 3: Robust flash memory operation: (a) Program-disturb (PD) verification
with high-speed $C-V$ measurement system. (Inset) $C-V$ traces from low speed
(gray-traces) and high speed (red and green traces) measurements. Red colored
lines indicate back-and-forth sweeps at high speed (time period($T$)=4 $\mu$s)
while the sample is in programmed state. Green colored lines indicate back-
and-forth sweeps at high speed (time period($T$)=4 $\mu$s) while the sample is
in erased state. (b) Endurance characteristics of the device. (Inset) Shape of
the pulses used in program ($P$), erase ($E$) and read operations. (c)
Retention characteristics of the flash memory device. Programming of the state
was done with a square pulse of height $\pm$33 V and having a duration of 500
ms. (d) Retention characteristics of the flash memory device after endurance
test of 10k cycles. (e) $P/E$ test of the flash memory devices at various
temperatures. Lines are the guide to eye. (f) The statistical distribution of
flash memory window for 47 devices. For this test applied $P/E$ voltage is
$\pm$33 V for 300 ms.

To check for any disturbance of the program/erase state during the read
operation, the device is set to $E$ or $P$ states first with a single long
square pulse ($T\sim 0.5~{}$s, see Fig. 2c) and then multiple high-frequency
triangular ($T\sim 4~{}\mu$s) pulses were applied representing multiple read
operations. Figure 3a shows the statistical distribution of $V_{FB}$ values
obtained from such multiple read operations for both the $E$ and $P$ states.
Narrow distributions of $V_{FB}$ indicate low read-disturb for both the
states. Inset of Fig. 3a shows typical $C$-$V$ traces for $E$, $P$ and read
operations. Here two red and two green lines indicate back-and-forth sweep of
the voltage ($V_{G}$) using the high speed triangular read pulse probing the
$P$ and $E$ states respectively.

Erase-program operation over 10 k cycles shows no degradation in memory window
thus demonstrates high endurance (Fig. 3b). In fact, with the increasing
operation cycle memory window increases which may be attributed to the
generation of additional trap states because of electrical stress. Data
retention was tested by programming a fresh device at room temperature with
pulses of amplitude 33 V and duration 500 ms. The device shows almost no
change in memory window within $\sim$104s which was the limit of experimental
time scale. This is the highest reported retention time for any memory device
without additional tunneling and blocking layers. Even after 10 k $P$/$E$
cycles devices show a degradation of only 9% memory window after $\sim
10^{4}$s (Fig. 3d). Temperature dependent data retention was also tested and
presented in Fig. 2e. After 104s of operation a memory window loss of 25% and
50% are observed at 60∘C and 80∘C respectively. The reliability is also tested
on the device made with lower thickness of ALPO film (91 nm) which shows
equivalent performance as obtained from the other devices(see Section VII,
Supporting Information)

To examine the scalability of these low temperatures processed memory devices,
we prepared and characterized more than 40 devices. Statistical distribution
of memory window for all these devices are shown in Fig. 3f. 80% of devices
lie within the expected memory window (3-4V) when programmed with $\pm$33 V
pulse height and 300 ms duration. The remaining 20% devices show a memory
window of less than 3V. These preliminary results show sufficiently high yield
for large scale production needed for practical applications.

Figure 4: Temperature dependent changes in trap states:  (a) Core level O 1s
spectra from different temperature heated film. Thick arrow indicate movement
of the peaks obtained from samples heated with different temperatures. (b)-(d)
Core level oxygen peaks from as$-$prepared, 200∘C and 400∘C heated ALPO films
respectively. The oxygen peak(red) in each case is de-convoluted into three
components corresponding to oxygen vacancies (M-Ovac) $\sim$ green, lattice
oxygen (M-O) $\sim$ blue and metal hydroxide (M-OH) $\sim$ cyan. (e) Atomic
composition ratios of oxygen in ALPO thin films as a function of temperature.
(inset) increment of traps OVAC at 200∘C with respect to AP ALPO film. (f)
Schematics showing condensation mechanism of oxide precursors by heating. The
first block denotes the individual molecules, second and third are low and
high temperature processed ALPO films respectively. All molecular models were
constructed using MolView(http://molview.org/). Escape of water molecules
(black circle) shown in third block leads to the decrease in thetrap states as
well as the decrease of the thickness of the film.

Microscopic origin of memory operation is understood from the temperature
controlled trap density variation which is confirmed from photo-emission
spectroscopy (XPS). Multiple samples were prepared at different heating
temperatures and studied with XPS. The shape of the oxygen peak changes
systematically with respect to heating temperature (Fig. 4a). De-convolution
of XPS signal into respective peaks of oxygen vacancies(M-Ovac), metal
hydroxide(M-OH) and lattice oxygen(M-O) helps understanding the contributions
of respective states (Fig. 4b-d). The ALPO film which was heated at 200∘C has
similar ratio (atomic %) of aluminum (Al), oxygen (O) and phosphorous (P) as
that of the mother solution (see Section VIII, Supporting Information). Due to
addition of HCl in the ALPO precursor, a small percentage of chlorine is
observed (see Section IX, Supporting Information) in low temperature processed
ALPO film. It is observed that M-Ovac intensity sharply decreases with
increase in heating where temperature reaches above 200∘C (Fig. 4b-d and
Section X of Supporting Information). As the heating temperature increases,
hydrogen is lost as water vapor and the film becomes denser. The M-O peaks
increase with higher heating temperature (Fig. 4e-f ). This is consistent with
the reduction in the number of oxygen vacancies and consequently lower trap
density in samples heated at higher temperature. A quantitative comparison
between the peak intensity and the trap density of the heated film reveals a
strong correlation between the oxygen vacancies and trap densities (see
Section X, Supporting Information), thus indicating that the oxygen vacancies
are the responsible entities for memory states.

In conclusion we have shown fabrication and high quality performance of low
temperature processed ($\lesssim$200 ∘C) inorganic flash memory devices which
do not require tunneling or blocking layers. Simple sample fabrication
technique involving spin-coating of solution is another advantage. Narrow
distribution of memory window obtained from more than 40 samples indicates the
scalability of the fabrication method. In spite of having no tunneling and
blocking layers, these devices show extremely low leakage current which is one
of the key features of memory operation, and demonstrate high endurance, high
retention, thus, outperforming other solution processed memory devices
reported so far. Temperature dependent control on trap density also helps
optimizing the memory window. ALPO based devices paves the way for designing a
new class of scalable two terminal flash memory devices for practical
applications.

Table - I $|$ Memory Technology Comparison  Device Fab. Technique Structure
P/E Vol (V) P/E Time Memory Window Ref. ALD & RF Sputtering p-Si/SiO2/Ge-
NC/TaZrOx/Al2O3/Al 10 V 60 s 3.8 V Lehninger2015c PECVD & Thermal
Si/SiO2/Al@Al2O3-NP/SiO2/Al $\pm$ 15 V 5 s / 5 s 2.5 V Park2014 PVD & e-beam
p-Si/Al2O3/Pt-NP/Al2O3/Ti $\pm$ 7 V 5 s / 5 s 4.3 V Yun2009 RF Sputtering
p-Si/Al2O3/GeNC1L/Al2O3/Al 5 V / -5 V 3 s / 3 s 2.3 V Bar2015c PECVD &
Sputtering p-Si/SiO2/DyTixOy/Al2O3/Al 7 V / -10 V 1 s / 10 s 4V Chen2011
Solution Processed p-Si/ALPO/Al 30 V / -30V 500 ms / 500 ms 9 V This work

## Methods

Material growth:  Precursor solution of Aluminium Oxide Phosphate, ALPO
[Al2O3-3X (PO4)2X] in water (18 M$\Omega$ cm) was prepared with Al(OH)3 (99%,
Alfa Aesar, USA) in 2 mole equivalents of HCl (AR Grade, Thermo Fisher
Scientific, USA) and an appropriate amount of H3PO4 (ExcelaR Grade, Thermo
Fisher Scientific, USA) was added to obtain P/Al = 0.5 with concentration of
0.5 M . The solution was stirred under heat of 90∘C in a water bath for 24
hours.

Device fabrication:  Two terminal MIS structures were fabricated on piranha
cleaned [H2SO4:H2O2 = 3:1] lightly doped silicon substrate (p$-$Si). The ALPO
solution was spin-coated at 3000 rpm for 30s to form the gate dielectric. The
films were then heated at 150oC for 1 min. This process was repeated for 2-3
times to achieve the desired thickness. The sample was exposed to oxygen
plasma for 5 to 10 mins at 0.5 mbar pressure before deposition of each layer.
Such fabrication process may help generation of stable traps in ALPO below
200∘C. Typically, ALPO based devices are treated at high temperature
($>$350∘C) to achieve better dielectric performance, thus, most of the
reported devices do not show any memory effectKim2011 . For control devices
the film was further heated at 800oC for one hour in ambient to achieve the
trap free oxide. A 200 nm aluminium gate was deposited by thermal evaporation
at $\sim$ 10-6 mbar pressure. Before deposition of the top aluminium gate, the
ALPO films were heated at different temperature in a preheated furnace for one
hour. All devices were stored under ambient conditions and no degradation of
CV curves was found even after three years.

Characterization:  ALPO precursor solutions prepared with different
concentrations and P/Al ratios were optically characterized using a UV-Visible
spectrophotometer. They are all transparent in visible wavelength range with
the main absorption peak in the ultra-violet range at 235 nm. After spin
coating and heating, the film shows a refractive index 1.5 (see Section I,
Supporting Information) with negligible absorption at 550 nm as measured by
ellipsometry (M-2000, J.A. Woollam Co. Inc., USA). The thickness of ALPO film
on silicon substrate extracted from ellipsometry is 139 nm and verified with
cross-sectional SEM (Ultra 55, Carl Zeiss). The surface of the ALPO film is
atomically smooth with a roughness of 0.08 nm as measured by AFM (ND$-$MDT,
Russia). The CV curve was measured with HIOKI 3532 LCR meter and Keithley 2400
source meter. Agilent Device Analyzer B1500A was used to measure the IV
characteristics in ambient environment. The leakage current and CV
measurements were performed on more than 50 devices. The entire measurement
was done in continuous mode of the instrument. The high speed CV measurement
was performed with home made CV measurement systemMondal2016 . The XPS
measurements were carried out by AXIS 165 with Al K$\alpha$ radiation (9mA,
13keV, 1486.6 eV) in ultra$-$high vacuum. The XPS spectra were calibrated with
C 1s peak (284.6 eV).

## References

  * (1) Hsin-Chiang You _et al._ SONOS-type flash memory using an HfO$_2$ as a charge trapping layer deposited by the sol-gel spin-coating method.  _IEEE Electron Device Letters_ 27, 653–655 (2006).
  * (2) Lee, J.-S. _et al._ Layer-by-layer assembled charge-trap memory devices with adjustable electronic properties.  _Nature Nanotechnology_ 2, 790–795 (2007).
  * (3) Kapetanakis, E. _et al._ Molecular storage elements for proton memory devices.  _Advanced Materials_ 20, 4568–4574 (2008).
  * (4) Tsai, T.-D., Chang, J.-W., Wen, T.-C. & Guo, T.-F.  Manipulating the Hysteresis in Poly(vinyl alcohol)-Dielectric Organic Field-Effect Transistors Toward Memory Elements.  _Advanced Functional Materials_ 23, 4206–4214 (2013).
  * (5) Chiu, Y. C., Otsuka, I., Halila, S., Borsali, R. & Chen, W. C.  High-performance nonvolatile transistor memories of pentacence using the green electrets of sugar-based block copolymers and their supramolecules.  _Advanced Functional Materials_ 24, 4240–4249 (2014).
  * (6) Chiu, Y. C. _et al._ Oligosaccharide Carbohydrate Dielectrics toward High-Performance Non-volatile Transistor Memory Devices.  _Advanced Materials_ 27, 6257–6264 (2015).
  * (7) Wei, Q. _et al._ Additive-driven assembly of block copolymer-nanoparticle hybrid materials for solution processable floating gate memory.  _ACS Nano_ 6, 1188–1194 (2012).
  * (8) Baeg, K. J. _et al._ Charge-Trap Flash-Memory Oxide Transistors Enabled by Copper-Zirconia Composites.  _Advanced Materials_ 26, 7170–7177 (2014).
  * (9) Park, Y.-S. & Lee, J.-S.  Design of an Efficient Charge-Trapping Layer with a Built-In Tunnel Barrier for Reliable Organic-Transistor Memory.  _Advanced Materials_ 27, 706–711 (2015).
  * (10) Kim, S. J. & Lee, J. S.  Flexible organic transistor memory devices.  _Nano Letters_ 10, 2884–2890 (2010).
  * (11) Faber, H. _et al._ Low-Temperature Solution-Processed Memory Transistors Based on Zinc Oxide Nanoparticles.  _Advanced Materials_ 21, 3099–3104 (2009).
  * (12) Park, Y. S. _et al._ High-performance organic charge trap flash memory devices based on ink-jet printed 6,13-bis(triisopropylsilylethynyl) pentacene transistors.  _Applied Physics Letters_ 96, 1–4 (2010).
  * (13) Han, S.-T., Zhou, Y., Xu, Z.-X., Roy, V. A. L. & Hung, T. F.  Nanoparticle size dependent threshold voltage shifts in organic memory transistors.  _Journal of Materials Chemistry_ 21, 14575 (2011).
  * (14) Wang, S. M., Leung, C. W. & Chan, P. K.  Nonvolatile organic transistor-memory devices using various thicknesses of silver nanoparticle layers.  _Applied Physics Letters_ 97 (2010).
  * (15) Kim, Y. M., Kim, S. J. & Lee, J. S.  Organic-transistor-based nano-floating-gate memory devices having multistack charge-trapping layers.  _IEEE Electron Device Letters_ 31, 503–505 (2010).
  * (16) Yi, M. _et al._ The effect of porous structure of PMMA tunneling dielectric layer on the performance of nonvolatile floating-gate organic field-effect transistor memory devices.  _Organic Electronics: physics, materials, applications_ 33, 95–101 (2016).
  * (17) Lee, J. S. _et al._ Tunable memory characteristics of nanostructured, nonvolatile charge trap memory devices based on a binary mixture of metal nanoparticles as a charge trapping layer.  _Advanced Materials_ 21, 178–183 (2009).
  * (18) Lee, J. S.  Review paper: Nano-floating gate memory devices.  _Electronic Materials Letters_ 7, 175–183 (2011).
  * (19) Xue, J. & Wang, G.  SONOS Flash Memory.  _United States Patent [US 7,977,734 B2]_ 2 (2012).
  * (20) Liu, S.-h., Yang, W.-l., Wu, C.-c. & Chao, T.-s.  A Novel Ion-Bombarded and Plasma-Passivated Charge Storage Layer for SONOS-Type Nonvolatile Memory.  _Elec. Dev. Lett._ 33, 1393–1395 (2012).
  * (21) Hu, S. M.  Evidence of hole injection and trapping in silicon nitride films prepared by reactive sputtering.  _Applied Physics Letters_ 10, 97 (1967).
  * (22) Roy S. Bass, Jr., Arup Bhattacharyya, G. D. G.  Non-volatile memory cell having Si rich silicon nitride charge trapping layer.  _United States Patent [19]_ 14 (1989).
  * (23) Park, Y. _et al._ Highly manufacturable 32Gb multi-Level NAND flash memory with 0.0098 $\mu$m2 cell size using TANOS(Si-Oxide-Al2O3 - TaN) cell technology.  _Technical Digest - International Electron Devices Meeting, IEDM_ 2, 5–8 (2006).
  * (24) Avis, C. & Jang, J.  High-performance solution processed oxide TFT with aluminum oxide gate dielectric fabricated by a sol-gel method.  _J. Mater. Chem._ 21, 10649 (2011).
  * (25) Suzuki, K. & Kato, K.  Characterization of high-k HfO2 films prepared using chemically modified alkoxy-derived solutions.  _Journal of Applied Physics_ 105, 061631 (2009).
  * (26) Dutta, S. _et al._ Growth and electrical properties of spin coated ultrathin ZrO2 films on silicon.  _Journal of Applied Physics_ 114 (2013).
  * (27) Kumar, A., Mondal, S., Kumar, S. G. & Koteswara Rao, K.  High performance sol-gel spin-coated titanium dioxide dielectric based MOS structures.  _Materials Science in Semiconductor Processing_ 40, 77–83 (2015).
  * (28) Sandip Mondal and V. Venkataraman.  Tunable electron affinity with electronic band alignment of solution processed dielectric.  _Appl. Phys. Lett._ 111, 041602 (2017).
  * (29) Chang, Y.-M. _et al._ A hot hole-programmed and low-temperature-formed SONOS flash memory.  _Nanoscale research letters_ 8, 340 (2013).
  * (30) Shih, C. C., Chiu, Y. C., Lee, W. Y., Chen, J. Y. & Chen, W. C.  Conjugated polymer nanoparticles as nano floating gate electrets for high performance nonvolatile organic transistor memory devices.  _Advanced Functional Materials_ 25, 1511–1519 (2015).
  * (31) Kim, K. M. _et al._ Competitive device performance of low-temperature and all-solution-processed metal-oxide thin-film transistors.  _Applied Physics Letters_ 99, 242109 (2011).
  * (32) Mondal, S. & Venkataraman, V.  All Inorganic Spin-Coated Nanoparticle-Based Capacitive Memory Devices.  _IEEE Electron Device Letters_ 37, 396–399 (2016).
  * (33) Meyers, S. T. _et al._ Solution-processed aluminum oxide phosphate thin-film dielectrics.  _Chemistry of Materials_ 19, 4023–4029 (2007).
  * (34) Wu, D., Huang, R., Wang, P., Tang, P. & Wang, Y.  A low-voltage flash memory cell utilizing the gate-injection program/erase method with a recessed channel structure.  _Semiconductor Science and Technology_ 23 (2008).
  * (35) Lue, H. T. _et al._ A novel gate-injection program/erase p-channel NAND-type flash memory with high (10M cycle) endurance.  _Digest of Technical Papers - Symposium on VLSI Technology_ 140–141 (2007).
  * (36) Lue, H.-t., Hsu, T.-h. & Lai, E.-k.  Cell oparation methods using gate-injection for floating gate NAND flash memory.  _US Patent 2008/0080248_ 1 (2008).
  * (37) Yeol Yun, D., Hyun Lee, N., Seong Kim, H., Wook Lee, S. & Whan Kim, T.  Multilevel charging and discharging mechanisms of nonvolatile memory devices based on nanocomposites consisting of monolayered Au nanoparticles embedded in a polystyrene layer.  _Applied Physics Letters_ 104, 023304 (2014).
  * (38) Perkins, C. K. _et al._ Aqueous process to limit hydration of thin-film inorganic oxides.  _Solid State Sciences_ 61, 106–110 (2016).
  * (39) Anderson, J. T. _et al._ Chemically amplified dehydration of thin oxide films.  _ACS Sustainable Chemistry and Engineering_ 3, 1081–1085 (2015).
  * (40) Lehninger, D. _et al._ Charge trapping of Ge-nanocrystals embedded in TaZrOx dielectric films.  _Applied Physics Letters_ 106 (2015).
  * (41) Chi-Chang Wu _et al._ A Robust Data Retention Characteristic of Sol Gel-Derived Nanocrystal Memory by Hot-Hole Trapping.  _IEEE Electron Device Letters_ 31, 746–748 (2010).
  * (42) Park, S. J. & Yoon, J. H.  Synthesis and memory properties of a self-assembled Al@Al2O3 core-shell nanoparticle layer for floating gate devices.  _Applied Physics Letters_ 104, 10–14 (2014).
  * (43) Yun, M., Mueller, D. W., Hossain, M., Misra, V. & Gangopadhyay, S.  Sub-2 nm size-tunable high-density Pt nanoparticle embedded nonvolatile memory.  _IEEE Electron Device Letters_ 30, 1362–1364 (2009).
  * (44) Bar, R. _et al._ Multilayer Ge nanocrystals embedded within Al2O3 matrix for high performance floating gate memory devices.  _Appl. Phys. Lett._ 107, 093102 (1–5) (2015).
  * (45) Chen, F. H., Pan, T. M. & Chiu, F. C.  Metal-oxide-high-k-oxide-silicon memory device using a Ti-doped Dy2O3 charge-trapping layer and Al2O3 blocking layer.  _IEEE Transactions on Electron Devices_ 58, 3847–3851 (2011).
  * (46) Zhang, G., Wang, X. P., Yoo, W. J. & Li, M. F.  Spatial distribution of charge traps in a SONOS-type Flash memory using a high-$\kappa$ trapping layer.  _IEEE Transactions on Electron Devices_ 54, 3317–3324 (2007).

## Acknowledgment

This work was supported by the grant (MITO$-$084) of Center for Nano Science
and Engineering (CeNSE), Indian Institute of Science, Bengaluru, India. S.M.
would like to thank Dr. Kallol Roy for useful discussions during manuscript
preparation. S.M. also thanks Chandan and Arvind Kumar for help to analyze the
data. S.M. also thanks the Rajiv Gandhi National Fellowship of the University
Grants Commission (UGC) for fellowship support.

## Author contributions

S.M., V.V. conceived the project. S.M. designed and performed the experiments,
completed characterization of the fully solution processed flash memory
device. S.M. analyzed results and wrote the paper.

## Additional information

Supplementary information is available in the online version of the paper.
Correspondence and requests for materials should be addressed to S.M.


# Exploring the LLM Journey from Cognition to Expression with Linear
Representations

Yuzi Yan  Jialian Li  Yipin Zhang  Dong Yan

###### Abstract

This paper presents an in-depth examination of the evolution and interplay of
cognitive and expressive capabilities in large language models (LLMs), with a
specific focus on Baichuan-7B and Baichuan-33B, an advanced bilingual (Chinese
and English) LLM series. We define and explore the model’s cognitive and
expressive capabilities through linear representations across three critical
phases: Pretraining, Supervised Fine-Tuning (SFT), and Reinforcement Learning
from Human Feedback (RLHF). Cognitive capability is defined as the quantity
and quality of information conveyed by the neuron output vectors within the
network, similar to the neural signal processing in human cognition.
Expressive capability is defined as the model’s capability to produce word-
level output. Our findings unveil a sequential development pattern, where
cognitive abilities are largely established during Pretraining, whereas
expressive abilities predominantly advance during SFT and RLHF. Statistical
analyses confirm a significant correlation between the two capabilities,
suggesting that cognitive capacity may limit expressive potential. The paper
also explores the theoretical underpinnings of these divergent developmental
trajectories and their connection to the LLMs’ architectural design. Moreover,
we evaluate various optimization-independent strategies, such as few-shot
learning and repeated sampling, which bridge the gap between cognitive and
expressive capabilities. This research reveals the potential connection
between the hidden space and the output space, contributing valuable insights
into the interpretability and controllability of their training processes.

Machine Learning, ICML

##  1 Introduction

Large Language Models (LLMs) are profoundly transforming the way we work and
live. To train these models, computational power worth billions is used daily
in the three-phase paradigm of Pretraining, Supervised Fine-Tuning (SFT), and
Reinforcement Learning from Human Feedback (RLHF). However, the specific roles
of these three stages are only broadly understood: Pretraining primarily
encodes knowledge, SFT aligns question-answer formats, and RLHF refines
outputs via human feedback (Achiam et al., 2023). Evidently, this
understanding is on the level of behavioral patterns and does not aid in
comprehending LLMs from a capability perspective, nor does it guide us on how
to refine the training process to enhance and control the model’s proficiency
in various tasks.

To analyze the three-phase training paradigm from a capability perspective,
researchers have introduced the concept of Alignment Tax to articulate the
discrepancy between the model’s inherent capabilities and its outward
performance (Lightman et al., 2023; Ouyang et al., 2022; Askell et al., 2021).
Beside the training paradigm, prompt engineering also significantly influences
the performance exhibited by the model. These pieces of evidence point towards
a hypothesis that sometimes LLMs internally comprehend and encode the answer
to a question in the internal representations but struggles to output it
effectively.

Prior research in interpretability has culminated in considerable
breakthroughs, aiding in demystifying the internal processes of LLMs. In Zou
et al. (2023), the authors propose representation engineering (RepE) as an
advanced method to improve AI transparency. Meanwhile, in Park et al. (2023),
the study suggests the possibility of a linear space structure within neuron-
level representations. Additionally, probing-based explanation techniques
offer novel insights into the abstract abilities of LLMs, as explored in Zhao
et al. (2023).

In this paper, we define and quantify the _cognitive capability_ and the
_expressive capability_ of a LLM and explore the establishment process of
them. The _cognitive capability_ is defined by the quantity and quality of
information conveyed by the neuron output vectors within the network, similar
to the neural signal processing in human cognition. This definition
corresponds to the way the human brain processes information and makes sense
of the world. The definition of cognitive capability exploits linear
representations within the hidden space, obtained particularly from a selected
intermediate layer. On the other hand, The _expressive capability_ is defined
as the model’s capability to produce word-level output, similar to the human
ability to express thoughts or feelings by language, art, or other means. Our
study includes a comprehensive series of experiments and analyses carried out
during the Pretraining, SFT, and RLHF phases of the Baichuan-7B and
Baichuan-33B. These models are part of an advanced bilingual LLM series
Baichuan2 (Baichuan, 2023). Notably, Baichuan-7B is an open-source model,
whereas Baichuan-33B is a closed-source model. We present the following key
findings: 1) Cognitive and expressive capabilities evolve at different paces.
Specifically, cognitive capability is primarily established during the
Pretraining stage, whereas expressive capability is developed during the SFT
and RLHF stages, with SFT playing a more significant role. 2) A robust
statistical correlation exists between cognitive and expressive capabilities.
The cognitive capability sets the upper boundary for the expressive
capability. 3) Our research illustrates that specific techniques, including
few-shot learning, repeated sampling, and prompt engineering, can
efficaciously bridge the gap between a LLM’s expressive and cognitive
capabilities.

In addition, we delve into the internal mechanisms governing the development
of cognitive and expressive capabilities, along with a theoretical analysis of
the gap between them, in Section 4. We conduct multiple experiments to
underpin our hypotheses. Specifically, the discrepancy between these
capabilities may stem from the differences in linear separability between the
embedding space of neuron output and the token-level semantic space. From the
standpoint of the LLM’s architecture, the diminution of this gap during the
SFT/RLHF stage could be attributed to enhancements in the vocabulary linear
layer at this phase. We anticipate that these discoveries will offer valuable
insights into the training process of LLMs.

Figure 1:  Schematic representation of the asynchronous capabilities
development process in LLMs. Initially, the model lacks the ability to
comprehend questions or generate relevant responses. Through the Pretraining
phase, the LLM primarily acquires cognitive capabilities, though its ability
to articulate responses remains underdeveloped. Subsequent SFT and RLHF
enhance the model’s expressive capability, aligning it closely with the
cognitive skills.

##  2 Related Work

It is very appealing to explain the model’s capability by analyzing the hidden
space’s linear properties of the language model, and it has attracted a lot of
research attention recently. These works significantly inspired this paper. A
comparative analysis is then presented to highlight the relationship and the
distinctions between our work and these prior studies.

Linear subspaces and geometry in language representations. The hypothesis of
linear subspaces was initially observed empirically in the context of word
embeddings by Mikolov et al. (2013b). Similar structures have been observed in
cross-lingual word embeddings (Mikolov et al., 2013a), as well as in sentence
embeddings and the representational spaces of Transformer-based LLMs
(Hernandez et al., 2023). There is a significant body of work studying the
geometry of word-level or sentence-level representations (Arora et al., 2016;
Mimno & Thompson, 2017; Li et al., 2023; Park et al., 2023). These
observations motivate our approach to measuring cognitive capability in LLMs
using linear representations in the hidden space. This paper lends further
credibility to the analysis of language model characteristics within a linear
hidden space by demonstrating a strong correlation between the measured
cognitive capabilities and the model’s upper limit of expressiveness.

Promoting LLM by linear representation. Recent advancements have leveraged
linear representations to augment the capabilities of LLMs. In Liu et al.
(2023), the authors introduce an innovative RLHF approach, which refines
linear representations to align closely with high-level human preferences,
thereby enabling more precise control over model behavior and enhancing its
performance. The findings of this study suggest two critical insights:
firstly, certain linear representations within the hidden layers may provide
information that is at least as significant as that conveyed by token-level
outputs. Secondly, these linear representations can serve as potent indicators
for directing the model’s refinement. In our work, we substantiate the first
hypothesis by delineating the gap between cognitive capabilities, as defined
by linear representations, and expressive capabilities, as indicated by direct
token-level outputs. Furthermore, we lend partial support to the second
hypothesis by identifying strategies to bridge this gap without parameter
optimization.

Measurement and mechanistic interpretability. A considerable volume of
research has been dedicated to the exploration of linear representations for
both interpreting (probing) (Alain & Bengio, 2016; Kim et al., 2018) and
manipulating (steering) (Turner et al., 2023) the behavior of models. Notably,
the work presented in Zou et al. (2023) posits that engaging with concept-
level representations within LLMs can substantially enhance the model’s
proficiency in specific concepts such as truthfulness and honesty. This
discovery underscores the potential of linear representations significantly
augment or modulate the expressive prowess of models from a top-down
perspective. Complementing this viewpoint, our study explore the interplay
between linear representations and the structural design of LLMs, offering a
bottom-up analysis.

##  3 Main Results

This section defines cognitive and expressive capabilities in LLMs and
outlines their quantification methods. We then present experimental results
from public datasets, highlighting the asynchronous development of these
capabilities during training. Finally, we demonstrate their statistical
correlation, underscoring their interdependence in LLM performance.

###  3.1 Definitions and Quantification of Cognitive and Expressive
Capabilities

We conceptualize the inference process in a LLM as follows: Given an input
prompt $x$ comprising $n$ tokens, where $x\in{\mathcal{T}}^{n}$ and
${\mathcal{T}}$ denotes the token-level space, the LLM initially maps $x$ to a
high-dimensional vector $c\in\mathcal{R}^{m}$ through a mapping function
$f(\cdot)$. The architecture of the LLM, such as in prevalent decoder-only
models like Llama 2 (Touvron et al., 2023) or GPT (Achiam et al., 2023),
determines the specifics of $f(\cdot)$, including the hidden size of the
Transformer block (Vaswani et al., 2017), network weights, the layer from
which $c$ is extracted, and other hyperparameters. Subsequently, the function
$g(\cdot)$ maps $f(x)$ to the next token output $y\in\mathcal{T}$, influenced
by the model’s remaining architecture, notably including a critical vocabulary
linear layer discussed further in Section 4. The inference process is
succinctly represented as:

$\displaystyle
x\in{\mathcal{T}}^{n}\xrightarrow{f(\cdot)}c\in\mathcal{R}^{m}\xrightarrow{g(\cdot)}y\in{\mathcal{T}}$

Our hypothesis, supported by empirical evidence presented in later sections,
posits that the intermediate vector $c$ harbors more insightful information
compared to the direct token output $y$. For instance, in binary
classification tasks such as ”True or False” questions, leveraging
unsupervised algorithms like PCA on $c$ has shown to surpass strategies that
directly analyze $y$. This suggests that LLMs may grasp the underlying problem
and possess the correct solution, yet lack the capability to articulate it
accurately.

We analogize $c$, emerging from the neuron outputs within the network and
resembling neural signals in human cognition, as the model’s cognitive
capability. Conversely, the ability to produce token-level outputs is defined
as the LLM’s expressive capability.

To assess these capabilities, we devise experiments with single-choice
questions, comprising one question ($<\text{QUESTION}>$) and multiple choices
($<\text{CHOICE}_{i}>$), where only one is correct. The evaluation methods are
detailed below:

Quantification of cognitive capability.

For each single-choice question within a dataset, we pair each
$(<\text{QUESTION}>,<\text{CHOICE}_{i}>)$ with Template A (see Appendix A.1)
to form an input $x$. The LLM’s cognitive capability on this dataset is
evaluated using Algorithm 1, drawing on the unsupervised Representation
Engineering (RepE) approach, which uses Principal Component Analysis (PCA)
(Zou et al., 2023).

Quantification of expressive capability: In the test set $D_{\text{test}}$, we
wrap each $(<\text{QUESTION}>,\\{<\text{CHOICE}_{i}\\}_{i=1}^{4})$ with
Template B (see Appendix A.1). For a given model, the wrapped prompt serve as
inputs to calculate the accuracy of direct token responses, defining the
expressive capability’s quantification. It is crucial to note that our method
diverges from frameworks like Gao et al. (2023), which employ greedy search to
assess the likelihood of each option in its entirety, as opposed to our focus
on the model’s direct token outputs.

Algorithm 1 Cognitive capability quantification

0: Model $M$, training set $D_{\text{train}}$, test set $D_{\text{test}}$.

1: Format the input prompts $\\{x^{\text{train}}\\}$ from $D_{\text{train}}$
and $\\{x^{\text{test}}\\}$ from $D_{\text{test}}$ using Template A.

2: PCA Direction Extraction:

3: for each Transformer block $i$ do

4: Use $\\{x^{\text{train}}\\}$ as input and extract embeddings
$\\{c^{\text{train}}_{i}\\}$ for the last token.

5: Calculate a PCA direction $v_{i}\in\mathcal{R}^{m}$ using
$\\{c^{\text{train}}_{i}\\}$.

6: Determine the sign function:
$S_{i}\in\\{\mathop{\mathrm{argmin}},\mathop{\mathrm{argmax}}\\}$ based on the
principle component extracted and the correct answers.

7: end for

8: Evaluation:

9: for each Transformer block $i$ do

10: Use $\\{x^{\text{test}}\\}$ as input and extract embeddings
$\\{c^{\text{test}}_{i}\\}$ for the last token.

11: Project $\\{c^{\text{test}}_{i}\\}$ onto $v_{i}$ and choose one answer by
$S_{i}$: $S_{i}(v_{i}^{\mathrm{T}}\cdot c^{\text{test}}_{i})$.

12: Compare the chosen answers and the correct answers, calculate the
accuracy: $\text{Acc}_{i}$

13: end for

14: Compute maximum evaluation accuracy across all Transformer blocks as the
quantification of the cognitive capability: $\max_{i}(\\{\text{Acc}_{i}\\})$.

###  3.2 Datasets and Experimental Setup

We carry out our quantification experiments using four standard benchmark
datasets: OpenbookQA (Mihaylov et al., 2018), CommonSenseQA (Talmor et al.,
2018), RACE (Lai et al., 2017) and ARC (Clark et al., 2018). OpenBookQA is
designed to test a language model’s ability for text understanding and
reasoning. It focuses on the application of common sense and general knowledge
in answering questions. CommonSenseQA is a benchmark for testing the common
sense of AI systems. It includes questions that require an understanding of
everyday concepts and relationships between objects and ideas. The RACE
dataset is a large-scale reading comprehension dataset collected from English
exams for middle and high school Chinese students. It consists of passages and
corresponding single-choice questions. The AI2 Reasoning Challenge (ARC) aims
to evaluate a system’s reasoning ability and understanding of scientific
texts, offering two levels of difficulty, referred to as ARC-challenge and
ARC-easy. All the datasets above can be formatted as single-choice questions
with 4 options.

Our experiments use checkpoints from the Pretraining, SFT, and RLHF stages of
the in-house developed Baichuan-7B and Baichuan-33B, both decoder-only,
bilingual LLMs. The 7B model is openly available (Baichuan, 2023), and the 33B
model extends the 7B architecture with increased parameters. For RLHF, we
implement the Proximal Policy Optimization (PPO) strategy, as elaborated in
Achiam et al. (2023).

###  3.3 Pretraining: Building Cognitive Capability

Figure 2:  Progression of cognitive capability during the Pretraining stage in
Baichuan-33B, as quantified by linear representations. The graph illustrates a
stabilization in cognitive performance when the volume of training data
reaches approximately 2.4T.

This section details the development of cognitive capability during the
Pretraining phase for Baichuan-33B and Baichuan-7B. Both models were trained
from the ground up, with training data incrementally increased up to 3.2T. The
progression of cognitive capability in Baichuan-33B, assessed by Algorithm 1,
is depicted in Figure 2. For Baichuan-7B, corresponding findings are presented
in Figure 11 within Appendix D.3.

In both Baichuan-33B and Baichuan-7B, we note a swift initial improvement in
cognitive capability that tapers off with increased training data. Initially,
both models exhibit decision-making akin to random guessing, with accuracy
around 0.25, indicative of their nascent state. The cognitive capability’s
growth stabilizes around 2.4T of data for Baichuan-33B and 1.5T for
Baichuan-7B, suggesting a nearing to the models’ cognitive limits in tasks
like reasoning, commonsense understanding, and information retrieval across
various datasets. Notably, the model with fewer parameters, Baichuan-7B,
demonstrates lower accuracy, reflecting its lower cognitive capacity compared
to Baichuan-33B.

Additionally, we applied the general lm-evaluation-harness framework (Gao et
al., 2023) to both models, which employs greedy search to evaluate each
option’s probability for answer selection. The outcomes are depicted in Figure
13 for Baichuan-33B and in Figure 14 for Baichuan-7B in Appendix D.4. Despite
the similarity in pattern between the two curves, a noticeable discrepancy
exists. Both sets of results distinctly illustrate the progression of the
models’ intrinsic cognitive capabilities, yet highlight a shortfall in
expressive capability, which we will explore further in the following section.

###  3.4 SFT and RLHF: Aligning Expressive and Cognitive Capabilities

This section explores how the SFT and RLHF stages align cognitive and
expressive capabilities. Post-Pretraining, despite high cognitive accuracy,
the models often fail to deliver correct token-level answers. We assess
expressive capability as outlined in Section 3.1, finding both zero-shot and
few-shot performances significantly lagging behind cognitive accuracy measured
by Algorithm 1 (see Figures 3 and 8). Case studies in Appendix E reveal
instances of incoherent responses, suggesting that while advanced cognition
may be achieved late in Pretraining, guiding accurate expressions through
zero-shot or few-shot approaches remains challenging. SFT and RLHF effectively
reduce this cognitive-expressive discrepancy.

Leveraging the pretrained Baichuan-33B model with 3.0T training data, we
developed variants through SFT and RLHF. The SFT phase involved training over
4 epochs, each with 1M tokens. For RLHF, we initially trained a reward model
on preference-ranked data annotated by our in-house annotation team, using the
pretrained Baichuan-33B as a base. This was followed by implementing the
standard PPO pipeline, commencing from the SFT model at epoch 4.

We assessed cognitive and expressive capabilities across various SFT epochs
and the concluding PPO model, with findings illustrated in Figure 3. Key
observations include: 1) Cognitive capability remains relatively stable
throughout SFT and RLHF phases. 2) Expressive capability significantly
improves during SFT, eventually nearing but not surpassing cognitive
capability. Observation 1 corroborates Section 3.3’s assertion that cognitive
development primarily transpires during Pretraining. Observation 2 highlights
the pivotal role of SFT (and to a lesser extent, RLHF) in enhancing expressive
capability, suggesting cognitive capability as a potential ceiling for
expressiveness. Approaches to optimize expressive capability and narrow this
gap are explored in Section 5.

######  Remark 3.1.

The quantification of cognitive capability, conducted in the hidden space via
Principal Component Analysis (PCA), presents a non-trivial approach for
assessing the internal capabilities of LLMs. This opens avenues for employing
additional linear analysis techniques for model analysis or control, as
evidenced in recent studies such as Zou et al. (2023); Liu et al. (2023).
These methods will constitute the core of our forthcoming research endeavors.

Figure 3:  Illustration of the diminishing gap between expressive and
cognitive capabilities in SFT and RLHF. Each SFT epoch processes 1 million
tokens. The dotted line signifies the cognitive capability, established during
the Pretraining phase and acting as the upper boundary for expressive
capability. The solid line represents the expressive capability. The diagram
highlights the gradual reduction of the disparity between these capabilities
as the model undergoes further refinement through SFT and RLHF.

###  3.5 Statistical Correlation between Cognitive Capability and Expressive
Capability

In this subsection, we evaluate the consistency between two quantification
methods and investigate the correlation between cognitive and expressive
capabilities using hypothesis testing. Detailed results are presented in Table
7 within Appendix D.1. An illustrative example for the RACE dataset is
provided in Table 1.

During quantification of the capabilities, the LLM answers single-choice
questions. Consistency between the two quantification methods for a question
is established when both yield the same outcome—correct or incorrect. We
assess a model’s consistency on a dataset by computing the ratio of questions
where the methods concur to the total question count. These consistency
metrics are documented in Column 3 of Table 1. An upward trend in consistency
is noted with progressing SFT epochs, indicative of the expressive capability
increasingly mirroring the stable cognitive capability at a granular level.

To assess the correlation between cognitive and expressive capabilities, we
employ hypothesis testing. We denote the accuracy from cognitive capability
quantification as cognitive accuracy $a^{\text{cog}}$, and that from
expressive capability quantification as expressive accuracy $a^{\text{exp}}$.
Our null hypothesis assumes $a^{\text{exp}}$ and $a^{\text{cog}}$ are
independent. Under this premise, the consistency count across methods for a
set of questions is expected to adhere to a binomial distribution
$\mathcal{B}(s,1-(1-a^{\text{exp}})\times(1-a^{\text{cog}}))$, where $s$ is
the total question count. We then calculate the likelihood of observing the
actual consistency level, finding it to consistently be less than $0.01\%$
across diverse datasets. Such low probabilities strongly suggest the null
hypothesis to be improbable, thereby indicating a significant correlation
between $a^{\text{exp}}$ and $a^{\text{cog}}$.

Table 1: The statistical correlation analysis of the cognitive capability and
the expressive capability in RACE. H-Test probability stands for hypothesis
testing probability with the hypothesis that these two capabilities are
irrelevant. We refer the expressive accuracy and the cognitive accuracy to the
accuracy that are obtained by the quantification of the expressive capability
and the cognitive capability as mentioned in Section 3.
Data set | Model | Consistency |  Expressive Accuracy |  Cognitive Accuracy |  Binomial Distribution |  H-Test probability
---|---|---|---|---|---|---
RACE | SFT Epoch2 | 72.26% |  68.72% |  79.42% |  $\mathcal{B}(3451,0.6101)$ |  $<$0.01%
| SFT Epoch3 | 73.24% |  73.17% |  78.94% |  $\mathcal{B}(3451,0.6341)$ |  $<$0.01%
| SFT Epoch4 | 74.24% |  78.94% |  74.00% |  $\mathcal{B}(3451,0.6563)$ |  $<$0.01%
| PPO | 73.92% |  77.01% |  80.10% |  $\mathcal{B}(3451,0.6410)$ |  $<$0.01%

###  3.6 Assessment of Cognitive Convergence across Training Phases

In this subsection, we explore the convergence of the cognitive capability by
assessing consistency across consecutive checkpoints. We quantify cognitive
capabilities for adjacent model checkpoints and compute the inconsistency
ratio—the proportion of questions where the two models diverge—to the total
question count. A lower inconsistency ratio indicates smaller cognitive
discrepancies between the models. The findings are depicted in Figure 4.

The results indicate a steady decrease in inconsistency among consecutive
models during Pretraining, signaling enhanced stability in cognitive
responses. Transitioning to SFT with a novel corpus leads to an initial
increase in inconsistency, which subsequently diminishes. Notably,
inconsistency experiences a minor uptick following RLHF, hinting that the
introduction of SFT and RLHF momentarily injects cognitive uncertainty,
despite a prevailing trend of improved consistency throughout Pretraining.

Figure 4: Convergence of cognitive capabilities by assessing consistency
across consecutive checkpoints. The y-axis quantifies the discrepancy in
judgments between each model and its predecessor. The red solid line is the
average result.

##  4 Theoretical Analysis

###  4.1 Explanation of the Capability Gap

The following theorem articulates our rationale for the observed gap between
cognitive and expressive capabilities.

######  Theorem 4.1.

The gap between cognitive and expressive capabilities stems from the superior
mapping efficiency of the function $f(\cdot)$ compared to $g(\cdot)$, along
with the greater linear separability afforded by the hidden space
$\mathcal{R}^{m}$ over the token-level space $\mathcal{T}$.

To corroborate this theorem, we use a simple linear classifier within both
$\mathcal{R}^{m}$ and $\mathcal{T}$, utilizing the HalluQA dataset (Cheng et
al., 2023) designed for assessing hallucination phenomena in Chinese LLMs via
counterfactual question-answer pairs. This dataset facilitates the creation of
clear positive and negative examples, essential for our comparative analysis:

Linear SVM on $\mathcal{R}^{m}$: In the trainset, each (QUESTION, {ANSWER})
pair is processed with Template A (see Appendix A.1) to extract the embedding
$c$ from a chosen layer. A linear-kernel SVM is then trained on these
embeddings $\\{c\\}$, with its classification accuracy assessed on the
testset.

Direct token generation on $\mathcal{T}$: The model’s accuracy in generating
responses is evaluated on the test set, comparing against the provided correct
answers as the reference.

The trainset is integrated into the SFT training data. The outcomes, detailed
in Table 2, reveal a pronounced disparity between the accuracies of SVM
classifications and direct token generation. Remarkably, the accuracy of SVM
classifications stays relatively constant, whereas direct token generation
accuracy progressively enhances with additional SFT epochs and through the
implementation of RLHF.

This discrepancy underscores the distinct classification landscapes offered by
$\mathcal{R}^{m}$ and ${\mathcal{T}}$. The linear SVM delineates a hyperplane
in $\mathcal{R}^{m}$ that effectively segregates the data into two categories,
in contrast to the hyperplane in ${\mathcal{T}}$ inferred by direct token
generation. The comparative analysis reveals that $\mathcal{R}^{m}$
facilitates lower intra-class variance and higher inter-class variance,
indicating more pronounced class separability than ${\mathcal{T}}$.

Figure 5: UMAP visualization of HalluQA classifier in Baichuan-33B. The red
line represents the delineation by SVM on neuron output while the green line
represents that of token-level output. The blue dots and orange dots
represents positive samples and negative samples in the datasets respectively.
SFT and RLHF demonstrates the potential to align the expressive capability to
the cognition capability in the fine-tuning stages.

Figure 5 conceptualizes this distinction through UMAP (McInnes et al., 2018)
reduction ${\mathcal{T}}$ from the final transformer block to two dimensions.
The delineation by the SVM and direct token generation classifiers,
represented by red and green lines respectively, visually captures the gap
between cognitive and expressive capabilities. SFT and RLHF demonstrates the
potential to bridge this gap.

Table 2: The performance gap between the Linear SVM and the direct token generation on HalluQA in Baichuan-33B.  Model | Pretrain | SFT-2 | SFT-3 | SFT-4 | PPO
---|---|---|---|---|---
L-SVM | 0.868 | 0.875 | 0.868 | 0.868 | 0.868
Direct | 0.0607 | 0.493 | 0.509 | 0.513 | 0.563

###  4.2 Establishment of Cognitive Capability

To elucidate the evolution of cognitive capabilities and pinpoint the layer
that most significantly represents cognitive capability within the LLM, we
apply Algorithm 1 to examine various layers in both Baichuan-7B and
Baichuan-33B models. The findings for Baichuan-33B are depicted in Figure 6,
while results for Baichuan-7B are presented in Figure 12 within Appendix D.3.

The peak accuracy of the curve serves as an indicator of cognitive capability.
During the Pretraining and SFT phases, there is a notable increase in
cognitive capability as training data volume expands. The effect of RLHF on
cognitive capability is ambiguous, seemingly influenced by the training data’s
distribution for the reward model. Performance assessments on datasets like
ARC-easy, ARC-challenge, and CommonSenseQA show the PPO-enhanced model
outperforming its SFT-only version. However, in OpenBookQA evaluations, the
introduction of PPO slightly detracts from performance, hinting at a potential
data-specific bias in the reward model.

The curve’s trajectory offers valuable insights into the model’s capability
development. In the initial phase of Pretraining, with training data under 1T,
the curve’s peak typically aligns with the model’s mid-section. For instance,
in the Baichuan-33B model, using 700B tokens for Pretraining, the peak value,
as assessed through ARC-Challenge, is identified in the 26th Transformer
block, depicted by the orange curve in the third figure of Figure 6. As the
training data expands to 1.5T and 3T, the accuracy within the model’s final 30
layers stabilizes at a high level. This pattern, observed across various
tests, suggests a characteristic feature of the Pretraining phase nearing
convergence.

######  Remark 4.2.

The establishment process of layer-wise cognitive capability is divided into
two periods, the bell curve (in the early stages of Pretraining, with less
training data) and the plateau curve (in the late stages of Pretraining, with
more training data). In the plateau period, the cognitive capability reaches
its peak at a certain intermediate layer. Based on this, we believe that
cognitive capability may be established in the first few layers of the model,
while the pleatue layers continuously strengthen this cognitive capability,
and are mapped to expressive capability in the final linear layer. This
phenomenon may result largely from _Residual Connection_ and _pre-Layer
Normalization_ in the model architecture. Besides, the appearance of the
cognitive capability plateau curve may represent some redundancy of the model.
We leave more discussion in Section C in the supplementary materail.

The SFT and RLHF phase mainly influences the performance of the model’s final
layers. The results in ARC-challenge and OpenBookQA (see Figure 10)
illustrates a notable performance dip in the ultimate layer of the pretrain-3T
model compared to its preceding layers. Nonetheless, SFT and RLHF helps in
rectifying this performance gap. The efficacy of the model’s final layers,
especially the last one, is significantly associated with its expressive
capability. This relationship will be explored in the following subsection.

Figure 6: Layer-wise performance of linear representations in Baichuan-33,
shedding light on the intricate architecture underlying cognitive capability
formation.

###  4.3 Establishment of Expressive Capability

This subsection explores the significance of the vocabulary linear layer in
shaping the expressive capabilities of a model. Positioned as the final MLP
(Multi-Layer Perceptron) layer with trainable parameters within a decoder-only
LLM, the vocabulary linear layer maps the last transformer block’s output to
the vocabulary space ${\mathcal{T}}$. During the model’s greedy output
generation, this layer effectively assesses the similarity between its input
and each of its row vectors (each corresponding to a token in the vocabulary),
ultimately selecting the token that exhibits the highest similarity for its
prediction. This process not only underpins the model’s ability to generate
coherent and contextually relevant text but also forges a structural link
between cognitive and expressive capabilities. Earlier sections have
illustrated that as the model approaches the convergence point during
Pretraining, the output performance of the last few transformer blocks
stabilizes, reflecting the model’s cognitive capabilities. Consequently, the
effective training of the vocabulary linear layer is paramount, as it directly
influences the model’s ability to articulate its ’thoughts’ and knowledge
accurately.

To demonstrate the dynamics of the vocabulary linear layer, we designed an
experiment using a set of prompts $\\{x_{i}\\}_{i=1}^{m}$ and performed
inference across a series of models. We define the output from the final
transformer block for each prompt $x_{i}$ in model $M$ as $c_{-1}(x_{i},M)$.
The weights of model $M$’s vocabulary linear layer are represented by
$\text{Vocal}(M)$. We used a series of Baichuan-33B models
$\\{M_{j}|j=1,2,\ldots,n\\}$ across Pretraining, SFT, and RLHF stages. An
average output
$c_{-1}(x_{i})=\frac{1}{n}\sumop\displaylimits_{j=1}^{n}c_{-1}(x_{i},M_{j})$
is computed. Subsequently, $\text{Vocal}(M_{i})$ was adjusted to examine the
resulting output distribution through:

$\displaystyle
d_{M_{j}}(x_{i})=\text{softmax}(c_{-1}(x_{i})\cdot\text{Vocal}(M_{j})),$

where $c_{-1}(x_{i})$ acts as a consistent reference for input $x_{i}$ across
all models, enabling the assessment of changes in the vocabulary linear layer
via the variation in average KL divergence:
$\text{KL}(M_{j}||M_{k})=\frac{1}{m}\sumop\displaylimits_{i=1}^{m}\text{KL}(d_{M_{j}}(x_{i})||d_{M_{k}}(x_{i}))$.
The results are depicted in Figure 7.

Using HalluQA for prompt input, our findings are depicted in Figure 7. Through
analysis of various models spanning Pretraining, SFT and RLHF, and by tracking
KL divergence changes per 1M training data increments, we found that
adjustments to the vocabulary linear layer weights during SFT and RLHF result
in more significant KL divergence fluctuations than those observed during
later Pretraining stages. This implies that the SFT phase, in particular,
notably bolsters the model’s expressive capabilities, thus narrowing the gap
with its cognitive abilities. This observation is consistent with the apparent
shift in LLM response behaviors following SFT and RLHF, suggesting a
fundamental shift in the model’s token response generation.

Figure 7: Dynamics of KL divergence per 1M training data, utilizing different
vocabulary linear layers, evaluated by HalluQA.

##  5 Methods for Bridging the Gap

Figure 8:  Performance of the optimization-free methods aimed at bridging the
gap between expressive and cognitive capabilities.

In this section, we explore a selection of optimization-free approaches to
evaluate their efficacy in narrowing the gap between cognitive and expressive
capabilities.

###  5.1 Few-shot Learning

In few-shot learning, we leverage a templated approach as delineated in
Appendix A.2. This involves prefacing the input with a series of question-and-
answer examples, thereby providing the model with a context that enhances its
ability to understand and respond to new, similar queries. The underlying
premise of this approach is that providing even a small set of examples can
substantially enhance the model’s capacity to draw upon its inherent
knowledge, thereby amplifying its expressive capabilities.

###  5.2 Repeated Sampling

Building upon the concept of Rejection Sampling as outlined in the work of
(Touvron et al., 2023), we explore the potential of repeated sampling as a
means to extract potentially accurate responses. By generating multiple
responses to a single question and considering the response set successful if
at least one meets the reference answer, We can explore the upper limit of the
model’s expressive capability. Specifically, we sample 8 responses per prompt
independently, adopting the criterion that the model’s output is deemed
accurate if any of these responses is correct.

###  5.3 Magical Additional Prompt

In light of findings from prior research (Kojima et al., 2022; Yang et al.,
2023), it has been suggested that the inclusion of specific, strategically
crafted prompts—either preceding or following the primary query—can
significantly augment an LLM’s performance. To this end, we introduce what we
term _magical additional prompts_ to our input:

_Let’s think step by step and take a deep breathe, the task is very important
for human society!_

###  5.4 Results Analysis

The outcomes of our experiments, as depicted in Figure 8, indicate that both
few-shot learning and repeated sampling methodologies exhibit considerable
promise in amplifying the expressive capabilities of LLMs. However, the impact
of the ’magical additional prompt’ was found to be less pronounced. Notably,
with repeated sampling employed up to eight times, the LLM’s expressive
capability is observed to match its cognitive capability. These findings
suggest that through strategic prompt engineering, the model can achieve
performance levels that surpass its expressive capabilities, hinting that the
cognitive capabilities of the model implicitly set the upper bound for its
expressive performance.

##  6 Conclusion

In this work, we delved into the distinctions between cognitive and expressive
capabilities in LLMs, specifically focusing on the bilingual Baichuan-7B and
Baichuan-33B series. Cognitive capability is quantified via linear
representations in the hidden space, while expressive capability is evaluated
through direct token outputs. Our extensive experimentation, encompassing
reasoning, common-sense comprehension, and logical inference, reveals that
cognitive capabilities are primarily developed during Pretraining, with
expressive capabilities further refined in subsequent SFT and RLHF phases.
Statistical analysis confirms a strong correlation between these capabilities.
Theoretically, we attribute the capability gap to the superior linear
separability of the hidden space $\mathcal{R}^{m}$ over the token-level space
${\mathcal{T}}$. Examination of various optimization-free strategies for
mitigating this gap shows that methods like repeated sampling and few-shot
learning significantly improve expressive capabilities, aligning them more
closely with cognitive capacities. Having established a correlation between
linear spaces and the cognitive capabilities of language models, the
extraction and the transformation of features within these linear spaces
becomes a compelling avenue for future research.

## Acknowledgements

The authors would like to thank Yiyu Li, Luyao Ma, Lin Bo and Yulong Wang from
Baichuan AI for helpful discussions and feedback.

## Impact Statement

This paper presents work whose goal is to advance the field of Machine
Learning. There are many potential societal consequences of our work, none
which we feel must be specifically highlighted here.

## References

  * Achiam et al. (2023) Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Alain & Bengio (2016) Alain, G. and Bengio, Y.  Understanding intermediate layers using linear classifier probes.  _arXiv preprint arXiv:1610.01644_ , 2016.
  * Arora et al. (2016) Arora, S., Li, Y., Liang, Y., Ma, T., and Risteski, A.  A latent variable model approach to pmi-based word embeddings.  _Transactions of the Association for Computational Linguistics_ , 4:385–399, 2016.
  * Askell et al. (2021) Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., et al.  A general language assistant as a laboratory for alignment.  _arXiv preprint arXiv:2112.00861_ , 2021.
  * Baichuan (2023) Baichuan.  Baichuan 2: Open large-scale language models.  _arXiv preprint arXiv:2309.10305_ , 2023.  URL https://arxiv.org/abs/2309.10305.
  * Cheng et al. (2023) Cheng, Q., Sun, T., Zhang, W., Wang, S., Liu, X., Zhang, M., He, J., Huang, M., Yin, Z., Chen, K., et al.  Evaluating hallucinations in chinese large language models.  _arXiv preprint arXiv:2310.03368_ , 2023.
  * Clark et al. (2018) Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _arXiv preprint arXiv:1803.05457_ , 2018.
  * Gao et al. (2023) Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac’h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A.  A framework for few-shot language model evaluation, 12 2023.  URL https://zenodo.org/records/10256836.
  * Goddard et al. (2024) Goddard, C., Siriwardhana, S., Ehghaghi, M., Meyers, L., Karpukhin, V., Benedict, B., McQuade, M., and Solawetz, J.  Arcee’s mergekit: A toolkit for merging large language models.  _arXiv preprint arXiv:2403.13257_ , 2024.
  * Hernandez et al. (2023) Hernandez, E., Sharma, A. S., Haklay, T., Meng, K., Wattenberg, M., Andreas, J., Belinkov, Y., and Bau, D.  Linearity of relation decoding in transformer language models.  _arXiv preprint arXiv:2308.09124_ , 2023.
  * Kim et al. (2018) Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., et al.  Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav).  In _International conference on machine learning_ , pp. 2668–2677. PMLR, 2018.
  * Kojima et al. (2022) Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y.  Large language models are zero-shot reasoners.  _Advances in neural information processing systems_ , 35:22199–22213, 2022.
  * Lai et al. (2017) Lai, G., Xie, Q., Liu, H., Yang, Y., and Hovy, E.  Race: Large-scale reading comprehension dataset from examinations.  _arXiv preprint arXiv:1704.04683_ , 2017.
  * Li et al. (2023) Li, K., Hopkins, A. K., Bau, D., Viégas, F., Pfister, H., and Wattenberg, M.  Emergent world representations: Exploring a sequence model trained on a synthetic task.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=DeG07_TcZvT.
  * Lightman et al. (2023) Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K.  Let’s verify step by step.  _arXiv preprint arXiv:2305.20050_ , 2023.
  * Liu et al. (2023) Liu, W., Wang, X., Wu, M., Li, T., Lv, C., Ling, Z., Zhu, J., Zhang, C., Zheng, X., and Huang, X.  Aligning large language models with human preferences through representation engineering.  _arXiv preprint arXiv:2312.15997_ , 2023.
  * McInnes et al. (2018) McInnes, L., Healy, J., and Melville, J.  Umap: Uniform manifold approximation and projection for dimension reduction.  _arXiv preprint arXiv:1802.03426_ , 2018.
  * Mihaylov et al. (2018) Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  _arXiv preprint arXiv:1809.02789_ , 2018.
  * Mikolov et al. (2013a) Mikolov, T., Le, Q. V., and Sutskever, I.  Exploiting similarities among languages for machine translation.  _arXiv preprint arXiv:1309.4168_ , 2013a.
  * Mikolov et al. (2013b) Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J.  Distributed representations of words and phrases and their compositionality.  _Advances in neural information processing systems_ , 26, 2013b.
  * Mimno & Thompson (2017) Mimno, D. and Thompson, L.  The strange geometry of skip-gram with negative sampling.  In _Empirical Methods in Natural Language Processing_ , 2017.
  * Ouyang et al. (2022) Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744, 2022.
  * Park et al. (2023) Park, K., Choe, Y. J., and Veitch, V.  The linear representation hypothesis and the geometry of large language models.  _arXiv preprint arXiv:2311.03658_ , 2023.
  * Talmor et al. (2018) Talmor, A., Herzig, J., Lourie, N., and Berant, J.  Commonsenseqa: A question answering challenge targeting commonsense knowledge.  _arXiv preprint arXiv:1811.00937_ , 2018.
  * Touvron et al. (2023) Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Turner et al. (2023) Turner, A., Thiergart, L., Udell, D., Leech, G., Mini, U., and MacDiarmid, M.  Activation addition: Steering language models without optimization.  _arXiv preprint arXiv:2308.10248_ , 2023.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Yadav et al. (2024) Yadav, P., Tam, D., Choshen, L., Raffel, C. A., and Bansal, M.  Ties-merging: Resolving interference when merging models.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Yang et al. (2023) Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., and Chen, X.  Large language models as optimizers.  _arXiv preprint arXiv:2309.03409_ , 2023.
  * Zhao et al. (2023) Zhao, H., Chen, H., Yang, F., Liu, N., Deng, H., Cai, H., Wang, S., Yin, D., and Du, M.  Explainability for large language models: A survey.  _ACM Transactions on Intelligent Systems and Technology_ , 2023.
  * Zou et al. (2023) Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., et al.  Representation engineering: A top-down approach to ai transparency.  _arXiv preprint arXiv:2310.01405_ , 2023.

##  Appendix A Prompt Template

###  A.1 Wrapper prompt for the quantification of cognitive and expressive
capabilities

To quantify the cognitive capability and the expressive capability, we use the
following templates as the wrapper for each
$(<\text{QUESTION}>,<\text{CHOICE}_{i}>)$ pair:

Template A

_Consider the correctness of the answer to the following question. Question:_
$<\text{QUESTION}>$_, Answer:_ $<\text{CHOICE}>$._Directly answer correct or
wrong:_

Template B

$<\text{QUESTION}>$ _1._ $<\text{CHOICE}_{1}>$_2._ $<\text{CHOICE}_{2}>$ _3._
$<\text{CHOICE}_{3}>$_4._ $<\text{CHOICE}_{4}>$. _Choose only one answer
directly._

###  A.2 Additional example prompt for few-shot learning

For each $(<\text{QUESTION}>,<\text{CHOICE}_{i}>)$ pair, we select 2-shot
exclusive examples shown as follows:

Which statement best explains why photosynthesis is the foundation of most
food webs?

A. Sunlight is the source of energy for nearly all ecosystems.

B. Most ecosystems are found on land instead of in water.

C. Carbon dioxide is more available than other gases.

D. The producers in all ecosystems are plants.

Choose only one answer directly. A. Sunlight is the source of energy for
nearly all ecosystems.

Which piece of safety equipment is used to keep mold spores from entering the
respiratory system?

A. safety goggles

B. breathing mask

C. rubber gloves

D. lead apron

Choose only one answer directly. B. breathing mask.

$<\text{QUESTION}>$

_A._ $<\text{CHOICE}_{1}>$

_B._ $<\text{CHOICE}_{2}>$

_C._ $<\text{CHOICE}_{3}>$

_D._ $<\text{CHOICE}_{4}>$.  Choose only one answer directly.

##  Appendix B Detailed Experimental Settings

Table 3: Size for each datasets. Dataset Name | trainset size | testset size
---|---|---
ARC-easy | 2251 | 2376
ARC-challenge | 1119 | 1172
CSQA | 9741 | 1140
OBQA | 4957 | 500
RACE | 62445 | 3498
HalluQA | 240 | 63
Table 4: Direct token generation hyperparameters Term | parameter
---|---
temperature | 1.2
top p | 0.9
top k | 50
max tokens | 2048
repetition penalty | 1.05

##  Appendix C Further Discussion about the Cognition Establishment Process

As is shown in Figure 6, the establishment process of layer-wise cognitive
capability is divided into two periods, the bell curve (in the early stages of
Pretraining, with less training data) and the plateau curve (in the late
stages of Pretraining, with more training data). In the plateau period, the
cognitive capability reaches its peak at a certain intermediate layer. Based
on this, we believe that cognitive capability may be established in the first
few layers of the model, while the pleatue layers continuously strengthen this
cognitive capability, and are mapped to expressive capability in the final
linear layer. This phenomenon may result largely from _Residual Connection_
and _pre-Layer Normalization_ in the model architecture. Here is our deduction
process.

######  Lemma C.1.

Suppose that $x$ is with components that are independent and identically
distributed with a zero mean. The magnitude of the $x$ is directly
proportional to its standard deviation:

$\|x\|=\sqrt{d}\cdot std(x)$

.

######  Lemma C.2.

The variance of the sum of independent variables equals the sum of the
variances.

$Var(x+y)=Var(x)+Var(y)$

######  Lemma C.3.

The relationship between the magnitude of the output $x_{L}$ and the depth of
layer $L$ is as follow:

$\|x_{L}\|\sim O(\sqrt{L})$

###### Proof.

Assuming that the input $x$ and output $y=f(x)$ of each layer are independent,
then we know from Lemma 2:

$Var(x_{L+1})=Var(x_{L})+Var(f_{L}(LN(x_{L})))$

The variance of each layer will increase. From Lemma 1, it is known that the
magnitude of each layer will increase. Suppose that $f_{L}(LN(x_{i}))$ has a
mean value of 0 and a variance of $k$ for each $i$, we have:

$\|x_{L}\|=\sqrt{kLd}$

The result that the magnitude of the output from each layer increases is
experimentally verified on most LLMs such as Baichuan-7B, Baichuan-33B and
LLaMa-7B. ∎

######  Claim C.4.

$LN(x_{L})$ and $LN(x_{L+1})$ will become similar as the number of layers $L$
increases.

Although the absolute value of $f_{L}(LN(x_{L}))$ of each layer $L$ may not be
small, the relative size compared to $x_{L}$ will be small. Specially we have:

$LN(x_{L})^{T}LN(x_{L+1})=\frac{x_{L}^{T}\cdot
x_{L+1}}{\|x_{L}\|\|x_{L+1}\|}=\frac{x_{L}^{T}(x_{L}+f_{L}(LN(x_{L})))}{\|x_{L}\|\|x_{L+1}\|}=\frac{\|x_{L}\|}{\|x_{L+1}\|}+\frac{x_{L}^{T}f_{L}(LN(x_{L}))}{\|x_{L}\|\|x_{L+1}\|}$

$=\sqrt{\frac{L}{L+1}}+\frac{x_{L}^{T}f_{L}(LN(x_{L}))}{\|x_{L}\|\|x_{L+1}\|}\geq\sqrt{\frac{L}{L+1}}-\frac{\|x_{L}\|\|f_{L}(LN(x_{L}))\|}{\|x_{L}\|\|x_{L+1}\|}=\sqrt{\frac{L}{L+1}}-\sqrt{\frac{kd}{k(L+1)d}}$

$=\sqrt{\frac{L}{L+1}}-\sqrt{\frac{1}{L+1}}$

This shows that as the number of layers $L$ increases, the angle between
$LN(x_{L})$ and $LN(x_{L+1})$ will tend to 0, and the cosine distance will
tend to 0, which further shows that these two are very close.

######  Claim C.5.

When $L$ is large, the gradients of two adjacent layers are also very close as
well.

$\frac{\partial J}{\partial x_{L}}\approx\frac{\partial J}{\partial x_{L+1}}$

We have:

$x_{L+1}=x_{L}+f(LN(x_{L}))$

then,

$\frac{\partial J}{\partial x_{L}}=\frac{\partial J}{\partial
x_{L+1}}+\frac{\partial J}{\partial x_{L+1}}\frac{\partial
f(LN(x_{L}))}{\partial x_{L}}=\frac{\partial J}{\partial
x_{L+1}}+\frac{\partial J}{\partial x_{L+1}}\frac{\partial
f(LN(x_{L}))}{\partial LN(x_{L})}\frac{\partial LN(x_{L})}{\partial x_{L}}$

We observe that $\frac{\partial f(LN(x_{L}))}{\partial LN(x_{L})}$ will not be
big. For example, if $f$ is a linear, then $\frac{\partial
f(LN(x_{L}))}{\partial LN(x_{L})}=W^{T}$, and $\|W\|$ cannot be big because of
weight decay. Another term $\frac{\partial LN(x_{L})}{\partial
x_{L}}=O(\frac{1}{\|x_{L}\|})$, with a big $L$, this term is going to be small
(according to Claim 3). So that

$\frac{\partial J}{\partial x_{L}}=\frac{\partial J}{\partial
x_{L+1}}(1+O(\frac{1}{\sqrt{L}}))\approx\frac{\partial J}{\partial x_{L+1}}$

Integrating Claim C.4 and C.5, we understand that when $L$ is large, the
inputs to layers $L$ and $L+1$ are very similar, as are the gradients
propagated back through them, and the network architecture remains identical.
Thus, these two layers will be extremely alike. The contextual information
accessible to layer $L$ is essentially accessible to layer $L+1$ as well. The
only difference is a slight reduction in the number of attention cycles, which
may lead to a decrease in the intensity of attention.

In summary, the _Residual Connections_ lead to an increasing norm of the
residual branches, and the addition of _pre-Layer Normalization_ results in
very similar inputs for the Attention and MLP layers between adjacent layers
in deeper networks. On the other hand, the increasing norm of the _Residual
Connections_ branches and the presence of _pre-Layer Normalization_ also cause
that the errors propagated back through adjacent layers in deeper networks are
very similar. These factors contribute to a high degree of similarity between
layers $L+1$ and $L$ and the plateau in the layer-wise cognition capability
measurement.

These theoretical results and the appearance of the cognitive capability
plateau curve in the Pretraining stage (Figure 6) may represent some
redundancy of the model. To see this, we did several additional experiments.

#### Experiment 1.

We delete the 23-th layer of Baichuan-7B (31 layers in total) and connect the
rest of the model directly, the performance on MMLU and CMMLU almost doesn’t
decline, which aligns with the result on the layer-wise cognition capability
curve.

Table 5: Deleting one redundant intermediate layer. Dataset Name | Baichuan-7B | Baichuan-7B (delete 23-th layer)
---|---|---
MMLU | 0.5416 | 0.5398
CMMLU | 0.5707 | 0.5659

#### Experiment 2.

We found that basically speaking, the larger/deeper the LLM, the more
redundancy is likely to be. This suggests that under the current training data
volume, the model size has a lot of room for optimization. To see this, we
conduct the cognition capability measurement on Phi-2, a 2.7B ”small” LLM that
is reported to have the on-par capability with LLaMa2-7B and LLaMa2-13B. The
plateau curve is also observed but the plateau part is much shorter. The
results are shown as follow:

Table 6: The relationship between the plateau propotion and the model size. Model Name | Model Depth | ”Plateau” propotion
---|---|---
Phi-2 | 32 | 0.225
Baichuan2-7B | 32 | 0.53125
LLaMa2-7B | 32 | 0.53125
Baichuan2-13B | 40 | 0.575
LLaMa2-13B | 40 | 0.575
Baichuan2-33B | 60 | 0.55
Baichuan2-53B | 64 | 0.5873

Some recent works regarding model merging (mainly describes the hard
structural merging of two different models without reducing model
capabilities) also suggest the same phenomenon (Yadav et al., 2024; Goddard et
al., 2024). One conjecture is that these layers in plateau may be redundant in
terms of cognitive capability, but play a role in establishing expressive
capability. We observe that in the tests in MMLU and CMMLU (experiment 1), if
one of the layers is deleted, the logits values of the four options A, B, C,
and D will become smaller. It is inferred that the $L+1$ layer only
strengthens the prediction results of the original $L$ layer, and help the
model to better output token-level results.

##  Appendix D Supplementary Experimental Results

###  D.1 Hypothesis testing

We provide the supplementary results of hypothesis testing in ARC-Challenge,
ARC-Easy, CommonSenseQA, OpenbookQA and RACE, in addition to the results shown
in the main part. The results are shown in Table 7.

Table 7: The statistical correlation analysis of the cognitive capability and
the expressive capability. H-Test probability stands for hypothesis testing
probability with the hypothesis that they are irrelevant. We refer the
expressive accuracy and the cognitive accuracy to the accuracy that are
obtained by the quantification of the expressive capability and the cognitive
capability as mentioned in Section 3.
Data set | Model | Consistency |  Expressive Accuracy |  Cognitive Accuracy |  Binomial Distribution |  H-Test probability
---|---|---|---|---|---|---
ARC Challenge | SFT Epoch2 | 75.00% |  65.88% |  78.59% |  $\mathcal{B}(300,0.5908)$ |  0.23%
| SFT Epoch3 | 77.00% |  71.01% |  78.59% |  $\mathcal{B}(300,0.6201)$ |  $<$0.01%
| SFT Epoch4 | 77.92% |  76.51% |  78.92% |  $\mathcal{B}(300,0.6533)$ |  $<$0.01%
| PPO | 80.20% |  78.26% |  82.60% |  $\mathcal{B}(300,0.6842)$ |  0.01%
ARC Easy | SFT Epoch2 | 79.43% |  81.13% |  92.40% |  $\mathcal{B}(570,0.7639)$ |  1.04%
| SFT Epoch3 | 89.28% |  84.01% |  92.04% |  $\mathcal{B}(570,0.7859)$ |  $<$0.01%
| SFT Epoch4 | 89.10% |  88.22% |  92.98% |  $\mathcal{B}(570,0.8285)$ |  0.01%
| PPO | 89.34% |  87.92% |  92.63% |  $\mathcal{B}(570,0.8233)$ |  $<$0.01%
CommonSenseQA | SFT Epoch2 | 60.28% |  54.60% |  76.98% |  $\mathcal{B}(1221,0.5248)$ |  $<$0.01%
| SFT Epoch3 | 64.70% |  55.62% |  76.26% |  $\mathcal{B}(1221,0.5295)$ |  $<$0.01%
| SFT Epoch4 | 66.83% |  65.98% |  77.39% |  $\mathcal{B}(1221,0.5870)$ |  $<$0.01%
| PPO | 68.14% |  66.75% |  77.23% |  $\mathcal{B}(1221,0.5917)$ |  $<$0.01%
OpenBookQA | SFT Epoch2 | 62.14% |  53.51% |  75.20% |  $\mathcal{B}(500,0.5176)$ |  $<$0.01%
| SFT Epoch3 | 70.28% |  60.00% |  74.80% |  $\mathcal{B}(500,0.5496)$ |  $<$0.01%
| SFT Epoch4 | 71.74% |  63.53% |  74.00% |  $\mathcal{B}(500,0.5649)$ |  $<$0.01%
| PPO | 68.33% |  61.52% |  73.20% |  $\mathcal{B}(500,0.5534)$ |  $<$0.01%
RACE | SFT Epoch2 | 72.26% |  68.72% |  79.42% |  $\mathcal{B}(3451,0.6101)$ |  $<$0.01%
| SFT Epoch3 | 73.24% |  73.17% |  78.94% |  $\mathcal{B}(3451,0.6341)$ |  $<$0.01%
| SFT Epoch4 | 74.24% |  78.94% |  74.00% |  $\mathcal{B}(3451,0.6563)$ |  $<$0.01%
| PPO | 73.92% |  77.01% |  80.10% |  $\mathcal{B}(3451,0.6410)$ |  $<$0.01%

###  D.2 Methods for bridging the gap

We provide the supplementary results of methods for bridging the gap between
cognitive and expressive capabilities in ARC-Challenge, ARC-Easy,
CommonSenseQA, OpenbookQA and RACE, in addition to the results shown in the
main part. The results are shown in Table 9.

Figure 9: Methods for bridging the gap between the expressive capability and
the cognitive capability.

Figure 10: The performance of each layer of representation engineering in
Baichuan-33B in different training stage, which reflect the internal
establishment process of the cognitive capability.

###  D.3 The establishment of cognitive capability in Pretraining

We provide the supplementary results of the cognitive capability (measured by
Algorithm 1) establishment process in Pretraining phase in both Baichuan-7B
and Baichuan-33B, in addition to the results shown in the main part.

The progression of cognitive capability in Baichuan-7B, assessed by Algorithm
1, is depicted in Figure 11.

The performance of the linear representations for each layer in Baichuan-33,
which reflect the internal establishment process of the cognitive capability.
The results in Baichuan-7B is shown in Figure 12 and the results in
Baichuan-33B is shown in Figure 10.

Figure 11: The figure shows the increasing process of the cognitive capability
in the Pretraining stage in Baichuan-7B.

Figure 12: The performance of each layer of representation engineering in
Baichuan-7B in different training stage, which reflect the internal
establishment process of the cognitive capability.

###  D.4 The gap between the quantified cognitive capability and the lm-
evaluation-harness performance

We applied the general lm-evaluation-harness framework (Gao et al., 2023) to
both models, which employs greedy search to evaluate each option’s probability
for answer selection. The outcomes are shown in Figure 13 for Baichuan-33B and
in Figure 14 for Baichuan-7B.

Figure 13: The gap between the quantified cognitive capability and the lm-
evaluation-harness performance in Baichuan-33B.

Figure 14: The gap between the quantified cognitive capability and the lm-
evaluation-harness performance in Baichuan-7B.

##  Appendix E Case Study

Figure 15: An example in RACE: The complete establishment process of the cognitive capability and the expressive capability in Pretraining, SFT and RLHF.  Question |  Which technology was developed most recently? A. cellular telephone B. television C. refrigerator D. airplane Choose only one answer directly.
---|---
Reference Answer |  A. cellular telephone
pretrain-2T4 |  e choice, click on the mouse and enter.You can make up to five attempts - it is for free.If you made the first choice (not correct), try other possibilities by clicking here: other options) How to solve this test correctly in order to save time and have successful progress within next tests?To do this you should have access (knowledge of) to all of the given answers. This is not a problem, all of you who want know an answer to a specific choice, just go to the end of my ePortafolio for a full explanation of any choice available. Do not miss the chance to learn, for the sake of saving some time while answering some questionnaires … You cannot progress or do the work unless you put knowledge before time, at least in this case. Take the opportunity to be really well prepared during this whole survey process!
pretrain-3T |  39) By which means did the Incas carry messages and information? A. telephone B. postal service C. human couriers D. radio wave Choose only one answer directly from the text. 40) To live in caves of rocks is called A nomadic life B agrarian life C sedentary life D fishing and seafaring life 41) Which type of food the hunters eat? A. meat B. meat and blood C. blood, skin and meat D. milk Choose only one answer from the text. 42) The hunter/gatherer made tools to chop or break food. What tool did they use when cutting food for cooking or serving it on plates, spoons and knives? A. scissors B. spuds C. knives and forks D. knives Choose only one answer. 43) Today some people live by moving from place to place eating, hunting and collecting roots, berries and other wild plants. What people are they? A. agriculturists B. forest farmers C. hunter/gatherers D. forest dwellers Choose only one word from the text for the blank. SECTION I B 50 MINUTE TIME Read the following passages and choose the correct answers according to the given instructions.
SFT-epoch3 |  They all appeared within about 1900, but I think television was introduced first (roughly 1925) and then refrigerators in the 1930s, followed by cellular phones in the 1980s. Airplanes were around well before this (as early as the beginning of the 20th century). So choosing only among these options the airplane would be the correct option.
SFT-epoch4 |  A. cellular telephone
PPO |  A. cellular telephone
Table 8: A test example: The response to the same question in different
training stages for Baichuan-33B.