# Electron density variations in the interstellar medium and the average
frequency profile of a scintle from pulsar scintillation spectra

N. Bartel  York University, 4700 Keele St., Toronto, ON M3J 1P3, Canada  M. S.
Burgin  Lebedev Physical Institute, Astro Space Center, Profsoyuznaya 84/32,
Moscow, 117997 Russia  E. N. Fadeev  Lebedev Physical Institute, Astro Space
Center, Profsoyuznaya 84/32, Moscow, 117997 Russia  M. V. Popov  Lebedev
Physical Institute, Astro Space Center, Profsoyuznaya 84/32, Moscow, 117997
Russia  N. Ronaghikhameneh  York University, 4700 Keele St., Toronto, ON M3J
1P3, Canada  University of Alberta, 116 St. & 85 Ave., Edmonton, AB T6G 2R3,
Canada  T. V. Smirnova  Lebedev Physical Institute, Pushchino Radio Astronomy
Observatory, Pushchino 142290, Moscow region, Russia  V. A. Soglasnov  Lebedev
Physical Institute, Astro Space Center, Profsoyuznaya 84/32, Moscow, 117997
Russia

###### Abstract

We observed the scintillation pattern of nine bright pulsars at 324 MHz and
three at 1.68 GHz and analyzed the wavenumber spectrum which is related to
electron density variations of the plasma turbulence of the interstellar
medium. For all pulsars the frequency section of the autocorrelation function
of the dynamic spectra to at least 45% of the maximum corresponds to
predictions of scattering theories with a range of power-law exponents of the
wavenumber spectrum of $3.56\leq\alpha\leq 3.97$ with errors $\leq 0.05$ and a
mean with standard deviation of $3.76\pm 0.13$. The range includes
$\alpha=3.67$ for the Kolmogorov spectrum. Similar results although with
larger errors were found from the Fourier transform of the autocorrelation
functions down to $\sim 10^{-3}$ of the maximum. No clear case of a
distinction between thin-screen and extended-medium scattering models was
found. The average frequency profile of the scintles can be characterized for
steep wavenumber spectra with $\alpha\lesssim 4$ by a cusp with a somewhat
rounded peak. For flatter spectra, down to at least $\alpha\sim 3.56$ the cusp
with its peak becomes more pronounced and its decay steepens. We discuss our
findings in the context of scattering characteristics of the interstellar
medium.

scattering — pulsars — radio continuum: ISM — techniques

††facilities: SRT, GBT, Aresibo, WSRT, Pushchino 22-m radio telescope (used as
Tracking Station)††software: CFITSIO

##  1 Introduction

Electron density variations in the interstellar medium (ISM) scatter radio
emission from cosmic sources. The variations can be characterized by a spatial
correlation function. Its Fourier transform is the spatial wavenumber spectrum
with an exponent, $\alpha$, that describes its steepness. Extensive studies of
scattering effects began with the discovery of pulsars, since pulsars are
quasi point-like sources that provide a coherent flux of radio waves where the
influence of the source structure can be neglected. Scattering leads to image
blurring, pulse broadening, and intensity modulation over time and frequency.
The observable parameters characterizing these phenomena are the scattering
angle, $\theta_{\mathrm{sc}}$, the scattering time, $\tau_{\mathrm{sc}}$, the
diffraction scintillation time, $t_{\mathrm{scint}}$, and the decorrelation
bandwidth, $\Delta f_{1/2}$. Theoretical considerations of scattering effects
by, e.g., Lovelace (1970), Lee & Jokipii (1975a, b, c), Rickett (1977),
Shishov et al. (2003a) have established early on a number of relationships
between characteristics of electron density variations and observable
scattering parameters such as those given above.

Previously we reported on a comparison of $\theta_{\mathrm{sc}}$ and
$\tau_{\mathrm{sc}}$ and our estimates of the distances to the effective
scattering screens (Gwinn et al., 2016; Fadeev et al., 2018; Popov et al.,
2017, 2020), as well as on the time characteristics of the scintillation
pattern (Popov & Smirnova, 2021).

In this paper we focus on the frequency characteristics of the scintillation
pattern. The frequency section of the time-averaged autocorrelation functions,
$ACF(\Delta f)$, of consecutive pulsar scintillation spectra, or dynamic
spectra, is often used to derive the parameter, $\Delta f_{1/2}$. This
section, or its Fourier transform, $\mathcal{FT}[ACF(\Delta f)]$, can also be
compared with predictions of thin-screen and extended-medium scattering models
of plasma inhomogeneities in the intervening ISM, both for a Gaussian and a
power-law spatial wavenumber spectrum with spectral index, $\alpha$. While
there have been many observational investigations of the parameter, $\alpha$,
only very few, and then for only a few pulsars, are based on observing
$ACF(\Delta f)$ functions and comparing them to model predictions (e. g.,
Armstrong & Rickett, 1981; Wolszczan, 1983).

New insights into the scattering process were obtained with the development of
space VLBI observations of pulsars with Radioastron which allowed the
predicted substructure (Goodman & Narayan, 1989) of the visibility function to
be discovered (Gwinn et al., 2016; Popov et al., 2017). On long ground-space
baseline projections for which the scattering disk is fully resolved the
visibility function retains values clearly larger than zero over a range of
delays corresponding to the scattering time, $\tau_{\mathrm{sc}}$. Since
$\tau_{\mathrm{sc}}$ is inversely related to $\Delta f_{1/2}$, pulsars with a
broad visibility function are expected to have a narrow decorrelation
bandwidth, $\Delta f_{1/2}$. Therefore, depending on the time and frequency
resolution of the observations, using the functions $ACF(\Delta f)$ and
$\mathcal{FT}[ACF(\Delta f)]$ can be considered complementary for providing a
more complete analysis of the dynamic spectra.

In this paper we report on an analysis of the dynamic spectra of 12 pulsars.
The time range of the dynamic spectra is for each pulsar clearly longer than
the scintillation time so that our results refer to an analysis of the
scintillations in the averaging mode (Narayan, 1992). We compare the dynamic
spectra with the predictions of thin-screen and extended-medium scattering
models in terms of the wavenumber spectral index, $\alpha$, of the ISM
electron density variations, discuss our results with measurements by others,
search for hints for an observational preference for either of the models,
relate the dynamic spectra to the characteristics of the substructure in the
visibility functions and infer the average frequency profile of the scintles
as a function of $\alpha$.

##  2 Observational parameters of the dynamic spectra

We studied the pulsar dynamic spectra for nine pulsars at a center frequency
of 324 MHz and for three pulsars at 1676 MHz. The data were obtained with VLBI
recorders over a bandwidth of 16 MHz in the context of the space VLBI
Radioastron scientific program (Kardashev et al., 2017). We list the pulsars
together with their periods, dispersion measures, galactic coordinates,
recording stations, dates of observations and observing frequencies in Table
1. The data were already used earlier in our other studies (refer to ”Code” in
Table 1) and more details on these observations and the first stage of data
analysis are given in our previous papers (see above).

Each dynamic spectrum, $S(f_{i},t_{j})$ as a function of frequency, $f$, and
time, $t$, consists of $N_{\mathrm{f}}\times N_{\mathrm{t}}$ values, with
$1\leq i\leq N_{\mathrm{f}}$ and $1\leq j\leq N_{\mathrm{t}}$ where
$N_{\mathrm{f}}$ is the number of frequency channels covering the frequency
range in the bandpass from 316 to 332 MHz or from 1668 to 1684 MHz, and
$N_{\mathrm{t}}$ the number of individual spectra in a given set of
observations. We calibrated the dynamic spectra as follows:

$S(f_{i},t_{j})=\frac{S^{\mathrm{on}}(f_{i},t_{j})-S^{\mathrm{off}}(f_{i},t_{j})}{S^{\mathrm{off}}(f_{i},t_{j})}$
(1)

and hereafter refer to them simply as $S(f,t)$. Here, $S^{\mathrm{on}}$ and
$S^{\mathrm{off}}$ are the spectra obtained during the on-pulse and off-pulse
windows. Usually, the time interval between successive spectra is equal to the
period of the pulsar, but in some cases averaging was done over several
periods to smooth out the intensity fluctuations from pulse to pulse. The time
span for the dynamic spectra, as well as $N_{\mathrm{f}}$ and
$N_{\mathrm{t}}$, is also listed in Table 1.

Table 1: List of pulsars

PSR | P | DM | $l_{II}$ | $b_{II}$ | Obs. | $N_{\mathrm{f}}$ | $N_{\mathrm{t}}$ | Station | Date | $f$ | Code
---|---|---|---|---|---|---|---|---|---|---|---
| (s) | (pc cm-3) | (deg) | (deg) | (min) |  |  |  | (yyyy mm dd) | (MHz) |
(1) | (2) | (3) | (4) | (5) | (6) | (7) | (8) | (9) | (10) | (11) | (12)
B0329+54 | 0.714 | 26.7 | 145.0 | -1.2 | 60 | 4096 | 504 | GB | 2012 11 26 | 324 | raes10a
B0525+21 | 3.745 | 50.9 | 183.4 | -6.9 | 168 | 512 | 2700 | AR | 2013 09 18 | 1676 | raks02ac
B0823+26 | 0.531 | 19.4 | 197.0 | 31.7 | 150 | 2048 | 16900 | AR | 2015 03 11 | 324 | rags04aj
B0834+06 | 1.274 | 12.8 | 219.7 | 26.3 | 55 | 8192 | 3300 | GB | 2014 12 08 | 324 | rags04ah
B0919+06 | 0.430 | 27.3 | 225.4 | 36.4 | 90 | 2048 | 5200 | AR | 2018 05 10 | 324 | rags29p
B1133+16 | 1.188 | 4.8 | 241.9 | 69.2 | 120 | 1024 | 6000 | AR | 2018 02 03 | 324 | rags29g
B1237+25 | 1.382 | 9.3 | 252.0 | 86.5 | 100 | 512 | 4340 | AR | 2017 12 22 | 324 | rags29c
B1642 – 03 | 0.387 | 35.7 | 14.1 | 26.1 | 90 | 512 | 13200 | WB | 2013 08 09 | 324 | raks02ab
B1749 – 28 | 0.562 | 50.8 | 1.5 | -1.0 | 250 | 192 | 2450 | PA | 2014 05 26 | 1676 | raks02az
B1929+10 | 0.226 | 3.2 | 47.4 | -3.9 | 100 | 512 | 26000 | AR | 2015 05 05 | 324 | rags04ao
B1933+16 | 0.359 | 158.5 | 52.4 | -2.1 | 90 | 8192 | 15036 | AR | 2013 08 01 | 1676 | rags02aa
B2016+28 | 0.558 | 14.1 | 68.0 | -4.0 | 45 | 2048 | 5300 | AR | 2015 05 22 | 324 | rags04aq

Note. — Columns are as follows: (1) pulsar name, (2) pulsar period, (3)
dispersion measure, (4) galactic longitude, (5) galactic latitude, (6)
observing time range for dynamic spectra, (7) number of frequency channels
across the bandpass, (8) number of spectra over the observing time range, (9)
radio telescope station: AR – Arecibo, GB – Green Bank, PA – Parkes, WB –
Westerbork, (10) observing date, (11) center of observing frequency, (12) code
of observing session.

Figure 1: Dynamic spectra of four pulsars analyzed in this paper. The color-
coded intensity as a function of time and frequency is given in arbitrary
units in the bar on the right side of each spectrum.

In Figure 1 we give examples of dynamic spectra for four pulsars: B0525+21,
B0919+06, B1133+16, and B1642-03. The dynamic spectra for the other eight
pulsars in our sample were already published earlier (Popov et al., 2016,
2021; Fadeev et al., 2018). One can see islands of increased intensity
distributed randomly over the frequency-time domain. These are the peaks of
diffractive structure, which we refer to as ’scintles.’ In particular we are
interested in the decorrelation bandwidth, $\Delta f_{1/2}$, the frequency
section of the autocorrelation function, $ACF(\Delta f)$, as well as the
Fourier transform of this function, their relation to parameters of the
turbulent interstellar plasma and the average frequency profile of the
scintles.

##  3 Analysis of the dynamic spectra

###  3.1 Frequency structure of the dynamic spectra and their relation to the
turbulent plasma of the intervening ISM

Scintillations of pulsars are related to electron density fluctuations in the
ISM which can be characterized by a spatial correlation function. Its Fourier
transform is the spatial wavenumber spectrum, $P(q)$. If the magnitude of the
three dimensional wavenumber, $q$, is within a limited range of wavenumbers
with inner and outer boundaries, then $P(q)$ can be described as a power-law,
$P_{\mathrm{P}}(q)=C_{n}q^{-\alpha}$, with $\alpha<4$ (Romani et al., 1986)
including $\alpha=11/3$ for a Kolmogorov spectrum. For a Gaussian the spectrum
is, $P_{\mathrm{G}}(q)=G_{n}\exp(-(q/q_{0})^{-2})$ corresponding to
$\alpha=4$. The value of $\alpha$ can be obtained from the two-dimensional
autocorrelation function of $S(f,t)$ at zero time lag, $ACF(\Delta f,\Delta
t=0)$. For simplicity we refer to this frequency section after normalization
as $ACF(\Delta f)$ with

$ACF(\Delta f)=\left\langle(S(f,t))(S(f+\Delta f,t))\right\rangle_{t}.$ (2)

The frequency section, $ACF(\Delta f)$, is predicted to depend on $\alpha$.
For a thin-screen model of the ISM analytical solutions exist for the Gaussian
and power-law models that can be fit to the autocorrelation functions. For an
extended-medium model of the ISM (e.g. Lee & Jokipii, 1975a) analytical
solutions for the frequency section of the ACF exist for the Gaussian spectrum
but only numerical solutions for the power-law spectrum (Lee & Jokipii,
1975b).

Instead of computing the frequency section, $ACF(\Delta f)$, for comparison
with predictions, some authors preferred to compare the Fourier transform of
this ACF with predictions. Both approaches can be considered complementary. In
this paper we use both approaches by following (Wolszczan, 1983) concerning
the computation of the predictions of the extended-medium scattering model and
Armstrong & Rickett (1981) concerning the thin-screen model.

###  3.2 Determination of the power-law spectral index, $\alpha$, of the
wavenumber spectrum

We studied 12 pulsars by analyzing the frequency section, $ACF(\Delta f)$, of
the dynamic spectra. From $ACF(\Delta f)$ we determined the decorrelation
bandwidth, $\Delta f_{1/2}$, as the half-width at half maximum (HWHM) and list
the values for each pulsar in Table 2. The fractional statistical (rms) error
can be estimated as the inverse square root of the number of scintles in each
dynamic spectrum (Backer, 1975). An approximate expression is given by
$\sigma_{\mathrm{stat}}=\left(f_{s}\frac{BT_{\mathrm{obs}}}{\Delta
f_{1/2}t_{\mathrm{scint}}}\right)^{-1/2}$ where $f_{s}$ is the filling factor,
$B$ the receiver bandwidth, $T_{\mathrm{obs}}$ the observing time, and
$t_{scint}$ the scintillation time. Assuming $f=0.5$ and taking values for
$t_{\mathrm{scint}}$ from Popov & Smirnova (2021) we list
$\sigma_{\mathrm{stat}}$ in Table 2. For a consideration of possible
systematic errors, see Popov & Smirnova (2021).

Table 2: Results of data reduction

PSR | $\Delta f_{1/2}$ | $\Delta\tau_{1/2}$ | $\sigma_{\mathrm{stat}}$ | $\alpha_{ACF}$ | $rms_{ACF}$ | $\alpha_{\mathcal{FT}(ACF)}$ | $rms_{\mathrm{scat.\,type}}$ | Scat. type
---|---|---|---|---|---|---|---|---
| (kHz) | $(\mu s)$ | $(10^{-2})$ |  | $(10^{-2})$ |  | $(10^{-2})$ |
(1) | (2) | (3) | (4) | (5) | (6) | (7) | (8) | (9)
B0329$+$54 | 30 | 7.90 | 1.0 | 3.67$\pm$0.02 | 0.05 | 3.80$\pm$0.05 | 0.15 | K
B0525$+$21 | 4000 | 0.04 | 8.6 | 3.73$\pm$0.02 | 0.5 | 3.80$\pm$0.05 | 3.1/4.2/4.2 | K/E/L
B0823$+$26 | 180 | 0.83 | 1.2 | 3.56$\pm$0.02 | 0.2 | 3.60$\pm$0.05 | 1.8 | K
B0834$+$06 | 400 | 0.55 | 5.8 | 3.74$\pm$0.02 | 0.2 | 4.00$\pm$0.10 | 1.8 | K
B0919$+$06 | 170 | 1.15 | 1.9 | 3.59$\pm$0.02 | 0.2 | 3.70$\pm$0.10 | 1.7 | K
B1133$+$16 | 96 | 2.50 | 0.7 | 3.80$\pm$0.02 | 0.3 | 4.00$\pm$0.10 | 2.3/3.7 | K/L
B1237$+$25 | 2300 | 0.11 | 3.3 | 3.90$\pm$0.04 | 0.7 | 3.80$\pm$0.10 | 2.7/3.8 | G/E
B1642$-$03 | 930 | 0.20 | 4.0 | 3.97$\pm$0.05 | 1.4 | 3.80$\pm$0.20 | 3.0/3.7 | G/E
B1749$-$28 | 690 | 0.31 | 3.5 | 3.78$\pm$0.02 | 0.2 | 3.90$\pm$0.10 | 3.7/3.8/3.9 | E/L/K
B1929$+$10 | 970 | 0.39 | 6.8 | 3.85$\pm$0.02 | 0.4 | 3.90$\pm$0.10 | 2.8 | E
B1933$+$16 | 120 | 6.85 | 1.1 | 3.64$\pm$0.04 | 0.7 | 3.80$\pm$0.10 | 1.7 | K
B2016$+$28 | 71 | 2.04 | 4.9 | 3.92$\pm$0.02 | 0.2 | 3.90$\pm$0.05 | 1.1/1.7 | L/E

Note. — Columns are as follows: (1) pulsar name; (2) HWHM of the ACFs of the
dynamic spectra at zero time lag (the smaller value for for B1643-03 of $\sim$
6 kHz is not considered in this paper); (3) HWHM of the FT(ACF)s; (4)
fractional rms uncertainty of values in columns (2) and (3); (5) parameter,
$\alpha$, from the fit of the extended-medium scattering model to the observed
ACFs down to 0.45 of the maximum. The errors were calculated from the
statistical errors derived from column (6) added in quadrature with systematic
errors estimated from slight deviations of the ACF from the model (same values
for extended-medium and thin screen models); (6) the rms values in units of
correlation coefficients for the ACFs with respect to the fit model; (7)
parameter, $\alpha$, from the fit of the extended-medium scattering model to
the Fourier transforms of the observed ACFs for the delay range from 0.2 to 20
$\Delta\tau/\Delta\tau_{1/2}$ or down to $\sim 10^{-3}$ of the maximum (same
values for extended-medium and thin screen models). The errors are largely
systematic errors estimated from non-random deviations of $\mathcal{FT}(ACF)$
from the models; (5, 7) values including errors larger than 4 are still valid
but nonphysical within the context of the models and may indicate deviations
from them; (8) the smallest rms values in units of $(10^{-2})$ of the maximum
of unity of the observed ACFs with respect to the analytical $ACF(f)$
functions in column (9) from Table 4 up to $2\times\Delta f/\Delta f_{1/2}$
(see Figure 7); (9) the best matching functions from the four functions from
Table 4, $ACF(G)$ to $ACF(K)$.

Figure 2: Comparison of the predictions from the extended-medium model of Lee
& Jokipii (1975b) with computed autocorrelation functions for our 12 pulsars.
The predictions are plotted as solid lines for indices of $\alpha$
corresponding to the best-fitting models. The ACF’s are plotted as data points
with different symbols for each of the pulsars listed on the upper right.

We fit the functions with a curve corresponding to the predictions for the
extended-medium model. The differences between the predictions for $ACF(\Delta
f)$ for the thin-screen and the extended-medium models are small for the inner
part but larger for the tail of the functions. Using, for instance, the
solutions for the Gaussian spectrum ($\alpha=4.0$) (Lovelace, 1970; Chashei &
Shishov, 1976; Lerche, 1979) with a maximum at unity and scaling them so that
the values for $\Delta f_{1/2}$ are identical, the functions differ by less
than 0.002 down to the HWHM and by less than 0.007 down to 20% of the maximum.
These differences are not large enough to be considered for our analysis of
the inner part of $ACF(\Delta f/\Delta f_{1/2})$ but have to be considered for
the tail of the ACFs where they could perhaps become measurable. Although for
some pulsars a good fit could be obtained even for the tail of $ACF(\Delta
f/\Delta f_{1/2})$, a consistently good fit for all pulsars was only obtained
up to frequency lags, $\Delta f/\Delta f_{1/2}\sim 1.2$, where $ACF(\Delta
f/\Delta f_{1/2})=0.45$. For this range, the fitted values of $\alpha$ for the
two scattering models were the same within our sensitivity limits. We list the
values of $\alpha$ together with their uncertainties in Table 2.

For a summary view of our 12 measured $ACF(\Delta f/\Delta f_{1/2})$ functions
we plot their inner portions down to half of the maximum and compare them with
the fit model predictions in Figure 2. It can be seen that the value of
$\alpha$ is mostly determined by the shape of the inner part of $ACF(\Delta
f/\Delta f_{1/2})$. All the functions are within a range of fit parameters,
$\alpha$, between 3.56 for the lower curve of PSR B0823+26 and 3.97 for the
upper curve of PSR B1642-03. The mean with standard deviation is
$\langle\alpha_{ACF}\rangle=3.76\pm 0.13$ and is therefore close to
$\alpha_{K}=3.67$ of the Kolmogorov spectrum. The quality of the fit varies.
We list rms variations of the observed ACFs from the model for each pulsar in
Table 2. The fits are for most pulsars quite good with rms values smaller or
equal to 0.005 on the scale of the ACF functions down to 0.45 of the maximum
of unity. Larger deviations with rms values of 0.007 to 0.014 were found for
PSRs B1237+25, B1642-03 and B1933+16. These are mostly not random variations
but to a large part systematic deviations from the model.

To inspect the fits in more detail down to lower correlation coefficients, we
plot $ACF(\Delta f/\Delta f_{1/2})$ with the models from Figure 2 for each of
the 12 pulsars separately for frequency lags up to 2 $\times$ $\Delta f_{1/2}$
in Figure 3. For about half the pulsars the data are still well fit by the
model over the wider range of frequency lags. However significant deviations
are now apparent for the other half.

Some of these deviations in the tail of $ACF(\Delta f/\Delta f_{1/2})$ could
be due to technical effects such as 1) superposition of the contribution from
neighboring scintles; 2) limited band of the receiver filter which distorts or
cuts off wide diffraction structures; 3) inaccuracy of the determination of
the off-pulse level for the computation of ACFs and 4) relatively low signal-
to-noise ratios (SNRs). Some of these effects if not all could possibly
increase the ACFs toward large lags. Such increase can be seen for PSRs
B0834+06 and B1133+16. For the other pulsars with deviations, the tail of the
ACFs is lower than the model predictions. In these cases the deviations are
likely indications of deficiencies of the scattering models.

Figure 3: Comparison of the $ACF(\Delta f)$ functions derived from the
observations (blue dots) with the theoretical predictions for the extended-
medium scattering theories with power-law spectral index, $\alpha$, as a
fitting parameter. The solid line shows the best-fitting model for the inner
part of the $ACF(\Delta f)$ down to 45% consistently for all pulsars. The
surrounding dotted lines indicate $\Delta\alpha=\pm 0.05$ deviations from the
best-fit value for illustration purposes. For better comparison all functions
were normalized by their HWHM frequency lags, $\Delta f_{1/2}$.

A complementary method to estimate $\alpha$ values is to fit the models to the
Fourier transform of the ACFs, $\mathcal{FT}(ACF)$ (e.g., Armstrong & Rickett,
1981; Wolszczan, 1983). We computed $\mathcal{FT}[ACF(\Delta f)]$ by using the
full range of frequency delays of $\pm 4$ MHz. To suppress fluctuations, we
applied to the observed ACF$(\Delta f)$ functions a Gaussian filter with a
half-width at the 1/e level of $4\times\Delta f_{1/2}$. Only for PSR B0525+21
with its relatively large value of $\Delta f_{1/2}$ we used a four times
narrower Gaussian.

A comparison of the observed functions $\mathcal{FT}[ACF(\Delta f)]$ of all
pulsars with the theoretical predictions for the extended-medium and for the
thin-screen models is shown in Figure 4. In this subsection we focus on the
fit of the former to the FT(ACF)s. All FT(ACF) functions are normalized in
amplitude and in width. Their HWHM values, $\Delta\tau_{1/2}$, are listed in
Table 2.

Figure 4: The magnitude of the Fourier transform of the observed ACFs,
|FT(ACF)|, (green dots), fit by the theoretical predictions for the extended-
medium scattering theory with rounded power-law spectral index $\alpha$ (solid
blue lines) and for the thin-screen model (dashed red lines) with the same
value for $\alpha$. Scattered points in the tail of some functions are due to
noise. Delay is given as $\Delta\tau$ with $\Delta\tau_{1/2}$ as the HWHM
listed in Table 2.

The values of $\alpha_{\mathcal{FT}(ACF)}$ and their estimated errors are
listed in Table 2 (column (7)). The mean with standard deviation is
$\langle\alpha_{\mathcal{FT}(ACF)}\rangle=3.83\pm 0.12$, larger by 0.07 than
$\langle\alpha_{ACF}\rangle$. The mean of the magnitudes of the differences to
those obtained from the fits to the ACFs with standard deviation is
$\langle|\alpha_{ACF}-\alpha_{\mathcal{FT}(ACF)}|\rangle=0.11\pm 0.07$. In
general the values are fairly consistent within the errors given that the
fitting ranges are quite different. The largest discrepancy is 0.26 for PSR
B0834+06. The difference could perhaps at least partly be due to the large
deviations from the model in the ACF for lags outside the ACF fitting range.

###  3.3 Comparison with other estimates of $\alpha$

The exponent, $\alpha$, of the wavenumber power-law spectrum of electron
density variations can be estimated by several methods. In Table 3 we compare
our estimates from Table 2 in columns (5) and (7) with those from other
authors derived from the Fourier transform of the frequency section of the
ACF’s, structure function, and frequency dependence of the decorrelation
bandwidth.

The most similar analysis to ours was done by Wolszczan (1983) (column (4))
and Armstrong & Rickett (1981) (column (5)) with data taken some 30 years
before ours who fit an extended-medium and a thin-screen scattering model,
respectively, to the FTs of the ACFs. Compared with the values of Wolszczan
(1983), our corresponding values in column (3) are all consistent with theirs
within our errors. Compared with the values of Armstrong & Rickett (1981)
which are given with relatively large ranges, ours are within their ranges or
slightly above but exclude the range below $\alpha=3.5$.

The values in the other columns were obtained with different analyses schemes.
The most interesting for a comparison are those listed in column (6) which are
based on the same data as ours in columns (2) and (3). They are derived from
the structure function of the time section of the ACFs and are therefore best
compared with our values in column (2). For five pulsars, the differences in
$\alpha$ are $\leq$0.10. For the other four pulsars, ignoring B2016+28 (see
caption of Table 3), the differences are as large as 0.51 as for B1237+25.

The last column lists estimates of $\alpha$ obtained from a fit of the
decorrelation bandwidth as a function of observing frequency. They are
consistent with our values in column (2) within 1.5 times their larger
uncertainties. We comment on these comparisons in Section 4.

Table 3: Comparison of estimates of $\alpha$

PSR | $\alpha_{ACF(\Delta f)}$ | $\alpha_{\mathcal{FT}(ACF)}$ | $\alpha_{\mathcal{FT}(ACF)}$ | $\alpha_{\mathcal{FT}(ACF)}$ | $\alpha_{\mathrm{SF}}$ | $\alpha_{\Delta f_{1/2}(f)}$
---|---|---|---|---|---|---
(1) | (2) | (3) | (4) | (5) | (6) | (7)
B0329$+$54 | $3.68\pm 0.02$ | $3.80\pm 0.05$ | 3.85 | 3.7-4.0 | $3.67\pm 0.01$ | $3.41_{-0.25}^{+0.41}$
B0525$+$21 | $3.73\pm 0.02$ | $3.80\pm 0.05$ |  |  |  |
B0823$+$26 | $3.56\pm 0.02$ | $3.60\pm 0.05$ |  |  | $3.66\pm 0.01$ |
B0834$+$06 | $3.74\pm 0.02$ | $4.00\pm 0.10$ |  |  | $3.528\pm 0.006$ |
B0919$+$06 | $3.59\pm 0.02$ | $3.70\pm 0.10$ |  |  | $3.57\pm 0.01$ |
B1133$+$16 | $3.80\pm 0.02$ | $4.00\pm 0.10$ | 3.90 | 3.4-3.8 | $3.86\pm 0.01$ |
B1237$+$25 | $3.90\pm 0.04$ | $3.80\pm 0.10$ |  |  | $3.39\pm 0.01$ |
B1642$-$03 | $3.97\pm 0.05$ | $3.80\pm 0.20$ |  | 3.2-3.9 |  | $3.63_{-0.21}^{+0.29}$
B1749$-$28 | $3.78\pm 0.02$ | $3.90\pm 0.10$ |  |  | $3.82\pm 0.01$ | $3.50_{-0.14}^{+0.18}$
B1929$+$10 | $3.85\pm 0.02$ | $3.90\pm 0.10$ |  |  | $3.65\pm 0.02$ |
B1933$+$16 | $3.64\pm 0.04$ | $3.80\pm 0.10$ | 3.90 |  | $3.18\pm 0.01$ | $3.80_{-0.12}^{+0.14}$
B2016$+$28 | $3.92\pm 0.02$ | $3.90\pm 0.05$ |  |  | $\;\,3.36\pm 0.02^{*}$ |

Note. — Columns are as follows: (1) pulsar name, (2) taken from Table 2,
column (5); (3) taken from Table 2, column (7); (4) estimates of $\alpha$ of
the extended medium power-law model from the Fourier transform of the
frequency ACFs, PSRs B0329+54 (327 and 480 MHz), B1133+16 (327 MHz), B1933
(1416 MHz) (Wolszczan, 1983); (5) estimate of $\alpha$ of the thin-screen
power-law model from the Fourier transform of the frequency ACFs, PSRs
B0329+54 (340, 408 MHz), B1133+16 (340 MHz), B1642-03 (340 MHz) (Armstrong &
Rickett, 1981); (6) estimate of $\alpha$ of the power-law spectrum from the
structure function of the time section of the ACFs, with ’∗’ indicating a
measurement biased toward lower values (Popov & Smirnova, 2021); (7) estimate
of $\alpha$ of the power-law model from the decorrelation bandwidth, $\Delta
f_{1/2}$, as a function of observing frequency in the range 80-8100 MHz,
(Cordes et al., 1985).

###  3.4 Thin screen or extended medium?

To search for any indications whether a thin-screen or an extended-medium
scattering model is preferred by our observed $ACF(\Delta f)$ functions, we
extend the functions to frequency lags of 4.5$\times\Delta f_{1/2}$ and plot
the observed ACFs from Figure 3 together with the predictions of the two
scattering models in Figure 5. For pulsars with broad scintles relative to the
bandwidth of 16 MHz (see, e.g., PSR B0525+21, Figure 1 and Table 2), we plot
the functions only for a portion of the total range in frequency lags.

Slight differences in the models can be seen at $\Delta f\sim 2.5\times\Delta
f_{1/2}$ that grow larger with further frequency lags. For most pulsars the
tail of the measured ACFs deviates substantially from either of the models
and, as discussed in subsection 3.2, can likely be interpreted in most cases
as being due to deficiencies in the theoretical scattering models.

Figure 5: Comparison of the $ACF(\Delta f)$ functions derived from our
observations (green dots) with the best-fit theoretical predictions for the
extended-medium model with power-law spectral index $\alpha$ as a fitting
parameter (solid blue lines) and for the thin-screen model (dashed red lines)
with the same spectral index for comparison. For better comparison all
functions were normalized by their HWHM frequency lags, $\Delta f_{1/2}$. Only
the inner part of the ACFs down to 45% or up to $\Delta f/\Delta f_{1/2}\sim
1.2$ for which a good fit could be obtained for all pulsars was used for the
fit. The parameter, $\alpha$, which is essentially the same for our ACFs for
both models for the fitting range is given for each pulsar. For some pulsars
the width of the scintles in terms of $\Delta f_{1/2}$ relative to the
receiver bandwidth of 16 MHz was relatively large so that only a shortened
version of the functions was plotted.

Three pulsars, B0329+54, B0823+26 and B0919+06, show fairly good consistency
in the observed ACFs with either, or both, of the models for frequency lags at
least up to $4.5\times\Delta f_{1/2}$. For PSRs B0329+54 and B0919+06,
deviations from either of the two models clearly grow for even larger lags.
PSR B0823+26, however, is a candidate that warrants more scrutiny. In Figure 6
we show the observed ACF and the two model curves but now up to $\Delta f\sim
6\times\Delta f_{1/2}$. The extended-medium model is drawn as in Figure 5 with
$\alpha_{em}=3.56$. The thin-screen model curve lies for the same $\alpha$
value above the extended-medium model curve and is therefore a candidate for
matching the tail of the ACF, but not that much that it would be a good fit to
the observed ACF. To investigate this case further, we plot the thin-screen
model for $\alpha_{ts}=3.49$, 3.5 $\sigma$ below the best fit value in an
attempt to deviate not too much from the best fit but also to match the tail
of the ACF. However, even in this extreme case, deviations much larger than
for the inner part of the ACF are still visible. We therefore think that also
in this case the deviations are due to effects discussed already for the other
pulsars. There is no convincing case that either of the two models is
preferred.

Figure 6: The $ACF(\Delta f)$ function (thick green line) for PSR B0823+26
compared with the extended medium (solid blue curve) and thin-screen model
(dashed red curve) but for frequency lags up to about $6\times\Delta f/\Delta
f_{1/2}$. The extended medium model is drawn as in Figure 4 with
$\alpha_{\mathrm{em}}=3.56$. The thin-screen model is plotted for
$\alpha_{\mathrm{ts}}=3.49$ as a test, in an attempt to match the tail of the
ACF without excessively changing the best-fit value of $\alpha=3.56$.

An alternative approach to search for differences is to analyze the Fourier
transform of the ACFs, $\mathcal{FT}[ACF(\Delta f)]$ (e.g., Armstrong &
Rickett, 1981; Wolszczan, 1983) and the respective thin-screen and extended-
medium models as shown in Figure 4. Differences between the models are
somewhat more pronounced than in Figure 5. However also in this case, the
observed functions $\mathcal{FT}[ACF(\Delta f)]$ could not conclusively
discriminate between either of the models.

###  3.5 Relation to the dynamic visibility function from space VLBI
Radioastron observations

The visibility function, $V(\tau,t)$, as a function of delay and time in our
space VLBI observations is obtained from the inverse Fourier transform of the
cross-spectrum between two different radio telescopes. In our case of
observations of dynamic spectra with single telescopes, the cross-spectrum
corresponds to the auto-spectrum at any particular time, $t$, in a dynamic
spectrum, $S(f,t)$. A Gaussian has been used to approximate the frequency
section of the ACF’s of dynamic spectra (e.g. Cordes, 1986). Since for a
Gaussian ACF, the underlying function and its Fourier transform are also
Gaussians, the auto-spectrum would also be expected to be a Gaussian. Table 4
lists the function $y$ as a Gaussian function, $G(f)$, together with three
other functions, $L(f)$, $E(f)$, $K(f)$, we consider further in our analysis.
In addition we list for each function, $y$, its Fourier transform,
$\mathcal{FT}(y)=\tilde{y}$, and its $ACF(y)$.

Table 4: Fourier transform and self-convolution of four parametric families of
functions

Name | $y$ | $\tilde{y}$ |  | $ACF(y)$
---|---|---|---|---
Gauss |  | G(1/(2πσ), t)2πσ | 12π | G(2 σ,Δf) —
Lorentz |  | E(1/(2πλ), t)π λ | 12π | L(2λ,Δf) —
Laplace |  | L(1/(2πϵ), t))2 ϵ | 12π | (ϵ+—Δf—)E(ϵ,Δf)2 ϵ —
Bessel |  | (L(1/(2πϕ), t)2 ϕ)1/2 |  | E(ϕ,Δf) —

Note. — Name of the function, $y$, definition of $y$, Fourier transform of $y$
with $\mathcal{FT}(y)=\tilde{y}$, The Fourier transform is defined here as
$\tilde{y}(p,t)=\int^{+\infty}_{-\infty}y(p,f)e^{-2\pi ift}\,\mathrm{d}f$,
with $p$ as a parameter. All the functions considered here are real and
normalized so that $\tilde{y}(0)=1$. Therefore, the autocorrelation function
is identical with self-convolution and $\widetilde{ACF}={\tilde{y}}^{2}$.
Because of the normalization chosen, the functions may be identified with
probability densities of random variables. The functions named “Lorentz”,
“Gauss”, and “Laplace” are related to the well-known statistical distributions
whose mathematical properties are described by Feller (1971). For the last
row, we used the term “Bessel” for the function since $K_{0}(z)$ is the
modified Bessel function of the second kind of order 0 and argument z which
rises logarithmically to infinity at $f\to 0$. The scale parameters $\sigma$
and $\phi$ are equal to rms deviations of the corresponding distributions, and
the parameters $\lambda$ and $\epsilon$ are half-widths of the Lorentz and
Laplace distributions at the level $1/2$ and $1/e$, respectively.

Figure 7: The four $ACF(y)$ functions listed in Table 4 and plotted as a
function of frequency lag $\Delta f$. The functions are normalized in
amplitude and with respect to their HWHM, $\Delta f_{1/2}$, for easy
comparison. All functions are two-sided, only one side is plotted. In addition
observed ACFs are plotted for four pulsars to display the range of ACF
variations for comparison.

Space VLBI with Radioastron revealed for some pulsars that the tail of the
delay cross-section of $V(\tau,t)$ retained significant magnitudes at baseline
projections where the pulsar scattering disk was completely resolved. The
delay cross-section, $ACF(\Delta\tau,\Delta t=0)$ of $V(\tau,t)$, was found to
be well approximated by a Lorentz function (Popov et al., 2020). The Fourier
transform of a Lorentz function is a two-sided exponential and therefore the
auto-spectrum from our $S(f,t)$ would also be expected to be a two-sided
exponential (second row in Table 4). The ACF of such an exponential, $ACF(E)$,
is a function given in the third row in Table 4.

Independent of any prior knowledge from space VLBI observation, an inspection
of our observed ACF functions in Figure 3 indicates that several of them have
up to $\sim 0.5\times\Delta f_{1/2}$ a concave profile, with a rounded top
where the second derivative is negative. Such a profile is similar to the
analytical functions, $ACF(G)$, $ACF(L)$ and $ACF(E)$ in Table 4. Some others
have in contrast a convex inner profile where the second derivative is
positive. This profile is similar to $ACF(K)$.

In Figure 7 we plot as examples the observed ACFs of four pulsars that define
the range in curvature for all 12 pulsars, and compare them to the four
analytical $ACF(y)$ functions. PSR 1642-03 has the most concave inner ACF of
all pulsars. It is between that of the $ACF(G)$ and $ACF(E)$ functions. For
the inner part it resembles $ACF(E)$ and for the outer part of the plot in
lies between $ACF(E)$ and $ACF(G)$. For PSR 2016+28, the ACF is closest to
$ACF(L)$ and $ACF(E)$ for the inner part and lies between the two functions
for the outer part. The least ambiguous case is found for PSR 0329+54. The ACF
is well matched by $ACF(K)$. PSR B0823+26 has the most convex ACF even
slightly more so than $ACF(K)$. In Table 2 we list the rms values of the
observed ACFs to the closest analytical $ACF(y)$ functions and indicate which
functions they are. The residuals for all pulsars vary between rms values of
0.0015 for B0329+54 and 0.037 for B1749-28. They are all larger by a factor 2
to 9 than those for the scattering models, however they are also from a fit
over a larger lag range and do not have a free parameter for adjustment. In
general the ACFs of all 12 pulsars are approximately within the range of the
four analytical functions.

Does the ACF of any of these pulsars resemble a Gaussian? We fit a function of
the form, $f_{g}(\Delta f)=\exp(-|\Delta f|^{\delta}/b_{f})$ to the ACF of PSR
B1642-03 which is one of the two most likely candidates for its ACF to have a
Gaussian shape. We obtained for the free exponent and its statistical error,
$\delta=1.79\pm 0.03$, still significantly smaller than $\delta=2$ for a
Gaussian. However, in general it appears that for a few pulsars with a concave
shape of the inner ACFs there can be a tendency in the shape of the ACF toward
a Gaussian.

Listing the 12 pulsars according to the shape of their ACFs it is clear that
the most concave inner ACFs are related to the highest values of $\alpha$ and
the most convex to the lowest value. PSR B0823+26 has the most convex ACF,
even slightly more so than $ACF(F)$, and the lowest value of $\alpha$ of 3.56.
At $\alpha=\sim 3.75$ the ACFs transition from concave to convex.

###  3.6 The average frequency profile of the scintles

The average frequency profile of the scintles could be obtained analytically
by finding the function, $y$ in Table 4 if our observed ACFs were exactly
represented by $ACF(y)$. However, only very few can be approximately
associated with one particular $ACF(y)$, and then only for a limited range of
frequency lags. Most are between neighboring analytical functions or are
closer to one function in their inner parts and to another function at the
tail. Therefore it is best to use the comparison of the observed ACFs with the
analytical functions as a guide. The inferences as to the scintle profile can
be only approximate.

In Figure 8 we plot the functions, $y$, which are the undelying functions to
$ACF(y)$. The functions $G$ and $L$ are concave up to $\sim$HWHM with a bell-
shaped profile and the other two functions, $E$ and $K$ are convex with a
profile of a cusp either with a moderate or an extreme peak. In the latter
case, the function goes to infinity for $f\to 0$.

Comparing the $ACF(y)$ with the $y$ functions in Figures 7, 8 and Table 4, it
is clear that the observed ACFs matched best by $ACF(G)$ and $ACF(L)$
functions are associated with bell-shaped scintle profiles. The $ACF(E)$
function is almost identical to $ACF(L)$ for small lags and becomes clearly
different for lags $\Delta f>\Delta f_{1/2}$. However the function $y$ assumes
the shape of a cusp. Five of our pulsars have clear similarities to $ACF(E)$
but also to any of the other two functions, $G$ and $L$. The scintle profile
is probably described best as a hybrid between a bell-shaped curve and a cusp.

For one pulsar, B1929+10, the ACF is best described only by $ACF(E)$. None of
the observed ACFs is best described only by the bell-shaped functions, G and
L. For three pulsars, B0525+21, B1133+16, and B1749+28, there is a transition
from concave to convex ACF shapes. Their ACFs are hybrids with $ACF(K)$ as one
element. For five other pulsars their ACFs are clearly convex and best
described by only $ACF(K)$. The scintle profile is less ambiguous. Although
the function, $K$, goes to infinity for $f\to 0$, because of deviations of the
observed ACFs from $ACF(K)$, the scintle profile would probably be best
described as a pronounced cusp with a sharp peak.

What is the relation of the wavenumber power-law spectral index to the scintle
profile? It appears that for pulsars with high $\alpha$ values , that is steep
wavenumber spectra, the scintle could have a profile intermediate between a
cusp and a Gaussian or Lorentzian because of a tendency of the shape of their
ACFs toward these functions. Below the transition from a concave shape of the
inner ACFs to a convex shape at $\alpha\simeq 3.75$ the cusp becomes more
clearly defined with a sharp peak and its sides decrease even faster than
exponentially in the case of pulsars with the flattest wavenumber spectra.

Figure 8: The four functions, $y$ in Table 4 plotted as a function of
frequency, $f$. Three functions are normalized in amplitude and width to
facilitate comparison. The fourth function, $K$, is plotted with the
respective normalization parameter corresponding to the function, $E$. It goes
to infinity at $f\to 0$. As in Figure 7, all functions are two-sided, only one
side is plotted.

##  4 Discussion

The electron density variations in terms of the wavenumber power-law spectral
index, $\alpha$, can be estimated by several methods by, e.g., analyzing the
shapes of ACFs of dynamic spectra in frequency and time, the frequency
dependence of the decorrelation bandwidth, $\Delta f_{1/2}$, the scattering
time, $\tau_{sc}$, and the scattering angle, $\theta_{sc}$. We estimated the
index, $\alpha$, for 12 pulsars by analyzing the frequency section of the time
averaged ACF of dynamic spectra and comparing it to extended-medium and thin-
screen scattering models. This is the larges sample we are aware of that has
been used for such analysis and has given us important insight into
characteristics of the turbulent interstellar plasma and the associated
frequency profile of the scintles.

The inner part of the ACFs of all 12 pulsars down to at least 45% of the
maximum or $1.2\times\Delta f_{1/2}$ is for most pulsars well fit by the
predictions of extended-medium and thin-screen scattering theories. The power-
law spectral indices range from $\alpha=3.56$ to 3.97 with errors $\leq 0.05$
and a mean with standard deviation of 3.76$\pm$0.13 which is close to the
index of 3.67 for the Kolmogorov spectrum. The largest deviations from either
of the models were found for PSR B1642-03 with an error for $\alpha$ of 0.05.

All pulsars show deviations of their ACFs from either of the models at
frequency lags starting at $\sim 1.2$ to $3\times\Delta f_{1/2}$. These
deviations are larger than any found for the inner part of the ACFs. For some
of them the ACFs turn to a level above the models which could possibly be due
to technical reasons. However for most of them the ACFs turn to a level
clearly below the models which likely has physical reasons. A more compact ACF
is equivalent to a broader $\mathcal{FT}(ACF)$ (compare Figures 4 and 6).
Longer delays correspond to more compact inhomogeneities which may indicate
deficiencies in either of the two scattering models.

The most direct comparison of our measurements of $\alpha$ are those obtained
from $\mathcal{FT}[ACF(\Delta f)]$. While the estimates of Armstrong & Rickett
(1981) have rather large allowed ranges, the estimates of Wolszczan (1983) (no
errors given), are equal within our errors of 0.1. Although only three pulsars
could be compared, this result is remarkable, since the observations were made
some thirty years apart and therefore give hints as to the stability of the
power-law spectrum for the direction of these pulsars.

Of particular interest is a comparison of an estimate of $\alpha$ by Popov &
Smirnova (2021) who used for 10 of our pulsars the same dynamic spectra but
focused on the time section of the frequency averaged ACFs and analyzed the
structure function. While for five pulsars their values are only somewhat
larger or smaller within a difference of $\leq$ 0.10, for the other pulsars
their values are all smaller by as much as 0.51 in the case of B1237+25. Such
large discrepancies are remarkable given that the estimates were obtained on
the basis of the same spectra but separately for their frequency and their
time characteristics.

The estimates of $\alpha$ from the frequency dependence of $\Delta f_{1/2}$ as
given in our example in Table 3 were largely consistent with our values,
albeit within the authors’ (Cordes et al., 1985) larger errors.

Independent information on $\alpha$ comes from measurements of the exponential
broadening of pulses through multiple path propagation in the ionized ISM. The
parametrization is generally given by the broadening time scale or scattering
time, $\tau_{sc}$. This parameter in turn is related to $\alpha$. For a power-
law wavenumber spectrum, $\tau_{sc}\propto f^{-\beta}$ and for $\alpha<4$,
$\beta=2\alpha/(\alpha-2)$. That requires $\beta<4$. Löhmer et al. (2004)
found for seven pulsars $3.3\leq\alpha\leq 4.0$ and for PSR B1933+16 in our
list, $\beta=3.4\pm 0.2$, too small for $\alpha<4$. Bhat et al. (2004) found
for eight out of 15 pulsars $3.1\pm 0.7\leq\alpha\leq 4\pm 1$ but for the
others also values of $\beta$ too small for $\alpha<4$. An even larger portion
of pulsars outside the $\alpha$ range was found by Geyer et al. (2017) with
$1.5\leq\beta\leq 4.0$ for 13 pulsars and only two of them with $\alpha<4$
within the errors. The largest sample of pulsars with multifrequency
observations was obtained by Lewandowski et al. (2015). From 48 pulsars only
16 were given with values of $\alpha$ within their errors $<4$ including the
only one from our list, PSR B1749-28 with $\alpha=4.24\pm 0.18$, different
from our value of 3.78$\pm$0.02 by 3 times their larger uncertainty. All the
others including PSRs B1642-03 and B1933+16 from our list, had correspondingly
smaller values of $\beta$, with an average of 3.89.

The consistent result from pulse broadening observations is that a large
fraction of pulsars have $\beta$ values too small to be consistent with
$\alpha<4$. Several effects could be considered that would lead to an increase
or changes of $\beta$. There is an indication that pulsars with $DM<500$ pc
cm-3 have with $\beta=3.95$ a larger average than those with $DM>500$ pc cm-3
with $\beta=3.49$ (Lewandowski et al., 2015). All of our pulsars have in
comparison very small dispersion measures (see, Table 2). Geyer et al. (2017)
find that their small values of $\beta$ may be indications of anisotropic
scattering since that assumption would lead to an increase of the values.
Anisotropic scattering mechanisms have also been considered by Stinebring et
al. (2001); Tuntsov et al. (2013). Other effects discussed include finite or
truncated scattering screen (Cordes & Lazio, 2001) and internal cutoff scale
effects (Rickett et al., 2009). Also, analysis of the structure function of
several pulsars, based on multifrequency observations, showed that the
spectrum of interstellar plasma in the direction of some pulsars follows a
piece-wise power law (Shishov et al., 2003b; Smirnova et al., 2006). Our
measurements of 12 pulsars with a different analysis scheme adds to the
discussion of the wavenumber spectrum and the electron density variations of
the plasma turbulence of the ISM. In general it appears that more complex
theoretical models are needed to describe the observed data or possibly that
there are deviations from the wavenumber power-law spectrum.

In our search for the average frequency profile of the scintles in terms of
analytical functions independent of scattering models we were guided by our
earlier results from VLBI observations of pulsars where we found that the
delay section of the visibility function of some pulsars could be well fit by
Lorentzians. For half of our pulsars we indeed found that the ACF’s could be
best fit by the corresponding $ACF(E)$ function or by a hybrid of functions
with $ACF(E)$ being part of it. However, for five others the more sharply
pointed two-sided exponential, $ACF(K)$ was warranted in addition to three
more where that function was part of a hybrid. It is interesting to note that
while the set of power-law spectral indices from $\alpha=3.97$ to 3.56 appears
to be a uniform distribution, the functions describing the ACFs and the shape
of the scintles are at least formally quite different. For the steep spectra
with $\alpha\gtrsim 3.75$, the inner part of the ACFs become increasingly
concave while for the flatter spectra with $\alpha\lesssim 3.75$ they become
convex. In other words, the value of $\alpha$ determines the average frequency
profile of the scintles. Steep wavenumber spectra with $\alpha<4$ correspond
to scintles with a somewhat rounded cusp. With smaller $\alpha$ values the
peak of the cusp becomes more pronounced. For $\alpha\lesssim 3.75$ and
further flattening, the cusp and its peak sharpen further and decay faster
than an exponential, approaching at least nominally the modified Bessel
function of the second kind of order zero.

Is there any correlation between the shape of the scintles and any of the
pulsar characteristics listed in Table 1 such as the dispersion measure, the
distance to the pulsar and the galactic coordinates? We searched for such a
correlation but no correlation is apparent.

##  5 Conclusions

  1. 1.

We analyzed the dynamic spectra of nine pulsars at a center frequency of 324
MHz and three pulsars at 1676 MHz and computed the frequency sections of the
two dimensional autocorrelation functions.

  2. 2.

For each pulsar the inner part of the function down to at least 45% of the
maximum is well fit by the prediction of a thin-screen or extended-medium
scattering model. The power law wavenumber spectral indices of the
interstellar plasma turbulence, $\alpha$, are all within a range of 3.56 and
3.97 with uncertainties $\leq 0.05$.

  3. 3.

The mean of the spectral indices with standard deviation is
$\langle\alpha\rangle=3.76\pm 0.13$ which is close to the Kolmogorov index of
3.67.

  4. 4.

The Fourier transforms of the model functions fit to those of the ACFs for the
full width gives similar values for alpha, although with larger uncertainties.

  5. 5.

Beyond the inner part of the function, clear misfits can be seen for all of
our pulsars that are larger than any possible deviations seen in the inner
part of the ACFs, indicating scattering characteristics more complex than
described in the models, or indicating that there are deviations from the
power law of the interstellar plasma turbulence.

  6. 6.

Comparison of extended-medium and thin-screen models with observed ACFs and
the respective $\mathcal{FT}(ACF)$s gives no clear evidence that either of the
models is preferred.

  7. 7.

The observed ACFs have a concave inner part down to about half of the maximum
for high $\alpha$ values that becomes less concave with flattening spectra and
turns convex for $\alpha\lesssim 3.75$.

  8. 8.

For six pulsars, with $3.75\lesssim\alpha<4.0$, the function $ACF(E)$ alone or
as a member of a hybrid fits the observed ACFs moderately well down to 20% of
the maximum. This function is expected for pulsars for which the Lorentzian
provides a fairly good fit to the visibility function from VLBI.

  9. 9.

For the pulsars, with $3.56\leq\alpha\lesssim 3.75$ a function, like the two
sided exponential, $ACF(K)$, is more warranted for the fit to the ACFs.

  10. 10.

A Gaussian was not an appropriate fit for any of the ACFs of the 12 pulsars.
The pulsars who came closest are B1237+25 and B1642-03 with ACFs described
best by a hybrid between $ACF(G)$ and $ACF(E)$.

  11. 11.

From the functional fit to the observed ACFs we found that the average
frequency profile of a scintle is for steep wavenumber spectra characterized
by a hybrid between a cusp and a function like a Gaussian or Lorentzian. With
increasing flattening of the spectrum with $\alpha\lesssim 3.75$, the cusp and
its peak becomes more pronounced and decays faster than an exponential.

We thank an unknown referee for valuable comments and suggestions. The
Radioastron project is led by the Astro Space Center of the Lebedev Physical
Institute of the Russian Academy of Sciences and the Lavochkin Scientific and
Production Association under a contract with the Russian Federal Space Agency,
in collaboration with partner organizations in Russia and other countries.
This paper was supported in part by the Russian Academy of Science Program
KP19-270 “The study of the Universe origin and evolution using the methods of
earth-based observations and space research”. NB was supported by the National
Sciences and Engineering Research Council of Canada. .

## References

  * Armstrong & Rickett (1981) Armstrong, J. W., & Rickett, B. J. 1981, MNRAS, 194, 623, doi: 10.1093/mnras/194.3.623
  * Backer (1975) Backer, D. C. 1975, A&A, 43, 395
  * Bhat et al. (2004) Bhat, N. D. R., Cordes, J. M., Camilo, F., Nice, D. J., & Lorimer, D. R. 2004, ApJ, 605, 759, doi: 10.1086/382680
  * Chashei & Shishov (1976) Chashei, I. V., & Shishov, V. I. 1976, Soviet Astr., 20, 13
  * Cordes (1986) Cordes, J. M. 1986, ApJ, 311, 183, doi: 10.1086/164764
  * Cordes & Lazio (2001) Cordes, J. M., & Lazio, T. J. W. 2001, ApJ, 549, 997, doi: 10.1086/319442
  * Cordes et al. (1985) Cordes, J. M., Weisberg, J. M., & Boriakoff, V. 1985, ApJ, 288, 221, doi: 10.1086/162784
  * Fadeev et al. (2018) Fadeev, E. N., Andrianov, A. S., Burgin, M. S., et al. 2018, MNRAS, 480, 4199, doi: 10.1093/mnras/sty2055
  * Feller (1971) Feller, W. 1971, An Introduction to Probability Theory and its Applications, 2nd edn., Vol. 2 (Wiley)
  * Geyer et al. (2017) Geyer, M., Karastergiou, A., Kondratiev, V. I., et al. 2017, MNRAS, 470, 2659, doi: 10.1093/mnras/stx1151
  * Goodman & Narayan (1989) Goodman, J., & Narayan, R. 1989, MNRAS, 229, 73
  * Gwinn et al. (2016) Gwinn, C. R., Popov, M. V., Bartel, N., et al. 2016, ApJ, 822, 96, doi: 10.3847/0004-637X/822/2/96
  * Kardashev et al. (2017) Kardashev, N. S., Alakoz, A. V., Andrianov, A. S., et al. 2017, Solar System Research, 51, 535, doi: 10.1134/S0038094617070085
  * Lee & Jokipii (1975a) Lee, L. C., & Jokipii, J. R. 1975a, ApJ, 196, 695, doi: 10.1086/153458
  * Lee & Jokipii (1975b) —. 1975b, ApJ, 201, 532, doi: 10.1086/153916
  * Lee & Jokipii (1975c) —. 1975c, ApJ, 202, 439, doi: 10.1086/153994
  * Lerche (1979) Lerche, I. 1979, MNRAS, 189, 137
  * Lewandowski et al. (2015) Lewandowski, W., Kowalińska, M., & Kijak, J. 2015, MNRAS, 449, 1570, doi: 10.1093/mnras/stv385
  * Löhmer et al. (2004) Löhmer, O., Mitra, D., Gupta, Y., Kramer, M., & Ahuja, A. 2004, A&A, 425, 569, doi: 10.1051/0004-6361:20035881
  * Lovelace (1970) Lovelace, R. V. E. 1970, Ph.D. Thesis (Cornell University)
  * Narayan (1992) Narayan, R. 1992, The Royal Soc. Publ., 341, 151
  * Popov et al. (2020) Popov, M. V., Bartel, N., Burgin, M. S., et al. 2020, ApJ, 888, 57, doi: 10.3847/1538-4357/ab5db6
  * Popov et al. (2021) Popov, M. V., Bartel, N., Burgin, M. S., Smirnova, T. V., & Soglasnov, V. A. 2021, MNRAS, 506, 4101, doi: 10.1093/mnras/stab1921
  * Popov & Smirnova (2021) Popov, M. V., & Smirnova, T. V. 2021, Astronomy Reports, 65, 1129, doi: 10.1134/S1063772921120052
  * Popov et al. (2016) Popov, M. V., Andrianov, A. S., Bartel, N., et al. 2016, Astronomy Reports, 60, 792, doi: 10.1134/S1063772916090067
  * Popov et al. (2017) Popov, M. V., Bartel, N., Gwinn, C. R., et al. 2017, MNRAS, 465, 978, doi: 10.1093/mnras/stw2353
  * Rickett et al. (2009) Rickett, B., Johnston, S., Tomlinson, T., & Reynolds, J. 2009, MNRAS, 395, 1391, doi: 10.1111/j.1365-2966.2009.14471.x
  * Rickett (1977) Rickett, B. J. 1977, Ann. Rev. Astron. Astrophys., 15, 479
  * Romani et al. (1986) Romani, R. W., Narayan, R., & Blandford, R. 1986, MNRAS, 220, 19, doi: 10.1093/mnras/220.1.19
  * Shishov et al. (2003a) Shishov, V. I., Smirnova, T. V., Sieber, W., et al. 2003a, A&A, 404, 557, doi: 10.1051/0004-6361:20030480
  * Shishov et al. (2003b) —. 2003b, A&A, 404, 557, doi: 10.1051/0004-6361:20030480
  * Smirnova et al. (2006) Smirnova, T. V., Shishov, V. I., Sieber, W., et al. 2006, A&A, 455, 195, doi: 10.1051/0004-6361:20041368
  * Stinebring et al. (2001) Stinebring, D. R., McLaughlin, M. A., Cordes, J. M., et al. 2001, ApJ, 549, L97, doi: 10.1086/319133
  * Tuntsov et al. (2013) Tuntsov, A. V., Bignall, H. E., & Walker, M. A. 2013, MNRAS, 429, 2562, doi: 10.1093/mnras/sts527
  * Wolszczan (1983) Wolszczan, A. 1983, MNRAS, 204, 591, doi: 10.1093/mnras/204.2.591


# Sentiment-Aware Automatic Speech Recognition pre-training for enhanced
Speech Emotion Recognition

###### Abstract

We propose a novel multi-task pre-training method for Speech Emotion
Recognition (SER). We pre-train SER model simultaneously on Automatic Speech
Recognition (ASR) and sentiment classification tasks to make the acoustic ASR
model more “emotion aware”. We generate targets for the sentiment
classification using text-to-sentiment model trained on publicly available
data. Finally, we fine-tune the acoustic ASR on emotion annotated speech data.
We evaluated the proposed approach on MSP-Podcast dataset, where we achieved
the best reported concordance correlation coefficient (CCC) of 0.41 for
valence prediction.

Index Terms—  Speech emotion recognition, automatic speech recognition,
sentiment analysis, pre-training

##  1 Introduction

The origins of Speech Emotion Recognition (SER) date back to Blanton’s work
[1] when he wrote that “the language of the tones is the oldest and most
universal of all our means of communication.”, and culminated in studies on
word-free voice samples with the goal of isolating qualitative features of the
voice from those related to the articulated sounds patterns [2]. Emotion
researchers define emotion either as a discrete construct or using emotion
dimensions. The discrete construction theory credited to Ekman[3] isolates six
basic emotion categories: anger, disgust, fear, happiness, sadness, surprise,
but more comprehensive emotion category sets have been proposed [4]. The
dimensional perspective, on the other hand, defines emotion as a point in a
space defined by emotion dimensions, most commonly Activation, Valence, and
Dominance (AVD).

A century later, the challenge of SER remains but new computational tools have
emerged permitting more complex modelling. The motivations have also changed,
it is no longer destined to understanding the emotion from a psychological
perspective alone. It is further fueled by the ubiquitous speech-based
interactions. Moreover, building smart systems capable of detecting the user
emotional state has the potential of enhancing the interactive experience with
different devices.

(Deep) neural network-based models have been a popular choice for SER in
recent years [5, 6, 7]. However, training a Neural Network for SER requires a
large training corpus, and non-neutral emotions are rare in speech. Recent
work has addressed this limitation by pre-training on speech tasks, such as
ASR [5, 8] or Speaker Identification [9, 10]. In this work, we propose a novel
pre-training approach for SER. The proposed pre-training consists of building
a “Sentiment-aware ASR” (SA2SR). The training of SA2SR objective is a
combination of ASR and text-based sentiment classification, where the text
sentiment labels are generated from a trained text sentiment model. This
approach allows us to amass a large amount of text sentiment labels for speech
data, and the effectiveness of these labels on improving SER is validated in
our experiments.

##  2 Related work

A suitable choice of input representation is crucial for Speech Emotion
Recognition (SER). Traditional approaches used classical signal processing
methods (pitch, filter banks) or statistical measures (mean, variance,
quantiles…) of the acoustic signal to train on classification/regression
(Ekman categories vs AVD). Recent work on SER attempted to infer the emotion
based on acoustic and textual cues, either simultaneously or separately.

In End-to-End SER with ASR [11], an acoustic-to-word ASR model is first
trained then fine-tuned on a multi-task learning to jointly optimize ASR and
SER objective. The emotion prediction block has two input fields: acoustic
features similar to those used in ASR and the states of the ASR decoder. The
authors also show that using this combination (raw acoustic inputs & ASR
decoder features) outperforms an SER based on any single element of this
combination.

Combining ASR and SER also outperforms a variant in which the SER block model
takes the transcript of the ASR (word embeddings) instead of the decoder
states. This result is expected since using the ASR transcript propagates the
transcription inaccuracies to the SER block. The same reasoning can applied to
text-based SER to point out their limitations, such as the one used in
Sentiment Analysis based on speaker data [12]. In this speech-based Sentiment
Analysis a pre-trained ASR is used for transcription of the utterance. The
text is then fed to a feature extractor in parallel to a Speaker
Identification feature block to provide an input to the emotion predictor.

A different approach that decouples text and acoustic features was introduced
in Multi-modal SER [13], where the SER model encodes information from audio
and text using two encoders. For the text encoder, the text input is the
transcription from an ASR. This approach leverages the ability of text
encoders to capture long term semantics better than the acoustic ones.
However, this multi-modal approach assumes that the transcript is provided
(via ASR), which limits its applicability when only the utterance (audio) is
accessible.

To the extent of our knowledge, the only previous work that leveraged text-
based sentiment labels was published recently[14] with three major
differences: 1) we start with analyzing the correlation between text sentiment
and speech emotion, thereby establishing a strong motivation for the proposed
method, and an explanation for the observed performance boost, 2) the focus of
this work is on SER with the widely used dimensional emotion representation
(activation, valence and dominance), while that of [14] is on three-way speech
sentiment classification, and 3) The approach in [14] uses out of the box
encodings (ASR followed by Bert) and is oblivious to the feedback from
sentiment labels. Our approach, on the other hand, leverages the proxy
sentiment labels to induce the ASR embedding to incorporate emotional
knowledge and yields better performance.

##  3 Proposed method

We propose building a SER model that is pre-trained on a sentiment-aware ASR
task. The sentiment-awareness is implemented by transferring sentiment
knowledge from a text domain to the acoustic domain.

###  3.1 Correlation between text sentiment and speech emotion

We conjecture that text sentiment correlates with the valence dimension of
speech emotion. Indeed, when a human listener tries to determine the emotion
from a speech segment, the words in the speech also plays a role – the obvious
examples are the speech segments that contain cursing words and phrases, or
strong praising adjectives (e.g. excellent, beautiful, etc).

We test this correspondence between text-based sentiment and valence on the
IEMOCAP dataset [15] and use a pre-trained text sentiment analysis (Roberta
[16]) to get the sentiment labels:  negative, positive, neutral. Table 1 shows
the confusion matrix between text sentiment classes and speech emotion, with
the dominant speech emotion highlighted in red in each sentiment class and the
second dominant ones highlighted in blue.

As can be seen from Table 1, the negative text-sentiment utterances are mostly
associated with negative speech emotion labels (Sad, Anger and Frustrated);
while the positive text-sentiment utterances correspond to positive speech
emotion labels (Happy). More interestingly, by investigating elements of the
cell $\\{Neutral,frustrated\\}$ we can find transcripts such as : “Nothing”,
“A vacation.”, “I’m just saying.”$\\}$, while the inferred sentiment is
neutral – this means the emotion in this case is likely conveyed through
speech style and tone. Furthermore, by grouping $\\{Sad,Frustrated,Anger\\}$
into one class, we get a Spearman correlation of $0.22$ between text sentiment
and speech emotion. These observations motivate us to employ readily available
text sentiment model to generate sentiment labels, which will serve as weak
signal to train speech emotion models.

| Text sentiment
---|---
| Negative | Neutral | Positive
Speech emotion | Sad | 339 | 604 | 137
Anger | 490 | 518 | 94
Frustrated | 658 | 1049 | 141
Neutral | 253 | 1251 | 204
Happy | 252 | 848 | 533

Table 1: Confusion matrix between text-based sentiment [16] and speech emotion
(IEMOCAP)

###  3.2 Sentiment-aware Automatic Speech Recognition (SA2SR)

In addition to the ASR model, a sentiment classifier (proxy classifier) is
trained jointly on the acoustic encoder states. The architecture logic is
similar to the one in combined ASR-SER[11]. The model takes as input the log
filterbank energy (LFBE) features and contains two classifiers (Figure 1) that
take the encoded acoustic sequence as input:

  * •

Sequence to sequence classifier: A softmax layer for token classification

  * •

Sentiment classifier: A sequence summarizer (recurrent neural network)
followed by a softmax layer over the sentiment classes (negative, neutral,
positive).

The proposed architecture is trained using a loss that is a linear combination
(Equation 1) of (a) Connectionist Temporal Classification (CTC) loss [17]
between the target sequence and the sequence of output probabilities over the
token set, and (b) cross entropy loss between the predicted and proxy
sentiment targets, i.e., sentiment obtained from the pre-trained text-to-
sentiment model. The global loss is defined as:

$\displaystyle L_{global}=L_{ASR}+\lambda L_{sentiment}$ (1)

where $\lambda\geq 0$ is a hyper-parameter reflecting how the importance of
sentiment classification vis-à-vis the ASR task.

LFBE FeaturesAcoustic Encoder (ASR)Sequence EncodingSoftmax layerProbabilities
over tokensTarget tokensCTC LossLSTM LayerSoftmax LayerPredicted proxy
labelProxy ClassifierTrue proxy labelCross-entropy lossGlobal loss Fig. 1:
Architecture of SA2SR pre-training network

###  3.3 Fine-tuning

During fine-tuning, we use the pre-trained acoustic encoder (Figure 2) and add
an emotion regression transformer block that takes the encoding sequence as
input and outputs the Activation, Valence, Dominance (AVD) values. The model
is then trained to maximize the Concordance Correlation Coefficient (CCC)
between prediction $\hat{y}$ and target values $y$.

$\displaystyle
CCC(y,\hat{y})=\frac{2Cov(y,\hat{y})}{\sigma^{2}_{y}+\sigma^{2}_{\hat{y}}+(\mu_{y}-\mu_{\hat{y}})^{2}}$
(2)

where $Cov$ denotes the co-variance, $\sigma$ and $\mu$ denote the sample
variance and mean, respectively. During model training, these statistics are
computed on mini-batches of data.

To enable easy comparison to previous works, the CCC objectives for each of
the activation, valence and dominance are simply combined as:

$\displaystyle\mathcal{L_{CCC}}=-\frac{1}{3}(CCC_{A}+CCC_{V}+CCC_{D})$ (3)

LFBE FeaturesAcoustic Encoder (ASR)Sequence EncodingPre-trained1-D Convolution
BlockMulti-head attention$\sigma$ and $\mu$ poolingLinear ProjectionPredicted
AVDTrue AVDCCC lossEmotion Regressor Fig. 2: Architecture of the fine-tuned
SA2SR

##  4 Experiment results

Datasets: We use the full Librispeech dataset [18] for pretraining a character
based SA2SR. The dataset contains audios and transcripts for around 960 hours
of audio, and we generate the proxy sentiment labels using text-to-sentiment
RoBerta model trained on Twitter sentiment data [19]. We extract LFBE features
with 40 frequencies using $25ms$ window and $10ms$ steps. Hence, for SA2SR
pre-training, an input tuple consists of (LFBE Features, transcript, proxy
label).

Model fine-tuning and evaluation are conducted on MSP-Podcast [20] dataset.
The MSP-Podcast contains 60K speech segments from podcast recordings which are
perceptually annotated using crowd-sourcing. The ground truth emotion
dimension scores are obtained averaging scores selected by individual
annotators on seven-point Likert scale. The dataset was split following the
same partitions provided in the official MSP corpus.

###  4.1 Networks Architecture

The Acoustic Encoder consists of 5 Bidirectional LSTM (Bi-LSTM) layers. All
LSTM layers have 192 units with $tanh$ activation and $sigmoid$ recurrent
activation.

The 1-D convolutional block in Emotion Regressor (Figure 2) is a stack of 2
one-dimensional masked convolutions with filter sizes (6,3) and strides (3,2)
respectively. The convolutions accept input masks and process them accordingly
to match the output of the convolution. The convolutions are followed by
sample level normalization and a $\text{LeakyRELU}_{\alpha=0,3}$ is applied to
the convolutional block output.

The multi-head attention uses 4 attention heads and 64 dimensions for all
encoding spaces and feed-forward outputs. The output of the attention
mechanism is pooled by computing the variance and the mean of the sequences
yielding 128 features that are linearly projected into the AVD space.

###  4.2 Pre-training

For the global loss weight, we chose $\lambda=200$. To optimize it, we use
Adam optimizer with the parameters: $lr=5\times 10^{-5}$,
$\beta_{1}=0.9,\beta_{2}=0.999$. The $\lambda$ was chosen such that the ASR
and sentiment classification losses achieve similar value on the validation
set.

The LFBE features are normalized on a sample level. The training set is then
augmented using speed augmentation of factors $0.9$ and $1.1$. This
augmentation step triples the training set size. The features are then masked
on time and frequency dimensions with probability $p=0.5$, as in the
SpecAugment method [21]. We finally stack every adjacent three frames and skip
every other two time steps, which leads to shorter sequences and improved
model training time. The ASR training uses a token set of 29 characters.

After each epoch, we use the validation set to compute the Character Error
Rate (CER) from the ASR task and Area Under the Curve (AUC) of the sentiment
classifier. The pre-training terminates when the metric $M=CER-AUC$ does not
improve for 25 epochs and the model that has the lowest $M$ is evaluated. For
the baseline model, we pre-train on the ASR task only and use the model from
the epoch with the best CER on the validation set.

####  4.2.1 Effectiveness of ASR features for SER

To start, we test the effectiveness of ASR trained features for SER. Some
previous work, for example [5, 8], reported limited transferability between
ASR and SER. In our experiments, however, we found ASR features to be quite
effective. It is possible that the differences are due to the large pre-
training dataset (full Librispeech) and modern model architectures such as
bidirectional LSTM and transformers.

Model | Activation | Valence | Dominance
---|---|---|---
No Pre-training | 0.603 | 0.304 | 0.511
ASR features | 0.649 | 0.393 | 0.544
Table 2: The baseline ASR pre-training achieves much improved CCC compared to
no pre-training

We report these results in Table 2. The “ASR features” are pre-trained on
Librispeech and fine-tuned on MSP-Podcast for SER. The “No pre-training”
baseline is directly trained with the MSP-Podcast. We observe large
performance boost when ASR pre-training is employed.

####  4.2.2 Effectiveness of SA2SR pre-training

In this experiment, we examine whether the additional proxy sentiment labels
enhance SER performance. In particular, we expect that valence recognition to
be improved, as we have seen correlation between text sentiment and valence in
Section 3.1. The results are reported in Table 3. As a comparison, we also
included the results from [7] that uses self-supervised pre-training as a
strong contender.

Model | Activation | Valence | Dominance
---|---|---|---
ASR features | 0.649 | 0.393 | 0.544
CPC-based pre-traing [7] | 0.706 | 0.377 | 0.639
SA2SR features | 0.679 | 0.412 | 0.564

Table 3: SA2SR produces good features for enhancing the CCC metric

We can see from Table 3 that recognition of valence, arguably the most
important dimension of emotion, is further improved compared to the strong
“ASR features” baseline. However, in this equal-weight multi-task emotion
training setting, we see that activation and dominance dimension performs
relatively weak compare to that of [7]. We view this as an encouraging result
as in many applications, valence (positive v.s. negative) is of most interest.

Additionally, during the pre-training of the SA2SR model, we observed that the
sentiment classification part achieves weighted-average-recall of 0.71 and
0.81 AUC. This indicates that indeed the model is trained to recognize these
proxy sentiment labels. Therefore, we expect the learned representation to be
suitable for the final SER task.

####  4.2.3 Importance of fine-tuning

Lastly, we are interested in examining whether we should freeze the learned
representations during SER training. For both ASR and SA2SR features, we train
models with and without freezing the acoustic encoder. The results are
reported in Table 4.

Pretraining | Activation | Valence | Dominance
---|---|---|---
ASR features (frozen) | 0.503 | 0.378 | 0.438
ASR features | 0.649 | 0.393 | 0.544
SA2SR features (frozen) | 0.508 | 0.403 | 0.483
SA2SR features | 0.679 | 0.412 | 0.564

Table 4: Impact of finetuning the pre-trained encoders on CCC

As can be seen from Table 4, fine-tuning the encoder gives better SER
performance compared to the methods with frozen encoders. This result proves
that using out of the box ASR models for transcription without any fine-tuning
would be outperformed when the gradient propagates to the ASR network weights.
It also proves the point we made earlier about the importance of sentiment
awareness in the SA2SR. Even without fine-tuning, SA2SR clearly outperforms
the ASR embedding on 2 out of the 3 emotion AVD dimensions.

##  5 Conclusion

We proposed a novel pre-training method that utilizes proxy sentiment labels
to aid ASR pre-training for SER. As text-sentiment and speech-emotion are
correlated, this way we train speech representations capturing both phonetic
and emotion-relevant info. We evaluated the proposed method on the MSP-Podcast
dataset achieving state of the art performance on the challenging valence
dimension.

Albeit we focused on ASR-based pre-training, the proxy sentiment
classification task can be combined with other pre-training techniques, such
as APC [22], CPC [23], which we will address in the future work.

## References

  * [1] Smiley Blanton,  “The voice and the emotions,”  Quarterly Journal of Speech, vol. 1, no. 2, pp. 154–172, 1915.
  * [2] William F. Soskin and Paul E. Kauffman,  “Judgment of Emotion in Word-Free Voice Samples,”  Journal of Communication, vol. 11, no. 2, pp. 73–80, 02 2006.
  * [3] Paul Ekman,  “An argument for basic emotions,”  Cognition and Emotion, vol. 6, no. 3-4, pp. 169–200, 1992.
  * [4] Alan S. Cowen and Dacher Keltner,  “Self-report captures 27 distinct categories of emotion bridged by continuous gradients,”  PNAS, published online September 5, 2017.
  * [5] Haytham M Fayek, Margaret Lech, and Lawrence Cavedon,  “On the correlation and transferability of features between automatic speech recognition and speech emotion recognition.,”  in Interspeech, 2016, pp. 3618–3622.
  * [6] Vasudha Kowtha, Vikramjit Mitra, Chris Bartels, Erik Marchi, Sue Booker, William Caruso, Sachin Kajarekar, and Devang Naik,  “Detecting emotion primitives from speech and their use in discerning categorical emotions,”  in ICASSP 2020. IEEE, 2020, pp. 7164–7168.
  * [7] Mao Li, Bo Yang, Joshua Levy, Andreas Stolcke, Viktor Rozgic, Spyros Matsoukas, Constantinos Papayiannis, Daniel Bone, and Chao Wang,  “Contrastive unsupervised learning for speech emotion recognition,”  in ICASSP 2021. IEEE, 2021, pp. 6329–6333.
  * [8] Egor Lakomkin, Cornelius Weber, Sven Magg, and Stefan Wermter,  “Reusing neural speech representations for auditory emotion recognition,”  arXiv preprint arXiv:1803.11508, 2018.
  * [9] Michelle Bancroft, Reza Lotfian, John Hansen, and Carlos Busso,  “Exploring the intersection between speaker verification and emotion recognition,”  in ACIIW 2019. IEEE, 2019, pp. 337–342.
  * [10] Raghavendra Pappagari, Tianzi Wang, Jesus Villalba, Nanxin Chen, and Najim Dehak,  “X-vectors meet emotions: A study on dependencies between emotion and speaker recognition,”  in ICASSP 2020. IEEE, 2020, pp. 7169–7173.
  * [11] Han Feng, Sei Ueno, and Tatsuya Kawahara,  “End-to-End Speech Emotion Recognition Combined with Acoustic-to-Word ASR Model,”  in Proc. Interspeech 2020, 2020, pp. 501–505.
  * [12] S Maghilnan and M Rajesh Kumar,  “Sentiment analysis on speaker specific speech data,”  in 2017 International Conference on Intelligent Computing and Control (I2C2), 2017, pp. 1–5.
  * [13] Seunghyun Yoon, Seokhyun Byun, and Kyomin Jung,  “Multimodal speech emotion recognition using audio and text,”  CoRR, vol. abs/1810.04635, 2018.
  * [14] Suwon Shon, Pablo Brusco, Jing Pan, Kyu J. Han, and Shinji Watanabe,  “Leveraging pre-trained language model for speech sentiment analysis,”  CoRR, vol. abs/2106.06598, 2021.
  * [15] Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan,  “Iemocap: Interactive emotional dyadic motion capture database,”  Language Resources and Evaluation, vol. 42, no. 4, pp. 335–359, 2008.
  * [16] Francesco Barbieri, José Camacho-Collados, Leonardo Neves, and Luis Espinosa Anke,  “Tweeteval: Unified benchmark and comparative evaluation for tweet classification,”  CoRR, vol. abs/2010.12421, 2020.
  * [17] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber,  “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,”  in Proceedings of the 23rd International Conference on Machine Learning, 2006, pp. 369–376.
  * [18] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur,  “Librispeech: An ASR corpus based on public domain audio books,”  ICASSP 2015, pp. 5206–5210, 2015.
  * [19] Mark Heitmann, Christian Siebert, Jochen Hartmann, and Christina Schamp,  “More than a feeling: Benchmarks for sentiment analysis accuracy,”  Communication & Computational Methods eJournal, 2020.
  * [20] R. Lotfian and C. Busso,  “Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings,”  IEEE Transactions on Affective Computing, vol. 10, no. 4, pp. 471–483, October-December 2019.
  * [21] Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le,  “Specaugment: A simple data augmentation method for automatic speech recognition,”  Interspeech 2019, Sep 2019.
  * [22] Yu-An Chung and James Glass,  “Generative pre-training for speech with autoregressive predictive coding,”  in ICASSP 2020. IEEE, 2020, pp. 3497–3501.
  * [23] Aaron van den Oord, Yazhe Li, and Oriol Vinyals,  “Representation learning with contrastive predictive coding,”  arXiv preprint arXiv:1807.03748, 2018.