# Improving Noise Robustness for Spoken Content Retrieval using semi-
supervised ASR and N-best transcripts for BERT-based
ranking models

###### Abstract

BERT-based re-ranking and dense retrieval (DR) systems have been shown to
improve search effectiveness for spoken content retrieval (SCR). However, both
methods can still show a reduction in effectiveness when using ASR transcripts
in comparison to accurate manual transcripts. We find that a known-item search
task on the How2 dataset of spoken instruction videos shows a reduction in
mean reciprocal rank (MRR) scores of 10-14%. As a potential method to reduce
this disparity, we investigate the use of semi-supervised ASR transcripts and
N-best ASR transcripts to mitigate ASR errors for spoken search using BERT-
based ranking. Semi-supervised ASR transcripts brought 2-5.5% MRR improvements
over standard ASR transcripts and our N-best early fusion methods for BERT DR
systems improved MRR by 3-4%. Combining semi-supervised transcripts with
N-best early fusion for BERT DR reduced the MRR gap in search effectiveness
between manual and ASR transcripts by more than 50% from 14.32% to 6.58%.

Index Terms—  spoken content retrieval, BERT re-ranking, BERT dense retrieval,
N-best BERT-based retrieval

##  1 Introduction

The growing archives of spoken multimedia content, such as those arising in
video streaming services and podcasts, are increasing demands for effective
spoken content retrieval (SCR) systems. Unlike textual documents, spoken
documents are not directly searchable based on their content. For this reason,
spoken multimedia content is often searched only based on user-created
summaries and metadata. The effectiveness of this approach relies on
availability of meaningful summaries and metadata for the documents. Further,
since many spoken content items are long, SCR systems should ideally provide
users with more sophisticated, finer granularity search functionalities based
on the content itself. Such functionalities can only be provided if
transcripts of the spoken contents of the documents are available.

The high cost of creating manual transcripts and the large amounts of content
mean that transcripts are generally made using automatic speech recognition
(ASR). However, transcription errors from ASR can impact on search
effectiveness. While information retrieval (IR) has been shown to generally be
robust to moderate word error rates (WERs), Larson and Jones report that WERs
greater than about 30% can significantly impact on search reliability [1]. The
accuracy of ASR systems continues to improve, with very low WERs reported for
some tasks [2, 3]. However, WERs can still though be high when there is domain
mismatch between the training of the ASR system and the data to be transcribed
[4], speech is informal or acoustic noise is present in audio data [2]. In
many settings, there is no control of domains or audio quality of the data,
e.g., user-generated data of video streaming services or Podcasts, unlike
audio corpora used in laboratory settings. Thus, even a state-of-the-art ASR
system can still suffer high transcription WERs. This motivates us to explore
development of methods to improve noise robustness of SCR systems.

Transformer-based IR models have recently emerged as a highly effective
alternative to traditional models such as BM25 [5]. These new models can
largely be classified into two types: re-ranking and dense retrieval (DR). The
BERT-based re-ranking model, namely MonoBERT [6] directly takes a query-
document pair to compute its relevance score. The drawback of MonoBERT and its
successors is that transformer-based inference is very slow and hardware
requirements can make them impractical for use in practice [7]. In DR, the
transformer-based model is used to encode documents into dense document
vectors separately from queries. At search time, only the query needs to be
encoded into vector representations. Then, any vector-based similarity
measure, such as nearest neighbour (NN), can be used to compute query-document
relevance scores. This approach is much faster to run than re-ranking [7].

The contributions of this paper are as follows. (i) We investigate BERT-based
re-ranking and DR for a known-item search task of spoken instruction videos.
We observe that, despite the use of BERT-based retrieval systems, the gap in
search effectiveness between manual and ASR transcripts was around 10-14%.
(ii) We augment an ASR system using semi-supervised training for domain
adaptation and examine the impact of improved ASR transcripts on the BERT-
based ranking systems. The use of semi-supervised ASR transcripts brought
2-5.5% improvement of the MRR score. (iii) We propose the use of N-best ASR
transcripts for BERT DR systems and observe that our proposed early fusion
approach brought a gain in the MRR score by around 3-4%. Combining these two
methods gives an average improvement in MRR of 6.5%.

The reminder of the paper is organised as follows. The next section gives a
brief review of BERT-based IR and SCR, followed by details of the models used
in this investigation. We then propose our early and late fusion extensions
using ASR N-best transcripts for a BERT DR system. Experimental results and
analysis of the BERT-based ranking models are then presented, followed by
conclusions and future work.

##  2 Relevance to Prior Work

Mainstream studies of spoken document retrieval (SDR) began with the Text
REtrieval Conference (TREC) [8]. Studies have continued with exploration of
SCR in activities such as those at MediaEval [9] and the Podcast retrieval
track at TREC 2020 and 2021 [10, 11]. Another line of recent studies on SCR is
described in [12, 13] , which investigate the use of BERT re-ranking on
retrieval of Mandarin news stories.

A number of retrieval models have been introduced over the years, among the
most consistently effective is the BM25 probabilistic model [5], which has
been applied successfully in many SCR tasks including [9, 10]. Recent years
have seen the introduction of neural IR models, including [14]. However, more
recently a new class of retrieval models using the pre-trained transformer
architecture (BERT) [15] has been introduced [16]. These models have been
shown to be better than the classic BM25 model [5] and non-BERT neural models,
such as Conv-KNRM [14]. These BERT-based search models can be classified into
two categories: BERT-based re-ranking [6] and dense retrieval (DR) [7, 17].
While BERT-based re-ranking takes as input a query-document pair and directly
produces a relevance score, DR encodes documents and queries into dense
vectors independently and applies approximate nearest neighbour (ANN) search
to these vectors.

N-best ASR hypotheses have been used as input to the BERT model for Spoken
Language Understanding as a form of lattices [18], confusion networks [19] and
plain texts [20]. Ganesan et al. conclude in [20] that text format input was
more suitable for BERT models than alternative formats since BERT models are
pre-trained on plain texts. N-best hypotheses have also been explored for ASR
error correction [21].

Our paper differs from existing work on BERT-based ranking systems for SCR
tasks as follows. (i) We experiment with both BERT re-ranking and BERT DR
systems on manual transcripts and ASR transcripts. The TREC Podcasts Tracks
[10, 11] have seen the use of BERT re-ranking and DR models. However, manual
transcripts were not available for system evaluation. BERT re-ranking was
investigated in [12, 13], while the BERT DR model was not applied to their
spoken broadcast retrieval. (ii) We employ semi-supervised ASR transcripts to
the SCR setting, where a domain-adapted ASR system is not available. Existing
literature [1, 8] suggests a near-linear relationship between transcription
WERs and search scores. Our investigation into the use of semi-supervised ASR
transcripts demonstrates that BERT re-ranking and DR models benefit from
improved transcripts using a semi-supervised ASR system. (iii) We propose the
use of ASR N-best transcripts for BERT DR systems. To the best of our
knowledge, BERT ranking models using ASR N-best for SCR has not been
investigated previously. We observe that both early and late fusion approaches
for BERT DR mitigate errors in ASR transcripts for the SCR task.

##  3 BERT-based Retrieval Models

In this section we provide details of BERT-based re-ranking and DR. We first
present the models, and then describe how they are trained.

###  3.1 BERT for retrieval systems

Fig. 1: Schematic diagram of a BERT-based re-ranking system and a BERT-based
dense retrieval (DR) system.

Figure 1 shows the structure of a BERT re-ranking system and a BERT DR system.
The BERT re-ranker takes a ranked document list from an initial search model
and re-ranks it using the BERT model, while the DR system uses BERT to
transform documents and a query into dense vectors and another search model to
obtain the final ranked list of documents.

BERT Re-ranker  The BERT model re-ranks a ranked document list retrieved from
a document archive using a lightweight search model such as BM25. The first
BERT model used for re-ranking was MonoBERT [6]. Suppose a query consists of
$N$ tokens and a document $M$ tokens, the input to the BERT model can be
expressed as $[CLS]q_{1},...,q_{N}[SEP]$ $d_{1},...,d_{M}[SEP]$, where $[CLS]$
and $[SEP]$ are special tokens of BERT. A relevance score of the query-
document pair is computed by feeding an output vector corresponding to the
$[CLS]$ token to a fully-connected layer.

BERT Dense Retrieval  A BERT model is used to encode a query and documents
into vector representations. While two BERT models were used by Karpukhin et
al. [17] to encode a query and documents independently, Xiong et al. [7]
applied a single BERT model to encode a query and documents (Dual encoder). In
this work, we used the dual encoder framework for our DR system. A
representation of a query and a document is a vector corresponding to the
$[CLS]$ token preceding query tokens or document tokens. In contrast to the
re-ranker BERT output which is a relevance score, a dense vector is a
representation of a document or a query in a fixed dimensional space, and the
final relevance score is computed by taking a similarity score of the query
and document vectors.

A similarity score can be computed using a dot product or cosine similarity.
This enables an efficient retrieval algorithm, such as approximate nearest
neighbour, to be used to create a final list of documents according to query-
document similarity scores [22]. In this work, we use a dot product for
similarity scoring following [7].

BERT re-ranking vs BERT dense retrieval  BERT re-ranking has generally been
found to produce better retrieval metrics than the BERT DR [16]. However,
Xiong et al. [7] demonstrated that using it can be 100 times slower than DR at
search time. Due to its inference speed, BERT re-ranking is often applied to
only the $k$ top documents returned by an initial search, while BERT DR needs
only to encode a query at search time and can be applied to the whole document
collection.

###  3.2 BERT training for retrieval

BERT Re-ranking  The BERT re-ranking model is trained using binary cross-
entropy loss [6]. Specifically, given a query and a relevant document pair,
the BERT model is trained to produce an output of 1, while it is trained to
produce 0 for a query and a non-relevant document. This can be expressed as

$L=-\log(f(q_{i},d_{i}^{+};\theta))-\sum_{j=1}^{J}\log(1-f(q_{i},d_{ij}^{-};\theta))$
(1)

where $f(q_{i},d_{i}^{+};\theta)$ is the score from the BERT model given the
$i$th query-document pair and $f(q_{i},d^{-}_{ij};\theta)$ the score from the
BERT model given the $j$th negative query-document pair corresponding to the
$i$th train example.

BERT Dense Retrieval  The BERT DR model is trained using the negative log
likelihood [7, 17]. Specifically, the model is trained to maximise a
similarity score of a pair of a query and a relevant document and to minimise
a similarity score of a pair of query and a non-relevant document. Formally,
this is:

$\displaystyle s(q,d)$ $\displaystyle=sim(g(q;\theta),g(d;\theta))$ (2)
$\displaystyle L$
$\displaystyle=-\log\frac{\exp(s(q_{i},d_{i}^{+}))}{\exp(s(q_{i},d_{i}^{+}))+\sum^{J}_{j=1}\exp(s(q_{i},d_{ij}^{-}))}$
(3)

where $g(q;\theta)$ is a function to obtain a query dense vector corresponding
to the BERT output vector of the [CLS] token and $g(d;\theta)$ its document
counterpart. $sim()$ is a function to compute a similarity score of the two
vectors. Similar to the cross entropy loss used for the BERT re-ranker, this
loss is obtained for the $i$th query-document pair and corresponding $j$
negative query-document pairs.

To obtain negative examples for each true query-document pair, the BM25 model
is often used to decide the initial negative examples. Xiong et al. [7]
proposed an improved algorithm to select negative examples named Approximate
nearest neighbour Negative Contrastive Learning (ANCE). This algorithm begins
with BERT training on negative examples chosen from the output of BM25
document ranking, and updates negative examples using the BERT model being
trained at every specified checkpoint. Xiong et al. [7] provides theoretical
grounds for ANCE and empirical results demonstrating that ANCE training
produces better search results and faster model convergence. While their
original paper proposes asynchronous updates of negative examples where
inference for ANCE and model training run in parallel, we stopped model
training at the end of every training epoch and ran ANCE updates to resume
model training on updated negative examples.

##  4 N-best Fusion for BERT Dense Retrieval

As outlined in Section 2, using ASR N-best transcripts has been effective in
SLU and ASR error correction [20, 21]. In this paper, we propose two N-best
fusion approaches for BERT DR. The first approach fuses N-best transcripts
before feeding text to BERT (early fusion). The second approach creates N
indexes for N-best transcripts and combines scores from each of the indexes to
compute the final score for each document (late fusion).

###  4.1 N-best early fusion

The input sequence length of the BERT model is often limited to 512 tokens
with tokens exceeding this maximum length being truncated [15]. For the task
of SLU, user utterances are generally very short and N-best transcripts can be
concatenated with the [SEP] token [20]. However, in case of SCR, document
lengths are generally much longer and N-best document representations cannot
be simply concatenated with the [SEP] token. This issue can be resolved by the
alignment approach introduced in [21], and taking the average of $N$ word
embeddings for $i$th token to compress $N$ documents into a single document
representation.

Fig. 2: Example of N-best alignment to create BERT input text.

Figure 2 shows an example of aligning 3-best transcripts. The 1-best
transcript is used as an anchor transcript and aligned with the other two
transcripts. The aligned transcripts are further combined to match the length
across N-best transcripts. As described in [21], “—” is used as a special
token to alleviate a length mismatch between transcripts. This token is added
to one of the word embedding vectors. The input of the BERT model is often
sub-word tokens [15]. It is important to run alignment of transcripts _after_
tokenisation is applied to texts. If this order is reversed, aligned
transcripts can create a length mismatch after tokenisation. Once the N-best
transcripts have been aligned, the tokens are transformed into word embeddings
with the $i$th position of the aligned tokens combined by taking the average.
For example, in Figure 2, the first position of the three transcripts are all
“i”. This results in the averaged embeddings of “i”. This could be extended by
concatenating embeddings and feeding concatenated embeddings to a linear
layer, as in [21], but the concatenated vector size can be large and we did
not apply this method in this paper.

###  4.2 N-best late fusion

Our N-best late fusion approach to BERT DR involved creation of an $N$ index
for each of the N-best transcripts. The BERT model encodes a query into a
vector and search is carried out for all $N$ indexes. The final result is
calculated by summing the scores for each document from each index, and
ranking in descending order of summed scores. While a drawback of this
approach is creating $N$ indexes, ANN search over $N$ indexes could be run in
parallel using $N$ CPUs after computing a query vector to maintain operational
speed.

##  5 Experimental Investigation

In this section we present our experimental investigation. We first describe
our experimental setup. This includes details of the data used for our
experiments, training and hyper-parameter details of BERT re-ranking and DR
models, and creation of standard ASR transcripts and semi-supervised ASR
transcripts. We then present experimental results of a known-item search task
using a corpus of spoken instruction videos. The results show search scores of
applying BERT ranking systems to different types of transcripts, inference
time of BERT re-ranking and DR systems, and the use of an N-best extension for
BERT DR systems.

###  5.1 Experimental Setup

Test Collection and Evaluation Our experimental investigation was carried out
using the How2 dataset [23]. How2 consists of 19,770 videos with manual
transcripts and metadata including video titles. For our experiments we used
How2 video titles as search queries to retrieve the associated video to form a
known-item search task where each query has one relevant target document. We
randomly selected 500 of the 19,770 video titles as evaluation queries.
Experiments used the standard known-item search of mean reciprocal rank (MRR),
defined as follows:

$MRR=\frac{1}{N}\sum^{N}_{i=1}\frac{1}{rank_{i}}$ (4)

where $N$ is the number of user queries and $rank_{i}$ is the rank of the
relevant document in the retrieval list for the $i$th query. In the How2
corpus, only around 2.5% of the collection used exceeds the length limit of
BERT. Thus, our results are unlikely to be affected by the BERT length limit.
Examination of any impact of this issue and strategies to address it could
form future work.

BERT Retrieval Model Data Setup For the re-ranking BERT model, an additional
500 validation queries were randomly selected. For each validation query, one
positive example and one negative example from the top of the BM25 ranked
document list were chosen (if the top document is the positive example, the
second ranked document was chosen) to compute validation loss using the cross
entropy. For the BERT DR model, 500 validation queries were randomly chosen,
and for each query, 1 relevant document and 99 irrelevant documents from the
BM25 model were ranked by the BERT encoder to compute the mean reciprocal rank
of the relevant document.

ASR Transcripts To obtain ASR transcripts of How2 audio data, a Kaldi “chain”
system was developed using the LibriSpeech corpus [24, 25]. This is a
traditional ASR system integrating an acoustic model with a hidden Markov
model. This system, referred to as “standard ASR”, trained on LibriSpeech
created a domain mismatch scenario to recognise How2 audio and produced an
overall WER of 31.1%. We used the “standard ASR” system to generate decode
lattices and 1-best transcripts of the How2 train data. Combined with
LibriSpeech data, the decode lattices were used to train a domain-adapted
semi-supervised acoustic model, while 1-best transcripts were used for
training of a semi-supervised N-gram for decoding and a re-scoring recurrent
neural network language model [4, 26, 27]. This system, referred to as “semi-
supervised ASR” produced an overall WER of 23.95%. Further, we generated
lattice oracle transcripts using the “standard ASR” system (i.e., the most
correct hypotheses present in lattices with respect to manual transcripts),
which produced an overall WER of 11.18%. For the N-best experiments, the
“standard ASR” and “semi-supervised ASR” systems were used to generate {2, 5,
10, 20}-best transcripts to develop the N-best version of the BERT DR system.

Retrieval Models The BM25 model used was the Python version of Lucene with
$k1$ 1.2 and $b$ 0.75111https://lucene.apache.org/. The re-rank system took as
input the top 1,000 documents returned for each query. The pre-trained BERT
model used was “bert-base-uncased” and its corresponding tokeniser from
HuggingFace [28]. For the DR system, after the whole document collection had
been encoded by the BERT model, Faiss software [22] was used to run similarity
search over a query vector and an index of document vectors.

The hyper-parameter settings for BERT model training were as follows. The re-
ranker BERT was trained with 20 epochs and used the model checkpoint which
produced the best validation score. Model training was generally converged by
epoch 15. The best initial learning rate was empirically selected, and was
$3\times 10^{-7}$. The DR BERT was converged by 10 epochs as reported by [7].
The best initial learning rate found was $1\times 10^{-5}$. Both re-ranker
BERT and DR BERT used 20 negative examples. Our BERT models were trained using
Quadro RTX 6000. The BERT models could potentially be improved by using more
negative examples, but the number of negative examples 20 was its memory
limit. Increasing the negative examples using a memory queue could form future
work. For training of the re-ranker and DR BERT, the AdamW optimizer was used
with a weight decay of $5\times 10^{-5}$ to avoid overfitting.

###  5.2 MRR scores using lattice oracle, manual, standard and semi-supervised
ASR transcripts

Table 1 presents MRR scores for BM25, BERT re-ranking and BERT DR systems
using manual (man), lattice oracle (oracle), semi-supervised ASR (semi ASR)
and standard ASR (std ASR) transcripts. We use the paired 2-tailed t-test to
compare distributions of MRR scores from BM25 and other BERT-based models.

Comparison of BERT ranking systems The largest MRR gain was obtained using the
BERT DR system trained with ANCE. This improved the MRR scores by around 26%
over the BM25 model on manual transcripts, and by 23% on standard ASR
transcripts, indicating that transcription errors affected the score. The BERT
re-ranking system improved the MRR scores by around 16-18% over the BM25 model
for all transcript types.

Comparison of searching over different transcripts Despite the improvement in
MRR obtained by BERT re-ranking and DR systems, the gap remains between manual
and ASR transcript scores. Searching over standard ASR transcripts resulted in
around 11% worse MRR scores than manual transcripts with BM25 and BERT re-
ranking. BERT DR systems were more affected by transcription errors showing
around 15% worse MRR scores when using standard ASR transcripts. Although
semi-supervised transcripts had a small impact on the improvement of MRR score
by the BM25 model, the BERT re-ranking system improved the MRR score by 2% and
the BERT DR systems improved the score by 5.5% without ANCE training and by
around 4% with ANCE training. The oracle experiments, however, show that the
gap in the MRR scores between manual transcripts and ASR transcripts could
potentially be further reduced. In comparison to searching over manual
transcripts, all of the models including BM25 produced only 3-4% worse MRR
scores when searching over lattice oracle transcripts.

Table 1: MRR results of using different transcripts for How2 known-item search evaluation. All of BERT results are statistically significant over BM25 with $p<0.05$ denoted by **. model | MRR
---|---
eval data
man | oracle | semi ASR | std ASR
BM25 | 40.27 | 36.02 | 30.85 | 29.37
rerank | 57.32** | 52.46** | 48.69** | 46.37**
DR | 50.61** | 45.96** | 40.38** | 34.81**
DR-ANCE |  66.73** |  63.88** |  56.22** |  52.41**

Comparison of search run time Similar to [7], we compare the inference time of
100 documents per query. Search operations of BM25 were measured using a CPU,
while inference of BERT re-ranking was performed using a GPU. BERT DR search
run time is the total for encoding the query using a GPU and ANN operations
using a CPU. For fair comparison, all of the search runs were performed on the
same computation node of our high performance computing system. We observe
that average run time of BM25 was 40 msecs, BERT re-ranking 1.84 secs, and
BERT DR 18 msecs, respectively. This result agrees with [7], where BERT re-
ranking was roughly 100x slower than BERT DR at search time and demonstrates a
practical advantage of the BERT DR system.

Table 2: MRR results of BERT-DR models with and without ANCE training using the N-best extension over standard ASR and semi-supervised ASR transcripts. % change shows relative improvement/regression of MRR over corresponding 1-best models. Asterisks * and ** denote statistical significance over the baseline 1-best system with $p<0.05$ and $p<0.01$, respectively. model | MRR | % change
---|---|---
N-size
1 | 2 | 5 | 10 | 20
standard ASR |  |  |  |  |  |
2-7 DR-early | 34.81 | 35.42 | 32.94 | 33.71 | 34.24 | +1.75 (2-best)
DR-late | 35.49 | 36.94** | 37.55** |  38.20** | +9.74 (20-best)
2-7 DR-ANCE-early | 52.41 | 51.77 | 52.14 | 52.75 | 54.45 | +3.49 (20-best)
DR-ANCE-late | 53.29 | 54.88** | 54.86** |  55.40** | +5.71(20-best)
semi-sup ASR |  |  |  |  |  |
2-7 DR-early | 40.38 | 39.40 | 38.21 | 37.91 | 38.72 | -2.43 (2-best)
DR-late | 40.05 | 40.21 | 41.20 | 40.93 | +2.03 (10-best)
2-7 DR-ANCE-early | 56.22 | 57.26 | 57.27 | 57.92 |  60.15** | +6.99 (20-best)
DR-ANCE-late | 57.79* | 58.19** | 58.79** |  59.20** | +5.30 (20-best)

###  5.3 MRR scores using ASR N-best for BERT DR

We next present the MRR scores of our proposed ASR N-best extension for BERT
DR. Table 2 summarises the MRR scores of BERT DR using {2,5,10,20}-best
transcripts either from the standard ASR system or from the semi-supervised
ASR system. MRR scores of N-size 1 correspond to the MRR scores from row 3 and
row 4 and semi ASR and std ASR columns in Table 1. Bold face indicates MRR
scores of each BERT DR system which are higher than the baseline 1-best result
and highest among other N sizes. A paired 2-tailed t-test was again carried
out to compare MRR distributions from the 1-best BERT model and other N-best
models.

The 20-best early and late fusion over semi-supervised transcripts using ANCE
BERT produced MRR of 60.15% and 59.20%, respectively (row 3 and 4 of semi-sup
ASR in Table 2). Considering that the 1-best BERT DR over standard ASR
transcripts produced MRR of 52.41% and the MRR gap between standard ASR and
manual transcripts was 14.32% ($66.73-52.41$), the MRR score gap was reduced
to 6.58% ($66.73-60.15$) using the 20-best early fusion approach. This
reduction represents a more than 50% relative improvement.

Early vs late fusion While the early fusion approach produced inconsistent
improvement of MRR with different N-sizes, the late fusion approach
consistently benefited from increase in N sizes. Despite the highest MRR score
of 60.15% obtained from early fusion of 20-best transcripts for BERT DR with
ANCE training, when the early fusion was applied to BERT DR without ANCE
training, the MRR scores did not improve with increase in the value of N.
While statistical significance over the 1-best system was observed over semi-
supervised ASR transcripts (row 3 semi-sup ASR), the MRR score of 20-best over
the standard ASR transcripts did not show statistical significance (row 3
standard ASR). In contrast, the late fusion approach brought consistent
improvement over 1-best systems. The highest MRR score observed was 59.20
using 20-best semi-supervised ASR transcripts for BERT DR with ANCE training
(row 4 semi-sup ASR). Generally, 20-best transcripts benefited the late fusion
approach the most, except when BERT DR without ANCE training was applied to
semi-supervised ASR transcripts (row 2 semi-sup ASR).

In further analysis we examined the average MRR improvement and regression per
query for the 20-best extension to ANCE BERT DR systems. The early fusion
approach brought an average improvement in MRR of 0.32 for 147 out of 500
evaluation queries over the 1-best system, while 123 out of these 500
evaluation queries had a lower MRR score with average 0.22. MRR scores of the
other 240 queries were unchanged. The late fusion approach with 20-best ANCE
BERT improved the MRR scores of 135 queries with by an average 0.17, while
reducing scores of 86 queries by an average of 0.10. MRR scores of the other
279 queries were unchanged by the late fusion approach. These results
demonstrate that the early fusion approach introduces more radical changes
than late fusion, in computation of document representations over 1-best
systems.

##  6 Conclusions and Future Work

We explored BERT re-ranking and DR systems for a known-item SCR task. Despite
improved search effectiveness using the BERT ranking model, MRR scores from
BERT re-ranking and DR systems were 10-14% lower when using ASR than manual
transcripts. We investigated the use of semi-supervised ASR transcripts and an
N-best extension for BERT DR. Use of semi-supervised transcripts improved MRR
by 2-5.5%, while use of 20-best transcripts brought a gain in 3-4%. Combining
them gave an average improvement of 6.5%.

Future work includes the use of decode lattices for BERT retrieval. As seen in
Table 1, applying BERT ranking models to lattice oracle transcripts further
closed the MRR gap in 3-5% MRR to manual transcripts. This motivates
development of BERT ranking which can effectively take as input decode
lattices to overcome ASR transcription errors.

Acknowledgement: This work was supported by Science Foundation Ireland as part
of the ADAPT Centre (Grant 13/RC/2106) at Dublin City University.

## References

  * [1] M. Larson and G. J. F. Jones,  “Spoken Content Retrieval: A survey of techniques and technologies,”  Foundations and Trends in Information Retrieval, vol. 4, no. 4-5, pp. 235–422, 2012.
  * [2] Y. Wang, A. Mohamed, D. Le, C. Liu, A. Xiao, J. Mahadeokar, H. Huang, A. Tjandra, X. Zhang, F. Zhang, C. Fuegen, G. Zweig, and M. L. Seltzer,  “Transformer-based acoustic modeling for hybrid speech recognition,”  in ICASSP, 2020, pp. 6874–6878.
  * [3] W-N. Hsu, B. Bolte, Y-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed,  “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”  Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021.
  * [4] Y. Moriya and G. J. F. Jones,  “Augmenting asr for user-generated videos with semi-supervised training and acoustic model adaptation for spoken content retrieval,”  in Statistical Language and Speech Processing, 2021, pp. 73–84.
  * [5] S. Robertson and H. Zaragoza,  “The probabilistic relevance framework: Bm25 and beyond,”  Foundations and Trends in Information Retrieval, vol. 3, no. 4, pp. 333–389, 2009.
  * [6] R. Nogueira and K. Cho,  “Passage re-ranking with bert,”  arXiv preprint, 2019.
  * [7] L. Xiong, C Xiong, Y Li, K-F. Tang, J. Liu, P. N. Bennett, J Ahmed, and A. Overwijk,  “Approximate nearest neighbor negative contrastive learning for dense text retrieval,”  in ICLR, 2021.
  * [8] J. S. Garofolo, C. G. P. Auzanne, and E. M. Voorhees,  “The TREC spoken document retrieval track: A success story,”  in Content-Based Multimedia Information Access - Volume 1, 2000, p. 1–20.
  * [9] M. Eskevich, R. Aly, D. Racca, R. Ordelman, S. Chen, and G.J.F. Jones,  “The search and hyperlinking task at mediaeval 2014,”  in Proceedings of MediaEval, 2014.
  * [10] R. Jones, B. Cartere, A. Clifton, M. Eskevich, G. J. F. Jones, J. Karlgren, A. Papu, S Reddy, and Y. Yu,  “TREC 2020 podcasts track overview,”  in Text REtrieval Conference (TREC), 2020.
  * [11] R. Jones, B. Cartere, A. Clifton, M Eskevich, G. J. F. Jones, J. Karlgren, A. Pappu, S. Reddy, and Y. Yu,  “TREC 2021 podcasts track overview,”  in Text REtrieval Conference (TREC), 2021.
  * [12] H-Y. Lin, T-H. Lo, and B. Chen,  “Enhanced bert-based ranking models for spoken document retrieval,”  in ASRU, 2019, pp. 601–606.
  * [13] S-W. Fan-Jiang, T-H. Lo, and B. Chen,  “Spoken document retrieval leveraging bert-based modeling and query reformulation,”  in ICASSP, 2020, pp. 8144–8148.
  * [14] Z. Dai, C. Xiong, J. Callan, and Z. Liu,  “Convolutional neural networks for soft-matching n-grams in ad-hoc search,”  in International Conference on Web Search and Data Mining, 2018, p. 126–134.
  * [15] J. Devlin, M-W. Chang, K. Lee, and K. Toutanova,  “BERT: Pre-training of deep bidirectional transformers for language understanding,”  in NAACL, 2019, pp. 4171–4186.
  * [16] J. Lin, R. Nogueira, and A. Yates,  “Pretrained transformers for text ranking: BERT and beyond,”  arXiv preprint, 2020.
  * [17] V. Karpukhin, B. Oguz, S. Min, P. Lewis, L.l Wu, S. Edunov, D. Chen, and W-T. Yih,  “Dense passage retrieval for open-domain question answering,”  in EMNLP, 2020, pp. 6769–6781.
  * [18] C-W. Huang and Y-N. Chen,  “Adapting pretrained transformer to lattices for spoken language understanding,”  in ASRU, 2019, pp. 845–852.
  * [19] C. Liu, S. Zhu, Z. Zhao, R. Cao, L. Chen, and K. Yu,  “Jointly Encoding Word Confusion Network and Dialogue Context with BERT for Spoken Language Understanding,”  in Interspeech, 2020, pp. 871–875.
  * [20] K. Ganesan, P. Bamdev, J. B, V. Amresh, and A. Tushar,  “N-best ASR transformer: Enhancing SLU performance using multiple ASR hypotheses,”  in ACL, 2021, pp. 93–98.
  * [21] L. Zhu, W. Liu, L. Liu, and E. Lin,  “Improving asr error correction using n-best hypotheses,”  in ASRU, 2021, pp. 83–89.
  * [22] J. Johnson, M. Douze, and H. Jégou,  “Billion-scale similarity search with GPUs,”  IEEE Transactions on Big Data, vol. 7, no. 3, pp. 535–547, 2019\.
  * [23] R. Sanabria, O. Caglayan, S. Palaskar, D. Elliott, L. Barrault, L. Specia, and F. Metze,  “How2: a large-scale dataset for multimodal language understanding,”  in Proceedings of the Workshop on Visually Grounded Interaction and Language (ViGIL). NeurIPS, 2018.
  * [24] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur,  “Librispeech: An ASR corpus based on public domain audio books,”  in ICASSP, 2015, pp. 5206–5210.
  * [25] H. Hadian, H. Sameti, D Povey, and S Khudanpur,  “End-to-end speech recognition using lattice-free mmi,”  in Interspeech, 2018, pp. 12–16.
  * [26] V. Manohar, H. Hadian, D. Povey, and S. Khudanpur,  “Semi-supervised training of acoustic models using lattice-free mmi,”  in ICASSP, 2018, pp. 4844–4848.
  * [27] H. Xu, T. Chen, D. Gao, Y. Wang, K. Li, N. Goel, Y. Carmiel, D. Povey, and S. Khudanpur,  “A pruned rnnlm lattice-rescoring algorithm for automatic speech recognition,”  in ICASSP, 2018, pp. 5929–5933.
  * [28] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush,  “Transformers: State-of-the-art natural language processing,”  in EMNLP, 2020, pp. 38–45.


# Visualising emergent phenomena at oxide interfaces

Michael Oberaigner
Graz University of Technology,
Graz, Austria
<EMAIL_ADDRESS>
Manuel Ederer
Technical University of Vienna,
Vienna, Austria
<EMAIL_ADDRESS>
Sandeep K. Chaluvadi
Istituto Officia dei Materiali-CNR,
Trieste, Italy
<EMAIL_ADDRESS>
Pasquale Orgiani
Istituto Officia dei Materiali-CNR,
Trieste, Italy
<EMAIL_ADDRESS>
Regina Ciancio
Istituto Officia dei Materiali-CNR,
Trieste, Italy
<EMAIL_ADDRESS>
Stefan Löffler
Technical University of Vienna,
Vienna, Austria
<EMAIL_ADDRESS>
Gerald Kothleitner
Graz University of Technology,
Graz, Austria
<EMAIL_ADDRESS>
Daniel Knez
Graz University of Technology,
Graz, Austria
<EMAIL_ADDRESS>
Corresponding author

###### Abstract

Knowledge of atomic-level details of structure, chemistry, and electronic
states is paramount for a comprehensive understanding of emergent properties
at oxide interfaces. We utilise a novel methodology based on atomic-scale
electron energy loss spectroscopy (EELS) to spatially map the electronic
states tied to the formation of a two-dimensional electron gas (2DEG) at the
prototypical non-polar/polar $TiO_{2}$/$LaAlO_{3}$ interface. Combined with
differential phase contrast analysis we directly visualise the microscopic
locations of ions and charge and find that 2DEG states and $Ti^{3+}$ defect
states exhibit different spatial distributions. Supported by density
functional theory (DFT) and inelastic scattering simulations we examine the
role of oxygen vacancies in 2DEG formation. Our work presents a general
pathway to directly image emergent phenomena at interfaces using this unique
combination of arising microscopy techniques with machine learning assisted
data analysis procedures.

_Keywords_ Transition metal interface $\cdot$ 2DEG $\cdot$ STEM-EELS $\cdot$
DPC $\cdot$ Lanthanum aluminate $\cdot$ Anatase $\cdot$ Oxygen Vacancies
$\cdot$ Electronic state mapping

##  1 Introduction

The electronic, magnetic and catalytic properties of transition metal oxide
(TMO) interfaces have received significant attention in recent years. This is
due to their promising potential for breakthroughs in materials science and
nanotechnology. The formation of an electrically conductive layer is one of
the most intriguing phenomena observed in these interfaces. Such two-
dimensional electron gases (2DEG) are studied at several TMO interfaces, with
the prototypical example being $SrTiO_{3}$/$LaAlO_{3}$ forming a non-
polar/polar interface Ohtomo and Hwang (2004). Strong electron coupling at
such interfaces trigger a metal-insulator transition by different external
stimuli, such as electric or magnetic fields, temperature, light and strain
(Hwang et al., 2012; Yang et al., 2011). Applications of these effects have
already been demonstrated, for instance in highly integrated transistors (Cen
et al., 2009), or in ultrahigh-density nanoelectronic memory and logic
elements (Park et al., 2010). The metal-insulator transition in such systems
is often attributed to an electron transfer at the interface due to a polar
discontinuity, referred to as electronic reconstruction or polar catastrophe
model (Nakagawa et al., 2006; Ohtomo and Hwang, 2004; Thiel et al., 2006).
Electric field formation, however, can also lead to the formation and
accumulation of oxygen vacancies (VO) at the interface region, which act as
n-type dopants (Chen et al., 2010; Bristowe et al., 2011). The absence of
impact on electron mobility at varying VO concentrations is attributed to the
separation of charge carriers and charged oxygen vacancies (Siemons et al.,
2007). Moreover, the intermixing of cations may generate charge carriers to
establish a conductive region, or it may impede charge mobility as a scatterer
within the conductive layer (Schlom and Mannhart, 2011; Bristowe et al., 2014;
Chen et al., 2013).
In order to explore the role of these mechanisms a large variety of techniques
like angle-resolved photoemission spectroscopy (ARPES), X-ray absorption
spectroscopy (XAS), X-ray photoemission spectroscopy (XPS),
magnetic/electrical transport measurements or scanning tunneling microscopy
and spectroscopy (XSTM/S) have already been applied (Salluzzo et al., 2009;
Drera et al., 2013; Troglia et al., 2022; McKeown Walker et al., 2022; Strocov
et al., 2022; Thiel et al., 2006; Bigi et al., 2020; Huang et al., 2012).
While highly valuable information could certainly be obtained from such
measurements, all these techniques either lack spatial resolution or are
surface sensitive only. Scanning transmission electron microscopy (STEM) is
the only technique that enables atomic imaging of such interfaces in cross-
section and, coupled with electron energy loss spectroscopy (EELS), allows to
study the spatial distribution of elements, valence states and even individual
orbitals (Knez et al., 2020; Maurice et al., 2008; Löffler et al., 2017;
Bugnet et al., 2022). Additional integrated differential phase contrast (iDPC)
imaging and differential phase contrast (DPC) field mapping enables us to
provide a complete picture of the contributing phenomena at unprecedented
detail.
By relying on the most recent technological and methodological advancements in
electron microscopy, such as direct electron detection and machine learning
assisted post-processing, combined with density functional theory (DFT)
simulations and inelastic scattering calculations, we reveal the structure,
chemistry and electronic states of the $TiO_{2}$/$LaAlO_{3}$ interface at
atomic scale. Such characterisations are crucial for designing and engineering
TMO interfaces with tailored 2DEG properties, which can have far-reaching
implications for advanced electronic devices and technologies. We believe that
our results will also help to understand similar systems, such as
$LaAlO_{3}$/$LaVO_{3}$(Wadati et al., 2008) or $SrVO$/$SrTiO$ (Mirjolet et
al., 2021).

In this work we present a visualisation of the spatial distribution of
electronic states associated with the presence of 2DEG and defects at the
anatase/LAO ($TiO_{2}$/$LaAlO_{3}$) interface (Fig. 1a). While this interface
is closely related to the STO/LAO system, it lacks the influence of
interfacial strain due to the almost perfectly matching lattices between
anatase and LAO, with only 0.1 % lattice mismatch compared to the 3 % mismatch
of STO/LAO interfaces (Gobaut et al., 2017; Cheng et al., 2017; Minohara et
al., 2014). Anatase $TiO_{2}$ (001) is grown epitactic on single crystalline
LAO (001) by pulsed laser deposition (PLD). Fig. 1b provides a representative
view of the cross-section of the anatase film and interface region,
demonstrating that the interface is atomically sharp and of high quality,
which has also been shown in previous studies (Knez et al., 2020; Islam et
al., 2022). The anatase film has a thickness of approximately 15 nm. Film and
substrate fully match in plane as a consequence of the very low lattice
mismatch of anatase with the LAO substrate. The interface can have two types
of termination as illustrated in Fig. 1c, which exhibit different electronic
properties. Since the 2DEG is only formed at the La-terminated interface
($Al$-termination leads to semiconducting behaviour), the focus of this study
is on La-terminated interfaces (Wang et al., 2010; Minohara et al., 2014;
Cheng et al., 2017). Proof for the presence of the La-termination is provided
by high-angle annular dark-field (HAADF) contrast and further supported by
EELS mapping of the $Ti$ L2,3 edge signal, as shown in Fig. 1d.

Figure 1: $TiO_{2}$ and $LaAlO_{3}$ unit cells (a) with HAADF-recorded
interface (b) demonstrating the high-quality epitaxial film. The two types of
terminations (c) can be determined by HAADF and EELS analysis, integrating the
$Ti$ L2,3 edges (d).

##  2 Electronic Analysis

The partial density of states (pDOS) calculated by DFT indeed indicates a
metallic behaviour for the oxygen layers close to the La-terminated interface
due to the crossing electronic states with the Fermi-level and accordingly
these states can be attributed to O atoms contributing to the 2DEG formation
(Fig. 2a). Two regions, one at a distance of approximately 1.5 nm from the
interface (denoted by a red rectangle in the corresponding HAADF image (Fig.
2b)) is compared with the region at the interface (green rectangle), showing
that the metallic features are only present close to the interface. In order
to detect the 2DEG at the interface experimentally, we performed monochromated
STEM EELS mapping of the O K edge. By using a post-processing sequence to
identify and automatically remove artefact-affected data, as well as
correcting sample drift and enhancing signal-to-noise ratio through multi-
frame imaging, we are able to detect faint spectral details with high
resolution in both space and energy. Details about the applied procedures are
provided in the supplementary file. The results of the EELS analysis are
summarised in Fig. 2c. The features of the O K edge from the first region
exhibits the typical features for anatase, with the t2g and eg peaks due to
the effect of crystal field splitting. A comparison, however, with the region
at the interface reveals the presence of an additional feature directly at the
edge onset at approximately 530 eV as highlighted in Fig. 2c. Such features
clearly indicate the occurrence of additional states at the Fermi-level, as
seen in the DFT calculations. It shall be noted that for the simulation no VO
or cation intermixing was considered, indicating that these states are
intrinsic to the interface and not related to VO accumulation. By selecting
the corresponding energy region, a map for the spatial distribution of these
states within this energy range is generated, as denoted in Fig. 2c. The
resulting map is shown in Fig. 2d. It becomes clear that these 2DEG states are
limited to a spatial range between the first two $Ti$ atoms from the
interface. For comparison, a similar map is generated from the simulation data
by performing STEM-EELS multislice simulations from the pDOS at a specific
energy loss (Fig. 2e). The simulations match well with the experiment and
indicate that the metallic features indeed originate from electronic
reconstruction. Despite the delocalised nature of the electrons contributing
to the 2DEG, the corresponding states are still localised to the atomic
columns due to the nature of EELS projecting onto core states and the
channelling by the atomic columns.
Similar features at the Fermi-level should also be visible at the $Ti$ L3 EELS
edge, however in this case the signal overlaps with features associated to the
presence of defect ($Ti^{3+}$) states (Bigi et al., 2020), as shown in
Supplementary Fig. 1. Furthermore, multiplet effects, which are not included
in DFT calculations (Krüger, 2010), affect the shape of Ti L edges and can
therefore not be interpreted directly as the pDOS, e.g. pre-peak features at
the L3 edge (Kucheyev et al., 2004). To include such effects into our
calculations would require to solve the Bethe-Salpeter equation, which is
computationally expensive and beyond the scope of this work. In contrast, such
effects caused by electron-electron interactions can be ignored for the $O$ K
edge because the interaction energy of the 1s core electron with valence
electrons is less than 1 meV (Frati et al., 2020; Groot, 2005).

Figure 2: 2DEG mapping with the O K edge. (a) pDOS calculation showing O atoms
with metallic features at the interface. (b) Corresponding HAADF image. (c)
EELS-Spectrum comparison of the O K edge between bulk and interface. (d)
Experimental map of the 2DEG using the marked regions in (b). (e) Simulated
map of the 2DEG using the marked regions in (a).

The electronic reconstruction model predicts that the positively charged
$(LaO)^{+}$ terminating layer should induce an electric field that compensates
for the discontinuity between the polar LAO and the non-polar $TiO_{2}$,
avoiding a polar catastrophe (Ohtomo and Hwang, 2004; Pauli and Willmott,
2008). In order to probe the presence of such local electric field changes
associated with the presence of a 2DEG, we performed STEM DPC measurements at
the interface region. DPC probes the momentum transfer caused by
electromagnetic fields within the specimen to the incident electrons. The
results are shown in Fig. 3a and indeed reveal an out-of-plane deflection of
the electron beam at the interface towards the LAO substrate, which indicates
the presence of an confining potential with a field gradient parallel to the
interface. The colors correspond to the direction of beam deflection, which
means that the field points in the opposite direction. The field region
extends approximately 1 nm into the anatase film, which matches with
predictions from DFT calculations (Bigi et al., 2020).
To account for the possibility of artefacts due to diffraction contrast
variations, which are known to occur particularly at interfaces and grain
boundaries (MacLaren et al., 2015), additional atomic displacement
measurements have been performed. A local electric field is expected to
manifest itself by shifts of the negatively charged $O$ ions relative to the
positively charged $Ti$ ions. This effect of an electric field on the anatase
ion positions is demonstrated by multislice calculations based on molecular
dynamics simulations (Fig. 3c). These atom displacements are measurable in a
high resolution integrated DPC (iDPC) micrograph of the interface region
(depicted in Fig. 3b). The signal obtained with iDPC has been shown to be
particularly efficient for light element imaging (Lazić et al., 2016). From
this image the atomic positions are determined with sub-pixel accuracy by
fitting 2D Gaussian functions to the columns, utilising the Atomap package
(Nord et al., 2017). The distances are then measured between pairs of $O$ and
$Ti$ atoms in out-of-plane direction. These measurements reveal a stretching
and compressing of the lattice close to the interface, dependent on the
orientation of the $Ti$-O dipoles relative to the field direction. The
distance between the ions changes by approximately 10-20 pm. The evident
orientation dependence also allows us to exclude the possibility that these
displacements are caused by other effects such as strain or vacancies and
clearly indicate the presence of an electric field, with the same out-of-plane
direction as found in the DPC field mapping data.

Figure 3: DPC experiments at the interface. (a) iDPC and DPC measurements.
Colors in the the DPC image correspond to the deflection of the beam with the
orientation indicated by the color wheel in the inset. (b) iDPC image showing
the stretching and compression of the Ti-O dipoles depending on their
orientation in the presence of an electric field. Error bars are estimated by
statistic sampling (around 5 measurements per point) and averaged over all
points. (c) Simulated iDPC images of anatase with and without electric field
demonstrating the atomic displacement in the profile plots.

##  3 Defects

Although our 2DEG mapping agrees with the defect-free simulations within the
experimental and numerical accuracy, VO are known to play a significant role
in influencing the properties of the interface and the formation of the 2DEG
(Kalabukhov et al., 2007). On the one hand VO act as donors, introducing
additional electrons and thereby increasing the carrier density at the
interface (Siemons et al., 2007). On the other hand, VO also induce lattice
distortions (Knez et al., 2020; Lee and Selloni, 2022), which affect the
electronic structure and band alignment at the interface (Zhang and Dong,
2019). To probe local variations in the VO concentration the EELS Ti L3 eg/t2g
peak ratio and the valley between them are useful indicators. Changes of these
features, which are often traced back to changes in the $Ti^{3+}$/$Ti^{4+}$
ratio (Maurice et al., 2008), are indeed visible in our EELS measurements as
shown in Fig. 4a, where three spectra extracted from regions indicated in (b)
are given. To measure the amplitudes Voigt profiles are fitted to the
individual peaks (see the Methods section for more details) yielding a map of
the eg/t2g peak amplitude ratio provided in Fig. 4c. This image reveals a
higher concentration of $Ti^{3+}$ at a distance up to the second and third
layer of $Ti$ from the interface while in contrast, the termination layer of
$Ti$ at the interface shows a ratio similar to the bulk, which is in close
agreement with the spatial extend of the confining potential and literature
reports (Bigi et al., 2020). Note that the fitting procedure would result in
artefacts and noise in the $LaAlO_{3}$ regions due to the absence of $Ti$,
which is ignored here. Further proof for the presence of a higher amount of VO
is provided by iDPC images by a slightly reduced intensity of oxygen columns
close to the interface, as shown in Fig. 4d. The terminating oxygen layer at
the interface, in contrast, exhibits a higher intensity, which again agrees
well with the EELS data. Such accumulation of VO cannot be observed in the LAO
(see Oxygen vacancies in $LaAlO_{3}$ in supplementary information).
The high spatial resolution of our data enables us to separate the location of
the VO related defect states from the $O$ K pre-edge states attributed to the
2DEG. Fig.4e shows a comparison of intensity profiles obtained from the
experimental maps of the pre-edge states from Fig. 2c and the peak ratio
eg/t2g from Fig.4b, corresponding to the respective phenomena. Thereby, a
spatial separation of the VO accumulation and the 2DEG is observed. While the
2DEG is found to be located close to the interface, VO are accumulated within
a range between approximately 0.5 and 1 nm from the interface.
Unlike oxygen vacancies, $Al$ and $La$ interdiffusion does not affect the
charge carrier density of the 2DEG due to their inefficiency as n-type dopant
in $TiO_{2}$ (Minohara et al., 2014), however cation intermixing can reduce
the carrier mobility (Chen et al., 2013). We were able to detect some degree
of cation intermixing of $Ti$ and $Al$ between the terminating layers
(Supplementary Fig. 3), which will affect the electron mobility.

Figure 4: VO at the interface. Normalised $Ti$ L3 edges (a) from regions
indicated in (b). (c) Ratio between eg and t2g from Ti L3 edges, which have at
least 600 counts. (d) iDPC image of the interface with corresponding profiles
revealing the lower $O$ peaks. (e) Normalised intensity profile of VO mapped
by the $Ti$ eg/t2g ratio and 2DEG mapped by the $O$ K pre-edge across the
interface.

## Summary

In this study, the spatial distributions of the 2DEG, oxygen vacancies, and
cation intermixing have been imaged with atomic resolution by STEM-EELS at a
non-polar/polar TMO interface. These distributions have a significant impact
on the properties of the 2DEG and are therefore important for engineering new
TMO interfaces. At the example of $TiO_{2}$/$LaAlO_{3}$ interface displayed in
this work, the electronic states that cause the 2DEG have been experimentally
localised in the first two $Ti$ layers from the termination by STEM EELS.
Corresponding inelastic scattering simulations of a defect-free interface
match with experimental EELS maps within the experimental and numerical
accuracy, indicating that VO are not primarily required for the 2DEG
formation, which is in agreement with previous work (Cheng et al., 2017). We
were further able to disentangle 2DEG states from $Ti^{3+}$ defect states in
EELS and showed that VO are accumulated at a distance of around 2-3 $Ti$
layers from the interface by a built-in electric field. We directly visualised
this field by DPC field mapping and measured its effects on the positions of
the ions within the field. Last, cation interdiffusion of $Al$ in the
conductive layer has been observed, which can affect the electron mobility.
Our study provides a complete picture off several phenomena accompanying 2DEG
formation at oxide interfaces and highlights the great power of combining
state-of-the-art STEM experiments, advanced data processing techniques and
high-level simulations to address emergent phenomena in complex oxide
heterostructures.

##  4 Methods

### Sample preparation

The thin layer of anatase is epitaxially grown on (001)-oriented $LaAlO_{3}$
substrates by pulsed laser deposition (PLD) using a $KrF$ excimer pulsed laser
source of wavelength 248 nm. A laser energy density of about 2 Jcm-2 and a
laser repetition rate of 1 Hz were used. The growth temperature was set to
700°C and the target-substrate distance was about 5 cm. A stoichiometric
single-crystalline rutile $TiO_{2}$ target was used and the oxygen pressure
was approximately 10-4 mbar during growth. The TEM sample is prepared by
focused ion beam milling (FIB) at 30 keV and 5 keV utilising a Nova 200
NanoLab from FEI. The FIB lamella is polished by low voltage $Ar^{+}$ ion
milling at 900 eV utilising a Fischione Nanomill. Before STEM investigations,
the specimens are baked at 190°C for 2 h in vacuum and cleaned 10 s with
$Ar$/$O$ plasma to reduce contamination.

### STEM-EELS and DPC

The experiments are performed on an FEI Titan3 G2 60-300 equipped with a
monochromator and probe Cs-corrector. The EELS signal is recorded by a direct
electron detection camera, K2 by Gatan Inc. A HAADF image is recorded
simultaneously with each spectrum image, which is used as reference image for
multi-frame spectrum summation and spatial alignments during post-processing.
The images are taken at 300 keV with a EELS semi-collection angle of 24.2 mrad
and a beam semi-convergence angle of 15.5 mrad. The beam current was
approximately 40 pA with an energy resolution of 250 meV (full width at half
maximum of the zero loss peak) by utilising the monochromator. For the DPC
measurements, the microscope is operated without an excitation of the
monochromator. The semi-convergence angle is varied between 15.6 mrad and 19.6
mrad. The signal is recorded by a segmented annular dark field detector, which
is divided into four quadrants with a collection angle of 6-34 mrad. The
specimen thickness values, which are used for corresponding simulations, are
determined by utilising the automatised PACBED analysis based on pre-trained
convolutional neural networks as described elsewhere in more detail (Xu and
LeBeau, 2018; Oberaigner et al., 2023).

### Data post processing

STEM-EELS data are denoised by a combination of a multi-frame integration
approach and multivariate analysis including automatic control mechanisms
assessing data quality, like clustering the SI and rating frames at the multi-
frame approach by a $\ell_{2}$-norm. The procedure is described in detail in
the supplementary information.
To map the $Ti^{+3}$/$Ti^{+4}$ ratio, Voigt profiles, which are a convolution
between a Lorentzian and Gaussian function, are fitted into the eg and t2g
peaks from the $Ti$ L3 edge. These profiles consider the Lorentzian component,
which is caused by the lifetime broadening, and the Gaussian part, which is
introduced by the energy spread of the primary electrons. The peak width and
the shape factor are determined by fitting into the spectrum of summed SI and
these parameters are locked for the amplitude fitting at each pixel
afterwards.

### DFT calculation

All ab-initio density functional theory calculations are performed using the
all-electron, full-potential augmented plane wave method as implemented in
WIEN2k (Blaha et al., 2020). For the exchange and correlation potential, the
PBE-GGA (Perdew et al., 1996) is chosen. The number of plane waves are limited
to 4818 plus 428 local orbitals. The full Brioullin-zone is divided into a
$13\times 13\times 1$ k-point mesh. For the atomic structure, we have
constructed a super-cell consisting of 2.5 anatase unit cells and 4.5
$LAlO_{3}$ unit cells. The periodic super-cell incorporates a $La$–$Ti$
terminated interface and an $Al$–$Ti$ terminated interface, although only the
first is of significance for this work.

### Image simulation

The STEM-EELS images are calculated using inelastic multislice simulations
(Löffler, 2013). The code incorporates a standard multislice algorithm
(Kirkland, 1998) which simulates the elastic electron propagation through the
sample. For the inelastic interaction between the probe beam and the sample
electrons, the mixed dynamic form factor (MDFF) (Löffler et al., 2013) is
calculated. The MDFF includes the atomic site dependent cross density of
states and the radial wave functions of the one-electron Kohn-Sham orbitals of
the corresponding electronic transition. Both are extracted from the DFT
calculation described in the previous section.

### Molecular dynamics

Molecular dynamics (MD) of an anatase supercell (15x15x8 unit cells) are
performed with and without an external applied electric field (0.1 Å) along
[001] crystallographic direction. MD simulations were performed with the
Large-scale Atomic/Molecular Massively Parallel Simulator (LAMMPS) Plimpton
(1995) using a modified embedded atom method potential Joost et al. (2015).
Energy minimisation was achieved by applying a conjugate gradient minimisation
with cell size optimisation, followed by a damped dynamics step Knez et al.
(2020).

## Data availability

Extra data is available from the corresponding author upon reasonable request.

## Code availability

The codes used for data analysis are available from the corresponding authors
upon reasonable request.

## Supplementary information

The Supplementary Information contains more information about

## Acknowledgments

The authors thank Martina Dienstleder for preparing the FIB-lamella and
Johannes Biskupek for additional Nanomill polishing. We thank also Thomas
Mairhofer for discussing the DPC experiments and Alessandro Barla for help
with EELS data interpretation. This project has received fundings from the
European Union’s Horizon 2020 Research and Innovation Programme (Grant No.
823717, project “ESTEEM3”) and the Austrian Science Fund (FWF) under grant
number I4309-N36 “Orbitalkartierung an Grenzflächen”. For the purpose of open
access, the authors have applied a CC BY public copyright license to any
Author Accepted Manuscript version arising from this submission. The authors
also acknowledge fundings by the Zukunftsfond Steiermark for purchasing the K2
camera and the Wirtschaftskammer Steiermark (WKO Stmk.) for providing
additional funds for maintenance.

## Author contributions

M.O. performed the electron microscopy experiments and data analysis. M.E.
performed the DFT calculations and STEM-EELS simulations. S.L, D.K. and G.K.
conceived the experiments and supervised the simulations and experiments. D.K
performed supporting atomistic simulations. S.K.C and P.O. grew the samples
and performed the pre-characterisation of the films. R.C. and P.O. contributed
to the discussion of results and to reviewing drafts. M.O. and D.K. wrote the
manuscript with the contribution from all the authors.

## Declarations

The authors declare that they have no competing interests.

## References

  * Ohtomo and Hwang [2004] A. Ohtomo and H. Y. Hwang.  A high–mobility electron gas at the $LaAlO_{3}$/$SrTiO_{3}$ heterointerface.  _Nature_ , 427(6973):423–426, 01 2004.  ISSN 1476-4687.  doi:10.1038/nature02308.
  * Hwang et al. [2012] H. Y. Hwang, Y. Iwasa, M. Kawasaki, B. Keimer, N. Nagaosa, and Y. Tokura.  Emergent phenomena at oxide interfaces.  _Nature Materials_ , 11(2):103–113, 02 2012.  ISSN 1476-4660.  doi:10.1038/nmat3223.
  * Yang et al. [2011] Zheng Yang, Changhyun Ko, and Shriram Ramanathan.  Oxide Electronics Utilizing Ultrafast Metal-Insulator Transitions.  _Annual Review of Materials Research_ , 41(1):337–367, 2011.  doi:10.1146/annurev-matsci-062910-100347.  URL https://doi.org/10.1146/annurev-matsci-062910-100347.
  * Cen et al. [2009] Cheng Cen, Stefan Thiel, Jochen Mannhart, and Jeremy Levy.  Oxide Nanoelectronics on Demand.  _Science_ , 323(5917):1026–1030, 2009.  doi:10.1126/science.1168294.  URL https://www.science.org/doi/abs/10.1126/science.1168294.
  * Park et al. [2010] J. W. Park, D. F. Bogorin, C. Cen, D. A. Felker, Y. Zhang, C. T. Nelson, C. W. Bark, C. M. Folkman, X. Q. Pan, M. S. Rzchowski, J. Levy, and C. B. Eom.  Creation of a two-dimensional electron gas at an oxide interface on silicon.  _Nature Communications_ , 1(1):94, 10 2010.  ISSN 2041-1723.  doi:10.1038/ncomms1096.  URL https://doi.org/10.1038/ncomms1096.
  * Nakagawa et al. [2006] Naoyuki Nakagawa, Harold Y. Hwang, and David A. Muller.  Why some interfaces cannot be sharp.  _Nature Materials_ , 5(3):204–209, 03 2006.  ISSN 1476-4660.  doi:10.1038/nmat1569.  URL https://doi.org/10.1038/nmat1569.
  * Thiel et al. [2006] S. Thiel, G. Hammerl, A. Schmehl, C. W. Schneider, and J. Mannhart.  Tunable Quasi-Two-Dimensional Electron Gases in Oxide Heterostructures.  _Science_ , 313(5795):1942–1945, 2006.  doi:10.1126/science.1131091.
  * Chen et al. [2010] Hanghui Chen, Alexie M. Kolpak, and Sohrab Ismail-Beigi.  Electronic and Magnetic Properties of $SrTiO_{3}$/$LaAlO_{3}$ Interfaces from First Principles.  _Advanced Materials_ , 22(26-27):2881–2899, 2010.  doi:https://doi.org/10.1002/adma.200903800.  URL https://onlinelibrary.wiley.com/doi/abs/10.1002/adma.200903800.
  * Bristowe et al. [2011] N. C. Bristowe, P. B. Littlewood, and Emilio Artacho.  Surface defects and conduction in polar oxide heterostructures.  _Phys. Rev. B_ , 83:205405, 5 2011.  doi:10.1103/PhysRevB.83.205405.  URL https://link.aps.org/doi/10.1103/PhysRevB.83.205405.
  * Siemons et al. [2007] Wolter Siemons, Gertjan Koster, Hideki Yamamoto, Walter A. Harrison, Gerald Lucovsky, Theodore H. Geballe, Dave H. A. Blank, and Malcolm R. Beasley.  Origin of Charge Density at $LaAlO_{3}$ on $SrTiO_{3}$ Heterointerfaces: Possibility of Intrinsic Doping.  _Phys. Rev. Lett._ , 98:196802, 05 2007.  doi:10.1103/PhysRevLett.98.196802.  URL https://link.aps.org/doi/10.1103/PhysRevLett.98.196802.
  * Schlom and Mannhart [2011] Darrell G. Schlom and Jochen Mannhart.  Interface takes charge over Si.  _Nature Materials_ , 10(3):168–169, 3 2011.  ISSN 1476-4660.  doi:10.1038/nmat2965.  URL https://doi.org/10.1038/nmat2965.
  * Bristowe et al. [2014] N C Bristowe, Philippe Ghosez, P B Littlewood, and Emilio Artacho.  The origin of two-dimensional electron gases at oxide interfaces: insights from theory.  _Journal of Physics: Condensed Matter_ , 26(14):143201, 3 2014.  doi:10.1088/0953-8984/26/14/143201.  URL https://dx.doi.org/10.1088/0953-8984/26/14/143201.
  * Chen et al. [2013] Y. Z. Chen, N. Bovet, F. Trier, D. V. Christensen, F. M. Qu, N. H. Andersen, T. Kasama, W. Zhang, R. Giraud, J. Dufouleur, T. S. Jespersen, J. R. Sun, A. Smith, J. Nygård, L. Lu, B. Büchner, B. G. Shen, S. Linderoth, and N. Pryds.  A high-mobility two-dimensional electron gas at the spinel/perovskite interface of $\gamma$-$Al_{2}O_{3}$/$SrTiO_{3}$.  _Nature Communications_ , 4(1):1371, 01 2013.  ISSN 2041-1723.  doi:10.1038/ncomms2394.  URL https://doi.org/10.1038/ncomms2394.
  * Salluzzo et al. [2009] M. Salluzzo, J. C. Cezar, N. B. Brookes, V. Bisogni, G. M. De Luca, C. Richter, S. Thiel, J. Mannhart, M. Huijben, A. Brinkman, G. Rijnders, and G. Ghiringhelli.  Orbital Reconstruction and the Two-Dimensional Electron Gas at the ${\mathrm{LaAlO}}_{3}/{\mathrm{SrTiO}}_{3}$ Interface.  _Phys. Rev. Lett._ , 102:166804, 04 2009.  doi:10.1103/PhysRevLett.102.166804.  URL https://link.aps.org/doi/10.1103/PhysRevLett.102.166804.
  * Drera et al. [2013] G. Drera, G. Salvinelli, A. Brinkman, M. Huijben, G. Koster, H. Hilgenkamp, G. Rijnders, D. Visentin, and L. Sangaletti.  Band offsets and density of Ti3+ states probed by x-ray photoemission on LaAlO3/SrTiO3 heterointerfaces and their LaAlO3 and SrTiO3 bulk precursors.  _Phys. Rev. B_ , 87:075435, 02 2013.  doi:10.1103/PhysRevB.87.075435.  URL https://link.aps.org/doi/10.1103/PhysRevB.87.075435.
  * Troglia et al. [2022] Alessandro Troglia, Chiara Bigi, Ivana Vobornik, Jun Fujii, Daniel Knez, Regina Ciancio, Goran Dražić, Marius Fuchs, Domenico Di Sante, Giorgio Sangiovanni, Giorgio Rossi, Pasquale Orgiani, and Giancarlo Panaccione.  Evidence of a 2D Electron Gas in a Single-Unit-Cell of Anatase $TiO_{2}$ (001).  _Advanced Science_ , 9(16):2105114, 2022.  doi:https://doi.org/10.1002/advs.202105114.
  * McKeown Walker et al. [2022] Siobhan McKeown Walker, Margherita Boselli, Emanuel A. Martínez, Stefano Gariglio, Flavio Y. Bruno, and Felix Baumberger.  A Laser-ARPES View of the 2D Electron Systems at LaAlO3/SrTiO3 and Al/SrTiO3 Interfaces.  _Advanced Electronic Materials_ , 8(7):2101376, 2022.  doi:https://doi.org/10.1002/aelm.202101376.  URL https://onlinelibrary.wiley.com/doi/abs/10.1002/aelm.202101376.
  * Strocov et al. [2022] VN Strocov, F Lechermann, A Chikina, F Alarab, LL Lev, VA Rogalev, T Schmitt, and MA Husanu.  Dimensionality of mobile electrons at x-ray-irradiated laalo3/srtio3 interfaces.  _Electronic Structure_ , 4(1):015003, 2022.  doi:https://doi.org/10.1088/2516-1075/ac4e74.
  * Bigi et al. [2020] Chiara Bigi, Zhenkun Tang, Gian Marco Pierantozzi, Pasquale Orgiani, Pranab Kumar Das, Jun Fujii, Ivana Vobornik, Tommaso Pincelli, Alessandro Troglia, Tien-Lin Lee, Regina Ciancio, Goran Drazic, Alberto Verdini, Anna Regoutz, Phil D. C. King, Deepnarayan Biswas, Giorgio Rossi, Giancarlo Panaccione, and Annabella Selloni.  Distinct behavior of localized and delocalized carriers in anatase $TiO_{2}$ (001) during reaction with $O_{2}$.  _Physical Review Materials_ , 4(2):025801, 2020.  ISSN 2475-9953.  doi:10.1103/PhysRevMaterials.4.025801.  URL https://link.aps.org/doi/10.1103/PhysRevMaterials.4.025801.
  * Huang et al. [2012] Bo-Chao Huang, Ya-Ping Chiu, Po-Cheng Huang, Wen-Ching Wang, Vu Thanh Tra, Jan-Chi Yang, Qing He, Jiunn-Yuan Lin, Chia-Seng Chang, and Ying-Hao Chu.  Mapping Band Alignment across Complex Oxide Heterointerfaces.  _Phys. Rev. Lett._ , 109:246807, 12 2012.  doi:10.1103/PhysRevLett.109.246807.  URL https://link.aps.org/doi/10.1103/PhysRevLett.109.246807.
  * Knez et al. [2020] Daniel Knez, Goran Dražić, Sandeep Kumar Chaluvadi, Pasquale Orgiani, Stefano Fabris, Giancarlo Panaccione, Giorgio Rossi, and Regina Ciancio.  Unveiling Oxygen Vacancy Superstructures in Reduced Anatase Thin Films.  _Nano letters_ , 20(9):6444–6451, 2020.  ISSN 1530-6992.  doi:10.1021/acs.nanolett.0c02125.
  * Maurice et al. [2008] Jean-Luc Maurice, Gervasi Herranz, Christian Colliex, Isabelle Devos, Cécile Carrétéro, Agnès Barthelemy, Karim Bouzehouane, Stéphane Fusil, Dominique Imhoff, Éric Jacquet, François Jomard, Dominique Ballutaud, and Mario Basletic.  Electron energy loss spectroscopy determination of Ti oxidation state at the (001) LaAlO3/SrTiO3 interface as a function of LaAlO3 growth conditions.  _EPL - Europhysics Letters_ , 82:17003, 04 2008.  doi:10.1209/0295-5075/82/17003.  URL https://hal.science/hal-00193115.
  * Löffler et al. [2017] Stefan Löffler, Matthieu Bugnet, Nicolas Gauquelin, Sorin Lazar, Elias Assmann, Karsten Held, Gianluigi A. Botton, and Peter Schattschneider.  Real-space mapping of electronic orbitals.  _Ultramicroscopy_ , 177:26–29, 2017.  ISSN 0304-3991.  doi:https://doi.org/10.1016/j .ultramic.2017.01.018.
  * Bugnet et al. [2022] M. Bugnet, M. Ederer, V. K. Lazarov, L. Li, Q. M. Ramasse, S. Löffler, and D. M. Kepaptsoglou.  Imaging the Spatial Distribution of Electronic States in Graphene Using Electron Energy-Loss Spectroscopy: Prospect of Orbital Mapping.  _Phys. Rev. Lett._ , 128:116401, 03 2022.  doi:10.1103/PhysRevLett.128.116401.
  * Wadati et al. [2008] Hiroki Wadati, Yasushi Hotta, Atsushi Fujimori, Tomofumi Susaki, Harold Y. Hwang, Yasutaka Takata, Koji Horiba, Masaharu Matsunami, Shik Shin, Makina Yabashi, Kenji Tamasaku, Yoshinori Nishino, and Tetsuya Ishikawa.  Hard x-ray photoemission study of $LaAlO_{3}$/$LaVO_{3}$ multilayers.  _Physical Review B_ , 77(4), 2008.  ISSN 1098-0121.  doi:10.1103/PhysRevB.77.045122.
  * Mirjolet et al. [2021] Mathieu Mirjolet, Hari Babu Vasili, Adrian Valadkhani, José Santiso, Vladislav Borisov, Pierluigi Gargiani, Manuel Valvidares, Roser Valentí, and Josep Fontcuberta.  Orbital occupancy and hybridization in strained $SrVO_{3}$ epitaxial films.  _Physical Review Materials_ , 5(9), 2021.  ISSN 2475-9953.  doi:10.1103/PhysRevMaterials.5.095002.
  * Gobaut et al. [2017] Benoit Gobaut, Pasquale Orgiani, Alessia Sambri, Emiliano di Gennaro, Carmela Aruta, Francesco Borgatti, Valerio Lollobrigida, Denis Céolin, Jean-Pascal Rueff, Regina Ciancio, Chiara Bigi, Pranab Kumar Das, Jun Fujii, Damjan Krizmancic, Piero Torelli, Ivana Vobornik, Giorgio Rossi, Fabio Miletto Granozio, Umberto Scotti di Uccio, and Giancarlo Panaccione.  Role of Oxygen Deposition Pressure in the Formation of Ti Defect States in $TiO_{2}(001)$ Anatase Thin Films.  _ACS Applied Materials & Interfaces_, 9(27):23099–23106, 2017.  doi:10.1021/acsami.7b03181.
  * Cheng et al. [2017] Jianli Cheng, Jian Luo, and Kesong Yang.  Comparison Studies of Interfacial Electronic and Energetic Properties of $LaAlO_{3}$/$TiO_{2}$ and $TiO_{2}$/$LaAlO_{3}$ Heterostructures from First-Principles Calculations.  _ACS applied materials & interfaces_, 9(8):7682–7690, 2017.  ISSN 1944-8252.  doi:10.1021/acsami.6b12254.
  * Minohara et al. [2014] Makoto Minohara, Takashi Tachikawa, Yasuo Nakanishi, Yasuyuki Hikita, Lena F. Kourkoutis, Jun-Sik Lee, Chi-Chang Kao, Masahiro Yoshita, Hidefumi Akiyama, Christopher Bell, and Harold Y. Hwang.  Atomically Engineered Metal–Insulator Transition at the $TiO_{2}$/$LaAlO_{3}$ Heterointerface.  _Nano Letters_ , 14(11):6743–6746, 2014.  doi:10.1021/nl5039192.
  * Islam et al. [2022] Mahabul Islam, Piu Rajak, Daniel Knez, Sandeep Kumar Chaluvadi, Pasquale Orgiani, Giorgio Rossi, Goran Dražić, and Regina Ciancio.  HAADF STEM and Ab Initio Calculations Investigation of Anatase $TiO_{2}$/$LaAlO_{3}$ Heterointerface.  _Applied Sciences_ , 12(3), 2022.  ISSN 2076-3417.  doi:10.3390/app12031489.
  * Wang et al. [2010] Zhongchang Wang, Wen Zeng, Lin Gu, Mitsuhiro Saito, Susumu Tsukimoto, and Yuichi Ikuhara.  Atomic-scale structure and electronic property of the $LaAlO_{3}/TiO_{2}$ interface.  _Journal of Applied Physics_ , 108(11):113701, 2010.  doi:10.1063/1.3516496.
  * Krüger [2010] Peter Krüger.  Multichannel multiple scattering calculation of ${L}_{2,3}$-edge spectra of $TiO_{2}$ and $SrTiO3$: Importance of multiplet coupling and band structure.  _Phys. Rev. B_ , 81:125121, Mar 2010.  doi:10.1103/PhysRevB.81.125121.  URL https://link.aps.org/doi/10.1103/PhysRevB.81.125121.
  * Kucheyev et al. [2004] S. O. Kucheyev, T. van Buuren, T. F. Baumann, J. H. Satcher, T. M. Willey, R. W. Meulenberg, T. E. Felter, J. F. Poco, S. A. Gammon, and L. J. Terminello.  Electronic structure of titania aerogels from soft x-ray absorption spectroscopy.  _Phys. Rev. B_ , 69:245102, 06 2004.  doi:10.1103/PhysRevB.69.245102.  URL https://link.aps.org/doi/10.1103/PhysRevB.69.245102.
  * Frati et al. [2020] Federica Frati, Myrtille O. J. Y. Hunault, and Frank M. F. de Groot.  Oxygen K-edge X-ray Absorption Spectra.  _Chemical Reviews_ , 120(9):4056–4110, 05 2020.  ISSN 0009-2665.  doi:10.1021/acs.chemrev.9b00439.
  * Groot [2005] Frank De Groot.  Multiplet effects in X-ray spectroscopy.  _Coordination Chemistry Reviews_ , 249(1):31–63, 2005.  ISSN 0010-8545.  doi:https://doi.org/10.1016/j.ccr.2004.03.018.
  * Pauli and Willmott [2008] S A Pauli and P R Willmott.  Conducting interfaces between polar and non-polar insulating perovskites.  _Journal of Physics: Condensed Matter_ , 20(26):264012, 06 2008.  doi:10.1088/0953-8984/20/26/264012.  URL https://dx.doi.org/10.1088/0953-8984/20/26/264012.
  * MacLaren et al. [2015] Ian MacLaren, LiQiu Wang, Damien McGrouther, Alan J. Craven, Stephen McVitie, Roland Schierholz, András Kovács, Juri Barthel, and Rafal E. Dunin-Borkowski.  On the origin of differential phase contrast at a locally charged and globally charge-compensated domain boundary in a polar-ordered material.  _Ultramicroscopy_ , 154:57–63, 2015.  ISSN 0304-3991.  doi:https://doi.org/10.1016/j.ultramic.2015.03.016.  URL https://www.sciencedirect.com/science/article/pii/S0304399115000650.
  * Lazić et al. [2016] Ivan Lazić, Eric G.T. Bosch, and Sorin Lazar.  Phase contrast STEM for thin samples: Integrated differential phase contrast.  _Ultramicroscopy_ , 160:265–280, 2016.  ISSN 0304-3991.  doi:https://doi.org/10.1016/j.ultramic.2015.10.011.  URL https://www.sciencedirect.com/science/article/pii/S0304399115300449.
  * Nord et al. [2017] Magnus Nord, Per Erik Vullum, Ian MacLaren, Thomas Tybell, and Randi Holmestad.  Atomap: a new software tool for the automated analysis of atomic resolution images using two-dimensional Gaussian fitting.  _Advanced Structural and Chemical Imaging_ , 3:9, 2017.  ISSN 2198-0926.  doi:10.1186/s40679-017-0042-5.
  * Kalabukhov et al. [2007] Alexey Kalabukhov, Robert Gunnarsson, Johan Börjesson, Eva Olsson, Tord Claeson, and Dag Winkler.  Effect of oxygen vacancies in the $SrTiO_{3}$ substrate on the electrical properties of the $LaAlO_{3}$/$SrTiO_{3}$ interface.  _Phys. Rev. B_ , 75:121404, 03 2007.  doi:10.1103/PhysRevB.75.121404.
  * Lee and Selloni [2022] Taehun Lee and Annabella Selloni.  Structure and Stability of Oxygen Vacancy Aggregates in Reduced Anatase and Rutile $TiO_{2}$.  _Journal of Physical Chemistry C_ , 2022.  ISSN 1932-7447.  doi:10.1021/acs.jpcc.2c06806.
  * Zhang and Dong [2019] Daoyu Zhang and Shuai Dong.  Challenges in band alignment between semiconducting materials: A case of rutile and anatase $TiO_{2}$.  _Progress in Natural Science: Materials International_ , 29(3):277–284, 2019.  ISSN 10020071.  doi:10.1016/j. pnsc.2019.03.012.
  * Xu and LeBeau [2018] W. Xu and J.M. LeBeau.  A deep convolutional neural network to analyze position averaged convergent beam electron diffraction patterns.  _Ultramicroscopy_ , 188:59–69, 2018.  ISSN 0304-3991.  doi:https://doi.org/10.1016/j .ultramic.2018.03.004.  URL https://www.sciencedirect.com/science/article/pii/S0304399117303340.
  * Oberaigner et al. [2023] Michael Oberaigner, Alexander Clausen, Dieter Weber, Gerald Kothleitner, Rafal E Dunin-Borkowski, and Daniel Knez.  Online Thickness Determination with Position Averaged Convergent Beam Electron Diffraction using Convolutional Neural Networks.  _Microscopy and Microanalysis_ , 29(1):427–436, 01 2023.  ISSN 1431-9276.  doi:10.1093/micmic/ozac050.
  * Blaha et al. [2020] Peter Blaha, Karlheinz Schwarz, Fabien Tran, Robert Laskowski, Georg K. H. Madsen, and Laurence D. Marks.  WIEN2k: An APW+lo program for calculating the properties of solids.  _The Journal of Chemical Physics_ , 152(7):074101, 2020.  doi:10.1063/1.5143061.  URL https://doi.org/10.1063/1.5143061.
  * Perdew et al. [1996] John P. Perdew, Kieron Burke, and Matthias Ernzerhof.  Generalized Gradient Approximation Made Simple.  _Phys. Rev. Lett._ , 77:3865–3868, Oct 1996.  doi:10.1103/PhysRevLett.77.3865.  URL https://link.aps.org/doi/10.1103/PhysRevLett.77.3865.
  * Löffler [2013] Stefan Löffler.  _Study of Real Space Wave Functions with Electron Energy Loss Spectrometry_.  PhD thesis, Vienna University of Technology, 2013.
  * Kirkland [1998] E.J. Kirkland.  _Advanced Computing in Electron Microscopy_.  Springer New York, NY, 1998.  doi:10.1007/978-1-4757-4406-4.
  * Löffler et al. [2013] S. Löffler, V. Motsch, and P. Schattschneider.  A pure state decomposition approach of the mixed dynamic form factor for mapping atomic orbitals.  _Ultramicroscopy_ , 131:39–45, 2013.  ISSN 0304-3991.  doi:https://doi.org/10.1016/j.ultramic.2013.03.021.  URL https://www.sciencedirect.com/science/article/pii/S0304399113000910.
  * Plimpton [1995] Steve Plimpton.  Fast Parallel Algorithms for Short-Range Molecular Dynamics.  _Journal of Computational Physics_ , 117(1):1–19, 1995.  ISSN 00219991.  doi:10.1006/jcph.1995.1039.
  * Joost et al. [2015] W. J. Joost, S. Ankem, and M. M. Kuklja.  A modified embedded atom method potential for the titanium–oxygen system.  _Modelling and Simulation in Materials Science and Engineering_ , 23(1):015006, 2015.  ISSN 0965-0393.  doi:10.1088/0965-0393/23/1/015006.