# SpeechCLIP: Integrating Speech
with Pre-Trained Vision and Language Model

###### Abstract

Data-driven speech processing models usually perform well with a large amount
of text supervision, but collecting transcribed speech data is costly.
Therefore, we propose SpeechCLIP, a novel framework bridging speech and text
through images to enhance speech models without transcriptions. We leverage
state-of-the-art pre-trained HuBERT and CLIP, aligning them via paired images
and spoken captions with minimal fine-tuning. SpeechCLIP outperforms prior
state-of-the-art on image-speech retrieval and performs zero-shot speech-text
retrieval without direct supervision from transcriptions. Moreover, SpeechCLIP
can directly retrieve semantically related keywords from speech.

Index Terms—  Visual grounding, vision and language, self-supervised learning

(0cm,11cm) Copyright 2022 IEEE. Published in the 2022 IEEE Spoken Language
Technology Workshop (SLT) (SLT 2022), scheduled for 19-22 January 2023 in
Doha, Qatar. Personal use of this material is permitted. However, permission
to reprint/republish this material for advertising or promotional purposes or
for creating new collective works for resale or redistribution to servers or
lists, or to reuse any copyrighted component of this work in other works, must
be obtained from the IEEE. Contact: Manager, Copyrights and Permissions / IEEE
Service Center / 445 Hoes Lane / P.O. Box 1331 / Piscataway, NJ 08855-1331,
USA. Telephone: + Intl. 908-562-3966.

##  1 Introduction

Conventionally, speech processing tasks like speech recognition need
transcribed speech data for machine learning. They usually require large
labeled datasets to perform well, but transcribing an enormous amount of
speech is expensive. Therefore, recent studies exploit unlabeled speech to
pre-train models with self-supervised learning (SSL) [1]. Models learn to
predict pseudo targets generated from raw data in SSL pre-training. Some
typical speech SSL methods include masked reconstruction [2, 3, 4, 5, 6],
contrastive learing [7, 8, 9, 10, 11], classification [12, 13, 14], multi-task
learning [15], and knowledge distillation [16, 17, 18]. These methods succeed
in a wide range of speech processing problems [19, 20, 21].

Besides SSL methods focusing on a single modality, researchers propose using
data from other modalities to boost machine performance on a specific
modality. E.g., pairing images with semantically related text or spoken
captions is a typical method since collecting parallel image-text or image-
speech data is fast and inexpensive [22]. Specifically, paired image-text data
can be obtained by crawling images and captions from the internet. Paired
image-speech data can be collected by uttering text captions or describing
images in speech. This paper uses paired image-speech data and an image-text
pre-trained model to enhance speech SSL models.

Fig. 1:  An overview of the proposed SpeechCLIP model.

(a) Parallel SpeechCLIP

(b) Cascaded SpeechCLIP

Fig. 2:  An illustration of SpeechCLIP models. (a) A pre-trained HuBERT [12]
extracts audio features. The features are concatenated with a learnable CLS
token and fed into a transformer encoder layer to obtain a single vector
representing the information of the entire sequence. The vector is then used
to compute contrastive loss with the CLIP image encoder’s output [23]. (b)
Cascaded SpeechCLIP uses $K$ CLS tokens to capture a small sequence of
keywords from the audio signal. The keywords are batch-normalized and vector-
quantized before passing to the CLIP text encoder. BN and VQ respectively
denote batch normalization and vector quantization.

Much effort was put into using paired images and spoken captions to help
speech processing [24], and they are usually called visually grounded speech
models (VGS). VGS models benefit many applications like speech recognition
[25], word discovery [26], speech generation [27], cross-modal alignment [22,
28, 29], and multilingual spoken language processing [30, 31, 32, 33]. Most
studies pre-train and evaluate VGS models on image-speech retrieval, showing
the capabilities of capturing the correspondence between images and speech
[34, 35]. E.g., the recent Fast-Slow Transformer for Visually Grounding Speech
(FaST-VGS and FaST-VGS+) succeeds in many speech processing tasks by utilizing
transformers and cross-modal attention mechanisms to perform image-speech
retrieval and semantic tasks [36, 37]. Moreover, VGS models trained with
retrieval objectives can extract semantic and word-level information from
speech [38], which is difficult to achieve by training solely with speech
[39].

While many studies obtain semantic information from speech without
transcriptions, some extent of assistance from text could be helpful for some
tasks. E.g., recent unsupervised ASR methods rely on nonparallel text data and
a pronunciation lexicon [40, 41]. To circumvent transcriptions or lexicons, we
propose to bridge speech and text domains via images, i.e., taking advantage
of paired image-speech and image-text data. Thus, this paper introduces
SpeechCLIP, a novel framework to integrate speech SSL models with a pre-
trained vision and language model as depicted in Fig. 1. We use Contrastive
Language-Image Pre-training (CLIP), a powerful model pre-trained to align
parallel image-text data [23]. Then, a speech encoder initialized by a pre-
trained speech SSL model is enhanced by aligning with CLIP using paired image-
speech data. By aligning a speech encoder’s and CLIP’s image embedding spaces,
the speech encoder is implicitly aligned with CLIP’s text encoder, forcing it
to capture more textual content.

We propose two SpeechCLIP architectures: parallel and cascaded. The parallel
model is similar to WAV2CLIP [42]. However, our speech encoder uses a pre-
trained speech SSL model and focuses on capturing local and global spoken
contents. Meanwhile, WAV2CLIP extracts global features in general audio for
classification and retrieval. Furthermore, AudioCLIP is an extension of
WAV2CLIP since it is trained with paired image, audio, and text data [43]. The
cascaded SpeechCLIP cascades CLIP’s text encoder on top of the speech encoder,
forcing the model to output subword embeddings. Eventually, the cascaded model
captures spoken words in speech signals.

In this paper, the proposed SpeechCLIP models achieve state-of-the-art image-
speech retrieval on two standard spoken caption datasets with minimal fine-
tuning. Moreover, we demonstrate SpeechCLIP’s capability of performing zero-
shot speech-text retrieval and capturing keywords directly from speech. We
also make our code available on
Github111https://github.com/atosystem/SpeechCLIP.

##  2 Method

###  2.1 Preliminaries

We briefly explain pre-trained models used in SpeechCLIP.

Contrastive Language-Image Pre-training (CLIP) [23]. CLIP uses contrastive
learning to pre-train visual models from natural language supervision on an
enormous scale, where the supervision comes from paired image-text data.
Composing two encoders processing image and text separately, CLIP aims to
align semantically similar images and text captions. CLIP can easily transfer
across various computer vision tasks with little supervision.

Hidden-unit BERT (HuBERT) [12]. HuBERT is a speech SSL method similar to
masked language modeling, predicting labels generated by clustered acoustic
features. HuBERT comprises a CNN feature extractor followed by a transformer
encoder [44] and offers good initialization for many speech processing tasks
[19, 21].

In SpeechCLIP, pre-trained CLIP and HuBERT models are frozen and serve as
feature extractors, as shown in Fig. 2. The CLIP model extracts image and
sentence embeddings to supervise SpeechCLIP. Following SUPERB [19], HuBERT’s
CNN output and transformer encoder’s hidden representations are weighted and
summed by a set of learnable weights. The weights automatically assign
importance to each hidden layer to minimize the overall objective function.
Only the newly added components excluding HuBERT and CLIP are learnable during
training, reducing the computational cost significantly, thus enabling a
larger batch size for contrastive pre-training. In the following sections, we
introduce two SpeechCLIP architectures: parallel and cascaded.

###  2.2 Parallel SpeechCLIP

Parallel SpeechCLIP is similar to CLIP, which aligns semantically related
images and spoken captions, as shown in Fig. 2(a). Since the weighted sum of
HuBERT’s output is a sequence of frame-level features, we add a learnable CLS
token at the beginning of each sequence. The sequence is passed through a
transformer encoder layer to obtain an utterance-level representation [44].
The representation is used to compute the cosine similarity with image
embeddings in a mini-batch for calculating the contrastive loss. Cosine
similarity scores are also used for retrieving speech and image samples.
Following CLIP, the loss function has a learnable temperature for scaling the
similarity scores.

By aligning speech and CLIP image encoders, parallel SpeechCLIP implicitly
bridges speech and text representations since CLIP’s image and text encoders
are well-aligned. Therefore, it can perform both image-speech and speech-text
retrieval. Still, this method is limited to summarizing utterances because it
has no explicit constraints to capture word-level content. Thus, the following
section introduces a novel method addressing this issue.

###  2.3 Cascaded SpeechCLIP

To force the speech encoder to capture semantic information from speech, we
propose cascaded SpeechCLIP by cascading speech encoder with CLIP’s text
encoder as shown in Fig. 2(b). Following parallel SpeechCLIP, the cascaded
model is trained with contrastive loss, but the difference lies in the
summarization process of utterances.

First, we add $K$ learnable CLS tokens at the beginning of an audio feature
sequence, where $K$ is a hyper-parameter for the number of keywords obtained
from an utterance. The sequence is fed into a transformer encoder and
projected to the CLIP input embedding dimension. Next, the projected CLS
tokens are batch-normalized to match the mean and variance of CLIP’s subword
embeddings. We apply vector quantization (VQ) to map the $K$ normalized
embeddings to CLIP’s $V$ subword embeddings. This operation produces keywords
indicating the essential concepts in each utterance.

The VQ process is described as follows. We first compute the cosine similarity
between the $k^{\text{th}}$ normalized CLS embedding ($\boldsymbol{z}_{k}$)
and the $v^{\text{th}}$ subword embedding ($\boldsymbol{e}_{v}$) as

$\vspace{-3pt}s_{kv}=\text{cos}\left(\boldsymbol{z}_{k},\boldsymbol{e}_{v}\right).$
(1)

Next, we choose the subword embedding with the highest similarity from the
vocabulary, which can be expressed as

$\vspace{-3pt}\boldsymbol{e}_{v^{\star}},\text{~{}where~{}}v^{\star}=\underset{1\leq
v\leq V}{\text{argmax}}~{}s_{kv}.$ (2)

Since $\boldsymbol{e}_{v^{\star}}$ is not differentiable, we compute another
embedding by weighted summing all $V$ subword embeddings as

$\vspace{-3pt}\overline{\boldsymbol{h}}_{k}=\left[\boldsymbol{e}_{1}\dots\boldsymbol{e}_{V}\right]\text{softmax}\left(\left[s_{k1}\dots
s_{kV}\right]^{\top}/\tau\right),$ (3)

where each embedding $\boldsymbol{e}_{v}$ is a column vector and $\tau$ is a
hyper-parameter ($\tau=0.1$). Combining Eqs. 2 and 3, we apply straight-
through gradient estimator [45] to obtain quantized keywords

$\vspace{-3pt}\boldsymbol{h}_{k}=\boldsymbol{e}_{v^{\star}}+\overline{\boldsymbol{h}}_{k}-\text{sg}\left(\overline{\boldsymbol{h}}_{k}\right),$
(4)

where $\text{sg}(x)=x$ and $\frac{d}{dx}\text{sg}(x)=0$ is the stop gradient
operator. The $K$ keywords are then fed into the CLIP text encoder for
computing the contrastive objective.

Overall, the cascaded SpeechCLIP encourages the speech encoder to extract
subwords because of the supervision from the CLIP text encoder. Hence, it is
expected to capture more semantic and content information from speech.

##  3 Experiment

Table 1:  Model details. The number of parameters varies since they include parallel and cascaded models.  | Audio | CLIP Image | Trainable | Total
---|---|---|---|---
Model | Encoder | Encoder | Params (M) | Params (M)
Base | HuBERT Base | ViT-B/32 | 2.8 – 7.5 | 252 – 257
(95 M) | (250 M)
Large | HuBERT Large | ViT-L/14 | 6.1 – 13.4 | 765 – 772
(316 M) | (422 M)

###  3.1 Setup

Dataset. SpeechCLIP is pre-trained and evaluated with retrieval on Flickr8k
Audio Captions Corpus [26] and SpokenCOCO dataset [27]. Each image in both
datasets is paired with five spoken captions produced by humans uttering text
captions. Flickr8k consists of 8k images and 46 hours of speech, while
SpokenCOCO has 123k images and 742 hours of speech. Following FaST-VGS, we use
the Karpathy split for SpokenCOCO [46].

Model. We implemented SpeechCLIP in two sizes: Base and Large, a detailed
comparison is shown in Table 1. Note that we omit the Base notation in the
following sections. The hidden dimension of the transformer encoder is the
same as that of the audio encoder. The feed-forward network in the cascaded
model’s transformer encoder is removed for better performance. Parallel and
cascaded models have respectively eight and one attention head. We set $K$ to
8 in all experiments. All models are trained with Adam optimizer with a weight
decay of 10${}^{-\text{6}}$, batch size of 256, and 50k steps in total. The
learning rate linearly increases to 10${}^{-\text{4}}$ in the first 5k steps
and linearly decreases to 10${}^{-\text{8}}$ afterward. All experiments are
conducted on a 32GB V100 GPU except for pre-training on SpokenCOCO, which uses
two. The largest model’s pre-training lasts approximately two days.

Table 2:  Recall scores for image-speech retrieval on Flickr8k and SpokenCOCO testing sets.  |  Speech $\rightarrow$ Image |  Image $\rightarrow$ Speech
---|---|---
Method | R@1 | R@5 | R@10 | R@1 | R@5 | R@10
| Flickr8k
FaST-VGS${}_{\text{CO}}$ [36] | 26.6 | 56.4 | 68.8 | 36.2 | 66.1 | 76.5
FaST-VGS${}_{\text{CTF}}$ [36] | 29.3 | 58.6 | 71.0 | 37.9 | 68.5 | 79.9
MILAN [35] | 33.2 | 62.7 | 73.9 | 49.6 | 79.2 | 87.5
Parallel | 26.7 | 57.1 | 70.0 | 41.3 | 73.9 | 84.2
Cascaded | 8.2 | 25.7 | 37.2 | 14.1 | 34.5 | 49.2
Parallel Large | 39.1 | 72.0 | 83.0 | 54.5 | 84.5 | 93.2
Cascaded Large | 14.7 | 41.2 | 55.1 | 21.8 | 52.0 | 67.7
| SpokenCOCO
ResDAVEnet [25] | 17.3 | 41.9 | 55.0 | 22.0 | 50.6 | 65.2
FaST-VGS${}_{\text{CO}}$ [36] | 31.8 | 62.5 | 75.0 | 42.5 | 73.7 | 84.9
FaST-VGS${}_{\text{CTF}}$ [36] | 35.9 | 66.3 | 77.9 | 48.8 | 78.2 | 87.0
Parallel Large | 35.8 | 66.5 | 78.0 | 50.6 | 80.9 | 89.1
Cascaded Large | 6.4 | 20.7 | 31.0 | 9.6 | 27.7 | 39.7

###  3.2 Image-Speech Retrieval

In this section, we evaluate SpeechCLIP on the image-speech retrieval task,
showing how well models can align speech with CLIP image embeddings. As shown
in Table 2, parallel SpeechCLIP models surpass almost all baseline methods,
especially for the Large SpeechCLIP models. The parallel Base model on
Flickr8k also shows competitive performance with the FaST-VGS ${}_{\text{CO}}$
model, indicating that utilizing powerful pre-trained models and a small set
of learnable parameters is sufficient. Moreover, the cascaded models obtain
the lowest recall scores because passing encoded speech through VQ and a CLIP
text encoder loses information. Overall, the results show the benefits of
integrating CLIP in VGS models even with minimal fine-tuning.

Table 3:  Recall for speech-text retrieval on Flickr8k and SpokenCOCO. ‘Sup.” indicates the supervised version of parallel SpeechCLIP by replacing the image encoder with CLIP text encoder in parallel SpeechCLIP.  |  Speech $\rightarrow$ Text |  Text $\rightarrow$ Speech
---|---|---
Method | R@1 | R@5 | R@10 | R@1 | R@5 | R@10
| Flickr8k
Random | 0.10 | 0.50 | 0.99 | 0.10 | 0.50 | 0.99
Parallel Large | 19.56 | 44.06 | 58.46 | 22.50 | 44.14 | 54.54
Parallel Large (Sup.) | 97.06 | 99.24 | 99.46 | 97.88 | 99.76 | 99.90
| SpokenCOCO
Random | 0.02 | 0.10 | 0.20 | 0.02 | 0.10 | 0.20
Parallel Large | 60.32 | 81.81 | 88.18 | 65.45 | 85.82 | 91.27
Parallel Large (Sup.) | 95.02 | 99.46 | 99.78 | 95.35 | 99.68 | 99.93

###  3.3 Zero-shot Speech-Text Retrieval

This section highlights parallel SpeechCLIP’s capability to perform zero-shot
speech-text retrieval. Speech and text representations are respectively
computed from a pre-trained parallel SpeechCLIP’s speech encoder and a CLIP
text encoder. The representations are then used to calculate cosine similarity
scores for retrieval. Although this problem has been studied for a while,
prior studies require either paired speech-text training data [47, 48] or
pretrained image tagger [49].

Additionally, two supervised parallel SpeechCLIP models respectively trained
with paired spoken and text captions in Flickr8k and SpokenCOCO are considered
as toplines. These models’ CLIP image encoders are replaced with CLIP text
encoders to align speech and text explicitly. When computing recall, we regard
retrieving speech and text captions related to the same image as successful.
Therefore, results only show whether models retrieve semantically related
samples, not exact matching of speech and transcriptions.

According to Table 3, proposed SpeechCLIP models yield considerably better
performance than random retrieval, showing that speech and text embedding
spaces are well aligned. Specifically, parallel SpeechCLIP performs better on
this task when trained on a larger dataset like SpokenCOCO. Although the
performance gap between the proposed methods and the supervised toplines
remains, we show that bridging speech and text with image is possible and
promising.

We demonstrate that parallel SpeechCLIP retrieves noisy transcriptions for
speech signals. These transcriptions can then be used for supervised or semi-
supervised speech recognition model training. Furthermore, by replacing CLIP
with Multilingual-CLIP222https://github.com/FreddeFrallan/Multilingual-CLIP,
we can retrieve noisy transcriptions of different languages, thus performing
speech translation.

###  3.4 Keyword Retrieval with Cascaded SpeechCLIP

Fig. 3:  Demonstration of a cascaded SpeechCLIP Large model retrieving words using its CLS tokens’ outputs. The two utterances are from the SpokenCOCO test set. For each keyword in each sample, we show the transformer encoder’s attention map over the whole sequence and the retrieved subwords on the right and sorted in decreasing cosine similarity. Subwords in boldface indicate they exist in the ground truth caption.  Table 4:  Keyword hit rates for cascaded SpeechCLIP. Avg denotes averaged hit rate. † and ‡ respectively denote models trained on Flickr8k and SpokenCOCO.  Model | kw1 | kw2 | kw3 | kw4 | kw5 | kw6 | kw7 | kw8 | Avg
---|---|---|---|---|---|---|---|---|---
Base† | 57.0 | 25.6 | 20.2 | 5.0 | 20.0 | 26.5 | 10.5 | 16.6 | 22.7
Large† | 56.5 | 19.6 | 20.5 | 37.5 | 21.7 | 34.6 | 26.4 | 44.7 | 32.7
Large‡ | 27.5 | 22.4 | 35.8 | 61.0 | 21.6 | 54.2 | 60.1 | 22.9 | 38.2

Due to the unique design of cascaded SpeechCLIP, we investigate what and how
well the speech encoder extracts keywords. For each encoded and normalized CLS
token $\boldsymbol{z}_{k}$, keywords are retrieved by finding subwords with
the highest cosine similarities between $\boldsymbol{z}_{k}$ and the
corresponding subword embeddings. Notice that previous works [49, 50] are also
capable of retrieving semantically related keywords from speech. Nonetheless,
they required pretrained image tagger and the size of keywords set is very
limited. For SpeechCLIP, we can apply the same method to other pretrained
langange models’ vocabulary, technically. Also, our setting is quite different
from [51], where the 8 keywords are discovered from speech utterance without
any text query in our work. Namely, SpeechCLIP can automatically summarize the
speech by selecting 8 keywords. We offer quantitative and qualitative analyses
in the following paragraphs.

We inspect how well keywords are retrieved from speech signals for the
quantitative analysis. The evaluation metric is hit rate, which is the
percentage of successful top-1 keyword retrieval of any word in the caption
averaged over all testing samples. In Table 4, some CLS tokens frequently
retrieve words in the ground truth captions, showing that the cascaded
architecture can directly capture words from speech. Moreover, the first
keyword’s hit rate for models trained on Flickr8k is relatively high compared
to other keywords. Probably because the first word in a sentence has a higher
chance to be “a”, which is also the top-1 commonly retrieved subword from the
first keyword in Flickr8k. Another finding is that the Large model obtains a
higher averaged keyword hit rate than the Base model on Flickr8k, which is
consistent with the trend in Table 2. Hence, retrieving correct keywords is
related to retrieving between speech and image samples. Although some CLS
tokens obtain reasonable hit rates, one might question whether the retrieved
words are meaningful instead of stopwords. Hence, we next analyze the results
qualitatively to address this concern.

For the qualitative analysis, we offer two samples from the SpokenCOCO testing
set in Fig. 3, showing their attention maps in the transformer encoder and
retrieved words for each CLS token. In the first example, although only a few
retrieved keywords are in the ground truth caption, some semantically related
words are found. For instance, attention maps of keywords 1, 2, and 6 focus on
segments uttering “tie” and “suit.” Meanwhile, they retrieve words related to
clothes and appearance, e.g., “dapper”, “tuxedo”, and “scarf.” A similar trend
can be found in the second sample, showing that the cascaded objective makes
the speech encoder captures semantic information. Moreover, looking at both
examples, each keyword seems to have a particular purpose, e.g., the 8th
keyword tends to retrieve specific nouns from utterances while the 7th
retrieves prepositions. This observation leads us to investigate the
properties of each keyword.

Table 5:  Top 10 successfully retrieved subwords for each keyword on SpokenCOCO test set using the cascaded Large model. The subwords are sorted in decreasing occurrence. kw1 | kw2 | kw3 | kw4 | kw5 | kw6 | kw7 | kw8
---|---|---|---|---|---|---|---
a | cat | bathroom | a | a | street | in | train
pizza | a | skateboard | of | tennis | bathroom | of | sign
the | room | room | in | with | kitchen | to | cake
giraffe | sheep | horse | man | eating | train | from | clock
bathroom | frisbee | elephant | woman | and | beach | for | is
skateboard | skis | motorcycle | dog | playing | bed | a | bus
living | bird | kitchen | train | the | bus | on | truck
gira | skateboard | clock | with | flying | grass | at | car
sheep | surf | tower | is | sitting | road | the | of
an | kite | bear | to | walking | room | – | signs

In Table 5, we list the top 10 successfully and frequently retrieved subwords
for each keyword in SpokenCOCO. Generally, commonly retrieved subwords are
either stopwords like “a” and “of” or objects like “skateboard” and “street.”
In the first case, the phenomenon might be caused by the supervision from the
CLIP text encoder because stopwords contain little information about speech
signals but are sometimes crucial for maintaining the syntactic structures.
Moreover, we find the frequently retrieved words for objects sometimes appear
in SpokenCOCO’s captions but not very frequently. Hence, these words might be
easier to be detected in speech, and the corresponding objects are more
concrete to be found in images.

Additionally, we find that some keywords predict specific subword categories
successfully. For instance, keyword 7 tends to output prepositions and
articles, while keyword 5 mostly retrieves action words. As for the rest of
the keywords, nouns are mostly retrieved. Particularly, for keyword 2,
“frisbee”, “skis”, “skateboard”, and “surf” are all related to outdoor
activities. As for keyword 8, “train”, “sign”, “bus”, “truck”, “car”, and
“signs” are all related to traffic. This section demonstrates the cascaded
SpeechCLIP for retrieving semantically related keywords from speech signals.

###  3.5 Layer Importance in SpeechCLIP Speech Encoder

In this section, we show which HuBERT hidden layers are crucial for SpeechCLIP
to perform well in various tasks discussed earlier. Hence, we visualize the
learned weights in the weighted sum mechanism mentioned in Sec. 2.1 in Fig. 4.
Both parallel and cascaded SpeechCLIP utilize the roughly the
8${}^{\text{th}}$ to the 10${}^{\text{th}}$ layers in HuBERT, inferring that
HuBERT’s top layers capture rich content and semantic information. This result
is consistent with prior works investigating the importance of different
hidden layers in speech SSL models [40, 39, 16], i.e., the top hidden layers
contain word meaning and content information. However, the cascaded model’s
weights distribute more evenly over the layers than parallel SpeechCLIP,
showing that the model architecture design affects the utilization of HuBERT’s
layers.

Fig. 4:  Normalized weights for layer summarization of HuBERT in parallel and
cascaded SpeechCLIP. CNN denotes the HuBERT CNN feature extractor.

###  3.6 Ablation Studies

Table 6:  Recall scores on Flickr8k for ablation studies.  |  Speech $\rightarrow$ Image |  Image $\rightarrow$ Speech
---|---|---
Method | R@1 | R@5 | R@10 | R@1 | R@5 | R@10
| Batch Normalization
Cascaded (w/ BN) | 8.2 | 25.7 | 37.2 | 14.1 | 34.5 | 49.2
Cascaded (w/o BN) | 1.1 | 4.7 | 8.4 | 1.4 | 5.7 | 9.4
| Keyword Num
Cascaded ($K$ $=$ 8) | 8.2 | 25.7 | 37.2 | 14.1 | 34.5 | 49.2
Cascaded ($K$ $=$ 4) | 3.5 | 13.2 | 21.1 | 5.2 | 17.5 | 27.4
Cascaded ($K$ $=$ 2) | 2.1 | 8.4 | 14.4 | 2.7 | 10.6 | 17.6

####  3.6.1 Batch Normalization in Cascaded SpeechCLIP

Here, we demonstrate the importance of batch normalization in the cascaded
SpeechCLIP. We compare cascaded SpeechCLIP with its variant without using
batch normalization, as shown in the first two rows of Table 6. Removing batch
normalization degrades retrieval performance significantly, showing the
significance of mean and variance matching described in Sec. 2.3.

####  3.6.2 Number of Keywords in Cascaded SpeechCLIP

This section discusses the impact of the number of keywords in cascaded
SpeechCLIP. We report retrieval results on Flickr8k using different amounts of
keywords in Table 6. Results show that reducing keywords degrades retrieval
performance, indicating that using fewer keywords is incapable of passing
information from the speech encoder to the CLIP text encoder. Furthermore, the
number of subword tokens in a Flickr8k utterance is 11.3 $\pm$ 4.1, and some
tokens carry less information like stopwords. Therefore, we suggest 8 is a
reasonable number for $K$ to obtain good performance with cascaded SpeechCLIP.
Although dynamically assigning $K$ for utterances of different lengths is more
appropriate, we leave this approach for future investigation.

##  4 Conclusion

This paper introduces SpeechCLIP, a novel framework integrating CLIP into
visually grounded speech models. We demonstrate significant improvements in
image-speech retrieval with CLIP’s supervision. Moreover, the proposed methods
can perform zero-shot speech-text retrieval and capture semantically related
keywords in speech signals. Results indicate that bridging speech and text
domains with CLIP’s supervision is possible and promising. Overall, SpeechCLIP
opens a new research direction of indirectly supervising speech models with
text via other modalities. We suggest some topics in SpeechCLIP are worth
investigating in the future, including integrating parallel and cascaded in
the same model and cascaded structure with variable length prediction aiming
for unsupervised ASR. Furthermore, extending SpeechCLIP to a multilingual
model is possible using spoken captions from other languages or Multilingual-
CLIP models. Finally, we wish to inspect how CLIP can enhance speech SSL
models’ performance on downstream problems like speech recognition and intent
classification.

##  5 ACKNOWLEDGMENTS

This work was supported by JSALT 2022 at JHU, with gift-funds from Amazon,
Microsoft, and Google. Also, we thank Taiwan Web Service (TWS) and National
Center for High-performance Computing (NCHC) of National Applied Research
Laboratories (NARLabs) for providing computational resources.

## References

  * [1] Abdelrahman Mohamed, Hung-yi Lee, Lasse Borgholt, Jakob D Havtorn, Joakim Edin, Christian Igel, Katrin Kirchhoff, Shang-Wen Li, Karen Livescu, Lars Maaløe, et al.,  “Self-supervised speech representation learning: A review,”  arxiv:2205.10643, 2022.
  * [2] Andy T Liu, Shu-wen Yang, Po-Han Chi, Po-chun Hsu, and Hung-yi Lee,  “Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders,”  in ICASSP, 2020.
  * [3] Andy T Liu, Shang-Wen Li, and Hung-yi Lee,  “TERA: Self-supervised learning of transformer encoder representation for speech,”  TASLP, vol. 29, 2021.
  * [4] Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass,  “An unsupervised autoregressive model for speech representation learning,”  in Interspeech, 2019.
  * [5] Yu-An Chung, Hao Tang, and James Glass,  “Vector-quantized autoregressive predictive coding,”  in Interspeech, 2020.
  * [6] Alexander H Liu, Yu-An Chung, and James Glass,  “Non-autoregressive predictive coding for learning speech representations from local dependencies,”  in Interspeech, 2021.
  * [7] Aaron van den Oord, Yazhe Li, and Oriol Vinyals,  “Representation learning with contrastive predictive coding,”  arxiv:1807.03748, 2018.
  * [8] Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli,  “wav2vec: Unsupervised pre-training for speech recognition,”  in Interspeech, 2019.
  * [9] Alexei Baevski, Steffen Schneider, and Michael Auli,  “vq-wav2vec: Self-supervised learning of discrete speech representations,”  in ICLR, 2020.
  * [10] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  in NeurIPS, 2020.
  * [11] Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu,  “W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,”  in ASRU, 2021.
  * [12] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed,  “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,”  arxiv:2106.07447, 2021.
  * [13] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al.,  “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,”  arxiv:2110.13900, 2021.
  * [14] Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu,  “Self-supervised learning with random-projection quantizer for speech recognition,”  ICML, 2022.
  * [15] Mirco Ravanelli, Jianyuan Zhong, Santiago Pascual, Pawel Swietojanski, Joao Monteiro, Jan Trmal, and Yoshua Bengio,  “Multi-task self-supervised learning for robust speech recognition,”  in ICASSP, 2020.
  * [16] Heng-Jui Chang, Shu-wen Yang, and Hung-yi Lee,  “Distilhubert: Speech representation learning by layer-wise distillation of hidden-unit bert,”  in ICASSP, 2022.
  * [17] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli,  “Data2vec: A general framework for self-supervised learning in speech, vision and language,”  arxiv:2202.03555, 2022.
  * [18] Rui Wang, Qibing Bai, Junyi Ao, Long Zhou, Zhixiang Xiong, Zhihua Wei, Yu Zhang, Tom Ko, and Haizhou Li,  “Lighthubert: Lightweight and configurable speech representation learning with once-for-all hidden-unit bert,”  arxiv:2203.15610, 2022.
  * [19] Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, et al.,  “SUPERB: Speech processing universal performance benchmark,”  in Interspeech, 2021.
  * [20] Solène Evain et al.,  “LeBenchmark: A reproducible framework for assessing self-supervised representation learning from speech,”  in Interspeech, 2021.
  * [21] Hsiang-Sheng Tsai et al.,  “SUPERB-SG: Enhanced speech processing universal PERformance benchmark for semantic and generative capabilities,”  in ACL, 2022.
  * [22] David Harwath, Adria Recasens, Dídac Surís, Galen Chuang, Antonio Torralba, and James Glass,  “Jointly discovering visual objects and spoken words from raw sensory input,”  in ECCV, 2018.
  * [23] Alec Radford et al.,  “Learning transferable visual models from natural language supervision,”  in ICML, 2021.
  * [24] Grzegorz Chrupała,  “Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques,”  JAIR, vol. 73, 2022.
  * [25] Wei-Ning Hsu, David Harwath, and James Glass,  “Transfer learning from audio-visual grounding to speech recognition,”  Interspeech, 2019.
  * [26] David Harwath and James Glass,  “Deep multimodal semantic embeddings for speech and images,”  in ASRU, 2015.
  * [27] Wei-Ning Hsu, David Harwath, Christopher Song, and James Glass,  “Text-free image-to-speech synthesis using learned segmental units,”  arxiv:2012.15454, 2020.
  * [28] Liming Wang, Xinsheng Wang, Mark Hasegawa-Johnson, Odette Scharenborg, and Najim Dehak,  “Align or attend? toward more efficient and accurate spoken word discovery using speech-to-image retrieval,”  in ICASSP, 2021.
  * [29] Khazar Khorrami and Okko Räsänen,  “Evaluation of audio-visual alignments in visually grounded speech models,”  Interspeech, 2021.
  * [30] David Harwath, Galen Chuang, and James Glass,  “Vision as an interlingua: Learning multilingual semantic embeddings of untranscribed speech,”  in ICASSP, 2018.
  * [31] Herman Kamper and Michael Roth,  “Visually grounded cross-lingual keyword spotting in speech,”  SLTU, 2018.
  * [32] William N Havard, Jean-Pierre Chevrot, and Laurent Besacier,  “Catplayinginthesnow: Impact of prior segmentation on a model of visually grounded speech,”  CoNLL, 2020.
  * [33] Yasunori Ohishi, Akisato Kimura, Takahito Kawanishi, Kunio Kashino, David Harwath, and James Glass,  “Trilingual semantic embeddings of visually grounded speech with self-attention mechanisms,”  in ICASSP, 2020.
  * [34] Gabriel Ilharco, Yuan Zhang, and Jason Baldridge,  “Large-scale representation learning from visually grounded untranscribed speech,”  CoNLL, 2019.
  * [35] Ramon Sanabria, Austin Waters, and Jason Baldridge,  “Talk, don’t write: A study of direct speech-based image retrieval,”  Interspeech, 2021.
  * [36] Puyuan Peng and David Harwath,  “Fast-slow transformer for visually grounding speech,”  in ICASSP, 2022.
  * [37] Puyuan Peng and David Harwath,  “Self-supervised representation learning for speech using visual grounding and masked language modeling,”  arxiv:2202.03543, 2022.
  * [38] Puyuan Peng and David Harwath,  “Word discovery in visually grounded, self-supervised speech models,”  Interspeech, 2022.
  * [39] Ankita Pasad, Ju-Chieh Chou, and Karen Livescu,  “Layer-wise analysis of a self-supervised speech representation model,”  arxiv:2107.04734, 2021.
  * [40] Alexei Baevski, Wei-Ning Hsu, Alexis Conneau, and Michael Auli,  “Unsupervised speech recognition,”  NeurIPS, 2021.
  * [41] Alexander H Liu, Wei-Ning Hsu, Michael Auli, and Alexei Baevski,  “Towards end-to-end unsupervised speech recognition,”  arxiv:2204.02492, 2022.
  * [42] Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello,  “Wav2clip: Learning robust audio representations from clip,”  in ICASSP, 2022.
  * [43] Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel,  “Audioclip: Extending clip to image, text and audio,”  in ICASSP, 2022.
  * [44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  in NeurIPS, 2017.
  * [45] Yoshua Bengio, Nicholas Léonard, and Aaron Courville,  “Estimating or propagating gradients through stochastic neurons for conditional computation,”  arxiv:1308.3432, 2013.
  * [46] Andrej Karpathy and Li Fei-Fei,  “Deep visual-semantic alignments for generating image descriptions,”  in CVPR, 2015.
  * [47] Paul-Ambroise Duquenne, Hongyu Gong, and Holger Schwenk,  “Multimodal and multilingual embeddings for large-scale speech mining,”  in NeurIPS, 2021.
  * [48] Sameer Khurana, Antoine Laurent, and James Glass,  “SAMU-XLSR: Semantically-aligned multimodal utterance-level cross-lingual speech representation,”  arxiv preprint, 2022.
  * [49] Herman Kamper, Gregory Shakhnarovich, and Karen Livescu,  “Semantic speech retrieval with a visually grounded model of untranscribed speech,”  in CVPR, 2018, pp. 2514–2517.
  * [50] Ankita Pasad, Bowen Shi, Herman Kamper, and Karen Livescu,  “On the contributions of visual and textual supervision in low-resource semantic speech retrieval,”  arXiv preprint, 2019.
  * [51] Kayode Olaleye, Dan Oneata, and Herman Kamper,  “Keyword localisation in untranscribed speech using visually grounded speech models,”  JSTSP, 2022.


#  RUNHETC-2023-27
My Personal History With the Quantum Theory of de Sitter Space

Tom Banks
NHETC and Department of Physics
Rutgers University, Piscataway, NJ 08854-8019
E-mail<EMAIL_ADDRESS>

###### Abstract

The purpose of this brief article is to clarify certain distortions of the
history of ideas about the quantum theory of de Sitter space that have
appeared in recent literature.

###### Contents

  1. 1 Introduction
  2. 2 Locality
    1. 2.1 The Limitations of Quantum Field Theory
  3. 3 The Back Reaction of Detectors
  4. 4 Conclusions

##  1 Introduction

I first heard about the cosmological constant problem from K. Johnson in a
lecture at the Scottish Summer School in St. Andrews in 1976. L. Susskind, who
had emphasized the importance of the gauge hierarchy problem to me since we
first worked together in 1975, often referred to it as an even bigger mystery.
In 1999 I was sitting on the patio of my new home in Big Sur, staring at the
ocean, when a number of thoughts came together in my head. We had learned from
AdS/CFT that when it was negative, the c.c. should be thought of as a
parameter characterizing the high energy behavior of particular models, and
was not renormalized in low energy effective field theory. My friends Fischler
and Susskind had proposed generalizing the Bekenstein-Hawking bound for black
holes to cosmological space-times[4], and this had led Bousso to a general
conjecture on entropy bounds for space-time regions[5]. The universe we
inhabited seemed to be approaching a de Sitter asymptotic future, and Gibbons
and Hawking had predicted that such a universe had finite entropy. Moreover,
conventional string theory seemed unable to cope with such a universe.

Suddenly it all made sense. The Gibbons-Hawking entropy should be thought of
as the log of the dimension of the Hilbert space, which implied that empty dS
space was a maximal entropy state, and indeed from the point of view of any
geodesic, anything not bound to the geodesic appears to fade into its
cosmological horizon in a time of order the dS radius. Furthermore, the black
hole entropy formula showed that objects bound to the geodesic lowered the
entropy111It wasn’t until somewhat later that I realized that this gave a
derivation of the precise value of the Gibbons-Hawking temperature independent
of quantum field theory (QFT). It was much later that Fischler and I realized
that this was the key clue to how local physics arises from a quantum theory
of horizon degrees of freedom..

I spoke about these ideas at the 60th birthday celebration for L. Susskind at
Stanford in the Spring of 2000. Fischler was there and told me he’d had the
same idea independently. He spoke about his version at Geoff West’s 60th
birthday celebration in Taos, NM, also in 2000. I’ve always tried to give him
equal credit, but the community has resisted this. I finally published a
version of this in connection with the talk I gave at the strings conference
in Michigan[6]. Some people[7] greeted the idea that dS space had a finite
dimensional Hilbert space with a certain amount of enthusiasm, but for the
most part it was met with skepticism and mockery. I spent hours trying to
convince E. Witten that the idea was correct, despite the fact that in global
coordinates the past and future asymptotic regions of dS space seemed to have
arbitrarily large space-like slices.

I gave two arguments, one mathematical, one physical. The mathematical
argument was that because there were no asymptotic space-like or null regions,
de Sitter isometries were all gauge transformations and all of dS space
consisted of gauge copies of a single static patch. The physical argument was
that generic asymptotic data from the past or future extrapolated to a
singularity that did not allow one to continue to the future or past of dS, so
that specifying both past and future dS asymptotics restricted one to a finite
dimensional Hilbert space. Gary Horowitz told me this sounded plausible, and
Bousso and Freivogel[12] eventually proved a theorem along these lines. Witten
seemed to partially concede in[8].

The conjecture that dS space has a finite number of states gets reinforced by
the computation of Coleman DeLuccia tunneling amplitudes between two dS
spaces[9], which obey a rule of detailed balance[10] appropriate to tunneling
in finite systems. This leads to a controversial reinterpretation of the
theory of eternal inflation[11], with by far the most sensible conjecture
about the meaning of transitions to Big Crunch cosmologies. It turns the
search for a measure on an ill defined infinite probability space into an
exercise in the non-equilibrium statistical mechanics of finite quantum
systems.

In the meantime, Fischler and I continued to develop these ideas into a
general formalism for dealing with cosmological space-times quantum
mechanically. Eventually we gave it the name Holographic Space-time. The
central idea was to generalize the conjecture about dS space to a conjecture
about the density matrix of every finite area causal diamond in ”empty space”.
Bousso pointed out to us that he had made a similar conjecture in his review
papers[13]. This led us to a model of the maximally entropic universe, which
is a flat FRW universe with equation of state $p=\rho$[15]. If we had been
more clear thinking, we would have realized that this model in fact
contradicted our initial ansatz for the density matrix. In fact, to fit the
data of the $p=\rho$ universe, we had to assume that the modular Hamiltonian
at any time was that of a $1+1$ dimensional conformal field theory (CFT), with
central charge that grew like the area of the apparent horizon. We did notice
the connection to earlier work of Carlip[16] on black hole horizons, but did
not pay enough attention. And it was only much later that we realized that
simply stopping the expansion of the Hilbert space of our model with a fixed
CFT, gave a description of the flat FRW model with $a(t)=\sinh^{1/3}(3t/R)$,
which asymptotes to dS space. This implies that the density matrix of dS space
is not proportional to the unit matrix. It does not affect the ability to
interpret the dS temperature in terms of a connection between energy measured
along a geodesic, and the number of constraints on boundary degrees of
freedom[27].

Fischler and I have always been loath to use the term observer because of all
of the nonsensical literature about the connection between quantum measurement
theory and consciousness. Instead, the HST formalism has always been based on
the Hilbert spaces associated with intervals along time-like trajectories. One
thinks of these as the trajectories of detectors that have no back-reaction on
space-time geometry. I’ll discuss the problem of back-reaction below. The
covariant entropy bound implies that for short enough proper time the maximal
entropy causally accessible to a detector on such a trajectory is always
finite, which means expectation values of operators are computable in a Type
$II_{1}$ von Neumann algebra. The hyperfinite Type $II_{1}$ algebra is just
the limit of a finite number of fermionic creation and annihilation operators.
For a black hole of mass $1$ gram in $4$ dimensions, the entropy is
approximately $10^{11}$, so one must be able to do measurements fine enough to
distinguish between the infinite dimensional algebra and that of $10^{11}$
q-bits in order to falsify the claim of HST that the Hilbert space is finite
dimensional. For the universe as a whole the number of q-bits is about
$10^{123}$ and it’s clearly impossible to imagine making the distinction. As
we’ll see, realistic measurements inside a causal diamond cannot actually
probe even a finite fraction (parametrically) of the quantum information in
the diamond. Thus, we’ve always assumed finite dimensional Hilbert spaces,
which also implies a discreteness of time evolution. We take the discrete
interval to be the Planck time.

A key feature of the formalism has always been the Quantum Principle of
Relativity (QPR) (which should have been a clue to the tone deaf that we were
talking about ”observers”). Any given space-time is described by multiple
quantum systems, living in a Hilbert bundle over a manifold of time-like
trajectories. The simplest collection of time-like trajectories that form a
manifold is the space of time-like geodesics. In examples like AdS space one
might want to add null-geodesics or accelerated trajectories that reach the
boundary. For any two choices of interval along a pair of trajectories in the
base space of the bundle there is an overlap region and a causal diamond,
unique up to symmetries, of maximal area in that region. We insist that there
be a Hilbert space associated with this diamond and unitary embedding of this
space into the Hilbert spaces of each individual trajectory as a tensor
factor. The density matrices on those two tensor factors must have identical
entanglement spectra. This contrains both the time dependent Hamiltonians and
the choices of initial state on all fibers of the Hilbert bundle.

This formulation of HST depends on the assumption of a background space-time.
The proper way to think about this is using Jacobson’s[14] demonstration that
Einstein’s equations (doubly projected onto an arbitrary null vector so that
the c.c. doesn’t contribute) are the hydrodynamic equations of the area law
for causal diamonds. Thus, we view a space-time as a hydrodynamic background,
into which we must fit a quantum theory whose hydrodynamics matches that
background. The refinement of Jacobson’s argument due to Carlip[16] and
Solodukhin[17], generalized from black holes to arbitrary diamonds in[18],
shows us how to do this. These authors show that the fluctuations of the
conformal factor of the background geometry of the holographic screen of each
diamond are the hydrodynamic equations of a $1+1$ dimensional CFT with central
charge proportional to the area of the diamond’s holographic screen. They
derive the area law for the diamond entropy, assuming the modular Hamiltonian
is the $L_{0}$ generator of that CFT. The HST formalism provides us with a
guess for what the CFT is[22].

##  2 Locality

Perhaps the most important insight into the relation between the quantum
theory of gravity and quantum field theory comes from Fiol’s explicit modeling
of the entropy reduction of localized objects in dS space[3]. The variables
are rectangular matrices, each matrix element being a $1+1$ dimensional
massless fermion field (with a UV cutoff described in[22]) . Localized states
are states on which matrices bilinear in the fermions are block diagonal. This
prescription reproduces the thermal properties of dS space[3][27], but also
leads to a model of scattering of particles in Minkowski space in the limit of
infinite dS radius[21]. This is an alternative view of the emergence of local
physics from a holographic theory of quantum gravity, which shows how locality
breaks down at the threshold of black hole formation. It has puzzled me that
so little attention has been paid to it. By using tensor network technology,
it can be made compatible with the standard tensor network picture of locality
in AdS space[25], on scales larger than the AdS radius. Although the original
implementation of Fiol’s idea was based on the assumption that the density
matrix of empty dS space was proportional to the unit matrix, it works just as
well, with one proviso, for the Carlip-Solodukhin ansatz[27]. The proviso is
that we must give up the requirement that the probability distribution for
states that have energy (as measured along a geodesic in the diamond) of order
$1/R_{\diamond}$ is just given by the ratio of the dimension of the
constrained subspace in which they live, to the total. This remains true for
those low energy states that can actually be detected on the geodesic, and are
well described by QFT, but is not true for most of the states on the boundary.

###  2.1 The Limitations of Quantum Field Theory

There is a widespread folklore in the string theory community that in various
”weak string coupling” or ”large AdS radius” limits, QFT is a good
approximation to everything in a model of quantum gravity. This is simply
incorrect and has been known to be incorrect since at least the 1970s. If one
considers a causal diamond in Minkowski space of proper time $T$, the field
theory Hilbert space one associates with it has a dimension which is the
exponential of $(TM_{c})^{d-1}$, where $M_{c}$ is some kind of UV cut-off.
Most of these states have semi-classical gravitational back-reaction that
produces black holes with size larger than $T$. Eliminating those, one is
left, generically, with an entropy bounded by something of order
$(A_{\diamond}/G_{N})^{\frac{d-1}{d}}$. In particular, one has to omit the
area’s worth of states that give rise to the UV divergent state independent
QFT entanglement entropy of the diamond222Note that in a pre-existing black
hole background, the near horizon states have smaller gravitational back
reaction due to the gravitational redshift. This is the reason they’ve been
viewed as candidates for the explanation of black hole entropy. Making this
assumption leads to the firewall paradox.. In an important paper[20], Cohen,
Kaplan and Nelson, showed that omitting these states did not affect the
agreement between QFT and any known experiment. The intuitive reason for this
is that experiments are done on near geodesic trajectories for finite proper
times, and are relatively unaffected by virtual excitations (in QFT language)
far from those trajectories. CKN used a crude correlation between UV and IR
cutoffs and found that their bounds were close to current experimental limits.
More sophisticated cutoff procedures[24] allow more high momentum modes near
the experimental trajectory and put experimental probes of the bound further
out of reach.

The upshot of these arguments is that we have no phenomenological reason to
believe the QFT picture of most of the states on the boundaries of a causal
diamond. Since that picture leads to the firewall problem, and does not
account microscopically for the large increase in entropy when a low entropy
object is dropped onto a large black hole (whereas the HST models referred to
above avoid the firewall and do explain the entropy increase), one should
simply accept the fact that QFT breaks down, not just at large energy and
short distance, but in any situation with large entropy. There are indications
from perturbative string theory[23] that this is the case.

##  3 The Back Reaction of Detectors

According to the modern rules of quantum measurement theory, a detector that
has to measure and store the quantum information about a quantum system with
$M$ dimensions has to have $M$ independent ”collective variables” with the
following properties

  * •

There is a large integer $N$ such that the uncertainties of all of the
collective variables scale like $N^{-1}$. The equations of motion of the
detector system relate collective variables to each other, so we can follow
histories of these variables with no ambiguity, for times of order $N$.

  * •

The interference terms in the probability sum rule for different histories of
the collective variables are exponentially small, of order $e^{-N^{p}}$ with
$p$ of order $1$ or larger.

  * •

The detector’s collective variables must retain their ”semi-classical” status
over time scales $N$ long enough to do multiple measurements on different
bases of the $M$ dimensional system, in order to be able to extract all of the
probability amplitudes for different time dependent transitions.

It follows from this that the detector must be a quantum system with a Hilbert
space whose dimension is $\gg M$. Just the requirement that the detector be a
subsystem of a closed finite entropy universe makes it impossible to imagine a
detector inside a closed universe that can verify every detail of the quantum
model of the universe. However, the features of the quantum theory of gravity
discussed above make the problem much more severe than this.

First, all experiments to date have taken place on near geodesic trajectories
in a causal diamond. The more complex a detector we imagine on such a
trajectory, the more constrained the state of the boundary quantum system it
is measuring. Thus, not only can’t we imagine a detector with enough
collective variables to determine the quantum behavior of the universe, but
the more precise and efficient our detector, the less of the Hilbert space of
the universe the detector can explore.

Furthermore, the kinds of systems that we know to exist on near geodesic
trajectories are those described by QFT, and black holes. We’ve argued that,
if our universe is a dS space with the observed value of the c.c. then the log
of the dimension of the Hilbert space of the maximal QFT detector allowed by
theory333Note that the actual entropy of localized matter in our universe is
about $10^{90}$ and is dominated by the entropy of supermassive black holes in
the centers of galaxies. is about $10^{92}$, while that of the universe is
$10^{123}$. Black holes have more quantum states for a given entropy deficit
on the horizon, but they have many fewer collective coordinates with which to
record and store data. So the quantum gravity constraints on detectors inside
a dS universe imply that much of the quantum information in the universe is
undetectable.

The combination of quantum measurement theory and the constraints of quantum
gravity thus imply that no observations in a dS universe can ever test most of
the details of a quantum model of the entire universe. Note that we have not
yet taken into account the constraint that the detector’s collective
coordinates survive long enough to make all the required measurements. If we
imagine a detector moving along a geodesic, these constraints are very severe.
Most time-like trajectories approach the boundary of the causal diamond of any
given geodesic in a time of order the dS radius. Quantum fluctuations of the
detector’s position, and disturbances of its position by gravitational
scattering from other localized matter, Gibbons-Hawking radiation etc. will
make it deviate from geodesic motion on time scales short compared to those
necessary to the collection of the maximal amount of quantum information it
can process. Fortunately, for detectors that belong to local groups of
galaxies like our own, geodesic motion is irrelevant. The huge set of quantum
states of the local group decohere the trajectory of its center of mass motion
through the universe. In principle then, the useful lifetime of such a massive
complex detector is limited by the time it takes it to collapse into a black
hole. After that point, the information it has collected has ”unhappened”, it
is lost in the detailed quantum microstate of the black hole, and no
macroscopic record remains.

##  4 Conclusions

I list the main take-aways from this article

  * •

The original claim that the density matrix of empty dS space is maximally
uncertain was made independently by myself and W. Fischler in 2000. The most
important part of that claim was the identification of the energy of localized
states as a count of the number of constrained q-bits in the subspace of fixed
localized energy. Fiol invented a fermionic matrix model of this effect[3],
and it leads to an understanding of how local particle physics emerges from a
holographic model of gravity in small causal diamonds. This is quite different
from the way that locality in the AdS part of space-time emerges in the tensor
network approach to the AdS/CFT correspondence.

  * •

Operator algebras/Hilbert spaces of finite area diamonds are finite
dimensional, and time evolution is consequently discrete, consisting of
entangling of larger and larger tensor factors of a large Hilbert space with
the rest of the degrees of freedom. The quantum of time is taken to be the
Planck scale.

  * •

The maximal uncertainty ansatz is probably wrong and should be replaced by the
Carlip-Solodukhin ansatz that the modular Hamiltonian of a diamond is the
(appropriately cut-off) Virasoro generator $L_{0}$ of a $1+1$ dimensional CFT
living on the stretched horizon of the diamond. This is valid for all causal
diamonds in space-times with non-negative c.c.. For negative c.c. the C-S
ansatz is valid for boundary anchored Ryu-Takayanagi diamonds, and for
diamonds smaller than the AdS radius. Diamonds larger than the AdS radius must
be treated by the tensor network construction of[25] and have the modular
Hamiltonian of a thermal CFT in higher dimensions. For small causal diamonds
or diamonds in space-times with non-negative c.c., the connection between
local physics and boundary constraints, and the derivation of the dS
temperature from this relation, are compatible with the C-S ansatz as long as
we restrict our assumptions about the range of validity of QFT in the bulk, in
the manner suggested by CKN. A particularly welcome consequence of abandoning
the maximal uncertainty hypotheses is that it implies fluctuations of the
modular Hamiltonian in inflationary horizon volumes. These give rise to the
observed temperature fluctuations in the Cosmic Microwave Background. A model
without them would be ruled out by observation.

  * •

QFT is valid in a causal diamond at most for states of entropy bounded by
$(A/G_{N})^{\frac{d-1}{d}}$.

  * •

The ”vacuum” in Minkowski space is a high entropy state, with the entropy
coming from zero energy states. This follows by viewing Minkowski space as a
limit of dS space, but also as a limit from negative c.c.. In the AdS case,
the Minkowski diamond is a tiny subsystem entangled with the much larger AdS
background, as seen in the tensor network formulation. In correlation
functions the entanglement comes from Witten diagrams in which arbitrary
numbers of massless lines are exchanged between operators focussed on the
Minkowski diamond (which converge to Minkowski S-matrix elements with finite
numbers of particles) and operators that create excitations that propagate far
from the diamond. The nature of the asymptotic Hilbert space and the existence
of a non-perturbative S-matrix remain open, particularly in $4$ dimensions.

  * •

The joint constraints of quantum measurement theory and quantum gravity imply
that no detector in a closed universe can probe most of the quantum
information that the covariant entropy bound allows in that universe.
Consequently, no model of such a universe with infinite dimensional algebras
of observables has any observational meaning, and even finite dimensional
models will have a lot of freedom that cannot be verified or falsified by
experiment.

For those with endless patience, a fairly complete list of my thoughts on the
entropy of states in dS space and their wider implications, full of mistakes
and false starts, can be found here[28].

Acknowledgments

The work of T.B. was supported by the Department of Energy under grant DE-
SC0010008. Rutgers Project 833012. I would like to thank W. Fischler for
collaboration over many years. He deserves equal credit for most of the
insights about entropy, space-time and locality that we’ve explore in the past
25 years. The opinions expressed in this paper are my own.

## References

  * [1] T. Banks, Talk at the 60th Birthday Celebration of L. Susskind, Stanford University, 2000 ; T. Banks, “Cosmological breaking of supersymmetry?,” Int. J. Mod. Phys. A 16, 910-921 (2001) doi:10.1142/S0217751X01003998 [arXiv:hep-th/0007146 [hep-th]].
  * [2] W. Fischler, Taking de Sitter Seriously, talk at the 60th Birthday Celebration for G. West, Taos, NM, 2000.
  * [3] T. Banks, B. Fiol and A. Morisse, “Towards a quantum theory of de Sitter space,” JHEP 12, 004 (2006) doi:10.1088/1126-6708/2006/12/004 [arXiv:hep-th/0609062 [hep-th]].
  * [4] W. Fischler and L. Susskind, “Holography and cosmology,” hep-th/9806039;
  * [5] R. Bousso, “A Covariant entropy conjecture,” JHEP 9907, 004 (1999) [hep-th/9905177].
  * [6] T. Banks, “Cosmological breaking of supersymmetry?,” Int. J. Mod. Phys. A 16, 910-921 (2001) doi:10.1142/S0217751X01003998 [arXiv:hep-th/0007146 [hep-th]].
  * [7] R. Bousso, “Positive vacuum energy and the N bound,” JHEP 11, 038 (2000) doi:10.1088/1126-6708/2000/11/038 [arXiv:hep-th/0010252 [hep-th]].
  * [8] E. Witten, “Quantum gravity in de Sitter space,” [arXiv:hep-th/0106109 [hep-th]].
  * [9] S. R. Coleman and F. De Luccia,“Gravitational Effects on and of Vacuum Decay,” Phys. Rev. D 21, 3305 (1980) doi:10.1103/PhysRevD.21.3305
  * [10] T. Banks,“Heretics of the false vacuum: Gravitational effects on and of vacuum decay. 2.,” [arXiv:hep-th/0211160 [hep-th]]; A. R. Brown and E. J. Weinberg, “Thermal derivation of the Coleman-De Luccia tunneling prescription,” Phys. Rev. D 76, 064003 (2007) doi:10.1103/PhysRevD.76.064003 [arXiv:0706.1573 [hep-th]].
  * [11] T. Banks,“Relaxation of the Cosmological Constant,” Phys. Rev. Lett. 52, 1461-1463 (1984) doi:10.1103/PhysRevLett.52.1461; T. Banks and M. Johnson,“Regulating eternal inflation,” [arXiv:hep-th/0512141 [hep-th]]; A. Aguirre, T. Banks and M. Johnson,“Regulating eternal inflation. II. The Great divide,” JHEP 08, 065 (2006) doi:10.1088/1126-6708/2006/08/065 [arXiv:hep-th/0603107 [hep-th]]; T. Banks and J. F. Fortin,“Tunneling Constraints on Effective Theories of Stable de Sitter Space,” Phys. Rev. D 80, 075002 (2009) doi:10.1103/PhysRevD.80.075002 [arXiv:0906.3714 [hep-th]]; T. Banks,“On the Limits of Effective Quantum Field Theory: Eternal Inflation, Landscapes, and Other Mythical Beasts,” [arXiv:1910.12817 [hep-th]]; T. Banks and B. Zhang,“Comment on Coleman-DeLuccia instantons,” SciPost Phys. 12, no.1, 015 (2022) doi:10.21468/SciPostPhys.12.1.015 [arXiv:2106.12696 [hep-th]].
  * [12] R. Bousso and B. Freivogel, “Asymptotic states of the bounce geometry,” Phys. Rev. D 73, 083507 (2006) doi:10.1103/PhysRevD.73.083507 [arXiv:hep-th/0511084 [hep-th]].
  * [13] R. Bousso, “Holography in general space-times,” JHEP 06, 028 (1999) doi:10.1088/1126-6708/1999/06/028 [arXiv:hep-th/9906022 [hep-th]]; R. Bousso, “The Holographic principle for general backgrounds,” Class. Quant. Grav. 17, 997-1005 (2000) doi:10.1088/0264-9381/17/5/309 [arXiv:hep-th/9911002 [hep-th]].
  * [14] T. Jacobson, “Thermodynamics of space-time: The Einstein equation of state,” Phys. Rev. Lett. 75, 1260-1263 (1995) doi:10.1103/PhysRevLett.75.1260 [arXiv:gr-qc/9504004 [gr-qc]].
  * [15] T. Banks, W. Fischler and L. Mannelli, “Microscopic quantum mechanics of the p = rho universe,” Phys. Rev. D 71, 123514 (2005) [hep-th/0408076];
  * [16] S. Carlip, “Black hole entropy from conformal field theory in any dimension,” Phys. Rev. Lett. 82, 2828-2831 (1999) doi:10.1103/PhysRevLett.82.2828 [arXiv:hep-th/9812013 [hep-th]].
  * [17] S. N. Solodukhin, “Conformal description of horizon’s states,” Phys. Lett. B 454, 213-222 (1999) doi:10.1016/S0370-2693(99)00398-6 [arXiv:hep-th/9812056 [hep-th]].
  * [18] T. Banks and K. M. Zurek, “Conformal description of near-horizon vacuum states,” Phys. Rev. D 104, no.12, 126026 (2021) doi:10.1103/PhysRevD.104.126026 [arXiv:2108.04806 [hep-th]].
  * [19] T. Banks and P. Draper, “Comments on the entanglement spectrum of de Sitter space,” JHEP 01, 135 (2023) doi:10.1007/JHEP01(2023)135 [arXiv:2209.08991 [hep-th]].
  * [20] A. G. Cohen, D. B. Kaplan and A. E. Nelson, “Effective field theory, black holes, and the cosmological constant,” Phys. Rev. Lett. 82, 4971 (1999) doi:10.1103/PhysRevLett.82.4971 [hep-th/9803132].
  * [21] T. Banks and W. Fischler, “Holographic Theory of Accelerated Observers, the S-matrix, and the Emergence of Effective Field Theory,” [arXiv:1301.5924 [hep-th]].
  * [22] T. Banks, “Hilbert Bundles and Holographic Space-time Models,” [arXiv:2306.07038 [hep-th]].
  * [23] S. Ghosh and S. Raju, “Breakdown of String Perturbation Theory for Many External Particles,” Phys. Rev. Lett. 118, no.13, 131602 (2017) doi:10.1103/PhysRevLett.118.131602 [arXiv:1611.08003 [hep-th]]; S. Ghosh and S. Raju,“Loss of locality in gravitational correlators with a large number of insertions,” Phys. Rev. D 96, no.6, 066033 (2017) doi:10.1103/PhysRevD.96.066033 [arXiv:1706.07424 [hep-th]].
  * [24] T. Banks and P. Draper, “Remarks on the Cohen-Kaplan-Nelson bound,” Phys. Rev. D 101, no.12, 126010 (2020) doi:10.1103/PhysRevD.101.126010 [arXiv:1911.05778 [hep-th]]; N. Blinov and P. Draper, “Densities of states and the Cohen-Kaplan-Nelson bound,” Phys. Rev. D 104, no.7, 076024 (2021) doi:10.1103/PhysRevD.104.076024 [arXiv:2107.03530 [hep-ph]].
  * [25] T. Banks and W. Fischler, “Holographic Space-time Models of Anti-deSitter Space-times,” [arXiv:1607.03510 [hep-th]].
  * [26] G. W. Gibbons and S. W. Hawking, “Cosmological Event Horizons, Thermodynamics, and Particle Creation,” Phys. Rev. D 15, 2738-2751 (1977) doi:10.1103/PhysRevD.15.2738
  * [27] T. Banks and P. Draper, “Comments on the entanglement spectrum of de Sitter space,” JHEP 01, 135 (2023) doi:10.1007/JHEP01(2023)135 [arXiv:2209.08991 [hep-th]].
  * [28] T. Banks and W. Fischler, “M theory observables for cosmological space-times,” [arXiv:hep-th/0102077 [hep-th]]. T. Banks and W. Fischler, “An Holographic cosmology,” hep-th/0111142; T. Banks, W. Fischler and S. Paban, “Recurrent nightmares? Measurement theory in de Sitter space,” JHEP 0212, 062 (2002) [hep-th/0210160]; T. Banks,“Some thoughts on the quantum theory of de sitter space,” [arXiv:astro-ph/0305037 [astro-ph]]; T. Banks and W. Fischler, “Holographic cosmology 3.0,” Phys. Scripta T 117, 56 (2005) [hep-th/0310288]; T. Banks,“Supersymmetry, the cosmological constant, and a theory of quantum gravity in our universe,” Gen. Rel. Grav. 35, 2075-2078 (2003) doi:10.1023/A:1027342603060 [arXiv:hep-th/0305206 [hep-th]]; T. Banks, W. Fischler and L. Mannelli, “Microscopic quantum mechanics of the p = rho universe,” Phys. Rev. D 71, 123514 (2005) [hep-th/0408076]; T. Banks, “Some thoughts on the quantum theory of stable de Sitter space,” [arXiv:hep-th/0503066 [hep-th]]. T. Banks and W. Fischler, “Holographic Theories of Inflation and Fluctuations,” arXiv:1111.4948 [hep-th]; T. Banks, “Holographic Space-Time: The Takeaway,” arXiv:1109.2435 [hep-th]; T. Banks and J. Kehayias, “Fuzzy Geometry via the Spinor Bundle, with Applications to Holographic Space-time and Matrix Theory,” Phys. Rev. D 84, 086008 (2011) [arXiv:1106.1179 [hep-th]]; T. Banks, “TASI Lectures on Holographic Space-Time, SUSY and Gravitational Effective Field Theory,” arXiv:1007.4001 [hep-th]; T. Banks, “Entropy and initial conditions in cosmology,” [arXiv:hep-th/0701146 [hep-th]]; T. Banks, “Holographic Space-time from the Big Bang to the de Sitter era,” J. Phys. A 42, 304002 (2009) doi:10.1088/1751-8113/42/30/304002 [arXiv:0809.3951 [hep-th]]. T. Banks, “Holographic space-time and its phenomenological implications,” Int. J. Mod. Phys. A 25, 4875 (2010) [arXiv:1004.2736 [hep-th]]; T. Banks, “Holographic Space-time from the Big Bang to the de Sitter era,” J. Phys. A A 42, 304002 (2009) [arXiv:0809.3951 [hep-th]]. T. Banks and W. Fischler, “CP Violation and Baryogenesis in the Presence of Black Holes,” arXiv:1505.00472 [hep-th]. T. Banks and W. Fischler, “Holographic Inflation Revised,” arXiv:1501.01686 [hep-th]. T. Banks and W. Fischler, “Holographic Space-time and Newton’s Law,” [arXiv:1310.6052 [hep-th]].