# Listen, Chat, and Edit: Text-Guided Soundscape Modification for Enhanced
Auditory Experience

Xilin Jiang  Cong Han  Yinghao Aaron Li  Nima Mesgarani

###### Abstract

In daily life, we encounter a variety of sounds, both desirable and
undesirable, with limited control over their presence and volume. Our work
introduces “Listen, Chat, and Edit” (LCE), a novel multimodal sound mixture
editor that modifies each sound source in a mixture based on user-provided
text instructions. LCE distinguishes itself with a user-friendly chat
interface and its unique ability to edit multiple sound sources simultaneously
within a mixture, without needing to separate them. Users input open-
vocabulary text prompts, which are interpreted by a large language model to
create a semantic filter for editing the sound mixture. The system then
decomposes the mixture into its components, applies the semantic filter, and
reassembles it into the desired output. We developed a 160-hour dataset with
over 100k mixtures, including speech and various audio sources, along with
text prompts for diverse editing tasks like extraction, removal, and volume
control. Our experiments demonstrate significant improvements in signal
quality across all editing tasks and robust performance in zero-shot scenarios
with varying numbers and types of sound sources. An audio demo is available
at: https://listenchatedit.github.io.

Machine Learning, ICML

##  1 Introduction

Imagine yourself at a cocktail party, listening to a conversation between two
people. Amidst their conversation, a soft guitar melody gracefully wafts
through the air, however woefully accompanied by the annoying noises of
passing cars outside. While you might find their conversation intriguing, one
of the speakers talks too loudly, making it difficult to hear the other person
clearly. Moreover, the persistent outdoor traffic adds an extra layer of
distraction. At this moment, you open the smart sound editor and type “Hey.
Can you reduce the volume of the excited male speaker and remove the
background traffic noise completely?” Instantly, you find it much easier to
follow their conversation, and the soothing music remains hovering in the air.

The abovementioned scenario where people want to focus on specific sounds
among multiple concurrent ones is known as the cocktail party problem (Cherry,
1953; Haykin & Chen, 2005; McDermott, 2009). Numerous studies have been
conducted for enhancing the hearing experience in multi-sound environments.
Traditional hearing aids (Kates, 2008; Clark & Swanepoel, 2014; Launer et al.,
2016) can amplify or suppress a wide spectrum of sounds within a mix, but they
fall short in targeting specific sound sources. Some recent models can extract
a target sound source based on neural auditory attention signals (Van
Eyndhoven et al., 2016; Han et al., 2019) or sound word labels (Kilgour et
al., 2022; Liu et al., 2022; Veluri et al., 2023). However, these models
concentrate on isolating a single source and lack the versatility to modify
multiple sound sources. Additionally, most of these models do not provide a
user-friendly interface for specifying the target. Therefore, we need a
solution that allows for soundscape modification by easily specifying
arbitrary sound components we would like to edit.

Figure 1: An overview of Listen, Chat, and Edit.

In this study, we introduce text-guided sound mixture-to-mixture editing, a
new problem that aims to simultaneously extract, remove, or control the volume
of various sounds in a mix based on the user’s open-vocabulary instruction. We
propose the first text-guided sound mixture editor: Listen, Chat, and Edit
(LCE). Figure 1 shows the schematic of LCE: To begin with, a user and LCE both
Listen to the sound mixture. Next, the user Chats with a text prompt in
natural language to specify which sound sources to be edited and in which
manner. Finally, LCE Edits the sound mixture according to the text prompt and
outputs a new sound mixture that aligns with the user’s command. LCE differs
from previous works in three distinctive features:

  * •

Selectivity: LCE recognizes and edits target talkers by semantic descriptions.
A user describes speaking styles for human speakers, including gender, pitch,
tempo, energy, and emotion, and the category for non-speech sounds, such as
the name of an animal or a musical instrument.

  * •

Accessibility: LCE employs a large language model (LLM) to interpret users’
open-vocabulary text prompts for different editing tasks. This textual
instruction is more user-friendly than other forms of instruction like neural
signals, speaker features, sound labels, and so on.

  * •

Sound mixture-to-mixture editing: LCE simultaneously edits multiple sound
sources in the sound mixture in one step, without separating them in advance
and editing each sound one by one.

We propose 16 sound mixture-to-mixture editing tasks as shown in Table 1 and
evaluate LCE on them. Since the sound mixtures and text prompts required to
train LCE did not exist, we have curated a new sound mixture dataset,
comprising around 160 hours or over 100k of speech and audio mixtures with
over 500k of text prompts written for these 16 editing tasks. Our experimental
results show that LCE trained on this dataset enhances the signal quality in
94.4% of sound mixtures and improves the signal-to-noise ratio by 10.4 dB
averaged across all editing tasks. Notably, LCE surpasses previous expert
models on target speech or audio extraction as one of the editing tasks.
Moreover, LCE can perform zero-shot editing on sound mixtures containing
unseen types of sounds or different numbers of sources.

##  2 Related Works

Sound Separation and Extraction. Sound mixture understanding and processing
(Settle et al., 2018; Turpault et al., 2020) relies on sound separation and
extraction, as the latter can provide the computational groundwork to isolate
each sound source in a mixture. There have been sound separation models
separating speeches (Hershey et al., 2015; Luo & Mesgarani, 2019; Subakan et
al., 2021), music (Bryan et al., 2013; Défossez et al., 2021; Luo & Yu, 2022),
and universal sounds (Kavalerov et al., 2019; Wisdom et al., 2021; Petermann
et al., 2022) in a mixture. However, they are not selective since they
unconditionally separate all sources in the mixture. On the other hand, sound
extraction models selectively extract one target sound from a sound mixture
(Elminshawi et al., 2022; Zmolikova et al., 2023). A clue, which can take
multiple forms, is provided to identify the target. Some common clues include
lip movement videos (Afouras et al., 2018; Ma et al., 2021), videos recording
the sound production (Zhao et al., 2018; Tzinis et al., 2021), target speaker
embeddings (Žmolíková et al., 2019; Wang et al., 2019), neural auditory
attention signals (Van Eyndhoven et al., 2016; Han et al., 2019), locations
(Gu et al., 2019; Heitkaemper et al., 2019), the language spoken (Borsdorf et
al., 2021), and recently emerging text prompts.

Text-Guided Audio Applications. Recent advancements in NLP and text-audio
learning have led to the use of text as the guidance for various audio
applications. In target sound extraction, semantic labels, descriptions of
audio sources (Kilgour et al., 2022; Liu et al., 2022, 2023c; Veluri et al.,
2023), transcriptions, or speaker attributes (Hao et al., 2023) has been used
as a new form of clue to extract the target sound source. Text prompts have
also been introduced in sound generation tasks, such as TextrolSpeech (Ji et
al., 2023), AudioGen (Kreuk et al., 2022), and Diffsound (Yang et al., 2023a).
Text-guided sound editing is similar to sound generation but provides a noisy
or corrupt sound as an additional input. Relevant work includes SpeechX (Wang
et al., 2023a) for speech enhancement, deletion, extraction, and content
editing, AUDIT (Wang et al., 2023b) for audio addition, removal, inpainting,
replacement, and super-resolution, and uSee (Yang et al., 2023b) for noise and
reverberation addition and deletion. The latest sound generation and editing
models, AudioLDM (Liu et al., 2023a, b), Voicebox (Le et al., 2023), and
Audiobox (Vyas et al., 2023) can perform style transfer, noise reduction, and
audio inpainting.

This work focuses on text-guided sound mixture-to-mixture editing, a new
category of text-guided audio applications. The problem we address here is a
generalized version of the target sound extraction, one of many tasks LCE can
solve. In addition, LCE is inherently different from existing sound generation
or editing models, which focus on generating a new sound or modifying sound
properties. LCE focuses on keeping every sound’s integrity and controls its
presence and volume for an enhanced hearing experience in a multi-sound
environment. The applications of LCE include hearing assistance, audio
engineering, and ambient sound control.

Multimodal Text-Audio Learning. The performance of text-guided audio
applications relies on the relevance of the text embedding to the audio
content. Given that most language models are trained without audio, achieving
better performance in text-guided audio tasks often involves a subsequent
language model finetuning step using paired text and audio data. There are two
common approaches for this. One approach is to use Contrastive Language-Audio
Pretraining (CLAP) (Elizalde et al., 2023a, b) or similar methods such as
Wav2clip (Wu et al., 2022), Audioclip (Guzhov et al., 2022), and CLAPSpeech
(Ye et al., 2023), which maximizes the similarity between embeddings of paired
text and audio. A pretrained language model is first finetuned with a
pretrained audio encoder using the CLAP objective. Then, the language model is
frozen and its text embeddings are used for various downstream tasks including
audio captioning and classification (Wang et al., 2023c; Liang et al., 2023),
sound extraction (Liu et al., 2023c), and audio generation (Liu et al.,
2023a). Another approach does not incorporate the intermediate CLAP step.
Instead, it directly optimizes the language model and the audio processing
model, such as a sound separation model (Kilgour et al., 2022; Hao et al.,
2023), together with the task-specific objective. We opt for the second
approach since the sound mixtures and text prompts in this work contain
multiple sources, and to our knowledge, there is no work using CLAP to learn
the text embedding corresponding to a mixture of sounds due to a lack of
relevant datasets and an increased complexity to learn every combination of
sources in a mixture. Nevertheless, we still implemented the CLAP method
trained on single sources for target speech or audio extraction and included
the results in Table 2 as a baseline.

Table 1: The task space $\mathcal{T}$ with 16 tasks grouped by what and how to
edit.

One Speech One Audio All Speeches or Audios Multiple Speeches and Audios All
Sounds Target Speech Extraction (TSE) Target Audio Extraction (TAE) Speech
Enhancement (SE) Multiple Sound Extraction or Removal (ME) Mult. Sound Ext. or
Rem. and Vol. Ctrl. (MEVC) Multiple Sound Volume Control (MVC) Target Speech
Removal (TSR) Target Audio Removal (TAR) Speech Removal (SR) Target Speech
Volume Up (TS$\uparrow$) Target Audio Volume Up (TA$\uparrow$) Speech Volume
Up (S$\uparrow$) Overall Volume Control (OVC) Target Speech Volume Down
(TS$\downarrow$) Target Audio Volume Down (TA$\downarrow$) Speech Volume Down
(S$\downarrow$)

Figure 2: An example sound mixture consists of a female speaker, a male
speaker, a helicopter, and a turkey. We write 6 example prompts covering
S$\uparrow$, SR, TAR, ME, MEVC, and TSE task. The unprocessed, edited, and the
target Mel spectrograms are plotted for comparison.

##  3 Sound Mixture-to-Mixture Editing

We formulate sound mixture-to-mixture editing in this section. An example
editing is shown in Figure 2.

###  3.1 Sources and Actions

Consider a sound mixture $x$, we can express it as the sum of $N$ different
sources:

$\displaystyle x=\sum_{i=1}^{N}s_{i}$ (1)

Each source can have different energy and can be either a human speech or a
non-speech sound:
$s_{i}\in\mathcal{S}=\mathcal{S}_{\text{speech}}\cup\mathcal{S}_{\text{audio}}$.
For the remainder of this paper, we will use the term audio for non-speech
sounds, including animal voices, sound effects, music, and noises. We can
describe each sound source $s_{i}$ by a semantic description $\tilde{s}_{i}$,
which can be the speaking style measured in one or multiple of gender, pitch,
tempo, energy, and emotion for a speech source or the class label for an audio
source. Let us define $\tilde{\mathcal{S}}$ as the set of all semantic
descriptions. In this work, we assume that no two sound sources share the same
semantic description in the sound mixture. In other words, no two speech
sources share the same speaking style, and no two audio sources share the same
class label. Therefore, every sound mixture is a combination of $N$ different
sources of different descriptions
$\\{\tilde{s}_{1},\tilde{s}_{2},...,\tilde{s}_{N}\\}$, with
$\tilde{s}_{i}\neq\tilde{s}_{j},\forall i\neq j\large$, and a description
$\tilde{s}_{i}$ can identify an unique source $s_{i}$ in the mixture.

Our goal is to edit the mixture $x$ to yield a new mixture $y$:

$\displaystyle y=\sum_{i=1}^{N}\alpha_{i}s_{i}$ (2)

which is made up by the same sources but different scaling factors applied to
them. The values of $\alpha_{i}$ are determined by the user’s desired editing
actions, which can be selected from the action space denoted as
$\mathcal{A}=\\{\textit{removing}\hskip 3.00003pt(0),\textit{keeping}\hskip
3.00003pt(1),\textit{increasing the volume}\hskip 3.00003pt(\uparrow),\\\
\textit{decreasing the volume}\hskip 3.00003pt(\downarrow)\\}$. These actions
correspond to scaling factors represented by
$\mathcal{A}_{\alpha}=\\{0,1,2,0.5\\}$. For example, a scaling factor
$\alpha_{i}=2$ implies increasing the volume of $s_{i}$ by 6 dB. If only one
$\alpha_{i}$ is equal to 1, and all others are equal to 0, it results the
extraction of $s_{i}$.

###  3.2 Editing Tasks

To specify both how and what to modify for a single source, we require a tuple
$(a_{i},\tilde{s}_{i})\in\mathcal{A}\times\tilde{\mathcal{S}}$ to specify both
the action $a_{i}$ and the source $\tilde{s}_{i}$. To specify how and what to
modify for a mixture of $N$ sources, we require a collection of $N$ such
tuples: $\pi=\\{(a_{1},\tilde{s}_{1}),...,(a_{N},\tilde{s}_{N})\\}$. We call
$\pi$ a sound mixture-to-mixture editing instruction. If the composition
$\\{\tilde{s}_{1},\tilde{s}_{2},...,\tilde{s}_{N}\\}$ of the sound mixture is
fixed, $\mathcal{A}^{N}$ represents the set of all possible edits to this
mixture. For instance, if a mixture contains four sources, there is a total of
$|\mathcal{A}|^{4}-2=254$ possible edits, excluding the trivial identity and
silence cases. One example editing
$[0,\downarrow,\uparrow,1]\in\mathcal{A}^{4}$ instructs the following: remove
$s_{1}$, turn down the volume of $s_{2}$, turn up the volume of $s_{3}$, and
keep $s_{4}$.

Notice that many elements in $\mathcal{A}^{N}$ are similar. For example,
$[0,\downarrow,\uparrow,1]$ and $[\downarrow,0,1,\uparrow]$ specify same
actions but applied to different sources. To more accurately evaluate the
performance of LCE and to maintain consistency with existing literature, we
categorize all elements in $\mathcal{A}^{N}$ into the task space
$\mathcal{T}$. For any sound mixture containing at least two speech sources
and at least two audio sources, there are 16 different tasks in $\mathcal{T}$.
In Table 1, we present $\mathcal{T}$ based on which source(s) are edited (in
columns) and the manner in which they are edited (in rows). Details and
examples of each task are provided in Appendix C.

The task space $\mathcal{T}$ is a super set of the target speech or audio
extraction and removal tasks studied in previous works (Liu et al., 2023c; Hao
et al., 2023; Veluri et al., 2023). It further includes volume control and
multiple sound editing tasks.

###  3.3 Text Prompts

Although $\pi=\\{(a_{1},\tilde{s}_{1}),...,(a_{N},\tilde{s}_{N})\\}$ itself
can accurately specify a sound mixture-to-mixture editing instruction, writing
an instruction in this format can be inconvenient and difficult for humans,
because we prefer to use natural language rather than numbers and symbols. As
a result, we adopted using $p=\mathcal{H}(\pi)$ as the input for LCE. Here,
$p$ is an open-vocabulary text prompt written in natural language by
$\mathcal{H}$, which can be either a human or a large language model (LLM).
Given the impracticality of hiring humans to write hundreds of thousands of
prompts, we leveraged a LLM to generate text prompts $p$ from $\pi$. The text
prompt generation pipeline can be found in Section 5 and Appendix B.

Figure 3: The components and the training paradigm of LCE.

##  4 Listen, Chat, and Edit

Listen, Chat, and Edit (LCE) is our solution to the sound mixture-to-mixture
editing problem. LCE edits a sound mixture $x$ given a natural language text
prompt $p$:

$\displaystyle\hat{y}=\text{LCE}(x,p)$ (3)

LCE includes two models: a PromptReader to read text prompts and a SoundEditor
to edit sound mixtures. A graphical illustration of it in details is depicted
in Figure 3.

###  4.1 Prompt Reader

The PromptReader is a language model that behaves as the inverse of
$\mathcal{H}$: It generates a $D$-dimensional text embedding
$z\in\mathbb{R}^{D}$ from a text prompt $p$, which encodes the information
about the sources and actions specified in $\pi$. We call $z$ a semantic
filter because it is generated from text to guide the SoundEditor to execute
proper filtering of sounds. We analyze and show the correspondence between the
semantic filter and the resulting sound filter in Appendix A.

$\displaystyle z=\textit{PromptReader}(p)$ (4)

We employed a pretrained language model GPT-2 (Radford et al., 2019) or LLaMA2
(Touvron et al., 2023) in the role of the PromptReader. We finetuned it, with
full parameters or low-rank approximation (Hu et al., 2022), using the
gradients backpropagated from the SoundEditor. While the pretrained language
model only grasps textual information, further joint training with SoundEditor
enables it to generate text embeddings that exhibit awareness of the acoustic
characteristics (e.g., pitch, emotion, animal voices, and instrument timbres)
written in the descriptions (see Appendix A). Consequently, the enhanced
embeddings facilitate the SoundEditor in more easily identifying the
corresponding sources noted in the text prompt.

###  4.2 Sound Editor

The SoundEditor is an acoustic model responsible for editing the sound mixture
$x$ based on the semantic filter $z$:

$\displaystyle\hat{y}=\textit{SoundEditor}(x,z)$ (5)

We adopted the architecture of source separation models for the SoundEditor
due to their abilities to isolate individual sources in a sound mixture.
However, it is essential to clarify that SoundEditor does not explicitly (in
any stage) separate all the sources in the mixture, individually editing them,
and sum them up. Instead, the model functions similar to a sound extraction
model to directly output a single sound that is the mix of all the edited
sources.

In particular, we followed the design of time-domain source separation network
Conv-TasNet (Luo & Mesgarani, 2019) and SepFormer (Subakan et al., 2021). Both
models incorporate a learnable linear encoder $\mathcal{E}:\mathbb{R}^{1\times
T}\rightarrow\mathbb{R}^{C\times L}$ for mapping the input waveform $x$ of $T$
samples into a $C$-dimensional latent representation $h_{x}$ and a learnable
linear decoder $\mathcal{D}:\mathbb{R}^{C\times
L}\rightarrow\mathbb{R}^{1\times T}$ for mapping the output sound from the
latent space $h_{\hat{y}}$ back to the waveform $\hat{y}$.

$\displaystyle h_{x}=\mathcal{E}(x),\quad\hat{y}=\mathcal{D}(h_{\hat{y}})$ (6)

Between the encoder and the decoder, a mask network denoted as $\mathcal{M}$
estimates a single editing mask $h_{m}\in\mathbb{R}^{C\times L}$ that
selectively extracts, removes, amplifies, or reduces samples in $h_{x}$
according to $z$ through element-wise multiplication on $h_{x}$:

$\displaystyle h_{m}=\mathcal{M}(x,z),\quad h_{\hat{y}}=h_{m}\otimes h_{x}$
(7)

$\mathcal{M}$ consists of $R$ editing blocks with the same structure: temporal
convolutional network for Conv-TasNet or dual-path transformer for SepFormer.
We guide the behavior of every editing block by the semantic filter $z$ using
Feature-wise Linear Modulation (FiLM) (Perez et al., 2018). Before the $i$th
block, the text-guided feature $\tilde{h}_{i}$ is computed by:

$\displaystyle\tilde{h}_{i}=\gamma_{i}h_{i}+\beta_{i},\quad 0\leq i\leq R-1$
(8)
$\displaystyle\gamma_{i}=\mathcal{F}_{i}(z),\quad\beta_{i}=\mathcal{G}_{i}(z)$
(9)

$\mathcal{F}_{i}:\mathbb{R}^{D}\rightarrow\mathbb{R}^{C}$ and
$\mathcal{G}_{i}:\mathbb{R}^{D}\rightarrow\mathbb{R}^{C}$ are modeled by two
two-layer perceptrons which project the $z$ to the same dimension of $h_{i}$.
Afterwards, $\gamma_{i}$ and $\beta_{i}$ are replicated along the time axis to
modulate $h_{i}$. This ensures that the text prompt maintains time-invariant
influence over the duration of the mixture signal.

###  4.3 Training Objective

Each training instance is a tuple $(x,y,p)$ including an input mixture $x$, a
target mixture $y$, and a text prompt $p$. The sources in the mixture
$\\{s_{1},s_{2},...,s_{N}\\}$ and the underlying symbolic instruction $\pi$
used in the creation of $y$ and $p$ are hidden from the system. LCE estimates
a mixture $\hat{y}$ given $x$ and $p$, following Equation 4 and 5. All model
components are jointly optimized by maximizing the signal-to-noise ratio (SNR)
between the target and the estimated mixture:

$\displaystyle\text{SNR}(\hat{y},y)=10\log_{10}\Biggl{(}\frac{\|y\|^{2}}{\|y-\hat{y}\|^{2}}\Biggr{)}$
(10)

Table 2: Editing performance (SNRi in dB) with various model choices for
extraction (E), removal (R), or volume control (VC), up ($\uparrow$) and down
($\downarrow$), applied to target (T) speech (S), audio (A), or the overall
(O) mixture.

SoundEditor PromptReader TSE TSR TS$\uparrow$ TS$\downarrow$ TAE TAR
TA$\uparrow$ TA$\downarrow$ SE Conv-TasNet CLAP (Speech) 8.8 - - - - - - - -
CLAP (Audio) - - - - 8.1 - - - - Conv-TasNet GPT-2 (finetuned) 8.0 3.1 3.4 3.7
7.5 1.6 2.6 2.4 7.8 LLaMA2 (frozen) 7.0 2.2 2.7 2.7 7.1 1.3 2.0 1.9 5.1 LLaMA2
(LoRA) 9.7 4.6 4.7 4.8 8.2 2.5 3.1 3.0 8.2 Sepformer GPT-2 (finetuned) 11.1
6.1 6.5 6.9 8.8 3.2 4.1 4.0 9.7 LLaMA2 (LoRA) 13.1 7.9 8.1 8.3 10.1 4.5 4.8
4.8 10.7 Continued SoundEditor PromptReader SR S$\uparrow$ S$\downarrow$ ME
MVC MEVC OVC Average Conv-TasNet GPT-2 (finetuned) 8.0 6.4 6.0 3.4 2.6 3.3
42.7 7.0 LLaMA2 (frozen) 6.8 5.0 5.0 2.8 1.3 2.0 39.5 5.9 LLaMA2 (LoRA) 8.5
7.4 7.0 4.2 3.9 4.6 40.3 7.8 Sepformer GPT-2 (finetuned) 10.0 9.1 8.6 5.2 4.5
5.0 43.8 9.2 LLaMA2 (LoRA) 11.1 10.2 9.6 6.5 6.3 6.8 44.2 10.4

##  5 Dataset

Due to the absence of natural language instruction-prompted sound editing
datasets, we generated our own sound mixtures and text prompts. This section
highlights key features of our dataset. See more details in Appendix B.

Sound Sources Speech sources were from TextrolSpeech (Ji et al., 2023) and
audio sources were from VGGSound (Chen et al., 2020) (for training and in-
domain evaluation) or FSD50K (Fonseca et al., 2021) (for zero-shot
evaluation). Speakers and audios were split into training, validation, and
testing sets. All sources were resampled to 16 kHz and cropped or padded to 5
seconds.

Sound Mixtures We generated 100k, 5k, and 10k mixtures of 2 speech + 2 audio
sources from TextrolSpeech and VGGSound for training, validation, and testing.
These four-source mixtures, featuring a variety of categories, simulate
acoustic scenes commonly encountered in daily life. We also generated 5,000
mixtures of 2 speech, 2 audio, 2 speech + 1 audio, 1 speech + 2 audio for
zero-shot evaluation on unseen numbers of sources, and 5,000 mixtures of 2
speech + 2 audio sources from FSD50K with either all classes or unseen classes
for zero-shot evaluation on unseen sounds.

Editing Tasks For every mixture, we first evenly assigned a task
$t\in\mathcal{T}$ (e.g. target speech extraction), and then assigned a
particular instruction $\pi$ of task $t$ (e.g. extracting a particular
speaker). The reported performance for each task was calculated by averaging
the editing performance of all mixtures assigned to that task.

Text Prompts We requested GPT-3.5 Turbo to write a text prompt five times from
the assigned instruction $\pi$ for each mixture. To improve the diversity of
the text prompts, we asked it to try imperative or interrogative sentences and
substitute synonyms for class labels and style keywords such as female, male,
low, high, pitch, and tempo. To improve the naturalness of the text prompts,
we omitted to mention a source if the action was keeping, unless we
specifically wanted to extract it. Furthermore, we described a speaker by a
random subset instead of all of five style attributes. More efforts to make
the text prompts more diverse and natural are discussed in Appendix B.

Table 3: Zero-shot editing SNRi (dB) for sound mixtures with unseen source
composition

Sound Mixtures TSE TSR TS$\uparrow$ TS$\downarrow$ TAE TAR TA$\uparrow$
TA$\downarrow$ 2 Speech 12.6 n/a 12.7 12.7 n/a n/a n/a n/a 2 Audio n/a n/a n/a
n/a 4.3 n/a 4.3 5.3 2 Speech + 1 Audio 9.7 7.1 9.6 10.0 n/a n/a n/a n/a 1
Speech + 2 Audio n/a n/a n/a n/a 7.4 3.6 5.0 4.7 2 Speech + 2 Audio (FSD50K,
all) 13.5 8.9 8.8 8.9 9.3 5.0 4.9 5.4 2 Speech + 2 Audio (FSD50K, unseen) 13.4
9.5 9.3 9.2 9.0 3.3 3.9 4.4 2 Speech + 2 Audio (VGGSound) 13.1 7.9 8.1 8.3
10.1 4.5 4.8 4.8 Continued Sound Mixtures SE SR S$\uparrow$ S$\downarrow$ ME
MVC MEVC OVC 2 Speech n/a n/a n/a n/a n/a 10.3 12.0 37.8 2 Audio n/a n/a n/a
n/a n/a 3.4 5.2 25.9 2 Speech + 1 Audio 9.4 12.8 10.9 9.8 n/a 7.0 29.5 8.1 1
Speech + 2 Audio 11.8 8.4 9.2 9.7 n/a 4.9 6.4 28.3 2 Speech + 2 Audio (FSD50K,
all) 12.4 12.8 11.4 11.1 5.9 6.8 7.6 47.2 2 Speech + 2 Audio (FSD50K, unseen)
12.6 12.5 11.5 11.7 5.8 6.0 7.2 47.6 2 Speech + 2 Audio (VGGSound) 10.7 11.1
10.2 9.6 6.5 6.3 6.8 44.2

##  6 Experiments

We evaluated LCE’s performance across 16 tasks using different sound editor
and prompt reader models. Because our model is the first of its kind, no other
model that can read text prompts and perform all 16 tasks as LCE can be used
as the baseline for comparison. Therefore, we used the specialized text-guided
target audio or speech extraction models in Table 2 and 5 as a baseline. Then,
we conducted an in-depth analysis of our top-performing system across all
editing tasks. Furthermore, we performed zero-shot evaluations on mixtures of
unseen number of sources and unseen types of sounds and studied the impact of
the speaking style for speech editing. For insights into alternative model
configurations, including a causal LCE model, and the impact of text prompt
quality, see Appendix E. Details on the model and training are available in
Appendix F. Analyses of text and audio features are presented in Appendix A.
Examples of edited spectrograms can be found in Appendix G.

###  6.1 Model-wise Performance Analysis

We evaluated the editing performance of LCE by SNR improvement (SNRi) with the
target sound mixtures. We used extraction-only system with CLAP in (Liu et
al., 2023c) as the baseline models 111We used the official CLAP checkpoint
(Elizalde et al., 2023a) for TAE, and trained another CLAP on TextrolSpeech
for TSE. Conv-TasNet was used in place of the separation model in (Liu et al.,
2023c) for a fair comparison with other experiments.. Tables 2 compares LCE to
the baseline and compares LCE with different SoundEditor and PromptReader.
Three of five LCE systems outperform the CLAP baselines trained on speech or
audio sources which are specialized at TSE or TAE. Moreover, results show the
editing performance depends on both SoundEditor and PromptReader: a stronger
SoundEditor, Sepformer, and a stronger PromptReader, LLaMA2, outperform their
weaker counterparts, Conv-TasNet and GPT-2, respectively. Sepformer & LLaMA2
(LoRA) together achieve the highest SNRi in all the 16 tasks. We also
investigated the importance of finetuning the language model, LLaMA2 (FROZEN)
or (LORA). With the same Conv-TasNet, the LLaMA2 model finetuned with LoRA
achieved higher SNRi in all the tasks compared to a frozen one. Although
several works in text-audio learning (Kreuk et al., 2022; Wang et al., 2023b)
used a frozen language model, our results demonstrate that finetuning the
language model is essential for effectively handling text prompts that
describe multiple sounds.

###  6.2 Task-wise Performance Analysis

We conducted a thorough analysis of the editing performance of the Sepformer &
LLaMA2 (LoRA) system across all the 16 tasks. The distributions of SNR for
both unprocessed and edited mixtures and the corresponding SNRi are plotted in
Figure 4. For every task, LCE correctly edits the sound mixture according to
the text prompt. This proficiency is evident as, depending on the specific
task, between 89.0% and 99.8% of the mixtures exhibited a positive SNRi. We
also notice that tasks vary in difficulty. Overall volume control (OVC),
target speech/audio extraction/removal (TSE, TSR, TAE, TAR), and speech
enhancement/removal (SE, SR) perform better, with an average SNRi exceeding 10
dB. These tasks are also more commonly studied in the existing literature. In
contrast, target speech/audio removal (TSR, TAR) or volume control
(TS$\uparrow$, TS$\downarrow$, TA$\uparrow$, TA$\downarrow$) and multiple
sound extraction or/and volume control (MVE, MVC, MEVC) are more challenging,
which are also less studied in previous research. Finally, we observe that LCE
performs on average 3 dB worse on audio tasks (TAE, TAR, TA$\uparrow$,
TA$\downarrow$) than speech tasks (TSE, TSR, TS$\uparrow$, TS$\downarrow$).
This performance gap could be explained by the presence of more natural noises
in VGGSound and the involvement of audios from a broader range of categories
compared to speech.

Figure 4: The distribution (25th, 50th, and 75th quartile) of SNR of the
unprocessed (blue) and LCE edited (orange) sound mixtures and the SNR
improvement (SNRi, pink), calculated with respect to the target mixtures. The
ratios of mixtures with an improvement (i.e. SNRi $>0$) for each task are
written below the task labels.

###  6.3 Zero-shot Evaluation on Unseen Mixtures

Table 3 presents the zero-shot performance of the Sepformer & LLaMA2 (LoRA)
system on sound mixtures with different number of sources (rows 1 to 4)
222Some entries are filled with N/A because some tasks are not defined (e.g.
TAE for 2 Speech) or are equivalent to another task (e.g. TSE = TSR, both
resulting in one speech left, for 2 Speech). or including unseen audio sources
(rows 5 and 6), compared to the in-domain performance (the last row). The
performances on 2 Speech and 2 Speech + 1 Audio mixtures are comparable to the
performance of 2 Speech + 2 Audio mixtures used for training, while the
performances on 1 Speech + 2 Audio and 2 Audio mixtures are worse but still at
least 3.4 dB better than the original mixtures. Interestingly, the zero-shot
performance on audio sources from FSD50K is better in most tasks compared to
VGGSound, possibly because FSD50K has higher audio quality and most audio
labels are covered in VGGSound. To eliminate the latter factor, we evaluated
LCE on audios with strictly unseen labels 333Synonyms were considered for
excluding seen labels. from FSD50K and observe a 0.3 to 1.7 dB performance
drop on audio tasks (TAE, TAR, TA$\uparrow$, TA$\downarrow$) but similar
performance on the rest. To summarize, LCE shows remarkable zero-shot editing
performance on mixtures with unseen compositions. Achieving better performance
may be possible through extra training with different numbers of sources and
high-quality audio sources from new categories.

Table 4: Zero-shot speech editing SNRi (dB) for two speakers with one or
multiple style variation.

Differ in TSE TS$\uparrow$ TS$\downarrow$ MVC Average Gender 11.4 12.5 13.0
10.3 11.8 Pitch 11.3 11.7 11.6 8.7 10.8 Tempo 6.7 5.4 6.6 3.3 5.5 Energy 10.9
11.5 11.8 10.7 11.2 Emotion 11.4 11.9 11.6 9.6 11.1 Multiple 13.2 13.2 13.0
10.6 12.5

Table 5: Zero-shot and finetuned TSE SI-SDR (dB) for two speakers differing in
gender or energy, compared to LLM-TSE (the number in parentheses is evaluated
with an additional audio clue).

Differ in zero-shot 2.8-hour Finetuned LLM-TSE Gender 8.4 12.1 10.4 (10.9)
Energy 7.0 10.8 8.9 (9.4)

###  6.4 Zero-shot Style-wise Performance Analysis

Lastly, Table 4 evaluates the zero-shot two-speaker editing performance of two
speakers with controlled style difference to analyze the influence of each
style attribute (see Appendix D for the full result with both SoundEditors).
Our results show that LCE can effectively edit speakers as long as they have
one distinct style attribute. A difference in gender, energy, pitch, or
emotion results in an average 5.3 dB or higher SNRi compared to a difference
in tempo, which requires a longer window to calculate the speaking speed.
LCE’s ability to edit the target speaker(s) is further improved when two
speakers differ in multiple style attributes.

Table 5 compares our performance with the target speech extraction performance
reported in LLM-TSE (Hao et al., 2023) in gender and energy evaluated on
similar two-speaker mixtures 444Mixtures were generated with the same SNR
distribution.. The performance was measured in Scale-Invariant Signal-to-
Distortion ratio (SI-SDR) (Roux et al., 2018). While LCE’s zero-shot
performance falls behind that of LLM-TSE by 2 dB, after finetuning with 2k
two-speaker mixtures (equivalent to 2.8 hours), LCE surpasses LLM-TSE, even if
an additional audio clue is provided to the latter. This demonstrates that
LCE, as a versatile sound mixture editor, can outperform an expert model in a
specific task after finetuning with limited data.

##  7 Conclusion

We develop Listen, Chat, and Edit, the first text-guided sound mixture editor
that can handle a variaty of editing tasks for speech and audio mixtures. Our
development also introduces the first text-prompted sound mixture dataset of
160 hours, featuring speakers with diverse styles, audio from hundreds of
classes, and five generic text prompts written for each mixture. LCE trained
on this dataset demonstrates an SNR improvement of over 10 dB averaged across
all 16 tasks, superior performance than ad-hoc sound extraction models, and a
generalization ability to unfamiliar sound mixtures of unseen sounds or
different numbers of sound sources. Our future efforts will target two
directions: One is to scale up the size of the model and the dataset to reach
a higher performance on the current 16-task benchmark. The other is to
introduce new tasks to sound mixture-to-mixture editing, such as
dereverberation, restoration, and content editing of speech and audio.

##  8 Broader Impact

In noisy environments with multiple sound sources present, human listeners
often find their listening abilities reduced (Schäfer et al., 2018;
Vanthornhout et al., 2018). This issue intensifies for those with hearing
impairments, making it hard to distinguish and focus on specific sounds
(Marrone et al., 2008; Shinn-Cunningham & Best, 2008; Dai et al., 2018). LCE,
as the first text-guided sound mixture editor, is capable of adjusting the
presence and volume of specific sound sources according to textual
instructions from listeners. Therefore, LCE could serve as an effective tool
in assisting those with hearing challenges. Importantly, LCE is safe to use.
It operates using general descriptions of speaker styles and audio categories
without retaining or employing specific characteristics of any speaker.
Furthermore, it only edits existing sources within the mix, ensuring it does
not generate malicious sounds that are not initially present. LCE offers a
promising and safe enhancement for auditory experience without any evident
ethical concerns or negative social impacts.

##  9 Acknowledgement

This work was funded by the National Institutes of Health (NIH-NIDCD) and a
grant from Marie-Josee and Henry R. Kravis. We would like to thank Gavin
Mischler for suggesting the name Listen, Chat, and Edit.

## References

  * Afouras et al. (2018) Afouras, T., Chung, J. S., and Zisserman, A.  The conversation: Deep audio-visual speech enhancement.  _arXiv preprint arXiv:1804.04121_ , 2018.
  * Borsdorf et al. (2021) Borsdorf, M., Li, H., and Schultz, T.  Target language extraction at multilingual cocktail parties.  In _2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_ , pp. 717–724, 2021.  doi: 10.1109/ASRU51503.2021.9688052.
  * Bryan et al. (2013) Bryan, N. J., Mysore, G. J., and Wang, G.  Source separation of polyphonic music with interactive user-feedback on a piano roll display.  In _ISMIR_ , pp. 119–124, 2013.
  * Chen et al. (2020) Chen, H., Xie, W., Vedaldi, A., and Zisserman, A.  VGGSound: A large-scale audio-visual dataset.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 721–725. IEEE, 2020.
  * Cherry (1953) Cherry, E. C.  Some experiments on the recognition of speech, with one and with two ears.  _The Journal of the acoustical society of America_ , 25(5):975–979, 1953.
  * Clark & Swanepoel (2014) Clark, J. L. and Swanepoel, D. W.  Technology for hearing loss–as we know it, and as we dream it.  _Disability and Rehabilitation: Assistive Technology_ , 9(5):408–413, 2014.
  * Dai et al. (2018) Dai, L., Best, V., and Shinn-Cunningham, B. G.  Sensorineural hearing loss degrades behavioral and physiological measures of human spatial selective auditory attention.  _Proceedings of the National Academy of Sciences_ , 115(14):E3286–E3295, 2018.
  * Défossez et al. (2021) Défossez, A., Usunier, N., Bottou, L., and Bach, F.  Music source separation in the waveform domain, 2021.
  * Elizalde et al. (2023a) Elizalde, B., Deshmukh, S., Al Ismail, M., and Wang, H.  Clap learning audio concepts from natural language supervision.  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 1–5. IEEE, 2023a.
  * Elizalde et al. (2023b) Elizalde, B., Deshmukh, S., and Wang, H.  Natural language supervision for general-purpose audio representations, 2023b.  URL https://arxiv.org/abs/2309.05767.
  * Elminshawi et al. (2022) Elminshawi, M., Mack, W., Chakrabarty, S., and Habets, E. A.  New insights on target speaker extraction.  _arXiv preprint arXiv:2202.00733_ , 2022.
  * Fonseca et al. (2021) Fonseca, E., Favory, X., Pons, J., Font, F., and Serra, X.  FSD50K: an open dataset of human-labeled sound events.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 30:829–852, 2021.
  * Gu et al. (2019) Gu, R., Chen, L., Zhang, S.-X., Zheng, J., Xu, Y., Yu, M., Su, D., Zou, Y., and Yu, D.  Neural spatial filter: Target speaker speech separation assisted with directional information.  In _Interspeech_ , 2019.  URL https://api.semanticscholar.org/CorpusID:197629146.
  * Guzhov et al. (2022) Guzhov, A., Raue, F., Hees, J., and Dengel, A.  AudioCLIP: Extending clip to image, text and audio.  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 976–980. IEEE, 2022.
  * Han et al. (2019) Han, C., O’Sullivan, J., Luo, Y., Herrero, J., Mehta, A. D., and Mesgarani, N.  Speaker-independent auditory attention decoding without access to clean speech sources.  _Science advances_ , 5(5):eaav6134, 2019.
  * Hao et al. (2023) Hao, X., Wu, J., Yu, J., Xu, C., and Tan, K. C.  Typing to listen at the cocktail party: Text-guided target speaker extraction, 2023.
  * Haykin & Chen (2005) Haykin, S. and Chen, Z.  The cocktail party problem.  _Neural computation_ , 17(9):1875–1902, 2005.
  * Heitkaemper et al. (2019) Heitkaemper, J., Fehér, T., Freitag, M. J., and Häb-Umbach, R.  A study on online source extraction in the presence of changing speaker positions.  In _International Conference on Statistical Language and Speech Processing_ , 2019.  URL https://api.semanticscholar.org/CorpusID:203565755.
  * Hershey et al. (2015) Hershey, J. R., Chen, Z., Roux, J. L., and Watanabe, S.  Deep clustering: Discriminative embeddings for segmentation and separation, 2015.
  * Hu et al. (2022) Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=nZeVKeeFYf9.
  * Ji et al. (2023) Ji, S., Zuo, J., Fang, M., Jiang, Z., Chen, F., Duan, X., Huai, B., and Zhao, Z.  TextrolSpeech: A text style control speech corpus with codec language text-to-speech models.  _arXiv preprint arXiv:2308.14430_ , 2023.
  * Kates (2008) Kates, J. M.  _Digital hearing aids_.  Plural publishing, 2008.
  * Kavalerov et al. (2019) Kavalerov, I., Wisdom, S., Erdogan, H., Patton, B., Wilson, K., Le Roux, J., and Hershey, J. R.  Universal sound separation.  In _2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)_ , pp. 175–179, 2019.  doi: 10.1109/WASPAA.2019.8937253.
  * Kilgour et al. (2022) Kilgour, K., Gfeller, B., Huang, Q., Jansen, A., Wisdom, S., and Tagliasacchi, M.  Text-driven separation of arbitrary sounds.  _arXiv preprint arXiv:2204.05738_ , 2022.
  * Kingma & Ba (2014) Kingma, D. P. and Ba, J.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kreuk et al. (2022) Kreuk, F., Synnaeve, G., Polyak, A., Singer, U., Défossez, A., Copet, J., Parikh, D., Taigman, Y., and Adi, Y.  AudioGen: Textually guided audio generation.  _arXiv preprint arXiv:2209.15352_ , 2022.
  * Launer et al. (2016) Launer, S., Zakis, J. A., and Moore, B. C.  Hearing aid signal processing.  _Hearing aids_ , pp. 93–130, 2016.
  * Le et al. (2023) Le, M., Vyas, A., Shi, B., Karrer, B., Sari, L., Moritz, R., Williamson, M., Manohar, V., Adi, Y., Mahadeokar, J., et al.  Voicebox: Text-guided multilingual universal speech generation at scale.  _arXiv preprint arXiv:2306.15687_ , 2023.
  * Liang et al. (2023) Liang, J., Liu, X., Liu, H., Phan, H., Benetos, E., Plumbley, M. D., and Wang, W.  Adapting language-audio models as few-shot audio learners.  _arXiv preprint arXiv:2305.17719_ , 2023.
  * Liu et al. (2023a) Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D., Wang, W., and Plumbley, M. D.  AudioLDM: Text-to-audio generation with latent diffusion models.  _Proceedings of the International Conference on Machine Learning_ , 2023a.
  * Liu et al. (2023b) Liu, H., Tian, Q., Yuan, Y., Liu, X., Mei, X., Kong, Q., Wang, Y., Wang, W., Wang, Y., and Plumbley, M. D.  AudioLDM 2: Learning holistic audio generation with self-supervised pretraining.  _arXiv preprint arXiv:2308.05734_ , 2023b.
  * Liu et al. (2022) Liu, X., Liu, H., Kong, Q., Mei, X., Zhao, J., Huang, Q., Plumbley, M. D., and Wang, W.  Separate what you describe: Language-queried audio source separation.  In _INTERSPEEH_ , pp. 1801–1805, 2022.
  * Liu et al. (2023c) Liu, X., Kong, Q., Zhao, Y., Liu, H., Yuan, Y., Liu, Y., Xia, R., Wang, Y., Plumbley, M. D., and Wang, W.  Separate anything you describe.  _arXiv preprint arXiv:2308.05037_ , 2023c.
  * Luo & Mesgarani (2019) Luo, Y. and Mesgarani, N.  Conv-tasnet: Surpassing ideal time–frequency magnitude masking for speech separation.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 27(8):1256–1266, August 2019.  ISSN 2329-9304.  doi: 10.1109/taslp.2019.2915167.  URL http://dx.doi.org/10.1109/TASLP.2019.2915167.
  * Luo & Yu (2022) Luo, Y. and Yu, J.  Music source separation with band-split rnn.  _arXiv preprint arXiv:2209.15174_ , 2022.
  * Ma et al. (2021) Ma, P., Petridis, S., and Pantic, M.  End-to-end audio-visual speech recognition with conformers, 2021.
  * Marrone et al. (2008) Marrone, N., Mason, C. R., and Kidd Jr, G.  The effects of hearing loss and age on the benefit of spatial separation between multiple talkers in reverberant rooms.  _The Journal of the Acoustical Society of America_ , 124(5):3064–3075, 2008.
  * McDermott (2009) McDermott, J. H.  The cocktail party problem.  _Current Biology_ , 19(22):R1024–R1027, 2009.
  * Perez et al. (2018) Perez, E., Strub, F., De Vries, H., Dumoulin, V., and Courville, A.  FiLM: Visual reasoning with a general conditioning layer.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 32, 2018.
  * Petermann et al. (2022) Petermann, D., Wichern, G., Wang, Z.-Q., and Roux, J. L.  The cocktail fork problem: Three-stem audio separation for real-world soundtracks.  In _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 526–530, 2022.  doi: 10.1109/ICASSP43922.2022.9746005.
  * Radford et al. (2019) Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I.  Language models are unsupervised multitask learners.  2019\.
  * Ravanelli et al. (2021) Ravanelli, M., Parcollet, T., Plantinga, P., Rouhe, A., Cornell, S., Lugosch, L., Subakan, C., Dawalatabad, N., Heba, A., Zhong, J., et al.  SpeechBrain: A general-purpose speech toolkit.  _arXiv preprint arXiv:2106.04624_ , 2021.
  * Roux et al. (2018) Roux, J. L., Wisdom, S., Erdogan, H., and Hershey, J. R.  SDR – half-baked or well done?  _ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 626–630, 2018.  URL https://api.semanticscholar.org/CorpusID:53246666.
  * Schäfer et al. (2018) Schäfer, P. J., Corona-Strauss, F. I., Hannemann, R., Hillyard, S. A., and Strauss, D. J.  Testing the limits of the stimulus reconstruction approach: Auditory attention decoding in a four-speaker free field environment.  _Trends in Hearing_ , 22, 2018.  URL https://api.semanticscholar.org/CorpusID:56531160.
  * Settle et al. (2018) Settle, S., Le Roux, J., Hori, T., Watanabe, S., and Hershey, J. R.  End-to-end multi-speaker speech recognition.  In _2018 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pp. 4819–4823. IEEE, 2018.
  * Shinn-Cunningham & Best (2008) Shinn-Cunningham, B. G. and Best, V.  Selective attention in normal and impaired hearing.  _Trends in amplification_ , 12(4):283–299, 2008.
  * Subakan et al. (2021) Subakan, C., Ravanelli, M., Cornell, S., Bronzi, M., and Zhong, J.  Attention is all you need in speech separation, 2021.
  * Touvron et al. (2023) Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T.  Llama 2: Open foundation and fine-tuned chat models, 2023.
  * Turpault et al. (2020) Turpault, N., Wisdom, S., Erdogan, H., Hershey, J., Serizel, R., Fonseca, E., Seetharaman, P., and Salamon, J.  Improving sound event detection in domestic environments using sound separation.  _arXiv preprint arXiv:2007.03932_ , 2020.
  * Tzinis et al. (2021) Tzinis, E., Wisdom, S., Jansen, A., Hershey, S., Remez, T., Ellis, D. P. W., and Hershey, J. R.  Into the wild with audioscope: Unsupervised audio-visual separation of on-screen sounds, 2021.
  * Van Eyndhoven et al. (2016) Van Eyndhoven, S., Francart, T., and Bertrand, A.  Eeg-informed attended speaker extraction from recorded speech mixtures with application in neuro-steered hearing prostheses.  _IEEE Transactions on Biomedical Engineering_ , 64(5):1045–1056, 2016.
  * Vanthornhout et al. (2018) Vanthornhout, J., Decruy, L., Wouters, J., Simon, J. Z., and Francart, T.  Speech intelligibility predicted from neural entrainment of the speech envelope.  _Journal of the Association for Research in Otolaryngology_ , 19:181–191, 2018.
  * Veluri et al. (2023) Veluri, B., Itani, M., Chan, J., Yoshioka, T., and Gollakota, S.  Semantic hearing: Programming acoustic scenes with binaural hearables.  In _Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology_ , UIST ’23, New York, NY, USA, 2023. Association for Computing Machinery.  ISBN 9798400701320.  doi: 10.1145/3586183.3606779.  URL https://doi.org/10.1145/3586183.3606779.
  * Vyas et al. (2023) Vyas, A., Shi, B., Le, M., Tjandra, A., Wu, Y.-C., Guo, B., Zhang, J., Zhang, X., Adkins, R., Ngan, W., Wang, J., Cruz, I., Akula, B., Akinyemi, A., Ellis, B., Moritz, R., Yungster, Y., Rakotoarison, A., Tan, L., Summers, C., Wood, C., Lane, J., Williamson, M., and Hsu, W.-N.  Audiobox: Unified audio generation with natural language prompts, 2023.
  * Wang et al. (2019) Wang, Q., Muckenhirn, H., Wilson, K., Sridhar, P., Wu, Z., Hershey, J. R., Saurous, R. A., Weiss, R. J., Jia, Y., and Moreno, I. L.  VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking.  In _Proc. Interspeech 2019_ , pp. 2728–2732, 2019.  doi: 10.21437/Interspeech.2019-1101.  URL http://dx.doi.org/10.21437/Interspeech.2019-1101.
  * Wang et al. (2023a) Wang, X., Thakker, M., Chen, Z., Kanda, N., Eskimez, S. E., Chen, S., Tang, M., Liu, S., Li, J., and Yoshioka, T.  SpeechX: Neural codec language model as a versatile speech transformer.  _arXiv preprint arXiv:2308.06873_ , 2023a.
  * Wang et al. (2023b) Wang, Y., Ju, Z., Tan, X., He, L., Wu, Z., Bian, J., and Zhao, S.  Audit: Audio editing by following instructions with latent diffusion models.  In _NeurIPS 2023_ , December 2023b.  URL https://www.microsoft.com/en-us/research/publication/audit-audio-editing-by-following-instructions-with-latent-diffusion-models/.
  * Wang et al. (2023c) Wang, Z., Subakan, C., Subramani, K., Wu, J., Tavares, T., Ayres, F., and Smaragdis, P.  Unsupervised improvement of audio-text cross-modal representations.  _arXiv preprint arXiv:2305.01864_ , 2023c.
  * Wisdom et al. (2021) Wisdom, S., Erdogan, H., Ellis, D. P. W., Serizel, R., Turpault, N., Fonseca, E., Salamon, J., Seetharaman, P., and Hershey, J. R.  What’s all the fuss about free universal sound separation data?  In _ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 186–190, 2021.  doi: 10.1109/ICASSP39728.2021.9414774.
  * Wu et al. (2022) Wu, H.-H., Seetharaman, P., Kumar, K., and Bello, J. P.  Wav2CLIP: Learning robust audio representations from clip.  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 4563–4567. IEEE, 2022.
  * Yang et al. (2023a) Yang, D., Yu, J., Wang, H., Wang, W., Weng, C., Zou, Y., and Yu, D.  Diffsound: Discrete diffusion model for text-to-sound generation.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 2023a.
  * Yang et al. (2023b) Yang, M., Zhang, C., Xu, Y., Xu, Z., Wang, H., Raj, B., and Yu, D.  uSee: Unified speech enhancement and editing with conditional diffusion models.  _arXiv preprint arXiv:2310.00900_ , 2023b.
  * Ye et al. (2023) Ye, Z., Huang, R., Ren, Y., Jiang, Z., Liu, J., He, J., Yin, X., and Zhao, Z.  CLAPSpeech: Learning prosody from text context with contrastive language-audio pre-training.  _arXiv preprint arXiv:2305.10763_ , 2023.
  * Yu et al. (2017) Yu, D., Kolbæk, M., Tan, Z.-H., and Jensen, J.  Permutation invariant training of deep models for speaker-independent multi-talker speech separation, 2017.
  * Zhao et al. (2018) Zhao, H., Gan, C., Rouditchenko, A., Vondrick, C., McDermott, J., and Torralba, A.  The sound of pixels.  In _The European Conference on Computer Vision (ECCV)_ , September 2018.
  * Zmolikova et al. (2023) Zmolikova, K., Delcroix, M., Ochiai, T., Kinoshita, K., Černocký, J., and Yu, D.  Neural target speech extraction: An overview.  _IEEE Signal Processing Magazine_ , 40(3):8–29, May 2023.  ISSN 1558-0792.  doi: 10.1109/msp.2023.3240008.  URL http://dx.doi.org/10.1109/MSP.2023.3240008.
  * Žmolíková et al. (2019) Žmolíková, K., Delcroix, M., Kinoshita, K., Ochiai, T., Nakatani, T., Burget, L., and Černocký, J.  Speakerbeam: Speaker aware neural network for target speaker extraction in speech mixtures.  _IEEE Journal of Selected Topics in Signal Processing_ , 13(4):800–814, 2019.  doi: 10.1109/JSTSP.2019.2922820.

##  Appendix A How can a text prompt filter a sound mixture?

Figure 5: The attention scores of semantic filter $z$ when reading text
prompts and the resulting editing mask $h_{m}$ plotted in Mel scale. The
attention scores are calculated at the last EOS token, ignoring token SOS and
EOS itself, and darker colors indicate larger scores. The ground truth mask is
plotted for comparison.

We try to find the connection between two filters of different modalities: the
semantic (text) filter $z$ calculated by the PromptReader when reading the
text prompt and the editing mask (sound filter) calculated by the SoundEditor
from the semantic filter. The attention scores of the last layer of
PromptReader LLaMA2 (frozen or LoRA finetuned) when reading text prompts, and
the editing mask of the SoundEditor Sepformer compared to the ground truth one
are plotted in Figure 5. Readers can refer to Figure 2 for the edited
spectrograms produced by these filters.

Visualization of the attention scores reveals that acoustic keywords are
highlighted in each editing instruction. These keywords include “background
sounds”, “people talking”, “helicopter”, “man”, “turkey”, and “elev(ated)
tone”. We also observe that the LoRA-finetuned PromptReader tends to pay more
attention to these keywords than a non-finetuned (frozen) one. Greater
attention suggests that the semantic filter (text embedding) generated by a
finetuned PromptReader is more effective in capturing the critical information
what to edit in the text prompts. This, in turn, aids the downstream
SoundEditor in making accurate edits. This finding provides an insight into
our experimental result that a finetuned PromptReader outperforms a frozen one
by more than 4.5 dB as shown in Table 2.

The connection between the semantic filter and the sound editing mask becomes
more apparent once we match sub-sentences from the text prompt to spectral-
temporal regions in the editing mask. We can easily discern patterns such as
“extract” (red color), “remove” (purple color), “increase the volume”
(brighter color), and “decrease the volume” (darker color) in the editing
masks plotted as Mel spectrograms, if the text prompts explicitly state these
editing actions. Interestingly, the editing mask is linear in each editing
action specified in the text prompt. An additional editing action in the text
prompt adds a new pattern to the editing mask. An opposite editing action
causes an opposite pattern in the editing mask. For example, the text prompt
“Why not get rid of the man’s voice and the turkey’s noise, and reduce the
helicopter’s volume?” specifies three editing actions: remove the man’s voice,
remove the turkey’s voice, and turn down the volume of the helicopter. The
resulting editing mask also displays three distinctive regions corresponding
to each one of the editing actions. Similarly, opposite instructions like
“pull out … the turkey” and “get rid of … the turkey” lead to opposite
patterns (red versus purple) for filtering turkey oppositely in the editing
mask.

##  Appendix B Text-Prompted Sound Mixture Editing Dataset

Due to the absence of text-prompted sound mixture editing datasets, we have
curated a 160-hour dataset of more than 100k sound mixtures and more than 500k
text prompts. We will make the mixing scripts and the text prompts publicly
available to promote future research. In this section, we used sound mixtures
of 2 Speech + 2 Audio for illustration. Other compositions of sound mixtures
can be generated and prompted in the same way.

###  B.1 Sound Sources

The speech sources were sourced from TextrolSpeech (Ji et al., 2023).
TextrolSpeech consists of 330 hours of speeches gathered from 1327 speakers,
with each speech source being tagged with five style attributes: gender
(female or male), pitch (low, normal, or high), tempo (low, normal, or high),
volume (low, normal, or high), and emotion (angry, contempt, disgusted, fear,
happy, sad, surprised, or neutral). The audio sources were sourced from
VGGSound (Chen et al., 2020) for training and in-domain evaluation and FSD50K
(Fonseca et al., 2021) for out-of-domain evaluation. VGGSound contains audio
clips from YouTube, with each clip being associated with one or more audio
class labels. We filtered out clips with multiple labels, indicating a mixture
of sounds, and those with labels related to human voices, such as people,
children, and police radio chatter. The filtered VGGSound dataset contains 430
hours of audio from 248 distinct classes. FSD50K is collected from the
Freesound database, and we only used the Eval set for testing. We filtered out
clips with labels under Human_voice or Human_group_actions, resulting in 22
hours of audio data from 171 different classes. Furthermore, we filtered out
clips with labels that overlapped (considering synonyms) with those in
VGGSound, resulting in 8 hours of audio data of 67 different classes. We also
shorten labels that contain parenthesis or are too long, e.g. vehicle horn and
car horn and honking to vehicle horn and keyboard (musical) to musical
keyboard.

We partitioned 1327 speakers from TextrolSpeech into three subsets: 1177 for
training, 50 for validation, and 100 for testing. We applied the same ratio to
partition the VGGSound into three sets. All remaining sources in FSD50K Eval
are reserved for zero-shot evaluation. Next, we resampled both speech and
audio sources to a 16 kHz sampling rate and adjusted their duration to be 5
seconds through cropping or padding as necessary.

To facilitate later discussion, we define a style vector
$\vec{s}=[\textit{gender},\textit{pitch},\textit{tempo},\textit{volume},\textit{emotion}]$,
which is the ordered sequence of values corresponding to the style attributes
of a speech source. Additionally, we use the symbol $c$ to denote the class
label associated to an audio source.

###  B.2 Mixture Generation

We randomly sampled two speech sources of different style vectors (i.e.
$\vec{s}_{1}\neq\vec{s}_{2}$, meaning at least one of the five style
attributes is different) and two audio sources of different class labels (i.e.
$c_{3}\neq c_{4}$). This makes sure that every source can be uniquely
identified by speaking style or class label in words.

We mixed audio sources with signal-to-noise ratios (SNR) that were uniformly
sampled from [-3, 3] dB. For speech sources, we used SNRs that were uniformly
sampled from three different ranges: [-3, -2] dB, [-1, 1] dB, and [2, 3] dB,
depending on volume values: low, normal, and high, respectively. In total, we
created 100,000 mixtures (equivalent to 139 hours), 5000 mixtures (7 hours),
and 10,000 mixtures (14 hours) for the training, validation, and testing sets,
respectively.

###  B.3 Instruction Generation

We assigned an editing instruction to every mixture in the dataset. Instead of
distributing instructions uniformly across $254=|\mathcal{A}|^{4}-2$ possible
edits, we opted to assign the instruction to every mixture uniformly across
the 16 tasks. Our intention is to ensure that all 16 tasks carry equal
importance and have an equal share in the dataset. Randomly selecting from 254
possible edits leads to training imbalances and evaluation unfairness, because
some tasks, such as SE and SR have significantly fewer edits associated
compared to other tasks like MVC and MEVC (as shown Appendix C).

For every mixture $x$ in the dataset, we first selected a task
$t\in\mathcal{T}$ from the task space. Subsequently, we uniformly sampled an
edit from all possible edits with the task $t$, resulting in
$[a_{1},a_{2},a_{3},a_{4}]\in\mathcal{A}^{4}(t)$. Finally, we translated the
actions into scaling factors
$[\alpha_{1},\alpha_{2},\alpha_{3},\alpha_{4}]\in\mathcal{A}_{\alpha}^{4}$ and
generated a target mixture $y$ according to Equation 2.

An editing instruction can be expressed as
$\pi=\\{(a_{1},\vec{s}_{1}),(a_{2},\vec{s}_{2}),(a_{3},c_{3}),(a_{4},c_{4})\\}$.
However, when people writing text prompts, we tend to not include every detail
in $\pi$. For instance, we might not mention sources we prefer not to change,
and might not describe a speaker using all five style attributes. To align
with human’s behavior, we built $\pi^{\prime}$ which contains a subset of
information from $\pi$ in two steps. First, we skipped sources marked as
$a_{i}=\textit{keeping}$ in $\pi$, except for extraction and removal tasks,
where we alternated to say extracting sources (dropping all removing) and
removing other sources (dropping all keeping) with equal probabilities.
Second, we simplified the speaker description by only using a subset of style
attributes that can distinguish one speaker from the other. For example, if
two speakers have different gender, pitch, and tempo, we might only describe
gender and tempo and omit the other three attributes for both speakers.

###  B.4 Prompt Generation

We generated two types of text prompts from $\pi^{\prime}$, the template
prompt and the generic prompt. The template prompt is generated by plugging in
actions, styles, and labels into a prompt template. We wrote three templates
as shown below555The presented templates are for four sources, i.e.
$|\pi^{\prime}|=4$. We used similar templates for other numbers of sources. :
"Please <wrd($a_{2}$)> <str($\vec{s}_{2}$)>, <wrd($a_{3}$)> <str($c_{3}$)>,
<wrd($a_{1}$)> <str($\vec{s}_{1}$)>, and <wrd($a_{4}$)> <str($c_{4}$)>."

"I want to <wrd($a_{4}$)> <str($c_{4}$)>, <wrd($a_{2}$)> <str($\vec{s}_{2}$)>,
<wrd($a_{1}$)> <str($\vec{s}_{1}$)>, and <wrd($a_{3}$)> <str($c_{3}$)>."

"Can you <wrd($a_{3}$)> <str($c_{3}$)>, <wrd($a_{4}$)> <str($c_{4}$)>,
<wrd($a_{1}$)> <str($\vec{s}_{1}$)>, and <wrd($a_{2}$)> <str($\vec{s}_{2}$)>?"
wrb samples a verb or phrase that signifies an action, such as “remove”,
“eliminate”, or “take away” for action removing. The str function is used to
transform a style vector or class label into a readable string like “the happy
male speaker characterized by normal pitch, high tempo, and high energy” and
“the playing cello sound”. To introduce more variability, we shuffled the
elements in $\pi^{\prime}$ before putting them into the template, ensuring
that the speech and audio sources do not always appear in fixed positions in
the prompt.

The generic prompt is generated by instructing a LLM to rephrase the template
prompt. For each mixture, we requested GPT-3.5 Turbo to generate five
different rephrased versions of the template prompt. We encouraged it to try
various sentence structures, including both imperative and interrogative
forms, and to use different word choices, such as synonyms for class labels
and style-related keywords like “female”, “male”, “pitch”, “volume”, and so
on. After generating these prompts, we manually removed and regenerated
prompts that were incomplete or irrelevant. We recognized and accepted a minor
level of hallucination, such as the inclusion of additional sources, in the
LLM’s rephrased results. This allowance is in line with the understanding that
humans, too, occasionally make errors when formulating prompts.

Below is our conversation with GPT-3.5 Turbo, whose role is configured as “You
are a user of a sound editing system”. Content in <> varies for each mixture.

Prompt: <template prompt>

This prompt is an instruction to a sound editing system. As a human user,
rephrase the prompt 5 times, as natural as possible. Be creative in words and
sentence structures. Use either imperative or interrogative instructions. Use
synonyms to words: <speaking styles and class labels>101010Words in {“male”,
“female”, “pitch”, “tempo”, “energy”, “volume”, str($c_{3}$), str($c_{4}$)}
that appear in <template prompt>.. Double-check that all <number of
sources>111111This is the number of sources appeared in <template prompt>, not
the number of sources in the sound mixture. sources are included, and the
instruction is correct, in each one of 5 responses. Keep your language simple
for daily use.

The average, the standard deviation, the lowest, and the highest word counts
of all the generated prompts are 20.7, 9.5, 4, and 74 words, respectively. We
used all five generic prompts in the training set for training, and used the
last one in the validation and testing set for evaluation to make the result
consistent.

Additionally, we wrote five special generic prompts for each one of the
special cases in which the same editing actions are applied to all speech
sources and the same actions are applied to all audio sources. In such cases,
it is unnecessary to identify every source by their unique styles or labels.
Instead, we can use more general terms like “speech”, “conversation”,
“background”, “noise”, or “everything”. For example, here are the five special
generic prompts we wrote for the Speech Enhancement (SE) task:  "Extract all
speakers."

"Enhance the audio by removing background noises."

"Extract all human voices from the recording."

"Eliminate any non-speech sounds in the surroundings."

"Make the conversation as clean as possible." We used special generic prompts
(when applicable) for training and evaluation with a probability of $50\%$.

##  Appendix C Editing Task Specification

We elaborate on the task space $\mathcal{T}$ discussed in Section 3.2 and
Table 1. Consider a sound mixture of two speech and two audio sources (the
composition of the sound mixtures we use for training), there are a total of
$254=|\mathcal{A}|^{4}-2$ possible edits, not counting the identity
($[1,1,1,1]$) and the silence ($[0,0,0,0]$) case. These edits can be
categorized into 16 editing tasks. Below, we provide descriptions, examples,
and the count of unique edits belong to each task:

  * •

Target Speech Extraction (TSE)
Extract a target speech source.
Examples: $[1,0,0,0],[0,1,0,0]$.
Count: 2

  * •

Target Speech Removal (TSR)
Remove a target speech source.
Examples: $[0,1,1,1],[1,0,1,1]$.
Count: 2

  * •

Target Speech Volume Up (TS$\uparrow$)
Up the volume of a target speech source.
Examples: $[\uparrow,1,1,1],[1,\uparrow,1,1]$.
Count: 2

  * •

Target Speech Volume Down (TS$\downarrow$)
Down the volume of a target speech source.
Examples: $[\downarrow,1,1,1],[1,\downarrow,1,1]$.
Count: 2

  * •

Speech Enhancement (SE)
Extract all speech sources. (Remove all audio sources).
Example: $[1,1,0,0]$.
Count: 1

  * •

Speech Removal (SR)
Remove all speech sources. (Extract all audio sources).
Example: $[0,0,1,1]$.
Count: 1

  * •

Speech Volume Up (S$\uparrow$)
Up the volume of all speech sources, down the volume of all audio sources, or
both.
Examples:
$[\uparrow,\uparrow,1,1],[1,1,\downarrow,\downarrow],[\uparrow,\uparrow,\downarrow,\downarrow]$.
Count: 3

  * •

Speech Volume Down (S$\downarrow$)
Up the volume of all audio sources, down the volume of all speech sources, or
both.
Examples:
$[1,1,\uparrow,\uparrow],[\downarrow,\downarrow,1,1],[\downarrow,\downarrow,\uparrow,\uparrow]$.
Count: 3

  * •

Target Audio Extraction (TAE)
Extract a target audio source.
Examples: $[0,0,1,0],[0,0,0,1]$.
Count: 2

  * •

Target Audio Removal (TAR)
Remove a target audio source.
Examples: $[1,1,0,1],[1,1,1,0]$.
Count: 2

  * •

Target Audio Volume Up (TA$\uparrow$)
Up the volume of a target audio source.
Examples: $[1,1,\uparrow,1],[1,1,1,\uparrow]$.
Count: 2

  * •

Target Audio Volume Down (TA$\downarrow$)
Down the volume of a target audio source.
Examples: $[1,1,\downarrow,1],[1,1,1,\downarrow]$.
Count: 2

  * •

Multiple Sound Extraction or Removal (ME)
Extract or remove multiple speech and audio sources.
Examples: $[1,0,1,0],[0,1,1,0]$.
Count: 4

  * •

Multiple Sound Volume Control (MVC)
Up or down the volume of multiple speech and audio sources.
Examples:
$[\uparrow,1,\uparrow,1],[\downarrow,\downarrow,\downarrow,\uparrow]$.
Count: 64

  * •

Multiple Sound Extraction or Removal and Volume Control (MEVC)
Extract or remove and turn up or down the volume of multiple speech and audio
sources.
Examples: $[\uparrow,0,\downarrow,1],[1,\downarrow,\downarrow,0]$.
Count: 160

  * •

Overall Volume Control (OVC)
Up or down the overall volume of the mixture.
Examples:
$[\uparrow,\uparrow,\uparrow,\uparrow],[\downarrow,\downarrow,\downarrow,\downarrow]$.
Count: 2

The existence and the count of edits for each task depend on the number of the
speech and the audio sources in the sound mixture. However, these 16 tasks
cover all editing cases for any mixture containing at least two speech and at
least two audio sources.

Figure 6: Example A: sound mixture editing on 2 Speech + 2 Audio (VGGSound)
mixture. Figure 7: Example B: sound mixture editing on 2 Speech + 2 Audio
(VGGSound) mixture. Figure 8: Example C: sound mixture editing on 2 Speech + 2
Audio (VGGSound) mixture. Figure 9: Example D: sound mixture editing on 2
Speech + 2 Audio (VGGSound) mixture.

##  Appendix D Style-wise Performance Analysis

Table 6: Zero-shot speech editing SNRi (dB) for two speakers with one or
multiple style variation.

SoundEditor Sepformer Conv-TasNet Differ in Task TSE TS$\uparrow$
TS$\downarrow$ MVC Average TSE TS$\uparrow$ TS$\downarrow$ MVC Average Gender
11.4 12.5 13.0 10.3 11.8 6.1 6.6 7.0 5.1 6.2 Pitch 11.3 11.7 11.6 8.7 10.8 4.8
5.1 5.6 2.9 4.6 Tempo 6.7 5.4 6.6 3.3 5.5 3.4 2.9 3.6 1.2 2.8 Energy 10.9 11.5
11.8 10.7 11.2 4.7 5.3 5.2 3.5 4.7 Emotion 11.4 11.9 11.6 9.6 11.1 5.7 6.3 5.7
4.1 5.5 Multiple 13.2 13.2 13.0 10.6 12.5 7.5 7.5 7.2 5.6 7.0

We compared the zero-shot two-speaker editing performance between SoundEditor
Sepformer and Conv-TasNet in Table 6, as a compliment to Section 6.4 and Table
4. Sepformer wins Conv-TasNet in all editing tasks under all speaking styles.
Conv-TasNet is particularly bad at speech editing based on tempo because of
the limited receptive field of convolution.

##  Appendix E Ablations

The ablations we conducted are summarized in Table 7. We studied the impact of
the text prompts, the number of the editing mask(s) estimated, and other model
configurations. We performed all ablations with Conv-TasNet & LLaMA2 for
computational efficiency, but we expect that similar observations can be drawn
with Sepformer & LLaMA2.

Table 7: Ablations on the text prompt quality, the training method, and the model configuration with Conv-TasNet & LLaMA2 Setting | TSE | TSR |  TS$\uparrow$ |  TS$\downarrow$ | TAE | TAR |  TA$\uparrow$ |  TA$\downarrow$ | SE
---|---|---|---|---|---|---|---|---|---
Template 100% | 7.7 | 3.4 | 4.2 | 2.2 | 6.4 | 2.1 | 2.5 | 1.6 | 7.5
Template 100% + | 7.7 | 4.0 | 4.5 | 3.0 | 6.7 | 2.3 | 2.9 | 2.3 | 7.7
Template 50% + | 9.7 | 4.6 | 4.8 | 4.9 | 8.2 | 2.5 | 3.1 | 3.0 | 8.3
MultiMask | 9.6 | 4.4 | 4.6 | 4.7 | 8.3 | 2.5 | 3.1 | 3.0 | 8.2
MultiMask + PIT | 9.7 | 4.6 | 4.8 | 4.9 | 8.2 | 2.5 | 3.1 | 3.0 | 8.3
Kernel Size $4\times$ | 9.5 | 4.3 | 4.7 | 4.7 | 8.3 | 2.7 | 3.3 | 3.3 | 8.1
Causal | 8.5 | 3.3 | 3.3 | 3.4 | 7.7 | 1.9 | 2.5 | 2.5 | 6.9
Default | 9.7 | 4.6 | 4.7 | 4.8 | 8.2 | 2.5 | 3.1 | 3.0 | 8.2
Continued
SETTING | SR |  S$\uparrow$ |  S$\downarrow$ | ME | MVC | MEVC | OVC | Average
Template 100% | 7.7 | 6.9 | 6.4 | 3.6 | 3.2 | 4.0 | 32.6 | 6.4
Template 100% + | 8.1 | 7.2 | 6.7 | 3.8 | 3.3 | 4.3 | 35.0 | 6.9
Template 50% + | 8.6 | 7.6 | 7.2 | 4.3 | 4.0 | 4.7 | 39.2 | 7.8
MultiMask | 8.5 | 7.4 | 7.0 | 4.3 | 3.8 | 4.6 | 42.5 | 7.9
MultiMask + PIT | 8.6 | 7.2 | 6.9 | 4.3 | 4.0 | 4.7 | 42.6 | 8.0
Kernel Size $4\times$ | 8.5 | 7.5 | 7.1 | 4.3 | 3.9 | 4.5 | 36.1 | 7.6
Causal | 7.2 | 6.0 | 5.7 | 3.7 | 3.2 | 3.8 | 37.2 | 6.7
Default | 8.5 | 7.4 | 7.0 | 4.2 | 3.9 | 4.6 | 40.3 | 7.8

###  E.1 Text Prompt Quality

By default, we train the LCE with open-vocabulary generic prompts. Here, we
instead use “closed-vocabulary” template prompts for training but still used
generic prompts for testing. The differences between generic and template
prompts are elaborated in Appendix B.4. In brief, template prompts are
generated by a program that fills in words within one of three predefined
templates, whereas generic prompts are rephrased from template prompts by
GPT-3.5 Turbo and have greater variability in both word choices and sentence
structures, thus closer to prompts written by humans. The advantage of using
template prompts lies in their ease of generation from a computational and
time-efficiency perspective. Compared to generic prompts which were written
offline for only one selected edit per sound mixture121212Due to resource
constraints, we only generated five text prompts corresponding to one of 254
possible edits per sound mixture., template prompts can be written on the fly
for all 254 possible edits for each sound mixture with minimal computational
overhead.

In Table 7, the first row “TEMPLATE 100%” used 100% of template prompts for
training. The second row “TEMPLATE 100% +” also used 100% of template prompts
for training, but it generated additional training samples: For every mixture,
we randomly sampled a new editing instruction, which might differ from the one
assigned and written the generic prompts, and generated a new template prompt
and target sound mixture on the fly. The third row “TEMPLATE 50% +” used 50%
of template prompts and 50% of generic prompts for training. For the generic
half, we used one of the five generic prompts written offline for the assigned
edit. For the template half, we did the same as the second row to randomly
sample a new edit, resulting in a new template prompt and a new target
mixture.

We observe that the LCE trained on template prompts of lower quality exhibits
an average SNRi 1.4 dB lower compared to the LCE trained on generic prompts.
However, when we introduce the generation of new editing instructions and new
target mixtures, the average SNRi increases by 0.5 dB. The best performance is
achieved when using a combination of 50% generic prompts and 50% template
prompts with the generation of new editing instructions and new target
mixtures, on par with the performance of using 100% generic prompts. We have
three insights from these results: First, including generic or real prompts in
training is crucial for good performance on open-vocabulary instructions.
Second, it is not necessary to train exclusively with 100% generic or real
prompts to achieve comparable results. Third, the performance of LCE can be
further improved by generating new editing instructions and target mixtures.

###  E.2 Multiple Masks

Recall that SoundEditor estimates a single editing mask
$h_{m}\in\mathbb{R}^{C\times L}$ for sound mixtures containing any number of
sources, as shown in Equation 7. In this section, we explore estimating $N$
masks for $N$ sources in the mixture, i.e. $h_{m}\in\mathbb{R}^{N\times
C\times L}$. This is equivalent to first separate all the sources in the
mixture, individually edit each of them with one of the mask, and sum the
edited sources up. We show two variants with multiple editing masks: In one
experiment, we only changed the output dimension of the output layer of the
mask network $\mathcal{M}$ to be $N$ channels to output $N$ masks. In the
other experiment, we further introduced a permutation invariant SNR (Yu et
al., 2017) (PIT-SNR) loss between the estimated edited sources
$\\{\hat{s_{i}}\\}$ and the ground truth edited sources $\\{\bar{s_{i}}\\}$.
The total loss in this case is shown in Equation 11 and $\hat{s_{i}}\sim\Pi$
denotes selecting the optimal permutation of the sources that maximizes the
sum of SNR for all sources:

$\displaystyle\mathcal{L}\bigl{(}\\{\hat{s_{i}}\\},\\{\bar{s_{i}}\\},\hat{y},y\bigr{)}=-\bigl{(}\frac{1}{N}\sum_{\hat{s_{i}}\sim\Pi}\text{SNR}(\hat{s_{i}},\bar{s_{i}})\bigr{)}-\text{SNR}(\hat{y},y)$
(11)

The performance with multiple editing masks, both without and with the PIT-SNR
loss on sources, is shown in the fourth and fifth rows of Table 7. Switching
from one mask to multiple ($N$) masks has a negligible impact on performance.
However, incorporating an additional PIT-SNR loss leads to an improvement of
0.2 dB in average performance. While this represents a slight improvement over
the default system with one mask, it necessitates access to each of the ground
truth edited sources ($\\{\bar{s_{i}}\\}$), in addition to the target mixture
$y$, and presumes knowledge of the number of sources ($N$) in the mixture.
Therefore, this approach is only included in this discussion for comparison
purposes.

###  E.3 Model Configurations

We tried two other configurations of SoundEditor Conv-TasNet. In one
experiment, the kernel size and the stride of both the encoder and decoder are
increased to 64 samples (4 milliseconds) and 32 samples (2 milliseconds),
respectively, which is four times the size used in the standard configuration.
This adjustment results in a receptive field that is four times larger but
with features in the latent space that are four times smaller. Despite this
configuration being less effective than the default in terms of speech
separation (Luo & Mesgarani, 2019), we want to explore if a longer context of
the sound input can compensate for the shortcoming of the reduced feature size
in better comprehending and executing editing instructions. The performance
with this four times larger kernel and stride size is only marginally lower by
an average of 0.2 dB, yet the inference speed of Conv-TasNet is approximately
four times faster. In another experiment, we develop a causal system targeting
real-time hearing assistance. The performance of the causal system is 1.1 dB
lower than the performance of the non-causal system. The inference speed is
around 90 times faster than real-time on a NVIDIA L40 GPU. The average memory
usage is about 14 GB, with the majority of the memory attributed to
instantiating and running PromptReader LLAMA2. Further research is needed to
reduce the memory usage for a commodity GPU or mobile devices.

##  Appendix F Models and Training

We jointly optimized either Conv-TasNet or Sepformer, implemented by
SpeechBrain (Ravanelli et al., 2021) with the default model architecture (in
particular a kernel size of 16 samples or one millisecond for the encoder and
decoder) in papers (Luo & Mesgarani, 2019; Subakan et al., 2021), as the
SoundEditor, and either GPT-2 (base) (Radford et al., 2019) or LLaMA2
(7b-chat-hf) (Touvron et al., 2023) as the PromptReader. The SoundEditor is
trained from scratch and the PromptReader is finetuned from pretrained
checkpoints131313LLaMA2: https://huggingface.co/meta-llama/Llama-2-7b-chat-hf.
GPT-2: https://huggingface.co/gpt2. We trained LCE for 100 epochs using 4
NVIDIA L40 GPUs with brain 16-bit floating point precision. The total batch
size was set to 16 for Conv-TasNet and 8 for Sepformer. We used an Adam
(Kingma & Ba, 2014) optimizer with a learning rate of $5e-4$ for Conv-TasNet
and $1e-4$ for Sepformer. GPT-2 and LLaMA2 were finetuned with a learning rate
of $1e-4$. GPT-2 was finetuned with all parameters, and LLaMA2 was finetuned
using LoRA with a rank of 16 and a dropout rate of 0.05 for the query and
value matrices in each self-attention layer. In addition, we applied a linear
learning rate warm-up for the first 5000 updates, and in case the validation
Signal-to-Noise Ratio (SNR) did not improve for 3 epochs, we halved the
learning rate for both the SoundEditor and PromptReader.

The training time is around 3 days for Conv-TasNet & GPT-2 system, 4 days for
Conv-TasNet & LLaMA2 system, 7 days for Sepformer & GPT-2 system, and 9 days
for Sepformer & LLaMA2 system.

##  Appendix G Examples

Examples of sound mixture editing are presented from Figure 6 to 13. These
examples span both in-domain and zero-shot editing applied to sound mixtures
comprising various numbers and categories of sound sources.

Figure 10: Example E: zero-shot sound mixture editing on 2 Speech + 2 Audio
(FSD50K, seen labels) mixture. Figure 11: Example F: zero-shot sound mixture
editing on 2 Speech + 2 Audio (FSD50K, seen labels) mixture. Figure 12:
Example G: zero-shot sound mixture editing on 2 Speech + 2 Audio (FSD50K,
unseen labels) mixture. Figure 13: Example H: zero-shot sound mixture editing
on 2 Speech + 2 Audio (FSD50K, unseen labels) mixture. Figure 14: Example I:
zero-shot sound mixture editing on 2 Speech mixture. Figure 15: Example J:
zero-shot sound mixture editing on 2 Speech mixture. Figure 16: Example K:
zero-shot sound mixture editing on 2 Audio mixture. Figure 17: Example L:
zero-shot sound mixture editing on 2 Audio mixture. Figure 18: Example M:
zero-shot sound mixture editing on 2 Speech + 1 Audio mixture. Figure 19:
Example N: zero-shot sound mixture editing on 2 Speech + 1 Audio mixture.
Figure 20: Example O: zero-shot sound mixture editing on 1 Speech + 2 Audio
mixture. Figure 21: Example P: zero-shot sound mixture editing on 1 Speech + 2
Audio mixture.