# From Words to Music: A Study of Subword Tokenization Techniques in Symbolic
Music Generation

Adarsh Kumar1 and Pedro Sarmento2 1Indian Institute of Technology Kharagpur,
India
2 Queen Mary University of London, UK
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Subword tokenization has been widely successful in text-based natural language
processing (NLP) tasks with Transformer-based models. As Transformer models
become increasingly popular in symbolic music-related studies, it is
imperative to investigate the efficacy of subword tokenization in the symbolic
music domain. In this paper, we explore subword tokenization techniques, such
as byte-pair encoding (BPE), in symbolic music generation and its impact on
the overall structure of generated songs. Our experiments are based on three
types of MIDI datasets: single track-melody only, multi-track with a single
instrument, and multi-track and multi-instrument. We apply subword
tokenization on post-musical tokenization schemes and find that it enables the
generation of longer songs at the same time and improves the overall structure
of the generated music in terms of objective metrics like structure indicator
(SI), Pitch Class Entropy, etc. We also compare two subword tokenization
methods, BPE and Unigram, and observe that both methods lead to consistent
improvements. Our study suggests that subword tokenization is a promising
technique for symbolic music generation and may have broader implications for
music composition, particularly in cases involving complex data such as multi-
track songs.

##  1 Introduction

Subword tokenization is a widely used technique for text representation in
natural language processing (NLP). Such tokenization techniques based on the
creation of subword tokens, like byte pair enconding (BPE) Sennrich et al.
(2016), Unigram Kudo (2018) and WordPiece Wu et al. (2016), have become
ubiquitous in various NLP tasks. Attributed to their efficiency in modeling
longer patterns, rather than simply characters, these subword tokenization
techniques became extremely successful with Transformer models like BERT
Devlin et al. (2018) and GPT Radford et al. (2019), achieving state-of-the-
results in multiple text-based NLP applications. Works like Park et al. (2020)
and Gallé (2019) have further shown the universality of its application across
languages, not just in English.

Figure 1: Example of similar frequently co-occurring structures within the
text (at character level) and musical representations (MIDI-like, at event
level).

Inspired by the success of Transformer models in text-NLP,111For the sake of
clarity and distinction, we will henceforth refer to tasks related to the text
in NLP as text-NLP. recent years have witnessed a shift in research work
towards leveraging Transformers Vaswani et al. (2017) in the domain of
symbolic music generation Huang et al. (2018); Muhamed et al. (2021); Guo et
al. (2022). This shift can be ascribed to the resemblance of symbolic music
post-musical-tokenization222Musical Tokenization refers to tokenization of
symbolic formats like MIDI or GuitarPro with Music Tokenization such as REMI
or MIDI-like. to that of text tokens. With Transformers’ extraordinary
capability to model longer sequences, we are able to generate coherent,
adequate pieces of music end-to-end Hsiao et al. (2021); Huang and Yang
(2020); Neves et al. (2022).

Despite all the success of the predecessors in improving the state of music
generation, these models are often accused of failing to entirely capture the
repetitive structure and overall musical development of songs Dai et al.
(2022); Wu and Yang (2020, 2022). This becomes more apparent as the structure
of the music becomes complex such as in the case of polyphonic music or multi-
track music. A reasonable explanation for this could be the significantly
longer sequence of symbolic music tokens, which limits the segment of a song
visible to the Transformer model, hindering its understanding of the overall
musical structure. As an analogy, this would be equivalent to representing a
piece of text as a sequence of individual characters.

One possible solution to this problem is at the token level. The idea is to
group individual events into subgroups, similar to subwords in text-NLP. There
have been works like Hsiao et al. (2021) and Liu et al. (2022), which tried to
group musical events, exploiting the properties of musical structure for
certain MIDI-based datasets. However, most of these works are dependent on the
musical structure of certain datasets involved, and hence cannot be
extrapolated to other formats easily. So, to the best of our knowledge, no
work has been done so far to assess the use of subword tokenization techniques
like BPE and Unigram that utilize the co-occurrence-based structure of musical
events, independent of the musical structure of the training dataset itself.

We are therefore motivated to study whether the use of subword tokenization
can improve the overall musical structure of generated songs, while at the
same time being independent of the dataset or format of symbolic music
involved. In this work, we specifically investigate the usefulness of subword
tokenization techniques like BPE Sennrich et al. (2016) and Unigram Kudo
(2018) in modeling the task of symbolic music generation. Through our
experiments, we try to answer the following two primary questions:

  * •

Q1: Can we use subword tokenization techniques to improve the overall musical
structure and musical quality of the generated examples?

  * •

Q2: How do these findings generalize between two different subword
tokenization techniques, namely BPE and Unigram?

In an effort to answer the above questions, our main contributions through
this paper are as follows:

  * •

Creation and implementation of an evaluation environment to objectively assess
the usefulness of subword tokenization techniques, namely BPE and Unigram, to
improve overall musical structure (in terms of quantitative metrics like Pitch
Class Entropy, Structureness Indicators etc.), independent of data-specific
factors like music file formats (e.g. MIDI or GuitarPro);

  * •

Establishing the efficiency of subword tokenization across datasets, data
formats and musical tokenization techniques, with our study involving melody-
only, polyphonic and multi-track datasets;

  * •

Demonstrating the usefulness of subword tokens towards facilitating longer
pieces of generated music within the same inference time.

##  2 Background and Related Work

###  2.1 Subword Tokenization

Tokenization has become a fundamental process in NLP which involves breaking
the macroscopic units of text such as ‘words’ or ‘sentences’ into smaller
units called tokens. Since representing the much longer textual data as
individual characters is highly ineffective, the concept of ‘subword
tokenization’ was introduced Mielke et al. (2021). It involves breaking the
word or sentence into sub-words (a subsequence of characters with length $\geq
1$), which are then used to represent the data. Over the years, several
subword tokenization techniques have been introduced such as BPE Gage (1994);
Sennrich et al. (2016), Unigram Kudo (2018), WordPiece Wu et al. (2016) and
SentencePiece Kudo and Richardson (2018). Most of these techniques involve the
selection of subwords based on their frequency within the training data.

Byte-Pair-Encoding (BPE) Sennrich et al. (2016) is a data compression
technique Gage (1994) which involves the replacement of the most frequent pair
of bytes by a single, unused byte. In text-NLP, BPE is applied to subword
units, rather than bytes, by finding the most frequent character n-grams in a
text corpus and merging them into a single token. This allows the model to
learn a more fine-grained representation of the language, especially for rare
or out-of-vocabulary words. The result is a variable-length subword
vocabulary, which balances the ability to capture complex language structure
with reducing the risk of overfitting.

In contrast to BPE, Unigram Kudo (2018) is another subword tokenization
technique, which starts from a large vocabulary and gradually trims the
vocabulary towards a smaller one. It involves a probabilistic Unigram language
model, which decides whether to keep a subword or not based on its likelihood
and loss function. Both these models have been extensively used in various NLP
applications such as Wang et al. (2020), Liu et al. (2019) and Conneau and
Lample (2019). Furthermore, papers like Gallé (2019) and Park et al. (2020)
have shown that these subword tokenization techniques can be applied
effectively across languages. This motivates us to explore whether these
results can be extended to symbolic music as a language, and what impact it
could have on the overall musical structure of generated songs.

Figure 2: Symbolic music processing pipeline with subword tokenization. Please
note that, here, musical symbols refer to mapping of musical events to unicode
symbols or characters, which are then processed with subword tokenizer.

###  2.2 Symbolic Music Generation

Symbolic Music Generation involves representation of music data from formats
like MIDI or GuitarPro with symbols or sequences of events which are then used
to train generative models. This has been an extensively researched area,
where researchers are trying to come up with algorithms and models that can
generate music at par with human performance Hiller (2019). Recently, the
domain of symbolic music generation has witnessed steady improvements, mostly
driven by advances in deep learning architectures. Overall, approaches towards
symbolic music generation with deep learning can be aggregated according to
the architecture used, namely Variational Autoencoder (VAEs) models Tan and
Herremans (2020), Generative Adversarial Networks (GANs) Dong and Yang (2018),
and models that stem from natural language processing (NLP) field, such as
Recurrent Neural Networks (RNNs) Meade et al. (2019), Long Short-Term Memory
(LSTMs) Sturm et al. (2016), or Transformers Vaswani et al. (2017). As stated
before, the Transformer architecture Vaswani et al. (2017) is suitable for
generating longer sequences when compared to previous approaches used by RNNs.
Transformer-like sequence-to-sequence models are able to learn the
dependencies and patterns among elements of a given sequence. The work by
Huang et al. (2019), the Music Transformer, pioneered the application of the
self-attention mechanism to generate longer sequences of symbolic piano music.
Other examples include works like Musenet Payne (2019), in which a large-scale
Transformer model, GPT-2, was used to generate symbolic multi-instrument music
from different musical genres, the Pop Music Transformer Huang and Yang
(2020), which uses Transformer-XL Dai et al. (2019a) as a backbone
architecture and is able to generate pop piano symbolic music with a better
rhythmic structure, the Compound Word Transformer Hsiao et al. (2021), that
presents novel and more efficient ways of tokenizing symbolic music for
training purposes, and GTR-CTRL Sarmento et al. (2023) explores genre and
instrument conditioning using special control tokens with multi-track music
dataset DadaGP, thereby controlling overall musical structure of generated
songs.

##  3 Methodology

In order to evaluate the usefulness of subword tokenization and to answer the
questions mentioned in Section 1, we designed an empirical study where we
objectively assess the performance of models with and without subword
tokenization methods, while keeping all the other factors identical. For
clarification, we here refer to the model without subword tokens as the ‘base
model’, while the other models are named according to the respective subword
tokenization technique used while modeling (e.g. ‘BPE model’ as the model that
used BPE for subword tokenization).

###  3.1 Datasets and Music Tokenization Schemes

As mentioned earlier, we used three datasets for our experiments, namely the
Folk Songs dataset Sturm et al. (2016), the MAESTRO datasetHawthorne et al.
(2019), and the DadaGP dataset Sarmento et al. (2021). Furthermore, for the
music tokenization procedure, we utilized REMI Huang and Yang (2020), Midi-
like Oore et al. (2018) and DadaGP tokenization Sarmento et al. (2021)
respectively, primarily to demonstrate the compatibility of subword
tokenization with existent different music tokenization approaches. Given that
training the models such as Music Transformer Huang et al. (2018) and
Transformer-XL Dai et al. (2019b); Huang and Yang (2020) on such large
datasets is very resource intensive, we restricted our study to subsets of
each dataset, using 1000 samples from the Folk Songs dataset, 400 samples from
the MAESTRO dataset and 2000 songs from the Rock genre in DadaGP. However,
since similar settings were kept for both the models with and without subword
tokenization, our restriction does not impact any conclusions from our
results.

###  3.2 Data Processing

While there are several subword tokenization methods available, we focused our
experiments on two of the most common ones, BPE and Unigram, primarily because
of their ease of applicability to any dataset. To leverage subword
tokenization with symbolic music, we first converted songs from symbolic
format (MIDI or GuitarPro) to musical events using music tokenization schemes
such as REMI Huang and Yang (2020), following which we create a mapping, from
musical events to unicode symbols. Thus, we obtained a relatively easy-to-
process corpus of symbols, in which we assume each song is a single entity
similar to SentencePiece Kudo and Richardson (2018). We then trained subword
tokenization methods on the respective datasets to create a larger vocabulary
of subword tokens. Furthermore, we used this vocabulary to process musical
tokens of all songs in a given dataset. A statistical summary of the original
vocabulary size of musical tokens and subword tokens used in these experiments
is given in Table 1.

Music Tokenization | Original | BPE | Unigram
---|---|---|---
REMI | 227 | 300 | 300
MIDI-Like | 331 | 1000 | 1000
DadaGP | 2104 | 5000 | 5000
Table 1: Vocabulary size of original tokens (using respective musical
tokenization), post-processing with BPE and Unigram.

###  3.3 Experimental Configuration

As stated before, in order to assess the independence of results in terms of
improvements while using subword tokenization methods against factors like
type of dataset, music tokenization procedure, and model, we experimented with
three different combinations:

  1. 1.

Folk Songs (monophonic, single instrument, MIDI format) + REMI tokenization +
Music Transformer;

  2. 2.

MAESTRO (polyphonic, single instrument, MIDI format) + MIDI-Like tokenization
+ Music Transformer;

  3. 3.

DadaGP (polyphonic, multi-instrument, GuitarPro format) + DadaGP tokenization
+ Transformer-XL;

The key idea here is to experiment with subword tokenization schemes in
different settings and configurations and to see if the results in terms of
improvement hold true irrespective of the rest of the factors. Furthermore,
the choice of monophonic, polyphonic, and multi-track music, allows us to
evaluate the usefulness of subword tokenization for music generation tasks
with different levels of complexities.

##  4 Evaluation Metrics

To objectively evaluate the results of applying subword tokenization in music
generation, we subdivided dedicated metrics into categories: musical quality
and structure and efficiency in representation.

###  4.1 Musical Quality and Structure

Structureness Indicator (SI): Proposed in Wu and Yang (2020),the structureness
indicator (SI) is designed to capture the structureness of music, induced by
repetitions of musical content. It is based on the fitness-scapeplot algorithm
Müller et al. (2011) and the self-similarity matrix (SSM) Foote (1999)
computed from fitness, where the degree of repetition is derived from SSM for
a given segment ($i$, $j$). Similar to Wu and Yang (2020), in our experiments,
we used $SI_{3}^{8}$, $SI_{8}^{15}$ and $SI_{15}$, in which $SI_{l}^{u}$, $l$
and $u$ represent the lower and upper bound of intervals of consideration (in
seconds). We respectively refer to these as SI-short, SI-medium, and SI-long,
as they are used to examine the short, medium, and long-term structureness of
generated songs. Here it is important to note that a higher SI-x value doesn’t
necessarily mean ‘better’ music, since the generated samples may then be too
repetitive. Instead, a more valid assumption is that it is better the closer
it is to the real music (i.e. testing corpus), which will be the basis of our
evaluation.

Pitch Class Entropy ($\mathcal{H}$): Also described in Wu and Yang (2020),
gives an insight about different pitches, and thereby tonality used in a song.
Here, the main idea is to calculate entropy from a normalized 12-dimensional
pitch class histogram (corresponding to 12 pitch classes C, C#, D, … B) and
analyze how close these values are to the real values.

Groove Pattern Similarity ($\mathcal{GS}$): Another metric defined in Wu and
Yang (2020), Groove Pattern Similarity helps in measuring the rhythmic
consistency within a song. It calculates the pairwise similarity of each bar’s
groove vector g (indicating the position in bar with at least a note onset) as
$1-HammingDist(g_{a},g_{b})$, across all pairs $g_{a}$ and $g_{b}$. Similar to
the other two metrics mentioned previously, in this case also, the closer the
values of groove similarity from the generated songs are to the real songs,
the better.

###  4.2 Efficiency in Representation

Average Number of Tokens per Song: Within this metric we measure the average
number of tokens per song present post-data processing, which are then fed
into a Transformer model for training. A more efficient representation will be
the one, which has smaller average number of tokens per song.

Average Number of Tokens per Song for the Same Inference Time: In this metric,
for a given dataset, we generate an equal number of tokens (i.e. same
inference)for each of the three models we experimented with. After a post-
conversion process back to the original musical tokens, we compare the average
number of tokens generated in terms of the original tokens. This metric helps
us assess how efficient the representation is in terms of generating longer
music within the same inference time. Hence, for this particular metric, the
larger the value, the better representation of data will be.

###  4.3 Other Metrics

NLL Loss: Negative log-likelihood is a common metric, often used to measure
how well a model fits the training dataset Huang et al. (2018); Peracha
(2020); Hsiao et al. (2021). While a relatively closer and smaller value of
NLL represents a well-fitted model, in our case we observe that it is not a
good metric to compare performances across models, since the model with
subword tokenization has a higher number of parameters. Furthermore, a lower
NLL loss doesn’t necessarily mean better generation quality. We have still
added this to give an idea of how well the BPE or the Unigram model fits
relative to the base model.

##  5 Results

###  5.1 Experimental Settings

The experiments were conducted using HuggingFace Wolf et al. (2020) and
PyTorch implementations of Music
Transformer333https://github.com/jason9693/MusicTransformer-pytorch and
Transformer-XL444https://github.com/YatingMusic/compound-word-transformer. For
musical tokenization, we used the MidiTok Fradet et al. (2021) library, which
we further processed for subword tokenization using Huggingface’s
tokenizers555urlhttps://huggingface.co/docs/tokenizers/index. For the first
two parts of our experiment (i.e. Folk Songs and MAESTRO songs dataset
involving the Music Transformer model), we used a 3-layer Transformer
architecture, with an embedding dimension 256, which we trained on Google
Colab Free Tier with a P100 16GB GPU machine. For the last part (i.e. DadaGP
with Transformer-XL), we used the same architecture as in Huang and Yang
(2020), training the model on a 24GB Quadro RTX 6000 GPU. For the evaluation,
we used implementation of the metrics described in the previous section in
MusPy Dong et al. (2020) and MusDr666https://github.com/slSeanWU/MusDr.
Finally, we evaluated the model performance by generating 20 songs for each
model configuration. Samples from the generation can be accessed here.

###  5.2 Objective Evaluation

Results from the three distinct experiments can be seen in tables 2, 3 and 4.

Metrics | Real | Original | BPE | Unigram
---|---|---|---|---
SI-short | 0.4637 | 0.2707 | 0.3376 | 0.2712
SI-medium | 0.4959 | 0.2759 | 0.3379 | 0.2719
SI-long | 0.4543 | 0.2583 | 0.3340 | 0.2451
$\mathcal{H}$ | 2.6011 | 2.6924 | 2.6754 | 2.6842
$\mathcal{GS}$ | 0.9987 | 0.9984 | 0.9986 | 0.9985
Table 2: Results with the Folk Songs dataset. Metrics | Real | Original | BPE | Unigram
---|---|---|---|---
SI-short | 0.3228 | 0.5119 | 0.3880 | 0.3483
SI-medium | 0.3066 | 0.4663 | 0.3334 | 0.2828
SI-long | 0.2343 | 0.4173 | 0.3031 | 0.2205
$\mathcal{H}$ | 3.0555 | 2.5152 | 2.8705 | 2.9297
$\mathcal{GS}$ | 0.9917 | 0.9971 | 0.9942 | 0.9936
Table 3: Results with the MAESTRO songs dataset. Metrics | Real | Base | BPE | Unigram
---|---|---|---|---
SI-short | 0.5069 | 0.5110 | 0.5125 | 0.4894
SI-medium | 0.5270 | 0.4219 | 0.4892 | 0.4539
SI-long | 0.4972 | 0.2943 | 0.4397 | 0.3924
$\mathcal{H}$ | 2.5842 | 2.0529 | 2.3084 | 2.4333
$\mathcal{GS}$ | 0.9991 | 0.9994 | 0.9993 | 0.9992
Table 4: Results with the DadaGP dataset.

As can be observed from the tables, the use of subword tokenization methods
outperforms the base models by a significant margin in both cases (i.e. BPE
and Unigram), in almost all the configurations, irrespective of the model,
dataset, data format, or music tokenization scheme used. The values of SI-
short, SI-medium, and SI-long, which closely resemble real song data, indicate
an overall improvement in the musical structure of songs using subword
tokenization. Additionally, longer repetitive structures exhibit more
significant improvements compared to shorter ones. The results suggest that
subword tokenization techniques have the potential to model the musical
structure better and can leverage the latent co-occurring structure within
musical tokens to improve the quality of generated music.

That being said, the relatively smaller improvements in the case of the Folk
Songs dataset, suggest a correlation between the complexity of the dataset
being modeled and the performance change, with more scope for improvements in
the case of datasets with higher complexities such as MAESTRO and DadaGP.
Adding to this, negligible improvements in terms of SI for Unigram could be
possible because the subword tokens generated with Unigram are harder to learn
by the model, thereby collapsing to simpler tokens (which is a case similar to
base model, which also works on simpler tokens). This again could be possibly
attributed to the relative simplicity in terms of structure and lesser scope
of co-occurrence when the dataset is melody-only. However, when we move to
more complicated datasets, we have musical co-occurring structures like
Chords, which improve the feasibility of using Subword Tokenization
Techniques.

Dataset | Base | BPE | Unigram
---|---|---|---
FOLK | 796 | 359 | 436
MAESTRO | 12925 | 8831 | 8618
DadaGP | 5332 | 2875 | 2954
Table 5: Average number of tokens per song in each representation. Dataset | Base | BPE | Unigram
---|---|---|---
FOLK | 500 | 1307 | 994
MAESTRO | 1000 | 1570 | 1534
DadaGP | 1000 | 1437 | 1828
Table 6: Average number of tokens generated for same inference time in a
dataset (i.e. for same time taken to generate $x$ base model (without subword
tokenization) tokens, corresponding $y$ and $z$ tokens generated (in terms of
original musical tokens) for BPE and Unigram models respectively.)

Furthermore, the figures from Tables 5 and 6 demonstrate the improved
efficiency of representation with the use of subword tokenization. This adds
up to the advantage of using subword tokenization with the base model, as it
allows to model of longer sequences within the same inference time from a
model. These results become particularly important in the case of complex
symbolic music datasets such as DadaGP. This dataset, being complex in
representation, requires that much longer sequences are generated even for a
short song segment. Using subword tokenization with such datasets can allow
shortening the sequences, thereby allowing longer music to be inferred.

A comparison between the performance of models with BPE or Unigram
tokenization throughout the results suggests that the improvements hold true
in general for subword tokenization, leveraging the frequent cooccurrence of
musical events within songs. Though there are some localized differences on
what method is more efficient in terms of structural modeling or musical
quality, with one working better than the other in certain cases or datasets,
in all, these perform better than the base model ‘without’ subword
tokenization. This answers our second question of how the results of our study
generalize for two different subword tokenization methodologies. Furthermore,
this generalization is independent of the constraints of a given musical
structure of a particular dataset.

Another interesting aspect of our results is to observe the improvement of
results with the experiments involving Transformer-XL i.e. beyond fixed length
modeling. While the main purpose of using this model, as proposed in Huang and
Yang (2020) for music generation, is to model musical structure beyond a fixed
length of the input, it is intriguing to see the improvements with subword
tokenization even in this case, particularly with longer repetitive
structureness. This suggests while the dedicated architecture of Transformer-
XL is capable of modeling shorter musical structure better, there is still a
loss of information as the model propagates through the windows of
Transformer-XL input, leading to lesser long-term repetitiveness. However, on
reducing this length with subword tokenization, this loss can be reduced,
thereby allowing improved representation of the musical structure, closer to
the real data, we are trying to model.

Dataset | Model | Train Time | Base | BPE | Unigram
---|---|---|---|---|---
FOLK | MT | $\sim 20mins$ | 0.08 | 0.13 | 0.11
MAESTRO | MT | $\sim 3hrs$ | 2.58 | 3.16 | 3.62
DadaGP | Tr-XL | $\sim 3days$ | 0.10 | 0.11 | 0.11

Table 7: Ngeative Log-Likelihood Loss for the models (MT $\to$ Music
Transformer, Tr-XL $\to$ Transformer XL). As design choices, here we decided
the train time based on the complexity of the dataset and the model in use,
having kept the same time for all three experiments.

Lastly, it is important to note the NLL loss performance of the models we
trained from Table 7. While the closer values of loss functions in the case of
the base, BPE and Unigram suggest the model is able to model the dataset
almost equivalently in all the cases, differences in the objective evaluation
of generated musical quality, indicate that not all the information is
captured within NLL loss. Furthermore, it supports our initial assumption (in
section 4.3)that NLL is not an appropriate metric to act upon conclusively
over the model performances. However, it can still provide an overview of how
well the model is fitting the dataset, which in the case of usage of subword
tokenization is almost the same as without it.

##  6 Discussion

In order to provide some qualitative insights from the generated content, we
here present an individual subjective analysis of some of the results. Despite
good results in terms of overall structureness presented in Table 3, we
noticed that on some occasions the model opts by resorting to rests (silence)
for a few measures. This can obviously be a desirable outcome sometimes, but
as is observable in Figure 3, the rests from measure 24 to 28 seem to detract
from the previous flow in terms of the musical idea behind it.

Figure 3: MuseScore screenshot of measures 17 to 34 of sample 19 from the
MAESTRO BPE experiment.

Furthermore, despite the frequentist approach in subword tokenization
procedures, often giving emphasis to combinations of words/subwords that are
more common in a given corpus, for the particular case of guitar-focused
symbolic music generation with the DadaGP dataset, it is interesting to
observe that tokens concerning guitar expressivity techniques are preserved,
despite its diminished frequency when compared to note tokens.

Figure 4: GuitarPro screenshot of the first five measures from sample 18 from
the DadaGP BPE experiment. Only the distorted guitar is visible.

As we can see from Figure 4, guitar-specific expressivity techniques such as
hammer-ons and pull-offs, bends, slides, and vibrato, are adequately used.

Figure 5: GuitarPro screenshot of the first nine measures from sample 10 from
the DadaGP Unigram experiment. Only the distorted guitar is visible.

In order to visually support the arguments made in Section 5.2 towards the
improvement in terms of ‘structureness’ of the generated examples from the
subword tokenization models, in Figure 5 we can observe that the model is able
to call-back to the motifs played on the first two measures (i.e. same pattern
repeating in measures seven and eight). From measure 5, it is also interesting
to observe that the model was able to refer to the same pattern that was
introduced in the first two measures but also incorporated a few connecting
notes in its first beats.

Figure 6: Fitness scape plots for two generated samples from each of Base, BPE
and Unigram along with real songs corresponding to the MAESTRO dataset. The
x-axis represents the segment center (in sec) and y-axis represents the
segment length (in sec) for any repetitive structure in music

A similar observation on improvements in terms of structural representation
can be made if we analyze the fitness scape plots for the generated songs
against real songs. A sample of such observation is shown in Figure 6 for
MAESTRO dataset samples. Contours in yellowish/brownish regions in scape plots
represent the repetitive structure, with the lower side implying short-term
and the higher side implying long-term repetitive structure. A better model is
one that generates songs having a scape-plot similar to that of real songs
since then the musical structure of the training dataset is captured in a more
accurate way. As we can observe from the scape plots given in Figure 6, there
is a much less long-term repetitive structure in the real data. Both BPE and
Unigram have their scape plots more similar to real songs, with lesser
yellowish regions on the upper side of plots, than that of the base model.
This suggests that the BPE and Unigram models capture the overall musical
structure of real songs better than the base model, with Unigram being the
best among the three. This observation is in coherence with Table 3,
suggesting improvements with the use of subword tokenization methods.

##  7 Conclusion and Future Work

In this paper, we conducted an empirical study to assess the usefulness of
subword tokenization methods in symbolic music generation. We objectively
studied the change in performance of music generation models with the use of
subword tokenization techniques such as BPE and Unigram while answering the
two fundamental questions posed in Section 1. Our study not only shows that
BPE and Unigram, as data compression techniques, are not only able to
represent the data in a more efficient manner but also improve the overall
structure of music generated with model-inculcating subword tokenization.
Furthermore, this result (and the trends in improvement) stands irrespective
of the model, dataset, or musical tokenization used. Overall, from our study,
we can conclude that the inclusion of subword tokenization in Symbolic Music
generation has a potential impact on the performance of the model and
structure of generated music, while allowing longer music to be generated at
the same time.

From this point onwards, future work can be multifaceted. One direction could
be to explore the impact of vocabulary size on the model performance, i.e. how
the vocal-performance trade-off. Similar to text-NLP, where we usually have a
vocabulary size in the range of 50k, we can increase the vocab size here as
well to see how performance varies with changes in vocabulary size. Another
interesting direction could be to explore whether the knowledge of Music
theory can be leveraged in coordination with BPE or Unigram-type techniques,
to develop a hybrid version of subword tokenization, involving concurrency
(such as in Musical BPE) musical events with adjacency of tokens. In all,
there is a large scope of future work that can be done, beyond this study.

## Ethical Statement

The training of large language models is an intensive computational process
that requires vast amounts of energy, resulting in a significant carbon
footprint. Cloud providers are increasingly offering services for training and
hosting these models, but not all of them have committed to being carbon
neutral, meaning they may contribute to greenhouse gas emissions. This is an
important consideration when selecting a cloud provider for training models,
as it has both environmental and ethical implications.

To minimize the impact of training large language models, one solution is to
release pre-trained models. This approach enables others to use these models
without having to undergo the energy-intensive process of training them from
scratch. By releasing pre-trained models, we aim to reduce the carbon
footprint associated with training and make it easier for others to use these
models in a more sustainable manner. Additionally, releasing pre-trained
models may also encourage collaboration and innovation in the field, further
advancing the development of AI technology.

## Acknowledgement

We would like to express our sincere gratitude to Dr. Yi-Hsuan Yang, for his
supervision and guidance throughout this research project. Dr. Yang’s expert
advice and feedback were invaluable in shaping the direction and outcome of
this study. His constant support and motivation were crucial in keeping us
focused and inspired. We would also like to thank Dr. Sourav Mukhopadhyay, IIT
Kharagpur for giving us an opportunity to work on this project as Adarsh’s
Master’s Thesis Project.

## References

  * Conneau and Lample [2019] Alexis Conneau and Guillaume Lample.  Cross-lingual language model pretraining.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * Dai et al. [2019a] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov.  Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context.  In Proc. of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
  * Dai et al. [2019b] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov.  Transformer-xl: Attentive language models beyond a fixed-length context, 2019.
  * Dai et al. [2022] Shuqi Dai, Huiran Yu, and Roger B. Dannenberg.  What is missing in deep music generation? a study of repetition and structure in popular music, 2022.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2018.
  * Dong and Yang [2018] Hao-Wen Dong and Yi-Hsuan Yang.  Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation.  In Proc. of the 19th Int. Soc. for Music Information Retrieval Conf. (ISMIR), 2018.
  * Dong et al. [2020] Hao-Wen Dong, Ke Chen, Julian J. McAuley, and Taylor Berg-Kirkpatrick.  Muspy: A toolkit for symbolic music generation.  CoRR, abs/2008.01951, 2020.
  * Foote [1999] Jonathan Foote.  Visualizing music and audio using self-similarity.  In Proceedings of the Seventh ACM International Conference on Multimedia (Part 1), MULTIMEDIA ’99, page 77–80, New York, NY, USA, 1999. Association for Computing Machinery.
  * Fradet et al. [2021] Nathan Fradet, Jean-Pierre Briot, Fabien Chhel, Amal El Fallah Seghrouchni, and Nicolas Gutowski.  Miditok: A python package for midi file tokenization.  In Extended Abstracts for the Late-Breaking Demo Session of the 22nd International Society for Music Information Retrieval Conference, 2021.
  * Gage [1994] Philip Gage.  A new algorithm for data compression.  The C Users Journal archive, 12:23–38, 1994.
  * Gallé [2019] Matthias Gallé.  Investigating the effectiveness of BPE: The power of shorter sequences.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1375–1381, Hong Kong, China, November 2019. Association for Computational Linguistics.
  * Guo et al. [2022] Rui Guo, Ivor Simpson, Chris Kiefer, Thor Magnusson, and Dorien Herremans.  Musiac: An extensible generative framework for music infilling applications with multi-level control, 2022.
  * Hawthorne et al. [2019] Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel, and Douglas Eck.  Enabling factorized piano music modeling and generation with the MAESTRO dataset.  In International Conference on Learning Representations, 2019.
  * Hiller [2019] Lejaren Hiller.  IV. Music Composed With Computers—A Historical Survey, pages 42–96.  Cornell University Press, Ithaca, NY, 2019.
  * Hsiao et al. [2021] Wen-Yi Hsiao, Jen-Yu Liu, Yin-Cheng Yeh, and Yi-Hsuan Yang.  Compound Word Transformer: Learning to Compose Full-Song Music Over Dynamic Directed Hypergraphs.  In Proc. of the AAAI Conf. on Artificial Intelligence, 2021.
  * Huang and Yang [2020] Yu-Siang Huang and Yi-Hsuan Yang.  Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions.  In Proc. of the 28th ACM Int. Conf. on Multimedia, 2020.
  * Huang et al. [2018] Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Curtis Hawthorne, Andrew M. Dai, Matthew D. Hoffman, and Douglas Eck.  An improved relative self-attention mechanism for transformer with application to music generation.  CoRR, abs/1809.04281, 2018.
  * Huang et al. [2019] Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck.  Music Transformer: Generating Music with Long-term Structure.  In Proc. of the 7th Int. Conf. on Learning Representations, 2019\.
  * Kudo and Richardson [2018] Taku Kudo and John Richardson.  Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  CoRR, abs/1808.06226, 2018.
  * Kudo [2018] Taku Kudo.  Subword regularization: Improving neural network translation models with multiple subword candidates.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia, July 2018. Association for Computational Linguistics.
  * Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized BERT pretraining approach.  CoRR, abs/1907.11692, 2019.
  * Liu et al. [2022] Jiafeng Liu, Yuanliang Dong, Zehua Cheng, Xinran Zhang, Xiaobing Li, Feng Yu, and Maosong Sun.  Symphony generation with permutation invariant language model.  2022\.
  * Meade et al. [2019] Nicholas Meade, Nicholas Barreyre, Scott C Lowe, and Sageev Oore.  Exploring Conditioning for Generative Music Systems with Human-Interpretable Controls.  2019\.
  * Mielke et al. [2021] Sabrina J. Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y. Lee, Benoît Sagot, and Samson Tan.  Between words and characters: A brief history of open-vocabulary modeling and tokenization in nlp, 2021.
  * Muhamed et al. [2021] Aashiq Muhamed, Liang Li, Xingjian Shi, Suri Yaddanapudi, Wayne Chi, Dylan Jackson, Rahul Suresh, Zachary C. Lipton, and Alexander J. Smola.  Symbolic music generation with transformer-gans.  In 35th AAAI Conference on Artificial Intelligence, AAAI 2021, 2021.
  * Müller et al. [2011] Meinard Müller, Peter Grosche, and Nanzhu Jiang.  A segment-based fitness measure for capturing repetitive structures of music recordings.  pages 615–620, 01 2011.
  * Neves et al. [2022] Pedro Neves, Jose Fornari, and João Florindo.  Generating music with sentiment using transformer-gans, 2022.
  * Oore et al. [2018] Sageev Oore, Ian Simon, Sander Dieleman, Douglas Eck, and Karen Simonyan.  This time with feeling: Learning expressive musical performance.  Neural Computing and Applications, 2018.
  * Park et al. [2020] Kyubyong Park, Joohong Lee, Seongbo Jang, and Dawoon Jung.  An empirical study of tokenization strategies for various Korean NLP tasks.  In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 133–142, Suzhou, China, December 2020. Association for Computational Linguistics.
  * Payne [2019] Christine Payne.  Musenet, 2019.
  * Peracha [2020] Omar A Peracha.  Improving polyphonic music models with feature-rich encoding.  2020\.
  * Radford et al. [2019] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019\.
  * Sarmento et al. [2021] Pedro Sarmento, Adarsh Kumar, CJ Carr, Zack Zukowski, Mathieu Barthet, and Yi-Hsuan Yang.  DadaGP: a Dataset of Tokenized GuitarPro Songs for Sequence Models.  In Proc. of the 22nd Int. Soc. for Music Information Retrieval Conf., 2021.
  * Sarmento et al. [2023] Pedro Sarmento, Adarsh Kumar, Yu-Hua Chen, CJ Carr, Zack Zukowski, and Mathieu Barthet.  Gtr-ctrl: Instrument and genre conditioning for guitar-focused music generation with transformers, 2023.
  * Sennrich et al. [2016] Rico Sennrich, Barry Haddow, and Alexandra Birch.  Neural machine translation of rare words with subword units.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany, August 2016. Association for Computational Linguistics.
  * Sturm et al. [2016] Bob L. Sturm, João Felipe Santos, Oded Ben-Tal, and Iryna Korshunova.  Music transcription modelling and composition using deep learning.  In Proc. on the 1st Conf. on Computer Simulation of Musical Creativity, 2016.
  * Tan and Herremans [2020] Hao Hao Tan and Dorien Herremans.  Music FaderNets: Controllable Music Generation Based On High-Level Features via Low-Level Feature Modelling.  In Proc. of the 21st Int. Soc. for Music Information Retrieval Conf., 2020.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention Is All You Need.  In Proc. of the 31st Conf. on Neural Information Processing Systems, 2017.
  * Wang et al. [2020] Ziyu Wang, Dingsu Wang, Yixiao Zhang, and Gus Xia.  Learning Interpretable Representation for Controllable Polyphonic Music Generation.  In Proc. of the 21st Int. Soc. for Music Information Retrieval Conf., Montréal, Canada, 2020.
  * Wolf et al. [2020] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush.  Huggingface’s transformers: State-of-the-art natural language processing, 2020.
  * Wu and Yang [2020] Shih-Lun Wu and Yi-Hsuan Yang.  The jazz transformer on the front line: Exploring the shortcomings of ai-composed music through quantitative measures.  CoRR, abs/2008.01307, 2020.
  * Wu and Yang [2022] Shih-Lun Wu and Yi-Hsuan Yang.  Compose & embellish: Well-structured piano performance generation via a two-stage approach, 2022.
  * Wu et al. [2016] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean.  Google’s neural machine translation system: Bridging the gap between human and machine translation, 2016.