# Image-Based Conditioning for Action Policy Smoothness in Autonomous
Miniature Car Racing with Reinforcement Learning

Bo-Jiun Hsu  Department of Computer Science, National Yang Ming Chiao Tung
University, Taiwan  Hoang-Giang Cao  Department of Computer Science, National
Yang Ming Chiao Tung University, Taiwan  Research Center for IT Innovation,
Academia Sinica, Taiwan  I Lee  Department of Computer Science, National Yang
Ming Chiao Tung University, Taiwan  Chih-Yu Kao  Department of Computer
Science, National Yang Ming Chiao Tung University, Taiwan  Jin-Bo Huang
Department of Computer Science, National Yang Ming Chiao Tung University,
Taiwan  I-Chen Wu† Department of Computer Science, National Yang Ming Chiao
Tung University, Taiwan  Research Center for IT Innovation, Academia Sinica,
Taiwan

###### Abstract

In recent years, deep reinforcement learning has achieved significant results
in low-level controlling tasks. However, the problem of control smoothness has
less attention. In autonomous driving, unstable control is inevitable since
the vehicle might suddenly change its actions. This problem will lower the
controlling system’s efficiency, induces excessive mechanical wear, and causes
uncontrollable, dangerous behavior to the vehicle. In this paper, we apply the
Conditioning for Action Policy Smoothness (CAPS) with image-based input to
smooth the control of an autonomous miniature car racing. Applying CAPS and
sim-to-real transfer methods helps to stabilize the control at a higher speed.
Especially, the agent with CAPS and CycleGAN reduces 21.80% of the average
finishing lap time. Moreover, we also conduct extensive experiments to analyze
the impact of CAPS components.

$\dagger$$\dagger$footnotetext: Correspondence.

##  I INTRODUCTION

In recent years, deep reinforcement learning (DRL) has archived many
milestones. AlphaZero, a computer program based on reinforcement learning,
achieved superhuman performance in chess, shogi, and Go through self-play[12].
For the real-time strategy (RTS) games, deep reinforcement learning also
reached the human level performance. For example, in 2018, DeepMind proposed
AlphaStar, and defeated professional players for the first time[13]; OpenAI
Five is the first AI to beat the world champions in an e-sports game [10]. DRL
is also applied to many real-world applications. Shixiang Gu et al. applied
DRL for robotic manipulation with asynchronous off-policy updates [4]. For
autonomous driving, Amazon Web Services (AWS) provided DeepRacer, an
autonomous racing experimentation platform for sim-to-real reinforcement
learning[1]. However, training DRL usually requires a large amount of data for
training. Collecting data in the real world is extremely costly and time-
consuming. In addition, to acquire an optimal policy or near-optimal policy,
the agent needs to perform random actions to explore, which can be unstable
and dangerous during real-world data collection for a task like autonomous
driving. Sim-to-real provides an efficient approach for training DRL. We often
train agents in the simulation and migrate them to the real world.

However, the intrinsic gap between simulation and the real world often causes
the agent trained in simulation to perform poorly in the real-world [7]. Prior
works mainly focus on domain randomization and domain adaptation methods. The
most widely used sim-to-real transfer method is domain randomization, but
domain randomization requires careful task-specific hyper-parameter tuning,
and the cost grows exponentially with the number of parameters [14, 7]. The
work in [3] proposed a student-teacher method for sim-to-real transfer in
DeepRacer. The student model after training can perform better than the
teacher model and other randomization methods. Recent works turned to
learning-based domain adaptation. RL-CycleGAN combined Generative Adversarial
Network (GAN) with a task-specific RL constraint in a robotics grasping task,
but collecting task specific data can be expensive [11]. RetinaGAN improved
GAN via object detection consistency auxiliary loss[6]. However, in autonomous
driving, the observation has no specific object for tracking to follow the
perception consistency loss as proposed.

Another critical problem in autonomous driving is the jerky behavior, where
the vehicle moves unstable or shaky. Jerky control causes many serious
problems, such as uncontrollable moving and power-consuming, which may reduce
the service life of the autonomous robot. Siddharth Mysore et al. proposed
Conditioning for Action Policy Smoothness (CAPS) for solving jerky actions by
adding regularization terms[9]. CAPS was originally applied to smooth the
control of quadrotor drones. There is less research about solving the jerky
control problem in autonomous driving.

In this paper, we extend the idea of Conditioning for Action Policy Smoothness
(CAPS)[9] to image-based CAPS and applying to solve jerky control in
autonomous miniature car racings. In the experiment, we analyze the impact of
CAPS components. We also compare the effectiveness of sim-to-real transfer
approaches in autonomous car racings.

Contributions. The main contributions of this paper can be summarized as
follow:  1) We extend CAPS[9] to image-based CAPS, and apply to solve jerky
control in autonomous car racing.  2) In the experiments, we show that CAPS
helps stabilize the car when moving at a higher speed. Especially, the method
that combines CycleGAN and CAPS outperforms other methods and reduces the
average finish lap time, while also improving the completion rate.  3) We also
conducted extensive experiments to study the impact of CAPS components.

##  II CAPS: Conditioning for Action Policy Smoothness

Siddharth Mysore et al. introduced Conditioning for Action Policy Smoothness
(CAPS)[9] and got significant improvements in controller smoothness and power
consumption on a quadrotors drone. To condition policies for smooth control,
the authors proposed two regularization terms:  1) Temporal Smoothness term.
2) Spatial Smoothness term.

The policy $\pi$ is a mapping function of states $s$ to actions $a=\pi(s)$.
The objective function of CAPS, $J_{\pi}^{CAPS}$, contains three components:
objective function of a model-free DRL $J_{\pi}$ (we used Soft Actor Critic in
this work); Temporal Smoothness regularization term $L_{T}$; and Spatial
Smoothness regularization term $L_{S}$. The regularization weights
$\lambda_{T}$ and $\lambda_{S}$ are used to balance the impact of two
regularization terms $L_{T}$ and $L_{S}$, respectively.

$J_{\pi}^{CAPS}=J_{\pi}-\lambda_{T}L_{T}-\lambda_{S}L_{S}$ (1)
$L_{T}=D_{T}(\pi(s_{t}),\pi(s_{t+1}))$ (2)
$L_{S}=D_{S}(\pi(s_{t}),\pi({s}_{t}^{\prime}))\quad where\quad
s_{t}^{\prime}\sim\Phi(s_{t})$ (3)

$D$ is the Euclidean distance, i.e $D(a,b)=||a-b||_{2}$.

The Temporal Smoothness term $L_{T}$ penalize the $J_{\pi}^{CAPS}$ when the
action of the next states $s_{t+1}$ are significantly different from the
actions of the current states $s_{t}$. The Spatial Smoothness term $L_{S}$
ensures that the policy takes the similar actions on the similar states
$S_{t}^{\prime}$, which are drawn from a distribution $\phi$ around states
$s$. Originally, CAPS is applied to smooth the control of a quadrotor drone
with the input are the inertial measurement unit (IMU) and the electronic
speed controller (ESC) sensors. Therefore, to generate similar states
$S_{t}^{\prime}$, the authors sample from normal distribution,
$\Phi(s)=N(s,\sigma)$ with standard deviation $\sigma$, around $s$. Figure 1
illustrates the Temporal Smoothness and Spatial Smoothness in CAPS.

In this paper, we extend the idea of CAPS to smooth the control of a miniature
car racing with the image-based input.

(a) Temporal Smoothness.

(b) Spatial Smoothness.

Figure 1: Condition policies for smooth behavior in CAPS.

##  III OUR APPROACH

###  III-A Image-based CAPS for Autonomous Car Racing

In this work, we extend the idea of using CAPS with image-based input and then
apply it to smooth the control of an autonomous miniature car racing. CAPS was
originally applied to smooth the control of a drone with the internal states
of the rotors. Then, to generate the similar state $S_{t}^{\prime}$, the
authors used Gaussian Noise to draw $S_{t}^{\prime}$ from a normal
distribution around state $S_{t}$ as describe in section II. Our approach use
image as the input, therefore, to generate the similar state $S_{t}^{\prime}$
in the Spatial Smoothness in CAPS, we implement 6 domain randomization
methods:

  1. 1.

Random Brightness: adjust the brightness of the image.

  2. 2.

Random Contrast: adjust the degree to which light and dark colors in the image
differ.

  3. 3.

Random Rotation: rotate the image with a random angle.

  4. 4.

Salt and Pepper: add salt and pepper noise to the image.

  5. 5.

Gaussian Blur: blur an image by a Gaussian function.

  6. 6.

Random Cut-off: random cut-off the image by overlaying a black rectangle at a
random position with random size on the top of the image.

Figure 2 demonstrates our domain randomization methods that we used to
generate the similar states $S_{t}^{\prime}$ from states $s$.

Figure 2: Domain randomization methods.

We also study the impact between Temporal Smoothness and Spatial Smoothness by
adjusting the regularization weights ${\lambda{{}_{T}}}$ and
${\lambda{{}_{S}}}$ in the subsection V-D

###  III-B Sim-to-real transfer for Autonomous Car Racing

Sim-to-real transfer plays an important role in the sim-to-real task. In sim-
to-real tasks, we trained in the policy in the simulation and then directly
transfer in the real world environment. There are two main approaches for sim-
to-real transfer: domain randomization and domain adaptation. Domain
randomization tries to enlarge the data distribution to cover the data as much
as possible; while domain adaptation tries to translate the data from this
domain to another domain, which is simulation and real in this paper.

To analize the effectiveness of different sim-to-real transfer approaches in
autonomous miniature car racings, we implement CycleGAN, a domain adaptation
method, and three different domain randomization methods:  1) Salt and Pepper
Noise [2].  2) Random Reflection.  3) Random HSV Shift.

##  IV EXPERIMENT SETUP

###  IV-A Environments Setup

In the real world, our track had two straight acceleration zones, two square
corners, three hairpin corners, and one s-curve. If the car crosses the
trapezoidal wall, it fails.

In the simulation, there is no reflections, noises, or car shake on the
simulator track, while the real track has distinct differences because of the
dynamic sunlight changing.

Figure 3 are our simulation track and real track. The autonomous miniature car
is set up with a camera placed on the top of the car. See Figure 4. The camera
captures RGB images with a resolution of 120x160 at 30fps. The computing
device used in the real car is NVIDIA Jetson Xavier NX.

(a) Simulation track.

(b) Real track.

Figure 3: Simulation track(left) and real track(right).

Figure 4: Simulation car(left) and real car(right).

###  IV-B Network Structure and Training

We implement a version of Ape-X[8] with Soft-Actor-Critic (SAC)[5] with the
following configurations: number of workers is 3; N-step is 4; gamma is 0.98;
initial alpha is 0.3; batch size is 512; learning rate is 0.0003; global
buffer size is 45000; local buffer size is 2000.

Our input is the images that are captured from a camera placed on the top of
the miniature car racing, Figure 4. The observation is a RGB image with the
resolution 120x160, and the color of each pixel ranges from 0 to 255. We stack
the current observation with the previous observation as the input. Therefore
the input size is 120x160x6. The output are the continuous values of steering
angle and speed with a range in [-1, 1]. Figure 5 illustrate our network
structure used in this paper.

Figure 5: Network structure.

###  IV-C Baselines

We implement a version of Deep Neural Network Ape-X[8] with Soft-Actor-Critic
(SAC)[5] without sim-to-real transfer method as the baseline. We then compare
the effectiveness of sim-to-real methods and CAPS for the smoothness control.
The different models are as follows:

  1. 1.

Baseline with CAPS and no CAPS.

  2. 2.

Domain randomization with CAPS and without CAPS.

  3. 3.

CycleGAN with and without CAPS.

###  IV-D Metrics

To evaluate the effectiveness of the smoothness method (CAPS), we use two
following metrics:

  1. 1.

Mean of selected steering angle: measure the angle difference when the agent
decide to change the moving direction.

  2. 2.

Smoothness value: proposed by [9], a method based on the Fast Fourier
Transform frequency spectrum as defined in Equation 4 . The lower the value,
the higher the low-frequency control and smoother the action.

$S_{m}=\frac{2}{nf_{s}}\sum_{i=1}^{n}M_{i}f_{i}$ (4)

where $M_{i}$ is the amplitude of the frequency component $f_{i}$, and the
$f_{s}$ is the sampling rate. We set $f_{s}=30$ in this work.

To evaluate the performance of the car racing, we use two metrics:

  1. 1.

Average finishing lap time: the average time to finish a run (in second).

  2. 2.

Completion rate: the % number of completion runs per all runs.

##  V EXPERIMENT RESULTS

In the experiment, we first verify the performance of image-based CAPS in V-A.
We then combine CAPS with different sim-to-real methods to compare the
effectiveness of sim-to-real transfer for autonomous miniature car racing
(V-B). Finally, we conduct extensive experiments to analyze the impact of CAPS
components (V-C, V-D, V-E).

###  V-A Smoothness with CAPS

| Without CAPS | With CAPS
---|---|---
Steering angle | 93.47 | 20.39
Speed | 78.35 | 13.72
TABLE I: Compare the smoothness value of steering angle and speed with CAPS
and without CAPS.

In this experiment, we study the smoothness ability of CAPS. We train the
agent in the simulation and test it in the real environment. The sim-to-real
method is CycleGAN.

Figure 6, 7, 8 and 9 show the comparison of the steering angle and the speed
with CAPS and without CAPS. The model with CAPS achieves more stable behaviors
in real-world testing. Table I compares the smoothness value of steering angle
and speed of the model with CAPS and without CAPS. The results show that CAPS
significantly improves the smoothness value in both speed and steering angle.

Figure 6: Smoothness value of speed without CAPS (left) and with CAPS (right).

Figure 7: Smoothness value of steering angle without CAPS (left) and with CAPS
(right).

(a) Speed of the car without CAPS.

(b) Speed of the car with CAPS.

Figure 8: Speed of the car without CAPS and with CAPS.

(a) Steering angle of the car without CAPS.

(b) Steering angle of the car with CAPS.

Figure 9: Steering angle of the car without CAPS and with CAPS. | Baseline | Domain Randomization | CycleGAN
---|---|---|---
| w/o CAPS | with CAPS | w/o CAPS | with CAPS | w/o CAPS | with CAPS
Real World (c1) | 0% | 0% | 0% | 100% | 73.33% | 73.33%
Real World (c2) | 0% | 0% | 0% | 26.67% | 86.67% | 80.00%
Real World (c3) | 0% | 0% | 0% | 0% | 13.33% | 20.00%
TABLE II: Completion rate. | Baseline | Domain Randomization | CycleGAN
---|---|---|---
| w/o CAPS | with CAPS | w/o CAPS | with CAPS | w/o CAPS | with CAPS
Real World (c1) | NaN | NaN | NaN | 32.47s | 22.20s | 17.36s
Real World (c2) | NaN | NaN | NaN | 24.33s | 19.77s | 16.05s
Real World (c3) | NaN | NaN | NaN | NaN | 17.20s | 14.71s
TABLE III: Average finishing lap time.

###  V-B Smoothness Control and Sim-to-Real Policy Transfer

In this experiment, we compare the effectiveness of both the action smoothing
method and sim-to-real transfer methods. We conduct the experiments to compare
the following models:

  1. 1.

SAC only (without CAPS, without sim-to-real transfer method).

  2. 2.

SAC + CAPS (without sim-to-real transfer method)

  3. 3.

SAC + DR (without CAPS)

  4. 4.

SAC + DR + CAPS

  5. 5.

SAC + CycleGAN (without CAPS)

  6. 6.

SAC + CycleGAN + CAPS

For the domain randomization methods, we apply three types: Salt and Pepper
noise, Random Reflection, Random HSV Shift. Figure 10 shows examples of
different domain randomization methods. For the domain adaptation method, we
implement CycleGAN. Figure 11 shows our CycleGAN result for translating
between the simulation and real images and vice versa. The agent is trained in
the simulation with the sim-to-real transfer methods and then tested 15 times
with three different speed configurations (denoted as c1, c2, c3) in the real
environment. The speed configuration is a range of speed values that the car
can take. We want to test the ability to stable the control of CAPS by
increasing the speed of the car. That is no doubt that running at a higher
speed is more difficult for the agent to make a decision. We setup different
speed configurations as follow:

  1. 1.

Real World (c1): from 1.125 to 8 (m/s).

  2. 2.

Real World (c2): from 1.125 to 9 m/s.

  3. 3.

Real World (c3): from 3.15 to 9.3 (m/s)

Table II and Table III show the results of completion rate and average finish
lap time. Without CAPS and without sim-to-real transfer method, the agents can
not transfer the policies from the simulation to the real environment. With
CAPS, the behaviors of the agents are more stable, therefore can finish a run
faster than the agents without CAPS.

Figure 10: Domain randomization methods for sim-to-real transfer.

(a) From left to right: the simulation image; the generated real image by
CycleGAN; the reconstructed simulation image.

(b) From left to right: the real image; the generated sim image by CycleGAN;
the reconstructed real image.

Figure 11: CycleGAN results.

###  V-C Study the impact of CAPS components

In this experiment, we want to study the impact of each component on the
objective function of CAPS (Equation 1). We compare the finishing lap time and
the completion rate of the following models:

  1. 1.

SAC only.

  2. 2.

SAC + Steering angle penalty: a reward engineering method with Steering Angle
Penalty = 0.003 * abs(degree).

  3. 3.

SAC + Temporal Smoothness term.

  4. 4.

SAC + Spatial Smoothness term.

  5. 5.

SAC + CAPS (Spatial + Temporal).

Figure 12 shows the result of comparison. We can see that the Temporal
Smoothness or the Spatial Smoothness individually improves the smoothness
value. Moreover, the Temporal Smoothness takes the most impact to the result
and then significantly reduces the finishing lap time.

(a) Smoothness value.

(b) Average finishing lap time.

Figure 12: Compare the impact of the components in CAPS.

###  V-D Temporal Smoothness Sensitivity Analysis

From the experiment above (V-C), we see that the Temporal Smoothness has the
most impact on improving the agent’s performance. In this experiment, we
analyze the sensitivity of the Temporal Smoothness in CAPS. We remove the
Spatial Smoothness term by setting the $\lambda_{S}=0$ and then test with
different values of $\lambda_{T}$: 0.5, 0.8, 1.0, 1.3.

From the results in Figure 13, we see that $\lambda_{T}=1.0$ gives the best
result for both smoothness value and finishing lap time.

(a) Smoothness value.

(b) Average finishing lap time.

Figure 13: Compare the sensitivity of Temporal Smoothness.

###  V-E Spatial Smoothness Ablation Study

We conduct an ablation study to investigate the influence of the randomization
methods on Spatial Smoothness. In this experiment, we individually remove each
randomization method to compare the impact of each method on the Spatial
Smoothness. The experiment shows that the performance is less reduced when
removing the Random Brightness while removing the Gaussian Blur or Salt and
Pepper clearly drop the result. A possible explanation is that the image
captured in the simulation has less noise than the image captured in the real
world. Adding Salt and Pepper helps the agent cover this type of image.
Moreover, when the car runs fast, the captured images are easily blurred;
thus, training with more blurred images improves the performance of the agent.
Figure 14 show the result of the Spatial Smoothness ablation study.

(a) Finishing lap time in Spatial Smoothness ablation study.

(b) Finishing lap time in Spatial Smoothness ablation study.

Figure 14: Spatial Smoothness ablation study.

##  VI Conclusion

This paper presents image-based CAPS, an image-based regularizing action
policies method to smooth the control of autonomous miniature car racing. The
model that combines image-based CAPS and CycleGAN achieved the best result in
real-world testing, which reduces the average finishing lap time; while
improving the completion rate. Analyzing the CAPS components shows that the
Temporal Smoothness term takes more impact on the performance of CAPS. An
ablation study of the Spatial Smoothness term indicates that Salt and Pepper
and Gaussian Blur are two important randomization methods influencing the
result.

## References

  * [1] Bharathan Balaji, Sunil Mallya, Sahika Genc, Saurabh Gupta, Leo Dirac, Vineet Khare, Gourav Roy, Tao Sun, Yunzhe Tao, Brian Townsend, Eddie Calleja, Sunil Muralidhara, and Dhanasekar Karuppasamy.  Deepracer: Educational autonomous racing platform for experimentation with sim2real reinforcement learning.  CoRR, abs/1911.01562, 2019.
  * [2] Yoshua Bengio, Frédéric Bastien, Arnaud Bergeron, Nicolas Boulanger–Lewandowski, Thomas Breuel, Youssouf Chherawala, Moustapha Cisse, Myriam Côté, Dumitru Erhan, Jeremy Eustache, Xavier Glorot, Xavier Muller, Sylvain Pannetier Lebeuf, Razvan Pascanu, Salah Rifai, François Savard, and Guillaume Sicard.  Deep learners benefit more from out-of-distribution examples.  15:164–172, 11–13 Apr 2011.
  * [3] Yeong-Jia Roger Chu, Ting-Han Wei, Jin-Bo Huang, Yuan-Hao Chen, I Wu, et al.  Sim-to-real transfer for miniature autonomous car racing.  arXiv preprint arXiv:2011.05617, 2020.
  * [4] Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine.  Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates.  In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 3389–3396, 2017.
  * [5] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1861–1870. PMLR, 10–15 Jul 2018\.
  * [6] Daniel Ho, Kanishka Rao, Zhuo Xu, Eric Jang, Mohi Khansari, and Yunfei Bai.  Retinagan: An object-aware approach to sim-to-real transfer, 2020.
  * [7] Sebastian Höfer, Kostas Bekris, Ankur Handa, Juan Camilo Gamboa, Florian Golemo, Melissa Mozifian, Chris Atkeson, Dieter Fox, Ken Goldberg, John Leonard, et al.  Perspectives on sim2real transfer for robotics: A summary of the r: Ss 2020 workshop.  arXiv preprint arXiv:2012.03806, 2020.
  * [8] Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado van Hasselt, and David Silver.  Distributed prioritized experience replay.  CoRR, abs/1803.00933, 2018.
  * [9] Siddharth Mysore, Bassel Mabsout, Renato Mancuso, and Kate Saenko.  Regularizing action policies for smooth control with reinforcement learning.  2021\.
  * [10] OpenAI, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang.  Dota 2 with large scale deep reinforcement learning.  2019\.
  * [11] Kanishka Rao, Chris Harris, Alex Irpan, Sergey Levine, Julian Ibarz, and Mohi Khansari.  Rl-cyclegan: Reinforcement learning aware simulation-to-real.  CoRR, abs/2006.09001, 2020.
  * [12] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis.  A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.  Science, 362(6419):1140–1144, 2018.
  * [13] Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, L. Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander Sasha Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Le Paine, Caglar Gulcehre, Ziyun Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy P. Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver.  Grandmaster level in starcraft ii using multi-agent reinforcement learning.  Nature, pages 1–5, 2019.
  * [14] Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund.  Sim-to-real transfer in deep reinforcement learning for robotics: a survey.  In 2020 IEEE Symposium Series on Computational Intelligence (SSCI), pages 737–744. IEEE, 2020.


aainstitutetext: Institute for Gravitation and the Cosmos and Physics
Department. Penn State University. PA 16802, USA. bbinstitutetext:
Departamento de Matemáticas, Universidad Carlos III de Madrid. Avda. de la
Universidad 30, 28911 Leganés, Spain. ccinstitutetext: Grupo de Teorías de
Campos y Física Estadística. Instituto Gregorio Millán (UC3M). Unidad Asociada
al Instituto de Estructura de la Materia, CSIC, Madrid, Spain.ddinstitutetext:
Instituto de Estructura de la Materia, CSIC, Serrano 123, 28006, Madrid, Spain

# Palatini gravity with nonmetricity, torsion, and boundaries in metric and
connection variables

J. Fernando Barbero G a,c Juan Margalef-Bentabol b,c Valle Varo b,c Eduardo
J.S. Villaseñor<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We prove the equivalence in the covariant phase space of the metric and
connection formulations for Palatini gravity, with nonmetricity and torsion,
on a spacetime manifold with boundary. To this end, we will rely on the
cohomological approach provided by the relative bicomplex framework. Finally,
we discuss some of the physical implications derived from this equivalence in
the context of singularity identification through curvature invariants.

###### Keywords:

First order gravity, Palatini action, boundaries, covariant methods

##  1 INTRODUCTION

The study of general relativity on manifolds with boundary is of great
importance as these can be used to account for the asymptotics of the
gravitational field and the presence of horizons [1, 2, 3]. The original
formulation of general relativity relied on a Lorentzian metric as the
fundamental field. However, it is also possible to use tetrads for the same
purpose. This is important from a physical point of view for several reasons.
First, the use of tetrads provides a natural way to incorporate fermion fields
into the theory. Second, the tetrad formalism leads naturally to the Ashtekar
formulation, which is one of the main avenues for the quantization of gravity
[4, 5]. Finally, local Lorentz gauge invariance plays a crucial role in the
tetrad formalism, which may introduce significant differences in the treatment
of the theory compared with the metric formulation. Within both frameworks,
there are several possible choices for the action that can be classified
according to their independent basic field variables. Palatini theories are
those in which the actions are written either in terms of a metric and a
connection (metric formalism) or a frame and a spin connection (tetrad
formalism) [6, 7, 8, 9].

The present paper attempts to unify and expand certain aspects of previous
works [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20], focusing on the comparison
of both formulations and the treatment of boundaries. In this regard, we would
like to highlight the pioneering work by Obukhov [21], where he introduced the
appropriate surface terms for Palatini gravity. Here we generalize those
results and improve on the variational treatment of the problem by relying on
the recently proposed CPS algorithm [22], which provides a clean, consistent,
and ambiguity-free procedure to obtain the solution spaces, the presymplectic
forms canonically associated with the actions, and some relevant charges.

The equivalence of the metric-Palatini and tetrad-Palatini formalisms will be
proven in two steps. First, we will obtain a precise description of the
solution spaces for both theories which will allow us to map them
appropriately. Second, we will show that the presymplectic forms given by the
CPS algorithm are equivalent in both cases.

In the following we consider a 4-dimensional spacetime manifold $M$
diffeomorphic to $\Sigma\times\mathbb{R}$, where $\Sigma$ is a 3-dimensional
manifold with boundary $\partial\Sigma$ (possibly empty). We will refer to
$\partial_{L}M\cong\partial\Sigma\times\mathbb{R}$ as the lateral boundary of
$M$ and restrict ourselves to the open set of metrics making $\partial_{L}M$
time-like. A few words on notation: Greek letters will denote abstract indices
for tensorial objects in $M$ and barred Greek indices will denote tensors on
$\partial_{L}M$ (quite often the object itself will also carry an overbar).
The inclusion map will be denoted as $\jmath:\partial_{L}M\hookrightarrow M$
and its tangent map as $\jmath^{\alpha}_{\overline{\alpha}}$.

##  2 METRIC PALATINI

Given a connection $\widetilde{\nabla}$, we define its torsion, Riemann, and
Ricci tensors as

$\displaystyle\tensor{\widetilde{\mathrm{T}}\mathrm{or}}{{}^{\alpha}_{\mu}{}_{\nu}}(\mathrm{d}\phi)_{\alpha}=-[\widetilde{\nabla}_{\\!\mu},\widetilde{\nabla}_{\\!\nu}]\phi\,,$
$\displaystyle\tensor{\widetilde{\mathrm{R}}\mathrm{iem}}{{}^{\alpha}_{\beta}{}_{\mu}{}_{\nu}}Z^{\beta}=([\widetilde{\nabla}_{\\!\mu},\widetilde{\nabla}_{\\!\nu}]+\tensor{\widetilde{\mathrm{T}}\mathrm{or}}{{}^{\beta}_{\mu}{}_{\nu}}\widetilde{\nabla}_{\\!\beta})Z^{\alpha}\,,$
$\displaystyle\widetilde{\mathrm{R}}\mathrm{ic}_{\beta\nu}:=\tensor{\widetilde{\mathrm{R}}\mathrm{iem}}{{}^{\mu}_{\beta}{}_{\mu}{}_{\nu}}\,.$

If we endow $M$ with a connection $\widetilde{\nabla}$ and a metric $g$, we
have the nonmetricity tensor, the $(g,\\!\widetilde{\nabla})$-scalar-
curvature, the $(g,\\!\widetilde{\nabla})$-extrinsic-curvature of
$\partial_{L}M$, and its trace

$\displaystyle\widetilde{M}_{\alpha\beta\gamma}:=\widetilde{\nabla}_{\\!\alpha}g_{\beta\gamma}\,,$
$\displaystyle\widetilde{R}:=g^{\alpha\beta}\widetilde{\mathrm{R}}\mathrm{ic}_{\alpha\beta}\,,$
$\displaystyle\widetilde{K}_{\overline{\alpha}\overline{\beta}}:=\frac{1}{2}\jmath^{\alpha}_{\overline{\alpha}}\jmath^{\beta}_{\overline{\beta}}\Big{(}\widetilde{\nabla}_{\\!\alpha}\nu_{\beta}+g_{\alpha\gamma}\widetilde{\nabla}_{\\!\beta}\nu^{\gamma}\Big{)},$
$\displaystyle\widetilde{K}:=\overline{g}^{\overline{\alpha}\overline{\beta}}\widetilde{K}_{\overline{\alpha}\overline{\beta}}\,,$

where $\overline{g}:=\jmath^{*}g$ is the induced metric, $\nu^{\alpha}$ the
outward unit vector field, and $\nu_{\beta}=g_{\beta\gamma}\nu^{\gamma}$.
Notice that $\widetilde{R}$ and
$\widetilde{K}_{\overline{\alpha}\overline{\beta}}$ are (non-standard)
generalizations of the $g$-scalar $\accentset{\circ}{R}$ and $g$-extrinsic
curvature $\accentset{\circ}{K}$ defined by the $g$-Levi-Civita connection
$\accentset{\circ}{\nabla}$. Moreover, in general,
$\widetilde{K}_{\overline{\alpha}\overline{\beta}}$ is not symmetric if the
nonmetricity is different from zero.

Given two connections $\nabla$ and $\widetilde{\nabla}$, their difference is a
$(2,1)$-tensor $Q\equiv\widetilde{\nabla}-\nabla$. For a $(1,1)$-tensor
$\tensor{S}{{}^{\beta}_{\gamma}}$ we have

$(\widetilde{\nabla}_{\\!\alpha}-\nabla_{\\!\alpha})\tensor{S}{{}^{\beta}_{\gamma}}=\tensor{Q}{{}^{\beta}_{\alpha}{}_{\mu}}\tensor{S}{{}^{\mu}_{\gamma}}-\tensor{Q}{{}^{\mu}_{\alpha}{}_{\gamma}}\tensor{S}{{}^{\beta}_{\mu}}$

and analogously for higher order objects. Observe that if we choose a fiducial
connection, usually the $g$-Levi-Civita one, there is a bijection between
connections $\widetilde{\nabla}$ and $(2,1)$-tensors $Q$. Working with tensors
is usually easier, as they form a vector space while connections form an
affine space. Thus, in the following, we will use the variables $(g,Q)$
instead of the equivalent ones $(g,\\!\widetilde{\nabla})$.

We will now use the CPS algorithm [22], which essentially consists in
introducing a pair of bulk and boundary Lagrangians, compute the variations,
extract the equations of motion and symplectic potentials, and get the
presymplectic form on the space of solutions. The power of this method lies in
its cohomological nature which renders it ambiguity-free: we can pick any
representative Lagrangians and symplectic potentials to describe the solution
spaces and compute the presymplectic form.

###  2.1 The action

We consider actions of the form

$\mathbb{S}=\int_{M}L-\int_{\partial M}\overline{\ell}\,,$

for some bulk Lagrangian $L$ and some boundary Lagrangian $\overline{\ell}$.
The metric-GR action and its generalization, known as metric-Palatini, are
respectively given by the Lagrangian pairs

$\displaystyle
L^{(m)}_{\mathrm{EH}}(g):=\big{(}\accentset{\circ}{R}-2\Lambda\big{)}\mathrm{vol}_{g}\,,$
$\displaystyle\overline{\ell}^{(m)}_{\mathrm{GHY}}(g):=-2\accentset{\circ}{K}\,\mathrm{vol}_{\overline{g}}\,,$
$\displaystyle
L^{(m)}_{\mathrm{PT}}(g,Q):=\big{(}\widetilde{R}-2\Lambda\big{)}\mathrm{vol}_{g}\,,$
$\displaystyle\overline{\ell}^{(m)}_{\mathrm{PT}}(g,Q):=-2\widetilde{K}\,\mathrm{vol}_{\overline{g}}\,.$

It is straightforward to rewrite the Palatini Lagrangians as those of standard
GR plus a coupling term

$\displaystyle
L^{(m)}_{\mathrm{PT}}(g,Q)=L^{(m)}_{\mathrm{EH}}(g)+L^{(m)}_{\mathrm{CP}}(g,Q)\,,$
$\displaystyle\overline{\ell}^{(m)}_{\mathrm{PT}}(g,Q)=\overline{\ell}^{(m)}_{\mathrm{GHY}}(g)+\overline{\ell}^{(m)}_{\mathrm{CP}}(g,Q)\,,$

where

$\begin{array}[]{l}L^{(m)}_{\mathrm{CP}}(g,Q):=\big{(}C_{\lambda}A^{\lambda}-\tensor{Q}{{}^{\alpha}{}^{\beta}_{\lambda}}\tensor{Q}{{}^{\lambda}_{\alpha}{}_{\beta}}\big{)}\mathrm{vol}_{g}+\mathrm{d}\big{(}\iota_{\vec{A}-\vec{C}}\mathrm{vol}_{g}\big{)}\,,\\\\[4.30554pt]
\overline{\ell}^{(m)}_{\mathrm{CP}}(g,Q):=\jmath^{*}\big{(}\iota_{\vec{A}-\vec{C}}\mathrm{vol}_{g}\big{)}\,,\end{array}\qquad\begin{array}[]{l}A^{\alpha}:=g^{\beta\gamma}\tensor{Q}{{}^{\alpha}_{\beta}{}_{\gamma}}\,,\\\
B_{\beta}:=\tensor{Q}{{}^{\mu}_{\beta}{}_{\mu}}\,,\\\
C_{\gamma}:=\tensor{Q}{{}^{\mu}_{\mu}{}_{\gamma}}\,.\end{array}$

As mentioned before, the results given by the CPS algorithm do not depend on
the choice of Lagrangians as long as they define the same action (the pairs
are equal up to a _relative_ exact form, see [22] for more details). In the
present case it is easy to see that

$\displaystyle\big{(}L^{(m)}_{\mathrm{PT}},\overline{\ell}^{(m)}_{\mathrm{PT}}\big{)}\\!=\\!\big{(}L^{(m)}_{\mathrm{EH}},\overline{\ell}^{(m)}_{\mathrm{GHY}}\big{)}\\!+(\hat{L}^{(m)}_{\mathrm{CP}},0)\\!+\underline{\mathrm{d}}(\iota_{\vec{A}-\vec{C}}\mathrm{vol}_{g},0)\,,$
$\displaystyle\hat{L}^{(m)}_{\mathrm{CP}}:=L^{(m)}_{\mathrm{CP}}-\mathrm{d}(\iota_{\vec{A}-\vec{C}}\mathrm{vol}_{g})\,.$

Considering $(\hat{L}^{(m)}_{\mathrm{CP}},0)$ actually makes the computations
a little shorter, but we will stick to
$(L^{(m)}_{\mathrm{CP}},\overline{\ell}^{(m)}_{\mathrm{CP}})$ as this will
facilitate the comparison of our results with the existing literature.

###  2.2 Variations

The variations of the $L^{(m)}_{\mathrm{EH}}$ and
$\overline{\ell}^{(m)}_{\mathrm{GHY}}$ terms are standard and they may be
found, for instance, in [23]. The remaining variations are those of the
coupling Lagrangians, which can be written as

$\displaystyle\mathchoice{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{4.84373pt}\raisebox{0.0pt}{\kern
0.17221pt\rotatebox{90.0}{\kern-0.17221pt\makebox[4.84373pt][l]{}}\kern
0.17221pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{3.44444pt}\raisebox{0.0pt}{\kern
0.12915pt\rotatebox{90.0}{\kern-0.12915pt\makebox[3.44444pt][l]{}}\kern
0.12915pt}\hskip
0.6458pt}L^{(m)}_{\mathrm{CP}}=(\mathfrak{E}^{\mathrm{CP}}_{(m)})^{\alpha\beta}\mathchoice{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{4.84373pt}\raisebox{0.0pt}{\kern
0.17221pt\rotatebox{90.0}{\kern-0.17221pt\makebox[4.84373pt][l]{}}\kern
0.17221pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{3.44444pt}\raisebox{0.0pt}{\kern
0.12915pt\rotatebox{90.0}{\kern-0.12915pt\makebox[3.44444pt][l]{}}\kern
0.12915pt}\hskip
0.6458pt}g_{\alpha\beta}+\tensor{(\mathcal{E}^{\mathrm{CP}}_{(m)})}{{}_{\gamma}^{\alpha}{}^{\sigma}}\mathchoice{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{4.84373pt}\raisebox{0.0pt}{\kern
0.17221pt\rotatebox{90.0}{\kern-0.17221pt\makebox[4.84373pt][l]{}}\kern
0.17221pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{3.44444pt}\raisebox{0.0pt}{\kern
0.12915pt\rotatebox{90.0}{\kern-0.12915pt\makebox[3.44444pt][l]{}}\kern
0.12915pt}\hskip
0.6458pt}\tensor{Q}{{}^{\gamma}_{\alpha}{}_{\sigma}}+\mathrm{d}\Theta^{(m)}_{\mathrm{CP}}\,,$
$\displaystyle\mathchoice{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{4.84373pt}\raisebox{0.0pt}{\kern
0.17221pt\rotatebox{90.0}{\kern-0.17221pt\makebox[4.84373pt][l]{}}\kern
0.17221pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{3.44444pt}\raisebox{0.0pt}{\kern
0.12915pt\rotatebox{90.0}{\kern-0.12915pt\makebox[3.44444pt][l]{}}\kern
0.12915pt}\hskip
0.6458pt}\overline{\ell}^{(m)}_{\mathrm{CP}}-\jmath^{*}\Theta^{(m)}_{\mathrm{CP}}=0\,,$

where

$\displaystyle(\mathfrak{E}^{\mathrm{CP}}_{(m)})^{\alpha\beta}(g,Q)$
$\displaystyle:=\Big{(}\tensor{Q}{{}^{\gamma}{}^{\alpha}_{\sigma}}\tensor{Q}{{}^{\sigma}_{\gamma}{}^{\beta}}-C_{\sigma}\tensor{Q}{{}^{\sigma}{}^{\alpha}{}^{\beta}}+\frac{1}{2}g^{\alpha\beta}(C_{\sigma}A^{\sigma}-\tensor{Q}{{}^{\gamma}{}^{\tau}_{\sigma}}\tensor{Q}{{}^{\sigma}_{\gamma}{}_{\tau}})\Big{)}\mathrm{vol}_{g}\,,$
$\displaystyle\tensor{(\mathcal{E}^{\mathrm{CP}}_{(m)})}{{}_{\gamma}^{\alpha}{}^{\sigma}}(g,Q)$
$\displaystyle:=\Big{(}\delta^{\alpha}_{\gamma}A^{\sigma}+g^{\alpha\sigma}C_{\gamma}-\tensor{Q}{{}^{\sigma}_{\gamma}{}^{\alpha}}-\tensor{Q}{{}^{\alpha}{}^{\sigma}_{\gamma}}\Big{)}\mathrm{vol}_{g}\,.$

Notice that the latter is an equation of motion by itself, but the former is
not. To obtain one we have to add $\mathfrak{E}^{\mathrm{CP}}_{(m)}$ to the
Einstein equations coming from the variation of $L^{(m)}_{\mathrm{EH}}$ (which
only depends on $g$).

In view of the previous result, we can choose the following representatives
(defined up to a relative exact form) as the contributions of the
$\mathrm{CP}$ terms to the symplectic potentials

$\displaystyle\Theta^{(m)}_{\mathrm{CP}}(g,Q):=\mathchoice{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{4.84373pt}\raisebox{0.0pt}{\kern
0.17221pt\rotatebox{90.0}{\kern-0.17221pt\makebox[4.84373pt][l]{}}\kern
0.17221pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{3.44444pt}\raisebox{0.0pt}{\kern
0.12915pt\rotatebox{90.0}{\kern-0.12915pt\makebox[3.44444pt][l]{}}\kern
0.12915pt}\hskip
0.6458pt}(\imath_{\vec{A}-\vec{C}}\mathrm{vol}_{\overline{g}})\,,\qquad\qquad\qquad\overline{\theta}^{(m)}_{\mathrm{CP}}(g,Q):=0\,.$

###  2.3 Space of solutions

The algebraic equation of motion $\mathcal{E}^{\mathrm{CP}}_{(m)}(g,Q)=0$ can
be solved for $Q$. Its general solution is
$Q_{0}^{\alpha\beta\gamma}=g^{\alpha\gamma}U^{\beta}$ with arbitrary
$U^{\beta}$ (we sketch the proof of a similar fact in section 3.3). This
solution, moreover, satisfies $\mathfrak{E}^{\mathrm{CP}}_{(m)}(g,Q_{0})=0$
for any $g$. Hence, $(g,Q)$ is a solution for Palatini if and only if
$Q=Q_{0}$ and $g$ satisfies the Einstein equations:

$\mathrm{Sol}^{(m)}_{\mathrm{PT}}=\\{(g_{\alpha\beta},\delta^{\alpha}_{\gamma}U_{\\!\beta})\
/\ g\in\mathrm{Sol}^{(m)}_{\mathrm{GR}},\ U_{\\!\beta}\text{ arbitrary}\\}\,.$

This, in turn, proves that the metric sector of Palatini is equivalent to
metric-GR. Notice that the boundary only plays a role in the metric sector
$\mathrm{Sol}^{(m)}_{\mathrm{GR}}$ of the solution space (which is discussed
in detail in [23], where both Dirichlet and Neumann boundary conditions are
considered). We have now the following (“on shell”) identities over the space
of solutions

$\begin{array}[]{ll}\mathrm{\widetilde{R}}\tensor{\mathrm{iem}}{{}^{\alpha}_{\beta}{}_{\mu}{}_{\nu}}=\mathrm{\accentset{\circ}{R}}\tensor{\mathrm{iem}}{{}^{\alpha}_{\beta}{}_{\mu}{}_{\nu}}+g^{\alpha}_{\beta}(\mathrm{d}U)_{\mu\nu}\,,\\\\[1.29167pt]
\mathrm{\widetilde{R}ic}_{\beta\nu}=\mathrm{\accentset{\circ}{R}ic}_{\beta\nu}+(\mathrm{d}U)_{\beta\nu}\,,&\widetilde{R}=\accentset{\circ}{R}\,,\\\
\widetilde{K}_{\overline{\alpha}\overline{\beta}}=\accentset{\circ}{K}_{\overline{\alpha}\overline{\beta}}\,,&\widetilde{K}=\accentset{\circ}{K}\,,\\\
\widetilde{M}_{\alpha\beta\gamma}=-2g_{\beta\gamma}U_{\alpha}\,,&\tensor{\widetilde{T}}{{}^{\gamma}_{\alpha}{}_{\beta}}=\delta^{\gamma}_{\beta}U_{\alpha}-\delta^{\gamma}_{\alpha}U_{\beta}\,.\end{array}$
(1)

The last two equations imply that, on solutions, $\widetilde{M}=0$ if and only
if $\widetilde{T}=0$. Thus, we have either the Levi-Civita connection or one
with nonmetricity and torsion.

###  2.4 Presymplectic form

The metric-Palatini presymplectic form on the space of solutions
$\includegraphics[height=6.83331pt]{./OOmega.pdf}^{\mathrm{PT}}_{(m)}$ has the
same functional form as the metric-GR presymplectic form
$\includegraphics[height=6.83331pt]{./OOmega.pdf}^{\mathrm{GR}}_{(m)}$ since
$\mathchoice{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{4.84373pt}\raisebox{0.0pt}{\kern
0.17221pt\rotatebox{90.0}{\kern-0.17221pt\makebox[4.84373pt][l]{}}\kern
0.17221pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{3.44444pt}\raisebox{0.0pt}{\kern
0.12915pt\rotatebox{90.0}{\kern-0.12915pt\makebox[3.44444pt][l]{}}\kern
0.12915pt}\hskip 0.6458pt}\Theta^{(m)}_{\mathrm{CP}}=0$. However, the solution
spaces are different. In order to compare them, we need the projection
$\pi_{(m)}(g,Q)=g$. In this way we get

$\includegraphics[height=6.83331pt]{./OOmega.pdf}^{\mathrm{PT}}_{(m)}=\pi_{(m)}^{*}\includegraphics[height=6.83331pt]{./OOmega.pdf}^{\mathrm{GR}}_{(m)}$
(2)

Clearly, all vectors of the form $(0,\mathbb{V})\in
T\mathrm{Sol}^{(m)}_{\mathrm{PT}}$, i.e. in the directions of the connection
sector, define gauge directions since they belong to the kernel of
$(\pi_{(m)})_{*}$ and hence annihilate
$\includegraphics[height=6.83331pt]{./OOmega.pdf}^{\mathrm{PT}}_{(m)}$.

##  3 TETRAD PALATINI

Given a tetrad $e_{\alpha}^{I}$, we have its $\eta_{IJ}$-dual $E^{\alpha}_{I}$
and the space-time metric $g=\eta_{IJ}e^{I}e^{J}=:\Phi_{\mathrm{GR}}(e)$ as
explained in more detail in [23]. We can consider the Levi-Civita connection
$\accentset{\circ}{\nabla}$ of $g$ which, in turn, allows us to build the
$1$-form connection

$\displaystyle\tensor{\accentset{\circ}{\omega}}{{}_{\mu}^{K}{}_{I}}:=e^{K}_{\alpha}\accentset{\circ}{\nabla}_{\\!\mu}E^{\alpha}_{I}\,.$
(3)

If we now consider a generic $1$-form connection
$\tensor{\widetilde{\omega}}{{}_{\mu}^{I}{}^{J}}$ (not necessarily
antisymmetric in the internal indices), we can define the covariant derivative

$\displaystyle\widetilde{\nabla}_{\\!\mu}Y^{\beta}=(\mathrm{d}Y^{J})_{\mu}E^{\beta}_{J}+E^{\beta}_{K}\tensor{\widetilde{\omega}}{{}_{\mu}^{K}{}_{J}}Y^{J}\,,\qquad
Y^{J}:=e^{J}_{\gamma}Y^{\gamma}\,,$

and extend it in the usual way to objects with more space-time and internal
indices. In particular, using this definition with $E^{\beta}_{I}$ leads to

$\tensor{\widetilde{\omega}}{{}_{\mu}^{K}{}_{I}}=e^{K}_{\alpha}\widetilde{\nabla}_{\\!\mu}E^{\alpha}_{I}\,,$
(4)

which is analogous to (3). We also have the $\widetilde{\omega}$-curvature and
the $\widetilde{\omega}$-covariant derivative over forms given by

$\displaystyle\widetilde{F}_{IJ}=\mathrm{d}\widetilde{\omega}_{IJ}+\widetilde{\omega}_{IK}\wedge\tensor{\widetilde{\omega}}{{}^{K}_{J}}\,,$
$\displaystyle\widetilde{\mathcal{D}}\alpha_{I}=\mathrm{d}\alpha_{I}+\tensor{\widetilde{\omega}}{{}_{I}^{J}}\wedge\alpha_{J}\,,$

where the latter is extended, as usual, to objects with more antisymmetric
internal indices.

Given two 1-form connections $\omega$ and $\widetilde{\omega}$, we have two
covariant derivatives $\nabla$ and $\widetilde{\nabla}$ which are related by a
$(2,1)$-tensor $Q$. In particular, if we take
$\nabla=\accentset{\circ}{\nabla}$, then

$\tensor{Q}{{}^{\beta}_{\mu}{}_{\alpha}}=E^{\beta}_{K}e^{J}_{\alpha}\big{(}\tensor{\widetilde{\omega}}{{}_{\mu}^{K}{}_{J}}-\tensor{\accentset{\circ}{\omega}}{{}_{\mu}^{K}{}_{J}}\big{)}=:\tensor{\varphi(e,\widetilde{\omega})}{{}^{\beta}_{\mu}{}_{\alpha}}\,.$

This allows us to define the map

$\Phi_{\mathrm{PT}}(e,\widetilde{\omega}):=\big{(}\Phi_{\mathrm{GR}}(e),\varphi(e,\widetilde{\omega})\big{)}$

which is surjective but not injective. In fact
$\Phi_{\mathrm{PT}}(e,\widetilde{\omega})=\Phi_{\mathrm{PT}}(e^{\prime},\widetilde{\omega}^{\prime})$
if and only if there exists some $\Psi\in SO(1,3)$ such that
$e^{\prime}_{I}=\tensor{\Psi}{{}_{I}^{J}}e_{J}$ and
$\widetilde{\omega}_{I}^{{}^{\prime}\,J}=\tensor{\Psi}{{}_{I}^{K}}\mathrm{d}\tensor{\Psi}{{}^{J}_{K}}+\tensor{\Psi}{{}_{I}^{K}}\tensor{\Psi}{{}^{J}_{L}}\widetilde{\omega}\tensor{}{{}_{K}^{L}}$.

###  3.1 The action

In order to introduce the tetrad formalism, we perform the “change of
variables” given by $\Phi_{\mathrm{PT}}$

$\displaystyle
L^{(t)}_{\mathrm{PT}}(e,\widetilde{\omega}):=L^{(m)}_{\mathrm{PT}}\circ\Phi_{\mathrm{PT}}(e,\widetilde{\omega})\,,$
$\displaystyle\overline{\ell}^{(t)}_{\mathrm{PT}}(e,\widetilde{\omega}):=\overline{\ell}^{(m)}_{\mathrm{PT}}\circ\Phi_{\mathrm{PT}}(e,\widetilde{\omega})\,.$

We could write again these Lagrangians as the GR part plus a coupling term,
but in this case it is more convenient to split the $1$-form connection in its
antisymmetric and symmetric parts (in its internal indices)

$\widetilde{\omega}_{IJ}=\widehat{\omega}_{IJ}+S_{IJ}\,,$

and consider the equivalent variables $(e,\widehat{\omega},S)$ instead.
Following the ideas of [23], one obtains the following expressions for these
Lagrangians

$\displaystyle L^{(t)}_{\mathrm{PT}}(e,\widehat{\omega},S)$
$\displaystyle=\frac{1}{2}\epsilon_{IJKL}\Big{(}\widehat{F}^{IJ}-\frac{\Lambda}{6}e^{I}\wedge
e^{J}+\tensor{S}{{}^{I}_{M}}\wedge S^{MJ}\Big{)}\wedge e^{K}\wedge e^{L}\,,$
$\displaystyle\overline{\ell}^{(t)}_{\mathrm{PT}}(e,\widehat{\omega})$
$\displaystyle=-\frac{1}{2}\epsilon_{IJKL}\left(2N^{I}\mathrm{d}N^{J}-\overline{\widehat{\omega}}{}^{IJ}\right)\wedge\overline{e}^{K}\wedge\overline{e}^{L}\,,$

where $\overline{e}^{I}:=\jmath^{*}e^{I}$,
$\overline{\widehat{\omega}}{}^{IJ}:=\jmath^{*}\widehat{\omega}^{IJ}$ and
$N^{I}=\nu^{\alpha}e^{I}_{\alpha}$. Notice that
$\overline{\ell}^{(t)}_{\mathrm{PT}}$ does not depend on $S$.

###  3.2 Variations

Computing the variations, one easily obtains

$\displaystyle\mathchoice{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{4.84373pt}\raisebox{0.0pt}{\kern
0.17221pt\rotatebox{90.0}{\kern-0.17221pt\makebox[4.84373pt][l]{}}\kern
0.17221pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{3.44444pt}\raisebox{0.0pt}{\kern
0.12915pt\rotatebox{90.0}{\kern-0.12915pt\makebox[3.44444pt][l]{}}\kern
0.12915pt}\hskip
0.6458pt}L^{(t)}_{\mathrm{PT}}=\mathfrak{E}^{(t)}_{L}\wedge\mathchoice{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{4.84373pt}\raisebox{0.0pt}{\kern
0.17221pt\rotatebox{90.0}{\kern-0.17221pt\makebox[4.84373pt][l]{}}\kern
0.17221pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{3.44444pt}\raisebox{0.0pt}{\kern
0.12915pt\rotatebox{90.0}{\kern-0.12915pt\makebox[3.44444pt][l]{}}\kern
0.12915pt}\hskip
0.6458pt}e^{L}+\mathcal{E}^{(t)}_{KL}\wedge\mathchoice{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{4.84373pt}\raisebox{0.0pt}{\kern
0.17221pt\rotatebox{90.0}{\kern-0.17221pt\makebox[4.84373pt][l]{}}\kern
0.17221pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{3.44444pt}\raisebox{0.0pt}{\kern
0.12915pt\rotatebox{90.0}{\kern-0.12915pt\makebox[3.44444pt][l]{}}\kern
0.12915pt}\hskip
0.6458pt}\widehat{\omega}^{KL}+\mathscr{E}^{(t)}_{JM}\wedge\mathchoice{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{4.84373pt}\raisebox{0.0pt}{\kern
0.17221pt\rotatebox{90.0}{\kern-0.17221pt\makebox[4.84373pt][l]{}}\kern
0.17221pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{3.44444pt}\raisebox{0.0pt}{\kern
0.12915pt\rotatebox{90.0}{\kern-0.12915pt\makebox[3.44444pt][l]{}}\kern
0.12915pt}\hskip 0.6458pt}S^{JM}+\mathrm{d}\Theta^{(t)}_{\mathrm{PT}}\,,$
$\displaystyle\mathchoice{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{4.84373pt}\raisebox{0.0pt}{\kern
0.17221pt\rotatebox{90.0}{\kern-0.17221pt\makebox[4.84373pt][l]{}}\kern
0.17221pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{3.44444pt}\raisebox{0.0pt}{\kern
0.12915pt\rotatebox{90.0}{\kern-0.12915pt\makebox[3.44444pt][l]{}}\kern
0.12915pt}\hskip
0.6458pt}\overline{\ell}^{(t)}_{\mathrm{PT}}-\jmath^{*}\Theta^{(t)}_{\mathrm{PT}}=\overline{b}^{(t)}_{I}\wedge\mathchoice{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{4.84373pt}\raisebox{0.0pt}{\kern
0.17221pt\rotatebox{90.0}{\kern-0.17221pt\makebox[4.84373pt][l]{}}\kern
0.17221pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{3.44444pt}\raisebox{0.0pt}{\kern
0.12915pt\rotatebox{90.0}{\kern-0.12915pt\makebox[3.44444pt][l]{}}\kern
0.12915pt}\hskip
0.6458pt}\overline{e}^{I}-\mathrm{d}\overline{\theta}^{(t)}_{\mathrm{PT}}\,,$

where the Euler-Lagrange equations are

$\displaystyle\mathfrak{E}^{(t)}_{L}$
$\displaystyle:=\epsilon_{IJKL}\Big{(}\widehat{F}^{IJ}+\tensor{S}{{}^{I}_{M}}\wedge
S^{MJ}-\frac{\Lambda}{3}e^{I}\wedge e^{J}\Big{)}\wedge e^{K}\,,$
$\displaystyle\mathcal{E}^{(t)}_{KL}$
$\displaystyle:=\frac{1}{2}\widehat{D}(\epsilon_{IJKL}e^{I}\wedge e^{J})\,,$
$\displaystyle\mathscr{E}^{(t)}_{JM}$
$\displaystyle:=\frac{1}{2}\big{(}\epsilon_{IKLJ}\delta_{M}^{R}+\epsilon_{IKLM}\delta_{J}^{R}\big{)}\tensor{S}{{}_{R}^{I}}\wedge
e^{K}\wedge e^{L}\,,$ $\displaystyle\overline{b}^{(t)}_{I}$
$\displaystyle:=\epsilon_{IJKL}(2N^{K}\mathrm{d}N^{L}-\overline{\widehat{\omega}}{}^{KL})\wedge\overline{e}^{J}+2\epsilon_{MJKL}N^{L}(\iota_{\overline{E}^{J}}\mathrm{d}\overline{e}^{K})\wedge\overline{e}^{M}N_{I}\,,$

and we take the symplectic potentials

$\displaystyle\Theta^{(t)}_{\mathrm{PT}}:=\frac{1}{2}\epsilon_{IJKL}e^{I}\wedge
e^{J}\wedge\mathchoice{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{4.84373pt}\raisebox{0.0pt}{\kern
0.17221pt\rotatebox{90.0}{\kern-0.17221pt\makebox[4.84373pt][l]{}}\kern
0.17221pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{3.44444pt}\raisebox{0.0pt}{\kern
0.12915pt\rotatebox{90.0}{\kern-0.12915pt\makebox[3.44444pt][l]{}}\kern
0.12915pt}\hskip 0.6458pt}\widehat{\omega}^{KL}\,,$
$\displaystyle\overline{\theta}^{(t)}_{\mathrm{PT}}:=\epsilon_{IJKL}\overline{e}^{I}\wedge\overline{e}^{J}\wedge
N^{K}\mathchoice{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{4.84373pt}\raisebox{0.0pt}{\kern
0.17221pt\rotatebox{90.0}{\kern-0.17221pt\makebox[4.84373pt][l]{}}\kern
0.17221pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{3.44444pt}\raisebox{0.0pt}{\kern
0.12915pt\rotatebox{90.0}{\kern-0.12915pt\makebox[3.44444pt][l]{}}\kern
0.12915pt}\hskip 0.6458pt}N^{L}\,.$ (5)

Notice that the boundary term $\overline{b}_{I}$ plays no role in the
Dirichlet case as in the variational principle one has
$\mathchoice{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{6.91037pt}\raisebox{0.0pt}{\kern
0.1873pt\rotatebox{90.0}{\kern-0.1873pt\makebox[6.91037pt][l]{}}\kern
0.1873pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{4.84373pt}\raisebox{0.0pt}{\kern
0.17221pt\rotatebox{90.0}{\kern-0.17221pt\makebox[4.84373pt][l]{}}\kern
0.17221pt}\hskip
0.6458pt}{\mathbbm{d}\rule[0.0pt]{0.0pt}{3.44444pt}\raisebox{0.0pt}{\kern
0.12915pt\rotatebox{90.0}{\kern-0.12915pt\makebox[3.44444pt][l]{}}\kern
0.12915pt}\hskip 0.6458pt}\overline{e}^{I}=0$, see [22] for a careful
discussion.

###  3.3 Space of solutions

As in the metric case, we can exactly solve some of the equations. We begin by
expanding $\tensor{S}{{}_{I}{}_{J}}=\tensor{S}{{}_{I}{}_{J}{}_{K}}e^{K}$ and
using the unique decomposition
$\tensor{S}{{}_{I}{}_{J}{}_{K}}=\not{S}_{IJK}+\eta_{IJ}U_{K}$ with
$\not{S}\tensor{}{{}^{I}_{I}{}_{K}}=0$. Plugging this into
$\mathscr{E}^{(t)}=0$ we get

$\displaystyle\tensor{\not{S}}{{}_{R}{}_{L}{}_{I}}+\tensor{\not{S}}{{}_{I}{}_{L}{}_{R}}-\not{S}\tensor{}{{}_{R}{}_{J}^{J}}\eta_{LI}-\not{S}\tensor{}{{}_{I}{}_{J}^{J}}\eta_{LR}=0\,,$

and taking its trace shows that $\not{S}\tensor{}{{}_{I}{}_{J}^{J}}=0$ which,
in turn, implies $\not{S}_{IJK}=0$. The general solution to
$\mathscr{E}^{(t)}=0$ is then $S_{IJK}=\eta_{IJ}U_{K}$, with arbitrary
$U_{\\!K}$. Similarly, from $\mathcal{E}^{(t)}=0$ we obtain the solution
$\widehat{\omega}_{IJ}=\accentset{\circ}{\omega}_{IJ}$. Thus

$\mathrm{Sol}^{(t)}_{\mathrm{PT}}=\\{(e^{I}_{\alpha},\accentset{\circ}{\omega}_{\mu}^{IJ}+\eta^{IJ}U_{\\!\mu})\
/\ e^{I}_{\alpha}\in\mathrm{Sol}^{(t)}_{\mathrm{GR}},\ U_{\\!\mu}\text{
arbitrary}\\}$

We see, again, that the tetrad sector of Palatini is equivalent to tetrad-GR
and that the boundary plays no role in the connection part. As in the metric
case, the Dirichlet or Neumann boundary conditions for the tetrads are
incorporated in $\mathrm{Sol}^{(t)}_{\mathrm{GR}}$, which is studied in detail
in [23].

###  3.4 Presymplectic form

It is easy to see that, over the space of solutions, both symplectic
potentials on (5) are given by the same expressions as the symplectic
potentials obtained in [23] for tetrad GR (remember, though, that they live in
different spaces). In fact, using the projection
$\pi_{(t)}(e,\widetilde{\omega})=e$ we have

$\includegraphics[height=6.83331pt]{./OOmega.pdf}^{\mathrm{PT}}_{(t)}=\pi_{(t)}^{*}\includegraphics[height=6.83331pt]{./OOmega.pdf}^{\mathrm{GR}}_{(t)}$
(6)

in analogy with equation (2). We conclude as well that the vectors of the form
$(0,\mathbb{W})\in T\mathrm{Sol}^{(t)}_{\mathrm{PT}}$ correspond to degenerate
directions of
$\includegraphics[height=6.83331pt]{./OOmega.pdf}^{\mathrm{PT}}_{(t)}$.

##  4 CONCLUSIONS AND COMMENTS

In this letter, we have studied Palatini gravity, both in the metric and
tetrad formulations, in a manifold with boundary, and including nonmetricity
and torsion. As can be seen in (2) and (6), the presymplectic structures
defined on the space of solutions of metric and tetrad Palatini are related to
those of metric and tetrad GR by the projection on the first factor. Also, the
solution spaces of metric and tetrad Palatini are related to each other by
$\Phi_{\mathrm{PT}}$. Finally, since we have from [23] that

$\includegraphics[height=6.83331pt]{./OOmega.pdf}^{\mathrm{GR}}_{(t)}=\Phi^{*}_{\mathrm{GR}}\includegraphics[height=6.83331pt]{./OOmega.pdf}^{\mathrm{GR}}_{(m)}$
(7)

it is immediate to obtain, using
$\pi_{(m)}\circ\Phi_{\mathrm{PT}}=\Phi_{\mathrm{GR}}\circ\pi_{(t)}$, the
following important relation

$\includegraphics[height=6.83331pt]{./OOmega.pdf}^{\mathrm{PT}}_{(t)}=\Phi^{*}_{\mathrm{PT}}\includegraphics[height=6.83331pt]{./OOmega.pdf}^{\mathrm{PT}}_{(m)}$
(8)

This is one of the main results of the paper: the equivalence, up to the
internal gauge transformations given by the kernel of
$(\Phi_{\mathrm{PT}})_{*}$, of metric and tetrad Palatini. Furthermore,
equations (2), (6), (7), and (8) show the equivalence (up to the gauge
transformations given by the kernel of the corresponding push-forwards) of all
these four formulations of gravity.

${\big{(}\includegraphics[height=6.83331pt]{./OOmega.pdf}_{(t)}^{\mathrm{PT}},\mathrm{Sol}_{(t)}^{\mathrm{PT}}\big{)}}$${\big{(}\includegraphics[height=6.83331pt]{./OOmega.pdf}_{(m)}^{\mathrm{PT}},\mathrm{Sol}_{(m)}^{\mathrm{PT}}\big{)}}$${\big{(}\includegraphics[height=6.83331pt]{./OOmega.pdf}_{(t)}^{\mathrm{GR}},\mathrm{Sol}_{(t)}^{\mathrm{GR}}\big{)}}$${\big{(}\includegraphics[height=6.83331pt]{./OOmega.pdf}_{(m)}^{\mathrm{GR}},\mathrm{Sol}_{(m)}^{\mathrm{GR}}\big{)}}$$\pi_{(t)}$$\Phi_{\mathrm{PT}}$$\Phi_{\mathrm{GR}}$$\pi_{(m)}$
(9)

This has been proven for both Dirichlet and homogeneous Neumann boundary
conditions, although the same result can be obtained in other instances as
long as all four formulations are related in the way shown in the diagram
(hence with the appropriate boundary Lagrangians, [23]). Notice that this may
also be seen by realizing that the projection of the spaces of solutions over
the metric/tetrad sector is well defined (in the sense that it is not
necessary to know the second factor to get the first). On the other hand, the
main result regarding the connection sector is that boundaries have no
influence whatsoever (once the correct boundary Lagrangian is considered). We
have also found the well-known projective symmetry in both metric and tetrad
formulations, which can be expressed, respectively, in the form

$\tensor{Q}{{}^{\alpha}_{\beta}{}_{\gamma}}=\delta^{\alpha}_{\gamma}U_{\\!\beta}\,,\qquad\qquad
S^{IJ}_{\beta}=\eta^{IJ}U_{\\!\beta}$ (10)

for arbitrary $U_{\\!\beta}$. This projective symmetry plays an important role
to define observables as they must be $U$-independent. For instance, the
$g$-Kretschmann curvature
$\mathrm{\accentset{\circ}{K}res}:=\mathrm{\accentset{\circ}{R}iem}_{\alpha\beta\gamma\delta}\mathrm{\accentset{\circ}{R}iem}{}^{\alpha\beta\gamma\delta}$
leads to an observable for every $(g,Q)$ as, in fact, it is independent of
$Q$. However, the $(g,Q)$-Kretschmann
$\widetilde{\mathrm{K}}\mathrm{res}:=\mathrm{\widetilde{R}iem}_{\alpha\beta\gamma\delta}\mathrm{\widetilde{R}iem}{}^{\alpha\beta\gamma\delta}$
cannot provide an observable. Indeed, using (1), it is easy to show that on
the space of solutions we have

$\widetilde{\mathrm{K}}\mathrm{res}=\mathrm{\accentset{\circ}{K}res}+(\mathrm{d}U)_{\alpha\beta}(\mathrm{d}U)^{\alpha\beta}$

which depends on $U$. This is important if one plans to use these geometric
objects as indicators of the presence of singularities [24].

Finally, it is relevant to mention that although we have proven the
equivalence of the four theories in vacuum, this equivalence may be broken by
matter fields. For instance, pregeodesics computed by using the connection are
insensitive to the $U$-arbitrariness [25]. However, if matter is introduced,
this may no longer be the case. Since we consider Palatini theories where both
nonmetricity and torsion are allowed to be non-zero, matter fields can be
coupled in ways that are not possible in standard GR. This may produce
interesting changes in the dynamics of the gravity-matter system.

###### Acknowledgements.

The authors are grateful to G. Olmo for helpful discussions. This work has
been supported by the Spanish Ministerio de Ciencia Innovación y
Universidades-Agencia Estatal de Investigación FIS2017-84440-C2-2-P grant.
Juan Margalef-Bentabol is supported by the Eberly Research Funds of Penn
State, by the NSF grant PHY-1806356 and by the Urania Stott fund of Pittsburgh
foundation UN2017-92945. E.J.S. Villaseñor is supported by the Madrid
Government (Comunidad de Madrid-Spain) under the Multiannual Agreement with
UC3M in the line of Excellence of University Professors (EPUC3M23), and in the
context of the V PRICIT (Regional Programme of Research and Technological
Innovation).

## References

  * [1] A. Ashtekar, M. Campiglia and A. Laddha, _Null infinity, the BMS group and infrared issues_ , _Gen. Rel. Grav._ 50 (2018) 140 [arXiv:1808.07093].
  * [2] A. Ashtekar and B. Krishnan, _Isolated and dynamical horizons and their applications_ , _Living Rev. Rel._ 7 (2004) 10 [arXiv:gr-qc/0407042].
  * [3] W. Wieland, _Null infinity as an open Hamiltonian system_ , _JHEP_ 04 (2021) 095 [arXiv:2012.01889].
  * [4] A. Ashtekar and J. Lewandowski, _Background independent quantum gravity: A Status report_ , _Class. Quant. Grav._ 21 (2004) R53 [arXiv:gr-qc/0404018].
  * [5] J.F. Barbero G., B. Díaz, J. Margalef-Bentabol and E.J.S. Villaseñor, _Concise symplectic formulation for tetrad gravity_ , _Phys. Rev. D_ 103 (2021) 024051 [arXiv:2011.00661].
  * [6] F.W. Hehl, P. von der Heyde and G.D. Kerlick, _General relativity with spin and torsion: Foundations and prospects_ , _Rev. Mod. Phys._ 48 (1976) 393.
  * [7] F.W. Hehl, J.D. McCrea, E.W. Mielke and Y. Ne’eman, _Metric affine gauge theory of gravity: Field equations, Noether identities, world spinors, and breaking of dilation invariance_ , _Phys. Rept._ 258 (1995) 1 [arXiv:gr-qc/9402012].
  * [8] G.J. Olmo, _Palatini Approach to Modified Gravity: f(R) Theories and Beyond_ , _Int. J. Mod. Phys. D_ 20 (2011) 413 [arXiv:1101.3864].
  * [9] V.I. Afonso, C. Bejarano, J. Beltran Jimenez, G.J. Olmo and E. Orazi, _The trivial role of torsion in projective invariant theories of gravity with non-minimally coupled matter fields_ , _Class. Quant. Grav._ 34 (2017) 235003 [arXiv:1705.03806].
  * [10] B. Julia and S. Silva, _Currents and superpotentials in classical gauge invariant theories. 1. Local results with applications to perfect fluids and general relativity_ , _Class. Quant. Grav._ 15 (1998) 2173 [arXiv:gr-qc/9804029].
  * [11] K. Banerjee, _Some aspects of Holst and Nieh-Yan terms in general relativity with torsion_ , _Class. Quant. Grav._ 27 (2010) 135012 [arXiv:1002.0669].
  * [12] N. Dadhich and J. M. Pons, _On the equivalence of the Einstein-Hilbert and the Einstein-Palatini formulations of general relativity for an arbitrary connection_ , _Gen. Rel. Grav._ 44 (2012) 2337 [arXiv:1010.0869].
  * [13] G. Barnich, P. Mao and R. Ruzziconi, _Conserved currents in the Cartan formulation of general relativity_ , _About Various Kinds of Interactions: Workshop in honour of Professor Philippe Spindel_ (2016)  [arXiv:1611.01777].
  * [14] A. Corichi, I. Rubalcava-García and T. Vukašinac, _Actions, topological terms and boundaries in first-order gravity: A review_ , _Int. J. Mod. Phys. D_ 25 (2016) 1630011 [arXiv:1604.07764].
  * [15] A. Delhom-Latorre, G. J.Olmo and M. Ronco, _Observable traces of non-metricity: New constraints on metric-affine gravity_ , _Phys. Lett. B_ 780 (2018) 294 [arXiv:1709.04249].
  * [16] S. Chakraborty and R. Dey, _Noether current, black hole entropy and spacetime torsion_ , _Phys. Lett. B_ 786 (2018) 432 [arXiv:1806.05840].
  * [17] E. De Paoli and S. Speziale, _A gauge-invariant symplectic potential for tetrad general relativity_ , _JHEP_ 07 (2018) 040 [arXiv:1804.09685].
  * [18] R. Oliveri and S. Speziale, _Boundary effects in general relativity with tetrad variables_ , _Gen. Rel. Grav._ 52 (2020) 83 [arXiv:1912.01016].
  * [19] R. Oliveri and S. Speziale, _A note on dual gravitational charges_ , _JHEP_ 12 (2020) 079 [arXiv:2010.01111].
  * [20] G. Barnich, P. Mao and R. Ruzziconi, _Conserved currents in the Palatini formulation of general relativity_ , _PoS_ CORFU2019 (2020) 171 [arXiv:2004.15002].
  * [21] Y.N. Obukhov, _The Palatini principle for a manifold with boundary_ , _Class. Quant. Grav._ 4 (1987) 1085.
  * [22] J. Margalef-Bentabol and E.J.S. Villaseñor, _Geometric formulation of the covariant phase space methods with boundaries_ , _Phys. Rev. D_ 103 (2021) 025011 [arXiv:2008.01842].
  * [23] J.F. Barbero G., J. Margalef-Bentabol, V. Varo and E.J.S. Villaseñor, “Covariant phase space for gravity with boundaries: Metric versus tetrad formulations.” 2021.  [arXiv:2103.06362].
  * [24] C. Bejarano, A. Delhom, A. Jiménez-Cano, G. J. Olmo and D. Rubiera-Garcia, _Geometric inequivalence of metric and Palatini formulations of general relativity_ , _Phys. Lett. B_ 802 (2020) 135275 [arXiv:1907.04137].
  * [25] A.N. Bernal, B. Janssen, A. Jimenez-Cano, J.A. Orejuela, M. Sanchez and P. Sanchez-Moreno, _On the (non-)uniqueness of the Levi-Civita solution in the Einstein-Hilbert-Palatini formalism_ , _Phys. Lett. B_ 768 (2017) 280 [arXiv:1606.08756].


# Generative Modelling for Controllable Audio Synthesis of
Expressive Piano Performance

Hao Hao Tan  Yin-Jyun Luo  Dorien Herremans

###### Abstract

We present a controllable neural audio synthesizer based on Gaussian Mixture
Variational Autoencoders (GM-VAE), which can generate realistic piano
performances in the audio domain that closely follows temporal conditions of
two essential style features for piano performances: articulation and
dynamics. We demonstrate how the model is able to apply fine-grained style
morphing over the course of synthesizing the audio. This is based on
conditions which are latent variables that can be sampled from the prior or
inferred from other pieces. One of the envisioned use cases is to inspire
creative and brand new interpretations for existing pieces of piano music.

Machine Learning, ICML

##  1 Introduction

Synthesizing audio of piano performances from MIDI requires either a huge
collection of recordings of individual notes, or a model that simulates an
actual piano. These approaches, however, have two main limitations: (i) the
“stitching” of individual notes might not optimally capture the various
interactions between notes (Hawthorne et al., 2019); and (ii) the quality of
the synthesized audio is restricted by the recordings in the sound library or
the piano simulator. This further motivates the effort to build realistic
piano synthesizers. In this work, we propose a neural network based
synthesizer which takes the onset roll as input, and generates realistic piano
performances in the audio domain. Our model takes the onset roll instead of
the complete piano roll as input, thereby waiving the necessity of fine-
grained frame and velocity information, which are often unavailable, to
synthesize expressive piano performances. Without the constraint of frame and
velocity information, performance style transfer could also be achieved by
interpreting each onset note with the style inferred from a given audio
performance. On top of the onset roll, the generation is further conditioned
on variables corresponding to style features, which enable the rendition of
expressive piano performances. In particular, we consider articulation and
dynamics, which are two significant features related to expressiveness in
piano performances, and base our framework on variational autoencoders (VAEs)
(Kingma & Welling, 2014) with a Gaussian mixture prior (Jiang et al., 2016).

The underlying task of this work is to build a neural audio synthesizer that
maps piano performances in MIDI to audio. The current literature either
focuses on such direct mapping without additional controllability with regard
to performance styles (Hawthorne et al., 2019; Thakkar Vijay Manzelli, 2018),
or concerns only the global attributes such as synthesizing different
instruments (Kim et al., 2019). We distinguish ourselves from the previous
works by incorporating GM-VAE (Jiang et al., 2016), which has been applied to
speech (Hsu et al., 2019) and instrument modelling (Luo et al., 2019), to
disentangle two significant expressive performance factors. This allows us to
achieve creative applications such as gradual style morphing over time. The
sequence of conditions can be sampled from the prior or inferred from other
pieces, as shown in Section 3.

##  2 Experimental Details

Figure 1: Model architecture.

Data Representation: We use the MAESTRO v2.0.0 dataset (Hawthorne et al.,
2019) which consists of 1,282 performances with aligned audio-MIDI pairs, and
we split the data as annotated. We train on 20-second random crops from audio
clips, which are converted into a log-scale Mel-spectrogram with 80 Mel-
filters ($\textbf{X}^{\scriptsize{\textrm{}}}$). The MIDI note sequences are
represented as piano rolls, from which the onset roll
($\textbf{Y}^{\scriptsize{\textrm{onset}}}$) is extracted. We represent both
articulation (staccato, legato) and dynamics (loud, soft) as binary sequences
$\textbf{c}^{\scriptsize{\textrm{art}}}_{1...T}$,
$\textbf{c}^{\scriptsize{\textrm{dyn}}}_{1...T}\in\\{0,1\\}$, each of which
corresponds to two components of a Gaussian mixture.
$c^{\scriptsize{\textrm{art}}}_{t}=1$ if at least one note is held at $t$; and
$c^{\scriptsize{\textrm{dyn}}}_{t}=1$ if the average velocity across all notes
is greater than 70 at $t$, which the threshold is determined by our
preliminary data analysis.

Model Formulation: Figure 1 shows our model architecture, which is adapted
from (Hsu et al., 2019). The joint distribution
$p_{\theta}(\textbf{X}^{\scriptsize{\textrm{}}},\textbf{z}^{\scriptsize{\textrm{art}}},\textbf{z}^{\scriptsize{\textrm{dyn}}}|\textbf{Y}^{\scriptsize{\textrm{onset}}},\textbf{c}^{\scriptsize{\textrm{art}}},\textbf{c}^{\scriptsize{\textrm{dyn}}})=p_{\theta}(\textbf{X}^{\scriptsize{\textrm{}}}|\textbf{Y}^{\scriptsize{\textrm{onset}}},\textbf{z}^{\scriptsize{\textrm{art}}},\textbf{z}^{\scriptsize{\textrm{dyn}}})p_{\theta}(\textbf{z}^{\scriptsize{\textrm{art}}}|\textbf{c}^{\scriptsize{\textrm{art}}})p_{\theta}(\textbf{z}^{\scriptsize{\textrm{dyn}}}|\textbf{c}^{\scriptsize{\textrm{dyn}}})$,
where $\theta$ refers to parameters of the generation network. Both the prior
distributions $p_{\theta}(\textbf{z}^{\scriptsize{\textrm{art}}})$ and
$p_{\theta}(\textbf{z}^{\scriptsize{\textrm{dyn}}})$ are Gaussian mixtures of
two components. A variational distribution
$q_{\phi}(\textbf{z}^{\scriptsize{\textrm{}}}|\textbf{X}^{\scriptsize{\textrm{}}})$
is introduced to approximate the true posterior, where $\phi$ refers to
parameters of the inference network. The model is trained to optimize the
evidence lower bound:

$\begin{split}\mathcal{L}(\theta,\phi;\textbf{X}^{\scriptsize{\textrm{}}})&=\mathbb{E}_{q_{\phi}(\textbf{z}^{\scriptsize{\textrm{art}}}|\textbf{X}^{\scriptsize{\textrm{}}})q_{\phi}(\textbf{z}^{\scriptsize{\textrm{dyn}}}|\textbf{X}^{\scriptsize{\textrm{}}})}[\log
p_{\theta}(\textbf{X}^{\scriptsize{\textrm{}}}|\textbf{Y}^{\scriptsize{\textrm{onset}}},\textbf{z}^{\scriptsize{\textrm{art}}},\textbf{z}^{\scriptsize{\textrm{dyn}}})]\\\
-&\mathcal{D}_{\mathrm{KL}}\big{(}q_{\phi}(\textbf{z}^{\scriptsize{\textrm{art}}}|\textbf{X}^{\scriptsize{\textrm{}}})\|p(\textbf{z}^{\scriptsize{\textrm{art}}}|\textbf{c}^{\scriptsize{\textrm{art}}})\big{)}\\\
-&\mathcal{D}_{\mathrm{KL}}\big{(}q_{\phi}(\textbf{z}^{\scriptsize{\textrm{dyn}}}|\textbf{X}^{\scriptsize{\textrm{}}})\|p(\textbf{z}^{\scriptsize{\textrm{dyn}}}|\textbf{c}^{\scriptsize{\textrm{dyn}}})\big{)}\end{split}$
(1)

Both the generation and inference network are implemented with two-layer
bidirectional LSTMs111Source code: https://github.com/gudgud96/piano-
synthesis. Note that we simplify
$\textbf{c}^{\scriptsize{\textrm{art}}}_{1...T}$ and
$\textbf{c}^{\scriptsize{\textrm{dyn}}}_{1...T}$ as
$\textbf{c}^{\scriptsize{\textrm{art}}}$ and
$\textbf{c}^{\scriptsize{\textrm{dyn}}}$ (similarly for the latent variables
$\textbf{z}^{\scriptsize{\textrm{art}}}$ and
$\textbf{z}^{\scriptsize{\textrm{dyn}}}$). For each of
$\textbf{c}^{\scriptsize{\textrm{art}}}$ and
$\textbf{c}^{\scriptsize{\textrm{dyn}}}$, an additional cross-entropy loss is
introduced such that the posterior
$p(\textbf{c}^{\scriptsize{\textrm{}}}|\textbf{z}^{\scriptsize{\textrm{}}})$
can also learn from the labelled ground-truth
$\textbf{c}^{\scriptsize{\textrm{}}}$.

Audio Synthesis: We leverage WaveGlow (Prenger et al., 2019) to invert the
Mel-spectrogram to audio, due to its fast inference and superior performance
(Govalkar et al., ; Zhao et al., 2020). We adopt the implementation from (Yu,
2019).

##  3 Results and Discussion

Gradual Style Morphing Over Time: Utilizing the Gaussian mixture prior
distribution enables style morphing by linear interpolation between mixture
components. In particular, given $\mu_{0}^{\textrm{art}}$ and
$\mu_{1}^{\textrm{art}}$ representing the mean vectors of mixture components
corresponding to staccato and legato, we can set
$\textbf{z}^{\scriptsize{\textrm{art}}}_{t}=\mu_{0}^{\textrm{art}}+(\mu_{1}^{\textrm{art}}-\mu_{0}^{\textrm{art}})\times\frac{t}{T}$
in the latent space of articulation (similarly for
$\textbf{z}^{\scriptsize{\textrm{dyn}}}_{t}$). In other words, the sequence of
conditioning vectors travel from $\mu_{0}^{\textrm{art}}$ to
$\mu_{1}^{\textrm{art}}$, whereby we can expect the articulation of the
synthesized piano performance to morph gradually from staccato to legato.
Figure 2 demonstrates the generated Mel-spectrograms of four different
scenarios. From staccato to legato, one can observe that notes are gradually
sustained longer; and from soft to loud, the amplitude increases over time and
more mid-high frequencies are covered, which results in a higher level of
perceived energy in the human auditory system (Fletcher & Munson, 1933).

Figure 2: Generated Mel-spectrograms of a given piece, with combinations of
gradual morphing of articulation and dynamics. Figure 3: An example of
performance style transfer. The piano roll for the final output is estimated
using the state-of-the-art model for piano transcription (Hawthorne et al.,
2018).

Performance Style Transfer: In addition to sampling from the prior
distribution, we can also infer the sequence of conditioning vectors from
another piece of music. Specifically, we let
$\textbf{z}^{\scriptsize{\textrm{art}}}\sim
q_{\phi}(\textbf{z}^{\scriptsize{\textrm{art}}}|\textbf{X}^{\scriptsize{\textrm{style}}})$
(similarly for $\textbf{z}^{\scriptsize{\textrm{dyn}}}$), where
$\textbf{X}^{\scriptsize{\textrm{style}}}$ is the style piece that would
determine the fine-grained style over time of the synthesized piano
performance. Figure 3 shows an example that renders a piece from the Baroque
era (which is more detached and constant in terms of dynamics) in the style of
a piece from the Romantic era (which is more legato and expressive in terms of
dynamics). By observing both the Mel-spectrograms and piano rolls, one can see
that the final output closely follows the style features of the style piece in
terms of note duration (articulation) and amplitude (dynamics), while
preserving the musical content. Audio examples can be found online.222
https://piano-performance-synthesis.github.io

We envision that this framework could learn to achieve fine-grained control on
multiple performance style factors, which allows us to explore new performance
directions for any given piece, even by taking inspirations from other pieces
via style transfer. Future work will involve extending the set of performance
features (e.g. onset deviation, pedalling) in order to generate more realistic
piano performance.

## Acknowledgements

We would like to thank the anonymous reviewers for their constructive reviews.
This work is supported by MOE Tier 2 grant no. MOE2018-T2-2-161, SRG ISTD 2017
129, and Singapore International Graduate Award (SINGA) provided by the Agency
for Science, Technology and Research (A*STAR), under reference number
SING-2018-01-1270.

## References

  * Fletcher & Munson (1933) Fletcher, H. and Munson, W. A.  Loudness, its definition, measurement and calculation.  _Bell System Technical Journal_ , 12(4):377–430, 1933.
  * (2) Govalkar, P., Fischer, J., Zalkow, F., and Dittmar, C.  A comparison of recent neural vocoders for speech signal reconstruction.
  * Hawthorne et al. (2018) Hawthorne, C., Elsen, E., Song, J., Roberts, A., Simon, I., Raffel, C., Engel, J., Oore, S., and Eck, D.  Onsets and frames: Dual-objective piano transcription.  In _18th International Society for Music Information Retrieval Conference, ISMIR_ , 2018.
  * Hawthorne et al. (2019) Hawthorne, C., Stasyuk, A., Roberts, A., Simon, I., Huang, C.-Z. A., Dieleman, S., Elsen, E., Engel, J., and Eck, D.  Enabling factorized piano music modeling and generation with the maestro dataset.  In _International Conference of Learning Representations, ICLR_ , 2019\.
  * Hsu et al. (2019) Hsu, W.-N., Zhang, Y., Weiss, R. J., Zen, H., Wu, Y., Wang, Y., Cao, Y., Jia, Y., Chen, Z., Shen, J., et al.  Hierarchical generative modeling for controllable speech synthesis.  In _International Conference of Learning Representations, ICLR_ , 2019\.
  * Jiang et al. (2016) Jiang, Z., Zheng, Y., Tan, H., Tang, B., and Zhou, H.  Variational deep embedding: An unsupervised and generative approach to clustering.  _arXiv preprint arXiv:1611.05148_ , 2016.
  * Kim et al. (2019) Kim, J. W., Bittner, R., Kumar, A., and Bello, J. P.  Neural music synthesis for flexible timbre control.  In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 176–180. IEEE, 2019.
  * Kingma & Welling (2014) Kingma, D. P. and Welling, M.  Auto-encoding variational bayes.  In _International Conference of Learning Representations, ICLR_ , 2014\.
  * Luo et al. (2019) Luo, Y.-J., Agres, K., and Herremans, D.  Learning disentangled representations of timbre and pitch for musical instrument sounds using gaussian mixture variational autoencoders.  In _International Society for Music Information Retrieval Conference, ISMIR_ , 2019.
  * Prenger et al. (2019) Prenger, R., Valle, R., and Catanzaro, B.  Waveglow: A flow-based generative network for speech synthesis.  In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 3617–3621. IEEE, 2019.
  * Thakkar Vijay Manzelli (2018) Thakkar Vijay Manzelli, R.  Combining deep generative raw audio models for structured automatic music.  In _19th International Society for Music Information Retrieval Conference, ISMIR_ , 2018.
  * Yu (2019) Yu, C. Y.  Constant memory waveglow: A pytorch implementation of waveglow with constant memory cost.  https://github.com/yoyololicon/constant-memory-waveglow, 2019.
  * Zhao et al. (2020) Zhao, Y., Wang, X., Juvela, L., and Yamagishi, J.  Transferring neural speech waveform synthesizers to musical instrument sounds generation.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 6269–6273. IEEE, 2020.


# Run LoRA Run: Faster and Lighter LoRA Implementations

Daria Cherniuk
Center for Artifical Intelligence Technology
Skolkovo Institute of Science and Technology
Moscow, Russia
<EMAIL_ADDRESS>
Aleksandr Mikhalev
Center for Artifical Intelligence Technology
Skolkovo Institute of Science and Technology
Moscow, Russia
<EMAIL_ADDRESS>
Ivan Oseledets
Artificial Intelligence Research Institute
<EMAIL_ADDRESS>
Center for Artifical Intelligence Technology
Skolkovo Institute of Science and Technology
Moscow, Russia
<EMAIL_ADDRESS>

###### Abstract

LoRA is a technique that reduces the number of trainable parameters in a
neural network by introducing low-rank adapters to linear layers. This
technique is used both for fine-tuning (LoRA, QLoRA) and full train (ReLoRA).
This paper presents the RunLoRA framework for efficient implementations of
LoRA that significantly improves the speed of neural network training and
fine-tuning using low-rank adapters. The proposed implementation optimizes the
computation of LoRA operations based on dimensions of corresponding linear
layer, layer input dimensions and lora rank by choosing best forward and
backward computation graph based on FLOPs and time estimations, resulting in
faster training without sacrificing accuracy. The experimental results show up
to 17% speedup on Llama family of models.

_Keywords_ Low-rank $\cdot$ LoRA $\cdot$ Llama

##  1 Introduction

LoRA Hu et al. (2022) paper has introduced low-rank adapters to fine-tune
large LLMs on downstream tasks. This approach quickly became popular due to
reduced cost of the update. Different modifications of LoRA followed, for
example, QLoRA Dettmers et al. (2023) utilizes quantization and further
reduces fine-tuning costs, and ReLoRA Lialin et al. (2023) which showed that
low-rank updates can also be used for full training.

However, all variations of LoRA use the same chain of operations while
calculating the output, which often leads to sub-optimal graph of
computations. We propose RunLora: a framework which contains different
variations of forward and backward pass through an adapter-induced linear
layer and chooses the best pair for a given architecture.

We evaluated our framework’s performance on a series of Llama models and
achieved up to 17% speedup only due to optimized chain of PyTorch operations.
Additionally, we managed to save up to 4Gb of memory due to reduction in
number of saved activations.

##  2 Problem setting and Methodology

Default forward pass of LoRA looks the following:

$LoRA(X)=XW+(XA)B,$ (1)

The backward pass is then automatically determined by the framework from an
autograd technique. All the optimizations are left to the neural network
training framework, which often performs sub-optimally.

Many scientists and engineers avoid the following chain of computations:

$LoRA(X)=X(W+AB).$ (2)

This is due to an implicit assumption that weights $W$ are quite large and
forming a same-size matrix AB is undesirable. However, real-world LoRA-adapter
training deals with large input $X$ in an attempt of increasing batch size to
utilize GPU RAM at its full capacity. Large batch size leads to a
contradiction to the assumption and inefficient LoRA implementation.

Our current implementation contains 2 major and 2 minor variants of the
forward pass and 5 major and 30 minor variants of the backward pass. Major
variants are different by opening brackets differently, while minor variants
are based on major variants with operations reordered. Formally, the major
forward variants are:

  1. 1.

$Y=(XW)+((XA)B)$

  2. 2.

$Y=X(W+AB)$

Unlike default LoRA implementation, neither forward function in RunLoRA saves
result of $XA$ to context.

The backward of the LoRA adapter requires us to calculate the following
tensors:

$\left\\{\begin{array}[]{l}\mathrm{d}A=X^{\top}\mathrm{d}YB^{\top},\\\
\mathrm{d}B=A^{\top}X^{\top}\mathrm{d}Y,\\\
\mathrm{d}X=\mathrm{d}YW^{\top}+\mathrm{d}YB^{\top}A^{\top}.\end{array}\right.$
(3)

Due to associativity of matrix multiplications, there are several ways of
doing it and getting the same result up to rounding errors:

$ABC=(AB)C=A(BC).$ (4)

There are 3 multiplications and each can be done in 2 ways, which leads to 8
variants of the backward pass:

  1. 1.

$\mathrm{d}A=X^{\top}(\mathrm{d}YB^{\top}),\mathrm{d}B=(A^{\top}X^{\top})\mathrm{d}Y,\mathrm{d}X=\mathrm{d}YW^{\top}+(\mathrm{d}YB^{\top})A^{\top}.$

  2. 2.

$\mathrm{d}A=X^{\top}(\mathrm{d}YB^{\top}),\mathrm{d}B=A^{\top}(X^{\top}\mathrm{d}Y),\mathrm{d}X=\mathrm{d}YW^{\top}+(\mathrm{d}YB^{\top})A^{\top}.$

  3. 3.

$\mathrm{d}A=(X^{\top}\mathrm{d}Y)B^{\top},\mathrm{d}B=A^{\top}(X^{\top}\mathrm{d}Y),\mathrm{d}X=\mathrm{d}YW^{\top}+(\mathrm{d}YB^{\top})A^{\top}.$

  4. 4.

$\mathrm{d}A=(X^{\top}\mathrm{d}Y)B^{\top},\mathrm{d}B=A^{\top}(X^{\top}\mathrm{d}Y),\mathrm{d}X=\mathrm{d}Y(W^{\top}+B^{\top}A^{\top}).$

  5. 5.

$\mathrm{d}A=X^{\top}(\mathrm{d}YB^{\top}),\mathrm{d}B=(A^{\top}X^{\top})\mathrm{d}Y,\mathrm{d}X=\mathrm{d}Y(W^{\top}+B^{\top}A^{\top}).$

  6. 6.

$\mathrm{d}A=(X^{\top}\mathrm{d}Y)B^{\top},\mathrm{d}B=(A^{\top}X^{\top})\mathrm{d}Y,\mathrm{d}X=\mathrm{d}YW^{\top}+(\mathrm{d}YB^{\top})A^{\top}.$

  7. 7.

$\mathrm{d}A=(X^{\top}\mathrm{d}Y)B^{\top},\mathrm{d}B=(A^{\top}X^{\top})\mathrm{d}Y,\mathrm{d}X=\mathrm{d}Y(W^{\top}+B^{\top}A^{\top}).$

  8. 8.

$\mathrm{d}A=X^{\top}(\mathrm{d}YB^{\top}),\mathrm{d}B=A^{\top}(X^{\top}\mathrm{d}Y),\mathrm{d}X=\mathrm{d}Y(W^{\top}+B^{\top}A^{\top}).$

However, out of 8 variants we implement only 5 as we assume others to require
more or equal number of FLOPs in any setting. For example, backward6 would
require more flops than backward5 for any architecture and training
configuration (Table 1).

  1. 1.

backward1, equivalent to the variant 1 above:

$Z_{1}=\mathrm{d}YB^{\top},Z_{2}=XA,\mathrm{d}A=X^{\top}Z_{1},\mathrm{d}B=Z_{2}^{\top}\mathrm{d}Y,\mathrm{d}X=\mathrm{d}YW^{\top}+Z_{1}A^{\top},$

  2. 2.

backward2, equivalent to the variant 2 above:

$Z_{1}=\mathrm{d}YB^{\top},Z_{2}=X^{\top}\mathrm{d}Y,\mathrm{d}A=X^{\top}Z_{1},\mathrm{d}B=A^{\top}Z_{2},\mathrm{d}X=\mathrm{d}YW^{\top}+Z_{1}A^{\top},$

  3. 3.

backward3, equivalent to the variant 3 above:

$Z_{1}=\mathrm{d}YB^{\top},Z_{2}=X^{\top}\mathrm{d}Y,\mathrm{d}A=Z_{2}B^{\top},\mathrm{d}B=A^{\top}Z_{2},\mathrm{d}X=\mathrm{d}YW^{\top}+Z_{1}A^{\top},$

  4. 4.

backward4, equivalent to the variant 4 above:

$Z_{1}=W+AB,Z_{2}=X^{\top}\mathrm{d}Y,\mathrm{d}A=Z_{2}B^{\top},\mathrm{d}B=A^{\top}Z_{2},\mathrm{d}X=\mathrm{d}YZ_{1}^{\top},$

  5. 5.

backward5, equivalent to the variant 5 above:

$Z_{1}=\mathrm{d}YB^{\top},Z_{2}=XA,Z_{3}=W+AB,\mathrm{d}A=X^{\top}Z_{1},\mathrm{d}B=Z_{2}^{\top}\mathrm{d}Y,\mathrm{d}X=\mathrm{d}YZ_{3}^{\top}.$

Method | FLOPs
---|---
forward1 | $2b\cdot s\cdot(i\cdot o+r\cdot i+o\cdot r)$
forward2 | $2(i\cdot o\cdot r+b\cdot s\cdot o\cdot i)$
forward3 | $2b\cdot s\cdot(i\cdot o+r\cdot i+o\cdot r)$
forward4 | $2(i\cdot o\cdot r+b\cdot s\cdot o\cdot i)$
backward1 | $2b\cdot s\cdot(2o\cdot r+3i\cdot r+o\cdot i)$
backward2 | $2b\cdot s(o\cdot r+2i\cdot r+2i\cdot o)+2i\cdot o\cdot r$
backward3 | $2b\cdot s\cdot(2i\cdot o+o\cdot r+i\cdot r)+4i\cdot r\cdot o$
backward4 | $2(2b\cdot s\cdot i\cdot o+3i\cdot o\cdot r)$
backward5 | $2b\cdot s\cdot(2o\cdot r+2i\cdot r+o\cdot i)+2i\cdot o\cdot r$
backward6 | $2b\cdot s\cdot(2o\cdot r+2i\cdot r+2o\cdot i)+4i\cdot o\cdot r$
backward7 | $2b\cdot s\cdot(o\cdot r+i\cdot r+2o\cdot i)+4i\cdot o\cdot r$
backward8 | $2b\cdot s\cdot(o\cdot r+i\cdot r+2o\cdot i)+4i\cdot o\cdot r$
Table 1: Number of floating-point operations per second(FLOPs) for our
implemented forward and backward passes. $b$ \- batch size; $s$ \- sequence
length; $i$ \- input dimension, $o$ \- output dimension; $r$ \- adapter rank;

##  3 Numerical experiments

We have conducted a number of experiments on LLama family of models ranging
from 60 millions of parameters up to 1.3 billion. We measured mean time of a
Forward-Backward pass through the network for different architectures and
training settings. As showed in Table 3, we managed to achieve up to 17%
speedup compared to default peft implementation.

Additionally, since RunLoRA forward functions do not save intermediate result
$XA$ we manage to save up to 4Gb of memory.

All experiments were performed on a single Nvidia A100 GPU 80Gb. Training was
conducted in bfloat16 format.

Implementation | Mean F-B loop, ms | Memory for F-B loop, MB | Speedup, % | Memory Saved, MB
---|---|---|---|---
llama 60m, b=64, r=8 [head to column names]llama_60mb64r8.csv
llama 250m, b=58, r=128 [head to column names]llama_250mb58r128.csv
llama 1.3b, b=24, r=512 [head to column names]llama_1bb24r512.csv

Table 2: Comparison between RunLora (with flops criterion) and default LoRA
implementation from peft library. $b$-batch size, $r$-lora rank.

##  4 Conclusion and Future Work

One of the possible directions for future work is utilizing quantization of
weights to conduct experiments on larger batch sizes and evaluate RunLoRA on
some of the current top LLMs, for example, Llama2-7b/13b/70b. We expect larger
batch sizes and architectures to show even greater performance improvement.

## References

  * Hu et al. [2022] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  2022\.
  * Dettmers et al. [2023] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.  Qlora: Efficient finetuning of quantized llms.  _ArXiv_ , abs/2305.14314, 2023.  URL https://api.semanticscholar.org/CorpusID:258841328.
  * Lialin et al. [2023] Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky.  Stack more layers differently: High-rank training through low-rank updates, 2023.