# Wind speed prediction using multidimensional convolutional neural networks

Kevin Trebing  Department of Knowledge Engineering
Maastricht University
Maastricht, The Netherlands
<EMAIL_ADDRESS>Siamak Mehrkanoon *corresponding
author. Department of Knowledge Engineering
Maastricht University
Maastricht, The Netherlands
<EMAIL_ADDRESS>
###### Abstract

Accurate wind speed forecasting is of great importance for many economic,
business and management sectors. This paper introduces a new model based on
convolutional neural networks (CNNs) for wind speed prediction tasks. In
particular, we show that compared to classical CNN-based models, the proposed
model is able to better characterise the spatio-temporal evolution of the wind
data by learning the underlying complex input-output relationships from
multiple dimensions (views) of the input data. The proposed model exploits the
spatio-temporal multivariate multidimensional historical weather data for
learning new representations used for wind forecasting. We conduct experiments
on two real-life weather datasets. The datasets are measurements from cities
in Denmark and in the Netherlands. The proposed model is compared with
traditional 2- and 3-dimensional CNN models, a 2D-CNN model with an attention
layer and a 2D-CNN model equipped with upscaling and depthwise separable
convolutions.

###### Index Terms:

Deep learning, Wind speed prediction, Convolutional neural networks, Feature
learning, Short-term forecasting

##  I Introduction

Weather conditions undeniably affect many aspects of life in modern societies.
Adverse weather conditions and events can have both direct and indirect
effects on a large number of economic and business sectors, such as transport,
logistics and agriculture. Accurate and timely weather forecasts are therefore
important for a variety of applications by facilitating planning and
management in response to weather conditions. Weather forecasts are crucial to
predict natural disasters such as extreme rainfalls, floods, hurricanes and
heat waves. With wind power gaining importance as a source of renewable energy
in recent years, wind speed forecasting has become an important tool for
efficient and adaptive management of wind parks [1].

To date, weather forecasting typically relies on numerical weather prediction
(NWP) models to solve complex mathematical equations that simulate the real
world’s atmosphere, fluid- and thermodynamics as closely as possible [2]. This
approach requires immense computing power and, even with current technological
equipment and tools, can take several hours to process [3]. Due to their
extensive computational requirements, the applicability of NWP models is, in
practice, restricted to more long-term predictions. That is, NWP models can be
used to predict for instance three hours ahead, however, as the time to
process this prediction takes longer than three hours, the practical utility
of this approach becomes limited [3]. This results in an important gap for
short-term weather predictions that could enable short-term planning.

One way this gap has been addressed so far is through data-driven machine
learning models that have been able to significantly reduce the processing
time compared to NWP models (e.g. [3, 4]). Machine learning-based models have
been successfully applied in various application domains [5, 6, 7, 8]. In
particular, the literature has witnessed the use of different classes of
machine learning models for time-series prediction tasks. They range from
kernel-based models such as support vector machines [9], Least squares support
vector machines [10] as well as deep artificial neural networks-based models
among others [11, 12, 13]. Machine learning-based models for weather
forecasting can potentially reduce the need for large computing power relative
to the requirements of current NWP models. A crucial difference between the
two approaches to weather forecasting lies in what they base their predictions
on. Unlike conventional NWP models, data-driven approaches often do not make
any assumptions about the underlying physical factors influencing weather.
Instead, they use historical weather observations to train a machine learning
model to map the exogenous input to a target output [14].

In recent years, convolutional neural networks (CNNs) and their variants have
emerged as a powerful computational architecture for addressing a range of
challenging tasks [15, 16]. Convolutions are kernel-based operations that
employ a sliding window approach over a matrix or tensor. The use of kernels
allows to capture local invariant features and utilising weight sharing
decreases the number of trainable parameters of the network significantly
[15].

The advancement of modern deep learning techniques has inspired many
researchers to explore the available massive amount of weather data. The
prediction performance of deep learning architectures for the purpose of
weather forecasting have been studied in [17]. The authors in [18] combined
predictive models with a deep neural network to model the joint statistics of
a set of weather-related variables. In this work, we propose a novel
convolutional neural network architecture to model the underlying complex
nonlinear dynamics of wind speed in multiple Danish and Dutch cities over the
course of up to 24 hours. This is achieved by reshaping the data in a way that
allows the use of convolutions over multiple input dimensions and fusing the
obtained feature maps per dimension. We show that our model is able to predict
multiple steps ahead with smaller error in comparison to the other examined
models.

This paper is organised as follows. A brief overview of related research works
for weather forecasting is given in section II. The proposed deep CNN-based
architecture for wind speed forecasting together with the models that we used
for comparison are presented in section III. Section IV describes the
conducted experiments and the obtained results. A discussion on models
performance is given in section V. We end with some conclusive remarks in
section VI.

##  II Related Work

A number of different machine learning models have been proposed for weather
forecasting. One of the most widely adopted approaches has seen Multilayer
Perceptron (MLP) as main model for this purpose [19, 20, 1, 21]. An MLP is a
rather simple neural network that consists of at least three layers: an input
layer, a hidden layer and an output layer. Although MLPs were shown to be
useful for a large range of applications, recently more sophisticated models
such as convolutional neural networks have been applied to more challenging
and complex tasks. Initially, CNNs have been used for image applications due
to their aptness at capturing spatial relations [15]. However, since then,
CNNs have also been successfully applied to time-series tasks. For instance,
the authors in [22] utilised a CNN-based model to achieve high classification
accuracy for a time-series task for detecting human transportation mode. The
input to their model consists of 1-D data because they only have access to one
sensor reading. Weather data, in contrast, often has multiple sensor readings
resulting in multi-dimensional data, thus presenting a more complex time-
series task that requires a more sophisticated model.

For weather prediction tasks, convolutions are often used in conjunction with
long short-term memory (LSTM). An LSTM is a particular type of recurrent
neural network which, unlike standard feedforward neural networks, includes
feedback connections which provide the network with some kind of memory of
previous signals, making it adept at learning from experience and well-suited
for predictions using time series data [23]. Taking advantage of this, the
authors in their seminal work in [24] were the first to combine convolutions
with an LSTM to create the ConvLSTM model for precipitation forecasting. An
important challenge in machine learning-based precipitation forecasting is
that the rain maps that are typically used as input are high-dimensional data
which requires the model to make efficient use of the available information.
Extending on the idea of [24], the authors in [25] therefore stacked multiple
convolutional LSTMs while also adding extra memory connections in order to
enable efficient flow of spatial information, resulting in their PredRNN
model.

In the field of precipitation forecasting, the authors in [3] employed a U-net
architecture [26] to predict rainfall one hour ahead using the last hour of
rain maps. A U-net is a type of CNN that employs an encoder-decoder structure
which at first applies several iterations of downsampling followed by several
iterations of upsampling. In addition, the upsampling integrates the output
from previous layers through so-called skip-connections. This particular
architecture combining up- and downsampling enables a synergistic learning of
both low- and high-level features. An interesting aspect of [3] is that they
predict four discrete classes of rainfall based on millimetres of rain per
hour, rather than predicting exact values of rainfall as has predominantly
been done in previous models. This classification approach has recently been
extended by [4] who map their output onto 512 bins, thereby resulting in a
much finer resolution.

A frequently used baseline in data-driven weather forecasting is the ‘Naïve
Predictor’ which predicts for time $t+\Delta t$ the same value as at time $t$
[2]. This method is also called ‘Persistence’ because it is based on the
assumption that the weather persists from one time window to the next.
Although this prediction method is rather simple, it proved to be very
accurate for very-short and short forecasts as described by [2]. In the next
section we describe the models that we use and explain our proposed model
architecture that was applied to real-life data for the wind speed prediction
task.

##  III Methods

###  III-A Proposed multidimensional model

The present study builds upon and extends the work of [14] who introduced
different CNN architectures, including 1-D, 2-D and 3-D convolutions, to
accurately predict wind speed in the next 6 to 12 hours. In particular, for
our new model, we apply depthwise-separable convolutions (DSCs) [27] to all
three input tensor dimensions (channel, width, height). In this way, the
kernels of the three convolutions go over the height-and-width, channel-and-
height and channel-and-width dimensions, respectively. For the studied wind
speed prediction task, this corresponds to time steps-features, cities-
features, cities-time steps. The output of the three DSCs are then
concatenated and fed into a fully connected layer. Thanks to the proposed
architecture, the underlying information within each view of the input are
learnt jointly in an end-to-end fashion.

Depthwise-separable convolutions have their convolution operation split up
into two convolutions: a depthwise and a pointwise convolution. The advantage
of using DSCs over normal convolutions is that they require a significant
amount less parameters while keeping a similar accuracy [28]. Here, we train
our model to predict the wind speed multiple steps ahead. As the prediction
time ahead increases, the prediction task becomes increasingly complex for the
model. After applying DSCs to all three dimensions we flatten the resulting
feature maps to feed it into a fully connected layer followed by the output
layer. In order to be able to apply a convolution to other dimensions, the
input tensor needs to be permuted. A visual interpretation of this approach
can be found in Fig. 1. In our approach we set the number of feature maps per
channel to 16 and the number of hidden neurons of the fully connected layer to
128. Following every pointwise convolution we apply a batch normalisation. A
rectified linear (ReLu) activation function [29] is used after batch
normalisation as well as after the dense layer.

Figure 1: Using depthwise-separable convolutions to go over different
dimensions of the input tensor. The resulting three tensors are concatenated
along the city-dimension and then flattened and fed into a dense layer with
128 hidden neurons. Lastly, a dense layer with three neurons calculates the
final output.

###  III-B Alternative convolutional models

We have compared the performance of our proposed model with four alternative
CNN-based models. This includes the 2D and 3D convolutional models from [14].
A 2D convolutional approach using attention [30] and a 2D convolution with a
transposed convolution of the original input followed by a double convolution
as it is used in a U-net structure [26], but instead of normal convolutions we
used depthwise-separable convolutions. A transposed convolution is used for
upscaling the input dimensions. It has a similar effect than upsampling, but
with a transposed convolution the network can adjust the kernels for the
upsampling as well, allowing it to better adapt to the underlying data. The
models with their respective trainable parameters can be found in Table I. The
amount of parameters differed for the two datasets, because their input
dimensions were of different size.

TABLE I: The number of parameters of the five models that were compared in the experiments. Model | # Feature maps | Parameters
---|---|---
|  | Denmark | Netherlands
2D | 32 | 46,115 | 112,167
2D + Attention | 32 | 47,059 | 113,367
2D + Upscaling | 32 | 27,974 | 77,568
3D | 10 | 54,749 | 200,929
Multidimensional | 16 | 37,258 | 102,832

It can be seen from Table I that there is a large difference between the
number of parameters in the models of the Dutch and Danish dataset. This
difference arises from different input sizes of the dataset. The Danish
dataset has an input shape of $5\times 4\times 4$ and the Dutch dataset has an
input shape of $7\times 6\times 6$ (see section IV for a more detailed
description of the datasets). Since we are flattening the output of the
convolutional layers and feed it to a dense layer, the number of parameters
increase dramatically. The architectures of the alternative models used for
comparison are shown in Fig. 2.

The number of feature maps for the 2D and 3D CNN-based models are set to 32
and 10, respectively. For our proposed model the number of feature maps is set
to 16 for every dimension. We train the models for a maximum of 150 epochs
with a batch size of 64. An early stopping is used to track the validation
error and if it does not decrease after 20 epochs the training will stop. We
use the Adam optimiser [31] with default values and a learning rate of 0.001
for updating the model parameters.

(a)

(b)

(c)

(d)

Figure 2: The model architecture of the models that were tested. (a) 2D
convolutional neural network. (b) 2D convolutional neural network with
attention. (c) 2D convolutional neural network with transposed convolution
(upscaling). (d) 3D convolutional neural network. The input dimensions and
number of output neurons are adjusted to the dataset. Here, the models for the
DK dataset are shown.

The code of our proposed model and the other trained models along with the
datasets can be found on Github111https://github.com/HansBambel/multidim_conv.

##  IV Experiments

We evaluate our model together with the other discussed four models on two
different real-life datasets from Denmark and the Netherlands. The cities from
which the weather measurements have been recorded and used in our experiments
are shown in Fig. 3.

(a) Danish weather stations

(b) Dutch weather stations

Figure 3: The cities where the wind data was taken: (a) Aalborg, Aarhus,
Esbjerg, Odense and Roskilde. (b) Schiphol, De Bilt, Leeuwarden, Eelde,
Rotterdam, Eindhoven, Maastricht.

The performance of the models is measured by their mean absolute error (MAE)
and mean squared error (MSE) on the test set. MAE calculates the mean of the
difference between the predicted values and true values and is calculated as
follows:

$MAE=\frac{\sum^{n}_{i=1}|y_{i}-\hat{y}_{i}|}{n},$ (1)

where $n$ is the number of samples and $y_{i}$ and $\hat{y}_{i}$ is the true
and predicted value, respectively. MSE calculates the mean of the squared
difference between the predicted values and the true values. As a result,
larger differences between $y$ and $\hat{y}$ contribute more to the error term
than smaller differences. MSE is calculated as follows:

$MSE=\frac{\sum^{n}_{i=1}(y_{i}-\hat{y}_{i})^{2}}{n}.$ (2)

MAE was chosen to be able to quantitatively compare the results with those in
[14] and MSE was chosen to have an indicator how reasonable the predictions of
the models are. In what follows, the two used datasets are described in more
detail.

###  IV-A Dataset Denmark

This is the same dataset as used in [14] and available
online222https://sites.google.com/view/siamak-mehrkanoon/code-data. It
contains hourly measurements of temperature, pressure, wind speed and wind
direction of five cities in Denmark (see Fig. 3a) from 2000-2010. The dataset
is split into training and validation (years 2000-2009) and test set (year
2010). The dataset is normalised using a min-max scaler that was trained on
the training set. Furthermore, the dataset is arranged in a way that a sample
contains the last four measurements in all the five cities and the target is
the wind speed in three of the cities (Esbjerg, Odense and Roskilde).
Therefore, the input dimensions have the shape $C\times T\times F$ where
$C=\\#cities$, $T=\\#timesteps$ and $F=\\#features$ (resulting in a shape of
size $5\times 4\times 4$). Those are the three input dimensions of the left-
most cube in Fig. 1. We trained all five models multiple times to predict the
wind speed multiple steps ahead which corresponds to 6, 12, 18 and 24 hours
ahead in the Danish dataset. Additionally, we created a persistence model as
baseline that uses the last wind speed measurement of each city as the
prediction.

The average of the obtained MAEs and MSEs, corresponding to different models,
over three target cities in Denmark are tabulated in Table II. The best MAE
and MSE are underlined for each tested prediction time in Table II. Our model
performs best or second best for all of the tested prediction time. In
particular, one can observe that our proposed multidimensional architecture
has the smallest overall MAE in the 6h and 24h prediction and the smallest MSE
in the 6h and 24h prediction. The error increases in all models the further
ahead into the future the model needs to predict. This makes sense because
values in the near future are less likely to change than those further away,
therefore they are easier to predict.

TABLE II: The average MAEs and MSEs of different models over target cities in Denmark dataset.  Model | MAE | MSE
---|---|---
| 6h | 12h | 18h | 24h | 6h | 12h | 18h | 24h
Persistence | 1.649 | 2.210 | 2.309 | 2.313 | 4.608 | 7.929 | 8.702 | 8.812
2D | 1.304 | 1.746 | 1.930 | 2.004 | 2.824 | 5.088 | 6.120 | 6.610
2D+Attention | 1.313 | 1.715 | 1.905 | 1.950 | 2.885 | 4.896 | 5.933 | 6.201
2D+Upscaling | 1.307 | 1.723 | 1.858 | 1.985 | 2.826 | 4.931 | 5.639 | 6.474
3D | 1.311 | 1.677 | 1.908 | 1.957 | 2.855 | 4.595 | 5.958 | 6.238
Multidimensional | 1.302 | 1.706 | 1.873 | 1.925 | 2.804 | 4.779 | 5.773 | 6.066

Furthermore, one can note that the error increases less in the proposed
multidimensional model than in the other models. This is an indicator that our
model has better generalisation capabilities and can adapt better to different
kinds of data. The baseline model is outperformed by a large margin in all
step ahead predictions.

In Fig. 4 the MAE of the models for each of the three target cities in all
four prediction times are shown. The plots show that our model has, for almost
all time steps, the lowest or second lowest error in each city. The
differences in the performance of the models get larger the further they have
to predict into the future.

(a) 6 hours ahead (b) 12 hours ahead

(c) 18 hours ahead (d) 24 hours ahead

Figure 4: The models performance for different time steps ahead for each
cities forecast of the Denmark dataset.

To illustrate further how well the models are performing across all time steps
we calculated the mean of the MAEs for each city across the time steps (see
Fig. 5). We can observe that our model is performing best for each of the
three cities.

Figure 5: Mean of the models performance across different time steps for each
city in the Danish dataset.

###  IV-B Dataset Netherlands

The second dataset for this study was acquired from the Royal Netherlands
Meteorological Institute. It contains hourly weather measurements from 7
cities in the Netherlands (Fig. 3b) from January 1, 2011 until March 29, 2020
resulting in 81.000 data points. The measured weather features are wind speed,
wind direction, temperature, dew point, air pressure and rain amount. We split
the dataset into training (January 2011 - December 2018) and testing (January
2019 - March 2020). The arrangement of the dataset is the same as the Denmark
dataset: $C\times T\times F$. We choose to use the last 6 measurements as
input and predict the wind speed 1-, 2-, 3- and 4-steps ahead (corresponding
to 1h, 2h, 3h and 4h ahead, respectively), which results in an input tensor of
shape $7\times 6\times 6$. Additionally, we create a persistence baseline by
predicting the last given input values.

The performance of the models trained on this dataset can be found in Table
III. From Table III, one can observe that our model has the best performance
in the 2h and 3h prediction when looking at the MAE and outperforms the other
models in the 2h, 3h and 4h ahead predictions when using MSE. The baseline is
outperformed by every model by a big margin.

TABLE III: The average MAEs and MSEs of different models over target cities in the Netherlands dataset. Model | MAE | MSE
---|---|---
| 1h | 2h | 3h | 4h | 1h | 2h | 3h | 4h
Persistence | 9.55 | 11.34 | 12.90 | 14.37 | 183.61 | 246.95 | 310.38 | 375.36
2D | 8.11 | 9.17 | 10.15 | 11.12 | 116.89 | 149.01 | 181.78 | 218.49
2D+Attention | 8.08 | 9.10 | 10.11 | 11.00 | 115.96 | 147.75 | 180.66 | 213.23
2D+Upscaling | 8.16 | 9.07 | 10.14 | 10.85 | 117.80 | 147.21 | 182.44 | 208.96
3D | 8.17 | 9.26 | 10.15 | 10.93 | 118.35 | 151.51 | 181.35 | 211.19
Multidimensional | 8.12 | 9.05 | 9.95 | 10.94 | 116.78 | 144.51 | 174.07 | 208.73

For this dataset we also calculated the mean of the MAEs for each individual
city across the four different time steps. The results are shown in Fig. 6. It
can be seen that our multidimensional model performs best or second best in
five out of the seven cities.

Figure 6: Mean of the models performance across different time steps for each
city in the Dutch dataset.

An example of a 1 hour and 4 hour prediction from our model is shown in Fig.
7. For this, we used a subset of the data and plotted the wind speed
prediction for the cities of Eindhoven and Maastricht at each time step. Note
that the MAE reported is for the whole test set.

(a)  (b)

(c)  (d)

Figure 7: Comparison of predictions using our model and the real value. It is
visible that the 1 hour predictions (a) and (b) follow the trend more closely
than the 4 hour predictions (c) and (d).

The Fig. 7 shows that 1-hour ahead prediction of our model more closely
follows the real measurements compared to that of the 4-hours ahead
predictions.

##  V Discussion

In this study, we propose a new convolution neural network for short-term wind
speed predictions using weather datasets from Denmark and the Netherlands.
Specifically, our new model applies depthwise-separable convolutions to three
dimensions of the input tensor. To assess the model performance, we compare it
to four previously described CNNs, including a 2D model, a 3D model, a 2D
model with attention and a 2D model with transposed convolution.

Using MAE and MSE as performance indicators, the analysis of our experiments
shows that our model outperforms previous prediction approaches for most time
steps on both of the tested datasets. We hypothesise that these performance
improvements can be attributed to our novel approach of applying convolutions
separately over different input dimensions, rather than applying them to only
the height and width dimensions as done by standard convolutional approaches.
The proposed architecture enables the network to more efficiently exploit
information from all dimensions.

Our findings of the 2D and 3D models are consistent with the results of [14].
They showed that a 3D-convolutional approach outperforms a 2D-convolutional
approach on the Danish dataset. Likewise, in our study, the 3D-convolution
model performed better than the regular 2D-convolution model for all time
steps on the Danish dataset except for the 6-hour prediction and thus,
overall, performed better than the 2D-convolution model (see Table II).
Interestingly, the 2D-convolutional models with attention and upscaling are
able to achieve better results than the regular 2D- and 3D-models.

We did not find the same high performance of the 3D-convolutional model for
the Dutch dataset when looking at different time steps. Indeed, when
disregarding the baseline model, the 3D model showed the worst or second worst
MAE and MSE for most time steps as can be seen in Table III. Interestingly,
the 3D model performs well when comparing the mean of the time steps for each
individual city (see Fig. 5 and 6). Our multidimensional model performed
better than the other models in the later time step prediction when
considering MSE as performance indicator. As previously mentioned, a smaller
MSE is an indicator of how reasonable the model predicts. Overall, the high
performance of our model generalises better across time steps and the two used
datasets (see Fig. 5 and 6), largely providing better predictions than
previously described standard based convolutional approaches.

An important limitation of our study concerns the comparability of the five
models. We used number of parameters as main indicator for model
comparability, rather than optimising every model individually which was
beyond the scope of the present study. Thus, we sought to keep the number of
parameters approximately similar in all models without modifying too many
other configurations. To this end, we set the number of feature maps for the
three 2D models to 32, for the 3D model to 10, and to 16 per dimension in our
model (see Table I). The number of feature maps for the different models was
based on the input dimensions of the Danish dataset and were subsequently also
applied to the Dutch dataset that has larger input dimensions than the Danish
dataset. This explains the greater range in number of parameters of the models
when applied to the Dutch dataset (77,568 - 200,929 parameters) compared to
the Danish dataset (27,974 - 54,749 parameters), as shown in Table I. We also
trained the 2D models with less feature maps (10 feature maps, as in [14]),
but observed no performance increase.

##  VI Conclusion and Future Work

In this paper a new multidimensional CNN based model is introduced for wind
speed prediction. We showed that applying convolutions to multiple dimensions
helps to capture more modalities of the data. Using real weather datasets we
tested our approach against conventional 2D- and 3D-CNNs and showed that our
proposed model is able to more accurately predict wind speeds multiple steps
ahead. Furthermore, our proposed multidimensional model performed best across
different time steps compared to the other tested models.

## References

  * [1] G. Li and J. Shi, “On comparing three artificial neural networks for wind speed forecasting,” _Applied Energy_ , vol. 87, no. 7, pp. 2313–2320, 2010\.
  * [2] S. S. Soman, H. Zareipour, O. Malik, and P. Mandal, “A review of wind power and wind speed forecasting methods with different time horizons,” in _North American Power Symposium 2010_. IEEE, 2010, pp. 1–8.
  * [3] S. Agrawal, L. Barrington, C. Bromberg, J. Burge, C. Gazen, and J. Hickey, “Machine learning for precipitation nowcasting from radar images,” _arXiv preprint arXiv:1912.12132_ , 2019.
  * [4] C. K. Sønderby, L. Espeholt, J. Heek, M. Dehghani, A. Oliver, T. Salimans, S. Agrawal, J. Hickey, and N. Kalchbrenner, “Metnet: A neural weather model for precipitation forecasting,” _arXiv preprint arXiv:2003.12140_ , 2020\.
  * [5] S. Mehrkanoon, S. Mehrkanoon, and J. A. Suykens, “Parameter estimation of delay differential equations: an integration-free ls-svm approach,” _Communications in Nonlinear Science and Numerical Simulation_ , vol. 19, no. 4, pp. 830–841, 2014.
  * [6] S. Mehrkanoon, “Cross-domain neural-kernel networks,” _Pattern Recognition Letters_ , vol. 125, pp. 474–480, 2019.
  * [7] S. Mehrkanoon and J. A. Suykens, “Deep hybrid neural-kernel networks using random fourier features,” _Neurocomputing_ , vol. 298, pp. 46–54, 2018.
  * [8] ——, “Regularized semipaired kernel CCA for domain adaptation,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 29, no. 7, pp. 3199–3213, 2017.
  * [9] N. I. Sapankevych and R. Sankar, “Time series prediction using support vector machines: a survey,” _IEEE Computational Intelligence Magazine_ , vol. 4, no. 2, pp. 24–38, 2009.
  * [10] T. Van Gestel, J. A. Suykens, D.-E. Baestaens, A. Lambrechts, G. Lanckriet, B. Vandaele, B. De Moor, and J. Vandewalle, “Financial time series prediction using least squares support vector machines within the evidence framework,” _IEEE Transactions on neural networks_ , vol. 12, no. 4, pp. 809–821, 2001.
  * [11] S. Scher, “Toward data-driven weather and climate forecasting: Approximating a simple general circulation model with deep learning,” _Geophysical Research Letters_ , vol. 45, no. 22, pp. 12–616, 2018.
  * [12] M. Hossain, B. Rekabdar, S. J. Louis, and S. Dascalu, “Forecasting the weather of nevada: A deep learning approach,” in _2015 international joint conference on neural networks (IJCNN)_. IEEE, 2015, pp. 1–6.
  * [13] E. R. Rodrigues, I. Oliveira, R. Cunha, and M. Netto, “Deepdownscale: a deep learning strategy for high-resolution weather forecast,” in _2018 IEEE 14th International Conference on e-Science (e-Science)_. IEEE, 2018, pp. 415–422.
  * [14] S. Mehrkanoon, “Deep shared representation learning for weather elements forecasting,” _Knowledge-Based Systems_ , vol. 179, pp. 120–128, 2019.
  * [15] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Advances in neural information processing systems_ , 2012, pp. 1097–1105.
  * [16] S. Mehrkanoon, “Deep neural-kernel blocks,” _Neural Networks_ , vol. 116, pp. 46–55, 2019.
  * [17] A. G. Salman, B. Kanigoro, and Y. Heryadi, “Weather forecasting using deep learning techniques,” in _2015 international conference on advanced computer science and information systems (ICACSIS)_. IEEE, 2015, pp. 281–285.
  * [18] A. Grover, A. Kapoor, and E. Horvitz, “A deep hybrid model for weather forecasting,” in _Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , 2015, pp. 379–386.
  * [19] E. Cadenas and W. Rivera, “Short term wind speed forecasting in la venta, oaxaca, méxico, using artificial neural networks,” _Renewable Energy_ , vol. 34, no. 1, pp. 274–278, 2009.
  * [20] M. Monfared, H. Rastegar, and H. M. Kojabadi, “A new strategy for wind speed forecasting using artificial intelligent methods,” _Renewable energy_ , vol. 34, no. 3, pp. 845–848, 2009.
  * [21] K. Rohrig and B. Lange, “Application of wind power prediction tools for power system operations,” in _2006 IEEE Power Engineering Society General Meeting_. IEEE, 2006, pp. 5–pp.
  * [22] X. Liang, Y. Zhang, G. Wang, and S. Xu, “A deep learning model for transportation mode detection based on smartphone sensing data,” _IEEE Transactions on Intelligent Transportation Systems_ , 2019.
  * [23] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [24] S. Xingjian, Z. Chen, H. Wang, D.-Y. Yeung, W.-K. Wong, and W.-c. Woo, “Convolutional lstm network: A machine learning approach for precipitation nowcasting,” in _Advances in neural information processing systems_ , 2015, pp. 802–810.
  * [25] Y. Wang, M. Long, J. Wang, Z. Gao, and S. Y. Philip, “Predrnn: Recurrent neural networks for predictive learning using spatiotemporal lstms,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 879–888.
  * [26] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [27] F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1251–1258.
  * [28] V. J. Lawhern, A. J. Solon, N. R. Waytowich, S. M. Gordon, C. P. Hung, and B. J. Lance, “Eegnet: a compact convolutional neural network for eeg-based brain–computer interfaces,” _Journal of neural engineering_ , vol. 15, no. 5, p. 056013, 2018.
  * [29] V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in _Proceedings of the 27th international conference on machine learning (ICML-10)_ , 2010, pp. 807–814.
  * [30] I. Bello, B. Zoph, A. Vaswani, J. Shlens, and Q. V. Le, “Attention augmented convolutional networks,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 3286–3295.
  * [31] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.


XX XX 8 September

Department: Head Editor: Name, xxxx@email

# Co-designing for a Hybrid Workplace Experience in Software Development

Zhendong Wang†∗ Yi-Hung Chou†∗ Kayla Fathi‡ Tobias Schimmer‡ Peter Colligan‡
David Redmiles† and Rafael Prikladnicki§ †University of California, Irvine,
USA
‡SAP Labs, USA
§Pontifical Catholic University of Rio Grande do Sul, Brazil

(2022)

###### Abstract

With increasing demands for flexible work models, many IT organizations have
adapted to hybrid work that promises enhanced team productivity as well as
work satisfaction. To achieve productive engineering practice, collaborative
product innovation, and effective mentorship in the ensuing hybrid work, we
introduce a workshop approach on co-designing for a hybrid workplace
experience and provide implications for continuously improving collaborative
software development at scale.

††journal: IEEE Software

As COVID-19 indiscriminately circled the globe, millions of workers had to
adapt to a fully remote work environment. ††footnotetext: *: This work was
conducted while Zhendong and Yi-Hung interning at SAP. Since this massive and
widespread shift in work conditions, software practitioners have been re-
evaluating their work preferences. According to increasing reports of shifts
in work preferences, a plurality of information workers prefer a new normal of
hybrid arrangements, which can yield improved flexibility and productivity
concurrently.

The present article describes our practice of designing a hybrid workplace
experience at one branch office of SAP Labs. This office tried to adopt a
version of hybrid work: a work model combining in-office and remote work
schedules. This model intends to improve collaboration quality with in-person
sessions while also providing flexible and focused work time [1].

We took on a co-design approach to address issues of transitioning towards a
hybrid work model. We deliberately included various roles of software
practitioners into the process of designing hybrid workplace experiences:
prior studies suggest that creativity can be a wellspring of inspiration for
everyone, especially when it comes to experiences that are closely meaningful
to them [2]. Besides, we argue that following a bottom-up decision-making
process improved developer satisfaction during the transition, and potentially
enhanced the future hybrid experience. Thus, we leveraged co-design workshops
to devise and improve software developers’ hybrid workplace, and also
incorporated the site’s design thinking practice for inspiring participants’
creativity.

The main question behind this work is:

How to design for and continuously improve a workplace experience with
software development teams while transitioning to a hybrid work arrangement?

To answer this question, SAP-affiliated authors facilitated three co-design
workshops at a branch office in the US. After providing some background, we
first describe goals of each workshop and their process with some intermediate
results respectively. This is followed by participant feedback and our
recommendations on improving the hybrid work model and workplace from the
perspectives of communication, change management, and organizational culture.
In particular, although these workshops created an initial structure of hybrid
schedules, software practitioners at this site still expected to regularly re-
visit their hybrid work model and calibrate its arrangements as an outcome of
their team retrospectives. We argue that our co-design adaptation, lessons
learned according to this practice, and recommendations for a hybrid work
model can provide valuable insights for other software organizations adapting
to the “new” normal in the post-pandemic world.

## BACKGROUND

Co-design refers to the collective creativity of both designers and non-
designers working together in the design process [2]. As a participatory
approach to design, it strives to cater various stakeholders’ fuzzy needs as
well as foster consensus, creativity, and collaboration. Human-Computer
Interaction (HCI) and Software Engineering (SE) research has increasingly
demonstrated interest in co-designing for marginalized populations and
underrepresented groups. In SE, prior co-design practice also enabled
developers to articulate imprecise requirements in software-intensive
development [3], and hence improve user satisfaction. While recent reports
often indicate developers’ diverse work preferences towards remote and on-site
arrangements [1], co-design could be a suitable approach to address an
organization’s fuzzy needs during transitioning to a particular work model.

To leverage practitioners’ creative potential, we incorporated this site’s
design thinking (DT) practice into the co-design process. For this SAP site,
DT has been applied as a collaborative and iterative approach that explores
and elaborates the value proposition of a product in the early stages. Its DT
approach aims to establish empathy for end users, and reduce the likelihood of
costly adaptations [4]. Building upon DT’s success with customer co-
innovation, i.e., an SAP activity that connects software developers and
product managers with customers and articulates business and technical
requirements of a product, we anticipate improved co-design outcomes by
leveraging participants’ skills acquired from the site’s reliable DT approach.

## CO-DESIGN WORKSHOPS

### Site and Participants

These workshops were organized at an SAP branch office at Newport Beach,
California. SAP is a multinational corporation known for its integrated
enterprise software. This branch office has started operating since 2019. This
office practices Large-Scale Scrum (LeSS) to scale and optimize multi-team
collaboration for software development [5]. Meanwhile, it has experienced
large growth in organization size while working remotely during the pandemic.
Until recently, many remotely onboarded software developers had not yet
experienced this newly operated office. As a global software development
organization, this site bears global dependencies, i.e., several major
products here depend on internal services that are under development by teams
at other locations. Starting 2022, this office has been actively preparing for
hybrid work arrangements within the framework of a global corporate initiative
called “flex work” [6].

Participants in the co-design workshops consisted of software engineers, UX
designers, and product managers representing the major software development
roles at the site. The number of participants varied from 17 to 22 across the
three workshops with a common core of people. The first workshop was organized
in March 2022 including a portion of dedicated remote participants, and in the
rest two they also participated in-person for improved workshop experience.
The site’s actual hybrid transition started in July 2022 (see Figure 1).

All co-design artifacts such as sticky notes and whiteboard collaboration
spaces were digitized for thematic analysis by the authors, and the emerging
themes are reported in the following sections. Additionally, we released
agendas and methodological guidelines for these co-design workshops in a
public repository111Co-design hybrid: https://github.com/co-design-hybrid/co-
design-hybrid.

Figure 1: The overview of our co-design approach and the timeline of
workshops. March 11th, 2022 was the initial Discovery and Co-design. March
16th introduced Jobs-to-Be-Done to specify the ideas produced in the first
workshop. A month later on April 15th, we started the Experimental Design for
hybrid work.

### Workshop 1: Discovery and Co-design

The first workshop was a daylong event for participants to co-design the
site’s hybrid experience, and set up initial action items for pilot
experiments and improvements. This workshop and the following were held in a
large conference room with whiteboard spaces and a wall display (see Figure
2). After a short introduction, participants were randomly assigned to four
co-design teams.

Reframe Challenge and Scope. The first step of this workshop was to
understand, refine and reframe the underlying challenges for transitioning to
hybrid at this site. We encouraged participants to raise questions about
hybrid challenges, and record and paste these questions onto a collaborative
space for team reviews. From this step, participants anchored the overall
direction for how to address issues in hybrid work for this entire workshop.

Research and Synthesis. The second step enabled participants to explore the
needs of their target personas in a hybrid workplace. Personas have been
widely used in design and DT projects as an approach to build empathy between
users and designers [7]. As a starter, facilitators provided the three main
persona examples in the domain of engineering, UX, and product management,
which were based on existing roles at the site. Participants were encouraged
to leverage a broad variety of curated research resources (e.g., internal
technical reports, research literature, and press articles on hybrid work) to
further elaborate on the underlying challenges of each given persona (see
example personas in the public repository1).

Figure 2: Participants might add notes onto whiteboards for refining the
personas, and identify core themes of a persona’s workplace experience and
needs.

Ideate, Prototype, and Validate. Finally, in DT terms, participants “ideated”
on solutions and action items collaboratively to address the needs of their
personas. As a prerequisite, each team first created several Point-Of-View
(POV) statements for their target persona to address specific user needs and
provide improvement insights, e.g., “as a UX designer, I need a collaborative
open space to run customer workshops with my team so that we can get maximum
empathy and feedback from customers.” Then each participant elaborated on
their ideas with POV statements, and proceeded with a team discussion. They
prototyped their top-team-rated ideas by creating a concrete plan that
included a set of action items for the organization to experiment and improve
its workplace. The prototype validation was further carried out with a team
discussion and voting activity. At the end of this workshop, each co-design
team provided a presentation of proposed improvements to the organization.

The first workshop enabled participants to brainstorm on explicit solutions to
address these statements, e.g., proposing the organization arrange training
and mentoring sessions for career development, and create interest groups for
re-establishing a sense of belonging at work, while some participants were
concerned with few action items strayed off-topic and general work discussion.
Co-design teams mainly voted for the ideas that were actionable for
themselves, and also the ones perceived as substantially leading to an
improved workplace experience.

### Workshop 2: Development Jobs-to-Be-Done

The second workshop aimed at specifying a focused action plan for an ideal
hybrid workplace experience. Since the first workshop created a diverse and
overwhelming number of action items, the second workshop aligned design goals
for a hybrid workplace experience around main development tasks. This
alignment process was guided by a business strategy, Jobs-to-Be-Done, often
adopt at this site, which aims for articulating the practical needs of end
users and customers [8]. Thus, this workshop could create an initial structure
of hybrid schedule, and set tracking that monitored these items’ progress.

We first introduced a categorization of activities that are essential for
overall software design and delivery. This category of activities incorporated
findings in the last workshop into the Collaboration Model, the site’s
software development standards for product management and engineering to
discover, design, and deliver products collaboratively. These categories
included Development Preparation, Heads-down Work, Reviews & Retrospectives,
Customer Co-innovation, Team Building and Personal Development. Participants
placed their recurring development activities into these categories for an
ideal workplace experience: distinguishing the activities that favor on-site
from tasks that prefer remote and quiet environments.

To manage the overwhelming action items that emerged from the first workshop,
the facilitators introduced a visual Kanban board where participants could
create improvement tickets for their action items. Moreover, all employees of
this office had access to the board and thus could comment on and keep track
of the organization’s progress with potential improvements.

The second workshop created an initial hybrid schedule for this site, which
facilitated most collaborative and mentoring activities on-site, and focused
work remotely. When creating the hybrid schedule, participants realized that
optimizing their work required the support of tools and on-site
infrastructure. Additionally, the Kanban board urged the organization to
progress with highly prioritized action items, e.g., providing office benefits
for working on-site days.

Figure 3: General guidelines of a two-week Sprint for collaborative activities
at the site. Mondays, Tuesdays and Fridays are on-site days for collaborative
events, and Wednesdays and Thursdays are off-site days for focused work. Daily
standup meetings can either be online or in-person forms to align with the
overall schedule.
*PBR is short for Product Backlog Refinement meetings as described in [5].

### Workshop 3: Experimental Designs and Piloting

The last workshop was a short session that designs small experimental
improvements for challenging action items and solicited teams to experience
hybrid schedules.

Facilitators organized participants to review the Kanban board together to
focus on several long-lasting and challenging topics (e.g., interactive
whiteboard or writable wall were not sufficient in conference rooms). The
workshop delegated each topic to at least two participants. These small groups
collaboratively plan and set weekly progress goals (e.g., contact facilitator
management team, purchasing, and installation) in a pair working session.

In addition, a few development teams solicited by participants voluntarily
experimented with working on a hybrid schedule. Meanwhile, all participants
continuously provided feedback on the advances during weekly check-ins on-site
and on the transparent Kanban board, e.g., concerns were raised about
allocating on-site review meetings on Fridays overlapping with most team-
building activities.

### Concluding Interviews

We conducted semi-structured concluding interviews with available participants
about their experiences with workshops and ensuing hybrid work, after two
weeks (one Sprint) of an office-wide hybrid schedule.

Participants perceived being a part of the workshops as an interactive,
enjoyable, and engaging experience. They enjoyed attending these creative
events in-person for timely confirmation and feedback, and also fostered the
collective creativity of the participant group. With the last two workshops,
they formed a self-organized group with a shared mission to transparently
address critical issues in their hybrid work. Their experience during this
collaborative process was perceived as a natural way to leverage
organizational resources, e.g., design thinking practice within SAP, existing
site infrastructure, and working as a cross-functional group on-site.

While overwhelming personal needs and concerns were raised, some participants
sensed a disconnect between the workshop results and the ensuing hybrid
practice. For instance, predefined on-site days needed to be agreed upon for
achieving high-quality team synchronization, and promoting highly
collaborative work such as Product Backlog Refinement meetings. Additionally,
some challenging action items took an extended time to progress, e.g.,
although participants preferred that product teams could collaborate with
their product neighborhoods in co-located sessions, compromises had to be made
under globally distributed settings.

## HYBRID WORKPLACE EXPERIENCE

Based on the workshop outcome, we initialized a hybrid schedule that organized
recurring software development tasks such as common Scrum ceremonies into on-
and off-site schedules (see Figure 3). For a typical five-day week during a
Sprint, this site practiced three aligned on-site days for Scrum ceremonies of
the basic Scrum and LeSS framework as well as architecture and UX design
sessions. These on-site days were primarily reserved for collaboration-intense
sessions in-person, and some optional and informal team-building events. The
remaining two days were remote days for focused standalone work sessions in-
between; one was the No-Meeting-Wednesday to reduce interruptions. The
rationale behind this formation was to fit Scrum activities in a 2-week time
frame: started with planning in week 1, and ended with reviews and
retrospectives in week 2. After all, developers might utilize their on-site
workplaces and assigned desks at any time they prefer.

### Bridging Distance between On-Site and Remote Teams

Communication and collaboration are key dimensions of productivity in team-
based software development [9]. Impaired communication and collaboration often
exert escalating effects on other productivity dimensions, which result in re-
work, distrust, and dissatisfaction. For this distributed software development
organization, hybrid meetings–i.e. meetings whose attendees comprise on-site
and remotely off-site–are bound to happen. According to our participants, most
of these meetings were unpleasant experiences due to stressful meeting times
caused by timezone differences, imbalanced awareness and common ground among
attendees who did not share physical meeting space, and skepticism towards
off-site attendees’ commitment.

Our primary suggestion is to coordinate and leverage on-site time deliberately
while limiting the number and frequency of hybrid meetings as such. Prior
lessons learned from this site suggested that the growing distrusts from
avoidable hybrid meetings substantially harmed team spirit and work morale,
especially for inexperienced or newly formed teams. Besides, hosting
unnecessary hybrid meetings could defeat the purpose of on-site hours for in-
person collaboration and fostering social interaction, as mentioned by several
in the concluding interviews.

For unavoidable hybrid meetings with off-site attendees, available and well-
equipped conferencing devices, a software toolset for hybrid collaboration,
and an inclusive meeting format are fundamentals. Based on participants’
feedback, sufficient phone booths and access to various sizes of well-equipped
conference rooms are not only essential for the meeting experience, but also
for creating a focused and distraction-free open workplace. For sharing design
artifacts in hybrid meetings, conference rooms with interactive whiteboards
(e.g. SurfaceHub and Jamboard) substantially help with cross-organization and
-function collaboration. Moreover, providing a complete set of collaboration
tools is equivalently essential as technical engineering systems [10]. This
site supplies each developer with a suite of hybrid collaboration tools,
including Mural, Teams, Slack, LucidChart, etc., to facilitate collaborations
with remote colleagues. Finally, experience from the first workshop suggested
that hybrid meetings could apply an inclusive meeting format such as
Roundtable during discussion. Such an inclusive format ensures remote
participants have equal opportunities to speak up.

### Building Cornerstones of a Seamless Transition

According to participant feedback from the hybrid experiment, the most
essential personal adjustments were allocating commute and core work hours
into in-person days. With the wave of fully remote work during COVID-19, the
work-from-home schedule provided workers with small breaks and personal time
throughout the workday. We provide two major suggestions on initiating a
seamless hybrid transition and preserving the flexibility since working from
home: one, providing local flexibility of working hours and location at the
team level; and two, incentivizing employees to experience hybrid arrangements
earlier.

First, adjustments on individual and team schedules are hardly completed
overnight, and decision-makers may consider providing some room for
flexibility. According to the concerns from participants, strict regulation on
core hours increased their stress over time management in addition to the
reluctance of commuting. Thus, when they attended meetings or were on-call
beyond regular core hours, providing flexibility on working hours and location
would reduce their stress, and safeguard trust within the team. For instance,
attending an early morning meeting with Europe teams at home and then
commuting to the office later at noon should not raise skepticism towards
their work ethics.

Second, participants with prior hybrid or on-site experience, e.g.,
participation in hybrid experimentation, or working in an office before the
pandemic often led to more positive feedback with the office-wide hybrid
schedule. Socializing with teammates in-person, incrementally adjusting to
individual and local teams’ hybrid schedules, and being familiar with office
infrastructure and benefits (especially for remotely onboard members)
substantially improved participants’ workplace experience. Besides monetary
incentives, there were many other events for motivating employees to
experience the hybrid workplace, e.g., customer site visits, social outings
with teammates at or near the office, interest group events with academic
collaborators and colleagues, and office-sponsored lunch events. Being
physically together during these events, cross-function groups could establish
common grounds and foster personal trust for efficient collaborations [11,
12].

### Fostering Culture and Trust for Collaboration and Deep Work

High-performing teams have strong foundations in trust [13]. Trust is a
combination of personal connection, consistency, and experience. Trust is also
a key element of efficient communication, exerting influence on organizations
and design. SAP has had a large success in integrating end-to-end business
processes that cross traditional internal departmental silos. As such, it has
gained a competitive advantage by establishing a culture of high trust and
collaboration. This is especially true for a new team and new product
development of any sufficient complexity where the interfaces between teams
and systems are unclear. Allowing teams to co-work and form together in a
modern workplace drives the desired function of rapid learning and trust
building and accelerating the development of integrated systems and
components.

While minimizing the distance between engineers, UX designers and product
managers is important from an information flow perspective, any sufficiently
complex work to be solved also needs time to focus on individual tasks for
longer periods of time; sometimes referred to as deep work, i.e., professional
activities performed in a state of distraction-free concentration [14].
Balancing highly collaborative work with deep work is consistent with the
site’s core business value around building enterprise-grade products that
leverage systems thinking and decision support planning systems efficiently.
Therefore, hybrid, from the view of mixing highly collaborative and close
proximity work with deep work is substantially ingrained in this
organization’s culture, and a large part of its customer and financial
success. Thus a “dual” style of working is naturally the mode of most
enterprise software engineers, but perhaps rather a more delineated work
pattern in complex sub-system programming and integration as required for
enterprise software.

With COVID-19 shifting SAP teams and thus communication patterns to a
completely distributed model, a shift to more asynchronous and remote
communication methods also occurred. It is therefore important for management
to re-establish existing and re-think new complimentary working principles
along people, process, and technology pillars, i.e., balancing healthy team
characteristics, organizational productivity, and individual flexibility while
ensuring equal career advancement opportunities for off- and on-site members.
This will be an area of interest and research for SAP and likely other large-
scale software firms in the coming years.

### Practicing Co-design to Articulate Specifics in Hybrid Workplace

To locally optimize the hybrid workplace experience and articulate the
specific needs of practitioners at other similar large software development
organizations, we recommend applying this co-design format1 on local teams
with the following considerations. First, participant selection and persona
example illustration of the workshop should be representative based on the
site’s team roles and demographics; second, design formats and resources
should align with the local team’s daily practice to achieve optimal outcomes
of co-design activities; and third, intermediate co-design results should be
communicated to organization members outside the workshops and solicit
feedback, in our case, through a Kanban board.

Moreover, practitioners may consider planning and adapting this format for
other types of organizations through some rigorous structure such as the GQM
approach [15], especially when there were doubts about the worthiness of a
hybrid transition. Particularly, the specific process could be developed based
on the goals of the hybrid transition (e.g., to improve collaborative
engineering and creative work), questions about what can be improved for
transitioning process and beyond (e.g., site infrastructure and career
development opportunities), and metrics to evaluate whether the transition was
satisfying (e.g., unplanned meeting amount and frequency, and employee self-
reported satisfaction).

## CONCLUDING REMARKS

In this article, we provide a novel approach of co-designing for a hybrid
workplace experience in software development, and present our takeaways for
improving such a hybrid arrangement continuously. Co-design workshop formats
can also help other organizations create their own paradigm of hybrid work via
a bottom-up approach. Following a continuous improvement process, we will
regularly re-visit our hybrid arrangement to calibrate organizational
preferences and the site’s collaborative development practice.

## ACKNOWLEDGEMENT

We would like to thank all anonymous reviewers. Chou, Wang and Redmiles are
supported in part by UC Irvine, Donald Bren School of ICS. Prikladnicki is
partially funded by Fapergs and CNPq, Brazil.

## References

  * [1] D. Ford, M.-A. Storey, T. Zimmermann, C. Bird, S. Jaffe, C. Maddila, J. L. Butler, B. Houck, and N. Nagappan, “A tale of two cities: Software developers working from home during the COVID-19 pandemic,” _ACM Transactions on Software Engineering and Methodology (TOSEM)_ , vol. 31, no. 2, pp. 1–37, 2021.
  * [2] E. B.-N. Sanders and P. J. Stappers, “Co-creation and the new landscapes of design,” _Co-design_ , vol. 4, no. 1, pp. 5–18, 2008.
  * [3] J. Hehn, D. Mendez, F. Uebernickel, W. Brenner, and M. Broy, “On integrating design thinking for human-centered requirements engineering,” _IEEE Software_ , vol. 37, no. 2, pp. 25–31, 2019.
  * [4] T. Hildenbrand and J. Meyer, “Intertwining lean and design thinking: Software product development from empathy to shipment,” in _Software for people_. Springer, 2012, pp. 217–237.
  * [5] C. Larman and B. Vodde, _Large-scale scrum: More with LeSS_. Addison-Wesley Professional, 2016.
  * [6] T. Tuan, L. Mendoza, N. Safrudin, J. Park, S. Kim, B. Riefer, and A. Sposato, “The Next WoW - Reimagining hybrid work for an innovation team,” 2021. [Online]. Available: https://apphaus.sap.com/resource/next-wow
  * [7] J. Pruitt and T. Adlin, _The persona lifecycle: keeping people in mind throughout product design_. Elsevier, 2010\.
  * [8] C. M. Christensen, T. Hall, K. Dillon, and D. S. Duncan, “Know Your Customers’ “Jobs to Be Done”,” _Harvard Business Review_ , Sep. 2016, section: Customer strategy. [Online]. Available: https://hbr.org/2016/09/know-your-customers-jobs-to-be-done
  * [9] N. Forsgren, M.-A. Storey, C. Maddila, T. Zimmermann, B. Houck, and J. Butler, “The SPACE of developer productivity: There’s more to it than you think.” _Queue_ , vol. 19, no. 1, pp. 20–48, 2021.
  * [10] V. Jackson, A. van der Hoek, R. Prikladnicki, and C. Ebert, “Collaboration tools for developers,” _IEEE Software_ , vol. 39, no. 2, pp. 7–15, 2022\.
  * [11] G. M. Olson and J. S. Olson, “Distance matters,” _Human–computer interaction_ , vol. 15, no. 2-3, pp. 139–178, 2000.
  * [12] N. B. Moe and D. Šmite, “Understanding a lack of trust in global software teams: a multiple-case study,” _Software Process: Improvement and Practice_ , vol. 13, no. 3, pp. 217–231, 2008.
  * [13] B. W. Tuckman and M. A. C. Jensen, “Stages of small-group development revisited,” _Group & organization studies_, vol. 2, no. 4, pp. 419–427, 1977.
  * [14] C. Newport, _Deep work: Rules for focused success in a distracted world_. Hachette UK, 2016.
  * [15] V. R. Basili, G. Caldiera, and D. H. Rombach, “The goal question metric approach,” _Encyclopedia of software engineering_ , pp. 528–532, 1994.

Zhendong Wang is currently working towards his Ph.D. degree in Software
Engineering at the University of California, Irvine. His research focuses on
leveraging and supporting developer expertise in distributed software
development. Wang received an M.S. degree in Software Engineering from the
University of California, Irvine in 2018. Contact him at<EMAIL_ADDRESS>
Yi-Hung Chou is currently a software development/research intern at SAP
Newport Beach. He is pursuing M.S. Software Engineering at University of
California, Irvine. His research interest is at the intersection of human-
computer interaction and software engineering. Contact him at
<EMAIL_ADDRESS>
Kayla Fathi is a user experience designer at SAP, working within an innovation
unit, Industry Cloud. Fathi received her B.S. in Cognitive Science from the
University of California, Berkeley. Contact her at<EMAIL_ADDRESS>
Tobias Schimmer is a software engineering practitioner with SAP, and an
adjunct professor at the University of Mannheim. He is responsible for SAP’s
Industry Cloud collaboration model and respective product engineering
operations. His research focuses on multi-team collaboration, empirical
software engineering, and data-driven process improvements. Schimmer received
his Ph.D. degree from the University of Mannheim in 2008 and has been a
visiting researcher with University of California, Irvine, since 2006. Contact
him at<EMAIL_ADDRESS>
Peter Colligan is a senior executive manager and head of engineering for SAP’s
Industry Cloud. Contact him at<EMAIL_ADDRESS>
David Redmiles is a Professor in the Department of Informatics at the
University of California, Irvine, in the Donald Bren School of Information and
Computer Sciences. Redmiles received his Ph.D. in Computer Science in 1992
from the University of Colorado, Boulder. His research integrates the areas of
software engineering, human-computer interaction, and computer-supported
cooperative work. Over the years, his research group has investigated themes
of cognitive support for software developers; issues of trust and emotion
affecting software teams; behaviors of participants in social software
development platforms; global software engineering; end-user software
development; and more. He is a member of the ACM and the IEEE Computer
Society. He was designated an ACM Distinguished Scientist in 2011 and a Fellow
of Automated Software Engineering in 2009. Contact him at
<EMAIL_ADDRESS>
Rafael Prikladnicki is an associate professor in the School of Technology at
the Pontifical Catholic University of Rio Grande do Sul, Porto Alegre,
90450171, Brazil. Contact him at https://www.pucrs.br/researchers/rafael-
prikladnicki/ or<EMAIL_ADDRESS>


# On-demand assembly of optically-levitated nanoparticle arrays in vacuum

Jiangwei Yan  The State Key Laboratory of Quantum Optics and Quantum Optics
Devices, Institute of Opto-Electronics, Shanxi University, Taiyuan 030006,
China  Collaborative Innovation Center of Extreme Optics, Shanxi University,
Taiyuan, Shanxi 030006, China  Xudong Yu  The State Key Laboratory of Quantum
Optics and Quantum Optics Devices, Institute of Opto-Electronics, Shanxi
University, Taiyuan 030006, China  Collaborative Innovation Center of Extreme
Optics, Shanxi University, Taiyuan, Shanxi 030006, China  Zheng Vitto Han  The
State Key Laboratory of Quantum Optics and Quantum Optics Devices, Institute
of Opto-Electronics, Shanxi University, Taiyuan 030006, China  Collaborative
Innovation Center of Extreme Optics, Shanxi University, Taiyuan, Shanxi
030006, China  Tongcang Li  Department of Physics and Astronomy, Purdue
University, West Lafayette, IN, USA  Jing Zhang  Email<EMAIL_ADDRESS><EMAIL_ADDRESS>The State Key Laboratory of Quantum Optics and Quantum
Optics Devices, Institute of Opto-Electronics, Shanxi University, Taiyuan
030006, China  Collaborative Innovation Center of Extreme Optics, Shanxi
University, Taiyuan, Shanxi 030006, China

(August 27, 2024)

###### Abstract

Realizing a large-scale fully controllable quantum system is a challenging
task in current physical research and has broad applications. Ultracold atom
and molecule arrays in optical tweezers in vacuum have been used for quantum
simulation, quantum metrology and quantum computing [1, 2, 3, 4, 5]. Recently,
quantum ground state cooling of the center-of-mass motion of a single
optically levitated nanoparticle in vacuum was demonstrated [6, 7, 8],
providing unprecedented opportunities for studying macroscopic quantum
mechanics [9, 10, 11, 12] and precision measurements [13, 14]. In this work,
we create a reconfigurable optically-levitated nanoparticle array in vacuum.
Our optically-levitated nanoparticle array allows full control of individual
nanoparticles to form an arbitrary pattern and detect their motion. As a
concrete example, we choose two nanoparticles without rotation signals from an
array to synthesize a nanodumbbell in-situ by merging them into one trap. The
nanodumbbell synthesized in-situ can rotate beyond 1 GHz. Our work provides a
new platform for studying macroscopic many-body physics [15, 16, 17, 18, 19]
and quantum sensing [13, 14].

Optical levitation employs forces exerted by strongly focused light fields to
capture and manipulate microparticles and nanoparticles [20, 21, 22]. As a
result, an optically levitated system is naturally isolated from environmental
disturbances. In particular, it possess extremely low damping in high vacuum
and thus has an ultrahigh mechanical quality factor as an excellent
optomechanical system. Hence, this system has attracted abroad attention and
provides a powerful platform for precision measurements [13, 14] and
fundamental physics investigations [9, 10, 11, 12]. The center-of-mass (CoM)
motion of an optically levitated nanoparticle has been cooled to the quantum
ground state [6, 7, 8], and can be used to generate non-Gaussian macroscopic
quantum states in future. Besides CoM motion, libration [23], rotation [24],
and their coupling with internal degrees of freedoms (e.g., phonons, magnons,
spin defects) of the levitated nanoparticle also provide rich physics to
explore. In particular, the free rotation of a rigid body exhibits fascinating
behaviors in both classical and quantum regimes due to its non-linear
dynamics. We can now drive a single levitated nanoparticle to rotate at GHz
frequencies [25, 26, 27, 28], control its rotation with ultra-high precision
[29, 24], and cool its librations to sub-kelvin temperatures [30, 31].

Figure 1: Experimental setup and an image of an optically-levitated
nanoparticle array. (a) The 2D trap array is produced by passing a 1064 nm
laser through a pair of orthogonal acousto-optic deflectors (AOD). The laser
beams created by the AOD are imaged with a 4f image system onto a high NA
(NA=0.95) objective lens, which creates an array of tightly focused optical
tweezers in a vacuum chamber. The optical tweezers propagate against gravity.
For particle assembly, we use an auxiliary moving tweezer at 1064 nm
superimposed on the trap array with a polarized beam splitter (PBS1). This
auxiliary beam is deflected by a motor-driven reflective mirror (MRM1). For
particle motion detection, we overlap a probe beam at 532 nm with the trapping
beams using a dichroic mirror (DM1). This 532 nm beam is deflected by another
motor-driven reflective mirror (MRM2) to measure the motion of an arbitrary
particle in the array. The 1064 nm trap beams and the 532 nm probe beam after
the high NA objective are collimated by an aspherical lens (AL) and seperated
by another dichroic mirror (DM2) for detecting the motion of trapped
nanoparticles. For particle imaging, an imaging beam at 532 nm is used to
illuminate the particles orthogonally. The scattered light is collected by the
same high NA objective to form an image on an charge-coupled device (CCD).
$\lambda/2$: half-wave plate; L1-4:spherical lens; BD: beam dump. (b) A
simulated trapping potential of a $3\times 3$ two-dimensional (2D) array of
optical tweezers. (c) An image of a $3\times 3$ array of optically levitated
nanoparticles.

In this work, we report the creation, detection, and rearrangement of an array
of optically levitated nanoparticles in vacuum. Ultracold atoms and molecules
in an array of optical tweezers in vacuum have provide a versatile platform
for large scale quantum simulation and quantum computing [1, 2, 3, 4, 5].
There are also significant progress in optical manipulation of multiple
microparticles and nanoparticles in a liquid [32, 33, 34]. However, creating a
reconfigurable array of optically levitated nanoparticles in vacuum is still
challenging due to the difficulty of loading multiple optical tweezers
simultaneously and the lack of damping in vacuum to stabilize the system. Here
we employ a high NA objective lens to tightly focus laser beams in the
vertical direction against the gravity to create an array of optical tweezers.
In this layout, the tightly focused optical tweezers provide large gradient
forces for trapping and the gravity helps compensate the scattering force and
photophoretic force to stabilize the system. As a result, our optical tweezers
array can levitate silica nanoparticles at low pressures without feedback
cooling. In addition, we use an independent detection laser beam to monitor
the motion of each nanoparticle in the array. A control laser beam is used to
move and rearrange nanoparticles to desired patterns. As an application of
this technique, we choose two nanoparticles without rotational signals in an
array to synthesize a silica nanodumbbell in-situ, and drive the created
nanodumbbell to rotate at GHz frequencies. On-demand assembly of optically
levitated nanoparticle arrays in vacuum will be important for creating
macroscopic quantum entanglement [9, 10, 11, 12] and studying complex phases
of interacting systems [16, 17, 19, 18, 15].

Figure 2:  Rearranging the pattern of a nanoparticle array. (a) The initial
pattern of 8 levitated nanoparticles in a $4\times 4$ array of optical
tweezers. (b)-(c) Rearranged patterns of the nanoparticle array.

The experimental protocol is illustrated in Fig. 1(a). A two-dimensional (2D)
array of 1064 nm laser beams is created by a pair of orthogonal acousto-optic
deflectors (AOD) driven by a multitone radio-frequency (RF) signal. The
resulting beams are focused by a NA=0.95 objective lens to create an array of
optical tweezers in a vacuum chamber. The optical tweezers propagate along
vertical direction against the gravity. The power of each trapping beam is
about 200 mW. More details can be found in Methods. Fig. 1(b) shows a
simulated trapping potential of a $3\times 3$ array of optical tweezers for
single 170 nm nanoparticles. An auxiliary 1064 nm laser controlled by a motor-
driven reflective mirror is used to rearrange nanoparticles into arbitrary
patterns on demand. In addition, we use a 532 nm detection beam to measure the
motion of trapped nanoparticles and a 532 nm illuminating beam for imaging. An
photo of 9 nanoparticles trapped in an $3\times 3$ array is shown in Fig.
1(c). The horizontal distance of two adjacent columns is 1.77 $\mu$m and the
vertical distance of two adjacent rows is 2.66 $\mu$m.

Figure 3: Characterization of each nanoparticle in an optically-levitated
nanoparticle array. (a) The power spectra of the CoM and torsional motions for
the $3\times 3$ trapped nanoparticles as shown in Fig. 1(c) at 2000 Pa. The
blue, red, and green traces correspond to the power spectra of the CoM motions
along the $z$ axis, $x$ axis, and $y$ axis. The pink traces are the power
spectra of torsional motions. (b) The damping rates of the CoM motions for the
9 trapped nanoparticles as a function of the pressure. The dots are measured
data and the solid lines are linear fittings.

We can use the auxiliary trapping beam to rearrange the pattern of an array.
By controlling the orientation of this laser, we can transport and remove the
nanoparticles at the near atmospheric pressures with almost perfect success
probabilities. Fig. 2 shows an example of such on-demand rearrangement. First,
we load the nanoparticles into a $4\times 4$ array of optical tweezers as
shown in Fig. 2(a). We can see the initial filling of the array is
probabilistic. Then we perform the rearrangement procedure to construct
arbitrary and flexible patterns as shown in Fig. 2(b) and (c).

Figure 4: In-situ synthesis of a nanodumbbell by merging two optically
levitated nanoparticles. (a) An image of two separated nanoparticles in two
optical tweezers. (b) An image of the assembled nanoparticle. (c), (d) and
(e), (f) CoM motion signals and the corresponding damping rates as functions
of the pressures for nanoparticle 1 and 2 respectively. (g) and (h) CoM motion
signals and the corresponding damping rates as a function of the pressure for
the assembled nanodumbbell. For (c)-(h), the trapping laser beams are linearly
polarized. (i) Rotational frequency of the nanodumbbell as a function of the
pressure. (j) Rotational signal of the assembled nanodumbbell at 0.06 Pa. For
(i) and (j), the trapping laser beam is circularly polarized.

To exactly characterize each optically-levitated nanoparticle in the array, we
use a probe beam to measure CoM and torsional motion of each nanoparticle. The
CoM and torsional motion signals for each nanoparticle in an $3\times 3$ array
[Fig. 1(c)] at 2000 Pa are shown in Fig. 3(a). We find that nanoparticles
labeled as “1”, “2”, “3”, “7”, and “9” have no signal of torsional motion,
which implicates these nanoparticles are near spherical. However, the
nanoparticles labeled as “4”, “5”, “6”, and “8” display torsional motion
signals, indicating anisotropy in their shapes. Note that a linearly polarized
trapping laser is used here to generate torsional motion of the trapped
nanoparticle. We can model the nanoparticle as an ellipsoid with different
sizes ($r_{1}>r_{2}>r_{3}$) along three major axes. The longest axis with size
$r_{1}$ of the nanoparticle will tend to align with the electric field of the
linearly polarized laser, which is defined as the $x$ axis. This is because
the polarizability of the nanoparticle along its longest size axis is the
largest [26]. The trapping frequencies of the CoM motion depend on the
polarization of the trapping laser [35]. The smaller radial trapping frequency
is along the electric field of the linear polarized laser ($x$ axis).

The damping rate $\gamma_{i}$ of the CoM motion for a nanoparticle depends on
the air pressure $p$ and its shape ($r_{i}$, $i\in\\{x,y,z\\}$). It is
proportional to $p$ at low pressures when the mean free path of air molecules
is much larger than the size of the particle. A larger size in one direction
will lead to a smaller damping rate of the CoM motion along that direction
[36, 26]. Therefore, we can estimate the shape of the nanoparticle via the
measured damping rates. Fig. 3(b) shows the damping rates of the CoM motion
along three orthogonal directions as a function of the pressure. When the
damping rates of the CoM motion along three orthogonal directions are almost
the same (the curves labeled “1”, “2”, “3”, “7”, and “9” in Fig. 3(b)), we can
infer that the shape of the nanoparticle is near spherical, which confirms the
observation of no torsional signal for these nanoparticles. In contrast, the
damping rate ratios $\gamma_{y}/\gamma_{x}$ and $\gamma_{z}/\gamma_{x}$ of CoM
motion are large for “4”, “5”, “6”, and “8” nanoparticles, which means these
nanoparticles are anisotropic. This agrees with the observation of torsional
motion for these nanoparticles in a linearly polarized laser.

After we obtain an optically-levitated nanoparticle array, we select two
nanoparticles without torsional signals from the array to synthesize a
nanodumbbell that supports fast rotation. Fig. 4(a) and (b) illustrate the
assembling process. First we choose two nanoparticles from an array, which
have no torsional motion signals as shown in Fig. 4(c)-(f). Then we control
the auxiliary trapping beam to move one of two nanoparticles into the same
trap. Two nanoparticles will stick together to become a nanodumbbell with
25$\%$ probability of success in a single trap. There are several other
different situations for the two nanoparticles merged to a single trap. For
example, the nanoparticles may be lost from the trap. The two nanoparticles
may also remain separated in a trap, which may be due to the repulsive Coulomb
force between them if they have the same sign of charges. These situations are
discussed further in the supplementary material.

Figure 4(b) shows the image of the assembled nanodumbbell, whose intensity is
larger than the individual nanoparticles. The CoM motion signals and the
torsional motion signal are measured simultaneously, as shown in Fig. 4(g).
There is almost no change in the trapping frequencies of the CoM motion after
assembly. The damping rates of the assembled nanodumbbell at different
pressures are shown in Fig. 4(h). It shows a larger difference in damping
rates for the CoM motion along three orthogonal directions and illustrates the
anisotropic shape of the nanodumbbell. When we change the polarization of the
trapping laser from linear to circular, the nanodumbbell is driven to rotate.
The rotation frequency as a function of the pressure is shown in Fig. 4(i). A
maximum rotation frequency of about 1.75 GHz at 0.06 Pa is observed (Fig.
4(j)). By further feedback cooling of CoM, the higher rotation frequency can
be reached [28]

In conclusion, we have experimentally realized a 2D array of optically
levitated nanoparticles in vacuum. The initial loading of the array is
probabilistic, whereas the rearrangement procedure allows us to create defect-
free arrays with high fidelity and construct flexible nanoparticle patterns on
demand. By measuring the motion information, we can characterize each trapped
nanoparticle, especially the anisotropic shape of the nanoparticle. As a solid
application, we choose two nanoparticles without the rotational signals from
the array to synthesize a nanodumbbell by moving the two nanoparticles into a
trap. This work opens up a variety of opportunities, ranging from the assembly
of complex systems with different types of nanoparticles to precision
measurements [13, 14]. By applying cooling techniques [19] and utilizing
optical binding [16, 17, 18], this system can be used to explore the many-body
macroscopic quantum physics with interacting nanoparticles [9, 10, 11, 12,
15].

## References

  * Barredo  _et al._ [2016] D. Barredo, S. de Léséleuc, V. Lienhard, T. Lahaye, and A. Browaeys, An atom-by-atom assembler of defect-free arbitrary two-dimensional atomic arrays, Science 354, 1021 (2016).
  * Endres  _et al._ [2016] M. Endres, H. Bernien, A. Keesling, H. Levine, E. R. Anschuetz, A. Krajenbrink, C. Senko, V. Vuletic, M. Greiner, and M. D. Lukin, Atom-by-atom assembly of defect-free one-dimensional cold atom arrays, Science 354, 1024 (2016).
  * Norcia  _et al._ [2019] M. A. Norcia, A. W. Young, W. J. Eckner, E. Oelker, J. Ye, and A. M. Kaufman, Seconds-scale coherence on an optical clock transition in a tweezer array, Science 366, 93 (2019).
  * Norcia  _et al._ [2018] M. A. Norcia, A. W. Young, and A. M. Kaufman, Microscopic control and detection of ultracold strontium in optical-tweezer arrays, Phys. Rev. X 8, 041054 (2018).
  * Anderegg  _et al._ [2019] L. Anderegg, L. W. Cheuk, Y. Bao, S. Burchesky, W. Ketterle, K.-K. Ni, and J. M. Doyle, An optical tweezer array of ultracold molecules, Science 365, 1156 (2019).
  * Delić  _et al._ [2020] U. Delić, M. Reisenbauer, K. Dare, D. Grass, V. Vuletić, N. Kiesel, and M. Aspelmeyer, Cooling of a levitated nanoparticle to the motional quantum ground state, Science 367, 892 (2020).
  * Magrini  _et al._ [2021] L. Magrini, P. Rosenzweig, C. Bach, A. Deutschmann-Olek, S. G. Hofer, S. Hong, N. Kiesel, A. Kugi, and M. Aspelmeyer, Real-time optimal quantum control of mechanical motion at room temperature, Nature 595, 373–377 (2021).
  * Tebbenjohanns  _et al._ [2021] F. Tebbenjohanns, M. L. Mattana, M. Rossi, M. Frimmer, and L. Novotny, Quantum control of a nanoparticle optically levitated in cryogenic free space, Nature 595, 378–382 (2021).
  * Li  _et al._ [2015] J. Li, I. M. Haghighi, N. Malossi, S. Zippilli, and D. Vitali, Generation and detection of large and robust entanglement between two different mechanical resonators in cavity optomechanics, New Journal of Physics 17, 103037 (2015).
  * Zhang  _et al._ [2017] J. Zhang, T. Zhang, and J. Li, Probing spontaneous wave-function collapse with entangled levitating nanospheres, Phys. Rev. A 95, 012141 (2017).
  * Bassi  _et al._ [2013] A. Bassi, K. Lochan, S. Satin, T. P. Singh, and H. Ulbricht, Models of wave-function collapse, underlying theories, and experimental tests, Rev. Mod. Phys. 85, 471 (2013).
  * Weiss and Romero-Isart [2019] T. Weiss and O. Romero-Isart, Quantum motional state tomography with nonquadratic potentials and neural networks, Phys. Rev. Research 1, 033157 (2019).
  * Geraci  _et al._ [2010] A. A. Geraci, S. B. Papp, and J. Kitching, Short-range force detection using optically cooled levitated microspheres, Phys. Rev. Lett. 105, 101101 (2010).
  * Carney  _et al._ [2021] D. Carney, G. Krnjaic, D. C. Moore, C. A. Regal, G. Afek, S. Bhave, B. Brubaker, T. Corbitt, J. Cripe, N. Crisosto, A. Geraci, S. Ghosh, J. G. E. Harris, A. Hook, E. W. Kolb, J. Kunjummen, R. F. Lang, T. Li, T. Lin, Z. Liu, J. Lykken, L. Magrini, J. Manley, N. Matsumoto, A. Monte, F. Monteiro, T. Purdy, C. J. Riedel, R. Singh, S. Singh, K. Sinha, J. M. Taylor, J. Qin, D. J. Wilson, and Y. Zhao, Mechanical quantum sensing in the search for dark matter, Quantum Science and Technology 6, 024002 (2021).
  * Liu  _et al._ [2020] S. Liu, Z.-q. Yin, and T. Li, Prethermalization and nonreciprocal phonon transport in a levitated optomechanical array, Advanced Quantum Technologies 3, 1900099 (2020).
  * Arita  _et al._ [2018] Y. Arita, E. M. Wright, and K. Dholakia, Optical binding of two cooled micro-gyroscopes levitated in vacuum, Optica 5, 910 (2018).
  * Svak  _et al._ [2021] V. Svak, J. Flajšmanová, L. Chvátal, M. Šiler, A. Jonáš, J. Ježek, S. H. Simpson, P. Zemánek, and O. Brzobohatý, Stochastic dynamics of optically bound matter levitated in vacuum, Optica 8, 220 (2021).
  * Rieser  _et al._ [2022] J. Rieser, M. A. Ciampini, H. Rudolph, N. Kiesel, K. Hornberger, B. A. Stickler, M. Aspelmeyer, and U. Delić, Observation of strong and tunable light-induced dipole-dipole interactions between optically levitated nanoparticles (2022).
  * Vijayan  _et al._ [2022] J. Vijayan, Z. Zhang, J. Piotrowski, D. Windey, F. van der Laan, M. Frimmer, and L. Novotny, Scalable all-optical cold damping of levitated nanoparticles (2022).
  * Yin  _et al._ [2013] Z.-Q. Yin, A. A. Geraci, and T. Li, Optomechanics of levitated dielectric particles, Int. J. Mod. Phys. B 27, 1330018 (2013).
  * Millen  _et al._ [2020] J. Millen, T. S. Monteiro, R. Pettit, and A. N. Vamivakas, Optomechanics with levitated particles, Rep. Prog. Phys. 83, 026401 (2020).
  * Gonzalez-Ballestero  _et al._ [2021] C. Gonzalez-Ballestero, M. Aspelmeyer, L. Novotny, R. Quidant, and O. Romero-Isart, Levitodynamics: Levitation and control of microscopic objects in vacuum, Science 374, eabg3027 (2021).
  * Hoang  _et al._ [2016] T. M. Hoang, Y. Ma, J. Ahn, J. Bang, F. Robicheaux, Z.-Q. Yin, and T. Li, Torsional optomechanics of a levitated nonspherical nanoparticle, Phys. Rev. Lett. 117, 123604 (2016).
  * Monteiro  _et al._ [2018] F. Monteiro, S. Ghosh, E. C. van Assendelft, and D. C. Moore, Optical rotation of levitated spheres in high vacuum, Phys. Rev. A 97, 051802 (2018).
  * Reimann  _et al._ [2018] R. Reimann, M. Doderer, E. Hebestreit, R. Diehl, M. Frimmer, D. Windey, F. Tebbenjohanns, and L. Novotny, Ghz rotation of an optically trapped nanoparticle in vacuum, Phys. Rev. Lett. 121, 033602 (2018).
  * Ahn  _et al._ [2018] J. Ahn, Z. Xu, J. Bang, Y.-H. Deng, T. M. Hoang, Q. Han, R.-M. Ma, and T. Li, Optically levitated nanodumbbell torsion balance and ghz nanomechanical rotor, Phys. Rev. Lett. 121, 033603 (2018).
  * Ahn  _et al._ [2020] J. Ahn, Z. Xu, J. Bang, P. Ju, X. Gao, and T. Li, Ultrasensitive torque detection with an optically levitated nanorotor, Nat. Nanotechnol. 15, 89 (2020).
  * Jin  _et al._ [2021a] Y. Jin, J. Yan, S. J. Rahman, J. Li, X. Yu, and J. Zhang, 6 ghz hyperfast rotation of an optically levitated nanoparticle in vacuum, Photon. Res. 9, 1344 (2021a).
  * Kuhn  _et al._ [2017] S. Kuhn, A. Kosloff, B. A. Stickler, F. Patolsky, K. Hornberger, M. Arndt, and J. Millen, Full rotational control of levitated silicon nanorods, Optica 4, 356 (2017).
  * Bang  _et al._ [2020] J. Bang, T. Seberson, P. Ju, J. Ahn, Z. Xu, X. Gao, F. Robicheaux, and T. Li, Five-dimensional cooling and nonlinear dynamics of an optically levitated nanodumbbell, Phys. Rev. Research 2, 043054 (2020).
  * van der Laan  _et al._ [2021] F. van der Laan, F. Tebbenjohanns, R. Reimann, J. Vijayan, L. Novotny, and M. Frimmer, Sub-kelvin feedback cooling and heating dynamics of an optically levitated librator, Phys. Rev. Lett. 127, 123605 (2021).
  * Sasaki  _et al._ [1991] K. Sasaki, M. Koshioka, H. Misawa, N. Kitamura, and H. Masuhara, Pattern formation and flow control of fine particles by laser-scanning micromanipulation, Opt. Lett. 16, 1463 (1991).
  * Dufresne  _et al._ [2001] E. R. Dufresne, G. C. Spalding, M. T. Dearing, S. A. Sheets, and D. G. Grier, Computer-generated holographic optical tweezer arrays, Review of Scientific Instruments 72, 1810 (2001), https://aip.scitation.org/doi/pdf/10.1063/1.1344176 .
  * MacDonald  _et al._ [2002] M. P. MacDonald, L. Paterson, K. Volke-Sepulveda, J. Arlt, W. Sibbett, and K. Dholakia, Creation and manipulation of three-dimensional optically trapped structures, Science 296, 1101 (2002), https://www.science.org/doi/pdf/10.1126/science.1069571 .
  * Jin  _et al._ [2019] Y. Jin, X. Yu, and J. Zhang, Polarization-dependent center-of-mass motion of an optically levitated nanosphere, J. Opt. Soc. Am. B 36, 2369 (2019).
  * Jin  _et al._ [2018] Y. Jin, X. Yu, and J. Zhang, Optically levitated nanosphere with high trapping frequency, Sci. China Phys. Mech. Astron. 61, 114221 (2018).
  * Jin  _et al._ [2021b] Y. Jin, J. Yan, S. J. Rahman, X. Yu, and J. Zhang, Imaging the dipole scattering of an optically levitated dielectric nanoparticle, Applied Physics Letters 119, 021106 (2021b), https://doi.org/10.1063/5.0053008 .
  * Jin  _et al._ [2022] Y. Jin, J. Yan, S. J. Rahman, X. Yu, and J. Zhang, Interference of the scattered vector light fields from two optically levitated nanoparticles, Opt. Express 30, 20026 (2022).

## Methods

Our scheme to create an optically levitated nanoparticle array in vacuum is
shown in Fig. 1 in the main text. A 2D laser beam array is created by passing
a 1064 nm laser through a pair of orthogonal acousto-optic deflectors (AOD)
driven by a multitone radio-frequency (RF) signal. The resulting beam array is
imaged with a 1:1 telescope onto a high NA (NA=0.95) objective lens, which
creates an array of tightly focused optical tweezers in a vacuum chamber. An
auxiliary 1064 nm laser with the orthogonal linear polarization is combined
with the trapping beam array with a polarizing beam splitter (PBS1) and
focused by the same objective. The orientation of this beam is controlled by a
motor-driven reflective mirror to rearrange nanoparticles trapped by the
optical tweezers array into any patterns. In order to obtain motion
information of the trapped nanoparticles, a 532 nm probe beam is combined with
the trapping beams by a dichroic mirror (DM1) before the vacuum chamber. The
orientation of this 532 nm beam can be adjusted to make it focus on an
arbitrary particle in the array. The strongly focused 1064 nm trapping laser
beams and the 532 nm probe beam are collimated by an aspherical lens with
NA=0.78. The output 532 nm laser and 1064 nm laser beams from the vacuum
chamber are separated by another diachroic mirror (DM2). The 532 nm probe beam
is then split into two parts for detecting the center-of-mass (CoM) motion,
and rotation or torsional motion of trapped nanoparticles. When measure the
high-speed rotation at low pressure, we directly use the 1064 nm trapping
laser to detection the rotation signal.

Commercial silica nanoparticles with a nominal diameter of 170 nm are
utilized. The monodisperse nanoparticles are dispersed into the vacuum chamber
by an ultrasonic nebulizer and trapped by the optical tweezer array with each
beam at about 200 mW. In order to image the optically-levitated nanoparticle
array, another 532 nm beam is used to illuminate those particles orthogonal to
the optical axis of the high NA objective lens. The scattering light is
collected by the same high NA objective to form an image on a charge-coupled
device (CCD). This configuration provides a dark background and a high signal-
noise ratio for imaging levitated nanoparticles [37, 38]. The scattering 532
nm light intensity shown in the image provides preliminary information about
the size of each trapped nanoparticle. The spatial distance among the
nanoparticles can be precisely measured according to the interference fringes
of the scattered 532 nm lights from the particles. After the loading
procedure, we start a vacuum pump to evacuate air from the vacuum chamber.

## CODE AVAILABILITY

The custom codes that support the findings of this study are available from
the corresponding author upon reasonable request.

## ACKNOWLEDGMENTS

This research was supported in part by National Natural Science Foundation of
China (NSFC) (Grant No. 61975101, 11234008, 11361161002, 61571276).

## AUTHOR CONTRIBUTIONS

J.Z. designed and supervised the project. J.Y., X.Y. and J.Z. performed the
experiments. All authors analysed the data and discussed the results. J.Z.,
J.Y., and T.L. wrote the manuscript. All authors interpreted the results and
reviewed the manuscript.