# Distributed Beamforming Techniques for Cell-Free Wireless Networks Using
Deep Reinforcement Learning

Firas Fredj, Yasser Al-Eryani, Setareh Maghsudi, Mohamed Akrout,
and Ekram Hossain, Fellow, IEEE F. Fredj, Y. Al-Eryani, M. Akrout, and E.
Hossain are with the Department of Electrical and Computer Engineering at the
University of Manitoba, Canada (emails<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Ekram.Hossain@umanitoba.ca).
S. Maghsudi is with the Department of Computer Science, University of
Tübingen, Germany (email: setareh.maghsudi@uni-tuebingen.de). This work was
supported in part by a Discovery Grant from the Natural Sciences and
Engineering Research Council of Canada (NSERC) and in part by Grant 01IS20051
from the German Federal Ministry of Education and Research (BMBF).

###### Abstract

In a cell-free network, a large number of mobile devices are served
simultaneously by several base stations (BSs)/access points(APs) using the
same time/frequency resources. However, this creates high signal processing
demands (e.g. for beamforming) at the transmitters and receivers. In this
work, we develop centralized and distributed deep reinforcement learning
(DRL)-based methods to optimize beamforming at the uplink of a cell-free
network. First, we propose a fully centralized uplink beamforming method (i.e.
centralized learning) that uses the Deep Deterministic Policy Gradient
algorithm (DDPG) for an offline-trained DRL model. We then enhance this
method, in terms of convergence and performance, by using distributed
experiences collected from different APs based on the Distributed
Distributional Deterministic Policy Gradients algorithm (D4PG) in which the
APs represent the distributed agents of the DRL model. To reduce the
complexity of signal processing at the central processing unit (CPU), we
propose a fully distributed DRL-based uplink beamforming scheme. This scheme
divides the beamforming computations among distributed APs. The proposed
schemes are then benchmarked against two common linear beamforming schemes,
namely, minimum mean square estimation (MMSE) and the simplified conjugate
symmetric schemes. The results show that the D4PG scheme with distributed
experience achieves the best performance irrespective of the network size.
Furthermore, although the proposed distributed beamforming technique reduces
the complexity of centralized learning in the DDPG algorithm, it performs
better than the DDPG algorithm only for small-scale networks. The performance
superiority of the fully centralized DDPG model becomes more evident as the
number of APs and/or UEs increases. The codes for all of our DRL
implementations are available at
https://github.com/RayRedd/Distributed_beamforming_rl

###### Index Terms:

Cell-free network, beamforming, successive interference cancellation, deep
reinforcement learning (DRL), deep deterministic policy gradient algorithm
(DDPG), distributed distributional deterministic policy gradients algorithm
(D4PG).

##  I Introduction

###  I-A Background

To provide ultra-reliable low-latency communications (URLLC) [1] in the beyond
5G wireless systems (B5G), the idea of cell-free networks has emerged [2]. A
fully centralized cell-free network will use a fully connected wireless
network architecture with centralized processing, control, and storage of
data. Precisely, in a cell-free wireless network (Fig. 1), all access points
(APs)/base stations (BSs) cooperate to simultaneously serve all user
equipments (UEs) within the network coverage area [3, 4, 5]. Such centralized
network operations mitigate the adverse effects of non-coordinated collisions
and interference among transmitted signals, especially in scenarios such as
massive machine-type communications (mMTC) [6]. Furthermore, in a cell-free
architecture, fast fronthaul/backhaul links connect all APs to an edge cloud
processor which is responsible for simultaneous downlink (uplink) beamforming
design for transmit(receive) signals to(from) different UEs [7, 8]. On the
downside, a fully centralized cell-free network architecture requires a huge
computational capacity. Moreover, without an efficient design, there would be
excessive control signaling [9, 10]. Note that the main benefits of fully
centralized cell-free networks include enhanced coverage, improved diversity,
and provisioning of efficient interference cancellation mechanisms.

Figure 1: A cell-free network model.

The challenges associated with cell-free networking such as multi-UE joint
beamforming and channel estimation [11, 12, 13, 14] can be addressed by using
artificial intelligence (AI), specifically, machine learning (ML) techniques.
Due to the computational complexity, these problems are often characterized by
an algorithmic deficit rather than a modeling deficit [15]. Among numerous
methods, deep reinforcement learning (DRL) is a notable candidate to design
cell-free networks that avoids having a training data sets a priori which are
hard to obtain for dynamic wireless environments. Also, DRL enables us to
achieve a trade-off between centralized- and distributed processing of
computational tasks. Recent results have shown that, for a cell-free network,
simultaneous uplink/downlink beamforming within a centralized processing unit
(CPU) in optimal system performance. On the downside, fully centralized
processing suffers from high computational complexity and excessive processing
delay, especially when many UEs utilize the same time-frequency resources.
However, as the signal processing tasks of an uplink/downlink cell-free
network becomes more distributed, the network performance becomes closer to
that of a cellular network with non-cooperative APs.

The state-of-the-art of cell-free networks focuses on uplink/downlink
beamforming [3, 8], estimation of channel state information (CSI) [11, 16],
fronthaul imperfections [17], and scalable cell-free network designs [18, 9,
19]. For example, in [20], the authors propose conjugate beamforming and zero-
forcing precoding scheme for a fully centralized downlink cell-free network.
They show that the zero-forcing technique outperforms the conjugate
beamforming technique. In [11], the authors develop a channel estimation
technique for mmWave-enabled massive cell-free network using supervised
learning-based denoising convolutional neural network. To reduce the
complexity of centralized signal processing, [18] proposes a partitioned cell-
free wireless network architecture. The architecture clusters the cooperating
APs based on current network CSI (UE-centric clustering). The scheme enables
an efficient design of practical mMTC systems by compensating the effect of
inter-cluster interference. The compensation is followed by network
partitioning and enabling multi-level successive interference cancellation
(SIC) at each receiver [5]. Another low-complexity design of cell-free network
architecture is presented in [19]. The core idea is to reduce the
dimensionality of beamforming matrices by using a dynamic clustering of APs.
Each cluster then represents a single multi-antenna AP (transmit/receive
diversity). In [13], the authors utilize supervised learning to solve the
beamforming problem in cell-free networks. They include a complete neural
network optimizer in each AP. Every AP then obtains the local CSI knowledge by
estimating only the large-scale fading while considering the small-scale
fading as a constant. Table I summarizes some of the important works in the
area of beamforming in cell-free networks. Here, the system model column
refers to the operational confguration of the APs in a cell-free network (i.e.
how the APs are grouped) as described below:

  * •

Centralized: All of the APs jointly serve the users and the processing is done
centrally in the CPU, which is referred to as the edge cloud processor (ECP).

  * •

Co-located: Each AP operates independently using its own processing
capabilities.

  * •

UE-centric clustering: Each user is assigned to cluster/group of APs.

  * •

Local partial zero forcing: The channel estimation is performed locally at the
APs.

  * •

Static: The APs are considered in fixed locations in the network.

  * •

Dynamic: The APs are grouped into clusters and the clusters are updated at
each time step.

TABLE I: Summary of beamforming schemes in cell-free networks Ref. | System model | Main objective | Techniques and characteristics
---|---|---|---
[21] . |  Centralized . |  Max-min fairness . |  Formulates downlink beamforming problem as a quasi-concave optimization and uses bisection method
[22] . |  Centralized . |  Max-min fairness . |  Designs an angle-of-arrival-based beamforming/combining scheme for FDD-based cell-free network
[3, 8, 23, 24] . . |  Centralized . |  Max-min fairness . |  Use conjugate beamforming and zero forcing techniques. MMSE processing is also used for (partially-) distributed operations of cell-free massive MIMO in [3]
[25] . . . |  Co-located, Cell-free . |  Max-min fairness . . |  Uses Lyapunov optimization techniques to develop a dynamic scheduling algorithm to perform user equipment (UE) scheduling based on time slot and transmission rate
[18, 26] . |  UE-centric clustering |  Maximize per-cluster sum-rate |  First paper solves beamforming problem with optimal CSI, Second paper proposes hybrid analog-digital beamforming
[27] . |  Local partial zero-forcing |  Max-min fairness |  Uses local CSI of the strongest connected UEs at each AP to conduct local zero-forcing beamforming
[28] . |  Static . |  Maximize sum-rate (uplink/downlink) |  Uses minimum-mean-square-error receiver in uplink, uses zero-forcing and regularized zero-forcing beamforming in downlink
[19] . |  Dynamic . |  Maximize sum-rate . |  Uses a hybrid DRL-based model for joint AP clustering and beamforming that utilizes the DDQN and DDPG algorithms
This paper . . . |  Static . . . |  Maximize sum-rate . . . |  The paper uses DDPG and D4PG algorithms for beamforming based on (i) centralized learning, (ii) centralized learning with distributed experiences, and (iii) completely distributed learning.

###  I-B Motivation and Contributions

A fully centralized cell-free architecture enhances the network coverage and
transmission performance. The complexity of signal processing can be reduced
by using scalable, dynamic, and low-complexity designs [19], and also by using
suitable network- and UE-centric architectures and algorithms [18], [29, 7].
The aforementioned low-complexity designs however sacrifice the performance
gain of centralized processing. The complexity of solving the beamforming
problem in a centralized manner (e.g. to obtain the beamforming vectors at the
ECP) can however be reduced by using a distributed learning or processing
approach while the detection of the transmitted data is still performed at the
central unit. Such a solution has not been investigated in the literature.

Several recent research papers have employed deep learning (DL) techniques to
find approximately optimal solution of the beamforming problem in the uplink
of cell-free networks based on centralized learning [30, 31]. However, these
papers use a data-driven supervised-learning approach, which requires a large
amount of training data for many different wireless propagation scenarios.
Such a requirement is a severe limitation for practical implementation of a
cell-free network. Furthermore, all previous DL-based beamforming techniques
utilize the concept of deep Q-learning (DQN) (or deep convolutional neural
network (DCNN)) with a discrete action space. However, DQN is a RL algorithm
that learns the state-value function and is designed to solve tasks with
discrete action space. One way of using DQN on continuous action space is
discretization. However, such a method suffers from the curse of
dimensionality because the action space would grow exponentially with the size
of the network. This results in sub-optimal beamforming designs. Therefore, in
this paper, we propose to utilize DRL methods based on DDPG algorithm that
handles continuous action space for uplink beamforming in a cell-free network.
Specifically, in this paper, we investigate several practical beamforming
designs for uplink cell-free network considering both centralized and
distributed learning settings. The main contributions of this paper can be
summarized as follows:

  * •

For a fully centralized uplink cell-free network, we formulate the optimal
beamforming problem in order to maximize the normalized sum-rate of the
network. We then use a centralized learning approach to realize the uplink
beamforming by using the deep deterministic policy gradient (DDPG) algorithm
[32].

  * •

We also propose a novel distributed experience-based beamforming system using
the distributed distributional deterministic policy gradient (D4PG) algorithm
[33] in which the APs act as the distributed agents.

  * •

To reduce the complexity of centralized learning, we propose a novel DRL-based
beamforming scheme with distributed learning.

  * •

We evaluate the performance of the proposed beamforming designs numerically
for different system settings considering non-orthogonal pilot contamination
and shadow fading.

  * •

We compare the performances of the proposed DRL methods with those of the
traditional conjugate method and MMSE method for beamforming.

The rest of this paper is organized as follows. The system model and the
beamforming problem are presented in Section II. Section III briefly reviews
the preliminaries of DRL and DRL-based beamforming. We design a centralized
DRL solution, a distributed experienced-based DRL solution, and another
beamforming solution based on distributed learning for the beamforming problem
in Section IV. In Section IV-C, we discuss the complexity of all of the DRL-
based solutions. In Section VI, we present and discuss the numerical results.
Section VII concludes the paper. Definitions of the major system model
parameters, DRL models parameters and the abbreviations used in the paper are
given in Table II.

Notations: For a random variable (rv) $X$, $F_{X}(x)$ and $f_{X}(x)$,
respectively, represent cumulative distribution function (CDF) and probability
density function (PDF). Moreover, $\mathbb{E}[\cdot]$ denotes the expectation.
For a given matrix $\bm{A}\in\mathbb{C}^{M\times N}$, $\bm{A}^{H}$ represents
the Hermitian transpose of $\bm{A}$. The PDF of a random variable $X$
following the Nakagami-${m}$ distribution is given by
$f_{X}(x)=\frac{2{m}^{{m}}}{\Gamma({m}){\Omega}^{{m}}}x^{2{m}-1}e^{-\frac{{m}}{\Omega}x^{2}}$.
A random variable $X$ that follows the Gamma distribution is denoted by
$X\thicksim\mathcal{G}(\alpha,\beta)$, with the PDF being
$f_{X}(x)=\frac{\beta^{\alpha}}{\Gamma(\alpha)}x^{\alpha-1}e^{-\beta x},\quad
x>0$, where $\beta>0$, $\alpha\geq 1$, and $\Gamma(z)$ is the Euler’s Gamma
function. Moreover, the distributions $O\mathcal{G}(\alpha_{i},\beta_{i})$,
$i=1,\dots,N$, refer to the $i$-th ascending-ordered rv from a set of $N$
gamma rvs with parameters $\alpha_{i}$ and $\beta_{i}$.

##  II System Model, Assumptions, and Problem Formulation

###  II-A Cell-free Network Model

We consider the uplink of a wireless network with $M$ single-antenna APs and
$K$ single-antenna UEs that have fixed locations within a certain coverage
area, as shown in Fig. 1. Each AP has a baseband processor to partially
process the signal received from all connected UEs. We refer to such an AP as
an “enhanced-AP” (eAP) to distinguish them from the conventional APs. The eAPs
are connected via the backhaul links, hence forming a cell-free network [2].
Such a network architecture enables the distributed eAPs to collaboratively
serve all the UEs within the network coverage area. The beamforming
optimization can be done either in a edge cloud processor (ECP) or in the eAPs
in a distributed manner (Fig. 1). The eAPs are connected to the ECP in a star
network topology. All of the eAPs use the same available spectrum to serve the
same set of UEs. That is, the frequency reuse factor is 1 and all of the eAPs
are allocated the same total channel bandwidth.

TABLE II: Definitions of major system model parameters, DRL model parameters and abbreviations Parameter, |  Definition
---|---
Abbreviation |
System model parameters |
IUI |  Intra-UE Interference
rv |  random variable
$M$ |  Number of eAPs
$K$ |  Number of UEs
eAP |  Enhanced Access Point
ECP |  Edge Cloud Processor
UE |  User Equipment
$\left(\alpha,\beta\right)$ |  parameters of gamma rv
$\mathcal{F}_{mk}$ |  Large-scale fading parameter
$\kappa$ |  Path-loss exponent
$\tau_{c}$ |  Channel coherence time
$\tau_{p}$ |  Duration of pilot symbol
$\bm{\phi}_{k}$ |  Pilot symbol of UE $k$
$\mathcal{E}_{mk}$ |  Pilot estimation factor at link $k\rightarrow m$
$\rho_{k}$ |  $k$-th Pilot transmission power
$p_{k}$ |  $k$-th UE transmission power
$\eta_{z}$ |  Additive White Gaussian Noise (AWGN) at point $z$
$\rho_{z}$ |  AWGN variance at point $z$
$\bm{W}$ |  Beamforming matrix
$\bm{\omega}_{m}$ |  $m$-th row of the beamforming matrix $\mathbf{W}$
DRL parameters |
$Q(s,a)$ |  Action-value function (DDPG critic network)
$\mu(s)$ |  Policy function (DDPG & D4PG actor network)
$Z(s,a)$ |  Action-value distribution function (D4PG critic network)
$\mathcal{S}$ |  State space
$s$ |  Sample state of DRL environment
$\mathcal{A}$ |  Action space
$a$ |  Sample action of actor $\mu(.)$
$r$ |  Reward of DRL environment
$\zeta$ |  Discount factor of DRL model
$\bm{\theta}^{Q}$ |  Weights of $Q(.,.)$
$\bm{\theta}^{\mu}$ |  Weights of $\mu(.)$
$\bm{\theta}^{Z}$ |  Weights of $Z(.,.)$

###  II-B Channel Model

Between the $k\text{-th}$ UE and the $m\text{-th}$ eAP, the channel gain is a
random variable

$g_{mk}=\mathcal{F}^{1/2}_{mk}h_{mk}.$ (1)

In (1), $h_{mk}$ is the small-scale channel fading gain that follows a
Nakagami-$\mathcal{M}_{mk}$ distribution with spreading and shape parameters
$\mathcal{M}_{mk}$ and $\Omega_{mk}$, respectively. Therefore,
$|h_{mk}|^{2}\thicksim\mathcal{G}\left(\alpha_{mk},\beta_{mk}\right)$, where
$\alpha_{mk}=\mathcal{M}_{mk}$ is the shape parameter and
$\beta_{mk}=\frac{\mathcal{M}_{mk}}{\Omega_{mk}}$ is the inverse-scale
parameter. Moreover, we have

$\mathcal{F}_{mk}={L}_{mk}^{-2\kappa}10^{\frac{\sigma_{\text{sh}}z_{mk}}{10}},$
(2)

where $L_{mk}=||d_{mk}||$ is the Euclidean distance between the $k\text{-th}$
UE and the $m$-th eAP. Also, $\kappa$ is the path-loss exponent ($\kappa\geq
2$) and $\sigma_{\text{sh}}$ is the shadow fading variance in dB. Furthermore,
$z_{mk}=\sqrt{\delta}a_{mk}+\sqrt{1-\delta}b_{mk}$, where $0\leq\delta\leq 1$
is the transmitter-receiver shadow fading correlation coefficient [34]. The
parameters $a_{mk}\thicksim\mathcal{N}\left(0,1\right)$ and
$b_{mk}\thicksim\mathcal{N}\left(0,1\right)$ characterize the shadow fading.
We assume that $L_{mk}^{-2\kappa}$ and $\mathcal{F}_{mk}~{}\forall~{}m$, $k$
are known. This assumption is justified since large-scale fading parameters
can be easily estimated given the received signal strength (RSS). Accordingly,
we have
$|g_{mk}|^{2}\thicksim\mathcal{G}(\alpha_{mk},\beta_{mk}/\mathcal{F}_{mk})$.

###  II-C Uplink Network Training

To estimate the CSI of the cell-free network, we train the network using a set
of orthonormal pilot sequences. Lemma 1 presents the MMSE estimation constant
of the channel gain $g_{mk}$ in the presented cell-free network.

######  Lemma 1.

Let
${\bm{\varphi}}_{k}=\left[\varphi_{k,1}\dots\varphi_{k,\tau_{p}}\right]^{H}$,
$||{\bm{\varphi}}_{k}||^{2}=1$, be the pilot sequence with sample size
$\tau_{p}$ that is assigned to the $k$-th UE. The MMSE estimation constant
$\mathcal{E}_{mk}$ of $g_{mk}$ is given by

$\mathcal{E}_{mk}=\frac{\sqrt{\tau_{p}\rho_{k}}\mathcal{F}_{mk}\left(\frac{\alpha_{mk}}{\beta_{mk}}\right)}{\tau_{p}\sum_{l=1}^{K}\rho_{l}\mathcal{F}_{ml}\left(\frac{\alpha_{ml}}{\beta_{ml}}\right)|\bm{\varphi}_{mk}^{H}\bm{\varphi}_{ml}|^{2}+1}.$
(3)

###### Proof.

See Appendix A. ∎

In the above, we use the fact that
$\mathbb{E}\left[|h_{mk}|^{2}\right]=\frac{\alpha_{mk}}{\beta_{mk}}$. If all
of the UEs receive a set of mutually orthogonal pilot sequences (i.e.
$|\bm{\varphi}^{H}_{k}\bm{\varphi}_{l}|=0,~{}\forall~{}k\neq l$), the
estimated small-scale channel fading in (A.2) reduces to a scaled version of
the exact fading gain plus a relatively small AWGN. However, depending on the
applications and due to the limitations concerning the length of the training
sequence, non-orthogonal pilot signals have to be used among some active UEs.
To decrease the computational complexity at the ECP, one approach would be to
estimate the CSI at distributed eAPs, where the $m$-th eAP estimates channel
gains $g_{mk},\forall~{}k=1,\dots,K$.

###  II-D Uplink Data Transmission

In a cell-free network, each eAP receives the composite of signals from all
UEs. For each UE, a weighted sum of composite signals from all eAPs is
constructed to maximize the signal component while minimizing the interference
plus noise. This process takes place at the baseband level in the ECP, before
forwarding the detected signal of each UE to its final destination. Formally,
the overall signal received by the ECP to be used in detecting the $k$-th UE’s
data is given by (4), where $w_{mk}$ is the $m$-th element of the beamforming
vector related to the $k$-th UE such that $0\leq w_{mk}\leq 1$.

$\begin{split}y_{k}&=\sum_{m=1}^{M}w_{mk}\left[\sum_{l=1}^{K}\hat{g}_{ml}\sqrt{p_{l}}x_{l}+{\tilde{\eta}}_{m}\right]\\\
&=\underbrace{\sqrt{\tau_{p}\rho_{k}p_{k}}x_{k}\sum_{m=1}^{M}w_{mk}\mathcal{E}_{mk}g_{mk}}_{\text{Desired
Signal}}+\sum_{m=1}^{M}w_{mk}\left[\underbrace{\sum_{l=1,l\neq
k}^{K}\sqrt{\tau_{p}\rho_{l}p_{l}}x_{l}\mathcal{E}_{ml}g_{ml}}_{\text{Inter-UE
Interference}}\right.\\\ &\;\;\;\;+\left.\underbrace{\sum_{v=1,v\neq
k}^{K}\sqrt{\tau_{p}\rho_{v}p_{k}}x_{k}\mathcal{E}_{mk}|\bm{\varphi}_{k}^{H}\bm{\varphi}_{v}|g_{mv}+\sum_{q=1,q\neq
k}^{K}\sum_{u=1,u\neq
q}^{K}\sqrt{\tau_{p}\rho_{u}p_{q}}x_{q}\mathcal{E}_{mq}|\bm{\varphi}_{q}^{H}\bm{\varphi}_{u}|g_{mu}}_{\text{non-
orthogonal pilot-related estimation error}}\right]\\\
&\;\;\;\;+\sum_{m=1}^{M}w_{mk}\left[\underbrace{\sqrt{p_{k}}\mathcal{E}_{mk}x_{k}|\bm{\varphi}_{k}^{H}\bm{\eta}_{m}|+\sum_{z=1,z\neq
k}^{K}\sqrt{p_{z}}\mathcal{E}_{mz}x_{z}|\bm{\varphi}_{z}^{H}\bm{\eta}_{m}|}_{\text{AWGN-
related estimation
error}}+\underbrace{\tilde{\eta}_{m}}_{\text{AWGN}}\right].\end{split}$ (4)

Moreover, $p_{k}$ is the uplink transmission power of the $k$-th UE such that
$0\leq p_{k}\leq P_{k}$, where $P_{k}$ is the power budget of the $k$-th UE.
Also, $x_{k}$ is the transmitted symbol of the $k$-th UE such that
$\mathbb{E}[|x_{k}|^{2}]=1$, and $\tilde{\eta}_{m}$ is the AWGN at the $m$-th
eAP with $\tilde{\eta}_{m}\thicksim\mathcal{CN}\left(0,1/2\right)$. The
instantaneous signal-to-interference-plus-noise ratio (SINR) for the $k$-th UE
is given by (5) [35]

$\gamma_{k}=\frac{\sum_{m=1}^{M}w_{mk}^{2}|\tilde{g}_{mk}|^{2}}{\sum_{m=1}^{M}w_{mk}^{2}\left[\sum_{l=1,l\neq
k}^{K}|\tilde{g}_{ml}|^{2}+\sum_{v=1,v\neq
k}^{K}|\tilde{g}_{m{v}}|^{2}+\sum_{q=1,q\neq k}^{K}\sum_{u=1,u\neq
q}^{K}|\tilde{g}_{m{u}}|^{2}\right]+1},$ (5)

where
$|\tilde{g}_{mi}|^{2}\thicksim\mathcal{G}\left(\tilde{\alpha}_{mi},\tilde{\beta}_{mi}\right)$,
for $i=k,l,v,u$. Moreover, $\tilde{\alpha}_{mi}=\mathcal{M}_{mi}$, for
$i=k,l,v,u$. In addition,
$\tilde{\beta}_{mi}=\frac{\mathcal{M}_{mi}\dot{\sigma}_{mi}}{\Omega_{mi}\mathcal{F}_{mi}\tau_{p}\rho_{i}p_{i}\mathcal{E}_{mi}^{2}}$,
for $i=k,l$. Also,
$\tilde{\beta}_{mv}=\frac{\mathcal{M}_{mk}\dot{\sigma}_{mk}}{\Omega_{mk}\mathcal{F}_{mvs}\tau_{p}\rho_{v}p_{k}\mathcal{E}_{mk}^{2}|\bm{\varphi}_{k}^{H}\bm{\varphi}_{v}|^{2}}$.
Similarly,
$\tilde{\beta}_{mu}=\frac{\mathcal{M}_{mq}\dot{\sigma}_{mk}}{\Omega_{mq}\mathcal{F}_{mu}\tau_{p}\rho_{u}p_{q}\mathcal{E}_{mq}^{2}|\bm{\varphi}_{q}^{H}\bm{\varphi}_{u}|^{2}}$,
with

$\dot{\sigma}_{mk}=\sum_{m=1}^{M}w_{mk}^{2}\left[\sum_{t=1}^{\tau_{p}}\left(p_{k}\mathcal{E}_{mk}^{2}\varphi_{k,t}^{2}+\sum_{z=1,z\neq
k}^{K}~{}p_{z}\mathcal{E}_{mz}^{2}\varphi_{z,t}^{2}\right)+1\right].$

Equation (5) is concluded from the following: When both the transmitter and
the receiver know the estimated CSI, one can replace the second moments of
channel fading parameters with their instantaneous values. For example, the
numerator of (5) can be written as
$\mathbb{E}\left[|\bm{\mathcal{G}}_{k}^{H}\bm{\mathcal{W}}_{k}x_{k}|^{2}\right]=\bm{\mathcal{W}}_{k}^{H}\bm{\mathcal{R}}_{k}\bm{\mathcal{W}}_{k}$,
where $\bm{\mathcal{G}}_{k}=\left[C_{1k}g_{1k}\dots
C_{M}g_{\mathcal{D}_{1}k}\right]$, $\bm{\mathcal{W}}_{k}=\left[w_{1}\dots
w_{M}\right]$, and
$C_{mk}=\sqrt{\tau_{p}\rho_{k}p_{k}}\mathcal{E}_{mk}G_{mk}$. Moreover,
$\bm{\mathcal{R}}_{k}$ represents the auto-correlation matrix of $k$-th UE’s
signal and is defined as
$\bm{\mathcal{R}}_{k}=\mathbb{E}\left[|\bm{\mathcal{G}}_{k}^{H}\bm{\mathcal{G}}_{k}|^{2}\right]=\bar{\bm{\mathcal{G}}}_{k}\bar{\bm{\mathcal{G}}}_{k}^{H}+{\bm{C}}_{\bm{\mathcal{G}}_{k}}$,
where $\bar{\bm{\mathcal{G}}}_{k}$ and ${\bm{C}}_{\bm{\mathcal{G}}_{k}}$ are
the mean and covariance matrices of $\bm{\mathcal{G}}_{k}$, respectively. If
both the transmitter and the receiver know the instantaneous CSI,
$\bm{\mathcal{R}}_{k}$ yields
$\bm{\mathcal{R}}_{k}=\bm{\mathcal{G}}_{k}\bm{\mathcal{G}}_{k}^{H}$ [35].
Using a similar procedure, one can characterize the interference power
component of (5). Additionally, one can compute the power of AWGN component by
utilizing the fact that all noise samples are i.i.d circularly symmetric
Gaussian rvs with zero-mean and constant variance
$\sigma_{m}^{2}=\sigma^{2}=1/2,~{}\forall~{}m$.

###  II-E Problem Formulation

Since all UEs transmit in the same time-frequency channel, the receiver can
deploy successive interference cancellation (SIC) to increase the UEs’ SINR
values [36], thereby enhancing the per-UE performance. To this end, the
beamforming vector of each UE is designed optimally to separate the signal
component of that UE from other components at least by some value
$\bar{P_{s}}$ referred to as receiver sensitivity. More precisely, to decode
UE $k$, the receiver first decodes the signal components of other UEs with
higher power. It subtracts these components from the overall signal. It then
decodes the desired signal treating the remaining UEs’ signals, i.e. those
with lower power, as interference. Formally, when detecting the signal from
the $k$-th UE, we first arrange the received signal components from the UEs in
an ascending order such that
$\sum_{m=1}^{M}~{}|\check{g}_{m1}|^{2}\leq\dots\leq\sum_{m=1}^{M}|\check{g}_{mk}|^{2}~{}~{}\leq\dots\leq\sum_{m=1}^{M}|\check{g}_{mK}|^{2}$
[18]. The beamforming vector of the $k$-th UE (denoted by
$\bm{w}_{k}=\left[w_{1k}\dots w_{Mk}\right]$) shall maximize an objective
function which is a function of $\gamma_{k},~{}\forall k=1,\dots,K$. For this,
$\gamma_{k}$ in (5) is modified as in (6),

$\gamma_{k}=\frac{\sum_{m=1}^{M}w_{mk}^{2}|\check{g}_{mk}|^{2}}{\sum_{m=1}^{M}w_{mk}^{2}\left[\sum_{l=k+1}^{K}|\check{g}_{ml}|^{2}+\sum_{v=1,v\neq
k}^{K}|\check{g}_{{m}{v}}|^{2}+\sum_{q=1,q\neq k}^{K}\sum_{u=1,u\neq
q}^{K}|\check{g}_{{m}{u}}|^{2}\right]+1}.$ (6)

in which $|\check{g}_{mi}|^{2}\thicksim
O\mathcal{G}(\check{\alpha}_{mi},\check{\beta}_{mi})$ and
$\check{\alpha}_{mi}=\mathcal{M}_{mi}$ for $i=k,l,v,u$. Moreover,
$\check{\beta}_{mi}=\frac{\mathcal{M}_{mi}\dot{\sigma}_{mk}}{\Omega_{mi}\mathcal{F}_{mi}\tau_{p}\rho_{i}p_{i}\mathcal{E}_{mi}^{2}}$
for $i=k,l$. In addition,
$\check{\beta}_{mv}=\frac{\mathcal{M}_{mk}\dot{\sigma}_{mk}}{\Omega_{mk}\mathcal{F}_{mv}\tau_{p}\rho_{v}p_{k}\mathcal{E}_{mk}^{2}|\bm{\varphi}_{k}^{H}\bm{\varphi}_{v}|^{2}}$,
and
$\check{\beta}_{mu}=\frac{\mathcal{M}_{mq}\dot{\sigma}_{mk}}{\Omega_{mq}\mathcal{F}_{mu}\tau_{p}\rho_{u}p_{q}\mathcal{E}_{mq}^{2}|\bm{\varphi}_{q}^{H}\bm{\varphi}_{u}|^{2}}$.
Also, we have

$\dot{\sigma}_{mk}=\sum_{m=1}^{M}w_{mk}^{2}\left[\sum_{t=1}^{\tau_{p}}\left(p_{k}\mathcal{E}_{mk}^{2}\varphi_{k,t}+\sum_{l=1,l\neq
k}^{K}p_{l}\mathcal{E}_{ml}^{2}\varphi_{l,t}\right)+1\right].$

The beamforming problem can be then formulated as

$\displaystyle\textbf{P}_{1}:$
$\displaystyle~{}~{}\underset{\bm{W}\in[0~{}1]^{M\times
K}}{\text{maximize}}~{}\sum_{k=1}^{K}\log_{2}\left(1+\gamma_{k}\right)$ (7)
s.t.
$\displaystyle~{}\textbf{C}_{1}:\sum_{m=1}^{M}\left(w_{m\delta_{l}}^{2}-\sum_{i=\delta_{l}+1}^{l}w_{mi}^{2}\right)\bar{\gamma}_{{m}l}\geq\bar{P_{s}},$
$\displaystyle~{}\textbf{C}_{2}:w_{mk}^{2}\in\left[0,1\right],~{}~{}\forall~{}m,k,$
$\displaystyle~{}~{}\forall k,\forall~{}\delta_{l}=1,\dots,l-1\;\mbox{,
and}\;l=2,\dots,K,$

where $\bar{\gamma}_{{m}l}=p_{l}|{g}_{ml}|^{2}$, and
$\bm{W}\in\left[0,1\right]^{M\times K}$ is the overall beamforming matrix in
which $\bm{w}_{k}=[w_{1k},\dots,w_{Mk}]$. Note that in (7), the objective is a
function of $\gamma_{k}$, which is a function of $\bm{\omega}_{k}$. The
constraint $\textbf{C}_{1}$ corresponds to the
$\sum_{l=2}^{K}(l-1)=\frac{K(K-1)}{2}$ conditions of successful SIC operation
with a receiver sensitivity of $\bar{P_{s}}$. Note that, for maximizing the
minimum transmission rate (max-min fairness), the objective function of the
optimization problem will be
$\underset{k=1,...,K}{\text{minimum}}\log_{2}\left(1+\gamma_{k}\right)$. The
receiver deals only with the measured (estimated) channel values that include
the estimation error and the AWGN component. The SINR value after the SIC
operation decreases due to pilot contamination.

The globally optimal solution of problem $\textbf{P}_{1}$ in (7) is the one
that gives the best performance among all possible matrices $\bm{W}$. However,
$\textbf{P}_{1}$ is a non-convex problem due to the non-convexity of the
objective function. Accordingly, the globally optimal solution of
$\textbf{P}_{1}$ in (7) can only be obtained by an exhaustive search. The
computational complexity of such as exhaustive search will be combinatorial in
terms of the number of UEs and eAPs, and therefore, will not be feasible for
practical implementation. Iterative optimization methods can be used, for
example, at the ECP of the cell-free system, that can converge to locally
optimal solutions after a finite number of iterations. However, this will
require all the CSI information to be transmitted to the ECP resulting in
increased network overhead as well as high processing complexity. DRL-based
solutions based on distributed learning can reduce the network overhead as
well as processing complexity at the ECP. More specifically, centralized
learning with distributed agents’ experience can improve the quality of the
solution, while a completely distributed learning-based scheme will reduce the
signal processing complexity at the ECP and the network overhead
significantly. In the later sections of this paper, we will develop three DRL-
based solutions, namely, the fully centralized training and learning-based
method, the centralized learning-based method with distributed training, and
the fully distributed training and learning-based method, for the formulated
problem. The centralized learning-based method can serve as a benchmark for
the two other methods.

##  III Background on DRL and Beamforming Optimization

###  III-A DRL Preliminaries

DRL is a solution method for Markov decision processes (MDPs). An MDP is
characterized by a tuple
$(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\zeta$), where $\mathcal{S}$
and $\mathcal{A}$ represent the state space and the agent’s action space,
respectively. Moreover, $\mathcal{P}$ is the transition probability matrix,
where $\mathcal{P}(s,a,s^{\prime})\in[0,1]$ is the probability that state $s$
changes to state $s^{\prime}$ by selecting action $a$.
$\mathcal{R}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ defines the
expected reward of performing action $a$ at state $s$. Finally, $\zeta$ is the
reward’s discount factor. The goal is to select the best action at each step
so as to maximize the accumulated discounted reward. In a DRL model, a neural
network learns to map the states to values or state-action to Q-values using
the historical outcomes. DRL algorithms fall into three categories: (i) value-
based methods that aim to learn a value function like deep Q-learning (DQN)
algorithm [37], (ii) policy-based methods that learn the optimal policy
function, and (iii) actor-critic methods that combine value-based and policy-
based methods.

In this paper, we use two actor-critic algorithms, namely, DDPG [32] and D4PG
[33]. These methods are suitable to optimize beamforming, since in both DDPG
and D4PG, the action space can be continuous. Moreover, for D4PG, the
exploration can be distributed among multiple agents. The DDPG algorithm uses
state-action Q-value critic based on deep Q-learning [37] and updates the
policy using its critic gradients. D4PG builds on the DDPG approach by making
several enhancements such as the Q-value estimation and the distributed
collection of experiences. Note that an experience is the process of exploring
a new action by executing it in the environment or simulating it, thereby
receiving some reward and observing the new state.

###  III-B DRL Agent for Beamforming Optimization

For a cell-free network with $M$ APs and $K$ UEs (Fig. 1), we develop a DRL
model to optimize the beamforming matrix $\bm{W}$. This matrix includes
beamforming vectors of all UEs within the network coverage area given the
complete CSI. Note that, since the beamforming vector of the $k$-th UE is
given by $\bm{w}_{k}=\left[w_{1k}\dots w_{Mk}\right]$, the matrix $\bm{W}$ has
a dimension of $M\times K$.

We cast the beamforming optimization problem as an MDP problem. We then train
a DRL model where an agent learns by interacting with a cell-free network as
the environment. Such a DRL model can be implemented either centrally or in a
distributed manner. The design of the environment for a cell-free network
includes the definition of state $\bm{s}$, the action $\bm{a}$, and the
immediate reward function $\bm{r}$, needed for the DRL algorithm to estimate
the policy and the Q-values. To improve the training process, new actions need
to be explored by the agent. Therefore, we add a noise generated from a random
process at each action taken in the training phase. Although originally an
Ornstein–Uhlenbeck (OU) process was proposed for exploration, later results
showed that, an uncorrelated, zero mean Gaussian noise gives the same
performance [38]. Therefore, due to its simplicity of implementation, the
Gaussian noise process is preferred to the OU process for exploration. The
state can be any key performance indicator. While the action of this model is
the optimization variable $\bm{W}$ (the beamforming matrix) of the problem
$\textbf{P}_{1}$, the reward can be any performance metric that jointly
quantifies the performance of all active UEs. In Table III, we summarize the
design parameters of a DRL model and the corresponding measures in the cell-
free network.

TABLE III: Design parameters for the DRL model Environment Variables | System Equivalence
---|---
State $s=\\{s_{1},....,s_{K}\\}$ | UE SINR: $\\{\gamma_{1},...,\gamma_{K}\\}$
Reward $r$ | Sum-rate for all UEs: $\sum_{k=1}^{K}\log_{2}(1+\gamma_{k})$
Action $a$ | Beamforming matrix: $\bm{W}$

##  IV DRL-Based Centralized and Distributed Beamforming Methods

###  IV-A The DDPG Algorithm: A Fully Centralized Solution

In this section, we propose a DRL-based centralized solution for the
beamforming problem in (7) using the DDPG learning algorithm. This solution
serves as a benchmark for the other beamforming techniques in the subsequent
sections. The actor-critic algorithm in DDPG can handle continuous state space
and continuous action space. Since the elements $w_{ij}$ of $\bm{W}$ are
continuous in the range $[0,1]$, to find the optimal beamforming matrix
$\bm{W}\in\left[0,1\right]^{M\times K}$, we use DDPG. It uses two neural
networks as function estimators: (i) the critic, $Q(s,a)$, whose parameters
are $\theta^{Q}$ and calculates the expected return given state $s$ and action
$a$; (ii) the actor, $\mu(s)$, whose parameter is $\theta^{\mu}$ and
determines the policy. In DDPG, the actor directly maps states to actions
instead of outputting a probability distribution across a discrete action
space. The starting point for learning an estimator to $Q^{*}(s,a)$ is the
Bellman equation given by

$Q^{*}(s,a)=\mathbb{E}_{(s,a,r,s^{\prime})\in\mathcal{R}}\left[r(s,a)+\zeta\;\underset{a^{\prime}}{\text{max}}\;Q^{*}(s^{\prime},a^{\prime})\right],$
(8)

where $\mathcal{R}$ is the set of the experiences and $\zeta$ is the discount
factor.

Computing the maximum over actions in the target is quite challenging in the
continuous action spaces. In particular, the training of the Q-value function
involves minimizing the loss function $L(\theta^{Q})$ in equation
(LABEL:eq:q_loss). However, the target value $y_{t}$ that we want our Q-value
to be close to depends on the weights of critic network $Q$. This dependency
will lead to an instability in the learning of critic network $Q$ which will
affect the learning of the policy $\mu$.

$\begin{split}&L(\theta^{Q})=\mathbb{E}_{s_{t}\sim\rho^{\mu},a_{t}\sim\beta,r_{t}\sim
E}\Big{[}(Q(s_{t},a_{t})|\theta^{Q})-y_{t})^{2}\Big{]}.\\\
&y_{t}=r(s_{t},a_{t})+\zeta Q(s_{t+1},\mu(s_{t+1})|\theta^{Q}).\end{split}$
(9)

To mitigate this instability, in [32] a lagged version of the actual critic
network and the actor network are created and updated through the Polyak
averaging at each iteration to enhance the convergence chances. DDPG handles
this coupling and instability by using two target networks, namely, a critic
target network $Q^{\prime}(s,a)$ and a policy target network
$\mu^{\prime}(s)$. These two networks use a set of parameters
$\theta^{Q^{\prime}}$ and $\theta^{\mu^{\prime}}$ updated by Poylak averaging
with factor $\tau$ as shown at (15) and (16) in Algorithm 1. These additional
networks represent a lagged version of the actual critic network and actor
network that are updated at each iteration and they increase the chance of
convergence of the algorithm. Therefore, in Algorithm 1, the critic network
minimizes the loss function presented at (12). In policy learning, DDPG learns
$\theta^{\mu}$ that maximizes $Q$. At every round, it maximizes the expected
return as

$J(\theta^{\mu})=\mathbb{E}\Big{[}Q(s,a)\Big{|}s=s_{t},a=a_{t}\Big{]},$ (10)

and updates the weights $\theta^{\mu}$ by following the gradient of (10)

$\nabla
J_{\theta^{\mu}}(\theta)\approx\nabla_{a}Q(s,a).\nabla\mu(s|\theta^{\mu}).$
(11)

Algorithm 1 DDPG learning process [32] (implemented in the ECP)

1:Randomly initialize the actor network $\mu(s|\theta^{\mu})$ and the critic
network $Q(s,a|\theta^{Q})$ with weights $\theta^{\mu}$, $\theta^{Q}$.

2:Initialize the target networks $Q^{\prime}$ and $\mu^{\prime}$ with weights
$\theta^{Q^{\prime}}\leftarrow\theta^{Q}$ and
$\theta^{\mu^{\prime}}\leftarrow\theta^{\mu}$.

3:Initialize the replay buffer $\mathcal{R}$.

4:for episode $=1\;to\;$ max-number-episodes do

5: Initialize a random process $\mathcal{N}$ for action exploration.

6: Observe the initial state $s_{1}$.

7: for $t=1\;to\;$ max-episode-steps do

8: Perform action $a_{t}=\mu(s_{t})+\mathcal{N}_{t}$. Observe reward $r_{t}$
and the next state $s_{t+1}$.

9: Store the transition $(s_{t},a_{t},r_{t},s_{t+1})$ in $\mathcal{R}$.

10: Let $N$ be the batch size. Sample random mini batch of $N$ transitions
from $\mathcal{R}$.

11: Update the critic by minimizing the loss
$L=\frac{1}{N}\sum_{j}(y_{j}-Q(s_{j},a_{j}|\theta^{Q}))^{2},$ (12)
$y_{j}=r_{j}+\zeta
Q^{\prime}(s_{j+1},\mu^{\prime}(s_{j+1})|\theta^{\mu^{\prime}})|\theta^{Q^{\prime}}).$
(13)

12: Update the actor policy using sampled policy gradient ascent as
$\nabla_{\theta^{\mu}}J\approx\frac{1}{N}\sum_{j}\nabla_{a}Q(s,a|\theta^{Q})|_{s=s_{j},a=\mu(s_{j})}\nabla_{\theta^{\mu}}\mu(s|\theta^{\mu})|_{s=s_{j}}.$
(14)

13: Update the target networks as
$\theta^{Q^{\prime}}\leftarrow\tau\theta^{Q}+(1-\tau)\theta^{Q^{\prime}},$
(15)
$\theta^{\mu^{\prime}}\leftarrow\tau\theta^{\mu}+(1-\tau)\theta^{\mu^{\prime}}.$
(16)

14: end for

15:end for

This update rule represents the Deterministic Policy Gradient theorem [38].
The term $\nabla_{a}\,Q(s,a)$ is obtained from the backpropagation of the
Q-network $Q(s,a|\theta^{Q})$ w.r.t the action input $\mu(s|\theta^{\mu})$.
Algorithm 1 summarizes the DDPG learning process with settings presented in
Table III (i.e. the states $s_{t}$, actions $a_{t}$ and rewards $r_{t}$ are
defined as UE SINR $\\{\gamma_{1},...,\gamma_{K}\\}$, the beamforming matrix
$W$, and the sum-rate $\sum_{k=1}^{K}\log_{2}(1+\gamma_{k})$, respectively, at
the $t$-th learning step).

###  IV-B D4PG-Based Beamforming With Distributed Agents’ Experience

In the previous section, we discussed the fully centralized DDPG-based
beamforming scheme as a replacement of conventional centralized optimization
at the ECP. In this section, we take a step toward distributed beamforming in
cell-free networks. We propose a DRL-based beamforming scheme which exploits
the experiences of the distributed actors (e.g. eAPs) that belong to a
centrally located agent (e.g. ECP). This scheme is based on the distributed
distributional deep deterministic policy gradient (D4PG) algorithm [39].
Generally, the D4PG algorithm applies a set of improvements on DDPG and make
it run in a distributional fashion. These improvements enable D4PG to
outperform DDPG [40]. This is achieved by having a multiple actor neural
networks to gather independent experiences and feed it to a single replay
buffer. However, it contains a single critic network that samples the
independently gathered experiences from the replay buffer to explore a new
Q-value. Traditionally, the independent actors, the replay buffer, and the
critic of the D4PG are implemented within the same operation area/module. In
this paper, we propose an implementation for the D4PG by setting the number of
independent actors to $M$ (equal to the number of eAPs) and placing an actor
at each eAP, and place the replay buffer and the critic network at the ECP. As
illustrated in Fig. 2, the actor networks implemented in the eAPs generate
independent experiences that are sent to the ECP to be pushed into the replay
buffer, which combines all experiences explored by the actor networks. In
Algorithm 3, we describe the exploration technique used in the actor network.
This approach improves the quality of training data since the eAPs
simultaneously generate multiple experiences with different exploration
processes.

Figure 2: D4PG-based beamforming design.

The design parameters of the DRL model for D4PG are represented in Table III,
therefore the states $s_{t}$, actions $a_{t}$ and rewards $r_{t}$ are defined
as the UE SINR $\\{\gamma_{1},...,\gamma_{K}\\}$, the beamforming matrix $W$,
and the sum-rate $\sum_{k=1}^{K}\log_{2}(1+\gamma_{k})$, respectively, at the
$t$-th learning step. D4PG is an actor-critic method that enhances the DDPG
algorithm to perform in a distributed manner, thereby improving the
estimations of the Q-values. Below we discuss these enhancements to the D4PG
algorithm, which we explicitly describe in Algorithm 2.

Algorithm 2 D4PG-Based Beamforming With Distributed Agents’ Experience

1:Input: batch size $P$, trajectory length $N$, number of actors $K$, replay
size $R$, initial learning rates $\alpha_{0}$ and $\beta_{0}$, time for
updating the target networks $t_{target}$, time for updating the actors policy
$t_{actor}$.

2:Initialize actor and critic networks $(\theta^{\mu}$,$\theta^{Z})$ at
random.

3:Initialize target weights
$(\theta^{\mu^{\prime}}$,$\theta^{Z^{\prime}})\leftarrow(\theta^{\mu}$,$\theta^{Z})$.

4:Launch $K$ actors and replicate their weights
$(\theta^{\mu^{\prime}}$,$\theta^{Z^{\prime}})$.

5:for $t=1,\dots,T$ do

6: Sample $P$ transitions $(s_{i:i+N},a_{i:i+N},r_{i:i+N})$ of length $N$ from
replay with priority $p_{i}$.

7: Construct the target distributions
$Y_{i}=\sum_{n=0}^{N-1}\zeta^{n}r_{i+n}+\zeta^{N}Z^{\prime}(s_{i+N},\mu^{\prime}(s_{i+N}|\theta^{\mu^{\prime}})|\theta^{Z^{\prime}})$.

8: Compute the actor and critic updates
$\delta_{\theta^{Z}}=\frac{1}{P}\sum_{i}\nabla_{\theta^{Z}}(R\,p_{i})^{-1}d(Y_{i},Z(s_{i},a_{i}|\theta^{Z})),$
$\delta_{\theta^{\mu}}=\frac{1}{P}\sum_{i}\nabla_{\theta^{\mu}}\mu(s_{i}|\theta^{\mu})\mathbb{E}[\nabla_{a}Z(s_{i},a|\theta^{Z})]|_{a=\mu(s_{i}|\theta^{\mu})}.$

9: Update network parameters
$\theta^{\mu}\leftarrow\theta^{\mu}+\alpha_{t}\;\delta_{\theta^{\mu}}$,
$\theta^{Z}\leftarrow\theta^{Z}+\beta_{t}\;\delta_{\theta^{Z}}$.

10: If $t\,\,\mathbf{mod}\,\,t_{target}=0$, update the target networks
$(\theta^{\mu^{\prime}},\theta^{Z^{\prime}})\leftarrow(\theta^{\mu},\theta^{Z})$.

11: If $t\,\,\mathbf{mod}\,\,t_{actors}=0$, replicate network weights to the
actors.

12:end for

13:Return policy parameters $\theta^{\mu}$.

Algorithm 3 Actor

1:repeat

2: Set $t=0$

3: Initialize random process $\mathcal{N}$ at each actor

4: At step t, select action $a_{t}=\mu(s_{t}|\theta^{\mu})+\mathcal{N}_{t}$.
Receive reward $r_{t}$ and observe state $s_{t}^{\prime}$.

5: Send the experience $(s_{t},a_{t},r_{t},s_{t}^{\prime})$ to ECP to be
stored in the replay buffer.

6:until learner finishes, i.e. Algorithm 2 stops.

The main features of the D4PG algorithm are described below.

####  IV-B1 Distributional Critic

In D4PG, the Q-value is a random variable following some distribution $Z$ with
parameters $\theta^{Z}$, thus
$Q_{\theta^{Z}}(s,a)=\mathbb{E}[Z(s,a|\theta^{Z})]$. The objective function
for learning the distribution minimizes some measure of the distance between
the distribution estimated of the target critic network and that of the critic
network, e.g. the binary cross-entropy loss. Formally,

$L(\theta^{Z})=\mathbb{E}[d(T_{\mu}\;Z^{\prime}(s,a|\theta^{Z^{\prime}}),Z(s,a|\theta^{Z})],$
(17)

where $T_{\mu}$ is the Bellman operator. The deterministic policy gradient
update yields

$\begin{split}\nabla_{\theta^{\mu}}J(\theta^{\mu})&\approx\mathbb{E}_{\rho}[\nabla_{\theta^{\mu}}\mu(s|\theta^{\mu})\nabla_{a}Q(s,a|\theta^{Z})|_{a=\mu(s|\theta^{\mu})}]\\\
&=\mathbb{E}_{\rho}[\nabla_{\theta^{\mu}}\mu(s|\theta^{\mu})\mathbb{E}[\nabla_{a}Z(s,a|\theta^{Z})]|_{a=\mu(s|\theta^{\mu})}].\end{split}$
(18)

####  IV-B2 N-step Returns

An agent in D4PG computes the $N$-step Temporal Difference (TD) target
instead. Formally,

$(T_{\mu}^{N}Q)(s_{0},a_{0})=r(s_{0},a_{0})+\mathbb{E}\left[\sum_{n=1}^{N-1}r(s_{n},a_{n})+\zeta^{N}Q(s_{N},\mu_{\theta}(s_{N}))|s_{0},a_{0}\right].$
(19)

####  IV-B3 Multiple Distributed Parallel Actors

This process takes place in parallel within $K$ actors, each one generating
samples independently. The samples are collected in a replay buffer from which
a learner samples batches to update the weights of the networks.

####  IV-B4 Prioritized Experience Replay (PER)

Finally, D4PG collects $R$ samples from the replay buffer with non-uniform
probability $p_{i}$. The $i$-th sample is selected with priority
$(Rp_{i})^{-1}$ that also indicates the importance of the sample.

###  IV-C DRL-Based Beamforming With Distributed Learning

In the previous section, we proposed a D4PG learning method to train a policy
that predicts the beamforming matrix given a cell-free network environment. We
enhanced the learning performance by allocating an agent per eAP and then
utilizing the distributed experience collected from several agents. The
enhanced method converges faster and shows better performance compared to the
fully centralized DDPG solution. However, the proposed D4PG method still
conducts the learning process at the ECP that involves a large body of the
computational tasks.

In this section, we propose a cell-free beamforming scheme, which, in addition
to distributing the agents’ experiences, splits the learning process of the
DRL among all eAPs. In such a model, the eAPs divide the computational tasks
equally and the ECP only performs limited control and coordination task. In
this scheme, every eAP is responsible to find the optimal beamforming vector
for all UEs. To this end, all vectors of the overall beamforming matrix are
considered to be constants, to be simultaneously found by other eAPs. As an
example, for a cell-free network with $M$ eAPs and $K$ UEs, eAPm is
responsible to optimize
$\bm{\omega}_{m}=\bm{W}\left(m,[1,~{}\dots,~{}k]\right)$. Thus, eAPm solves
the following subproblem:

$\displaystyle\textbf{P}_{m}:$
$\displaystyle~{}\underset{\bm{\omega}_{m}\in\left[0~{}1\right]^{1\times
K}}{\text{maximize}}~{}\sum_{k=1}^{K}\log_{2}\left(1+\gamma_{k}\right)$ (20)
S.t.
$\displaystyle~{}\textbf{C}_{1}:\sum_{m=1}^{M}\left(w_{m\delta_{l}}^{2}-\sum_{i=\delta_{l}+1}^{l}w_{mi}^{2}\right)\bar{\gamma}_{{m}l}\geq\bar{P_{s}},$
$\displaystyle~{}\textbf{C}_{2}:w_{mk}^{2}\in\left[0,1\right],$
$\displaystyle~{}~{}\forall k,\forall~{}\delta_{l}=1,\dots,l-1\;\mbox{,
and}\;l=2,\dots,K.$

In the following, we present a new system design for the cell-free network
environment where the eAPs interact with each other to find the optimal
beamforming vector. We intend to solve the optimization problem in (7) through
multiple eAPs solving the problem in (20). Therefore, as shown in Fig. 3, we
distribute the learning process among the eAPs by implementing a DDPG agent in
each eAP with its local experience buffer, actor network, and critic network.

Figure 3: Example of the proposed distributed learning-based DRL beamforming.

The design parameters of the DRL model are slightly changed compared to Table
III. The states $s_{t}$ and $r_{t}$ are the UE SINR
$\\{\gamma_{1},...,\gamma_{K}\\}$ and sum-rate
$\sum_{k=1}^{K}\log_{2}\left(1+\gamma_{k}\right)$ (as before), but the actions
$a_{t}$ are changed to $\bm{\omega}_{m}$, since the $m$-th eAP learns to
optimize the $m$-th row of the beamforming matrix $\mathbf{W}$. Beamforming is
then optimized as follows: The first step is the DDPG process that includes:
(i) generating experiences, and (ii) updating the network parameters using
Bellman equation (8), and the policy gradient in (11). Each agent learns to
optimize one row of the beamforming matrix. Here, the ECP, which is connected
to all the eAPs through backhaul links, acts as the coordinator to facilitate
sharing of the optimized rows with other agents. At each episode, the eAPs
discover multiple actions. They select the action with the highest reward in
terms of the maximum normalized sum-rate during the episode and send it to the
coordinator. The coordinator receives all the rows from the eAPs, and then
concatenates them to create a new beamforming matrix. Finally, it broadcasts
the new matrix to all eAPs. The number of updates of the beamforming matrix,
i.e. the horizon, can be selected based on the required accuracy. Algorithm 4
summarizes the learning process in each eAP, and Algorithm 5 summarizes the
coordination process among the eAPs by the the ECP.

Algorithm 4 Local DDPG (implemented in each eAP)

1:Initiate the cell-free network environment with random beamforming matrix
$\bm{W}_{\textrm{local}}$.

2:Initialize actor network $\mu(\cdot)$ and critic network $Q(\cdot)$ with
parameters $\theta^{\mu}$ and $\theta^{Q}$, respectively.

3:Initiate the random process $\mathcal{N}$.

4:Initiate $r_{\textrm{max}}=0$.

5:for each episode do

6: Observe initial state $s$ from the cell-free network environment.

7: for each step do

8: Apply $a=\mu(s)+\mathcal{N}_{t}$ on the cell-free network environment and
observe new state $s^{\prime}$ and reward $r$.

9: Store $(s,a,r,s^{\prime})$ in the replay buffer $\mathcal{R}$.

10: Update $\theta^{\mu}$ and $\theta^{Q}$ weights with a batch from
$\mathcal{R}$ using Bellman equation (8) and policy gradient (11).

11: if $r\geq r_{\textrm{max}}$ then

12: $a_{\textrm{optimal}}=a$.

13: $r_{\textrm{max}}=r$.

14: end if

15: end for

16: if update time of beamforming matrix then

17: Send $a_{\textrm{optimal}}$ to the coordinator.

18: Receive $\bm{W}_{\textrm{local}}$ from the coordinator.

19: $\bm{W}_{\textrm{local}}=\bm{W}_{\textrm{new}}$.

20: end if

21:end for

1:repeat

2: $\bm{W}_{\textrm{new}}=0$

3: repeat

4: if $a_{\textrm{optimal}}$ is received from $i$-th eAP then

5: $\bm{W}_{i,\,\textrm{new}}=a_{\textrm{optimal}}$.

6: end if

7: until $a_{\textrm{optimal}}$ is received from every eAP.

8: Broadcast $\bm{W}_{\textrm{new}}$ to all agents.

9:until all agents stop training.

Algorithm 5 Coordinator (implemented in the ECP)

##  V Complexity Analysis and Communication Overhead

As we have mentioned before, the beamforming optimization problem (7) is non-
convex. Conventional approaches to solve a non-convex problem include
iterative algorithms such as exhaustive search, steepest descent, gradient
ascent, and interior-point methods111Note that the objective function in (7)
is twice differentiable.. Considering an exhaustive search, we quantize the
beamforming vector of $k$-UE, i.e. $\bm{\omega}_{k}=\left[w_{1k}\dots
w_{Mk}\right]$, by a certain step size $\Delta$. For the beamforming
optimization problem, this results in a complexity of
$O\left(\left(\frac{1}{\Delta}\right)^{M+K}\right)$. The combinatorial
complexity renders the conventional beamforming techniques impractical, in
particular, for dense cell-free networks. This calls for novel methods such as
DRL-based solutions that retain low complexity while guaranteeing efficiency.

To evaluate the time-complexity of a DNN, the conventional measure is the
floating-point operations per second (FLOPs). For any fully connected layer
$L_{i}$ of input size $I_{i}$ and output size $O_{i}$, the number of FLOPs is
given by

$\text{FLOPS}(L_{i})=2\,I_{i}\,O_{i}.$ (21)

Thus the total number of FLOPS of the DRL-based method during the inference
for a policy with $L$ hidden layers yields:

$\displaystyle\text{FLOPs}_{\text{DRL}}$
$\displaystyle=\sum_{i=1}^{3}\,\text{FLOPs}(L_{i})$ (22)
$\displaystyle=2\cdot\Big{(}H_{1}\cdot\mathcal{|S|}+H_{L}\cdot\mathcal{|A|}+\sum_{i=2}^{L}H_{i-1}\times
H_{i}\Big{)}$ $\displaystyle=2\cdot\Big{(}H_{1}\times K+H_{L}\left(K\times
M\right)+\sum_{i=2}^{L}H_{i-1}\times H_{i}\Big{)},$

where $H_{1}$, $H_{L}$, and $H_{i}$ denote the size of the first hidden layer,
$L$-th hidden layer, and $i$-th hidden layer, respectively.

Table IV compares the order of complexity of inference, as well as the
convergence rate, of the three proposed DRL-based methods. Note that the
policy implemented in this paper has 2 hidden layers of sizes 258 and 128.

TABLE IV: Complexity analysis Model | FLOPS for inference | Convergence
---|---|---
MMSE Solution . |  $\textit{O}\left(\left[M\times K\right]^{2}\right)$ |  Polynomial Convergence
Centralized (DDPG) |  $2\cdot\Big{(}256\times 128$ \+ 256$\cdot{K}$ \+ 128$\cdot{(M\times K)\Big{)}}$ | Quadratic
Distributed Experience (D4PG) |  $2\cdot\Big{(}256\times 128$ \+ 256$\cdot K$ \+ 128$\cdot(M\times K)\Big{)}$ . |  Quadratic .
Distributed Learning |  $2\cdot\Big{(}256\times 128$ \+ 256$\cdot K$ \+ 128$\cdot K\Big{)}$ | Quadratic

The number of FLOPS during the inference is mainly determined by the matrix
multiplications of the policy network, which has four layers with size
$\mathcal{|S|}$, 256, 128, and $\mathcal{|A|}$. The number of FLOPs for a
connected layer is twice the product of the size of the hidden layers. That
is, given a hidden layer of size 128 that has a previous layer of size 256,
the weight matrix should be of size $128\times 256$ and the FLOPs is
$2\times(256\times 128)$. For D4PG, which uses distributed training, the
number of FLOPS is equal to that of DDPG since the feed-forward operation on
the policy network remains identical during the inference. In contrast, for
the fully distributed approach, each of the $M$ eAPs predicts one row of the
beamforming matrix. Therefore, the number of FLOPS is smaller than that of
D4PG with distributed experience and DDPG. In the next section, we will
investigate the performance of DRL models in terms of transmission sum-rate
and the convergence time.

Also, note that, for the D4PG method, efficient communication required for
sharing the experiences, generated locally at the eAPs, with the ECP can be
challenging. Moreover, in case of heterogeneous processing capabilities at the
eAPs, a synchronization method will be required at the ECP for reception of
the experiences from the eAPs. This method should avoid any bias toward
experiences coming from the actors that contribute more to the replay buffer.
In a cell-free network, optimizing the uplink beamforming necessitates some
level of control signaling among the distributed entities (eAPs and UEs). In
this regard, each of our proposed beamforming solutions imposes a different
requirement. Considering CSI, all of the solutions require the same amount of
signaling overhead, which is due to the collection of the received pilot
signals at each eAP. This is followed by application of the MMSE algorithm and
estimation of the CSI among all UEs and the intended eAP. To find the
beamforming matrices, the fully distributed solution requires the highest
amount of signaling exchanges to transfer the beamforming vectors among
different eAPs. This is obviously the cost paid to enable a distributed
learning-based solution.

##  VI Numerical Analysis

###  VI-A Simulation Parameters

We numerically evaluate the proposed beamforming methods in terms of
transmission performance (sum-rate) and convergence rate. Table V summarizes
the most important parameters of the simulation setting.

TABLE V: Simulation parameters Parameter | Value
---|---
AWGN PSD per UE |  $-169$ dBm/Hz
Path-loss exponent, $\kappa$ | $2$
Nakagami fading parameters, $\left(\mathcal{M},\Omega\right)$ | $\left(1,1\right)$
Training sequence length, $\tau_{p}$ |  $K$ Samples
Pilot transmission power, $\rho_{k}$ |  $100$ mW, ${\forall~{}k}$
SIC sensitivity, $P_{s}$ |  $1$ dBm

For simplicity, we assume the following: (i) For every AP $m$ and every UE
$k$, $h_{mk}$ is a random variable with $\mathcal{M}_{mk}=\mathcal{M}=1$ and
$\Omega_{mk}=\Omega=1$; (ii) For every AP $m$, the AWGN has PSD
$\sigma_{m}/2=-169$ dBm/Hz; (iii) Concerning large-scale fading, all eAPs and
UEs are uniformly distributed over a disc of radius $r=18$ meters, implying a
coverage area of $1017.88\text{m}^{2}$. The ECP knows the large-scale fading
of each UE. Furthermore, to evaluate different DRL models, we assume that each
iteration of the DRL corresponds to one coherent block (one CSI realization).
Note that the DRL models can also be trained under varying CSI (at each
channel use); however, this requires a larger number of iterations. We avoid
such a training procedure to simplify the simulation.

We train the proposed models by using Python and TensorFlow 2.1.0 for $10$
episodes with $1000$ steps per one episode. The actor- and critic networks
have fully connected layers with two hidden layers of 256 and 128 neurons
followed by the Relu activation function in each. The dimension of the final
layer for the actor network and its corresponding target network is defined in
the cell-free network environment, depending on the approach followed. In the
first two approaches, the dimension of the output layer is equal to the number
of elements in the beamforming matrix. In the third approach, it is equal to
the number of elements of one row.

The constraint C1 in the problem formulation P1 is implemented by affecting a
negative reward to the actions. The constraint C2 is imposed by defining a
softmax activation function in the output layer which takes as input a vector
of real values and normalizes it into a vector with values between 0 and 1.
The hyperparameters of the DRL-model are as follows: discount factor
$\zeta=0.99$, learning rate $\nu=0.001$ for both actor and critic networks, a
Poylak averaging parameter $\tau=0.005$, and size of experience replay buffer
$|\mathcal{R}|=10^{6}$. We use Adam for the critic and actor optimizer. In the
D4PG approach, we use $N$-step returns $N=5$ and $51$ atoms in the
distributional representation with $V_{min}=-20$ and $V_{max}=100$ as defined
in [41]. We use the binary cross-entropy as the metric of the distance between
distributions.

To verify the effectiveness of the proposed scheme, we evaluate the
performance in terms of instantaneous reward defined by (7). We first train
the proposed methods under three possible network models, namely, (i) small-
scale cell-free network with $M=15$ and $K=5$; (ii) medium-scale cell-free
network with $M=50$, $K=15$; (iii) large-scale cell-free network with $M=70$
and $K=20$.

###  VI-B Results

Fig. 5 shows the normalized transmission sum-rate versus the overall number of
training steps in the small-scale setting. In this figure, we compare the
performances of the proposed beamforming schemes with those for the two most
common techniques used in the literature namely, MMSE and conjugate
beamforming. Note that conjugate beamforming is considered as the simplest
beamforming scheme, where by knowing the CSI, each AP multiplies the received
signal by the intended UE’s CSI vector without considering interference from
other UEs. It can be noticed from this figure that the MMSE method gives the
best performance while the conjugate beamforming method results in the worst
performance. Among the proposed DRL-based beamforming schemes, the scheme with
distributed learning and centralized training (i.e. D4PG) exhibits the best
performance. This is due to the fact that, the D4PG actors distributed among
different eAPs generate more independent experiences all of which can be
exploited by the critic network at the ECP. Moreover, the performance of the
fully distributed DRL-based beamforming is better than that of the fully
centralized scheme (DDPG). Indeed, in a small network, since the beamforming
vector has low dimensions, the performance of the DRL method at each eAP is
almost identical to those of the conventional optimization methods (such as
the steepest ascent-based iterative algorithms). Furthermore, the D4PG
achieves the closest performance to the MMSE beamforming solution (without
learning) with about $80\%$ of the performance of the MMSE method after 6000
learning steps. Note that the performance degradation in the DRL-based systems
is the cost we pay while we achieve a significant reduction in computational
complexity compared to that of centralized MMSE method. Furthermore, even
though the conjugate beamforming method can be implemented in a fully
distributed manner with low processing requirements, it suffers from
significant performance degradation compared to all other methods (as shown in
Fig. 5).

Figure 4: Performance of different models under small-scale scenario.

Figure 5: Performance of different models under medium-scale scenario.

Fig. 5 shows the results for a medium-scale cell-free network. The centralized
DRL-based beamforming with distributed experience (i.e. D4PG) retains its
superiority over other methods, although the performance gap with the fully
centralized DDPG is smaller in this case. Moreover, the fully distributed DRL-
based beamforming is no longer superior to the centralized DDPG. The reason is
that in the distributed setting, every eAP uses the beamforming vectors found
by the other eAPs in previous iterations. This introduces inaccuracy in
treating the inter-user interference from other eAPs.

In Fig. 7, we consider a larger-scale network setting. The gap between the
distributed DRL-based beamforming and other methods with centralized learning
(DDPG and D4PG) increases significantly. Additionally, the centralized DRL-
based beamforming with distributed experience (D4PG) maintains better
performance compared to the fully centralized DDPG method in the sense that it
converges in fewer steps. Nevertheless, after a relatively large number of
training steps, the DDPG algorithm is observed to perform slightly better than
the D4PG algorithm.

Figure 6: Performance of different models under large-scale scenario.

Figure 7: Performance of DRL models under different actor-critic network
sizes.

In Fig. 7, we evaluate the performance of the proposed DRL models with larger
sizes of actor-critic networks. Specifically, we increase the size of both the
actor and critic networks from $256\times 128$ to $400\times 300$. With larger
sizes of the neural networks, we observe a smaller performance gap between the
fully centralized method based on DDPG and the DRL-based beamforming with
distributed experience (i.e. D4PG). The performance of the DRL-based
beamforming with distributed learning is not affected by the increase in the
size of the neural networks. The reason is that, in distributed learning, the
number of the optimization variables per eAP (i.e. $K$ elements in a row of
the beamforming matrix) is small so that the best possible performance is
achievable even for relatively smaller actor-critic neural networks (e.g.
$256\times 128$).

Moreover, we compare the running times of the gradient ascent method and the
proposed DRL models as a function of the problem dimension. For the DRL
models, the running time is considered to be the time required to obtain the
solution in the inference mode. We recall that the inference in deep learning
is a feed-forward propagation for a trained neural network. In the
simulations, we use the inference time of the policy network proposed in the
DRL-based beamforming approach (DDPG) since its policy network architecture is
identical to that of DRL-based beamforming with distributed experience (D4PG).
Moreover, it is bigger than the policy network architecture of DRL-based
beamforming with distributed learning (since the output layer of the
centralized method is much bigger than that of the distributed approach). The
learning rate of the gradient ascent algorithm to solve the optimization
problem is $\alpha=0.1$. The problem dimension is defined by the number of
eAPs $M$ and the number of UEs $K$ in the network. Here we set $K=\frac{M}{3}$
and we vary $M$ in the range of $[15,\dots,150]$.

Figure 8: Running time comparison.

Figure 9: Performance of DRL models in a testing environment.

In Fig. 8, we benchmark the gradient ascent method with the centralized DRL
method in the inference mode. Note that, based on the calculation of the
number of FLOPs in the inference mode, as given in Table IV, the centralized
DDPG has the same complexity as the D4PG model and has higher complexity than
the distributed DDPG model. That is, among the three DRL models, the
centralized model involves the most computational operations. Also, we compare
the convergence times (i.e. time needed to reach a stable solution) of the
algorithms for different network size.

Fig. 9 shows that the DRL-models converge to a stable solution faster than the
gradient ascent algorithm. The solution time of the gradient ascent algorithm
grows exponentially with the dimension of the network, whereas the DRL models
require less than a second for finding the optimal beamforming matrix.

Finally, we consider the case of various coherent blocks (different CSI
realizations) and we study the performance of the DRL-based methods. We setup
a cell-free network environment where each episode composed of 1000 iterations
uses one CSI realization. We train the DRL-models on 50 episodes, 50000
iterations in totals. For benchmarking, we test the trained policies in a test
environment for 10 episodes, 10 different CSI realizations, and we compare
them with the centralized MMSE method and the conjugate beamforming method.
Fig 9 shows that the MMSE method retains its superiority over the DRL-based
methods, while the DRL methods outperform the conjugate beamforming method.
Since the training can be done offline, the DRL methods are suitable for
practical implementation due to their low inference time. This time is almost
negligible compared to the convergence time of MMSE and conjugate beamforming
methods, especially for huge number of eAPs and UEs. A DRL-based method with
centralized learning gives higher performance compared to the DRL-based method
with distributed learning. For the latter, we split the computational task,
which is costly for large cell-free networks, among the eAPs. Therefore, we
have the classical trade-off between performance and computational
requirements for convergence where the centralized approaches have higher
performance but at the cost of high computation at the ECP.

##  VII Conclusion

We have studied the beamforming optimization problem in cell-free networks.
First, we have considered a fully centralized network and designed a DRL-based
beamforming method based on the DDPG algorithm with continuous optimization
space. We have also enhanced this method in D4PG by collecting distributed
experiences from geographically-distributed eAPs. Afterward, we have developed
a DRL-based beamforming design with distributed learning, which divides the
beamforming optimization tasks among the eAPs. Even though the D4PG
beamforming technique demonstrates a promising performance, it still conducts
the learning process at the ECP. A future research direction could be to
develop DRL models for channel estimation and pilot assignment for cell-free
networks, and also investigate the robustness of the DRL solutions in presence
of estimation errors as well as errors in reward signals.

## Appendix

Let $\tau_{p}\leq\tau_{c}$, where $\tau_{c}$ is the coherence time of the
channel via which the sequence is sent to all the eAPs with constant power.
The received pilot vector at the $m$-th eAP yields

$\bm{y}_{\textit{p},m}=\sum_{k=1}^{K}\sqrt{\tau_{p}\rho_{k}}g_{mk}\bm{\varphi}_{k}+\bm{\eta}_{m},$
(A.1)

where $\rho_{k}$ is the normalized transmission power for each symbol of the
$k$-th UE pilot vector. Moreover, $\bm{\eta}_{m}\in\mathbb{C}^{\tau_{p}\times
1}$ is the zero-mean complex additive white Gaussian noise (AWGN) vector
related to pilot symbols with independent and identically distributed (i.i.d)
rvs, i.e. ${\eta}_{m}\thicksim\mathcal{CN}\left(0,1/2\right)$. To find the
best estimate of $g_{mk}$ (denoted by
$\hat{g}_{mk}=\mathcal{F}_{mk}^{1/2}\hat{h}_{mk}$) given the vector of
observations $\bm{y}_{\textit{p},m}$, we first project $\bm{y}_{p,m}$ over
$\bm{\varphi}^{H}_{k}$. Therefore,

$\dot{y}_{p,m}=\bm{\varphi}^{H}_{k}\bm{y}_{\textit{p},m}=\underbrace{\sqrt{\tau_{p}\rho_{k}}g_{mk}}_{\text{Desired
Value}}+\underbrace{\sum_{l=1,l\neq
k}^{K}\sqrt{\tau_{p}\rho_{l}}g_{ml}\bm{\varphi}_{k}^{H}\bm{\varphi}_{l}+\bm{\varphi}_{k}^{H}\bm{\eta}_{m}}_{\text{Estimation
Error}}.$ (A.2)

Here $g_{mk}$ can be estimated from (A.2) by using the maximum a posteriori
decision rule (MAP), which is identical to the minimum mean square method
(MMSE) [42, 43]. Furthermore, given that the pilot signals are partially
orthogonal and partially non-orthogonal, $\bm{\varphi}_{k}^{H}\bm{y}_{p,m}$ in
(A.2) represents a sufficient statistics for the optimal estimation of
$g_{mk}$ (MMSE). Thus the best estimate of $g_{mk}$ is given by [6]

$\hat{g}_{mk}=\frac{\mathbb{E}\left[\dot{y}_{p,m}^{*}g_{mk}\right]}{\mathbb{E}\left[|\dot{y}_{p,m}|^{2}\right]}\dot{y}_{p,m}=\mathcal{E}_{mk}\dot{y}_{p,m}.$
(A.3)

Under the assumption that for all $~{}m$ and $k$, $g_{mk}$s are proper
independent but non-identically distributed (i.n.d) complex Gaussian rvs, and
that $\bm{\eta}_{m}$s are zero-mean i.i.d random variables, we get
$\mathcal{E}_{mk}$ as shown in Lemma 1.

## References

  * [1] P. Popovski, K. F. Trillingsgaard, O. Simeone, and G. Durisi, “5G wireless network slicing for eMBB, URLLC, and mMTC: A communication-theoretic view,” _IEEE Access_ , vol. 6, pp. 55 765–55 779, 2018.
  * [2] S. Zhou, M. Zhao, X. Xu, J. Wang, and Y. Yao, “Distributed wireless communication system: A new architecture for future public wireless access,” _IEEE Mag. Commun._ , vol. 41, no. 3, pp. 108–113, March 2003.
  * [3] E. Björnson and L. Sanguinetti, “Making cell-free massive MIMO competitive with MMSE processing and centralized implementation,” _IEEE Trans. on Wireless Commun._ , vol. 19, no. 1, pp. 77–90, 2020.
  * [4] M. Bashar, K. Cumanan, A. G. Burr, M. Debbah, and H. Q. Ngo, “On the uplink max-min SINR of cell-free massive MIMO systems,” _IEEE Trans. on Wireless Commun._ , vol. 18, no. 4, pp. 2021–2036, April 2019.
  * [5] Y. Al-Eryani and E. Hossain, “The D-OMA method for massive multiple access in 6G: Performance, security, and challenges,” _IEEE Veh. Technol. Mag._ , vol. 14, no. 3, pp. 92–99, Sep. 2019.
  * [6] H. Q. Ngo, A. Ashikhmin, H. Yang, E. G. Larsson, and T. L. Marzetta, “Cell-free massive MIMO versus small cells,” _IEEE Trans. on Wireless Commun._ , vol. 16, no. 3, pp. 1834–1850, March 2017.
  * [7] G. Interdonato, E. Björnson, H. Quoc Ngo, P. Frenger, and E. G. Larsson, “Ubiquitous cell-free massive MIMO communications,” _EURASIP J. on Wireless Commun. and Net._ , vol. 2019, no. 1, pp. 197–210, Aug. 2019.
  * [8] M. Attarifar, A. Abbasfar, and A. Lozano, “Modified conjugate beamforming for cell-free massive MIMO,” _IEEE Trans. on Wireless Commun._ , vol. 8, no. 2, pp. 616–619, April 2019.
  * [9] E. Björnson and L. Sanguinetti, “Scalable cell-free massive MIMO systems,” _IEEE Trans. on Commun._ , vol. 68, no. 7, pp. 4247–4261, 2020\.
  * [10] A. Liu and V. K. N. Lau, “Joint BS-user association, power allocation, and user-side interference cancellation in cell-free heterogeneous networks,” _IEEE Trans. Signal Processing_ , vol. 65, no. 2, pp. 335–345, 2017.
  * [11] Y. Jin, J. Zhang, S. Jin, and B. Ai, “Channel estimation for cell-free mmWave massive MIMO through deep learning,” _IEEE Trans. Veh. Technol._ , vol. 68, no. 10, pp. 10 325–10 329, 2019.
  * [12] S. J. Nawaz, S. K. Sharma, S. Wyne, M. N. Patwary, and M. Asaduzzaman, “Quantum machine learning for 6G communication networks: State-of-the-art and vision for the future,” _IEEE Access_ , vol. 7, pp. 46 317–46 350, 2019.
  * [13] S. Chakraborty, E. Björnson, and L. Sanguinetti, “Centralized and distributed power allocation for max-min fairness in cell-free massive MIMO,” in _2019 53’rd Asilomar Conf. on Signals, Systems, and Computers_ , 2019, pp. 576–580.
  * [14] R. Nikbakht, A. Jonsson, and A. Lozano, “Unsupervised-learning power control for cell-free wireless systems,” in _IEEE 30’th Annual Int’l Symposium on Pers., Indoor and Mobile Radio Commun. (PIMRC’19)_ , 2019, pp. 1–5.
  * [15] O. Simeone, “A very brief introduction to machine learning with applications to communication systems,” _IEEE Trans. on Cognitive Commun. and Networking_ , vol. 4, no. 4, pp. 648–664, 2018.
  * [16] A. Abdallah and M. M. Mansour, “Angle-based multipath estimation and beamforming for FDD cell-free massive MIMO,” in _IEEE 20’th Inter’l Workshop on Signal Proc. Advances in Wireless Commun. (SPAWC’19)_ , 2019, pp. 1–5.
  * [17] G. Femenias and F. Riera-Palou, “Cell-free millimeter-wave massive MIMO systems with limited fronthaul capacity,” _IEEE Access_ , vol. 7, pp. 44 596–44 612, 2019.
  * [18] Y. Al-Eryani, E. Hossain, and D. I. Kim, “Generalized coordinated multipoint (GCoMP)-enabled NOMA: Outage, capacity, and power allocation,” _IEEE Trans. on Commun._ , vol. 67, no. 11, pp. 7923–7936, Nov. 2019.
  * [19] Y. Al-Eryani, M. Akrout, and E. Hossain, “Multiple access in cell-free networks: Outage performance, dynamic clustering, and deep reinforcement learning-based design,” _IEEE J. Select. Areas Commun._ , pp. 1–1, 2020\.
  * [20] E. Nayebi, A. Ashikhmin, T. L. Marzetta, H. Yang, and B. D. Rao, “Precoding and power optimization in cell-free massive MIMO systems,” _IEEE Trans. on Wireless Commun._ , vol. 16, no. 7, pp. 4445–4459, 2017\.
  * [21] A. Zhou, J. Wu, E. G. Larsson, and P. Fan, “Max-min optimal beamforming for cell-free massive MIMO,” _IEEE Commun. Lett._ , pp. 1–1, 2020.
  * [22] A. Abdallah and M. M. Mansour, “Efficient angle-domain processing for FDD-based cell-free massive MIMO systems,” _IEEE Trans. on Commun._ , vol. 68, no. 4, pp. 2188–2203, 2020.
  * [23] H. Yang and T. L. Marzetta, “Energy efficiency of massive MIMO: Cell-free vs. cellular,” in _IEEE 87’th Veh. Tech. Conf. (VTC Spring)_ , 2018, pp. 1–5.
  * [24] Y. Zhang, H. Cao, M. Zhou, and L. Yang, “Cell-free massive mimo: Zero forcing and conjugate beamforming receivers,” _Journal of Commun. and Networks_ , vol. 21, no. 6, pp. 529–538, 2019.
  * [25] Z. Chen, E. Björnson, and E. G. Larsson, “Dynamic resource allocation in co-located and cell-free massive MIMO,” _IEEE Trans. on Green Commun. and Net._ , vol. 4, no. 1, pp. 209–220, 2020.
  * [26] M. Alonzo, S. Buzzi, A. Zappone, and C. D’Elia, “Energy-efficient power control in cell-free and user-centric massive MIMO at millimeter wave,” _IEEE Trans. on Green Commun. and Networking_ , vol. 3, no. 3, pp. 651–663, 2019.
  * [27] G. Interdonato, M. Karlsson, E. Björnson, and E. G. Larsson, “Local partial zero-forcing precoding for cell-free massive MIMO,” _IEEE Trans. on Wireless Commun._ , vol. 19, no. 7, pp. 4758–4774, 2020.
  * [28] D. Wang, M. Wang, P. Zhu, J. Li, J. Wang, and X. You, “Performance of network-assisted full-duplex for cell-free massive MIMO,” _IEEE Trans. on Commun._ , vol. 68, no. 3, pp. 1464–1478, 2020.
  * [29] G. Interdonato, P. Frenger, and E. G. Larsson, “Scalability aspects of cell-free massive MIMO,” in _IEEE Int’l Conf. on Commun. (ICC’19)_ , 2019, pp. 1–6.
  * [30] C. D’Andrea, A. Zappone, S. Buzzi, and M. Debbah, “Uplink power control in cell-free massive MIMO via deep learning,” in _2019 IEEE 8’th Int’l Workshop on Comput. Advances in Multi-Sensor Adaptive Proc. (CAMSAP’19)_ , 2019, pp. 554–558.
  * [31] M. Bashar, A. Akbari, K. Cumanan, H. Q. Ngo, A. G. Burr, P. Xiao, M. Debbah, and J. Kittler, “Exploiting deep learning in limited-fronthaul cell-free massive MIMO uplink,” _IEEE J. on Sel. Areas in Commun._ , pp. 1–1, 2020.
  * [32] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” _arXiv preprint arXiv:1509.02971_ , 2015.
  * [33] G. Barth-Maron, M. W. Hoffman, D. Budden, W. Dabney, D. Horgan, D. Tb, A. Muldal, N. Heess, and T. Lillicrap, “Distributed distributional deterministic policy gradients,” _arXiv preprint arXiv:1804.08617_ , 2018\.
  * [34] H. Q. Ngo, A. Ashikhmin, H. Yang, E. G. Larsson, and T. L. Marzetta, “Cell-free massive MIMO: Uniformly great service for everyone,” in _2015 IEEE 16th Int’l Workshop on Signal Proc. Advances in Wireless Commun. (SPAWC)_ , 2015, pp. 201–205.
  * [35] W. Liao, T. Chang, W. Ma, and C. Chi, “QoS-based transmit beamforming in the presence of eavesdroppers: An optimized artificial-noise-aided approach,” _IEEE Trans. Signal Processing_ , vol. 59, no. 3, pp. 1202–1216, 2011.
  * [36] S. Buzzi, C. D’Andrea, and C. D’Elia, “User-centric cell-free massive mimo with interference cancellation and local zf downlink precoding,” in _2018 15th International Symposium on Wireless Communication Systems (ISWCS)_ , 2018, pp. 1–5.
  * [37] H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double Q-learning,” in _Thirtieth AAAI conf. on artificial intelligence_ , 2016.
  * [38] D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller, “Deterministic policy gradient algorithms,” in _Proceedings of the 31st Int’l Conf. on Machine Learning_ , ser. Proceedings of Machine Learning Research, E. P. Xing and T. Jebara, Eds., vol. 32, no. 1. Bejing, China: PMLR, 22–24 Jun 2014, pp. 387–395.
  * [39] R. Barth-Maron, M. W. Hoffman, D. Budden, W. Dabney, D. Horgan, D. TB, A. Muldal, N. Heess, and T. Lillicrap, “Distributed distributional deterministic policy gradients,” _Conference at ICLR_ , 2018. [Online]. Available: https://arxiv.org/pdf/1804.08617.pdf
  * [40] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. de Las Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, T. Lillicrap, and M. Riedmiller, “Deepmind control suite,” 2018.
  * [41] M. G. Bellemare, W. Dabney, and R. Munos, “A distributional perspective on reinforcement learning,” _arXiv preprint arXiv:1707.06887_ , 2017.
  * [42] R. M. Gray and L. D. Davisson, _An Introduction to Statistical Signal Processing_ , 1st ed. New York, NY, USA: Cambridge University Press, 2010.
  * [43] H. Yin, D. Gesbert, M. Filippou, and Y. Liu, “A coordinated approach to channel estimation in large-scale multiple-antenna systems,” _IEEE J. on Sel. Areas in Commun._ , vol. 31, no. 2, pp. 264–273, Fe. 2013.