#  Spectrally Multiplexed Hong-Ou-Mandel Interference

Oriol Pietx-Casas  Corresponding author<EMAIL_ADDRESS>QuTech,
Delft University of Technology, 2600 GA Delft, The Netherlands  Kavli Institue
of Nanoscience, Delft University of Technology, 2600 GA Delft, The Netherlands
Gustavo Castro do Amaral  QuTech, Delft University of Technology, 2600 GA
Delft, The Netherlands  Center for Telecommunication Studies, Pontifical
Catholic University of Rio de Janeiro,
22451-900, Rio de Janeiro, Brazil  Tanmoy Chakraborty  QuTech, Delft
University of Technology, 2600 GA Delft, The Netherlands  Kavli Institue of
Nanoscience, Delft University of Technology, 2600 GA Delft, The Netherlands
Remon Berrevoets  QuTech, Delft University of Technology, 2600 GA Delft, The
Netherlands  Thomas Middelburg  QuTech, Delft University of Technology, 2600
GA Delft, The Netherlands  Joshua Slater  QuTech, Delft University of
Technology, 2600 GA Delft, The Netherlands  Wolfgang Tittel  QuTech, Delft
University of Technology, 2600 GA Delft, The Netherlands  Kavli Institue of
Nanoscience, Delft University of Technology, 2600 GA Delft, The Netherlands
Department of Applied Physics, University of Geneva, 1211 Geneva 4,
Switzerland  Schaffhausen Institute of Technology in Geneva, Switzerland

###### Abstract

We explore the suitability of a Virtually-Imaged Phased Array (VIPA) as a
Spectral-to-Spatial Mode-Mapper (SSMM) for applications in quantum
communication such as a quantum repeater. To this end we demonstrate
spectrally-resolved two-photon “Hong-Ou-Mandel” (HOM) interference. Spectral
sidebands are generated on a common optical carrier and weak coherent pulses
are prepared in each spectral mode. The pulses are subsequently sent to a
beamsplitter followed by two SSMMs and two single-photon detectors, allowing
us to measure spectrally-resolved HOM interference. We show that the so-called
HOM dip can be observed in the coincidence detection pattern of matching
spectral modes with visibilities as high as 45% (with the maximum being 50%).
For unmatched modes, the visibility drops significantly, as expected. Due to
the similarity between HOM interference and a linear-optics Bell-state
measurement (BSM), this establishes this simple optical arrangement as a
candidate for the implementation of a spectrally-resolved BSM. Finally, we
simulate the secret key generation rate using current and state-of-the-art
parameters in a Measurement-Device Independent Quantum Key Distribution
scenario and explore the trade-off between rate and complexity in the context
of a spectrally-multiplexed quantum communication link.

A quantum repeater will allow long-distance connectivity of quantum nodes,
ultimately enabling the establishment of the quantum internet. With a working
quantum internet comes the possibility of implementing protocols such as cloud
quantum computing, quantum-key-distribution (QKD) without trusted nodes, and
dense coding Ekert (1991); Bennett and Wiesner (1992). However, for these
applications to run between users – say A and B – over arbitrary distances, a
quantum repeater, in some form, becomes necessary, the simplest of which is
that of a memory-assisted quantum relay: a chain of elementary links that
consist of sources of entangled photon-pairs, quantum memories, a linear-
optics-based Bell-state measurement (BSM), and a quantum channel. Within an
elementary link, direct connection through entanglement swapping is possible
by restricting the distances such that the losses in the quantum channel are
constrained. To close the chain between end nodes A and B, the elementary
links are interconnected by entanglement swapping, as depicted in Fig. 1.

Figure 1: Three different possible configurations of a long-distance quantum
communication link between end-users A and B: direct connection; the quantum
relay; and the quantum repeater. The quantum repeater consists of a memory-
assisted quantum relay that decouples the elementary links and allows
overcoming channel loss through the use of multiplexing and feed-forward mode
mapping. L is the total distance and n is the number of links.

Multiplexing enhances the entanglement distribution rates of a quantum
repeater, and temporal multiplexing figures as the most common flavour of
quantum repeater architectures, where the quantum memories must be capable of
accessing the stored quantum states at any desired time. These so-called on-
demand quantum memories allow decoupling transmission through the elementary
links such that the successful entanglement distribution across the entire
link does not depend on simultaneous successful entanglement swapping
operations within all the constituent elementary links Duan et al. (2001).
Successfully swapped entangled states are stored in the memories until either
the storage time limit is reached or entanglement distribution across the
neighboring elementary link has been successful.

A quantum repeater making use of multiplexing is able to overcome the rates
achieved by a quantum relay. The rate enhancement can be easily quantified in
a simple model (see Fig. 1). We start by taking, as a reference, the rate
achieved by a quantum relay: since there is no rate boost for a quantum relay
architecture when compared to direct communication, $R^{\text{relay}}$ can be
computed, in the ideal case of unit efficiencies (including ideal BSMs), as
$R^{\text{relay}}=R^{\text{source}}10^{-\alpha L}$, where $L$ is the distance
between A and B in meters, $\alpha$ is the fiber’s loss coefficient in dB/m,
and $R^{\text{source}}$ is the maximum repetition rate achieved by the sources
Sinclair et al. (2014). For a temporally-multiplexed quantum repeater, we must
first define the total number of attempts $M$ that can be performed during the
memory’s storage time $t^{\text{store}}$, i.e.,
$M=t^{\text{store}}R^{\text{source}}$. It can be shown that Wang et al.
(2021):

$R^{\text{repeater}}=R^{\text{source}}\left(1-\left(1-10^{-\frac{\alpha
L/n}{10}}\right)^{M}\right)^{n},$ (1)

which represents a considerable boost over $R^{\text{relay}}$ if $M$
increases.

Unfortunately, some (if not all) quantum memories that allow read-out on
demand introduce deadtime and hence limit the entanglement generation rate of
a temporally multiplexed quantum repeaterAskarani et al. (2021). For instance,
for the atomic frequency comb protocol Afzelius et al. (2009), this problem is
caused by the need for control pulses that transfer optically excited
coherence onto spin coherence and, at a desired time, back. As described in
Askarani et al. (2021), no more qubits can be added to the memory after the
first control pulse. This creates a bottleneck for the rate since the memories
must potentially store states for long times. A spectrally-multiplexed quantum
repeater can overcome this bottleneck Sinclair et al. (2014). Through a
suitable design of the spectra of the photons generated by the sources (and
absorbed by the memories), it is possible to perform, via spectral-
multiplexing, $M$ attempts _during a single temporal window_. Furthermore, an
arbitrary number of temporal modes can be stored concurrently in the memory’s
optical coherence, contrasting the above-mentioned case of spin mapping. Under
these conditions, the spectrally-multiplexed quantum memory operates with a
fixed-storage time, which creates a constraint on the length of the elementary
links: the heralded results of the entanglement swapping operation must be
available at the memory location before its storage time elapses. This
condition can be encapsulated in mathematical form as
$t^{\text{fixed}}=L/n\cdot\left(v_{\text{fiber}}\right)^{-1}$, where
$v_{\text{fiber}}$ is the speed of light in the optical fiber. Extending the
optical storage time of ensemble-based broadband quantum memories is an on-
going effort, and recent experiments Askarani et al. (2021) have demonstrated
$t^{\text{fixed}}\\!=\\!100$ $\mu$s – enough for a 20-km-long elementary link.

In the same way that temporally-multiplexed quantum memories use on-demand
retrieval to reconcile entanglement in different temporal modes, spectrally-
multiplexed memories require a feed-forward spectral mode-mapper to reconcile
different spectral modes Sinclair et al. (2014). The information about which
mode has been successful, imperative for the repeater’s operation, must be
determined by the remote BSM stations. They therefore require the necessary
technological overhead to distinguish between different spectral modes. Since
the frequency bandwidth of quantum repeaters is likely to be limited by the
forthcoming quantum memories (e.g. 56 GHz in Askarani et al. (2021)), DWDM
technology is unsuitable due to the limited amount of available channels
within this bandwidth. Our work addresses this problem: we demonstrate high-
resolution spectrally-resolved two-photon interference by making use of
virtually-imaged phased arrays (VIPAs). This paves the way towards a
spectrally-multiplexed quantum repeater.

The Bell-State measurement is ubiquitous for long distance quantum
communications—in particular for quantum repeaters—but also in functionally
simpler (and thus simpler to performance benchmark) applications, such as
Measurement Device-Independent QKD (MDI-QKD) Rubenok et al. (2013). To
investigate the performance enhancement of the proposed spectrally-multiplexed
BSM, we implement a spectrally-multiplexed MDI-QKD channel and use the
extracted secret key rate as a figure of merit for the BSM’s performance. The
proposal involves routing spectrally-multiplexed ($M>1$) attenuated laser
pulses to a single beam-splitter, mapping each spectral mode to a different
spatial mode (spectral-to-spatial mode mapping – SSMM) using the VIPA,
detecting each spatial mode at a different detector, and correlating the
events of all possible detector combinations. We expect to observe a so-called
Hong-Ou-Mandel (HOM) dip when detectors for the same spectral/spatial mode are
selected.

Figure 2: Experimental setup. A single beam-splitter (BS) and two SSMMs based
on VIPAs allow demonstrating $M=2$ spectrally-resolved two-photon
interference. PM: Phase Modulator; IM: Intensity Modulator;
INT${}_{\text{A,B}}$: Phase-Controlled Interferometers; VOA: Variable Optical
Attenuator; $\Delta t$: Relative arrival time between pulses at the BS.

The experimental setup is depicted in Fig. 2. A phase modulator driven by a 4
GHz sinusoidal signal modulates two sidebands onto a continuous-wave laser
beam with central wavelength $\lambda_{0}=1532.68$ nm. We will refer to the
red-shifted spectral mode (-4 GHz away from the central optical carrier) as
spectral mode 1, and to the blue-shifted one (+4 GHz away) as spectral mode 2.
The remaining optical carrier will henceforth be ignored. The light is then
distributed to two preparation stations (A and B) – located in the same
laboratory – that encode time-bin qubits using electro-optical intensity
modulators (used to carve 625-ps-long pulses at an 80 MHz repetition rate) and
unbalanced, phase-stabilized interferometers with the same optical path-length
difference. The resulting pulses are furthermore attenuated to the single-
photon level and transmitted to the measurement station C. See Rubenok et al.
(2013) for more information.

At the measurement station, the two beams—one from A and one from B—are
combined in a symmetrical fiber-optic beam splitter, and the outputs are cast
onto free-space and shaped by a cylindrical lens. Each output is subsequently
directed to a different VIPA, which maps different spectral modes onto
distinct spatial modes that propagate into different directions. These modes
are collected into different fibres, positioned in the back focal plane of a
lens with positive focal length. This arrangement constitutes an SSMM able to
differentiate between different frequency modes (spectral-to-temporal
solutions have also been recently demonstrated Saglamyurek et al. (2016);
Merkouche et al. (2021)). Note that the number of beam splitters and VIPAs
remains the same as the number N of spectral/spatial modes grows. However, the
number of required detectors scales as 2N, which poses a concern in terms of
system scalability. Recently, arrays of single-photon detectors based on
superconducting nanowires have been reported Allmaras et al. (2020), showing
that the scaling issue can be alleviated by increasing the density of
detectors within a single cryogenic system.

The alignment of the fibers in the focal plane of the collimating lens is
critical to enable efficient coupling of the relevant frequency modes. In
order to characterize the spectral resolution and the cross-talk of the
assembled SSMMs, the four possible outputs – pictorially represented in Fig. 2
– are coupled into fibers and directed to four SNSPDs. Their count rates were
recorded as the frequency of the phase-modulator’s modulating tone was swept
in a range of [-6;6] GHz in steps of 100 MHz, yielding the frequency response
of the SSMM system. In practice, the coupling can be optimized individually
for each detector; however, the visibility of the two-photon interference
pattern depends on the relative efficiencies of the detectors Moschandreou et
al. (2018). Therefore, we roughly equalized the coupling for all 4 outputs.
The frequency response of SSMM 1 is presented in Fig. 3, where the two
distinct spectral windows separated by 8 GHz can be observed (the result
obtained for SSMM 2 is similar). In these measurements, the frequency
reference is taken as $\nu_{0}$, the center frequency of the common laser
source.

Figure 3: Frequency-response of the spectral-to-spatial mode mapper based on
VIPA 1. The similar result obtained with VIPA 2 is not shown. Two distinct
spectral windows can be identified at $\pm$4 GHz with respect to the center
wavelength of the common laser source.

The current experimental setup suffers from a limited adjacent mode rejection
ratio (i.e. crosstalk) and a low beamsplitter-through-VIPA-into-fiber coupling
efficiency. Fig. 3 shows that the former is well below 10 dB, while we
experimentally established the latter to be merely 5%. This is in stark
contrast with estimated values of 18 dB and 50%, respectively Xiao et al.
(2005). Therefore, with an optimized optical design, the performance of the
SSMM can be dramatically increased. But even under these sub-optimal
conditions, it is possible to extract reasonable coincidence count rates while
employing attenuated laser pulses with an average mean photon number of 0.1
for A and B. For instance, we found rates on the order of 1000 per second
outside the region of indistinguishability (i.e. outside the HOM dip).

To characterise the performance of our SSMMs, we begin with a HOM dip
measurement; an important figure of merit in the experiment is the visibility
of the HOM dip, which is limited to 50% for attenuated laser pulses. It is
defined as $V=\tfrac{C_{\text{dist}}-C_{\text{indist}}}{C_{\text{dist}}}$,
where $C_{\text{dist}}$ and $C_{\text{indist}}$ are the coincidence counts,
accumulated during the same time, when the wave-packets are distinguishable
and indistinguishable, respectively. In order to reach these two regions, the
relative time delay of the emission of the pulses ($\Delta t$ in Fig. 2) is
swept in a 3 ns range, centered at the point of maximum indistinguishability,
as depicted in Fig. 4. We extract visibilities for all four possible
combinations of spectral modes using time-bin qubits prepared in
$|late\rangle$. These states are created by blocking the short path of the
interferometers depicted in Fig. 2, and setting the mean-photon number per
late temporal mode to 0.1. We found visibilities for the matching spectral
modes as high as 45%, approaching the theoretical limit of 50% thus attesting
the suitability of our setup for spectrally-resolved two-photon interference.
Ideally, the visibility for unmatched spectral modes should be zero since the
wave-packets should be spectrally distinguishable (i.e. feature no spectral
overlap). However, as discussed above, our setup features crosstalk, resulting
in visibilties of 5% and 21% instead (see Fig. 4).

Figure 4: Coincidence counts as the relative arrival time of the attenuated
laser pulses is swept for all four combinations of matched and unmatched
spectral modes. Solid lines are Gaussian best fits from which the visibilities
are extracted. Inset: Measured visibility of a HOM-like interference
experiment where the relative phase between early and late qubit modes is
varied. The solid line is a sinusoidal best fit.

Next, we go one step further in our characterization and make use of the
interferometers to create qubits of the form

$\begin{split}|\psi_{A}\rangle=\tfrac{1}{\sqrt{2}}\left(|e\rangle+|\ell\rangle\right)\hskip
10.0pt;\hskip
10.0pt|\psi_{B}\rangle=\tfrac{1}{\sqrt{2}}\left(|e\rangle+e^{i\theta}|\ell\rangle\right).\end{split}$
(2)

To perform a HOM-like interference measurement, we repeat the procedure used
to extract the HOM visibility but look only for the coincidence events of
spectral mode 1. In addition, the temporal windows within which coincidences
are registered now extend over early and late temporal modes. We vary the
phase difference $\theta$ between the early and late temporal modes of the
qubits created in B in the range of [$-\pi$;$\pi$], and record the visibility
of the HOM dip for each value. As depicted in the inset of Fig. 4, the
visibility changes from maximum to minimum as the phase causes the time-bin
encoded photonic qubits to become orthogonal (this is the case if
$\theta=\pm\pi$).

The final step in our analysis targets the potential performance enhancement
due to the use of our SSMMs and spectral-multiplexing: how many distinct
spectral modes $M$ are required to overcome the performance of a non-
multiplexed system in MDI-QKD? Ideally, the answer to this question would be
$M>1$; however, as we have pointed out, the inclusion of our technology
induces loss and crosstalk, which significantly impacts on the system’s
overall performance. This likely requires $M$ to be much higher.

We base our calculations on Yu et al. (2013), and the imperfections associated
with the creation of a generic time-bin-encoded photonic qubit are
encapsulated in the model developed in Chan et al. (2014):

$|\psi\rangle=\tfrac{1}{\sqrt{1+2b}}\Big{(}\sqrt{m+b}|e\rangle+e^{i\theta}\sqrt{1-m+b}|l\rangle\Big{)},$
(3)

with $m=S_{e}/\left(S_{e}+S_{\ell}\right)$ and
$b=B/\left(S_{e}+S_{\ell}\right)$. The experimentally determined parameters
$S_{e}$, $S_{\ell}$, and $B$ are established independently for each pair of
qubits belonging to a mutually unbiased basis (Z, spanned by $|e\rangle$ and
$|\ell\rangle$; and X, spanned by
$|\pm\rangle=(|e\rangle\pm|\ell\rangle)\sqrt{2}$), as depicted in Fig. 5 for
the states $|\ell\rangle$ and $|+\rangle$. Apart from $m$ and $b$, the model
takes as input the observed visibility of the HOM-dip—we choose 42% as the
average between spectral mode 1 and spectral mode 2, and then allows
calculating the secret key rate for the MDI-QKD channel.

Figure 5: The set of experimentally measured parameters that constitute the
inputs for the key rate estimation model. Highlighted are $S_{\ell Z}$,
$B_{Z}$ and $S_{eX}$, $B_{X}$ for bases Z and X, respectively.

To assess the potential impact of the proposed spectrally-multiplexed BSM, we
compute secret key rates for three different scenarios: (1) the current
experimental scenario; (2) the current scenario but assuming state-of-the-art
coupling efficiency of 50% for the central mode (see the inset of Fig. 6); and
(3) a scenario where the coupling is again state-of-the-art, and the channel
spacing is reduced from 8 to 3.2 GHz, i.e. limited only by the width of the
spectral channels in Fig. 3. These rates are compared to the regular single-
mode case in which the losses from the SSMM are removed but all other
parameters, (e.g. qubit descriptions as in Eq. 3, and $R^{\text{source}}$) are
kept the same.

The results are displayed in Fig. 6. Our analysis includes the constraint that
the VIPA in our current SSMM has a 60 GHz spectral bandwidth (matching that of
the memory in Askarani et al. (2021)). Thus, assuming a mode spacing of 8 GHz,
this constraints the maximum number of modes to 7. In scenario (3), this
number increases to 20. Note, however, that the VIPA allows for smaller
channel spacing. For instance, a spectral resolution around 600 MHz has been
reported in Xiao et al. (2005)). In addition, VIPAs with larger spectral
bandwidth can be manufactured, but with a direct impact on maximum coupling
efficiency and spatial resolution; therefore, we picked the same value for the
spectral bandwidth for all three scenarios. Note that the rate scales non-
linearly with the number of modes, which is due to the decreasing coupling
efficiency for spatial modes away from the VIPA’s nominal operating wavelength
– refer to the inset of Fig. 6.

Figure 6: Enhancement of the secret key rate over the single mode case for
different numbers of spectral modes assuming three different experimental
conditions. SoA denotes state-of-the-art. The fits are exponential best fits
that represent upper bounds. The inset showcases the decreasing coupling
efficiency experienced by modes away from the VIPA’s nominal operating
wavelength Xiao et al. (2005) . Spatial separation is calculated based on a
collimating lens with focal length $f=1$ meter; the blue dashed square
highlights the current VIPA’s spectral bandwidth.

We find that the coupling loss in the current setup prevents increasing the
rates, even when the number of spectral modes reaches its maximum of 7
(scenario (1)). However, we also see that the proposed spectrally-resolved BSM
outperforms a single-mode BSM for M$\geq$5 modes if the coupling is improved
from its current value of 5% to 50% (scenario (2)). A crossover is confirmed
for scenario (3), in which the mode spacing is reduced from 8 to 3.2 GHz.
There, due to the larger number of possible modes, the rate enhancement grows
up to a factor of 3.4.

Assuming the use of a SSMM with increased spectral range while keeping all
other parameters as before allows us to extend the prediction to larger
numbers of spectral modes. The observed saturation of the secret key rate with
growing M indicates that it is advantageous to increase the density of
spectral modes rather than to increase the total spectral window. It is
interesting to observe how the time-bandwidth product (TBP) plays a role in
the maximum achievable rate in the frequency-multiplexed scenario: reducing
the pulse width allows increasing the rate. In turn, this widens the spectral
channels, thereby reducing the maximum number that can be resolved within the
total spectral bandwidth. Hence, by increasing $R^{\text{source}}$, a limit is
set for $M$, which corresponds to the most efficient usage of the available
spectrum and saturates Eq. 1. But note that the optimal number of spectral
modes (and hence the temporal mode duration and the emission rate that
maximize the TBP) depends on the use of the SSMM. For instance, in MDI-QKD
Rubenok et al. (2013) it may be best to limit the setup to one spectral mode,
while a larger number of modes is needed for a spectrally multiplexed quantum
repeater Sinclair et al. (2014).

In conclusion, we have proposed and experimentally demonstrated a VIPA-based
spectral-to-spatial mode-mapper capable of performing spectrally-resolved two-
photon interference. We characterized its properties, current limitations, and
potential to implement a spectrally-resolved Bell-state measurement in the
near-future, where rate enhancements in a spectrally-multiplexed quantum
repeater scenario can be reached.

Funding. The authors acknowledge funding through the Netherlands Organization
for Scientific Research (NWO), and the European Union’s Horizon 2020 Research
and Innovation Program under Grant Agreement No.820445 and Project Name
Quantum Internet Alliance.

## References

  * Ekert (1991) A. K. Ekert, Phys. Rev. Lett. 67, 661 (1991), URL https://link.aps.org/doi/10.1103/PhysRevLett.67.661.
  * Bennett and Wiesner (1992) C. H. Bennett and S. J. Wiesner, Phys. Rev. Lett. 69, 2881 (1992), URL https://link.aps.org/doi/10.1103/PhysRevLett.69.2881.
  * Duan et al. (2001) L.-M. Duan, M. D. Lukin, J. I. Cirac, and P. Zoller, Nature 414 (2001).
  * Sinclair et al. (2014) N. Sinclair, E. Saglamyurek, H. Mallahzadeh, J. A. Slater, M. George, R. Ricken, M. P. Hedges, D. Oblak, C. Simon, W. Sohler, et al., Phys. Rev. Lett. 113, 053603 (2014), URL https://link.aps.org/doi/10.1103/PhysRevLett.113.053603.
  * Wang et al. (2021) P.-C. Wang, O. Pietx-Casas, M. F. Askarani, and G. C. Do Amaral, JOSA B pp. 1140–1146 (2021).
  * Askarani et al. (2021) M. F. Askarani, A. Das, J. H. Davidson, G. C. Amaral, N. Sinclair, J. A. Slater, S. Marzban, C. W. Thiel, R. L. Cone, D. Oblak, et al., Physical Review Letters 127, 220502 (2021).
  * Afzelius et al. (2009) M. Afzelius, C. Simon, H. De Riedmatten, and N. Gisin, Phys. Rev. A 79, 052329 (2009).
  * Rubenok et al. (2013) A. Rubenok, J. A. Slater, P. Chan, I. Lucio-Martinez, and W. Tittel, Phys. Rev. Lett. 111 (2013), URL https://link.aps.org/doi/10.1103/PhysRevLett.111.130501.
  * Saglamyurek et al. (2016) E. Saglamyurek, M. G. Puigibert, Q. Zhou, L. Giner, F. Marsili, V. B. Verma, S. W. Nam, L. Oesterling, D. Nippa, D. Oblak, et al., Nature communications 7, 1 (2016).
  * Merkouche et al. (2021) S. Merkouche, V. Thiel, and B. J. Smith, arXiv preprint arXiv:2104.05655 (2021).
  * Allmaras et al. (2020) J. P. Allmaras, E. E. Wollman, A. D. Beyer, R. M. Briggs, B. A. Korzh, B. Bumble, and M. D. Shaw, Nano letters 20, 2163 (2020).
  * Moschandreou et al. (2018) E. Moschandreou, J. I. Garcia, B. J. Rollick, B. Qi, R. Pooser, and G. Siopsis, Journal of Lightwave Technology 36, 3752 (2018).
  * Xiao et al. (2005) S. Xiao, A. M. Weiner, and C. Lin, Journal of Lightwave Technology 23, 1456 (2005).
  * Yu et al. (2013) Z.-W. Yu, Y.-H. Zhou, and X.-B. Wang, Physical Review A 88, 062339 (2013).
  * Chan et al. (2014) P. Chan, J. A. Slater, I. Lucio-Martinez, A. Rubenok, and W. Tittel, Optics express 22, 12716 (2014).


# Sustainable Palm Tree Farming: Leveraging IoT and Multi-Modal Data for Early
Detection and Mapping of Red Palm Weevil

Yosra Hajjaji
RIADI Laboratory, National School of Computer Science, University of Manouba,
Tunisia
<EMAIL_ADDRESS>
& Ayyub Alzahem
Robotics and Internet-of-Things Laboratory, Prince Sultan University, Riyadh,
Saudi Arabia
<EMAIL_ADDRESS>
& Wadii Boulila
Robotics and Internet-of-Things Laboratory, Prince Sultan University, Riyadh,
Saudi Arabia
RIADI Laboratory, National School of Computer Science, University of Manouba,
Tunisia
<EMAIL_ADDRESS>
Imed Riadh Farah
RIADI Laboratory, National School of Computer Science, University of Manouba,
Tunisia
<EMAIL_ADDRESS>
& Anis Koubaa
Robotics and Internet-of-Things Laboratory, Prince Sultan University, Riyadh,
Saudi Arabia
<EMAIL_ADDRESS>

###### Abstract

The Red Palm Weevil (RPW) is a highly destructive insect causing economic
losses and impacting palm tree farming worldwide. This paper proposes an
innovative approach for sustainable palm tree farming by utilizing advanced
technologies for early detection and management of RPW. Our approach combines
computer vision, deep learning (DL), the Internet of Things (IoT), and
geospatial data to effectively detect and classify RPW-infested palm trees.
The main phases include; (1) DL classification using sound data from IoT
devices, (2) palm tree detection using YOLOv8 on UAV images, and (3) RPW
mapping using geospatial data. Our custom DL model achieves 100% precision and
recall in detecting and localizing infested palm trees. The integration of
geospatial data enables the creation of a comprehensive RPW distribution map
for efficient monitoring and targeted management strategies. This technology-
driven approach benefits agricultural authorities, farmers, and researchers in
managing RPW infestations, safeguarding palm tree plantations’ productivity.

_Keywords_ Smart Agriculture $\cdot$ Palm Tree Detection $\cdot$ Early
Detection of Red Palm Weevil $\cdot$ Multi-Modal Data $\cdot$ Sound Data
$\cdot$ UAV Images $\cdot$ Deep Learning $\cdot$ Internet of Things.

##  1 Introduction

Palm trees, with over 2,500 species and the ability to produce more than 1000
products, hold immense importance worldwide. They play a vital role in the
agricultural economies of many countries, particularly in the Middle East,
where they are the leading producers of dates by generating more than 8
million tons per year. Additionally, over 50% of the world’s palm oil is
produced in Indonesia, which is also the leading producer Ammar and El-Naggar
(2011); eco . The global palm oil industry is valued at around $65 billion
annually. Millions of people throughout the world have work prospects thanks
to the palm tree business, especially in rural regions where palm agriculture
is the main source of income Haraz et al. (2021).

However, the quality and quantity of palm trees face constant risks due to
various diseases, including Leaf Spots, Leaf Blights, and the notorious Red
Palm Weevil (RPW) , have significant consequences Ashry et al. (2022); Wibowo
et al. (2022). Due to damaged palm trees and decreased harvests, the RPW
infestation alone has caused billions of dollars in economic losses worldwide.
In addition, diseases like Leaf Spot and Leaf Blight can reduce palm oil
production output by up to 40%. This not only impacts farmers’ livelihoods but
also has negative implications for ecosystems and biodiversity. The
destructive impact of the RPW has been observed in Southeast Asia, parts of
Asia, northern Africa, Europe, and certain regions of North America. Human
activity plays a role in facilitating the rapid spread of RPW, as diseased
palm trees are inadvertently transported to non-infested areas. The RPW’s
ability to remain hidden inside the palm tree trunk poses challenges for early
identification. If left untreated, infected palm trees can suffer long-term
damage, as the RPW eventually exits the hollowed tree in search of a new host
Yarak et al. (2021); Ferry et al. (2019); Hajjaji et al. (2021a). The use of
Internet of Things (IoT) devices has emerged as a critical aspect of palm tree
identification and management. These gadgets have a variety of sensors,
including sound, temperature, and humidity sensors. The sound sensors are
crucial in the context of detecting palm tree infestations. They make it
possible to gather the audio data required to use deep learning (DL)-based
classification techniques.

The DL algorithms can precisely recognize and classify afflicted palm trees by
examining the distinctive acoustic signals generated by Red Palm Weevil (RPW)
larvae or their feeding activities. Early detection of RPW infestations is
essential for proactive monitoring and quick response to reduce the hazards
that come with them.

This integration of DL classification based on sound data from IoT devices, as
well as palm tree identification and RPW mapping phases, forms a basis of a
successful method for tackling palm tree diseases and pests. This strategy
provides effective and data-driven management strategies improving the general
health and production of palm tree farms by utilizing the power of IoT sensors
and DL algorithms. Hajjaji et al. (2022); Ghandorh et al. (2022); Hajjaji et
al. (2021b). The value of IoT devices stems from their capacity to offer real-
time data from palm tree farms, allowing for proactive monitoring, early
identification, and rapid reaction to illnesses and insect infestations. The
gathered sound data can be processed using DL-based classification approaches
to identify individual palm tree species and assess their health state.
Furthermore, combining the palm tree detection and RPW mapping phases allows
for the identification and tracking of RPW-infested palms, allowing for
targeted interventions to prevent and reduce the spread of this damaging
pestElshafie et al. (2019); Jintasuttisak et al. (2022).

In this research, we provide a complete approach that combines deep learning
health categorization based on IoT device sound data, palm tree detection
algorithms, and RPW mapping methods Driss et al. (2020). We want to build an
effective solution for palm tree disease and pest management by integrating
these components. We hope to provide precise diagnosis, early intervention,
and timely actions to protect palm tree populations by analyzing sound data
and applying powerful computer vision techniques. The main contributions of
our paper can be summarized as follows:

  1. 1.

Capturing of UAV remote sensing images and Arduino-based acoustic sensor RPW
noise;

  2. 2.

Creating a dataset of Palm Tree Images Samples (PTIS) and sound Weevil data;

  3. 3.

DL palm tree health classification based on sound Weevil data and inception
model;

  4. 4.

Training, validating, and testing the selected YOLOv8 models to detect and
count palm trees based on PTIS;

  5. 5.

Evaluating the accuracy and performance of the optimal YOLOv8-based model for
palm tree detection;

  6. 6.

Mapping the location and spatial distribution of infested and non-infested
palm trees based on palm tree health classification and object detection
results.

This paper is organized as follows. Section 2 reviews relevant literature on
the detection of palm tree infestation, highlighting key findings and gaps in
the existing research. Section 3 describes the research design and analysis
procedures used in the study with detailed explanations for every component
presented in the proposed approach. Section 4, is a description of the used
dataset and the experiments done on it including the results of each
experiment. Finally, the main findings of the proposed study, their
implications, and recommendations for future research are summarized in
Section 5.

##  2 Related works

Karar et al. Karar et al. (2022) proposed a smart IoT-based system for
detecting RPW larvae in date palms using mixed depthwise convolutional
networks (MixConvNet). The system used a modified MixConvNet as a deep
learning classifier to identify RPW infestation cases based on short audio
recordings of feeding and/or moving RPWs. The system was tested on the public
TreeVibes dataset, which contains 146 infested and 351 clean sounds. The
system achieved the best accuracy score of 97.38% and outperformed other deep
learning classifiers, such as Xception and Resnet models. The system can
provide an efficient and accurate solution for protecting date palm trees at
the early stage of infestation.

Koubaa et al. Koubaa et al. (2020) developed an IoT-based smart palm
monitoring prototype to prevent the damage caused by the red palm weevil, a
critical pest of palms. The prototype collected data from palms using smart
agriculture sensors, and consumers could monitor their palm farms remotely via
web/mobile applications. To interface between the sensor layer and the user
layer, the authors used an industrial-level IoT platform. To examine the data
and determine the infection fingerprint, they also used signal processing and
statistical approaches.

Alsanea et al. Alsanea et al. (2022) used region-based convolutional neural
network (R-CNN) and convolutional neural network (CNN) techniques to detect
and categorize the red palm weevil (RPW), a damaging pest of palm trees. The
model located the RPW in an image using bounding boxes and retrieved
characteristics to classify it. At an early stage in the Al-Qassim region, the
model identified the RPW with 100% accuracy in infested palm trees.

Ashry et al. Ashry et al. (2022) introduced a unique approach for the early
identification of RPW, a devastating pest of palm trees using optical fiber
distributed acoustic sensing (DAS) and machine learning (ML). The authors
classified the signals captured by the DAS system and filtered by a 100-800 Hz
bandpass filter using CNN models. In both controlled and outdoor conditions,
the approach produced good accuracy and minimized false alarms, with the
latter being influenced by wind noise.

The identification and monitoring of RPW have benefited greatly from these
connected efforts. However, they frequently concentrated on certain elements,
such as monitoring systems, image-based methods, or larvae identification. In
contrast, our method offers a more complete solution for RPW identification
and localization since it incorporates sound analysis, CQCC feature
extraction, deep learning classification, palm tree object detection, and
coordinate matching. Our method improves accuracy and allows focused
intervention tactics by combining numerous data sources.

##  3 Proposed approach

The suggested approach’s process is depicted in Figure 1 and consists of three
major phases: DL classification based on sound data from IoT devices, Palm
tree recognition, and RPW mapping. Each of these phases is discussed in detail
in the sections that follow. The proposed methodology intends to contribute to
the successful monitoring and preservation of palm tree health and
productivity by integrating innovative technologies and data-driven
approaches.

Figure 1: Overview of our proposed approach of palm tree detection, counting
and health classification based on integrated UAV remote sensing and sound RPW
sound data.

###  3.1 DL Health Classification

We provide a quick method for spotting Red Palm Weevil (RPW) infestation in
palm plants that makes use of reliable data and deep learning (DL) methods.
The distinctive noises made by afflicted palm trees are captured using
specialized equipment, which is then processed to obtain Constant-Q Cepstral
Coefficients (CQCC), which stand for important characteristics of the recorded
signals. The InceptionV3 model, a sophisticated DL approach, is then used to
categorize and identify the existence of RPW infestation in palm trees using
the retrieved data. This easy-to-use technique, though effective, permits
early identification of RPW infestations and facilitates proactive steps to
protect palm trees’ health and wellbeing.

####  3.1.1 Constant-Q Cepstral Coefficients (CQCC)

CQCC feature extraction algorithm is a strong technique that is commonly
utilized in audio signals processing tasks such as voice recognition, sound
categorization, and music analysis. CQCC captures both the spectral and
temporal properties of audio signals, making it an effective representation
for a wide range of audio-based applications. We present the CQCC algorithm in
detail in this section, highlighting its major phases and equations.

  1. 1.

Constant-Q Transform: The first step in the CQCC algorithm is to compute the
input audio signal’s Constant-Q Transform (CQT). The CQT is comparable to the
well-known Short-Time Fourier Transform (STFT), but with frequency bins that
are logarithmically separated to better mimic human auditory perception. The
following Equation defines it:

$X[n,k]=\sum_{m=0}^{M-1}x[n+m]w[m]e^{-j2\pi kQm/M}$ (1)

where, $X[n,k]$ represents the CQT coefficients at time frame $n$ and
frequency bin $k$. $x[n]$ denotes the input audio signal, $w[m]$ is a
windowing function (e.g., Hamming window), $Q$ is the Q-factor determining the
number of frequency bins per octave, and $M$ is the window length.

  2. 2.

Log-compression and Power Normalization: After computing the CQT coefficients,
a log-compression operation is applied to enhance the perceptual
representation of the audio signal. It helps to emphasize the lower magnitude
components while compressing the higher ones. The log-compressed coefficients,
denoted as $X_{\text{log}}[n,k]$, are obtained using the following equation:

$X_{\text{log}}[n,k]=\log(1+\mu|X[n,k]|)$ (2)

where $\mu$ is a small positive constant to avoid the logarithm of zero and
control the compression.

To normalize the power across different frequency bins, a power normalization
operation is performed. It aims to ensure that the resulting features are
robust to variations in signal amplitude. The power-normalized coefficients,
denoted as $X_{\text{norm}}[n,k]$, are computed as follows:

$X_{\text{norm}}[n,k]=\frac{X_{\text{log}}[n,k]-\bar{X}_{k}}{\sigma_{k}}$ (3)

where $\bar{X}_{k}$ and $\sigma_{k}$ represent the mean and standard deviation
of the CQT coefficients within each frequency bin $k$.

  3. 3.

Cepstral Coefficients: Next, the Cepstral Coefficients are obtained by
performing the Discrete Cosine Transform (DCT) on the power-normalized
coefficients. The DCT decorrelates the coefficients and retains the most
salient information. The resulting Cepstral Coefficients, denoted as $C[n,k]$,
are given by the equation:

$C[n,k]=\sum_{m=0}^{M-1}X_{\text{norm}}[n,k]\cos\left(\frac{\pi
k(m+0.5)}{M}\right)$ (4)

  4. 4.

CQCC Features: Finally, the CQCC features are obtained by concatenating the
Cepstral Coefficients across multiple time frames. This step creates a compact
and discriminative representation of the audio signal suitable for further
processing and analysis.

####  3.1.2 InceptionV3

InceptionV3 is a Convolutional Neural Network (CNN) architecture developed by
Google for image recognition and classification tasks. It incorporates
Inception modules, and parallel convolutional layers with different filter
sizes, to capture multi-scale information efficiently. InceptionV3 utilizes
auxiliary classifiers for additional supervision, global average pooling for
spatial information aggregation, and a softmax classifier for prediction.
Trained on large-scale datasets, InceptionV3 achieves state-of-the-art
performance in various computer vision tasks due to its effective multi-scale
feature extraction and computational efficiency Al Husaini et al. (2022).

###  3.2 Palm tree detection using YOLOv8

In this section, we aim to leverage the YOLOv8 object detection framework to
achieve accurate and efficient detection of date palm trees.

YOLOv8, released in 2023 Ultralytics (2023), represents a culmination of
advancements from various real-time object detectors, aimed to combine the
best of many real-time object detectors. It integrates the strengths of
multiple models to deliver a comprehensive solution for object detection
tasks, including palm tree detection. Our motivation for choosing YOLOv8 stems
from its ability to leverage the best features and optimizations from
different approaches, resulting in enhanced performance and accuracy. YOLOv8
incorporates state-of-the-art backbone and neck architectures (i.e., YOLOv5),
improving the quality of feature extraction and enabling precise palm tree
detection. The network employs a sigmoid activation function to calculate the
output probability for each detection instance being a palm tree, as shown in
Equation (5), where $\sigma$ represents the sigmoid function and
$\mathrm{net}$ denotes the input to the sigmoid.

$\text{Output}=\sigma(\text{net})$ (5)

YOLOv8 introduces the Anchor-Free approach, departing from the Anchor-Based
approach used in previous versions. Additionally, YOLOv8 incorporates the
dynamic TaskAlignedAssigner for its matching strategy. To determine the
alignment degree at the Anchor level for each instance, Equation 6 is
employed, where "s" represents the classification score, "u" represents the
IOU value, and $\alpha$ and $\beta$ denote the weight hyperparameters.
Positive samples are selected by choosing "m" anchors with the maximum value
"t" in each instance, while the remaining anchors are considered negative
samples. The model is then trained using the loss function. These enhancements
have resulted in YOLOv8 achieving a 1% increase in accuracy compared to
YOLOv5, establishing it as the most accurate object detector to date.

$t=s^{\alpha}\cdot u^{\beta}$ (6)

To improve the performance of the model, YOLOv8 employs a comprehensive loss
function that incorporates multiple components, including classification loss,
localization loss, and confidence loss, as shown in Equation (7). The loss
function is designed to guide the training process and optimize the detection
performance, where each term contributes to different aspects of the detection
task, penalizing incorrect predictions and encouraging accurate detections.

$\mathrm{Loss}=\text{Classification Loss}+\text{Localization
Loss}+\text{Confidence Loss}$ (7)

This optimized balance between accuracy and speed makes YOLOv8 an ideal choice
for real-time palm tree detection in applications such as environmental
monitoring, urban planning, and agriculture. Furthermore, YOLOv8 provides a
variety of pre-trained models tailored to different performance requirements.
These models offer a convenient starting point, leveraging the expertise and
optimizations already embedded in YOLOv8. This simplifies the deployment
process and saves time and resources in developing an effective palm tree
detection system. Hence, YOLOv8 was chosen as the reference version as a
baseline of our work Lou et al. (2023).

####  3.2.1 Network achitecture

YOLOv8 shares critical components/modules with YOLOv5, such as the Focus, CBL,
SPP, and CSP modules. These modules serve similar purposes and functionalities
in YOLOv8 as in YOLOv5. The Focus module splits the input image into four
parallel slices and uses the CBL module for feature map generation. The CBL
module performs convolution operations with batch normalization and leaky-ReLU
activation for feature extraction. The CSP module in YOLOv8 is based on CSPNet
and consists of CSP1 and CSP2 modules, similar to YOLOv5. These modules are
utilized in the network backbone and neck parts of YOLOv8, where they split
the input feature map, fuse cross-level features, and determine the
architecture’s depth. The SPP module, also present in YOLOv5, downsamples
input features using max pooling layers and combines them with the starting
features to capture spatial information. While the architectural elements and
configurations may differ between YOLOv8 and YOLOv5, the core concepts and
functions of these modules remain the same. The subsequent subsections discuss
the dataset preparation, training, validation, and testing processes.

####  3.2.2 Network training

The training dataset included two classes: palm and tree, with accurate
labeling distinguishing between palm trees and other types of trees.
Rectangular bounding boxes were used to enclose the identified palm trees and
trees, accounting for variations in size, overlap, occlusion, and diverse
backgrounds. To enrich the training dataset, data augmentation techniques were
employed, including common transformations like scaling, cropping, rotation,
and color space adjustments. Mosaicking, a technique involving the combination
of multiple images to create a composite image, was also used for further
augmentation, introducing variations in scene composition and context. This
augmented dataset significantly increased the number of training images for
each class, enhancing the model’s ability to generalize and accurately detect
date palm trees. Various hyperparameters were adjusted to optimize the
training process, including the SGD optimizer with a momentum value of 0.9 and
a decay rate of 0.0005. A random seed of 45 was set for reproducibility, and
the training progress was monitored with model saving every 10 epochs. Early
stopping with patience of 20 epochs was implemented to prevent overfitting.
These configuration settings ensured consistent and effective training of the
YOLO-V8 model for accurate palm tree detection.

####  3.2.3 Palm tree detection

After training the YOLO-V8 network, we resized the test images to 1280x1280
pixels to maintain consistency in aspect ratio. This size was chosen as it
struck a balance between preserving details and reducing computational load.
The resized images were then fed into the trained network, generating three
scales of feature maps (80x80, 40x40, and 20x20). Each scale of prediction in
YOLO-V8 produced three bounding boxes per cell, containing information such as
coordinates, size, confidence score, and class probability. By setting a
threshold and applying Non-Maximum Suppression (NMS), low confidence boxes
were removed, and redundant detections were suppressed. Finally, the bounding
boxes were rescaled to the original image size to accurately locate the palm
trees. This process allowed for the efficient and accurate detection of palm
trees across various sizes and improved the overall performance of the system.

###  3.3 RPW map generation

In the process of generating an RPW map, the main focus is on detecting RPW
infestation in palm trees and localizing their presence in UAV images. This
involves a coordinated approach using recording devices attached to palm
trees, UAV imagery, and palm tree object detection algorithms.

To begin, recording devices are strategically deployed and attached to
individual palm trees, with the coordinates of each device pre-saved in a
cloud database. These devices capture the sound of the palm trees, which are
then subjected to sound analysis. By extracting CQCC features from the
recorded sound, important spectral and temporal characteristics are captured.

The CQCC features are passed through a previously trained InceptionNetV3
model, which has been trained on the TreeVibes dataset. This model classifies
whether a palm tree is infested or not infested with RPW based on sound
analysis. The classification results are stored, along with the coordinates of
each palm tree, to establish a link between the infestation status and
location.

Concurrently, UAVs outfitted with cameras gather high-resolution images of the
farm, focusing on the palm tree area of interest. These UAV captures show an
overview of the entire farm, including the palm trees.

The next step is to apply palm tree object detection algorithms to the UAV
photos, such as YOLOv8. These methods detect and localize palm tree
occurrences in images, returning bounding box coordinates for each discovered
palm tree.

Now comes the essential process of coordinate matching and visualization. The
discovered palm tree coordinates from the object detection stage are compared
to the palm tree coordinates recorded with the associated recording devices.
This matching procedure aids in establishing a link between the discovered
palm trees and their RPW infestation categorization.

The RPW map visualization is made once the matching is completed. The bounding
boxes of the identified palm trees are colored dependent on their RPW
infestation condition. If a palm tree is afflicted with RPW, the appropriate
bounding box is highlighted in red, showing the existence of the infestation.
If a palm tree is identified as not infected, the bounding box is depicted in
blue, indicating a healthy palm tree.

The RPW map is created by integrating the findings of sound analysis, RPW
classification, UAV images, and object recognition. This map depicts the
spread of RPW-infested palm trees on the farm, allowing for efficient
monitoring, targeted intervention, and effective RPW infestation control.

##  4 Experimentation and results

###  4.1 Dataset description

The experiments of our approach are carried out on a multi-modal dataset
composed of RPW sound data and UAV image data.

####  4.1.1 RPW sound data

In this paper, we utilize the TreeVibes dataset Rigakis et al. (2021),
consisting of over 54,000 10-second audio clips recorded from different types
of trees in various locations and seasons. The dataset provides labels
indicating tree species, location, season, weather, and time of day. It
includes a wide range of tree species across continents, capturing variations
in tree sounds across seasons and weather conditions. Each audio clip is
accompanied by metadata, such as GPS coordinates, date and time, temperature,
humidity, and air pressure. The dataset’s richness is summarized in Table 1.

Label |  Number of records
---|---
Not infested |  967
Infested |  53676
Table 1: Number of records in TreeVibes dataset

####  4.1.2 UAV image data

We have also collected a dataset consisting of aerial images from two
different locations, including a total of 349 images. The images were captured
using two different drones. The first drone used was a DJI Phantom 4 Pro
equipped with a DJI FC6310 camera, which has resolutions of 4864x3648 and
4096x2160. The second drone used was a DJI Mavic Pro equipped with a DJI FC220
camera, with a resolution of 4000x3000. To analyze and classify the images,
the dataset was manually labeled using the Labelbox platform. A total of
13,071 instances were labeled, with 11,150 instances identified as palm trees
and 1,921 instances labeled as other trees.

###  4.2 Results

####  4.2.1 DL classification based on sound data

We used a dataset of red palm weevil audio recordings, converting them into
images using CQCC feature extraction. The image dataset was split into
training, validation, and testing sets with ratios of 0.8, 0.1, and 0.1,
respectively. InceptionV3 was employed as the deep learning model, trained
with a batch size of 16, a learning rate of 0.0001, and 200 iterations. Adam
optimizer and binary cross-entropy loss function were utilized.

The trained InceptionV3 model was evaluated on the testing dataset, showing
highly promising results. It achieved perfect performance with an accuracy,
precision, recall, and F1-score of 1.000 across all metrics.

Validation accuracy and loss were visualized at different iterations. The
plots showed a steady improvement in accuracy and a decreasing trend in the
loss function throughout the training process.

(a) Mean validation accuracy over 10 iterations.

(b) Mean validation loss over 10 iterations.

Figure 2: Comparison of accuracy and loss.

####  4.2.2 YoloV8 for object detection

We trained the YOLOv8 object detection model on a carefully curated dataset
for palm tree detection. Our goal was to create a precise and robust model
capable of identifying palm trees in various contexts. The YOLOv8 large model
architecture was used with 60 epochs of training and an image size of 1280.
Evaluation results showed the effectiveness of our trained YOLOv8 model for
palm tree detection. The precision achieved was 0.841, correctly identifying
palm trees in 84.1% of instances. The recall value reached 0.865, successfully
detecting 86.5% of actual palm trees in the dataset. To assess overall
accuracy, we used Mean Average Precision (mAP) metrics. The mAP@50 value was
0.899, indicating high accuracy with a 50% confidence threshold. The mAP@50-95
value was 0.541, showing decreased performance with a wider range of
confidence thresholds.

Figure 3 presents the training progress for the palm trees object detection
matrices.

Figure 3: Precision, Recall, mAP50, and mAp50-95 history.

####  4.2.3 RPW map generation

The RPW map generation process has yielded promising results in detecting and
visualizing RPW infestation in palm trees on the farm. The results provide
valuable insights into the distribution of RPW-infested palm trees, aiding in
effective management strategies. Figure 4 showcases the RPW map, displaying
the UAV image with bounding boxes representing infested and not infested palm
trees.

Figure 4: RPW Map Visualizing Infested and Not Infested Palm Trees.

##  5 Conclusion

In this study, our proposed approach utilizing IoT and multi-modal data offers
a promising solution for the early detection and mapping of Red Palm Weevil
(RPW) infestations in palm trees. By converting sound data into images using
the CQCC feature and employing the InceptionV3 model for classification, we
achieved 100% accuracy in RPW detection, a significant milestone in RPW
management. The integration of the YOLOv8 model for palm tree detection in UAV
images and RPW mapping further enhances the effectiveness of our approach. The
generated RPW distribution map provides valuable insights for farm managers to
take targeted actions and protect the health and productivity of palm tree
ecosystems. However, implementing the proposed approach on a larger scale,
particularly in industrial palm tree farms, brings both advantages and
challenges. Geographic variations, such as differences in vegetation density
or topography, can affect the accuracy of palm tree recognition in UAV images.
Therefore, these factors need to be carefully considered throughout the
system’s development and deployment. While the adoption of the suggested
strategy may entail initial financial investments for IoT devices, sensors,
and data infrastructure, it can lead to significant cost savings by enabling
early identification and mapping of RPW infestations. Prompt intervention and
targeted actions based on accurate information not only reduce economic losses
caused by RPW infestations but also protect the productivity of palm tree
ecosystems. Conducting cost-effectiveness studies considering long-term
benefits and potential yield losses is essential to determine the economic
feasibility of adopting this approach. The scalability of IoT devices and
sensors facilitates comprehensive monitoring in large plantation areas, but
managing the substantial volume of generated data and ensuring the reliability
and maintenance of the IoT infrastructure pose challenges. Efficient data
processing, analysis, and storage infrastructure, along with high-performance
computing systems, are necessary to support the integration of deep learning
algorithms. Future research should focus on validating and generalizing the
proposed approach across different regions and palm tree species, while
exploring the integration of additional sensing technologies such as thermal
imaging or multi-spectral analysis, may provide complementary information for
improved RPW detection and mapping.

## References

  * Ammar and El-Naggar [2011] M.I. Ammar and M.A. El-Naggar.  Date palm (Phoenix dactylifera L.) fungal diseases in najran, saudi arabia.  _International Journal of Plant Pathology_ , 2(3):126–135, 2011.
  * [2] Top 10 date producing countries in the world.  https://ecob.com.br/top-10-date-producing-countries-in-the-world/.  Accessed on 26 July 2020.
  * Haraz et al. [2021] Osama M. Haraz, Waleed Saad, Mohamed Mamdouh M. Ali, and Tayeb A. Denidni.  Early detection and prevention of red palm weevil along with irrigation management system.  In _2021 IEEE USNC-URSI Radio Science Meeting (Joint with AP-S Symposium)_ , pages 56–57, 2021.  doi:10.23919/USNC-URSI51813.2021.9703522.
  * Ashry et al. [2022] Islam Ashry, Biwei Wang, Yuan Mao, Mohammed Sait, Yujian Guo, Yousef Al-Fehaid, Abdulmoneim Al-Shawaf, Tien Khee Ng, and Boon S Ooi.  Cnn–aided optical fiber distributed acoustic sensing for early detection of red palm weevil: A field experiment.  _Sensors_ , 22(17):6491, 2022.
  * Wibowo et al. [2022] Hery Wibowo, Imas Sukaesih Sitanggang, Mushthofa Mushthofa, and Hari Agung Adrianto.  Large-scale oil palm trees detection from high-resolution remote sensing images using deep learning.  _Big Data and Cognitive Computing_ , 6(3):89, 2022\.
  * Yarak et al. [2021] Kanitta Yarak, Apichon Witayangkurn, Kunnaree Kritiyutanont, Chomchanok Arunplod, and Ryosuke Shibasaki.  Oil palm tree detection and health classification on high-resolution imagery using deep learning.  _Agriculture_ , 11(2):183, 2021.
  * Ferry et al. [2019] Matthieu Ferry, Renaud Cousin, David Chabernaud, and Franck Ferrero.  An effective strategy to obtain very rapidly the red palm weevil decline in an area planted with ornamental palms.  _Arabian Journal of Plant Protection_ , 37:188–197, 2019\.  doi:10.14429/djlit.39.8.11875.  URL https://scholar.google.com/scholar_lookup?title=.
  * Hajjaji et al. [2021a] Yosra Hajjaji, Wadii Boulila, and Imed Riadh Farah.  An improved tile-based scalable distributed management model of massive high-resolution satellite images.  _Procedia Computer Science_ , 192:2931–2942, 2021a.
  * Hajjaji et al. [2022] Yosra Hajjaji, Wadii Boulila, and Imed Riadh Farah.  Leveraging artificial intelligence techniques for smart palm tree detection: A decade systematic review.  _Procedia Computer Science_ , 207:2823–2832, 2022.
  * Ghandorh et al. [2022] Hamza Ghandorh, Wadii Boulila, Sharjeel Masood, Anis Koubaa, Fawad Ahmed, and Jawad Ahmad.  Semantic segmentation and edge detection—approach to road detection in very high resolution satellite images.  _Remote Sensing_ , 14(3):613, 2022.
  * Hajjaji et al. [2021b] Yosra Hajjaji, Wadii Boulila, Imed Riadh Farah, Imed Romdhani, and Amir Hussain.  Big data and iot-based applications in smart environments: A systematic review.  _Computer Science Review_ , 39:100318, 2021b.
  * Elshafie et al. [2019] Hazem Elshafie, José Faleiro, Hazem Abdel Farag El-Shafie, and José Romeno Faleiro.  Red palm weevil, Rhynchophorus Ferrugineus (coleoptera: Curculionidae): Global invasion, current management options, challenges, and future prospects characterization of local entomopathogenic bacillus strains.  _Arabian Journal of Plant Protection_ , 37:170–177, 2019\.  doi:10.14429/djlit.39.8.11875.  URL https://scholar.google.com/scholar_lookup?title=Red.
  * Jintasuttisak et al. [2022] Thani Jintasuttisak, Eran Edirisinghe, and Ali Elbattay.  Deep neural network based date palm tree detection in drone imagery.  _Computers and Electronics in Agriculture_ , 192:106560, 2022.
  * Driss et al. [2020] Maha Driss, Amani Aljehani, Wadii Boulila, Hamza Ghandorh, and Mohammed Al-Sarem.  Servicing your requirements: An fca and rca-driven approach for semantic web services composition.  _IEEE Access_ , 8:59326–59339, 2020.  doi:10.1109/ACCESS.2020.2982592.
  * Karar et al. [2022] Mohamed Esmail Karar, Abdel-Haleem Abdel-Aty, Fahad Algarni, Mohd Fadzil Hassan, MA Abdou, and Omar Reyad.  Smart iot-based system for detecting rpw larvae in date palms using mixed depthwise convolutional networks.  _Alexandria Engineering Journal_ , 61(7):5309–5319, 2022.
  * Koubaa et al. [2020] Anis Koubaa, Abdulrahman Aldawood, Bassel Saeed, Abdullatif Hadid, Mohanned Ahmed, Abdulrahman Saad, Hesham Alkhouja, Adel Ammar, and Mohamed Alkanhal.  Smart palm: An iot framework for red palm weevil early detection.  _Agronomy_ , 10(7):987, 2020.
  * Alsanea et al. [2022] Majed Alsanea, Shabana Habib, Noreen Fayyaz Khan, Mohammed F Alsharekh, Muhammad Islam, and Sheroz Khan.  A deep-learning model for real-time red palm weevil detection and localization.  _Journal of Imaging_ , 8(6):170, 2022.
  * Al Husaini et al. [2022] Mohammed Abdulla Salim Al Husaini, Mohamed Hadi Habaebi, Teddy Surya Gunawan, Md Rafiqul Islam, Elfatih AA Elsheikh, and FM Suliman.  Thermal-based early breast cancer detection using inception v3, inception v4 and modified inception mv4.  _Neural Computing and Applications_ , 34(1):333–348, 2022.
  * Ultralytics [2023] Ultralytics.  YOLOv8.  https://docs.ultralytics.com/models/yolov8/, 2023.  Accessed: [10/04/2023].
  * Lou et al. [2023] Haitong Lou, Xuehu Duan, Junmei Guo, Haiying Liu, Jason Gu, Lingyun Bi, and Haonan Chen.  Dc-yolov8: Small-size object detection algorithm based on camera sensor.  _Electronics_ , 12(10):2323, 2023.
  * Rigakis et al. [2021] Iraklis Rigakis, Ilyas Potamitis, Nicolaos-Alexandros Tatlas, Stelios M Potirakis, and Stavros Ntalampiras.  Treevibes: modern tools for global monitoring of trees for borers.  _Smart Cities_ , 4(1):271–285, 2021.