#  $4C$: A Computation, Communication, and Control Co-Design Framework for
CAVs

Liangkai Liu1, Shaoshan Liu2 and Weisong Shi1
1Wayne State University
2PerceptIn

###### Abstract

Connected and autonomous vehicles (CAVs) are promising due to their potential
safety and efficiency benefits and have attracted massive investment and
interest from government agencies, industry, and academia. With more computing
and communication resources are available, both vehicles and edge servers are
equipped with a set of camera-based vision sensors, also known as Visual IoT
(V-IoT) techniques, for sensing and perception. Tremendous efforts have been
made for achieving programmable communication, computation, and control.
However, they are conducted mainly in the silo mode, limiting the
responsiveness and efficiency of handling challenging scenarios in the real
world. To improve the end-to-end performance, we envision that future CAVs
require the co-design of communication, computation, and control. This paper
presents our vision of the end-to-end design principle for CAVs, called $4C$,
which extends the V-IoT system by providing a unified communication,
computation, and control co-design framework. With programmable
communications, fine-grained heterogeneous computation, and efficient vehicle
controls in $4C$, CAVs can handle critical scenarios and achieve energy-
efficient autonomous driving. Finally, we present several challenges to
achieving the vision of the $4C$ framework.

##  I Introduction

The recent proliferation of computing technologies, e.g., sensors, computer
vision, machine learning, and hardware acceleration, has promoted connected
and autonomous vehicles (CAVs) [1]. The global CAVs market is expected to grow
to $173.15 B by 2030, with shared mobility services contributing to 65.31%
[2]. In general, there are three main benefits of CAVs. The first benefit is
safety. According to the fatality analysis from the National Highway Traffic
Safety Administration (NHTSA), 94 percent of serious crashes are caused by
human error [3]. Compared with a human driver, the machine does not fatigue,
drunk, and speeding. The sensors are better at distance detection, blind space
detection, emergency obstacle avoidance, etc. The second benefit is efficiency
for many aspects: less traffic congestion, less fuel consumption, less
greenhouse gas emission, less travel time, etc. The improvement is because
CAVs get more comprehensive traffic information, and it has better planning
and controls than human drivers. The third benefit is that CAVs could support
third-party applications, including applications for public safety like AMBER
Alert and criminal face detection. All these applications leverage the on-
vehicle computation and communication resources.

CAV is one of the typical applications for V-IoT technologies [4] since it
relies on the underlying visual sensors with computation and communication
devices to understand the road and context environment. In the typical design
of CAVs, sensors usually include cameras, LiDAR, radar, GPS/GNSS, etc., and
computation platform includes accelerator for DNN inference, with supporting
vehicle communications like dedicated short-range communications (DSRC) or
cellular vehicle-to-everything (C-V2X). Besides, a real-time operating system,
drivers, and complex algorithms for sensor data processing are usually
implemented. However, compared with other V-IoT application scenarios like
smart city, the real deployment of CAVs becomes more challenging because it is
safety-critical. Gigabytes of raw sensor data are generated by the visual
sensors every second. How to process them in real-time with limited
computation and communication resources is the key to achieving CAVs.

Lots of efforts have been made to build V-IoT systems for CAVs. For visual
sensor data collection, auto-grade visual sensors like cameras, LiDAR, and
radar are developed, and the sensing range increased from less than 100 meters
to over 300 meters. For visual data processing, since the perception is mainly
based on deep neural networks (DNN), many accelerators have been introduced
into the computation platform, including the GPU-based NVIDIA DRIVE platform,
FPGA-based Zynq UltraScale+ platform, and ASIC-based Mobileye EyeQ5 [5]. DNN
compression techniques, including parameter pruning and quantization and low-
rank factorization, are designed to reduce the computation demands to decrease
the DNN inference time. The development of DSRC and C-V2X also promotes the
development of autonomous driving. Through DSRC, the vehicle could read basic
safety messages (BSM) from other vehicles. Besides, traffic lights, stop
signs, etc., can also be shared with the vehicle from the traffic
infrastructure. The IEEE and 3GPP have also made efforts to achieve high
reliability, low latency, and high throughput vehicle communications. The DSRC
standard is evolving towards 802.11bd, while C-V2X is transitioning toward NR
V2X. For vehicle control, vehicle drive-by-wire technology makes up the
interface from the driving systems to the vehicle’s control. Recent efforts in
the edge cloud system and 6G networks propose co-designing the computation,
communication, and caching for task offloading algorithms [6, 7]. However,
these approaches are still theoretical analyses and cannot satisfy safety-
critical CAVs applications. Therefore, current developments in computation,
communication, and control are conducted either in silo mode or in theory,
limiting the responsiveness and efficiency of handling challenging real-world
scenarios.

Generally, with tremendous computation and communication resources equipped,
V-IoT systems are deployed to both the vehicle and the edge server [4]. The
real challenge for supporting CAVs applications is coordinating the V-IoT
resources to handle complex road environments, especially critical scenarios
like snowing roads, heavy rain/fog, work-zone, etc. To improve the end-to-end
performance for state-of-the-art driving systems, we proposed our vision of
the $4C$ framework, which extends the V-IoT system by providing communication,
computation, and control co-design capability for CAVs applications. 4C is
built on several promising technologies in communication virtualization,
heterogeneous computation, and vehicle control. Unlike previous works trying
to enable CAVs applications in silo mode, $4C$ is the first programming
framework that provides unified APIs for managing the programmable
computation, communication, and control resources. It is supposed to be
deployed both on the vehicle and the edge server (V-IoT). $4C$ is composed of
three main components: programmable communications support, fine-grained
heterogeneous computation, and efficient vehicle controls. Programmable
communications make it possible to upload the most appropriate tasks onto the
edge server (V-IoT) for computation. Fine-grained heterogeneous computation
gives the programmer the ability to request heterogeneous by demand,
significantly improves the flexibility. Finally, efficient vehicle controls
are designed to make the feedback of the vehicle’s physical control system
available to the driving system. To illustrate the availability and importance
of the $4C$ framework, we discuss two case studies: autonomous driving in
critical scenarios, energy-efficient autonomous driving. Finally, we discuss
the challenges in naming, predictability, and security.

The rest of the paper is organized as follows. Section II presents the
background and motivations of this work. Section III discusses the proposed
$4C$ framework. Sections IV presents two case studies of the $4C$ framework.
Sections V presents the challenges and discussion. Section VI concludes the
paper.

##  II Background and Motivation

###  II-A Visual IoT

In the era of the internet of everything, billions of end devices are
connected to support applications like smart cities, surveillance systems,
unmanned aerial vehicles, autonomous vehicles, etc. [4]. Visual IoT is an
enabling technology which model traffic at device-level to support various
applications. V-IoT is built on top of video processing technologies since
rich visual sensors are deployed for environment sensing and perception [8].
However, the actual deployment of V-IoT still faces several challenges. One of
the biggest challenges is the visual data collection and process. Take
autonomous vehicles as an example. Gigabytes of raw sensor data are generated
every second and need to be processed in real-time while the vehicle is
running. Therefore, the design of the visual data compression algorithm should
cope with real-time requirements. Besides, the co-design of communication and
computation resources would increase the V-IoT system’s resource utilization.
Furthermore, edge computing technology enables smart infrastructures (edge
servers) equipped with sensors and computation and communication devices. More
sensing and perception tasks can be executed on the edge server and share the
perception results with the vehicle. Extending the current V-IoT system to
support safety-critical systems becomes essential.

###  II-B Autonomous Driving Systems

CAV is one of the typical applications for V-IoT technologies. However, unlike
other application scenarios like smart cities and surveillance systems, CAV is
built with dynamic wireless communications. It is a safety-critical scenario
with hard real-time requirements. These characteristics make it challenging
for the current V-IoT system to support CAVs applications [1]. Figure 1 shows
an overview of the end-to-end autonomous driving system. Typically, multiple
sensors are integrated to enable the vehicle’s sensing and perception. To
simplify the autonomous driving pipeline, we discuss a design which is purely
relied on cameras for sensing the environment. As shown in the figure, eight
primary components can be divided into three parts: sensing, perception, and
decision [9]. Sensing is the process of sensors that captures information from
the environment. Perception represents the understanding environment with
algorithms applied to the sensing data, including localization, detection,
semantic segmentation, and sensor fusion. The role of sensor fusion is to
collaborate with each component’s perception results and generate locations
for objects, lanes, and open space for the planning module. The decision is
composed of global planning, local planning, and vehicle control. Global
planning generates the routes between origin and destination, while local
planning generates the trajectory and controls on break, acceleration, and
steering. Finally, the control commands will be applied to the drive-by-wire
system and applied to the vehicle. In general, the Robot Operating System [10]
(ROS) is used to manage these components and provide data communications
between different components.

Figure 1: An overview of the end-to-end autonomous driving system.

###  II-C Why Are Vehicle Communications Important?

Massive innovations in computer vision, machine learning, and accelerators
improve the end-to-end autonomous driving systems’ performance. However,
accidents and fatalities caused by early deployed autonomous vehicles arise
from time to time. There are still several challenges in the development and
deployment of the autonomous driving system.

The biggest challenge is safety, which means the sensing, perception,
decision, and control should be finished in real-time. As a safety-critical
system, real-time means the guarantee to finish the driving tasks before the
deadline. According to [11], when the vehicle drives 40 km per hour in urban
areas and that autonomous functions should be effective every 1 m, each real-
time task’s execution should be less than 100ms. However, limited by DNN
algorithms’ performance and lack of adequate field testing, the real traffic
environment is too complicated for the current driving systems to handle. The
reliability of sensors makes this problem even more challenging. The quality
of images captured by cameras is affected by the lighting conditions [9].
Although the point clouds from LiDAR are precise and independent of
illumination, there can be noisy and sparse. Another challenge is the cost.
The cost of a level 4 autonomous driving vehicle can attain 300,000 dollars,
which the sensors and computing platform cost almost 200,000 dollars [1].
Besides, to ensure AV’s reliability and safety, a backup of hardware devices
may be necessary. The high cost of the autonomous driving vehicle remains a
significant obstacle to its complete deployment in the real world.

Vehicle communication mechanisms, e.g., DSRC, C-V2X, 5G, address the above
challenges from another perspective. Unlike the general approaches, the
vehicle communication-based approaches obtain traffic information like traffic
lights, vehicle speed, pedestrians, etc., from the traffic infrastructure,
decreasing computation demands for the vehicle’s embedded system. Besides,
with raw data updated from the vehicle, sensing and perception tasks are
executed at the edge server or the cloud. In this case, co-designing the
computation and communication resources for CAVs applications becomes a
fundamental problem. The main reason is that as a safety-critical system, the
driving system should be predictable. However, when the vehicle is driving
autonomously, both the available computation and communication resources are
unpredictable.

###  II-D Why Is Integration with Controls Important?

Typically, an essential interface for the computing system to communicate with
the real vehicle is the drive-by-wire system, which provides interfaces for
executing the driving commands. The drive-by-wire system sets up interactions
with lower layers (vehicle system) to ensure the control commands from the
upper layer (computing system) are executed correctly in real-time. However,
the lack of backward interaction with the computing system makes it impossible
for the upper layer to respond. Table I presents vehicle control delays for
acceleration, braking, and steering for selected commercialized vehicles.
Control delay is defined as the time interval between the control message’s
generation and execution. We can observe that the delay for any controls for
all these five vehicles is larger than 100ms, while the highest exceeds 480ms.
The control delay could significantly affect the vehicle’s safety. Therefore,
a fully autonomous driving system requires the co-design of computing and
control systems.

TABLE I: Vehicle control delay for acceleration, braking, and steering for
selected commercialized vehicles

Delay (ms) |  | Lincoln
---
MKZ
| Hongqi
---
H7
| Hongqi
---
EV
| NIO
---
ES8
| GAC Group
---
Aion LX
Acceleration | 280 | 200 | 484.2 | 120 | 236
Braking | 230 | 362 | 191.7 | 120 | 266
Steering | 136 | 128 | 124.8 | 108 | 120

There are several benefits of integrating vehicle controls with the driving
system; the first is safety. The difference between the control and odometer
output is road conditions like pod holes and icy roads. Under these
circumstances, human drivers are supposed to decrease the speed to ensure
safety, which means putting more effort into the sensing and perception
stages. The second benefit is fuel efficiency. If the vehicle’s control model
is integrated with the path planning model, it can generate a path with the
lowest fuel consumption with a safety guarantee. The third benefit is
convenience, which is more related to the passenger experience for autonomous
vehicles. Unlike human drivers, the machine-based system could create bang-
bang controls when control switches abruptly from one extreme to the other,
especially for emergency braking and obstacle avoidance. However, with V2X
communications and vehicle control integrated with the driving system, a wider
range of environment perception is achieved, and the vehicle controls could
become smoother.

##  III $4C$ Framework for CAVs

In the actual deployment of CAVs, how to leverage the computation,
communication resources to generate and execute adequate controls under a
dynamic traffic environment is a fundamental challenge. We propose the $4C$
framework, which co-designs the communication, computation, and controls to
solve this challenge. In this section, we first present the overview of $4C$.
Then we show a detailed discussion of computation, communication, and control,
respectively. Finally, we discuss the abstracted application programming
interface (APIs) in 4C.

###  III-A $4C$ Overview

The overview of $4C$ is shown in Figure 2. Generally, the autonomous driving
pipeline can be divided into five modules: sensing, perception, fusion,
planning, and decision, where each module has several tasks running in
parallel [1].As a safety-critical system, the driving system requires high
predictability and flexibility. To support the autonomous driving pipeline,
$4C$ has three main components, including programmable communications, fine-
grained heterogeneous computation, and efficient vehicle controls. On top of
these components, unified APIs are abstracted. $4C$ could be deployed both on
the vehicle and the edge server [8]. The key innovation for 4C is that it
provides unified APIs to manage the programmable computation, communication,
and control resources.

Figure 2: The overview of the $4C$ framework.

###  III-B Programmable Communications Support

As a complement to the vehicle’s embedded system, vehicle communications
enable the vehicle to access information and computation resources with a
broader range. Typically, other vehicle’s speed and control information is
obtained through communications. Some tasks are uploaded to the roadside unit
(RSU) or cloud for computation because they have more powerful computing
devices. All these capabilities require programmable communications, which
could provide flexibility. Traditionally, cellular communications like 4G,
LTE, and vehicle communications like DSRC are deployed in a “black-box”
fashion, where the communication hardware and software are plug-and-play. This
approach’s limitation is that the devices and software cannot configure,
contributing to low flexibility and programmability [12]. Recently, many
efforts have been made in pushing the softwarization and virtualization of
network resources, including software-defined radio (SDR), network function
virtualization (NFV), network slicing, etc.

The communication module in the $4C$ framework is built on SDR devices that
have reconfiguration capability. On top of that, open-source software includes
OpenAirInterface (OAI), and srsLTE are implemented to provide full-stack
cellular network functions [12]. Both OAI and srsLTE include the software
stack for the core network (EPC), the base station (eNodeB), and user
equipment (UE). Typically, the EPC and eNodeB are deployed on the traffic
infrastructure or cloud, while the UE is deployed on vehicles. This setting
makes the communication module provide 4G, LTE, 5G, and Cellular Vehicle-to-
Everything (C-V2X) links between vehicles and the edge server. In the vehicle
communications scenario, coverage of cellular links is limited, and lots of
handovers will happen when the vehicle moves at high speed. Therefore, network
management and slicing are introduced to monitor and allocate network
resources at the core and base stations. Besides, context information,
including the vehicle’s control, the vehicle’s path, the task’s priority,
etc., is necessary for more effective network resources management.
Previously, vehicular communications only share safety-related messages like
the Basic Safety Message (BSM) in DSRC. In contrast, the API module in $4C$
provides reconfiguration capabilities of network resources to design CAVs
applications.

###  III-C Fine-grained Heterogeneous Computation

Visual IoT sensors are one of the primary components of the computation
module, including cameras, LiDAR, Radar, GNSS, etc., which generate Gigabytes
of raw sensor data every second. Processing such a huge amount of sensor data
in real-time brings massive challenges for the driving systems of autonomous
vehicles. Due to its good performance and capability to process raw sensor
data, DNN has been widely utilized in the driving system. How to support DNN
models’ execution in real-time becomes the key in driving systems design.
Heterogeneous computation architecture is promising for better performance
with low cost and energy budget [13]. However, the flexibility of current
approaches to access heterogeneous computation resources is still limited,
contributing to a considerable waste of computation resources. Therefore, in
$4C$, we propose a fine-grained heterogeneous computation module that enables
the higher-level applications to share lower-level hardware.

From Figure 2, we can find four types of computing architectures are chosen,
including CPU, GPU, Field Programmable Gate Array (FPGA), and Application
Specific Integrated Circuit (ASIC) [13]. CPU is the most versatile
architecture in computing systems, and it supports generic computations,
especially good at logistic controls and sequential processing. GPU is
designed mainly for graphics processing. With many cores, GPU shows strength
in parallel processing like DNN model training and inference. However, the
power consumption of GPU is a big concern. FPGA is an architecture based on a
matrix of configurable logic blocks (CLB) connected by programmable
interconnects. Therefore, FPGA shows the best reconfiguration capability, and
it’s much more energy-efficient than GPU. However, the programmability is much
lower than the CPU and GPU. Unlike general-purpose architectures, ASIC is
customized architecture for a specific logic function. This design makes ASIC
shows better performance and energy efficiency than others, but the
flexibility is limited. A combination of these architectures is leveraged to
handle both general-purpose applications and customized functions.

Another challenge is how to access the heterogeneous hardware in fine-grained.
Many efforts have been made to split the DNN model inference into several sub-
tasks and execute them on the most suitable hardware. For example, the image
resizing and compression can be executed on the CPU, while the matrix addition
and multiplication are on GPU. Besides, the DNN model could also be split into
several layers and execute on a different architecture. $4C$ is designed to
support both DNN layers splitting and scheduling. To program on heterogeneous
architecture, $4C$ utilizes the hardware drivers in the operating system (OS)
with open-source libraries, including OpenCL, CUDA, and OneAPI. On top of
these libraries, $4C$ defines configuration scripts that set the destination
hardware with the maximum accelerator resources to manage the splitting and
scheduling of DNN layers flexibly. Similar to communication resources, APIs
are defined for higher-level CAVs applications to access fine-grained
heterogeneous computation.

###  III-D Efficient Vehicle Controls

Vehicle control is the last step of the autonomous driving pipeline, which
applies decisions on the wheels. The drive-by-wire system is the bond that
provides the interface to send control messages to the electronic control unit
(ECU) via the controller area network (CAN) bus [14]. Like the I/O bus in
computers, the CAN bus supports communications between different ECU and the
drive-by-wire system. Typically, the vehicle is equipped with up to 80 ECUs,
including engine control module (ECM), powertrain control module (PCM), Brake
Control Module (BCM), etc. Each of them is an embedded system to ensure the
real-time response to the commands read from the CAN bus. Take throttle
control as an example; the drive-by-wire system sends a throttle command to
the ECM via CAN bus, then the message will be sent to the motors to turn.
Meanwhile, the throttle position sensor will monitor the throttle and send
feedback to the ECM. The ECM will adjust the motor control to decrease the
difference to achieve the desired throttle control.

Unlike traditional design, in the control module of the $4C$ framework, the
drive-by-wire system is leveraged to bridge the driving system to ECUs like
ECM, PCM, and BCM. Besides, the drive-by-wire system also reads feedback from
the ECUs. The feedback of controls from the ECUs, the real-time diagnostics,
and other information from the vehicle’s sensor will be grouped and shared
with the autonomous driving applications. The feedback of controls to the
driving system sets up the stringent connection between the application and
the actual vehicle, making it possible to have more customized optimizations.

###  III-E APIs for Co-Design

From the design of the $4C$ framework, all the communication, computation, and
control resources and messages are abstracted as unified APIs for the
autonomous driving pipeline. This abstraction aims to enable each application
to access communication and computation resources in fine-grained and
customized for the specific vehicle. In summary, we can find that the APIs can
be divided into two categories: management APIs and data APIs. The management
APIs include managing communications bandwidth, computation resources,
controls to the vehicle through the drive-by-wire system, etc. In contrast,
data APIs are defined to enable each application to access runtime and context
information from the lower level communication, computation, and vehicle
control devices.

We could co-design communication, computation, and control for specific
applications through APIs, like energy-efficient autonomous driving,
autonomous driving in critical scenarios, etc.

##  IV Case Study

From the design of the $4C$ framework, all the communication, computation, and
control resources and messages are abstracted as unified APIs for the
autonomous driving pipeline. The $4C$ framework can support various CAVs
applications. This section presents two case studies of the $4C$ framework:
autonomous driving in critical scenarios and energy-efficient autonomous
driving.

###  IV-A Autonomous Driving in Critical Scenarios

Critical scenarios have always been a big challenge for the design and
deployment of fully autonomous driving vehicles. Critical scenarios refer to
the traffic environments that are extremely hard for the current driving
system to understand. Currently, the field-testing of autonomous driving
vehicles mostly happens in places with good weather and light conditions like
Arizona and Florida. However, accidents and fatalities happen because the real
traffic environment is much more complicated than the testing scenarios [1].
Besides, DNN-based approaches are widely used in perception and planning
tasks. It’s tough to explain how DNN learns the pattern and knowledge as a
“black-box” approach, contributing to many perception failure in the real
traffic environment. Other critical scenarios are currently less studied, like
snowing roads, heavy rain/fog, work-zone, etc. Current driving systems of
autonomous vehicles are not expected to handle these scenarios.

With the $4C$ co-design framework, some critical scenarios are addressed. The
vehicle can access traffic information from other vehicles and infrastructures
with the communication module, eliminating some perception failures. For
critical scenarios like snowing roads, heavy rain/fog, and work-zone, the
control module’s feedback to the driving system could make the vehicle
decelerate or emergency stop. Besides, with communications like 5G and C-V2X,
real-time high-definition (HD) maps are shared with the vehicle. Furthermore,
teleoperation could be launched to take over the vehicle’s control for safety.

###  IV-B Energy-Efficient Autonomous Driving

With massive tasks running on the driving systems, energy consumption has been
a big concern. Generally, the driving system’s total power consumption exceeds
1,000 watts, and this value would be doubled if considering a duplicated
system as a backup. Another major part of energy consumption is
fuel/electricity consumption, mainly affected by the engine model and
controls. Based on a report of the operating cost per mile for heavy-duty
trucks, the fuel cost accounted for 39 percent, which is the most significant
contributor to operational costs [15]. Reducing energy consumption improves
the vehicle’s mileage range and solving heat dissipation issues brings by
overheating.

Figure 3: A typical design of energy efficient autonomous driving based on
$4C$.

A practical energy-efficient autonomous driving system requires the co-design
of communication, computation, and control. Figure 3 shows a typical design
based on the $4C$ framework. Using the recorded control data, including the
throttle, brake, and torque, as the input while the corresponding fuel
consumption as the label, an engine fuel model could be trained, predicting
the instant and total fuel consumption for input controls. This engine fuel
model could be used in path planning and the autonomous driving pipeline’s
decision to get a trajectory with the lowest fuel consumption. To show the
availability of the $4C$ framework in energy-efficient controlling, we trained
a fuel rate prediction model based on recorded control data of throttle,
brake, torque, etc., from the vehicle’s Engine Management System (EMS). The
model is trained using AutoML, which achieves a coefficient of determination
($R^{2}$) to 0.97. Next, we apply this fuel rate prediction model to the
vehicle control module to generate the most energy-efficient control commands.
We evaluate the system through a road test which is around 14 kilometers and
costs 700 seconds. The comparisons of AutoML (Our Model) and state-of-the-art
energy-efficient control algorithm VT-CPFM are shown in Figure 4. The top
subfigure shows the altitude of the road test and the optimal speed from the
control module. With the optimal speed, the AutoML model and VT-CPFM model’s
total consumption is 10570 grams and 11240 grams, respectively, 5.97% of total
fuel saving on this road.

In addition to energy-efficient controls, the communication module enables the
vehicle to access a longer range of traffic information. The maximum range of
most autonomous vehicle sensors is less than 200 meters, but the range of
C-V2X communications goes to 1000 meters. Increasing the sensing range also
helps the planning module to get a more energy-efficient route. Furthermore,
with fine-grained access to heterogeneous computing architectures, energy-
efficient scheduling becomes practical, and the power consumption of the
driving system is also decreased.

Figure 4: Preliminary results of energy efficient autonomous driving by co-
designing vehicle planning (computation) with vehicle controls.

##  V Challenges and Discussion

This section presents several challenges and open problems to realize the
vision of $4C$, including naming, predictability, and security.

Naming Since $4C$ is supposed to be deployed both on vehicles and the edge
server, a naming scheme becomes an essential part to support programming,
addressing, and data communication between vehicles and traffic
infrastructures. Unlike cellular communications, vehicle communications are ad
hoc, and the duration of communication links are usually short because
vehicles are driving, and the V2X communication range is limited. A
traditional naming scheme like DNS cannot support this dynamic mobile
scenario. In the $4C$ framework, virtualized and programmable communications
allow the management of V2X communications in link-level, which brings higher
flexibility. However, mapping these V2X links uniquely with vehicles with low
complexity is still an open question for CAVs.

Predictability An autonomous driving vehicle relies on the driving system to
understand the road environment and get safe controls in real-time, requiring
the whole system’s predictability. The achieve the system level
predictability, each stage within the autonomous driving pipeline should be
predictable, which brings a massive challenge for $4C$’s communication and
computation modules. In 4C, with V2X communications and computations
abstracted in fine-grain granularity, the behavior of tasks with link-level
communications becomes more predictable. However, sophisticated designs are
still needed to leverage abstracted resources to guarantee a real-time
response.

Security As a complicated distributed system that combines computation,
communication, and control, security is one of the biggest challenges for the
actual deployment of CAVs applications. Currently, vehicle communication, like
DSRC, is not encrypted. Attacks could happen to any hardware and software of
the $4C$ framework. However, with all the APIs for lower-layer resources are
abstracted for higher-layer applications, plenty of mature techniques in
building secure computing systems could be applied to the vehicle system.

##  VI Conclusion

CAVs have attracted massive attention from both the academic and automotive
communities. The driving system plays a crucial role in understanding the
traffic environment and making decisions for CAVs. However, the limited
system-level reliability and the missing integration with vehicle
communications and controls narrow the testing of CAVs to constraint
scenarios. This paper presents our vision of a future driving system for CAVs,
called $4C$, which provides a unified communication, computation, and control
co-design framework. With two case studies, we present the usability of the
$4C$ framework. Finally, we discuss the challenges in the $4C$ framework.

## References

  * [1] L. Liu, S. Lu, R. Zhong, B. Wu, Y. Yao, Q. Zhang, and W. Shi, “Computing systems for autonomous driving: State-of-the-art and challenges,” _arXiv preprint arXiv:2009.14349_ , 2020.
  * [2] (2018) Global Autonomous Driving Market Outlook, 2018. [Online]. Available: https://www.prnewswire.com/news-releases/global-autonomous-driving-market-outlook-2018-300624588.html
  * [3] (2020) National Highway Traffic Safety Administration. Automated Vehicles for Safety. [Online]. Available: https://www.nhtsa.gov/technology-innovation/automatedvehicles#topic-benefits
  * [4] W. Ji, J. Xu, H. Qiao, M. Zhou, and B. Liang, “Visual IoT: Enabling internet of things visualization in smart cities,” _IEEE Network_ , vol. 33, no. 2, pp. 102–110, 2019.
  * [5] S. Grigorescu, B. Trasnea, T. Cocias, and G. Macesanu, “A survey of deep learning techniques for autonomous driving,” _Journal of Field Robotics_ , vol. 37, no. 3, pp. 362–386, 2020.
  * [6] S. Barbarossa, S. Sardellitti, E. Ceci, and M. Merluzzi, “The edge cloud: A holistic view of communication, computation, and caching,” in _Cooperative and Graph Signal Processing_. Elsevier, 2018, pp. 419–444.
  * [7] E. C. Strinati and S. Barbarossa, “6g networks: Beyond shannon towards semantic and goal-oriented communications,” _Computer Networks_ , vol. 190, p. 107930, 2021.
  * [8] W. Ji, B. Liang, Y. Wang, R. Qiu, and Z. Yang, “Crowd V-IoE: visual internet of everything architecture in AI-driven fog computing,” _IEEE Wireless Communications_ , vol. 27, no. 2, pp. 51–57, 2020.
  * [9] S. Liu, L. Li, J. Tang, S. Wu, and J.-L. Gaudiot, “Creating autonomous vehicle systems,” _Synthesis Lectures on Computer Science_ , vol. 6, no. 1, pp. i–186, 2017.
  * [10] M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, and A. Y. Ng, “ROS: an open-source robot operating system,” in _ICRA workshop on open source software_ , vol. 3, no. 3.2. Kobe, Japan, 2009, p. 5.
  * [11] S. Kato, E. Takeuchi, Y. Ishiguro, Y. Ninomiya, K. Takeda, and T. Hamada, “An open approach to autonomous vehicles,” _IEEE Micro_ , vol. 35, no. 6, pp. 60–68, 2015.
  * [12] L. Bonati, M. Polese, S. D’Oro, S. Basagni, and T. Melodia, “Open, programmable, and virtualized 5g networks: State-of-the-art and the road ahead,” _arXiv preprint arXiv:2005.10027_ , 2020.
  * [13] S.-C. Lin, Y. Zhang, C.-H. Hsu, M. Skach, M. E. Haque, L. Tang, and J. Mars, “The architectural implications of autonomous driving: Constraints and acceleration,” in _Proceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems_. ACM, 2018, pp. 751–766.
  * [14] R. Isermann, R. Schwarz, and S. Stolzl, “Fault-tolerant drive-by-wire systems,” _IEEE Control Systems Magazine_ , vol. 22, no. 5, pp. 64–81, 2002\.
  * [15] (2020) The Real Cost of Trucking – Per Mile Operating Cost of a Commercial Truck. [Online]. Available: https://www.thetruckersreport.com/infographics/cost-of-trucking/


# BEFUnet: A Hybrid CNN-Transformer Architecture for Precise Medical Image
Segmentation

Omid Nejati Manzari 111Corresponding author
omid_nejaty@alumni.iust.ac.ir${}^{\;\,,1}$ Javad Mirzapour Kaleybar2 Hooman
Saadat3 Shahin Maleki2
1 School of Electrical Engineering, Iran University of Science and Technology,
Tehran, Iran
2 Department of Computer Engineering, University College of Nabi Akram,
Tabriz, Iran
3 Department of Electrical Engineering, Qazvin Branch, Iran

###### Abstract

The accurate segmentation of medical images is critical for various healthcare
applications. Convolutional neural networks (CNNs), especially Fully
Convolutional Networks (FCNs) like U-Net, have shown remarkable success in
medical image segmentation tasks. However, they have limitations in capturing
global context and long-range relations, especially for objects with
significant variations in shape, scale, and texture. While transformers have
achieved state-of-the-art results in natural language processing and image
recognition, they face challenges in medical image segmentation due to image
locality and translational invariance issues. To address these challenges,
this paper proposes an innovative U-shaped network called BEFUnet, which
enhances the fusion of body and edge information for precise medical image
segmentation. The BEFUnet comprises three main modules, including a novel
Local Cross-Attention Feature (LCAF) fusion module, a novel Double-Level
Fusion (DLF) module, and dual-branch encoder. The dual-branch encoder consists
of an edge encoder and a body encoder. The edge encoder employs PDC blocks for
effective edge information extraction, while the body encoder uses the Swin
Transformer to capture semantic information with global attention. The LCAF
module efficiently fuses edge and body features by selectively performing
local cross-attention on features that are spatially close between the two
modalities. This local approach significantly reduces computational complexity
compared to global cross-attention while ensuring accurate feature matching.
BEFUnet demonstrates superior performance over existing methods across various
evaluation metrics on medical image segmentation datasets. The synergy of CNN
and transformer architectures in BEFUnet excels in handling irregular and
challenging boundaries, establishing it as a promising approach for advancing
medical image segmentation tasks. Our code is publicly available at GitHub.

##  1 Introduction

Medical image segmentation is of utmost importance in computer vision,
offering vital insights into anatomical regions for comprehensive analysis and
aiding healthcare professionals in injury visualization, disease monitoring,
and treatment planning. CNNs, particularly fully convolutional networks (FCNs)
[32] like U-Net [36], have emerged as dominant methods, showing remarkable
success in various medical applications such as cardiac, organ, and polyp
segmentation. The field of automated medical image segmentation has gained
significant attention for its potential to alleviate labor-intensive tasks for
radiologists. FCNs and their variants, including SegNet [2], U-Net [36],
DenseNet [4], and DeepLab [9], have played a crucial role in enhancing medical
diagnostics and treatment by enabling accurate segmentation from MRI, CT
scans, PET/CT scans, and other medical imaging modalities.

CNN-based approaches are widely favored for image segmentation due to their
powerful local translation invariance, representation learning capabilities,
and filter sharing properties. However, these methods have limitations in
capturing long-range relations and explicit global context, especially for
objects with large inter-patient variation in texture, scale, and shape.
Various strategies like dilated convolution [51], image pyramids [52], prior-
guided methods [11], multi-scale fusion [18], and self-attention mechanisms
[35] have been attempted to address these limitations in medical image
segmentation but have shown weaknesses in extracting global context features.
Among the successful architectures for medical image segmentation are the FCN
and the U-Net, both utilizing convolutional layers on encoder and decoder
modules. The U-Net [36], with its skip connections, has proven to be versatile
for segmentation tasks, but the convolution layer’s locality restriction
limits its representational power in capturing shape and structural
information crucial for medical image segmentation. The restricted performance
of CNN models in building long-range dependencies and global contexts further
hinders their effectiveness in image segmentation. Addressing these challenges
remains an ongoing pursuit in CNN architectures for medical image
segmentation.

The recent success of transformers in Natural Language Processing (NLP) [44]
has led to the development of vision transformers to overcome CNN limitations
in image recognition tasks [14]. Vision transformers leverage multi-head self-
attention (MSA) to establish long-range dependencies and capture global
contexts. However, vanilla vision transformers demand large amounts of data to
generalize and suffer from quadratic complexity. Various approaches like DeiT
[40], Swin Transformer, and pyramid vision transformer aim to address these
limitations. While transformers have achieved state-of-the-art (SOTA) results
in image classification and semantic segmentation, their high computation
power requirements hinder real-time applications, such as radiotherapy. To
address the limitations of CNN models, Vision Transformer (ViT) models utilize
the MSA mechanism and achieve SOTA performance compared to convolution-based
methods. Hybrid-transformer architectures integrate transformers with CNN-
based networks, while pure transformers like Swin-Unet and others apply
transformers to both the encoder and decoder for global feature representation
at multiple levels. Despite transformers’ ability to capture global
dependencies, they struggle with image locality and translational invariance,
affecting accurate segmentation of organ boundaries.

Recent advances in vision transformers emphasize the significance of multi-
scale feature representations. DS-TransUNet [29], CrossViT [7], and HRViT [18]
introduce dual-branch transformer architectures to extract contextual
information, enhance fine-grained features, and improve semantic segmentation.
In medical image processing, researchers explore multi-branch architectures to
address challenges, such as Valanarasu et al. [43] using undercomplete and
overcomplete encoders, Lin et al. [29] employing a dual-branch Swin
Transformer, and Zhu et al. [55] leveraging different branches for processing
various data modalities and achieving superior performance. These multi-branch
models show promise in tackling complex features in medical image processing.

Despite the ability of vision transformers to capture global contextual
representations, their self-attention mechanism can overlook low-level
features. Hybrid CNN-transformer approaches, such as TransUnet [8] and LeVit-
Unet [49], have been proposed to address this issue by combining the locality
of CNNs with the long-range dependency of transformers to encode both global
and local features in medical image segmentation. However, these approaches
face challenges in effectively combining high-level and low-level features
while maintaining feature consistency and utilizing multi-scale information
produced by the hierarchical encoder properly. Furthermore, many existing deep
learning-based methods for medical image segmentation tend to focus solely on
body features, neglecting the significance of edge information. Some studies,
like Kuang et al. [26] and Yang [50], attempted to incorporate edge features
by separating them from body features or using them as additional constraints.
However, these approaches did not fully exploit the potential of edge features
in medical image segmentation.

In this paper, we introduce an innovative network called Body and Edge Fusion
Unet (BEFUnet), which aims to achieve precise medical image segmentation by
enhancing the fusion of edge and body information. The proposed BEFUnet
consists of three key modules: a dual-branch encoder, a Double-Level Fusion
(DLF) module, and a Local Cross-Attention Feature fusion module (LCAF). The
dual-branch encoder is designed to simultaneously extract edge and body
information using a lightweight CNNs branch with pixel-wise convolution and a
hierarchical Transformer branch based on SwinTransformer. The LCAF effectively
fuses cross-modal features by considering features that are closely located in
position between the two branches, leading to improved accuracy while reducing
computational complexity. Moreover, the novel DLF module is a multi-scale
vision transformer that employs a cross-attention mechanism to fuse two
obtained feature maps. The proposed BEFUnet not only addresses the
aforementioned issues but also outperforms other methods in terms of various
evaluation metrics. Our main contributions:

$\bullet$ A novel hybrid method that combines the edge local semantic
information of CNN with the body contextual interactions of the transformer,
resulting in enhanced integration of complementary features. This approach
proves to be particularly advantageous in handling irregular and challenging
boundaries in medical image segmentation.

$\bullet$ We introduce a Double-Level Fusion module, which effectively fuses
coarse and fine-grained feature representations.

$\bullet$ Our experimental results underscore the remarkable effectiveness of
BEFUnet, as we meticulously trained and evaluated our model on three distinct
medical image segmentation datasets. In our comprehensive comparison, BEFUnet
consistently outperforms a diverse set of state-of-the-art models, affirming
its robustness and superiority in handling various segmentation challenges
across different datasets.

##  2 Related Works

###  2.1 CNN-based Segmentation Networks

In early medical image segmentation, traditional machine learning algorithms
[41] were widely used. However, the landscape changed with the emergence of
deep CNNs, U-Net [36], and its variants became powerful alternatives. The
simplicity and superior performance of U-shaped structures have made them
popular choices for both 2D and 3D medical image segmentation scenarios. In
the realm of medical image segmentation, the focus primarily shifted toward
the application of CNNs and their variations to achieve accurate and efficient
segmentation results. Researchers introduced various methods to address the
limitations of vanilla FCNs [32]. These methods include fusing the output of
different layers, using dilated convolutions [51], and employing context
modeling [52]. Furthermore, U-shaped encoder-decoder structures like U-Net and
its variants garnered significant attention as they offered further
enhancements in performance for medical imaging tasks. Among these
architectures, U-Net [36] has become the standard choice for medical image
analysis. It consists of a symmetric encoder and decoder network with skip
connections, which prove highly effective. Building upon this success, several
U-Net-like architectures have been proposed, such as Res-UNet [47], Dense-UNet
[4], Kiu-Net [43], and U-net++ [54]. Each of these variants offers
improvements and adaptations to cater to different medical imaging tasks. For
example, U-net++ introduces dense skip connections between modules, leading to
improved results. As a result, these structures have demonstrated outstanding
performance across various medical domains. Despite the remarkable progress
made with CNN-based methods in medical image segmentation, there are still
limitations in terms of segmentation accuracy and efficiency due to the
inherent locality of convolution operations and complex data access patterns.
Efforts continue to address these challenges and further improve the
capabilities of medical image segmentation models.

###  2.2 Vision Transformers

In recent years, Transformer-based models have demonstrated remarkable success
across various domains, particularly in natural language processing. These
models have achieved SOTA performance in tasks such as machine translation
[44]. Moreover, their application has extended to vision tasks with the
introduction of the Vision Transformer (ViT) [14]. The ViT has showcased an
impressive speed-accuracy trade-off for image recognition; however, it
requires pre-training on large datasets. To address this limitation, efforts
have been made to enhance ViT’s performance on ImageNet, resulting in Deit
[40], which incorporates training strategies for improved outcomes.
Additionally, another noteworthy vision Transformer is the Swin Transformer
[31], a hierarchical model that serves as an efficient and effective vision
backbone. The Swin Transformer has attained SOTA performance in image
classification, object detection, and semantic segmentation tasks.

###  2.3 Transformers for Medical Image Segmentation

In the context of medical segmentation, researchers have explored the
potential of Transformer-based models [46, 53, 17, 37]. Several variants have
been introduced, such as MedT [42], which integrates gated axial transformer
layers into existing architectures, and TransUNet [8], which combines
Transformers and CNNs to leverage their respective advantages. Moreover, Swin-
UNet [5] has adopted pure transformers within a U-shaped encoder-decoder
architecture for global semantic feature learning. These Transformer-based
approaches show promise in overcoming the limitations of CNN models, which
struggle with long-range dependency modeling due to their restricted receptive
fields. By incorporating both global and local information, these hybrid
models aim to improve feature extraction for 3D and 2D segmentation of medical
images [55, 3, 28].

To further enhance the fusion of edge and body information, the proposed
BEFUnet architecture stands out. BEFUnet employs a novel transformer-based
fusing scheme, ensuring feature consistency and richness in 2D medical image
segmentation. This approach addresses the limitations of simple feature-fusing
mechanisms found in other hybrid models and guarantees more effective feature
maps that contain rich information, leading to improved segmentation results.

Figure 1: The structure of our proposed BEFUnet. BEFUnet consists of a dual-
encoder that simultaneously extracts edge and body features. Moreover, it
incorporates an efficient fusion module called LCAF, which facilitates the
merging of edge and body features. Additionally, there is a DLF module
integrated into the skip connection of the encoder-decoder structure to fully
integrate features from adjacent scales.

##  3 Methodology

The framework of BEFUnet is depicted in Figure 1. It consists of a dual-branch
encoder, a cross-attention feature fusion module called LCAF, and a DLF
module, which effectively fuses coarse and fine-grained feature
representations. In the upcoming sections, we will provide a detailed
description of these components.

###  3.1 Dual-branch encoder

The dual-branch encoder comprises a body encoder and one edge encoder. The
edge encoder utilizes CNNs architecture with Pixel Different Convolution (PDC)
[39] blocks for extracting edge information effectively. On the other hand,
the body encoder employs Transformer architecture for capturing semantic
information using global attention.

####  3.1.1 Edge encoder

The extraction of edge features is crucial for image segmentation, and to
address the issue of inadequate edge information extraction by conventional
medical segmentation networks, a dedicated edge detection branch is
incorporated. As illustrated in Figure 1, the edge encoder consists of four
stages, each containing four PDC blocks for feature detection. Max-pooling
layers are utilized for downsampling the feature maps between stages to obtain
hierarchical features. The first stage expands the original 3-channel image to
C channels and reduces the feature maps’ size to 1/4, ensuring compatibility
with the body encoder’s output size.

The PDC block consists of a depth-wise convolution layer, a ReLU layer, and a
convolution layer with a kernel size of 1. A residual connection is also
included to aid in model training. Traditional convolutional networks face
difficulties in extracting edge-related information due to the lack of
explicitly encoded gradient information. To overcome this, PDC is introduced,
which integrates gradient information into convolutional operations and
enhances edge features.

Vanilla convolution calculates the weighted sum of pixel values within the
convolution kernel, while PDC calculates the weighted sum of differences
between pixel values within the kernel. The expressions for vanilla
convolution and PDC are given by Equations (1) and (2), respectively:

$y=f(x,\theta)=\sum_{i=1}^{k\times k}w_{i}\cdot x_{i}\quad(\text{vanilla
convolution})$ (1)

$y=f(\nabla x,\theta)=\sum_{(x_{i},{x}^{\prime}_{i})\in
P}w_{i}\cdot(x_{i}-x_{i}^{\prime})\quad(\text{PDC})$ (2)

Here, $w_{i}$ represents the weights in the k × k convolution kernel, $x_{i}$
and ${x}^{\prime}_{i}$ denote the pixels covered by the kernel, and $P$
represents the collection of selected pixel pairs in the local area covered by
the kernel.

To further enhance edge extraction, a supervision strategy [48] is employed,
where an edge map is generated for the output feature of each stage, and the
loss is computed between the generated edge maps and the ground truth.

####  3.1.2 Body encoder

The body encoder utilizes the Transformer architecture, specifically the Swin
Transformer [31], for encoding high-level feature representations with global
long-range modeling. The Swin Transformer employs a sliding window mechanism
to construct hierarchical features and is well-suited for segmenting organs
with irregular shapes.

Figure 1 illustrates the overall structure of the Swin Transformer in red
blocks, consisting of four stages. The first stage performs feature encoding
on the original image using a patch embedding layer and two Swin Transformer
blocks. The image is divided into patches of size $P\times P$, reshaped into
1D vectors, and projected to a C-dimensional space. Positional parameters are
added to encode positional information, followed by passing the sequence
through the Swin Transformer blocks. The subsequent stages downsample the
feature map and extract higher-level features using patch merging and Swin
Transformer blocks.

Each stage of the Swin Transformer comprises alternating arrangements of two
different types of blocks. The first block consists of Layer Normalization
(LN), Window-based Multi-head Self-Attention (W-MSA), Multi-Layer Perceptron
(MLP), and residual connections. The second block is similar, but it replaces
W-MSA with Shifted Window-based Multi-head Self-Attention (SW-MSA). The
operations within these blocks can be expressed using Equations (3)-(6).

$\displaystyle\ \hat{z}^{l}=W\mbox{-}MSA(LN(z^{l-1}))+z^{l-1}$ (3)
$\displaystyle\ z^{l}=MLP(LN(\hat{z}^{l}))+\hat{z}^{l}$ (4) $\displaystyle\
\hat{z}^{l+1}=SW\mbox{-}MSA(LN(z^{l}))+z^{l}$ (5) $\displaystyle\
z^{l+1}=MLP(LN(\hat{z}^{l+1}))+\hat{z}^{l+1}$ (6)

Between each stage, patch merging is performed to downsample the feature map
and gather contextual features. It merges adjacent $2\times 2$ patches into a
larger patch, reducing the number of patches and concatenating their
dimensions to minimize information loss. Patch merging downsamples the
features by a factor of 2. The output feature size of each stage is
$\frac{H}{4}\times\frac{W}{4}\times C$, $\frac{H}{8}\times\frac{W}{8}\times
2C$, $\frac{H}{16}\times\frac{W}{16}\times 4C$, and
$\frac{H}{32}\times\frac{W}{32}\times 8C$, respectively, assuming the input
image size is $H\times W\times 3$.

The extracted edge and body features are then fed into the LCAF module for
further fusion.

Figure 2: A block diagram of the LCAF.

###  3.2 LCAF module

The Local Cross-Attention Fusion (LCAF) module is designed to fuse edge and
body features accurately and efficiently by selectively performing cross-
attention on features that are spatially close between the two image
modalities. This approach overcomes the limitations of conventional cross-
attention methods, which require storing global information for each patch and
cannot leverage the one-to-one correspondence of pixels between modalities for
accurate matching and comprehensive fusion.

Figure 2 illustrates the structure of the LCAF module. It utilizes a
transformer block with local cross-attention to modulate features in close
proximity. The input edge and body features are divided into local regions,
projected into query, key, and value vectors, and attention scores are
calculated through dot product attention. The resulting vectors are weighted
and summed to obtain the fused feature representation. The LCAF module also
incorporates a multi-head operation to project vectors onto distinct subspaces
and compute cross-attention scores in diverse subspaces. The output of the
LCAF module is obtained by applying feed-forward networks (FFN) to the
modulated features, as described in Equation (9). Residual connections are
used to maintain feature integrity.

$LCA(X_{f})=softmax(\frac{X_{edge}W_{Q}(X_{body}W_{K})^{T}}{\sqrt{d_{k}}})X_{body}W_{v}$
(7)

$M\mbox{-}LCA(X_{f})=X_{f}+Concat[LCA(X_{f})_{1},...,LCA(X_{f})_{h}]W_{o}$ (8)

$X_{f}=FFN(M\mbox{-}LCA(X_{f}))+M\mbox{-}LCA(X_{f})$ (9)

In this context, $X_{body}$, $X_{edge}$, and $X_{f}$ refer to features
corresponding to the edge modality, body modality, and fused modality,
respectively. The learnable matrices $W_{k}$, $W_{Q}$, $W_{V}$, and $W_{o}$
represent the key matrix, query matrix, value matrix, and a matrix for output,
respectively. Compared to conventional global cross-attention, LCAF
significantly reduces computational complexity by performing selective local
cross-attention. The computational complexities of global cross-attention
(GCA) and Local Cross-Attention (LCA) are given by Equations (10) and (11),
respectively, assuming $h$ and $w$ patches in the features and $h_{l}$ and
$w_{l}$ patches in the LCA window.

$\displaystyle\ \Omega(M\mbox{-}GCA)=4hwC^{2}+2(hw)^{2}C$ (10) $\displaystyle\
\Omega(M\mbox{-}LCA)=4hwC^{2}+2h_{l}h_{w}\cdot hwC$ (11)

The LCAF module outputs the fused features, which are then fed into the next
stage.

Figure 3: The Cross Attention process entails several steps. Initially, the
class token of the small level, denoted as $CLS^{s}$, is projected for
dimension alignment and then appended to $P^{l}$. The resulting embedding
operates as both the key and value. Moreover, the query is made using
$CLS^{\prime s}$. Subsequently, attention computation and back projection are
performed to obtain $Z^{s}$. Noteworthily, this process can also be extended
to the large level.

####  3.2.1 Double-Level Fusion Module (DLF)

The primary obstacle lies in effectively merging CNN and Swin Transformer
level features while maintaining feature consistency. A direct approach would
be to input the sum of CNN levels and their corresponding Swin Transformer
levels into a decoder to obtain the segmentation map. However, this approach
fails to ensure feature consistency between the levels, resulting in subpar
performance. Therefore, we propose a novel module called DLF that addresses
this issue by taking the smallest ($P^{s}$) and largest ($P^{l}$) levels as
inputs and employing a cross-attention mechanism to fuse information across
scales.

Typically, shallow levels contain more precise localization information, while
deeper levels carry more semantic information that is better suited for the
decoder. To balance the trade-off between computational cost and the marginal
effect of the middle-level feature map on model accuracy, we opted not to
include the middle level in the feature fusion process, thus saving
computational resources. Consequently, we encourage representation by multi-
scaling the shallowest ($P^{s}$) and the last ($P^{l}$) levels while
preserving localization information.

In the proposed DLF module, the class token plays a significant role as it
summarizes all the information from the input features. Each level is assigned
a class token derived from global average pooling (GAP) over the level’s norm.
The class tokens are obtained as follows:

$\displaystyle CLS^{s}$ $\displaystyle=GAP(Norm(P^{s}))$ (12) $\displaystyle
CLS^{l}$ $\displaystyle=GAP(Norm(P^{l}))$

where $CLS^{s}$ $\in R^{4D^{\prime}\times 1}$ and $CLS^{l}$ $\in
R^{D^{\prime}\times 1}$. The class tokens are then concatenated with the
respective level embeddings before being passed into the transformer encoders.
The small level is followed by $S$ transformer encoders, and the large level
is followed by $L$ transformer encoders to compute global self-attention.
Notably, a learnable position embedding is also added to each token of both
levels to incorporate position information into the transformer encoders’
learning process.

After passing the embeddings through the transformer encoders, the features of
each level are fused using the cross-attention module. Prior to fusion, the
class tokens of the two levels are swapped, meaning the class token of one
level is concatenated with the tokens of the other level. Each new embedding
is then individually fed through the module for fusion and finally back-
projected to its original level. This interaction with tokens from the other
level allows the class tokens to share rich information across levels.

In particular, for the small level, this interaction is illustrated in Fig. 3.
$f^{s}(.)$ first projects $CLS^{s}$ to the dimensionality of $P^{l}$, and the
resulting output is denoted as $CLS^{\prime s}$. $CLS^{\prime s}$ is
concatenated with $P^{l}$ to serve as the key and value for the cross-
attention computation, while also independently acting as the query. Since
only the class token is queried, the cross-attention mechanism operates in
linear time. The final output $Z^{s}$ can be mathematically represented as
follows:

$\displaystyle y^{s}=f^{s}(CLS^{s})+MCA(LN([f^{s}(CLS^{s})\parallel P^{l}]))$
$\displaystyle Z^{s}=[P^{s}\parallel g^{s}(y^{s})]$ (13)

###  3.3 Loss function

Since the encoder is designed to detect both edge and body features
simultaneously, we employ two loss components: $L_{edge}$ and $L_{body}$.

####  3.3.1 Edge supervision loss

For edge detection, a significant majority of samples are negative, requiring
the use of an annotator-robust loss [30].

We apply this loss function to the edge maps generated by each stage of the
encoder. For the $i$-th pixel in the $j$-th edge map, denoted as $y^{j}_{i}$,
the loss is computed according to the following conditions:

$l^{j}_{i}=\begin{cases}\alpha\cdot\log(1-y^{j}_{i})&\text{if }y_{i}=0\\\
0&\text{if }0<y_{i}<\eta\\\ \beta\cdot\log
y^{j}_{i}&\text{otherwise}\end{cases}$ (14)

Here, $y^{j}_{i}$ represents the predicted value of the $i$-th pixel in the
$j$-th edge map, and $\eta$ is a predefined threshold. Pixels that are
annotated as positive by annotators with a proportion smaller than $\eta$ are
considered negative samples. The proportion of negative samples in the dataset
is denoted as $\beta$. Additionally, we define $\alpha=\lambda\cdot(1-\beta)$,
where $\lambda$ is a hyperparameter used to balance positive and negative
samples. The overall $L_{edge}$ loss is obtained by summing the losses of each
pixel:

$L_{edge}=\sum_{i,j}l^{j}_{i}$ (15)

####  3.3.2 Body supervision loss

Medical image segmentation faces a severe class imbalance problem, which we
address by employing a combination of Binary Cross-Entropy Loss and Dice Loss
[34] as the body loss.

Binary Cross-Entropy Loss measures pixel-level prediction errors and is
suitable for most semantic segmentation scenarios. It can be expressed as
follows:

$L_{Bce}=-\sum_{i=0}^{N}[(1-\hat{y}_{i})\log(1-y_{i})+\hat{y}_{i}\log(y_{i})]$
(16)

Dice Loss is a widely used loss function in medical image segmentation,
effectively handling imbalanced samples. It is given by the following formula:

$L_{Dice}=1-2\times\frac{2\sum_{i=0}^{N}y_{i}\hat{y}i}{\sum{i=0}^{N}(y_{i}+\hat{y}_{i})}$
(17)

The $L_{body}$ loss can be expressed as a combination of Binary Cross-Entropy
Loss and Dice Loss:

$L_{body}=\lambda_{1}L_{Bce}+\lambda_{2}L_{Dice}$ (18)

Finally, combining $L_{edge}$ and $L_{body}$ with a weighted hyperparameter
$\gamma$, we obtain the final loss $L$:

$L=L_{body}+\gamma\cdot L_{edge}$ (19)

Table 1: Comparison results of the proposed method on the Synapse dataset.
Blue indicates the best result, and red displays the second-best.

Methods | DSC $\uparrow$ | HD $\downarrow$ | Aorta | Gallbladder | Kidney(L) | Kidney(R) | Liver | Pancreas | Spleen | Stomach
---|---|---|---|---|---|---|---|---|---|---
DARR [15] | 69.77 | - | 74.74 | 53.77 | 72.31 | 73.24 | 94.08 | 54.18 | 89.90 | 45.96
V-Net [34] | 68.81 | - | 75.34 | 51.87 | 77.10 | 80.75 | 87.84 | 40.05 | 80.56 | 56.98
U-Net [36] | 76.85 | 39.70 | 89.07 | 69.72 | 77.77 | 68.60 | 93.43 | 53.98 | 86.67 | 75.58
R50 U-Net [8] | 74.68 | 36.87 | 87.74 | 63.66 | 80.60 | 78.19 | 93.74 | 56.90 | 85.87 | 74.16
Att-UNet [38] | 77.77 | 36.02 | 89.55 | 68.88 | 77.98 | 71.11 | 93.57 | 58.04 | 87.30 | 75.75
R50 Att-UNet [8] | 75.57 | 36.97 | 55.92 | 63.91 | 79.20 | 72.71 | 93.56 | 49.37 | 87.19 | 74.95
R50 ViT [8] | 71.29 | 32.87 | 73.73 | 55.13 | 75.80 | 72.20 | 91.51 | 45.99 | 81.99 | 73.95
Swin-Unet [5] | 79.13 | 21.55 | 85.47 | 66.53 | 83.28 | 79.61 | 94.29 | 56.58 | 90.66 | 76.60
TransUnet [8] | 77.48 | 31.69 | 87.23 | 63.13 | 81.87 | 77.02 | 94.08 | 55.86 | 85.08 | 75.62
LeVit-Unet [49] | 78.53 | 16.84 | 78.53 | 62.23 | 84.61 | 80.25 | 93.11 | 59.07 | 88.86 | 72.76
DeepLabv3+ (CNN) [10] | 77.63 | 39.95 | 88.04 | 66.51 | 82.76 | 74.21 | 91.23 | 58.32 | 87.43 | 73.53
M2SNet [53] | 79.70 | 21.39 | 85.48 | 66.89 | 83.03 | 79.12 | 94.40 | 56.26 | 90.11 | 76.97
UTNetV2 [16] | 78.35 | 20.88 | 87.57 | 64.75 | 84.21 | 80.20 | 94.77 | 58.93 | 89.23 | 75.46
MISSFormer [22] | 79.74 | 19.65 | 85.31 | 66.47 | 83.37 | 81.65 | 94.52 | 63.49 | 91.51 | 79.63
BEFUnet | 80.47 | 16.26 | 87.03 | 73.89 | 85.23 | 80.47 | 95.49 | 60.31 | 91.56 | 81.54

Figure 4: Segmentation results of the proposed method on the Synapse dataset.
The red rectangles identify organ regions where the superiority of our
proposed method can be clearly seen.

##  4 Experiments

###  4.1 Dataset

Synapse Multi-Organ Segmentation: To assess the performance of HiFormer, we
conducted evaluations using the synapse multi-organ segmentation dataset [6].
This dataset consists of 30 cases containing 3779 axial abdominal clinical CT
images. Each CT volume comprises 85 to 198 slices of 512x512 pixels, with a
voxel spatial resolution ranging from $([0.54\sim 0.54]\times[0.98\sim
0.98]\times[2.5\sim 5.0])mm^{3}$.

Multiple Mylomia Segmentation: We evaluated our methodology on the multiple
myeloma cell segmentation grand challenges provided by SegPC 2021 [19, 20].
The challenge dataset includes a training set with 290 samples, as well as
separate validation and test sets with 200 and 277 samples, respectively.

Skin Lesion Segmentation: Extensive experiments were conducted on skin lesion
segmentation datasets, including the ISIC 2017 dataset [13]. This dataset
consists of 2000 dermoscopic images for training, 150 for validation, and 600
for testing. Additionally, we used the ISIC 2018 dataset [12] and followed the
methodology described in previous literature [45] to split the dataset into
train, validation, and test sets. We also incorporated the $\mathrm{PH}^{2}$
dataset [33], which is a dermoscopic image database designed for segmentation
and classification tasks. The $\mathrm{PH}^{2}$ dataset contains 200
dermoscopic images, including 160 nevi and 40 melanomas.

###  4.2 Implementation Details

The proposed model was implemented using the PyTorch framework and all
experiments were conducted on an Nvidia Titan RTX 24G GPU. For improved
computational efficiency, the resolution of all training, validation, and test
images was set to $224\times 224$. To achieve better model initialization, we
employed PiDiNet [39] as the pre-trained model for the edge encoder and swin-
tiny [31] for the body encoder.

During training, the proposed model utilized the AdamW optimizer with a weight
decay of 0.01. The initial learning rate was set to 0.01, and the
ReduceLROnPlateau algorithm was employed for learning rate scheduling. A batch
size of 24 and 80 training epochs were chosen. The weight parameters
$\lambda_{1}$, $\lambda_{2}$, and $\gamma$ were set to 0.6, 0.4, and 0.2,
respectively.

Furthermore, we utilize ReduceLROnPlateau for learning rate optimization. All
experiments are performed on four datasets using identical train, validation,
and test data. Additionally, we train other SOTA models with default
parameters, while simultaneously employing a pretrained ViT model during the
training of TransUNet and LeViT-UNet. On the other hand, the remaining models
are trained from scratch.

##  5 Evaluation Results

We assessed the accuracy performance of our proposed method using commonly
employed metrics, such as Sensitivity (SE), Specificity (SP), Accuracy (ACC),
Intersection over Union (IoU), Dice coefficient (Dice), and Hausdorff Distance
(HD). To ensure a fair and unbiased comparison, we evaluated BEFUnet against
CNN and transformer-based methods, as well as models combining elements of
both approaches.

###  5.1 Results of Synapse Multi-Organ Segmentation

Table 1 presents a comparison of our proposed BEFUnet with previous SOTA
methods on the Synapse multi-organ CT dataset. The experimental results
demonstrate that our Unet-like transformer method achieves outstanding
performance, with a segmentation accuracy of 80.47% (DSC) and 16.26% (HD). Our
algorithm show significant improvement in the DSC evaluation metric compared
to Swin-Unet [5] and the recently introduced LeVit-Unet [49], we achieved
noteworthy accuracy enhancements of approximately 1% and 5% on the HD
evaluation metric, indicating superior edge prediction capabilities. Notably,
BEFUnet consistently outperforms existing literature in the segmentation of
most organs, especially for the kidney, stomach, and liver.

Figure 4 displays the segmentation results of various methods on the Synapse
dataset, revealing that these approaches generally struggle with edge region
segmentation of organs. Our proposed BEFUnet, on the other hand, effectively
addresses this challenge by leveraging a powerful edge encoder, resulting in
enhanced segmentation, particularly in complex background scenarios.

Table 2: Performance evaluation on the SegPC challenge 2021

Method | Accuracy | Precision | Recall | F1-score | mIoU
---|---|---|---|---|---
R2U-Net [1] | 0.933 | 0.852 | 0.831 | 0.834 | 0.744
U-Net [36] | 0.939 | 0.842 | 0.879 | 0.855 | 0.766
Unet++ [54] | 0.942 | 0.855 | 0.876 | 0.857 | 0.770
DoubleU-Net [23] | 0.937 | 0.833 | 0.896 | 0.858 | 0.763
Att-UNet [35] | 0.940 | 0.845 | 0.866 | 0.849 | 0.757
ResUNet++ [24] | 0.934 | 0.838 | 0.858 | 0.840 | 0.736
UNet3+ [21] | 0.939 | 0.848 | 0.866 | 0.852 | 0.766
LeViT-UNet [49] | 0.939 | 0.850 | 0.837 | 0.837 | 0.738
TransUNet [8] | 0.939 | 0.822 | 0.869 | 0.838 | 0.741
M2SNet [53] | 0.945 | 0.839 | 0.866 | 0.848 | 0.751
UTNetV2 [16] | 0.926 | 0.848 | 0.845 | 0.842 | 0.743
MISSFormer [22] | 0.947 | 0.865 | 0.871 | 0.851 | 0.752
BEFUnet | 0.951 | 0.873 | 0.908 | 0.871 | 0.776

Figure 5: Visual representation of the proposed method on the SegPC cell
segmentation dataset.

###  5.2 Results of Multiple Mylomia Segmentation

In the domain of medical image analysis, it is common to encounter medical
images that contain multiple classes of objects requiring segmentation. To
address this requirement, we evaluated all models using the SegPC-2021
dataset, which comprises two distinct types of cells. The quantitative results
can be found in Table 2. Our proposed model outperforms other SOTA approaches,
achieving a higher Accuracy score of 95.1.6%, which represents a significant
improvement of 1.2% over Swin-Unet and a 1.3% increase in F1-score compared to
the DoubleU-Net architecture.

Furthermore, in Figure 5, we provide visual illustrations of the segmentation
outputs produced by our proposed model and LeVit-Unet. As demonstrated, our
predictions align closely with the provided ground truth masks, indicating the
effectiveness of our approach. One of the notable advantages of BEFUnet is its
capability to model multi-scale representations. This property enables it to
effectively suppress background noise, which is particularly valuable when
dealing with datasets featuring highly overlapped backgrounds, such as SegPC.
In summary, BEFUnet surpasses CNN-based methods that rely solely on local
information modeling, as well as transformer-based counterparts, which exhibit
poorer performance in this context.

Table 3: Results of BEFUnet on ISIC2017 dataset compared with the latest methods. Method | Dice | SE | SP | ACC | IoU
---|---|---|---|---|---
U-Net [36] | 0.783 | 0.806 | 0.954 | 0.933 | 0.696
Att U-Net [35] | 0.808 | 0.800 | 0.978 | 0.915 | 0.717
UNet++ [54] | 0.832 | 0.830 | 0.965 | 0.925 | 0.743
FocusNet [25] | 0.832 | 0.767 | 0.980 | 0.921 | 0.756
DAGAN [27] | 0.859 | 0.835 | 0.976 | 0.935 | 0.771
DoubleU-Net [23] | 0.845 | 0.841 | 0.967 | 0.933 | 0.760
TransUnet [8] | 0.841 | 0.807 | 0.979 | 0.932 | 0.755
FAT-Net [45] | 0.850 | 0.840 | 0.973 | 0.933 | 0.765
Swin-Unet [5] | 0.862 | 0.842 | 0.950 | 0.936 | 0.764
M2SNet [53] | 0.853 | 0.828 | 0.964 | 0.931 | 0.751
UTNetV2 [16] | 0.831 | 0.791 | 0.972 | 0.927 | 0.742
MISSFormer [22] | 0.863 | 0.848 | 0.980 | 0.939 | 0.764
Ours(BEFUnet) | 0.868 | 0.853 | 0.985 | 0.946 | 0.768
Figure 6: Visual comparison with different SOTA methods on the ISIC 2017
dataset. The green contours are ground truth, and the red contours are the
segmentation results of indicated methods.

###  5.3 Results of Skin Lesion Segmentation

Table 3 displays the comparison results for the ISIC 2017 benchmarks. Our
method stands out with the highest scores for most metrics, achieving 86.8%
for Dice, 85.3% for SE, 98.5% for SP, 94.6% for ACC, , and 76.8% for IoU.

Additionally, we visually compared the segmentation results of several
representative competitors in our experiments, namely U-Net, Swin-Net, and
FAT-Net (Figure 6). The observations reveal that our method consistently
outperforms the competition, particularly in challenging examples. U-Net
struggles to accurately predict skin lesions when there is low contrast
between foreground and background pixels due to the lack of sufficient global
contextual information. FAT-Net, by incorporating a pyramidal module for
multi-scale feature extraction, mitigates some of the weaknesses in global
context information extraction, but it still produces mis-segmentation in
complex boundary cases. In contrast, our BEFUnet successfully integrates
multi-scale encoders to capture both edge features and body context
information. This enables us to achieve superior segmentation results compared
to other competitors, especially in challenging scenarios involving ambiguous
brightness contrast and boundary situations.

Table 4: The ablation experiment results of BEFUnet on the Synapse dataset.

Method |  DSC$\uparrow$ |  HD$\downarrow$ | Aorta | Gallbladder | Kidney(L) | Kidney(R) | Liver | Pancreas | Spleen | Stomach
---|---|---|---|---|---|---|---|---|---|---
Baseline | 70.19 | 30.88 | 80.70 | 67.08 | 72.85 | 65.03 | 89.92 | 55.53 | 82.22 | 68.18
Baseline + DLF | 71.72 | 27.97 | 81.59 | 68.73 | 75.05 | 67.96 | 91.15 | 56.03 | 84.13 | 69.09
Baseline + EE | 74.54 | 25.84 | 84.98 | 69.27 | 75.43 | 69.16 | 92.53 | 57.20 | 87.18 | 73.58
Baseline + EE + LCAF | 78.91 | 20.86 | 86.89 | 71.01 | 81.57 | 76.18 | 92.86 | 58.14 | 87.62 | 75.19
Baseline + EE + LCAF + DLF (BEFUnet) | 80.47 | 16.26 | 87.03 | 73.89 | 85.23 | 80.47 | 95.49 | 60.31 | 91.56 | 81.54

##  6 Ablation Study

Efficiency of Different Components Ablation experiments were performed to
showcase the effectiveness of various components within the LCAUnet model. The
baseline configuration consists of a single-branch U-shaped architecture
comprising a body encoder and a decoder, utilizing a straightforward
concatenation of adjacent-scale features. Subsequently, the baseline network
was enhanced by individually incorporating the LCAF module, Edge-Encoder (EE),
and DLF module, resulting in three distinct methods denoted as baseline+DLF,
baseline+EE, and baseline+EE+LCAF. Comparative evaluations were carried out on
these modified networks with different modules using the Synapse dataset.

In Table 4, we present a comparison between the baseline method and two
augmented methods: baseline+DLF and baseline+EE. The results demonstrate
notable improvements for both augmented methods on the Synapse dataset.
Specifically, the baseline+DLF method shows a remarkable enhancement of
approximately 2.91% in the HD metric and approximately 1.53% in the DSC
metric. Similarly, the baseline+EE method achieves significant improvements,
with approximately 5.04% lower HD and approximately 4.35% higher DSC. These
results underscore the importance of the DLF module, as it effectively
facilitates the fusion of features across adjacent scales. Additionally, the
Edge-Encoder proves to be crucial in extracting edge information, leading to
enhanced segmentation performance. Furthermore, the combination of
baseline+EE+LCAF method offers even better results, with a further improve of
4.98% in the HD metric and 4.37% in the DSC metric compared to the baseline+EE
method. This further highlights the valuable role played by the LCAF module in
promoting the fusion of edge and body information, contributing to improved
performance. Ultimately, by integrating all three modules into our
comprehensive approach (baseline+EE+LCAF+DLF), we achieve the best performance
across all metrics. Specifically, this approach surpasses the baseline method
by approximately 10.28% in DSC, and 14.62% in HD metrics on the Synapse
dataset.

##  7 Discussion

Our extensive experiments conducted on various medical image segmentation
datasets showcase the effectiveness of our novel BEFUnet model when compared
to traditional CNN and transformer-based approaches. Our approach encompasses
two key advancements. To enhance the design rationality of the network, we
have incorporated Pixel Different Convolution and Transformer into the initial
layers. Furthermore, the LCAF module facilitates feature reuse by merging the
Edge features from PDC with the body features provided by the transformer
module. The quantitative evaluation of the BEFUnet network on three
challenging datasets demonstrates its exceptional segmentation performance,
outperforming SOTA methods in most cases. This result is also supported by
visual analysis, as depicted in Fig. 4, which shows noise-free segmentation of
organs such as the Kidney and Liver, consistent with the quantitative
benchmarks. However, our model may encounter failure cases in certain
instances (e.g., Aorta), aligning with the numerical results. Additionally,
our model still faces notable challenges when dealing with low-contrast skin
images. Overall, BEFUnet has demonstrated the ability to effectively learn
critical anatomical relationships from medical images.

##  8 Conclusions

this paper makes three significant contributions. Firstly, we introduce the
concept of pixel difference convolution, which combines the strengths of
traditional edge encoders and deep CNNs. This integration results in robust
and precise edge segmentation. Secondly, we propose a novel medical image
segmentation structure called BEFUnet, which extracts both body and edge
features and effectively integrates them to enhance segmentation performance.
To achieve this, we construct a local cross-modal fusion module, LCAF, that
combines body and edge modalities for feature fusion. Furthermore, we employ a
DLF module to achieve finer feature fusion from the aforementioned
representations. Lastly, we conduct comprehensive experiments on three
publicly available datasets to evaluate the proposed model. Compared to
existing methods, the BEFUnet model demonstrates superior performance. It
accurately segments boundary-blurred, irregular, and interference-present
organs regions, while exhibiting strong generalization capabilities. The
results validate that the proposed BEFUnet model outperforms most SOTA
methods.

## References

  * [1] Md Zahangir Alom, Mahmudul Hasan, Chris Yakopcic, Tarek M Taha, and Vijayan K Asari.  Recurrent residual convolutional neural network based on u-net (r2u-net) for medical image segmentation.  arXiv preprint arXiv:1802.06955, 2018.
  * [2] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  IEEE transactions on pattern analysis and machine intelligence, 39(12):2481–2495, 2017.
  * [3] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai.  Transfusion: Robust lidar-camera fusion for 3d object detection with transformers.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1090–1099, 2022.
  * [4] Sijing Cai, Yunxian Tian, Harvey Lui, Haishan Zeng, Yi Wu, and Guannan Chen.  Dense-unet: a novel multiphoton in vivo cellular image segmentation model based on a convolutional neural network.  Quantitative imaging in medicine and surgery, 10(6):1275, 2020.
  * [5] Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang.  Swin-unet: Unet-like pure transformer for medical image segmentation.  arXiv preprint arXiv:2105.05537, 2021.
  * [6] MICCAI 2015 Multi-Atlas Abdomen Labeling Challenge.  Synapse multi-organ segmentation dataset.  https://www.synapse.org/#!Synapse:syn3193805/wiki/217789, 2015.  Accessed: 2022-04-20.
  * [7] Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda.  Crossvit: Cross-attention multi-scale vision transformer for image classification.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 357–366, 2021.
  * [8] Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L Yuille, and Yuyin Zhou.  Transunet: Transformers make strong encoders for medical image segmentation.  arXiv preprint arXiv:2102.04306, 2021.
  * [9] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE transactions on pattern analysis and machine intelligence, 40(4):834–848, 2017.
  * [10] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In Proceedings of the European conference on computer vision (ECCV), pages 801–818, 2018.
  * [11] Feng Cheng, Cheng Chen, Yukang Wang, Heshui Shi, Yukun Cao, Dandan Tu, Changzheng Zhang, and Yongchao Xu.  Learning directional feature maps for cardiac mri segmentation.  In Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part IV 23, pages 108–117. Springer, 2020.
  * [12] Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al.  Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic).  arXiv preprint arXiv:1902.03368, 2019.
  * [13] Noel CF Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al.  Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic).  In 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), pages 168–172. IEEE, 2018.
  * [14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [15] Shuhao Fu, Yongyi Lu, Yan Wang, Yuyin Zhou, Wei Shen, Elliot Fishman, and Alan Yuille.  Domain adaptive relational reasoning for 3d multi-organ segmentation.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 656–666. Springer, 2020.
  * [16] Yunhe Gao, Mu Zhou, Di Liu, and D Metaxas.  A multi-scale transformer for medical image segmentation: Architectures, model efficiency, and benchmarks.  arXiv preprint arXiv:2203.00131, 2022.
  * [17] Yunhe Gao, Mu Zhou, Di Liu, Zhennan Yan, Shaoting Zhang, and Dimitris N Metaxas.  A data-scalable transformer for medical image segmentation: architecture, model efficiency, and benchmark.  arXiv preprint arXiv:2203.00131, 2022.
  * [18] Jiaqi Gu, Hyoukjun Kwon, Dilin Wang, Wei Ye, Meng Li, Yu-Hsin Chen, Liangzhen Lai, Vikas Chandra, and David Z Pan.  Multi-scale high-resolution vision transformer for semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12094–12103, 2022.
  * [19] Anubha Gupta, Ritu Gupta, Shiv Gehlot, and Shubham Goswami.  Segpc-2021: Segmentation of multiple myeloma plasma cells in microscopic images, 2021.
  * [20] Anubha Gupta, Pramit Mallick, Ojaswa Sharma, Ritu Gupta, and Rahul Duggal.  Pcseg: Color model driven probabilistic multiphase level set based tool for plasma cell segmentation in multiple myeloma.  PloS one, 13(12):e0207908, 2018.
  * [21] Huimin Huang, Lanfen Lin, Ruofeng Tong, Hongjie Hu, Qiaowei Zhang, Yutaro Iwamoto, Xianhua Han, Yen-Wei Chen, and Jian Wu.  Unet 3+: A full-scale connected unet for medical image segmentation.  In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1055–1059. IEEE, 2020.
  * [22] Xiaohong Huang, Zhifang Deng, Dandan Li, and Xueguang Yuan.  Missformer: An effective medical image segmentation transformer.  arXiv preprint arXiv:2109.07162, 2021.
  * [23] Debesh Jha, Michael A Riegler, Dag Johansen, Pål Halvorsen, and Håvard D Johansen.  Doubleu-net: A deep convolutional neural network for medical image segmentation.  In 2020 IEEE 33rd International symposium on computer-based medical systems (CBMS), pages 558–564. IEEE, 2020.
  * [24] Debesh Jha, Pia H Smedsrud, Michael A Riegler, Dag Johansen, Thomas De Lange, Pål Halvorsen, and Håvard D Johansen.  Resunet++: An advanced architecture for medical image segmentation.  In 2019 IEEE international symposium on multimedia (ISM), pages 225–2255. IEEE, 2019.
  * [25] Chaitanya Kaul, Suresh Manandhar, and Nick Pears.  Focusnet: An attention-based fully convolutional network for medical image segmentation.  In 2019 IEEE 16th international symposium on biomedical imaging (ISBI 2019), pages 455–458. IEEE, 2019.
  * [26] Hulin Kuang, Yixiong Liang, Ning Liu, Jin Liu, and Jianxin Wang.  Bea-segnet: Body and edge aware network for medical image segmentation.  In 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 939–944. IEEE, 2021.
  * [27] Baiying Lei, Zaimin Xia, Feng Jiang, Xudong Jiang, Zongyuan Ge, Yanwu Xu, Jing Qin, Siping Chen, Tianfu Wang, and Shuqiang Wang.  Skin lesion segmentation via generative adversarial networks with dual discriminators.  Medical Image Analysis, 64:101716, 2020.
  * [28] Yingwei Li, Adams Wei Yu, Tianjian Meng, Ben Caine, Jiquan Ngiam, Daiyi Peng, Junyang Shen, Yifeng Lu, Denny Zhou, Quoc V Le, et al.  Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17182–17191, 2022.
  * [29] Ailiang Lin, Bingzhi Chen, Jiayu Xu, Zheng Zhang, Guangming Lu, and David Zhang.  Ds-transunet: Dual swin transformer u-net for medical image segmentation.  IEEE Transactions on Instrumentation and Measurement, 2022.
  * [30] Yun Liu, Ming-Ming Cheng, Xiaowei Hu, Kai Wang, and Xiang Bai.  Richer convolutional features for edge detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3000–3009, 2017.
  * [31] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021.
  * [32] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.
  * [33] Teresa Mendonça, Pedro M Ferreira, Jorge S Marques, André RS Marcal, and Jorge Rozeira.  Ph 2-a dermoscopic image database for research and benchmarking.  In 2013 35th annual international conference of the IEEE engineering in medicine and biology society (EMBC), pages 5437–5440. IEEE, 2013\.
  * [34] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi.  V-net: Fully convolutional neural networks for volumetric medical image segmentation.  In 2016 fourth international conference on 3D vision (3DV), pages 565–571. Ieee, 2016.
  * [35] Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et al.  Attention u-net: Learning where to look for the pancreas.  arXiv preprint arXiv:1804.03999, 2018.
  * [36] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [37] Jiacheng Ruan, Mingye Xie, Jingsheng Gao, Ting Liu, and Yuzhuo Fu.  Ege-unet: an efficient group enhanced unet for skin lesion segmentation.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 481–490. Springer, 2023.
  * [38] Jo Schlemper, Ozan Oktay, Michiel Schaap, Mattias Heinrich, Bernhard Kainz, Ben Glocker, and Daniel Rueckert.  Attention gated networks: Learning to leverage salient regions in medical images.  Medical image analysis, 53:197–207, 2019.
  * [39] Zhuo Su, Wenzhe Liu, Zitong Yu, Dewen Hu, Qing Liao, Qi Tian, Matti Pietikäinen, and Li Liu.  Pixel difference networks for efficient edge detection.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 5117–5127, 2021.
  * [40] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In International Conference on Machine Learning, pages 10347–10357. PMLR, 2021.
  * [41] Andy Tsai, Anthony Yezzi, William Wells, Clare Tempany, Dewey Tucker, Ayres Fan, W Eric Grimson, and Alan Willsky.  A shape-based approach to the segmentation of medical imagery using level sets.  IEEE transactions on medical imaging, 22(2):137–154, 2003.
  * [42] Jeya Maria Jose Valanarasu, Poojan Oza, Ilker Hacihaliloglu, and Vishal M Patel.  Medical transformer: Gated axial-attention for medical image segmentation.  In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part I 24, pages 36–46. Springer, 2021.
  * [43] Jeya Maria Jose Valanarasu, Vishwanath A Sindagi, Ilker Hacihaliloglu, and Vishal M Patel.  Kiu-net: Towards accurate segmentation of biomedical images using over-complete representations.  In International conference on medical image computing and computer-assisted intervention, pages 363–373. Springer, 2020.
  * [44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [45] Huisi Wu, Shihuai Chen, Guilian Chen, Wei Wang, Baiying Lei, and Zhenkun Wen.  Fat-net: Feature adaptive transformers for automated skin lesion segmentation.  Medical Image Analysis, 76:102327, 2022.
  * [46] Hanguang Xiao, Li Li, Qiyuan Liu, Xiuhong Zhu, and Qihang Zhang.  Transformers in medical image segmentation: A review.  Biomedical Signal Processing and Control, 84:104791, 2023.
  * [47] Xiao Xiao, Shen Lian, Zhiming Luo, and Shaozi Li.  Weighted res-unet for high-quality retina vessel segmentation.  In 2018 9th international conference on information technology in medicine and education (ITME), pages 327–331. IEEE, 2018.
  * [48] Saining Xie and Zhuowen Tu.  Holistically-nested edge detection.  In Proceedings of the IEEE international conference on computer vision, pages 1395–1403, 2015.
  * [49] Guoping Xu, Xingrong Wu, Xuan Zhang, and Xinwei He.  Levit-unet: Make faster encoders with transformer for medical image segmentation.  arXiv preprint arXiv:2107.08623, 2021.
  * [50] Haonan Yang and Dapeng Yang.  Cswin-pnet: A cnn-swin transformer combined pyramid network for breast lesion segmentation in ultrasound images.  Expert Systems with Applications, 213:119024, 2023.
  * [51] Fisher Yu and Vladlen Koltun.  Multi-scale context aggregation by dilated convolutions.  arXiv preprint arXiv:1511.07122, 2015.
  * [52] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881–2890, 2017.
  * [53] Xiaoqi Zhao, Hongpeng Jia, Youwei Pang, Long Lv, Feng Tian, Lihe Zhang, Weibing Sun, and Huchuan Lu.  Msnet: Multi-scale in multi-scale subtraction network for medical image segmentation.  arXiv preprint arXiv:2303.10894, 2023.
  * [54] Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang.  Unet++: A nested u-net architecture for medical image segmentation.  In Deep learning in medical image analysis and multimodal learning for clinical decision support, pages 3–11. Springer, 2018.
  * [55] Zhiqin Zhu, Xianyu He, Guanqiu Qi, Yuanyuan Li, Baisen Cong, and Yu Liu.  Brain tumor segmentation based on the fusion of deep semantics and edge information in multimodal mri.  Information Fusion, 91:376–387, 2023.