# HOPPY: An Open-source Kit for Education with Dynamic Legged Robots

Joao Ramos1,2, Yanran Ding1, Young-woo Sim1, Kevin Murphy1, and Daniel Block2
Authors are with the 1Department of Mechanical Science and Engineering and the
2Department of Electrical & Computer Engineering at the University of Illinois
at Urbana-Champaign, USA<EMAIL_ADDRESS>
###### Abstract

This paper introduces HOPPY, an open-source, low-cost, robust, and modular kit
for robotics education. The robot dynamically hops around a rotating gantry
with a fixed base. The kit is intended to lower the entry barrier for studying
dynamic robots and legged locomotion with real systems. It bridges the
theoretical content of fundamental robotic courses with real dynamic robots by
facilitating and guiding the software and hardware integration. This paper
describes the topics which can be studied using the kit, lists its components,
discusses preferred practices for implementation, presents results from
experiments with the simulator and the real system, and suggests further
improvements. A simple heuristic-based controller is described to achieve
velocities up to $1.7m/s$, navigate small objects, and mitigate external
disturbances when the robot is aided by a counterweight. HOPPY was utilized as
the subject of a semester-long project for the Robot Dynamics and Control
course at the University of Illinois at Urbana-Champaign. The positive
feedback from the students and instructors about the hands-on activities
during the course motivates us to share this kit and continue improving in the
future.

##  I Introduction

The imminent robotics revolution will employ robots as ubiquitous tools in our
lives. Many machines are already being widely used in factories, assembly
lines, and, more recently, in automated warehouses [9]. However, most of the
tasks performed by these robots are quasi-static, which means that the robot
can stop mid-motion without destabilizing (falling down). In contrast, humans
possess the capability to efficiently execute dynamic tasks such as running or
weightlifting, where the task cannot be interrupted mid-motion. For instance,
a runner cannot instantaneously freeze motion between steps without falling;
an Olympic weightlifter cannot statically lift the payload above his/her head.
Thus, to make robots with human-level abilities accessible in the future, we
must train the next generation of roboticists to create machines that are
capable of robustly performing dynamic physical tasks. Dynamic motions such as
legged locomotion impose unique challenges regarding mechanical robustness,
actuation saturation, control rate, state estimation, and more. However,
performing experiments with dynamic robot motions is challenging because
capable hardware is expensive and not readily available, and errors can
quickly lead to terminal hardware damage [10]. To address this issue, many
researchers created platforms focused on physical robustness, low-cost, and
modularity [11, 22].

Figure 1: The hopping robot HOPPY for hands-on education in dynamic control
and legged locomotion. Available at https://github.com/robodesignlab/hoppy-
project.

So why create another open-source robot for education? Although remarkable
platforms such as the Open Dynamics Solo ($4800) [7] and the Stanford Doggo
($3000) [12] have a significantly lower cost than most legged robots used for
research, their use in large-scale education is still impractical. Firstly,
the base cost of acquiring tens of robot units for a typical robotics class
would be formidable. In addition, fabrication equipment is needed, such as CNC
machining and 3D printing, and impractical for most courses. Moreover, these
robots demand base-level knowledge of mechanical fabrication and electronic
prototyping, in addition to low-level programming skills for communication,
controls, and simulation. These requirements prohibit the use of existing
open-source robot platforms in large cross-disciplinary Engineering courses,
especially at the undergraduate level. Feasible educational kits must be
student-friendly, mechanically robust, and easy to fabricate, maintain, and
program [15]. Other commercial kits such as the Robotis Biolod ($1,200), and
SoftBank’s NAO ($14,000) have easy-to-use hardware and software, but are
driven by highly geared servo motors that limit their ability to perform
dynamic motions like hopping [24]. On the other hand, toy-grade products such
as LEGO Mindstorms ($350) with plastic parts, do not have the mechanical
robustness to sustain continuous impacts inherent to dynamic legged
locomotion. To fill the gap in large-scale robotic education, this paper
introduces HOPPY (Fig. 1), a robust and low-cost kit designed for studying
robotics through dynamic legged locomotion. The kit costs under $500, does not
require any fabrication skills, and can be built exclusively with off-the-
shelf components, allowing the students, not the instructor, to assemble the
kit and making it a viable option for online and remote education. In addition
to the hardware, the kit includes a physics simulator based on MATLAB, a
software adopted by most Universities. The kit covers many of the topics of
fundamental robotics courses such as kinematics, dynamics, position and force
control, trajectory planning, physics simulation, and more. Although
sophisticated controllers can be used to push robot performance, students can
achieve stable dynamic hopping with simple heuristics [17], as illustrated by
the simple controller we present. As a tool for nurturing active-learning in
robotics, the kit was implemented as a hands-on and semester-long project for
the Spring 2020 Robot Dynamics and Control (ME446) course at the University of
Illinois at Urbana-Champaign (UIUC) (http://youtu.be/6O6czr0YyC8). This hands-
on approach to education is becoming increasingly popular due to its proven
effect on enabling enduring, deeper, and more significant learning [19, 16].

The contributions of this manuscript include open-sourcing the robot
mechanical, kinematic, and dynamic models, a baseline hopping controller, its
MATLAB-based simulator with instructions, the bill of materials (BoM), and
assembly instructions videos. In addition, we list the basic robotics-related
topics we covered in the course and how they relate to HOPPY. We suggest
hardware modifications for other educational applications and possible
improvements if the budget permits. Finally, we discuss our experience
implementing the kit in the ME446 course at UIUC and address future work. This
manuscript and the kit are intended for educators interested in implementing
hands-on robotics activities, and for students, researchers, and hobbyists
interested in experimentally learning about dynamic robots and legged
locomotion.

##  II Kit design and components

HOPPY is designed exclusively with off-the-shelf components to lower its cost,
enable modularity, customization, and facilitate maintenance. In addition, it
is lightweight (total weight about $3.8kg$), portable, and mechanically
robust. All components withstanding heavy load are designed to be metal parts
available from https://gobilda.com/. We intentionally avoid the use of plastic
3D printed parts for their short working life under impact loads. Because of
the kit’s durability, only the initial cost for setting up the course is
required and the equipment can be reutilized in future semesters with minimal
maintenance costs. The kit available at
https://github.com/robodesignlab/hoppy-project includes:

  * •

Complete list of components and quantities (BoM)

  * •

CAD model files in SolidWorks2019 and .STEP

  * •

Video instructions for mechanical assembling (https://youtu.be/CDxhdjob2C8)

  * •

List of parameters for rigid-body dynamics (link dimensions, mass, center of
mass location, and inertia tensor)

  * •

Kinematic and dynamic models

  * •

The baseline controller described in Section III-B

  * •

The MATLAB code for dynamic simulation with instructions (version 2016a or
later)

  * •

Electrical wiring diagram and basic code for the microcontroller ($\mu$C).

###  II-A Mechanical System

Fig. 1 shows that the mechanical system of the HOPPY kit consists of a robot
leg attached to a gantry, similar to the experimental set up for legged robot
research [6] [5]. The two joints of the gantry ($\theta_{1,2}$) are passive,
allowing the robot to translate in a circle and jump vertically. The hip motor
on joint 3 ($\theta_{3}$) drives link 3, while the knee motor on joint 4
($\theta_{4}$) drives link 4. As shown in Fig. 2, both motors are placed near
the hip joint to reduce the moving inertia of the leg and enable fast swing
motions. The second actuator is mounted coaxially with the hip joint and
drives the knee joint through a timing belt. The motors have a minimal gearing
ratio in order to be backdrivable and mitigate impacts with the ground [24].
To reduce the torque requirements for hopping, springs are added in parallel
with the knee, and a counterweight is attached at the opposite end of the
gantry. We employ cheap $2.3kg$ ($5lb$) weights typically utilized for
gymnastics. The addition of an excessively heavy counterweight reduces the
achievable frictional forces between the foot and the ground, and the robot is
more likely to slip. However, this low-actuation design strategy provides
inherent safety for human-robot-interaction, and elongates the working life of
the system.

Figure 2: Kinematic transformations and the five rigid-bodies which compose
HOPPY. The first two joints ($\theta_{1}$ and $\theta_{2}$) are passive while
joints three and four are driven by electric motors.

###  II-B Electrical System

The diagram of electrical connections and communications protocols are
depicted in Fig. 3. The user computer communicates with the embedded $\mu$C
(Texas Instruments LAUNCHXL-F28379D) via a USB port. The $\mu$C commands the
desired voltage to the motors (goBilda 5202-0002-0027 for the hip and 0019 for
the knee) via Pulse Width Modulation (PWM) to the drivers (Pololu VNH5019) and
receives analog signals proportional to the motor current and the pulse counts
from the incremental encoders. The motor drivers are powered by a 12V power
supply (ALITOVE 12Volt 10 Amp 120W). A linear potentiometer with spring return
(TT Electronics 404R1KL1.0) attached in parallel with the foot works as a foot
contact switch.

Figure 3: Electrical components of the kit and communication diagram.

###  II-C Modularity and customization

The kit is designed to facilitate the adjustment of physical parameters and
the replacement of components. For instance, goBilda provides motor options
with a wide range of gear ratios from $3.7:1$ to $188:1$ to be easily adapted
to the HOPPY system according to the needs. The length of the lower leg is
continually tunable between $60mm$ and $260mm$, and knee springs can be added
or removed as deemed necessary. The gantry length is also continually
adjustable between $0.1m$ and $1.1m$ and its height has multiple discrete
options between $0.2m$ and $1.1m$. The dynamic effect of the counterweight can
be adjusted by shifting its position further away from the gantry joint or
adding more mass.

##  III Education with Dynamic Legged Robots

This section describes how the HOPPY kit supports the curriculum of typical
fundamental robotics courses. As shown in Fig. 2, the HOPPY platform consists
of a two degrees-of-freedom (DoF) active leg attached to a supporting gantry
with two passive DoFs, forming a serial chain. To make HOPPY successfully hop,
students employ concepts of classical Robotics textbooks [21, 4, 20, 14]
including:

  1. 1.

Coordinate frames and transformations

  2. 2.

Forward, inverse, and differential kinematics

  3. 3.

Forward, inverse, and contact dynamics

  4. 4.

Actuation, gearing, and friction

  5. 5.

Trajectory planning

  6. 6.

Position and force control

HOPPY can be integrated into a robotics course curriculum as a semester-long
project. During the first half of the course, students derive the mathematical
models and explore the simulator while assembling their kits. The second half
of the course is dedicated to experimentation and exploration. Section V
describes our approach using this format at UIUC.

###  III-A Kinematics, Dynamics, and Actuation

The complete kinematic transformations and dynamic equations of HOPPY can be
found in the separate document HOPPY_Technical_Guideline.pdf in the
repository, and here we summarize the key concepts. The coordinate frames of
HOPPY are presented in Fig. 2, where the origin coordinate frame at the base
of the rotating gantry is fixed to the ground. Joints $\theta_{1,2}$ are
passive while joints $\theta_{3,4}$ are driven by electric brushed motors.
These angles define the robot joint configuration vector
$q=[\theta_{1}\quad\theta_{2}\quad\theta_{3}\quad\theta_{4}]^{\top}$ and the
coordinate transformations between link-fixed frames. The rigid-body
properties of each link are provided to the students to derive the standard
dynamic equations of motion (EoM) from energy principles using the system’s
Lagragian. The derived manipulator equations [21] is:

$M(q)\ddot{q}+C(q,\dot{q})\dot{q}+G(q)=B_{e}u+J_{c}^{\top}F_{GRF},$ (1)

where $\dot{q}$ and $\ddot{q}$ are the joint velocity and acceleration; $M(q)$
represents the symmetric and positive definite inertia matrix as a function of
the robot joint configuration $q$; $C(q,\dot{q})$ is the Coriolis and
centrifugal forces matrix; $G(q)$ is the vector of torques due to gravity; and
$u$ is the vector of input torques, including the torque exerted by the knee
spring. The spring attached in parallel with the knee alleviates the joint’s
torque requirements for hopping. Matrix $B_{e}$ maps the two-dimensional hip
and knee torque vector $u$ to the full four-dimensional system dynamics. The
robot is underactuated because the system has four degrees-of-freedom but only
two control inputs [23]. Finally, the matrix $J_{c}$ is the foot contact
Jacobian and $F_{GRF}$ represents the ground reaction forces between the foot
and the ground.

Since hopping is a physically demanding task, the actuator limits play a
significant role in the stability and performance of the robot. Students could
explore how the force and torque requirements of the gait push the velocity
and torque boundaries of electric rotary actuators. In addition, the friction
in the gearbox is added to the EoM as well as the reflected inertia of the
rotors. These governing equations are implemented in the MATLAB-based physics
simulator provided with the kit. The simulator serves as a sandbox where
students gain familiarity with the fundamentals of the robot kinematics and
dynamics before experimenting with the real hardware.

###  III-B Planning and Control

Despite the challenges in legged locomotion control, HOPPY is designed such
that a simple heuristic could be employed to achieve stable hopping. The robot
alternates between the stance phase and the aerial phase and dedicated
controllers are be promptly switched. We employ a simple strategy where the
robot applies a prescribed force profile during the stance phase and holds a
desired leg configuration during the aerial phase. Because the gantry is
designed to be sufficiently long, HOPPY is assumed to perform a motion that is
constrained to the $Y_{H}Z_{H}$ plane [25]. Further discussion and insights on
intuitive hopping control are provided in [13, 17].

####  III-B1 Stance Phase

During the stance phase, the robot leg applies a pre-defined force profile
against the ground, which results in a net impulse on the robot. The force
profile is generated using simple functions, such as polynomials for fast
online computation. Here we utilize Bézier polynomials to create a force
profile for the vertical and horizontal components of the ground contact force
within a projected stance duration of $0.15s$ as shown in Fig. 4. The peak
value of the force profile is modulated to regulate jumping height and forward
velocity. The desired contact force during stance $F_{d}$ is mapped to the
required joint torques $u_{st}$ using the foot contact Jacobian:

$u_{st}=J_{c}^{\top}F_{d}.$ (2)

Equation (2) assumes that the inertial effects of the lightweight leg in
respect to the body are negligible when compared to the contact forces that
propel the robot [17]. However, more advanced control strategies can be used
to incorporate the inertial properties of the leg.

####  III-B2 Aerial Phase

When HOPPY is in the aerial phase, the leg is held at a desired configuration
in respect to the hip in preparation for touchdown. To achieve this, a simple
task-space Proportional-Derivative (PD) controller is implemented:

$u_{ar}=J^{\top}_{c}\left[K_{p}(p_{ref}-p_{f})-K_{d}\dot{p}_{f}\right],$ (3)

where $u_{ar}$ is the vector of joint torques in aerial phase; $K_{p}$ and
$K_{d}$ are diagonal and positive definite gain matrices, $p_{ref}$ is the
desired foot position with respect to the hip at touchdown; $p_{f}$ and
$\dot{p}_{f}$ are the current foot position and velocity. More sophisticated
controllers such as workspace inverse dynamics control could be implemented to
improve the tracking performance.

####  III-B3 Touch-down, Impact, and Lift-off

When the touch-down event is detected, the robot transitions from the aerial
phase controller to the stance phase controller. The default contact sensor
with the kit is a spring-loaded linear potentiometer (see Fig. 3) attached to
the foot. However, other more capable options exist [3]. Alternatively, the
robot can detect contact with the ground by measuring the deviation of the
foot from the desired position after impact [1]. The impact at touch-down
instantaneously dissipates part of the total mechanical energy of the system.
The MATLAB simulator incorporates a hard contact model [25] for the impact
mechanics and assume an inelastic collision that instantaneously brings the
foot to a complete stop. The robot controller in the real hardware is
oblivious to impact forces due to the fast propagation of shock loads.
Instead, large impact loads are avoided by minimizing the reflected inertia of
the foot during touchdown [14]. The lift-off event is declared when the
contact sensor returns to its original unloaded state.

####  III-B4 Finite State Machine (FSM)

The hopping control framework is summarized as an FSM shown in Fig. 4. This
FSM consists of two states, namely, the stance phase and the aerial phase. The
transition events, lift-off, and touchdown are described in the previous
Section III-B3.

Figure 4: The finite state machine for the hopping control framework. The
robot alternates between (A) aerial and (B) stance phases with transition
events of lift-off and touch-down. (C) Prescribed force profile for the
horizontal ($Y_{H}$) and vertical ($Z_{H}$) components of $F_{GRF}$ during a
projected stance of $0.15s$. The peak horizontal and vertical forces are
modulated to control locomotion speed and hopping height, respectively.

###  III-C Hands-on Robotics Labs

Since HOPPY is tailored for active learning via experimentation, students are
exposed to challenges inherent to real robotic systems. Some of the topics
are:

####  III-C1 Embedded and Discrete control

The robot controller runs on an onboard microcontroller ($\mu$C) which
interfaces with peripherals and compute inputs at discrete time instances. The
deterministic execution of these events is fundamental for the implementation
of discrete control policies. To effectively control dynamic motions, the
targeted control rate is in the order of $1~{}kHz$.

####  III-C2 Communication Protocols

The $\mu$C receives a digital signal from the incremental encoders, an analog
signal proportional to the motor current, and a binary signal from the foot
contact switch sensor. The $\mu$C regulates the motor voltage via PWM.

####  III-C3 Signal Processing

The encoder has a coarse resolution of 28 counts per revolution (CPR) and the
gearbox has a noticeable backlash. To avoid noise amplification, motor
velocity is estimated using a filtered derivative
$\frac{\dot{\theta}(s)}{\theta(s)}=\frac{\lambda s}{s+\lambda}$ converted to
discrete-time with a period of $T=1ms$, where $s$ is the Laplace variable for
frequency, and $\lambda$ is a tunable constant (usually $\lambda\approx 10$).
Fast sampling rates are essential to avoid delayed velocity estimation. The
analog signal from the motor current estimation is also noisy and requires
filtering if used for feedback control.

####  III-C4 Control Input Saturation

The achievable joint torques and speeds are limited by the available voltage
supply ($V_{max}=12V$) and the maximum current that the driver can provide
($I_{max}=30A$) [14].

###  III-D Advanced Topics

Compared with manipulators commonly used as examples in conventional robotics
classes, the behavior of HOPPY is richer and the control more challenging
because it must be dynamic. In addition, HOPPY continuously makes and breaks
contact with the ground, which characterizes it as a system with hybrid
dynamics [25]. The hybrid nature of legged locomotion provides the students
with hands-on experience not easily obtained when working with manipulators.
Moreover, the heuristic-based hopping control framework presented here is the
basis on which more advanced topics such as model-based control, trajectory
optimization, state estimation, and machine learning can studied. Advanced
approaches tailored for hybrid dynamical systems, such as limit cycles,
Poincare Return Maps [23], and Hybrid Zero Dynamics approaches [26] can also
be explored. Finally, thanks to the modularity of HOPPY, the effect of
mechanical design, for instance by varying link dimensions and the gearing
ratio of the actuators, can be evaluated with the real robot [2]. Finally,
additional modules such as cameras and LIDARs can also be used.

##  IV Limitations, Improvements, and Safety

Despite the advantages of HOPPY, the inexpensive components employed also have
limitations. For instance, voltage control of the motor is not ideal as it
hinders the precise and rapid regulation of ground reaction forces. The coarse
resolution of the encoders and the backlash in the gearbox also impair
position tracking performance. The hobby-grade brushed motors have limited
torque density (peak torque divided by mass) [18]. Thus, the robot will not be
able to hop without the aid of the springs and the counterweight. In addition,
the simulator assumes simple dynamic models for impact and contact and
neglects friction, structural compliance, vibrations, and foot slip. Due to
these limitations, it is unlikely that the physical robot will behave exactly
as the simulator predicts. Nevertheless, the simulator is a valuable tool to
obtain insights into the behavior of the real robot, and it is useful for
preliminary tuning of controllers before implementation in the hardware.
Therefore, the robot simulator is a fundamental tool for teaching.

We note that the kit described in this paper is the bare minimum required to
enable robot functionality. However, if the budget allows, many components of
HOPPY can be upgraded. A first priority improvement would be (i) enabling
unrestricted rotations around the gantry using an onboard battery to power the
robot and a USB slip ring on the first joint for communication between the
$\mu$C and the host’s computer. The battery can also be used as the gantry
counterweight. In addition, (ii) employing encoders with a finer resolution of
around 12bits (4096 counts per revolution) would improve position control and
joint velocity estimation. (iii) Implementation of motor drivers which can
perform high-frequency current control, such as the Advanced Motion Control
AZBDC12A8 would enable precise torque control and improved force tracking
during stance. An even better solution would be (iv) the utilization of
brushless motors similar to those employed in [7, 11]. These have superior
torque density and would enable hopping without the aid of springs or
counterweights. (v) Additional sensors (encoders) could be added to the gantry
joints ($\theta_{1,2}$) or an Inertial Measurement Unit (IMU) can be mounted
on the robot for state estimation [8]. And finally, (vi) data logging can be
done using external loggers such as the SparkFun OpenLog.

When experimenting with HOPPY (or any dynamic robot), safety guidelines must
be followed. Students should wear safety goggles and clear the robot’s path.
It is advised to constantly check the temperature of the motor’s armature. If
they are too hot to be touched, the experiments should be interrupted until
they cool down. Aggressive operations (high torque and speeds) within negative
work regimes should be done carefully because electric motors regenerate part
of the input energy back into the power supply when backdriven [18].
Therefore, utilizing a battery is more appropriated if considerable negative
work is expected. For instance, a conventional $12V$ car battery can be used
both as a power supply and a mechanical counterweight. Users should be mindful
of the power and USB cables to not be pinched by the joints or excessively
twisted during experiments. Pre-winding the cables around the gantry in the
opposite direction of motion before experiments may help mitigate this issue.
The $\mu$C can also be programmed to run standalone. Finally, applying thread
locker to fasteners is recommended during assembly and we advise to often
check for loose components due to the constant impacts and vibrations.

##  V Experiments and Discussion

Figure 5: Snapshot of the simulator GUI. It shows the robot animation in
isometric and Sagittal plane views along with the trajectories of the hip
joint in blue and the foot in cyan. Plot (B) shows the horizontal and vertical
tangential velocities of the hip base around the gantry. Areas shaded in red
represent stance phases. Hip and knee joint torques are shown in (D) and their
angular velocities in (D). Plots (E) and (F) display the polygons that define
the limits of the hip and knee actuators along with their torque versus
velocity trajectory. Feasible trajectories must be always contained inside the
polygon. Finally, plot (G) shows the hip an knee joint angles over time.

A snapshot of the output of the MATLAB-based simulator is shown in Fig. 5. It
displays a 3D animation of the robot during hopping with a trace of the hip
joint and foot spatial trajectories. The view of the robot’s planar motion is
shown in (A). The visualizer displays the hopping velocity in (B), required
torques in (C), the joint velocities in (D), the joint angles in (G), and the
working regions of the hip (E) and knee (F) actuators. Areas shaded in red
represent stance phases. More information about the simulator is provided in
instructions Code_Instructions.pdf in the repository.

For the experiments in companion video (https://youtu.be/AvJIx4CQarM), we
employed the controller described in section III-B using aerial phase gains of
$K_{p}=500[1_{2\times 2}]$ and $K_{s}=50[1_{2\times 2}]$, a prescribed stance
duration of $T_{s}=0.15s$, and a prescribed peak vertical force of $80N$.
Where $1_{2\times 2}$ is a $2\times 2$ identity matrix. The peak horizontal
force was modulated to control the locomotion speed. Fig. 7-(A) to (D) shows
the data of the robot accelerating from about $0.65m/s$ to around $1.2m/s$ by
increasing the horizontal force peak. The joint torques are estimated by
multiplying the measured motor current by the actuator’s torque constant. The
saturation occurs when the requested input voltage is above the supplied
$12V$. Fig. 7-(D) shows the stance duration, measured from the contact switch,
which decreases according to the locomotion speed. Notice that at fast speeds,
the robot touches the ground for less than $0.1s$. Finally, 7-(E) shows the
achieved steady-state hopping velocities according to the commanded peak
horizontal force. However, we note that the locomotion speed is likely
overestimated because it was computed using the relative velocity between the
foot and the hip during stance, assuming that the foot does not slip during
contact.

HOPPY was used in a semester-long project in the ME446 Robot Dynamics and
Control course at UIUC. The course is attended by senior undergraduate and
first-year graduate students from the Departments of Mechanical Science and
Engineering, Electrical and Computer Engineering, and Industrial & Enterprise
Systems Engineering. Ten cross-departmental teams of four to five students
each received their own leg (hip base of link 2 plus links 3 and 4), and all
teams shared six gantries (fixed base plus links 1 and 2). During the first
half of the course, students derived the kinematic and dynamic model of the
robot, developed different controllers, and created a simple version of the
robot simulator in MATLAB. They assembled their kits half-way through the
semester and performed experiments with the robot in the second half. After
developing their own dynamic simulator, students were provided with our code
for comparison. The results from experiments with the real robot from one of
the teams are shown in the snapshots of Fig. 6. The team developed a custom
foot contact sensor to detect the stance phase. The robot was able to regulate
hopping speed, overcome small obstacles, and mitigate disturbances from gentle
kicks. With this hands-on kit, we aim for students to develop a better
understanding of the applications of complementary tools (mathematical model,
computer simulation, and physical prototype) used to create real dynamic
robots.

Figure 6: Snapshots of the hopping experiment with HOPPY by one of the student
teams (https://youtu.be/6O6czr0YyC8). Figure 7: Experimental data with HOPPY
speeding up from around $0.65m/s$ to $1.2m/s$. Shaded magenta areas represent
stance periods. (A) Hip ($\theta_{3}$) joint angles in the left vertical axis
and knee ($\theta_{4}$) angle in right axis. (B) Hip joint commanded torque
(black) and actual torque (red) estimated from the measured current. (C) Knee
joint torque. (D) Stance phase duration (duration of following shaded areas).
And (E) steady-state hopping speed as a function of the commanded peak
horizontal contact force $F_{yGFP}$.

##  VI Conclusion and future work

This paper introduced HOPPY, an open-source, low-cost, robust, and modular kit
for robotics education with dynamic legged robots. The goal of the kit is to
lower the barrier for studying dynamic robot behaviors and legged locomotion
with real systems. The control of dynamic motions present unique challenges to
the robot software and hardware, and these are often overlooked in
conventional robotics courses, even those with hands-on sessions. Here we
describe the topics which can be explored using the kit, list its components,
discuss preferred practices for implementation, and suggest further
improvements. HOPPY was utilized as the topic of a semester-long project for a
first-year graduate-level course at UIUC. Students provided positive feedback
from the hands-on activities during the course. The instructors will continue
to improve the kit and course content for upcoming semesters. To nurture
active learning, future activities will include a friendly competition between
teams to elect the fastest robot and the most energy-efficient robot. Speed is
estimated by timing complete laps around the gantry. Energy efficiency is
determined by the Cost of Transport, which is proportional to the ratio
between the total electrical power consumed during one hopping cycle and the
average translation speed [18]. All future improvements to the kit will be
made open-source.

## References

  * [1] G. Bledt, P. M. Wensing, S. Ingersoll, and S. Kim.  Contact model fusion for event-based locomotion in unstructured terrains.  In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 4399–4406, 2018.
  * [2] G. Bravo-Palacios, A. D. Prete, and P. M. Wensing.  One robot for many tasks: Versatile co-design through stochastic programming.  IEEE Robotics and Automation Letters, 5(2):1680–1687, 2020.
  * [3] M. Y. Chuah and S. Kim.  Enabling force sensing during ground locomotion: A bio-inspired, multi-axis, composite force sensor using discrete pressure mapping.  IEEE Sensors Journal, 14(5):1693–1703, 2014.
  * [4] J. J Craig.  Introduction to robotics: mechanics and control, 3/E.  Pearson Education India, 2009.
  * [5] Y. Ding, C. Li, and H.-W. Park.  Single leg dynamic motion planning with mixed-integer convex optimization.  In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1–6. IEEE, 2018.
  * [6] Y. Ding and H.-W. Park.  Design and experimental implementation of a quasi-direct-drive leg for optimized jumping.  In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 300–305. IEEE, 2017.
  * [7] F. Grimminger, A. Meduri, M. Khadiv, J. Viereck, M. Wüthrich, M. Naveau, V. Berenz, S. Heim, F. Widmaier, T. Flayols, J. Fiene, A. Badri-Spröwitz, and L. Righetti.  An open torque-controlled modular robot architecture for legged locomotion research.  IEEE Robotics and Automation Letters, 5(2):3650–3657, 2020.
  * [8] R. Hartley, M. G. Jadidi, J. W. Grizzle, and R. M. Eustice.  Contact-aided invariant extended kalman filtering for legged robot state estimation.  CoRR, abs/1805.10410, 2018.
  * [9] IEEE Spectrum.  Kiva systems: Three engineers, hundreds of robots, one warehouse, 2008\.
  * [10] IEEE Spectrum.  DARPA robotics challenge: A compilation of robots falling down, 2015\.
  * [11] B. Katz, J. D. Carlo, and S. Kim.  Mini cheetah: A platform for pushing the limits of dynamic quadruped control.  In 2019 International Conference on Robotics and Automation (ICRA), 2019.
  * [12] N. Kau, A. Schultz, N. Ferrante, and P. Slade.  Stanford doggo: An open-source, quasi-direct-drive quadruped.  In 2019 International Conference on Robotics and Automation (ICRA), pages 6309–6315, 2019.
  * [13] J. Koechling and H. Marc.  How fast can a legged robot run?  In Robots and Biological Systems: Towards a New Bionics? Springer Berlin Heidelberg, 1993.
  * [14] K. Lynch and F Park.  Modern Robotics: Mechanics, Planning, and Control.  Cambridge University Press, USA, 1st edition, 2017.
  * [15] T. K. Morimoto, M. Orta Martinez, R. Davis, P. Blikstein, and A. M. Okamura.  Teaching with hapkit: Enabling online haptics courses with hands-on laboratories.  IEEE Robotics and Automation Magazine, In press, 2020.
  * [16] M. Orta Martinez, C. M. Nunez, T. Liao, T. K. Morimoto, and A. M. Okamura.  Evolution and analysis of hapkit: An open-source haptic device for educational applications.  IEEE Transactions on Haptics, 13(2):354–367, 2020.
  * [17] M. H. Raibert.  Legged Robots That Balance.  Massachusetts Institute of Technology, USA, 1986.
  * [18] S. Seok, A. Wang, Meng Yee Chuah, D. Otten, J. Lang, and S. Kim.  Design principles for highly efficient quadrupeds and implementation on the MIT cheetah robot.  In 2013 IEEE International Conference on Robotics and Automation, pages 3307–3312, May 2013.
  * [19] S. D. Sheppard, K. Macatangay, A. Colby, and W. M. Sullivan.  Educating Engineers: Designing for the Future of the Field.  Carnegie Foundation for the Advancement of Teaching, Jossey-Bass, USA, 2009.
  * [20] B. Siciliano, L. Sciavicco, L. Villani, and G. Oriolo.  Robotics: modelling, planning and control.  Springer Science & Business Media, 2010.
  * [21] M. W. Spong.  Robot Dynamics and Control.  John Wiley and Sons, Inc., USA, 1st edition, 1989.
  * [22] A. T. Spröwitz, A. Tuleu, M. Ajallooeian, M. Vespignani, R. Möckel, P. Eckert, M. D’Haene, J. Degrave, A. Nordmann, B. Schrauwen, J. Steil, and A. J. Ijspeert.  Oncilla robot: A versatile open-source quadruped research robot with compliant pantograph legs.  Frontiers in Robotics and AI, 5:67, 2018.
  * [23] R. Tedrake.  Underactuated robotics: Algorithms for walking, running, swimming, flying, and manipulation (course notes for mit 6.832). downloaded on 09/09/2020 from http://underactuated.mit.edu/.
  * [24] P. M. Wensing, A. Wang, S. Seok, D. Otten, J. Lang, and S. Kim.  Proprioceptive actuator design in the mit cheetah: Impact mitigation and high-bandwidth physical interaction for dynamic legged robots.  IEEE Transactions on Robotics, 33(3):509–522, June 2017.
  * [25] E. R. Westervelt, J. W. Grizzle, C. Chevallereau, J. H. Choi, and B. Morris.  Feedback control of dynamic bipedal robot locomotion.  CRC press., 2007.
  * [26] E. R. Westervelt, J. W. Grizzle, and D. E. Koditschek.  Hybrid zero dynamics of planar biped walkers.  IEEE Transactions on Automatic Control, 48(1):42–56, 2003.


# SONIC: Synergizing VisiON Foundation Models for Stress RecogNItion from ECG
signals

Orchid Chetia Phukan1∗†, Ankita Das2∗, Arun Balaji Buduru1, Rajesh Sharma1,3
1IIIT-Delhi, India
2IIIT-Guwahati, India
3University of Tartu, Estonia
<EMAIL_ADDRESS>
###### Abstract

Stress recognition through physiological signals such as Electrocardiogram
(ECG) signals has garnered significant attention. Traditionally, research in
this field predominantly focused on utilizing handcrafted features or raw
signals as inputs for learning algorithms. However, there is now a burgeoning
interest within the community in leveraging large-scale vision foundation
models (VFMs) like ResNet50, VGG19, and others. These VFMs are increasingly
preferred due to their ability to capture complex features, enhancing the
accuracy and effectiveness of stress recognition systems. However, no
particular focus has been given on combining these VFMs. The combination of
VFMs offers promising benefits by harnessing their collective knowledge to
extract richer representations for improved stress recognition. So, to
mitigate this research gap, we focus on combining different VFMs for stress
recognition from ECG and propose SONIC, a novel framework that combines VFMs
through their logits and training a fully connected network on the combined
logits. Through extensive experimentation, SONIC showed the top performance
against individual VFMs performance on the WESAD benchmark. With SONIC, we
report state-of-the-art (SOTA) performance in WESAD with 99.36% and 99.24%
(stress vs non-stress) and 97.66% and 97.10% (amusement vs stress vs baseline)
in accuracy and F1 respectively.

Index Terms: Stress Recognition, ECG, Vision Foundational Models, WESAD

**footnotetext: The authors contributed equally to this work

##  I Introduction

Stress is a human affective condition and it refers to the physiological and
psychological response that arises when an individual perceives a potential
threat, challenge, or demand in their environment. Stress is not always
detrimental; rather it can serve as a mechanism to help us cope with different
situations. However, prolonged chronic stress can have adverse effects on
physical well-being such as increased heart rate [1], increased blood pressure
[2], cancer [3], etc., and mental well-being leading to cognitive problems
such as anxiety, depression [2], etc. Therefore, recognizing stress at an
early stage is essential to prevent it from becoming chronic. By fostering
increased awareness of elevated stress levels that could otherwise go
unreported, technology for stress recognition has the potential to improve
people’s understanding of stress and facilitate its treatment.

Traditionally, experts in psychology and physiology used a questionnaire-based
stress analysis[4] to assess a person’s level of stress. However, this
technique lacks reliability because it only uses the subject’s self-reported
responses. The accuracy of the assessment may also be hampered by people’s
hesitation when answering the questionnaire. To overcome these limitations,
researchers have investigated various means such as voice [5], facial
landmarks [6], physiological signals captured by wearable devices [7], etc. In
this work, we specifically concentrate on one of the commonly used
physiological signals used for stress recognition i.e. ECG
(Electrocardiogram).

Stress recognition from ECG has largely benefitted from the availability of
various large-scale datasets such as WESAD [8], StressID [9], and so on.
Initial works involved usage of classical machine learning (ML) algorithms
such as Random Forest, AdaBoost, etc. [8, 10] with handcrafted features as
well as deep learning techniques such as CNN [11, 12, 13, 14]. Researchers
also built a transformer-based model that comprises of a convolutional block
followed by a transformer encoder achieving state-of-the-art (SOTA)
performance for stress recognition from ECG [15].

Moreover, there is a rising interest in the community towards the utilization
of large-scale vision foundation models (VFMs) such as ResNet50, VGG19, and
similar architectures [16] in stress recognition from ECG. These VFMs despite
being trained primarily on visual data provide enhanced performance in
detecting stress through ECG. Additionally, this contributes to saving both
time as well as resources and prevents training models from scratch. VFMs come
in different types of model architectures. They can be either CNN-based [17]
or transformer-based [18] architectures and pre-trained on very-large-scale
visual data.

However, no attention towards combining these VFMs has been given by previous
works for stress recognition from ECG. Combining VFMs with distinct
architectural strengths comes with improved and robust performance also shown
in tasks in medical imaging [19, 20]. Each VFM can excel at capturing certain
aspects of input patterns due to variations in their training data and
architectural design. By combining VFMs, we can capture a broader spectrum of
features, enabling a more holistic representation of the input features. This
rich feature representation can significantly improve the discrimination
between stress and non-stress signals. Furthermore, combining VFMs can
facilitate complementary behavior, wherein the strengths of individual VFMs
complement each other while their weaknesses are mitigated. To cover this
research gap, we focus on the combination of VFMs and propose SONIC
(Synergizing VisiON Foundation Models for Stress RecogNItion from
EleCtrocardiogram Signals), a novel modeling framework that merges VFMs by
fusing their logits and training a fully connected network (FCN) using the
fused logits for improved stress recognition from ECG.

The main contributions are summarized as follows:

  * •

We propose a novel framework, namely SONIC, which illustrates methodology to
combine selected VFMs by fusing their logits.

  * •

With SONIC, we report SOTA performance in terms of accuracy and F1 in 2-class
classification (stress vs non-stress) as well as 3-class classification
(amusement vs stress vs baseline). The proposed method reports almost 9% and
20% improvement in accuracy in comparison to previous SOTA works in 2-class
and 3-class classification.

We will release the codes and models curated as part of our study after the
review process to maintain reproducibility and reference for future works to
build upon our work.

##  II Vision Foundation Models

In this section, we discuss the VFMs considered for our study. We select these
VFMs as they have substantially led to improvements in ImageNet performance as
well as in various downstream tasks. They are as follows:

VGG19[17]: It’s a part of the VGG family of models built for large-scale image
recognition and was proposed as a model to test the hypothesis that increasing
model depth leads to improvements in performance. VGG-19 comprises of 19
layers in total and trained on ImageNet dataset. Its architecture consists of
16 convolutional layers followed by 3 fully connected layers. Central to its
design, VGG-19 employs small 3x3 convolutional filters, which are effective in
capturing detailed features within images. This choice of filter size
contributes to the network’s ability to discern intricate patterns, enhancing
its performance in image recognition tasks.

ResNet50[21]: It is a CNN comprising of 50 layers and trained on ImageNet. Its
distinguishing feature lies in the incorporation of residual connections or
skip connections and these connections enable the network to bypass one or
more layers, thereby mitigating the vanishing gradient problem commonly
encountered in deep neural networks. Furthermore, skip connections play a
crucial role in optimization by enabling the model to converge to a better
local minimum. By providing alternative paths for gradient flow, ResNet-50 can
navigate through the parameter space more efficiently, leading to improved
convergence and potentially higher performance.

EfficientNet [22]: It is groundbreaking CNN architecture trained on ImageNet
and designed for superior performance and computational efficiency. Its
innovation lies in the implementation of compound scaling, a method that
uniformly adjusts the network’s depth, width, and resolution to strike an
optimal balance between model size and accuracy across diverse computational
constraints. With variants ranging from EfficientNet-B0 to B7, each is
meticulously scaled for different resource requirements.

ViT [18]: It is a revolutionary convolution-free model architecture built upon
original transformer [23] architecture. However, contrary to the original
transformer architecture built for NLP tasks, ViT leverages only the encoder
part of it. It replaces the convolution networks with self-attention heads. It
is designed primarily for image classification tasks and trained on ImageNet
and showed competitive performance in comparison to its previous ConvNet
architectures. ViT divides an image into smaller patches of a fixed size
instead of processing an entire image at once. These patches are transformed
into vectors through linear embeddings. Positional embeddings are introduced
to maintain spatial information, and the transformer encoder then processes
this sequence of vectors.

We use the pre-trained VFMs openly available in Keras library for
VGG19***https://keras.io/api/applications/vgg/\\#vgg19-function,
ResNet50***https://keras.io/api/applications/resnet/\\#resnet50-function, and
EffecientNet***https://keras.io/api/applications/efficientnet/\\#efficientnetb0-function.
For ViT, we make use of keras_vit***https://pypi.org/project/keras-vit/
package and we select ViT_B32 version. For the VFMs in our study, the number
of parameters for each of them are as follows: 143.7M for VGG19, 25.6M for
ResNet50, 5.3M for EfficientNet (we select BO version), and 88.22M for ViT.

Figure 1: Architecture of SONIC; VFM, GAP, FCN stands for Vision Foundational
Model, Global Average Pooling, Fully Connected Network: B, S, A stands for
logits of Baseline, Stress, Amusement labels; Here, SONIC is given for 3-class
classification, similarly, for 2-class classification is implemented.

##  III Methodology

In this section, we discuss the proposed method, SONIC. With SONIC, we aim to
effectively tap into the complementary behavior of the VFMs for improved
stress recognition from ECG. The modeling architecture for SONIC is shown in
Figure 1. SONIC merges VFMs by fusing their logits, which are the unnormalized
predictions generated by each VFM before they are transformed into
probabilities through a softmax function. It is important to note that higher
logit values indicate a higher likelihood for the corresponding class,
portraying greater confidence in the classification.

We anticipate that SONIC can significantly outperform individual VFMs in
stress recognition tasks. This expectation stems from its unique capability to
capture raw, discriminative information while retaining the distinct
contributions of each VFM and harnessing the strengths of individual VFMs
while effectively mitigating their weaknesses.

We give a detailed walkthrough of the proposed methodology and its algorithm
is shown in Algorithm 1. First, the input ($X$) is sent simultaneously to
individual VFM. Each VFM converts the input to a unique feature space
tailoring its representative behavior. To be noted, we don’t freeze the VFM
weights. Each VFM is succeeded by a global pooling (GAP) layer. GAP aids in
spatial information retention and parameter reduction by averaging each
feature map. Further, we add a fully connected network (FCN) block consisting
of two cascading dense layers, the first one with 128 neurons and secondly, 2
(For 2-class classification, representing stress and non-stress labels) or 3
neurons (For 3-class classification, representing baseline, stress and
amusement labels). The second layer neurons of the FCN represent the logits
for each class. Further, the logits from the VFMs are fused together through
concatenation ($Z$). Then passed through FCN consisting of 12 neurons and
lastly, the classification head outputs the probabilities corresponding to
different classes. We also make use of dropout to reduce overfitting.

Algorithm 1 SONIC Algorithm

1:Input: Training data $(X,Y)$

2:Output: Trained model

3:procedure SONIC($X,Y,M$)

4: Send input $X$ to individual VFMs of $M$ and compute logits $Z_{1}$ and
$Z_{2}$

5: Concatenate logits: $Z\leftarrow[Z_{1},Z_{2}]$

6: Initialize FCN parameters: $W_{\text{FC}},b_{\text{FC}}$

7: for each training epoch do

8: $L=-\sum_{i=1}^{N}Y_{i}\log(\hat{Y}_{i})$, here, $N$ stands for the number
of training datapoints

9: Updating VFMs parameters: $W_{\text{VFM}}\leftarrow
W_{\text{VFM}}-\alpha\frac{\partial L}{\partial W_{\text{VFM}}}$,
$b_{\text{VFM}}\leftarrow b_{\text{VFM}}-\alpha\frac{\partial L}{\partial
b_{\text{VFM}}}$

10: Updating FCN parameters: $W_{\text{FC}}\leftarrow
W_{\text{FC}}-\alpha\frac{\partial L}{\partial W_{\text{FC}}}$,
$b_{\text{FC}}\leftarrow b_{\text{FC}}-\alpha\frac{\partial L}{\partial
b_{\text{FC}}}$, here, $\alpha,W,b$ stands for learning rate, weights, bias

11: end for

12: return Trained model

13:end procedure

Training SONIC: We train SONIC for 20 epochs for different VFMs combinations
with a learning rate as 1e-3 and batch size of 32. We use cross-entropy as the
loss function and Adam as the optimizer.

##  IV Experiments

###  IV-A Benchmark Datasets:

We use WESAD (Wearable Stress and Affect Detection) [8] dataset for our
experiments. It is a widely used benchmark dataset in the field of affective
computing and wearable computing. It is often employed to formulate and
evaluate algorithms for stress and affect detection using physiological
signals collected from wearable sensors. It includes higher resolution
physiological data such as ECG, electrodermal activity (EDA), electromyography
(EMG), respiratory activity, and temperature. It also includes motion
(acceleration) data from chest-worn and wrist-worn devices. The data was
collected from 15 participants sampled at 700 Hz. This dataset focuses on both
binary class (stress vs non-stress) and multi-class classification (amusement
vs stress baseline).

###  IV-B Data pre-processing:

Here, we give detailed information about the data preprocessing steps. First,
raw 1D ECG signals are segmented by creating a window size of 5 seconds and
maintaining a window gap of 2 seconds. Then we convert them to 2D Spectrograms
using Fourier
transform***https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.spectrogram.html.
These spectrograms are then reshaped to shape (224, 224). Further, as the VFMs
are pre-trained on 3-channel images, we extended the spectrograms to shape
(224, 224, 3) by replicating it to three dimensions. The spectrograms are
normalized to ensure consistent scaling. We perform 5-fold cross-validation
for all the modeling experiments with four-fold as the training set and one-
fold as a test set across 5 runs.

###  IV-C Modeling Baselines

As baseline experiments for our study, we first fine-tune the VFMs on WESAD
for both 2-class and 3-class classification. We fine-tune the VFMs (VGG-19,
ResNet50, EfficientNet, ViT) by adding a GAP layer followed by FCN with dense
layers of 128 neurons, and 2 or 3 neurons depending on the classification type
that represents the classes. We retrain all the weights of the VFMs on WESAD.
We employ dropout for mitigating overfitting. We set the number of epochs as
20 and learning rate as 1e-3. We use cross-entropy as the loss function and
Adam as the optimizer.

###  IV-D Metrics

We make use of accuracy and F1 (macro average) as the metrics for the
evaluation of the models.

TABLE I: Performance Evaluation of Models 2-class classification: Scores are average of 5-folds and in %; A(2), F(2) stands for accuracy and F1 for 2-class classification respectively; A(3), F(3) stands for accuracy and F1 for 3-class classification respectively; SOTA(2), SOTA(3) represents the SOTA works on 2-class and 3-class classification respectively Model | A(2) | F(2) | A(3) | F(3)
---|---|---|---|---
SOTA (3) [8] | 85.44 | 81.31 | 66.29 | 56.03
ViT | 87.36 | 84.06 | 72.49 | 54.62
SOTA (2)[16] | 90.47 | —— | —— | ——
VGG19 | 90.78 | 88.85 | 73.45 | 54.04
ResNet50 | 92.39 | 90.80 | 78.63 | 64.93
EfficientNet | 92.60 | 91.03 | 80.26 | 72.49
SONIC (ViT, VGG-19) | 92.84 | 91.30 | 73.67 | 61.33
SONIC (ViT, ResNet50) | 93.69 | 91.87 | 75.39 | 66.86
SONIC (ViT, EfficientNet) | 93.88 | 91.98 | 85.98 | 79.34
SONIC (VGG-19, ResNet50) | 94.82 | 93.97 | 83.77 | 73.45
SONIC (VGG-19, EfficientNet) | 97.36 | 95.30 | 96.76 | 95.95
SONIC (ResNet50, EfficientNet) | 99.36 | 99.24 | 97.66 | 97.10

(a)

(b)

Figure 2: Confusion Matrix of SONIC with the best VFMs combination (ResNet50,
EfficientNet) for 2-class and 3-class classification

(a)

(b)

Figure 3: t-SNE plots of EfficientNet for 2-class and 2-Class classification

(a)

(b)

Figure 4: t-SNE plots of SONIC with combination of ResNet and EfficientNet for
2-class and 3-class classfication

###  IV-E Results

Table I presents the scores obtained from various models. Firstly, we discuss
the results obtained by fine-tuning the VFMs individually. We observe that
fine-tuning EfficientNet leads to the best performance among the VFMs showing
its higher transferability for stress recognition from ECG. This behavior of
EfficientNet is also supported by Kornblith et al. [24] which showed that
better models in ImageNet transfer better to other downstream tasks. Similar
behavior can be observed between ResNet50 and VGG19, as ResNet50 performs
better than VGG19 in both 2-class and 3-class classification.

We observe that CNN-based VFMs (VGG19, ResNet50, EfficientNet) achieved
greater accuracy than transformer-based VFM (ViT), and this can be attributed
to CNNs efficacy in capturing local patterns and spatial hierarchies than ViT.
In contrast, ViT is designed with self-attention mechanisms which adept at
capturing broad-scale relationships within images.

From Table I, we can see that combining various VFMs leads to improvement in
stress recognition performance. This shows that through SONIC, VFMs are
showing complementary behavior and improves over their individual
performances. Among different combinations, ResNet50 and EfficientNet showed
the top most performance, further, proving the effectiveness of SONIC. The
confusion matrix of SONIC with the best VFMs combination (ResNet 50 and
EfficientNet) for 2-class and 3-class classification is shown in Figure 2.
Additionally, we show the t-SNE plots of EfficientNet in Figure 3 and SONIC
with ResNet50 and EfficientNet combination in Figure 4 for 2-class and 3-class
classification respectively. We observe far better clustering across the
classes with SONIC, supporting its topmost performance. For plotting the t-SNE
plots, we extract the representations of the trained models for a test set
from the penultimate layer before the output layer.

###  IV-F Comparison to State-of-the-art

We compare the results with SOTA works. Table I shows the comparison. We
obtain around 9%, 20% in 2-class and 3-class classification respectively in
comparison to previous SOTA models.

###  IV-G Conclusion

In this work, we introduce SONIC, a novel framework for stress recognition
from ECG by combining VFMs through their logits and then training FCN on top
of the combined logits. Through extensive experimentation, SONIC demonstrates
superior performance compared to individual VFMs on the WESAD benchmark.
Remarkably, SONIC achieves SOTA results on WESAD, boasting accuracy rates of
99.36% and 99.24% for stress vs non-stress classification, and 97.66% and
97.10% for amusement vs stress vs baseline classification, in terms of both
accuracy and F1, respectively. We believe our work can open up new
possibilities for stress recognition from ECG by harnessing the common
complementary feature space of VFMs through the combination of various VFMs.

## References

  * [1] Sheldon Cohen, Denise Janicki-Deverts, and Gregory E Miller,  “Psychological stress and disease,”  Jama, vol. 298, no. 14, pp. 1685–1687, 2007.
  * [2] Omar Hahad, Jürgen H Prochaska, Andreas Daiber, Thomas Muenzel, et al.,  “Environmental noise-induced effects on stress hormones, oxidative stress, and vascular dysfunction: key factors in the relationship between cerebrocardiovascular and psychological disorders,”  Oxidative medicine and cellular longevity, vol. 2019, 2019.
  * [3] Boris Mravec, Miroslav Tibensky, and Lubica Horvathova,  “Stress and cancer. part i: Mechanisms mediating the effect of stressors on cancer,”  Journal of neuroimmunology, vol. 346, pp. 577311, 2020.
  * [4] Alexandra D Crosswell and Kimberly G Lockwood,  “Best practices for stress measurement: How to measure psychological stress in health research,”  Health psychology open, vol. 7, no. 2, pp. 2055102920933072, 2020\.
  * [5] Zihan Wu, Neil Scheidwasser-Clow, Karl El Hajal, and Milos Cernak,  “Speaker Embeddings as Individuality Proxy for Voice Stress Detection,”  in Proc. INTERSPEECH 2023, 2023, pp. 1838–1842.
  * [6] P Ramesh Naidu, S Pruthvi Sagar, K Praveen, K Kiran, and K Khalandar,  “Stress recognition using facial landmarks and cnn (alexnet),”  in Journal of Physics: Conference Series. IOP Publishing, 2021, vol. 2089, p. 012039.
  * [7] Ritu Tanwar, Orchid Chetia Phukan, Ghanapriya Singh, Pankaj Kumar Pal, and Sanju Tiwari,  “Attention based hybrid deep learning model for wearable based stress recognition,”  Engineering Applications of Artificial Intelligence, vol. 127, pp. 107391, 2024.
  * [8] Philip Schmidt, Attila Reiss, Robert Duerichen, Claus Marberger, and Kristof Van Laerhoven,  “Introducing wesad, a multimodal dataset for wearable stress and affect detection,”  in Proceedings of the 20th ACM international conference on multimodal interaction, 2018, pp. 400–408.
  * [9] Hava Chaptoukaev, Valeriya Strizhkova, Michele Panariello, Bianca Dalpaos, Aglind Reka, Valeria Manera, Susanne Thümmler, Esma Ismailova, Massimiliano Todisco, Maria A Zuluaga, et al.,  “Stressid: a multimodal dataset for stress identification,”  Advances in Neural Information Processing Systems, vol. 36, 2024\.
  * [10] Mouna Benchekroun, Baptiste Chevallier, Hamza Beaouiss, Dan Istrate, Vincent Zalc, Mohamad Khalil, and Dominique Lenne,  “Comparison of stress detection through ecg and ppg signals using a random forest-based algorithm,”  in 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE, 2022, pp. 3150–3153.
  * [11] Giorgos Giannakakis, Eleftherios Trivizakis, Manolis Tsiknakis, and Kostas Marias,  “A novel multi-kernel 1d convolutional neural network for stress recognition from ecg,”  in 2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW). IEEE, 2019, pp. 1–4.
  * [12] Pengfei Zhang, Fenghua Li, Rongjian Zhao, Ruishi Zhou, Lidong Du, Zhan Zhao, Xianxiang Chen, and Zhen Fang,  “Real-time psychological stress detection according to ecg using deep learning,”  Applied Sciences, vol. 11, no. 9, pp. 3838, 2021.
  * [13] Pengfei Zhang, Fenghua Li, Lidong Du, Rongjian Zhao, Xianxiang Chen, Ting Yang, and Zhen Fang,  “Psychological stress detection according to ecg using a deep learning model with attention mechanism,”  Applied Sciences, vol. 11, no. 6, pp. 2848, 2021.
  * [14] Wonju Seo, Namho Kim, Sehyeon Kim, Chanhee Lee, and Sung-Min Park,  “Deep ecg-respiration network (deeper net) for recognizing mental stress,”  Sensors, vol. 19, no. 13, pp. 3021, 2019.
  * [15] Behnam Behinaein, Anubhav Bhatti, Dirk Rodenburg, Paul Hungler, and Ali Etemad,  “A transformer architecture for stress detection from ecg,”  in Proceedings of the 2021 ACM International Symposium on Wearable Computers, 2021, pp. 132–134.
  * [16] Syem Ishaque, Naimul Khan, and Sri Krishnan,  “Detecting stress through 2d ecg images using pretrained models, transfer learning and model compression techniques,”  Machine Learning with Applications, vol. 10, pp. 100395, 2022.
  * [17] Karen Simonyan and Andrew Zisserman,  “Very deep convolutional networks for large-scale image recognition,”  arXiv preprint arXiv:1409.1556, 2014.
  * [18] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.,  “An image is worth 16x16 words: Transformers for image recognition at scale,”  arXiv preprint arXiv:2010.11929, 2020.
  * [19] Wilson Bakasa and Serestina Viriri,  “Stacked ensemble deep learning for pancreas cancer classification using extreme gradient boosting,”  Frontiers in Artificial Intelligence, vol. 6, 2023.
  * [20] Amira Alotaibi, Tarik Alafif, Faris Alkhilaiwi, Yasser Alatawi, Hassan Althobaiti, Abdulmajeed Alrefaei, Yousef Hawsawi, and Tin Nguyen,  “Vit-deit: An ensemble model for breast cancer histopathological images classification,”  in 2023 1st International Conference on Advanced Innovations in Smart Cities (ICAISC). IEEE, 2023, pp. 1–6.
  * [21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [22] Mingxing Tan and Quoc Le,  “Efficientnet: Rethinking model scaling for convolutional neural networks,”  in International conference on machine learning. PMLR, 2019, pp. 6105–6114.
  * [23] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  Advances in neural information processing systems, vol. 30, 2017\.
  * [24] Simon Kornblith, Jonathon Shlens, and Quoc V Le,  “Do better imagenet models transfer better?,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 2661–2671.