#  : High Performance OS I/O Architecture with Protected Data Plane and
Lightweight Activation

Anonymous Submission #803

###### Abstract

Increasing data center network speed coupled with application requirements for
high throughput and low latencies have raised the efficiency bar for network
stacks. To reduce substantial kernel overhead in network processing, recent
proposals bypass the kernel or implement the stack as user space OS service –
both with performance isolation, security, and resource efficiency trade-offs.
We present , a new network stack architecture that combines the performance
and resource efficiency benefits of kernel-bypass and the security and
performance enforcement of in-kernel stacks. runs the OS I/O stack in user-
level threads that share both address spaces and kernel threads with
applications, avoiding almost all kernel context switch and cross-core
communication overheads. To provide sufficient protection, leverages x86
protection keys (MPK) extension to isolate the I/O stack from application
code. And to enforce timely scheduling of network processing and fine-grained
performance isolation, implements lightweight scheduler activations with
preemption timers.

##  1 Introduction

Modern data center applications and hardware have posed unprecedented
challenges to operating system I/O subsystems. User requests for these
applications are commonly served by hundreds, or even thousands of services
[31, 14]. To ensure responsiveness, each service needs to provide both low and
predictable latency [22], with tail latency service-level objectives (SLO)
commonly in the range of tens to hundreds of microseconds [38]. At the same
time, operators co-locate multiple services on the same physical host to fully
utilize precious computing resources, including CPU cycles, memory, and
network bandwidth. Consequently, resource efficiency and performance isolation
are critical metrics for data center systems.

Strict latency requirements, coupled with increasing network hardware speed,
have inspired an influx of work seeking to improve OS I/O subsystem
performance. One line of work [15, 26, 34, 36, 47] implements the I/O stack in
user-space libraries that bypass the kernel in the common case. In exchange
for reduced context switching overhead, these solutions compromise or
completely forgo protection of the I/O stack from untrusted applications. For
better scalability, they use hardware-based packet steering (e.g., receive
side scaling) and run-to-completion on each core, leading to non-preemptive,
multi-queue FIFO scheduling. Queuing theory dictates that such scheduling
policy suffers from higher tail latency due to head-of-line blocking,
particularly for highly skewed workloads or workloads with high dispersion
[23, 16].

In an attempt to address the limitations of kernel-bypass, another line of
work [27, 19] implements centralized I/O stack OS services running on
dedicated CPU cores. The stack is protected and isolated from untrusted
applications using process isolation. Recent work [16, 32, 9] goes a step
further: They leverage centralized scheduling services to realize preemptive,
single-queue FIFO, leading to better tail latency. Unfortunately, an inherent
issue for a centralized architecture is degraded performance isolation, as
multiple applications are sharing and contending for the shared OS service. By
design, these services allocate entire cores for network processing and
scheduling. Such coarse-grained allocation leads to suboptimal resource
utilization, as the service cores are either a system bottleneck if fully
saturated, or waste CPU cycles when under-utilized.

In this work, we address this dilemma by proposing , a new data plane OS
design. Our goal is to simultaneously offer strong protection, high
throughput, good tail latency guarantees, efficient utilization of resources,
and strict performance isolation. Similar to prior kernel-bypass systems, runs
the data plane OS in a distributed fashion: It spawns an instance of the data
plane for each application, and shares the same address space and CPU cores as
the application. By multiplexing CPU cores between the data plane and the
application, utilizes CPU resources more efficiently; by distributing I/O
processing and scheduling, ensures strong performance isolation between
applications. To protect the data plane from untrusted applications, leverages
Intel MPK [29], a hardware-based user-space memory protection scheme, to
ensure strong memory isolation. Our new protection scheme enables the data
plane OS to safely run in user-level threads, with a context switching
overhead of only 70 CPU cycles. To prevent untrusted applications from
escalating privileges, we also propose hardware mechanisms to safeguard MPK
registers, without mode switches in the critical path.

Low and predictable tail latency requires fine-grained preemptive scheduling
[16]. The high overhead of the OS kernel scheduler, coupled with our user-
level threading approach, makes it a challenging task. In , we address this
challenge by implementing lightweight scheduler activations [2]. We use the
high resolution timer subsystem in Linux, and implement a kernel module that
efficiently redirects timer interrupts together with the preempted context,
into the user-space data plane scheduler. also maps local Advanced
Programmable Interrupt Controller (APIC) registers into the data plane OS for
protected finer control over preemption intervals. Our activation mechanism
incurs an overhead of 1098 CPU cycles, allowing to perform preemptive
scheduling every 10$\mu$s.

We compare with both the kernel-bypass and central service architectures on a
variety of application workloads. achieves performance comparable to other
performance oriented network stacks on small-sized RPCs. On workload with
bimodal service times, outperforms IX by 6.9$\times$ in throughput under a
tail latency SLO. When allocated a fixed number of CPU cores, achieves
1.6$\times$ higher throughput than TAS due to its more efficient CPU
utilization. When co-locating a latency-critical (LC) and a high-throughput
(HT) application, provides better performance isolation than the central
service architecture – 83.8% lower tail latency for LC and 16.8% higher
sustainable throughput for HT.

##  2 Background and Motivation

In this section, we outline the goals of a modern data center OS network stack
to meet these demands, and discuss deficiencies of existing solutions.

###  2.1 Goals for Data Center Network Stacks

#### High-Performance.

Modern data center applications, such as high-performance key value stores,
web servers, and in-memory databases, require high message rates. Small
message workloads are common as many applications use remote procedure calls
(RPCs) to coordinate distributed components or invoke external services,
leading to many small control messages. Moreover, applications require RPCs to
complete in tens to hundreds of $\mu$s, even in the tail case and when servers
are processing high traffic volumes.

#### Performance Isolation.

Data center and cloud systems run concurrent tasks, be it OS and applications,
separate applications, different tasks in an individual application, or
requests on multiple client connections. Workload characteristics across
different tasks are diverse: Some tasks have low throughput but strict latency
requirements, while others issue large batches of requests; some tasks perform
simple computation or memory lookups, while others involve long-running jobs
such as table scans and garbage collection. Applications require mechanisms to
effectively isolate operations from performance interference from other
ongoing tasks.

#### Resource Efficiency.

Beyond performance, the cost of running data center applications is another
critical metric for both users and operators. Cost is directly related to
resource efficiency, i.e., how much hardware resources are required to achieve
the service-level agreement (SLA). As such, a crucial goal of the system is to
efficiently use resources such as CPU cycles and memory, and to avoid leaving
resources idle when there is pending work.

#### Multi-Tenant Enforcement.

Finally, a critical responsibility of an OS network stack is protecting
mutually non-trusting or buggy applications. The network stack has raw access
to network hardware, a shared system resource.Without protection, malicious or
buggy applications can violate integrity or confidentiality of other
applications’ network traffic, and interfere with their network performance.
Moreover, correct operation and protocol conformance of the stack requires
timely handling of protocol events, e.g., acknowledgements, re-transmissions,
and congestion control.

###  2.2 Prior Network Stack Architectures

Network
Stack |  High
Perf. |  Perf.
Isolation |  Resource
Efficiency |  Multi-Ten.
Enforcement
---|---|---|---|---
Linux |  |  () |  |
IX |  () |  |  |
mTCP |  |  |  |
TAS |  |  |  |
|  |  |  |
Table 1: Network architectures compared by design goals.

A long line of prior work [34, 4, 15, 47, 36, 16, 32, 9, 19, 27] explores
network stack architectures with different trade-offs among these goals. We
divide these architectures into three major categories: in-kernel,
decentralized kernel-bypass, and central services. tab:stackarchs lists and
compares the trade-offs with representative systems for each architecture.

#### In-Kernel.

The Linux in-kernel network stack performs poorly on small RPC throughput and
latency [4, 34]. This is primarily due to expensive context switches between
applications and the kernel, and the large code footprint of a general purpose
stack. The millisecond kernel scheduling quantum also leads to long tail
latency for short $\mu$s-scale requests. However, in-kernel network stacks can
guarantee protection for multi-tenant enforcement. The processor’s _kernel-
mode_ prevents applications from directly accessing or manipulating the
network stack. Combined with _preemptive scheduling_ with hardware timers, an
in-kernel stack also ensures correct and timely execution of protocol events,
as well as fair scheduling of different tasks, albeit at a coarse granularity.
By centrally managing all resources, the kernel can also make idle resources
available to applications with demand for capacity.

The IX [4] protected data plane retains similar protection through CPU
virtualization features [3]. IX’s custom non-POXSIX API amortizes context
switching and protection overhead through _batching_ , and improves latency by
requiring applications to process incoming requests to completion in order.
However, IX implements _non-preemptive and decentralized scheduling_ of
applications and the network stack, which increases tail latency and precludes
fine-grained performance isolation at connection or request level [36, 16].

#### Decentralized Kernel-Bypass.

Kernel-bypass network stacks, such as mTCP [15], Arrakis [34], or Demikernel
[47], go a step further and avoid context switching by implementing the
network stack as user space libraries that applications invoke through
function calls. To avoid synchronization, these stacks use hardware packet
steering to ensure packets arrive at the correct application core and then
process them locally. Implemented in a user space library OS, they can also be
tailored to and optimized for specific application needs [8].

While reducing common-case latency and increasing throughput, kernel-bypass
stacks fall short in fine-grained performance isolation due to cooperative
scheduling that results in first-come-first-serve request processing on each
core. More critically, these network stacks forgo protection from untrusted
applications. Application code can directly manipulate protocol state or delay
protocol processing. Hardware-based solutions [34] using SR-IOV and IOMMUs
only offer limited and coarse-grained protection and isolation, such as
filtering packets based on addresses and enforcing rate limits.

#### Central Service.

Inspired microkernels, TAS [19] and SNAP [27] run the network stack as a
separate service on _dedicated_ processor cores. Applications interface with
the network stack through fast shared memory message passing. While incurring
a small cache coherence overheads for message passing, central service stacks
avoid context switching, improve cache locality, and prevent interference from
applications. As a result, they offer throughput and common-case latency
comparable to kernel-bypass.

However, shared service architectures complicates performance isolation as
they handle network processing for all applications and connections. Finally,
resource efficiency is a major concern; while services can dynamically add or
remove network stack cores, this occurs at coarse granularity, both spatially
— entire cores are allocated — and temporally — reallocation occurs at
millisecond granularity. Sharing cores with applications to improve resource
utilization, forgoes the performance benefit of this architecture [27, 19].

###  2.3 A Dilemma

We now present a series of motivating experiments to demonstrate an
unfortunate dilemma: neither decentralized kernel-bypass nor central service
architectures satisfies all our goals outlined in sec:background:goals. The
full experiment setup is detailed in sec:eval.

eval-synthetic-uniform

(a) Synthetic uniform workload

eval-synthetic-imbalanced

(b) Synthetic imbalanced workload

eval-synthetic-bimodal

(c) Synthetic bimodal workload

eval-isolation

(d) Performance isolation

motiv-tas-const

(e) Core allocation (constant service time)

motiv-tas-exp

(f) Core allocation (exponential service time)

Figure 1: In (a) – (c), a single application runs on 16 server cores. The
application performs a synthetic workload with different service time in each
experiment: uniform distribution with a mean of 2.5$\mu$s in (a), exponential
distribution with 5 cores receiving 10 times more traffic in (b), and bimodal
(99.5%–1, 0.5%–1000) distribution in (c). In (d), two applications collocate
on the same server. The high-throughput (HT) application sends request at a
fixed 4 Gbps rate. Load of the latency-critical (LC) application gradually
increases. In (e) and (f), maximum throughput for different core allocation
schemes of a central service stack is shown. The application runs a constant
service time workload in (e), and an exponential workload in (f).

#### Poor Tail Latency.

We first show that existing decentralized kernel-bypass stacks suffer from
long tail latency when application workloads exhibit high variance in service
time or load distribution. We deploy a server application that performs dummy
work on each request, and control the duration of the work to simulate various
service-time distribution. The application runs on 16 CPU cores, and we
gradually increase the client load. We first run a workload with exponential
service-time distribution, but unevenly distributes the requests: 5 cores
receive 136 Kops while the remaining cores process 13.6 Kops. For a kernel-
bypass stack that implements decentralized first-come-first-serve (d-FCFS)
scheduling, e.g., IX, the cores that receive higher traffic experience high
CPU load, while the remaining cores stay idle. The effect is shown in
fig:motiv-imbalanced: IX suffers a significantly higher tail latency than TAS,
which implements centralized FCFS scheduling.

We next switch to a workload that evenly distributes the request, but with a
service time that follows a bimodal distribution with 99.5% of requests take
1$\mu$s and 0.5% of requests take 1000$\mu$s. Existing decentralized stacks
lack support for _preemptive scheduling_ , which is critical to approximate
processor sharing (PS) scheduling. As a result, long-running tasks in the
bimodal distribution block subsequent short tasks waiting in the scheduler
queue. This is demonstrated in fig:motiv-bimodal, where IX incurs a high 99%
tail latency even at moderate load.

#### Weak Performance Isolation.

While central service stacks excel at tail latency metrics, they fall short in
performance isolation guarantees. To illustrate this issue, we co-locate two
types of applications on the same server: a latency critical application (LC)
that generates small TCP messages and has strict tail latency requirement
(representative of a fast in-memory key-value store system like memcached
[28]), and a high-throughput application (HT) that transmits larger messages
and expects higher overall throughput for shorter completion
time(representative of a data analytics applications like Spark [45]). We fix
the load of HT at 4 Gbps and gradually increase the load of LC. We measure
both the tail latency of LC and the sustainable throughput of HT, and show the
result in fig:motiv-isolation.

For a centralized architecture such as TAS, the two applications are
contending over the shared I/O cores, interfering with each other’s
performance. Consequently, once the load of LC exceeds 2.0 Mops, throughput of
HT starts to drop compared to when running HT alone. More critically, tail
latency of LC is also negatively impacted, as more LC requests are
experiencing longer queuing delay.

#### Low Resource Efficiency.

Another metric where central service stacks struggle at is efficient
utilization of hardware resources. Due to their coarse allocation of entire
cores to I/O processing, CPU resources can be either left idle when there is
not enough I/O workload, or become a bottleneck during bursty traffic. We
demonstrate this phenomenon by running a simple benchmark on TAS, in which a
maximum of 8 cores are allocated to both the application and the TAS fast path
I/O cores. The application runs a simple echo server with two types workloads:
constant and exponential service-time. We then sweep the configuration space
of core allocation, and show the maximum throughput each configuration
achieves in fig:motiv-res-const and fig:motiv-res-exp. The total throughput
varies across configurations by more than 400% even when all the cores are
allocated, demonstrating that it is non-trivial to determine the best
configuration for a workload. The issue becomes even more severe in more
dynamic situations where application and I/O workloads fluctuate across time.

##  3 A Decentralized Protected Data Plane

A decentralized architecture naturally avoids two fundamental limitations of
the centralized service approach: Instead of processing packets with shared
resources, the architecture can leverage NIC hardware packet steering to
process packets on the same CPUs where the application consumes them. This not
only minimizes cache coherence overheads but also effectively _eliminates I/O
performance interference_ among competing applications. Instead of allocating
whole CPU cores to tasks, a decentralized archiecture can flexibly multiplex
inividual cores among application code, the I/O stack, and other OS
functionality. The architecture therefore can flexibly and dynamically
allocate CPU cycles, _improving CPU resource utilization_.

We argue that the missing or expensive protection and poor tail latency
(section 2.3) in prior decentralized data planes [15, 34, 4, 47], are not
fundamental, but merely design and implementation limitations. To overcome
these deficiencies, we need to address the following technical challenges:

#### Lightweight protection for data plane OS.

$\mu$s-scale data center applications issue frequent I/O calls leading to
frequent context switches between application and network stack. In-kernel
network stacks leverage CPU kernel-user isolation for confidentiality and
integrity protection of OS data and privileged CPU features. Micro-kernels use
address space isolation to protect the I/O stack. Both approaches incur
significant hardware overhead for context switches. Achieving our goals,
therefore, requires a lighter-weight solution that efficiently switches
between the I/O stack and the application, but without compromising
protection.

#### Fast preemptive scheduling.

Correct network protocol processing and low, predictable tail latency require
timely preemption of application-level tasks. Prior decentralized
architectures, however, only implement cooperative scheduling for two reasons:
1) As I/O stack and the application run in the same thead, the kernel
scheduler cannot switch execution between them.2) Current preemption
mechanisms incur high overhead, and occur at millisecond-scale intervals.
Consequently, we need to implement efficient decentralized $\mu$s-scale
preemption, while allowing flexible scheduling between I/O and user tasks.

###  3.1 Our Approach

We propose to address these challenges in using a combination of user-level
threading, hardware-assisted memory protection for I/O stack isolation, and
fine-grained timer interrupt coupled with lightweight scheduler activation.

#### User threading with virtual memory for confidentiality.

To minimize context switching overhead, runs the I/O stack in user-level
threads that share the same CPU privilege level and address space as the
application. We ensure _confidentiality_ between mutually non-trusting
applications through virtual memory; each application data plane only has
access to buffers and protocol state for that application. Application
visibility into protocol state for its connections is innocuous.

#### Memory-key integrity protection.

We augment this and separately protect network stack memory _integrity_
(section 4.2). leverages hardware-assisted user-space memory protection
through Intel MPK [29]. MPK enforces similar page-level protection as
conventional virtual memory, while adding ISA support to efficiently switch
protection domains from user-space. reserves a hardware protection key for
data plane OS memory pages. We propose a new hardware mechanism to safeguard
the protection key from unauthorized application code and through a call-gate-
like mechanism enable controlled transitions. Compared to privilege mode and
address space switches, our MPK-based protection scheme incurs a fraction of
the overhead but only protects integrity.

#### Preemptive scheduling with scheduler activations.

Scheduler activation [2] is a classic solution for kernel preemption of user-
level threads. However, the original design creates additional activations –
equivalent to new kernel threads – on each preemption. The cost of spawning
kernel threads is prohibitively high 152057 at our required preemption
frequency. instead implements lightweight activations that enable preemptive
scheduling of user-level threads at microsecond granularity (section 4.3).
leverages high-resolution timers on modern CPUs to trigger $\mu$s-scale timer
interrupts. A timer callback function, implemented in a kernel module,
redirects these interrupts directly to the user-space data plane scheduler. To
reduce activation overheads, we reuse the current activation and bypass the
kernel scheduler. also maps local Advanced Programmable Interrupt Controller
(APIC) timer registers directly into the data plane OS, allowing the user-
level scheduler to dynamically configure preemption intervals based on runtime
information. Leveraging lightweight preemption, effectively approximates PS
scheduling and enforces timely I/O stack execution.

##  4 Design

In this section, we present the concrete design of the OS, including details
of data plane protection, fine-grained preemption, lightweight activation, and
scheduling policies.

###  4.1 Design Overview

1.0design

Figure 2: system architecture

fig:design shows the high-level architecture of . Similar to prior designs,
separates the control plane from the data plane OS. The control plane runs in
the kernel. It manages system-wide resources, allocates CPU cores, and grants
I/O queues to the data plane. The data plane OS contains both an I/O stack and
a scheduler, and runs in the user address space of each application. spawns
user-level threads for both the I/O stack and the application (by overriding
pthread_create() using LD_PRELOAD), multiplexed over a kernel thread running
on each allocated core. Using user-level threads reduces both context
switching and thread creation overhead. The latter is particularly important
for designs that spawn a thread for each user request [37, 16, 32]. leverages
Intel MPK to provide memory isolation for the data plane OS.

The control plane configures NIC flow steering rules to route network packets
to the appropriate RX queues assigned to the data plane 1. Once the I/O thread
is scheduled, it polls a batch of packets from the RX queue 2, and processes
them through the stack 3. Since the data plane OS is deployed in a per-
application manner, the I/O stack can be optimized and tailored to the
application’s need [8, 15, 34, 26]. After the entire batch is processed, the
I/O thread yields to the data plane user-level scheduler 4. The scheduler
maintains both a per-core and a global task queue. A task scheduling policy
(section 4.4) determines how the scheduler enqueues/dequeues tasks to/from the
two queues. After the scheduler picks an application task to run, it performs
a context switch and downgrades the user-space protection level (section 4.2)
before switching to the thread 5. This ensures that the application has no
access right to memory pages of the data plane OS. The application may perform
networking I/O during the task. exposes a data plane system call interface for
requesting services in the data plane OS. The system call routine upgrades the
user-space protection level, calls the requested service handler 6, and
downgrades the protection level before returning to the application 7. When
the application thread yields or is blocked, control is returned to the user-
space scheduler, who can choose a different task or the I/O thread to run.

uses fine-grained preemptive scheduling, with a configurable preemption
interval down to 10$\mu$s. When timer interrupt occurs a, the kernel interrupt
handler performs a lightweight scheduler activation that upcalls into the data
plane user-level scheduler b, passing along the hardware-saved context of the
preempted thread. Departing from the original scheduler activation design,
reuses the same kernel thread for activation, saving the expensive cost of
creating a new activation in the kernel. If the user-level scheduler decides
that the preempted task has run for too long or the I/O stack needs to be
scheduled (section 4.4), it chooses another application thread or the I/O
thread c to run, moving the preempted task to the end of the run queue d.
Otherwise, it resumes execution of the preempted thread e.

###  4.2 Data Plane OS Protection

runs the data plane OS and application threads in the same user address space.
To protect the data plane, leverages user-space memory protection keys (MPK)
in recent Intel CPUs [29] to implement efficient memory isolation.

#### MPK primer.

MPK enables associating groups of memory pages in an address space to one of
16 hardware protection keys. To control accesses to each page group, MPK
introduces a new PKRU CPU register that contains the permission bits for each
of 16 protection keys in the local hardware thread. To modify group access
permissions, a user space program invokes the new non-privileged WRPKRU
instruction to update the corresponding bits in the PKRU register. WRPKRU
requires no context switches or TLB flushes, and takes less than 20 CPU cycles
[33]. Moreover, MPK enables a finer, per-thread view of memory access
permissions.

1.0dpcall

Figure 3: data plane call procedure.

#### MPK-based data plane OS protection.

fig:dpcall shows ’ data plane protection mechanism. We reserve the last memory
protection key, , for protecting the data plane OS (bits 30-31 of PKRU). The
remaining protection keys are still available for the applications. When
allocating memory to the data plane OS (including code, stack, and data
pages), associates with each memory page using pkey_mprotect(). To isolate the
data plane from applications, sets the Access Disable (AD) and Write Disable
(WD) bit for in the PKRU register when running application threads
(disable_ckey()). The MMU hardware ensures that memory regions of the data
plane OS are neither readable nor writable, but still executable, for
application threads. The AD and WD bits are cleared when entering the data
plane OS (enable_ckey()). To prevent users from circumventing our protection
mechanism, we instrument the mprotect() and pkey_mprotect() system calls to
reject attempts to modify access rights and protection keys for pages in the
data plane OS.

When an application thread requests services from the data plane (e.g., TCP
socket send()) or performs scheduling related operations (e.g., thread
synchronization and yield), performs a fast user-level context switch to the
data plane OS. We name these fast path system calls s. fig:dpcall shows the
procedure of performing a : The application thread writes the call number and
function arguments into registers, and jumps to a data plane entry function.
The entry function is also mapped in the data plane memory region –
application code can execute the entry point, but are prohibited to read or
modify it. The function clears AD and WD for in the PKRU register. It then
saves the application thread context, switches to the data plane stack, and
jumps to the appropriate handler using the call number and an indirection
table. Once the handler finishes serving the request, it calls a data plane
exit function, which restores the application thread context, sets AD and WD
bits for in PKRU, and jumps to the application thread return address.

#### Protecting data plane’s memory protection key.

PKRU is an unprotected x86 register which can be modified by application code
using non-privileged instruction. By changing the access bits in PKRU, user
applications can gain unwarranted read and write access to the protected data
plane memory. One solution is to apply ERIM [41] style binary inspection to
prevent unsafe application code from modifying the PKRU register. Such binary
inspection can be integrated into compilers.

Listing 1: WRPKRU instruction operation. Compared to the original x86 WRPKRU
instruction, we add instruction pointer tests to restrict updating
(PKRU[31:30]) at the program locations set by the control plane.

⬇

IF (ECX = 0 AND EDX = 0)

IF (RIP = CR9 OR RIP = CR10)

THEN PKRU := EAX;

ELSE PKRU[29:0] := EAX[29:0]

ELSE #GP(0);

FI;

However, to avoid the expensive binary inspection and including compilers in
the trusted computing base, we propose a lightweight hardware mechanism to
prevent unauthorized updates to . We use two 64 bit reserved control registers
(e.g., CR9 and CR10 on x86-64) to hold the addresses of the two instructions
that update the access rights for , one in the data plane entry function and
one in the exit function. Like other control registers, only the control plane
OS running in privileged-mode can modify their content. We then modify the
WRPKRU instruction, as shown in code:wrpkru, to perform the following check:
if the program counter does not equal to either of the control registers, the
instruction only updates the lower 30 bits of the PKRU register. The effect of
this hardware mechanism is that only the data plane entry and exit functions
can modify the hardware thread’s access rights to .

#### Security analysis of the data plane.

Our hardware-based protection mechanism is secure against indirect jump and
return-oriented programming (ROP) attacks. Only a single instruction in
enable_ckey() is permitted to modify in PKRU. We manually program
enable_ckey(), save_context(), and switch_stack() in assembly code, and ensure
only direct jumps are used. This enforces a single control flow between is
updated and the data plane stack is switched. Since the data plane stack
resides in protected memory, user programs are prohibited to write return
addresses into the stack. The same mechanism is applied to the switch_stack(),
restore_context(), and disable_ckey() sequence. Consequently, user programs
are incapable of escalating their privilege by running with AD and WD of
disabled.

Our mechanism, however, suffers the same security vulnerabilities of Intel MPK
itself. Most notably, Spectre [20] and Meltdown [25] attacks can bypass
hardware protection of MPK [5]. therefore offers weaker security guarantees
than Linux kernel with kernel page-table isolation (KPTI) [11].

#### Hardware Implementation and Resource Overhead

To validate the feasibility of our hardware proposal, we implement our MPK
design in RTL in the open-source CVA6 [46] 64-bit RISC-V CPU. CVA6 is a
6-stage, single-issue, in-order processor core. It implements RISC-V I, M, A,
and C extensions, plus three privilege levels M, S, and U. Unfortunately,
current RISC-V extensions have no support for user-level memory protections.
We therefore use the Donky [39] modified CVA6 core as the base design, which
implements MPK-like memory protection in CVA6.

Our implementation adds two RISC-V supervisor mode CSRs, CSR_SMPK_ENTRY and
CSR_SMPK_EXIT, which content can only be modified in supervisor mode. We
reserve the last protection key slot (slot 3, bits [43:33]) in the MPK
register (CSR_MPK) to . When updating the MPK register in user mode, we
additionally check the program counter (pc_i) against the two added CSRs. If
the program counter differs from the content of both registers, the reserved
protection key remains unmodified.

CPU | LUT | Register | BRAM | DSP
---|---|---|---|---
Donky | 30.97% | 11.73% | 11.24% | 3.21%
| 31.07% | 11.76% | 11.24% | 3.21%
Overhead | 0.10%(205) | 0.03%(128) | 0.00% | 0.00%
Table 2: FPGA resource usage of RISC-V CPU core

We synthesized our hardware design using Xilinx Vivado [42] $2021.2$. The
hardware target is a Digilent Genesys2 [10] board which equips with a Xilinx
Kintex-7 XC7K325T FPGA. tab:hw-res shows the generated resource utilization
report. Compared to the original Donky design, only uses $0.10\%(205)$ and
$0.03\%(128)$ additional LUT and register resources on the FPGA. Moreover, our
hardware modifications do not slow down the processor: the frequency of all
the generated clocks remains unchanged from the original design. The synthesis
result testifies that our MPK proposal is highly feasible on real hardware.
This should not come as a surprise, given the simplicity of the design.

###  4.3 Preemptive Scheduling and Lightweight Activation

relies on frequent preemption of user-level threads to ensure strong tail
latency guarantee and timely execution of the I/O stack. Existing timer
interrupt and preemptive scheduling mechanisms in Linux, however, are
inadequate for two reasons: First, the high overhead of running the kernel
scheduler and context switching among kernel threads limits the preemption
interval to milliseconds, much longer than our $\mu$s-scale tail latency
target. Second, the kernel scheduler has no visibility of user-level threads.
After a preemption, the scheduler is incapable of switching to other ready
user-level threads in the same kernel thread.

Scheduler activation [2] is a classic solution to the system integration
issues of user-level threads. However, the original design creates additional
activations – equivalent to new kernel threads – on each preemption. The cost
of spawning kernel threads is prohibited high 152057 at our intended
preemption frequency. To address this challenge, we implement a lightweight
activation mechanism to enable fine-grained preemptive scheduling of user-
level threads. Since our target preemption interval is much lower than the
kernel jiffy, we leverage the high resolution timer (hrtimer) system provided
by Linux [12], which internally uses the Advanced Programmable Interrupt
Controller (APIC) timers on x86 machines.

#### Lightweight activation mechanism.

The control plane registers a hrtimer callback function with the OS kernel
when initializing the data plan OS. uses this callback function as a
trampoline to jump to the data plane user-level scheduler. When the callback
is invoked by the kernel hrtimer IRQ handler, it stores the context of the
interrupted thread (saved by the hardware interrupt handler) into the stack of
the data plane OS. It then modifies the saved context to that of the data
plane OS, with the return address set to the entry function of the data plane
scheduler. Once the kernel completes the interrupt and returns to user-space,
it restores the modified context and jumps to the data plane scheduler,
instead of the original interrupted user-level thread. The data plane
scheduler finds the context of the interrupted user-level thread in its stack,
and decides if it should switch to a different user-level thread to run
(section 4.4). The I/O stack can additional register timers (e.g., for TCP
retransmission) with the data plane scheduler. The scheduler switches to the
I/O thread when one of the timers expire.

| POSIX signal | IPI |  hrtimer
---|---|---|---
Sender | 2418 | 42077.95 | -
Receiver | - | 884.67 | 1097.92
Total | 5645 | 32417.18 | 1097.92
Table 3: Interrupt overhead in CPU cycles for POSIX signal, IPI, and hrtimer-
based activation.

Our hrtimer-based preemption scheme is faster than alternative approaches. For
instance, the centralized dispatcher in Shinjuku [16] uses POSIX signals or
inter-processor interrupts (IPIs) to preempt worker threads. The overhead of
processing signals or IPIs in Linux, however, is substantial. In
tab:interrupt, we compare the performance (in CPU cycles) of the various
approaches to deliver interrupts to the data plane scheduler. We show in
section 5.1 that the low-overhead of our activation mechanism enables to
perform preemptive scheduling at 10$\mu$s granularity.

#### Configurable preemption in the data plane OS.

The above design works well for preemption that occurs at regular interval:
The control plane configures the interval in the callback function, and when
the callback is invoked, it resets the hrtimer to expire after the interval. A
fixed interval, however, may lead to many unnecessary interrupts that
negatively impact the efficiency of the system. As we later explain in section
4.4, only preempts a task if its runtime exceeds a threshold. For shorter
tasks, handling highly frequent interrupts simply waste CPU cycles. We
therefore augment the data plane scheduler to dynamically configure hrtimer
preemption. Specifically, the control plane maps local APIC timer registers
(APIC_TDCR, APIC_TMICT, and APIC_TMCCT) into data plane OS memory during
initialization. Writing to these registers allows the data plane to directly
set, reset, and cancel APIC timers. When scheduling a task to run, the
scheduler sets the hardware timer to expire after a threshold; it cancels the
timer if the task completes before the threshold, avoiding unnecessary
interrupts. Since the registers are mapped to data plane OS memory, our
protection mechanism (section 4.2) ensures hardware timers are inaccessible to
user programs. Note that in the more recent x2APIC [44] architecture, APIC
registers can only be accessed through Model Specific Register (MSR)
interfaces, which require privileged instructions. As a workaround, we
explicitly disable x2APIC in the kernel parameters (nox2apic).

###  4.4 Task Scheduling Policy

One benefit of running the data plane scheduler in user-space is that the
scheduling policy can be tailored to the requirements of the target
application [13]. In this work, we implement a scheduling policy that offers
good tail latency guarantees across a wide range of application workloads.

#### Approximating c-FCFS and PS scheduling.

As we argued in section 2, a scheduling policy that combines c-FCFS and PS
theoretically gives the best tail latency behavior for many classes of service
time distribution. To approximate c-FCFS, the data plane scheduler maintains
both a core local- and a global task queue. Note that the global queue is only
for threads within the application process. When new user-level threads are
created or become ready, they are enqueued to the tail of the global task
queue. The data plane scheduler runs threads from the local queue in FCFS
order, and if the local queue is empty, it pops up to $N$ threads from the
global queue and adds them to the local task queue. $N$ is a configurable
parameter that controls the trade-off between scheduler efficiency and
accuracy of c-FCFS approximation: When a smaller $N$ is used, tasks are more
likely to be scheduled in the order they arrive, but the scheduler suffers
higher synchronization overhead contending on the global task queue; when
using a larger $N$, frequency of thread synchronization is reduced, but the
task scheduling deviates more from c-FCFS towards a d-FCFS policy.

To approximate PS scheduling, the data plane OS defines a time quanta $T$.
When the data plane scheduler receives a timer activation (section 4.3), it
checks if the current running thread has been scheduled for more than $T$. If
the thread has exceeded the quanta, the scheduler pops the task from the local
queue, enqueues it to the end of the global task queue, and schedules the
thread at the head of the local queue.

#### Choosing between I/O and application threads.

The data plane scheduler has two options when making scheduling decisions:
picking an application thread to run, or scheduling the data plane I/O thread.
To ensure good application-level tail latency behavior, application threads
typically are given higher scheduling priority than the I/O thread. However,
there are two cases where the I/O thread takes precedence. Firstly, certain
I/O tasks are time sensitive and need to be scheduled with real-time
guarantees. Examples include TCP acknowledgement and retransmission. For these
tasks, the I/O stack registers special timers with the scheduler. The
scheduler runs the I/O thread if any of the timers has expired, even when
application threads are ready. Second, long-running application tasks, such as
garbage collection, should not block I/O processing indefinitely. This ensures
both high I/O performance and predictable latency for short tasks. Towards
this goal, we apply a dynamic priority adaptation policy. When an application
task is preempted and placed at the end of the run queue (i.e., they have
exceeded their scheduling quota), the task is marked as a low priority job.
When the head of the queue is a low priority task, the scheduler runs the I/O
thread, and only schedules the task after the I/O thread completes.

##  5 Evaluation

We implemented in 12863 lines of C and 150 lines of assembly code, including
both the control plane and the data plane OS. The kernel module is written in
301 lines of C code. Our testbed consists of two servers each equipped with
dual-socket 16-core Intel Xeon Gold 6326 2.90 GHz CPUs, 256 GB RAM, and a
Mellanox ConnectX-5 EN 100 GbE single-port NIC. The two servers are directly
connected by a 100 GbE QSFP28 cable. For software environment, our servers run
Ubuntu Linux 20.04 with kernel version 5.4.0 and DPDK 20.11.3 LTS. We reserve
2048 2 MB huge pages for DPDK. When evaluating Linux performance, we pin
benchmark processes to CPU cores and set their scheduler priority to real-
time. Note that when evaluating IX, we use Intel XL710 40Gbps NICs, Ubuntu
Linux 16.04 with kernel version 4.4.0, and DPDK version 16.04, as those are
the only configurations supported by IX.

#### Comparison systems.

We compare to five other systems: a baseline Linux-based system, mTCP [15], IX
[4], ZygOS [36], and TAS [19]. The Linux-based system runs the in-kernel
network stack, which we configured and tuned using the instructions in [23]
(denoted as Linux in the graphs). mTCP and IX are recent kernel-bypassed,
libOS-style I/O stacks that optimize host networking performance for data
center applications. To protect the stack from untrusted applications, IX uses
hardware virtualization techniques in Dune [3] to run the data plane in VMX
non-root mode. ZygOS builds upon IX and implements work stealing for better
tail latency. TAS implements a centralized TCP service running in dedicated
processes and CPU cores. TAS additionally splits TCP processing into a fast-
path and a slow-path.

###  5.1 Micro-benchmarks

To evaluate the basic performance characteristics of , we run a simple TCP
echo benchmark. We deploy open-loop clients which generate TCP request
messages with a fixed payload size (64 bytes if not otherwise specified). We
run a server process on the other machine that replies each request with the
same payload. The application runs a single server thread on each server core.
For TAS, we allocate one extra core to run the centralized fast path TCP
stack.

eval-echo

(a) Single core echo

eval-netpipe

(b) NetPIPE benchmark

eval-overhead

(c) Overhead of protection and activation mechanisms

eval-multicore

(d) Multicore scalability

Figure 4: Micro-benchmarks to show performance characteristics of . (a) shows
tail latency vs. throughput when running an open-loop TCP echo benchmark. We
run a NetPIPE benchmark in (b) to evaluate the throughput of with increasing
message size. In (c), we disable MPK-based protection and scheduler activation
in turn to measure their impact on performance. In (d), we run the TCP echo
benchmark with increasing number of CPU cores to measure multicore
scalability.

#### Single core performance.

We first deploy a single server core, gradually increase the offered client
load, and measure both the average latency and the throughput of each system.
outperforms all comparison systems in latency by a wide margin, even though
those systems are highly optimized for single core small-sized I/O. This level
of processing efficiency is attributed to the low overhead of ’ protection and
activation mechanisms. does attain lower single core throughput than the other
systems. Both mTCP and IX use aggressive batching for higher throughput. , on
the other hand, optimizes for latency. TAS’ high throughput is attributed to
the additional CPU resource. When we allocate a single core to TAS, its
throughput drops significantly.

We also use the NetPIPE [30] benchmark with increasing buffer size to measure
how performs when processing larger messages, and show the result in fig:eval-
netpipe. attains above 15 Gbps throughput when processing 64 KB messages on a
single core. This throughput is 0.95$\times$ higher than Linux, and within
6.18% of TAS which uses two cores.

#### Overhead of data plane protection and activation.

To understand the overhead of our protection (section 4.2) and activation
(section 4.3) mechanisms, we use the same TCP benchmark, and test with the
following configurations: 1) no protection for the data plane and disable
timer activation, 2) enable protection but keep timer activation disabled, and
3) full with both protection and timer activation. fig:eval-overhead shows the
average latency with increasing offered load for each configuration.

Enabling protection and timer activation has minimum impact on the end-to-end
latency of (close to 0% higher latency). As discussed in sec:design, the MPK-
based protection mechanism adds around 30ns overhead to context switching.
These overheads are dwarfed by the network and other software components in
the end-to-end latency. For a simple echo application that has near-zero
application-level processing, these overheads have bigger impact on the
maximum throughput. The configuration with both protection and preemption
disabled achieves a throughput of 1.16 Mops, translating to 858.4 ns per
request on average. Enabling protection and adding preemptive scheduling
reduce the throughput by 7.9%. This throughput impact, however, will decrease
with higher application-level work.

#### Multicore scalability benchmark

To test how well scales, we deploy the same TCP echo application on an
increasing number of CPU cores and measure the maximum attainable throughput.
As shown in fig:eval-scale, throughput of scales close to linearly to the
number of cores. With RSS, network packets are steered evenly across all the
cores, distributing the I/O work roughly equally to each core. Since does not
steal I/O work across cores, each I/O stack can perform packet processing
independently, minimizing thread synchronization that limits scalability. Work
stealing done by the scheduler also has minimum scalability impact, as our
simple echo benchmark can hardly lead to skewed application-level work across
cores.

###  5.2 Synthetic Workloads

Next, we test and the comparison systems with a more realistic set of
application workloads. We use the same set of motivating experiments in
sec:background:dilemma: one server application running on 16 server cores. The
application performs dummy work to simulate various service-time distribution:
(a) a uniform distribution with a mean of 2.5$\mu$s, (b) a bimodal
distribution with 99.5% of requests take 1$\mu$s and 0.5% of requests take
1000$\mu$s, and (c) an imbalanced workload where 5 cores receive 136 Kops of
requests while the remaining cores receive request load of 13.6 Kops.

In fig:motiv-uniform, we use the uniform workload, gradually increase the
offered client load, and measure the median latency of each system. All the
comparison systems are optimized for this simple workload. Even though pays
the cost of MPK-based protection, work-stealing for approximating c-FCFS, and
timer activation for preemptive scheduling, it still outperforms IX and ZygOS
both in throughput (31%) and latency (59%). achieves equivalent latency as
TAS, while attains lower maximum throughput.

fig:motiv-bimodal shows the same tail latency vs. offered load measurement
when running our bimodal service-time workload. As explained in
sec:background:dilemma, the lack of preemptive scheduling support in IX leads
to long tail latency behavior. Work stealing in ZygOS alleviates this issue to
a certain extent, as idle cores can steal short tasks that are being blocked.
However, its effectiveness is rather limited beyond moderate client load. In
contrast, aggressively preempts long-running tasks, allowing subsequent short
tasks to complete without suffering long queuing delay. maintains tail latency
below 1000$\mu$s before hitting load saturation (5 Mops), while IX and ZygOS
already show tail latency above 1000$\mu$s at load beyond 0.2 Mops.

Lastly, we run our imbalanced service-time workload and show the result in
fig:motiv-imbalanced. By performing dynamic work stealing, effectively
addresses the load imbalance issue caused by d-FCFS scheduling – it improves
99% tail latency by more than 6.4$\times$ comparing to IX. It also outperforms
ZygOS in tail latency by 4.1$\times$ as ZygOS lacks preemptive scheduling
support, therefore suffering from head-of-line blocking.

###  5.3 Resource Efficiency

eval-resource-allocation-15cores

Figure 5: Evaluation of resource efficiency. The application simulates an
exponentially distributed service-time. We measure the maximum throughput
achieved by and TAS. For TAS, we manually sweep all CPU allocation space.

To evaluate ’ efficiency at utilizing CPU resources, we run an exponentially
distributed service-time workload, and measure the maximum throughput
attainable by and TAS. We allocate 16 CPU cores to both systems. allocates one
core to the control plane, while TAS uses one core for TCP slow path.
allocates the remaining 15 cores to the application. For TAS, we manually
adjust the partition of cores between TCP fast path and the application.
fig:eval-resource shows the maximum throughput attained by each configuration.
For TAS, when less than four cores are allocated to TCP fast path, network
processing becomes the bottleneck of the system, while the application cores
are under-utilized. When more than four cores are allocated to TCP fast path,
application cores are fully saturated, but the fast path cores have idle
cycles that are not being utilized. In contrast, is able to efficiently
multiplex CPU cores between I/O processing and application-level work, fully
utilizing all the available CPU resources. Our scheduling policy and
preemption mechanism guarantee that neither I/O nor application threads will
block the other component, ensuring fair sharing of CPU cycles. Even comparing
to the best configuration of TAS (4 cores for TCP fast path and 11 cores for
application), achieves 6.6% higher throughput.

###  5.4 Performance Isolation

Lastly, we evaluate the performance isolation property of , and compare to
TAS. This is the same isolation experiment we used in sec:background:dilemma:
two colocating applications; one latency critical application (LC) that
generates small TCP messages, and one high-throughput application (HT) that
transmits larger messages and requires higher overall throughput. We allocate
15 total cores to both and TAS, with one core assigned to the control plane
and TCP slow path respectively. When deploying , we evenly divide the
remaining 14 cores to LC and HT. For TAS, we allocate 2 cores for TCP fast
path, and evenly divide the remaining cores to the two applications. We fix
the load of HT at 4 Gbps and gradually increase the load of LC.

As shown in fig:motiv-isolation, the decentralized model of allows I/O
processing, scheduling, and application-level work to be physically
partitioned between the two applications. Increase in the LC load therefore
does not affect HT at all: sustainable throughput of HT stays constantly at 40
Gbps, regardless of the LC load. At the same time, bursty I/O processing of HT
has no impact on the tail latency of LC. This is in contrast to the behavior
of TAS. As we explained in sec:background:dilemma, the two applications are
contending over the shared fast path cores, resulting in performance
interference. Consequently, achieves a 0.94$\times$ higher sustainable LC load
under a tail latency SLO of 100$\mu$s.

###  5.5 Memcached

eval-memcached

Figure 6: Memcached evaluation

Lastly, we evaluate how well performs on real world applications. We use the
popular memcached [28] key-value store as our target application, and compare
the performance across Linux, TAS, and . Porting memcached to requires minimum
effort, since exposes a POSIX-compliant API. We only need to instrument the
memcached server code to use our threading library. We allocate four total
cores to each system. For Linux and , we spawn four memcached server threads,
each pinned to a dedicated core. For TAS, we allocate one core for TCP fast
path processing and three cores for three memcached threads. We use the
popular YCSB a – d benchmarks for evaluation.

fig:eval-memcached shows the maximum throughput achieved by each system.
outperforms both Linux and TAS in throughput by a wide margin for three of the
four benchmarks. For instance, in YCSB workload c, a read-only workload,
attains 343% and 41% higher throughput than Linux and TAS respectively. The
improvement in throughput can be attributed to the fast I/O processing and
lightweight protection mechanisms of . Unfortunately, current has a
performance issue in mutex, and thus obtains a lower throughput in YCSB
workload a, which contains 50% write operations. We are currently working on a
fix of the issue.

##  6 Related Work

#### Kernel bypass data plane OS:

The library operating system [8, 35] architecture has inspired a recent line
of work that implements data plane OSes in user-space [34, 4, 36, 47, 24, 18],
bypassing the kernel for higher I/O performance. Arrakis [34] leverages
hardware I/O virtualization (SR-IOV and IOMMU) to provide protection to the
data plane OS. However, the approach is constrained by the available virtual
functions and limited coarse-grain resource controls such as rate limiters and
filtering. IX [4] uses CPU hardware virtualization techniques such as VT-x
[40] to deploy a protected data plane kernel in privileged non-root ring 0.
ZygOS [36] implements work stealing atop IX to approximate c-FCFS scheduling
for better tail latency. Demikernel [47] implements a general-purpose,
portable data plane OS that works on heterogeneous kernel-bypass devices.
Compared to prior kernel-bypass library OSes, offers stronger protection for
the data plane without compromising performance, and enables aggressive
preemptive scheduling for stronger tail latency guarantees.

#### Centralized I/O and scheduling:

The non-optimal task scheduling and weaker protection model have lead to
proposals of centralized I/O and scheduling architectures [19, 27, 16, 32, 9,
13]. TAS [19] and Snap [27] implement networking I/O as a centralized OS
service, similar to a microkernel architecture, that runs in a separate
process on dedicated cores. Shinjuku [16] uses a centralized core to schedule
all application-level tasks in a c-FCFS manner and preempts long-running tasks
to simulate PS scheduling. Shenango [32] and Caladan [9] implements a
centralized scheduler that collects control signals and allocates CPU cores to
applications at fine granularity. As we have demonstrated in section 2 and
section 5, a centralized architecture can lead to non-optimal usage of CPU
resources. addresses this issue by multiplexing CPU cores among I/O,
scheduler, and application work.

#### I/O stack specialization:

An orthogonal line of work implements specialized I/O stack in user-space,
using the DPDK [7] library or RDMA hardware. Sandstorm [26], mTCP [15], eRPC
[17], Andromeda [6], and SocksDirect [21] fall under this category. These
work, however, do not offer the protection and manageability benefits of an
operating system. The I/O stack in can apply optimization techniques proposed
by these work to further improve performance.

#### User-level threading:

User-level threading system [37, 43, 1] offers better performance and
predictability than kernel threads, thus becoming an attractive option for
deploying microsecond-scale applications [47, 16]. Prior user-level threading
systems, however, only support cooperative scheduling, which can result in
poor tail latency and delaying of time-sensitive I/O tasks. addresses this
issue by implementing fine-grained preemptive scheduling for user-level
threads using lightweight timer activation.

##  7 Conclusion

In this paper, we described , a new OS networking architecture that provides
high performance but without compromising protection, functionality, or
manageability. Unlike previous approaches, runs the OS networking service in
shadow user-level threads, and protects the network stack through memory
isolation using Intel Memory Protection Keys. We have also implemented I/O
optimized scheduler activations to enforce correct and timely execution of
network protocols in user-level threads.

## References

  * [1] A. Adya, J. Howell, M. Theimer, W. J. Bolosky, and J. R. Douceur.  Cooperative task management without manual stack management.  In Proceedings of the General Track of the Annual Conference on USENIX Annual Technical Conference, ATEC ’02, page 289–302, USA, 2002. USENIX Association.
  * [2] T. E. Anderson, B. N. Bershad, E. D. Lazowska, and H. M. Levy.  Scheduler Activations: Effective Kernel Support for the User-Level Management of Parallelism.  In Proceedings of the Thirteenth ACM Symposium on Operating Systems Principles, SOSP ’91, pages 95–109, Pacific Grove, California, USA, 1991. Association for Computing Machinery.
  * [3] A. Belay, A. Bittau, A. Mashtizadeh, D. Terei, D. Mazières, and C. Kozyrakis.  Dune: Safe User-Level Access to Privileged CPU Features.  In Proceedings of the 10th USENIX Conference on Operating Systems Design and Implementation, OSDI ’12, pages 335–348, Hollywood, CA, USA, 2012. USENIX Association.
  * [4] A. Belay, G. Prekas, A. Klimovic, S. Grossman, C. Kozyrakis, and E. Bugnion.  IX: A Protected Dataplane Operating System for High Throughput and Low Latency.  In Proceedings of the 11th USENIX Conference on Operating Systems Design and Implementation, OSDI ’14, pages 49–65, Broomfield, CO, 2014. USENIX Association.
  * [5] C. Canella, J. V. Bulck, M. Schwarz, M. Lipp, B. von Berg, P. Ortner, F. Piessens, D. Evtyushkin, and D. Gruss.  A systematic evaluation of transient execution attacks and defenses.  In 28th USENIX Security Symposium (USENIX Security 19), pages 249–266, Santa Clara, CA, Aug. 2019. USENIX Association.
  * [6] M. Dalton, D. Schultz, J. Adriaens, A. Arefin, A. Gupta, B. Fahs, D. Rubinstein, E. C. Zermeno, E. Rubow, J. A. Docauer, J. Alpert, J. Ai, J. Olson, K. DeCabooter, M. De Kruijf, N. Hua, N. Lewis, N. Kasinadhuni, R. Crepaldi, S. Krishnan, S. Venkata, Y. Richter, U. Naik, and A. Vahdat.  Andromeda: Performance, isolation, and velocity at scale in cloud network virtualization.  In Proceedings of the 15th USENIX Conference on Networked Systems Design and Implementation, NSDI’18, page 373–387, USA, 2018. USENIX Association.
  * [7] Intel data plane development kit.  https://www.dpdk.org/.
  * [8] D. R. Engler, M. F. Kaashoek, and J. O’Toole.  Exokernel: An Operating System Architecture for Application-Level Resource Management.  In Proceedings of the Fifteenth ACM Symposium on Operating Systems Principles, SOSP ’95, pages 251–266, Copper Mountain, Colorado, USA, 1995. Association for Computing Machinery.
  * [9] J. Fried, Z. Ruan, A. Ousterhout, and A. Belay.  Caladan: Mitigating Interference at Microsecond Timescales.  In 14th USENIX Symposium on Operating Systems Design and Implementation, OSDI ’20, Virtual Event, Nov. 2020. USENIX Association.
  * [10] Digilent Genesys 2 FPGA board.  https://digilent.com/reference/programmable-logic/genesys-2/start.
  * [11] D. Gruss, M. Lipp, M. Schwarz, R. Fellner, C. Maurice, and S. Mangard.  Kaslr is dead: Long live kaslr.  In E. Bodden, M. Payer, and E. Athanasopoulos, editors, Engineering Secure Software and Systems, pages 161–176, Cham, 2017. Springer International Publishing.
  * [12] Linux subsystem for high-resolution kernel timers.  https://docs.kernel.org/timers/hrtimers.html.
  * [13] J. T. Humphries, N. Natu, A. Chaugule, O. Weisse, B. Rhoden, J. Don, L. Rizzo, O. Rombakh, P. Turner, and C. Kozyrakis.  GhOSt: Fast and Flexible User-Space Delegation of Linux Scheduling.  In Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles, SOSP ’21, Virtual Event, Germany, 2021\. Association for Computing Machinery.
  * [14] V. Jalaparti, P. Bodik, S. Kandula, I. Menache, M. Rybalkin, and C. Yan.  Speeding up Distributed Request-Response Workflows.  In Proceedings of the ACM SIGCOMM 2013 Conference on SIGCOMM, SIGCOMM ’13, pages 219–230, Hong Kong, China, 2013. Association for Computing Machinery.
  * [15] E. Y. Jeong, S. Woo, M. Jamshed, H. Jeong, S. Ihm, D. Han, and K. Park.  mTCP: A Highly Scalable User-Level TCP Stack for Multicore Systems.  In Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation, NSDI ’14, pages 489–502, Seattle, WA, 2014. USENIX Association.
  * [16] K. Kaffes, T. Chong, J. T. Humphries, A. Belay, D. Mazières, and C. Kozyrakis.  Shinjuku: Preemptive Scheduling for µSecond-Scale Tail Latency.  In Proceedings of the 16th USENIX Conference on Networked Systems Design and Implementation, NSDI ’19, Boston, MA, USA, 2019\. USENIX Association.
  * [17] A. Kalia, M. Kaminsky, and D. G. Andersen.  Datacenter rpcs can be general and fast.  In Proceedings of the 16th USENIX Conference on Networked Systems Design and Implementation, NSDI’19, page 1–16, USA, 2019. USENIX Association.
  * [18] R. Kapoor, G. Porter, M. Tewari, G. M. Voelker, and A. Vahdat.  Chronos: Predictable low latency for data center applications.  In Proceedings of the Third ACM Symposium on Cloud Computing, SoCC ’12, New York, NY, USA, 2012. Association for Computing Machinery.
  * [19] A. Kaufmann, T. Stamler, S. Peter, N. K. Sharma, A. Krishnamurthy, and T. Anderson.  TAS: TCP Acceleration as an OS Service.  In Proceedings of the Fourteenth EuroSys Conference 2019, EuroSys ’19, Dresden, Germany, 2019. Association for Computing Machinery.
  * [20] P. Kocher, J. Horn, A. Fogh, , D. Genkin, D. Gruss, W. Haas, M. Hamburg, M. Lipp, S. Mangard, T. Prescher, M. Schwarz, and Y. Yarom.  Spectre attacks: Exploiting speculative execution.  In 40th IEEE Symposium on Security and Privacy (S&P’19), 2019.
  * [21] B. Li, T. Cui, Z. Wang, W. Bai, and L. Zhang.  Socksdirect: Datacenter sockets can be fast and compatible.  In Proceedings of the ACM Special Interest Group on Data Communication, SIGCOMM ’19, page 90–103, New York, NY, USA, 2019. Association for Computing Machinery.
  * [22] J. Li, N. K. Sharma, D. R. K. Ports, and S. D. Gribble.  Tales of the Tail: Hardware, OS, and Application-Level Sources of Tail Latency.  In Proceedings of the ACM Symposium on Cloud Computing, SOCC ’14, pages 1–14, Seattle, WA, USA, 2014. Association for Computing Machinery.
  * [23] J. Li, N. K. Sharma, D. R. K. Ports, and S. D. Gribble.  Tales of the Tail: Hardware, OS, and Application-Level Sources of Tail Latency.  In Proceedings of the ACM Symposium on Cloud Computing, SOCC ’14, Seattle, WA, USA, 2014. Association for Computing Machinery.
  * [24] H. Lim, D. Han, D. G. Andersen, and M. Kaminsky.  Mica: A holistic approach to fast in-memory key-value storage.  In Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation, NSDI’14, page 429–444, USA, 2014. USENIX Association.
  * [25] M. Lipp, M. Schwarz, D. Gruss, T. Prescher, W. Haas, A. Fogh, J. Horn, S. Mangard, P. Kocher, D. Genkin, Y. Yarom, and M. Hamburg.  Meltdown: Reading kernel memory from user space.  In 27th USENIX Security Symposium (USENIX Security 18), 2018\.
  * [26] I. Marinos, R. N. Watson, and M. Handley.  Network Stack Specialization for Performance.  In Proceedings of the 2014 ACM Conference on SIGCOMM, SIGCOMM ’14, pages 175–186, Chicago, Illinois, USA, 2014. Association for Computing Machinery.  SandStorm.
  * [27] M. Marty, M. de Kruijf, J. Adriaens, C. Alfeld, S. Bauer, C. Contavalli, M. Dalton, N. Dukkipati, W. C. Evans, S. Gribble, N. Kidd, R. Kononov, G. Kumar, C. Mauer, E. Musick, L. Olson, E. Rubow, M. Ryan, K. Springborn, P. Turner, V. Valancius, X. Wang, and A. Vahdat.  Snap: A Microkernel Approach to Host Networking.  In Proceedings of the 27th ACM Symposium on Operating Systems Principles, SOSP ’19, pages 399–413, Huntsville, Ontario, Canada, 2019. Association for Computing Machinery.
  * [28] memcached distributed memory object caching system.  https://memcached.org/.
  * [29] Intel 64 and IA-32 architectures software developer manuals.  https://software.intel.com/content/www/us/en/develop/articles/intel-sdm.html.
  * [30] Network Protocol Independent Performance Evaluator.  https://linux.die.net/man/1/netpipe.
  * [31] R. Nishtala, H. Fugal, S. Grimm, M. Kwiatkowski, H. Lee, H. C. Li, R. McElroy, M. Paleczny, D. Peek, P. Saab, D. Stafford, T. Tung, and V. Venkataramani.  Scaling Memcache at Facebook.  In Proceedings of the 10th USENIX Conference on Networked Systems Design and Implementation, NSDI ’13, pages 385–398, Lombard, IL, 2013. USENIX Association.
  * [32] A. Ousterhout, J. Fried, J. Behrens, A. Belay, and H. Balakrishnan.  Shenango: Achieving High CPU Efficiency for Latency-sensitive Datacenter Workloads.  In 16th USENIX Symposium on Networked Systems Design and Implementation, NSDI ’19, Boston, MA, Feb. 2019. USENIX Association.
  * [33] S. Park, S. Lee, W. Xu, H. Moon, and T. Kim.  Libmpk: Software Abstraction for Intel Memory Protection Keys (Intel MPK).  In Proceedings of the 2019 USENIX Conference on Usenix Annual Technical Conference, USENIX ATC ’19, pages 241–254, Renton, WA, USA, 2019. USENIX Association.
  * [34] S. Peter, J. Li, I. Zhang, D. R. K. Ports, D. Woos, A. Krishnamurthy, T. Anderson, and T. Roscoe.  Arrakis: The Operating System is the Control Plane.  In Proceedings of the 11th USENIX Conference on Operating Systems Design and Implementation, OSDI ’14, pages 1–16, Broomfield, CO, 2014. USENIX Association.
  * [35] D. E. Porter, S. Boyd-Wickizer, J. Howell, R. Olinsky, and G. C. Hunt.  Rethinking the library os from the top down.  In Proceedings of the Sixteenth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS XVI, page 291–304, New York, NY, USA, 2011. Association for Computing Machinery.
  * [36] G. Prekas, M. Kogias, and E. Bugnion.  ZygOS: Achieving Low Tail Latency for Microsecond-Scale Networked Tasks.  In Proceedings of the 26th Symposium on Operating Systems Principles, SOSP ’17, Shanghai, China, 2017. Association for Computing Machinery.
  * [37] H. Qin, Q. Li, J. Speiser, P. Kraft, and J. Ousterhout.  Arachne: Core-Aware Thread Management.  In Proceedings of the 13th USENIX Conference on Operating Systems Design and Implementation, OSDI ’18, pages 145–160, Carlsbad, CA, USA, 2018. USENIX Association.
  * [38] S. M. Rumble, D. Ongaro, R. Stutsman, M. Rosenblum, and J. K. Ousterhout.  It’s Time for Low Latency.  In Proceedings of the 13th USENIX Conference on Hot Topics in Operating Systems, HotOS ’13, page 11, Napa, California, 2011\. USENIX Association.
  * [39] D. Schrammel, S. Weiser, S. Steinegger, M. Schwarzl, M. Schwarz, S. Mangard, and D. Gruss.  Donky: Domain Keys – Efficient in-Process Isolation for RISC-V and X86.  In Proceedings of the 29th USENIX Conference on Security Symposium, SEC ’20, USA, 2020. USENIX Association.
  * [40] R. Uhlig, G. Neiger, D. Rodgers, A. L. Santoni, F. C. M. Martins, A. V. Anderson, S. M. Bennett, A. Kagi, F. H. Leung, and L. Smith.  Intel virtualization technology.  Computer, 38(5):48–56, May 2005.
  * [41] A. Vahldiek-Oberwagner, E. Elnikety, N. O. Duarte, M. Sammler, P. Druschel, and D. Garg.  ERIM: Secure, Efficient in-Process Isolation with Protection Keys (MPK).  In Proceedings of the 28th USENIX Conference on Security Symposium, SEC ’19, pages 1221–1238, Santa Clara, CA, USA, 2019. USENIX Association.
  * [42] Xilinx Vivado design suit.  https://www.xilinx.com/products/design-tools/vivado.html.
  * [43] R. von Behren, J. Condit, F. Zhou, G. C. Necula, and E. Brewer.  Capriccio: Scalable Threads for Internet Services.  In Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles, SOSP ’03, pages 268–281, Bolton Landing, NY, USA, 2003. Association for Computing Machinery.
  * [44] Intel 64 architecture x2APIC specification.  https://www.naic.edu/ phil/software/intel/318148.pdf.
  * [45] M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. J. Franklin, S. Shenker, and I. Stoica.  Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing.  In Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation, NSDI’12, page 2, USA, 2012. USENIX Association.
  * [46] F. Zaruba and L. Benini.  The cost of application-class processing: Energy and performance analysis of a linux-ready 1.7-ghz 64-bit risc-v core in 22-nm fdsoi technology.  IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 27(11):2629–2640, Nov 2019.
  * [47] I. Zhang, A. Raybuck, P. Patel, K. Olynyk, J. Nelson, O. S. N. Leija, A. Martinez, J. Liu, A. K. Simpson, S. Jayakar, P. H. Penna, M. Demoulin, P. Choudhury, and A. Badam.  The Demikernel Datapath OS Architecture for Microsecond-Scale Datacenter Systems.  In Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles, SOSP ’21, pages 195–211, Virtual Event, Germany, 2021. Association for Computing Machinery.