# On-the-fly object detection using StyleGAN with CLIP guidance

###### Abstract

We present a fully automated framework for building object detectors on
satellite imagery without requiring any human annotation or intervention. We
achieve this by leveraging the combined power of modern generative models
(e.g., StyleGAN [1]) and recent advances in multi-modal learning (e.g., CLIP
[2]). While deep generative models effectively encode the key semantics
pertinent to a data distribution, this information is not immediately
accessible for downstream tasks, such as object detection. In this work, we
exploit CLIP’s ability to associate image features with text descriptions to
identify neurons in the generator network, which are subsequently used to
build detectors on-the-fly.

Index Terms—  StyleGAN, CLIP, Object Detection

##  1 Introduction

The advances of generative modeling in image understanding and computer vision
motivate their application on remote sensing and satellite imagery [3]. This
application space presents unique challenges, where large amount of unlabeled,
high-resolution data is available while being constantly updated. However,
obtaining annotations and labels often is often prohibitively expensive, not
only due to its demand for domain expertise but also its sensitive nature in
various applications. As a result, the efficacy of modern deep architectures
in handling high-resolution images (e.g., convolution networks [4],
transformers [5]) and in supporting representation learning on unlabeled data
have had a transformative impact in these applications. While the inherent
challenge in curating fine-grained labels for satellite images still persists
[3], the use of pre-trained representations have significantly relaxed the
annotation requirement for different predictive modeling tasks [6]. Despite
these advances, the performance of the resulting models tends to be poor when
the labeled data is extremely sparse or unavailable in practice.

In this paper, we investigate the problem of constructing object detectors
“on-the-fly” – i.e., a completely zero-shot setting where we cannot access
_any_ training annotations. Note, in contrast to existing zero-shot learning
(ZSL) problems that attempt to extend a classifier to include additional un-
sampled classes, our goal is to build a detector for a specific semantic
concept (for e.g. plane), without requiring the users to provide any examples
or explicit specification of the attributes of interest (e.g., shape, color,
size). Designing such a zero-shot approach requires two key ingredients: (i)
an accurate description of all attributes (local/global) in satellite images
using only unlabeled data; (ii) ability to align semantic concepts (text
description) with image characteristics in a joint latent space. While the
former component offers controllability of specific features even in complex
satellite imagery, the latter automatically provides a mapping between image
features and human-understandable semantic descriptors. Thankfully, recent
advances in deep generative models (e.g., StyleGAN [7]) and multimodal
embedding techniques (e.g., CLIP [2]) make this goal feasible.

Our approach begins with the user specifying a semantic concept that needs to
be detected (i.e., localize the region), which we transform using a pre-
trained text encoder from CLIP [2]. Next, we propose a new approach to
automatically identify specific neurons from the pre-trained StyleGAN’s
generator network that are most aligned with the semantic concept. Note, we
are able to measure this alignment by projecting the reconstruction from
StyleGAN into the joint latent space using the image encoder from CLIP.

Remarkably, we found that a naïve detector that we design based on the
activations of the selected StyleGAN’s neurons is able to effectively identify
different variants of an object without any user guidance (different locations
in the image, varying sizes, varying number of occurrences in a scene).
Moreover, we showed that curating a “pseudo”-labeled dataset using our naïve
detector and training an explicit object segmentation model from scratch
(e.g., U-Net) leads to improved performance. Using standard benchmarks, we
demonstrate the effectiveness of our zero-shot detector and present
comparisons to an oracle supervised training. In summary, this work sheds
light into how recent large-scale, pre-trained models can be leveraged in lieu
of expensive data annotation.

##  2 Related Work

Several existing works have demonstrated StyleGAN’s abilities to capture
physical attributes (size, shape, color, texture etc.) automatically from
unsupervised data [8, 9], but still require human supervision to explicitly
label the discovered attributes. Instead of discovering attributes in the
latent space, one can also define concepts by characterizing individual
neurons’ functionality. To this end, Bau et al. proposed a neuron
interpretation method that utilizes an image dataset with rich segment-level
labels [10, 11]. In the proposed work, we also adopt a similar neuron-level
semantic modeling approach. However, instead of relying on the human
annotation of objects in the image as in [10, 12], we leverage the recently
CLIP multimodal embeddings [13, 14]. Discovering semantically aligned neurons
from a StyleGAN will naturally enable us to produce an arbitrarily large
“pseudo”-labeled dataset. Existing works have also adopted a similar approach,
but require at least a few annotated examples. For example, the recently
proposed datasetGAN [15] uses a few human annotations of GAN-generated images
to construct an “annotated” GAN data factory, which is capable of producing a
large number of examples with fine-grained labels.

Fig. 1: The proposed algorithm for quantifying a neuron’s capability as a
specific object detector. Given a StyleGAN pretrained on the target domain and
a CLIP model with image and text encoders, we first gather topk images for
each neuron in all convolutional layers. Then, we mask out regions with low
activation values so that only features the neuron focuses on are retained. We
feed these masked topk images to CLIP’s image encoder and a set of prompts
containing objects people are interested in to CLIP’s text encoder to get
image embeddings $I$ and text embeddings $T$, respectively. Finally, we
compute the cosine similarity between $I$ and $T$ to get a similarity matrix
$M$. The object a neuron detects is contained in the prompt that has the
maximum average similarity with the neuron’s topk masked images.

##  3 CLIP-Guided Object-Centric Neuron Discovery in StyleGAN

One of the key ingredients that facilitates the proposed approach is the
expressiveness of generative models (e.g., StyleGAN) and the semantic control
they exhibit [10, 16, 17, 18]. However, existing unsupervised methods for
discovering task-relevant attributes from generative models often produce a
large number of unlabeled suggestions, which require human supervision to be
leveraged for any downstream task. In this work, we overcome some of these
shortcomings by utilizing the multimodal CLIP model [2] to provide guidance.
The inherent capability of CLIP to embed both image and language descriptors
into a joint feature space enables us to ‘annotate’ each of the StyleGAN
neurons (in the generator) by the semantic concepts that they align with. We
achieve this in two stages: (i) for each neuron, gather quantile statistics of
its activation value and save top activated images and (ii) leverage CLIP’s
joint embedding space to select neurons that are maximally aligned to a
semantic concept of interest.

As illustrated in Fig.1, for a given neuron, we first identify the top
$\mathrm{K}$ activation images based on maximum activation intensity,
$I_{k}=\mathcal{G}(z_{k})$, where $0<k\leq\mathrm{K}$, from a set of randomly
generated images from the GAN $\mathcal{G}$ pretrained on a satellite image
dataset. We then isolate and focus on the high-activation regions using a
simple mask, $M_{k}$, following which the masked image
$I_{k}^{\prime}=M_{k}\circ I_{k}$ is embedded into the the CLIP embedding
space along with $\mathrm{N}$ candidate prompts (for e.g., for airplanes we
use _“an aerial photograph of airplanes”_ , i.e., $N=1$).

These prompts describe the objects we aim to identify, denoted by $T_{n}$. The
most related concept of the given neuron can then assigned by examining which
prompt achieves the largest average cosine similarity between all $\mathrm{K}$
image and text embeddings in the CLIP latent space:
$\sum_{k=1}^{\mathrm{K}}(Sim(\hat{I}_{k},\hat{T_{n}}))/\mathrm{K}$, where
$\hat{I}_{k}=\mathrm{CLIP}_{image}(I_{k}^{\prime})$,
$\hat{T}_{n}=\mathrm{CLIP}_{text}(T_{n})$, are the corresponding image and
text embedding in CLIP model, and $Sim$ is the cosine similarity between the
embedding vectors.

Given a prompt, we can identify the most relevant neuron at each convolutional
layer by ranking the average cosine similarities. In practice, we choose the
neuron with the highest cosine similarity from each convolutional layer with a
feature map size of $16\times 16$ and above. This way, we leverage neurons
with multiple scales. We did not consider neurons in earlier convolutional
layers as their feature maps were too coarse. As illustrated in
Fig.LABEL:fig:teaser, using the prompts _“an aerial photograph of airplanes”_
and _“an aerial photograph of a parking lot”_ , our method is able to identify
neurons that can localize aircrafts and parking lots respectively.

Table 1: The performance of trained U-Nets using different datasets measured by F1 score. The numbers in parentheses are recall and precision, respectively. Here the last row represents the Oracle, i.e., real satellite images with detailed human annotation. Image | Annotation | Random | Pretrained
---|---|---|---
GAN generated images | Neurons | 0.31 (0.39 / 0.26) | 0.30 (0.42 / 0.24)
GAN generated images | Neurons + Label Smoothing | 0.35 (0.43 / 0.27) | 0.30 (0.36 / 0.26)
GAN generated images | Neurons + CLIP Consistency | 0.42 (0.49 / 0.36) | 0.26 (0.29 / 0.23)
Satellite images (_Rareplane_ dataset) | Human | 0.78 (0.81 / 0.74) | 0.82 (0.87 / 0.79)

##  4 Object detection with annotated StyleGAN

Once we have identified neurons as relevant object detectors, we can use them
to label images. However, utilizing these neurons directly on real images are
not feasible without a highly accurate GAN inversion model, i.e., during
inference time, for an unseen image, we first invert it into the GAN’s latent
space, and then using the identified the detector neuron (see Section 3) we
can instantaneously identify the object and highlight its location. While
current state-of-the-art GAN inversion models such as [19] work extremely well
for aligned images such as faces, we found through experimentation that they
fail on satellite images, which are unaligned and much more complex. As a
result, in order to distill the knowledge of the detector neuron, we trained a
separate detector by utilizing the StyleGAN generated images along with the
corresponding annotation generated solely by neurons .

For the detector, we used an U-Net [20], a classic architecture consisting of
an encoder to extract image features and a symmetric decoder to enable precise
localization, as our downstream segmentation model. The encoder we used is
ResNet18[4]. We split the original Rareplanes [3] training set into 5233
training images and 582 validation images while keeping the test set of size
2710 intact; the training images and corresponding label annotations are never
used during training our model. Correspondingly, we sampled 5233 random images
from GAN and gather their annotations generated by neurons.

Since these neuron-generated annotations are inherently noisy, we adopted two
methods to improve their quality: (a) Label smoothing: is a widely used method
[21] to tackle the overconfidence of neural networks, that adds a small
penalty to encourage predictions with higher entropy; and (b) Label
consistency with CLIP: we discard images whose masked version based on neuron-
annotation have low cosine similarity ($<0.24$) with prompts in CLIP’s
embedding space and keep sampling until reaching the target training set size.
This cosine similarity threshold ($0.24$) is the mean similarity between a
masked training set and the prompt for airplanes. We emphasize that generating
these datasets and corresponding annotations requires no form of human
supervision and is completely automated. We used dice loss [22] for training
the U-Net model, and models with the best validation loss were selected for
testing.

During testing, we convert both the predicted and ground truth segmentations
to the tightest bounding box and evaluate models’ performance by F1 score,
setting prediction threshold to IoU $>$ $0.5$. We opted in this evaluation
protocol because for satellite images, localization is often sufficient. To
establish a performance reference, we trained a fully supervised model using
5233 training samples from Rareplanes dataset. In all experiments, we trained
the model using Adam optimizer and a learning rate of 0.001 for 20 epochs.

Results: Our experiment results are summarized in table 1. The first column
indicates the source of the training images. The Oracle utilizes the original
real images, whereas the experiments of the proposed method trained on GAN
generate images don’t. The second column shows what supervision is provided
for training. For the Oracle, detailed human annotations are provided. For the
proposed method, we experimented with several variations of neuron annotation
discussed previously: 1) use only neuron annotation in GAN; 2) augment neuron
annotation with label smoothing; 3) augment neuron annotation with label
consistency with CLIP. We can see that the proposed method achieved
outstanding performance under the restrictive problem setting, i.e., utilizing
only GAN-generated images and no human annotations or input. The performance
gap between the Oracle and the proposed method is expected, as the Oracle
represents the most optimal and more importantly supervised setting.

When training U-Net with random initialization, both label smoothing and label
consistency alleviate the noisiness in the training set and help improve
performance, with label consistency by CLIP boosting the F1 score by 0.11.
However, when starting with pre-trained weights, the proposed methods suffer
from a performance drop. We hypothesize that there is a gap between GAN-
generated images and real images, and label consistency by CLIP seems to
further amplify the difference. This problem can be alleviated if we improve
the quality of the GAN by including more training datasets and/or increasing
its resolution. Lastly, we also include examples of failure cases. As we can
see in Fig. 2), the complex and snowy background hides the planes in the left
image. And the model mistaken part of the terminals, which is also white, as
airplanes.

Fig. 2: Here we show failure cases of our trained U-Net. In left example, the
model missed airplanes that are blended in the background. In the right
example, the model mistakes part of terminals for airplanes.

##  5 Discussion and Future Work

In this work, we propose a novel approach for building object detectors i in a
challenging application domain without utilizing any human annotation or
supervision. To the best of our knowledge, this work demonstrates the first
effort to directly annotate neurons with the help of another neural network
model (CLIP). By removing human involvement in the predictive pipeline, we
achieved an on-the-fly zero-shot setup that can accommodate unseen objects in
an unknown dataset. However, as we rely on CLIP for identifying objects of
interest, we also inherit the potential limitation and bias of the CLIP model.
The CLIP model is trained on a large number of natural images that are
potentially different from the top-down satellite images, so not all prior
knowledge is transferable. To mitigate this issue, we adopted a CLIP model
that is fine-tuned on satellite images (the original CLIP model produces
slightly worse but still comparable results). For future directions, we plan
to explore the GAN-based knowledge mining approach and experiment with the
proposed methods on more diverse datasets and other multimodal models. Even
though in this work we only focused on satellite imagery, the proposed method
can be adopted for other applications and dataset and potentially work better
as satellite images present many unique challenges that are not seen
elsewhere. Moreover, we also plan to perform more thorough analysis on failure
cases to understand the performance drop caused by using pretrained weights.

##  6 Acknowledgement

This work performed under the auspices of the U.S. Department of Energy by
Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344.

## References

  * [1] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila,  “Training generative adversarial networks with limited data,”  in Proc. NeurIPS, 2020.
  * [2] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.,  “Learning transferable visual models from natural language supervision,”  in International Conference on Machine Learning. PMLR, 2021, pp. 8748–8763.
  * [3] Jacob Shermeyer, Thomas Hossler, Adam Van Etten, Daniel Hogan, Ryan Lewis, and Daeil Kim,  “Rareplanes dataset,” June 2020.
  * [4] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Identity mappings in deep residual networks,”  in European conference on computer vision. Springer, 2016, pp. 630–645.
  * [5] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  Advances in neural information processing systems, vol. 30, 2017\.
  * [6] Oscar Manas, Alexandre Lacoste, Xavier Giró-i Nieto, David Vazquez, and Pau Rodriguez,  “Seasonal contrast: Unsupervised pre-training from uncurated remote sensing data,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 9414–9423.
  * [7] Tero Karras, Samuli Laine, and Timo Aila,  “A style-based generator architecture for generative adversarial networks,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 4401–4410.
  * [8] Shusen Liu, Rushil Anirudh, Jayaraman J Thiagarajan, and Peer-Timo Bremer,  “Sparsity improves unsupervised attribute discovery in stylegan,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 3388–3392.
  * [9] Ali Jahanian, Lucy Chai, and Phillip Isola,  “On the ”steerability” of generative adversarial networks,”  arXiv preprint arXiv:1907.07171, 2019.
  * [10] David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba,  “Understanding the role of individual units in a deep neural network,”  Proceedings of the National Academy of Sciences, vol. 117, no. 48, pp. 30071–30078, 2020.
  * [11] David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman, and Antonio Torralba,  “Visualizing and understanding generative adversarial networks,”  in International Conference on Learning Representations, 2019.
  * [12] Yuzhe Lu and Adam Perer,  “An interactive interpretability system for breast cancer screening with deep learning,”  arXiv preprint arXiv:2210.08979, 2022.
  * [13] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski,  “Styleclip: Text-driven manipulation of stylegan imagery,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 2085–2094.
  * [14] Gwanghyun Kim and Jong Chul Ye,  “Diffusionclip: Text-guided image manipulation using diffusion models,”  2021\.
  * [15] Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler,  “Datasetgan: Efficient labeled data factory with minimal human effort,”  in CVPR, 2021.
  * [16] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris,  “GANspace: Discovering interpretable GAN controls,”  arXiv preprint arXiv:2004.02546, 2020.
  * [17] Yujun Shen and Bolei Zhou,  “Closed-form factorization of latent semantics in GANs,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 1532–1540.
  * [18] Andrey Voynov and Artem Babenko,  “Unsupervised discovery of interpretable directions in the GAN latent space,”  in International Conference on Machine Learning. PMLR, 2020, pp. 9786–9796.
  * [19] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or,  “Encoding in style: a stylegan encoder for image-to-image translation,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 2287–2296.
  * [20] Olaf Ronneberger, Philipp Fischer, and Thomas Brox,  “U-net: Convolutional networks for biomedical image segmentation,”  in International Conference on Medical image computing and computer-assisted intervention. Springer, 2015, pp. 234–241.
  * [21] Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton,  “Regularizing neural networks by penalizing confident output distributions,”  arXiv preprint arXiv:1701.06548, 2017.
  * [22] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi,  “V-net: Fully convolutional neural networks for volumetric medical image segmentation,”  in 2016 fourth international conference on 3D vision (3DV). IEEE, 2016, pp. 565–571.


# Trusting the Cloud-Native Edge: Remotely Attested Kubernetes Workers
††thanks: This work has been partially funded by the European Union (NATWORK
project, Grant Agreement 101139285, https://natwork-project.eu/).

Jordi Thijsman , Merlijn Sebrechts , Filip De Turck and Bruno Volckaert
0009-0007-2333-7051 0000-0002-4093-7338 0000-0003-4824-1199
0000-0003-0575-5894 IDLab, Department of Information Technology
Ghent University - imec, Ghent, Belgium
<EMAIL_ADDRESS>
###### Abstract

A Kubernetes cluster typically consists of trusted nodes, running within the
confines of a physically secure datacenter. With recent advances in edge
orchestration, this is no longer the case. This poses a new challenge: how can
we trust a device that an attacker has physical access to? This paper presents
an architecture and open-source implementation that securely enrolls edge
devices as trusted Kubernetes worker nodes. By providing boot attestation
rooted in a hardware Trusted Platform Module, a strong base of trust is
provided. A new custom controller directs a modified version of Keylime to
cross the cloud-edge gap and securely deliver unique cluster credentials
required to enroll an edge worker. The controller dynamically grants and
revokes these credentials based on attestation events, preventing a possibly
compromised node from accessing sensitive cluster resources. We provide both a
qualitative and a quantitative evaluation of the architecture. The qualitative
scenarios prove its ability to attest and enroll an edge device with role-
based access control (RBAC) permissions that dynamically adjust to attestation
events. The quantitative evaluation reflects an average of 10.28 seconds delay
incurred on the startup time of the edge node due to attestation for a total
average enrollment time of 20.91 seconds. The presented architecture thus
provides a strong base of trust, securing a physically exposed edge device and
paving the way for a robust and resilient edge computing ecosystem.

###### Index Terms:

remote attestation, TPM, measured boot, secure boot, edge, node enrollment,
k8s

††publicationid: pubid: Pre-print of article accepted to IEEE ICCCN 2024 ©2024
IEEE

##  I Introduction

As computing environments evolve, so do the challenges associated with
ensuring their security. Typically, Kubernetes [1] nodes reside within secure
data centers, shielded from physical tampering and unauthorized access.
However, recent advancements in edge container orchestration, such as Fledge
[2] and KubeEdge [3] , have ushered in a new era where clusters can extend
beyond the confines of datacenters to embrace even the smallest devices. These
devices are often deployed in remote and physically exposed locations making
them exceptionally vulnerable to compromise.

While existing research has explored various aspects of Kubernetes security,
such as protecting workloads from each other [4] or safeguarding them against
privileged user threats [5], our focus is on protecting the entire cluster
from physically compromised nodes. By providing hardware-based boot
attestation of the edge device, a trusted base can be provided that other
features such as runtime attestation or kernel security features can rely on.

Keylime [6] provides an abstraction on top of TPM boot and runtime
attestation, allowing for easier secure bootstrapping of managed hardware
attested devices. Efforts are underway to make it easily deployable to cloud-
native clusters such as Kubernetes and OpenShift. But integration of the
Kubernetes API into its various parts is still lacking, making it impractical
to automate attestation management. More importantly in this context, it lacks
the ability to manage cluster permissions granted to an edge node in relation
to attestation events. A critical aspect of our approach involves integrating
flexible role-based access control (RBAC) mechanisms, to comply with
Kubernetes security best practices [7]. Specifically adapted to the context of
edge worker nodes, these RBAC policies enable swift and granular adjustments
to access permissions in response to attestation events. These responses can
prevent untrusted devices from accessing sensitive information or quarantine
it to prevent scheduling of sensitive workloads. By swiftly limiting an edge
node’s access to cluster resources upon compromise, our architecture aims to
bolster the overall security posture of distibuted Kubernetes cloud-edge
clusters.

This paper presents an exploration of our proposed open-
source111https://github.com/idlab-discover/trust-edge [8] platform, addressing
key research questions:

RQ1. How can a toolchain automate enrollment of an edge device as a trusted
Kubernetes worker node, ensuring full boot attestation verified by the
cluster?

RQ2. How can such a platform integrate into the authentication and
authorization framework of the orchestrator itself, in order to ensure
granular permission adjustments in response to attestation events?

This paper is structured as follows: section II provides essential context on
TPM attestation, measured boot, and the Keylime software and outlines other
work related to these topics. Section III delves into the components
constituting the proposed architecture. Section IV details the full enrollment
and trust management process, explaining the functionalities and interactions
of various components. Section V presents the evaluation results and addresses
RQ1 and RQ2. Finally, section VI explores potential avenues for future
research.

##  II Background and related work

###  II-A TPMs and the boot process

A Trusted Platform Module is a secure cryptographic coprocessor, one of its
most important functionalities is its ability to measure digests into so-
called platform configuration registers (PCRs) [9]. The core idea of these
PCRs is built around two basic operations: extend and reset. The extend
operation performs an XOR operation with the current PCR state and the new
digest as operands and saves the hash of the result as the new PCR value.

Proving that a device is running trusted software is a very complex process,
stretching all the way from the lowest levels of the boot sequence to the
kernel, the operating system and the applications it runs. These components
form a chain, relying on a lower level to tie the next to a trusted state by
measuring the next binary into a TPM platform configuration register in a
process called a measured boot. The lowest level of the chain is the so called
core root of trust for measurement (CRTM) [10] which anchors the entire chain
to a piece of immutable CPU code. If a system lacks a proper measurement for
any part of this chain, it could compromise the system’s ability to protect
workloads as many container isolation mechanisms rely on security features
provided by the Linux kernel (such as cgroups and namespaces).

Figure 1: A measured boot extends a digest of each part of the boot chain into
the TPM, this is in contrast with a secure boot which only verifies a part’s
signature.

It is important to recognize the distinction between such a measured boot and
a secure boot. Fig. 1 provides a schematic comparison between the two. Where
measured boot stores digests of each part of the boot chain in the TPM, a
secure boot only validates the parts’ signatures against a known good public
key. If one of the system’s components does not contain an approved signature,
the system will simply interrupt the boot and shut down. A device using secure
boot thus makes its own trust decision, compared to a measured boot, which
just securely documents the boot process in PCRs and allows an external
verifier to judge the TPM measurements.

TPMs are often used in edge research to provide hardware-based trust. It is,
however, essential to acknowledge a critical oversight regularly present in
such implementations. These designs are often implemented on devices lacking a
CRTM (e.g., Raspberry Pi). Without a CRTM, the chain of trust, crucial for
ensuring the integrity of the system, lacks a solid anchor [11]. Consequently,
reliance on a potentially compromised kernel to transmit measurements to the
TPM can render the entire attestation process unreliable. Simply plugging a
TPM into an edge device and trusting it is not enough. This oversight
highlights the importance of ensuring that the foundational elements, such as
a complete boot attestation anchored in the CRTM, are present.

This lack of manufacturer provided CRTM renders many edge devices useless in
an environment where trust is required. Recent research into software TPMs
could, however, provide a future for such devices. In contrast to hardware-
based TPMs, software TPMs leverage Trusted Execution Environments (TEEs) to
establish trust without necessitating a built-in CRTM. There are some research
projects developing such TPMs [12] [13] for various platforms such as ARM
TrustZone and Intel SGX, however no implementation has been widely adopted.

A final issue with TPMs is their complexity. Interacting with a TPM requires a
very good understanding of the complex architecture [14] and is generally done
by transmitting low level binary commands [15] to the device. In recent years,
the Trusted Computing Group (TCG) [16] has put a lot of effort into developing
and standardizing higher level APIs [17] to interact with the TPM resulting in
a toolset called tpm2-tools. These tools allow for higher-level interactions
with the TPM using the CLI. While the low-level code complexity has been
reduced significantly, a deep knowledge of the inner workings and mechanisms
of a TPM is still required to actively use it in an application. Research
projects like Keylime could help alleviate this complexity issue.

###  II-B Keylime

Keylime is an open-source CNCF project originally developed by MIT’s Lincoln
Laboratory [18]. It has since seen increased adoption with the support of
RedHat, who are actively developing it for RHEL and OpenShift. Keylime
provides another abstraction layer to TPM attestation on top of the existing
tpm2-tools and allows developers to easily integrate boot and runtime
attestation into their architectures. Keylime consists of cloud components
written in Python (verifier, registrar, and tenant) and a Rust agent running
on the machine that is to be attested.

Figure 2: An overview of the Keylime architecture: a system administrator uses
the Keylime tenant CLI to set up attestation of a device that runs an agent
and deliver a payload.  Figure 3: An overview of our architecture: the
attestation controller interacts with the tenant API to enroll a trusted edge
device into the cluster, it monitors the EdgeNode custom resource for
attestation events and adjust RBAC permissions accordingly.

The Keylime architecture is controlled by the tenant CLI application, which
allows a system administrator to enroll a device and set it up for TPM-based
boot and runtime attestation using specific golden values for each. These
values contain known good states of the system and serve as a reference for
future attestation. The tenant interacts with the other cloud components to
deliver a secure payload to an agent. This agent is a device’s connection
point to the Keylime cloud components. It provides an API over HTTPS, which
allows for an abstracted interaction with the device’s TPM.

Efforts are underway to develop a cloud operator [19] that can manage TPM-
enabled nodes in a Kubernetes compatible environment. This operator is
designed to simplify the deployment of Keylime in cloud Kubernetes
environments to provide node attestation. It is currently still in the early
development phase. While it provides useful functionality, mainly the ability
to easily deploy the complex network of Keylime cloud components using Helm,
it is directed towards cloud environments and does not address the specific
needs of the cloud-edge use-case outlined by this paper. Mainly the
interaction with a device external to the cluster, the unique RBAC identity
tied to such a device, and its dynamic RBAC permissions. For our platform we
make use of an early fork of this project to deploy our customized tenant,
verifier and registrar using Helm.

Berbecaru et al. [20] implemented a prototype that uses Keylime to provide
attestation on a Raspberry Pi edge device and proposed some solutions to the
lack of CRTM on such a device. It does however not integrate with any
orchestration systems such as Kubernetes nor does it provide any automated
management of the device.

##  III Architecture

The next paragraphs outline the various components that compose the
architecture proposed in this paper. An overview can be found in Fig. 3.

The EdgeNode custom resource is the center of this architecture, as it
represents the edge device in the Kubernetes API. The custom resource contains
all the information required to identify and attest the device, which is
provided by a system administrator. The EdgeNode’s status is the most
important field, as it directs the controller’s actions. An EdgeNode resource
is initially deployed with an ‘unregistered’ status and will move through the
‘registered’ state towards either an ‘attested’ or an ‘unattested’ state
throughout its lifetime.

Kubebuilder [21] serves as a scaffold for the development of the EdgeNode
controller and provides easy deployment to the cluster. The controller
monitors the state of EdgeNode resources and is responsible for enrolling an
edge node and steering its corresponding custom resource to an ‘attested
state’. It interacts with the Kubernetes API to manage unique RBAC roles for
each edge device and provides it with the right permissions to join the
cluster. The controller interfaces with the tenant through its API to deliver
these credentials to attested nodes and sets up continuous attestation.
Throughout the node’s lifecycle, it is responsible for modifying the unique
permissions in reaction to attestation events.

The tenant application is built on top of the Keylime tenant. It provides an
API layer that integrates into the source of Keylime to solve shortcomings in
the original design. This original design is CLI based and is meant to be
called by a system administrator to enroll an attested device. The command
takes a description of the node’s desired state and the payload that is to be
delivered as an argument. It reads these files from disk and executes the
necessary operations to securely deliver the payload to an attested device.
Keylime has already attempted to bypass this reliance on an interactive system
administrator by offering a script that uses kubectl commands to automatically
copy the necessary files to the tenant pod and execute the appropriate tenant
command. This reliance on an external script however, introduces a few
additional issues that can be improved upon. First, it requires multiple disk
read/write operations to achieve, likely increasing latency. The controller
must write the payload it generates in memory to disk, kubectl then reads it
again and writes it to the tenant’s disk after which the tenant can finally
read it again to process it. Calling an external bash script presents a second
issue, introducing extra overhead both in code complexity and code efficiency.
Our API allows the tenant’s functionality to be called natively by the
controller using a REST interface instead of outsourcing the interactions to
an external script. The API’s business logic extends into the tenant’s source
code to carry the in-memory payload and reference boot state, generated by the
controller, into the tenant to avoid expensive disk read/write operations.

Our verifier extends the Keylime verifier by integrating its attestation
responses into the cluster. The verifier is responsible for providing
continuous attestation managed by the tenant. It polls the agent for signed
TPM quotes and validates them against the reference. When an attestation
fails, Keylime has built in functionality to notify other components using
either ZeroMQ or a webhook. To provide a better integration of the verifier
into the cluster and avoid having to add extra components to the architecture,
we add a third option to our verifier: a native Kubernetes revocation. By
integrating the Kubernetes Python client, our verifier can patch the EdgeNode
custom resource’s status to reflect its unattested state. This will notify the
controller, allowing it to take corrective actions.

The registrar plays a crucial role in storing and managing essential data for
agent operation, including identity, attestation information, and metadata.
Acting as the initial point of contact for edge devices, the registrar
facilitates the verification of agent identity using its Endorsement Key (EK)
and mTLS certificate. To automate the response to this initial contact, we
extended the Keylime registrar to include Kubernetes API support. After
validating the identity, it can notify the controller of the edge device’s
presence by patching the EdgeNode custom resource with the ‘registered’ state,
kickstarting the enrollment process.

##  IV Enrollment workflow

This section covers the complete enrollment of an attested Kubernetes edge
node with unique RBAC credentials. Fig. 3 shows an overview of the complete
flow with numbers indicating the sequence of operations.

###  IV-A Registration of edge device

First, the system administrator uses a Helm chart [22] to create an instance
of the EdgeNode custom resource (0), loaded with the reference state and TPM
identification information. The reference state consists of a JSON containing
information on the allowed state of various boot components such as the CRTM,
firmware, kernel, and keys. To provide identification for the TPM, an
endorsement key (EK) certificate signed by the TPM’s manufacturer must be
included in the custom resource. An administrator can use the tpm2-tools to
extract the EK certificate from the TPM’s non-volatile memory. It is important
to note that the system administrator is considered to be a fully trusted
entity that determines the definition of trust for the edge device.

When an edge device is initially turned on, it will attempt to contact the
registrar (1). The registrar will subject it to an identity verification (2)
by making the device encrypt a nonce with the private EK stored in its TPM.
This is validated using the matching public key contained in the EK
certificate provided by the device. After successfully passing this check, the
registrar adds the node’s information to its records. As described in section
III, we integrated Kubernetes API support into the registrar which allows it
to set the EdgeNode status to ‘registered’ (3), notifying the controller.

###  IV-B Generation of RBAC credentials

After device registration, the controller has to execute a series of tasks to
steer the EdgeNode resource to an ‘attested’ state and deploy the device as an
edge worker node. First, the controller must verify that the entity previously
registered with the registrar matches the one provided by the system
administrator using the EdgeNode custom resource (4). Simply comparing the two
certificates for a match is enough since the registrar has already performed a
verification of the certificate provided by the edge device.

Next, the controller needs to provide the edge device with a unique identity
and the necessary permissions to enroll as an edge worker node using an edge
orchestration framework like Fledge. It generates a new key pair and uses it
to request a signed certificate from the Kubernetes cluster (5), linking the
edge device to a unique Kubernetes user. This generated user does not yet have
permissions to access any of the cluster resources. Next the controller
creates a unique Kubernetes role for the edge device with the required RBAC
permissions and links it to the user certificate using a Kubernetes
rolebinding (6).

###  IV-C Enrollment of edge node

The payload required to enroll the edge device in the cluster is now complete.
The controller packs it into a ZIP archive and together with the reference
boot state from the EdgeNode custom resource, sends it to the tenant’s REST
endpoint (7). The tenant encrypts the payload with a composite U-key (which is
a bitwise xor of two other keys called the K- and V-keys), and starts the
delivery process. First, it contacts the registrar (8) to obtain the edge
device’s information, which includes its IP and port as well as the stored EK
certificate. The tenant contacts the agent (9) running on the edge device and
once again verifies that the device’s TPM does in fact own the EK certificate.
Once the identity is established, the tenant sends the first part of the
composite encryption key (U-key) as well as the encrypted payload to the edge
device.

Next, the tenant forwards the second part of the payload encryption key
(K-key) as well as the reference state to the verifier to set up continuous
attestation (10). The verifier requests an integrity check from the agent and
validates its PCR values and boot record. After successful attestation, the
verifier delivers the key to the agent (11), allowing it to decrypt and
execute the payload that starts the Fledge service and enrolls the device as
an edge worker node (12). Finally, the verifier sets the status of the
EdgeNode custom resource to ‘attested’ to notify the controller of the
attestation event (13).

The verifier will periodically check the integrity of the edge node, should a
failure occur at any point it will change the EdgeNode custom resource’s
status to ‘unattested’, allowing the controller to revoke RBAC permissions for
the edge device’s unique identity.

##  V Evaluation results

We evaluated the design through three scenarios to assess its effectiveness in
ensuring the integrity and security of the Kubernetes edge workers. A
quantitative analysis was also performed to measure the impact of attestation
on the startup time of a Kubernetes edge node. This evaluation was performed
using a Dell Latitude 5540 as an edge device with 16GB RAM and an Intel Core
i7. The Kubernetes cluster is provisioned using CloudNativeLab [23] with 1
worker node with 2 CPU cores and 8 GB RAM. The raw data is published on GitLab
[24].

  1. 1.

In the first scenario, an edge device successfully passes initial attestation
and receives its payload, including credentials linked to a unique RBAC role,
which it uses to start Fledge and enroll as an edge node. This demonstrates
the system’s capability to verify the integrity of an edge device and provide
them with the necessary granular credentials. The Kubernetes event log
indicates a successful attestation process, with attestation being achieved 11
seconds after first contact by the edge device.

Figure 4: Kubernetes event log shows successfull detection of node attestation
failure during continous monitoring.

  2. 2.

The second scenario involved intentionally causing a node attestation failure
to evaluate the system’s response. We booted a node with an unknown kernel to
simulate an attacker starting a custom, possibly compromised kernel. The
Kubernetes event log captures the detection of an unattested node, indicating
that the system appropriately identifies integrity failures. The verifier
notified the controller of this failure within 7 seconds. No payload was
delivered to the edge device and its RBAC permissions were taken away, denying
it the ability to enroll as a worker node.

  3. 3.

In the third scenario, the system’s continuous monitoring capability was
assessed by inducing a node attestation failure after the system had already
been successfully attested. This simulates an attacker rebooting an attested
system that already serves as a worker node and loading a compromised kernel.
Fig. 4 illustrates the Kubernetes event log detecting a node attestation
failure during continuous monitoring. In response to this failure, the
controller swiftly retracted the RBAC permissions linked to this worker’s
unique identity, preventing it from accessing cluster resources. This scenario
validates the system’s effectiveness in maintaining security posture over time
by promptly identifying attestation failure and revoking cluster access for
compromised nodes.

Finally, we present a quantitative evaluation (Fig. 5). We executed the first
scenario 50 times and measured the time between the agent service and the
Fledge service starting as well as the time Fledge needs to register as an
edge worker. The results indicate an increase in startup time due to
attestation of 10.28 seconds (std 0.30s). This is in addition to the startup
time of Fledge, which measured on average, 10.52 seconds (std 0.20s), for a
total average startup time of an attested edge node of 20.91 seconds (std
0.36s).

The evaluation results of the proposed architecture effectively address the
research questions. Scenario one demonstrates the architecture’s capability in
securely enrolling edge nodes into a Kubernetes cluster (RQ1) by verifying
device integrity and delivering necessary credentials. Furthermore, scenario
two and three show that the system provides granular cluster permissions in
response to attestation events (RQ2), dynamically adjusting RBAC permissions
based on node status. Thus, the evaluation validates the system’s ability to
achieve secure enrollment and provide tailored permissions, aligning with the
research objectives.

Figure 5: Attestation takes on average 10.28 seconds for a total node
registration time of 20.91 seconds.

##  VI Future work

A major challenge remains the binary nature of attestation. In traditional
schemes, a node either passes or fails attestation. When using boot
attestation, this means that a node running a newer kernel that was not
included in the reference will automatically be considered insecure. This
binary decision is at odds with the flexible and granular nature of Kubernetes
access control and scheduling. Instead of deciding on a pass/fail basis if a
certain combination of CRTM, firmware, and kernel should be trusted, a more
nuanced description of trust must be employed. An older kernel version is not
necessarily insecure, but to comply with security SLAs one could limit such a
node’s access to sensitive resources and prevent the scheduling of certain
workloads to such nodes while still allowing it to execute non-sensitive ones.

A final problem that is outside the scope of this paper but requires future
attention is network access control. This paper focuses on setting up a unique
RBAC identity for an attested edge node that allows flexible adjustments to
Kubernetes resource access. It does however, not cover that node’s access to
the different networks. The node currently has full access to the local
network through its VPN connection, and thus any resources exposed on any of
the cloud nodes. This network access should also be managed by the controller,
initially limiting the connection to only the registrar and expanding network
access as the attestation state evolves.

##  VII Conclusion

In conclusion, this research paper proposes a novel architecture and open-
source implementation for secure and automated enrollment of Kubernetes edge
nodes after remote attestation, ensuring the integrity of the entire cluster
even in physically exposed environments. By leveraging hardware-based boot
attestation rooted in Trusted Platform Modules (TPMs), the architecture
establishes a strong base of trust for edge devices. The integration of role-
based access control (RBAC) mechanisms enables granular adjustments to access
permissions in response to attestation events, further protecting sensitive
cluster resources from untrusted edge nodes.

Through qualitative scenarios and a quantitative evaluation, the effectiveness
of the proposed architecture has been demonstrated. The system successfully
enrolls edge nodes into the Kubernetes cluster, verifies their integrity, and
dynamically adjusts RBAC permissions based on attestation events. Attestation
adds on average 10.28 seconds to the enrollment time, for a total average of
20.91 seconds. This addresses the research questions regarding automated
enrollment and integration with authentication and authorization frameworks,
showcasing the architecture’s capability in enhancing security within cloud-
edge computing ecosystems.

We identify future work in addressing the binary nature of attestation
decisions and implementing network access control to further bolster the
security of edge deployments. Overall, this research paves the way for
resilient and secure edge computing environments, crucial for the
proliferation of distributed and decentralized computing infrastructures.

## References

  * [1] “Kubernetes: open-source system for automating deployment, scaling, and management of containerized applications.” [Online]. Available: https://kubernetes.io/
  * [2] T. Goethals, F. De Turck, and B. Volckaert, “Extending Kubernetes Clusters to Low-Resource Edge Devices Using Virtual Kubelets,” _IEEE Transactions on Cloud Computing_ , vol. 10, no. 4, pp. 2623–2636, 10 2022.
  * [3] “KubeEdge: a Kubernetes Native Edge Computing Framework.” [Online]. Available: https://kubeedge.io/
  * [4] A. Goel and B. Thangaraju, “Authenticating Distributed Systems Using SPIRE over Kubernetes Cluster,” in _2022 IEEE International Conference on Electronics, Computing and Communication Technologies (CONECCT)_. IEEE, 7 2022, pp. 1–6.
  * [5] G. P. Fernandez and A. Brito, “Secure container orchestration in the cloud,” in _Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing_. New York, NY, USA: ACM, 4 2019, pp. 138–145.
  * [6] “Keylime: Bootstrap & Maintain Trust on the Edge / Cloud and IoT.” [Online]. Available: https://keylime.dev/
  * [7] M. S. Islam Shamim, F. Ahamed Bhuiyan, and A. Rahman, “XI Commandments of Kubernetes Security: A Systematization of Knowledge Related to Kubernetes Security Practices,” in _2020 IEEE Secure Development (SecDev)_. IEEE, 9 2020, pp. 58–64.
  * [8] “Trust Edge: Trusted edge nodes for Kubernetes.” [Online]. Available: https://github.com/idlab-discover/trust-edge
  * [9] S. Kinney, “6 - Platform Configuration Registers,” in _Trusted Platform Module Basics_ , ser. Embedded Technology, S. Kinney, Ed. Burlington: Newnes, 2006, pp. 53–64. [Online]. Available: https://www.sciencedirect.com/science/article/pii/B9780750679602500075
  * [10] Cooper David, Polk William, Regenscheid Andrew, and Souppaya Muragiah, “Special Publication 800- 147: BIOS Protection Guidelines,” NIST, Gaithersburg, Tech. Rep., 4 2011.
  * [11] W. Arthur, D. Challener, and K. Goldman, “A Practical Guide to TPM 2.0.” Apress, 2015, p. 152.
  * [12] H. Raj, S. Saroiu, A. Wolman, R. Aigner, J. Cox, P. England, C. Fenner, K. Kinshumann, J. Loeser, D. Mattoon, M. Nystrom, D. Robinson, R. Spiger, S. Thom, and D. Wooten, “fTPM: A Software-Only Implementation of a TPM Chip,” in _25th USENIX Security Symposium (USENIX Security 16)_. Austin, TX: USENIX Association, 8 2016, pp. 841–856. [Online]. Available: https://www.usenix.org/conference/usenixsecurity16/technical-sessions/presentation/raj
  * [13] H. Sun, R. He, Y. Zhang, R. Wang, W. H. Ip, and K. L. Yung, “eTPM: A Trusted Cloud Platform Enclave TPM Scheme Based on Intel SGX Technology,” _Sensors_ , vol. 18, no. 11, 2018. [Online]. Available: https://www.mdpi.com/1424-8220/18/11/3807
  * [14] “Trusted Platform Module Library Part 1: Architecture,” Trusted Computing Group, Tech. Rep., 11 2019.
  * [15] “Trusted Platform Module Library Part 3: Commands,” Trusted Computing Group, Tech. Rep., 11 2019.
  * [16] Trusted Computing Group, “Trusted Computing Group,” https://trustedcomputinggroup.org/.
  * [17] “tpm2-software: Developer community for those implementing APIs and infrastructure from the TCG TSS2 specifications.” [Online]. Available: https://github.com/tpm2-software
  * [18] N. Schear, P. T. Cable, T. M. Moyer, B. Richard, and R. Rudd, “Bootstrapping and maintaining trust in the cloud,” in _Proceedings of the 32nd Annual Conference on Computer Security Applications_. New York, NY, USA: ACM, 12 2016, pp. 65–77.
  * [19] “keylime/attestation-operator: Keylime easily deployable on Kubernetes/Openshift.” [Online]. Available: https://github.com/keylime/attestation-operator
  * [20] D. G. Berbecaru and S. Sisinni, “Counteracting software integrity attacks on IoT devices with remote attestation: a prototype,” in _2022 26th International Conference on System Theory, Control and Computing (ICSTCC)_. IEEE, 10 2022, pp. 380–385.
  * [21] “kubebuilder.” [Online]. Available: https://book.kubebuilder.io/
  * [22] “Helm: The package manager for Kubernetes.” [Online]. Available: https://helm.sh/
  * [23] “CloudNativeLab: Kubernetes testbed.” [Online]. Available: https://practicum.cloudnativelab.ilabt.imec.be/
  * [24] “Trust Benchmark: Benchmark results for trust edge architecture.” [Online]. Available: https://gitlab.ilabt.imec.be/edge-keylime/trust-benchmark