# KOSMOS: Knowledge-graph Oriented Social media and Mainstream media Overview
System

Chua Hao Yang, Yong Shan Jie

Nanyang Technological University
Singapore
{chua0808,yo0001ie}
@e.ntu.edu.sg
Boon Kok Chin, Lander Chin,
Lynnette Hui Xian Ng
Defence Science Technology Agency
Singapore
{boonkc,lanchin,nhuixia1}
@dsta.gov.sg

###### Abstract

We introduce KOSMOS, a knowledge retrieval system based on the constructed
knowledge graph of Social media and Mainstream media documents. The system
first identifies key events from the documents at each time frame through
clustering, extracting a document to represent each cluster, then describing
the document in terms of 5W1H (Who, What, When, Where, Why, How). The event-
centric knowledge graph is enhanced by relation triplets and entity
disambiguation from the representative document. Knowledge retrieval is
supported by a web interface that presents a graph visualisation of related
nodes and relevant articles based on a user query. The interface facilitates
understanding relationships between events reported in mainstream and social
media journalism through the KOSMOS information extraction pipeline, which is
valuable to public office to understand media slant and public opinions.
Finally, we explore a use case in extracting events and relations from
documents to understand the media and community’s view to the 2020 COVID19
pandemic.

##  1 Introduction

Understanding relationships between events reported in mainstream news
articles and social media chatter is a key goal of information extraction,
aiding policy makers to sense make the vast information space and understand
citizen chatter on specific issues.

We present KOSMOS, a web interface to understand online information through an
event-centric knowledge graph. We draw our data from mainstream news articles
and social media articles, henceforth collectively referring to the data as
documents. This paper presents the information processing pipeline through
identifying salient events (Section 3.1), relation extraction and knowledge
graph construction (Section 3.2), and knowledge retrieval (Section 3.3).
Finally, we explore a use case in extracting events and relations from
documents to understand the media and community’s view to the 2020 COVID19
pandemic (Section 4).

##  2 Related Works

KOSMOS is made up of several key components in its pipeline; Data Scraping,
Events Identification, Knowledge Graph, and Knowledge Retrieval. Some related
work would include NIFTY [Suen et al., 2013], which performs clustering of
topics from various sources across the internet by partitioning phrases from
document, then presenting each cluster as a Directed Acyclic Graph. VisIRR,
[Choo et al., 2014] seeks to ease information retrieval and recommendation
through interactive scatter plots.

Besides identifying key themes in the documents, the next goal is to turn
unstructured information from documents into a structured form of relation
tuples to express the relationship between entities and by extension,
documents. Several approaches have been developed to construct knowledge bases
using news data [Rospocher et al., 2016] [Rudnik et al., 2019] and social
media data [Gottschalk and Demidova, 2019], representing events, entities and
relations between the data. Many query methods include BERT for natural
language understanding [Dhingra et al., 2020], followed by knowledge graph
embedding [Huang et al., 2019] to find the relevant graph nodes by breaking
Natural Language Queries (NLQ) into its entities and predicates.

Most existing solutions build knowledge graphs based on static Wikipedia
databases or structured QA data sets such as DBpedia and Wikidata, focusing on
a single type of document structure, and presenting results in one single
presentation view. The aim of KOSMOS is to put together the classification of
multi-source information, as well as to provide an intuitive information
retrieval interface, displaying what is relevant in a graphical and temporal
representation, to enhance sense making through finding entities related, and
how sequence of events unfold.

Front-End and Server Frameworks. To implement KOSMOS, we relied on several
frameworks and libraries to implement and integrate the full pipeline. Start
from the front-end, we used JavaScript React for how it renders changes very
rapidly with its Virtual DOM changing only what has been updated. We coupled
the front end with a web server. For the web server, we chose Python Flask
over other framework such as Django because of its lightweight interface and
modularity, allowing us to build the back end server linking up all other
components through REST API.

Libraries for Data Processing. On the backend, there are a couple more
libraries used to extract crucial information from unstructured text. Firstly,
to perform document clustering, spaCy’s word vector [Honnibal and Montani,
2017] was used to vectorise the document. spaCy’s library was used for this
task because of its faster computational time compared to Stanford NLP’s GloVe
library. However, For relation triples $<$subject-relation-object$>$
extraction and recognition of named entity, we used Stanford NLP’s OpenIE
extractor [Angeli et al., 2015] and Named Entity Recognizer [Finkel et al.,
2005] because spaCy offered only entity extraction without any identifying
relations between extracted entities.

Each document that enters the KOSMOS pipeline needs to have a concise
breakdown of the document at the end of its processing. A candidate for this
task would be Bert Extractive Summarizer [Miller, 2019]. However, text
summarizers, despite it returning a concise summary of the document, is still
largely unstructured. We thus adopted the Giveme5W1H library [Hamborg et al.,
2019], which takes the document and answers the 6 general questions - Who,
What, When, Why, How - about the text.

Database and Data Retrieval Frameworks. Two form of database is required for
KOSMOS - Graph Database and Document Database. Neo4j was the choice for Graph
Database over others because Neo4j allowed for properties to be stored within
nodes, unlike other Graph Database, such as Ontotext’s GraphDB, which
primarily adopts the RDF store database model, which represents each property
of a node as additional node and edge. This perks of Neo4j would mean only
entities are made into nodes, and properties are stored within the created
entities, allowing the output graph to be significantly more condensed. On top
of that, Neo4j also has a Desktop application for us to verify our graph with
great ease.

Lastly, for a Document Database, we need a framework that not only ensures
data integrity, but also able to retrieve documents with ease and rapidly. For
KOSMOS, we used Elasticsearch as a NoSQL database. Elasticsearch is a highly
scalable and fast textual-based search engine on top of being a database. It
has many useful features to enhance our document retrieval, such as multi-
field matching, and fuzzy search.

##  3 System Overview

Figure 1 presents an overview of the pipeline of the KOSMOS system. We
collected articles from mainstream news sources using RSS feeds and Reddit
posts using Pushshift.io [Baumgartner et al., 2020]. The collected data are
stored in an ElasticSearch database, which facilitates document search with an
expandable schema. The knowledge graph stores entity relational data in a
Neo4J database. The pipeline is implemented in Python, while the user
interface for the Knowledge Retrieval module is implemented with Javascript
React and supported by a Python Flask server.

The pipeline first identifies salient events by performing document clustering
to group documents across time periods, then identify a representative
document that represents the theme of the cluster. To further describe the
event, event descriptors of 5W1H (Who, What, When, Where, Why, How) are
extracted on the representative document. The knowledge graph is constructed
in the next step from all the representative documents. This is done by
extracting relations through identifying relation triplets and performing
entity disambiguation before forming nodes and links in the graph database. A
user interface facilitates information retrieval of the relationships between
documents and entities related to the user query.

Figure 1: KOSMOS System Overview

###  3.1 Identifying Events in Documents

At each time period, a large amount of documents enter the pipeline. Many
mainstream articles and Reddit posts may elaborate on the same event, but a
handful of Reddit posts are incoherent sentences. To remove these noise and
identify salient news events, we first perform document clustering, which
results in clusters of documents and further highlights one representative
document per cluster. Each event in a cluster is processed using Hamborg’s
5W1H library to extract a more concise event descriptors.

Document Clustering. The documents are first pre-processed by stop-word and
non-ASCII character removal, deduplication, tokenisation and lemmatisation.
Non-English documents are disregarded to focus on an English-based system.

Each document is vectorised into 300 dimensions by sum average using spaCy’s
word vector [Honnibal and Montani, 2017]. To pick out salient features in the
documents, we reduce the documents to 100 dimensions by Principal Component
Analysis, giving a variance ratio of 0.92 and 0.84 on news and Reddit
documents respectively, intuitively representing the amount of initial
semantics in each document type.

Density-based spatial clustering is then applied on the vector collection with
a cosine distance metric. We perform a parameter search to find the optimal
distance between two documents as the distance resulting in the maximum number
of clusters. To keep the documents in each cluster concise and consistent,
each cluster is represented by one representative document. This
representative document is determined by finding the one closest to the
cluster center.

Extracting 5W1H. For the representative document in each cluster, the system
extracts the main event descriptors in the form of 5W1H from the article using
the library Giveme5W1H [Hamborg et al., 2019]. We find that refining the
search results to retain the top two descriptors based on the probability of
confidence gives a more reasonable and desired result than its default, based
on empirical testing on our articles. The extraction of event descriptors for
each document cluster provides a quick summary of the events at each time
period.

###  3.2 Knowledge Graph Construction.

We enhance the knowledge graph to capture entity relationships from the
representative document extracted in the previous section. This knowledge
graph is constructed using Neo4J, which serves as a base for extracting
information with the aid of a user search module.

Relation extraction. Stanford NLP’s libraries are used to perform relation
extraction on the representative document. Relation triples are extracted from
the article’s text using OpenIE extractor [Angeli et al., 2015]. The triplets
are then filtered using the Named Entity Recognizer [Finkel et al., 2005] to
keep essential entities to a knowledge graph and trimmed to eliminate
duplicated triples, before performing entity disambiguation against the
knowledge graph.

Entity Disambiguation. In insertion of extracted relations into the Neo4J
database, entity disambiguation is required as many news articles have similar
relations and/or entities. In this step, we perform deduplication of triplets
with the existing knowledge base and addition of nodes and corresponding links
to existing nodes. We harness the GeoNames database[Wick and Boutreux, 2019]
to map the geographical information from extracted location nodes to make
geographical sense of these nodes by tying respective cities to their
countries. In addition, we eliminate the creation of separate nodes from the
mention of last names with full names, presented in Figure 2. The resulting
set of triplets are then inserted into the knowledge graph. The edge between
nodes stores the information of their respective document’s data. The
extracted entities are then tokenised and the tokens stored in the
ElasticSearch database to facilitate retrieval of nodes during query of
knowledge retrieval module.

Figure 2: Named Entity Disambiguation Workflow

###  3.3 Knowledge Retrieval

KOSMOS knowledge retrieval module presents a simple to access web browser
based interface built using Javascript React library and Python Flask.
Multiple users can access and perform search queries at the same time. The
application facilitates a user search query, then displays a graph of related
entities to the query retrieved from the Neo4J database, uses a timeline to
display the documents.

Deconstructing the User Query To facilitate the search through the knowledge
base, we built a Natural Language Query (NLQ) search to translate the user’s
query into the corresponding ElasticSearch Lucene Language, followed by Neo4J
Cypher language to retrieve the relevant information. While there are query
methods using graph vertices comparison [Truong et al., 2008], we found that
leveraging on the Cypher and Lucene search language produces acceptable
results. The user inputs a query from the search box, which is then passed to
the ElasticSearch database to retrieve relevant entity/relation’s ids by
leveraging on the ability of the Lucene query syntax to perform multi-match
queries across tokens.

Retrieving Knowledge Graph from Neo4J. After the ElasticSearch query returns
the relevant node and relation id, a Neo4J Cypher query is constructed to
retrieve the nodes and the first degree relations from the knowledge graph.
This knowledge association is displayed to the user using the ReactForceGraph
Javascript library.

Retrieving Articles from ElasticSearch. Relationships between Nodes are formed
from article clusters, with each cluster represented by a representative
document (Section 3.2). The corresponding cluster’s representative document is
displayed upon clicking a relation edge, and one may drill down to individual
documents. We note that there are many documents that do not fit into any
cluster, and have elected to present the documents alongside the clustered
articles, leveraging on the Lucene query language to search through the
document database to provide a comprehensive overview of the event.

##  4 Use Case Demonstration

We collected documents during the period of January to March 2020, mapping
documents related to the COVID19 world pandemic. We currently have 13,709 news
articles collected from mainstream sources, and 36860 Reddit posts from
r/coronavirus forming 158 clusters of documents. This forms 5525 nodes and
5441 relationships. We characterised clusters by days to keep up with the
COVID19 pandemic news.

Figure 3 showcases the KOSMOS user interface. A knowledge search begins with
(1), where the user inputs a search query, which brings up a knowledge graph
of entities that match to the search query and the first degree relationships
(2). A timeline displays retrieved articles matched through ElasticSearch
query feature (3). The presented graph have been filtered (through (4) node
type and data source selection). The edge (5) is expanded, where a timeline of
related articles is displayed on (6).

Figure 3: KOSMOS system presenting relations and documents to ’Singapore’

##  5 Conclusion and Future Work

We have presented KOSMOS as the first event-centric knowledge graph
information retrieval system, dynamically constructed from news and social
media documents. Our system fuses document clustering and knowledge graph
construction to understand relationships between documents. Through
identifying key document themes via document clustering, the noise in the data
is reduced, and with it, and the system’s existing knowledge graph is enhanced
with salient entities and relationships. Based on the knowledge graph, KOSMOS
delivers an overview of events through knowledge retrieval of nodes and
documents, allowing easier sensemaking through the vast information space.
This work naturally entails a couple of interesting direction for future
research: (1) enhancing the breadth of information through incorporation of
more mainstream and social media sites; (2) incorporating a question-and-
answer query system to parse natural language queries.

## References

  * [Angeli et al., 2015]  Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D. Manning.  2015\.  Leveraging linguistic structure for open domain information extraction.  In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 344–354, Beijing, China, July. Association for Computational Linguistics.
  * [Baumgartner et al., 2020]  Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn.  2020\.  The pushshift reddit dataset.
  * [Choo et al., 2014]  J. Choo, C. Lee, H. Kim, H. Lee, Z. Liu, R. Kannan, C. D. Stolper, J. Stasko, B. L. Drake, and H. Park.  2014\.  Visirr: Visual analytics for information retrieval and recommendation with large-scale document data.  In 2014 IEEE Conference on Visual Analytics Science and Technology (VAST), pages 243–244.
  * [Dhingra et al., 2020]  Bhuwan Dhingra, Manzil Zaheer, Vidhisha Balachandran, Graham Neubig, Ruslan Salakhutdinov, and William W. Cohen.  2020\.  Differentiable reasoning over a virtual knowledge base.  In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
  * [Finkel et al., 2005]  Jenny Rose Finkel, Trond Grenager, and Christopher Manning.  2005\.  Incorporating non-local information into information extraction systems by gibbs sampling.  In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, ACL ’05, page 363–370, USA. Association for Computational Linguistics.
  * [Gottschalk and Demidova, 2019]  Simon Gottschalk and Elena Demidova.  2019\.  Eventkg - the hub of event knowledge on the web - and biographical timeline generation.  Semantic Web.
  * [Hamborg et al., 2019]  Felix Hamborg, Corinna Breitinger, and Bela Gipp.  2019\.  Giveme5w1h: A universal system for extracting main events from news articles.  In Proceedings of the 13th ACM Conference on Recommender Systems, 7th International Workshop on News Recommendation and Analytics (INRA 2019), Sept.
  * [Honnibal and Montani, 2017]  Matthew Honnibal and Ines Montani.  2017\.  spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing.  To appear.
  * [Huang et al., 2019]  Xiao Huang, Jingyuan Zhang, Dingcheng Li, and Ping Li.  2019\.  Knowledge graph embedding based question answering.  In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, WSDM ’19, page 105–113, New York, NY, USA. Association for Computing Machinery.
  * [Miller, 2019]  Derek Miller.  2019\.  Leveraging bert for extractive text summarization on lectures.
  * [Rospocher et al., 2016]  Marco Rospocher, Marieke [van Erp], Piek Vossen, Antske Fokkens, Itziar Aldabe, German Rigau, Aitor Soroa, Thomas Ploeger, and Tessel Bogaard.  2016\.  Building event-centric knowledge graphs from news.  Journal of Web Semantics, 37-38:132 – 151.
  * [Rudnik et al., 2019]  Charlotte Rudnik, Thibault Ehrhart, Olivier Ferret, Denis Teyssou, Raphael Troncy, and Xavier Tannier.  2019\.  Searching news articles using an event knowledge graph leveraged by wikidata.  In Companion Proceedings of The 2019 World Wide Web Conference, WWW ’19, page 1232–1239, New York, NY, USA. Association for Computing Machinery.
  * [Suen et al., 2013]  Caroline Suen, Sandy Huang, Chantat Eksombatchai, Rok Sosic, and Jure Leskovec.  2013\.  NIFTY: a system for large scale information flow tracking and clustering.  In Daniel Schwabe, Virgílio A. F. Almeida, Hartmut Glaser, Ricardo Baeza-Yates, and Sue B. Moon, editors, 22nd International World Wide Web Conference, WWW ’13, Rio de Janeiro, Brazil, May 13-17, 2013, pages 1237–1248. International World Wide Web Conferences Steering Committee / ACM.
  * [Truong et al., 2008]  Quoc Dinh Truong, Taoufiq Dkaki, Josiane Mothe, and Pierre-Jean Charrel.  2008\.  Gvc: a graph-based information retrieval mode.  In CORIA.
  * [Wick and Boutreux, 2019]  Mark Wick and Christophe Boutreux.  2019\.  Geonames api.  http://www.geonames.org/about.html.  Online; accessed 07 May 2020.


# Zero-Shot Segmentation of Eye Features Using the Segment Anything Model
(SAM)

Virmarie Maquiling
Human-Centered Technologies for Learning
Technical University of Munich
<EMAIL_ADDRESS>
& Sean Anthony Byrne
MoMiLab
IMT School for Advanced Studies Lucca
<EMAIL_ADDRESS>
& Diederick C. Niehorster
Lund University Humanities Lab &
Dept. of Psychology
Lund University
<EMAIL_ADDRESS>
& Marcus Nyström
Lund University Humanities Lab
Lund University
<EMAIL_ADDRESS>
& Enkelejda Kasneci
Human-Centered Technologies for Learning
Technical University of Munich
<EMAIL_ADDRESS>

###### Abstract

The advent of foundation models signals a new era in artificial intelligence.
The Segment Anything Model (SAM) is the first foundation model for image
segmentation. In this study, we evaluate SAM’s ability to segment features
from eye images recorded in virtual reality setups. The increasing requirement
for annotated eye-image datasets presents a significant opportunity for SAM to
redefine the landscape of data annotation in gaze estimation. Our
investigation centers on SAM’s zero-shot learning abilities and the
effectiveness of prompts like bounding boxes or point clicks. Our results are
consistent with studies in other domains, demonstrating that SAM’s
segmentation effectiveness can be on-par with specialized models depending on
the feature, with prompts improving its performance, evidenced by an IoU of
93.34% for pupil segmentation in one dataset. Foundation models like SAM could
revolutionize gaze estimation by enabling quick and easy image segmentation,
reducing reliance on specialized models and extensive manual annotation.

_Keywords_ Eye-tracking, Segmentation, Segment Anything Model, Zero-shot
learning, Foundational models, Prompt Engineering

Figure 1: Visualization of SAM’s segmentation performance according to
strategy. On the leftmost column are the ground truth masks for key eye
features: pupil, iris, and sclera. The remaining columns show representative
prompt strategies used in the experiment overlayed with the resulting masks
generated by SAM. Green and red stars represent foreground and background
point prompts. The light blue rectangle represents a bounding box prompt.

##  1 Introduction

Foundation models and Generative AI are already being celebrated as the
groundbreaking developments of the decade in the realm of artificial
intelligence. They mark a significant paradigm change in the way users
interact, develop, and deploy deep learning models Bommasani et al. (2022).
Compared to traditional deep learning models, foundation models have scalable
attributes, such as the vast number of trainable parameters and the extensive
amounts of data they are trained on. Due to these scaling attributes
foundational models demonstrate a remarkable ability to adjust to downstream
tasks and exhibit proficiency on data distributions previously unseen during
their training phase Kirillov et al. (2023); Bommasani et al. (2022). From a
non-technical standpoint, these models have made it easier to utilize tools
powered by artificial intelligence, managing complex tasks that previously
required specialized supervised learning models, and often custom datasets
collected specifically for the problem at hand Bommasani et al. (2022); Zhou
et al. (2023b). In contrast, foundation models simplify the deployment process
through their innate zero-shot learning capabilities—where the model makes
accurate predictions on tasks without having seen any examples during training
Romera-Paredes and Torr (2015). This potential can be further improved with
straightforward prompts. These prompts can be as easy to create as a single
mouse click or circling a significant feature within an image, thereby
diminishing the complexity and time needed to apply deep learning models on a
large scale Mazurowski et al. (2023).

In this study, we harness the power of the Segment Anything Model (SAM)
Kirillov et al. (2023), the first foundation model developed for image
segmentation, and which has demonstrated remarkable abilities within a variety
of different domains, including both natural and specialized forms of imagery,
such as medical imaging Ma et al. (2023), and remote sensing Archit et al.
(2023). We apply SAM in the context of gaze estimation, where we task the
model to segment eye regions in near-eye images obtained from a virtual
reality (VR) setting. We investigate a variety of prompting strategies to
bolster SAM’s performance and closely examine its effectiveness in identifying
three specific ocular regions: the pupil, iris, and sclera. Annotating eye
regions, a well-known task within video-based eye-tracking, is both subjective
and costly in terms of time. These segmented features are essential not only
for precise gaze estimation but also as training data for specialized deep
learning models that require numerous annotated examples in order to make
accurate predictions.

In addition to their ease of deployment, foundation models like SAM hold
promise in overcoming the domain generalization issue—a notable obstacle
across machine learning, where models typically falter when applied to data
different from their training set Zhou et al. (2022). This challenge is
particularly evident in gaze estimation, where it has been noted that many
supervised learning models, such as convolutional neural networks, often fail
to generalize their understanding of eye components, resulting in subpar
performance on data not encountered during their training due to differences
in recording setup such as lighting, camera angle, and any physiological
differences across participants Kothari et al. (2022); Byrne et al. (2023a);
Kim et al. (2019). The development of extensive datasets recorded across
multiple devices Fuhl et al. (2021); Kothari et al. (2022) and the generation
of synthetic data Kim et al. (2019); Nair et al. (2020); Maquiling et al.
(2023); Byrne et al. (2023a) have been suggested to bypass the domain
generalization problem in gaze estimation. Yet, these approaches haven’t
completely resolved the issue and also introduce significant entry barriers,
including the need for substantial computational power and specialized
expertise in computer vision. However, foundation models have showcased
exceptional generalization abilities, both in zero-shot and few-shot
scenarios, adeptly adapting to unseen tasks and data distributions Brown et
al. (2020); Bommasani et al. (2022).

Our investigation is threefold. First, we provide a literature review on
foundation models, zero-shot learning, and eye image segmentation to give the
reader a clear understanding of the problem and potential of foundation models
in this domain. Second, we explain what it means for a model like SAM to be
“promptable” and introduce a set of prompting strategies designed to explore
SAM’s capabilities for eye image segmentation. Third, using these tailor-made
prompting strategies, we evaluate SAM’s performance on two public eye image
segmentation datasets, providing quantitative and qualitative analysis with
the aim of illuminating the potential and limitations of applying SAM to the
domain of eye tracking. Our results reveal that SAM, in a zero-shot learning
context, can produce annotations that rival manual annotations derived from
experts and annotations from traditional supervised learning models in
segmenting the pupil, even with minimal manual guidance. Nevertheless, the
model exhibits certain limitations in the accurate annotation of the sclera
and, to a lesser degree, the iris. We have made all code available to users at
the following repository [HIDDEN FOR REVIEW] for both benchmarking and
labeling their own datasets.

##  2 RELATED WORK

###  2.1 An Introduction to Foundation models

Foundation models are a relatively new concept in deep learning, and as such,
there is not an extensive amount of literature available on them yet. Although
these models first gained prominence in the realm of Natural Language
Processing (NLP), they have recently made significant strides in the field of
computer vision. Foundation models are large-scale machine-learning models
trained on massive amounts of data that can be adapted for a wide range of
downstream tasks. These models mark a departure from conventional deep
learning approaches due to their scale, boasting not only an unprecedented
number of parameters (up to hundreds of billions) but also being pre-trained
on extensive datasets. This enables them to adapt to a variety of tasks
through fine-tuning and to exhibit emergent behaviors, such as strong
performance on previously unseen data distributions Bommasani et al. (2022).
Leading the way are models like BERT Devlin et al. (2018), Generative Pre-
Trained Transformers (GPTs) Brown et al. (2020), and CLIP Radford et al.
(2021), all of which leverage large-scale data for training. ChatGPT, fine-
tuned from the GPT-3 model with its 175 billion parameters, excels in various
tasks through natural language prompts and has significantly influenced AI
over the last year Zhou et al. (2023b). Already, there is a growing trend
toward the creation of specialized foundation models, such as MedSAM for
medical imaging Ma et al. (2023), and even more domain-specific, RETFound, a
model designed specifically for the analysis of retinal images Zhou et al.
(2023a). For a comprehensive review of foundation models see Bommasani et al.
(2022).

###  2.2 The Segment Anything Model

The Segment Anything model (SAM) Kirillov et al. (2023) is a state-of-the-art
vision transformer trained on the SA-1B dataset, a collection of diverse
images (11 million) accompanied by high-quality segmentation masks (1.1
billion) created specifically for SAM’s training, making it the largest
publicly available image segmentation dataset to date. The dataset was
generated in three stages. In the first stage, a set of images was labeled by
a team of professional annotators by clicking on foreground and background
points and refining masks generated by an early version of SAM that was
trained on common public segmentation datasets. In the second stage, SAM
automatically generated masks of a subset of the dataset, while the annotators
focused on annotating the rest of the objects to increase mask diversity. In
the last stage, the remaining masks were automatically generated by SAM by
prompting the model with a $32\times 32$ grid of foreground points Kirillov et
al. (2023).

In order to enable zero-shot generalization in image segmentation tasks, a
novel promptable segmentation task was introduced Kirillov et al. (2023), in
which, given an image and any prompt, the goal is to return a valid
segmentation mask. In this context, a “prompt” specifies which object needs to
be segmented by the model, e.g. by providing spatial information. This can be
manually given by a user or generated by a different model (e.g. using an
object detection model).

To support this novel promptable task, the model, as depicted in Fig. 2,
consists of three main components (for a more detailed description, refer to
the original paper Kirillov et al. (2023)):

Figure 2: A high-level schematic of the Segment Anything Model (SAM) featuring
an image encoder, a prompt encoder, and a lightweight mask encoder. The image
is fed to the image encoder, while a set of prompts (in this example, depicted
as point- and bounding box prompts drawn on the image) are fed to the prompt
encoder. The embeddings generated by the image and prompt encoders are passed
to the mask decoder which outputs valid masks with varying confidence scores.
Alternatively, it may output a single mask averaged over all valid masks.

  * •

Image encoder: This uses a Vision Transformer Dosovitskiy et al. (2010), pre-
trained using Masked Autoencoders (MAE) He et al. (2022), as its backbone and
can be run once per image before prompting the model. The image encoder uses
an input resolution of $1024\times 1024$ by resizing and padding the input
image. It then outputs a $16\times$ downscaled image embedding of size
$64\times 64$.

  * •

Prompt encoder: This encoder accepts sparse (points, boxes, text) and dense
(masks) prompts and encodes them as positional embeddings (points and boxes),
by CLIP Radford et al. (2021) (text), or through convolutions (masks). Point
prompts are given as pixel coordinates in the image, accompanied by point
labels indicating whether each corresponding point is in the foreground or the
background. Bounding box prompts consist of a pair of pixel coordinates
indicating the top-left corner and the bottom-right corner of the box. For
information regarding other prompt modes, refer to the original paper Kirillov
et al. (2023).

  * •

Mask decoder: This lightweight decoder maps the image and prompt embeddings to
an output mask.

SAM supports two modes: an automatic “everything” mode, in which the user
simply inputs an image to SAM, which then returns all predicted masks found
within the image, and a manual mode which expects the user to input an image
and a combination of manual prompts to indicate the location of the object to
be segmented, e.g. by simply clicking on the image or drawing a bounding box
around the region of interest.

Three pre-trained models of varying network sizes are available in SAM’s
GitHub repository111https://github.com/facebookresearch/segment-anything/.
From smallest to largest, they are named ViT Base (ViT-B), Vit Large (ViT-L),
and Vit Huge (ViT-H). The authors report substantial improvement of ViT-H over
ViT-B, while showing only marginal improvement over ViT-L. However, due to its
increased complexity, inference time is also multiplied Kirillov et al.
(2023). Nevertheless, previous work reported that ViT-H did not show a
significant advantage over ViT-B, with ViT-B even surpassing larger variants
in some segmentation tasks Huang et al. (2023); Mattjie et al. (2023).

###  2.3 Applications of the Segment Anything Model

The application of the Segment Anything Model (SAM) has already been broad,
extending to various domains and specialized tasks, with adaptations that have
seen it being used in video tracking Yang et al. (2023) and three-dimensional
environments Shen et al. (2023). However, this literature review section will
focus exclusively on the downstream performance of the original SAM
architecture. To our knowledge, there exists only a singular study where SAM
has been integrated with eye-tracking data: GazeSAM Wang et al. (2023a). This
model enhances the automation of medical image segmentation by capturing the
eye movements of radiologists and using this gaze data to prompt SAM for
interactive segmentation tasks. SAM has been notably utilized in medical
imaging in many studies. One study demonstrates SAM’s versatility across 19
different medical imaging datasets by employing interactive point and box
prompts in a zero-shot segmentation task. The outcomes showed variable
Intersection Over Union (IOU) scores depending on the specific dataset and
segmentation task, with scores fluctuating from 0.1135 in spine datasets to
0.8650 in MRI images Mazurowski et al. (2023). These findings were largely
corroborated in a separate study, which emphasized SAM’s capacity to minimize
annotation time in medical image analysis Huang et al. (2023). This study also
concluded that SAM benefits from point and box prompts to enhance object
detection in medical images, yet much like in the aforementioned work
Mazurowski et al. (2023) its performance remains inconsistent across different
objects and modalities. Taking this a step further, researchers created
MedSAM, a foundational model specifically for medical images Ma et al. (2023).
The study leveraged the SAM dataset to pre-train a transformer model, which
was then fine-tuned using over 1 million medical images. The MedSAM model not
only surpassed the standard SAM in performance but also matched or exceeded
specialized models, like a U-net, which are tailored for particular medical
imaging tasks Ma et al. (2023). Outside of medical imaging, SAM has been
applied in domains such as graphs Jing et al. (2023), remote sensing Wang et
al. (2023b) and microscopy Archit et al. (2023) all with various degrees of
success depending on the task.

###  2.4 Segmentation of eye-images recorded using video-based eye-tracking
systems

In the context of image analysis for video-based eye-tracking, segmentation is
an essential process that typically involves the detection and segmentation of
areas such as the pupil, sclera, iris, and any corneal reflections according
to the specific requirements of the task. Pupil center localization is a core
part of many gaze estimation methodologies Kim et al. (2019). Additionally,
techniques that track both the pupil and corneal reflections (P-CR eye
tracking) are widely employed for gaze estimation, representing the leading
approach in the video-based eye-tracking sector Dunn et al. (2023). Limbus
tracking, which uses the boundary between the iris and sclera, is another
technique utilized for gaze and eye movement monitoring Holmqvist et al.
(2011).

Historically, the segmentation of eye images from video footage was often done
using computer vision techniques, such as ellipse fitting or thresholding
methods. These techniques rely on handcrafted algorithms and heuristics to
identify features Kothari et al. (2022); Santini et al. (2018). Despite their
effectiveness, they can fail in the presence of, for instance, occlusions,
changing illumination conditions, or reflections in the eye images Fuhl et al.
(2016b). A potential remedy has been the adoption of machine learning and,
subsequently, deep learning models, which have considerably enhanced the
robustness of gaze estimation algorithms Kothari et al. (2022); Fuhl et al.
(2016a, 2022). The issue of data scarcity and the aspiration to tackle domain
generalization issues have led to propositions for both large-scale Fuhl et
al. (2021); Kothari et al. (2022) and synthetic datasets Byrne et al. (2023b,
a); Kim et al. (2019); Maquiling et al. (2023); Nair et al. (2020). These are
intended as solutions, yet the challenge of domain generalization remains
largely unaddressed in the literature. To date, the necessity for a new
specialized model, which requires significant expertise to construct, persists
without a convincing alternative.

##  3 Methodology

###  3.1 Gaze Datasets

To assess SAM’s efficacy, we utilized two datasets: the Open Eye Challenge
dataset (OpenEDS2019) Garbin et al. (2020), which is widely used to benchmark
gaze estimation methods Nair et al. (2020); Chaudhary et al. (2022); Kim et
al. (2019); Kothari et al. (2021, 2022); Byrne et al. (2023a), and its
successor, the OpenEDS 2020 Challenge dataset (OpenEDS2020) Palmero et al.
(2021). OpenEDS2019 Garbin et al. (2020) was collected using a VR head-mounted
display (HMD) equipped with two synchronized eye-facing cameras under
controlled illumination at a frame rate of 200 Hz. The dataset encompasses
eye-region video footage from 152 individual participants, comprising a total
of 12,759 images at a resolution of $400\times 640$ pixels, coupled with
pixel-level annotation masks for key eye-regions, namely the pupil, iris, and
sclera. For a complete description of the data, refer to the original paper
Garbin et al. (2020). The OpenEDS 2020 Challenge dataset Palmero et al. (2021)
consists of eye-image sequences recorded through a VR HMD mounted with dual
eye-facing cameras under controlled lighting conditions at a frame rate of 100
Hz and recorded from 80 participants while performing various gaze-related
tasks. It is further separated into two subsets: 1) Gaze Prediction Dataset
and 2) Eye Segmentation Dataset. The latter consists of 200 sequences sampled
at 5 Hz accompanied by manually annotated segmentation labels for 5% of the
dataset, totaling 2605 images at a resolution of $640\times 400$ pixels. A
representative image from each dataset is displayed in Fig. 3. Before being
fed into SAM’s image encoder, each image is converted to RGB with a pixel
intensity range of $[0,255]$. No further adjustments to the images were
performed.

As part of each challenge, the challenge authors conducted baseline
experiments to evaluate performance on both datasets. Specifically, they
trained an encoder-decoder neural network derived from the SegNet architecture
Badrinarayanan et al. (2017) for each dataset. For OpenEDS2019, the model was
trained for 200 epochs on the training set provided in the challenge dataset,
achieving a mean intersection over union (mIoU) score of 91.4% on the test set
consisting of all three features. For OpenEDS2020, the authors trained the
model for 150 epochs on the subset of images that had corresponding
segmentation masks, reporting a mIoU score of 84.1% across all features.

Figure 3: Representative images from OpenEDS2019 Garbin et al. (2020) (left)
and OpenEDS2020 Palmero et al. (2021) (right) datasets. Both datasets are
captured using a VR head-mounted display equipped with dual synchronous eye-
facing cameras. The OpenEDS2019 dataset contains images with a resolution of
$400\times 640$ while the images from the OpenEDS2020 dataset have a
resolution $640\times 400$.

###  3.2 Prompting Strategies

Figure 4: Prompt strategies for segmenting pupil, iris, and sclera. Green
points represent foreground point prompts, while red points represent
background point prompts. Bounding box prompts are visualized as light blue
rectangles surrounding the feature of interest.

We assess SAM’s performance on eye images, utilizing both the automatic
“everything” mode and the manual prompting mode, and employing a combination
of bounding box and point prompts. SAM’s main strength lies in its capability
to incorporate prompts, making it imperative to evaluate the impact of
different prompt strategies on model performance. We designed a set of
prompting strategies that could be employed by a human annotator using SAM for
eye image annotation, as depicted in Fig. 4. Each prompt strategy is described
below.

####  3.2.1 Automated/no prompting

This mode generates all potential segmentations identified by SAM within the
image. In the context of data annotation, upon feeding the image to the model,
the user receives a series of masks as output. However, not all these
segmentations may align with the user’s needs. Consequently, the user may opt
to manually choose the masks that best align with the features relevant to
them. To simulate this, we adopt the mask-matching mechanism proposed by Huang
et al. [2023], which involves identifying masks with the closest Dice scores
to the ground truth masks representing the pupil, iris, and sclera.
Subsequently, the masks with the highest scores are considered as the model’s
predicted features for the given image. We refer to this strategy as E
(everything) in our results.

####  3.2.2 Manual prompting

SAM gives users the flexibility to use diverse prompt strategies to select the
feature to segment. Possible strategies include clicking within or outside the
region of interest, drawing a loose or tight bounding box around it, or a
combination of these approaches. While an exhaustive comparison of all
potential prompt strategies is unfeasible, we can categorize these strategies
into the following actions: using a point prompt, using multiple point
prompts, using a bounding box, and a combination thereof. In this mode, we
specifically focus our analysis of SAM’s performance on the following prompt
designs, visually depicted in Fig. 4:

  1. 1.

Point prompts (P1, P4): single or multiple (4) points are randomly positioned
anywhere on the ground truth mask.

  2. 2.

Positive and negative point prompts (P4-4): taking inspiration from Huang et
al. [2023], 4 positive (foreground) points are randomly positioned on the
ground truth mask. A bounding box around the ground truth mask is then
calculated and doubled in size. 4 negative points are placed outside the
ground truth mask within the resulting bounding box.

  3. 3.

Bounding Box (BBOX): a tight bounding box surrounding the ground truth mask.

  4. 4.

Bounding Box with point prompts (BBOXP1, BBOXP4): a tight bounding box
surrounding the ground truth mask and a single or multiple (4) point prompts
positioned randomly anywhere on the ground truth mask.

  5. 5.

Bounding Box with positive and negative point prompts (BBOXP1-1, BBOXP4-4): To
assist SAM in excluding “holes” within the iris and sclera segmentations, we
provide it with a tight bounding box and a single or 4 positive point prompts
placed randomly on the ground truth mask (the same as BBOXP1 and BBOXP4). We
furthermore provide SAM with the same number of negative point prompts placed
on the ground truth iris mask (when segmenting the sclera) or on the pupil
mask (when segmenting the iris). In this experiment, images where the “holes”
are not present (i.e. in cases where the eye is half-covered by the eyelid or
in the process of blinking such that the sclera is visible but not the iris,
or the iris is visible but not the pupil) are skipped. This strategy
furthermore does not apply to pupil segmentation.

Furthermore, we explore the impact on SAM’s performance when introducing
variability in the bounding box strategy (BBOX). Taking inspiration from
Mattjie et al. [2023], we adjust the bounding box around the ground truth
mask, modifying its size and position by 5%, 10%, and 20%. This variation
mirrors the potential differences that may arise when different annotators
draw bounding boxes over the image.

Across all strategies, images where the targeted feature is absent are
excluded. In other words, if SAM’s ability to segment a specific feature
(e.g., pupil) is being evaluated, images in which that feature is not present
are skipped.

###  3.3 Evaluation

To evaluate SAM’s segmentation quality, we want to analyze the overlap between
its segmentations and the ground truth masks, while also gauging the alignment
of the two masks. To measure this, we chose well-established metrics that are
commonly used in evaluating image segmentation models:

Dice similarity coefficient (Dice) Milletari et al. (2016): a similarity
metric that gauges the overlap between the ground truth mask and the predicted
mask. A higher value signifies better model performance. The Dice coefficient
is defined as:

$Dice=\frac{2\times|A\cap B|}{|A|+|B|}$ (1)

Intersection-over-union (IoU) Zhou et al. (2019): similar to Dice, it measures
the overlap between two masks by getting the ratio of the intersection area to
the union area of the segments. IoU requires more precision than Dice as it
penalizes both under- and over-segmentation more rigorously. Higher values
signify a better model:

$\text{IoU}=\frac{{|A\cap B|}}{{|A\cup B|}}$ (2)

Hausdorff Distance (HD) Huttenlocher et al. (1993): a boundary-based metric
that quantifies the dissimilarity between two shapes by measuring the maximum
distance between points on one boundary to the nearest points on the other
boundary. This metric can be used to evaluate the degree of alignment between
the boundaries of the predicted and ground truth masks. A lower Hausdorff
distance indicates better model performance. Given two sets of finite points A
and B and with $d(a,b)$ as the distance between points a and b, the Hausdorff
Distance is defined as:

$H(A,B)=max(h(A,B),h(B,A))$ (3)

where

$h(A,B)=\sup_{a\in A}\inf_{b\in B}\,d(a,b)$ (4)

where $\sup$ represents the supremum operator and $\inf$ the infimum operator.

##  4 Results

Figure 5: Performance of each strategy on different segmentation tasks (pupil,
iris, and sclera segmentation). Each row represents a different metric. On
each plot, the best-performing strategy (highest Dice/IoU and lowest HD) is
colored red. OpenEDS-2019
---
| Pupil | Iris | Sclera
Perturbance | 5% | 10% | 20% | 5% | 10% | 20% | 5% | 10% | 20%
Dice $\uparrow$ | 0.9546 | 0.8423 | 0.4493 | 0.8916 | 0.8775 | 0.7165 | 0.6184 | 0.6021 | 0.4574
IoU $\uparrow$ | 0.9194 | 0.7903 | 0.4086 | 0.8083 | 0.7917 | 0.6261 | 0.4616 | 0.4493 | 0.3330
HD $\downarrow$ | 4.1031 | 12.1731 | 37.5265 | 52.4545 | 52.9094 | 63.3940 | 78.6238 | 79.8332 | 92.8560
OpenEDS-2020
| Pupil | Iris | Sclera
Perturbance | 5% | 10% | 20% | 5% | 10% | 20% | 5% | 10% | 20%
Dice $\uparrow$ | 0.9458 | 0.7849 | 0.3583 | 0.8887 | 0.8763 | 0.7795 | 0.5124 | 0.4701 | 0.2932
IoU $\uparrow$ | 0.9029 | 0.7229 | 0.3244 | 0.8038 | 0.7895 | 0.6830 | 0.3526 | 0.3213 | 0.1967
HD $\downarrow$ | 4.8235 | 13.9860 | 37.8557 | 40.7677 | 42.4274 | 54.2700 | 58.6688 | 72.2898 | 144.7311
Table 1: SAM’s performance on OpenEDS-2019 and -2020 datasets when perturbing
the bounding box by 5%, 10%, and 20%

Our findings across both datasets demonstrate that SAM’s performance is highly
dependent on the type of feature it is segmenting. When segmenting the pupil
(refer to Fig. 5, left column), SAM consistently had an IoU score above 88%
across both datasets in all strategies, with BBOXP4 yielding the best results
with an IoU score of 93.30 % and 93.34% for OpenEDS-2019 and OpenEDS-2020
respectively. When segmenting the iris (Fig. 5, middle column), E performed
far more poorly with an IoU score of 36.45% for OpenEDS-2019 and 31.92% for
OpenEDS-2020. Meanwhile, the prompting strategies all scored above 60% and
BBOXP1-1 was the best-performing strategy with an IoU score of 86.63% and
86.14% for OpenEDS-2019 and OpenEDS-2020 respectively. Lastly, when segmenting
the sclera (Fig. 5, right column), SAM performed very poorly compared to when
segmenting the pupil and sclera with an IoU score of 8.39% and 5.86% when
using the automatic mode. While introducing manual prompts boosted its
performance, it reached only an IoU score of 62.19% and 47.06% using BBOXP4-4
for OpenEDS-2019 and OpenEDS-2020 respectively, indicating the need for
additional manual guidance.

Comparing across prompting strategies (Fig. 5), E performed reasonably well
when segmenting the pupil and even outperformed P1 in both datasets. However,
its performance decreased dramatically when segmenting the iris and sclera, in
some cases, not overlapping at all with the ground truth mask. This is
exemplified by E’s predictions on the sclera in Fig. 7, and 7 which show in
both cases that the closest matching segmentation that SAM produces is nowhere
near the ground truth mask of the sclera. When introducing point prompts (P1,
P4, P4-4), SAM’s performance increased with the number of point prompts used.
Using a simple bounding box prompt (BBOX), however, outperformed the use of
only point prompts. This is likely because the bounding box more explicitly
delineates the area where the feature of interest is located, while point
prompts may be ambiguous regarding exactly which object the prompt is
referring to. Lastly, a combination of bounding box and point prompts (BBOXP1,
BBOXP4, BBOXP1-1, and BBOXP4-4) resulted in the best performance of SAM on
both datasets for both the iris and sclera.

Analyzing the impact of perturbed bounding boxes (refer to Table 1), SAM
demonstrates a decline in performance with decreasing accuracy of the bounding
box placement. However, even when the bounding box is perturbed, SAM still
consistently outperforms the majority of the point prompt strategies when
segmenting the iris and sclera, particularly with perturbations of 5% and 10%.
This observation aligns with the findings of Mattjie et al. [2023],
underlining the robustness of the bounding box approach compared to point
prompts.

When evaluating SAM’s performance based on the alignment of mask boundaries
(refer to Fig. 5, third and last rows), it is evident that strategies
incorporating the bounding box prompt consistently yield the best results.
However, it is noteworthy that not all strategies deliver the highest degree
of alignment (measured by the lowest HD) and the best overlap (measured by the
highest IoU and Dice scores) simultaneously. This is particularly evident when
examining performance on the sclera. While BBOXP4-4 showed the best overlap
for both datasets, it did not show the best alignment. Meanwhile, the BBOXP1-1
strategy showed the best alignment for OpenEDS2019 and BBOX performed best on
OpenEDS2020 while these showed up to almost 30% worse overlap scores. This
indicates that, although incorporating point prompts with the bounding box
prompt may help SAM to capture the general shape of the sclera, it might
actually hurt its ability to accurately delineate its exact boundaries.
Although less clear, the same trend is seen in the iris data, where adding a
single prompt point to the bounding box lead to better alignment than adding
four prompt points.

Choosing the best-performing strategy for the pupil (BBOXP4), iris (BBOXP1-1),
and sclera (BBOXP4-4), we calculated the mIoU for each dataset. The results
indicated a mIoU of 85.70% and 83.90% on the OpenEDS-2019 and OpenEDS-2020
datasets, respectively. These values fell short of the previously established
baselines of 91.4% Garbin et al. (2020) and 84.1% Palmero et al. (2021). This
is likely attributed to SAM’s limitations in accurately segmenting the sclera
as previously observed. Indeed, upon excluding the results related to sclera,
the mIoU improved significantly to 91.38% for OpenEDS2019 and 91.02% for
OpenEDS2020. However, these scores still lag behind the top-performing results
from the accompanying leaderboards222https://eval.ai/web/challenges/challenge-
page/353/leaderboard/1002,333https://eval.ai/web/challenges/challenge-
page/603/leaderboard/1680, which achieved a mIoU score of 95.28% for
OpenEDS2019 and 95.17% for OpenEDS2020.

Figure 6: Visualization of SAM’s performance on an image from OpenEDS2019
Garbin et al. (2020). The leftmost column shows the ground truth masks for
pupil (top row), iris (middle row), and sclera (bottom row) overlayed on the
input image. The remaining columns show SAM’s segmentations using different
strategies.

Figure 7: Visualization of SAM’s performance on an image from OpenEDS2020
Palmero et al. (2021)

##  5 Discussion

Evaluating the capability of the Segment Anything Model (SAM) in delineating
critical eye features for eye tracking, our study reveals that SAM effectively
performs zero-shot segmentation on the pupil, achieving an impressive IoU
(Intersection over Union) of 93.34%. With the assistance of bounding box and
point prompts, it also successfully segments the iris, reaching an IoU of
86.63%. Nevertheless, SAM encounters difficulties with the sclera, achieving a
best IoU of only 62.19%, and this limitation persists regardless of the type
of prompt used, even though prompts do enhance the performance over its
automatic mode. These findings are consistent with previous investigations
mainly from the medical imaging sector Mazurowski et al. (2023); Huang et al.
(2023).

The fundamental question for those in the eye-tracking field may boil down to
“Is SAM suitable for segmenting eye images?” The answer hinges on the specific
feature in question. For example, SAM exhibits strong capabilities in pupil
segmentation which can be used in pupil detection pipelines, a crucial element
in numerous gaze-tracking frameworks Kim et al. (2019), and the primary
subject of various studies Fuhl et al. (2016a); Santini et al. (2018); Fuhl et
al. (2015). The effectiveness of SAM for segmenting the iris is comparatively
lower, compared to the pupil. We attribute this reduction in performance to
the iris having a lower contrast and relatively blurred edge with the sclera,
conditions that have previously been observed to pose problems for SAM Mattjie
et al. (2023); Huang et al. (2023). Moreover, our results reveal that SAM
struggles to recognize the sclera as a distinct object and requires extensive
guidance from an annotator through the use of manual prompts. We speculate
that shadows from the eyelids and non-uniform illumination contribute to large
variations in scleral luminance in the images. This, combined with the blurry
edge between the iris and the sclera, likely poses a significant challenge for
SAM.

##  6 Limitations and Future Work

Our study presents several limitations. First, our study assessed only the
zero-shot capabilities of SAM within a Virtual Reality (VR) environment.
Future research should extend the evaluation of SAM’s performance to eye
images acquired from wearable as well as high-resolution laboratory eye
trackers. High-resolution eye tracking may demand greater precision and the
segmentation of different features Byrne et al. (2023a) for which the
segmentation techniques utilized in our study may not be effective. The
complexity of “gaze-in-the-wild” contexts, characterized by considerable
variability in eye appearance and challenges with low image quality, may
present a more arduous challenge than VR environments investigated in this
paper Kothari et al. (2020); Fuhl et al. (2016b, 2022). Second, there is
potential for uncovering more effective prompting strategies that could
further enhance the model’s performance in segmenting eye regions. Finally,
our study did not address the segmentation of corneal reflections, despite
their presence in the images (as demonstrated in Chugh et al. (2021);
Maquiling et al. (2023); Byrne et al. (2023a)) because they were not annotated
in the dataset. Future work should consider including these reflections, as
the detection and matching of these features can be integral to comprehensive
eye-tracking analyses and P-CR pipelines Chugh et al. (2021).

This research opens the door to a variety of future research avenues. One
promising direction is the fine-tuning of SAM on a small amount of eye images,
to investigate if this fine-tuning step improves the results compared to the
standard model. Second, developing a foundation model trained on a
comprehensive dataset, such as TEyeD Fuhl et al. (2021), could become a
pivotal asset for the eye-tracking community. However, this endeavor would
necessitate computational resources beyond those typically accessible to a
standard laboratory. Another intriguing potential research path involves
integrating text prompting into the model to further simplify the annotation
process for users without technical expertise.

##  7 Conclusion

This study explored the applicability of the Segment Anything Model (SAM) for
segmenting eye images, the first study into a foundation model’s performance
in this specific task to our knowledge. SAM showed promising zero-shot
capabilities, particularly in pupil segmentation. While we recommend a
cautious approach to using SAM for eye image segmentation in its current
state, this technology may represent a paradigm shift away from specialized
supervised learning models towards foundational models. However, what the eye-
tracking community may need for this technology to be fully realized is a
bespoke foundation model trained on eye images, akin to how MedSAM Ma et al.
(2023) is tailored for medical images or RETFound Zhou et al. (2023a) for
retinal imaging. Regardless, foundation models offer an exciting new direction
for the field. They could reduce the technical barriers associated with
developing specialized models, mitigate the acute shortage of annotated
datasets required for training deep learning systems, and achieve domain
generalization—critical challenges in gaze estimation. This shift has the
potential to democratize eye-tracking technologies, particularly as VR systems
become increasingly mainstream, enabling new market entrants to compete with
established companies by leveraging the scalable utility of foundation models.

## References

  * (1)
  * Archit et al. (2023) Anwai Archit, Sushmita Nair, Nabeel Khalid, Paul Hilt, Vikas Rajashekar, Marei Freitag, Sagnik Gupta, Andreas Dengel, Sheraz Ahmed, and Constantin Pape. 2023.  Segment Anything for Microscopy.  _bioRxiv_ (2023), 2023–08.
  * Badrinarayanan et al. (2017) Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. 2017.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  _IEEE transactions on pattern analysis and machine intelligence_ 39, 12 (2017), 2481–2495.
  * Bommasani et al. (2022) Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2022.  On the Opportunities and Risks of Foundation Models.  arXiv:2108.07258 [cs.LG]
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners.  arXiv:2005.14165 [cs.CL]
  * Byrne et al. (2023a) Sean Anthony Byrne, Virmarie Maquiling, Marcus Nyström, Enkelejda Kasneci, and Diederick C Niehorster. 2023a.  LEyes: A Lightweight Framework for Deep Learning-Based Eye Tracking using Synthetic Eye Images.  _arXiv preprint arXiv:2309.06129_ (2023).
  * Byrne et al. (2023b) Sean Anthony Byrne, Marcus Nyström, Virmarie Maquiling, Enkelejda Kasneci, and Diederick C Niehorster. 2023b.  Precise localization of corneal reflections in eye images using deep learning trained on synthetic data.  _arXiv preprint arXiv:2304.05673_ (2023).
  * Chaudhary et al. (2022) Aayush K Chaudhary, Nitinraj Nair, Reynold J Bailey, Jeff B Pelz, Sachin S Talathi, and Gabriel J Diaz. 2022.  Temporal RIT-Eyes: From real infrared eye-images to synthetic sequences of gaze behavior.  _IEEE Transactions on Visualization and Computer Graphics_ 28, 11 (2022), 3948–3958.
  * Chugh et al. (2021) Soumil Chugh, Braiden Brousseau, Jonathan Rose, and Moshe Eizenman. 2021.  Detection and correspondence matching of corneal reflections for eye tracking using deep learning. In _2020 25th International Conference on Pattern Recognition (ICPR)_. IEEE, 2210–2217.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Dosovitskiy et al. (2010) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2010\.  An image is worth 16x16 words: Transformers for image recognition at scale. arXiv 2020.  _arXiv preprint arXiv:2010.11929_ (2010).
  * Dunn et al. (2023) Matt J Dunn, Robert G Alexander, Onyekachukwu M Amiebenomo, Gemma Arblaster, Denize Atan, Jonathan T Erichsen, Ulrich Ettinger, Mario E Giardini, Iain D Gilchrist, Ruth Hamilton, et al. 2023\.  Minimal reporting guideline for research involving eye tracking (2023 edition).  _Behavior research methods_ (2023), 1–7.
  * Fuhl et al. (2021) Wolfgang Fuhl, Gjergji Kasneci, and Enkelejda Kasneci. 2021.  TEyeD: Over 20 Million Real-World Eye Images with Pupil, Eyelid, and Iris 2D and 3D Segmentations, 2D and 3D Landmarks, 3D Eyeball, Gaze Vector, and Eye Movement Types. In _2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)_. IEEE.  https://doi.org/10.1109/ismar52148.2021.00053
  * Fuhl et al. (2015) Wolfgang Fuhl, Thomas Kübler, Katrin Sippel, Wolfgang Rosenstiel, and Enkelejda Kasneci. 2015.  Excuse: Robust pupil detection in real-world scenarios. In _Computer Analysis of Images and Patterns: 16th International Conference, CAIP 2015, Valletta, Malta, September 2-4, 2015 Proceedings, Part I 16_. Springer, 39–51.
  * Fuhl et al. (2016a) Wolfgang Fuhl, Thiago Santini, Gjergji Kasneci, and Enkelejda Kasneci. 2016a.  Pupilnet: Convolutional neural networks for robust pupil detection.  _arXiv preprint arXiv:1601.04902_ (2016).
  * Fuhl et al. (2016b) Wolfgang Fuhl, Marc Tonsen, Andreas Bulling, and Enkelejda Kasneci. 2016b.  Pupil detection for head-mounted eye tracking in the wild: an evaluation of the state of the art.  _Machine Vision and Applications_ 27 (2016), 1275–1288.
  * Fuhl et al. (2022) Wolfgang Fuhl, Daniel Weber, and Shahram Eivazi. 2022.  Pistol: Pupil invisible supportive tool to extract pupil, iris, eye opening, eye movements, pupil and iris gaze vector, and 2d as well as 3d gaze.  _arXiv preprint arXiv:2201.06799_ (2022).
  * Garbin et al. (2020) Stephan Joachim Garbin, Oleg Komogortsev, Robert Cavin, Gregory Hughes, Yiru Shen, Immo Schuetz, and Sachin S Talathi. 2020.  Dataset for eye tracking on a virtual reality platform. In _ACM Symposium on Eye Tracking Research and Applications_. 1–10.
  * He et al. (2022) Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022.  Masked autoencoders are scalable vision learners. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 16000–16009.
  * Holmqvist et al. (2011) Kenneth Holmqvist, Marcus Nyström, Richard Andersson, Richard Dewhurst, Halszka Jarodzka, and Joost Van de Weijer. 2011.  _Eye tracking: A comprehensive guide to methods and measures_.  OUP Oxford.
  * Huang et al. (2023) Yuhao Huang, Xin Yang, Lian Liu, Han Zhou, Ao Chang, Xinrui Zhou, Rusi Chen, Junxuan Yu, Jiongquan Chen, Chaoyu Chen, Haozhe Chi, Xindi Hu, Deng-Ping Fan, Fajin Dong, and Dong Ni. 2023.  Segment Anything Model for Medical Images?  arXiv:2304.14660 [eess.IV]
  * Huttenlocher et al. (1993) Daniel P Huttenlocher, Gregory A. Klanderman, and William J Rucklidge. 1993.  Comparing images using the Hausdorff distance.  _IEEE Transactions on pattern analysis and machine intelligence_ 15, 9 (1993), 850–863.
  * Jing et al. (2023) Yongcheng Jing, Xinchao Wang, and Dacheng Tao. 2023.  Segment Anything in Non-Euclidean Domains: Challenges and Opportunities.  arXiv:2304.11595 [cs.CV]
  * Kim et al. (2019) Joohwan Kim, Michael Stengel, Alexander Majercik, Shalini De Mello, David Dunn, Samuli Laine, Morgan McGuire, and David Luebke. 2019.  Nvgaze: An anatomically-informed dataset for low-latency, near-eye gaze estimation. In _Proceedings of the 2019 CHI conference on human factors in computing systems_. 1–12.
  * Kirillov et al. (2023) Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. 2023.  Segment Anything.  arXiv:2304.02643 [cs.CV]
  * Kothari et al. (2020) Rakshit Kothari, Zhizhuo Yang, Christopher Kanan, Reynold Bailey, Jeff B Pelz, and Gabriel J Diaz. 2020.  Gaze-in-wild: A dataset for studying eye and head coordination in everyday activities.  _Scientific reports_ 10, 1 (2020), 2539.
  * Kothari et al. (2022) Rakshit S Kothari, Reynold J Bailey, Christopher Kanan, Jeff B Pelz, and Gabriel J Diaz. 2022.  EllSeg-Gen, towards Domain Generalization for head-mounted eyetracking.  _Proceedings of the ACM on Human-Computer Interaction_ 6, ETRA (2022), 1–17.
  * Kothari et al. (2021) Rakshit S Kothari, Aayush K Chaudhary, Reynold J Bailey, Jeff B Pelz, and Gabriel J Diaz. 2021.  Ellseg: An ellipse segmentation framework for robust gaze tracking.  _IEEE Transactions on Visualization and Computer Graphics_ 27, 5 (2021), 2757–2767.
  * Ma et al. (2023) Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. 2023.  Segment Anything in Medical Images.  arXiv:2304.12306 [eess.IV]
  * Maquiling et al. (2023) Virmarie Maquiling, Sean Anthony Byrne, Marcus Nyström, Enkelejda Kasneci, and Diederick C Niehorster. 2023.  V-ir-Net: A Novel Neural Network for Pupil and Corneal Reflection Detection trained on Simulated Light Distributions. In _Proceedings of the 25th International Conference on Mobile Human-Computer Interaction_. 1–7.
  * Mattjie et al. (2023) Christian Mattjie, Luis Vinicius de Moura, Rafaela Cappelari Ravazio, Lucas Silveira Kupssinskü, Otávio Parraga, Marcelo Mussi Delucis, and Rodrigo Coelho Barros. 2023.  Zero-shot performance of the Segment Anything Model (SAM) in 2D medical imaging: A comprehensive evaluation and practical guidelines.  arXiv:2305.00109 [cs.CV]
  * Mazurowski et al. (2023) Maciej A Mazurowski, Haoyu Dong, Hanxue Gu, Jichen Yang, Nicholas Konz, and Yixin Zhang. 2023.  Segment anything model for medical image analysis: an experimental study.  _Medical Image Analysis_ 89 (2023), 102918.
  * Milletari et al. (2016) Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. 2016.  V-net: Fully convolutional neural networks for volumetric medical image segmentation. In _2016 fourth international conference on 3D vision (3DV)_. Ieee, 565–571.
  * Nair et al. (2020) Nitinraj Nair, Rakshit Kothari, Aayush K Chaudhary, Zhizhuo Yang, Gabriel J Diaz, Jeff B Pelz, and Reynold J Bailey. 2020.  RIT-Eyes: Rendering of near-eye images for eye-tracking applications. In _ACM Symposium on Applied Perception 2020_. 1–9.
  * Palmero et al. (2021) Cristina Palmero, Abhishek Sharma, Karsten Behrendt, Kapil Krishnakumar, Oleg V Komogortsev, and Sachin S Talathi. 2021.  Openeds2020 challenge on gaze tracking for vr: Dataset and results.  _Sensors_ 21, 14 (2021), 4769.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning Transferable Visual Models From Natural Language Supervision.  arXiv:2103.00020 [cs.CV]
  * Romera-Paredes and Torr (2015) Bernardino Romera-Paredes and Philip Torr. 2015.  An embarrassingly simple approach to zero-shot learning. In _Proceedings of the 32nd International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 37)_ , Francis Bach and David Blei (Eds.). PMLR, Lille, France, 2152–2161.  https://proceedings.mlr.press/v37/romera-paredes15.html
  * Santini et al. (2018) Thiago Santini, Wolfgang Fuhl, and Enkelejda Kasneci. 2018.  PuRe: Robust pupil detection for real-time pervasive eye tracking.  _Computer Vision and Image Understanding_ 170 (2018), 40–50.
  * Shen et al. (2023) Qiuhong Shen, Xingyi Yang, and Xinchao Wang. 2023.  Anything-3d: Towards single-view anything reconstruction in the wild.  _arXiv preprint arXiv:2304.10261_ (2023).
  * Wang et al. (2023a) Bin Wang, Armstrong Aboah, Zheyuan Zhang, and Ulas Bagci. 2023a.  Gazesam: What you see is what you segment.  _arXiv preprint arXiv:2304.13844_ (2023).
  * Wang et al. (2023b) Di Wang, Jing Zhang, Bo Du, Dacheng Tao, and Liangpei Zhang. 2023b.  Scaling-up remote sensing segmentation dataset with segment anything model.  _arXiv preprint arXiv:2305.02034_ (2023).
  * Yang et al. (2023) Jinyu Yang, Mingqi Gao, Zhe Li, Shang Gao, Fangjing Wang, and Feng Zheng. 2023.  Track Anything: Segment Anything Meets Videos.  arXiv:2304.11968 [cs.CV]
  * Zhou et al. (2023b) Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et al. 2023b.  A comprehensive survey on pretrained foundation models: A history from bert to chatgpt.  _arXiv preprint arXiv:2302.09419_ (2023).
  * Zhou et al. (2019) Dingfu Zhou, Jin Fang, Xibin Song, Chenye Guan, Junbo Yin, Yuchao Dai, and Ruigang Yang. 2019.  Iou loss for 2d/3d object detection. In _2019 international conference on 3D vision (3DV)_. IEEE, 85–94.
  * Zhou et al. (2022) Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. 2022.  Domain generalization: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ (2022).
  * Zhou et al. (2023a) Yukun Zhou, Mark A Chia, Siegfried K Wagner, Murat S Ayhan, Dominic J Williamson, Robbert R Struyven, Timing Liu, Moucheng Xu, Mateo G Lozano, Peter Woodward-Court, et al. 2023a.  A foundation model for generalizable disease detection from retinal images.  _Nature_ (2023), 1–8.