# Modeling the Sacred: Considerations when Using Religious Texts
in Natural Language Processing

Ben Hutchinson
Google Research, Australia
<EMAIL_ADDRESS>
###### Abstract

This position paper concerns the use of religious texts in Natural Language
Processing (NLP), which is of special interest to the Ethics of NLP. Religious
texts are expressions of culturally important values, and machine learned
models have a propensity to reproduce cultural values encoded in their
training data. Furthermore, translations of religious texts are frequently
used by NLP researchers when language data is scarce. This repurposes the
translations from their original uses and motivations, which often involve
attracting new followers. This paper argues that NLP’s use of such texts
raises considerations that go beyond model biases, including data provenance,
cultural contexts, and their use in proselytism. We argue for more
consideration of researcher positionality, and of the perspectives of
marginalized linguistic and religious communities.

##  1 Introduction

> _“a particular string of speech may be viewed as data by a researcher but as
> sacred incantation by language users”_
> — Holton et al. (2022)

The Association for Computational Linguistics (ACL) is a secular institution.
Its constitution, resolutions and policies make no mention of religion other
than forbidding harassment on the basis of religion.111www.aclweb.org,
accessed September 2023 Nevertheless the Christian Bible and the Islamic
Quran222This paper follows several style guides in using “Quran”, although
mentions of the alternate Latinization “Koran” are also considered in the
corpus studies we report on. are often used in the scientific and professional
activities of ACL, as measured by papers published in the ACL Anthology (see
Section 3). Some of the reasons that NLP researchers use the Bible are aptly
expressed by Resnik et al. (1999). The Bible is the world’s most translated
book, with translations in over 2,000 languages, and often multiple
translations per language. Furthermore, great care is taken with the
translations, so from an NLP perspective data quality is high. It is often
easily available in electronic form, and is in the public domain, hence free
to use. It has a standard structure which allows parallel alignment verse-by-
verse. For these reasons, as recently as 2006 it was said to be “perhaps
surprising that the Bible has not been more widely used as a multilingual
corpus by the computational linguistics and information retrieval community”
Chew et al. (2006).

Despite the use of religious texts in NLP, the ethical considerations of such
use have not previously received much attention. Religious texts have
generally been treated as just data, overlooking their sacred dimensions,
their cultural significance, and their histories, which are sometimes entwined
with colonial projects. This position paper contends that responsible
secularism demands engaging with the ethical considerations of the use of
these texts.

Religion | Sacred texts | Est. 2020 population | Proselytizing
---|---|---|---
Christianity | Bible, New Testament, Old Testament | 2,382,750,000 | Yes
Islam | Quran (alt. spellings include Koran), Old Testament | 1,907,110,000 | Yes
Hinduism | Vedas, Upanishads, Puranas | 1,161,440,000 | No
Buddhism | Tripitaka, Mahayana Sutras, Tibetan Book of the Dead | 506,990,000 | Yes
Traditional Chinese Religion | Zhuangzi, Tao-te Ching, Daozang | 310,000,000 | No
Judaism | Talmod, Torah, Tanakh, Old Testament | 14,660,000 | No
Table 1: Some major world religions and their texts. Population estimates are
from the US-based Pew Research Center (www.pewresearch.org), which conducts
demographic and other research.

We take the position that it is important to contextualize religious texts
within not just their religious but also their broader cultural and historical
contexts. We argue that it is important to acknowledge the motivations, goals,
and positionalities of researchers using religious texts, particularly as it
is difficult to have intimate first-hand knowledge of the moral and cultural
concerns of a religion beyond one’s own (if any).

Section 2 provides relevant background material, including discussing the
relationship between academic linguistics and missionary linguistics. In
Section 3, we present an empirical study of NLP papers using sacred texts.
Section 4 discusses various ethical considerations when using sacred texts in
NLP. In doing so, we consider a range of approaches to the topic, including
ethical theories, Indigenous perspectives,333Following style guides such as
Younging (2018), this paper capitalizes the first letter of “Indigenous”.
human rights, and the AI principles commonly espoused by institutions. Our
goal in doing so is not to evaluate past NLP projects that use religious
texts, but rather to encourage more reflecting in and on future work. Based on
these considerations, we then make some recommendations for the NLP community
in Section 5.

##  2 Background

###  2.1 Religion

Precisely defining what constitutes a religion might be notoriously difficult
(see, e.g., Spiro, 2013; Neville, 2018), and lies beyond the scope of this
paper. Typical properties of religions center around giving meaning to
existence, and include moral values, spiritual beliefs, theistic beliefs,
rituals, stories and mythologies, kinship systems and marriage practices,
artistic practices, significant locations, and in some cases a language which
plays a special role. These are all closely related to questions concerning
values.

Like languages, religions may exhibit regional variation and incorporate local
practices, so thinking of them as discrete entities may be somewhat
misleading. Some widely cited estimates put the number of worldwide religions
at several thousand, although these claims are disputed. What seems more
certain is that the imminent extinction of thousands of Indigenous languages
will accompany an “impending loss of so many religions and worldviews”
(Harrison, 2007, p. 153). Acknowledging these challenges, we nevertheless
provide a summary of some of the world’s most populous religions in Table 1.
Surveying or defining each of these religions is beyond the scope of this
paper. Also not included here are the various sects and branches within each
religion, nor texts which might be important only to specific branches.

One important distinction is that between proselytizing and non-proselytizing
religions. The former attempt to convert new populations, whereas the latter
do not. The former are more intricately related to historical practices of
colonialism—especially in Africa, the Americas, Asia, Australia, and the
Pacific—and hence also to neocolonial legacies. Some religions hold that a
certain language is privileged for communicating sacred texts to the faithful,
while on the other hand Protestant Christianity exemplifies a commitment to
communicating in vernacular languages. (Article XXIV of the Articles of
Religion of the Anglican Church calls for “such a Tongue as the people
understandeth”.) Although originally in Ancient Greek, Christian texts have
been widely distributed in many European languages since the Protestant
Reformation, and the global reach of Christianity is associated with European
colonial practices.

###  2.2 The academy and Bible translation

The September 2009 issue of the journal _Language_ has a special feature of
five articles by anthropologists and linguists concerning the relationships
between the US-based Bible translation organization SIL International (SIL)
and academic linguistics. In this issue, Dobrin and Good (2009) explore how
academic linguists have at times become reliant on, and benefitted from, the
technological infrastructures of SIL, in part because creating and maintaining
these infrastructures have not been valued by the academy. This “partnership
of convenience” causes tensions between differing objectives, and raises
questions about what kind of relationships secular research institutions ought
to have with organizations with very different agendas. These practices
presage similar ways in which some areas of NLP research have become reliant
on Bible translations.

Many linguists and NLP practitioners working on Indigenous languages see their
research as addressing issues of human rights and cultural extinction.
However, as Dobrin and Good point out, languages which are most endangered are
least likely to receive SIL’s attention. Handman (2009) draws attention to how
SIL ideology separates linguistic identity from religious identity, differing
from unesco’s position that sustaining endangered languages entails sustaining
cultural worldviews, knowledge systems, and identity practices. Epps and
Ladley (2009) argue that evangelical success entails the displacement or
transformation of traditional beliefs, often leading to social upheaval, and
argues that the academy has a moral interest in supporting local self-
determination which is at odds with evangelical agendas.

##  3 The Use of Sacred Texts in NLP

This section presents empirical data on the use of religious texts in the
field of NLP.

###  3.1 Sacred texts in the ACL Anthology

In this section we consider four research questions: i) _which_ sacred texts
are used in NLP?, ii) how does NLP _characterize_ those sacred texts?, iii)
for what purposes does NLP _use_ sacred texts?, and iv) are there any _trends_
over time? To explore these, we use the Anthology of the Association of
Computational Linguistics444www.aclanthology.org, accessed August 2023.
(henceforth, the ACL Anthology) Bird et al. (2008), which we consider to
represent the research publications of the NLP community.

Regarding _which_ sacred texts are used in NLP, the number of ACL Anthology
entries for sacred texts are shown in Table 2.555Non-deterministic result
counts seem to be an artefact of the ACL Anthology’s use of Google’s
Programmable Search Engine Bollmann et al. (2023). Thousands of papers in the
ACL Anthology seem to use religious texts. There is a strong bias towards the
texts of the Judaism, Christianity, and Islam.

Search term/phrase | Results [min, max]
---|---
bible | [1920, 3890]]
quran | [291, 547]
new testament | [294, 294]
koran | [131, 248]
old testament | [73, 206]
torah | [25, 153]
talmud | [21, 22]
vedas | [22, 51]
tripitaka | [7, 7]
upanishads | [6, 6]
mahayana sutras | [4, 4]
tanakh | [3, 4]
zhuangzi | [3, 3]
puranas | [3, 3]
tibetan book of the dead | [0, 0]
tao-te ching | [0, 0]
daozang | [0, 0]
Table 2: Number of search results for religious texts in the ACL Anthology on
August 10, 2023. Since search result counts are non-deterministic, we report
the min and max of 10 searches for each term.

To understand how sacred texts are _characterized_ in NLP, we analysed ACL
Anthology papers containing the phrases “the Bible is” or “the Quran is”.
Thematic analysis of the results reveal that descriptions of the Bible mostly
emphasize its availability, convenience, size, and multilingual character (see
Appendix A). The religious nature of the Bible is only mentioned infrequently,
in stark contrast with how Christian websites describe the Bible. These
findings agree with the claim that NLP has a dominant ideology of “language as
data” Bird and Yibarbuk (2024). One paper describes the Bible as “one of the
most familiar documents”. NLP papers are more likely to mention religious
aspects of the Quran. No ACL anthology papers mention “the Koran is”, nor
related appositions “Koran, a[n]”; presumably the papers mentioning ‘Koran’
felt it needed no explanation.

To understand how NLP _uses_ sacred texts, we examined the first 100 results
(sorted by relevance) from the ACL Anthology for each of the terms ‘bible’ and
‘koran’. We manually looked at each of the 200 search results in order to
verify its relevance to our study, and omitted 48 search results for not being
relevant (see Appendix B). We manually coded the 88 papers mentioning the
Bible for their application domain. The most common application domain was
machine translation (23%), while many (18%) introduced a new corpus or lexical
resource. Three papers were concerned specifically with Bible translation, and
four with literary analysis of the Bible. A variety of other application
domains were represented, including various morpho-syntactic analysis tasks,
and language modeling or pretraining (see Appendix C). 48% of the 88 papers
concerned one or more Indigenous languages.666These categorizations were done
by the author, taking into account historical and social context, however an
ideal approach might engage with language communities to understand whether
they consider themselves to be Indigenous. The 64 ACL papers mentioning
‘koran’ were less varied and largely (73%) were concerned with machine
translation, typically using verses from the Quran in evaluation datasets (see
Appendix D).

To get a sense of _trends_ in the use of sacred texts by the NLP community, we
examined the year of publication of the 150 of the 152 papers described above
which were published between Jan 2004 and July 2023 (just 2 were published
prior to 2004). Of these, we see increasing use over time of mentions of the
Bible, and especially of mentions of ‘Koran’ (see Figure 1)777Since our 2023
sample was limited to Jan-July, paper counts for 2023 are multiplied by $12/7$
when creating sparklines, to make them comparable with previous years.,
including over 60% of papers being published between 2019 and 2023 alone. Of
course, the number of papers published in the ACL anthology has also been
increasing, for example ACL 2023 accepted about 12 times as many papers as ACL
2004 did. We are not necessarily seeing an increase in terms of the fraction
of NLP papers using the Bible, and thus there may only be weak evidence of
trends _within_ NLP research. We do however hypothesize that the absolute
counts of papers using religious texts may be a better proxy for the
_externalities_ (both positive and negative) of NLP research on linguistic and
religious communities, and thus relevant to this paper’s topic of ethical
considerations. There also seems to be a trend towards papers using sacred
texts handling very large numbers of languages, with ten papers since 2014 in
our sample handling over 500 languages.

Figure 1: Trend sparklines for counts of papers in the ACL Anthology
mentioning ‘bible’ or ‘koran’ (see §3).

###  3.2 Four case studies

To complement the broad analysis of NLP research above, we also report here on
some recent noteworthy cases which have attracted acclaim and attention. These
illustrate in more detail ways in which the NLP community is encountering and
using religious texts.

Our first example is a paper which aims to improve Speech Recognition and
Speech Synthesis for over a thousand languages Pratap et al. (2023). It was
uploaded to arXiv, a popular online archive for computer science papers, in
May 2023. The researchers train their model using translations of the New
Testament, as well as audio of readings of those translations, obtained from
Faith Comes by Hearing (faithcomesbyhearing.com), goto.bible and bible.com.
They also use spoken recordings in many languages, without paired texts, of
Bible stories, evangelistic messages, scripture readings, and songs, obtained
from Global Recordings Network (globalrecordings.net), whose mission is to
communicate “the Good News of Jesus Christ” via a strategy of recording,
distribution, and promotion.

Our second example aims to scale language models to 500 languages Imani et al.
(2023), and was awarded the ACL Area Chair Award for best Multilingualism and
Cross-Lingual NLP paper in July 2023. The researchers “crawl or download” data
from 150 sources, including religious texts and observe a “higher proportion
of religious data” compared to previous comparable work. Parallel verses from
Bible translations are used for model training and testing, and performance is
reported for Sentence Retrieval from the Bible.

Our third example concerns the use of JW300 Agić and Vulić (2019), a dataset
of around 100k sentences in each of 300 languages crawled from jw.org, a
website run by the US-based Jehovah’s Witnesses, a Christian denomination. A
majority of the texts come from the Jehovah’s Witnesses’ magazines _Awake!_
and _Watchtower_. Released as a corpus in 2019, JW300 has been cited over 180
times as of March 2024. The African grassroots open-source NLP project
Masakhane (masakhane.io) had been using JW300 to train Machine Translation
models, until receiving legal advice in 2023 that this was breaching
copyright. A subsequent request by Masakhane to the Jehovah’s Witnesses for
permission to use the data was declined.888See
https://infojustice.org/archives/45258.

Our final recent example concerns the release of madlad-400, a new text
dataset containing 3T tokens in 419 languages Kudugunta et al. (2023). It uses
2022 snapshots of the CommonCrawl web crawl (commoncrawl.org) and the paper
was uploaded to arXiv in September 2023. Auditing of a preliminary version of
the dataset, spanning 498 languages, revealed that for 141 languages there
were “significant amounts” of Bible data. Significant amounts of Jehovah’s
Witnesses data was also found for 37 languages, and of Church of Jesus Christ
of the Latter Day Saints (LDS) data for 2 languages. (No Quran data was
reported to be found in significant amounts.)

##  4 Considerations around Sacred Texts

Having demonstrated above that religious texts have been used in thousands of
NLP papers, we now discuss some of the ethical considerations. This topic has
not received much prior attention, although it is touched upon by Mager et al.
(2023, sec. 3) and Pratap et al. (2023, sec. 8). Our goal is not to critique
the works identified in the previous section, but rather to provide a toolbox
for assisting critical thinking in the future. In this section we lay out our
positions that relevant considerations include not just technological
outcomes, but also cultural standpoints, cultural inclusion, and relationships
between global and marginalized cultures. With this in mind, we now consider
the use of sacred texts in NLP from a range of lenses.

###  4.1 Ethical theories

Ethical theories are attempts at explaining why actions might be called right
or wrong. In this section we consider the use of sacred texts through just two
ethical theories, however our broader point is that if analyses of NLP
projects focus narrowly on using a single theory then they risk minimizing
concerns which other theories bring more focus to.

The term consequentialism refers to a family of normative theories which
emphasize the importance of considering the consequences of actions, including
analysis of risks and benefits. One example of a consequentialist theory is
utilitarianism, which holds that the best action is one that maximizes
wellbeing and minimizes suffering. One challenge with applying these theories
in an NLP research context is that in practice research activities are often
far removed from applications, and any eventual path between the two can be
unknowable or uncertain. Even if the impacts on users of NLP applications can
be known, challenges arise in calculating aggregate benefits across disparate
stakeholders with different objectives. Perhaps the easiest consequences to
reason about are the impacts of NLP research and applications on the lives of
the researchers and developers themselves, since research papers and software
bring rewards in the forms of measurable kudos, citations, career progression,
etc. How can such benefits be weighed against uncertain risks of causing
serious offense to religious communities? If translations of religious texts
are purchased from a proselytizing organization, what are the downstream
impacts of the organization’s re-investing those payments into their projects?

AI Principle |  Example possible considerations
---|---
Safety |  Could the errors made by an MT model trained on sacred texts perpetuate harmful misinformation and hate speech about religiously significant topics?
Privacy |  In speech-to-speech translation, could spoken recordings of religious texts (cf. Section 3.2) enable speakers to be identified and their religious beliefs compromised?
Bias and fairness |  In what specific ways are religious texts not representative of other domains (cf. Mayhew et al., 2017; Adelani et al., 2021; Evans and Sasse, 2004)? Do these biases cause MT models to have unfair impacts? What are the specific limitations of MT eval sets which are largely constituted by religious texts?
Accountability |  What mechanisms exist for non-English speaking religious communities to report and escalate offensive translations resulting from the use or religious texts? Who is responsible for responding to such reports and mitigating the harms?
Transparency |  Do we have accessible evidence that religious communities were consulted when building corpora of parallel religious texts to use for MT?
Table 3: Non-exhaustive list of possible AI Principles considerations related
to using religious texts for MT.

The term deontology refers to a normative theory which posits that there are
rules or principles which determine the rightness of wrongness of actions,
rather than the consequences. Within an NLP context, this might lead to a
focus on the upstream actions such as sourcing of data, rather than downstream
actions such as usage of NLP applications. Within the context of religious
texts and their translations, this might lead to questions such as: Were the
appropriate people involved in the creation of a dataset of sacred texts? Did
they have the right roles and relationships from within the perspective of
followers of the religion? Do translators of sacred texts have the right
specialist linguistic, religious, and and cultural knowledge? Were the
translators paid fairly, both individually and collectively? Were consent
practices around voice recordings followed? Generally, was the dataset
collected in a manner aligned with best research practices, e.g., as
operationalized by research ethics boards? Prabhumoye et al. (2020) discuss
the importance of informed consent for deontological approaches to NLP ethics,
and community-level consent might be an appropriate lens for thinking about
communities of religious practice.

###  4.2 AI principles

We use the term AI principles here to refer broadly to the hundreds of sets of
principles for responsible and ethical AI that have been released by companies
and governments in recent years. Unlike the ethical theories described above,
these are focused on AI technologies. Although all unique in their own way,
the various sets of principles also have many facets in common Floridi and
Cowls (2022), such as safety, privacy, bias, fairness, accountability, and
transparency. We expect the AI Principles-related concerns to be highly
dependent on NLP system functionality and thus on the application domain.

Given the frequent use of sacred texts for training and evaluating MT systems,
we provide a sketch in Table 3 of how AI Principles might raise different
flavors of considerations.999Mager et al. (2023) also provide a useful
discussion of Ethics and Machine Translation. Relevant to the potential for MT
bias, Evans and Sasse (2004, p. 4) observe that parallel Bibles will provide
signal on how to translate “arise!” and “Cain fought with Abel”, but will not
help with translating culture-specific concepts of the world’s diverse
languages. Similarly, MT evaluation will have limitations whenever eval sets
are largely constituted by parallel religious texts.

###  4.3 Cultural considerations

Authors of NLP publications do not represent global diversity Rungta et al.
(2022), and hence special consideration needs to be given to cultural
standpoints. A useful starting premise might be that ethical consideration of
ACL papers should not be biased either against nor towards any religions. This
might seem to suggest that Rawls’ Veil of Ignorance could provide a guide
Rawls (1971). That is, one’s consideration of the use of sacred texts within
NLP should proceed as if we are each in ignorance of which is our own religion
(if any). For example, how would I feel about NLP’s reliance on texts from
major religions if my own culture and religion might be ones which are
marginalized and endangered?

Although the Veil of Ignorance might seem attractive at face value, we take
the position that our ability in practice to truly avoid being informed by our
own cultural backgrounds and affiliations is extremely questionable. We must
instead accept that we have cultural standpoints. The emic/etic distinction
originated in linguistics in the 1950s for describing different standpoints
for language research Mostowlansky and Rota (2020). _Emic_ is commonly used to
describe research on a culture from the perspective of people of that culture.
This contrasts with _etic_ research, which takes an outsider’s perspective.
When NLP handles religious texts, we can distinguish research problems and
applications which are within the researcher’s own religious context, from
those applications which impact those having other religious beliefs (e.g.,
translation for the purpose of proselytizing).

####  4.3.1 On relativism and the ACL community

One possible objection to the argument that greater consideration of NLP’s use
of sacred texts is needed is based on an argument of cultural relativism. Such
an argument would contend that by using sacred texts for an extended period,
the ACL community has demonstrated that such practices are judged as
acceptable by the norms of the ACL community.

We counter that such an argument would be stronger if the ACL community both
had a stronger history of reflexive practices, and was more culturally
diverse. Compared to many other disciplines, we find that ACL’s interest in
ethics to be relatively recent (see Appendix E). The ACL adopts the ACM Code
of Ethics,101010https://www.aclweb.org/portal/content/acl-code-ethics a
general code for computing professionals which makes no mention of working
with cultural data such as language. Unlike some other disciplines,
positionality statements are rare in ACL papers, with “researcher
positionality” and “author positionality” each having only a single (and
recent) result in the ACL Anthology as of September 2023. We also find that
the ACL community does not represent the diversity of the world’s local
languages and religions; in fact disparities in sources of ACL publications
might be increasing Rungta et al. (2022).

###  4.4 Power and marginalization

The more different that a culture is from the global cultures which dominate
NLP research, the less we should expect the NLP community to understand local
concerns. Consideration of marginalized local cultures might bring our focus
to questions of how NLP technologies serve to re-arrange power. Which actions
are encouraged or discouraged by NLP technologies, and by who? Blodgett has
called for NLP researchers to focus their ethical considerations on power
relations between technologists and communities Blodgett et al. (2020). As
discussed in Section 3, many NLP papers using the Bible use translations of
its texts into Indigenous languages. In this section we give special attention
in this section to concerns of local Indigenous communities. Their voices are
among the least likely to be represented in ACL’s prestigious conferences and
journals. And their relationships with global projects are likely to involve
large power disparities such that “the asymmetry of power is the cause of
domination” Mager et al. (2023).

####  4.4.1 Indigenous perspectives

Indigenous concerns regarding data sovereignty describe the importance of data
governance and Indigenous Cultural and Intellectual Property (ICIP), but also
a broader set of concerns regarding practices for projects involving
Indigenous data (e.g., Carroll et al., 2020; National Health and Medical
Research Council (2018), Australia; Taiuru, 2021; Janke, 2021; Smith, 2021).
Surveying the plurality of perspectives is beyond the scope of this paper,
however they share common themes around respect, cooperative relationships
which include consultation and negotiation, shared control, and providing
benefits to community Mager et al. (2023); Cooper et al. (2024).

####  4.4.2 Human rights

As tools of colonizing projects, translations of sacred texts into Indigenous
languages have been described as a “well documented example of the non-ethical
misuse of translation” Mager et al. (2023). One consideration for NLP is
whether using such translations constitutes complicity with, and promotion of,
projects which might violate international human rights to maintain Indigenous
cultures.

Kenyan human rights scholar Makua Mutua describes the “basic contradictions”
between proselytizing religions and Indigenous cultures Mutua (2004).
Observing that religion is woven into every aspect of Indigenous social and
cultural life, including dances, storytelling, and marriage practices, Mutua
argues that such cultures’ meeting with proselytizing Christian and Islam
faiths amounts to cultural genocide. In some cases, this characterization
seems valid. In other cases, it might perhaps be too strong for the complex
realities of syncretic responses by Indigenous communities to proselytizing
cultures.

Mutua argues that the right to freedom of religious belief cannot be
considered to exist in a level playing field in which local cultures can
compete with global ones. Rather, the contexts of cultural invasion unfairly
privilege global religions, including histories of missionaries making access
to education and health services conditional on the “salvation” of “infidels”.
This echoes arguments by legal scholars that the power and sovereignty
dynamics between source and target cultures constitute important factors
between proper and improper proselytism Stahnke (1999). The Human Rights
Committee has acknowledged that the cultural rights protected under Article 27
depend on the ability of a minority group “to maintain its culture, language
or religion”.111111General Comment No2̇3, UN Doc. ICCPR/C/21/Rev.1/Add.5
(1994), para. 6.2. As such, Prof. Mutua argues that the (then Draft) UN
Declaration on the Rights of Indigenous People appears to prohibit
proselytizing by agents external to the the Indigenous culture in order to
create space for Indigenous peoples to maintain their cultures amidst external
threats.

##  5 Discussion and Recommendations

We take the position that the ACL should strive to be aware of risks of harms
to religious communities, including dignitive harms, that NLP research may
cause or be complicit with. It is beyond the scope of this paper to weigh
possible benefits of NLP technologies developed using religious texts against
the ethical considerations of such use, and any such benefits may be context-
and application-specific. However, we note that benefits are often asserted
for languages for which NLP has few resources, and we advocate that such
claims of benefits should be evaluated paying attention to local language
ecologies Bird (2022), to the roles of technology in supporting minority
languages Holton (2011), and to the real technology needs of communities
(e.g., Liu et al., 2022).

We now suggest five specific actions for the NLP community to mitigate the
risks of consequential, procedural, and dignitive harms.

##### 1\. Discuss ethical considerations.

If one should never speak of religion in polite company, then perhaps ACL
forums should be less polite. We suggest that discussion of ethical
considerations in NLP papers using religious data should be much more
prevalent than it currently is. We propose that Section 4 provides a good
starting point for NLP researchers to think about possible concerns, however,
building on HCI best practices we also encourage researchers to talk to
members of relevant religious communities in order to understand concerns
which might lie outside the researchers’ domains of lived experience.

##### 2\. Consider different ethical theories.

It has been argued that NLP research often implicitly adopts a utilitarian
lens, in the process minimizing other ethical considerations (e.g., Hutchinson
et al., 2022b). We encourage NLP researchers using sacred texts to contemplate
approaching ethical considerations from numerous angles, including not just
deontology (discussed in Section 4.1), but also alternatives such as virtue
ethics and ethics of care.

##### 3\. Delve into domain specifics.

We encourage more work on specific AI principle risks of different NLP domains
when using religious texts (cf. Section 4.2). What are the specific ways in
which various religious texts are not representative of other domains? What
are the model biases that result? Are there specific risks around offensive
misinformation on religious topics? Have datasets incorporating sacred texts
followed best practices around transparency (e.g. Bender and Friedman, 2018;
Gebru et al., 2021; Pushkarna et al., 2022)?

##### 4\. Situate NLP work culturally.

When should an NLP practitioner tread more carefully? We suggest that when
handling sacred data from other cultures and religions there will be more
risks, since the NLP practitioner will have almost certainly have gaps in
their knowledge, and will likely also have differing values Hershcovich et al.
(2022); Prabhakaran et al. (2022b). We suggest it is useful to talk of _etic
NLP_ and _emic NLP_ (cf. Section 4.3), according to whether the language
technology is for the own linguistic and religious cultures of the NLP
practitioner vs. those of others. Linguistic positionality statements have
been recommended by researchers who are aware of how different priorities and
agendas between researchers and language communities can impact projects
(e.g., Rolland et al., 2023; Cormier, 2018). Similarly, we suggest that
religion positionality statements, for NLP research working with religious
data, can also provide a useful signal for the NLP community concerning
agendas. For example, hypothetically, would non-Muslim researchers understand
the Muslim community’s concerns regarding automated mis-translations of the
Quran?

##### 5\. Attend to concerns of marginalized cultures.

Given the NLP community’s skew towards the West Rungta et al. (2022), the
values of, and possible harms to, local communities of diverse cultures are
not known by most NLP researchers. For Indigenous language NLP projects using
translations of sacred texts, we echo calls for more consideration of local
colonial contexts, to consider community opinions, and for research to
prioritize the needs of Indigenous communities Bird (2020); Schwartz (2022);
Alvarado Garcia et al. (2021). Mager et al. (2023) demonstrate one way in
which community opinions can be sought regarding NLP projects, and deeper
relationships with communities will provide more insights Bird and Yibarbuk
(2024); Cooper et al. (2024). NLP researchers working with Indigenous
languages should become familiar with Indigenous perspectives mentioned in
Section 4.4, and with codes of conduct such as that of the Endangered
Languages Project.121212https://fpcc.ca/wp-
content/uploads/2023/02/CodeOfConduct_Web.pdf With the imminent extinction of
many Indigenous languages and religions, we also suggest there may be a role
for NLP to play in documenting and maintaining “first-person accounts of what
people once believed in and how they talked to and about their gods”
(Harrison, 2007, p, 153).

One pertinent question is how often the papers surveyed in Section 3 adhere to
these recommendations. Regarding discussing ethical considerations, Pratap et
al. (2023, sec. 8.3) do include a section, albeit short, on Ethical
Considerations of the use of religious texts, although we found that doing so
is rare. Kudugunta et al. (2023) and Imani et al. (2023) both have short
ethics statements, but neither mentions religion. Despite translation of the
Quran being a “problematic and controversial issue for Muslims” Fatani (2006),
none of the many papers surveyed in Section 3.1 that evaluate MT systems
discuss ethical considerations.

Regarding ethical theories, papers using religious texts do not seem to break
the general NLP trend of adopting a utilitarian mindset Hutchinson et al.
(2022b) and focusing ethical discussion on narrow questions such as
decontextualized bias measurement Blodgett et al. (2020).

Regarding domain specifics, Pratap et al. (2023) attempt to understand biases
in their data, although the details are scarce, and their comparison is
limited to a dataset based on English language Wikipedia. Kudugunta et al.
(2023) and Imani et al. (2023) have no discussions of biases, although the
latter do include a datasheet Gebru et al. (2021).

Regarding situating work culturally, positionality statements are extremely
rare in the ACL Anthology papers. Some rare examples of NLP papers mentioning
author positionality or standpoint are McMillan-Major et al. (2021);
Hutchinson et al. (2022a); Santy et al. (2023); Yoder et al. (2023); Ghosh and
Caliskan (2023). Only half of ACL papers even mention the language(s) they
study Ducel et al. (2022), giving a false veneer of independence Bender
(2019).

Regarding attending to concerns of marginalized cultures, this is rarely done
in NLP research. NLP papers that consider ethical topics tend to focus on
English, and most papers on languages other than English focus on model
accuracy Ruder et al. (2022). Pratap et al. (2023) mention how their bias
analysis was constrained by lack of easily available native speakers for most
languages, which also precluded the possibility of discovering community
concerns beyond bias.

##  6 Conclusion

Thousands of NLP research papers have used religious texts, due to their
availability, convenience, and multilinguality. However these papers rarely
acknowledge the religious significance of these texts, nor their cultural
contexts, and the ethical considerations of such use have not previously been
explored. This position paper presented the first detailed study of the use of
religious texts in NLP research, finding that common scenarios include machine
translation and dataset creation. We have argued that responsible secularism
requires the NLP community to engage with concerns about how such NLP
activities might impact religious communities—especially the most marginalized
ones—or might be complicit with projects which do. We provided a detailed
account of some of the considerations, with a focus on ethical theories, AI
principles, cultural considerations, and marginalized communities. We
suggested that the field of NLP would benefit from more discussion of the
diverse and specific concerns around using religious texts. We proposed that
the NLP community engage more with questions of researcher positionality and
cultural standpoints, especially with regards to marginalized cultures.

##  7 Researcher Positionality

I live and work in a secular, multicultural, majority English-speaking,
colonized country of the Global North. I am linguistically competent only in
English. My experiences include studies and employment in linguistics and NLP,
but I have no background in religious studies. I grappled while writing this
paper with my lack of first-hand experiential understanding of religion, and
thus too with my personal role in arguing for more consideration of global and
marginalized religious communities.

## Limitations

Any position paper is limited by the experiences of the authors. A Researcher
Positionality statement is included partly to address this.

We acknowledge that the ACL Anthology used in Section 3 might not be
representative of the entire field of NLP, e.g., missing relevant work such as
Chandra and Ranjan (2022); Bashir et al. (2023), and NLP projects in industry
might not be well represented in the ACL Anthology. Another limitation is that
we exclude publications in languages other than English. Our reliance on
keyword searches, rather than more sophisticated analyses of the ACL corpus,
is a limitation. We also acknowledge that the ACL Anthology’s non-determinate
search result counts are a limitation, seemingly an artefact of the use under
the hood of Google’s Programmable Search Engine, however reproducing the ACL
Anthology’s search functionality is beyond the scope of this position paper.

This paper was unable to accommodate lengthy discussion of religious texts in
specific NLP domains or specific NLP projects. As the topic of religious texts
in NLP has not previously received attention, we took the position that
focusing on breadth was most useful. However in Section 5 we do suggest that
future work focus more on ethical considerations of using religious texts in
specific NLP domains.

This paper does not do justice to the complexities of many of the topics it
touches upon. Perhaps chief among these are the histories of Indigenous
communities, proselytizing religions and colonial projects. Colonial
missionaries saw their own roles in a positive way despite the accompanying
cultural destruction Nakata (2007). However in recent decades, there have at
times been apologies from some church bodies for histories of religious
domination which marginalized Indigenous languages and religious practices
Bush (2015). Indigenous communities have at times, facing many colonial
pressures including threats of language extinction, used Bible translations as
tools for language maintenance (e.g., Davis, 2015). Cultural protocols can
also be maintained, for example spoken recordings of Bible translations in
Kanien’kéha are still subject to cultural practices around speakers who have
passed away Pine et al. (2022), while on Groote Eylandt, in Northern
Australia, the Anindilyakwa people reinterpreted Christianity of the missions
in their own ways leading to a “hybridisation of cultures” Rademaker (2014).

This paper also does not fully consider relevant topics concerning human
rights. While we agree with scholars such as Prabhakaran et al. (2022a) that
human rights provide a useful framework for considering AI ethics, we note
that agreement on how to interpret collective and cultural rights is not
universal; for some perspectives from human rights scholars see Jakubowski
(2016); Sanders (1991); Ramcharan (1993).

## References

  * Adelani et al. (2021) David I Adelani, Dana Ruiter, Jesujoba O Alabi, Damilola Adebonojo, Adesina Ayeni, Mofe Adeyemi, Ayodele Awokoya, and Cristina España-Bonet. 2021.  The effect of domain and diacritics in Yorùbá-English neural machine translation.  _Proceedings of Machine Translation Summit XVIII: Research Track_.
  * Agić and Vulić (2019) Željko Agić and Ivan Vulić. 2019.  JW300: A wide-coverage parallel corpus for low-resource languages.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3204–3210.
  * Alfano et al. (2018) Mark Alfano, Dirk Hovy, Margaret Mitchell, and Michael Strube. 2018.  Proceedings of the second acl workshop on ethics in Natural Language Processing.  In _Proceedings of the Second ACL Workshop on Ethics in Natural Language Processing_.
  * Alvarado Garcia et al. (2021) Adriana Alvarado Garcia, Juan F Maestre, Manuhuia Barcham, Marilyn Iriarte, Marisol Wong-Villacres, Oscar A Lemus, Palak Dudani, Pedro Reynolds-Cuéllar, Ruotong Wang, and Teresa Cerratto Pargman. 2021.  Decolonial pathways: Our manifesto for a decolonizing agenda in HCI research and design.  In _Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems_ , pages 1–9.
  * Bashir et al. (2023) Muhammad Huzaifa Bashir, Aqil M Azmi, Haq Nawaz, Wajdi Zaghouani, Mona Diab, Ala Al-Fuqaha, and Junaid Qadir. 2023.  Arabic natural language processing for Qur’anic research: A systematic review.  _Artificial Intelligence Review_ , 56(7):6801–6854.
  * Bender (2019) Emily Bender. 2019.  The #BenderRule: On naming the languages we study and why it matters.  _The Gradient_ , 14:34.
  * Bender and Friedman (2018) Emily M Bender and Batya Friedman. 2018.  Data statements for Natural Language Processing: Toward mitigating system bias and enabling better science.  _Transactions of the Association for Computational Linguistics_ , 6:587–604.
  * Bird (2020) Steven Bird. 2020.  Decolonising speech and language technology.  In _Proceedings of the 28th international conference on computational linguistics_ , pages 3504–3519.
  * Bird (2022) Steven Bird. 2022.  Local languages, third spaces, and other high-resource scenarios.  In _60th Annual Meeting of the Association for Computational Linguistics, ACL 2022_ , pages 7817–7829. Association for Computational Linguistics (ACL).
  * Bird et al. (2008) Steven Bird, Robert Dale, Bonnie J Dorr, Bryan R Gibson, Mark Thomas Joseph, Min-Yen Kan, Dongwon Lee, Brett Powley, Dragomir R Radev, Yee Fan Tan, et al. 2008\.  The ACL anthology reference corpus: A reference dataset for bibliographic research in computational linguistics.  In _LREC_.
  * Bird and Yibarbuk (2024) Steven Bird and Dean Yibarbuk. 2024.  Centering the speech community.  In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 826–839.
  * Blodgett et al. (2020) Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020.  Language (technology) is power: A critical survey of "bias" in NLP.  _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Bollmann et al. (2023) Marcel Bollmann, Nathan Schneider, Arne Köhn, and Matt Post. 2023.  Two decades of the ACL Anthology: Development, impact, and open challenges.  In _Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023)_ , pages 83–94.
  * Bush (2015) Peter G Bush. 2015.  The Canadian churches’ apologies for colonialism and residential schools, 1986-1998.  _Peace research_ , pages 47–70.
  * Carroll et al. (2020) Stephanie Russo Carroll, Ibrahim Garba, Oscar L Figueroa-Rodríguez, Jarita Holbrook, Raymond Lovett, Simeon Materechera, Mark Parsons, Kay Raseroka, Desi Rodriguez-Lonebear, Robyn Rowe, et al. 2020.  The CARE principles for Indigenous data governance.  _Data Science Journal_ , 19:43–43.
  * Chandra and Ranjan (2022) Rohitash Chandra and Mukul Ranjan. 2022.  Artificial intelligence for topic modelling in Hindu philosophy: Mapping themes between the Upanishads and the Bhagavad Gita.  _Plos one_ , 17(9):e0273476.
  * Chew et al. (2006) Peter A Chew, Steve J Verzi, Travis L Bauer, and Jonathan T McClain. 2006.  Evaluation of the Bible as a resource for cross-language information retrieval.  In _Proceedings of the workshop on multilingual language resources and interoperability_ , pages 68–74.
  * Cooper et al. (2024) Ned Cooper, Courtney Heldreth, and Ben Hutchinson. 2024.  “It’s how you do things that matters”: Attending to process to better serve Indigenous communities with language technologies.  In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 204–211, St. Julian’s, Malta. Association for Computational Linguistics.
  * Cormier (2018) Gail Cormier. 2018.  The language variable in educational research: An exploration of researcher positionality, translation, and interpretation.  _International Journal of Research & Method in Education_, 41(3):328–341.
  * Davis (2015) Jenny L Davis. 2015.  Intersections of religion and language revitalization.  _The Changing World Religion Map: Sacred Places, Identities, Practices and Politics_ , pages 1091–1101.
  * Dobrin and Good (2009) Lise M. Dobrin and Jeff Good. 2009.  Practical language development: Whose mission? _Language_ , 85(3):619–629.
  * Dodge et al. (2019) Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A Smith. 2019\.  Show your work: Improved reporting of experimental results.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing_.
  * Ducel et al. (2022) Fanny Ducel, Karën Fort, Gaël Lejeune, and Yves Lepage. 2022.  Do we name the languages we study? the #BenderRule in LREC and ACL articles.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 564–573.
  * Epps and Ladley (2009) Patience Epps and Herb Ladley. 2009.  Syntax, souls, or speakers? On SIL and community language development.  _Language_ , 85(3):640–646.
  * Evans and Sasse (2004) Nicholas Evans and Hans-Jürgen Sasse. 2004.  Searching for meaning in the Library of Babel: field semantics and problems of digital archiving.  Open Conference Systems, University of Sydney, Faculty of Arts.
  * Fatani (2006) Afnan Fatani. 2006.  Translation and the Qur’an.  In Oliver Leaman, editor, _The Qur’an: an encyclopaedia_ , pages 657––669.
  * Floridi and Cowls (2022) Luciano Floridi and Josh Cowls. 2022.  A unified framework of five principles for AI in society.  _Machine learning and the city: Applications in architecture and urban design_ , pages 535–545.
  * Gebru et al. (2021) Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2021.  Datasheets for datasets.  _Communications of the ACM_ , 64(12):86–92.
  * Ghosh and Caliskan (2023) Sourojit Ghosh and Aylin Caliskan. 2023.  ChatGPT perpetuates gender bias in machine translation and ignores non-gendered pronouns: Findings across Bengali and five other low-resource languages.  _Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society_.
  * Handman (2009) Courtney Handman. 2009.  Language ideologies, endangered-language linguistics, and Christianization.  _Language_ , 85(3):635–639.
  * Harrison (2007) K David Harrison. 2007.  _When languages die: The extinction of the world’s languages and the erosion of human knowledge_.  Oxford University Press.
  * Hershcovich et al. (2022) Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. 2022.  Challenges and strategies in cross-cultural NLP.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6997–7013, Dublin, Ireland. Association for Computational Linguistics.
  * Holton (2011) Gary Holton. 2011.  The role of information technology in supporting minority and endangered languages.  In Peter K Austin and Julia Sallabank, editors, _The Cambridge handbook of endangered languages_. Cambridge University Press.
  * Holton et al. (2022) Gary Holton, Wesley Y Leonard, and Peter L Pulsifer. 2022.  Indigenous peoples, ethics, and linguistic data.  In Andrea L. Berez-Kroeker, Bradley McDonnell, and Eve Koller, editors, _The Open Handbook of Linguistic Data Management_ , pages 49–60.
  * Hovy et al. (2017) Dirk Hovy, Shannon L Spruit, Margaret Mitchell, Emily M Bender, Michael Strube, and Hanna Wallach. 2017.  Proceedings of the first ACL workshop on ethics in Natural Language Processing.  In _Proceedings of the First ACL Workshop on Ethics in Natural Language Processing_.
  * Hutchinson et al. (2022a) Ben Hutchinson, Jason Baldridge, and Vinodkumar Prabhakaran. 2022a.  Underspecification in scene description-to-depiction tasks.  In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1172–1184, Online only. Association for Computational Linguistics.
  * Hutchinson et al. (2022b) Ben Hutchinson, Negar Rostamzadeh, Christina Greer, Katherine Heller, and Vinodkumar Prabhakaran. 2022b.  Evaluation gaps in machine learning practice.  In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_ , pages 1859–1876.
  * Imani et al. (2023) Ayyoob Imani, Peiqin Lin, Amir Hossein Kargaran, Silvia Severini, Masoud Jalili Sabet, Nora Kassner, Chunlan Ma, Helmut Schmid, André FT Martins, François Yvon, et al. 2023.  Glot500: Scaling multilingual corpora and language models to 500 languages.  _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
  * Jakubowski (2016) Andrzej Jakubowski. 2016.  _Cultural Rights as Collective Rights: an international law perspective_ , volume 7.  Brill.
  * Janke (2021) Terri Janke. 2021.  _True tracks: Indigenous cultural and intellectual property principles for putting self-determination into practice_.  University of New South Wales Press.
  * Kudugunta et al. (2023) Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Christopher A Choquette-Choo, Katherine Lee, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, et al. 2023.  MADLAD-400: A multilingual and document-level large audited dataset.  _arXiv preprint arXiv:2309.04662_.
  * Liu et al. (2022) Zoey Liu, Crystal Richardson, Richard Hatcher Jr, and Emily Prud’hommeaux. 2022\.  Not always about you: Prioritizing community needs when developing endangered language technology.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
  * Mager et al. (2023) Manuel Mager, Elisabeth Mager, Katharina Kann, and Ngoc Thang Vu. 2023.  Ethical considerations for machine translation of Indigenous languages: Giving a voice to the speakers.  _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers_.
  * Mayhew et al. (2017) Stephen Mayhew, Chen-Tse Tsai, and Dan Roth. 2017.  Cheap translation for cross-lingual named entity recognition.  In _Proceedings of the 2017 conference on empirical methods in natural language processing_ , pages 2536–2545.
  * McMillan-Major et al. (2021) Angelina McMillan-Major, Salomey Osei, Juan Diego Rodriguez, Pawan Sasanka Ammanamanchi, Sebastian Gehrmann, and Yacine Jernite. 2021.  Reusable templates and guides for documenting datasets and models for natural language processing and generation: A case study of the HuggingFace and GEM data and model cards.  In _Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)_ , pages 121–135, Online. Association for Computational Linguistics.
  * Mostowlansky and Rota (2020) Till Mostowlansky and Andrea Rota. 2020.  Emic and etic.  In Felix Stein, editor, _The Open Encyclopedia of Anthropology_. University of Cambridge.
  * Mutua (2004) Makau Mutua. 2004.  Proselytism and cultural integrity.  In _Facilitating Freedom of Religion or Belief: A Deskbook_ , pages 651–668. Brill Nijhoff.
  * Nakata (2007) Martin N Nakata. 2007.  _Disciplining the savages, savaging the disciplines_.  Aboriginal Studies Press.
  * National Health and Medical Research Council (2018) (Australia) National Health and Medical Research Council (Australia). 2018.  _Ethical conduct in research with Aboriginal and Torres Strait Islander Peoples and communities: Guidelines for researchers and stakeholders_.  National Health and Medical Research Council.
  * Neville (2018) Robert Cummings Neville. 2018.  _Defining religion: Essays in philosophy of religion_.  State University of New York Press.
  * Pine et al. (2022) Aidan Pine, Dan Wells, Nathan Brinklow, Patrick Littell, and Korin Richmond. 2022\.  Requirements and motivations of low-resource speech synthesis for language revitalization.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7346–7359.
  * Prabhakaran et al. (2022a) Vinodkumar Prabhakaran, Margaret Mitchell, Timnit Gebru, and Iason Gabriel. 2022a.  A human rights-based approach to responsible AI.  _arXiv preprint arXiv:2210.02667_.
  * Prabhakaran et al. (2022b) Vinodkumar Prabhakaran, Rida Qadri, and Ben Hutchinson. 2022b.  Cultural incongruencies in Artificial Intelligence.  In _Proceedings of the NeurIPS 2022 Workshop on Cultures in AI/AI in Culture_.
  * Prabhumoye et al. (2020) Shrimai Prabhumoye, Brendon Boldt, Ruslan Salakhutdinov, and Alan W Black. 2020\.  Case study: Deontological ethics in NLP.  _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * Pratap et al. (2023) Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, et al. 2023\.  Scaling speech technology to 1,000+ languages.  _arXiv preprint arXiv:2305.13516_.
  * Pushkarna et al. (2022) Mahima Pushkarna, Andrew Zaldivar, and Oddur Kjartansson. 2022.  Data cards: Purposeful and transparent dataset documentation for responsible AI.  In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_ , pages 1776–1826.
  * Rademaker (2014) Laura Rademaker. 2014.  Language and Australian Aboriginal history; Anindilyakwa and English on Groote Eylandt.  _History Australia_ , 11(2):222–239.
  * Ramcharan (1993) Bertrand G Ramcharan. 1993.  Individual, collective and group rights: History, theory, practice and contemporary evolution.  _International Journal on Minority and Group Rights_ , 1(1):27–43.
  * Rawls (1971) John Rawls. 1971.  _A theory of justice_.  Cambridge, MA: Harvard University Press.
  * Resnik et al. (1999) Philip Resnik, Mari Broman Olsen, and Mona Diab. 1999.  The Bible as a parallel corpus: Annotating the ‘book of 2000 tongues’.  _Computers and the Humanities_ , 33:129–153.
  * Rogers et al. (2021) Anna Rogers, Timothy Baldwin, and Kobi Leins. 2021.  ‘Just what do you think you’re doing, Dave?’ a checklist for responsible data use in NLP.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 4821–4833, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Rolland et al. (2023) Louise Rolland, Hannah M King, and Pernelle Lorette. 2023.  Methodological implications of participant and researcher multilingualism: making language dynamics visible.  _Journal of Multilingual and Multicultural Development_ , 44(8):645–656.
  * Ruder et al. (2022) Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2022.  Square one bias in NLP: Towards a multi-dimensional exploration of the research manifold.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 2340–2354, Dublin, Ireland. Association for Computational Linguistics.
  * Rungta et al. (2022) Mukund Rungta, Janvijay Singh, Saif M Mohammad, and Diyi Yang. 2022.  Geographic citation gaps in NLP research.  _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_.
  * Sanders (1991) Douglas Sanders. 1991.  Collective rights.  _Human Rights Quarterly_ , 13:368.
  * Santy et al. (2023) Sebastin Santy, Jenny Liang, Ronan Le Bras, Katharina Reinecke, and Maarten Sap. 2023.  NLPositionality: Characterizing design biases of datasets and models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 9080–9102, Toronto, Canada. Association for Computational Linguistics.
  * Schwartz (2022) Lane Schwartz. 2022.  _Primum Non Nocere_ : Before working with Indigenous data, the ACL must confront ongoing colonialism.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_ , volume 2.
  * Smith (2021) Linda Tuhiwai Smith. 2021.  _Decolonizing methodologies: Research and Indigenous peoples_.  Bloomsbury Publishing.
  * Spiro (2013) Melford E Spiro. 2013.  Religion: Problems of definition and explanation.  In _Anthropological approaches to the study of religion_ , pages 85–126. Routledge.
  * Stahnke (1999) Tad Stahnke. 1999.  Proselytism and the freedom to change religion in international human rights law.  _BYU L. Rev._ , page 251.
  * Taiuru (2021) Karaitiana Taiuru. 2021.  Kaitiakitanga Māori data sovereignty licences.  Retrieved from https://www.taiuru.maori.nz/maori-data-sovereignty-licences/.
  * Yoder et al. (2023) Michael Yoder, Ahmad Diab, David Brown, and Kathleen Carley. 2023.  A weakly supervised classifier and dataset of white supremacist language.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 172–185, Toronto, Canada. Association for Computational Linguistics.
  * Younging (2018) Gregory Younging. 2018.  _Elements of Indigenous style: A guide for writing by and about Indigenous peoples_.  Brush Education.

## Appendix A: Characterizations of the Bible and the Quran

To select NLP papers for the purposes of examining how they describe the Bible
and the Quran, we searched the ACL Anthology for papers mentioning “the bible
is” or “the quran is”. The Bible is characterized by NLP papers in the ACL
Anthology using phrases such as:

  * •

‘available in electronic form’, ‘most available’, ‘only available’, ‘readily
available’, ‘available online’ and ‘often the only resource available for many
languages’;

  * •

‘parallel’, ‘natural source of parallel data’, ‘ideal source of parallel
texts’, and ‘aligned’;

  * •

‘short’, ‘large’, and ‘[comparatively] small’;

  * •

‘massively multilingual’, and ‘most translated’;

  * •

‘central religious text’;

  * •

other descriptions including ‘narrative of a reasonably straightforward kind’,
‘representative of core vocabulary’, ‘one of the most familiar documents’.

The Quran is characterized by NLP papers in the ACL Anthology using phrases
such as:

  * •

‘significant religious text’, ‘[believed by Muslims to be] God’s word’,
‘[believed by Muslims to be] God’s divine words’;

  * •

‘fully diacritized’;

  * •

‘written in a unique literary style, close to very poetic language’.

## Appendix B: Selection of ACL Papers

To select papers to analyze for NLP application domain, we took the first 100
search results on the ACL Anthology for the searches ‘bible’ and ‘koran’. We
omitted 12 ‘bible’ search results from our analysis due to either:

  * •

not being in English (2);

  * •

not being research papers (6, e.g., book reviews, invited talks, or
proceedings);

  * •

or being duplicates (4)

We omitted 36 ‘koran’ search results from our analysis due to either:

  * •

not being research papers (16);

  * •

being false positive search results (13, e.g., 10 had typos for ‘Korean’);

  * •

only using the word ‘koran’ in references (3);

  * •

only using the word ‘koran’ in footnotes (2);

  * •

only using the word ‘koran’ in content generated by a model (2).

## Appendix C: Application Domains of ACL Papers using the Bible

Application domain | count
---|---
Machine translation | 20
New corpus or resource | 16
Morpho-syntactic tasks other than POS-tagging and dependency parsing | 9
POS tagging | 6
Language modeling/pretraining | 5
Literary analysis of the bible | 4
Bible translation | 3
Identify text re-use | 2
Dependency parsing | 2
Other, including sentiment analysis, named entity, CLIR, patronizing language detection | 21
Total in selected sample | 88
Filtered out (see Appendix B) | 12
Total in sample | 100
Table 4: Application domains of ACL papers using the Bible.

## Appendix D: Application Domains of ACL Papers using the Quran

Application domain | count
---|---
Machine translation | 47
Text generation | 2
Language models | 2
NER | 2
Other, including hate speech and dialogue act classification | 11
Total in selected sample | 64
Filtered out (see Appendix B) | 36
Total in sample | 100
Table 5: Application domains of ACL papers mentioning ‘koran’.

## Appendix E: A Brief Recent History of Ethics in ACL

  * •

2017: The First Workshop on Ethics in NLP is held, at EACL Hovy et al. (2017).

  * •

2018: The Second Workshop on Ethics in NLP is held, at NAACL-HLT Alfano et al.
(2018).

  * •

2018: “Ethics and Fairness in NLP” is a possible submission topic for EMNLP.

  * •

2019: NAACL-HLT has a theme track of data privacy and model bias.

  * •

2020: “Ethics and NLP” becomes a possible submission topic for the ACL
conference. This topic, or slight variations of it, are then regular in *ACL
and EMNLP conferences from 2021 onwards.

  * •

2020: EMNLP introduces an Ethics Policy, reserving the right to reject papers
on ethical grounds. ACL and NAACL follow in 2021, and EACL in 2023. (There was
no EACL in 2022.)

  * •

2021: ACL, NAACL and EMNLP begin allowing extra space in papers for
discussions of ethical considerations. Reviewers are asked to consider Ethics
Review Questions. EACL follows in 2023.

  * •

2021: EACL has a track on “Green and Sustainable NLP”. NAACL has a submission
topic “Green NLP”.

  * •

2022: ACL Rolling Review (ARR) emerges as a centralized reviewing service for
ACL conferences. It incorporates a Responsible Research Checklist which
includes a section on Ethics and is based on the NeurIPS 2021 paper checklist
as well as the work of Rogers et al. (2021) and Dodge et al. (2019).


In self-driving applications, LiDAR data provides accurate information about distances in 3D but lacks the semantic richness of camera data.
Therefore, state-of-the-art methods for perception in urban scenes fuse data from both sensor types.
In this work, we introduce a novel self-supervised method to fuse LiDAR and camera data for self-driving applications.
We build upon masked autoencoders (MAEs) and train deep learning models to reconstruct masked LiDAR data from fused LiDAR and camera features.
In contrast to related methods that use birds-eye-view representations, we fuse features from dense spherical LiDAR projections and features from fish-eye camera crops with a similar field of view.
Therefore, we reduce the learned spatial transformations to moderate perspective transformations and do not require additional modules to generate dense LiDAR representations.
Code is available at: <https://github.com/KIT-MRT/masked-fusion-360>

Reconstruct LiDAR data by querying camera features. Spherical projections of LiDAR data are transformed into patches, afterwards, randomly selected patches are removed and a MAE encoder is applied to the unmasked patches. The encoder output tokens are fused with camera features via cross-attention. Finally, a MAE decoder reconstructs the spherical LiDAR projections.

§ INTRODUCTION

Simple algorithms that scale well [He et al., 2016, Vaswani et al., 2017, Brown et al., 2020] drive deep learning research forward.
A recent addition to this trend are self-supervised pre-training methods that leverage un-labeled data to improve downstream tasks in computer vision [He et al., 2020, Chen et al., 2020, Caron et al., 2021].
For self-driving applications, self-supervised pre-training of perception models [Ma et al., 2019] or algorithms for domain adaptation [Wang et al., 2020] are examples.
LiDAR data provides accurate information about distances in 3D but lacks the semantic richness of camera data.
Therefore, state-of-the-art methods for perception in urban scenes [Li et al., 2022, Piergiovanni et al., 2021] fuse data from both sensor types.
In this work, we introduce a novel self-supervised method to fuse LiDAR and camera features for self-driving applications.
We build upon masked autoencoders (MAEs) [He et al., 2022] and train vision transformers (ViTs) [Dosovitskiy et al., 2020] to reconstruct masked LiDAR data from fused LiDAR and camera features.
Related methods [Ku et al., 2018, Li et al., 2022, Bai et al., 2022] fuse LiDAR and camera data in birds-eye-view (BEV) representations, which involves learning complex transformations from camera coordinates to birds-eye-view coordinates.
[Hess et al., 2023] and [Min et al., 2022] apply masked autoencoding to voxel representations of LiDAR data, which requires additional modules for learning dense voxel embeddings from sparse LiDAR voxels.
In contrast, we fuse features from dense spherical LiDAR projections with features from fish-eye camera crops with a similar field of view.
Therefore, we reduce the learned spatial transformations to moderate perspective transformations.
Moreover, our method does not require additional modules to generate dense LiDAR representations.

§ METHOD

Our proposed fusion method builds upon masked autoencoding [He et al., 2022], which is a recent form of denoising autoencoding.
Masked autoencoders (MAEs) consist of a ViT-based encoder and a ViT-based decoder.
Accordingly, input images are divided into patches and processed as a sequence of tokens.
During training, 50% of these patches are randomly masked and the training target is the reconstruction of the masked patches.
We use a MAE encoder as LiDAR encoder (Figure <ref>).
After transforming the LiDAR data into patches, randomly selected patches are removed and the MAE encoder is only applied to the remaining patches.
Therefore, our LiDAR encoder achieves the same computational efficiency as a vanilla MAE encoder.
The encoder output tokens are projected to the decoder tokens by re-introducing tokens for masked patches.
The decoder tokens are fused with camera features via cross-attention.
For this attention mechanism, the LiDAR tokens serve as queries $Q$ and the camera tokens as keys $K$ and values $V$ (Figure <ref>, Appendix <ref>).
Finally, a MAE decoder reconstructs the spherical LiDAR projections from the fused tokens. Experiment. We perform an initial evaluation of our method with the KITTI-360 dataset [Liao et al., 2022] (Appendix <ref>).
Figure <ref> shows crops of a masked LiDAR intensity channel and the corresponding reconstruction and target.
The importance of camera features is demonstrated by the superior reconstruction quality (higher MSSIM scores [Wang et al., 2003]) when LiDAR and camera features are used.
We simulate missing camera features by using zero matrices as camera input.

grid style=dashed,gray
ylabel=Validation MSSIM,
legend style=font=, at=(0.4, 0.60),anchor=north west,legend columns=1,
xlabel=Epoch, every axis plot/.append style=ultra thick,
] table [x=epoch, y=ssim_full_input];

With camera features
] table [x=epoch, y=ssim_no_cam];

Without camera features

Left: Masked spherical LiDAR projection and corresponding reconstruction and target. Right:  Comparison of reconstruction quality with camera features vs. without camera features.

§ CONCLUSION

In contrast to related methods that use birds-eye-view representations, we fuse features from dense spherical LiDAR projections and features from fish-eye camera crops.
Therefore, we reduce the learned spatial transformations to moderate perspective transformations and do not require additional modules to generate dense LiDAR representations.
Future steps include evaluating the performance of our fusion method as pre-training for semantic scene understanding in urban scenarios.

§.§.§ Acknowledgements

This work was accomplished within the project HAIBrid (FKZ 01IS21096A).
We acknowledge the financial support for the project by the Federal Ministry of Education and Research of Germany (BMBF).
Furthermore, this work was supported by the Helmholtz Association's Initiative and Networking Fund on the HAICORE@FZJ partition.

§.§.§ URM Statement

The authors acknowledge that at least one key author of this work meets the URM criteria of ICLR 2023 Tiny Papers Track.

[Bai et al., 2022]
Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and
Chiew-Lan Tai.
Transfusion: Robust lidar-camera fusion for 3d object detection with
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp.  1090–1099, 2022.

[Brown et al., 2020]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla
Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell,
et al.
Language models are few-shot learners.
Advances in neural information processing systems,
33:0 1877–1901, 2020.

[Caron et al., 2021]
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal,
Piotr Bojanowski, and Armand Joulin.
Emerging properties in self-supervised vision transformers.
In Proceedings of the IEEE/CVF international conference on
computer vision, pp.  9650–9660, 2021.

[Chen et al., 2021]
Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda.
Crossvit: Cross-attention multi-scale vision transformer for image
In Proceedings of the IEEE/CVF international conference on
computer vision, pp.  357–366, 2021.

[Chen et al., 2020]
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.
A simple framework for contrastive learning of visual
In International conference on machine learning, pp.  1597–1607. PMLR, 2020.

[Dosovitskiy et al., 2020]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn,
Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg
Heigold, Sylvain Gelly, et al.
An image is worth 16x16 words: Transformers for image recognition at
arXiv preprint arXiv:2010.11929, 2020.

[He et al., 2016]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pp.  770–778, 2016.

[He et al., 2020]
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.
Momentum contrast for unsupervised visual representation learning.
In Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition, pp.  9729–9738, 2020.

[He et al., 2022]
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross
Masked autoencoders are scalable vision learners.
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp.  16000–16009, 2022.

[Hess et al., 2023]
Georg Hess, Johan Jaxing, Elias Svensson, David Hagerman, Christoffer
Petersson, and Lennart Svensson.
Masked autoencoder for self-supervised pre-training on lidar point
In Proceedings of the IEEE/CVF Winter Conference on
Applications of Computer Vision, pp.  350–359, 2023.

[Hunter, 2007]
J. D. Hunter.
Matplotlib: A 2d graphics environment.
Computing in Science & Engineering, 90 (3):0
90–95, 2007.

[Ku et al., 2018]
Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven L Waslander.
Joint 3d proposal generation and object detection from view
In 2018 IEEE/RSJ International Conference on Intelligent Robots
and Systems (IROS), pp.  1–8. IEEE, 2018.

[Li et al., 2022]
Yingwei Li, Adams Wei Yu, Tianjian Meng, Ben Caine, Jiquan Ngiam, Daiyi Peng,
Junyang Shen, Yifeng Lu, Denny Zhou, Quoc V Le, et al.
Deepfusion: Lidar-camera deep fusion for multi-modal 3d object
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp.  17182–17191, 2022.

[Liao et al., 2022]
Yiyi Liao, Jun Xie, and Andreas Geiger.
Kitti-360: A novel dataset and benchmarks for urban scene
understanding in 2d and 3d.
IEEE Transactions on Pattern Analysis and Machine
Intelligence, 2022.

[Loshchilov & Hutter, 2017]
Ilya Loshchilov and Frank Hutter.
Sgdr: Stochastic gradient descent with warm restarts.
In International Conference on Learning Representations, 2017.

[Loshchilov & Hutter, 2019]
Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization.
In International Conference on Learning Representations, 2019.

[Ma et al., 2019]
Fangchang Ma, Guilherme Venturelli Cavalheiro, and Sertac Karaman.
Self-supervised sparse-to-dense: Self-supervised depth completion
from lidar and monocular camera.
In 2019 International Conference on Robotics and Automation
(ICRA), pp.  3288–3295. IEEE, 2019.

[Meyer et al., 2019]
Gregory P Meyer, Ankit Laddha, Eric Kee, Carlos Vallespi-Gonzalez, and Carl K
Lasernet: An efficient probabilistic 3d object detector for
autonomous driving.
In Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition, pp.  12677–12686, 2019.

[Min et al., 2022]
Chen Min, Dawei Zhao, Liang Xiao, Yiming Nie, and Bin Dai.
Voxel-mae: Masked autoencoders for pre-training large-scale point
arXiv preprint arXiv:2206.09900, 2022.

[Piergiovanni et al., 2021]
AJ Piergiovanni, Vincent Casser, Michael S Ryoo, and Anelia Angelova.
4d-net for learned multi-modal alignment.
In Proceedings of the IEEE/CVF International Conference on
Computer Vision, pp.  15435–15445, 2021.

[Vaswani et al., 2017]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
Advances in neural information processing systems, 30, 2017.

[Wang et al., 2020]
Yan Wang, Xiangyu Chen, Yurong You, Li Erran Li, Bharath Hariharan, Mark
Campbell, Kilian Q Weinberger, and Wei-Lun Chao.
Train in germany, test in the usa: Making 3d object detectors
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp.  11713–11723, 2020.

[Wang et al., 2003]
Zhou Wang, Eero P Simoncelli, and Alan C Bovik.
Multiscale structural similarity for image quality assessment.
In The Thrity-Seventh Asilomar Conference on Signals, Systems
& Computers, 2003, volume 2, pp.  1398–1402. IEEE, 2003.

[Xie et al., 2021]
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping
Segformer: Simple and efficient design for semantic segmentation with
Advances in Neural Information Processing Systems,
34:0 12077–12090, 2021.

§ FUSION VIA CROSS-ATTENTION

The ViT-based encoders for LiDAR and camera data process their inputs as sequences of patch tokens ([Patch]) and a learnable class token ([CLS]).
We use the class token of the LiDAR encoder and the patch tokens of the camera encoder to fuse information via cross-attention [Chen et al., 2021].
As shown in Figure <ref>, the LiDAR class token is used as queries vector and concatenated with the camera patch tokens to generate keys and values matrices for a standard attention module.
Afterwards, the LiDAR class token is added to the attention output to compute a fused class token.
In this way, the additional LiDAR class token can learn where LiDAR data is sparse or masked and query the camera tokens accordingly.
Furthermore, this cross-attention mechanism is computed in a token-to-sequence manner.
This reduces the computational complexity compared to the vanilla sequence-to-sequence manner from $O(n^2)$ to $O(n)$, where $n$ is the sequence length.

Fuse LiDAR and camera tokens via cross-attention. The attention module computes: $\text{softmax}(\frac{QK^T}{\sqrt{d_K}}) \cdot V$, where $Q$, $K$, and $V$ are query, key, and value vectors, and $d_K$ is the size of key vectors.

§ EXPERIMENTAL SETUP AND FURTHER RESULTS

Dataset. KITTI-360 contains recordings from driving in urban and sub-urban areas in Germany.
The dataset contains 76k sets of fish-eye camera images and LiDAR scans as training data and 13k sets as testing data.
We use randomly selected 70k and 6k samples from the training data as training and validation splits and the testing data as test split.
Following [Meyer et al., 2019], we transform all LiDAR scans into spherical projections and store intensity, range, and height data as 3 channels (Figure <ref>).

Model and training.
We use a patch size of $8\times8$ pixels for our model.
The smaller patch size than in a vanilla ViT ($16\times16$) is chosen to better handle fine granular semantic correlations [Xie et al., 2021].
Both our ViT-based encoders have an embedding dimension of 2048 and a depth of 8.
The cross-attention block for fusion has an embedding dimensions of 1024 and a depth of 2.
The reconstruction decoder has an embedding dimensions of 1024 and a depth of 8.
As loss, we compute the mean squared error between the target and the reconstructed patches.
We choose AdamW [Loshchilov & Hutter, 2019] as optimizer with an initial learning rate of $10^{-4}$ and use cosine annealing [Loshchilov & Hutter, 2017] to reduce the learning rate while training for 57 epochs.
During training, we track the multiscale structural similarity index measure (MSSIM) between the learned reconstruction and the training target to asses the reconstruction quality.

Results. Table <ref> shows the achieved MSSIM scores after training for 57 epochs.
For all dataset splits, the reconstruction quality with camera features is significantly higher (30% higher MSSIM).

$\text{MSSIM}_{\text{train}}$     $\text{MSSIM}_{\text{val}}$     $\text{MSSIM}_{\text{test}}$

With camera features     0.9694     0.9691     0.9612

Without camera features     0.6772     0.6771     0.6410

Reconstruction quality for different dataset splits

Comparison to masked autoencoding without camera features.
We evaluate the reconstruction performance of our model with and without camera features.
The evaluation mode without camera features is similar to vanilla masked autoencoding but with spherical projections of LiDAR data instead of natural images as inputs and targets.
In this mode, the camera features are zero matrices and the corresponding camera patch tokens are zero vectors.
Therefore, in the cross-attention mechanism for information fusion (Figure <ref>), the LiDAR class token only attends to itself and is forwarded via the skip connection.
Hence, our model becomes a masked autoencoder with only one encoder.
As shown in Figure <ref> on the right, on average 30% higher MSSIM scores are achieved with camera features vs. without camera features.
Furthermore, the reconstruction quality without camera features decreases during training (0.72 MSSIM at epoch 2 vs. 0.68 MSSIM at epoch 57).
This shows that our model learns to leverage camera features during training.

Qualitative results. Figure <ref> shows additional qualitative results of our method.
Overall, after training, our model is able to reconstruct the spherical LiDAR projections using our fusion algorithm.

Validation sample and corresponding LiDAR reconstructions. To improve the visibility, we pseudo-colorize all channels using the viridis colormap [Hunter, 2007].