# From Fitting Participation to Forging Relationships: The Art of
Participatory ML

Ned Cooper<EMAIL_ADDRESS>0000-0003-1834-279X Australian National
UniversityCanberraAustralia and  Alexandra Zafiroglu
<EMAIL_ADDRESS>0000-0003-0918-3977 Australian National
UniversityCanberraAustralia

(2024)

###### Abstract.

Participatory machine learning (ML) encourages the inclusion of end users and
people affected by ML systems in design and development processes. We
interviewed 18 participation brokers—individuals who facilitate such inclusion
and transform the products of participants' labour into inputs for an ML
artefact or system—across a range of organisational settings and project
locations. Our findings demonstrate the inherent challenges of integrating
messy contextual information generated through participation with the
structured data formats required by ML workflows and the uneven power dynamics
in project contexts. We advocate for evolution in the role of brokers to more
equitably balance value generated in Participatory ML projects for design and
development teams with value created for participants. To move beyond
`fitting' participation to existing processes and empower participants to
envision alternative futures through ML, brokers must become educators and
advocates for end users, while attending to frustration and dissent from
indirect stakeholders.

participatory methods, machine learning, artificial intelligence, design

††journalyear: 2024††copyright: rightsretained††conference: Proceedings of the
CHI Conference on Human Factors in Computing Systems; May 11–16, 2024;
Honolulu, HI, USA††booktitle: Proceedings of the CHI Conference on Human
Factors in Computing Systems (CHI '24), May 11–16, 2024, Honolulu, HI,
USA††doi: 10.1145/3613904.3642775††isbn: 979-8-4007-0330-0/24/05††ccs: Human-
centered computing††ccs: General and reference Design

##  1\. Introduction

A growing body of research and practice in the field of participatory machine
learning (`Participatory ML') encourages the inclusion of end users and people
affected by ML systems in design and development processes (Kulynych et al.,
2020). This nascent field builds on a long history of participatory approaches
to computing research and development and has emerged in response to examples
of sub-par performance of ML systems for marginalised groups. Participatory
approaches have been enacted across each stage of ML design and
development—from problem formulation to model evaluation—and include
collaborative approaches to construct datasets (Theodorou et al., 2021;
Pushkarna et al., 2022), design and validate ML algorithms (Suresh et al.,
2022; Nekoto et al., 2020), and guide advocacy for algorithmic accountability
(Queerinai et al., 2023; Katell et al., 2020). At the same time, several
authors have raised concerns about ``participation-washing'' (Sloane et al.,
2022), co-optation of participatory work (Birhane et al., 2022), and the
limited evidence across Participatory ML projects of equitable partnerships
with participants (Delgado et al., 2023; Feffer et al., 2023; Corbett et al.,
2023).

In this paper, we take a ``studying up'' lens (Nader, 1972), focusing on the
researchers, practitioners, and activists who facilitate the inclusion of
participants in the development of an ML artefact or system and implement
feedback from those participants into development processes. We describe these
individuals as participation brokers, and the work they lead and champion as
participatory projects. Such projects sit within the context of new or ongoing
ML development and the work of brokers is bifocal; deciding and facilitating
who becomes a participant and what they participate in, and transforming the
products of participants' labour into inputs for an ML artefact or system.

Despite growing research in the field, a gap exists in the literature
regarding how brokers facilitate meaningful participation and manage power
dynamics in successful Participatory ML projects. This limits our
understanding of the practical strategies and challenges involved in
implementing effective projects. To address this gap, we conducted 18
interviews with participation brokers from a range of organisational settings
and project locations to understand the different incentives, strategies,
challenges and outcomes of participatory work in the varied contexts of ML
development. In particular, we committed to expanding the scope of
Participatory ML by including projects in our study based outside North
America and Europe, which have been most reported to date.

Specifically, three exploratory questions guided our interviews:

  1. (1)

Who is defined as a participant and included in Participatory ML projects?

  2. (2)

In what stages of the ML design and development process are those participants
included?

  3. (3)

How is feedback from participants during those stages implemented?

Through a reflexive thematic analysis, we demonstrate the challenges brokers
encounter in `fitting' participants into ML workflows, and we describe
tensions between the messy nature of participatory work and the rigid
structure of ML data formats. We conclude with a reflection on our findings,
advocating for evolution in the role of brokers to more equitably balance
value generated in Participatory ML projects for design and development teams
with value created for participants. We argue that projects that explicitly
include education for and engagement with end users, and create processes for
mediating frustration and dissent of indirect stakeholders within the limits
of the ML development process, demonstrate how brokers can move beyond fitting
outside voices into established engineering processes.

##  2\. Background

###  2.1. Participation in Computing Research and Development

The participation of intended users and stakeholders affected by a policy,
product or intervention has long been a principle pursued across diverse
fields, such as international development and environmental justice.
Participatory approaches have also been a feature of computing research and
development for many decades, with roots in Action Research (AR) (Hayes, 2011)
and the rise of Participatory Design (PD) in Scandinavia in the 1970s
(Sundblad, 2011; Schuler and Namioka, 1993). PD was initially driven by the
desire of labour unions to have input into how computer systems were
introduced in the workplace (Kensing and Blomberg, 1998; Gregory, 2003). The
participatory approach was not only a methodological choice but also a
political stance, advocating for the democratisation of design and
acknowledging the value of domain expertise (Björgvinsson et al., 2010).

In HCI, PD developed into a distinct method for merging the theoretical
knowledge of researchers and designers with the practical knowledge of users
(Muller and Druin, 2012; Muller and Kuhn, 1993; Bødker, 2006), while Value
Sensitive Design broadened HCI's scope to include not only direct users but
also indirect stakeholders (Friedman, 1996; Friedman and Hendry, 2019). Over
recent decades, there has also been a shift in HCI towards community-
collaborative approaches to research, expanding beyond engagement with
individual users and towards engagement with groups and communities (Cooper et
al., 2022a; DiSalvo et al., 2013).

###  2.2. The Emergence of Participatory ML

Academics and journalists have documented algorithmic bias and discrimination
for decades, as part of critical internet studies, though discourse in this
area has accelerated since the `techlash' in 2018 (Dubber et al., 2020). Over
the past five years, this critical discourse has uncovered many examples of
sub-par performance of ML systems for marginalised groups, including women
(Buolamwini and Gebru, 2018; Bolukbasi et al., 2016; Savoldi et al., 2021),
people of colour (Noble, 2018; Benjamin, 2019), people with a disability
(Hutchinson et al., 2020; Venkit et al., 2022), and LGBTQ+ communities (Keyes,
2018).

The growing field of AI ethics has been criticised, however, for focusing
inwardly on technologists and their agency (Greene et al., 2019). Research in
AI ethics is often limited to lab environments, where technologists design,
develop, and, perhaps most importantly, test systems, with ``the intention of
deploying those systems in a different context'' (Gansky and McDonald, 2022).
Hoffmann (2019) argues that such work is subject to the same limitations as
anti-discrimination law in addressing racial discrimination, which arises from
deeper inequalities in social systems. Similarly, Selbst et al. (2019) note
that the abstraction of problems from social contexts to develop a computing
response fails to consider how social contexts affect the distribution and use
of computing systems.

The turn to participatory approaches to developing ML systems may be
understood as part of a broad shift away from the inward focus of some AI
ethics discourse, and towards understanding and addressing the lived
experience of data subjects111Data subjects are defined in this paper as
people whose personal data can be used to identify them. Data subjects include
end users, defined in this paper as the direct users of an ML artefact or
system. However, a person does not need to be a direct user of a data-driven
system to be a data subject. Many data subjects are tracked, scored, and
analysed indirectly by data-driven systems—e.g., subjects of algorithmic
credit scoring systems (Ziewitz and Singh, 2021). Proponents of Participatory
ML encourage a shift in research and development activities out of labs and
into the communities in which people live, to build more ``inclusive'' and
``democratic'' systems (Kulynych et al., 2020). Katell et al. (2020) argue
that the shift is driven in part by end users, who are starting to demand
control over algorithms embedded in products and services they interact with
as they have previously demanded control over their data—control over what
models optimise for and an understanding of how the models work.

The growth of Participatory ML is exemplified by its inclusion in papers
[e.g., Lee et al., 2019; Halfaker and Geiger, 2020; Bondi et al., 2021],
panels (Zytko et al., 2022), and workshops at computing conferences—most
significantly, the Participatory ML workshop at the 2020 International
Conference on Machine Learning (ICML) (Kulynych et al., 2020). Additionally,
industry-based research teams are considering how to enable Participatory ML,
by broadening the circle of people who can engage in the development of ML
systems [e.g., Google People + AI Research (2020), PAIR]. Participation has
also emerged as a key principle for government and civil society stakeholders
in ML-related research, development, and monitoring [e.g., Farthing et al.,
2021; Hu et al., 2019; White House Office of Science and Technology Policy,
2021; Berditchevskaia et al., 2021].

###  2.3. Power and Participatory ML

The turn to Participatory ML has been accompanied, however, by cautions
against ``participation-washing'' that have beset other fields of design
(Sloane et al., 2022). Sloane et al. (2022) argue that the ``ML community''
must interrogate all forms of inclusion in ML design and development,
including those that are passive or extractive—where people become data points
in datasets used to train ML systems, with little or no reciprocal value
(Mayer-Schönberger and Cukier, 2013). Similarly, Birhane et al. (2022) retrace
a genealogy of participation to highlight potential limitations of and risks
for Participatory ML, such as the conflation of inclusion and participation
and the co-optation of participatory work by corporate actors.

Since the ICML workshop, several authors have reviewed Participatory ML
projects and demonstrated a gap between the field's goals and actual
practices. Feffer et al. (2023) and Robertson and Salehi (2020) argue that
many Participatory ML projects are limited to developing computational methods
to elicit participant preferences, rather than re-imagining participants as
co-designers throughout the ML process. Similarly, Corbett et al. (2023) and
Delgado et al. (2021) draw on Arnstein's Ladder of Citizen Participation
(Arnstein, 1969) to note the lack of consistency in what gets classified as a
participatory approach in ML development, and a tendency towards informing or
consulting with participants, rather than engaging in equitable partnerships.
Others have raised specific concerns about projects in industry settings,
including the use of algorithmic proxies for participants (Delgado et al.,
2023) and a lack of transparency about practices or adequate incentives for
sharing power with participants (Groves et al., 2023).

Some authors have proposed frameworks or guiding questions for Participatory
ML, to shift practice towards more equitable partnerships with participants
(Delgado et al., 2023; Feffer et al., 2023; Corbett et al., 2023), and to
ensure participation informs institutional decisions (Zhang et al., 2023).
These frameworks address concerns that participation alone cannot redistribute
power and, without attention, may be more instrumental for researchers than
beneficial for participants, reflecting longstanding discussions in PD about
other computing systems (Muller and Kuhn, 1993; Vines et al., 2013; Bratteteig
and Wagner, 2014).

While existing research has investigated the risks and limitations of
Participatory ML, and highlighted a gap between principles and practice,
successful case studies of meaningful participation have nevertheless gained
recognition within the research community. This paper seeks to answer: what
are the incentives, strategies, and challenges of brokers in facilitating such
meaningful forms of participation? If participatory approaches are
fundamentally about dismantling power and information asymmetries between data
scientists and end users (or other stakeholders), how do participation brokers
actively manage power dynamics throughout the ML development process? Our
research contributes an outline of how brokers exercise and manage their power
through their decisions and actions in projects recognised by the research
community for their collaborative approach. We hope these contributions
provide guidance for future participation brokers across diverse ML
development processes, in varied organisational settings and project
locations.

##  3\. Method

We used expert interviews to derive interpretations of participatory work,
rather than facts, geared toward meaning-making by the researchers (Warren,
2001). In-depth interviews provide ``thick'' descriptions (Geertz, 1973) of
the experience of conducting participatory work, beyond what is reported in
publications. Below we outline our approach to recruitment, data collection
and data analysis. We end this section by reflecting on our positionality in
relation to the study and outlining the limitations of our methodological
approach.

###  3.1. Recruitment

To recruit interviewees, we used a two-step approach combining purposeful and
snowball sampling (Palinkas et al., 2015). First, we contacted authors of
papers published at the aforementioned ICML workshop that reported on
information, datasets, or models developed from Participatory ML projects. We
then asked these authors to recommend other relevant projects for our
consideration. Second, we contacted authors of projects identified by our
first-round interviewees, prioritising projects conducted outside North
America and Europe. We also asked for recommendations for additional projects
from our second-round interviewees.

###  3.2. Data Collection

We conducted 18 semi-structured, 60-minute interviews via video conference in
November 2022 and the first half of 2023. All interviews were approved by a
Human Research Ethics Committee, transcribed, and shared with interviewees for
review. Table 1222Four projects traversed two locations. We document the
location to which our interview primarily related; NFP = Not-for-profit;
Several projects included multiple artefacts comprising a system, rather than
one artefact. We document those projects in the ‘model’ category. provides
details about interviewees and related projects, with some information removed
to preserve anonymity.

While each interview followed the same general format, covering descriptive
and reflective questions, we tailored the interview guides based on each
interviewee's background and published work. Descriptive questions focused on
the stages and people involved in the interviewee's project(s), while
reflective questions asked about the interviewee's incentives for, challenges
in, and outcomes of participatory practices. The interviewer also facilitated
open-ended discussion by probing interviewees for details and reflections when
interviewees mentioned examples of their decisions and actions in projects.

ID |  Project Location |  Organisational Setting |  Key Artefact
---|---|---|---
P1 |  USA |  Industry |  Model
P2 |  Oceania |  Academia |  Information
P3 |  USA |  Civil society (NFP) |  Information
P4 |  Latin America |  Academia |  Model
P5 |  South Asia |  Civil society (NFP) |  Information
P6 |  Africa |  Government |  Model
P7 |  Europe |  Academia |  Dataset
P8 |  USA |  Civil society (Advocacy) |  Information
P9 |  Africa |  Academia |  Model
P10 |  USA |  Academia |  Model
P11 |  Europe |  Academia |  Information
P12 |  USA |  Academia |  Information
P13 |  USA |  Academia |  Model
P14 |  South Asia |  Civil society (NFP) |  Model
P15 |  USA |  Industry |  Dataset
P16 |  USA |  Academia |  Dataset
P17 |  USA |  Academia |  Model
P18 |  South Asia |  Civil society (Advocacy) |  Information
Table 1. Interviewee details

###  3.3. Data Analysis

Reflexive thematic analysis encourages researchers to embrace their
subjectivity as they interpret themes from the data (Braun and Clarke, 2006).
Coding in reflexive thematic analysis is an active and reflexive process that
bears the mark of the researcher(s) (Braun and Clarke, 2019). We used an
experiential and inductive approach, focusing on both semantic and latent
content to generate themes from the analysis (Byrne, 2022; Braun and Clarke,
2020)

Coding was flexible, evolving throughout the coding process. After conducting
the interviews, the first author reviewed all transcripts and openly developed
codes based on the semantic content of the transcripts and latent content of
the interviews from their notes (initially by hand, in free-form, and later
using coding software for subsequent rounds of review). During code
development, the first author met in multiple rounds with the other author to
share and collaboratively revise the codes. Following three coding rounds, the
authors convened to discuss relationships between the codes and generate the
themes presented in this paper.

###  3.4. Positionality

Both authors have experience facilitating the inclusion of participants in
processes of development within and outside computing contexts (e.g., in law
and policy reform processes). We are motivated by the agenda of Participatory
ML, and this affects our position in relation to the interviewees—we seek to
learn and build on their experiences, at the same time recognising that
participation is `made' in the micro-level activities of brokers and
participants in projects. Our position also influences the thematic
analysis—we acknowledge our primary role in participatory projects as
researchers, not participants, which grants us certain societal privileges
that may not be accessible to many of the participants in the projects we
studied. Finally, we note that we are based outside North America and Europe,
which encouraged us to expand the geographic scope of projects included in
this study.

###  3.5. Limitations

Our study has several limitations. Firstly, our study prioritised interpreting
themes across projects, limiting in-depth analysis for individual projects.
Secondly, we engaged directly with brokers—researchers, data scientists, and
activists—and not with participants in ML projects. While this aligns with our
goal of understanding the processes and challenges faced by participation
brokers, it limits our insights into participants' experiences to the
interpretation and reporting of brokers. Thirdly, while our sampling approach
aimed to identify `successful' projects, it does not guarantee that the
collaborative approaches represented were effective or equitable for all
brokers or participants. Additionally, the practices and experiences
represented by our sample reflect an engaged subset of participation brokers,
which limits perspectives in our study of those sceptical of or new to
participatory approaches in ML. Fourthly, our study includes a small sample of
projects from industry and government. Despite contacting 34 potential
interviewees, many from industry and government were unavailable due to
workload and anonymity concerns. Lastly, despite our efforts, our study
includes a small number of projects located in the Global South. We hope this
study broadens Participatory ML literature to consider activities and
experiences outside the Global North, which are underrepresented and may be
described differently to current terms of the field.

##  4\. Findings

Our study demonstrates that participatory practices are not a uniform set of
activities, but rather a means used by brokers to create, structure, and
manage relationships between categories of people during the development of ML
artefacts and systems. In the sections below, we highlight the themes and
tensions in the work of participation brokers, illustrating how participatory
projects, as both methodological and political practices, challenge and are
simultaneously challenged by the established processes of ML design and
development.

###  4.1. Becoming Brokers, Making Participants

All brokers we interviewed sat between ML systems and end users of or people
affected by those systems. In some cases, brokers were data scientists
themselves, while others worked with data scientists during the project. In
civil society settings, brokers often connected affected participants with
decision-makers about systems or related policies.

While interviewees acknowledged pragmatic reasons for their organisations to
undertake a participatory project—such as improving the performance or
legitimising the deployment of an ML artefact or system—most interviewees also
expressed strong personal motivations driving their project. Several
interviewees spoke of using their position to ``open up space for others to do
something with ML'' [P1]: not only voicing concerns about a model, or deciding
what it should learn, but also who should use it and to what purpose it might
be applied. For example, some interviewees from the Global South spoke of a
desire to ``shape the futures of machine learning and AI in [our communities]
… we don't just want to become consumers of these technologies". [P9]

Across the interviews it was clear that brokers `make' certain people and
groups into participants. As Cooper et al. (2022b) notes, each stage of the ML
design and development pipeline implicates and affects many actors—which of
those actors should formally participate? Brokers we interviewed typically
prioritised the intended users of a system, such as medical professionals
using diagnostic tools. In some cases, participants were the intended
beneficiaries of an ML system, rather than the direct user of the artefact
produced from the project (e.g., a dataset of objects for use by computer
vision engineers). Notably, brokers across all settings and locations relied
heavily on prior relationships with organisations or communities to recruit
participants—``it was crucial for us to cultivate a relationship with a
trusted partner, a convener." [P3] Some brokers also mentioned leveraging
established relationships with groups or organisations, instead of forming new
groups for projects, as a strategy to balance power dynamics with
participants.

In our interviews, brokers frequently reported difficulty engaging with
indirect stakeholders (Friedman and Hendry, 2019) (or `people affected by a
system') relative to end users. For example, patients affected by a diagnostic
tool's predictions may be considered indirect stakeholders, yet one
interviewee noted that: "We're nowhere near having patients participate in
algorithm development. We're at the point now where I just hope there's more
transparency." [P17] Indifference among indirect stakeholders also complicated
recruitment: ``Recruiting people is hard… who cares enough to be part of it in
the first place?'' [P11] Some interviewees questioned the value of including
indirect stakeholders, asking whether "someone who only consumes a product
needs to be included" [P13] or highlighting the potential tension between
inclusion and innovation: ``I can't emphasise enough that innovation is not a
consensus-building process." [P17] The range of perspectives underscore the
debate regarding the feasibility and desirability of incorporating voices
beyond direct users of ML systems into Participatory ML projects.

###  4.2. Problem Formulation as Problematising ML

Across all settings and projects, the projects included in this study were
predominately initiated by the interviewees acting as brokers. A common
challenge noted by interviewees was translating general discussions with
partners about organisational or community problems and interests into
potential ML capabilities: ``if you go in and say, hey, let's have a
conversation about data or … automated decision-making, people are like `what
is that and why is it relevant to me?''' [P2] Given the novelty and technical
dimensions of ML, several interviewees encouraged demo-ing models to encourage
people engaged as participants to imagine applications, rather than
formulating ideas in the abstract: ``Very few people understand what we're
even trying to achieve until they see a model do something in practice.'' [P7]
In some cases, brokers combined prototypes with processes that engaged
participants in critical questioning about the societal impact of ML systems.
For example, one interviewee designed prototypes reflecting existing public ML
systems to ``probe the environment and allow us [researchers] to ask questions
… more broadly'' [P11] about the impact of ML in society.

Brokers in civil society organisations mostly engaged participants after a
system was deployed, in the context of advocacy about (or against) an existing
system. Civil society interviewees envisaged projects as opportunities to
build participants' confidence to ask questions about technology, or
technology policy, in areas that may previously have been obscure: "there are
a lot of people who may have questions about [technology policy] or be
uncertain, but they lack the confidence to ask hard questions to push back."
[P3]

In a few projects conducted in the Global South, brokers engaged with people
interacting with deployed systems to surface problems with those systems.
While acknowledging that the issue is not limited to ML systems, brokers in
these projects noted that the relevant systems were often developed by
organisations based in the Global North and deployed in (or on) the local
community. In those projects, brokers sought the participation of domain
experts (non-data scientists) who were frustrated and reconciling with the
systems. As one interviewee noted, ``the amount of work that you have to do in
order to get these systems to work for you'' [P5] is an under-recognised form
of participation. To capture this information, interviewees suggested using
individual stories to clarify the experience of broader collectives: ``When we
talk about communities impacted by data, for example, that is too abstract.
But if we ground it in the experience of a person who belongs to that
community, then you understand that community in the abstract better as
well.'' [P5]

###  4.3. Infrastructure to Express Interests Through Data

Once a problem is formulated, data collection is the first opportunity for
brokers to engage participants in the model iteration process. However,
interviewees who had worked with participants to collect data for an ML
artefact or system noted the challenge of getting participants to ``express
their concerns and needs correctly'' [P15], or, at least, as data in a form a
machine would understand. To overcome this challenge, several interviewees
created infrastructure to support the data collection stage, including
tutorials and training: ``"If you know how AI works, then it's much clearer
why we need your data and also why we need to have the data in a particular
way. For example, why we ask [participants] to contribute objects in various
locations and in various kind of lighting conditions… if that isn't clear,
then the data is rubbish, and then you get a rubbish AI system." [P7] Others
included feedback mechanisms for participants to report whether the collection
process was clear and whether data reflected participants' interests (not only
ML needs).

Similarly, labelling and annotation with participants challenged brokers to
translate contextual information related to labels into machine-readable form.
Despite the valuable knowledge participants brought to the annotation process,
some brokers reported that the translation challenge limited the scope of
engagement. In these cases, interviewees reported that participants became
frustrated when conversations narrowed to the data formats required for
classification models, instead of addressing the participants' problems and
interests: ``there are specific ways that you're taught to engage communities
and sometimes the community just doesn't want to engage that way.'' [P16]
Additionally, several interviewees highlighted a lack of emphasis from their
organisations on their ability to facilitate participatory activities during
data labelling, compared to their technical skills in data processing and
modelling: ``many of the ones who are designing these tasks, do not
necessarily feel that they have the right background or training … that's a
set of skills that can take a very long time to develop.'' [P12]

###  4.4. Decoupling Model Development From Model Evaluation

While brokers noted that data scientists make ``a lot of choices.. even if you
have a very well-developed set of requirements'' [P11] across data processing,
model training and fine-tuning, most interviewees identified model development
as the "most challenging stage" [P11] to involve participants. Some
interviewees referred to frameworks facilitating the inclusion of end users
and people affected by algorithmic systems during these activities (e.g.,
translating high-level design principles into development decisions (Zhu et
al., 2018) and developing interfaces to allow participants to make preferences
clear during model development (Lee et al., 2019; Cheng et al., 2021)).
However, interviewees noted there are few dependable tools or methods: ``I
just don't think the tools are there yet to make [Participatory ML] easy.''
[P10]

Several interviewees offered two reasons for prioritising participant
involvement during model evaluation rather than model development: 1) model
development activities are technically complicated, requiring infrastructure
and resources to support non-data scientists to participate beyond that of
data collection and labelling; 2) model development may be less consequential
for the performance and value of resulting systems for participants than
earlier stages: ``Once you have labelled data, you are most of the way
there.'' [P1] Therefore, some brokers elevated the voices of end users engaged
as participants during model evaluation: ``Evaluation is really important to
have people involved because that's when you're saying, does this actually
work? But maybe model development isn't as important … because I can create
iterations quickly and modify them based on people's feedback.'' [P4] However,
perspectives on participant inclusion for model evaluation diverged. Some
suggested re-engaging data collectors for a technical assessment of the
model's results. In contrast, others—especially in areas with specialist
expertise (e.g., health, law)—argued that evaluation should be performed by
the intended users of a system (e.g., general physicians, not specialist
physicians who collected and labelled data), focusing on how an application
embedded with a model operates in real-world scenarios.

Within projects conducted in the Global South, some interviewees highlighted a
tension between building and deploying ``good enough'' [P14] models and those
perfectly tailored for specific contexts. In these projects, model evaluation
was a critical opportunity for participants to exercise power over the
development process, particularly where data scientists engaged in the project
were not based in the local community. By drawing on their domain expertise,
participants could ``take control of testing and validating outputs'' [P6] and
decide whether a model improved upon existing systems and was ready for
deployment. As one broker of such projects noted: ``This challenges some of
the narratives we have around participatory AI—the need for [the model] to be
perfect and … who makes the decisions about when a model is good enough … to
be offered and used.'' [P14]

###  4.5. Scaling Relationships Post-deployment

A key feature of all projects described by brokers was the transfer of
knowledge between data scientists and participants through brokers. Many
interviewees described how participatory projects became a mechanism to bridge
knowledge gaps in the broader ML development process—for participants, about
how ML systems function and possible applications, and for data scientists,
about the social, economic, political, and cultural contexts in which ML
systems are or will be deployed.

However, many interviewees described encountering disciplinary divides in
their projects. At the start of projects, brokers often faced the challenge of
convincing data scientists in their teams to go through the process of
building relationships with participants: ``there were times where it was
really frustrating for [the data scientists] because it involved rethinking
the typical pipeline that they follow. And there was less time for them to
spend on the technical questions that they really thrive on.'' [P14]

Once a project was completed, many brokers reported encountering a tension
between desires for generalisability in publication venues and the situated
nature of their projects. While participatory projects resulted in ``more
complex designs [and] … more drawn out work'' [P16], the features of
participatory projects, such as the quantity and diversity of participants,
were rarely recognised in traditional evaluation criteria for ML development.
As one interviewee noted: ``in terms of engineering, [participation] is not
traceable.'' [P11] While noting challenges for recognition of situated
projects within research communities, several interviewees discussed
strategies to scale project outcomes—in summary, getting the participatory
approach right with one group, then replicating and adapting the approach with
others while fine-tuning the ML system: "Machine learning scales through mass
customisation and mass localisation." [P17]

##  5\. Discussion

Participatory ML projects redefine ``who gets to be considered a data
scientist’’ [P3], recognising practical experience and domain expertise as
critical to ML development (D'Ignazio and Klein, 2020). Our findings uncovered
how participation is achieved across such projects, focusing on the decisions
and actions of participation brokers. While prior research suggests that most
projects involve consultation to elicit stakeholder preferences, we focused on
projects recognised by the research community for their collaborative
approach.

Nonetheless, brokers in these projects encountered two sets of tensions that
made collaborative engagement with participants difficult: firstly, the
conflict between messy contextual information generated through participation
and neat, machine-readable forms of information required in ML; and secondly,
the inherent tensions within project contexts, characterised by uneven
organisational, economic, and decision-making power between brokers, data
scientists, and the end users or stakeholders engaged as participants. In this
section, we reflect on these tensions and the strategies our interviewees
employed to address them, while advocating for an evolution in the role of
brokers.

###  5.1. `Fitting' Participation Into the Process of ML

Our interviews demonstrated the challenges brokers face to `fit' participants
into the ML development process. While brokers expressed personal motivations
to share power with participants, in most cases, brokers and data scientists
brought the requisite technical knowledge and resources to undertake
Participatory ML projects. As such, brokers exercised power in deciding who
became participants, what they participated in, and how their contributions
were implemented.

Brokers described a range of strategies across the development process to
navigate power dynamics with participants. Initially, some engaged with
existing organisations, rather than forming their own groups, to balance
organisational power. Others used prototypes of models during problem
formulation, to engage participants who may have been previously unfamiliar
with ML. Before data collection, several brokers conducted tutorials and
training on data requirements for models, ensuring participants grasped the
significance of data to ML and could exert influence over that stage.
Furthermore, by prioritising participants' perspectives during model
evaluation, some brokers empowered participants to influence decisions
regarding the deployment readiness of ML systems.

Nonetheless, the process of `fitting’ participants to the ML development
process implies an asymmetric division of learning, with brokers holding power
in deciding what participants learn about ML while also gaining insights from
participants' situated knowledge (Zuboff, 2019). For example, several brokers
used demos of models to engage participants at the beginning of projects.
However, as Prabhakaran and Martin (2020) note, using ML tools during problem
formulation may foreclose the decision not to use ML in a project. Where there
is a power imbalance between participants and brokers or data scientists
during problem formulation, one interviewee argued that ``there's often a rush
to adopt technical solutions to really adaptive and complex problems.'' [P8]

Further into the development process, brokers were able to open ML systems to
feedback from participants during some stages more than others—e.g., during
data collection and labelling. While data collection and labelling may be an
opportunity for participants to learn about ML and have an impact on the shape
of systems (Patton et al., 2020; Suresh et al., 2022), the power to decide
what categories exist in the world of ML and the correct categories for
labelling rested with brokers (Bowker and Star, 2000). Additionally, as noted
by several brokers, the bounds of engagement with participants during data
collection and labelling were constrained by the instrumental needs of the
broader ML development project. The fundamental challenge of balancing the
instrumental requirements of ML development while affording participants power
over the process indicates the need for evolution in the role of brokers.

###  5.2. Re-Imagining Brokers as Activists

If Participatory ML projects are to change more than the definition of a data
scientist and address the democratic deficit in AI (Kulynych et al., 2020),
then the role of brokers must expand beyond acclimatising intended users to
ML. In recent decades, researchers and designers in AR and PD projects have
been urged to embrace activism, committing to social change and ``taking a
stance'' in support of participants (Bannon et al., 2018; Bødker and Kyng,
2018; Chatterton et al., 2007). Similarly, D'Ignazio and Klein (2020) recently
outlined an approach to data science guided by activism, which has guided
recent work in Participatory ML [e.g., Suresh et al., 2022].

Our findings reveal variation in how brokers engage with participants in
projects. While some interviewees focused on gathering information from
participants, others engaged in a reciprocal exchange, educating participants
about ML and using insights from projects to advocate for participants'
interests in broader computing research and development contexts. Such actions
indicate the potential for brokers to evolve into activists, forging ongoing
relationships with participants to envision "possibilities and alternatives"
through ML (Bannon et al., 2018). Below, we briefly outline strategies to
guide brokers in embracing this activist role.

####  5.2.1. Education and Advocacy for End Users

Brokers working with end users may scaffold participation through tools to
educate and engage end users about ML and its potential applications
(Theodorou et al., 2021). We see this as an extension of the educational work
several brokers reported conducting in relation to data collection and
labelling activities. For example, this may involve engaging intended or
potential end users in speculative design activities to contest ML design
assumptions [e.g., Alfrink et al., 2022, 2023]. Additionally, brokers must be
attuned to the possibility of ``reverse tutelage'' (Mohamed et al., 2020)—not
only acquiring information from end users, but also learning about the
situations in which ML systems are or will be used, and advocating for the
interests of end users in computing research and development contexts.

####  5.2.2. From Stakeholder Frustration to Feedback

Engaging with indirect stakeholders was particularly challenging for the
projects included in our study. While many brokers described working with
supportive partners, interviewees from civil society and some brokers of
Global South projects engaged with frustrated stakeholders to expose points of
contention with deployed systems. As Sambasivan and Veeraraghavan (2022) note,
domain experts are often reduced to data collectors in AI research,
particularly in Global South settings. Elevating such expertise requires
attending to frustration, refusal, and dissent concerning ML systems as viable
forms of agency and key sources of feedback (Dobbe et al., 2021; Zong and
Matias, 2023).

To bridge the gap between frustration and feedback first requires brokers to
educate indirect stakeholders about ML, similarly to end users, as
participating with respect to deployed systems ``requires a certain kind of
literacy … so that you can actually navigate your data representation.'' [P5]
Additionally, bridging this gap requires the development of methodologies that
enable people interacting with ML systems to express their interests in ways
that can be input into ML development and related policy processes. For
example, this may require further work on transforming qualitative, personal
stories into feedback for technical development processes (Singh et al.,
2022), and further development of collaborative algorithmic audit platforms to
encourage relationships between data scientists and dissenters (Deng et al.,
2023).

##  6\. Conclusion

In our interview study with 18 participation brokers across a range of
organisational settings and locations, we explored how participation is
achieved across the stages of a participatory project within the ML design and
development context. While we hope our findings will guide future
participation brokers, we acknowledge the inherent tension between messy
contextual information generated through participation and the rigid data
formats of ML workflows, as well as uneven power dynamics within project
contexts. Recognising these challenges, we advocate for a more pronounced
activist role for participation brokers. This role involves not only `fitting'
participants into established engineering processes, but also forging
relationships with participants over time to empower participants to envision
alternatives and possibilities through ML. By educating and advocating for end
users, and attending to frustration and dissent from indirect stakeholders,
participation brokers can help steer Participatory ML towards its democratic
potential.

###### Acknowledgements.

This research is supported by an Australian Government Research Training
Program (RTP) Scholarship and a Florence Violet McKenzie scholarship. Our
sincere thanks to the interviewees who shared their knowledge and experience
with us: Aleks Berditchevskaia, Anna Brown, Aaron Halfaker, Ken Holstein,
Michael Katell, Jennifer Lee, Vukosi Marivate, Rebeca Moreno Jiménez,
Rajendran Narayanan, Harini Suresh, Haiyi Zhu, and others who wish to remain
anonymous. We also thank our reviewers for their feedback and suggestions.

## References

  * (1)
  * Alfrink et al. (2023) Kars Alfrink, Ianus Keller, Neelke Doorn, and Gerd Kortuem. 2023.  Contestable Camera Cars: A Speculative Design Exploration of Public AI That Is Open and Responsive to Dispute. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ (Hamburg, Germany) _(CHI '23, Article 8)_. Association for Computing Machinery, New York, NY, USA, 1–16.
  * Alfrink et al. (2022) Kars Alfrink, Ianus Keller, Gerd Kortuem, and Neelke Doorn. 2022.  Contestable AI by Design: Towards a Framework.  _Minds & Machines_ (Aug. 2022), 1–27.  https://doi.org/10.1007/s11023-022-09611-z
  * Arnstein (1969) Sherry R Arnstein. 1969.  A Ladder Of Citizen Participation.  _Journal of the American Institute of Planners_ 35, 4 (July 1969), 216–224.
  * Bannon et al. (2018) Liam Bannon, Jeffrey Bardzell, and Susanne Bødker. 2018.  Reimagining participatory design.  _Interactions_ 26, 1 (Dec. 2018), 26–32.
  * Benjamin (2019) Ruha Benjamin. 2019.  _Race After Technology: Abolitionist Tools for the New Jim Code_.  Polity Press, Cambridge, UK.
  * Berditchevskaia et al. (2021) Aleks Berditchevskaia, Eirini Malliaraki, and Kathy Peach. 2021.  _Participatory AI for humanitarian innovation: a briefing paper_.  Technical Report. Nesta.  https://www.nesta.org.uk/report/participatory-ai-humanitarian-innovation-briefing-paper/
  * Birhane et al. (2022) Abeba Birhane, William Isaac, Vinodkumar Prabhakaran, Mark Diaz, Madeleine Clare Elish, Iason Gabriel, and Shakir Mohamed. 2022.  Power to the People? Opportunities and Challenges for Participatory AI. In _Equity and Access in Algorithms, Mechanisms, and Optimization_ (Arlington, VA, USA) _(EAAMO '22, Article 6)_. Association for Computing Machinery, New York, NY, USA, 1–8.
  * Björgvinsson et al. (2010) Erling Björgvinsson, Pelle Ehn, and Per-Anders Hillgren. 2010.  Participatory Design and ``Democratizing Innovation''. In _Proceedings of the 11th Biennial Participatory Design Conference_ (Sydney, Australia) _(PDC '10)_. Association for Computing Machinery, New York, NY, USA, 41–50.
  * Bødker (2006) Susanne Bødker. 2006.  When second wave HCI meets third wave challenges. In _Proceedings of the 4th Nordic Conference on Human-Computer Interaction: Changing Roles_ (Oslo, Norway) _(NordiCHI '06)_. Association for Computing Machinery, New York, NY, USA, 1–8.
  * Bødker and Kyng (2018) Susanne Bødker and Morten Kyng. 2018.  Participatory Design That Matters—Facing the Big Issues.  _ACM Transactions on Computer-Human Interaction_ 25, 1 (Feb. 2018), 1–31.
  * Bolukbasi et al. (2016) Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016.  Man Is to Computer Programmer as Woman Is to Homemaker? Debiasing Word Embeddings. In _Proceedings of the 30th International Conference on Neural Information Processing Systems_ _(NIPS'16)_. Curran Associates Inc., Red Hook, NY, USA, 4356–4364.
  * Bondi et al. (2021) Elizabeth Bondi, Lily Xu, Diana Acosta-Navas, and Jackson A Killian. 2021.  Envisioning Communities: A Participatory Approach towards AI for Social Good. In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_ _(AIES '21)_. Association for Computing Machinery, New York, NY, USA, 425–436.
  * Bowker and Star (2000) Geoffrey C Bowker and Susan Leigh Star. 2000.  _Sorting Things Out: Classification and Its Consequences_.  MIT Press, Cambridge, MA, USA.
  * Bratteteig and Wagner (2014) Tone Bratteteig and Ina Wagner. 2014.  _Disentangling Participation: Power and Decision-making in Participatory Design_.  Springer, Switzerland.
  * Braun and Clarke (2006) Virginia Braun and Victoria Clarke. 2006.  Using Thematic Analysis in Psychology.  _Qualitative Research in Psychology._ 3, 2 (2006), 77–101.
  * Braun and Clarke (2019) Virginia Braun and Victoria Clarke. 2019.  Reflecting on reflexive thematic analysis.  _Qualitative Research in Sport, Exercise and Health_ 11, 4 (Aug. 2019), 589–597.
  * Braun and Clarke (2020) Virginia Braun and Victoria Clarke. 2020.  One Size Fits All? What Counts as Quality Practice in (Reflexive) Thematic Analysis?  _Qualitative Research in Psychology_ 18, 3 (Aug. 2020), 1–25.
  * Buolamwini and Gebru (2018) Joy Buolamwini and Timnit Gebru. 2018.  Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In _Proceedings of the 1st Conference on Fairness, Accountability and Transparency_ _(Proceedings of Machine Learning Research, Vol. 81)_ , Sorelle A Friedler and Christo Wilson (Eds.). PMLR, New York, NY, USA, 77–91.
  * Byrne (2022) David Byrne. 2022.  A worked example of Braun and Clarke's approach to reflexive thematic analysis.  _Quality & Quantity_ 56, 3 (June 2022), 1391–1412.
  * Chatterton et al. (2007) Paul Chatterton, Duncan Fuller, and Paul Routledge. 2007.  Relating Action to Activism: Theoretical and Methodological Reflections.  In _Participatory Action Research Approaches and Methods: Connecting People, Participation and Place_ , Sara Kindon, Rachel Pain, and Mike Kesby (Eds.). Routledge, London, UK, 216–222.  https://play.google.com/store/books/details?id=daKkkt0NiPQC
  * Cheng et al. (2021) Hao-Fei Cheng, Logan Stapleton, Ruiqi Wang, Paige Bullock, Alexandra Chouldechova, Zhiwei Steven Steven Wu, and Haiyi Zhu. 2021.  Soliciting Stakeholders' Fairness Notions in Child Maltreatment Predictive Systems.  In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. Association for Computing Machinery, New York, NY, USA.
  * Cooper et al. (2022b) A Feder Cooper, Emanuel Moss, Benjamin Laufer, and Helen Nissenbaum. 2022b.  Accountability in an Algorithmic Society: Relationality, Responsibility, and Robustness in Machine Learning. In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_ (Seoul, Republic of Korea) _(FAccT '22)_. Association for Computing Machinery, New York, NY, USA, 864–876.
  * Cooper et al. (2022a) Ned Cooper, Tiffanie Horne, Gillian R Hayes, Courtney Heldreth, Michal Lahav, Jess Holbrook, and Lauren Wilcox. 2022a.  A Systematic Review and Thematic Analysis of Community-Collaborative Approaches to Computing Research. In _CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI '22, Article 73)_. Association for Computing Machinery, New York, NY, USA, 1–18.
  * Corbett et al. (2023) Eric Corbett, Emily Denton, and Sheena Erete. 2023.  Power and Public Participation in AI. In _Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization_ (Boston, MA, USA) _(EAAMO '23, Article 8)_. Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3617694.3623228
  * Delgado et al. (2021) Fernando Delgado, Stephen Yang, Michael Madaio, and Qian Yang. 2021.  Stakeholder Participation in AI: Beyond ``Add Diverse Stakeholders and Stir''.  arXiv:2111.01122 [cs.AI]
  * Delgado et al. (2023) Fernando Delgado, Stephen Yang, Michael Madaio, and Qian Yang. 2023.  The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice. In _Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization_ (Boston, MA, USA) _(EAAMO '23, Article 37)_. Association for Computing Machinery, New York, NY, USA, 1–23.  https://doi.org/10.1145/3617694.3623261
  * Deng et al. (2023) Wesley Hanwen Deng, Boyuan Guo, Alicia Devrio, Hong Shen, Motahhare Eslami, and Kenneth Holstein. 2023.  Understanding Practices, Challenges, and Opportunities for User-Engaged Algorithm Auditing in Industry Practice. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ (Hamburg, Germany) _(CHI '23, Article 377)_. Association for Computing Machinery, New York, NY, USA, 1–18.
  * D'Ignazio and Klein (2020) Catherine D'Ignazio and Lauren F Klein. 2020.  _Data Feminism_.  The MIT Press, Cambridge, MA, USA.
  * DiSalvo et al. (2013) Carl DiSalvo, Andrew Clement, and Volkmar Pipek. 2013.  Communities: Participatory Design for, with and by Communities.  In _Routledge International Handbook of Participatory Design_ , Jesper Simonsen and Toni Robertson (Eds.). Taylor & Francis Group, London, UK.
  * Dobbe et al. (2021) Roel Dobbe, Thomas Krendl Gilbert, and Yonatan Mintz. 2021.  Hard Choices in Artificial Intelligence.  _Artificial Intelligence_ 300 (2021), 103555.
  * Dubber et al. (2020) Markus D Dubber, Frank Pasquale, and Sunit Das. 2020.  _Oxford Handbook of Ethics of AI_.  Oxford University Press, Oxford, UK.
  * Farthing et al. (2021) Sophie Farthing, John Howell, Katerina Lecchi, Zoe Paleologos, Phoebe Saintilan, and Edward Santow. 2021.  _Human Rights and Technology Final Report_.  Technical Report. Australian Human Rights Commission, Sydney.
  * Feffer et al. (2023) Michael Feffer, Michael Skirpan, Zachary Lipton, and Hoda Heidari. 2023.  From Preference Elicitation to Participatory ML: A Critical Survey & Guidelines for Future Research. In _Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society_ (Montréal , QC, Canada) _(AIES '23)_. Association for Computing Machinery, New York, NY, USA, 38–48.  https://doi.org/10.1145/3600211.3604661
  * Friedman (1996) Batya Friedman. 1996.  Value-Sensitive Design.  _Interactions_ 3, 6 (Dec. 1996), 16–23.
  * Friedman and Hendry (2019) Batya Friedman and David G Hendry. 2019.  _Value Sensitive Design: Shaping Technology with Moral Imagination_.  The MIT Press, Cambridge, MA, USA.
  * Gansky and McDonald (2022) Ben Gansky and Sean McDonald. 2022.  CounterFAccTual: How FAccT Undermines Its Organizing Principles. In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_ (Seoul, Republic of Korea) _(FAccT '22)_. Association for Computing Machinery, New York, NY, USA, 1982–1992.
  * Geertz (1973) Clifford Geertz. 1973.  Thick Description: Toward an Interpretive Theory of Culture.  In _The Interpretation of Cultures_. Basic Books, New York, NY, USA, 41–51.
  * Google People + AI Research (2020) (PAIR) Google People + AI Research (PAIR). 2020.  People + AI Research.  https://pair.withgoogle.com/.  Accessed: 2022-10-12.
  * Greene et al. (2019) Daniel Greene, Anna Lauren Hoffmann, and Luke Stark. 2019.  Better, Nicer, Clearer, Fairer: A Critical Assessment of the Movement for Ethical Artificial Intelligence and Machine Learning. In _Proceedings of the 52nd Hawai`i International Conference on System Sciences_. IEEE Computer Society Press, Honolulu, Hawai`i, USA, 2122–2131.
  * Gregory (2003) Judith Gregory. 2003.  Scandinavian approaches to participatory design.  _International Journal of Engineering Education_ 19, 1 (2003), 62–74.
  * Groves et al. (2023) Lara Groves, Aidan Peppin, Andrew Strait, and Jenny Brennan. 2023.  Going public: the role of public participation approaches in commercial AI labs. In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_ (Chicago, IL, USA) _(FAccT '23)_. Association for Computing Machinery, New York, NY, USA, 1162–1173.
  * Halfaker and Geiger (2020) Aaron Halfaker and R Stuart Geiger. 2020.  ORES: Lowering Barriers with Participatory Machine Learning in Wikipedia.  _Proceedings of the ACM on Human-Computer Interaction_ 4, CSCW2 (Oct. 2020), 1–37.
  * Hayes (2011) Gillian R Hayes. 2011.  The Relationship of Action Research to Human-Computer Interaction.  _ACM Transactions on Computer-Human Interaction_ 18, 3 (Aug. 2011), 1–20.
  * Hoffmann (2019) Anna Lauren Hoffmann. 2019.  Where fairness fails: data, algorithms, and the limits of antidiscrimination discourse.  _Information, Communication & Society_ 22, 7 (June 2019), 900–915.
  * Hu et al. (2019) Xianhong Hu, Bhanu Neupane, Lucia Flores Echaiz, Prateek Sibal, and Macarena Rivera Lam. 2019.  _Steering AI and Advanced ICTs for Knowledge Societies_.  United Nations Educational, Scientific and Cultural Organization (UNESCO), Paris, France.
  * Hutchinson et al. (2020) Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020.  Social Biases in NLP Models as Barriers for Persons with Disabilities. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 5491–5501.
  * Katell et al. (2020) Michael Katell, Meg Young, Dharma Dailey, Bernease Herman, Vivian Guetler, Aaron Tam, Corinne Bintz, Daniella Raz, and P M Krafft. 2020.  Toward situated interventions for algorithmic equity: lessons from the field. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ (Barcelona, Spain) _(FAT* '20)_. Association for Computing Machinery, New York, NY, USA, 45–55.
  * Kensing and Blomberg (1998) Finn Kensing and Jeanette Blomberg. 1998.  Participatory Design: Issues and Concerns.  _Computer Supported Cooperative Work_ 7, 3 (Sept. 1998), 167–185.
  * Keyes (2018) Os Keyes. 2018.  The Misgendering Machines: Trans/HCI Implications of Automatic Gender Recognition.  _Proceedings of the ACM on Human-Computer Interaction_ 2, CSCW (Nov. 2018), 1–22.
  * Kulynych et al. (2020) Bogdan Kulynych, David Madras, Smitha Milli, Inioluwa Deborah Raji, Zhou, and Richard Zemel. 2020.  Participatory Approaches to Machine Learning.  https://participatoryml.github.io/.  Accessed: 2022-10-12.
  * Lee et al. (2019) Min Kyung Lee, Daniel Kusbit, Anson Kahng, Ji Tae Kim, Xinran Yuan, Allissa Chan, Daniel See, Ritesh Noothigattu, Siheon Lee, Alexandros Psomas, and Ariel D Procaccia. 2019.  WeBuildAI: Participatory Framework for Algorithmic Governance.  _Proceedings of the ACM on Human-Computer Interaction_ 3, CSCW (Nov. 2019), 1–35.
  * Mayer-Schönberger and Cukier (2013) Viktor Mayer-Schönberger and Kenneth Cukier. 2013.  _Big Data: A Revolution that Will Transform how We Live, Work, and Think_.  Houghton Mifflin Harcourt, Boston, MA, USA.
  * Mohamed et al. (2020) Shakir Mohamed, Marie-Therese Png, and William Isaac. 2020.  Decolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligence.  _Philosophy & Technology_ 33, 4 (Dec. 2020), 659–684.
  * Muller and Druin (2012) Michael J Muller and Allison Druin. 2012.  Participatory Design: The Third Space in Human–Computer Interaction.  In _The Human–Computer Interaction Handbook_. CRC Press, Boca Raton, FL, USA, 1125–1153.
  * Muller and Kuhn (1993) Michael J Muller and Sarah Kuhn. 1993.  Participatory Design.  _Commun. ACM_ 36, 6 (June 1993), 24–28.
  * Nader (1972) Laura Nader. 1972.  Up the Anthropologist: Perspectives Gained From Studying Up.  In _Reinventing Anthropology_ , Dell Hymes (Ed.). Pantheon Books, Random House, New York.
  * Nekoto et al. (2020) Wilhelmina Nekoto, Vukosi Marivate, Tshinondiwa Matsila, Timi Fasubaa, Taiwo Fagbohungbe, Solomon Oluwole Akinola, Shamsuddeen Muhammad, Salomon Kabongo Kabenamualu, Salomey Osei, Freshia Sackey, Rubungo Andre Niyongabo, Ricky Macharm, Perez Ogayo, Orevaoghene Ahia, Musie Meressa Berhe, Mofetoluwa Adeyemi, Masabata Mokgesi-Selinga, Lawrence Okegbemi, Laura Martinus, Kolawole Tajudeen, Kevin Degila, Kelechi Ogueji, Kathleen Siminyu, Julia Kreutzer, Jason Webster, Jamiil Toure Ali, Jade Abbott, Iroro Orife, Ignatius Ezeani, Idris Abdulkadir Dangana, Herman Kamper, Hady Elsahar, Goodness Duru, Ghollah Kioko, Murhabazi Espoir, Elan van Biljon, Daniel Whitenack, Christopher Onyefuluchi, Chris Chinenye Emezue, Bonaventure F P Dossou, Blessing Sibanda, Blessing Bassey, Ayodele Olabiyi, Arshath Ramkilowan, Alp Öktem, Adewale Akinfaderin, and Abdallah Bashir. 2020.  Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. Association for Computational Linguistics, Online, 2144–2160.  https://doi.org/10.18653/v1/2020.findings-emnlp.195
  * Noble (2018) Safiya Umoja Noble. 2018.  _Algorithms of Oppression: How Search Engines Reinforce Racism_.  NYU Press, New York, NY, USA.
  * Palinkas et al. (2015) Lawrence A Palinkas, Sarah M Horwitz, Carla A Green, Jennifer P Wisdom, Naihua Duan, and Kimberly Hoagwood. 2015.  Purposeful Sampling for Qualitative Data Collection and Analysis in Mixed Method Implementation Research.  _Adm. Policy Ment. Health_ 42, 5 (Sept. 2015), 533–544.
  * Patton et al. (2020) Desmond U Patton, Fei-Tzin Lee, William R Frey, Kathleen McKeown, Kyle A McGregor, and Emanuel Moss. 2020.  Contextual Analysis of Social Media: The Promise and Challenge of Eliciting Context in Social Media Posts with Natural Language Processing.  _Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society_ 2020 (Feb. 2020), 337–342.
  * Prabhakaran and Martin (2020) Vinodkumar Prabhakaran and Donald Martin, Jr. 2020.  Participatory Machine Learning Using Community-Based System Dynamics.  _Health and Human Rights Journal_ 22, 2 (Dec. 2020), 71–74.
  * Pushkarna et al. (2022) Mahima Pushkarna, Andrew Zaldivar, and Oddur Kjartansson. 2022.  Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI. In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_ (Seoul, Republic of Korea) _(FAccT '22)_. Association for Computing Machinery, New York, NY, USA, 1776–1826.  https://doi.org/10.1145/3531146.3533231
  * Queerinai et al. (2023) Organizers Of Queerinai, Anaelia Ovalle, Arjun Subramonian, Ashwin Singh, Claas Voelcker, Danica J Sutherland, Davide Locatelli, Eva Breznik, Filip Klubicka, Hang Yuan, Hetvi J, Huan Zhang, Jaidev Shriram, Kruno Lehman, Luca Soldaini, Maarten Sap, Marc Peter Deisenroth, Maria Leonor Pacheco, Maria Ryskina, Martin Mundt, Milind Agarwal, Nyx Mclean, Pan Xu, A Pranav, Raj Korpan, Ruchira Ray, Sarah Mathew, Sarthak Arora, St John, Tanvi Anand, Vishakha Agrawal, William Agnew, Yanan Long, Zijie J Wang, Zeerak Talat, Avijit Ghosh, Nathaniel Dennler, Michael Noseworthy, Sharvani Jha, Emi Baylor, Aditya Joshi, Natalia Y Bilenko, Andrew Mcnamara, Raphael Gontijo-Lopes, Alex Markham, Evyn Dong, Jackie Kay, Manu Saraswat, Nikhil Vytla, and Luke Stark. 2023.  Queer In AI: A Case Study in Community-Led Participatory AI. In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_ (Chicago, IL, USA) _(FAccT '23)_. Association for Computing Machinery, New York, NY, USA, 1882–1895.  https://doi.org/10.1145/3593013.3594134
  * Robertson and Salehi (2020) Samantha Robertson and Niloufar Salehi. 2020.  What If I Don't Like Any Of The Choices? The Limits of Preference Elicitation for Participatory Algorithm Design.  arXiv:2007.06718 [cs.CY]
  * Sambasivan and Veeraraghavan (2022) Nithya Sambasivan and Rajesh Veeraraghavan. 2022.  The Deskilling of Domain Expertise in AI Development. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI '22, Article 587)_. Association for Computing Machinery, New York, NY, USA, 1–14.
  * Savoldi et al. (2021) Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2021.  Gender Bias in Machine Translation.  _Transactions of the Association for Computational Linguistics (TACL)_ 9 (Aug. 2021), 845–874.
  * Schuler and Namioka (1993) Douglas Schuler and Aki Namioka. 1993.  _Participatory Design: Principles and Practices_.  CRC Press, Boca Raton, FL, USA.
  * Selbst et al. (2019) Andrew D Selbst, Danah Boyd, Sorelle A Friedler, Suresh Venkatasubramanian, and Janet Vertesi. 2019.  Fairness and Abstraction in Sociotechnical Systems. In _Proceedings of the Conference on Fairness, Accountability, and Transparency_ (Atlanta, GA, USA) _(FAT* '19)_. Association for Computing Machinery, New York, NY, USA, 59–68.
  * Singh et al. (2022) Ranjit Singh, Rigoberto Lara Guzmán, and Patrick Davison. 2022.  Parables of AI in/from the Majority World. (Dec. 2022).  https://doi.org/10.2139/ssrn.4258527
  * Sloane et al. (2022) Mona Sloane, Emanuel Moss, Olaitan Awomolo, and Laura Forlano. 2022.  Participation Is not a Design Fix for Machine Learning. In _Equity and Access in Algorithms, Mechanisms, and Optimization_ (Arlington, VA, USA) _(EAAMO '22, Article 1)_. Association for Computing Machinery, New York, NY, USA, 1–6.
  * Sundblad (2011) Yngve Sundblad. 2011.  UTOPIA: Participatory Design from Scandinavia to the World.  In _History of Nordic Computing 3_ , John Impagliazzo, Per Lundin, and Benkt Wangler (Eds.). Springer, Berlin, Germany; London, UK, 176–186.
  * Suresh et al. (2022) Harini Suresh, Rajiv Movva, Amelia Lee Dogan, Rahul Bhargava, Isadora Cruxen, Angeles Martinez Cuba, Guilia Taurino, Wonyoung So, and Catherine D'Ignazio. 2022.  Towards Intersectional Feminist and Participatory ML: A Case Study in Supporting Feminicide Counterdata Collection. In _2022 ACM Conference on Fairness, Accountability, and Transparency_. Association for Computing Machinery, Seoul, Republic of Korea, 667–678.
  * Theodorou et al. (2021) Lida Theodorou, Daniela Massiceti, Luisa Zintgraf, Simone Stumpf, Cecily Morrison, Edward Cutrell, Matthew Tobias Harris, and Katja Hofmann. 2021.  Disability-first Dataset Creation: Lessons from Constructing a Dataset for Teachable Object Recognition with Blind and Low Vision Data Collectors. In _Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility_ (Virtual Event, USA) _(ASSETS '21, Article 27)_. Association for Computing Machinery, New York, NY, USA, 1–12.  https://doi.org/10.1145/3441852.3471225
  * Venkit et al. (2022) Pranav Narayanan Venkit, Mukund Srinath, and Shomir Wilson. 2022.  A Study of Implicit Bias in Pretrained Language Models against People with Disabilities. In _Proceedings of the 29th International Conference on Computational Linguistics_. International Committee on Computational Linguistics, Gyeongju, Republic of Korea, 1324–1332.
  * Vines et al. (2013) John Vines, Rachel Clarke, Peter Wright, John McCarthy, and Patrick Olivier. 2013.  Configuring participation: on how we involve people in design. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ (Paris, France) _(CHI '13)_. Association for Computing Machinery, New York, NY, USA, 429–438.
  * Warren (2001) Carol A B Warren. 2001.  Qualitative Interviewing.  In _Handbook of Interview Research_. SAGE Publications Ltd, Thousand Oaks, California, USA, 83–102.
  * White House Office of Science and Technology Policy (2021) White House Office of Science and Technology Policy. 2021.  Artificial Intelligence and Democratic Values.
  * Zhang et al. (2023) Angie Zhang, Olympia Walker, Kaci Nguyen, Jiajun Dai, Anqing Chen, and Min Kyung Lee. 2023.  Deliberating with AI: Improving Decision-Making for the Future through Participatory AI Design and Stakeholder Deliberation.  _Proceedings of the ACM on Human-Computer Interaction_ 7, CSCW1 (April 2023), 1–32.
  * Zhu et al. (2018) Haiyi Zhu, Bowen Yu, Aaron Halfaker, and Loren Terveen. 2018.  Value-Sensitive Algorithm Design: Method, Case Study, and Lessons.  _Proceedings of the ACM on Human-Computer Interaction_ 2, CSCW (Nov. 2018), 1–23.
  * Ziewitz and Singh (2021) Malte Ziewitz and Ranjit Singh. 2021.  Critical Companionship: Some Sensibilities for Studying the Lived Experience of Data Subjects.  _Big Data & Society_ 8, 2 (July 2021), 20539517211061122.  https://doi.org/10.1177/20539517211061122
  * Zong and Matias (2023) Jonathan Zong and J Nathan Matias. 2023.  Data Refusal From Below: A Framework for Understanding, Evaluating, and Envisioning Refusal as Design.  _ACM Journal on Responsible Computing_ (Oct. 2023).  https://doi.org/10.1145/3630107
  * Zuboff (2019) Shoshana Zuboff. 2019.  _The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power_.  Profile Books, London, UK.
  * Zytko et al. (2022) Douglas Zytko, Pamela J. Wisniewski, Shion Guha, Eric P. S. Baumer, and Min Kyung Lee. 2022.  Participatory Design of AI Systems: Opportunities and Challenges Across Diverse Users, Relationships, and Application Domains. In _Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI EA '22, Article 154)_. Association for Computing Machinery, New York, NY, USA, 1–4.