# Challenges and outcomes in remote undergraduate research programs during the
COVID-19 pandemic

Dina Zohrabi Alaee  Benjamin M. Zwickl  School of Physics and Astronomy,
Rochester Institute of Technology, 84 Lomb Memorial Drive, Rochester, NY,14623

###### Abstract

In the Summer of 2020, as COVID-19 limited in-person research opportunities
and created additional barriers for many students, institutions either
canceled or remotely hosted their Research Experience for Undergraduates (REU)
programs. The present qualitative phenomenographic study was designed to
explore some of the possible limitations, challenges and outcomes of this
remote experience. Overall, 94 interviews were conducted with paired
participants; mentees ($N$=10) and mentors ($N$=8) from six different REU
programs. By drawing on Cultural-Historical Activity Theory (CHAT) as a
framework, our study uncovers some of the challenges for mentees around the
accomplishment of their research objectives and academic goals. These
challenges included motivation, limited access to technologies at home,
limited communication among REU students, barriers in mentor-mentee
relationships, and differing expectations about doing research. Despite the
challenges, all mentees reported that this experience was highly beneficial.
Comparisons between outcomes of these remote REUs and published outcomes of
in-person undergraduate research programs reveal many similar benefits such as
integrating students into STEM culture. Our study suggests that remote
research programs could be considered as a means to expand access to research
experiences for undergraduate students even after COVID-19 restrictions have
been lifted.

††preprint: APS/123-QED

##  I Introduction

Undergraduate research experiences (UREs) affect students’ academic pathways
and career preparation towards STEM by providing authentic research-based
learning situations [1, 2, 3, 4, 5]. A large body of literature has reported
both academic and psychosocial benefits as outcomes of in-person UREs.
Academically, UREs have been reported to help students achieve a higher level
of content knowledge [6], while also improving their eventual career outcomes
[7, 8]. Meanwhile, psychosocial benefits refer to the positive growth in a
student’s perceptions, emotions, attitudes, and social dimensions around their
academic experiences. Psychosocially, in-person UREs have been shown to help
students increase self-confidence [9, 10, 11, 12], develop communication
skills [6, 11], improve scientific identity [7, 13, 14], and grow a sense of
belonging to the community [7, 15, 16, 17].

One structure of a URE, which is common in the United States, is the Research
Experiences for Undergraduates (REU) program, which is a ten-week summer
research experience funded by the U.S. National Science Foundation. Due to the
COVID-19 pandemic, some REU programs transitioned to a remote format in the
summer of 2020.

Despite differences in the individual goals of each research project and
research program, there are several common goals between all undergraduate
research programs; they all hope to increase retention in a STEM career
pathway, promote STEM knowledge and practices, and integrate students into
STEM culture [18]. While many studies show these goals are often accomplished
through UREs, they do not fully describe why a URE would lead to increased
retention in STEM among participants. In most studies of URE outcomes, data
have been derived from self-reported surveys [8, 19], and end-of-program
formal evaluations [8, 19, 7, 20], while fewer studies have used in-depth
multiple interviews [9, 11]. Furthermore, most studies have exclusively
explored the outcomes of in-person undergraduate research experience, and very
little research has yet focused on a remote research experience [21]. We
designed the current study to identify the range of challenges that students
experienced and characterize some of the possible outcomes of the remote REU
program. We used longitudinal semi-structured interviews with mentor-mentee
pairs over the undergraduate summer research programs. In the current study,
we used Cultural-Historical Activity Theory (CHAT) [22, 23, 24] analysis to
describe the challenges and outcomes of the remote research experiences. The
Research Experience for Undergraduates (REU) program as a whole unit it
modeled as an activity system. Fig. 1 shows a simplified representation of the
REU as an activity system in the standard CHAT format developed by Engeström,
which represents the system as a triangle that connects a subject to their
objectives and outcomes [24]. The research questions that guided our analysis
were:

  * •

RQ1: What challenges were observed within the remote goal-directed REU
activity?

  * •

RQ2: What are some of the outcomes of the remote REU programs?

Figure 1: An summary of an activity system is represented as a triangular
model [24]. The Research Experiences for Undergraduates (REU) program is an
activity system. Subject refers to mentees who participated in the remote REU
programs. The tools include lab environment, software, and REU activities such
as seminars. The objective is a goal for the remote REU programs. The norms
refer to embedded rules among members of the community. The division of labor
is how tasks are shared between members of the lab group. The community refers
to all members of the research lab and REU participants. The outcome of the
REU programs is the result of participation in this remote REU experience.

##  II Theoretical background

Our theoretical framework applies Cultural-Historical Activity Theory (CHAT)
[22, 23, 24] to help us reveal the challenges and obstacles in the remote REU
setting, as well as identify and describe the possible outcomes of research
practices in a remote environment. Engeström’s third-generation Cultural-
Historical Activity Theory (CHAT) argued that every human activity is goal-
oriented and mediated by different components such as tools, division of
labor, rules, and community (see Fig. 1).

The links between the elements indicate that they are dynamic and interact
with the other components to describe how students’ learning can be
experienced across diverse educational settings [24]. Subject refers to an
individual or group whose perspective is considered for analysis. For
instance, we identify each individual mentee as a subject with particular
personal inputs (e.g., emotions, family background). The objective is the goal
that is the focus of the activity system. In our study, objectives are
mentees’ goals for their REU projects and participation in undergraduate
research programs. Tools are material, symbolic, and conceptual resources used
to meditate between the student and their goals. For example, tools include
the physical lab environment, mathematics, software, professional development
seminars, and experimental kits. The community refers to the people who have
shared goals, including research mentors, lab group members, REU participants,
and possibly a larger scientific community. Norms refer to the values,
expectations, and guidelines for the subject to participate effectively as a
community member. An example of a norm within the research group might be how
often and by what means the mentee should reach out to the mentor or other
group members to receive help. The division of labor describes the different
roles performed by the community as they work toward a common objective. For
an REU, the division of labor is how different tasks are shared between REU
students and their mentors or other lab mates. Lastly, the outcome refers to
the results of participating in an activity such as experiencing the remote
REU programs.

One useful feature of a systems-level analysis is that it can reveal aspects
of the system that have conflicting influences on the activity. The observable
effects of this misalignment are referred to as tensions that result from an
underlying systemic contradiction. A plausible example would be a mentee who
wants to be recognized by their research mentor for doing a good job at their
research. If the mentee believes that asking questions is a sign of ignorance
(a norm) then they may avoid asking questions to make it appear they
understand more and gain their mentor’s approval. However, by not asking
questions and communicating with their mentor, they may be less likely to
progress in their scientific research and have a successful mentor-mentee
relationship, which ultimately lessens the chance they will receive the
recognition they desire for their accomplishments. Hence, there is a
contradiction between the norms (question-asking indicates ignorance) and
their use of the community (mentor) to help accomplish their goals
(recognition). Generally, tensions, problems, or frustrations that affect the
achievement of goals can be interpreted as contradictions within and between
nodes in the activity system. Identifying contradictions is important because
it provides a mechanism that explains why the problem occurred, which can be
used to improve remote REU activities in the future.

This study is a part of an in-depth examination of an REU from multiple
perspectives including psychosocial growth [25], challenges and outcomes (this
paper), and a more comprehensive description of the tools, communities, norms,
and division of labor that occurred with remote REU programs (forthcoming).
This paper focuses on findings around challenges and outcomes of the remote
research experiences from multiple interviews with undergraduate physics
mentees and their mentors who participated in a remote REU program in the
summer of 2020.

##  III Methodology

In this article, we explore REU programs from both mentees’ and their paired
mentors’ perspectives. Both mentors and mentees were aware of each other’s
participation in this study. The interviewer assured students and mentors that
their own perspectives would be confidential and not divulged. We adopted a
qualitative longitudinal phenomenographic approach [26, 27] combined with the
CHAT framework [24] to collect data throughout the REU program and after it
finished. The CHAT lens help us to organize and interpret relationships
between multiple components of the REU activity system. The phenomenographic
aspect of our study design aimed to investigate the variation in the ways that
the mentees talked about each components of the remote REU activity in
addition to the outcomes. The longitudinal design helped us to capture many
features of the activity system and examined students academic and
psychosocial growth over the time of the program, which was an aim of our
extensive analysis [25, 28].

###  III.1 Data collection

Figure 2: Timeline of longitudinal study that explored multiple facets of the
REU as an activity system. The bold words indicate intentional alignment
between the interview protocol and the CHAT framework. The interviews included
perspectives from mentees (light gray) and mentors (dark gray). Interviews 9
and 10 occurred after the REU was complete and emphasized many of the outcomes
that are discussed in this paper.

We sent 64 physics REU program coordinators an email asking if their REU would
be offered in a remote format in the summer of 2020. We received eight
positive answers. Mentees were recruited into this study by an invitation
email from their REU coordinators on behalf of us. After students volunteered
to participate in our study, we contacted their mentors. Our overall sample
included ten mentees and eight paired mentors from six REU programs.
Demographics of all participants are shown in Table 1. The sample of mentees
was gender and ethnically mixed, while the sample of mentors was all men. We
were unable to recruit women mentors. While the percentage of women among
physics and astronomy faculty members has grown in recent years, in 2019, 19%
of physics faculty members and 23% of astronomy faculty members were women
[29]. Since the COVID-19 pandemic started, all faculty were negatively
affected [30]. However, women faculty often faced greater challenges due to
additional responsibilities of caregiving for other family members [31, 32].
All mentees were physics majors. Additional information about the mentees’
projects can be found in the Appendix section (all participant names are
pseudonyms). The students were compensated with a $20 gift card for their
participation in each separate interview, which was sent weekly.

Mentees’ Characteristics |  |  Number ($N$=10)
---|---|---
Gender |  Women | 4
|  Men | 6
|  White | 5
Race |  Asian | 3
|  Mixed | 2
|  Rising senior | 7
Year of college |  Rising junior | 2
|  Rising sophomore | 1
Type of home |  Ph.D. granting institutions | 4
institutions |  Master’s granting institutions | 2
|  Bachelor’s granting institutions | 4
Mentors’ Characteristics |  |  Number ($N$=8)
Gender |  Men | 8
Type of REU |  Doctoral Universities | 6
institution |  Baccalaureate Colleges | 2
Table 1: Participants’ characteristics

All participants were individually interviewed at multiple points throughout
the REU program in the summer of 2020 and one time after the REU program
finished (interview nine with mentees and interview ten with mentors). We
video-recorded all interviews via Zoom with the permission of the
interviewees. Mentees’ interviews took between 60 and 90 minutes, while
mentors’ interviews took between 30 and 45 minutes. Overall, 94 interviews
were conducted. Fig. 2 shows the overall protocol content for each week of the
interview and how it was aligned with the CHAT framework.

###  III.2 Data analysis

Each interview was recorded and auto-transcribed with Zoom for analysis. After
the interviews were completed, the transcripts were cleaned to fix errors and
punctuation. The transcripts became the focus of our phenomenographic
analysis. To answer the research questions posed in this article, we focused
on the REU activity system as a whole to understand mentees’ desired and
observed outcomes and their challenges.

#### Analysis needed for research question 1.

To address research question 1, we needed to identify challenges that mentees
experienced during the REU program. Mentees were asked directly about
challenges in multiple interviews. However, many of these challenges also came
up when mentees reflected on their research experiences more broadly. The
language that mentees used to describe challenges included emotive
difficulties (e.g., “it’s kind of frustrating”), procedural difficulties(e.g.,
“when they’re not working right, I guess it has been one of the big hiccups”),
and social difficulties (e.g., “it’s hard to get feedback”).

The analysis process is divided into multiple steps, including data immersion,
identifying CHAT nodes and coding them, searching for sub-codes, and
reconstructing an activity system diagram to understand the challenge. The
process of data immersion started with transcribing, cleaning, and listening
to the interviews repeatedly. Analysis of the transcripts was executed using
Dedoose software [33]. In the primary analysis for the RQ1, DZ identified
contradictions based on what mentees explained to her when they were facing
challenges or other difficult situations. After several rounds of discussion
between DZ and co-another BMZ to reflect on the challenges, we noticed that
tensions could occur both within one element of the activity system, such as a
lack of community, or between components of the activity system, such as a
lack of connection between tools and objectives. We used the CHAT lens to
reconstruct the features that led to the particular challenge. CHAT helped us
to examine the interactions between the nodes that participants mentioned
during their research experience.

#### Analysis needed for Research question 2.

Mentees were asked directly about the outcomes of the REU experience at
successive points during the REU program and after it finished. Most of the
results from this research question emerged from interview 9 and 10 which
focused on the impact of REU experience on mentees’ academic goals from
mentors’ and mentees’ perspectives. To address research question 2, DZ
identified moments when participants talked about the outcomes of the REU
program based on what they achieved and how they thought this remote research
experience would impact their professional and academic decisions. DZ then
applied a phenomenographic analysis process by reviewing every excerpt under
the outcome node and searching for variation and significant themes. Next, DZ
sorted them into themes based on similarities. Our main goal when using
phenomenographic methods was to gain insights into our participants’
perspectives regarding desirable and possible REU outcomes. DZ met regularly
with BMZ to discuss the coding process. Finally, in the discussion (Sec. VI)
we compare our results elucidating the outcomes of remote research experiences
with previous findings on the outcomes of in-person research experiences.

##  IV Results on research question 1

What challenges were observed within the remote goal-directed REU activity?
The COVID-19 pandemic impacted both mentors and mentees who participated in
the remote REU program. They all joined this program with their own
expectations, concerns, and emotions about the new remote research experience.
In general, challenges included factors that may be due to unique
circumstances around COVID-19, particular features of the remote format, or
could occur in any research opportunity. For instance, Grace said, “I know we
were very delayed and getting our access to the supercomputers at the
beginning, which was a little frustrating”–Interview 9, (Solid State Physics).
This challenge was not only due to the remote REU format, since the
supercomputer access is typically remote. Additionally, COVID-19 likely have
disrupted other workers who were responsible in helping obtain supercomputer
access. This section will provide several examples of ways in which the remote
format both created unique challenges for students and exacerbated existing
difficulties for REU students.

Figure 3: The diagrammatic representation of various tensions between the
mentee and their objectives that has been disrupted, resulting in a challenge
within the remote REU experience. The dashed lines represent tension between
nodes.

#### Working from home introduced technical challenges.

Several mentees mentioned they had a hard time accessing the right
technologies, partly due to the lack of resources and partly due to the lack
of in-person contacts who could provide technical support. Fig. 3a is a
diagrammatic representation of the challenge showing that tools were not able
to mediate the research goals due to a lack of access to the proper tools and
community support to use those tools. For instance, Helen, who simulated the
decay process of short-lived isotopes, had some difficulty trying to “compile
different software… I think the actual processes aren’t that difficult, but
where I’ve struggled is like downloading the different software and like
trying to become familiar with this new software well. Like analyzing
something and I think you kind of understand what the general approach should
be, but figuring out how to analyze it with this new software has a learning
curve, and downloading it and trying to compile it was challenging… It’s hard
to do it remotely, I have reached out to my main professor and then also the
director because the director helped me set up a remote connection to access
the software on his Linux system since it wasn’t working on my Mac.”–Interview
4, (Nuclear Physics). According to her, she was struggling with software
issues most of the weeks during the REU program; “I have been frustrated with
the whole software thing”–Interview 6. Similarly, by week 3 of the REU
program, Grace said, “We’re still waiting for some access to their
supercomputers.” David said, “One of the most challenging things was getting
used to the way the software works because obviously with the modeling
software… Especially since I have to do it on my computer as opposed to a more
powerful computer.”–Interview 4, (Acoustic).

#### Working from home introduced motivational challenges.

The sudden transition from working in a lab environment to working at home in
pajama pants and hanging out with family members introduced several
motivational challenges. A lack of social interaction (working from home in
isolation) and lack of communication with other members of their research lab
also made some mentees feel discouraged. Fig. 3b represents that apart from
the remote workspace issues, the lack of work norms and communication reduces
the level of motivation and impacts mentees’ progress towards achieving their
goals. Mentees were usually asked by their mentors to embody the work norms of
a research lab (e.g., staying on task, reaching out). However, it was harder
in such a remote format without seeing other members of their lab embody those
same norms, or feeling some of the “peer pressure” to work. For instance,
Helen said, “In terms of seeing the end goal, I haven’t really necessarily
seen that. Sometimes that’s been hard since I feel like I need to know what
I’m doing is on the right track. But I think also I’m doing less than what I
want to do. That’s like working online. It’s kind of hard to know what point,
you should be working on. Like, what other people are up to in their programs
as much because you don’t get to see them every day and have the subtle
interaction” –Interview 6, (Nuclear Physics).

A lack of social interaction made Bruce feel more isolated, which reduced his
productivity. He said, “Being personally motivated was kind of difficult…
Because I was living at home. There are currently five people at home. Most of
the rooms are taken up by other people’s activities… There were certainly some
things that were distractions. Now that I’m looking back when I wished I had
gone outside more. Because I really kind of got clumped inside and that
probably wasn’t great for me. I felt like my mind got stuck sometimes because
of that. My parents and my sisters were constantly trying to ask for my time
as well as trying to convince me to go out and stuff like that. I lost a lot
of time in that sense as well”–Interview 3 & 9, (Quantum Nonlinear Optics).
Fig. 3c shows that how the lack of the physical and social lab environment of
Bruce impacted his motivation.

#### Information overload and lack of time introduced challenges.

Even the most motivated and organized mentees can get frustrated by the amount
of new essential knowledge that they have to learn in a limited time of the
REU program. As one example, Joshua had experienced frustration with his lack
of preparation. He said, “The reason is that I need to learn a lot of basic
background knowledge. So, that is hard”–Interview 9, (Nuclear Physics).
Similarly, Bruce said, “I am pretty limited on time… I know there’s a lot more
information to go on it, but I just haven’t really done that and dissuaded
from moving in that direction, just because of the limited time I
have”–Interview 7, (Quantum Nonlinear Optics). The mentees’ REU experience can
have competing goals such as learning more physics knowledge and completing
the research project goals. Frustration can arise when mentees are trying to
gain a deeper understanding of their project content from one side and still
need to finish their project at the end of the program from the other side.

While it can be difficult for mentors to determine when mentees feel
frustration during in-person research programs, it is even harder to
understand these issues in a remote format when mentees are not physically
present in a lab. Mentors don’t see mentees most of the time and they can
assume how are mentees doing based on their body language, and facial
expressions in Zoom meetings, or directly ask them about their challenges. The
second issue is the missing learning opportunities from not being physically
present in an environment all focused on a topic (the books, the chats, and
the whiteboards all provide learning opportunities in less formal ways). This
informal communication allows them to learn a large number of background
materials.

A lack of understanding can make a mentee feel like an outsider in the lab.
Namely, a lack of enough background knowledge on the project topic may lead
mentees to experience a poor sense of belonging to the research community.
Fig. 3d shows that broken links between the top portion of the triangle
combined with the lower portion of the triangle (e.g., sense of belonging to
the community). Norms also play a role given the mentee and mentor might have
differing expectation about the appropriate balance of learning content and
accomplishing research goals. Frieda talked about the efforts her mentor made
to identify and teach the essential knowledge, which supported her sense of
belonging.

For instance, Frieda talked about her mentor who asked her to join their big
research group meetings with all the other researchers in the field. She
recalled her mentor telling her that, “You’re going to come to these group
meetings and you’re not going to understand most of what they say, but that’s
okay. We’re going to get there.” She continued “Mostly I’ve been listening
because, I don’t know that much yet… I think that he just preparing me, so
hopefully I will [know] eventually and be able to contribute as well. After
each of those meetings, my mentor has met with another REU student and me and
worked through some of the terminology and stuff [that they talked about in a
big group meeting]. He’s doing a really good job making us feel like we can
understand it”–Interview 3, (High energy physics). Although this challenge is
likely an issue in every REU program, due to the remote setting and lack of
face-to-face interactions, mentees are likely to feel less connected and less
cared for or accepted by their lab group.

#### Lack of communication opportunities among REU students introduced
challenges.

All REU programs have the common goal of preparing undergraduate students for
careers in STEM fields by integrating them into research projects and building
scientific community of peers. Different REU programs may develop their own
plans and structures to achieve these goals. However, our data indicate that
REU participants persistently lacked community throughout the REU program.

Sometimes tensions in the REU system happened. For example, a particular REU
program or the REU activity may aim to pursue multiple goals at the same time,
such as introducing students to different research areas (e.g., professional
development activities) and creating a community of peers (e.g., opportunity
for all REU participants to get together). Fig. 3e shows in addition to
developing a community among REU students, the overarching goal of many
undergraduate programs is to provide a long-term benefit by presenting
professional development activities. Although, the professional development
activity brings everyone into the same Zoom room, it doesn’t really form
community among students. According to Bruce, “[Having a community] really has
kind of fallen by the wayside, to the point where there has not been a single
chat in the group chat since the first week.” He continued that during social
events and seminars, the REU students did not interact because they were
listening to lectures, and he wished they had some sort of ‘all groups’
meeting” where they could present their work to each other. Similarly, Frieda
said, “We have bi-weekly meetings on Tuesdays and Thursdays. At first, there
was one about grad schools. And then, we each gave five-minute presentations
about our research. Then there was one about each area of research that’s
going on at the REU institution. So they got a representative professor to
come talk for the meeting”–Interview 3, (High Energy Physics). Although the
REU coordinators aimed to provide career support during these group sessions,
these sessions sometimes had more of a focus on information delivery rather
than peer discussions and interactions between mentees.

#### Barriers in mentor-mentee relationships introduced challenges.

A good mentor-mentee relationship is key to having a successful REU
experience. Analysis of the data identified that most mentees maintained a
positive relationship with their mentors. However, due to a lack of tools and
lack of physical proximity in a remote lab setting, it was harder to form this
good mentor-mentee relationship. Over the course of the COVID-19 pandemic,
meeting over instant communication apps such as Zoom became the norm for
communicating remotely. However, these tools have limitations and
shortcomings. For example, the Zoom platform was blocked or had a bad
connection in China. Joshua’s mentor said, “We use Zoom. But, we also use
WeChat. Because sometimes Zoom does not work very well, we lost
connection”–Interview 10.

Having multiple weekly meetings was one of the productive norms among many REU
groups in a remote lab setting. Weekly check-ins were vital to support
mentees’ research goals and helping them develop more open communication with
their mentors. However, for some mentees, they felt they were not a priority
for their mentors and the way they communicated with their mentors was mostly
via email. For instance, Helen said, “We zoomed in the beginning, and I’ve
talked about what my role would be. Then we’ve just been emailing out the next
plan for what I’m doing. I know he has two kids. I think that it’s just been
like a little hard for him to stay in touch as much”–Interview 3, (Nuclear
Physics). This challenge was probably unique to COVID-19 due to the juggling
of childcare, school, and full-time work responsibilities with children at
home. Fig. 3f shows the lack of communication tools and physical proximity can
impact both mentor’s and mentee’s contribution to the new norm of weekly
check-ins in the online setting.

Ivan also didn’t have any weekly meetings over Zoom, but for a different
reason. Ivan lived in a very different time zone from his mentor. Ivan said,
“We do not have virtual meetings. We send emails, and my mentor sends emails
back”–Interview 3, (Nuclear Physics). Ivan’s mentor described challenges when
collaborating with him, saying, “I should have told the REU director that I
couldn’t manage a student who is in a 14 hour different time zone.
Unfortunately, the only available hours for me to meet with Ivan overlapped
with my meeting hours with people at CERN. In the past, I had always managed
the REU time commitments by meeting with people in CERN in the morning and the
REU students in the afternoon. Because they were local and that was just not
an option [this summer] and there just weren’t enough hours in the day for me
to meet with Ivan”–Interview 10.

#### Differing expectations between mentors and mentees introduced
motivational challenges.

Barriers also arose when the expectations of mentees appeared to be different
from mentors’ expectations. Ivan, who was also enrolled in summer courses at
his home institution, said, “I had to take lots of courses and I did not have
too much time to do research… I was too busy to do that… My mentor gives me
lots of information that I can read it on my own. I have to use lots of time
to do it. He treats me as a researcher and a physicist and he gave me lots of
raw materials and I have to do it on my own”–Interview 6, (Nuclear Physics).

On the other hand, his mentor thought “It was hard to really get his attention
and get him working on the projects as a remote student with the time zone
difference and apparently a miscommunication about the amount of time that he
was supposed to have budgeted for the REU experience… I don’t really know that
he has a good idea of what it’s like to be a researcher and to work in high
energy physics because the validity to pursue a research question fairly
independently is a pretty central aspect to that experience and I don’t think
he got there”–Interview 10. The representation for this challenge is very
similar to Fig. 3b that shows that program coordinators, mentors, and mentees
need to communicate over norms and rules in the research labs in order
properly allocate time and gain the major goal which is having a positive
research experience and finish the research project. For Ivan, he stated, “I
am just a follower in this program”–Interview 9, (Nuclear Physics). However,
his mentor expected him to do some more independent work and dedicated more
time to his research. Interestingly, differences in expectations between Ivan
and his mentor were about more than how time was used on the project. Ivan
thought his research group was not serious and formal, which perhaps affected
the time and effort he dedicated to his REU. He said his lab members were
“Very kind and funny, [but] it is just different from my
expectations”–Interview 3, (Nuclear Physics). He said, “My mentor gives me
some tasks or some problem to solve and some information about those tasks… My
mentor just sent me some instructions and if I am working along those
directions, I get the work done”–Interview 3 & 6\. He wished his mentor asked
him to bring “some ideas about doing some research.”. His feeling about his
project did not change over the REU programs. After the REU program finished,
he said, “I just follow some things just have been done before. So maybe not
very big [research], it is maybe just a little part in this size”–Interview 9,
(Nuclear Physics). Meanwhile his mentor thought “He might have been a little
bit disappointed with the research project he had because I think he wanted to
look at big overarching questions, and we wanted him to look at the data that
came out and some checks… We all have dreams of sitting in a room and dreaming
up some theory that explains everything, and there is nothing tedious about
that process. However, when you actually get to work on an experiment, that is
tedious, and it is required to make progress. You have to learn a certain
amount of discipline and commitment to seeing through the tedious part so you
can get to the fun parts of having a discovery or learning
something”–Interview 10. The representation for this tension in Fig. 3g shows
that both the mentor and mentee did not speak regarding their expectations and
did not communicate properly as a group to consider each other’s goals for
this REU program. For Ivan, he expected to do a bigger project and test his
ideas in the project other than following some instructions and solving some
problems. On the other hand, his mentor expected him to check some codes.

The mismatched expectations and norms between mentors and mentees were not a
very common theme across our data set. However, the authors mentioned this
challenge here since the mentor-mentee’s communication is extremely important
factor in the students’ REU experience and their decisions around their future
career.

##  V Results on research question 2

What are some of the outcomes of remote REU programs?
REU programs are multifaceted learning activities that involve challenges and
benefits. Outcomes are the final part of the goal-oriented REU activity
system. A few weeks after the program finished, in the 9${}^{\textrm{th}}$
interview, we asked all mentees to describe the outcomes of their REU
experience. We found that almost every mentee, despite their different
projects and different circumstances during COVID-19, described having
achieved their goals as a result of their research experience. They all stated
that attending the REU program was the right decision in their academic
journey. The following subsections explain several of the specific outcomes of
the remote research experience (see Fig. 4).

Figure 4: Overview of the outcomes of the remote REU programs.

#### Mentees described learning physics content.

All mentees expressed this remote research experience deepened their knowledge
in discipline-specific project content and also introduced them to new areas.
An example is Grace, who said, “I am enjoying it a lot. I am learning a lot
about chemistry and biology and just how much physics intersects with
everything, and how it is fundamental science. I definitely had some
frustrations in some of the earlier weeks reading some of the papers and not
understanding them… Feeling like I had to understand everything that was being
thrown at me. But, overall, I am definitely enjoying learning these, learning
how to use the program, learning the chemistry and physics involved has been
really interesting and enlightening”–Interview 7, (Solid State Physics).
Joshua, an international student, talked about how much he learned from
extracurricular activities during the REU program; “I learned the skills such
as organizing the introduction or other parts of a paper better. In GRE
preparation courses, I learned some required physics concepts in English…
These are important for my future career. In seminars, I can broaden my
horizons, such as I learned some new physics knowledge in other physical
areas”–Interview 7, (Nuclear Physics).

#### Mentees described their future career decisions.

The remote REU experience positively impacted most mentees’ future career
decisions. Most of the mentees ($N$=8) said the REU experience helped them
understand the nature of research work and think about their future career in
a STEM field. David stated, “Especially because I was able to get a better
idea. Not only I was able to learn more about the software and about the
research going on in one of the fields that I am interested in, but also
because I was able just to get a better understanding of how research works
and that sort of dynamic. I get a better feel for what doing research with a
professor looks like. So that I have a better understanding of how that will
work when I am, for example, in graduate school doing research”–Interview 9,
(Acoustic). Caleb learned that he might prefer a balance of hands-on and
computer-based work. He explained he enjoys computational theoretical
research, but “I would like to be in a lab-type setting; some days, I am doing
hands-on research, and other days, I am doing computational stuff. My biggest
takeaway from the REU program is that I do not think I could just be on a
computer for eight hours a day”–Interview 6, (Atomic physics). Due to the lack
of information about different career choices and fields at the beginning of
the REU program, some mentees were uncertain about they wanted to do. For
instance, Ivan said, he is interested in studying nuclear physics in graduate
school now, but “Before the REU program, I did not even know what I was
interested in because there were too many fields”–Interview 9, (Nuclear
physics).

#### Mentees intend to apply for graduate school.

After finishing the REU program, most mentees ($N$=9) were sure that they
would like to apply to graduate programs. Frieda said, “It was really good for
where I was in my education and made me seriously consider this field as a
graduate school field, which not that I was not considering it, but I am much
more serious about considering it now”–Interview 9, (High energy physics).
Similarly, Andrew describes how the REU contributed to desire to do research
in graduate school. Andrew said, the REU program “has influenced me in the
fact that it makes me want to do graduate school and research more. Because
beforehand, I was like, I do not know if that is going to be a lifestyle I
want to get into, because I did not know. I have not done research before, so
I did not know if it [would] be something I enjoyed or something I [would]
absolutely hate. So after doing this REU, I was kind of like, I enjoyed this.
I can see myself doing that”–Interview 9, (Nuclear Physics).

Bruce was unsure about his future career during most weeks of the REU program.
However, in the last interview, he said that his mentor provided him with a
new perspective and insight on doing research. He said, “I definitely think
that [my REU mentor] has a strong influence on making my decision about
graduate school”–Interview 9, (Quantum Nonlinear Optics). Bruce explained that
this influence came from stories that his mentor told him about his previous
students’ career decisions and also from giving him a sense of how he does
research. He said, “There’s just so many different ways that they’re either
the professors or they’re working in research labs or in different things…
It’s just stuff that I really would like to be doing in the future as a
career… I think another [influence] is just he’s giving me a sense of how he
does research… He’s not always super stressed about everything. He’s not
overworking himself. This idea that you can be a researcher and not overwork
yourself all the time is also quite enticing”–Interview 9, (Quantum Nonlinear
Optics).

#### Mentees described engaging in new learning opportunities in the STEM
fields.

Some mentees explained how they planned to tailor their interests toward their
future career goals by choosing elective classes, attending the seminars, or
reading new related articles and books after the REU finished. One example was
Joshua (Nuclear) who decided to take some elective courses in astrophysics and
nuclear physics the following semester to narrow down his interests.

Other mentees tried to read and search through the literature related to their
summer research experience. For instance, Helen said, “I like to read
articles. I am part of the American Nuclear Society and get updates and stay
informed, especially around medical physics. I always have liked reading
Physics World news about medical physics, listening to podcasts, and
connecting with people. So I have definitely been more interested in learning
more in that area after that REU”–Interview 9, (Nuclear Physics). Similarly,
Andrew said, “What I did over the summer made me want to learn more about
that… I got a book on nuclear physics, and I was kind of reading that as the
program went along”–Interview 9, (Nuclear Physics).

#### Mentees valued doing independent research.

Due to the lack of in-person interaction between mentees and other members of
the lab community, they are had to engage in substantial independent learning.
For instance, Caleb said, “[The remote format] makes [the research] a lot more
independent. Because you are still able to reach out via email, Zoom, or
whatever platform you use, but there is that added step of composing the email
asking for Zoom chat. Whereas in in-person, just like walked out to the office
or they are in the room with you, and just like look over your shoulder. So,
it adds a lot more independence”–Interview 7, (Atomic Physics). Likewise,
David explained that he became more independent in his learning since, “I will
be in the middle of a project, trying to figure something out in the middle of
the day. Then I will just go look it up and try to figure it out. I usually
use it as a resource on my own, and I think that might be a bit different if I
was working with the professor, more closely in a physical
environment”–Interview 7, (Acoustic). Although, independence is an important
factor in any research work, some mentees were hoping to be more reliant on
their mentors, such as Ivan’s experience who had a lot of resources from his
mentor and wished his mentor let him bring his ideas about doing some
research, instead of “I am just a follower in this program.” More details
describe in section IV.

#### Mentees were comfortable with level of their knowledge and able to
engagement in scientific practices.

Regardless of the project area, every REU program aims to provide
opportunities for students to develop research practices, and become community
members by supporting them to work collaboratively. During interview nine
after the REU program finished, eight mentees described feeling more
comfortable with their ability to get involved in their community (e.g., by
asking questions and learning new concepts) and became more comfortable with
the level of their knowledge. For instance, Grace said, “[I can] ask a
question and not feel like I am asking a stupid question. It is very hard to
look back on my near past self and reflect because… It is hard to see [how
much I] change”–Interview 9, (Solid State Physics). David said, “I had a
better understanding of what was going on. So I was able to contribute a lot
more”–Interview 9, (Acoustic). Six out of eight mentors also were very happy
and satisfy with their mentees’ engagement in scientific practices.

#### Mentees described increasing in their sense of ownership of their
projects.

In our previously published extensive study, some mentees talked about a sense
of ownership as a factor that linked to their sense of belonging [25] through
scientific contribution. We found that, in order to achieve the ownership
construct, mentees need to produce potentially new results for their project.
Additionally, five mentors described how they intentionally helped mentees’
sense of ownership by offering them responsibility in their project. These
mentors explained that their mentoring philosophy includes helping mentees
increase their sense of ownership. Three mentors mentioned giving their
mentees the freedom to challenge themselves intellectually. Caleb’s mentor
said, “By respecting and giving them great freedom, that is, say if he had
ideas on how to do things. I would say, ‘Do it and show me.’ I did not have to
tell them this is how you do it step by step. In the beginning, yes, but it
did not take long to get going on his own and then come up with his own
improvements and extensions and search. He was very good at that. So I
certainly think that encouraged him. I think he liked that. I think he liked
the research. But, you would have to ask him, I guess”–Interview 10.

Two other mentors cultivated ownership by providing the bigger picture of the
project and giving students responsibilities. As Frieda’s mentor explained,
“She had a project, and it was very well specified that this was her project.
There was no one else working on it, and again, I am working with graduate
students and postdocs, making sure they were not working on the project. So,
it was hers and hers alone”–Interview 10. No one else would do what Frieda was
working on so the team needed her to get her part done.

#### Mentees described growth in their sense of belonging, self-efficacy, and
physics identity.

A higher level of sense of belonging and identity is associated with improved
academic performance and possibly enhanced persistence in the field of
physics. Many mentees reported growth in their psychosocial attributes, such
as a sense of belonging, self-efficacy, and physics identity. We have already
described the psychosocial benefits of these remote REU experiences in more
detail in another paper [25]. However, for completeness we summarize these
benefits here because they were an important outcome of the remote REU
experience. We found having a supportive lab research community and mentor-
mentee relationship helped most mentees to exhibit psychosocial growth.

Almost all mentees reported a higher level of sense of belonging to their lab
community and possibly to the disciplinary community at the end of the REU
program. David, who made a resonator model to learn an acoustic modeling
software tool, said he had a “place in the field now”–Interview 9, (Acoustic).
He thought his research experience over the summer positively helped him feel
like a part of the physics community. He said, “Just because I have been able
to experience and contribute as well as being able to look through the work
that other people have done in the field a lot more in-depth and having done
my little bit of work in the field helps me to understand better the work that
others have done in it… Even now that I am not doing that research full time,
I feel much more in part of that than when I was only taking classes
before”–Interview 9, (Acoustic). Frieda said, “I definitely consider myself
part of the physics community… A lot more than I did at the beginning. Because
now I feel I know a lot about it, I have gotten a lot done, I am informed, and
I feel more in it, like I am more submerged in it personally. Maybe you could
say my physics identity has gone from like an amateur to like a beginner to
intermediate now. I feel like I am actually part of the field”–Interview 9,
(High Energy Physics). Both Helen and Bruce, who had a lower level of sense of
belonging during the program, believed they felt more belonging after the
program finished since they were back on campus and communicating with more
people. For example, Bruce said, “[I became] more comfortable in the physics
community and because of that I became less stressed about interacting with
the community… I feel I’m somewhat part of the bigger physics group because I
understand how it works now. I am collaborating with some people. I know a
finite number of researchers and exactly what they’re doing. I’ve worked with
them or I know the kind of research they’re doing, so that’s why I feel like
that.”

Findings for the self-efficacy construct indicate that self-efficacy stemmed
from various sources, such as getting more physics content knowledge, doing
independent research, producing new results, and scientific communication with
other members of the community. For instance, David said, “I was just trying
to learn to catch up and understand what was going on. After I had been doing
it for about a month, I felt a lot more confident in what I had been able to
learn… I feel a lot more comfortable than before because I had a chance to
work full-time doing physics, even if it was just for summer. Since I enjoyed
that, I feel a lot more confident proceeding with that as a career
goal”–Interview 9, (Acoustic). His confidence stemmed from getting more
physics knowledge and contributing to his research group projects.

Sense of belonging to the physics labs or community and physics identity both
describe the personal experience around varying degrees of attachment. For
example, physics identity can be defined as the self-recognition of being a
physics kind of person or a physicist. For most mentees, developing a more
robust physics and researcher identity resulted from a stronger sense of
belonging and stronger self-efficacy [25]. Positive psychosocial changes that
resulted from research also impacted career decisions. During the REU program,
six mentees believed they were physics apprentices who were on the right path
and needed more knowledge and experience to be a physicist. After the REU
program ended, five mentees felt a stronger identity around physics and doing
research in the field. For instance, Ivan said, “The REU has taught me
something about confidence in doing research. The REU has made you think
yourself as a physicist”–Interview 9, (Nuclear Physics).

##  VI Discussion

We had two research questions in this study. The first research question was:
What contradiction were observed within the remote goal-directed REU activity?
We used the combination of phenomenography and CHAT framework to understand
the challenges in a new remote complex educational system. Based on our semi-
structured interviews, mentees shared with us different types of challenges
they faced during the remote REU program. As we outline in section IV,
COVID-19 and the remote format impacted students’ learning. For instance,
essential components of effective learning, such as easy availability of
helpful feedback and having a good mentor-mentee relationship, were harder in
a remote format.

Goals expressed in literature for students participating in in-person UREs [18] |  Short term outcomes of the remote Research Experiences for Undergraduates based on our data
---|---
Increase mentee’s retention in STEM fields |  -Clarifying future career interest
|  -Applying for graduate school
|  -Engaging to new learning opportunities in the STEM field
Promote STEM disciplinary knowledge |  -Learning physics content
|  -Appreciating value of doing research independently
|  -Utilizing disciplinary research practices by asking questions and
directing projects
Integrate students into STEM culture |  -Finding interest in physics field
|  -Expressing higher level of sense of ownership of their project during REU
experience
|  -Becoming enculturated in the physics community and expressing a stronger
sense of belonging, self-efficacy, and physics identity as a result of the
remote REU experience [25]
Table 2: Comparison between the primary goals for in-person UREs and the
actual outcomes for the remote REU program.

Our second research question asked: What are some of the outcomes of the
remote REU program during COVID-19? Although, the first research question
identified some challenges, our data still showed that the outcomes of the
remote REU program are not too far from the generally accepted benefits of any
in-person undergraduate research experience. Table 2 compares the primary
goals for UREs described in the literature [18] (left column) with outcomes of
the remote research experience from our findings (right column). It should be
noted that mentees describe outcomes that are short term (e.g., preparing
graduate school application materials) and long-term (e.g., future career
decisions). Most previous studies have focused on three major outcomes for in-
person UREs: increasing retention and persistence in STEM, promoting STEM
disciplinary knowledge and practices, and integrating students into STEM
culture [34, 35, 14]. We use these same three categories in Table 2 and link
our results (Sec. V) to each of those categories.

#### Increasing mentees’ retention in STEM fields.

Undergraduate research experiences can impact students’ retention in STEM
degree programs and provide them with a new way of thinking about their future
career paths in STEM fields. In-person UREs often aim to help mentees
understand what it means to do research and what a science career might look
like [18]. The outcomes associated with this goal across our data included:
clarifying future career interests; applying for graduate school; and engaging
to new learning opportunities in the STEM field. This result also is
consistent with the evidence in substantial prior research that has examined
the influence of UREs on students’ long term outcomes, such as clarifying
future career goals [10, 9, 11].

#### Promoting STEM disciplinary knowledge and practices.

UREs can help students to develop new skills, learn new knowledge, and engage
in the practices of their STEM discipline, such as using computational models
or analyzing and interpreting data. The outcomes associated with this goal in
our data include learning content knowledge through the REU experience,
gaining independence as a researcher, and developing questioning skills. In
terms of the activity system CHAT triangle, the disciplinary knowledge and
practices would be considered research tools that support the research
objectives of a student’s project. Several previous studies on UREs outcomes
indicated that meaningful research practices could facilitate students’
learning and development of their technical knowledge in addition to promoting
their communication skills [36, 11, 6, 18]. Similarly, in our data multiple
mentees said they felt like they better understood physics concepts and what
was going on in their lab research group. Besides, most mentees were
comfortable reaching out to lab members for help.

#### Integrating students into STEM culture.

Other studies of in-person UREs mentioned that undergraduate research
experiences could strengthen students’ motivation and interest in STEM
culture. Here, integration into STEM can be thought of as a psychosocial
growth process (e.g., a gain in physics identity and sense of belonging) where
students learn the values of the discipline. Our findings around integrating
mentees into STEM culture include: developing interest in particular subfields
of physics, a sense of ownership of their project, and growth of psychosocial
constructs.

Integrating into STEM culture requires integration into a community. There is
considerable evidence across our data that positive remote REU experiences
require community interactions to achieve desired outcomes. Nevertheless, as
described in the challenges in Sec. IV, a lack of community and communication
with other REU participants persisted throughout the REU program.

##  VII CONCLUSIONS AND RECOMMENDATIONS

Prior to early 2020, the vast majority of articles about online education
debated whether such methods were effective and whether academia should more
broadly use remote learning. In a perfect world, maybe! However, when the
COVID-19 pandemic began in early 2020, many institutions rapidly experienced a
remote learning transition. In this study, We attempted to elucidate some of
the challenges and outcomes associated with remote research experiences. Our
data shows that all participants described their remote experience as a “real”
research opportunity that achieved desirable outcomes. The remote format still
allowed mentees to meet their personal goals by the end of the program.

Following here we discuss recommendations to address challenges (see Sec.
(IV)) and increase the quality of remote research opportunities in the future.

Recommendation 1: “Provide technical support to help students working from
home.” Students can face technical challenges at home due to a lack of
resources and support. Highlighting the importance of technology in a remote
research setting, it is necessary to provide more human support and more
training than what they received in using technology effectively and
fulfilling their project goals. Students need more technical support to
practice research and learn physics content. The training documentation
including manuals, software handouts, and resources could be available online
before the program starts. In addition, to facilitate REU students’ engagement
with technology and resources immediately upon entry into the REU program,
having a point of contact who can respond quickly for technical related help
(e.g., software installation) would be helpful. For example, this could be a
graduate student or a computer support technician.

Recommendation 2: “Provide emotional and environmental support to help
students working from home.” Working remotely brings its own set of norms and
challenges that may impact students’ productivity. Lack of motivation may be
intrinsic to a remote research experience, but it’s important to identify
these challenges in order to mitigate future problems and increase self-
motivation. Mentors and REU coordinators need to ensure that mentees have all
the physical and social tools they need (e.g., dedicated office space, a good
chair, and good communication technology to facilitate communication) to make
progress and handle the isolation challenges. In addition, mentors and REU
coordinators can plan social activities outside of their research projects
such as an online movie night, a book club, an online yoga class, online game-
night, or other events so students can share their daily life together.
Mentors and coordinators could offer peer check-in opportunities or schedule
times that REU students can work together in the same Zoom room. Students need
to support and encourage each other when facing similar challenges. These
informal interactions could make students feel connected to their community
and increase their motivation.

Recommendation 3: “Define clusters of planning skills that are important when
doing research remotely.” Mentors and REU coordinators can develop a mentoring
strategy for helping their students to move forward in their research
experience in addition to providing them opportunities to gain enough new
content knowledge. Mentors could help mentees to organize their resources. For
example, mentors could differentiate between resources that are essential to
achieve the project goals and which are optional. In addition, they can
schedule more short check-in meetings. Sometimes, having a scheduled meeting
at a certain time helps students prepare to work around it and manage their
time. Mentors can set out clear goals and deadlines and help mentees divide
their research work into manageable chunks. This guidance will help students
not get overwhelmed by the extensive amount of new knowledge and what they
have to do.

Recommendation 4: “Provide communication opportunities among REU students.”
Mentors can require each student to make a presentation at a weekly seminar,
with one or two faculty assigned and all the other REU students to ask
questions and provide feedback. Students need more communication in order to
get familiar with the other community members and the broad range of research
they are doing. Mentors and REU coordinators can create structured activities
to facilitate REU student interactions.

As a part of this community experience in the remote REU setting, mentors
should help mentees to strengthen their peer community and get them in touch
with other mentees in the program. For instance, ask all students to gather
for lunch once a week to discuss various aspects of remote research and life
events. However, this recommendation takes time to build trust among students.
Mentors also can ask students to develop and share with peers a work plan that
includes short-term and long-term goals for their projects and provide each
other feedback. Building these communities may impact students’ future career
decision. We recommend mentors ask students to work on a common goal through
creative activities and communication. For instance, making an REU T-shirt
requires lots of group work and informal communication. The graduate school
preparation sessions could be served as another form of opportunity that
students can share their resources, support each other and exchange their
learning experiences.

Recommendation 5: “Find the best technology to communicate with students.” It
is important for both mentors and mentees to contribute to the new norm by
keeping communication ways open. Mentors can define a cluster of social skills
that are important in doing research remotely. For instance, mentors and
mentees can discuss these questions: how often can the student meet their
mentors? How often can the student expect to receive feedback for their
progress? How long will the meeting be? How should mentees document their
progress? Learning about these norms may also benefit students who want to go
to graduate school and provide an opportunity for them to learn how to
communicate with their Ph.D. advisor.

Recommendation 6: “Discuss about expectations and goals.” It is important to
understand the mentees’ previous educational experiences, future career goals,
and their REU goals, and identify a potential barrier to these goals. For
instance, the programs can be explicit about their goals, so students
understand what they are getting into.Besides, mentors can be open to hearing
students’ experiences and perspectives around doing research. Mentors and
mentees need to communicate clearly from the start about their roles,
expectations, and responsibilities. For instance, mentors can define project
goals and expected outcomes at the start of the REU programs. We believe these
communication opportunities help students to take more ownership of their
projects and their learning.

In summary, we hope our study provides the PER community with an insightful
introduction to the challenges and different outcomes of remote REU programs.
We also hope this study provides encouragement and practical advice for REU
mentors and coordinators to design programs that foster the best outcomes and
avoid likely pitfalls.

##  VIII ACKNOWLEDGEMENT

The authors would like to thank all participants who shared their REU
experiences. Without their consistent participation during the early months of
the pandemic this work would not have been possible. This work was supported
by the National Science Foundation under Grant No. 1846321.

##  Appendix A Projects’ characteristics

Table 3: Projects’ characteristics. Name | Area of research | Description of project
---|---|---
Andrew | Nuclear Physics | Understand the efficiency of detector, learn about details of the old nuclear
|  | reaction simulations, and refine the new simulation
Bruce | Quantum Nonlinear | Numerically model quantum optical devices, learn PyBoard coding, use digital
| optics | time delay and construct the circuit with equipment that shipped to his home
Caleb | Atomic Physics | Examine at the atomic structure and different spectroscopies and make a model for
|  | specific properties
David | Acoustic | Make a basic resonator model to learn the modeling program and then make an
|  | acoustic model of a reed instrument
Emma | Physics Education | She had two projects: 1-PER: science outreach and 2- Build a 3D-printed particle
| and Particle Physics | trap and work with electronics that shipped to her home to make circuits
Frieda | High Energy Physics | Use different models in high-energy physics to predict the probability of different
|  | decay modes in collisions
Grace | Solid State Physics | Learn the density functional theory and model certain molecules and look at the
|  | dynamics of the system
Helen | Nuclear Physics | Simulate the decay process of short lived isotopes, learn about different isotopes
|  | and different spectra fields, and literature half lives
Ivan | Nuclear Physics | Gain knowledge about CMS and LHC and use simulations and experimental data to
|  | refine the codes for detection of the charged particles in a large collider experiment
Joshua | Nuclear Physics | Understand neutron mirror model and add new equations into the old code to solve
|  | problems related to nuclear physics

## References

  * Council [1999] N. R. Council,  _Transforming Undergraduate Education in Science, Mathematics, Engineering, and Technology_ (The National Academies Press, Washington, DC, 1999).
  * Council [2002] N. R. Council,  _Improving undergraduate instruction in Science, Technology, Engineering, and Mathematics: report of a workshop_ (The National Academies Press, Washington, DC, 2002).
  * Wenzel and Karukstis [2004] T. J. Wenzel and K. K. Karukstis, J. Chem. Educ. 81, 468 (2004), publisher: American Chemical Society.
  * Kenny  _et al._ [2001] S. S. Kenny, E. Thomas, W. Katkin, M. Lemming, P. Smith, M. Glaser, and W. Gross,  _Reinventing undergraduate education: three years after the Boyer report_ (Stony Brook University: Office of the President, NY, 2001).
  * Kuh [2008] G. D. Kuh,  _High-impact educational practices: what they are, who has access to them, and why they matter_ (Association of American Colleges and Universities, Washington, D.C., 2008).
  * Kardash [2000] C. M. Kardash, J. Educ. Psychol. 92, 191 (2000).
  * Lopatto [2007] D. Lopatto, CBE Life Sci. Educ. 6, 297 (2007).
  * Hathaway  _et al._ [2002] R. S. Hathaway, B. R. A. Nagda, and S. R. Gregerman, J. Coll. Stud. Dev. 43, 614 (2002).
  * Seymour  _et al._ [2004] E. Seymour, A.-B. Hunter, S. L. Laursen, and T. DeAntoni, Sci. Educ. 88 (2004).
  * Lopatto [2004] D. Lopatto, CBE Life Sci. Educ. 3, 270 (2004).
  * Hunter  _et al._ [2007] A.-B. Hunter, S. L. Laursen, and E. Seymour, Sci. Educ. 91, 36 (2007).
  * Laursen  _et al._ [2010] S. Laursen, A.-B. Hunter, E. Seymour, H. Thiry, and G. Melton,  _Undergraduate research in the sciences: engaging students in real science_ (Jossey-Bass, 2010).
  * Estrada  _et al._ [2011] M. Estrada, A. Woodcock, P. R. Hernandez, and P. W. Schultz, J. Educ. Psychol. 103, 206 (2011).
  * Graham  _et al._ [2013] M. J. Graham, J. Frederick, A. Byars-Winston, A.-B. Hunter, and J. Handelsman, Science 341, 1455 (2013).
  * Hausmann  _et al._ [2007] L. R. M. Hausmann, J. W. Schofield, and R. L. Woods, Res. High. Educ. 48, 803 (2007).
  * Dolan and Johnson [2009] E. Dolan and D. Johnson, J. Sci. Educ. Technol. 18, 487 (2009).
  * Eagan  _et al._ [2013] M. K. Eagan, S. Hurtado, M. J. Chang, G. A. Garcia, F. A. Herrera, and J. C. Garibay, Am. Educ. Res. J. 50, 683 (2013).
  * of Sciences  _et al._ [2017] N. A. of Sciences, Medicine, and Engineering,  _Undergraduate Research Experiences for STEM students: successes, challenges, and opportunities_ , edited by J. Gentile, K. Brenner, and A. Stephens (The National Academies Press, Washington, DC, 2017).
  * Blockus [2016] L. Blockus, National Academies of Sciences, Engineering, and Medicine (NASEM) , 31 (2016).
  * Russell  _et al._ [2007a] S. H. Russell, M. P. Hancock, and J. McCullough, Science 316, 548 (2007a).
  * Forrester [2021] N. Forrester, Nature (2021), DOI: 10.1038/d41586-021-01209-2.
  * Leont’ev [1974] A. N. Leont’ev, Soviet Psychology 13, 4 (1974), publisher: Routledge.
  * Roth and Lee [2007] W.-M. Roth and Y.-J. Lee, Rev. Educ. Res. 77, 186 (2007), publisher: American Educational Research Association.
  * Engeström [1987] Y. Engeström,  _Learning by expanding: A activity-theoretical approach to developmental research_ (Cambridge University Press, New York, NY, 1987).
  * Zohrabi-Alaee  _et al._ [2022] D. Zohrabi-Alaee, M. K. Campbell, and B. M. Zwickl, Phys. Rev. Phys. Educ. Res. 18, 010101 (2022).
  * Marton and Booth [1997a] F. Marton and S. A. Booth,  _Learning and awareness_ (Mahwah, N.J. : L. Erlbaum Associates, 1997).
  * Marton [2000] F. Marton, Phenomenography , 102 (2000).
  * Zohrabi-Alaee and Zwickl [2021] D. Zohrabi-Alaee and B. M. Zwickl, in  _Physics Education Research Conference 2021_ , PER Conference (Virtual Conference, 2021) pp. 480–485.
  * Helba  _et al._ [2019] C. Helba, A. M. Porter, S. Nicholson, and R. Ivie, “Women among physics and astronomy faculty results from the 2018 academic workforce survey,”  (2019).
  * Walsh and Tyler [2022] C. Walsh and J. Tyler, “Self-reported changes in quality of work as a result of the covid-19 pandemic for faculty members in physics and astronomy,”  (2022).
  * Myers  _et al._ [2020] K. R. Myers, W. Y. Tham, Y. Yin, N. Cohodes, J. G. Thursby, M. C. Thursby, P. Schiffer, J. T. Walsh, K. R. Lakhani, and D. Wang, Nat. Hum. Behav. 4, 880 (2020).
  * Krukowski  _et al._ [2021] R. Krukowski, R. Jagsi, and M. Cardel, J. Womens Health 30, 341 (2021).
  * Deddose [2018] Deddose, “Dedoose version 8.0.35, web application for managing, analyzing, and presenting qualitative and mixed method research data,”  (2018).
  * Nagda  _et al._ [1998] B. A. Nagda, S. R. Gregerman, J. Jonides, W. von Hippel, and J. S. Lerner, Rev. High. Ed. 22, 55 (1998).
  * Rodenbusch  _et al._ [2016] S. E. Rodenbusch, P. R. Hernandez, S. L. Simmons, and E. L. Dolan, CBE Life Sci. Educ. 15 (2016).
  * Junge  _et al._ [2010] B. Junge, C. Quiñones, J. Kakietek, D. Teodorescu, and P. Marsteller, CBE Life Sci. Educ. 9, 119 (2010).
  * Vygotsky  _et al._ [1978] L. S. Vygotsky, M. Cole, V. John-Steiner, S. Scribner, and E. Souberman,  _Mind in society: development of higher psychological processes_ (Harvard University Press, 1978).
  * Gorgi [2009] A. Gorgi,  _The descriptive phenomenological method in psychology: a modified Husserlian approach_ (Duquesne University, Pittsburg, PA, 2009).
  * Groenewald [2004] T. Groenewald, Int. J. Qual. Methods 3, 42 (2004).
  * Engeström [2001] Y. Engeström, J. Educ. Work. 14, 133 (2001).
  * M. Hubenthal and Frassetto [2007] R. A. M. Hubenthal, J. Taber and A. Frassetto, in  _Fall Meeting Abstracts_ , Vol. 2007 (2007) pp. 33–1226.
  * Pascarella and Terenzini [2005] E. T. Pascarella and P. T. Terenzini,  _How College Affects Students: a third decade of research_ , Vol. 2 (Jossey-Bass, San Francisco, CA, 2005).
  * Pascarella and Terenzini [1991] E. T. Pascarella and P. T. Terenzini,  _How college affects students: findings and insights from twenty years of research_ (Jossey-Bass Inc., San Francisco, CA, 1991).
  * Lopatto  _et al._ [2008] D. Lopatto, C. Alvarez, D. Barnard, C. Chandrasekaran, H. M. Chung, C. Du, T. Eckdahl, A. L. Goodman, C. Hauser, C. J. Jones, O. R. Kopp, G. A. Kuleck, G. McNeil, R. Morris, J. L. Myka, A. Nagengast, P. J. Overvoorde, J. L. Poet, K. Reed, G. Regisford, D. Revie, A. Rosenwald, K. Saville, M. Shaw, G. R. Skuse, C. Smith, M. Smith, M. Spratt, J. Stamm, J. S. Thompson, B. A. Wilson, C. Witkowski, J. Youngblom, W. Leung, C. D. Shaffer, J. Buhler, E. Mardis, and S. C. R. Elgin, Science 322, 684 (2008).
  * Geisinger and Raman [2013] B. N. Geisinger and D. R. Raman, Int. J. Eng. Educ. 29, 914 (2013).
  * Zhao [2007] Y. Zhao, Intercultural Communication Studies 16, 129 (2007).
  * Hu [2002] G. Hu, Language culture and curriculum 15, 93 (2002).
  * Wilkinson and Olliver-Gray [2006] L. Wilkinson and Y. Olliver-Gray, Psychologia 49, 74 (2006).
  * Rachel Zhou  _et al._ [2005] Y. Rachel Zhou, D. Knoke, and I. Sakamoto, International Journal of Inclusive Education 9, 287 (2005).
  * Wan [1999] G. Wan, (1999).
  * Beaver and Tuck [1998] B. Beaver and B. Tuck, New Zealand Journal of Educational Studies (1998).
  * Byars-Winston  _et al._ [2015] A. M. Byars-Winston, J. Branchaw, C. Pfund, P. Leverett, and J. Newton, International journal of science education 37, 2533 (2015).
  * Chemers  _et al._ [2011] M. M. Chemers, E. L. Zurbriggen, M. Syed, B. K. Goza, and S. Bearman, Journal of Social Issues 67, 469 (2011).
  * Russell  _et al._ [2007b] S. H. Russell, M. P. Hancock, and J. McCullough, Science 316, 548 (2007b).
  * Engeström [2014] Y. Engeström,  _Learning by Expanding: An Activity-Theoretical Approach to Developmental Research_ (Cambridge University Press, Cambridge, 2014).
  * Engeström [1999] Y. Engeström, Comput. Support. Coop. Work. 8, 63 (1999).
  * Gronn [2002] P. Gronn, Leadersh. Q. 13, 423 (2002).
  * Marton and Booth [1997b] F. Marton and S. A. Booth,  _Learning and Awareness_ (Psychology Press, 1997).
  * Sin [2010] S. Sin, Int. J. Qual. Methods (2010).
  * Booth [1992] S. A. Booth,  _Learning to program: a phenomenographic perspective_ (Göteborg studies in educational sciences–Acta universitatis Gothoburgensis, 1992).
  * Marton [1981] F. Marton, Instr. Sci. 10, 177 (1981).
  * Marton [1986] F. Marton, Journal of Thought 21, 28 (1986), publisher: Caddo Gap Press.
  * Mendoza  _et al._ [2015] N. Mendoza, J. Richard, and T. D. Wickliff, in  _Paper Presented at the 7th First Year Engineering Experiences Conference_ (Roanoke, VA., 2015).
  * Prior and Bilbro [2012] P. Prior and R. Bilbro, in  _University Writing: Selves and Texts in Academic Societies_ , Studies in Writing, edited by M. Castello and C. Donahue (Emerald, 2012) pp. 19–31.
  * Zydney  _et al._ [2002] A. L. Zydney, J. S. Bennett, A. Shahid, and K. W. Bauer, J. Eng. Educ. 91, 151 (2002).
  * Gonzalez-Ramirez  _et al._ [2021] J. Gonzalez-Ramirez, K. Mulqueen, R. Zealand, S. Silverstein, C. Mulqueen, and S. BuShell, College Student Journal 55, 29 (2021).
  * Yamagata-Lynch [2010] L. C. Yamagata-Lynch,  _Activity systems analysis methods: Understanding complex learning environments_ (Springer Science & Business Media, 2010).
  * Zhang  _et al._ [2020] W. Zhang, Y. Wang, L. Yang, and C. Wang, “Suspending classes without stopping learning: China’s education emergency management policy in the covid-19 outbreak,” (2020).
  * Nambiar [2020] D. Nambiar, The International Journal of Indian Psychology 8, 783 (2020).
  * Yaffe  _et al._ [2014] K. Yaffe, C. Bender, and L. Sechrest, J. Coll. Sci. Teach. 44, 25 (2014), publisher: National Science Teachers Association.
  * Carpi  _et al._ [2017] A. Carpi, D. M. Ronan, H. M. Falconer, and N. H. Lents, J. Res. Sci. Teach. 54, 169 (2017).
  * Mabrouk and Peters [2000] P. A. Mabrouk and K. Peters, Council on Undergraduate Research (CUR) Quarterly , 20 (2000).
  * l. Gafney [2001] l. Gafney, Council on Undergraduate Research (CUR) Quarterly 21, 172 (2001).
  * Kardash  _et al._ [2008] C. M. Kardash, M. Wallace, and L. Blockus, in  _Creating Effective Undergraduate Research Programs in Science_ (Teachers College Press, New York, NY, US, 2008) pp. 191–205.