# Novice Programmers Strategies for Online Resource Use and Their Impact on
Source Code
††thanks: This research was sponsored by Saudi Electronic University, College
of Computing and Informatics, Kingdom of Saudi Arabia.

Omar Alghamdi12, Sarah Clinch1, Mohammad Alhamadi1, and Caroline Jay1
2College of Computing and Informatics, Saudi Electronic University,
Riyadh,6867, Saudi Arabia
Email:
<EMAIL_ADDRESS><EMAIL_ADDRESS>1 Department of Computer Science, University of
Manchester, M13 9PL,United Kingdom

###### Abstract

Websites are frequently used by programmers to support the development
process. This paper investigates programmer-Web interactions when coding, and
combines observations of behaviour with assessments of the resulting source
code. We report on an online observational study with ten undergraduate
student programmers as they engaged in programming tasks of varying
complexity. Screens were recorded of participants’ activities, and each
participated in an interview. Videos and interviews were thematically
analysed. Novice programmers employed various strategies for seeking and
utilising online knowledge. The resulting source code was examined to
determine the extent to which it met requirements and whether it contained
errors. The source code analysis revealed that coding with the websites
involved more coding time and effort, but increased the possibility of
producing correct code. However, coding with websites also introduced
instances of either incorrect or non-executable source code.

###### Index Terms:

Web Search, Knowledge reuse, Information Search and Retrieval, Software
Engineering.

##  I Introduction

The internet has made information more accessible than ever before, with
significant implications for how people learn and practice programming.
Website usage is viewed as an integral part of the software engineering
process [1]. Developers sometimes spend more time searching for information
online than coding, assisting their learning and clarifying terms [2].

Searching is considered a vital process that is conducted pervasively to
augment programmers’ knowledge, even for familiar issues [3, 4, 5, 6]. In
addition to providing information, searching the websites also aids in code
understanding and error correction [7, 2]. Programmers repeatedly access
previously visited Web pages [8]. However, searching can affect programmers’
productivity, as using the websites for simple tasks can take more time than
coding from scratch, and searches may not always be successful [9]. It can be
difficult for programmers to locate and assess online code snippets [10],
which are often of low-quality [11]. Students, in particular, experience
challenges, given their lack of vocabulary to assist their searches [12].

Seeking knowledge online is usually coupled with utilising code [13],
providing a structural template and supporting similar functionalities [14].
Copying code from online sources is commonplace during development, especially
from the website Stack Overflow[2, 15, 16, 11]. There is evidence that online
queries are focused primarily on code reuse [17] and that increased complexity
in source code leads to greater online information seeking [18]. To date,
there has been little exploration of how seeking and utilising online
knowledge affects the production of source code.

Online code is often of poor quality or unsuited to the task at hand, making
it challenging to use effectively. Prior research shows that code snippets on
Stack Overflow contain problematic code such as security issues [19, 20, 21]
and outdated statements [22].

The current study documents how programmers use the websites to complete a
series of coding tasks, and examines how these activities may affect the
resulting source code. We screen recorded ten undergraduate students solving
four programming tasks with varying levels of difficulty. We also collected
the resulting source code for analysis and interviewed the programmers to
understand in more depth why they used particular strategies and how these
affected their code. Our research questions were as follows:

  1. RQ1

How do programmers use websites during programming?

  2. RQ2

What are the effects of websites use on the resulting code?

Our results show that programmers use various strategies to seek and utilise
online knowledge. These strategies may increase task completion time and
effort but increases the chances of producing correct code. However, website
use does not guarantee functional code – in several instances participants
still fail to produce code that executes and/or meets requirements. The
results of this study shed the light in coding with the websites with a unique
approach applying multiple inputs of data to the investigation, as well as in
the assessment of the resulting code to discover problematic code propagation.

##  II Related work

Our work builds on research from software engineering that seeks to understand
programmers’ activities, and examines the resulting code to investigate
possible implications.

###  II-A Understanding coding activities

In computer science education, learning analytics such as Error Quotient [23,
24] and Watwin Score [25] (both based on compilation behaviour and errors),
NPSM [26, 27] (based on a more holistic set of IDE-captured measures), and RED
[28] (a measure of repeated errors), have been used to quantify programming
behavior and predict outcomes. Outside of teaching, IDE instrumentation has
been used to identify problem solving strategies [29] and code/documentation
interaction patterns [18, 9] (e.g. investigate, edit, validate [30]).

Qualitative approaches are also used to characterise programmer behaviours
[31, 32, 33, 10, 34, 35, 9, 12, 36]. For example, Ko _et al._ analysed screen
captures for experts engaging in software maintenance tasks, identifying three
common activities – code search, dependency following, and code collection
[34]. Combining IDE instrumentation with surveys and interviews has been used
to understand professionals’ debugging activities, with printf and breakpoints
used in preference to more advanced IDE functionality [37]; interviews have
also been used to understanding student debugging [35].

Whilst the above focus on in-editor activity, this study seeks to better
understand the usage of websites during coding activities. Such behaviors have
previously been observed by Han _et al._ , who combined eye-tracking with IDE
and browser logs to capture behaviour in Python tasks. In addition to
quantifying code production, their study identifies patterns of external
resource use (e.g. searching, copy-paste) [38].

####  II-A1 Search

Programmers use their coworkers for information [36], and search both within
their own codebases [39, 34, 6, 40, 4, 41] and for online information (e.g.
through Google’s Web search) [19, 2, 39, 38, 42, 5, 43]. In this paper, we
focus on the interactions when coding with websites, and our discussion of
search therefore centers on this latter behavior. Google has been observed to
dominate programming-related search, with the Stack Overflow Q&A website most
prominent in results [19, 10, 17, 12, 11]. However, the behaviour of novices
and students may differ, e.g. using search to identify online tutorials in
preference to Stack Overflow [2, 42, 44].

Search engines are used to support code comprehension and reuse [17, 45, 11,
44], debugging [7, 17, 11], information acquisition and reference [2, 7, 17,
11]. Search frequency increases when tasks become complex [18, 43], and both
searches and their results are often consulted multiple times in a single
session [42, 8]. For example, Astromskis _et al._ found low rates of search
among professionals ($6\%$ of sessions), but observed intense online
consultation during some sessions with more complex code leading to greater
use of the websites [18].

Searching websites are not always (immediately) successful. In observations,
Wang noted that some participants did not click any of the search results, or
that viewing a website was immediately followed by new search behavior [9].
Other studies confirm that multiple queries may be needed to identify
appropriate resources [18, 8, 12, 11]. This may be a result of difficulties
representing code and symbols in queries; retrieval of unanswered questions;
and retrieval of resources that are of poor quality, missing exemplars, or are
difficult to understand [10, 11]. Effective search may be particularly
problematic for novices, who lack strategies and vocabulary and have
difficulty assessing the relevance of results [32, 5, 46, 12]. However,
professionals also report challenges, particularly with regard to the volume
of information retrieved [33]. Finally, context-switches arising from search
may themselves be problematic, reducing productivity [18].

####  II-A2 Code cloning/reuse

A common objective for searching websites is the identification of code
snippets for reuse [17, 13, 16, 46, 15, 11]. In one analysis of search
queries, $46\%$ were seeking code for reuse [17].

Studies indicate that both students [47, 13, 48, 49] and professionals [18,
50, 51, 14] engage in code cloning, although there is some evidence that
students may adopt alternative approaches as they develop expertise [49].
Professionals report using code clones as a tool to help refactor code, act as
a structural template/example, and support forks/branches [14, 3], and both
groups engage in follow-on modification activity, e.g.: complete or partial
removal, correction and compilation cycles, modification and ‘beautification’
to fit context, and addition of new code [52, 47, 50, 8, 5].

###  II-B Impact on code

The prevalence of code cloning [17, 11] has led researchers to investigate the
degree to clones could propagate problematic code. Users of online code
snippets report them to be outdated, incomplete, incorrect, poorly structured,
verbose, lacking meaningful variable names and rationale, and indicate that
code often does not address their problems [53, 22, 54, 55]. These problems
have also been identified in data mining studies [22, 56, 57, 58], and
observation and lab studies suggest that use of websites typically leads to
poorer code [19, 52].

Data mining has also been a valuable tool for measuring propagation [59, 20,
1]. Abdalkareem _et al._ examined $1,496$ Android apps finding that $377$ had
source in common with Stack Overflow, and the introduction of this code
increased the number of subsequent bug fixes [59]. Propagation of security
vulnerabilities is a particular concern. Lab study has shown that Stack
Overflow users produce functional but insecure code [19], and data mining
shows that a very high proportion of security-related snippets contain
potential vulnerabilities [20], even in accepted answers, up-voted answers and
answers from high reputation users [21]. However, not all studies agree that
cloning is problematic – in a study of $1,244$ open-source projects, code
reuse predicted neither the presence or absence of security vulnerabilities
[60].

Novice programmers may be particularly likely to generate and propagate
problematic code. Static analysis of code written in an introductory
programming exam identified both functional (syntax, logic) and stylistic
errors [61].

In this paper, we provide in-depth qualitative analysis of novice programmers’
online information-seeking and code-cloning behaviours. Our research builds on
knowledge from prior studies, using a combination of methods to provide a rich
picture of online and in-editor activities, augmented with programmers’
motivation and experiences of those activities, and the impact on code
outputs.

##  III Methodology

We conducted an online study in three sequential phases: a set of four video-
captured programming tasks, source code collection, and then an interview (see
Fig 1). This approach allows us to triangulate multiple data sources to build
a rich understanding of our participants’ behaviour [62].

The experiment was conducted with undergraduate students recruited from our
university. Prior to recruitment, a pilot was used to ensure the clarity and
validity of tasks, and to assess overall experiment duration. Procedures for
the experiment were reviewed and approved by the Ethics Committee at our
university All data was anonymized at time of collection.

Figure 1: Overview of research methodology: an online study in three
sequential phases. A set of four video-captured programming tasks are followed
by source code collection, and then an interview. The resulting qualitative
data is thematically analysed.

###  III-A Task design

This study phase required participants to engage in a number of programming
tasks appropriate to the participants’ experience and expertise. Having
limited our recruitment to a single cohort of undergraduate students in their
second year at our University, we could design our tasks to build on course
materials from their first year of study. This included a course textbook
[63], lecture slides, instructions for programming exercises, and sample
solutions associated with those exercises. We extracted an initial set of
thirty-three programming tasks from these materials.

To ensure task completion could be supported by online materials, specifically
the Stack Overflow website111Prior research has indicated that Stack Overflow
is a dominant resource for students and programmers when seeking help using
websites, we conducted a preliminary search of Stack Overflow using task
keywords. Tasks with fewer than twenty Stack Overflow answers were removed
from the candidate pool, leaving fourteen tasks for further consideration. The
remaining tasks were classified into three difficulty levels. Tasks that used
only concepts/topics that were explicitly taught during the course unit
lectures/exercises were classified as ‘easy‘ (n=5), those that used a
combination of explicitly taught and additional material (e.g. from the
textbook) were classified as ‘medium’ (n=5), and those that required
substantial additional material were classified as ‘difficult‘ (n=4). From the
classified tasks, we selected four tasks with a variety of difficulty levels:
two easy, one medium and one difficult (See Table I).

TABLE I: Study Tasks, Difficulty Levels, and Topics. Fixed Width Text Denotes Java Classes, Methods and Keywords. Topics Marked With an Asterisk (*) Were Not Explicitly Taught in the First Year Java Course, but Were Covered in Additional Material. Task |  Difficulty |  Description |  Topics assessed
---|---|---|---
1 |  Easy |  Write a program that contains a two-dimensional array with the following values: 10, 20, 30, 40, 50, and 60. Declare and initialise a two-dimensional array with three rows and two columns. Then, using nested loops, print the array contents one by one in the same order. |  multi-dimensional arrays, nested loops
2 |  Easy |  Write and implement a program which converts a sum of money into a different currency. The user will enter the amount of money to be converted and the exchange rate. The program will contain separate methods for getting the sum of money from the user, getting the exchange rate from the user, calculating the conversion, and displaying the result. |  method definition, string input, string output
3 |  Medium |  Write a program to enter and confirm a suitable code name for a spy agent. Declare a String variable and then get the user to enter a suitable name as a codename. Check that the codename meets the following requirements:  1. The codename is greater than 6 characters in length. 2. The codename starts with the word “Agent”. 3. The codename ends with an “X” character. If one of the conditions above was not met, print “INVALID CODENAME” and ask the user to re-enter a code name. |  String, .startsWith*, .endsWith*.
4 |  Difficult |  Using Java threading feature, create three threads where each has a unique name. Then, each thread should print the numbers from 1 to 100 in sequence order. For example, thread A will print numbers 1 to 100, thread B will print numbers 1 to 100, and thread C will print numbers 1 to 100. |  threading*

###  III-B Study Procedure

Our three-phase study took place over the Zoom video conferencing software,
and using the Dropbox file hosting service.

####  III-B1 Preliminaries

All participants were provided with a briefing sheet prior to participation
(available as supplementary material222DOI: 10.48420/c.6366063). A video call
with each participant was initiated, and the researcher summarised the phases
of the study. Once participants had a good understanding of the nature of
their involvement, they were asked to provide verbal consent and recording of
the call (audio and screen capture) commenced.

####  III-B2 Phase 1: Video-captured programming tasks

At the beginning of this phase, participants were presented with a document
summarising the four programming tasks. Participants were instructed that they
should not overly concern themselves with the need to correctly solve all
tasks, and that they could use any of their usual software, resources, or
websites during task completion. Participants were then asked to share their
screen and could begin solving the tasks using their preferred IDE or editor.

During the programming tasks, the researcher muted their mic and turned off
their own video camera so as to minimise distraction. The researcher did not
intervene or interrupt the coding tasks unless participants directly requested
that they did so (e.g. to help resolve a technical issues or task). The
researcher did not provide any programming-related information that could help
the participant solve tasks, even if requested to do so by the participant.

The programming task phase ended after $50$ minutes of elapsed time, or
earlier if a participant indicated that all four tasks had been completed.
Upon completion, screen sharing and recording was stopped, and the resulting
video files (one per task) were saved for subsequent analysis.

####  III-B3 Phase 2: Source code collection

Participants were then requested to share their final set of source code files
for the four programming tasks. Zoom chat was used to direct participants to a
Dropbox upload link.

####  III-B4 Phase 3: Interview

The final phase was an audio-recorded structured interview. Participants were
asked to provide their age, gender and describe their programming
expertise/experience. They were asked about their experiences of completing
the programming tasks, and the degree to which their behavior was
representative of the participant’s usual programming activity. Participants
were additionally asked targeted questions about their use of websites during
typical programming and during the experiment’s programming task phase, code
cloning behaviours during the programming task phase, and the relationship
between website-based information retrieval and existing knowledge (i.e. are
they looking up things to refresh/jog their memory). The full questionset is
provided in the supplementary material2.

###  III-C Participants and Recruitment

Our target population were invited to participate through emails (sent to all
second year computer science students) sent by course directors. Ten
participants (five males and five females, aged $19-22$ years) responded and
participated in the study (see Table II). Participants were compensated with a
£10 GBP Amazon gift certificate333Approximately $14 USD..

TABLE II: Participants’ Demographic and Expertise Responses. Experience is a Self-reported Value From 1 to 10, Where Novice=0, Intermediate=5, and Professional=10. Duration Refers to the Length of Time for Which They Have Programmed. Participant | Gender | Age | Experience | Duration |  Programming languages
---|---|---|---|---|---
P1 | Male | 20 | 5 | Not available |  Python, Java, C#, JavaScript, and R
P2 | Male | 20 | 7 | One semester |  Python, Java, C, C#
P3 | Female | 19 | 5 | One year |  Python and Java
P4 | Female | 19 | 5 | 5 years |  4 programming languages
P5 | Male | 20 | 5 | 8 years |  Python, Java, JavaScript, Typescript, C#, .NET, C, C++, and Swift.
P6 | Female | 20 | 5 | 6 years |  Java, Python, C, C++, Web development
P7 | Female | 19 | 3 | One year |  Python
P8 | Male | 21 | 5 | 4 years |  5 programming languages
P9 | Male | 20 | 5 | 3 years |  Python, Java, C++ and Pascal
P10 | Female | 22 | 5 | 5 years |  Java along with $9$ programming languages

###  III-D Analysis

Interview recordings were transcribed for analysis. We analysed interview
(phase 3) transcripts first, and subsequently used the knowledge gained to
inform behavioral coding of the videos (phase 1) and source code (phase 2).
Analysis was done by first author. To ensure reliability of the coding, 30% of
the data (i.e. three interview transcripts, ten videos, and source code from
three participants) were additionally coded by third author, and any
disagreements were resolved through a process of discussion and consensus,
with near-perfect agreement (Cohen’s $k=0.83$, $k=0.87$, and $k=0.83$ for
interviews, videos and source respectively; agreement $96\%$, $96\%$ and
$93\%$).

Interview transcripts ($n=10$) were inductively coded using Braun and Clarke’s
six phases for reflexive thematic analysis [64] and NVivo 12.

Videos (screen recordings) were qualitatively analysed to observe [62, 65] and
quantify participants engagement with websites when solving the study tasks.
The same researcher again followed a reflexive thematic analysis approach [64]
and linked the recorded videos to NVivo 12, given the large size of the video
files.

In the first step of the reflexive thematic analysis, the researcher watched
the videos multiple times for content familiarisation. Then, the coding phase
started with watching and coding the related behaviours, using a deductive
approach. In contrast to the interviews, this approach was analyst-driven,
using findings from our phase 3 interviews as a framework to identify relevant
behaviors [66]. The choice of such an approach ensured behavioural coding of
the videos that focused on the study’s research questions and objectives, and
helped the researcher to know what to expect when coding the videos. However,
any interesting and related behaviours outside this framework were also
captured to ensure unbiased data. To facilitate the coding, a marker was
placed at the start of each task in the videos. Coding the videos was an
iterative process involving many rounds, despite requiring more time and
effort.

Initial coding observed behaviours based on the initial framework. Th
subsequent round sought to identify related behaviours not covered by the
framework. The third round was to extract behaviours related to the source
code for later analysis. After this round, two criteria were established to
ensure that the captured behaviours were meaningful and relevant. The criteria
were that the behaviours had to be observed on the videos to constitute clear
and reliable evidence and the behaviours should be measurable, in the sense
that the videos provide enough information to support their serving as
relevant evidence. The fourth round considered the mentioned criteria, checked
the extracted behaviours and extracted any missing relevant behaviours. These
multiple rounds ensured greater familiarity with the data and filtered
behaviours down to relevant activities with multiple observed instances.

The next phase of the reflexive thematic analysis was grouping similar codes
that form data patterns and share one overarching concept for the themes. For
instance, codes that captured participants’ behaviour toward online code
adoption were grouped. The final set of themes was reviewed to ensure that no
theme constituted a sub-theme to another theme or was not a theme in itself.
Then, a thematic map was drawn to illustrate the final themes. Finally, a
report on these behaviours was written.

A total of $33$ videos (from a possible $40$) were analysed. Five participants
did not attempt Task 4, and so no videos were captured for these participant-
task pairings. A further two missing videos (P10 Tasks 1 and P10 Task 2) were
attributed to a failure of the Zoom recording facility.

Source code produced by participants was analysed for the $33$ tasks for which
video was captured. A mixture of quantitative and qualitative measures were
established:

  1. 1.

Quantitative: Compilation/Execution – Was the researcher able to compile and
run the source code either using an IDE or Command line in Windows?

  2. 2.

Quantitative: Correctness – Does code execution produce the expected output
(based on the specified tasks)?

  3. 3.

Quantitative: Online Clones – the number of lines copied and pasted from
online sources (final non-commented source lines only) retrieved from
reviewing the videos.

  4. 4.

Quantitative: Online Sources – the URLs from which cloned code was sourced
(retrieved from reviewing the videos).

  5. 5.

Qualitative: Clone Purpose – the perceived motivation for including cloned
lines (retrieved from reviewing the videos).

##  IV Results

###  IV-A The interview

Our thematic analysis revealed three themes: reasons for website use,
experiences using websites, and study specifications and issues.

####  IV-A1 Theme 1: Usage of websites

Participants used websites for various purposes during their coding.

##### Websites as a reminder

Nearly all participants reported that the websites, such as Stack Overflow,
could serve as a reminder for information even if it were already known.

> I do not try to remember the code if I know where it is exactly. […] I am
> just reminding myself from the sources. (P3)

> […] it is often the same thing looking up just to check if you remember it
> correctly. (P6)

##### Copy online code

All participants reported copying and pasting online code, even the previously
encountered code or problems, due to the easy-to-find online code.

> [Q: What about copying code that you have previously copied during websites
> search?]
>
> Yeah, I would copy it again; some code I know where I can find it, so
> whenever I cannot remember it by heart, I know exactly where to find and
> then just copied it and then it works. (P8)

Most participants felt that the websites urged them to adopt available online
content without checking, where the ease of obtaining online code gave a sense
of no need to understand the code.

> […] I went ahead and copied the first things I saw […] I think this is very
> common from write code pull up a page get something from the internet and
> continue coding. (P8)

##### Searching for information

Participants described the procedures they took when searching the website for
programming material. At the start, they get familiarised with the
requirements to help determine the following step.

> I think my approach would usually be the same, read though the questions a
> few times and give it a go and see where I feel it does struggling and then
> start seeing if there anyone else done it in a smart way or whatever. (P9)

The following step is searching using the Google search engine without
predetermining preference, as mentioned by most participants, and structuring
the search query in a more appealing way related to the requested tasks.

> I did not use a particular website, just try to figure most concise way to
> word my question and just see what comes up. (P9)

####  IV-A2 Theme 2: Websites experience

This theme tackles programmers’ websites’ preferences and potential concerns.

##### Websites choices

Half of the participants reported that Tutorialwebsites provide clear, simple,
understandable, self-sufficient, and easy to locate examples that offset the
need to read the corresponding contents.

> The one I usually chose is GeeksforGeeksor W3school. These two because it
> explains the best what I need to find, and the answers are on the beginning,
> so I do not need to scroll and read through everything. (P10)

More than half of the participants mentioned using of Stack Overflow because
it provides reliable and genuine examples aiding problem-solving.

> […] I use Stack Overflow for for this one specific problem I was having
> because those questions are more specific. (P7)

The usage of the Stack Overflow is approached with care, as P5 noted:

> And there is Stack Overflow, but you need take that in with some caution
> sometime. (P5)

##### Issues with the websites

Many participants reported issues when using the websites. Some online answers
were functional, but they were not necessarily great.

> [Q: When searching the websites, were you confident that you got the right
> answer?]
>
> Getting answers that works, but it is not always the best answers I would
> say. (P4)

Online answers additionally contributed to poor outcomes unsuitable for
participants’ intentions.

> [Q: Did you find the websites easy to use, helpful, and correct?]
>
> One give me a misleading data, so I would say no. (P8)

Thus, participants reported the possibility of facing problematic code or
incomplete code that misses important information.

> Sometimes the code on Stack Overflow can be a bit outdated, new version of
> framework or languages. Also, the users select an answer as the best answers
> that probably should not be, […] like there are other answers better but not
> chosen by the user […] who answer but it did not show up in the top. (P5)

> The things it annoys me is website not showing the entire piece of code, so
> the thing that missing is something outside of the immediate bit of code
> they are showing. Like import statements, if the import statement is wrong
> and I cannot work out what it is, they very rarely show that I guess. (P4)

Participants were uncertain about their ability to integrate online code due
to the associated problems.

> Some problems are sometimes you are not working with your own code, so you
> are do not know everything about it, you do not know how the code interact
> with the code you find online. (P1)

Another impact was coding with unfamiliar programming aspects, where the
websites did not help resolve such uncertainty.

> […] for the threading, I am not too familiar with threading, so I am not
> sure if it is what I wanted. (P8)

####  IV-A3 Theme 3: Experiment’s specifications and issues.

This theme addresses participants’ thoughts and concerns regarding aspects of
the experiment.

##### Video recording

More than half of the participants stated that their behaviours during the
experiment followed their normal programming behaviours.

> [Q: To what extends does the behaviours that you used today reflect the
> behaviours that you would normally use?]
>
> I think pretty accurately. (P4)

Nevertheless, some behaviours were either not captured or did not reflect the
programmers’ normal behaviours. Two participants expressed that recording
could conceal some behaviour or introduce unwanted factors to their
programming activities.

> […] I would probably open up my notes and some written notes which obviously
> that I cannot show on the screen. (P2)

> [Q: To what extends does the behaviours that you used today reflect the
> behaviours that you would normally use?]
>
> It sorts of similar but in smaller scale. Usually I have got two monitors
> set up. (P5)

##### Time constraints

Half of the participants felt that time constraints affected their performance
in the experiment.

> So today I try do it relatively fast because of the time thing. Normally I
> will take it a bit slower and maybe read more things. (P8)

The other half of the participants indicated no stress from the restricted
time, and using some approaches would assist in regulating the time.

> What I do when I started problems, I know it is time constrain I sort try to
> get the first few really quick or the one I spot easier, so I try manage the
> time better, and I know I have done it so it does not much so time constrain
> does not stress me. (P5)

##### Task Difficulty

Participants reported that the tasks were not difficult but required more
searching.

> The tasks was not challenging. (P5)

However, four participants thought the fourth task could be challenging
because they had no previous information, which may lead to not approaching
the task.

> I have not got the time to read it [fourth task], but I am not familiar with
> threading, so that why I left it. (P3)

Two participants thought the problems were not from the task but from
forgetting the information and time.

> My Java was a bit rusty, slight, I did not remember a lot of things how to
> do in Java in particular. But no, I think if I have a bit more time I will
> finish them all. (P2)

###  IV-B Behavioural coding

Participants used both Web-based and non Web-based resources to search. All
participants used websites at various stages. Most participants used Tutorial
websites to retrieve syntax, including GeeksforGeeks, W3school, JavaTpoint,
“Java671”, BeginnerBook and TutorialPoint. The choice of Tutorial websites
changed based on the tasks. Other than Tutorial websites, Stack Overflow was
used for resolving errors, and blogs were used to aid for understanding. In
addition, participants consulted the requirements document for tasks’
requirements, course-related materials, previously written code, and the IDE.
Participants edited and compiled their code, going through a cycle of
observing and fixing errors post-compilation.

####  IV-B1 Theme 1: Acquiring knowledge

Participants searched websites for syntax and to clarify understanding.
Searches yielded useful knowledge, but also caused problems.

##### Looking for syntax

All participants searched websites for syntax either before or during coding.
Before writing code, P6, P7, P8, and P10 in the fourth task and P3 and P8 in
the first task searched for specific terms such as threading, “two-dimensional
array” and the main method. The search for syntax while coding was noticed by
nearly all participants who divided the tasks into meaningful searchable
pieces and searched looking for syntax. Specifically, all participants in the
first task, except P10, searched for “two-dimensional array” and basic syntax
like for loop, main method and array. In a similar manner, all participants in
the second task, except P2 and P10, searched regarding converting data types
and getting user inputs, such as Scanner and Console. P4 and P9 searched for
basic syntax, including Java function structure and .println, similar to the
previous task. All participants in the third task searched syntax to retrieve
String methods like .length, .startsWith, .endsWith, .charAt and .substring.
P9 and P6 searched for basic syntax, include while loop while loop and “AND”
symbol. In the fourth task, four participants (P5, P6, P8, P10) performed
superficial searches, such as creating threads, assigning threads names and
importing threads; P8 used the whole question text to search at the task’s
start.

##### Increase understanding

Participants sought programming information related to the task along with
searching for syntax. P1 and P4 on the second task and P7 on the third task
searched for syntax clarifications, such as the use of the .equals method for
character type, the difference between Float and Double and the use of
currency in Java. P7 and P10 also searched to gather information about the
threading feature in the fourth task.

##### Experiencing problems during search

It can be observed that some online searches were unhelpful, causing
additional time and effort. Eight participants struggled with online search
across coding stages in all the tasks, and half of them returned to the
previously visited Web pages. At the first task, P2 accessed various websites
and searched for the nested “foreach” loop, resulting in more search time and
attempts, then eventually changing the chosen method. In another instance, P2
searched for “two-dimensional array”, and the outcomes did not help succeed in
printing array elements accurately but motivated further searches that also
did not help fix the issue. Similarly, P7 searched repeatedly for a “two-
dimensional array” before writing it correctly. P9 searched for the main
method and the .println statements without beneficial outcomes and resolved
without searching. While coding the second task, P4 struggled the most as she
searched five times for Float and Double, truncate Float, Decimal format and
currency without valuable outcome; she sometimes returned to the previously
searched results and chose another website. P6 and P7 searched for Scanner
syntax, but their searches were not complete and caused further searches, and
P4 and P9 conducted fruitless searches when converting data types from String
to Float.

Online searching issues continued in the third task. Five participants faced
issues related to String. P2 was affected the most as he searched for the
getchar method in Java but adopted String declaring instead, then continued
searching for the equivalent of getchar from C++ in Java and accessed three
websites without reaching an answer. The previously adopted String declaring
resulted in issues when compiling, causing two further search attempts, which
was finally resolved by adopting a new code from websites; P10 shared a
similar struggle while searching for the getchar method. In addition, search
instances conducted by P3, P9 and P10 caused further search attempts, such as
searching for String .length methods, split String and .substring. P8 searched
for unnecessary syntax like “regex” and Matcher. Lastly, three participants
(P7, P8, P10) in the fourth task searched for how to start one thread after
another and count threads, then faced unproductive content, causing them to
reformulate their query and repeat the search.

####  IV-B2 Theme 2: Utilising knowledge to write the code

Through seeking online knowledge, participants utilised the knowledge to
develop their code. The following sub-themes will list participants’ observed
strategies when reusing online code.

##### Exploiting search results to copy code

All the participants copied online code snippets using two ways: a clipboard
as a normal way and a visual way by looking at online code while writing it.
There also appeared to be a further copy instance where participants examined
particular code snippets from the websites and then wrote it in their code
afterwards, and this copy instance will be labelled as a mental way. These
three copy instances were conducted throughout the tasks with variance.
Participants reused online code following the tasks’ requirements. Seven
participants in the first task, except P5, P6 and P10, copied “two-dimensional
array”. Similarly, all participants in the second task, except P10, copied
syntax, like declaring and importing Scanner and converting syntax such as
String to Float or Integer or the other way around. In the third task, all
participants, except P5 and P10, copied .startsWith, .length, .endsWith,
.charAt, .substring, Matcher and Scanner. At last, all five participants
copied online code, including a class header, printing outputs and declaring
the variables. Participants within all the tasks copied basic syntax from
online, such as main method, for loop and .println statements.

##### Implementing complicated approaches

Participants followed suggestions imposed by either websites or their code
practices causing additional coding time and effort. Videos observed such
approaches that did not necessarily appear in participants’ source code. Five
participants (P2, P3, P4, P5, P9) in the first task followed online complex
suggestions. In particular, P2 copied an online code that printed the array’s
contents using the toString method, which caused printing memory places, but
not the array contents. Three participants (P3, P4, P5) used online
suggestions to manually assign values to the array without using a loop, and
P5 placed the array contents using three variables, each holding two array
elements. P2 created four loops instead of two based on the online
suggestions. A similar observation is valid for P1 but not from the online
suggestions. In the second task, four participants (P3, P6, P8, P9) copied a
user entry that used a String data type from online, then converted it to
Integer data type. P5 and P1 followed the same path without online
suggestions. In the third task, six participants (P2, P3, P4, P5, P8, P10)
followed their code practices. P2 and P3 faced issues declaring multiple
String variables and P4 and P10 made multiple unnecessary if statements that
could be eliminated. P8 chose “regex” and Matcher, which resulted in several
searches, debugging and more time. As most of the code was copied online in
the fourth task, the online suggestions were only problematic for P7, who used
three Run() methods with different names and loops, causing more searches to
write them correctly.

###  IV-C Source code

Table III shows that nearly all participant-provided source code files
compiled and executed successfully (28/33 files, 85%), and just over half
(19/33, 58%) were considered to be correct (i.e. they met the requirements
described in the task). Across the four tasks, participants produced at least
two executables (mean: $2.80$, median $2.50$) and one correct source file
(mean: $1.88$, median $2.00$). Progressively fewer source code files were
correct as the tasks progressed (max: $9$, min: $2$).

TABLE III: Results of Source Code Analysis.

| Not Provided | Non-Executable | Incorrect | Correct | Online Clones
---|---|---|---|---|---
Task 1 | 1: P10 | 0 | 0 | 9: P1-P9 | 9/9 files: 1–5 lines/file
|  |  |  |  |  $\bar{x}$: $2.78$, $\tilde{x}$: $3.0$
Task 2 | 1: P10 | 0 | 4: P1, P3, P6, P8 | 5: P2, P4, P5, P7, P9 | 9/9 files: 2–7 lines/file
|  |  |  |  |  $\bar{x}$: $4.67$, $\tilde{x}$: $5.0$
Task 3 | 0 | 5: P2-P5, P9 | 2: P1, P7 | 3: P6, P8, P10 | 9/10 files: 2–6 lines/file
|  |  |  |  |  $\bar{x}$: $3.80$ ($4.22$), $\tilde{x}$: $4.0$ ($4.0$)
Task 4 | 5: P1-P4, P9 | 0 | 3: P5, P6, P10 | 2: P7, P8 | 5/5 files: 6–24 lines/file
|  |  |  |  |  $\bar{x}$: $12.00$, $\tilde{x}$: $10.0$

Note: In the First Four Columns, the Overall Number of Source Files in Each
Category Is Followed By the List of Participants Whose Source Fell Into Each
Category. Non-Executable Indicates That the Source Either Failed to Compile or
Did Not Run to Completion. Incorrect Indicates That the Code Compiled and
Executed Successfully but Produced Output That Was Not Compliant With The
Requirements. In the Final Column, the Number of Files and Lines Containing
Cloned Source Are Given, Together With the Mean ($\bar{X}$) and Median
($\tilde{X}$) Number of Cloned Lines Per File (Values in Brackets For Task 3
Indicate Averages for Files Containing Cloned Code).

All but one of the supplied source files (P5 Task 3) contained cloned code,
with each participant including an average of $2.67$ (mean, P1) to $8.50$
(mean, P6) cloned lines per file (medians ranged from $1.5$, P5, to $6.0$, P4
& P10).

####  IV-C1 Correct

There were instances of unnecessary syntax on the first and fourth tasks
increasing the complexity of the correct source code. In the first task, P1
and P2 in the first task used four loop statements instead of two to solve the
“two-dimensional array” where P2 wrote such code based on online suggestions,
P5 copied online code and identified the “two-dimensional array” using three
array locations, and P3 printed the locations of the array along with the
contents. In the fourth task, P7 followed the online suggestions and included
multiple Run() methods for each thread, for loop, and .println statements.

Each of the correct source code were linked with video recordings to
understand the strategies participants used to produce correct code. Regarding
searching strategies, syntax searches were apparent throughout coding sessions
for all participants, where they located items related to the task. Other than
syntax searching, few searching occasions were attributed to clarifications.
In addition, the online search resulted in multiple complicated issues faced
by more than half of the participants who produced correct source code, which
caused non-beneficial results and motivated further search attempts.
Furthermore, all participants referenced the requirements document to check
the requirements, and exhibited medium referencing in the first task, low in
the second task, and high in the third task.

In terms of copying online code strategies, nearly all participants, who
produced correct source code, copied online code in varied ways. On the
contrary, copying from non Web-based resources is a rare strategy done by one
participant. In addition, participants in twelve correct code instances ran
into complex issues, half of them from their choice and the other half from
the online suggestions.

####  IV-C2 Non-executable

Half of the third task participants (P2, P3, P4, P5, P9) ($50\%$) failed to
produce a functional compiled code. By analysing each source code within non-
executable type, multiple reasons caused the code to be non-executable.
Participants thought their code carried enough information while the code was
incomplete, contained incorrect elements, or missed critical elements. In
particular, P2 missed System.in that caused incorrect Scanner, used return()
in the method while the method is void, used variable before declaring it, and
wrote else if statement without including a condition. Similarly, P3 used else
without a condition and missed the Scanner import, and P2 and P3 copied from
online incomplete Scanner syntax and encountered difficulty when adopting the
code. In addition, P4 wrote four non-executable if statements, along with an
incomplete .println statement. At last, the P5 and P9 source code contained
incomplete main method and missed a semicolon.

The strategies that lead to non-executable code were explored for a more in-
depth understanding. Participants excessively searched for syntax during
coding, except for P5, and other searches were for learning. Searching produce
issues as P2 conducted a repeating search with no successful output, and P3
and P9 conducted unnecessary searches. Most participants referred to the
requirements document moderately, except P5 and P9. In the code strategies,
online code was copied by all participants following mainly the visual way,
except P5. Most participants made choices based on their practices that
complicated their coding and used non Web-based code.

####  IV-C3 Incorrect

Multiple reasons caused the source code to not comply with the specified
requirements. In the second task, P1 and P6 did not address Float or Double
user entries (caused by copying online code for P1) and P3 and P8 missed doing
methods for each requirement. In the third task, the String conditions wrote
by P1 code did not function because it accepts non-string entries without
processing, and P7 missed the while loop for continuous user entry. In the
fourth task, P5, P6, and P10 printed threads but not in sequence order, and
all solved the task engaging online.

Participants in this source code type conducted some common behaviours. In
syntax search, the online search was generally prevalent, where participants
looked for tasks related to syntax. Facing complex issues were not high during
searching activities. Nearly all the participants resorted to the requirements
document with moderate to high access. For the code strategies, copying the
code has normal observations where participants copy online code and faced
some complex issues.

##  V Discussion

This section discusses the study results and summarises the findings for each
research question.

###  V-A RQ1: How do programmers use websites during programming?

We found evidence from observations that the predominant use of websites was
for retrieving syntax. Participants in our experiment referred to various
resources, including websites, the requirements document, course-related
materials, previous code and the IDE; websites were used predominately to
search for knowledge. Participants preferred referring to the Tutorial
websites to seek syntax, consistent with the findings of [52] but in contrast
with other studies that suggest programmers use Stack Overflow for coding
[19]. They started their search by decomposing the tasks into searchable
parts, similar to observations in [40].

Two interesting observations were made in relation to searching: it was common
to search for Java basic syntax such as main method, .println statements and
loops syntax, and previously taught syntax such as “two-dimensional array” and
user entry methods. Interviews with participants showed they relied on
websites to remind them of this basic information, which they may have been
able to retrieve from memory if they had not had access to the websites. This
tendency to search for basic syntax has also been seen in [9, 5]. Participants
in our study preferred not to trust their memory of the syntax itself, but
rather to rely on their ability to find it online, supporting the notion of
using the websites as a reminder [2]. Participants also searched websites for,
e.g., the equivalent of getchar from C++ in Java, suggesting that previous
knowledge (especially knowledge of another programming language) plays a role
in the search process. In some cases, participants searched continuously,
without appearing to get relevant results. Failing to obtain relevant
information in spite of repeated searchers has also been noted in [19, 9].

Participants copied code from websites and used it within their programs.
Participants used code found online more in the fourth task than other tasks
suggesting that the more difficult or unfamiliar the task, the greater the
need to use online code. Using websites to help with unfamiliar tasks involves
dealing with uncertainty about the information found, as expressed in the
interviews. Participants were observed to copy online code, including basic
and taught syntax, in three ways: using copy and paste functionality;
observing it and retyping it; memorising it then retyping it. They reported in
the interviews that the reason for using online code is the ease with which it
can be located. It appears easier to search for basic syntax every time it is
required than to commit it to memory.

###  V-B RQ2: What are the effects of websites use on the resulting code?

Using the websites while coding helped participants to produce correct code
that met requirements. Where participants produced non-executable code, this
was not due to using websites per se, but rather poor programming practices.
Nevertheless, there were many instances where participants did not exploit
online content efficiently. Copied code was sometimes neither suitable for nor
required by the task. Examples are the manual assignment of array values, not
printing array contents, unsuitable data types for user entries, redundant
code and unnecessary multiple Run() methods. Participants appeared to trust
the code without giving it much scrutiny, at the expense of efficiency and
effectiveness. It is not possible to know whether better code would have been
written were the websites not available, but it is certainly the case that
wholesale or unthinking inclusion of online code did not always result in
satisfactory code. In the interviews participants reported that websites urged
them to use code as presented, but we can see that this sometimes resulted in
negative outcomes. The difficulty of reusing online code is also observed
superficially in [10, 11]. The findings of this study suggest that while
reusing online code can be effective, it can also cause complex problems that
require time and effort to resolve. Programmers need to consider the
associated time and effort while coding using the websites.

Issues we observed in the code included: increased complexity; extraneous
unwanted or unexpected outputs; and bugs [57, 59]. Participants reported two
additional issues in the interviews: outdated and incomplete problematic code
[56, 54, 22]. A possible explanation for these issues is that Websites state
the code is safe to use as is; this was also mentioned in the interviews.
Thus, it is necessary to reflect on the online presented content and ensure
its accuracy.

Other factors may have affected the accuracy of source code, including task
difficulty, time constraints and previous experience. While participants had
previously been taught how to achieve most of the tasks, difficulty appeared
to be a factor – production of valid code reduced as task difficulty
increased. However, participants reported in the interview that the tasks were
not difficult to code, indicating they may have been unaware that code did not
meet requirements.

###  V-C Recommendations

The findings in this paper have applications for stakeholders across software
engineering, including owners of the Websites, educators, researchers and
tools builder. The following sections distill these findings into some
concrete recommendations.

####  V-C1 Recommendations for owners of the Websites

  * •

Owners of the websites should advise their content authors to consider the
problematic code from the programmers’ perspective and fully explain their
online posted code.

  * •

Owners of the websites should advise their content authors to consider that
users with various experiences may consider their posted code.

####  V-C2 Recommendations for educators

  * •

Educators should providing training in online information seeking, including
how to search effectively, source selection, and appropriate expectations for
online content.

  * •

Educators should train students to engage in judicious code reuse, equipping
them to make sound judgements about the suitability of code snippets. This
includes supporting them in determining when (and which parts of) code
snippets are relevant and recognising problematic code.

####  V-C3 Recommendations for researchers

  * •

Researchers can further investigate the search and code reuse strategies and
propagation of problematic code by collecting or analysing ready sets of code.

####  V-C4 Recommendations for tools builder

  * •

Tools builder could exploit the findings by designing a tool that identify
programmers’ copy-and-paste activities when using websites and suggests
follow-on activities such as the review of copied code for understanding, fit
with requirements, and quality control.

###  V-D Limitation

Participants’ source code contained relatively few lines, providing little
room to analyse the impact of website use. The study’s design introduced no
baseline with which to compare coding with or without using the websites.
Asking participants to code without websites would not have been meaningful,
and participants may have been reluctant to take part in a study designed that
way. Also, repeating the study with the same participants would introduce
previous exposure to the tasks.

In addition, to minimise threats to construct validity, a pilot study was used
to validate and refine the tasks and the study’s phases. One threat to
construct validity could have been a concern about using appropriate tasks to
reflect upon the coding activities. Task design ensures participant
familiarity by using previous materials, and ensures online content support
solving the tasks. Other threat may have emerged as a property of recruiting
student participants from the University of Manchester. Whilst it was clearly
indicated to participants that their data would be treated anonymously, to
prevent impact on their study or outcomes, students may still have been
reluctant to divulge behaviours that they thought academe would perceive
negatively. Students at UK universities are regularly advised against
activities that might constitute plagiarism, such as copying and pasting from
external sources, and this therefore may have led students to minimise
disclosure of these behaviours during their coding sessions and interviews.

This study does not set out to examine causal relationships, and the internal
validity is of limited concern. However, potential influence by external
factors could have included experiment time, settings and researcher
availability influencing their coding. The study uses multiple data sources to
ensure triangulation, including observations, source code and interviews.
While the findings of this experiment resulted from solving programming tasks,
the observed behaviours may vary based on the tasks. Time also play an
important role in solving the tasks. In addition, steps were taken to involve
multiple members of the research team at every step. An inter-rater
reliability method was further conducted to increase the reliability of the
findings.

##  VI Conclusion

We conducted an online experiment to explore programmers’ activities during
coding tasks using websites, analysing the resulting source to uncover
possible consequences on the code. Recordings and source code for ten
programmers solving four programming tasks were collected, and participants
were interviewed. Recordings and interviews were thematically analysed, and
source code analysed through a combination of quantitative and qualitative
measures.

The observations have revealed that the vast majority of searches on websites
were for syntax and involved tutorial websites. Syntax search comprised
breaking down the task into searchable chunks and was usually for syntax that
was basic, and had been previously taught. Participants copied code in three
ways: using copy and paste functionality; observing it and retyping it;
memorising it then retyping it. The copied syntax was not always appropriate
or necessary for solving the task.

Participants produced source code in three categories: correct, which was
working according to the requirements; non-executable; and incorrect, which
was compiling/running but non-compliant with the requirements. Participants
used various strategies during their coding using the websites, and these
strategies were linked with participants’ source code to investigate their
outputs for any implications. Although coding with websites and encountering
complex issues that increased task completion time and effort helped
participants produce correct code, using websites impacted the resulting code,
producing either incorrect or non-executable code. Thus, using websites during
coding produced incorrect code.

Programmers need to consider the time and effort it takes to use websites,
reflect carefully on their requirements to help filter online content, and not
presume that online content is accurate. Future work should explore these
findings with other samples, for example, professional programmers.

## References

  * [1] D. Yang, P. Martins, V. Saini, and C. Lopes, “Stack Overflow in Github: any snippets there?” in _Proc. of 14th International Conference on Mining Software Repositories_. IEEE, 2017, pp. 280–290.
  * [2] J. Brandt, P. J. Guo, J. Lewenstein, M. Dontcheva, and S. R. Klemmer, “Two studies of opportunistic programming: interleaving web foraging, learning, and writing code,” in _Proc. of SIGCHI Conference on Human Factors in Computing Systems_. New York, NY, USA: ACM, 2009, pp. 1589–1598.
  * [3] T. D. LaToza, G. Venolia, and R. DeLine, “Maintaining mental models: A study of developer work habits,” in _Proc. of 28th International Conference on Software Engineering_. New York, NY, USA: ACM, 2006, pp. 492–501.
  * [4] J. Singer, T. Lethbridge, N. Vinson, and N. Anquetil, “An examination of software engineering work practices,” in _CASCON First Decade High Impact Papers_. Riverton, NJ, USA: IBM Corp., 2010, pp. 174–188.
  * [5] H. Li, Z. Xing, X. Peng, and W. Zhao, “What help do developers seek, when and how?” in _Proc. of 20th Working Conference on Reverse Engineering_ , 2013, pp. 142–151.
  * [6] C. Sadowski, K. T. Stolee, and S. Elbaum, “How developers search for code: a case study,” in _Proc. of 10th Joint Meeting on Foundations of Software Engineering_. New York, NY, USA: ACM, 2015, pp. 191–201.
  * [7] R. E. Gallardo-Valencia and S. E. Sim, “What kinds of development problems can be solved by searching the web? a field study,” in _Proc. of 3rd International Workshop on Search-Driven Development: Users, Infrastructure, Tools, and Evaluation_. New York, NY, USA: ACM, 2011, pp. 41–44.
  * [8] G. Gao, F. Voichick, M. Ichinco, and C. Kelleher, “Exploring programmers’ API learning processes: Collecting web resources as external memory,” in _Proc. of Symposium on Visual Languages and Human-Centric Computing_. IEEE, 2020, pp. 1–10.
  * [9] Y. Wang, “Characterizing developer behavior in cloud based IDEs,” in _2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement_ , 2017, pp. 48–57.
  * [10] J. Escobar-Avila, D. Venuti, M. Di Penta, and S. Haiduc, “A survey on online learning preferences for computer science and programming,” in _Proc. of 41st International Conference on Software Engineering: Software Engineering Education and Training_. IEEE, 2019, pp. 170–181, https://ieeexplore.ieee.org/document/8802102/.
  * [11] X. Xia, L. Bao, D. Lo, P. S. Kochhar, A. E. Hassan, and Z. Xing, “What do developers search for on the web?” _Empirical Software Engineering_ , vol. 22, no. 6, pp. 3149–3185, 2017, https://doi.org/10.1007/s10664-017-9514-4.
  * [12] D. Wong-Aitken, D. Cukierman, and P. K. Chilana, “It Depends on Whether or Not I’m Lucky” How Students in an Introductory Programming Course Discover, Select, and Assess the Utility of Web-Based Resources,” in _Proc. of 27th Conference on Innovation and Technology in Computer Science Education_. ACM, 2022, pp. 512–518.
  * [13] M. Fuchs, M. Heckner, F. Raab, and C. Wolff, “Monitoring students’ mobile app coding behavior data analysis based on IDE and browser interaction logs,” in _Proc. of Global Engineering Education Conference_. Istanbul: IEEE, 2014, pp. 892–899, http://ieeexplore.ieee.org/document/6826202/.
  * [14] M. Kim, L. Bergman, T. Lau, and D. Notkin, “An ethnographic study of copy and paste programming practices in oopl,” in _Proceedings. 2004 International Symposium on Empirical Software Engineering, 2004. ISESE’04._ IEEE, 2004, pp. 83–92.
  * [15] M. Umarji, S. E. Sim, and C. Lopes, “Archetypal internet-scale source code searching,” in _IFIP International Conference on Open Source Systems_ , B. Russo, E. Damiani, S. Hissam, B. Lundell, and G. Succi, Eds. Springer US, 2008, pp. 257–263.
  * [16] M. Hucka and M. J. Graham, “Software search is not a science, even among scientists: A survey of how scientists and engineers find software,” _Journal of Systems and Software_ , vol. 141, pp. 171–191, Jul. 2018.
  * [17] A. Hora, “Googling for software development: what developers search for and what they find,” in _2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR)_. IEEE, 2021, pp. 317–328.
  * [18] S. Astromskis, G. Bavota, A. Janes, B. Russo, and M. Di Penta, “Patterns of developers behaviour: A 1000-hour industrial study,” _Journal of Systems and Software_ , vol. 132, pp. 85–97, 2017, https://linkinghub.elsevier.com/retrieve/pii/S016412121730136X.
  * [19] Y. Acar, M. Backes, S. Fahl, D. Kim, M. L. Mazurek, and C. Stransky, “You get where you’re looking for: The impact of information sources on code security,” in _2016 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2016, pp. 289–305. [Online]. Available: https://doi.org/10.1109/SP.2016.25
  * [20] F. Fischer, K. Böttinger, H. Xiao, C. Stransky, Y. Acar, M. Backes, and S. Fahl, “Stack Overflow considered harmful? the impact of copy& paste on Android application security,” in _2017 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2017, pp. 121–136.
  * [21] N. Meng, S. Nagy, D. Yao, W. Zhuang, and G. Arango-Argoty, “Secure coding practices in java: Challenges and vulnerabilities,” in _2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE)_. IEEE, 2018, pp. 372–383, https://doi.org/10.1145/3180155.3180201.
  * [22] C. Ragkhitwetsagul, J. Krinke, M. Paixao, G. Bianco, and R. Oliveto, “Toxic code snippets on stack overflow,” _IEEE Transactions on Software Engineering_ , vol. 47, no. 3, pp. 560–581, 2021.
  * [23] M. C. Jadud, “A first look at novice compilation behaviour using BlueJ,” _Computer Science Education_ , vol. 15, pp. 25–40, 2005, https://doi.org/10.1080/08993400500056530.
  * [24] ——, “Methods and tools for exploring novice compilation behaviour,” in _Proc. of 2nd international workshop on Computing education research_. New York, NY, USA: ACM, 2006, pp. 73–84, https://doi.org/10.1145/1151588.1151600.
  * [25] C. Watson, F. W. Li, and J. L. Godwin, “Predicting performance in an introductory programming course by logging and analyzing student programming behavior,” in _Proc. of 13th International Conference on Advanced Learning Technologies_. IEEE, 2013, pp. 319–323.
  * [26] A. S. Carter, C. D. Hundhausen, and O. Adesope, “The normalized programming state model: Predicting student performance in computing courses based on programming behavior,” in _Proc. of 11th annual International Conference on International Computing Education Research_. Omaha, Nebraska, USA: ACM, 2015, pp. 141–150, http://dl.acm.org/citation.cfm?doid=2787622.2787710.
  * [27] ——, “Blending measures of programming and social behavior into predictive models of student achievement in early computing courses,” _ACM Transactions on Computing Education_ , vol. 17, no. 3, pp. 1–20, Aug. 2017, http://dl.acm.org/citation.cfm?doid=3135995.3120259.
  * [28] B. A. Becker, “A new metric to quantify repeated compiler errors for novice programmers,” in _Proc. of Conference on Innovation and Technology in Computer Science Education_. ACM, 2016, pp. 296–301, https://dl.acm.org/doi/10.1145/2899415.2899463.
  * [29] M. Robillard, W. Coelho, and G. Murphy, “How effective developers investigate source code: an exploratory study,” _IEEE Transactions on Software Engineering_ , vol. 30, no. 12, pp. 889–903, Dec. 2004.
  * [30] M. Schröer and R. Koschke, “Recording, visualising and understanding developer programming behaviour,” in _Proc. of International Conference on Software Analysis, Evolution and Reengineering_. IEEE, 2021, pp. 561–566.
  * [31] D. M. Arya, J. L. Guo, and M. P. Robillard, “How programmers find online learning resources,” _Empirical Software Engineering_ , vol. 28, no. 2, 2023, https://doi.org/10.1007/s10664-022-10246-y.
  * [32] P. Chatterjee, M. Kong, and L. Pollock, “Finding help with programming errors: An exploratory study of novice software engineers’ focus in Stack Overflow posts,” _Journal of Systems and Software_ , vol. 159, p. 110454, 2020, https://www.sciencedirect.com/science/article/pii/S0164121219302286.
  * [33] M. J. de Dieu, P. Liang, and M. Shahin, “How do developers search for architectural information? an industrial survey,” in _Proc. of 19th International Conference on Software Architecture_ , 2022, pp. 58–68.
  * [34] A. J. Ko, B. A. Myers, M. J. Coblenz, and H. H. Aung, “An exploratory study of how developers seek, relate, and collect relevant information during software maintenance tasks,” _IEEE Transactions on Software Engineering_ , vol. 32, no. 12, pp. 971–987, Dec. 2006, http://ieeexplore.ieee.org/document/4016573/.
  * [35] L. Murphy, G. Lewandowski, R. McCauley, B. Simon, L. Thomas, and C. Zander, “Debugging: the good, the bad, and the quirky – a qualitative analysis of novices’ strategies,” _ACM SIGCSE Bulletin_ , vol. 40, no. 1, pp. 163–167, Mar. 2008, http://doi.org/10.1145/1352322.1352191.
  * [36] A. J. Ko, R. DeLine, and G. Venolia, “Information needs in collocated software development teams,” in _29th International Conference on Software Engineering (ICSE’07)_. IEEE, 2007, pp. 344–353.
  * [37] M. Beller, N. Spruit, D. Spinellis, and A. Zaidman, “On the dichotomy of debugging behavior among programmers,” in _Proc. of 40th International Conference on Software Engineering_. New York, NY, USA: ACM, 2018, pp. 572–583, https://doi.org/10.1145/3180155.3180175.
  * [38] L. Han, T. Chen, G. Demartini, M. Indulska, and S. Sadiq, “On understanding data worker interaction behaviors,” in _Proc. of 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , 2020, pp. 269–278.
  * [39] S. Chattopadhyay, N. Nelson, T. Nam, M. Calvert, and A. Sarma, “Context in programming: an investigation of how programmers create context,” in _Proc. of 11th International Workshop on Cooperative and Human Aspects of Software Engineering_. ACM, 2018, pp. 33–36, https://dl.acm.org/doi/10.1145/3195836.3195861.
  * [40] J. Sillito, K. De Voider, B. Fisher, and G. Murphy, “Managing software change tasks: An exploratory study,” in _2005 International Symposium on Empirical Software Engineering_ , 2005.
  * [41] J. Starke, C. Luce, and J. Sillito, “Searching and skimming: An exploratory study,” in _Proc. of International Conference on Software Maintenance_. IEEE, 2009, pp. 157–166.
  * [42] A. P. Koenzen, N. A. Ernst, and M.-A. D. Storey, “Code duplication and reuse in Jupyter notebooks,” in _Proc. of Symposium on Visual Languages and Human-Centric Computing_. IEEE, 2020, pp. 1–9.
  * [43] R. Michaels, N. Tula, S. Ramisetty-Mikler, D. Nurmuradov, and R. Bryce, “An empirical study of how novice programmers search the web for help,” _The Journal of Computing Sciences in Colleges_ , p. 42, 2020.
  * [44] G. R. Bai, J. Kayani, and K. T. Stolee, “How graduate computing students search when using an unfamiliar programming language,” in _Proceedings of the 28th International Conference on Program Comprehension (ICPC ’20)_. ACM, 2020, pp. 160–171.
  * [45] W. Maalej, R. Tiarks, T. Roehm, and R. Koschke, “On the comprehension of program comprehension,” _ACM Transactions on Software Engineering and Methodology_ , vol. 23, no. 4, pp. 1–37, 2014, https://dl.acm.org/doi/10.1145/2622669.
  * [46] H. Nygren, J. Leinonen, and A. Hellas, “Tracking students’ Internet browsing in a machine exam,” in _Proc. of 6th Computer Science Education Research Conference_. ACM, 2017, pp. 91–95.
  * [47] P. Blikstein, “Using learning analytics to assess students’ behavior in open-ended programming tasks,” in _Proc. of 1st International Conference on Learning Analytics and Knowledge_. ACM, 2011, pp. 110–116, http://dl.acm.org/citation.cfm?doid=2090116.2090132.
  * [48] H. Meier, E. Tõnisson, M. Lepp, and P. Luik, “Behaviour patterns of learners while solving a programming task: An analysis of log files,” in _Proc. of Global Engineering Education Conference_. IEEE, 2020, pp. 685–690.
  * [49] A. Vihavainen, J. Helminen, and P. Ihantola, “How novices tackle their first lines of code in an IDE: analysis of programming session traces,” in _Proc. of 14th Koli Calling International Conference on Computing Education Research_. ACM, 2014, pp. 109–116, http://dl.acm.org/citation.cfm?doid=2674683.2674692.
  * [50] A. Ciborowska, N. A. Kraft, and K. Damevski, “Detecting and characterizing developer behavior following opportunistic reuse of code snippets from the Web,” in _Proc. of 15th IEEE/ACM International Conference on Mining Software Repositories_ , 2018, pp. 94–97.
  * [51] J. T. Jacques and P. O. Kristensson, “Studying programmer behaviour at scale: A case study using Amazon Mechanical Turk,” in _Companion Proc. of 5th International Conference on the Art, Science, and Engineering of Programming_. New York, NY, USA: ACM, 2021, pp. 36–48.
  * [52] G. R. Bai, B. Clee, N. Shrestha, C. Chapman, C. Wright, and K. T. Stolee, “Exploring tools and strategies used during regular expression composition tasks,” in _2019 IEEE/ACM 27th International Conference on Program Comprehension_ , 2019, pp. 197–208.
  * [53] J. Escobar-Avila, D. Venuti, M. Di Penta, and S. Haiduc, “A survey on online learning preferences for computer science and programming,” in _Proc. of 41st International Conference on Software Engineering: Software Engineering Education and Training_. IEEE, 2019, pp. 170–181, https://ieeexplore.ieee.org/document/8802102/.
  * [54] C. Treude and M. P. Robillard, “Understanding Stack Overflow code fragments,” in _Proc. of International Conference on Software Maintenance and Evolution_. IEEE, 2017, pp. 509–513, http://ieeexplore.ieee.org/document/8094452/.
  * [55] Y. Wu, S. Wang, C.-P. Bezemer, and K. Inoue, “How do developers utilize source code from stack overflow?” _Empirical Software Engineering_ , vol. 24, no. 2, pp. 637–673, 2019, http://link.springer.com/10.1007/s10664-018-9634-5.
  * [56] D. Yang, A. Hussain, and C. V. Lopes, “From query to usable code: An analysis of Stack Overflow code snippets,” in _Proc. of 13th Working Conference on Mining Software Repositories_. New York, NY, USA: ACM, 2016, pp. 391–401, https://doi.org/10.1145/2901739.2901767.
  * [57] T. Zhang, G. Upadhyaya, A. Reinhardt, H. Rajan, and M. Kim, “Are code examples on an online q&a forum reliable?: a study of api misuse on Stack Overflow,” in _Proc. of 40th International Conference on Software Engineering_. IEEE, 2018, pp. 886–896.
  * [58] J. Zhou and R. J. Walker, “Api deprecation: A retrospective analysis and detection method for code examples on the web,” in _Proc. of 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering_. New York, NY, USA: ACM, 2016, p. 266–277, https://doi.org/10.1145/2950290.2950298.
  * [59] R. Abdalkareem, E. Shihab, and J. Rilling, “On code reuse from StackOverflow: An exploratory study on Android apps,” _Information and Software Technology_ , vol. 88, pp. 148–158, 2017.
  * [60] A. Gkortzis, D. Feitosa, and D. Spinellis, “Software reuse cuts both ways: An empirical analysis of its relationship with security vulnerabilities,” _Journal of Systems and Software_ , vol. 172, p. 110653, 2021, https://www.sciencedirect.com/science/article/pii/S0164121220301199.
  * [61] T. Delev and D. Gjorgjevikj, “Static analysis of source code written by novice programmers,” in _Proc. of Global Engineering Education Conference_. IEEE, 2017, pp. 825–830, http://ieeexplore.ieee.org/document/7942942/.
  * [62] C. Seaman, “Qualitative methods in empirical studies of software engineering,” _IEEE Transactions on Software Engineering_ , vol. 25, no. 4, pp. 557–572, Jul. 1999.
  * [63] Q. Charatan and A. Kans, _Java in Two Semesters: Featuring JavaFX_ , 4th ed. Springer International Publishing, 2019.
  * [64] V. Braun and V. Clarke, “Using thematic analysis in psychology,” _Qualitative Research in Psychology_ , vol. 3, no. 2, pp. 77–101, 2006.
  * [65] K.-J. Stol and B. Fitzgerald, “The abc of software engineering research,” _ACM Transactions on Software Engineering and Methodology (TOSEM)_ , vol. 27, no. 3, pp. 1–51, 2018.
  * [66] R. Bakeman and V. Quera, “Behavioral observation,” in _APA handbook of research methods in psychology_. American Psychological Association, 2012, vol. 1, Foundations, planning, measures, and psychometrics, pp. 207–225.