Only a few studies <cit.> make use of proprietary dataset for their experiments.
<cit.> collect railway software systems from their industry partner (CAF) to evaluate their work. Other studies analyze proprietary search logs from Bing search engine <cit.> and Stack Overflow <cit.> to generate their reformulated queries.
According to existing literature <cit.>, open source and closed source systems could be different in various aspects (e.g., development practices, bug resolution, code reviews) and thus findings from these two types of systems might not always align with each other.

Other studies <cit.> use a single code search engine ( Lucene)
to evaluate their reformulated queries.
However, the same query might not deliver the same performance across multiple search engines <cit.>.
A number of studies <cit.> also make use of a limited set of queries ( $\le$50) for their evaluation.
A few studies <cit.> use only one subject system. Thus, their findings might
suffer from the lack of generalizability as well. Several primary studies <cit.> might not perform well for all types of queries due to their unique requirements for a given query (e.g., textual similarity to API classes <cit.>, presence of nouns <cit.>, syntactic correctness <cit.>), which cannot be always satisfied.

From Fig. <ref>, we note that the primary studies suffering from this limitation (i.e., CH3) frequently adopt three methodologies - M1, M3, and M6. However, unlike the above ones, the lack of generalizability is closely associated to the dataset or evaluation mechanism rather than the methodologies adopted in query reformulations.

Mitigation strategy. To improve the generalizability of findings, future work should focus on designing comprehensive experiments. For example, subject systems should come from multiple programming languages (e.g., Java, C/C++, Python, Javascript) <cit.>, development practices (e.g., open-source, closed-source) <cit.>, and application domains. More subject systems and queries should also be chosen to rigorously evaluate any proposed technique. Furthermore, multiple search engines should be used to evaluate the original and reformulated queries to avoid spurious findings <cit.>.

CH4–Issues towards practical adoption: About 33% (23/70) of our primary studies suffer from

major issues that might prevent their adoption by software developers. Several studies <cit.>
make use of Latent Semantic Indexing (LSI) and term-document co-occurrence matrix to reformulate their search queries.
Although the LSI might be useful to mitigate the vocabulary mismatch problem <cit.>, it suffers from the curse of dimensionality problem[http://mlwiki.org/index.php/Vector_Space_Models].
That is, a large, sparse matrix is needed to store the semantic space, which can be both costly and inefficient.
Besides, the semantic space needs to be updated frequently for each incoming query before its execution, which is time-consuming and undesirable.
Several approaches require non-trivial items such as test cases <cit.>, code examples <cit.>, and salient keywords <cit.> from developers, which could make the developers reluctant to use these approaches.
Several studies <cit.> make use of ground truth information to identify appropriate search keywords from a bug report. Although these studies might be useful to provide a proof of concept, they are not suitable for a practical use since the ground truth is not generally known during query construction <cit.>.
Several other studies involve costly computations such as genetic algorithms <cit.>, frequent AST construction <cit.>, or ontology construction <cit.>, which might outweigh the benefit of query reformulation.

A few studies <cit.> need significant configuration management or parameter tuning, whereas the others use proprietary, publicly unavailable datasets (e.g., Bing search logs <cit.>, Stack Overflow search logs <cit.>).

There also exist other approaches <cit.> that involve the developers in costly trials and errors.
Furthermore, 47% of our studies do not have a replication package and thus might not be replicated properly.
All these constraints and issues might prevent the developers from using $\approx$33% of our primary studies to reformulate their search queries.

Mitigation strategy. Future work can take several steps to mitigate this threat. First, any cost-benefit analysis through human studies will inform the usability of a technique. Developer frequently used general-purpose search engines (e.g., Google) for code search that support traditional, natural language queries <cit.>.

Thus they should not be forced to design non-traditional queries (e.g., test cases, example code) for code search. The designed tools should be easily integrated into the working context of a developer <cit.>. Furthermore, the replication packages should be available to foster reuse and adoption.

CH5–Human bias and lack of rigour in evaluation: About 56% (39/70) of our primary studies
might suffer from subjective bias and the lack of rigour in their evaluation. Several studies <cit.> evaluate their approaches involving
undergraduate or graduate students in a laboratory setting. These participants neither represent the real users ( professional developers) nor are adequate in number. For example, 80% of our studies that engage humans have less than 25 participants each (Fig. <ref>).
These participants were also responsible for both the ground truth preparation and the query evaluation.
Thus, there exists a good chance of subjective bias in the evaluation since the appropriate measures (e.g., inter-rater agreement, pooling method <cit.>) were not always taken.
Replication of an existing work might also introduce implementation bias especially if the replication package is not available. A few studies <cit.> might suffer from this bias.
Many primary studies <cit.> report their best performances using their well-tuned configurations against certain subject systems (Table <ref>).

Unfortunately, the same performance might not be guaranteed for other subject systems with the same configurations.
Several studies <cit.> do not make use of any frequently adopted metrics whereas the others <cit.> do not compare with established baselines from the literature. Besides, 49% of the primary studies do not perform any statistical test (e.g., significance tests), which is an important step of rigorous validation.

From Fig. <ref>, we note that the studies suffering from the above limitation or challenge (i.e., CH5) frequently adopt four methodologies: M1, M3, M6, and M8. Unlike many of the above ones, this challenge is closely associated to the experiment design rather than the methodologies adopted in search query reformulations.

Mitigation strategy. Future work should involve appropriate participants (e.g., software practitioners) and a sufficient number of participants to mitigate the threats from human studies. The ground truth should be also carefully constructed using either polling method or inter-rater agreement analysis <cit.>. The source code, experimental data, and other details should be made publicly available to facilitate smooth replication by third parties <cit.>. Furthermore, the conducted experiments should use appropriate performance metrics and statistical tests based on evaluation type and working contexts (check RQ2 for details).

CH6–Inappropriate use of external tools and dependencies: Over the last two decades, many tools, methods and technologies from Information Retrieval (IR) and Natural Language Processing (NLP) domains have been used to solve Software Engineering problems. Although they work with their default configurations, they need to be configured properly for Software Engineering problems for an optimal performance. They also need to be used with caution <cit.>.
For example, several studies make use of WordNet <cit.> and
Stanford CoreNLP POS tagger <cit.> in the context of Software Engineering problems.
Both these tools were trained on regular English texts ( Wall Street Journal) rather than the source code.
Thus, the semantic inferences and POS tag predictions made by these tools could be either noisy or even inaccurate, as confirmed by the existing evidence <cit.>. Many primary studies <cit.> also use TF-IDF <cit.> as a

de facto term weighting method to select important keywords from source code.
However, TF-IDF was originally designed for regular texts ( news articles, legal documents) rather than source code.
Regular texts and source code differ in their syntax, structures, semantics, and vocabulary <cit.>. Thus, the adoption of TF-IDF to select keywords from source code might not be appropriate. Several studies heavily depend on third-party items such as code change history <cit.>, semantic web technologies (e.g., SPARQL) <cit.>, tags from Stack Overflow <cit.>, and input code examples <cit.>, and thus might be negatively affected by their absence.

Mitigation strategy. Future work can take several steps to mitigate this threat. For example, the established tools from other domains (e.g., WordNet <cit.>, Word2Vec <cit.>) need to be carefully re-trained using software engineering data (e.g., source code, bug reports). Traditional techniques for keyword selection (e.g., TF-IDF <cit.>) should be adapted and tuned to software engineering data. Besides, the future techniques should depend on open-source, well-maintained dependencies to avoid any potential integration issues.

CH7–Lack of a sound theory: Several primary studies <cit.> make counter-intuitive or unsubstantiated theoretical assumptions in their approach.

<cit.> attempt to determine a query's intent using the code change history.
First, they mine the code change history of 631 GitHub projects and construct a database that contains thousands of $<$, $>$ pairs. Then they analyze the code segment retrieved by a query, identify its frequently changed identifier names
using the database as query intent, and then reformulate the query with the intent.

However, the idea of capturing a query's intent with the changed code

lacks a strong theoretical justification. A few other studies <cit.> might also have similar issues.

According to <cit.>, an ideal bug report contains three parts- (a) observed behaviour (OB), (b) expected behaviour (EB) and (c) steps to reproduce (S2R) the bug. Using an empirical study,  demonstrate that OB can be used as a reformulated query for bug localization.
However, how OB could be less noisy than the S2R was demonstrated neither theoretically nor empirically. According to a recent work <cit.>, S2R often contains terms that better indicate what the bug is about and includes key terms. Thus,  solely relying on OB might lead to the removal of important search keywords from a bug report <cit.>.

According to <cit.>, WordNet could be effective for query expansion during interface driven code search (IDCS) rather than keyword based code search (KBCS). However, such a claim was also not theoretically justified.

Mitigation strategy. Future work can take a few steps to mitigate this threat. For example, the intuition or logical reasoning behind any proposed technique should be clearly articulated before presenting any experimental evidence. Comprehensive understanding of the relevant literature can help develop such intuition. Then any proposed theory should be also substantiated by strong empirical evidence to place it in the literature.

CH8–Miscellaneous: About 21% (15/70) of our primary studies suffer from miscellaneous issues and limitations as follows. (a) Chronology of bug reports is not respected. <cit.> do not respect the chronology of bug reports through their use of 10-fold cross validation. That is, they might have identified previous bugs from the history
by training their approach on the later bug reports, which is impractical. (b) Underlying semantics are either overlooked or misused. The primary studies on query quality prediction <cit.> make use of lexical and syntactic properties of a query but mostly overlook the semantic aspect. They also target binary classification (e.g., good query and bad query), which might not be sufficient since the search queries can have non-binary quality levels (e.g., very good, good, bad, very bad). <cit.> consider two code segments similar if they have similar AST nodes. However, their approach does not guarantee that both code segments have similar semantic, i.e., implement the same programming task. <cit.> also assume that a query's intent can be better expressed with the changed identifier names rather than focusing on the semantics of the query, which is counter-intuitive.
(c) Corner-cases are not considered. <cit.> extend the vocabulary of source code by annotating program entities (e.g., ) with their equivalent concepts from natural language texts (e.g., integer array). However, their approach, as the authors report, might fall short for the nested program entities with subsumption problems. <cit.> reformulate natural language  query (i.e., change request) into source code elements by mining version history. However, their approach might be restricted to existing code elements and thus might be insufficient for change requests where new software features are requested.

Mitigation strategy. To mitigate the above threats, future work can take several steps including performing time-aware training with bug reports, leveraging query semantics, and considering corner-cases in their techniques.

Summary for RQ$\mathbf{_3}$:

Almost half of our primary studies might add unexpected noise to a query during its reformulation whereas 26% of them impose extra cognitive burden on the developers. The majority of these studies also suffer from a lack of generalizability due to their limited datasets, queries, and evaluation. About 33% of our studies suffer from the issues that might prevent their adoption by software developers whereas more than half of them could have subjective bias in their evaluation. Furthermore, several studies heavily depend of third-party items for their query reformulations and the others make counter-intuitive assumptions in their approaches.

§.§ Answering RQ$\mathbf{_4}$: Evidence of existing researches on automated query reformulation

We collect publication year, venue, and authors of each primary study to construct the evidence of

existing researches on automated query reformulations.
Such an evidence not only demonstrates the interest and enthusiasm around the research topic
but also shows the quality of the work being produced and published.
Tables <ref>, and <ref> and Fig. <ref>
present various statistics on the conducted studies over the last 15+ years.

Table <ref> shows the frequency of publications on automated query reformulations per calendar year.

We see that the first work on automated query reformulation got published back in 2004. <cit.> first introduced query reformulation in the context of concept location where they used word similarity derived from latent semantic space to automatically reformulate a given search query ( “font style").
From 2005 to 2012, there had been a few studies targeting query reformulations, and on average 2-3 papers got published every year. However, the query reformulation received much attention as a research topic since 2013. We believe that such an attention or enthusiasm could be partially attributed to a few contemporary doctoral theses on query reformulations and code searches <cit.>. Since 2013, there has been a significant increase in the research activities and publication counts. For example, during the last six years (2015–2021), 47 studies on query reformulations got published where 2016 and 2017 were the most productive years (e.g., 10 studies per year). It also should be noted that $\approx$80% of our selected primary studies were conducted during the last nine years (2013–2021).

Existing publications on query reformulation targeting source code search

Year     Published studies     Total

2004     S50 (WCRE)      1

2006     S18 (LNCS)     1

2007     S43 (ASOD)     1

2009     S12 (ICSE), S30 (ICSM), S55 (LNCS)     3

2010     S48 (ICSE), S51 (ASE)     2

2011     S3 (IST), S31 (ICPC), S47 (ASE)     3

2012     S8 (ASE)     1

2013     S6 (MSR), S10 (ICSE), S24 (ICSE), S28 (MSR), S52 (ASE)     5

2014     S1 (IST), S4 (ASE), S11 (ICSE), S27 (EMSE), S42 (MSR), S45 (MSR)     6

2015     S13 (SCAM), S14 (ASE), S15 (ASE), S33 (ICSME), S36 (SANER), S46 (ICSE), S53 (SANER)     7

2016     S5 (Internetware), S17 (VLC), S23 (SPLC), S29 (ICSE), S34 (TSC), S37 (Internetware), S38 (ASE), S39 (SANER), S41 (ICSE), S56 (ASE)     10

2017     S2 (ICCIT), S20 (TSE), S21 (JSEP), S22 (ASE), S26 (ASE), S32 (TOSEM), S40 (SANER), S44 (ICSME), S49 (WSDM), S54 (JSEP)     10

2018     S7 (EMSE), S9 (DKE), S16 (FCS), S19 (ICSME), S25 (FSE), S35 (SPE), S60 (Access), S67 (ICSME), S68 (ICSME)     9

2019      S57 (IST), S58 (MAPL), S59 (AUSE), S61 (SAC), S63 (SPE), S65 (Access), S69 (TSE), S70 (ICPC)     8

2020     S62 (IST), S64 (Access)     2

2021     S66 (ICSE)     1

2c      70

Existing Publications on Query Reformulation Targeting Source Code Search

Venue     Published studies     Total

ASE     S4 (2014), S8 (2012), S14 (2015), S15 (2015), S22 (2017), S26 (2017), S38 (2016), S47 (2011), S51 (2010), S52 (2013), S56 (2016)     11

ICSE     S10 (2013), S11 (2014), S12 (2009), S24 (2013), S29 (2016), S41 (2016), S46 (2015), S48 (2010), S66 (2021)     9

ICSM/ICSME     S19 (2018), S30 (2009), S33 (2015), S44 (2017), S67 (2018), S68 (2018)     6

WCRE/SANER     S50 (2004), S36 (2015), S39 (2016), S40 (2017), S53 (2015)     5

IST     S1 (2014), S3 (2011), S57 (2019), S62 (2020)     4

MSR     S6 (2013), S28 (2013), S42 (2014), S45 (2014)     4

Access     S60 (2018), S64 (2020), S65 (2019)     3

Internetware     S5 (2016), S37 (2016)     2

SPE     S35 (2018), S63 (2019)     2

LNCS     S18 (2006), S55 (2009)     2

JSEP     S21 (2017), S54 (2017)     2

EMSE     S7 (2018), S27 (2014)     2

TSE     S20 (2017), S69 (2019)      2

ICPC     S31 (2011), S70 (2019)     2

VLC      S17 (2016)     1

WSDM      S49 (2017)     1

AUSE      S59 (2019)     1

FSE      S25 (2018)     1

ASOD      S43 (2007)     1

SCAM      S13 (2015)     1

DKE      S9 (2018)     1

TOSEM      S32 (2017)     1

TSC      S34 (2016)     1

ICCIT      S2 (2017)     1

SPLC      S23 (2016)     1

MAPL      S58 (2019)     1

SAC      S61 (2019)     1

FCS      S16 (2018)     1

2c      70

Publications on code search and query reformulation

Table <ref> shows how many primary studies got published at each venue in different calendar years.
We see that 50% (35/70) of our selected studies got published in the top venues of Software Engineering
such as ICSE, ASE, ICSME, SANER, and MSR.

About 28% of them (20/70) also made into the flagship conferences such as ICSE (A*) and ASE (A*). Many of these 70 studies were extended later and got published in the reputed journals such as EMSE and JSEP.

There were also a number of Journal-First studies on query reformulations that got published in several high-quality journals such as TSE, TOSEM, EMSE, JSEP, SPE and TSC. While ASE and ICSE communities pioneered this research topic, other communities ( ICSME, MSR, SANER) have also been catching up to speed especially in the last few years.

We also investigate the comparative research interest in query reformulation and in code search. It should be noted each primary study is related to code search but each study on code search might not be related to query reformulation. Thus, we repeat the relevant part of our study selection process and identify the studies on code search as well. Fig. <ref> summarizes our analysis using a bar plot.
According to our investigation,  34% (70/207) of the code search studies focus on query reformulations.
As shown in Fig. <ref>, a significant fraction of code search studies target query reformulation every year. For example, a maximum of 23 code search studies were conducted in 2015, and 7 of them focused on search query reformulations.

We also identify the authors from each of our 70 primary studies and calculate their contribution frequency.

We found a total of 172 Software Engineering researchers who contributed to the research of automated query reformulations
to improve code search. Appendix <ref> shows the authors who contributed to at least two primary studies.

The demand for low-cost, accurate, and easy-to-use solutions for code search is likely to grow in the coming years due to
the constant growth of open-source repositories ( GitHub, BitBucket) and the increasing complexity or size of modern software systems.

Reformulation of a query is an inevitable step of code search regardless of the search context.
Thus, automated query reformulation is likely to be an active, interesting research topic for the years to come.

Summary of RQ$\mathbf{_4}$: To date, according to our investigation, a total of 70 primary studies
on automated query reformulations supporting code search have been conducted. About 80% of them were conducted in the last nine years. This trend not only indicates the timeliness or relevance but also the attention or enthusiasm around this research topic. A large group 172 researchers contributed to this topic. About 50% of the primary studies got published at the top venues of Software Engineering such as ICSE, ASE, ICSME, MSR, and SANER.

Given the increasing size and complexity of modern software systems, the research activities on automated query reformulations and code searches are likely to increase in the coming years.

§.§ Answering RQ$\mathbf{_5}$: A closer look at local code search and Internet-scale code search

Local code search such as bug localization, concept location, or feature location
attempts to find the location of a software bug, a concept, or a feature respectively within the code of a software system.
On the other hand, Internet-scale code search attempts to find out relevant, reusable code segments from thousands of software projects stored at Internet-scale code repositories ( GitHub). More specifically, while the former detects code locations that need to be modified, the later finds code segments that can be reused.

Although these two searches differ in their working context and end goal, they could overlap in several other aspects.
In this systematic literature review, we analyze 70 primary studies where 58% of them reformulate queries for local code searches and the remaining 42% studies attempt to improve Internet-scale code search through query reformulations (e.g., Fig. <ref>).
We refer to these two groups of studies as QR$_{LCS}$ and QR$_{ICS}$ respectively.

A closer look at the query reformulations supporting these two searches

can provide valuable insights.  In this section, we compare and contrast between these two groups of primary studies based on

their underlying methodologies, algorithms, evaluation methods, challenges, and limitations as follows.

Use of methodologies by the primary studies supporting (a) local code search, and (b) Internet-scale code search

Types of reformulation provided by the primary studies supporting (a) local code search, and (b) Internet-scale code search

Relevance feedback by the primary studies supporting (a) local code search, and (b) Internet-scale code search

Underlying methodologies and algorithms in query reformulation. Among 70 primary studies, 40 studies (QR$_{LCS}$)
reformulate queries to support local code searches ( bug localization, concept location) whereas the remaining 30 studies (QR$_{ICS}$) target Internet-scale code search. Fig. <ref> shows the methodologies adopted by these two groups of primary studies.
We see that term weighting and relevance feedback mechanism are frequently used to select keywords and to reformulate queries by the primary studies associated to local code searches. About 23% of them adopt this methodology whereas the ratio is only 11% for the studies from QR$_{ICS}$.
In local code search, there are several software artifacts such as bug reports and feature requests from which
the keywords can be chosen based on their weight or appropriateness, which explains the popularity of term weighting methods.
On the other hand, in the Internet-scale code search, query keywords are often chosen by the developers on the fly according to their information need.
Semantic relations, word co-occurrences, and thesauri have also been used by 15% and 25% of the primary studies from these two groups respectively.
However, the most popular methodologies for query reformulations that support Internet-scale code search are data mining and API recommendation.
About 28% of the studies from QR$_{ICS}$ adopt this methodology whereas the ratio is only 11% for the counterpart.
In Internet-scale code search, search queries are often short (e.g., 2–3 keywords) and thus they require an expansion to retrieve the desired code <cit.>. Many primary studies <cit.> expand these queries with relevant API classes or methods mined from large software repositories (e.g., GitHub, Stack Overflow), which explains the popularity of API recommendation methodology (i.e., M6).
Several studies <cit.> also expand these short queries with synonyms or similar words
from well-known thesauri such as WordNet <cit.>. This explains a high usage ratio (25%) of thesaurus based methodology (i.e., M3) by the studies from QR$_{ICS}$
(Fig. <ref>-(b)).

Fig. <ref> shows the types of query reformulations performed by primary studies supporting both local and Internet-scale code searches.
We see that query expansion is very common for both groups (e.g., 41%–77%). However, queries are more frequently expanded in the Internet-scale code search due to their short length <cit.>. Query reduction is less common for both groups of studies. Interestingly, in local code search, a significant number  of queries (e.g., 44%) receive other types of reformulations such as the combination of query expansion, reduction, or replacement.

Fig. <ref> shows how relevance feedback is used to reformulate queries by the primary studies.
We see that query reformulations in the local code search use comparatively more feedback. About 54% of them use either direct feedback from developers or pseudo-relevance feedback from a search engine. On the other hand, only 33% of studies from the Internet-scale code search (QR$_{ICS}$) use any form of feedback to reformulate their queries. It should be noted that these QR$_{ICS}$ studies capture less feedback

although they involve human developers more frequently than their counterpart (check Section <ref>-(d)).

Any local code search takes place on a single software project whereas the Internet-scale code search deals with thousands of cross-domain software projects.
Thus, the results from Internet-scale code search might be noisier, and capturing relevance feedback against them could be costly and challenging.

Such a high cost might have discouraged any form of relevance feedback. According to Fig. <ref>, 46%–67% of studies from both groups did not use any form of relevance feedback to reformulate their search queries.

Evaluation methods used by the primary studies supporting (a) local code search, and (b) Internet-scale code search

Popular evaluation metrics used by the primary studies supporting (a) local code search, and (b) Internet-scale code search

Evaluation and validation in query reformulation. Among 70 primary studies,
45 (64%) studies use re-enactment-based evaluation, 18 (26%) studies involve human participants whereas 6 (9%) studies use a combination of both (see Fig. <ref>).

Fig. <ref> shows a breakdown of these statistics for two groups of primary studies.
We see that 73% of the studies supporting local code searches evaluate their queries through re-enactment using historical artifacts (e.g., bug reports, version control history). On the contrary, 52% of the studies supporting Internet-scale code search
adopt the same where they make use of existing benchmarks and programming Q&A sites (e.g., Stack Overflow).
However, human participants were more frequently involved in the Internet-scale code searches. For example,

$\approx$50%  (28% + 21%) of the studies from the Internet-scale code search involve human developers in their evaluation. Such a ratio is only 24% for the counterpart. Existing primary studies from local code search

often construct their dataset and ground-truth from the version control history (e.g., bug-fixing commits) of open source projects that are widely available. On the contrary, historical data from Internet-scale code search such as search engine logs are proprietary artifacts that might not be publicly available. Thus, these studies often involve human developers to evaluate their queries.

Fig. <ref> further contrasts between two groups of our primary studies
based on their use of popular performance metrics.
We see that Hit@K, Recall, MAP and MRR are generally popular among both groups. However, the studies from local code search are more interested about query effectiveness (QE).

In local code searches such as concept location, bug localization, or feature location, the quick identification of the first correct result is important. The underlying idea is that once the first correct document is located,
the other relevant documents can be easily located through their structural dependencies ( call graphs) within a software project <cit.>. The same might not be true for the Internet-scale code search and 7% of their studies use QE.
Their search results are retrieved from thousands of cross-domain software projects and static dependencies might not be applicable beyond a single project. We also see that unlike QR$_{LCS}$,  QR$_{ICS}$ studies are more interested in developer-oriented metrics such as spent time, NDCG, and Likert that capture a developer's impression on the results retrieved by a query.

Challenges and limitations of query reformulations targeting (a) local code search, and (b) Internet-scale code search

Challenges and limitations in query reformulation. Since the studies supporting local code search (QR$_{LCS}$) and the studies supporting Internet-scale code search (QR$_{ICS}$) overlap in their methodologies (check Fig. <ref>),

they might suffer from similar types of challenges or limitations. Fig. <ref>
summarizes our analysis in this aspect. We see that the studies from both groups are similarly affected by several challenges and limitations (Table <ref>). About 17% of QR$_{LCS}$ studies could suffer from noise in their queries whereas the corresponding statistic is 21% for the QR$_{ICS}$ studies. In local code search, developers can select keywords from existing artifacts (e.g., bug reports) whereas they need to select keywords on the fly during Internet-scale code search. Thus, the later queries need more extension (Fig. <ref>) and might have a higher chance of getting noisy keywords.

The existing query reformulation approaches in the local code search impose more cognitive burden on the developers than their counterpart. Changes to wrong code often have major negative consequences ( subsequent bugs) <cit.>. Thus, precisely locating the source code of interest is a major step in the local code searches. On the contrary, developers might accept the partially relevant code examples during Internet-scale code search, which they can customize based on their needs.
Both groups of studies also suffer from a lack of generalizability, which is the most prevalent issue. The findings from
23%–28% of the studies from both groups might not be generalizable, as acknowledged or implied by their authors.
Although existing software artifacts (e.g., software, bug reports, version control history) are extensively used to evaluate query reformulations in the local code search, the majority of them target Java-based systems. According to an existing evidence <cit.>, the findings from Java-based systems might not generalize for other systems such as C/C++ systems.
On the other hand, the datasets used to evaluate query reformulations in the Internet-scale code search are either small or proprietary in nature. For example, 60% of the QR$_{ICS}$ studies use 75 or less number of queries each in their evaluation.
A few studies make use of proprietary items such as search logs from Bing <cit.> and Stack Overflow search engine <cit.>, which might not be available for a public reuse and replication.  From Fig. <ref>, we also see that both groups suffer similarly with the remaining issues including human bias, weak evaluation (CH5, Table <ref>), extreme external dependencies (CH6, Table <ref>), and a lack of sound theoretical justification (CH7, Table <ref>).

Comparison between query reformulations for local code search and Internet-scale code search

Dimension     Local code search (LCS)     Internet-scale code search (ICS)

3cQuery & search intent

Initial query (ad hoc)     Preprocessed version of a bug report, a change request, or a feature request     Developer chosen keywords

Initial query length     Comparatively long     Short ( 1-3 keywords <cit.>)

Keyword selection     Performed with term weighting and relevance feedback (M1)     Performed with data mining and API recommendation (M6)

Query reduction     Used by 15% of the studies     Used by 10% of the studies

Operational context     Software change tasks (e.g., bug fixing)     General-purpose programming tasks

Search intent     Precisely locating an entity of interest     Locating the most relevant code example to a programming task

End goal     Making changes to the existing code     Implementing a programming task

Relevance feedback     Used by 54% of the studies     Used by only 33% of the studies

3cCorpus, ground truth, & evaluation

Corpus documents     Source documents (e.g., classes, methods) from a single software project ( JEdit <cit.>)     Source documents from thousands of software projects ( Sourcerer <cit.>)

Ground truth construction     Commits from the version control history <cit.>     Relevance judgement from developers using pooling method <cit.>

External resources     Past bug reports, bug-fixing history, author history, knowledge graph <cit.>     WordNet, Stack Overflow, GitHub

Ranking priority     First correct result should rank high     All correct results should rank high

Popular metrics     Hit@K, Recall, MAP     Hit@K, Recall, MAP

Major challenges     Lack of generalizability, developer bias, weak evaluation, and noisy queries     Lack of generalizability, developer bias, weak evaluation, and noisy queries

Experience required     Domain knowledge ( design, architectures) of a system under study     Relevant API libraries

Research evidence     58% of the primary studies      42% of the primary studies

Table <ref> further compares and contrasts between the query reformulation approaches supporting local code search and Internet-scale code search. We see that they are different from each other in terms of the majority of  dimensions such as query length, keyword selection methodology, operational context, search intent, and even in the ranking priority.
For example, in a local code search, developers attempt to precisely locate the source code entity that needs a change.
Changing the wrong code entity can have serious consequences <cit.>. On the contrary, in the Internet-scale code search, developers attempt to locate the most relevant code example that can be reused to implement a programming task.
Despite these differences, the two groups of studies are also surprisingly similar across several dimensions such as their performance metrics (e.g., Hit@K, Recall, MAP) and experienced challenges or limitations (e.g., lack of generalizability).

Summary of RQ$\mathbf{_5}$:
About 58% of our primary studies support local code searches (e.g., concept location, bug localization) whereas the remaining 42% focus on the Internet-scale code search. The query reformulation approaches supporting these searches share their methodologies, algorithms, performance metrics, and even suffer from similar challenges and limitations. Appropriate keyword selection is an essential step of any query reformulation.
In local code search, keywords are often selected using

various term weighting methods (e.g., TF-IDF) whereas data mining and API recommendations are adopted
in the context of Internet-scale code search. The primary studies from these two searches also

differ in their reformulation types, evaluation methods, and human participation. However, both groups of studies suffer from a common set of issues such as a lack of generalizability, weak evaluation, and noisy queries.

§.§ Answering RQ$\mathbf{_6}$: Recommended practices in query reformulations for code search

Query reformulation is an integral part of source code search regardless of working context. Our systematic review using 70 primary studies informs the frequently adopted methodologies (RQ1) and evaluation methods (RQ2), and identifies their common challenges and limitations (RQ3). While query reformulation practices could be significantly different for local and Internet-scale code searches (RQ5),
there are several recommended, well-established practices that should be followed.

In this section, we discuss several recommended practices in query reformulations and code search based on experimental evidence and the general understanding of community.

(a) Capture search queries from human users. In local code search, the title and description of a change request
have been frequently used as baseline queries assuming that they need a reformulation <cit.>. On the other hand, in the Internet-scale code search, several primary studies <cit.> use the title of a question from Stack Overflow as baseline query.
However, these queries might not always represent an actual user's queries <cit.>. Thus, although capturing search queries from secondary sources might be necessary for an exhaustive evaluation, queries should be also collected from actual users (e.g., professional developers).

(b) Choose appropriate granularity for corpus documents. Unlike regular texts (e.g., news article), source code has different granularity levels: package, file, class/module, method, and line. To search for code using IR methods, the source code of a project needs to be indexed at an appropriate granularity level. Many primary studies <cit.> choose files and classes to index their corpus as a part of bug localization. Since software bugs might be scattered through multiple methods <cit.>, a granularity level such as file or class might be appropriate. On the other hand, most studies from concept or feature location <cit.> use methods to index their corpus. They identify the relevant entity in the code that needs a change <cit.>. Choosing a small granularity such as method is likely to help the post-search analysis by developers who can leverage their cognitive abilities <cit.>. In the Internet-scale code search, developers often look for short, working code examples <cit.>, which also makes method as a feasible choice for corpus indexing <cit.>.

(c) Apply standard preprocessing to source code. Corpus documents (e.g., classes, methods) need to be preprocessed as a part of indexing. Traditionally, three preprocessing steps have been applied to source code: stop word removal, token splitting, and stemming. First, stop words are frequently used words, but they convey very little semantics. Our primary studies often use a standard list of stop words <cit.> and programming keywords <cit.> to preprocess their corpora. Several studies <cit.> remove such words that have a length of less than three characters or that occur in more than 25% of the corpus documents. Second, source code is full of structured entities such as camel-case tokens, snake-case tokens, method invocations, field invocations, and so on. Our primary studies <cit.> split these entities using appropriate regular expressions and advanced algorithms (e.g., Samurai <cit.>). Third, stemming is a widely used preprocessing step that extracts the root of any given word. Two stemming algorithms –Porter <cit.> and Snowball <cit.> – have been frequently used by the primary studies. However, the impacts of stemming have been mixed according to a few studies <cit.>. For example, stemming might have a strong impact on short queries but negligible impact on large queries <cit.>. Thus, it should be used selectively based on the type or size of queries and the search task at hand (e.g., bug localization).

(d) Choose appropriate reformulations to improve a query. Queries can be reformulated in three different ways: expansion, reduction, and replacement (Section <ref>). Given an initial query, appropriate reformulation techniques should be chosen to reformulate the query. Developers frequently use short queries to search for reusable code in the Internet <cit.>. Similarly, they choose a few keywords from a change request as a query
and execute the query to detect the code of interest (e.g., faulty code, software feature). If a query is short, it is expanded with relevant, complementary keywords <cit.>. On the other hand, long queries (e.g., preprocessed bug report) are reduced by discarding noisy, irrelevant keywords from them <cit.>. Similarly, hard, difficult queries are replaced with better alternatives <cit.>. Sometimes, a combination of multiple reformulation techniques might be required where machine learning can be employed to suggest the best reformulated query <cit.>.

(e) Use multiple search engines to evaluate the same query. To retrieve relevant source code, a query needs to be executed with a search engine. In a local code search (e.g., bug localization, concept location), primary studies frequently use two search engines – Lucene <cit.> or Indri <cit.>  – to execute their queries. Lucene is a Vector Space Model (VSM)-based search engine that combines Boolean search and BM25 algorithm <cit.>. On the other hand, Indri combines probabilistic language modelling and BM25 algorithm <cit.>. These engines can be installed against a local code repository and are easy to use. Several studies <cit.> also use Lucene to execute their queries in Internet-scale code search. A few studies make use of well-established search engines such as Google <cit.>, GitHub search <cit.>, and Stack Overflow search <cit.>. According to existing literature <cit.>, the same query can perform differently with different search configurations, which could pose a threat to the generalizability of findings. Thus, the same reformulated query needs to be evaluated with multiple search engines to mitigate the threat.

(f) Choose appropriate performance metrics for evaluation. Queries need to be evaluated using appropriate performance metrics <cit.>.
Our investigation (Fig. <ref>) suggests that several metrics such as Hit@K, MAP, MRR, QE, precision, and recall have been frequently used by the primary studies for the last 15+ years. While precision and recall are traditional metrics, Hit@K, MAP, MRR, and QE have received significant attention since 2013. We found that these metrics have been widely used in both local and Internet-scale code searches (Fig. <ref>). However, they are more prevalent in the re-enactment based evaluation than in the developer studies (Fig. <ref>). Thus, future studies can choose them for their re-enactment based evaluation. On the other hand, likert, spent time can be used for developer study-based evaluation.

(g) Choose appropriate dataset for experiments. Our analysis suggests that the use of experimental dataset (e.g., subject systems) could vary based on the context of code search (Table <ref>). For example, primary studies frequently use individual systems in local code searches (e.g., bug localization, concept location) whereas large-scale software repositories (e.g., GitHub, SourceForge, Stack Overflow) are frequently used in Internet-scale code search. The widely used subject systems can be found in Fig. <ref>, which could be a good starting point for the future work.

We also note that only a few studies evaluate their query reformulation techniques using proprietary systems (e.g., CAAF <cit.>) whereas the majority use open-source software systems, which could lead to threats to generalizability (e.g., CH3, Table <ref>) <cit.>. Thus, the future work should use appropriate systems from both development practices to achieve generalizability <cit.>.

(h) Compare with the state-of-the-art techniques. Comparison with the state-of-the-art techniques is essential to place one's work in the literature <cit.>. Table <ref> records all the techniques used by our primary studies for their validation. We see that validation targets could be different based on working context. However, a few techniques (e.g., BM25 <cit.>, Rocchio <cit.>, RSV <cit.>) were used for validation across multiple code search contexts. Fig. <ref> shows the frequently used validation targets, which could be a good starting point for the future work.

(i) Use appropriate statistical tests to answer research questions. To test any null hypothesis, statistical tests are essential <cit.>. According to our investigation, 51% (36/70) of our primary studies conduct one or more statistical tests. Table <ref> outlines the use cases of various tests whereas Figures <ref>, <ref> focus on their uses by the primary studies. From Fig. <ref>, we see that three non-parametric tests such as Wilcoxon Signed Rank (WSR), Mann-Whitney Wilcoxon (MWW), and Cliff's delta, and one parametric test namely Student's t have been frequently used by the primary studies. Non-parametric tests are used when the probability distributions of test samples are not assumed whereas parametric tests should be chosen when the samples have a normal distribution <cit.>. All these items could be a reference point for the future work for any statistical analysis.

(j) Discuss limitations and threats to validity. Threats or limitations in a research
could emerge due to various choices made by the researchers <cit.>. According to our investigation, primary studies in query reformulations suffer from eight different challenges or limitations (Table <ref>). Interestingly, several challenges (e.g., lack of generalizability (CH3), injection of noise into the original query (CH1), human bias (CH5)) are common in both local and Internet-scale code searches (Fig. <ref>). As done by earlier work, future work should also mitigate and discuss their challenges and threats, which is key to advance the current state of knowledge.

(k) Provide replication package for reuse. Among 70 primary studies, we found that 24 studies provide complete replication package including dataset and executable code, 13 studies offer only partial information (e..g, dataset only), whereas 33 studies do not provide any replication package (Appendix <ref>). Replication of any research work could be the key to its validity and long-term impacts <cit.>. Thus, making replication package publicly available is one of the recommended practices to foster extended work and reuse.

Summary of RQ$\mathbf{_6}$: Although query reformulation practices could be significantly different between local and Internet-scale code searches, there are a set of practices that have been well-accepted and recommended by the community. For example, collecting search queries from professional developers and selecting both open and closed-source systems are recommended for designing an experiment. Choosing appropriate granularity levels and preprocessing steps is essential to index the corpus. The same reformulated query should be executed against multiple search engines to ensure a fair evaluation.

On the other hand, comparison with state-of-the-art techniques and conducting statistical tests are required to validate a proposed technique. Furthermore, threats and limitations should be discussed and the replication package should be shared publicly to advance the current state of knowledge.

§ ANSWERING RQ$\MATHBF{_7}$: FUTURE RESEARCH DIRECTIONS

Our selection criteria and personal knowledge led to a set of 70 primary studies being identified since 1998 supporting code search through query reformulations.

Among them, 40 studies focus on local code searches that involve software change tasks ( bug localization, feature location, concept location). On the other hand, 30 studies reformulate queries to support the Internet-scale code search.

Despite these significant number of studies, we believe that there is still room for future work and many novel dimensions are yet to be explored. Based on our empirical and qualitative analyses, we present a list of future research directions that can
improve query reformulations and support code search as follows.

§.§ Adapt traditional keyword selection algorithms to bug reports for bug localization

Information Retrieval (IR) has been adopted by at least 20 Software Engineering tasks including bug localization <cit.>. Bug localization is a type of local code search where bug reports serve as a major source of queries.
A few recent studies <cit.> point out potential biases and limitations in this local code search. According to them, IR-based localization is only good when the bug reports contain localization hints ( program entity names). However, these empirical studies use the whole texts of a bug report as a query, which are often noisy, and overlook the potential of optimal search queries. <cit.> recently conduct a large-scale empirical study using Genetic Algorithms and present  positive evidence for both bug reports and IR-based localization. They suggest that the bug reports contain sufficient keywords to return the faulty source code at high ranks using Information Retrieval.

In particular, the IR-based localization can succeed 67%–88% of the times even if their bug reports do not contain any localization hints <cit.>. Similar finding has been reported by a recent work of <cit.>.
Thus, the real challenge becomes the extraction of right search keywords from a bug report regardless of its localization hints.

Future work. Appropriate term weighting algorithms can play a major role in selecting the search keywords.
To date, existing
term weighting studies exploit each term's frequency <cit.>, statistical relations and syntactic dependencies <cit.>, or occurrence probability <cit.> as a proxy to its relative importance (a.k.a., term weight).

However, many of these algorithms were designed for regular texts in the Information Retrieval domain and

might not be well-configured for bug reports containing a mix of regular texts and structured entities ( stack traces, test cases, program elements, commit diffs). Future studies can focus on better adapting the IR algorithms to these structured entities. For example, rather than treating these entities as regular texts, they can be represented as graphs and thus can be better leveraged with graph-based algorithms (e.g., PageRank algorithm <cit.>) for keyword selection.

§.§ Design practical GA-based query reformulations for IR-based bug localization

<cit.> demonstrate that Genetic Algorithms (GA) are capable of generating optimal or near-optimal queries from a bug report (that can localize faulty code) when the ground truth information is provided. Unfortunately, in practice, the ground truth might not be known when a search query is constructed to find a bug. Thus, the ground truth should not be a part of fitness calculation when designing a GA-based query reformulation technique. In other words, we need a fitness function that can identify the better one from two different search queries without needing their ground truth, which is a major challenge.

Future work. Several studies <cit.> make use of query difficulty metrics ( specificity, coherency <cit.>) and machine learning algorithms to predict a query's success against the search engines.

However, these metrics have non-linear relationships with a query's real performance according to recent studies <cit.>.

One possible way to address this is to formulate the query construction as a multi-objective optimization problem, as demonstrated by <cit.>. They design a complex fitness function that attempts to optimize 15 objectives while reformulating a bug report into an optimal query for IR-based bug localization.

Future work can design one or more fitness functions that can be used in practice for GA-based keyword selection from bug reports.
For example, difficulty of a query could be a part of its fitness calculation where the easy queries will be prioritized over the difficult ones during evolutionary search.

§.§ Leverage word embeddings in query reformulation & code search

Several primary studies <cit.> use English language thesauri such as WordNet <cit.> to expand their query with synonyms or semantically similar words.
However, <cit.> suggest that the same word can have two different meanings
when used in source code and when used in regular texts. Since WordNet is based on regular English texts, its suggested keywords might not be appropriate for source code search.

Thus, several studies <cit.> construct software-specific thesauri from the mining of software repositories (e.g., open-source projects, programming Q&A site).

However, construction and update of these thesauri are costly and their effectiveness in the query reformulation is not well tested.
Recently, several studies <cit.> make use of word embedding technology
to determine semantic similarity or relevance between any two software specific words.
A few of them <cit.> use embeddings to reformulate queries for code search and report positive findings.
However, the true potential of word embeddings in query reformulations might not have been explored yet.

Future work. Word embedding technology approximates the meaning of a word using a high dimensional numeric vector <cit.>. The vector places each word as a single coordinate within a high dimensional semantic space (a.k.a., semantic hyperspace).  Such a geometric approximation of word semantics can be leveraged for various tasks in both local and Internet-scale code searches. For example, by placing all keywords from a document (e.g., bug report, feature request, source code) in the semantic hyperspace, important keywords could be detected using various geometric theories (e.g., Hopkins statistics <cit.>). By visualizing the semantic hyperspace using appropriate tools (e.g., t-SNE <cit.>), we could make a better sense of (a) relevance between a query and its target code, or (b) the difference between observed behaviour and expected behaviour of a reported bug.

§.§ Complement term weighting algorithms with contextual information

Determining importance of a term within a body of texts ( bug report, source document) has been

as a major challenge <cit.>. TF-IDF <cit.> is a term weighting algorithm that has been extensively used both in Information Retrieval and in Software Engineering. The algorithm determines a term's importance without considering its contextual information ( surrounding terms). However, a term's semantics are often determined by its contexts <cit.>. Besides, several primary studies demonstrate the benefits of incorporating context in their term weighting algorithms. To date, several contextual items such as spatial code proximity <cit.>, positional relevance <cit.>, term co-occurrences <cit.>, syntactic dependencies <cit.>, time-awareness <cit.>, and structural awareness <cit.> have been incorporated in term weighting. However, their use was ad hoc and their roles in query reformulations were not investigated exhaustively.

Future work. While context can deliver useful information in term weighting, it should be used carefully to avoid noise.
Future work can investigate how multiple contexts can be incorporated in the term weight calculation.

Genetic Algorithms can be used to optimize the relative weights of multiple contexts during their combination <cit.>. Machine learning can also be used to determine complex, non-linear relationships between these contexts and a term's importance <cit.>.

§.§ Leverage Stack Overflow in query reformulation for local code search

Software developers frequently use natural language queries to find relevant code on the web <cit.>.
However, their queries are often short and thus fail to

capture their information need <cit.>.

Existing findings <cit.> suggest that inclusion of relevant API classes or methods in a query can improve its chance of retrieving the relevant code.

A few studies <cit.> analyze the Q&A threads from Stack Overflow
to expand a natural language query and to support the Internet-scale code search.
For example, <cit.> analyze co-occurrences of query keywords and API classes in question title and accepted answer respectively

and expand a query with frequently co-occurred API classes. However, the true potential of Stack Overflow in query reformulation might not have been explored yet.

Future work. Several studies <cit.> perform machine translation where they translate a natural language query into relevant API classes or methods. However, they make use of proprietary search logs (e.g., Bing logs) that might not be publicly available for reuse. Interestingly, Stack Overflow can be leveraged to generate such logs where keywords and API classes can be extracted from question titles and accepted answers respectively. Given the size of Stack Overflow, millions of logs can be captured at low costs. Then the machine translation techniques based on these logs can be used to suggest (a) relevant API classes for a new feature request <cit.> or (b) potentially buggy API classes for a bug report.

§.§ Minimize query worsening during reformulation

Query reformulations have both benefits and costs. Existing studies suggest that automated query reformulation
might improve the performance of a query up to 20% during code search <cit.>. However, several studies <cit.> also question the idea of a complete automation in reformulation.

Without human intervention, automated reformulations might add noisy keywords that could drift a query away from its original topic <cit.>. Thus, we need a mechanism to

maximize the benefits and minimize the costs of automated query reformulations.

Future work. Several studies <cit.> use query difficulty metrics and machine learning to automatically identify the difficult queries and then reformulate them.

However, unless the models are 100% accurate, there will be always false-positive results. Thus, the risk of query worsening due to reformulations is still prevalent. According to <cit.>, human developers might perform well in removing irrelevant terms from a search query due to their cognitive abilities. On the contrary, automated techniques perform well in adding the relevant terms to a query <cit.>. Thus, query reformulation task can be considered as a multi-step changes to a query (e.g., query expansion, query reduction) where human and machine capabilities are combined together.

§.§ Apply pseudo-relevance feedback (PRF) to Internet-scale code search

Relevance feedback is an important mechanism to reformulate a search query <cit.>.
However, capturing developers' feedback on queries by analyzing their results could be costly and sometimes impractical.
Many primary studies <cit.> thus use pseudo-relevance feedback (PRF) (Section <ref>) as a feasible choice to reformulate their queries. PRF has been reported to improve the queries during code search <cit.>. However, the majority of studies except a few <cit.> focus on local code searches such as concept location, bug localization, and feature location.

Future work. In the Internet-scale code search, the results are retrieved from thousands of open source projects, which could be noisier and less homogeneous than that from local code search. PRF naively assumes the top few results retrieved by a query as relevant <cit.>, which might not apply to the Internet-scale search. Thus, future work can adapt the PRF mechanism to mitigate the noise and to support the query reformulations in Internet-scale code search.

§.§ Leverage PageRank algorithm in term weighting and source code retrieval

adapt PageRank <cit.>, a popular algorithm from Information Retrieval,
to reformulate queries in several Software Engineering tasks

such as bug localization <cit.>, concept location <cit.> and Internet-scale code search <cit.>. Other studies <cit.> make use of PageRank algorithm to rank code snippets during their code search. However, the true potential of PageRank algorithm in query reformulation and code search might not have been explored exhaustively.

Future work. PageRank operates on a graph with nodes and edges, implements a notion of voting, and then identifies the most important nodes from the graph using a recursive computation <cit.>. Source code
has an unrestricted vocabulary <cit.> and is full of structures, entities, and
dependencies among them (e.g., control dependency, data dependency), which can be represented as a graph <cit.>.
Thus, unlike TF-IDF, PageRank is a natural choice for analyzing the source code. While there have been a few attempts using static relationships <cit.>, future work can use PageRank algorithm to exploit the code structures to support query reformulations and code search. For example, PageRank can be used to (a) detect meaningful terms from PRF documents for query expansion and (b) shortlist terms from source code for corpus indexing.

§.§ Standardize query reformulation in IR-based bug localization

<cit.> first use Vector Space Model (VSM) to recover traceability links.
<cit.> later refine VSM as rVSM and incorporate past bug reports in the IR-based bug localization. <cit.> make use of structures both from bug reports and from source code documents to localize the bugs. <cit.> boost up the bug-proneness score of a source document based on the stack traces found in a bug report. <cit.> and <cit.> incorporate version control history in the IR-based bug localization. Finally, <cit.> incorporate five major items – past bug reports, structures, stack traces, version history, and author history – from the literature, and outperform the earlier approaches on IR-based localization. Thus, existing literature often adopt an incremental approach and  include more and more external artifacts in their approaches. Although these artifacts have positive impact upon the localization performance, their inclusion makes the proposed approaches less scalable and less usable Besides, these approaches become subject to external dependencies, which is not an ideal choice from the design point of view.
Such limitations might partially explain the reluctance of software practitioners in adopting the IR-based bug localization techniques in their workflow <cit.>.

Future work. Several approaches equipped with query reformulation <cit.> make an effective use of primary resources available to practitioners (e.g., bug report and source code), and
localize the bugs with a competitive accuracy and at low cost. Given that Information Retrieval has been adopted by at least 20 software engineering tasks, query reformulation has the potential to significantly improve them.

For example, several tasks such as bug localization <cit.>, duplicate bug report detection <cit.>, bug triaging and bug report summarization can be benefited from query reformulations.

Summary of RQ$\mathbf{_7}$: Despite significant research, there is still room for further work in automated query reformulations and code searches (e.g., bug localization, concept location, feature location, Internet-scale code search).

Existing findings suggest that up to 88% of bug reports contain appropriate keywords that make good queries and can find the bugs using Information Retrieval methods.

Unfortunately, many existing techniques are not effective enough to identify these keywords. The selection of keywords could be further improved by adding more contexts (e.g., time-awareness), designing an appropriate fitness function (for GA-based solutions), leveraging the structures from source code, or using neural language modeling (e.g., word embeddings). The adverse effects of automated query reformulations could be mitigated by combining both human cognitive power and tools' strengths. Large software repositories (e.g., GitHub) and programming Q&A site (e.g., Stack Overflow) have a lot to offer to the research of query reformulations and code search. They store invaluable technical knowledge curated by a large technical crowd, which can be leveraged to support not only code searches but also other software development and maintenance activities.

§ THREATS TO VALIDITY

We identify several threats to the validity of our findings in this work. While we did our best to mitigate the majority of these threats, a few of them can be addressed in the future surveys on automated query reformulations. We classify our identified threats into several standard categories and discuss them as follows.

§.§ Threats to internal validity

These threats are associated with experimental or internal errors and human biases <cit.>. Selection of primary studies and their classification into certain categories could be a source of such threats. However, we take careful steps to mitigate them.

First, we choose 70 primary studies using a well established, systematic approach (Section <ref>)
that involves (a) generating keywords from research questions using PIO criterion, (b)
extensive searching against 11 widely used publication databases (e.g., IEEE Xplore, ACM Digital library), and (c) multi-level filtration based on specific criteria. We first collect a large set of 2,970 studies that match our search keywords, and then carefully select 70 studies from them using six levels of noise filtration. Despite these careful, time-consuming steps, relevant studies might be omitted due to sub-optimal queries or the limitations of

search engines provided by the publication databases (Section <ref>).
To mitigate this threat, we also added a few studies that were relevant (according to our prior work experience) but were accidentally missed by the study selection process. However, we still cannot rule out the possibility of missing a few relevant studies on the topic of interest.

Second, we use the Grounded Theory approach (Section <ref>)
to classify the primary studies based on their adopted methodologies, algorithms (Section <ref>), and experienced challenges (Section <ref>).

Grounded Theory has been a popular choice for qualitative analysis in
Software Engineering  research for decades <cit.>. Thus, the threats identified above might be mitigated.

Third, local code search and Internet-scale code search differ in their working context and end goals. While former targets software change tasks ( bug localization, feature location), the latter one focuses on code reuse or general-purpose programming tasks. One might argue about the design of our survey since the query reformulation studies from both searches were captured. However, according to our investigation, query reformulation approaches from these two searches have a significant overlap in their reformulation techniques (Table <ref>, Fig. <ref>), evaluation methods (Figures <ref>, <ref>), and even strengths or limitations (Fig. <ref>).
Thus, analysis restricted to only either local code searches or Internet-scale code search misses a significant number of strongly related studies from the literature. Besides, collecting both set of studies allows us to
compare and contrast between them, which has led us to meaningful insights (Section <ref>) and novel ideas for future work (Section <ref>).

There also exists a large body of work on query reformulations in the Information Retrieval domain <cit.>. Many of our primary studies adapt their ideas from them. However, since we deal with Software Engineering problems such as code search, those studies from IR literature were not included in our survey.

Fourth, our systematic review is restricted to only such studies that deliver either automated or semi-automated
tool supports to reformulate queries for code search.

That is, it does not include empirical studies, developer surveys and sometimes even the simple incremental works. Such a choice was made to construct a homogeneous set of primary studies that solve a single research problem, i.e., automated query reformulation to support code search.

§.§ Threats to external validity

These threats relate to the generalizability of any reported findings <cit.>. Our primary studies are carefully chosen from a large population of 2,970 results that were retrieved from 11 popular publication databases (Fig. <ref>). Thus, these primary studies are likely to represent the core studies on query reformulation supporting code search in the literature.

Consequently, our findings and insights derived from these studies could also generalize for the remaining, accidentally missed studies from the literature. Furthermore, we leverage the Grounded Theory approach for our qualitative analysis, which has been a standard practice for years <cit.>.
Thus, the threats to external validity might be mitigated.

§.§ Threats to construct validity

These threats relate to the appropriateness of evaluation methodology used in a study <cit.>. Since our selection of primary studies involves certain quality control mechanism, such threats may arise. We select 70 studies through a well-established approach of extensive database search and well-documented filtration criteria (Fig. <ref>). Furthermore, we control the quality of these studies by asking 10 common questions about their quality (Section <ref>). While the majority of these studies pass the quality check, the remaining few do not meet certain quality standards ( lack of rigorous evaluation and validation). However, we retain them in our analysis due to their strong relevance to our survey topic– query reformulation in source code search. It also should be noted that three studies <cit.> on traceability link recovery were also included in our survey due to their solid contributions in query reformulation algorithms.

§.§ Threats to conclusion validity

These threats arise when the relationships between two variables are not backed up by solid evidence or reliable data <cit.>. In our qualitative analysis (Section <ref>), we compare and contrast between local code searches and Internet-scale code search and make several statements about their characteristics, methodologies, strengths, and limitations.

We also outline several future research directions based on our analyses.
Our claims and suggestions were backed up not only by substantial analytical data (RQ$_1$, RQ$_3$) but also by empirical evidence (RQ$_4$).

Thus, threats to the conclusion validity of our systematic survey might also be mitigated.

Comparison between existing surveys/reviews and our systematic reviews

Survey     Topic     FQR     Period     #Studies     SLR     Dimension

<cit.>     Information retrieval     Yes     1993-2009     27     No     Performance limitations

<cit.>     Feature location     No      1992-2011     89     Yes     Type of analysis, user input, data sources, output, programming language support, evaluation methods, and software systems

<cit.>     Feature location     No     1995–2010     24     No     Underlying technology, program representation, input, and user interactions

<cit.>     Fault localization     No     1977–2014     331     No     Underlying technology, subject programs, evaluation metrics, and critical aspects

<cit.>     Internet-scale code search     No     2002–2020     81     Yes      Code search technique, evaluation, and publication trends

Our systematic review     Concept location, feature location, bug localization, and Internet-scale code search     Yes     2004–2021     70     Yes     Underlying methodology, evaluation design, publication trends, limitations, and challenges

FQR=Focus on query reformulation, SLR=Use of systematic literature review

§ RELATED WORK

Source code search has two major steps – (a) query construction and (b) retrieval of relevant code <cit.>.
There have been several surveys  <cit.> that examine the evidence of existing researches on local code searches ( feature location, bug localization).

They mostly focus on the studies that deal with the second step of code search, i.e., retrieval of relevant code.
Unfortunately, to the best of our knowledge, there exists no systematic survey
that deals with query reformulation approaches adopted in source code search.
The closely related work is a similar survey <cit.> from the Information Retrieval (IR) domain. Although IR-based methods have been widely adopted by at least 20 Software Engineering tasks <cit.>, they experience unique challenges (Section <ref>), which warrants further investigation <cit.>. Our survey identifies the existing work on query reformulations supporting Software Engineering tasks, categorize them based on their underlying methodologies or limitations, and then discusses their
open issues.

<cit.> first provide a comprehensive survey of 89 studies on feature location techniques (FLT). They analyze each of these studies using seven different dimensions : (a) type of analysis, (b) user input, (c) data sources, (d) output, (e) programming language support, (f) evaluation method, and (g) software systems used. Based on an analysis, they also provide a detailed taxonomy on the FLT studies. <cit.> perform a similar survey using 24 studies on feature location techniques. They provide necessary guidelines for software practitioners to choose the right feature location technique for their tasks. <cit.> analyze 331 studies on software fault localization and provide a detailed classification of their studies. Unlike the above work, our survey deals with not only feature location or bug localization but also other types of code searches such as concept location and Internet-scale code search.
More specifically, we deal with a cross-cutting concept namely query reformulation that concerns each of these code search activities.

Our survey also analyzes 70 primary studies on query reformulations and categorize them based on several aspects such as query reformulation methodology (RQ$_1$), evaluation method (RQ$_2$), and their experienced challenges or limitations (RQ$_3$).

<cit.> present a survey on bug report analysis where they classify the existing work on bug-report optimization ( bug severity prediction), bug-report triage ( bug report assignment), and bug-fixing ( fixing time prediction). Similarly, we also deal with the primary studies that perform bug report analysis. However, our focus was to capture the studies that accept a bug report as an input, construct appropriate queries from the input, and then localize the bug using these queries.

A few other studies <cit.> also investigate the existing work on query reformulations supporting code searches and other Software Engineering tasks. However, their analysis was
limited and restricted to either local <cit.> or Internet-scale code search <cit.>.

Many of them also perform traditional literature review rather than a systematic review <cit.>, which

might not guarantee that all eligible primary studies were captured from literature.

On the other hand, we conduct

a systematic literature review that was guided by a set of research questions and well-crafted search queries (e.g., PIO criterion). The quality of our primary studies was also controlled by clear exclusion/inclusion criteria and they were collected from 11 popular publication databases. Thus, our goal was to establish a comprehensive understanding of the existing research on query reformulations targeting code search. We also

perform qualitative analysis using the Grounded Theory approach <cit.>, a widely used qualitative technique, to answer several of our research questions. No earlier survey addresses the same topic in this scale, which makes our work novel. Table <ref> also demonstrates how our survey fills in the gap of literature by focusing on query reformulations in the context of source code searches.

§ CONCLUSION

Software maintenance can cost up to 80% of total budget in modern software development. Searching for the code of interest (e.g., buggy code, reusable code example) is an integral part of software maintenance.
Developers often choose a few important keywords capturing their information need and attempt to find the code of interest either from a local software codebase or an Internet-scale code repository ( GitHub). Unfortunately, existing evidence suggests that the developers often perform poorly in choosing their keywords. As a result, they spend a significant amount of their time manually reformulating queries and analyzing the corresponding search results. Towards this goal, a number of studies attempt to support the developers through automated query reformulations. In this article, we conduct a systematic literature review using 70 primary studies on automated query reformulations intended for source code search. We summarize our quantitative and qualitative findings below and answer our research questions briefly as follows.

* RQ$_1$: Existing approaches for query reformulation adopt eight major methodologies
including term weighting, relevance feedback, semantic relations, thesaurus lookup, and data mining to reformulate their queries and to support code searches in various working contexts.

* RQ$_2$: Existing studies evaluate their approaches through re-enactment, developer surveys, or using both. They use a total of 30 performance metrics including the popular ones (e.g., Hit@K, MAP).
However, developer involvement is generally low, and the number of queries used to evaluate queries in the Internet-scale code search is often small. About 30% of the previous approaches were selected for comparison by the later approaches.

* RQ$_3$: Existing query reformulation approaches suffer from eight major challenges and limitations including
noisy keywords in their queries, vocabulary mismatch problem, lack of generalizability, human-induced biases, weak evaluation, and other prevailing issues that might prevent them from adoption by the software practitioners.

* RQ$_4$: Our selection criteria and personal knowledge led to a set of 70 primary studies from the last 15+ years

where 80% of them were done in the last nine years. These studies appeared in the top Software Engineering venues such as ICSE, ASE, ICSME, TSE, TOSEM, and EMSE. Given the increasing size and complexity of modern software systems, the research
topics such as automated query reformulation and code search are likely to gain increased attention in the coming years.

* RQ$_5$: About 58% of our primary studies perform query reformulations to support local code searches (e.g., concept location, bug localization, feature location) whereas the remaining 42% focus on Internet-scale code search.

The primary studies from these two searches differ in their adopted methodologies, reformulation types, evaluation methods, and human participation. However, both groups of studies

suffer from a common set of issues such as a lack of generalizability, weak evaluation, and noisy queries.

* RQ$_6$: Despite noticeable differences between local and Internet-scale code searches, there are a set of common, well-established practices in query reformulations targeting code search. These practices include selection of search queries from professional developers, selection of subject systems from both open and closed-source domains, execution of a query against multiple search engines, and comparison of a proposed technique with state-of-the-art techniques through replication.

* RQ$_7$: Constructing the right queries for code search has been a challenge for decades. Despite significant research, there is still room for further investigation. Future work can focus on

adding more contexts (e.g., time-awareness) to their keyword selection algorithms, designing an appropriate fitness function for GA-based solutions, leveraging the structures from source code, or using neural language modeling (e.g., word embeddings) to deliver better queries for the code search.

§ ACKNOWLEDGEMENT

This research was supported by Tenure-track startup grant, Dalhousie University, International Dean's Scholarship from University of Saskatchewan (2014–2017), Saskatchewan Innovation & Opportunity Scholarship (2017–2018), and the Natural Sciences and Engineering Research Council of Canada (NSERC).

§ SELECTED PRIMARY STUDIES

Table <ref> shows the 70 primary studies selected for our systematic literature review.

Selected Primary Studies for Systematic Literature Review

ID     Primary Study     Description

– Continued from previous page

ID     Primary Study     Description

3rContinued on next page

S1     <cit.>     Query reduction using noun term selection and time-aware term weighting for concept location

S2     <cit.>     Query expansion using keyword co-occurrences across the past queries from code search logs

S3     <cit.>     Ad hoc keyword selection from given test cases for semantic code search

S4     <cit.>     Query expansion using modified Rocchio's method and the re-ranking of results for improved code search

S5     <cit.>     Query reduction by extracting nouns, verbs, adjectives and type information from a free-form query for code search

S6     <cit.>     Query reformulation using spatial code proximity for Information Retrieval-based bug localization

S7     <cit.>     Query expansion using structured code entities from the relevant Q & A threads of Stack Overflow

S8     <cit.>     Automatic performance prediction of a given search query using query difficulty analysis and machine learning without executing the query

S9     <cit.>     Query reformulation for feature location using Rocchio, RSV and Dice methods with software model families

S10     <cit.>     Best query reformulation suggestion using Rocchio, RSV, Dice methods, query difficulty analysis and machine learning

S11     <cit.>     Query reduction using advanced heuristics and logistic regression

S12     <cit.>     Semi-automated reformulation of NL queries with query keyword contexts and NL phrase extraction from method and field signatures

S13     <cit.>     Query reformulation with NL thesaurus ( WordNet) in the context of interface-driven and keyword-based code searches

S14     <cit.>      Query reformulation using frequent identifier names from result files and other contextual information

S15     <cit.>     Query reformulation with relevant APIs by matching query texts and corresponding API documentations

S16     <cit.>     Query expansion by using tag co-occurrences in the same and duplicate questions of Stack Overflow

S17     <cit.>     Query reformulation using pre-search ( identifier and Verb-DO completion, term co-occurrences) and post-search recommendations ( synonym suggestion, spelling corrections)

S18     <cit.>     Query reformulation using ontology models for web service discovery

S19     <cit.>      Query reformulation using crowd generated knowledge and large-scale data analytics derived from Stack Overflow corpus

S20     <cit.>     Query reformulation using semantically related API classes for the improved Internet-scale code search

S21     <cit.>     Query reformulation using spatial code proximity, term ordering, Markov Random Field and query conditioning

S22     <cit.>     Query reformulation using term weighting method, CodeRank, query difficulty analysis, and machine learning

S23     <cit.>     Query reformulation of NL query using textual similarity with product requirements for feature location

S24     <cit.>      Semi-automated query reformulation using structure, intent and dependency facets of search results

S25     <cit.>     Query reformulation using bug report quality dynamics and graph-based term weighting algorithms

S26     <cit.>     Query reformulation using relevant API entities from RecoDec, conceptual knowledge, TF-IDF, and advanced heuristics

S27     <cit.>     Query reformulation using semantically similar word pairs mined from comment-code contexts of source documents

S28     <cit.>     Query reformulation using semantically similar word pairs mined from leading comments and method signatures across projects

S29     <cit.>     Query reduction using gradual term removal with the help of query effectiveness

S30     <cit.>     Query reformulation using explicit relevance feedback from the developers and Rocchio's expansion method

S31     <cit.>     Query reformulation with AST generation and XPath representation of example code snippet

S32     <cit.>     Best query prediction using query difficulty analysis and machine learning for concept location and traceability recovery

S33     <cit.>      Query reduction by subtree generation from the AST of input code

S34     <cit.>     Query expansion using Rocchio's method and pseudo-relevance feedback from Stack Overflow

S35     <cit.>      Query expansion based on intent mining from code change history and machine learning

S36     <cit.>     Query expansion using POS tagging and synonym addition from WordNet for improved concept location

S37     <cit.>     Query reformulation by term replacement with synonymous tags and by term expansion with related tags from Stack Overflow

S38     <cit.>     Query reformulation using frequently co-occurred keywords from source code and Stack Overflow threads

S39     <cit.>      Query reformulation using keyword-API co-occurrences in the Q & A threads of Stack Overflow

S40     <cit.>     Query reduction by selecting  important keywords using graph-based term weighting namely PageRank

S41     <cit.>     Query reformulation using co-occurrence probability of keyword and API classes in the search engine logs

S42     <cit.>     Query expansion using synonyms/antonyms from three thesauri– WordNet, <cit.>, and type thesaurus

S43     <cit.>     Query reformulation using Verb-DO extraction and their expansion using stemmed words, synonyms and similar usages

S44     <cit.>     Query reduction using observed behaviour identification, Grounded Theory and natural language discourse analysis

S45     <cit.>     Query reformulation using NL-SCL mapping between changeset and IDE interaction history, and term weighting

S46     <cit.>     Query reformulation with API method signatures using sentence parse tree extraction, uni-gram, and probabilistic context free grammar (PCFG)

S47     <cit.>     Query reduction using POS tagging, selection of nouns and verbs and source code ontology

S48     <cit.>     Query reformulation using semantic web based ontology models, Ginseng, and SPARQL

S49     <cit.>     Query/corpus reformulation using the mapping between programming concepts and their syntactic forms from Stack Overflow Q&A threads

S50     <cit.>     Query reformulation using similar terms from the source code with Latent Semantic Indexing (LSI)

S51     <cit.>     Query reformulation using web data mining and term weighting methods–domain term frequency, domain specificity, and concept generality

S52     <cit.>     Query reformulation using trace query transformation rules generated by FP-growth algorithm

S53     <cit.>     Query reformulation using ontology models across multiple repositories and candidate navigation path ranking

S54    <cit.>     Query/corpus reformulation by weight boosting ( repeating) of structured entities from various contexts in the source code

S55     <cit.>     Query/corpus expansion by appending corresponding API documentations against method invocations.

S56     <cit.>     Query suggestion by mining program dependency graphs from a given code segment

S57     <cit.>     Query reformulation by expanding methods with neighboring methods based on semantic similarity, temporal proximity, and call dependency

S58     <cit.>     Query expansion using co-occurring API methods from the underlying corpus using encoder-decoder transformation and conditional probability analysis

S59     <cit.>     Query expansion using experts' descriptions of a feature, their confidence level, and Rocchio's method

S60     <cit.>     Query expansion using first and second order word co-occurrences, and object-oriented relations among the words

S61     <cit.>     Query reformulation by adding attachment contents and by reducing noisy keywords from a bug report

S62     <cit.>     Query reformulation using Genetic algorithm and multi-objective optimization (e.g., query difficulty, OB, S2E, EB)

S63     <cit.>     Query expansion by deep learning change sequences and predicting the potentially changed terms for improved code search

S64     <cit.>     Query reformulation for feature location using Rocchio's method, relevance feedback, and expert's descriptions of a feature as the feedback documents

S65     <cit.>     Query reformulation using old code examples as relevance feedback and then extracting expansion keywords (i.e., intent) from the change history of those code examples

S66     <cit.>     Query reformulation by learning to predict reformulated query from search logs of Stack Overflow using sequence-to-sequence learning and attention mechanism

S67     <cit.>     Query selection for bug localization using Genetic algorithm and information need analysis as a fitness function

S68     <cit.>     Query reformulation for bug localization using Genetic algorithm and query effectiveness as a fitness function

S69     <cit.>     Query reformulation for feature location using Genetic algorithm using expansion, reduction, and selection as a mutation operation, and textual similarity as a fitness function

S70     <cit.>     Query expansion for code example search using semantic similarity analysis, API suggestion, and term weighting (e.g., TF-IDF)

§ QUALITY ASSESSMENT OF PRIMARY STUDIES

Table <ref> shows answers for the questions outlined in Section <ref>. We represent our responses as follows– 360 as “Yes", 0 as “No", and 270 or 180 as “Somewhat Yes".

Quality Assessment of Primary Studies

ID     Primary Study     Q1     Q2     Q3     Q4     Q5     Q6     Q7     Q8     Q9     Q10

– Continued from previous page

ID    Primary Study     Q1     Q2     Q3     Q4     Q5     Q6     Q7     Q8     Q9     Q10

11rContinued on next page

S1     <cit.>     360     360     360      360     360       360     360     360     360      180

S2     <cit.>     360     360      360     360     360     360     360     0     180     360

S3     <cit.>     360     360     360     180     360     360     360     360     360     360

S4    <cit.>     360     360     360      360      360     360     360     360     360     0

S5     <cit.>     360     360      360     360     180     360      180     0     180     0

S6     <cit.>     360     360     360      360      360       360     360     360     360     0

S7    <cit.>     360      360     360      180     360     360      360     360     0     360

S8     <cit.>     360     360     360      0     360     360     360     360     360      360

S9     <cit.>     360     360     360     360      360     360     360     360     0     0

S10     <cit.>     360     360       360     360      360      360     360     360     180      180

S11     <cit.>     360     360      360       360     180     360     360     0     360     360

S12    <cit.>     360     360     360     360      360     360     360     360     360      180

S13    <cit.>     360     360     360     360      360     360     360     360     360       360

S14    <cit.>     360     360      360     360     360     360     360     360     360     360

S15    <cit.>     360     360     360      360     360     360     360     360     360     0

S16    <cit.>     360     360     360     360     360     360     360     360     360     0

S17    <cit.>     360     360     360     360     360     360     360     360     360     360

S18     <cit.>     360     360      360     180     180     0     180     0     360      0

S19    <cit.>     360    360     360     360     360      360    360    360    360     360

S20     <cit.>     360     360     360      360      360      360     360     360     360       180

S21    <cit.>     360     360     360     360     360      360     360     360      360     180

S22    <cit.>     360     360     360    360     360      360    360     360     360      360

S23    <cit.>     360     360      360       360       270     360     360     360     360      0

S24    <cit.>     360     360     360     90     360     360     360     360     360     180

S25    <cit.>     360     360     360     360     360      360    360   360     360     360

S26   <cit.>     360     360      360      360      360       360    360     360     360      0

S27   <cit.>     360     360     360      270     360     360     360     360     360     360

S28   <cit.>     360     360     360     180     360     360     360     360     360     0

S29   <cit.>     360     360     0     180     180      360     360     0     180     360

S30   <cit.>     360     360     360     360      360     360     360     360     360     180

S31   <cit.>     360     360     360      360       360      360     360     360      360     0

S32   <cit.>     360     360      360     0      360      360     360      360     360     0

S33   <cit.>     360     360      360      360      360       360     360     0      360      0

S34   <cit.>     360     360     360      360     360     360     360     360     360      360

S35   <cit.>     360     360     360      360      360      360     360     360     360     0

S36   <cit.>     360     360       360      360     270     360     360     0     360     0

S37   <cit.>     360      360     360      360      360     360     360     360     360     0

S38   <cit.>     360     360      360      360      180     360     360     0     360     180

S39   <cit.>     360     360     360     360     360      360     360     360     360      360

S40   <cit.>     360     360      360     360      360      360     360     360     360      360

S41   <cit.>     360     360      360      360      360      360     360     0      360      0

S42   <cit.>     360     360      360      360      360       360     360     360     360       180

S43   <cit.>     360     360     360      360      360       360     360     360     360      180

S44   <cit.>     360     360      360      360      360      360     360     360     360     360

S45   <cit.>     360     360      360      360      180     360     360     0     360       360

S46   <cit.>     360     360      360      180      180      360     360     0     360      0

S47   <cit.>     360     360      360     360     180      360     360     0      360      0

S48   <cit.>     360     360      360     360     360      360     360     0      360       270

S49   <cit.>     360     360      360      360      360       360     360     360     360      360

S50   <cit.>     360     360      360      180      360       360     360     0     360     0

S51   <cit.>     360     360     360      360      360       360     360     0     360     360

S52   <cit.>     360      360      360      360       180     360     360     360     360      0

S53   <cit.>     360     360       360     360      360     360     360     360      360      0

S54   <cit.>     360      360     360      0     360      360     360     360      360      180

S55   <cit.>     360     360      360     180     360      360     360    0      360       0

S56   <cit.>     360     360     360     360     360     360     360     360     360     0

S57   <cit.>     360     360     360     360     360     360     360     360     360     0

S58   <cit.>     360     360     360     360      360     360     360      0     360     0

S59   <cit.>     360     360     360     360     360     360     360     360     90     0

S60   <cit.>      360      360     360     360     360     360     360     360     360     0

S61   <cit.>     360     360     360     360     360     360     360     0     180     0

S62   <cit.>     360     360     180     270     180     360     180     0     270     0

S63   <cit.>     360     360     360     360     360      360     360     360     0     0

S64   <cit.>     360     360     360     360     180     360     360     360     360     0

S65   <cit.>     360     360     360     360     180     270     360     180     360     0

S66   <cit.>     360     360     360     360     360     360     360     0     360     360

S67   <cit.>     360     360     360     180     180     360     360     0     360     180

S68   <cit.>     360     360     360     180     180     360     360     360     360     360

S69   <cit.>     360     360     360     360     360     360     360     360     360     360

S70   <cit.>     360     360     360     360     360     360     360     360     360     360

§ EVALUATION METRICS

Table <ref> defines the performance metrics frequently used by the primary studies.

Metrics for performance evaluation

Metric     Overview     Formula

Hit@K     Percentage of all search queries ($Q$) for each of which at least one relevant result is returned within the top K results.     Hit@K = $\frac{\sum_{q\in Q}rel(q, ~K)}{|Q|}$

QE     Query Effectiveness (QE) refers to the rank of the first relevant result returned by a search query.

MRR     Reciprocal Rank is the multiplicative inverse of the rank of the first relevant result returned by a search query ($q \in Q$).
Mean Reciprocal Rank (MRR) averages the reciprocal ranks for all search queries ($Q$).     MRR(Q) = $\frac{1}{|Q|}\sum_{q\in Q} \frac{1}{rank(q)}$

Precision     Precision refers to the fraction of retrieved results that are relevant. Here TP means true positive and FP means false positives.     P = $\frac{TP}{TP+FP}$

Recall     Recall refers to the fraction of ground truth results that are retrieved. Here, FN means false negatives.     R = $\frac{TP}{TP+FN}$

F1-score     F1-score calculates the harmonized mean of precision and recall measures.     F1-score = $\frac{2\times P\times R}{P+R}$

MAP     Precision@k calculates precision at every occurrence of a relevant result within a ranked list. Average Precision (AP) averages over all Precision@k measures for a given query $q$.     AP(q) = $\frac{\sum_{k=1}^{K}P_k}{|RR|}$

Mean Average Precision (MAP) calculates the mean of AP for all queries ($Q$). Here $P_k$ refers to precision@k and  $RR$ means the relevant results retrieved by a query.     MAP(Q) = $\frac{\sum_{q\in Q}AP(q)}{|Q|}$

NDCG     Discounted Cumulative Gain (DCG) determines the quality of a ranking technique by rewarding when the highly relevant results are found at the top positions and by penalizing when the highly relevant results are found at low positions of ranked list.
DCG(q) = $\sum_{k=1}^{K}\frac{grel_k}{log_2(k+1)}$

Normalized Discounted Cumulative Gain (NDCG) calculates to the ratio between DCG and Ideal Discounted Cumulative Gain (IDCG) which is based on ground truth information.     NDCG(q) = $\frac{DCG(q)}{IDCG(q)}$

Here $grel_k$ refers to graded relevance of $k_{th}$ result based on ground truth information.     NDCG(Q)=$\frac{1}{|Q|}\sum_{q\in Q}NDCG(q)$

Spent time     Time spent to accomplish a certain programming task.

Likert     Assessment made by human participants on a numerical scale.

§ EXPERIMENTAL DATASET

Table <ref> shows the datasets used by the primary studies.

Dataset used by primary studies

ID     Queries     Systems     Origin of queries

– Continued from previous page

ID     Queries     Systems     Origin of queries

4rContinued on next page

S1     200     04     Change requests, version control history

S2     22     -     Code search logs

S3     34     -     Test cases

S4     70     01     Existing benchmark

S5     70     -     Stack Overflow questions

S6     4,393     02     Bug reports

S7     174     01     Stack Overflow questions, study participants

S8     164     05     Change requests

S9     217     02     Feature requests, feature description

S10     282     05     Change requests, study participants

S11     06     01     Change requests

S12     28     -     Study participants

S13     128     -     Study participants

S14     06     -     Code search logs

S15     129     -     Existing benchmark, code search logs, Stack Overflow questions, study participants

S16     82     -     Study participants

S17     4,917     -     Code search logs

S18     07     -     Study participants

S19     310     04     Existing benchmark, programming tutorial sites

S20     74     24,666     Existing benchmark, Stack Overflow questions

S21     4,684     03     Bug reports

S22     1,675     08     Change requests

S23     21     07     Bug reports, product descriptions

S24     04     -     Existing benchmark

S25     5,139     06     Bug reports

S26     300     03     Stack Overflow questions

S27     08     04     Existing benchmark

S28     97     -     Code comment

S29     424     08     Bug reports

S30     30     03     Change requests

S31     08     -     Code search logs

S32     1043     15     Bug reports, use cases

S33     05     -     Study participants

S34     20     01     Stack Overflow questions

S35     54     -     Existing benchmark

S36     19     01     Existing benchmark

S37      14     -     Study participants

S38     510     02     Change requests

S39     175     03     Programming tutorial sites

S40     1,939     08     Change requests

S41     30     1     Code search logs

S42     13     100     Existing benchmark

S43     09     04     Bug reports

S44     451     21     Bug reports

S45     1,787     04     Change requests

S46     45     -     Hand-picked

S47     276     -     Study participants

S48     -     -     Case study

S49     169     -     Study participants

S50     88     01     Study participants, source code

S51     59     10     Regulatory documents

S52     383     01     Requirement documents

S53     5     01     Task description

S54     372     04     Feature description

S55     12     -     Study participants

S56     47     04     Task description

S57     531     05     Bug reports

S58     209     737     Existing benchmark

S59     43     23     Study participants

S60     20     1     Bug reports

S61     1,546     6     Existing benchmark

S62     1,546     6     Existing benchmark

S63     54     625     Existing benchmark

S64     43     1     Industry partner

S65     34     2,151     Existing benchmark

S66     65,103     1     Stack Overflow search logs

S67     613     13     Existing benchmark

S68     620     13     Bug reports

S69     121     23     Feature descriptions

S70     121     1     Stack Overflow questions

§ CONTRIBUTING AUTHORS

Table <ref> shows the authors who contributed to at least two primary studies.

Contributing authors

Author     # Studies

A. Marcus   7

M. M. Rahman   6

C. Roy   6

D. Lo   5

S. Haiduc   4

S. Wang   4

L. Pollock   3

G. Bavota   3

C. Lopes   3

R. Oliveto   3

Z. Li   2

O. Chaparro   2

Y. Zou   2

A. Kak   2

A. Lucia    2

K. Vijay-Shanker   2

Y. Zhang   2

E. Hill   2

F. Pérez   2

O. Lemos   2

D. Shepherd   2

J. Zhao   2

L. Jiang   2

J. Font   2

X. Peng   2

W. Zhao   2

S. Jain   2

T. Fritz   2

T. Menzies   2

T. Wang   2

G. Yin   2

B. Sisman   2

K. Kevic   2

C. Cetina    2

#1     #1#1#1     #1     #1     #1     #1#1          #1#1

[??sto[n. d.]]
year[n. d.].
titleStop words.
(year[n. d.]).

titleJava keywords.

titleReport: Software failure caused $1.7 trillion in
financial losses in 2017.

titleBoeing eyes Lion Air crash software upgrade in 6 to 8

titleThe 737Max and Why Software Engineers Might Want to
Pay Attention.

titleApache Lucene Core.

titleBoeing 737 jets grounded globally as officials
investigate technical issues behind fatal crash.

titleCodota Code Search.

titleGitHub Code Search.

titleHere's the terrifying reason Boeing's 737 MAX 8 is
grounded across the globe.

titleReplication package: A Systematic Review of
Automated Query Reformulations in Source Code Search.

[Antoniol, Canfora, Casazza, De Lucia, and
MerloAntoniol et al2002]
authorpersonG Antoniol, personG Canfora,
personG Casazza, personA De Lucia, and
personE Merlo. year2002.
Recovering Traceability Links between Code
and Documentation.
journalTSE volume28,
number10 (year2002), pages970–983.

[Anvik, Hiew, and MurphyAnvik
et al2005]
authorpersonJ. Anvik, personL. Hiew, and
personG. C. Murphy. year2005.
Coping with an Open Bug Repository. In
booktitleProc. OOPSLA/Eclipse. pages35–39.

[Asaduzzaman, Roy, Schneider, and
HouAsaduzzaman et al2016]
authorpersonM. Asaduzzaman, personC. K.
Roy, personK. A. Schneider, and personD. Hou.
A Simple, Efficient, Context-Sensitive Approach for
Code Completion.
journalJSEP volume28,
number7 (year2016), pages512–541.

[Bajracharya and LopesBajracharya and
authorpersonS. Bajracharya and personC.
Lopes. year2009.
Mining search topics from a code search engine
usage log. In booktitleProc. MSR.

[Bajracharya, Ngo, Linstead, Dou, Rigor, Baldi,
and LopesBajracharya et al2006]
authorpersonS. Bajracharya, personT. Ngo,
personE. Linstead, personY. Dou, personP.
Rigor, personP. Baldi, and personC. Lopes.
Sourcerer: A Search Engine for Open Source Code
Supporting Structure-based Search. In booktitleProc.
OOPSLA-C. pages681–682.

[Bajracharya and LopesBajracharya and
authorpersonS. K. Bajracharya and
personC. V. Lopes. year2012.
Analyzing and Mining a Code Search Engine Usage
journalEMSE volume17,
number4-5 (year2012), pages424–466.

authorpersonV. Balachandran.
Query by example in large-scale code repositories.
In booktitleProc. SANER. pages467–476.

[Banerjee and DaveBanerjee and Dave2004]
authorpersonA. Banerjee and personR. N
Dave. year2004.
Validating clusters using the Hopkins statistic.
In booktitleProc. FUZZY, Vol. volume1.

[Bassett and KraftBassett and Kraft2013]
authorpersonB Bassett and personN A
Kraft. year2013.
Structural Information based Term Weighting
in Text Retrieval for Feature Location. In
booktitleProc. ICPC. pages133–141.

[Blanco and LiomaBlanco and Lioma2012]
authorpersonR Blanco and personC Lioma.
Graph-based Term Weighting for Information
journalInf. Retr. volume15,
number1 (year2012), pages54–92.

[Blei, Ng, and JordanBlei
et al2003]
authorpersonD. M. Blei, personA. Y. Ng,
and personM. I. Jordan. year2003.
Latent Dirichlet Allocation.
journalJ. Mach. Learn. Res.  volume3
(year2003), pages993–1022.

[Bojanowski, Grave, Joulin, and
MikolovBojanowski et al2016]
authorpersonP. Bojanowski, personE. Grave,
personA. Joulin, and personT. Mikolov.
Enriching Word Vectors with Subword Information.
journalarXiv preprint arXiv:1607.04606

authorpersonA. Bosu. year2014.
Characteristics of the Vulnerable Code Changes
Identified Through Peer Code Review. In booktitleProc.
ICSE-C (seriesICSE Companion 2014).

[Bosu, Carver, Hafiz, Hilley, and JanniBosu
et al2014]
authorpersonA. Bosu, personJ. C. Carver,
personM. Hafiz, personP. Hilley, and
personD. Janni. year2014.
Identifying the Characteristics of Vulnerable
Code Changes: An Empirical Study. In
booktitleProc. FSE. pages257–268.

[Brandt, Guo, Lewenstein, Dontcheva, and
KlemmerBrandt et al2009]
authorpersonJ Brandt, personP J Guo,
personJ Lewenstein, personM Dontcheva, and
personS R Klemmer. year2009.
Two Studies of Opportunistic Programming:
Interleaving Web Foraging, Learning, and Writing Code. In
booktitleProc. SIGCHI. pages1589–1598.

[Brin and PageBrin and Page1998]
authorpersonS Brin and personL Page.
The Anatomy of a Large-Scale Hypertextual
Web Search Engine.
journalComput. Netw. ISDN Syst.
volume30, number1-7 (year1998),

[Cai, van Rijsbergen, and JoseCai
et al2001]
authorpersonD. Cai, personC. J. van
Rijsbergen, and personJ. M. Jose.
Automatic Query Expansion Based on Divergence. In
booktitleProc. CIKM. pages419–426.

[Cao, Chen, Baltes, Treude, and ChenCao
et al2021]
authorpersonK. Cao, personC. Chen,
personS. Baltes, personC. Treude, and
personX. Chen. year2021.
Automated Query Reformulation for Efficient Search
based on Query Logs From Stack Overflow. In booktitleProc.
ICSE. pages13.

[Carmel and Yom-TovCarmel and
authorpersonD Carmel and personE
Yom-Tov. year2010.
booktitleEstimating the Query Difficulty for
Information Retrieval.
publisherMorgan & Claypool. 89– pages.

[Carmel, Yom-Tov, Darlow, and PellegCarmel
et al2006]
authorpersonD Carmel, personE Yom-Tov,
personA Darlow, and personD Pelleg.
What Makes a Query Difficult?. In
booktitleProc. SIGIR. pages390–397.

[Carpineto, de Mori, Romano, and
BigiCarpineto et al2001]
authorpersonC. Carpineto, personR. de Mori,
personG. Romano, and personB. Bigi.
An Information-theoretic Approach to Automatic
Query Expansion.
journalACM Trans. Inf. Syst. volume19,
number1 (year2001), pages1–27.

[Carpineto and RomanoCarpineto and
authorpersonC Carpineto and personG
Romano. year2012.
A Survey of Automatic Query Expansion in
Information Retrieval.
journalACM Comput. Surv. volume44,
number1 (year2012), pages1:1–1:50.

[Chan, Cheng, and LoChan
et al2012]
authorpersonW Chan, personH Cheng, and
personD Lo. year2012.
Searching Connected API Subgraph via Text
Phrases. In booktitleProc. FSE.

[Chaparro, Florez, and MarcusChaparro
et al2017a]
authorpersonO Chaparro, personJ M Florez,
and personA Marcus. year2017a.
Using Observed Behavior to Reformulate Queries
during Text Retrieval-based Bug Localization. In
booktitleProc. ICSME. pages376–387.

[Chaparro, Florez, Singh, and MarcusChaparro
et al2019]
authorpersonO. Chaparro, personJ. M.
Florez, personU. Singh, and personA. Marcus.
Reformulating Queries for Duplicate Bug Report
Detection. In booktitleProc. SANER. pages12.

[Chaparro, Lu, Zampetti, Moreno, Di Penta,
Marcus, Bavota, and NgChaparro et al2017b]
authorpersonO Chaparro, personJ Lu,
personF Zampetti, personL Moreno, personM
Di Penta, personA Marcus, personG Bavota, and
personV Ng. year2017b.
Detecting Missing Information in Bug
Descriptions. In booktitleProc. ESEC/FSE.

[Chaparro and MarcusChaparro and
authorpersonO Chaparro and personA
Marcus. year2016.
On the Reduction of Verbose Queries in Text
Retrieval Based Software Maintenance. In booktitleProc.
ICSE-C. pages716–718.

[Chatterjee, Juvekar, and SenChatterjee
et al2009]
authorpersonS. Chatterjee, personS.
Juvekar, and personK. Sen. year2009.
SNIFF: A Search Engine for Java Using Free-Form
Queries. In booktitleProc. FASE.

[Chen, Xing, and LiuChen
et al2019]
authorpersonC. Chen, personZ. Xing, and
personY. Liu. year2019.
What's Spain's Paris? Mining Analogical Libraries
from Q&A Discussions.
journalEMSE volume24,
number3 (year2019), pages1155–1194.

[Cordeiro, Antunes, and GomesCordeiro
et al2012]
authorpersonJ Cordeiro, personB Antunes,
and personP Gomes. year2012.
Context-based Recommendation to Support
Problem Solving in Software Development. In
booktitleProc. RSSE. pages85–89.

[Da Silva, Roy, Rahman, Schneider, Paixo, and
MaiaDa Silva et al2019]
authorpersonR. F. G. Da Silva, personC. K.
Roy, personM. M. Rahman, personK. Schneider,
personK. Paixo, and personM. Maia.
Recommending Comprehensive Solutions for
Programming Tasks by Mining Crowd Knowledge. In
booktitleProc. ICPC. pages358–368.

[Dagenais and RobillardDagenais and
authorpersonB. Dagenais and personM. P.
Robillard. year2012.
Recovering traceability links between an API and
its learning resources. In booktitleProc. ICSE.

[Dietrich, Cleland-Huang, and ShinDietrich
et al2013]
authorpersonT. Dietrich, personJ.
Cleland-Huang, and personY. Shin.
Learning effective query transformations for
enhanced requirements trace retrieval. In booktitleProc.
ASE. pages586–591.

[Dit, Revelle, Gethers, and PoshyvanykDit
et al2013]
authorpersonB. Dit, personM. Revelle,
personM. Gethers, and personD. Poshyvanyk.
Feature location in source code: a taxonomy and
journalJSEP volume25,
number1 (year2013), pages53–95.

[Dourdas, Zhu, Maiden, Jones, and
ZachosDourdas et al2006]
authorpersonN. Dourdas, personX. Zhu,
personN. Maiden, personS. Jones, and
personK. Zachos. year2006.
Discovering Remote Software Services that Satisfy
Requirements: Patterns for Query Reformulation. In
booktitleAdvanced Information Systems Engineering.

[Eddy, Kraft, and GrayEddy
et al2018]
authorpersonBrian P. Eddy, personNicholas A.
Kraft, and personJeff Gray. year2018.
Impact of structural weighting on a latent
Dirichlet allocation based feature location technique.
journalJSEP volume30,
number1 (year2018), pagese1892.

[Ensan, Bagheri, and KahaniEnsan
et al2007]
authorpersonF. Ensan, personE. Bagheri,
and personM. Kahani. year2007.
The Application of Users' Collective Experience for
Crafting Suitable Search Engine Query Recommendations. In
booktitleProc. CNSR. pages148–156.

[Enslen, Hill, Pollock, and
Vijay-ShankerEnslen et al2009]
authorpersonE Enslen, personE Hill,
personL Pollock, and personK Vijay-Shanker.
Mining Source Code to Automatically Split
Identifiers for Software Analysis. In booktitleProc.
MSR. pages71–80.

authorpersonL Favre. year2008.
Modernizing Software & System Engineering
Processes. In booktitleProc. ICSENG.

[Feldt and MagaziniusFeldt and
authorpersonR. Feldt and personA.
Magazinius. year2010.
Validity Threats in Empirical Software Engineering
Research - An Initial Survey. In booktitleProc. SEKE.

authorpersonR. Fisher.
Statistical Methods and Scientific Induction.
journalJournal of the Royal Statistical Society.
Series B (Methodological) volume17, number1
(year1955), pages69–78.

[Furnas, Landauer, Gomez, and DumaisFurnas
et al1987]
authorpersonG W Furnas, personT K Landauer,
personL M Gomez, and personS T Dumais.
The Vocabulary Problem in Human-system
journalCommun. ACM volume30,
number11 (year1987), pages964–971.

[Gay, Haiduc, Marcus, and MenziesGay
et al2009]
authorpersonG Gay, personS Haiduc,
personA Marcus, and personT Menzies.
On the Use of Relevance Feedback in
IR-based Concept Location. In booktitleProc. ICSM.

[Ge, Shepherd, Damevski, and Murphy-HillGe
et al2017]
authorpersonX. Ge, personD. C. Shepherd,
personK. Damevski, and personE. Murphy-Hill.
Design and evaluation of a multi-recommendation
system for local code search.
journalJournal of Visual Languages and Computing
volume39 (year2017), pages1 – 9.

[Ghafari and MoradiGhafari and
authorpersonM. Ghafari and personH.
Moradi. year2017.
A framework for classifying and comparing source
code recommendation systems. In booktitleProc. SANER.

[Gibiec, Czauderna, and Cleland-HuangGibiec
et al2010]
authorpersonM. Gibiec, personA. Czauderna,
and personJ. Cleland-Huang. year2010.
Towards Mining Replacement Queries for
Hard-to-retrieve Traces. In booktitleProc. ASE.

[Glaser and StraussGlaser and
authorpersonB. G. Glaser and personA. L.
Strauss. year1967.
booktitleThe discovery of grounded theory :
strategies for qualitative research.
publisherChicago : Aldine Publishing.

authorpersonR. L. Glass.
Frequently forgotten fundamental facts about
software engineering.
journalIEEE Software volume18,
number3 (year2001), pages112–111.

[Gvero and KuncakGvero and Kuncak2015]
authorpersonT Gvero and personV Kuncak.
Interactive Synthesis Using Free-form Queries.
In booktitleProc. ICSE. pages689–692.

authorpersonS Haiduc. year2011.
Automatically Detecting the Quality of the
Query and its Implications in IR-based Concept Location. In
booktitleProc. ASE. pages637–640.

authorpersonSonia Haiduc.
titleSupporting Text Retrieval Query Formulation In
Software Engineering.
PhD dissertation. schoolWayne State University.

[Haiduc, Bavota, Marcus, Oliveto, De Lucia,
and MenziesHaiduc et al2013]
authorpersonS Haiduc, personG Bavota,
personA Marcus, personR Oliveto, personA
De Lucia, and personT Menzies.
Automatic Query Reformulations for Text
Retrieval in Software Engineering. In booktitleProc.
ICSE. pages842–851.

[Haiduc, Bavota, Oliveto, De Lucia, and
MarcusHaiduc et al2012]
authorpersonS Haiduc, personG Bavota,
personR Oliveto, personA De Lucia, and
personA Marcus. year2012.
Automatic Query Performance Assessment
During the Retrieval of Software Artifacts. In
booktitleProc. ASE. pages90–99.

[Haiduc and MarcusHaiduc and Marcus2011]
authorpersonS Haiduc and personA Marcus.
On the Effect of the Query in IR-based
Concept Location. In booktitleProc. ICPC.

authorpersonS. Hanneman.
Design, Analysis, and Interpretation of
Method-Comparison Studies.
journalAACN advanced critical care
volume19 (date04 year2008),

authorpersonD. Harman.
Relevance Feedback Revisited. In
booktitleProc. SIGIR. pages1–10.

[Hayes, Dekhtyar, and SundaramHayes
et al2006]
authorpersonJ. H. Hayes, personA. Dekhtyar,
and personS. K. Sundaram. year2006.
Advancing candidate link generation for
requirements tracing: the study of methods.
journalTSE volume32,
number1 (year2006), pages4–19.

[Hellendoorn and DevanbuHellendoorn and
authorpersonV. J. Hellendoorn and personP.
Devanbu. year2017.
Are Deep Neural Networks the Best Choice for
Modeling Source Code?. In booktitleProc. ESEC/FSE.

authorpersonEmily Hill.
titleIntegrating natural language and program
structure information to improve software search and exploration.
PhD dissertation. schoolUniversity of Delaware.

[Hill, Pollock, and Vijay-ShankerHill
et al2009]
authorpersonE Hill, personL Pollock, and
personK Vijay-Shanker. year2009.
Automatically Capturing Source Code
Context of NL-queries for Software Maintenance and Reuse. In
booktitleProc. ICSE. pages232–242.

[Hill, Pollock, and Vijay-ShankerHill
et al2011]
authorpersonE Hill, personL Pollock, and
personK Vijay-Shanker. year2011.
Improving Source Code Search with Natural
Language Phrasal Representations of Method Signatures. In
booktitleProc. ASE. pages524–527.

[Hill, Rao, and KakHill
et al2012]
authorpersonE Hill, personS Rao, and
personA Kak. year2012.
On the Use of Stemming for Concern
Location and Bug Localization in Java. In
booktitleProc. SCAM. pages184–193.

[Holmes and MurphyHolmes and Murphy2005]
authorpersonR Holmes and personG C
Murphy. year2005.
Using Structural Context to Recommend
Source Code Examples. In booktitleProc. ICSE.

[Howard, Gupta, Pollock, and
Vijay-ShankerHoward et al2013]
authorpersonM J Howard, personS Gupta,
personL Pollock, and personK Vijay-Shanker.
Automatically Mining Software-based,
Semantically-Similar Words from Comment-Code Mappings. In
booktitleProc. MSR. pages377–386.

[Huang, Xia, Xing, Lo, and WangHuang
et al2018a]
authorpersonQ. Huang, personX. Xia,
personZ. Xing, personD. Lo, and personX.
Wang. year2018a.
API Method Recommendation without Worrying about
the Task-API Knowledge Gap. In booktitleProc. ASE
(seriesASE 2018). pages293–304.

[Huang, Yang, and ChengHuang
et al2019]
authorpersonQ. Huang, personY. Yang, and
personM. Cheng. year2019.
Deep learning the semantics of change sequences for
query expansion.
journalSPE volume49,
number11 (year2019), pages1600–1617.

[Huang, Yang, Wang, Wan, Wang, and WuHuang
et al2017]
authorpersonQ. Huang, personY. Yang,
personX. Wang, personH. Wan, personR.
Wang, and personG. Wu. year2017.
Query Expansion via Intent Predicting.
journalIJSEKE volume27,
number09n10 (year2017), pages1591–1601.

[Huang, Yang, Zhan, Wan, and WuHuang
et al2018b]
authorpersonQ. Huang, personY. Yang,
personX. Zhan, personH. Wan, and
personG. Wu. year2018b.
Query expansion based on statistical learning from
code changes.
journalSPE volume48,
number7 (year2018), pages1333–1351.

[Hussain and BissonHussain and
authorpersonS. F. Hussain and personG.
Bisson. year2010.
booktitleText Categorization Using Word Similarities
Based on Higher Order Co-occurrences.

[Jiang, Shen, Peng, Lv, and ZhaoJiang
et al2015]
authorpersonS. Jiang, personL. Shen,
personX. Peng, personZ. Lv, and personW.
Zhao. year2015.
Understanding developers' natural language queries
with interactive clarification. In booktitleProc. SANER.

authorpersonK S Jones.
A Statistical Interpretation Of Term
Specificity And Its Application In Retrieval.
journalJournal of Documentation
volume28, number1 (year1972),

authorpersonN. Juristo.
Towards understanding replication of software
engineering experiments. In booktitleProc. ESEM.

[Juristo and GómezJuristo and
authorpersonN. Juristo and personO. S.
Gómez. year2012.
booktitleReplication of Software Engineering

[Kelly and TeevanKelly and Teevan2003]
authorpersonD. Kelly and personJ.
Teevan. year2003.
Implicit Feedback for Inferring User Preference: A
journalSIGIR Forum volume37,
number2 (year2003), pages18–28.

[Kevic and FritzKevic and Fritz2014a]
authorpersonK Kevic and personT Fritz.
A Dictionary to Translate Change Tasks to
Source Code. In booktitleProc. MSR.

[Kevic and FritzKevic and Fritz2014b]
authorpersonK Kevic and personT Fritz.
Automatic Search Term Identification for
Change Tasks. In booktitleProc. ICSE.

[Khanjani and SulaimanKhanjani and
authorpersonA. Khanjani and personR.
Sulaiman. year2011.
The aspects of choosing open source versus closed
source. In booktitleIEEE Symposium on Computers &
Informatics. pages646–649.

[Kim and LeeKim and Lee2018]
authorpersonM. Kim and personE. Lee.
Are Information Retrieval-based Bug Localization
Techniques Trustworthy?. In booktitleProc. ICSE.

[Kim and LeeKim and Lee2019]
authorpersonM. Kim and personE. Lee.
A Novel Approach to Automatic Query Reformulation
for IR-Based Bug Localization. In booktitleProc. SAC.

[Kim and LeeKim and Lee2020]
authorpersonM. Kim and personE. Lee.
ManQ: Many-objective optimization-based automatic
query reduction for IR-based bug localization.
journalIST  volume125

[Kimmig, Monperrus, and MeziniKimmig
et al2011]
authorpersonM. Kimmig, personM. Monperrus,
and personM. Mezini. year2011.
Querying source code with natural language. In
booktitleProc. ASE. pages376–379.

[Kitchenham and BreretonKitchenham and
authorpersonB. Kitchenham and personP.
Brereton. year2013.
A Systematic Review of Systematic Review Process
Research in Software Engineering.
journalIST volume55,
number12 (year2013), pages2049–2075.

[Kitchenham and ChartersKitchenham and
authorpersonB. Kitchenham and personS
Charters. year2007.
titleGuidelines for performing Systematic Literature
Reviews in Software Engineering.

[Ko, Myers, Coblenz, and AungKo
et al2006]
authorpersonAndrew J. Ko, personBrad A.
Myers, personMichael J. Coblenz, and
personHtet Htet Aung. year2006.
An Exploratory Study of How Developers Seek,
Relate, and Collect Relevant Information During Software Maintenance Tasks.
journalTSE volume32,
number12 (year2006), pages971–987.

[Kochhar, Tian, and LoKochhar
et al2014]
authorpersonP. S. Kochhar, personY. Tian,
and personD. Lo. year2014.
Potential Biases in Bug Localization: Do They
Matter?. In booktitleProc. ASE.

[Lapeña, Font, Pérez, and
CetinaLapeña et al2016]
authorpersonR. Lapeña, personJ. Font,
personF. Pérez, and personC. Cetina.
Improving Feature Location by Transforming the
Query from Natural Language into Requirements. In
booktitleProc. SPLC. pages362–369.

[Lavrenko and CroftLavrenko and
authorpersonV. Lavrenko and personW. B.
Croft. year2001.
Relevance Based Language Models. In
booktitleProc. SIGIR. pages120–127.

[Lawrie and BinkleyLawrie and
authorpersonD. Lawrie and personD.
Binkley. year2018.
On the Value of Bug Reports for Retrieval-Based Bug
Localization. In booktitleProc. ICSME.

[Lawrie, Morrell, Feild, and BinkleyLawrie
et al2006]
authorpersonD. Lawrie, personC. Morrell,
personH. Feild, and personD. Binkley.
What's in a Name? A Study of Identifiers. In
booktitleProc. ICPC. pages3–12.

[Lemaire and DenhièreLemaire and
authorpersonB. Lemaire and personG.
Denhière. year2008.
Effects of High-Order Co-occurrences on Word
Semantic Similarities.
journalCoRR (year2008).

[Lemos, Bajracharya, Ossher, Masiero, and
LopesLemos et al2011]
authorpersonO. A. L. Lemos, personS.
Bajracharya, personJ. Ossher, personP. C. Masiero,
and personC. Lopes. year2011.
A test-driven approach to code search and its
application to the reuse of auxiliary functionality.
journalIST volume53,
number4 (year2011), pages294 – 306.

[Lemos, de Paula, Sajnani, and LopesLemos
et al2015]
authorpersonO. A. L. Lemos, personA. C. de
Paula, personH. Sajnani, and personC. V. Lopes.
Can the use of types and query expansion help
improve large-scale code search?. In booktitleProc. SCAM.

[Lemos, de Paula, Zanichelli, and LopesLemos
et al2014]
authorpersonO. A. L. Lemos, personA. C. de
Paula, personF. C. Zanichelli, and personC. V.
Lopes. year2014.
Thesaurus-based Automatic Query Expansion for
Interface-driven Code Search. In booktitleProc. MSR.

[Li, Wang, Zhang, Zhan, and YinLi
et al2016]
authorpersonZ. Li, personT. Wang,
personY. Zhang, personY. Zhan, and
personG. Yin. year2016.
Query Reformulation by Leveraging Crowd Wisdom for
Scenario-based Software Search. In booktitleProc.
Internetware. pages36–44.

[Li, Yin, Wang, Zhang, Yu, and WangLi
et al2018]
authorpersonZ. Li, personG. Yin,
personT. Wang, personY. Zhang, personY.
Yu, and personH. Wang. year2018.
Correlation-based software search by leveraging
software term database.
journalFrontiers of Computer Science
volume12, number5 (year2018),

[Lin and MurrayLin and Murray2005]
authorpersonJ. Lin and personG. C.
Murray. year2005.
Assessing the Term Independence Assumption in Blind
Relevance Feedback. In booktitleProc. SIGIR.

[Lin, Zou, Zhao, and XieLin
et al2017]
authorpersonZ. Lin, personY. Zou,
personJ. Zhao, and personB. Xie.
Improving software text retrieval using conceptual
knowledge in source code. In booktitleProc. ASE.

[Linstead, Bajracharya, Ngo, Rigor, Lopes, and
BaldiLinstead et al2009]
authorpersonE. Linstead, personS.
Bajracharya, personT. Ngo, personP. Rigor,
personC. Lopes, and personP. Baldi.
Sourcerer: mining and searching internet-scale
software repositories.
journalData Mining and Knowledge Discovery
volume18, number2 (year2009),

[Liu, Xia, Lo, Gao, Yang, and GrundyLiu
et al2020]
authorpersonC. Liu, personX. Xia,
personD. Lo, personC. Gao, personX. Yang,
and personJ. C. Grundy. year2020.
Opportunities and Challenges in Code Search Tools.
journalCoRR  volumeabs/2011.02297

[Liu, Marcus, Poshyvanyk, and RajlichLiu
et al2007]
authorpersonD Liu, personA Marcus,
personD Poshyvanyk, and personV Rajlich.
Feature Location via Information Retrieval
Based Filtering of a Single Scenario Execution Trace. In
booktitleProc. ASE. pages234–243.

[Liu, Kim, Murali, Chaudhuri, and ChandraLiu
et al2019]
authorpersonJ. Liu, personS. Kim,
personV. Murali, personS. Chaudhuri, and
personS. Chandra. year2019.
Neural Query Expansion for Code Search. In
booktitleProc. MAPL. pages29–37.

[Lu, Wei, Sun, Li, Wen, and ZhouLu
et al2018]
authorpersonJ. Lu, personY. Wei,
personX. Sun, personB. Li, personW. Wen,
and personC. Zhou. year2018.
Interactive Query Reformulation for Source-Code
Search With Word Relations.
journalIEEE Access  volume6

[Lu, Sun, Wang, Lo, and DuanLu
et al2015]
authorpersonMeili Lu, personX. Sun,
personS. Wang, personD. Lo, and
personYucong Duan. year2015.
Query expansion via WordNet for effective code
search. In booktitleProc. SANER.

[Lucia, Oliveto, and SguegliaLucia
et al2006]
authorpersonA. D. Lucia, personR. Oliveto,
and personP. Sgueglia. year2006.
Incremental Approach and User Feedbacks: a Silver
Bullet for Traceability Recovery. In booktitleProc. ICSM.

[Lv, Zhang, Lou, Wang, Zhang, and ZhaoLv
et al2015]
authorpersonF. Lv, personH. Zhang,
personJ. Lou, personS. Wang, personD.
Zhang, and personJ. Zhao. year2015.
CodeHow: Effective Code Search Based on API
Understanding and Extended Boolean Model. In booktitleProc.
ASE. pages260–270.