# Is GPT-4 a reliable rater? Evaluating Consistency in GPT-4’s Text Ratings
††thanks: This investigation served as a preliminary study preceding an
extensive field study conducted as part of the BMBF-funded DeepWrite project
at the University of Passau. The primary objective was to ascertain the
consistency of GPT-4’s assessments before their integration into authentic
scenarios involving students within the realm of Higher Education. We extend
our gratitude towards Johann Graf von Lambsdorff, Deborah Voss, and Stephan
Geschwind for their contributions in designing the questions, sample
solutions, and the field study associated with this investigation.

Veronika Hackl University of Passau, Faculty of Social and Educational
Sciences, Innstrasse 41, 94032 Passau, Germany<EMAIL_ADDRESS>Main author. Alexandra Elena Müller University of Passau, Faculty of Law,
Innstrasse 41, 94032 Passau, Germany. Research assistant. Maximilian Sailer
University of Passau, Faculty of Social and Educational Sciences, Innstrasse
41, 94032 Passau, Germany. Methodology contributor. Michael Granitzer
University of Passau, Faculty of Computer Science and Mathematics, Innstrasse
41, 94032 Passau, Germany. Hypotheses formulation contributor.

###### Abstract

This study investigates the consistency of feedback ratings generated by
OpenAI’s GPT-4, a state-of-the-art artificial intelligence language model,
across multiple iterations, time spans and stylistic variations. The model
rated responses to tasks within the Higher Education (HE) subject domain of
macroeconomics in terms of their content and style. Statistical analysis was
conducted in order to learn more about the interrater reliability, consistency
of the ratings across iterations and the correlation between ratings in terms
of content and style. The results revealed a high interrater reliability with
ICC scores ranging between 0.94 and 0.99 for different timespans, suggesting
that GPT-4 is capable of generating consistent ratings across repetitions with
a clear prompt. Style and content ratings show a high correlation of 0.87.
When applying a non-adequate style the average content ratings remained
constant, while style ratings decreased, which indicates that the large
language model (LLM) effectively distinguishes between these two criteria
during evaluation. The prompt used in this study is furthermore presented and
explained. Further research is necessary to assess the robustness and
reliability of AI models in various use cases.

##### Keywords:

GPT-4, Consistency, Higher Education, Feedback, Prompt Engineering, Large
Language Models

##  1 Introduction

The integration of AI models, particularly LLMs, into the evaluation of
written tasks within educational environments is a burgeoning trend. This
trend is driven by the potential of these models to enhance learning outcomes
by transforming traditional pedagogical methods.

As the use of these models becomes increasingly pervasive, it is imperative to
thoroughly understand and quantify their reliability and consistency. Elazar
et al. have defined consistency as ’the ability to make consistent decisions
in semantically equivalent contexts, reflecting a systematic ability to
generalize in the face of language variability’ (Elazar et al., 2021).

In the context of automated essay grading, inconsistent ratings could lead to
unfair outcomes for students, undermining the credibility of the assessment
process. Trust in the system ’is highly influenced by users’ perception of the
algorithm’s accuracy. After seeing a system err, users’ trust can easily
decrease, up to the level where users refuse to rely on a system’ (Conijn et
al., 2023, p.3). Similarly, in the context of personalized learning,
unreliable predictions could result in inappropriate learning recommendations.
Therefore, scrutinizing the consistency of AI models is a necessary step
towards ensuring the responsible and effective use of these technologies in
education (Conijn et al., 2023).

GPT-4, through its emergent Automated Writing Evaluation capabilities,
presents a significant advancement in overcoming traditional obstacles
inherent in the evaluation of writing tasks. One such obstacle is discourse
coherence, a fundamental aspect of writing that refers to the logical and
meaningful connection of ideas in a text. In traditional manual grading,
assessing discourse coherence can be a subjective and time-consuming process,
often leading to inconsistencies in grading. However, GPT-4 with its advanced
language understanding capabilities, can analyze the logical flow of ideas in
a text, thereby providing a more objective and efficient evaluation of
discourse coherence (Naismith et al., 2023).

Feedback plays a crucial role in bridging the gap between a learning objective
and the current level of competence and effective feedback, as outlined by
Hattie and Timperley; encompassing three perspectives: Feed-Back, Feed-Up, and
Feed-Forward. Feed-Back involves providing information about the current
performance, Feed-Up clarifies the goals, and Feed-Forward gives guidance on
how to improve. (Hattie and Timperley, 2007) ’Feedback is a core component of
formative assessment processes and has been identified as a powerful factor
influencing learning in various instructional contexts, including higher
education’ (Narciss and Zumbach, 2020). Regarding the development of writing
skills, feedback on the text plays a crucial role, as it’s nearly impossible
to improve one’s writing abilities without such feedback (Schwarze, 2021).

In the context of this study, the AI-generated feedback primarily focuses on
the Feed-Back perspective, providing an analysis of the content and style
produced by the student. In this scenario of analytic rating, ’the rater
assigns a score to each of the dimensions being assessed in the task’ (Jonsson
and Svingby, 2007), in our case scores for style and content. The AI-generated
feedback in this study is constructed to be adaptive and to assist the learner
in figuring out options for improvement. This forms a contrast to non-adaptive
or static feedback (e.g. the presentation of a sample solution) which is often
used in HE scenarios due to its resource efficiency (Sailer et al., 2023).
Comprehensive feedback, which includes not only a graded evaluation but also
detailed commentary on the students’ performance, has been shown to lead ’to
higher learning outcomes than simple feedback, particularly in regard to
higher order learning outcomes’ (der Kleij et al., 2015). To make the feedback
comprehensive and adaptive, it is prompted to include comments on the
students’ performance as well as numerical ratings and advice on how to
improve.

A key advantage of AI-generated feedback is its immediacy. As noted by Wood
and Shirazi (2020), ‘Prompt feedback allows students to confirm whether they
have understood a topic or not and helps them to become aware of their
learning needs.’(Wood and Shirazi, 2020, p. 24). This immediacy, which is
often challenging to achieve in traditional educational settings due to
constraints such as class size and instructor workload, can significantly
enhance the learning experience by providing students with timely and relevant
feedback (Haughney et al., 2020). Kortemeyer’s observation that ’The system
performs best at the extreme ends of the grading spectrum: clearly correct and
clearly incorrect solutions are generally reliably recognized […]’
(Kortemeyer, 2023) further underscores the potential of AI models like GPT-4
in assisting human graders. This is particularly relevant in large-scale
educational settings where human graders may struggle to consistently identify
clearly correct or incorrect solutions due to the sheer volume of work.

##  2 Hypotheses

The stability of GPT-4’s performance is of significant interest given its
potential implications for educational settings where the consistent grading
of students’ work is paramount. In this investigation, GPT-4 was employed to
assess responses to questions within the subject domain of macroeconomics with
a focus on both the content and style of the responses. For content, the AI
was prompted to evaluate how close the test answer is semantically to the
sample solution. A sample solution inserted as demonstration in the prompt
allows in-context learning and serves to control the quality of the output
(Min et al., 2022). For style, the AI was asked to check whether the language
used in the test answer is appropriate for a HE setting and if the response is
logically structured and plausible. The questions displayed different levels
of complexity. The answers in the test set were created by the authors and
subject domain experts, imitating the differing quality of student answers.

The primary objective of this study is to evaluate the consistency of ratings
generated by GPT-4 across multiple iterations, time spans and variations. We
demonstrate the agreement between raters and examine various dimensions of
consistency. The term raters in our case refers to the different GPT-4
ratings. To provide a comprehensive analysis of GPT-4’s performance and
application, we propose the following hypotheses:

  1. H1:

The ratings generated by GPT-4 are consistent across multiple iterations.

  2. H1.1:

The ratings generated by GPT-4 are consistent across different time spans,
specifically within one week (short-term) and over several months (long-term).

  3. H1.2:

The complexity of the evaluated task does not influence the consistency of
GPT-4’s ratings.

  4. H1.3:

Different types of feedback (e.g. style, content) do not affect the
consistency of GPT-4’s performance.

  5. H2:

There is a significant correlation between the ratings for content and style
in GPT-4’s evaluations.

  6. H3:

A certain type of prompt framework enables adaptation to new questions and
answers, while maintaining consistency in the generated text.

While each of these hypotheses explores a distinct aspect related to GPT-4’s
performance and application, collectively, they contribute to a comprehensive
and multi-dimensional understanding of GPT-4’s potential and limitations in
HE.

##  3 Methods

The methods section of this study is designed to provide a comprehensive
overview of the research process, detailing the steps taken to address the
hypotheses. The research process involves a series of statistical analyses,
with the data collection process specifically designed to evaluate the
consistency of a LLM in providing feedback and rating students’ responses
within the subject domain of macroeconomics.

###  3.1 Data Collection

The data collection phase was conducted over a 14-week period from April 2023
to July 2023, with API calls being made at different times and on different
days to mimic a realistic usage scenario. The assumption underlying this
approach is that the behavior of the model changes over time (Chen et al.,
2023). The API was called through a key within the Audience Response System
classEx, which was used to interface with the AI model (Giamattei and
Lambsdorff, 2019).

Figure 1: Flowchart: Generated texts

###  3.2 Prompt Framework and Test Responses

The first step in the research process involved the establishment of a prompt
framework that serves as a universal structure within the context of this
investigation. The goal was to insert new pairs of question and sample
solutions without altering the consistency of the output, namely the LLM-
generated feedback. Pairs of questions (Ruth and Murphy, 1988), along with
corresponding sample solutions pertinent to macroeconomics, were prepared and
integrated into the prompt framework. This integration set the stage for the
model to assess students’ responses and to generate feedback. First taxonomies
aim at structuring prompt formulation approaches. The prompt used in this
study would be a Level 4 on the Proposed Prompt Taxonomy TELeR (Turn,
Expression, Level of Details, Role) by (Santu and Feng, 2023).

####  3.2.1 Establishing the prompt framework

The prompt framework was adapted to ensure consistency in the AI-generated
feedback. A tight scaffold or rubric for rating to gain comparable results was
used (Jonsson and Svingby, 2007). The system settings were adjusted to control
the randomness of the model’s responses, with a temperature setting of 0 used
to minimize variability. (Si et al., 2023; Schulhoff and Contributors, 2022).
By forcing the model into a deterministic behavior, it becomes more consistent
in its outputs, while the chances to produce very good or very bad generations
decrease. This is a brief documentation of the problems we encountered and the
main changes we applied on the path to creating a prompt that works
consistently for the use case:

Problem |  Changes made in prompt
---|---
Output format varies |  very clear instructions, ordinal numbers, examples
evaluations not strict enough |  role prompting, clear evaluation criteria and application
robustness |  shortening the prompt reduces calculation time, fewer outages
multiple identical inputs |  different inputs can be tested at the same time, identical inputs must not be tested in one run as the parameters will then be passed incorrectly and/or the result is homogeneous
informal address with ’Du’ |  giving a clear instruction in the prompt with example
show star symbols |  add the symbol in the prompt

This is the final scheme of the prompt framework used for the collection of
data (shortened and translated, original language: German):

Element/Function |  Prompt Formulation
---|---
Role Prompting |  You are a professor of macroeconomics and you pose this question to your students:
Variable |  <Insert Question here>
Task Description |  You evaluate the student’s response based on the sample solution using the criteria of content and style, and provide suggestions for improvement. This is the sample solution. It is clearly structured and builds the argument coherently. This solution is both correct in terms of content and very good in terms of style. It would receive 5 out of 5 stars for content and style. Sample solution:
Variable |  <Insert sample solution here>
Stepwise Task Description |  Please evaluate the student’s response based on the sample solution in three steps.
Set Behavior |  Here are some general tips for evaluation: Good feedback is honest and motivating. Always address the student directly using "you," for example: "Your response." Explain or mention the relevant points you are referring to.
Step 1: Evaluation of content (text feedback) |  Step 1: Provide feedback on the content. Answer the following questions: Is the student’s response correct in terms of content? Orient yourself to the meaning of the sample solution, but do not mention the sample solution. Are there any areas for improvement? Use a maximum of 2 sentences for this feedback.
Step 2: Evaluation of style (text feedback) |  Step 2: Provide feedback on the style: Is the language used by the student appropriate for the field of study? Is the response logically structured and does the argumentation make sense? Are there any areas for improvement? Use a maximum of 2 sentences for this feedback.
Step 3: Evalution (numeric feedback) |  Step 3: Evaluate the content and style of the response on a scale of 1 to 5 stars. The rating is based on the feedback on content and style. 1 star indicates a very poor performance. 5 stars indicate a very good performance. Only display the following for Step 3: Content: Number of stars (Please also provide the number of stars as a numeral in parentheses) Style: Number of stars (Please also provide the number of stars as a numeral in parentheses)
Set Format |  You provide a concise evaluation divided into 1. to 3.. Always display the stars as follows: star. Output:

####  3.2.2 Test Responses

Following the establishment of the prompt framework, test responses were
created by domain experts to mimic potential student answers to the given
questions. The test set included a variety of responses, ranging from very
good answers to nonsense answers and potential prompt injections, to ensure a
comprehensive evaluation of the model’s performance (Liu et al., 2023). An
initial set of ten test responses was prepared for the first question. Based
on our experience from this initial set, we expanded the test response set to
14 for the subsequent seven questions. We included prompt injections in the
test set of answers to fine-tune the prompt and increase the probability to
withstand potential attacks (Perez and Ribeiro, 2022).

Type Number | Type of Test Answer
---|---
1 | copy of sample solution
2 | correct, bullet points
3 | nonsense
4 | very good answer
5 | opposite of sample solution, but good style
6 | solid answer
7 | incorrect, average style
8 | correct, slightly informal style
9 | average answer
10 | bad content, bad style
11 | only symbols
12 | copy paste of question
13 | prompt injection 1
14 | prompt injection 2
Table 3: Scheme of test set of answers

####  3.2.3 Format of the generated feedback

The AI model generated feedback texts for each student response, accompanied
by two ratings on a scale of 1 to 5 - one for content ("Inhalt") and one for
style ("Stil"). This process was repeated eleven times for each question,
resulting in a robust dataset comprising 2.596 generated ratings.

This is the format of the AI-generated feedback:

1\. Your answer is factually correct and aligns with the essential points of
the model solution. You have well explained the impacts of the high population
growth rate on the level of the domestic product and its growth. 2\.
Stylistically, your answer is well-formulated, and the argumentation is
logically structured. However, you could explicitly mention the growth model
to fully answer the question. 3\. Content: (5)
Style: (4)

####  3.2.4 Changing styles

We further sought to investigate the effects of stylistic changes on the
evaluation of content. To serve this goal, the test set of answers was
stylistically transformed, keeping the content. Each original student response
was rephrased in the style of Warren G., a US-American rapper, with the
intention of maintaining the semantic content while changing the stylistic
presentation. The corresponding ratings for content and style were again
generated through GPT-4. To assess whether the rephrasing affected the content
and style ratings, paired t-tests were conducted comparing the original and
rephrased ratings for each response. Additionally, the mean and standard
deviation of the rating differences were calculated to provide an overview of
the magnitude and variation of the changes. This is an illustrative example of
an answer out of the original test set (1) and a rephrased version (2). Both
texts were translated, the original text is German. The original text, the
rephrased text and the corresponding ratings for these examples were as
follows:

Original answer, Feedback and Rating
(1) Macroeconomics as engineering starts from human moods and habits. Negative
expectations or moods can lead to a decline in domestic production. This is
due to fewer expenses being incurred. To reverse this trend, the government
should increase its spending. As a result, the domestic product becomes more
stable, and no underemployment occurs. 1. Your answer is substantively correct
and covers the essential points of the question. However, you could delve a
bit more into the role of the government and why it should act expansively. 2.
The style of your answer is appropriate, and the argument is logically
structured. However, you could try to elucidate the connections a bit more
clearly and formulate the reasons more precisely. 3. Content: (4)
Style: (4)

Rephrased answer, Feedback and Rating
(2) Macroeconomics, baby, is like fine-tuning an engine block - it’s all about
the vibes and routine of the people. If folks have a negative vibe or their
attitude isn’t positive, it can delay the economic performance, the GDP,
because they’re simply spending less dough. But you know what, the government
can change the game if they themselves start splashing more cash. This keeps
the GDP in a stable zone, and we don’t have people who have to sit at home
twiddling their thumbs. 1. Your answer is substantively correct as you
highlight the impacts of moods and expectations on domestic production and the
role of government spending. However, you could define and explain the terms
"domestic product" and "underemployment" more precisely. 2. Your style is
informal and could be considered inappropriate in an academic context. Try to
use a more formal language and structure your argument more clearly. 3. 3\.
Content: (3)
Style: (2)

###  3.3 Statistical Analysis

####  3.3.1 Intraclass Correlation Coefficient

The Intraclass Correlation Coefficient (ICC) is a statistical measure in order
to assess the level of agreement or consistency among raters. A perfect ICC
score of 1 indicates perfect agreement or consistency among raters, while a
score of 0 indicates no agreement nor consistency. ICC estimates and their 95%
confident intervals were calculated using RStudio based on a mean-rating,
absolute-agreement, 2-way mixed-effects model. For the decision which ICC
calculation to use the flowchart proposed by Koo et.al was used. The type of
the reliability study is ’inter-rater reliability’. We assign the different
iterations of GPT-4 the role of different raters and assume that the same set
of raters (GPT-4 at different points of time) rates all subjects. The model
chosen is the two-way mixed effects model as we assume to have a specific
sample of raters. The model type decided for is based on the mean of multiple
raters. Both the model definitions “absolute agreement” and “consistency” were
chosen. This results in the 2-way mixed-effects model. The caveat in the ICC
model chosen in the analysis is that it only represents the reliability of the
specific raters involved in this experiment (Koo and Li, 2016). As Generative
AI remains a "black box" system, this was considered to be the most suitable
model (Cao et al., 2023).

The extracted numerical ratings from the feedback texts formed the dataset for
the statistical analyses and were utilized to calculate the ICC, providing a
measure of the consistency of the ratings generated by the AI model.

####  3.3.2 Correlation Analysis and Rating Differences

In order to answer H2, a correlation analysis was conducted. This analysis
involved calculating the correlation coefficient between the content and style
ratings generated by the AI model. The correlation coefficient provides a
measure of the strength and direction of the relationship between the content
and style ratings, thereby providing insights into the model’s grading
criteria. Skewness is a measure of the asymmetry of the probability
distribution of a real-valued random variable about its mean. In this study,
the skewness of the rating distributions was calculated to examine the
symmetry of the data. The purpose of this analysis was to evaluate the extent
to which the ratings deviated from a normal distribution.

##  4 Results

The results section of this study presents the findings from the statistical
analyses conducted to address the hypotheses. The analyses include the
computation of Intraclass Correlation Coefficients (ICCs), skewness measures
for content and style ratings, and a correlation analysis between content and
style ratings.

###  4.1 Intraclass Correlation Coefficients

The tables 4 and 5 present the ICCs for ratings on content (Inh) and style
(Stil) based on two different measurements. Table 4 reports ICCs from the
initial ten measurements conducted between April and June 2023. The ICC values
for both absolute agreement and consistency for content and style are
extremely high (0.999), suggesting almost perfect agreement and consistency
among raters. The 95% confidence intervals (CI) are as well tight, ranging
from 0.998 to 0.999, indicating if the study was replicated, the true ICC
would be expected to fall within this range 95% of the time. The F-tests are
significant (p < 0.001), providing statistical evidence that the raters are
reliably consistent and in agreement with each other in their ratings.

Table 5 reports ICCs from a control measurement. The ratings were obtained
from two raters: the first being an average rating compiled from ten raters
across the months of April to June, and the second being a single rater
evaluating in July. The result shows lower ICC values of 0.944 for both Inh
and Stil. While these are still high values indicating good agreement, they
are not as high as the ICC values in table 4. This implies that while a robust
agreement persists between the mean rating and the July rater, it is not as
pronounced as the concordance among the ten raters. This inference suggests a
temporal evolution in the model’s behavior, necessitating diligent continuous
assessment for its utilization in educational tasks.

ICC Type | ICC Value | 95% CI | F-Test
---|---|---|---
Absolute agreement (Inh) | 0.999 | 0.999 - 0.999 | F(107,971) = 1332, p < 0.001
Absolute agreement (Stil) | 0.999 | 0.998 - 0.999 | F(107,971) = 689, p < 0.001
Consistency (Inh) | 0.999 | 0.999 - 0.999 | F(107,963) = 1332, p < 0.001
Consistency (Stil) | 0.999 | 0.998 - 0.999 | F(107,963) = 689, p < 0.001
Table 4: Reporting of Intraclass Correlation Coefficients (ICC) ICC Type | ICC Value | 95% CI | F-Test
---|---|---|---
Absolute agreement (Inh) | 0.944 | 0.918 - 0.962 | F(107,108) = 17.8, p < 0.001
Absolute agreement (Stil) | 0.944 | 0.918 - 0.962 | F(107,108) = 17.8, p < 0.001
Consistency (Inh) | 0.944 | 0.918 - 0.962 | F(107,107) = 17.8, p < 0.001
Consistency (Stil) | 0.944 | 0.918 - 0.962 | F(107,107) = 17.8, p < 0.001
Table 5: Reporting of Intraclass Correlation Coefficients (ICC) (mean rating
of 10 raters from April to June, contrast rating of July)

###  4.2 Correlation between Content and Style Ratings

The relationship between the average content (Inh) and style (Stil) ratings
was examined to assess the interplay between these two dimensions of
evaluation. A correlation analysis was conducted, yielding a correlation
coefficient of 0.87. This high value indicates a strong positive relationship
between content and style ratings, suggesting that responses rated highly in
terms of content were also likely to receive high style ratings, and vice
versa.

This strong correlation underscores the interconnectedness of content and
style in the evaluation process, suggesting that the AI model does not
distinctly separate these two aspects but rather views them as interrelated
components of a response’s overall quality. When the student answers were
rephrased in a different style, we found that the average difference in
content ratings before and after rephrasing was approximately 0.056 (stars
rating), with a standard deviation of around 1.33. The paired t-test revealed
no significant difference in content ratings between the original and
rephrased responses (t = 0.434, p = 0.665). In terms of style ratings, the
average difference before and after rephrasing was approximately 0.241, with a
standard deviation of around 1.37. The paired t-test suggested a marginally
significant difference between the original and rephrased style ratings (t =
1.813, p = 0.073).

The skewness of the content and style ratings was calculated to assess the
distribution of these ratings. A positive skewness value indicates right-
skewness, while a negative value indicates left-skewness. In this study, the
positive skewness values for content suggest that the AI model tended to give
higher ratings for content (see Table 6). Conversely, the majority negative
skewness values for style suggest a left-skewness, indicating that the model
was more critical in its ratings for style (see Table 7).

These skewness values provide insights into the AI model’s rating tendencies.
The right-skewness for content ratings suggests that the AI model may be more
lenient in its content evaluations or that the student responses were
generally of high quality. The left-skewness for style ratings, on the other
hand, suggests that the AI model may have stricter criteria for style or that
the style of the student responses varied more widely. These insights can
inform future refinements of the AI model to ensure more balanced and fair
evaluations.

Rater | Skewness
---|---
1_Inh | 0.107009
2_Inh | 0.080385
3_Inh | 0.094007
4_Inh | 0.116521
5_Inh | 0.076956
6_Inh | 0.096934
7_Inh | 0.126752
8_Inh | 0.089091
9_Inh | 0.094007
10_Inh | 0.090488
11_Inh | 0.299014

Table 6: Skewness for Content Ratings

Rater | Skewness
---|---
1_Stil | -0.037198
2_Stil | -0.043986
3_Stil | 0.029177
4_Stil | -0.017839
5_Stil | -0.047688
6_Stil | 0.000873
7_Stil | -0.040248
8_Stil | -0.050956
9_Stil | -0.013981
10_Stil | -0.017839
11_Stil | -0.147365

Table 7: Skewness for Style Ratings

##  5 Discussion

The findings of this study provide insights into the potential of AI models,
specifically GPT-4, in evaluating student responses in the context of
macroeconomics.

  * •

The high ICC values for both content and style ratings suggest that the AI
model was able to consistently apply well-defined evaluation criteria at
different points of time and with variations of style and content.

  * •

The ICC values were lower when calculated with another set of feedbacks
generated after a timespan of several weeks.

  * •

The high level of concurrence between ratings underlines the dependability of
the evaluation method employed in this study.

  * •

The positive correlation between content and style ratings underscores the
interconnectedness of content and style in the evaluation process.

  * •

Rephrasing the answers stylistically did not significantly affect the content
ratings, implying that GPT-4 was able to separate content from style in its
evaluations.

  * •

The ICC values show that forcing GPT-4 into a deterministic behavior through
prompt- and system settings works.

It is important to note the limitations of AI models as their application in
educational settings is not free of challenges. As stated in this paper, the
ICC values differ for ratings at different points of time. There are
variations in consistency for different levels of question complexity. Other
limitations are being mentioned in OpenAI’s technical report on GPT-4: AI
models can sometimes make up facts, double-down on incorrect information, and
perform tasks incorrectly (OpenAI, 2023). Another challenge is the "black box"
problem, as discussed by (Cao et al., 2023). This refers to the lack of
transparency and interpretability of AI models, which can hinder their
effective use in educational settings. Further research is needed to address
this issue and enhance the transparency and interpretability of AI models.

Despite these challenges, there are promising avenues for enhancing the
capabilities of AI models. The provision of feedback to macroeconomics
students can be characterized as an emergent capability of the AI model.
Emergence is a phenomenon wherein quantitative modifications within a system
culminate in qualitative alterations in its behavior. This suggests that
larger-scale models may exhibit abilities that smaller-scale models do not, as
suggested by (Wei et al., 2022). However, a direct comparison with GPT-3.5 is
needed to substantiate this claim. The potential of AI models in providing
feedback can be further enhanced by improving their "Theory of Mind" or human
reasoning capabilities, as suggested by (Moghaddam and Honey, 2023). This
could lead to more nuanced and contextually appropriate feedback, thereby
enhancing the learning experience of students. Furthermore, the study by (Fu
et al., 2023) points to the potential of using AI models to audit generative
AI. This opens up new avenues for ensuring the quality and reliability of AI-
generated content. Above that, the use of smaller models should be encouraged
(Bursztyn et al., 2022) as well as the idea to evaluate AI-generated feedbacks
either by a human rater or an AI before shown to the student (Perez and et
al., 2022).

In conclusion, while the results of this study are encouraging, they
underscore the need for further research to fully harness the potential of AI
models in educational settings. Future studies should focus on addressing the
long-term performance, but also the limitations of AI models and exploring
ways to enhance their reliability, transparency, and interpretability.

## References

  * Bursztyn et al. [2022] Bursztyn, V., Demeter, D., Downey, D. and Birnbaum, L. (2022). Learning to perform complex tasks through compositional fine-tuning of language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, pp. 1676–1686.
  * Cao et al. [2023] Cao, Y., Li, S., Liu, Y., Yan, Z., Dai, Y., Yu, P. S. and Sun, L. (2023). A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT. ArXiv:2303.04226.
  * Chen et al. [2023] Chen, L., Zaharia, M. and Zou, J. (2023). How is ChatGPT’s behavior changing over time? ArXiv:2307.09009.
  * Conijn et al. [2023] Conijn, R., Kahr, P. and Snijders, C. (2023). The Effects of Explanations in Automated Essay Scoring Systems on Student Trust and Motivation. Journal of Learning Analytics, 10 (1), 37–53.
  * der Kleij et al. [2015] der Kleij, F. M. V., Feskens, R. C. W. and Eggen, T. J. H. M. (2015). Effects of feedback in a computer-based learning environment on students’ learning outcomes: A meta-analysis. Review of Educational Research, 85 (4), 475–511.
  * Elazar et al. [2021] Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H. and Goldberg, Y. (2021). Measuring and Improving Consistency in Pretrained Language Models. ArXiv:2102.01017.
  * Fu et al. [2023] Fu, J., Ng, S. K., Jiang, Z. and Liu, P. (2023). GPTScore: Evaluate as You Desire. ArXiv:2302.04166.
  * Giamattei and Lambsdorff [2019] Giamattei, M. and Lambsdorff, J. G. (2019). classEx – an online tool for lab-in-the-field experiments with smartphones. Journal of Behavioral and Experimental Finance, 22, 223–231.
  * Hattie and Timperley [2007] Hattie, J. and Timperley, H. (2007). The power of feedback. Review of Educational Research, 77 (1), 81–112.
  * Haughney et al. [2020] Haughney, K., Wakeman, S. and Hart, L. (2020). Quality of feedback in higher education: A review of literature. Educ. Sci., 10 (3), 60.
  * Jonsson and Svingby [2007] Jonsson, A. and Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2 (2), 130–144.
  * Koo and Li [2016] Koo, T. K. and Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15 (2), 155–163.
  * Kortemeyer [2023] Kortemeyer, G. (2023). Can an AI-tool grade assignments in an introductory physics course? ArXiv:2304.11221.
  * Liu et al. [2023] Liu, Y., Deng, G., Li, Y., Wang, K., Zhang, T., Liu, Y., Wang, H., Zheng, Y. and Liu, Y. (2023). Prompt Injection attack against LLM-integrated Applications. ArXiv:2306.05499.
  * Min et al. [2022] Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H. and Zettlemoyer, L. (2022). Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? ArXiv:2202.12837.
  * Moghaddam and Honey [2023] Moghaddam, S. R. and Honey, C. J. (2023). Boosting Theory-of-Mind Performance in Large Language Models via Prompting. ArXiv:2304.11490.
  * Naismith et al. [2023] Naismith, B., Mulcaire, P. and Burstein, J. (2023). Automated evaluation of written discourse coherence using GPT-4. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), Toronto, Canada: Association for Computational Linguistics, pp. 394–403.
  * Narciss and Zumbach [2020] Narciss, S. and Zumbach, J. (2020). Formative Assessment and Feedback Strategies, Cham: Springer International Publishing, pp. 1–28.
  * OpenAI [2023] OpenAI (2023). GPT-4 Technical Report. ArXiv:2303.08774.
  * Perez and et al. [2022] Perez, E. and et al. (2022). Discovering Language Model Behaviors with Model-Written Evaluations. ArXiv:2212.09251.
  * Perez and Ribeiro [2022] Perez, F. and Ribeiro, I. (2022). Ignore Previous Prompt: Attack Techniques For Language Models. ArXiv:2211.09527.
  * Ruth and Murphy [1988] Ruth, L. and Murphy, S. M. (1988). Designing Writing Tasks for the Assessment of Writing.
  * Sailer et al. [2023] Sailer, M., Bauer, E., Hofmann, R., Kiesewetter, J., Glas, J., Gurevych, I. and Fischer, F. (2023). Adaptive feedback from artificial neural networks facilitates pre-service teachers’ diagnostic reasoning in simulation-based learning. Learning and Instruction, 83, 101620\.
  * Santu and Feng [2023] Santu, S. K. K. and Feng, D. (2023). TELeR: A General Taxonomy of LLM Prompts for Benchmarking Complex Tasks. ArXiv:2305.11430.
  * Schulhoff and Contributors [2022] Schulhoff, S. and Contributors, C. (2022). Learn Prompting.
  * Schwarze [2021] Schwarze, C. (2021). Feedbackpraktiken im Schreibcoaching: Texte besprechen in der Hochschullehre. Coaching Theor. Prax., 7, 117–134.
  * Si et al. [2023] Si, C., Gan, Z., Yang, Z., Wang, S., Wang, J., Boyd-Graber, J. and Wang, L. (2023). Prompting GPT-3 To Be Reliable. ArXiv:2210.09150.
  * Wei et al. [2022] Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J. and Fedus, W. (2022). Emergent Abilities of Large Language Models. ArXiv:2206.07682.
  * Wood and Shirazi [2020] Wood, R. and Shirazi, S. (2020). A systematic review of Audience Response Systems for Teaching and Learning in Higher Education: The Student Experience. Computers & Education, 153.


particles, protons and carbon, are combined into one beam line.

Electrostatic deflection extraction is used for the $\EC$ and $\EHe$ beams. A
single ESD located in a valley is used, with an electric field of
${\approx}15\UMV/\UmZ$. From the right panel of fig:c400_2, it can be seen
that the beams from both extraction systems do not exit in the same direction
and position: they are combined by two external beam lines onto the common
degrader. Downstream, both beams travel in the same beam line.

###  4.6 Regenerative extraction in the IBA S2C2

In the IBA superconducting synchro-cyclotron (S2C2), extraction with an
electrostatic deflector cannot be used, because the turn separation at
extraction is far too small. Instead, the regenerative method of extraction
based on the $2\nu_{r}=2$ resonance is used. The extraction system is fully
passive: only soft iron elements are used. The layout is shown in
fig:extraction3. The regenerator creates a strong magnetic field bump, of
which the quadrupole component increases the radial focusing and locks the
horizontal betatron frequency $\nu_{r}$ to one. The orbit becomes unstable and
is pushed towards the extraction channel by the first-harmonic component of
the magnetic field, also produced in the bump. It is essential to avoid the
Walkinshaw resonance in this process, as illustrated in fig:extraction4. When
the extraction sets in, the displacement of the beam towards the extraction
channel steadily and exponentially builds up. The magnetic septum of the
extraction channel separates the extracted orbit from the last internal orbit.
The locking of the betatron frequency $\nu_{r}$ to one can be observed in the
left panel of fig:extraction4 by the fact that the subsequent azimuth of the
nodes of the oscillation is fixed for multiple turns.

Figure 69: Passive extraction system of the IBA superconducting synchro-
cyclotron S2C2: PMQ, permanent magnet quadrupole. Figure 70: Regenerative
extraction: a strong quadrupole bump increases $\nu_{r}$ and locks it to one.
A steady shift of the beam towards the extraction channel is built up. The
Walkinshaw resonance ($\nu_{r}=2\nu_{z}$) must be avoided.

A series of iron correction bars is needed to compensate the magnetic field
undershoots towards inner radii that are produced by the regenerator and the
extraction channel. Farther downstream, between the main coils, a three-bar
gradient corrector is placed, which reduces the radial defocusing in the
fringe field of the pole. Finally a permanent magnet quadrupole is used to
further match the extracted beam to the external beam line.

##  5 Some aspects of cyclotron magnetic design

A good overview of considerations and methods applied for the design of
compact cyclotron magnets has already been made in previous topical CAS
courses by Jongen and Zaremba[59, 60]. Therefore, after some general
considerations, we limit ourselves in this section to some recent examples and
some new methods that have been developed.

###  5.1 Some general considerations

In industrial practice, many design choices of the cyclotron magnet are often
already fixed before the start of any design calculations. Such choices are
determined by more general considerations, such as:

  1. 1.

the application of the cyclotron, which will determine the particles to be
accelerated and their required maximum kinetic energy;

  2. 2.

from the maximum particle rigidity, the choice of maximum $B$-field will
determine the pole radius, or vice versa;

  3. 3.

the choice of the cyclotron type: isochronous or synchro-cyclotron;

  4. 4.

the coil technology: superconducting or normally conducting;

  5. 5.

the type of extraction to be applied: stripping extraction, ESD, or
regenerative extraction;

  6. 6.

when choosing, for example, an isochronous solution, the following parameters
are often already determined beforehand:

    1. (a)

number of sectors, pole gap, pole angle, valley depth, pole spiral, ;

    2. (b)

number of dees, dee voltage, harmonic mode, and RF frequency;

    3. (c)

for injection, the use of an internal or external ion source.

Other choices can be made with some rough back-of-the-envelope calculations:
here are a few examples.

  1. 1.

The maximum magnet rigidity is obtained from the particle maximum kinetic
energy as follows:

$B\rho=\frac{\sqrt{T^{2}+2TE_{0}}}{300Z}\ ,$ (30)

where $Z$ is the charge ionization number of the particle. The rest-energy,
$E_{0}$, and the kinetic energy, $T$, are expressed in MeV and the rigidity
$B\rho$ in Tm.

  2. 2.

For a magnet that is far from magnetic saturation, the relation between the
magnetic field in the pole gap and the number of ampere turns in the coil can
be approximated as:

$\oint\vec{H}\cdot\mathrm{d}\vec{l}=(NI)_{\mathrm{tot}}\approx\frac{1}{\mu_{0}}\left(gB_{\mathrm{gap}}\right)+\frac{1}{\mu_{0}\mu_{\mathrm{iron}}}\left(L_{\mathrm{iron}}B_{\mathrm{iron}}\right)\
.$ (31)

Here, the integration is made over a closed loop that crosses the gap and
closes via the return yoke around the coils of the magnet. Further
$(NI)_{\mathrm{tot}}$ is the total number of ampere turns in the coils, $g$ is
the pole gap, $L_{\mathrm{iron}}$ is the length of the loop in the iron,
$B_{\mathrm{gap}}$ and $B_{\mathrm{iron}}$ are the magnetic fields in the gap
and iron, respectively, $\mu_{0}$ is the magnetic permeability in vacuum and
$\mu_{\mathrm{iron}}$ is the relative magnetic permeability in iron. In
practice, for a conventional magnet far from saturation, the contribution of
the iron is smaller than a few percent.

  3. 3.

Assuming a hard-edge approximation for the fields produced from the iron pole
sectors, the average field $\bar{B}$ can be estimated as:

$\bar{B}\approx\alpha B_{\mathrm{h}}+(1-\alpha)B_{\mathrm{v}}\ .$ (32)

Here, $\alpha$ is the fraction of the hill angle on one symmetry period and
$B_{\mathrm{h}}$ and $B_{\mathrm{v}}$ are the magnetic field values in the
hill and valley, respectively.

  4. 4.

To estimate the width of the return yoke, the total magnetic flux, $\Phi$,
produced between the poles must be roughly equal to the flux guided in the
yoke:

$\Phi\approx 2\pi R_{\mathrm{p}}^{2}\bar{B}\approx
B_{\mathrm{ret}}A_{\mathrm{ret}}\ .$ (33)

Here $R_{\mathrm{p}}$ is the (effective) pole radius, $B_{\mathrm{ret}}$ is
the magnetic field in the return yoke, and $A_{\mathrm{ret}}$ is the total
surface cross-section of the return yoke.

  5. 5.

The required increase in average magnetic field, necessary for isochronous
operation, may be estimated from the relativistic mass increase of the
particles as follows:

$\bar{B}(r)\approx B_{0}\gamma_{\mathrm{rel}}\ ,$ (34)

where $B_{0}$ is the magnetic field in the centre and $\gamma_{\mathrm{rel}}$
is the relativistic gamma.

  6. 6.

The magnetic flutter may be obtained from the hard-edge approximation, as
given in flutje.

  7. 7.

The betatron frequencies may be related to the flutter and the pole spiral
angle, as given in Eqs. (14) and (15).

###  5.2 Tools for magnetic modelling in Opera

Almost all magnet designs at IBA are done using Opera simulation software from
Cobham (VectorFields)[7]. The main packages used are as follows.

####  5.2.1 Opera-2d

This is the 2D version of the software, which can be used to model magnets
with rotational symmetry. It is a perfect choice for an initial design of a
synchro-cyclotron magnet (without the extraction system). However, it can also
be useful for an initial design of a 3D isochronous cyclotron magnet. In this
case, stacking factors can be used to take into account the hill–valley
structure of such a magnet. This yields a preliminary idea of the average
magnetic field (as a function of radius), the stray field around the
cyclotron, and the dimensions of the return yoke.

####  5.2.2 Opera-3d—modeller interface

This module allows for full 3D simulations. It is easy to use and easy to
include fine geometrical details. The 3D finite-element mesh is generated
automatically. The most commonly used is the tetrahedral mesh. This mesh is
not always very regular, which may result in some noise in the calculated
magnetic field.

####  5.2.3 Opera-3d—preprocessor interface

This module also allows for full 3D simulations, but it is more difficult to
use and even more difficult to include fine geometrical details. However, the
3D finite-element mesh is fully created by the user and therefore fully under
control. A hexahedral mesh is used and less noisy magnetic fields can be
obtained. This module may be useful, for example, for the precise calculation
of magnetic forces.

###  5.3 Design approach using Opera-3d modelling

The cyclotron magnet design approach used at IBA can be characterized with the
following features.

  1. 1.

The 3D models are fully parameterized.

  2. 2.

Models are automatically generated using macros. Together with the previous
feature, this allows some parameters to be changed easily, so that the
corresponding new model is quickly obtained.

  3. 3.

The macros for all different types of IBA cyclotron (such as S2C2, C230,
C19/9-family, C30-family, C70-family) have similar structures. This enables
the quick creation of new macros for new machines if needed.

  4. 4.

The magnetic properties of the iron (the $BH$-curves) are verified with an in-
house permeability meter.

The following types of property are parameterized:

  1. 1.

all important dimensional parameters and pole profiles;

  2. 2.

main coil settings;

  3. 3.

material properties, such the different $BH$-curves for different iron parts;

  4. 4.

the sizes and other properties of the finite-element mesh;

  5. 5.

solver tolerances;

  6. 6.

switches for the selection or de-selection of individual subsystems;

  7. 7.

switches for filling individual subsystems with air or iron.

The main structure of a cyclotron modelling macro is illustrated in
fig:opera4. The first column (blocks in green) shows the main steps, namely:
(i) creation of the model in the Opera-3d modeller, (ii) solving the model
with the Opera-3d solver, and (iii) analysing the result with the Opera-3d
postprocessor. The second column (blocks in blue) illustrates how a model is
created in the modeller: (i) read from a file all the parameters that are
needed to create the model, (ii) create individual subsystems, (iii) assemble
these subsystem into one model that represents the full cyclotron, (iv) create
and save the database that serves at input for the solver. The third column
(yellow blocks) shows some details of some of the blue blocks. Subsystems can
be, for example, the yoke (upper, return), pole, or main coils, but also, for
example, a passive extraction system, a three-bar gradient corrector, or a
permanent magnet quadrupole. Creation of the full cyclotron model includes
steps like (i) loading all subsystems, (ii) defining material properties, and
(iii) defining boundary conditions or symmetries. The creation of the database
requires surface and volume meshing but also enables the inclusion of
different cases to be solved, such as, a list of different main coil currents.
The last column (in orange) shows some details for the creation of one
subsystem, such as (i) creation of the geometry, attaching material labels and
mesh-properties to different cells in the subsystem, and saving.

Figure 71: Logical structure of a typical input file used for the fully
parameterized and automated creation of an Opera-3d model of a cyclotron.

###  5.4 Some examples

In this section, we consider as an example the recent development of the IBA
superconducting synchro-cyclotron S2C2. [b] 72 shows a model of the S2C2 made
in Opera-2d. The vertical axis is the axis of rotational symmetry. The main
coils are shown in red and the magnet iron in deep blue. The iron of the S2C2
magnet is heavily saturated; therefore, additional iron placed around the yoke
might introduce a median plane error or increase the vertical forces on the
cold mass. The goal of this study was to calculate the vertical asymmetry
introduced by the cyclotron support feet. These are shown in light blue. The
feet do not obey rotational symmetry (there are four of them), but a stacking
factor was used to approximate their average effect. To compensate for the
effect of the feet, an iron ring is placed on top of the yoke. [b] 73 shows
some results of these simulations. The left panel shows the total vertical
force acting on the cold mass during ramp-up of the main coil current. Three
cases are shown: (i) for the blue curve, the feet were present in the model
but the ring was not; (ii) for the red curve, the ring was present but the
feet were not; and (iii) for the green curve, both the feet and ring were
present. The dimensions of the ring were optimized to minimize the total
vertical force during ramp-up. It can be seen that the total asymmetry due to
the feet (about $25000\UN$ at full current) is reduced to about $8000\UN$ by
placing the ring. The right panel shows the magnetic median plane error (the
radial field in the median plane) as a function of radius for the same three
cases at the maximum excitation current. It can be seen that for the same
optimized dimensions of the ring, the maximum field error is reduced from
$7\UG$ to ¡$1\UG$. It is noted that in such a synchro-cyclotron, median plane
errors must be avoided as much as possible, because of the weak vertical
focusing and also because of the large number of turns in such a machine.
Based on these calculations, the described method has successfully been used
in the actual machine.

Figure 72: The Opera-2d model of the S2C2 Figure 73: Opera-2d simulation of
the median plane error in the S2C2, as produced by the feet underneath the
cyclotron, and compensation by an iron ring placed on top of the yoke.

The left panel of fig:opera3 shows a model of the S2C2 that was created using
the Opera-3d preprocessor. The typical hexahedral mesh is clearly visible in
this model. The goal of this model was to calculate precisely the forces and
torques on the cold mass as a function of its position and rotation. An
understanding of these forces is crucial for the design of the tie rods by
which the cold mass is suspended in the cryostat. To have little heat flow
from the cold mass to the exterior, the diameter of these tie rods should be
as small as possible. Conversely, the diameter should be large enough to
withstand the magnetic forces acting on the cold mass. The differential forces
on the cold mass due to translations or rotations can be calculated with
better precision in the preprocessor. Results of these calculations are
summarized in the table on the right of fig:opera3. It was found that all
forces and torques vary linearly with displacement or rotation. Furthermore,
it can be seen that all coil movements are unstable: a given displacement will
create a force in the same direction as the displacement. Horizontal
displacements result in a force of about 2 tonnes per mm. For vertical
displacements this is about 0.5 tonnes per mm.

Figure 74: Opera-3d preprocessor model of the S2C2, to estimate forces and
torques due to displacement acting on the cold mass with respect to the
geometrical centre of the cyclotron.

The optimization of the magnetic circuit of the S2C2 has been a long process,
carried out by three main contributors: IBA, AIMA Developpement, and ASG (the
company that made the coil and the cryostat). Many aspects were considered,
such as:

  1. 1.

optimization of the pole-gap profile;

  2. 2.

definition of the pole radius and the $230\UMeV$ extraction radius;

  3. 3.

optimization of coil current density and dimensions to assure a margin with
respect to the critical surface of the used superconducting material and to
allow operation up to $250\UMeV$;

  4. 4.

dimensions of the yoke to balance outside stray fields reasonably;

  5. 5.

dimensions and placement of all horizontal and vertical yoke penetrations
(ports are needed for RF system, ion source, vacuum, beam exit, cryo-coolers,
3 horizontal and $2\times 3$ vertical tie rods);

  6. 6.

optimization of the extraction system;

  7. 7.

shielding required for external systems, such as the rotating capacitor and
the cryogenic coolers;

  8. 8.

the influence of the external iron systems on the accelerated beam;

  9. 9.

the influence of the fringe field on the external beam line;

  10. 10.

median plane errors introduced by the vertical asymmetry in the magnetic
design and compensation of these errors;

  11. 11.

magnetic forces acting on the return yoke, the coils, the extraction system
elements, external components, ;

  12. 12.

compensation of first-harmonic field errors;

  13. 13.

the influence of the cyclotron feet and the yoke lifting system.

For the design of the superconducting coil, the transient behaviour of the
magnet with eddy currents and AC losses needs to be studied in detail, along
with the quenching behaviour.

To make this optimization, magnetic finite-element models were produced, using
Opera-2d, Opera-3d, and CST. In Opera-3d especially, very detailed models were
made, as illustrated in fig:opera5. Besides the yoke, poles and main coils,
more detailed elements and features were included, such as the yoke
penetrations, the extraction system (regenerator and extraction channel with
their first-harmonic corrector bars, gradient corrector, and permanent magnet
quadrupole), and external systems, such as the cyclotron feet, yoke lifting
system, cryo-cooler shields, rotating capacitor shield, and external
quadrupoles with support structure.

Figure 75: A complete and detailed Opera-3D model was made of the IBA S2C2

##  6 Conclusions

The three main subjects discussed in this report (magnetic field design, beam
injection, and beam extraction) are the most difficult problems in cyclotron
design. For industrial applications and isotope production, there is generally
a need for ever-increasing beam intensity. This implies, at the same time, a
minimization of beam losses during both injection and extraction and also an
understanding of space charge effects in these processes. For particle therapy
applications,the importance lies much more in understanding beam quality,
reproducibility, and stability. Except perhaps for Baartman’s paper on space
charge effects in cyclotrons [61], nothing really new has appeared in
theoretical and analytical methods during the last 5–10 years that was helpful
in this aspect. However, increasing importance is given to the computational
tools that are needed to optimize the design. This concerns 3D finite-element
software packages that are used to model the magnetic field as well as
electrical field, but also more specialized orbit tracking codes that can be
produced in sufficient detail. The latter are usually developed in house.

## Acknowledgements

We would like to thank our colleagues Jarno Van de Walle, Vincent Nuttens,
Emma Pearson, and Eric Forton for their help in preparing this paper.

## References

  * [1] E.O. Lawrence and M.S. Livingston, _Phys. Rev._ 40(1) (1932) 9.
http://dx.doi.org/10.1103/PhysRev.40.19

  * [2] C. Oliver _et al._ , Proceedings of the 2013 Cyclotron Conference, Vancouver, Canada, p. 429.
  * [3] V. Veksler, _J. Phys. USSR_ 9(3) (1945) 153.
  * [4] E.M. McMillan, _Phys. Rev._ 68(5-6) (1945) 143L. http://dx.doi.org/10.1103/PhysRev.68.143
  * [5] J.L. Livingood, Principles of Cyclic Particle Accelerators (Van Nostrand, New York, 1961), Chap. 6.
  * [6] L.H. Thomas, _Phys. Rev._ 54(8) (1938) 580. http://dx.doi.org/10.1103/PhysRev.54.580
  * [7] http://operafea.com
  * [8] K.L. Brown _et al._ , Transport: a computer program for designing charged particle beam transport system, CERN 90/04 (1980), p. 39.
  * [9] N. Christofilos, Focussing system for ions and electrons, US patent 2736799, priority date March 10, 1950.
  * [10] E.D. Courant et al., _Phys. Rev._ 88(5) (1952) 1190. http://dx.doi.org/10.1103/PhysRev.88.1190
  * [11] H.L. Hagedoorn and N.F. Verster, _Nucl. Instrum. Methods_ 18–19 (1962) 201.
http://dx.doi.org/10.1016/S0029-554X(62)80032-9

  * [12] H. Willax, Proposal for a 500 MeV isochronous cyclotron with ring magnet, Proc. Int. Conf. on Sector-Focused Cyclotrons, Geneva, 1963, p. 386.
  * [13] J.R. Richardson, AIP Conf. Proc. 9 (1972) 126. http://dx.doi.org/10.1063/1.2946355
  * [14] C.B. Bigham et al., _Phys. Canada_ 29(4) (1973) 29.
  * [15] H.G. Blosser et al., VII Int. Conf. on Cyclotrons and their Applications (1975), p. 584. http://dx.doi.org/10.1007/978-3-0348-5520-4_123
  * [16] X.Y. Wu, Ph.D. thesis, Michigan State University, 1990.
  * [17] N.F. Verster and H.L. Hagedoorn, _Nucl. Instrum. Methods_ 18–19 (1962) 327.
http://dx.doi.org/10.1016/S0029-554X(62)80041-X

  * [18] M.M. Gordon, _Part. Accel._ 16 (1984) 39.
  * [19] S. Zaremba et al., Magnetic field design and calculations for the IBA C70 cyclotron, 18th Int. Conf. on Cyclotrons and their Applications (2007), p. 75.
  * [20] S. Galès, AGOR, a superconducting cyclotron for light and heavy ions, 11th Int. Conf. on Cyclotrons and their Applications (1986), p. 194.
  * [21] H.W. Schreuder (AGOR team), AGOR: initial beam tests, transport and commissioning, 14th Int. Conf. on Cyclotrons and their Applications (1995), p. 6.
  * [22] E.L. Kelly, _Nucl. Instr. Meth._ 18–19 (1962) 33. http://dx.doi.org/10.1016/S0029-554X(62)80006-8
  * [23] N.F. Verster et al., _Nucl. Instr. Meth._ 18–19 (1962) 88.
http://dx.doi.org/10.1016/S0029-554X(62)80014-7

  * [24] http://cyclotron.lbl.gov/cyclotron-history
  * [25] W. Kleeven et al., The influence of magnetic field imperfections on the beam quality in an $\EH^{-}$ cyclotron, 13th Int. Conf. on Cyclotrons and their Applications (1992), p. 380.
  * [26] http://www.mevion.com/triniobium-core
  * [27] W. Kleeven et al., The IBA superconducting synchrocyclotron project S2C2, 20th Int. Conf. on Cyclotrons and their Applications (2013), p. 115.
  * [28] P. Heikkinen, Injection and extraction for cyclotrons, CAS CERN 94/01 II (1994), p. 819.
  * [29] P. Mandrillon, Injection into cyclotrons, CAS CERN 96/02 (1996), p. 153.
  * [30] W. Kleeven, Injection and extraction for cyclotrons, CAS CERN 2006/012 (2006), p. 271.
  * [31] J.-L. Belmont, Ion transport from the source to the first cyclotron orbit, Proc. XXXIII European Cyclotron Progress Meeting [_Nukleonika_ 48 supplement 2 (2003) S13].
  * [32] B.F. Gavin, in The Physics and Technology of Ion Sources, 1st ed., Ed. I.G. Brown (Wiley, New York, 1989), p.167.
  * [33] M. Reiser, _Nucl. Instrum. Methods_ 18–19 (1962) 370.
http://dx.doi.org/10.1016/S0029-554X(62)80047-0

  * [34] N. Hazewindus et al., _Nucl. Instrum. Methods_ 118(1) (1974) 125.
http://dx.doi.org/10.1016/0029-554X(74)90692-2

  * [35] H. Houtman et al., _Comput. Phys._ 8(4) (1994) 469. http://dx.doi.org/10.1063/1.168506
  * [36] M. Reiser and J. Mullendore, _Nucl. Instrum. Methods_ 59(1) (1968) 93.
http://dx.doi.org/10.1016/0029-554X(68)90350-9

  * [37] E. Liukkonen et al., Design of the central regions for the MSU 500 MeV superconducting cyclotron, Proc. 8th Int. Conf. on Cyclotrons and their Applications [_IEEE Trans. Nucl. Sci._ NS-26(2) (1979) 2107]. http://dx.doi.org/10.1109/tns.1979.4329816
  * [38] W. Kleeven et al., The IBA self-extracting cyclotron project, Proc. 33rd European Cyclotron Progress Meeting [_Nukleonika_ 48 supplement 2 (2003) S59].
  * [39] D. Bohm and L.L. Foldy, _Phys. Rev._ 72(8) (1947) 649. http://dx.doi.org/10.1103/PhysRev.72.649
  * [40] R. Beurtly _et al._ , _Nucl. Instrum. Methods_ 33(2) (1965) 338.
http://dx.doi.org/10.1016/0029-554X(65)90069-8

  * [41] J.-L. Belmont and J.L. Pabot, _IEEE Trans. Nucl. Sci._ NS-13(4) (1966) 191.
http://dx.doi.org/10.1109/TNS.1966.4324204

  * [42] J.-L. Belmont, Axial injection and central region of AVF cyclotrons, Lecture notes of 1986 RCNP KIKUCHI Summer School on Accelerator Technology. Osaka, Research Center for Nuclear Physics, Osaka, p. 79.
  * [43] L. Root, Ph.D. thesis, University of British Columbia, 1972.
  * [44] R. Baartman and W. Kleeven, _Part. Accel._ 41 (1993) 41.
  * [45] W. Kleeven and R. Baartman, _Part. Accel._ 41 (1993) 55.
  * [46] W. Kleeven et al., _Nucl. Instrum. Methods Phys. Res. B_ 269(24) (2011) 2857.
http://dx.doi.org/10.1016/j.nimb.2011.04.031

  * [47] http://www.d-pace.com/products_hion.html
  * [48] W. Kleeven et al., _Nucl. Instrum. Methods Phys. Res. B_ 64(1-4) (1992) 367.
http://dx.doi.org/10.1016/0168-583X(92)95496-E

  * [49] J.I.M. Botman and H.L. Hagedoorn, Extraction from cyclotrons, CAS CERN 96/02 (1996), p. 169.
  * [50] G. Dutto, The TRIUMF 520 MeV cyclotron, 18 Int. Conf. on Cyclotrons and their Applications, Vancouver 1992, (World Scientific, 1993), p. 138.
  * [51] W. Joho, Extraction from medium and high energy cyclotrons, Proc. 5 Int. Conf. on Cyclotrons (Butterworths, London, 1971), p. 159.
  * [52] H.L. Hagedoorn and P. Kramer, _IEEE Trans. Nucl. Sci._ NS-13(4) (1966) 64.
http://dx.doi.org/10.1109/TNS.1966.4324177

  * [53] H.L. Hagedoorn and N.F. Verster, The extraction of the beam of the Philips AVF cyclotron, CERN 63-19 (1963), p. 228.
  * [54] M.M. Gordon and H.H. Blosser , Orbit calculations on the extraction system for the MSU cyclotron, CERN 63-19 (1963), p. 236.
  * [55] W. Kleeven, Ph.D. thesis, Eindhoven University, 1988.
  * [56] Y. Jongen et al., High intensity cyclotrons for radioisotope production or the comeback of the positive ions, Proc. 14th Int. Conf. on Cyclotrons and their Applications, Cape Town, Ed. J.C. Cornell (1995), p.115.
  * [57] Y. Jongen et al., _Nucl. Instrum. Methods Phys. Res A_ 624(1) (2010) 47.
http://dx.doi.org/10.1016/j.nima.2010.09.028

  * [58] Y. Jongen et al., IBA-JINR 400 MeV/u superconducting cyclotron for hadron therapy, Proc. of Cyclotrons, 2010, Lanzhou, China, p. 404.
  * [59] Y. Jongen and S. Zaremba, Cyclotron magnet calculations, CAS CERN 96/02 (1996), p. 139.
  * [60] S. Zaremba, Magnets for cyclotrons, CAS CERN 2006/012 (2006), 253.
  * [61] R. Baartman, Space charge limit in separated turn cyclotrons, 20th Int. Conf. on Cyclotrons and their Applications (2013) p. 305.