# Digital Comprehensibility Assessment of Simplified Texts among Persons with
Intellectual Disabilities

Andreas Säuberli<EMAIL_ADDRESS>0000-0001-9613-334X Department of
Computational LinguisticsUniversity of ZurichZurichSwitzerland ,  Franz
Holzknecht<EMAIL_ADDRESS>0000-0002-1218-2062 Institute for Language
and CommunicationUniversity of Teacher Education in Special Needs
ZurichZurichSwitzerland ,  Patrick Haller<EMAIL_ADDRESS>0000-0002-8968-7587 Department of Computational LinguisticsUniversity of
ZurichZurichSwitzerland ,  Silvana Deilen<EMAIL_ADDRESS>0000-0002-2933-9257 Johannes Gutenberg University MainzGermersheimGermany ,
Laura Schiffl<EMAIL_ADDRESS>0009-0003-4570-7472 Klinikum rechts der
Isar, Technische Universität MünchenMünchenGermany ,  Silvia Hansen-Schirra
<EMAIL_ADDRESS>0000-0002-3277-7717 Johannes Gutenberg University
MainzGermersheimGermany and  Sarah Ebling<EMAIL_ADDRESS>0000-0001-6511-5085 Department of Computational LinguisticsUniversity of
ZurichZurichSwitzerland

(2024)

###### Abstract.

Text simplification refers to the process of increasing the comprehensibility
of texts. Automatic text simplification models are most commonly evaluated by
experts or crowdworkers instead of the primary target groups of simplified
texts, such as persons with intellectual disabilities. We conducted an
evaluation study of text comprehensibility including participants with and
without intellectual disabilities reading unsimplified, automatically and
manually simplified German texts on a tablet computer. We explored four
different approaches to measuring comprehensibility: multiple-choice
comprehension questions, perceived difficulty ratings, response time, and
reading speed. The results revealed significant variations in these
measurements, depending on the reader group and whether the text had undergone
automatic or manual simplification. For the target group of persons with
intellectual disabilities, comprehension questions emerged as the most
reliable measure, while analyzing reading speed provided valuable insights
into participants’ reading behavior.

Automatic text simplification, evaluation, intellectual disabilities, reading
comprehension, readability, digital testing

††journalyear: 2024††copyright: rightsretained††conference: Proceedings of the
CHI Conference on Human Factors in Computing Systems; May 11–16, 2024;
Honolulu, HI, USA††booktitle: Proceedings of the CHI Conference on Human
Factors in Computing Systems (CHI ’24), May 11–16, 2024, Honolulu, HI,
USA††doi: 10.1145/3613904.3642570††isbn: 979-8-4007-0330-0/24/05††ccs: Human-
centered computing Empirical studies in accessibility††ccs: Computing
methodologies Natural language processing

##  1\. Introduction

Text simplification refers to the process of improving the comprehensibility
of texts by reducing complexity at several linguistic levels, for instance, by
using simpler vocabulary and syntactic structures, reorganizing text
structure, and explaining difficult words and concepts. Primary target groups
of simplified language111We use the term _simplified language_ as an umbrella
term, including many (and often language-specific) varieties such as _Easy
Language_ and _Plain Language_ (Maaß, 2020). include persons with intellectual
disabilities, persons with dementia, prelingually deaf persons, and non-native
readers (Maaß, 2020). In recent years, both the demand for simplified texts
and the amount of available data have been growing. Therefore, the development
of quantitative human evaluation methods which include and represent the
primary target groups becomes increasingly important. This is all the more
true with increasing numbers of automatic text simplification (ATS) models
being developed (Al-Thanyyan and Azmi, 2021). However, current research in ATS
mostly resorts to evaluations based on opinions of experts (e.g., simplified
language professionals) or crowdworkers who are not part of the primary target
groups.

Meanwhile, the use of information and communication technology, and mobile
touchscreen devices in particular, is becoming an integral part of the daily
lives of persons in these target groups (Ramsten et al., 2018; Menéndez
Álvarez-Dardet et al., 2020). This offers the potential of conducting human
evaluations with target groups of simplified language in digital form. Apart
from the efficiency gain in data collection and analysis compared to paper-
and-pencil methods, digital assessment methods also allow participants to read
texts in a more natural environment (possibly at home, on their own device),
and make it possible to record detailed user interactions, enabling
measurements such as reading speed or scrolling interactions as proxies for
reading comprehension. However, there is currently little research on the most
suitable methods for measuring text comprehensibility among the different
target groups as well as on the effects of text simplification on these
measurements. This issue also fundamentally concerns human-computer
interaction, since persons with intellectual disabilities differ not only in
their reading skills but also in their requirements for accessible user
interfaces (Braun et al., 2020).

The aim of the present study is to explore different ways of utilizing digital
tools for measuring comprehensibility. We determine the _comprehensibility_
(sometimes also referred to as _readability_) of a text by measuring its
_comprehension_ on the part of members of a specific group of readers, while
taking into account the fact that comprehensibility may differ between these
groups. To discuss the suitability of these methods for evaluating ATS, we
will also investigate the effect of the automatic simplification process on
these measurements. More specifically, the study is guided by the following
three research questions:

  1. (1)

Which methods for measuring comprehensibility can distinguish between
simplified and non-simplified texts?

  2. (2)

What is the effect of manual and automatic text simplification on these
measurements?

  3. (3)

How do these effects differ between persons with intellectual disabilities (as
a primary target group of simplified language) and a control group of persons
without intellectual disabilities?

To answer these questions, we present results from an empirical study
including participants with and without intellectual disabilities, using
unsimplified, manually simplified (i.e., simplified by human experts), and
automatically simplified German texts. To the best of our knowledge, this is
the first study evaluating ATS for German with this target group.

##  2\. Related work

###  2.1. Human evaluation of automatic text simplification

While human evaluation is the preferred way of evaluating the quality of ATS
output, there is no consensus on best practices (Alva-Manchego et al., 2020,
2021; Štajner, 2021; Stodden, 2021). In recent ATS research where human
evaluation was used, the most commonly applied methods were Likert scale
ratings, usually for the categories _simplicity_ , _fluency_ /_grammaticality_
, and _adequacy_ /_meaning preservation_ (Ryan et al., 2023; Martin et al.,
2022; Štajner and Nisioi, 2018; Mallinson et al., 2020). Less commonly, text
comprehensibility or difficulty is evaluated using multiple-choice
comprehension questions (Alonzo et al., 2021; Leroy et al., 2013; Laban et
al., 2021) or free recall questions (Leroy et al., 2013). Reading behavior,
e.g., by measuring reading speed (Alonzo et al., 2021; Crossley et al., 2014;
Saggion et al., 2015; Rello et al., 2013), scrolling interactions (Gooding et
al., 2021a), or eye movements (Rello et al., 2013), is rarely considered.

In most cases, the participants of such comprehensibility studies are persons
without disabilities or crowdworkers without specific inclusion criteria, who
are not part of the primary target group of simplified language. This can be
problematic, because what is considered difficult varies between reader groups
(Gooding et al., 2021b; Štajner, 2021), and the requirements for text
simplification should not be considered universal (Gooding, 2022). Some
exceptions of studies assessing ATS output among the target groups include
experiments with deaf and hard-of-hearing adults (Alonzo et al., 2021),
persons with intellectual disabilities (Huenerfauth et al., 2009; Saggion et
al., 2015) or dyslexia (Rello et al., 2013), and language learners (Crossley
et al., 2014). Among these, Saggion et al. (2015) is the most similar to our
study, as they evaluated both manually and automatically simplified texts with
persons with Down syndrome based on comprehension questions and reading time,
in addition to an expert evaluation using Likert scale ratings. Their
quantitative results did not show significant differences in comprehensibility
between the different text versions, but they reported positive subjective
perception of the simplified texts among target readers. Our study differs
from this contribution in that it is fully digital, also making use of
recorded user interactions, and we conduct the same comprehension assessment
with persons with and without intellectual disabilities, which allows us to
compare its effectiveness between the two groups.

###  2.2. Comprehension of simplified language by persons with intellectual
disabilities

Fajardo et al. (2014) conducted a study with 28 students with intellectual
disability reading news articles in easy-to-read Spanish on paper, and
correlated response accuracy in literal and inferential comprehension
questions with linguistic measures such as word and sentence length. In a
pilot study by Saletta and Winberg (2019), 20 participants with intellectual
or developmental disabilities read English texts that had undergone (among
others) controlled manipulations reducing lexical and syntactic complexity.
They measured errors while reading aloud and comprehension question response
accuracy and found a significant effect on the former but not on the latter.
They also found a high variability in reading comprehension among
participants.

For German, several studies have investigated the effect of specific features
of simplified language on comprehension by persons with intellectual
disabilities. Schiffl (2020) conducted an experiment using eye-tracking with
more than 80 participants, investigating the effects of word length and
frequency. They found fundamental differences in eye movements while reading
between persons with and without intellectual disabilities. Pappert and Bock
(2019) studied compound segmentation (a feature in several varieties of
simplified German) using a lexical decision task with participants with
intellectual disability or functional illiteracy. Bock and Lange (2017) tested
sentence and text comprehension skills of 28 persons with intellectual
disabilities and showed that certain phenomena that are assumed to be too
difficult for this target group (such as negation and personal pronouns)
hardly caused any problems for the participants.

More generally, reading comprehension by target groups of simplified language
has been studied by Jones et al. (2006), using several (adapted) standardized
tests with participants with mild and borderline learning disabilities. This
study revealed that the participant group was highly heterogeneous with
respect to reading comprehension abilities.

###  2.3. Digital assessment of reading comprehension

As mentioned in Section 1, digital assessment has the advantage of enabling
measurements of reading behavior even without expensive equipment and
expertise necessary for eye-tracking experiments. Some previous work has
studied the connection between behavioral measurements such as reading speed
or scrolling behavior and comprehension (Dyson and Haselgrove, 2000; DYSON and
HASELGROVE, 2001; Wallot et al., 2014; Støle et al., 2020), but there is only
little research on exploiting these measurements for assessing comprehension
or comprehensibility (Gooding et al., 2021a). Our work contributes to this
line of research by studying reading speed and response time as proxies for
comprehension.

While there is a relatively large body of literature both in human-computer
interaction and in language assessment dealing with differences in
comprehension and behavior when reading on digital devices compared to paper
(Al-Sulaimi and Al-Shihi, 2017; Chen and Catrambone, 2015; Kim and Kim, 2013;
Kong et al., 2018; Säuberli et al., 2023), almost no research has been
conducted on how digital reading assessments need to be adapted for persons
with intellectual disabilities. This is a significant research gap, given that
these user groups have very different needs in terms of interface
accessibility (Braun et al., 2020). By comparing different assessment
approaches between readers with and without intellectual disabilities, the
present paper represents a first step towards addressing this research gap.

##  3\. Materials and methods

###  3.1. Texts and comprehension questions

The texts used in this study originate from a parallel corpus of original and
simplified German documents. The documents were created at _capito_ , a
provider of commercial text simplification services for German. Each document
in the corpus was manually simplified by trained experts into one to three
levels of simplification following the levels of the Common European Framework
of Reference for Languages (A1, A2, and B2) (Council of Europe, 2020). All
manual simplifications used in this study are at level A2. This means that
most of the information from the original text is retained (i.e., there is
little to no summarization involved, as would be expected on a level of A1),
but simpler syntactic structures and vocabulary are used, complex terminology
is explained either inline or at the end of the text, and the layout is more
readable, e.g., using bullet point lists and shorter line lengths. Level A2 is
roughly comparable to Easy Language (in German: _Leichte Sprache_), for which
persons with intellectual disabilities are commonly listed as a primary target
group (Capito, 2023; Bock, 2014).

We used a subset of this parallel corpus to train a neural ATS system (fine-
tuned _mBART_ transformer model (Liu et al., 2020)) using the method described
in Rios et al. (2021). From the remaining documents, we selected twelve texts
according to several criteria: (1) The texts should be between 100 and 600
words in length, (2) they should cover a diverse range of topics but exclude
topics known to be familiar to a wide audience, and (3) the texts should not
require extensive additional context for comprehension. For each of the twelve
documents, we generated an automatic simplification at the A2 level using the
trained model and created four multiple-choice comprehension questions. The
first question was always “What is the text mainly about?” (four answer
options, one correct), the remaining three questions were about specific
details present in the text (three answer options, one correct). We created
these questions such that they can be answered based on the original and the
manually simplified text, without looking at the automatic simplifications in
order to avoid an unfair bias in favor of the system output. Since the ATS
model sometimes erroneously omits information present in the original, the
latter three questions have an additional fourth answer option “Information
does not appear in the text”. Care was taken that the questions are
unambiguous, independent of each other (i.e., being able to answer a question
was not contingent on getting the correct answer to a previous question), and
unanswerable using world knowledge alone. Each question was double-checked for
these criteria by two co-authors.

###  3.2. Participants

We recruited two groups of participants from different populations, described
in the following. All participants took part on a voluntary basis and were
compensated monetarily.

####  3.2.1. Target group

After approval by the institutional ethics review board, we recruited 18
participants from an educational program for persons with intellectual
disabilities in Austria. Eight were female and ten were male, and they were
aged between 18 and 32 (median: 23) at the time of recruitment. All
participants had some form of cognitive impairment (most commonly: autism
spectrum disorder, Down syndrome, or developmental delay), and a degree of
disability of at least 50% according to regulations concerning the assessment
of the degree of disability in Austria222BGBl. II Nr. 261/2010,
https://www.ris.bka.gv.at/eli/bgbl/II/2010/261/20100818. Therefore, these
participants represent a primary target group for simplified language. All
participants were legally allowed to sign the consent forms themselves.

In a questionnaire, which all participants filled in before the first session,
seven participants stated that they read texts in simplified language at least
once per week, five at least once per month. A total of 17 stated that they
used a touchscreen device on a daily basis, one person only weekly. Three
participants did not list German as their native language, but all have
completed compulsory education in German and are proficient at the CEFR level
of A2 or higher.

####  3.2.2. Control group

To compare the effects of text simplification on people outside the primary
target groups, we additionally recruited 18 people without cognitive
impairment—mostly current or former students—through university mailing lists.
Twelve were female, six were male, and they were aged between 20 and 36
(median: 25). All were native German speakers.

Unlike in the target group, most participants in the control group were not
used to reading simplified language (only 6 participants indicated reading
simplified texts at least once per month). However, the information and
consent forms which the participants received before the study were written in
A2 simplified language to establish a basic level of familiarity.

###  3.3. Procedure

All experiments were conducted using the _Okra_ app ((Säuberli et al., 2023);
version `0.3.1-alpha`) on Apple iPads (9.7-inch). _Okra_ is an app for
conducting reading experiments on mobile touchscreen devices, and it was
specifically designed for and tested with users with intellectual
disabilities. For instance, it reduces the complexity of the user interface
and the amount of text on screen in order to decrease the cognitive load
(Säuberli et al., 2023).

Each participant took part in three sessions on separate days. The target
group sessions took place at the facilities of the educational program, the
control group sessions in a university seminar room. Each session consisted of
reading tasks, two sessions also included cognitive tasks. The app presented
all instructions and guided participants through the entire session such that
several participants could be tested simultaneously without interruptions.
Each control group session included up to 12 participants, whereas for the
target group, only up to 5 individuals participated per session. This was
intended to provide better support in case of problems and to shorten waiting
times, as reading speeds varied widely in the target group. One or two test
administrators were present in the room and available for questions.

Before the main study, we conducted a usability test with 3 people from the
same educational program to improve the usability and accessibility of the
instructions and tasks implemented in the app. After finalizing the material
and procedure, we piloted the entire experiment with 3 participants from the
target group. Participants in the usability test and the pilot study were not
recruited for the main study.

####  3.3.1. Cognitive tasks

We included a total of four tasks testing several low-level cognitive skills
related to reading. The purpose of these tasks was to provide a basic
understanding of some of the differences between the two groups and the
heterogeneity within each group. The tests we used were adaptations of tasks
commonly used in psychological research (see references below). We adapted the
tasks to the target group (by adjusting the difficulty and number of trials
based on results from the usability test) and to the technical setup in the
present study (by making the interface usable on a touchscreen).

  * •

Digit span: Memorizing and repeating an increasingly long sequence of digits
in the same order (Wechsler, 2003); two trials, each of which ended after two
consecutive mistakes; measurement: longest correctly repeated sequence. This
task tests short-term memory, sequencing ability, attention and automated
learning (Wechsler, 2003).

  * •

Lexical decision: Deciding as quickly as possible whether the displayed
strings of characters are words or pseudowords (Meyer and Schvaneveldt, 1971);
37 stimuli; measurements: reaction time on correctly recognized words, ratio
of correct responses. This task tests vocabulary knowledge and lexical access.

  * •

Reaction time: Tapping randomly appearing balloons as quickly as possible; 15
stimuli; measurement: mean time between stimulus appearance and tap. Apart
from motor aspects, reaction time also depends on cognitive factors such as
visual processing speed and attention (Amini Vishteh et al., 2019).

  * •

Trail making: Tapping randomly positioned numbers in ascending order as
quickly as possible (Reitan and Wolfson, 1993); 3 trials; measurement: mean
time between taps. We only included part A of the trail making task, which
primarily assesses visual attention and psychomotor speed (Arnett and
Labovitz, 1995).

Each task was preceded by a practice task, which participants could optionally
repeat and whose results were excluded from the analysis.

####  3.3.2. Reading tasks

Participants read four texts per session. The texts were presented in one of
three versions (original, manually simplified, automatically simplified). No
participant read the same text in more than one version. The design was
counterbalanced so that all texts in all versions were read by the same number
of participants in both groups. After reading the text, participants were
asked to rate the text’s difficulty on a 5-point scale (1 = _very difficult_ ,
5 = _very easy_), whereby the level descriptions were marked with textual
labels, colors, and emoticons333We are aware that the interpretation of facial
expressions, including emoticons, can differ between individuals. We used
redundant coding with text, colors, and emoticons in the interface to avoid
ambiguity, while also reducing the amount of text visible on screen.. The text
was then displayed again, along with the comprehension questions. Only one of
four questions was shown at a time, and participants could switch back and
forth between questions until they submitted their final answers. The
screenshots in Figure 1 show this procedure for one text. After finishing the
text, participants were asked to take a break if necessary, and then continue
with the next text.

[Screenshots of the reading task]Three screenshots of the reading task. On the
first screenshot, there is a German text in a scrollable white box in the
center, surrounded by a gray background and a blue button at the bottom saying
“Continue” in German. In the second screenshot, the question “How difficult
was the text for you?” in German is shown in the center and five emoticons
(frowning and red on the left, smiling and blue on the right); the leftmost
emoticon is labeled “very difficult”, the rightmost “very easy”; beneath,
there is a button saying “Continue” in German. In the third screenshot, the
text is shown again in a white box, and a panel containing a comprehension
question with four answer options next to radio buttons below.

Figure 1. Screenshots of the reading task in _Okra_. (1) Initial reading
screen, where only the text is visible. (2) Text difficulty rating screen. (3)
Comprehension question screen; tapping the arrow buttons switches between
questions.

Apart from the responses, we also recorded timestamped user interactions such
as reading times and scrolling interactions. In the present paper, we will
focus on the following measurements:

  * •

Responses to comprehension questions [with our assessment: correct/incorrect]

  * •

Responses to text difficulty ratings [1–5]

  * •

Time taken to answer each question, i.e., the total time during which the
question was visible to the participant [seconds]

  * •

Reading speed when initially reading the text [words per minute, WPM]

Since the ATS model sometimes does not transfer all information accurately and
we designed the comprehension questions without looking at the ATS output, the
correct answers in the automatic simplification could be different from the
other versions. For example, the ATS model at times deleted a sentence from
the original which included relevant information for answering a question,
changing the correct answer for this question with respect to the
automatically simplified text to “Information does not appear in the text”.
Therefore, we manually recoded the answer correctness for the automatically
simplified texts. We removed instances where the correct answer in the
automatic simplification was “Information does not appear in the text” in
order not to give the ATS model an unfair advantage in the analysis. In total,
we removed 9 out of 48 questions from the results of the automatically
simplified texts.

###  3.4. Statistical analysis

When analyzing responses to comprehension questions or ratings, we took into
account that some participants may be more proficient than others, and some
questions may be more difficult to answer than others. To model these
differences, we analyzed our data using the Rasch model, also called the one-
parameter logistic model in item response theory (IRT). These models are
widely used in language assessment and psychometrics, and formally comparable
to generalized linear models with (fixed or random) effects for persons and
items (Fox, 2010, p. 143–145)(De Boeck et al., 2011). Whereas the classic
Rasch model only considers persons and items in the analysis, the many-facets
Rasch model allowed us to also take additional parameters (so-called _facets_)
into account in the modeling of the data (Linacre, 1989). As we were
interested in the effects of the text version (original, manually simplified,
automatically simplified) on participants’ performance, we specified a many-
facets Rasch model with three facets (persons, items, and text version). We
used the estimated parameter values (the “latent traits”) of the text version
facet to compare the effect of manual and automatic text simplification.

We applied a dichotomous Rasch model for the comprehension questions (Fox,
2010, p. 7–9) (equivalent to logistic regression) and a graded response model
for the difficulty rating (Samejima, 1997). For modeling response time and
reading speed, we used log-linear regression models as in (Van der Linden et
al., 1999) and (Fox, 2010, p. 228–231), fitting person, item, and text version
parameters in the same way as for the Rasch models. All models are defined in
Table 1.

Measurement | Range | Model definition
---|---|---
Comprehension question accuracy | $y_{p,q,v}\in\\{0,1\\}$ | $P(y_{p,q,v}=1)=\operatorname{logit}^{-1}(\mu+\alpha_{p}-\beta_{q}-\delta_{v})$
Perceived difficulty ratings | $y_{p,t,v}\in\\{1,2,3,4,5\\}$ | $\begin{aligned} P(y_{p,t,v}=k)=&\operatorname{logit}^{-1}(\mu+\alpha_{p}-\gamma_{t}-\delta_{v}-c_{k})\\\ &-\operatorname{logit}^{-1}(\mu+\alpha_{p}-\gamma_{t}-\delta_{v}-c_{k+1})\end{aligned}$
Response time | $y_{p,q,v}\in[0,\infty)$ |  $\log(y_{p,q,v})=\mu+\alpha_{p}+\beta_{q}+\delta_{v}+\epsilon$, $\epsilon\sim\mathcal{N}(0,\sigma)$
Reading speed | $y_{p,t,v}\in[0,\infty)$ |  $\log(y_{p,t,v})=\mu+\alpha_{p}+\gamma_{t}+\delta_{v}+\epsilon$, $\epsilon\sim\mathcal{N}(0,\sigma)$
Table 1. Model definitions for each measurement in the reading tasks.
$\alpha_{p}$, $\beta_{q}$, $\gamma_{t}$, $\delta_{v}$ are parameters for a
given person $p$, comprehension question $q$, text $t$, and text version $v$,
$\mu$ and $\sigma$ are parameters for mean and standard deviation, and $c_{k}$
is a parameter for the threshold of each rating category $k$ (fixed to
$-\infty$ for $k=1$).

We used Bayesian inference with a Markov chain Monte Carlo (MCMC) algorithm
for fitting the models. This has several advantages compared to frequentist
statistics: We get posterior distributions for parameter values, which provide
more information than point estimates, it allows including prior knowledge,
and Bayesian models are usually more accurate for complex IRT models and small
sample sizes (Fox, 2010, p. 2)(Gao and Chen, 2005)(You, 2022). We defined wide
normal distributions as priors for person, question/document, and text version
parameters (cf. Table 2). For each measurement, we fit two separate models for
the target and control groups, since we did not want to generalize across the
different populations they are sampled from.

We used _Stan_ (Carpenter et al., 2017) with the _PyStan_ interface (Riddell
et al., 2021) for sampling and _ArviZ_ (Kumar et al., 2019) for analysis. For
MCMC, we used 4 chains with 2000 iterations, including 1000 warmup iterations.
Model code and convergence diagnostics are published in the supplementary
material.

##  4\. Results

Anonymized data and code for reproducing the analyses are available in the
supplementary material. Five participants did not consent to publishing their
anonymized raw data. Therefore, the data for these participants is not
included in the supplementary material. The numbers and plots in the paper are
based on the complete data.

###  4.1. Cognitive tasks

Figure 2 compares the measurements from the cognitive tasks between the target
and control groups. The largest difference is in the digit span task for
measuring working memory, with median scores of 4.5 for the target group and 7
for the control group. We also measured a longer reaction time in the lexical
decision task, longer reaction times in general, and slower trail making in
the target group. Moreover, variability in the target group is generally much
higher than in the control group, which is likely to affect results in reading
behavior and comprehension (Johann et al., 2020).

[Boxplots of cognitive measurements]Boxplots for each of the five measurements
from the cognitive tasks, each comparing target and control group. First
boxplot: longest correct sequence in the digit span task; interquartile
ranges: target group between 4 and 5, control group between 7 and 7.75. Second
boxplot: Correct word reaction time in seconds in the lexical decision task;
interquartile ranges: target group between 0.94 and 1.50, control group
between 0.73 and 0.87. Third boxplot: Correct response ratio in the lexical
decision task; interquartile ranges: target group between 0.92 and 1.00,
control group between 0.97 and 1.00. Fourth boxplot: Reaction time in seconds
in the reaction time task; interquartile ranges: target group between 0.58 and
0.77, control group between 0.52 and 0.56. Fifth boxplot: Time per tap in
seconds in the trail making task; interquartile ranges: target group between
0.94 and 1.45, control group between 0.45 and 0.68.

Figure 2. Boxplots of the measurements from the cognitive tasks, compared
between target and control group. Each data point is the measured values for a
single participant aggregated across all trials/stimuli (maximum for digit
span, mean for all others), excluding practice trials.

###  4.2. Reading tasks

In total, 1680 responses to comprehension questions (excluding the 108
responses to unanswerable questions in the automatically simplified versions,
see Section 3.3.2) and 432 difficulty ratings are included in the analysis.

The estimated effects of the three text versions (original, manually
simplified, automatically simplified) on the four measurements are visualized
in Figure 3. Effects are centered around zero, and parameters for the two
groups were estimated independently (as explained in Section 3.4), therefore
the estimates cannot be compared across groups. We calculate the distribution
of the difference between the three text version parameters at each MCMC
sampling step and use highest density intervals (HDI) to quantify the
credibility of the difference between the text version effects.

[Point plots comparing comprehension question response accuracy between text
versions]Two point plots (one for the target group, one for the control
group), each comparing the estimated difficulty between the three text
versions, with error bars for credible intervals.

(a) Comprehension question response accuracy.

[Point plots comparing perceived difficulty ratings between text versions]Two
point plots (one for the target group, one for the control group), each
comparing the estimated difficulty between the three text versions, with error
bars for credible intervals.

(b) Perceived difficulty rating.

[Point plots comparing comprehension question response time between text
versions]Two point plots (one for the target group, one for the control
group), each comparing the estimated effect on response time between the three
text versions, with error bars for credible intervals.

(c) Comprehension question response time.

[Point plots comparing reading speed between text versions]Two point plots
(one for the target group, one for the control group), each comparing the
estimated effect on reading speed between the three text versions, with error
bars for credible intervals.

(d) Reading speed.

Figure 3. Posterior distributions of the text version parameters for the four
measurements in the reading task. Points are medians, error bars are 80%, 90%,
and 95% credible intervals (CI). A bracket with $\blacktriangle$ indicates
that the 80% CI of the difference between the two parameters does not include
zero (i.e., we are 80% confident that there is a difference). Similarly with
$\blacktriangle\blacktriangle$ for 90% CI and
$\blacktriangle\blacktriangle\blacktriangle$ for 95% CI.

####  4.2.1. Comprehension questions

Overall, the target group answered 47.5% of the questions correctly, whereas
the control group answered 92.8% correctly. In the control group, 25 questions
were answered correctly by all participants, and one participant answered all
48 questions correctly. In other words, in the control group, about half of
the questions were uninformative because they were too easy and therefore
unable to discriminate between more and less proficient readers and between
more and less difficult text versions. This ceiling effect means that the
parameter estimates of the Rasch model are less precise in the control group,
as the wider credible intervals in Figure 3(a) show. Still, the estimated
difficulty of the manual simplifications is measurably lower than both the
original ($\text{CI}_{95\%}=[0.15,1.61]$) and the automatic simplifications
($\text{CI}_{95\%}=[0.49,2.01]$), meaning that participants had a
significantly higher probability of answering questions correctly with the
manually simplified version. The automatic simplifications appear to have been
slightly more difficult than the originals ($\text{CI}_{80\%}=[0.01,0.86]$).
In the target group, the effects are less pronounced, the original being the
most difficult and the manual simplification the least difficult.

####  4.2.2. Perceived difficulty ratings

In Figure 3(b), again, the differences between text versions are much smaller
in the target group compared to the control group. The target group seems to
rate the automatically simplified texts slightly easier than the originals
($\text{CI}_{90\%}=[0.08,1.06]$), whereas the control group rated the
automatically simplified texts on par with the unsimplified ones. The control
group had a strong tendency to rate the manually simplified texts as less
difficult than the original ($\text{CI}_{95\%}=[1.22,2.58]$) and the
automatically simplified texts ($\text{CI}_{95\%}=[1.26,2.60]$).

####  4.2.3. Comprehension question response times

In response time models, a larger effect means a longer response time, which
is generally associated with a higher item difficulty in tests (Van der Linden
et al., 1999). In the target group, from Figure 3(c) we can observe that
manual simplifications led to slightly faster response times
($\text{CI}_{80\%}=[0.02,0.18]$), while the automatic simplifications are on
par with the originals. In the control group, the differences are even
stronger, and the automatic simplifications appear to have been the most
difficult. The effects on response time (Figure 3(c)) look very similar to the
effects on response accuracy (Figure 3(a)). This is in line with research on
psychological research on test design (Van der Linden et al., 1999), but in
our case, the observations from the two groups of participants do not agree on
the relative difficulty of the automatically simplified texts.

####  4.2.4. Reading speed

In terms of reading times, the behavior of the target group was much more
variable and less predictable, as becomes obvious from Figure 3(d). Some
participants had implausible reading speeds of up to thousands of words per
minute, meaning that many only skimmed or even skipped reading the text the
first time it was displayed. We found that a small number of target group
participants had a stronger tendency towards skimming or skipping, but most of
them did not do so consistently, and reading speeds were not distributed
bimodally, such that there was no obvious threshold to discriminate between
reading and skipping. The slowest reading speeds (50 WPM and slower) were also
observed in the target group. Mean reading speeds were 203 WPM in the target
group and 168 WPM in the control group. For comparison, a standardized
assessment of reading speed reported a mean of 179 WPM for native German
speakers (Trauzettel-Klosinski et al., 2012).

In the target group, the original texts tended to be read (or, more plausibly,
skipped) more quickly than the two simplified versions (manual:
$\text{CI}_{80\%}=[0.04,0.42]$, automatic: $\text{CI}_{80\%}=[0.05,0.43]$),
while in the control group, the automatic simplifications were read more
slowly than the other two (original: $\text{CI}_{95\%}=[0.02,0.20]$, manual:
$\text{CI}_{95\%}=[0.06,0.24]$).

##  5\. Discussion

The primary goal of this study was to investigate four different measurement
methods, comparing them with regards to two different text simplification
methods and two different reader groups, with the ultimate aim of improving
methods for human evaluation of ATS. We will discuss these aspects mainly
based on the results in Figure 3. The purpose of the cognitive tasks was to
characterize the participant groups to support the interpretation of results.
Therefore, we will not discuss them in further detail here.

###  5.1. Comparison of measurement methods and reader groups

By design, there are several fundamental differences between the four
measurement methods: Comprehension questions measure objective comprehension,
while difficulty ratings measure subjective perception. Measurements such as
response time and reading speed can only serve as proxies for comprehension
and require specific assumptions about the behavior of participants. For any
of these measurements to be considered suitable for evaluating text
simplification, they need to be able to capture a difference between less
comprehensible and more comprehensible texts. Since the manually simplified
texts were professionally edited by trained experts and according to
guidelines developed and checked with target readers, it is safe to assume
that there should be some measurable difference in comprehensibility between
original and manually simplified texts. From this perspective, our results
suggest that the measurement of comprehension question response accuracy was
most successful, and perceived difficulty ratings were least successful with
the target group. For the control group, all measurements except reading speed
were successful in differentiating between original and manually simplified
texts.

There are two possible factors which may explain why ratings were less
reliable than comprehension questions for the target group: First, the target
group was quite heterogeneous (as evidenced by the cognitive tasks), which led
to larger differences in subjective judgments of texts, especially because we
did not give more specific instructions to calibrate ratings in order to
reduce cognitive load. Second, when readers lose motivation and stop reading,
which happened in the target group, rating responses may be more random,
whereas responses to comprehension questions will reliably show a random-
guessing accuracy. Both of these may be arguments against using perceived
difficulty ratings with the target group.

Familiarity is a confounding factor, because participants in the target group
were mostly very familiar with the specific variety of simplified language in
the study, which may have biased their perception of the texts. However, if
this bias was strong, we would expect a larger effect in the perceived
difficulty ratings compared to the control group participants, who were mostly
unfamiliar with simplified language.

While previous work heavily relied on ratings for evaluating the
comprehensibility of simplified texts (see Section 2.1), our results show that
this is not always sufficient, especially for readers with intellectual
disabilities. The results also revealed significant differences in
comprehensibility and perception between persons with and without intellectual
disabilities, highlighting the importance of including the primary target
groups in studies on text simplification and bridging the gap to insights from
psycholinguistic research (see Section 2.2).

Although reading speed was mostly unsuccessful in discriminating between text
versions, it revealed important behavioral patterns in the target group
(skimming/skipping texts), which supports the interpretation of other results.
Previous work has suggested that reading time is to be considered separately
from comprehension (Wallot et al., 2014). Our observations support this view,
but our interpretation is limited by our study design: Since the text was
shown again after the initial reading, participants were free to decide not to
read the entire text the first time around.

Overall, the fact that reading behavior can be measured through a mobile
application is a major advantage of using digital evaluation tools such as the
one described in this paper compared to paper-and-pencil assessment. Our work
represents a first step towards exploiting this advantage to make
comprehensibility assessment more inclusive (see Section 2.3).

###  5.2. Effect of automatic simplification

We have seen that manual simplification resulted in noticeable differences for
most measures. By comparing the difficulty estimates in the automatic
simplification to the original and manual simplification, we can evaluate the
ATS output in terms of comprehensibility.

Based on the target group measurements, automatic simplification only had a
modest effect. The largest improvement compared to the original texts was
observed in the perceived difficulty ratings, which were generally less
reliable with this group, as discussed in Section 5.1. However, in terms of
reading speed, ATS had the same effect as manual simplification, which
suggests that ATS was somewhat successful in keeping up motivation to continue
reading for the target group. A possible explanation for this is that at the
surface level, the texts looked more like the simplified texts the
participants were familiar with.

In the control group, all measurements agreed that ATS outputs were equally
difficult or more difficult than the original text. Apart from lack of quality
in the automatic simplifications, several factors may have contributed to
these results: First, as described in Section 3.1, the comprehension questions
were written and optimized for the original and manually simplified texts.
Although we removed responses to questions which were not answerable based on
the ATS output, the wording of the questions may still have made the questions
more difficult in the automatic simplification. Second, the control group may
be more perceptive or sensitive to grammatical and semantic errors in the text
than the target group. Evidence for this are the control group’s higher
difficulty ratings and lower reading speed for the automatically simplified
texts.

Particularly the second factor requires further experimental research, as it
could have a major influence on human evaluation of text simplification: If
the linguistic fluency of ATS output only has a weak influence on
comprehension in primary target groups of simplified texts, this should be
accounted for when evaluating ATS systems. In addition, different types of
linguistic errors may have a different effect on reading behavior depending on
the specific type of cognitive impairment of the reader (Micai et al., 2019).

In the present work, we focused on the evaluation methodology as opposed to
pinpointing specific problems in the ATS output. However, our findings on the
comprehensibility of automatically simplified texts are in line with current
research showing that ATS systems are still quite limited in the effective
gain of simplicity they can achieve (Ryan et al., 2023). Recent experiments
with large instruction-tuned language models have already suggested
significant improvements in this regard, and these models would likely
outperform our models (Maddela et al., 2023; Kew et al., 2023). It is all the
more important that these improvements are evaluated with primary target
reader groups in the future.

##  6\. Conclusion

We conducted a study exploring different ways of measuring text
comprehensibility using a mobile application and investigating the effect of
manual and automatic text simplification on comprehension, including
participants with and without intellectual disabilities. The results revealed
several types of differences which must be taken into account when designing
human evaluation studies:

  * •

Differences between measurement methods: Comprehension questions, difficulty
ratings, and behavioral measurements can lead to different conclusions and
complement each other when combined.

  * •

Differences between manual and automatic simplification: Issues in the ATS
output may significantly impair objective comprehension without affecting
subjective perception (especially in the target group), whereas manually
simplified texts lead to more predictable results across measurement methods.

  * •

Differences between reader groups: Results from persons with intellectual
disabilities can be different from (or even in contradiction to) those of
persons without disabilities, particularly in terms of reading behavior and
subjective perception of difficulty.

We consider measuring interactions of users reading on touchscreen devices to
be a promising approach, especially for assessing comprehensibility with
diverse target groups, as traditional tests based on comprehension questions
can be cognitively demanding. Another advantage is that this approach allows
assessing reading behavior in a more natural environment. However, further
research is still required on other aspects of human-computer interaction,
e.g., regarding the exact relationship between user interactions and text
comprehension, the ways in which interactions differ between persons with and
without intellectual disabilities, and how they can be used to design more
reliable and accessible comprehensibility assessments with diverse user
groups.

Overall, we show that applying digital assessment methods for
comprehensibility evaluation to persons with intellectual disabilities is
viable, and that combining subjective and objective comprehensibility
assessment with behavioral measurements provides valuable insights into the
impact of text simplification.

###### Acknowledgements.

We are greatly indebted to CFS GmbH (capito), in particular to Ursula
Semlitsch, for recruiting and running the sessions with the target group
participants, and helping to translate the material into simplified language.
We would also like to thank Silke Gutermuth for helpful feedback on the
experiment design and the comprehension questions. We thank the anonymous
reviewers for their constructive comments. This work was funded by the Swiss
Innovation Agency (Innosuisse) Flagship IICT (PFFS-21-47).

## References

  * (1)
  * Al-Sulaimi and Al-Shihi (2017) Aisha Al-Sulaimi and Hafedh Al-Shihi. 2017.  The effects of reading mode (digital vs printed text) on reading comprehension: A literature review of the key assessment factors. In _2017 6th International Conference on Information and Communication Technology and Accessibility (ICTA)_. 1–6.  https://doi.org/10.1109/ICTA.2017.8336053
  * Al-Thanyyan and Azmi (2021) Suha S. Al-Thanyyan and Aqil M. Azmi. 2021.  Automated Text Simplification: A Survey.  _ACM Comput. Surv._ 54, 2, Article 43 (mar 2021), 36 pages.  https://doi.org/10.1145/3442695
  * Alonzo et al. (2021) Oliver Alonzo, Jessica Trussell, Becca Dingman, and Matt Huenerfauth. 2021.  Comparison of Methods for Evaluating Complexity of Simplified Texts among Deaf and Hard-of-Hearing Adults at Different Literacy Levels. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–12.
  * Alva-Manchego et al. (2020) Fernando Alva-Manchego, Carolina Scarton, and Lucia Specia. 2020\.  Data-Driven Sentence Simplification: Survey and Benchmark.  _Computational Linguistics_ 46, 1 (2020), 135–187.  https://doi.org/10.1162/coli_a_00370
  * Alva-Manchego et al. (2021) Fernando Alva-Manchego, Carolina Scarton, and Lucia Specia. 2021\.  The (Un)Suitability of Automatic Evaluation Metrics for Text Simplification.  _Computational Linguistics_ 47, 4 (Dec. 2021), 861–889.  https://doi.org/10.1162/coli_a_00418
  * Amini Vishteh et al. (2019) Rasoul Amini Vishteh, Ali Mirzajani, Ebrahim Jafarzadehpour, and Samireh Darvishpour. 2019.  Evaluation of simple visual reaction time of different colored light stimuli in visually normal students.  _Clinical Optometry_ (2019), 167–171.
  * Arnett and Labovitz (1995) James A Arnett and Seth S Labovitz. 1995.  Effect of physical layout in performance of the Trail Making Test.  _Psychological Assessment_ 7, 2 (1995), 220\.
  * Bock (2014) Bettina M. Bock. 2014\.  _“Leichte Sprache”: Abgrenzung, Beschreibung und Problemstellungen aus Sicht der Linguistik_.  Frank & Timme, Berlin, 17–51.
  * Bock and Lange (2017) Bettina M. Bock and Daisy Lange. 2017.  _Empirische Untersuchungen zu Satz- und Textverstehen bei Menschen mit geistiger Behinderung und funktionalen Analphabeten_.  Frank & Timme, Berlin, 253–274.
  * Braun et al. (2020) Melinda Braun, Matthias Wölfel, Gregor Renner, and Christian Menschik. 2020.  Accessibility of Different Natural User Interfaces for People with Intellectual Disabilities. In _2020 International Conference on Cyberworlds (CW)_. 211–218.  https://doi.org/10.1109/CW49994.2020.00041
  * Capito (2023) Capito. 2023.  Easy Language: What is it and why is it important?  https://capito.eu/en/easy-language/ Accessed: 2023-12-07.
  * Carpenter et al. (2017) Bob Carpenter, Andrew Gelman, Matthew D. Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Marcus Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. 2017\.  Stan: A Probabilistic Programming Language.  _Journal of Statistical Software_ 76, 1 (2017), 1–32.  https://doi.org/10.18637/jss.v076.i01
  * Chen and Catrambone (2015) Dar-Wei Chen and Richard Catrambone. 2015.  Paper vs. Screen: Effects on Reading Comprehension, Metacognition, and Reader Behavior.  _Proceedings of the Human Factors and Ergonomics Society Annual Meeting_ 59, 1 (2015), 332–336.  https://doi.org/10.1177/1541931215591069 arXiv:https://doi.org/10.1177/1541931215591069
  * Council of Europe (2020) Council of Europe. 2020\.  _Common European framework of reference for languages: Learning, teaching, assessment. Companion volume_.  Council of Europe Publishing, Strasbourg.
  * Crossley et al. (2014) Scott A Crossley, Hae Sung Yang, and Danielle S McNamara. 2014\.  What’s so Simple about Simplified Texts? A Computational and Psycholinguistic Investigation of Text Comprehension and Text Processing.  _Reading in a Foreign Language_ 26, 1 (2014), 92–113.
  * De Boeck et al. (2011) Paul De Boeck, Marjan Bakker, Robert Zwitser, Michel Nivard, Abe Hofman, Francis Tuerlinckx, and Ivailo Partchev. 2011.  The Estimation of Item Response Models with the lmer Function from the lme4 Package in R.  _Journal of Statistical Software_ 39, 12 (2011), 1–28.  https://doi.org/10.18637/jss.v039.i12
  * Dyson and Haselgrove (2000) Mary Dyson and Mark Haselgrove. 2000.  The effects of reading speed and reading patterns on the understanding of text read from screen.  _Journal of Research in Reading_ 23, 2 (2000), 210–223.  https://doi.org/10.1111/1467-9817.00115 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/1467-9817.00115
  * DYSON and HASELGROVE (2001) MARY C. DYSON and MARK HASELGROVE. 2001.  The influence of reading speed and line length on the effectiveness of reading from screen.  _International Journal of Human-Computer Studies_ 54, 4 (2001), 585–612.  https://doi.org/10.1006/ijhc.2001.0458
  * Fajardo et al. (2014) Inmaculada Fajardo, Vicenta Ávila, Antonio Ferrer, Gema Tavares, Marcos Gómez, and Ana Hernández. 2014.  Easy-to-read texts for students with intellectual disability: linguistic factors affecting comprehension.  _Journal of applied research in intellectual disabilities_ 27, 3 (2014), 212–225.
  * Fox (2010) Jean-Paul Fox. 2010\.  _Bayesian Item Response Modeling: Theory and Applications_.  Springer New York, New York, NY.  https://doi.org/10.1007/978-1-4419-0742-4
  * Gao and Chen (2005) Furong Gao and Lisue Chen. 2005.  Bayesian or Non-Bayesian: A Comparison Study of Item Parameter Estimation in the Three-Parameter Logistic Model.  _Applied Measurement in Education_ 18, 4 (oct 2005), 351–380.  https://doi.org/10.1207/s15324818ame1804_2
  * Gooding (2022) Sian Gooding. 2022\.  On the Ethical Considerations of Text Simplification. In _Ninth Workshop on Speech and Language Processing for Assistive Technologies (SLPAT-2022)_. Association for Computational Linguistics, Dublin, Ireland, 50–57.  https://doi.org/10.18653/v1/2022.slpat-1.7
  * Gooding et al. (2021a) Sian Gooding, Yevgeni Berzak, Tony Mak, and Matt Sharifi. 2021a.  Predicting Text Readability from Scrolling Interactions. In _Proceedings of the 25th Conference on Computational Natural Language Learning_. Association for Computational Linguistics, Online, 380–390.  https://doi.org/10.18653/v1/2021.conll-1.30
  * Gooding et al. (2021b) Sian Gooding, Ekaterina Kochmar, Seid Muhie Yimam, and Chris Biemann. 2021b.  Word Complexity is in the Eye of the Beholder. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, Online, 4439–4449.  https://doi.org/10.18653/v1/2021.naacl-main.351
  * Huenerfauth et al. (2009) Matt Huenerfauth, Lijun Feng, and Noémie Elhadad. 2009\.  Comparing evaluation techniques for text readability software for adults with intellectual disabilities. In _Proceedings of the 11th international ACM SIGACCESS conference on Computers and accessibility_. 3–10.
  * Johann et al. (2020) Verena Johann, Tanja Könen, and Julia Karbach. 2020\.  The unique contribution of working memory, inhibition, cognitive flexibility, and intelligence to reading comprehension and reading speed.  _Child Neuropsychology_ 26, 3 (2020), 324–344.  https://doi.org/10.1080/09297049.2019.1649381 PMID: 31380706.
  * Jones et al. (2006) FW Jones, K Long, and WML Finlay. 2006.  Assessing the reading comprehension of adults with learning disabilities.  _Journal of Intellectual Disability Research_ 50, 6 (2006), 410–418.
  * Kew et al. (2023) Tannon Kew, Alison Chi, Laura Vásquez-Rodríguez, Sweta Agrawal, Dennis Aumiller, Fernando Alva-Manchego, and Matthew Shardlow. 2023.  BLESS: Benchmarking Large Language Models on Sentence Simplification.  arXiv:2310.15773 [cs.CL]
  * Kim and Kim (2013) Hak Joon Kim and Joan Kim. 2013.  Reading from an LCD monitor versus paper: Teenagers’ reading performance.  _International Journal of Research Studies in Educational Technology_ 2, 1 (April 2013).  https://doi.org/10.5861/ijrset.2012.170
  * Kong et al. (2018) Yiren Kong, Young Sik Seo, and Ling Zhai. 2018.  Comparison of reading performance on screen and on paper: A meta-analysis.  _Computers & Education_ 123 (2018), 138–149.
  * Kumar et al. (2019) Ravin Kumar, Colin Carroll, Ari Hartikainen, and Osvaldo Martin. 2019.  ArviZ a unified library for exploratory analysis of Bayesian models in Python.  _Journal of Open Source Software_ 4, 33 (2019), 1143\.  https://doi.org/10.21105/joss.01143
  * Laban et al. (2021) Philippe Laban, Tobias Schnabel, Paul Bennett, and Marti A. Hearst. 2021.  Keep It Simple: Unsupervised Simplification of Multi-Paragraph Text. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Association for Computational Linguistics, Online, 6365–6378.  https://doi.org/10.18653/v1/2021.acl-long.498
  * Leroy et al. (2013) Gondy Leroy, James E Endicott, David Kauchak, Obay Mouradi, and Melissa Just. 2013.  User evaluation of the effects of a text simplification algorithm using term familiarity on perception, understanding, learning, and information retention.  _Journal of medical Internet research_ 15, 7 (2013), e144.
  * Linacre (1989) John Michael Linacre. 1989\.  _Many-faceted Rasch measurement_.  Ph. D. Dissertation. The University of Chicago.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020\.  Multilingual Denoising Pre-training for Neural Machine Translation.  _Transactions of the Association for Computational Linguistics_ 8 (2020), 726–742.  https://doi.org/10.1162/tacl_a_00343
  * Maaß (2020) Christiane Maaß. 2020\.  _Easy Language–Plain Language–Easy Language Plus: Balancing comprehensibility and acceptability_.  Frank & Timme.
  * Maddela et al. (2023) Mounica Maddela, Yao Dou, David Heineman, and Wei Xu. 2023\.  LENS: A Learnable Evaluation Metric for Text Simplification. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 16383–16408.  https://doi.org/10.18653/v1/2023.acl-long.905
  * Mallinson et al. (2020) Jonathan Mallinson, Rico Sennrich, and Mirella Lapata. 2020\.  Zero-Shot Crosslingual Sentence Simplification. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, Online, 5109–5126.  https://doi.org/10.18653/v1/2020.emnlp-main.415
  * Martin et al. (2022) Louis Martin, Angela Fan, Éric de la Clergerie, Antoine Bordes, and Benoît Sagot. 2022.  MUSS: Multilingual Unsupervised Sentence Simplification by Mining Paraphrases. In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_. European Language Resources Association, Marseille, France, 1651–1664.  https://aclanthology.org/2022.lrec-1.176
  * Menéndez Álvarez-Dardet et al. (2020) Susana Menéndez Álvarez-Dardet, Bárbara Lorence Lara, and Javier Pérez-Padilla. 2020.  Older adults and ICT adoption: Analysis of the use and attitudes toward computers in elderly Spanish people.  _Computers in Human Behavior_ 110 (2020), 106377.  https://doi.org/10.1016/j.chb.2020.106377
  * Meyer and Schvaneveldt (1971) David E. Meyer and Roger W. Schvaneveldt. 1971.  Facilitation in recognizing pairs of words: Evidence of a dependence between retrieval operations.  _Journal of Experimental Psychology_ 90, 2 (Oct. 1971), 227–234.  https://doi.org/10.1037/h0031564
  * Micai et al. (2019) Martina Micai, Mila Vulchanova, and David Saldaña. 2019\.  Do Individuals with Autism Change Their Reading Behavior to Adapt to Errors in the Text?  _Journal of Autism and Developmental Disorders_ 49, 10 (July 2019), 4232–4243.  https://doi.org/10.1007/s10803-019-04108-8
  * Pappert and Bock (2019) Sandra Pappert and Bettina M Bock. 2019.  Easy-to-read German put to the test: Do adults with intellectual disability or functional illiteracy benefit from compound segmentation?  _Reading and Writing_ (2019), 1–27.
  * Ramsten et al. (2018) Camilla Ramsten, Lene Karine Martin, Munir Dag, and Lena Marmstål Hammar. 2018.  Information and communication technology use in daily life among young adults with mild-to-moderate intellectual disability.  _Journal of Intellectual Disabilities_ 24 (2018), 289–308.
  * Reitan and Wolfson (1993) Ralph Reitan and Deborah Wolfson. 1993.  _The Halstead-Reitan neuropsychological test battery: Theory and clinical interpretation_.  Neuropsychology Press.
  * Rello et al. (2013) Luz Rello, Ricardo Baeza-Yates, Stefan Bott, and Horacio Saggion. 2013.  Simplify or help? Text simplification strategies for people with dyslexia. In _Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility_. 1–10.
  * Riddell et al. (2021) Allen Riddell, Ari Hartikainen, and Matthew Carter. 2021\.  pystan.  PyPI.
  * Rios et al. (2021) Annette Rios, Nicolas Spring, Tannon Kew, Marek Kostrzewa, Andreas Säuberli, Mathias Müller, and Sarah Ebling. 2021.  A New Dataset and Efficient Baselines for Document-level Text Simplification in German. In _Proceedings of the Third Workshop on New Frontiers in Summarization_. Association for Computational Linguistics, Online and in Dominican Republic, 152–161.  https://doi.org/10.18653/v1/2021.newsum-1.16
  * Ryan et al. (2023) Michael Ryan, Tarek Naous, and Wei Xu. 2023.  Revisiting non-English Text Simplification: A Unified Multilingual Benchmark. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Toronto, Canada, 4898–4927.  https://doi.org/10.18653/v1/2023.acl-long.269
  * Saggion et al. (2015) Horacio Saggion, Sanja Štajner, Stefan Bott, Simon Mille, Luz Rello, and Biljana Drndarevic. 2015\.  Making It Simplext: Implementation and Evaluation of a Text Simplification System for Spanish.  _ACM Trans. Access. Comput._ 6, 4, Article 14 (may 2015), 36 pages.  https://doi.org/10.1145/2738046
  * Saletta and Winberg (2019) Meredith Saletta and Jennifer Winberg. 2019.  Leveled Texts for Adults With Intellectual or Developmental Disabilities: A Pilot Study.  _Focus on Autism and Other Developmental Disabilities_ 34, 2 (2019), 118–127.  https://doi.org/10.1177/1088357618803332 arXiv:https://doi.org/10.1177/1088357618803332
  * Samejima (1997) Fumiko Samejima. 1997\.  _Graded Response Model_.  Springer New York, New York, NY, 85–100.  https://doi.org/10.1007/978-1-4757-2691-6_5
  * Schiffl (2020) Laura Schiffl. 2020\.  Hierarchies in Lexical Complexity: Do Effects of Word Frequency, Word Length and Repetition Exist for the Visual Word Processing of People with Cognitive Impairments?  In _Easy Language Research: Text and User Perspectives_ , Silvia Hansen-Schirra and Christiane Maaß (Eds.). Frank & Timme GmbH, 227–239.
  * Štajner and Nisioi (2018) Sanja Štajner and Sergiu Nisioi. 2018.  A Detailed Evaluation of Neural Sequence-to-Sequence Models for In-domain and Cross-domain Text Simplification. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_. European Language Resources Association (ELRA), Miyazaki, Japan.  https://www.aclweb.org/anthology/L18-1479
  * Stodden (2021) Regina Stodden. 2021\.  When the Scale is Unclear - Analysis of the Interpretation of Rating Scales in Human Evaluation of Text Simplification. In _Proceedings of the First Workshop on Current Trends in Text Simplification (CTTS 2021) co-located with the 37th Conference of the Spanish Society for Natural Language Processing (SEPLN2021), Online (initially located in Málaga, Spain), September 21st, 2021_ _(CEUR Workshop Proceedings, Vol. 2944)_ , Horacio Saggion, Sanja Štajner, Daniel Ferrés, and Kim Cheng Sheang (Eds.). CEUR-WS.org.  http://ceur-ws.org/Vol-2944/paper6.pdf
  * Støle et al. (2020) Hildegunn Støle, Anne Mangen, and Knut Schwippert. 2020\.  Assessing children’s reading comprehension on paper and screen: A mode-effect study.  _Computers & Education_ 151 (2020), 103861.  https://doi.org/10.1016/j.compedu.2020.103861
  * Säuberli et al. (2023) Andreas Säuberli, Silvia Hansen-Schirra, Franz Holzknecht, Silke Gutermuth, Silvana Deilen, Laura Schiffl, and Sarah Ebling. 2023\.  Enabling text comprehensibility assessment for people with intellectual disabilities using a mobile application.  _Frontiers in Communication_ 8 (2023).  https://doi.org/10.3389/fcomm.2023.1175625
  * Trauzettel-Klosinski et al. (2012) Susanne Trauzettel-Klosinski, Klaus Dietz, and the IReST Study Group. 2012.  Standardized Assessment of Reading Performance: The New International Reading Speed Texts IReST.  _Investigative Ophthalmology & Visual Science_ 53, 9 (08 2012), 5452–5461.  https://doi.org/10.1167/iovs.11-8284
  * Van der Linden et al. (1999) Wim J Van der Linden, David J Scrams, and Deborah L Schnipke. 1999.  Using response-time constraints to control for differential speededness in computerized adaptive testing.  _Applied psychological measurement_ 23, 3 (1999), 195–210.
  * Štajner (2021) Sanja Štajner. 2021\.  Automatic text simplification for social good: Progress and challenges. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_. Association for Computational Linguistics, Online, 2637–2652.  https://doi.org/10.18653/v1/2021.findings-acl.233
  * Wallot et al. (2014) Sebastian Wallot, Beth A. O’Brien, Anna Haussmann, Heidi Kloos, and Marlene S. Lyby. 2014.  The role of reading time complexity and reading speed in text comprehension.  _Journal of Experimental Psychology: Learning, Memory, and Cognition_ 40, 6 (2014), 1745–1765.  https://doi.org/10.1037/xlm0000030
  * Wechsler (2003) David Wechsler. 2003\.  Wechsler Intelligence Scale for Children, Fourth Edition.  https://doi.org/10.1037/t15174-000
  * You (2022) Hyesun You. 2022\.  Bayesian Versus Frequentist Estimation for Item Response Theory Models of Interdisciplinary Science Assessment.  _Interdisciplinary Journal of Environmental and Science Education_ 18, 4 (jul 2022), e2297.  https://doi.org/10.21601/ijese/12299

##  Appendix A Priors for model parameters

Measurement | Parameter | Prior distribution
---|---|---
Comprehension question | $\mu\in\mathbb{R}$ | $\mathcal{N}(0,1)$
accuracy | $\alpha\in\mathbb{R}^{16}$ | $\mathcal{N}(0,1)$
| $\beta\in\mathbb{R}^{48}$ | $\mathcal{N}(0,1)$
| $\delta\in\mathbb{R}^{3}$ | $\mathcal{N}(0,1)$
Perceived difficulty | $\mu\in\mathbb{R}$ | $\mathcal{N}(0,1)$
ratings | $\alpha\in\mathbb{R}^{16}$ | $\mathcal{N}(0,1)$
| $\gamma\in\mathbb{R}^{12}$ | $\mathcal{N}(0,1)$
| $\delta\in\mathbb{R}^{3}$ | $\mathcal{N}(0,1)$
| $c_{k}-c_{k-1}\in\mathbb{R}^{3}$ | $\mathcal{N}(0,1)$
Response time | $\mu\in\mathbb{R}$ | $\mathcal{N}(0,5)$
| $\sigma\in\mathbb{R}$ | $\Gamma(1,5)$
| $\alpha\in\mathbb{R}^{16}$ | $\mathcal{N}(0,5)$
| $\beta\in\mathbb{R}^{48}$ | $\mathcal{N}(0,5)$
| $\delta\in\mathbb{R}^{3}$ | $\mathcal{N}(0,5)$
Reading speed | $\mu\in\mathbb{R}$ | $\mathcal{N}(0,5)$
| $\sigma\in\mathbb{R}$ | $\Gamma(1,5)$
| $\alpha\in\mathbb{R}^{16}$ | $\mathcal{N}(0,5)$
| $\gamma\in\mathbb{R}^{12}$ | $\mathcal{N}(0,5)$
| $\delta\in\mathbb{R}^{3}$ | $\mathcal{N}(0,5)$
Table 2. Overview of the prior distributions used for the Bayesian models

Table 2 shows the prior distributions we chose for all model parameters.

Note about $c_{k}$: For five rating categories, there are four threshold
parameters in the graded response model. They need to be in ascending order
and sum to zero. Therefore, instead of sampling the threshold parameters
directly, we sample the differences between adjacent threshold parameters, of
which there are three. For all the other parameters, the sum-to-zero
constraint is achieved by dividing by the mean. For implementation details,
refer to the _Stan_ code in the supplementary material.