P vs RR | 0.7151 | 0.6322 | -11.60% | 0.8126 | 0.6049 | -25.56% | 0.7615 | 0.6764 | -11.18%
P vs RBP_p03 | 0.7494 | 0.7858 | +4.86% | 0.8503 | 0.8604 | +1.19% | 0.7798 | 0.7963 | +2.11%
P vs RBP_p08 | 0.8641 | 0.8447 | -2.24% | 0.9009 | 0.8944 | -0.72% | 0.8748 | 0.8465 | -3.23%
P vs DCG_b02 | 0.9352 | 0.8962 | -4.17% | 0.9623 | 0.9312 | -3.23% | 0.9478 | 0.9203 | -2.90%
P vs DCG_b10 | 0.9866 | 0.9243 | -6.32% | 0.9861 | 0.9254 | -6.15% | 0.9871 | 0.9360 | -5.17%
RBP_p05 vs RBP_p03 | 0.9498 | 1.0000 | +5.29% | 0.9704 | 1.0000 | +3.05% | 0.9601 | 1.0000 | +4.16%
RBP_p05 vs RBP_p08 | 0.9045 | 0.9082 | +0.40% | 0.9250 | 0.9351 | +1.09% | 0.9068 | 0.8998 | -0.78%
RBP_p05 vs RR | 0.8840 | 0.6733 | -23.84% | 0.9046 | 0.6228 | -31.15% | 0.9230 | 0.7590 | -17.77%
RBP_p05 vs DCG_b02 | 0.8461 | 0.8166 | -3.49% | 0.8896 | 0.8593 | -3.41% | 0.8489 | 0.8082 | -4.79%
RBP_p05 vs DCG_b10 | 0.7931 | 0.7686 | -3.09% | 0.8687 | 0.7958 | -8.39% | 0.8082 | 0.7886 | -2.42%
RBP_p03 vs RBP_p08 | 0.8606 | 0.9082 | +5.53% | 0.9069 | 0.9351 | +3.10% | 0.8701 | 0.8998 | +3.42%
RBP_p03 vs RR | 0.9144 | 0.6733 | -26.37% | 0.9154 | 0.6228 | -31.97% | 0.9394 | 0.7590 | -19.21%
RBP_p03 vs DCG_b02 | 0.8054 | 0.8166 | +1.39% | 0.8773 | 0.8593 | -2.06% | 0.8215 | 0.8082 | -1.62%
RBP_p03 vs DCG_b10 | 0.7547 | 0.7686 | +1.84% | 0.8564 | 0.7958 | -7.08% | 0.7918 | 0.7886 | -0.40%
RBP_p08 vs RR | 0.8133 | 0.6430 | -20.94% | 0.8656 | 0.6307 | -27.13% | 0.8547 | 0.7213 | -15.61%
RBP_p08 vs DCG_b02 | 0.9266 | 0.8822 | -4.79% | 0.9372 | 0.8939 | -4.62% | 0.9233 | 0.8646 | -6.36%
RBP_p08 vs DCG_b10 | 0.8745 | 0.8333 | -4.71% | 0.9149 | 0.8304 | -9.23% | 0.8857 | 0.8434 | -4.77%
RR vs DCG_b02 | 0.7668 | 0.5971 | -22.13% | 0.8375 | 0.5902 | -29.52% | 0.7998 | 0.6545 | -18.17%
RR vs DCG_b10 | 0.7214 | 0.5896 | -18.28% | 0.8209 | 0.5475 | -33.30% | 0.7700 | 0.6514 | -15.41%
DCG_b02 vs DCG_b10 | 0.9435 | 0.9273 | -1.72% | 0.9747 | 0.9221 | -5.40% | 0.9593 | 0.9632 | +0.41%
R vs P | 0.7948 | 1.0000 | +25.82% | 0.8901 | 1.0000 | +12.35% | 0.8191 | 1.0000 | +22.08%
R vs RBP_p05 | 0.6848 | 0.7858 | +14.76% | 0.8102 | 0.8604 | +6.20% | 0.8098 | 0.7963 | -1.67%
R vs RBP_p03 | 0.6648 | 0.7858 | +18.20% | 0.8052 | 0.8604 | +6.86% | 0.8169 | 0.7963 | -2.52%
R vs RBP_p08 | 0.7424 | 0.8447 | +13.78% | 0.8391 | 0.8944 | +6.59% | 0.8082 | 0.8465 | +4.74%
R vs RR | 0.6611 | 0.6322 | -4.37% | 0.7841 | 0.6049 | -22.86% | 0.7971 | 0.6764 | -15.14%
R vs DCG_b02 | 0.7756 | 0.8962 | +15.54% | 0.8730 | 0.9312 | +6.67% | 0.8169 | 0.9203 | +12.67%
R vs DCG_b10 | 0.7957 | 0.9243 | +16.15% | 0.8853 | 0.9254 | +4.54% | 0.8184 | 0.9360 | +14.37%
R vs AP | 0.8859 | 0.8932 | +0.83% | 0.8709 | 0.9319 | +7.01% | 0.9078 | 0.9211 | +1.46%
R vs nDCG_b02 | 0.7899 | 0.8962 | +13.45% | 0.8788 | 0.9312 | +5.97% | 0.8427 | 0.9203 | +9.21%
R vs nDCG_b10 | 0.8338 | 0.9243 | +10.85% | 0.8911 | 0.9254 | +3.86% | 0.8490 | 0.9360 | +10.25%
AP vs P | 0.8244 | 0.8932 | +8.35% | 0.8583 | 0.9319 | +8.58% | 0.8481 | 0.9211 | +8.61%
AP vs RBP_p05 | 0.7524 | 0.7936 | +5.48% | 0.8326 | 0.8413 | +1.04% | 0.8614 | 0.7980 | -7.36%
AP vs RBP_p03 | 0.7271 | 0.7936 | +9.14% | 0.8160 | 0.8413 | +3.09% | 0.8575 | 0.7980 | -6.94%
AP vs RBP_p08 | 0.8081 | 0.8587 | +6.27% | 0.8672 | 0.8730 | +0.67% | 0.8685 | 0.8466 | -2.52%
AP vs RR | 0.7030 | 0.5869 | -16.52% | 0.7761 | 0.5707 | -26.47% | 0.8249 | 0.6545 | -20.66%
AP vs DCG_b02 | 0.8316 | 0.9673 | +16.32% | 0.8651 | 0.9632 | +11.34% | 0.8748 | 0.9757 | +11.54%
AP vs DCG_b10 | 0.8280 | 0.9411 | +13.67% | 0.8672 | 0.9444 | +8.90% | 0.8575 | 0.9750 | +13.69%
AP vs nDCG_b02 | 0.8454 | 0.9673 | +14.42% | 0.8709 | 0.9632 | +10.60% | 0.8912 | 0.9757 | +9.49%
AP vs nDCG_b10 | 0.8563 | 0.9411 | +9.90% | 0.8687 | 0.9444 | +8.72% | 0.8849 | 0.9750 | +10.17%
nDCG_b02 vs P | 0.9330 | 0.8962 | -3.95% | 0.9579 | 0.9312 | -2.79% | 0.9305 | 0.9203 | -1.10%
nDCG_b02 vs RBP_p05 | 0.8449 | 0.8166 | -3.36% | 0.8925 | 0.8593 | -3.72% | 0.8638 | 0.8082 | -6.43%
nDCG_b02 vs RBP_p03 | 0.8051 | 0.8166 | +1.42% | 0.8773 | 0.8593 | -2.06% | 0.8395 | 0.8082 | -3.73%
nDCG_b02 vs RBP_p08 | 0.9283 | 0.8822 | -4.96% | 0.9401 | 0.8939 | -4.91% | 0.9319 | 0.8646 | -7.22%
nDCG_b02 vs RR | 0.7684 | 0.5971 | -22.30% | 0.8389 | 0.5902 | -29.65% | 0.8179 | 0.6545 | -19.98%
nDCG_b02 vs DCG_b02 | 0.9804 | 1.0000 | +2.00% | 0.9913 | 1.0000 | +0.87% | 0.9695 | 1.0000 | +3.15%
nDCG_b02 vs DCG_b10 | 0.9418 | 0.9273 | -1.54% | 0.9719 | 0.9221 | -5.12% | 0.9429 | 0.9632 | +2.16%
nDCG_b02 vs nDCG_b10 | 0.9368 | 0.9273 | -1.01% | 0.9690 | 0.9221 | -4.84% | 0.9577 | 0.9632 | +0.57%
nDCG_b10 vs P | 0.9568 | 0.9243 | -3.40% | 0.9832 | 0.9254 | -5.88% | 0.9525 | 0.9360 | -1.73%
nDCG_b10 vs RBP_p05 | 0.7841 | 0.7686 | -1.98% | 0.8644 | 0.7958 | -7.93% | 0.8278 | 0.7886 | -4.73%
nDCG_b10 vs RBP_p03 | 0.7472 | 0.7686 | +2.86% | 0.8535 | 0.7958 | -6.76% | 0.8129 | 0.7886 | -2.99%
nDCG_b10 vs RBP_p08 | 0.8684 | 0.8333 | -4.05% | 0.9120 | 0.8304 | -8.94% | 0.8928 | 0.8434 | -5.52%
nDCG_b10 vs RR | 0.7212 | 0.5896 | -18.25% | 0.8194 | 0.5475 | -33.18% | 0.7881 | 0.6514 | -17.35%
nDCG_b10 vs DCG_b02 | 0.9288 | 0.9273 | -0.16% | 0.9719 | 0.9221 | -5.12% | 0.9507 | 0.9632 | +1.32%
nDCG_b10 vs DCG_b10 | 0.9610 | 1.0000 | +4.06% | 0.9899 | 1.0000 | +1.02% | 0.9616 | 1.0000 | +3.99%

We can observe from Table 4, as very coarse and general trends, that
correlation is overestimated ($\Delta\%$ column negative) in the range
$[-33.30\%,\,-0.16\%]$, i.e. two evaluation measures are less close to each
other than what we would be induced to think; conversely, correlation is
underestimated ($\Delta\%$ column positive) in the range $[0.4\%,\,25.82\%]$,
i.e. two evaluation measures are closer to each other than what we would be
induced to think. This observation opens up a relevant question for IR
experiments: are IR measures really that different? Do we need all of them?
Are we really scoring runs according to different user viewpoints or are these
differences just an artefact of violating the scale assumptions? How much of
what reported in the literature is due just to this scale violation bias?

In the following sections we discuss a few examples from Table 4 of how the
correlation may change.

####  7.2.1 Measures not depending on the recall base

Let us start from the correlation between Precision and RBP with $p=0.5$. We
already know that they are interval scales and, therefore, their ranked
version is just another mapping of their respective interval scales – and this
is why in Table 2 their overall correlation is $1.0$. We can observe from
Table 4 that on T08_30 the correlation between RBP_p05 and P is $0.7858$ and,
as expected, the correlation between their ranked versions is the same, since
the interval scale behind the original measures and their ranked version is
the same. The same happens for the other tracks, i.e. T26_30 and T27_30.

Note that “the correlation between Precision and RBP with $p=0.5$ on T08_30 is
$0.7858$” is an example of a _meaningful statement_ in IR, since it is
invariant to a permissible transformation of the interval scales of these two
measures and it does not change its truth value.

The correlation between RBP with $p=0.3$ and RBP with $p=0.5$ is $0.9498$
while the correlation between their ranked versions is $1.0$. As we discussed
in Section 7.1, RBP_p03 and RBP_p05 order runs in the same way and so their
correlation should be $1.0$. Therefore, this $5.3\%$ underestimation of the
similarity between them is just due to RBP_p03 not being an interval scale.
Note that this case is somehow particularly severe since it induces us to
attribute this $5.3\%$ change to other reasons; typical explanations for such
changes you may find in studies about correlation among evaluation measures
are: “the user model behind RBP_p03 slightly differs from the RBP_p05 since it
represents a more impatient or less motivated user” or “due to the smaller
value of $p$, RBP_p03 is a slightly more top-heavy measure”; unfortunately,
none of these explanations would be correct since this $5.3\%$ change is just
due to fact that the values of RBP_p03 are not equi-spaced, still ordering
runs in exactly the same way as RBP_p05.

For the sake of completeness, we can observe that $0.9498$ is the same
correlation value reported in Table 2 between RBP_p03 and its ranked version.
This is indeed correct since both RBP_p03 and RBP_p05 are mapped to the same
ranked interval scale, which is just another mapping of the interval scale of
RBP_p05; therefore, the correlation between RBP_p03 and its ranked version is
the same as the correlation between RBP_p03 and RBP_p05.

Another interesting case is RR: its correlation with respect to P, RBP, and
DCG is way over-estimated – in the range $12\%-26\%$ more on T08_30,
$26\%-33\%$ more on T26_30, and $11\%-19\%$ more on T27_30, – mistakenly
suggesting us that this measure is closer to the others much more than what it
actually is. As it emerges from the previous discussion, RR is one of the
measures which departs more from being an interval scale and which also has
the highest number of tied values. Therefore, the computation of averages on
RR and on its ranked version leads to sensibly different  Rankings of Systems
(RoS), as it is clearly shown in Table 2 when comparing RR to its ranked
version. As a consequence, the correlation of the ranked version of RR with
the other measures changes more than in the other cases.

####  7.2.2 Measures depending on the recall base

Before proceeding, a word of caution has to be made remembering that in the
case of measures depending on the recall base our approach is just a
surrogate, which improves the “intervalness” of a measure but stretches the
steps of the scale. Therefore, all the increases/decreases in correlations
should be taken as tendency to overestimation/underestimation rather than
exact quantification of it.

Let us consider Precision and Recall: we know that on each topic they are the
same interval scale and this is reflected in Table 4 in the correlation
between their ranked version being $1.0$. On the other hand, the correlation
between the original measures tends to be underestimated by $26\%$ on T08_30,
$12\%$ on T26_30, and $22\%$ on T27_30. Apart from suggesting that these two
measures should be considered closer to each other than what they usually are,
this wide range of underestimation further stresses how much just the recall
base can affect the averaging across topics and how careful we should be with
such averages – not to say that we should avoid them at all.

Coherently with what discussed in Section 7.1.2, we can observe that the
correlation between DCG and nDCG tends to be underestimated by just $1\%-4\%$,
suggesting that they are practically equivalent. Therefore, even if usually
nDCG is preferred over DCG because it is bounded and normalised, it could be
actually better to use DCG instead, since it avoids issues with the recall
base and it can be easily turned into a proper interval scale by using our
transformation approach.

Let us now discuss AP with respect to Precision and Recall: the correlation
with R is higher than the one with P and this is usually attributed to AP
embedding the recall base in the same way as R does. However, when we turn to
the ranked measures, we can see how the correlation between AP and R and AP
and P is the same (for all the reasons already explained) and, especially, how
this tends to be underestimated in the range $1\%-9\%$, suggesting that AP is
slightly closer to these two measures than what is usually thought.

Finally, let us consider AP with respect to DCG: their correlation tends to be
underestimated in the range $9\%-16\%$ and the correlation between their
ranked versions is actually quite – high, between $0.94$ and $0.97$. This
suggests that, even if these two measures have two quite different
formulations and the user model of DCG is considered much more realistic than
the somehow artificial one of AP, when they are turned into their interval
scale version, they are much closer than expected and that part of their
difference could have been just due to their lack of “intervalness”.

###  7.3 Significance Testing Analysis. What systems are actually different,
or not?

In this section, we analyse how the results of statistical significance tests
change when using a measure or its ranked version. In other terms, we study
how much statistical significance tests are impacted by using or not a proper
interval scale. Indeed, as discussed in Section 7.3, there are significance
tests which assume to work with just an ordinal scale and others which assume
to work with an interval scale and they should be somehow affected by using a
measure which matches or not their assumptions. By impacted, we mean that we
can observe some change in which systems are considered significantly
different or not.

Moreover, as discussed in the previous sections, the recall base makes working
across topics problematic at best and statistical significance tests typically
perform some aggregation across topics. Therefore, they may be further
affected by the recall base.

As described in Section 7.3, we consider the following tests: Sign (ordinal
scale assumption), Wilcoxon Rank Sum (ordinal scale assumption), Wilcoxon
Signed Rank (interval scale assumption), Student’s t (interval scale
assumption), ANOVA (interval scale assumption), Kruskal-Wallis (ordinal scale
assumption), and Friedman (ordinal scale assumption). For the ANOVA case we
consider two alternatives:

  * •

One-way ANOVA for the System Effect:
$y_{ij}=\mu_{\cdot\cdot}+\alpha_{j}+\varepsilon_{ij}$ checks for the effect of
$\alpha_{j},\,j=1,\ldots,q$ different systems. It can be considered as an
extension of the Student’s t test to the comparison of multiple systems at the
same time and it is the parametric counterpart of the Kruskal-Wallis Test.

  * •

Two-way ANOVA for the Topic and System Effects: a more accurate model
$y_{ij}=\mu_{\cdot\cdot}+\tau_{i}+\alpha_{j}+\varepsilon_{ij}$ which accounts
also for the effect of $\tau_{i},\,i=1,\ldots,p$ topics, thus improving the
estimation of the system effect as well. Note that this is the ANOVA model
adopted by Tague-Sutcliffe and Blustein [88] and Banks et al. [11] when
analysing TREC data. It is the parametric counterpart of the Friedman Test.

In the case of the ANOVA, Kruskal-Wallis, and Friedman tests we performed a
Tukey  Honestly Significant Difference (HSD) adjustment for multiple
comparisons [48, 90].

Table 5 (measures not depending on the recall base) and Table 6 (measures
depending on the recall base) show the results of the analyses in the case of
the T08_30, T26_30, and T27_30 tracks. Results for the other tracks are
similar but not shown here for space reasons. For each test, the tables
report:

  * •

Sig: the total number of significantly different system pair using the
original measure;

  * •

S2NS: number of pairs changed from significantly to not significantly
different when passing from the original measure to its ranked version; within
parentheses we report their ratio with respect to Sig;

  * •

NS2S: number of pairs changed from not significantly to significantly
different when passing from the original measure to its ranked version; within
parentheses we report their ratio with respect to Sig;

  * •

$\Delta\%$: $\frac{\text{S2NS}+\text{NS2S}}{\text{Sig}}$ the ratio of the
total number of pairs that changed significance when passing from the original
measure to its ranked version.

In Tables 5 and 6 rows corresponding to significance tests based on an ordinal
scale assumption are highlighted in grey.

In an ideal situation an oracle would have told us which pairs of systems are
significantly different and which are not and this would have allowed us to
exactly determine which pairs of systems were correctly detected by each
measure and test. Unfortunately, this a priori knowledge is not available in
practice. On the other hand, we are comparing a measure to its ranked version
and we know that changes in the decision about what is significantly different
and what is not are a consequence of the steps of the scale being rearranged
from not equi-spaced and un-evenly distributed across the their range to equi-
spaced and evenly distributed across their range. Therefore, we can interpret
the S2NS count as a tendency to _false positives_ , since it accounts for
significantly different systems which are not significant when you remove the
effect of uneven steps in the scale; in other terms, S2NS can be interpreted
as a tendency of the ranked measure (the interval scale) to reduce _Type I
errors_. Symmetrically, we can interpret the NS2S count as a tendency to
_false negatives_ , since it accounts for not significantly different systems
which are significant when you remove the effect of uneven steps in the scale;
in other terms, NS2S can be interpreted as a tendency of the ranked measure
(the interval scale) to reduce _Type II errors_. Note that we are not claiming
that an interval scale detects/removes false positives/negatives in any
absolute sense; we are rather saying that, starting from whatever unknown
level of false positive/negatives, we can interpret the S2NS and NS2S counts
as a relative tendency to reduce false positives/negatives.

As a side note not regarding measures being interval scales or not, in Tables
5 and 6 we can observe that, as expected, parametric significance tests are
more powerful than non-parametric ones, since they discover more significantly
different pairs. Moreover, we can also observe as the Sign, Wilcoxon Rank Sum,
Wilcoxon Signed Rank, and Student’s t tests find many more significantly
different system pairs than the ANOVA, Kruskal-Wallis, and Friedman tests.
This increase is not due to more powerful tests but rather to the increase in
Type I errors due to the lack of adjustment in multiple comparisons for the
former tests. This further stresses the need for always adjusting for multiple
comparisons, as also pointed out by Fuhr [40], Sakai [75].

We can observe from Tables 5 and 6, as very coarse and general trends, that
the less close an evaluation measure is to be an interval scale, the stronger
are the changes in statistical significance tests based on an interval scale
assumption while those based on an ordinal scale assumption are not affected.
On the other hand, the presence of the recall base generally affects both
tests based on the ordinal scale assumption and those based on the interval
scale assumption, being the latter more affected.

In particular, we have found that:

  * •

for measures not depending on the recall base and significance test assuming
an interval scale, we have an overall average increase in the S2NS count
around $13\%$ and in the NS2S around $5\%$. This suggests that the major
impact is on reducing Type I errors still improving Type II errors and making
the test more powerful;

  * •

for measures depending on the recall base and significance test assuming an
ordinal scale, we have an overall average increase in the S2NS count around
$2\%$ and in the NS2S around $4\%$. This suggests that there is a small
reduction in Type I errors and some improvement in Type II errors, making the
test a bit more powerful;

  * •

for measures depending on the recall base and significance test assuming an
interval scale, we have an overall average increase in the S2NS count around
$10\%$ and in the NS2S around $45\%$. This suggests that there is a sizeable
reduction in Type I errors and a quite substantial improvement in Type II
errors, making the test much more powerful.

In general, these results indicate that adopting a proper interval scale tends
to reduce the Type I errors and, when the situation get more complicated
because of the effect of the recall base across topics, it also brings
substantially more power to the test.

Overall, if we consider the grand mean across all the tracks, measures, and
significance test, we observe an overall change $\Delta\%$ around $25\%\pm
11\%$ in the decision about what is significantly different and what is not.
Even without wishing to interpret it in term of Type I or Type II errors, this
figure let us understand how big is the impact of using an interval scale or
not, as well as the effect of the recall base.

As in the case of the correlation analysis, these observations open some
questions about IR experimentation: since violating the scale assumptions has
an impact on the number of significant/not-significant detected pairs and on
Type I and Type II errors, when we compare systems and algorithms, how much of
the observed differences is just due to the scale violation bias? How many
false positives/negatives are we observing? How much have the findings in the
literature been affected by these phenomena?

####  7.3.1 Measures not depending on the recall base

Table 5: Measures not depending on the recall base: changes in significance
test analyses between using a measure and its ranked version on tracks T08_30,
T26_30, and T27_30, using the unq tie breaking approach.

| T08_30 – 8256 system pairs | T26_30 – 2775 system pairs | T27_30 – 2556 system pairs
---|---|---|---
P | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$ | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$ | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$
Sign Test | 5153 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1848 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1746 | 0 (0.00%) | 0 (0.00%) | 0.00%
Wilcoxon Rank Sum Test | 4276 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1453 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1473 | 0 (0.00%) | 0 (0.00%) | 0.00%
Wilcoxon Signed Rank Test | 5644 | 0 (0.00%) | 0 (0.00%) | 0.00% | 2017 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1857 | 0 (0.00%) | 0 (0.00%) | 0.00%
Student’s t Test | 5633 | 0 (0.00%) | 0 (0.00%) | 0.00% | 2007 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1830 | 0 (0.00%) | 0 (0.00%) | 0.00%
One-way ANOVA | 1923 | 0 (0.00%) | 0 (0.00%) | 0.00% | 454 | 0 (0.00%) | 0 (0.00%) | 0.00% | 724 | 0 (0.00%) | 0 (0.00%) | 0.00%
Kruskal-Wallis Test | 1740 | 0 (0.00%) | 0 (0.00%) | 0.00% | 391 | 0 (0.00%) | 0 (0.00%) | 0.00% | 692 | 0 (0.00%) | 0 (0.00%) | 0.00%
Two-way ANOVA | 3362 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1155 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1214 | 0 (0.00%) | 0 (0.00%) | 0.00%
Friedman Test | 2595 | 0 (0.00%) | 0 (0.00%) | 0.00% | 883 | 0 (0.00%) | 0 (0.00%) | 0.00% | 925 | 0 (0.00%) | 0 (0.00%) | 0.00%
RBP_p05 | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$ | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$ | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$
Sign Test | 4724 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1414 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1613 | 0 (0.00%) | 0 (0.00%) | 0.00%
Wilcoxon Rank Sum Test | 4302 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1328 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1545 | 0 (0.00%) | 0 (0.00%) | 0.00%
Wilcoxon Signed Rank Test | 5108 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1683 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1679 | 0 (0.00%) | 0 (0.00%) | 0.00%
Student’s t Test | 5091 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1711 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1635 | 0 (0.00%) | 0 (0.00%) | 0.00%
One-way ANOVA | 1945 | 0 (0.00%) | 0 (0.00%) | 0.00% | 377 | 0 (0.00%) | 0 (0.00%) | 0.00% | 660 | 0 (0.00%) | 0 (0.00%) | 0.00%
Kruskal-Wallis Test | 1678 | 0 (0.00%) | 0 (0.00%) | 0.00% | 326 | 0 (0.00%) | 0 (0.00%) | 0.00% | 680 | 0 (0.00%) | 0 (0.00%) | 0.00%
Two-way ANOVA | 2861 | 0 (0.00%) | 0 (0.00%) | 0.00% | 733 | 0 (0.00%) | 0 (0.00%) | 0.00% | 927 | 0 (0.00%) | 0 (0.00%) | 0.00%
Friedman Test | 2228 | 0 (0.00%) | 0 (0.00%) | 0.00% | 588 | 0 (0.00%) | 0 (0.00%) | 0.00% | 806 | 0 (0.00%) | 0 (0.00%) | 0.00%
RBP_p03 | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$ | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$ | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$
Sign Test | 4724 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1414 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1613 | 0 (0.00%) | 0 (0.00%) | 0.00%
Wilcoxon Rank Sum Test | 4302 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1328 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1545 | 0 (0.00%) | 0 (0.00%) | 0.00%
Wilcoxon Signed Rank Test | 5027 | 29 (0.58%) | 110 (2.19%) | 2.77% | 1638 | 10 (0.61%) | 55 (3.36%) | 3.97% | 1651 | 3 (0.18%) | 31 (1.88%) | 2.06%
Student’s t Test | 4801 | 70 (1.46%) | 360 (7.50%) | 8.96% | 1551 | 19 (1.23%) | 179 (11.54%) | 12.77% | 1486 | 7 (0.47%) | 156 (10.50%) | 10.97%
One-way ANOVA | 1730 | 18 (1.04%) | 233 (13.47%) | 14.51% | 317 | 3 (0.95%) | 63 (19.87%) | 20.82% | 606 | 4 (0.66%) | 58 (9.57%) | 10.23%
Kruskal-Wallis Test | 1678 | 0 (0.00%) | 0 (0.00%) | 0.00% | 326 | 0 (0.00%) | 0 (0.00%) | 0.00% | 680 | 0 (0.00%) | 0 (0.00%) | 0.00%
Two-way ANOVA | 2432 | 8 (0.33%) | 437 (17.97%) | 18.30% | 573 | 2 (0.35%) | 162 (28.27%) | 28.62% | 807 | 0 (0.00%) | 120 (14.87%) | 14.87%
Friedman Test | 2228 | 0 (0.00%) | 0 (0.00%) | 0.00% | 588 | 0 (0.00%) | 0 (0.00%) | 0.00% | 806 | 0 (0.00%) | 0 (0.00%) | 0.00%
RBP_p08 | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$ | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$ | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$
Sign Test | 5105 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1615 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1678 | 0 (0.00%) | 0 (0.00%) | 0.00%
Wilcoxon Rank Sum Test | 4375 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1378 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1487 | 0 (0.00%) | 0 (0.00%) | 0.00%
Wilcoxon Signed Rank Test | 5508 | 165 (3.00%) | 114 (2.07%) | 5.07% | 1879 | 70 (3.73%) | 30 (1.60%) | 5.32% | 1846 | 110 (5.96%) | 3 (0.16%) | 6.12%
Student’s t Test | 5425 | 221 (4.07%) | 123 (2.27%) | 6.34% | 1924 | 144 (7.48%) | 60 (3.12%) | 10.60% | 1853 | 236 (12.74%) | 14 (0.76%) | 13.49%
One-way ANOVA | 2183 | 200 (9.16%) | 44 (2.02%) | 11.18% | 469 | 59 (12.58%) | 15 (3.20%) | 15.78% | 712 | 98 (13.76%) | 1 (0.14%) | 13.90%
Kruskal-Wallis Test | 1781 | 0 (0.00%) | 0 (0.00%) | 0.00% | 384 | 0 (0.00%) | 0 (0.00%) | 0.00% | 691 | 0 (0.00%) | 0 (0.00%) | 0.00%
Two-way ANOVA | 3474 | 257 (7.40%) | 20 (0.58%) | 7.97% | 950 | 104 (10.95%) | 21 (2.21%) | 13.16% | 1105 | 187 (16.92%) | 6 (0.54%) | 17.47%
Friedman Test | 2589 | 0 (0.00%) | 0 (0.00%) | 0.00% | 741 | 0 (0.00%) | 0 (0.00%) | 0.00% | 851 | 0 (0.00%) | 0 (0.00%) | 0.00%
DCG_b02 | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$ | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$ | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$
Sign Test | 5227 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1781 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1738 | 0 (0.00%) | 0 (0.00%) | 0.00%
Wilcoxon Rank Sum Test | 4425 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1457 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1499 | 0 (0.00%) | 0 (0.00%) | 0.00%
Wilcoxon Signed Rank Test | 5743 | 260 (4.53%) | 147 (2.56%) | 7.09% | 2020 | 127 (6.29%) | 40 (1.98%) | 8.27% | 1887 | 91 (4.82%) | 21 (1.11%) | 5.94%
Student’s t Test | 5664 | 408 (7.20%) | 161 (2.84%) | 10.05% | 2026 | 176 (8.69%) | 47 (2.32%) | 11.01% | 1881 | 155 (8.24%) | 15 (0.80%) | 9.04%
One-way ANOVA | 2125 | 633 (29.79%) | 59 (2.78%) | 32.56% | 495 | 113 (22.83%) | 3 (0.61%) | 23.43% | 734 | 153 (20.84%) | 2 (0.27%) | 21.12%
Kruskal-Wallis Test | 1812 | 0 (0.00%) | 0 (0.00%) | 0.00% | 400 | 0 (0.00%) | 0 (0.00%) | 0.00% | 695 | 0 (0.00%) | 0 (0.00%) | 0.00%
Two-way ANOVA | 3609 | 601 (16.65%) | 25 (0.69%) | 17.35% | 1129 | 179 (15.85%) | 36 (3.19%) | 19.04% | 1201 | 328 (27.31%) | 4 (0.33%) | 27.64%
Friedman Test | 2706 | 0 (0.00%) | 0 (0.00%) | 0.00% | 876 | 0 (0.00%) | 0 (0.00%) | 0.00% | 912 | 0 (0.00%) | 0 (0.00%) | 0.00%
DCG_b10 | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$ | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$ | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$
Sign Test | 5095 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1813 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1755 | 0 (0.00%) | 0 (0.00%) | 0.00%
Wilcoxon Rank Sum Test | 4307 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1458 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1485 | 0 (0.00%) | 0 (0.00%) | 0.00%
Wilcoxon Signed Rank Test | 5663 | 260 (4.59%) | 127 (2.24%) | 6.83% | 2019 | 105 (5.20%) | 32 (1.58%) | 6.79% | 1861 | 60 (3.22%) | 35 (1.88%) | 5.10%
Student’s t Test | 5636 | 370 (6.56%) | 132 (2.34%) | 8.91% | 2021 | 169 (8.36%) | 50 (2.47%) | 10.84% | 1843 | 114 (6.19%) | 36 (1.95%) | 8.14%
One-way ANOVA | 1958 | 747 (38.15%) | 37 (1.89%) | 40.04% | 468 | 86 (18.38%) | 12 (2.56%) | 20.94% | 727 | 187 (25.72%) | 0 (0.00%) | 25.72%
Kruskal-Wallis Test | 1750 | 0 (0.00%) | 0 (0.00%) | 0.00% | 390 | 0 (0.00%) | 0 (0.00%) | 0.00% | 687 | 0 (0.00%) | 0 (0.00%) | 0.00%
Two-way ANOVA | 3438 | 623 (18.12%) | 41 (1.19%) | 19.31% | 1139 | 130 (11.41%) | 36 (3.16%) | 14.57% | 1216 | 256 (21.05%) | 4 (0.33%) | 21.38%
Friedman Test | 2637 | 0 (0.00%) | 0 (0.00%) | 0.00% | 882 | 0 (0.00%) | 0 (0.00%) | 0.00% | 925 | 0 (0.00%) | 0 (0.00%) | 0.00%
RR | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$ | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$ | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$
Sign Test | 4074 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1305 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1309 | 0 (0.00%) | 0 (0.00%) | 0.00%
Wilcoxon Rank Sum Test | 3951 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1198 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1280 | 0 (0.00%) | 0 (0.00%) | 0.00%
Wilcoxon Signed Rank Test | 4578 | 479 (10.46%) | 362 (7.91%) | 18.37% | 1497 | 201 (13.43%) | 180 (12.02%) | 25.45% | 1462 | 94 (6.43%) | 111 (7.59%) | 14.02%
Student’s t Test | 4582 | 903 (19.71%) | 454 (9.91%) | 29.62% | 1517 | 455 (29.99%) | 180 (11.87%) | 41.86% | 1454 | 202 (13.89%) | 110 (7.57%) | 21.46%
One-way ANOVA | 1691 | 288 (17.03%) | 162 (9.58%) | 26.61% | 285 | 203 (71.23%) | 2 (0.70%) | 71.93% | 696 | 92 (13.22%) | 148 (21.26%) | 34.48%
Kruskal-Wallis Test | 1500 | 0 (0.00%) | 0 (0.00%) | 0.00% | 218 | 0 (0.00%) | 0 (0.00%) | 0.00% | 633 | 0 (0.00%) | 0 (0.00%) | 0.00%
Two-way ANOVA | 2233 | 511 (22.88%) | 131 (5.87%) | 28.75% | 527 | 349 (66.22%) | 23 (4.36%) | 70.59% | 868 | 145 (16.71%) | 92 (10.60%) | 27.30%
Friedman Test | 1813 | 0 (0.00%) | 0 (0.00%) | 0.00% | 428 | 0 (0.00%) | 0 (0.00%) | 0.00% | 739 | 0 (0.00%) | 0 (0.00%) | 0.00%

Let us start from Precision and RBP_p05 in Table 5. As we already know, both
of them are interval scales and, as expected, we do not observe any changes in
using them or their ranked version.

As in the case of the correlation, we can take them as an example of
_meaningful statements_ in IR, since a statement like “There are 1,923
significantly different system pairs for Precision according to one-way ANOVA
on T08_30” does not change its truth value for a permissible transformation of
the scale.

As said, RBP_p03 orders systems in the same way as RBP_p05 but it is no more
an interval scale. Coherently with this, we can see how the significance tests
assuming just an ordinal case detect the same number of significantly
different pairs for both RBP_p03 and RBP_p05. On the other hand, significance
tests assuming an interval scale are affected by this difference between
RBP_p03 and RBP_p05, causing an overall change $\Delta\%$ in the range
$2\%-29\%$. In particular, we observe a marked increase in the number of
significantly different pairs (NS2S up to $28\%$), i.e. the reduction in the
number of false negatives, and a very marginal decrease in the number of not
significantly different ones (S2NS around $1\%$), i.e. the reduction in the
number of false positives. In the case of RBP_p08 we can note a much more
marked increase in the number of not significantly different pairs (S2NS up to
$17\%$), i.e. a reduction in the number of false positives; on the other hand,
the increase in the number of significantly different pairs (NS2S around
$2\%$), i.e. the reduction in the number of false negatives, is more marginal.

Why do we observe such a different behaviour between RBP_p03 and RBP_p08? If
we look at Figure 4, we can see that RBP_p03 condenses values at the top and
the bottom of the range of possible values, in spans with a very small range
of values but containing the same number of runs. As a consequence, when the
ranked version of RBP_p03 equi-spaces these values, runs that before were very
close, and possibly not significantly different (NS) in the ranked version
become more distant, and possibly significantly different (S); and this can
explain why the NS2S case is more prominent for RBP_p03. On the other hand,
RBP_p08 uses all the possible range of values but very few runs, roughly
$20\%$ packed at the bottom and at the top, cover almost $50\%$ of the range
of values while the remaining $80\%$ of the runs, in the middle part, cover
the other $50\%$ of the range. Therefore, when we pass to the ranked version
of the measure, very few runs which were very distant, and possibly
significantly different (S), become closer, and possibly not significantly
different (NS); viceversa, many runs which were very close, and possibly not
significantly different (NS), may become a little bit more distant, and
possibly (but not necessarily) significantly different (S). As a consequence,
the effect on S2NS is more prominent than the one on NS2S.

In the case of DCG we observe a behaviour similar to the one of RBP_p08, being
the increase on S2NS and the reduction in the false positives even more
prominent. If we look at Figure 5, we can see how DCG is sharper than RBP_p08
at the top and bottom of the range – less than $10\%$ of the runs account for
almost $50\%$ of the range of values – making even fewer runs falling more
apart.

Finally, RR exhibits both effects: a very remarkable increase in S2NS, i.e. a
reduction in the false positives, and a sizeable increase in NS2S, i.e. a
reduction in the false negatives, causing an overall change $\Delta\%$ up to a
$72\%$. If we consider Figure 3, we can see how most of the runs, over $90\%$,
are concentrated in just 4 possible values which are quite distant, possibly
making them significantly different (S); when we move to the ranked version,
these 4 values become much closer, possibly making the runs not significantly
different (NS); and this explains the big S2NS effect. Vice versa, few runs,
less than $10\%$, account for just the $20\%$ of the range of values in the
lower quartile; when we move to the ranked version, these values become more
distant, possibly making the runs significantly different (S); since this
change may affect a smaller number of runs, this explains why NS2S tends to be
more moderate with respect to S2NS.

####  7.3.2 Measures depending on the recall base

Table 6: Measures depending on the recall base: changes in significance test
analyses between using a measure and its ranked version on tracks T08_30,
T26_30, and T27_30, using the unq tie breaking approach.

| T08_30 – 8256 system pairs | T26_30 – 2775 system pairs | T27_30 – 2556 system pairs
---|---|---|---
R | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$ | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$ | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$
Sign Test | 5153 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1848 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1746 | 0 (0.00%) | 0 (0.00%) | 0.00%
Wilcoxon Rank Sum Test | 3494 | 14 (0.40%) | 796 (22.78%) | 23.18% | 1239 | 25 (2.02%) | 239 (19.29%) | 21.31% | 1254 | 17 (1.36%) | 236 (18.82%) | 20.18%
Wilcoxon Signed Rank Test | 5434 | 185 (3.40%) | 395 (7.27%) | 10.67% | 1892 | 35 (1.85%) | 160 (8.46%) | 10.31% | 1799 | 27 (1.50%) | 85 (4.72%) | 6.23%
Student’s t Test | 5073 | 303 (5.97%) | 863 (17.01%) | 22.98% | 1723 | 62 (3.60%) | 346 (20.08%) | 23.68% | 1554 | 34 (2.19%) | 310 (19.95%) | 22.14%
One-way ANOVA | 409 | 0 (0.00%) | 1514 (370.17%) | 370.17% | 69 | 0 (0.00%) | 385 (557.97%) | 557.97% | 574 | 3 (0.52%) | 153 (26.66%) | 27.18%
Kruskal-Wallis Test | 1417 | 28 (1.98%) | 351 (24.77%) | 26.75% | 259 | 0 (0.00%) | 132 (50.97%) | 50.97% | 653 | 16 (2.45%) | 55 (8.42%) | 10.87%
Two-way ANOVA | 2440 | 47 (1.93%) | 969 (39.71%) | 41.64% | 683 | 3 (0.44%) | 475 (69.55%) | 69.99% | 923 | 15 (1.63%) | 306 (33.15%) | 34.78%
Friedman Test | 2595 | 0 (0.00%) | 0 (0.00%) | 0.00% | 883 | 0 (0.00%) | 0 (0.00%) | 0.00% | 925 | 0 (0.00%) | 0 (0.00%) | 0.00%
AP | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$ | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$ | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$
Sign Test | 5233 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1801 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1727 | 0 (0.00%) | 0 (0.00%) | 0.00%
Wilcoxon Rank Sum Test | 4067 | 25 (0.61%) | 385 (9.47%) | 10.08% | 1455 | 84 (5.77%) | 85 (5.84%) | 11.62% | 1446 | 48 (3.32%) | 96 (6.64%) | 9.96%
Wilcoxon Signed Rank Test | 5659 | 242 (4.28%) | 169 (2.99%) | 7.26% | 1896 | 91 (4.80%) | 128 (6.75%) | 11.55% | 1853 | 97 (5.23%) | 38 (2.05%) | 7.29%
Student’s t Test | 5156 | 339 (6.57%) | 559 (10.84%) | 17.42% | 1683 | 145 (8.62%) | 336 (19.96%) | 28.58% | 1759 | 118 (6.71%) | 109 (6.20%) | 12.91%
One-way ANOVA | 391 | 3 (0.77%) | 955 (244.25%) | 245.01% | 59 | 0 (0.00%) | 295 (500.00%) | 500.00% | 529 | 60 (11.34%) | 65 (12.29%) | 23.63%
Kruskal-Wallis Test | 1593 | 19 (1.19%) | 225 (14.12%) | 15.32% | 380 | 20 (5.26%) | 37 (9.74%) | 15.00% | 700 | 21 (3.00%) | 11 (1.57%) | 4.57%
Two-way ANOVA | 2508 | 171 (6.82%) | 560 (22.33%) | 29.15% | 636 | 31 (4.87%) | 360 (56.60%) | 61.48% | 913 | 105 (11.50%) | 73 (8.00%) | 19.50%
Friedman Test | 2706 | 0 (0.00%) | 0 (0.00%) | 0.00% | 883 | 0 (0.00%) | 0 (0.00%) | 0.00% | 915 | 0 (0.00%) | 0 (0.00%) | 0.00%
nDCG_b02 | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$ | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$ | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$
Sign Test | 5227 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1781 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1738 | 0 (0.00%) | 0 (0.00%) | 0.00%
Wilcoxon Rank Sum Test | 4420 | 49 (1.11%) | 54 (1.22%) | 2.33% | 1457 | 14 (0.96%) | 14 (0.96%) | 1.92% | 1530 | 52 (3.40%) | 21 (1.37%) | 4.77%
Wilcoxon Signed Rank Test | 5726 | 245 (4.28%) | 149 (2.60%) | 6.88% | 2004 | 115 (5.74%) | 44 (2.20%) | 7.93% | 1892 | 96 (5.07%) | 21 (1.11%) | 6.18%
Student’s t Test | 5623 | 384 (6.83%) | 178 (3.17%) | 9.99% | 2013 | 174 (8.64%) | 58 (2.88%) | 11.53% | 1883 | 163 (8.66%) | 21 (1.12%) | 9.77%
One-way ANOVA | 2159 | 667 (30.89%) | 59 (2.73%) | 33.63% | 498 | 116 (23.29%) | 3 (0.60%) | 23.90% | 820 | 237 (28.90%) | 0 (0.00%) | 28.90%
Kruskal-Wallis Test | 1827 | 34 (1.86%) | 19 (1.04%) | 2.90% | 418 | 20 (4.78%) | 2 (0.48%) | 5.26% | 718 | 25 (3.48%) | 2 (0.28%) | 3.76%
Two-way ANOVA | 3642 | 631 (17.33%) | 22 (0.60%) | 17.93% | 1120 | 172 (15.36%) | 38 (3.39%) | 18.75% | 1201 | 330 (27.48%) | 6 (0.50%) | 27.98%
Friedman Test | 2706 | 0 (0.00%) | 0 (0.00%) | 0.00% | 876 | 0 (0.00%) | 0 (0.00%) | 0.00% | 912 | 0 (0.00%) | 0 (0.00%) | 0.00%
nDCG_b10 | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$ | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$ | Sig | S2NS (%) | NS2S (%) | $\mathbf{\Delta\%}$
Sign Test | 5095 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1813 | 0 (0.00%) | 0 (0.00%) | 0.00% | 1755 | 0 (0.00%) | 0 (0.00%) | 0.00%
Wilcoxon Rank Sum Test | 4312 | 52 (1.21%) | 47 (1.09%) | 2.30% | 1453 | 15 (1.03%) | 20 (1.38%) | 2.41% | 1532 | 62 (4.05%) | 15 (0.98%) | 5.03%
Wilcoxon Signed Rank Test | 5632 | 250 (4.44%) | 148 (2.63%) | 7.07% | 2021 | 111 (5.49%) | 36 (1.78%) | 7.27% | 1851 | 65 (3.51%) | 50 (2.70%) | 6.21%
Student’s t Test | 5579 | 370 (6.63%) | 189 (3.39%) | 10.02% | 2012 | 162 (8.05%) | 52 (2.58%) | 10.64% | 1816 | 122 (6.72%) | 71 (3.91%) | 10.63%
One-way ANOVA | 2012 | 814 (40.46%) | 50 (2.49%) | 42.94% | 480 | 97 (20.21%) | 11 (2.29%) | 22.50% | 830 | 290 (34.94%) | 0 (0.00%) | 34.94%
Kruskal-Wallis Test | 1782 | 50 (2.81%) | 18 (1.01%) | 3.82% | 397 | 7 (1.76%) | 0 (0.00%) | 1.76% | 728 | 44 (6.04%) | 3 (0.41%) | 6.46%
Two-way ANOVA | 3478 | 670 (19.26%) | 48 (1.38%) | 20.64% | 1125 | 121 (10.76%) | 41 (3.64%) | 14.40% | 1203 | 281 (23.36%) | 42 (3.49%) | 26.85%
Friedman Test | 2637 | 0 (0.00%) | 0 (0.00%) | 0.00% | 882 | 0 (0.00%) | 0 (0.00%) | 0.00% | 925 | 0 (0.00%) | 0 (0.00%) | 0.00%

As in the case of the correlation among measures, a word of caution has to be
made remembering that in the case of measures depending on the recall base our
approach is just a surrogate, which improves the “intervalness” of a measure
but stretches the steps of the scale. Therefore, all the changes in the
significantly different system pairs should be taken as tendency rather than
exact quantification.

From a glance at Table 6 we can note as, in this case, also the significance
tests assuming just an ordinal scale, with the exception of the Sign and
Friedman tests, are affected by the transformation to an interval scale for an
overall change $\Delta\%$ up to a $51\%$. This further confirms that
aggregating across topics when the recall base changes can cause variations
which go well beyond the loss of “intervalness”.

As another general trend, we can see that significance tests based on an
interval scale assumption are generally more affected, since they experience
both the violation of their assumptions and the effect of the recall base.

If we consider Recall, we can see how the most prominent effect is the
underestimation of significant differences with a very big increase in the
number of significantly different pairs (NS2S), i.e. a reduction in the number
of false positives, up to a striking $558\%$ for the one-way ANOVA.
Considering that the interval scale behind Recall is the same as the one
behind Precision, these figures tell us how big is the loss of power for
Recall, mostly due to the impact of aggregating across topics with different
recall bases.

In the case of nDCG we can observe a behaviour quite similar to one of DCG,
with an overall change $\Delta\%$ just a bit bigger than the one of DCG.
Considering that both DCG and nDCG share the same interval scale, this further
suggests to use DCG to avoid the further bias due to the recall base.

##  8 Conclusions and Future Work

We have addressed the problem that IR measures typically are not interval
scales. This issue has severe consequences: you should neither compute means,
variances, and confidence intervals nor perform statistical significance tests
which assume an interval scale. We have provided a detailed discussion on the
motivations and needs behind the interval scales, both in the general field of
the representational theory of measurement and in the IR context in
particular, presenting viewpoints and opinions both supporting and opposing
these two “prohibitions”. It is a matter of fact that these two “prohibitions”
have been constantly overlooked in the IR community. However, when applying
improper methods, the results should not be called valid (according to general
scientific standards), especially before the impact of these violations has
been thoroughly investigated, as has so far been the case in IR.

The main motivation for IR measures not being interval scales is that their
values are not equi-spaced. Therefore, we have proposed a straightforward yet
powerful way to turn any measure into an interval scale by considering how all
the possible runs are ranked by the measure and keeping the unique ranks, i.e.
after removing tied values, as values of the mapped measures. These ranks are
equi-spaced by construction and preserve the same order of runs of the
original measure. In this way, we obtain an interval scale able to represent
the order of runs produced by the original measure.

We have also shown that the situation in IR is worsened by the fact that
mixing runs of different length and different recall bases for different
topics actually means mixing different scales, being them interval or not.
Therefore, computing aggregations across runs and topics in such a way can
lead to invalid results. While the run length issue can be mitigated by
ensuring that all the runs have the same length, the recall base one is more
problematic, since you cannot force a single recall base for all the topics.
Therefore, this discourages the use of measures depending on the recall base.

Overall, this discussion led us to raise the fundamental question that IR
should be more concerned with being able to rely on _meaningful statements_ ,
i.e. statements whose truth values do not change when you perform legitimate
transformations of the underlying scale, since they ensure for more valid and
generalisable inferences.

Relying on several TREC collections, we have conducted a thorough
experimentation on several (popular) state-of-the-art evaluation measures in
order to assess the differences between using an evaluation measure and its
interval-scaled version.

The correlation analysis has shown that the relationship between evaluation
measures and their interval-scaled version matches the expected theoretical
properties and that not using an interval scale somehow inflates the
differences among evaluation measures. Notably, RR represents an exception
since its departure from being an interval scale makes it look to be more
similar to other measures than what it actually is.

Most importantly, the correlation analysis provides us with a rough estimator
of how much interval scale an evaluation measure is and it represents the
first attempt to quantify how much evaluation measures depart from their
assumptions.

The analysis on many different types of statistical significance tests has
clearly shown the impact of passing from an evaluation measure to its
interval-scaled version. In particular, for measures not depending on the
recall base, the transformation provides benefits in terms of reduced Type I
error and some increase in power of the test. While for measures depending on
the recall base, it produces sizeable improvements in terms of Type II error
and power of the test, still delivering substantial enhancements in terms of
Type I error. Even apart from any interpretation in terms of Type I and Type
II errors, we observed an overall mean change around $25\%$ in the decision
about which systems are significantly different and which are not.

Our results on both the correlation analysis and the statistical significance
tests open the question about which claims and findings in the IR literature
would be impacted by these difference or, in other terms, which statements
made in IR so far would be actually _meaningful_.

The main limitation of the proposed approach is practical, since first you
need to generate all the possible runs of $REL^{N}$ and then you have compute
the evaluation measures on all these runs. For increasing values of $N$, and
even more in the case of multi-graded relevance, this becomes practically
infeasible. Therefore, our future work will concern approximating this
generation process in order to make it possible to deal with runs of whatever
length.

## References

  * Abelson and Tukey [1959] R. P. Abelson and J. W. Tukey.  Efficient Conversion Of Non-Metric Information Into Metric Information.  In _Proc. of the Social Statistics Section of the American Statistical Association_ , pages 226–230. American Statistical Association, Washington, USA, 1959.
  * Adams et al. [1965] E. W. Adams, R. F. Fagot, and R. E. Robinson.  A theory of appropriate statistics.  _Psychometrika_ , 30:99–127, June 1965.
  * Allan et al. [2018] J. Allan, D. K. Harman, E. Kanoulas, D. Li, C. Van Gysel, and E. M. Voorhees.  TREC 2017 Common Core Track Overview.  In E. M. Voorhees and A. Ellis, editors, _The Twenty-Sixth Text REtrieval Conference Proceedings (TREC 2017)_. National Institute of Standards and Technology (NIST), Special Publication 500-324, Washington, USA, 2018.
  * Allan et al. [2019] J. Allan, D. K. Harman, E. Kanoulas, and E. M. Voorhees.  TREC 2018 Common Core Track Overview.  In E. M. Voorhees and A. Ellis, editors, _The Twenty-Seventh Text REtrieval Conference Proceedings (TREC 2018)_. National Institute of Standards and Technology (NIST), Special Publication 500-331, Washington, USA, 2019.
  * Amigó and Mizzaro [2020] E. Amigó and S. Mizzaro.  On the nature of information access evaluation metrics: a unifying framework.  _Information Retrieval Journal_ , 23(3):318–386, June 2020.
  * Amigó et al. [2009] E. Amigó, J. Gonzalo, J. Artiles, and M. F. Verdejo.  A comparison of extrinsic clustering evaluation metrics based on formal constraints.  _Information Retrieval_ , 12(4):461–486, August 2009.
  * Amigó et al. [2013] E. Amigó, J. Gonzalo, and M. F. Verdejo.  A General Evaluation Measure for Document Organization Tasks.  In G. J. F. Jones, P. Sheridan, D. Kelly, M. de Rijke, and T. Sakai, editors, _Proc. 36th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2013)_ , pages 643–652. ACM Press, New York, USA, 2013.
  * Amigó et al. [2018] E. Amigó, D. Spina, and J. Carrillo-de Albornoz.  An Axiomatic Analysis of Diversity Evaluation Metrics: Introducing the Rank-Biased Utility Metric.  In K. Collins-Thompson, Q. Mei, B. Davison, Y. Liu, and E. Yilmaz, editors, _Proc. 41st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2018)_ , pages 625–634. ACM Press, New York, USA, 2018.
  * Amigó et al. [2019] E. Amigó, J. Gonzalo, M. F. Verdejo, and D. Spina.  A comparison of filtering evaluation metrics based on formal constraints.  _Information Retrieval Journal_ , 22(6):581–619, December 2019.
  * Anderson [1961] NN. H. Anderson.  Scales and Statistics: Parametric and Nonparametric.  _Psychological Bulletin_ , 58(4):305–316, 1961\.
  * Banks et al. [1999] D. Banks, P. Over, and N.-F. Zhang.  Blind Men and Elephants: Six Approaches to TREC data.  _Information Retrieval_ , 1(1-2):7–34, May 1999\.
  * Bollmann [1984] P. Bollmann.  Two Axioms for Evaluation Measures in Information Retrieval.  In C. J. van Rijsbergen, editor, _Proc. of the Third Joint BCS and ACM Symposium on Research and Development in Information Retrieval_ , pages 233–245. Cambridge University Press, UK, 1984.
  * Bollmann and Cherniavsky [1980] P. Bollmann and V. S. Cherniavsky.  Measurement-theoretical investigation of the MZ-metric.  In C. J. van Rijsbergen, editor, _Proc. 3rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 1980)_ , pages 256–267. ACM Press, New York, USA, 1980.
  * Bollmann and Cherniavsky [1981] P. Bollmann and V. S. Cherniavsky.  Restricted Evaluation in Information Retrieval.  In C. J. Crouch, W. S. Cooper, and J. Herr, editors, _Proc. 4th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 1981)_ , pages 15–21. ACM Press, New York, USA, 1981.
  * Buckley and Voorhees [2005] C. Buckley and E. M. Voorhees.  Retrieval System Evaluation.  In Harman and Voorhees [46], pages 53–78.
  * Busin and Mizzaro [2013] L. Busin and S. Mizzaro.  Axiometrics: An Axiomatic Approach to Information Retrieval Effectiveness Metrics.  In O. Kurland, D. Metzler, C. Lioma, B. Larsen, and P. Ingwersen, editors, _Proc. 4th International Conference on the Theory of Information Retrieval (ICTIR 2013)_ , pages 22–29. ACM Press, New York, USA, 2013.
  * Campbell [1920] N. R. Campbell.  _Physics: The Elements_.  Cambridge University Press, UK, 1920.
  * Campbell [1928] N. R. Campbell.  _An account of the principles of measurement and calculation_.  Longmans, Green, Lonndono, UK, 1928.
  * Carmel and Yom-Tov [2010] D. Carmel and E. Yom-Tov.  _Estimating the Query Difficulty for Information Retrieval_.  Morgan & Claypool Publishers, USA, 2010.
  * Carterette [2011] B. A. Carterette.  System Effectiveness, User Models, and User Utility: A Conceptual Framework for Investigation.  In W.-Y. Ma, J.-Y. Nie, R. Baeza-Yaetes, T.-S. Chua, and W. B. Croft, editors, _Proc. 34th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2011)_ , pages 903–912. ACM Press, New York, USA, 2011.
  * Carterette [2012] B. A. Carterette.  Multiple Testing in Statistical Analysis of Systems-Based Information Retrieval Experiments.  _ACM Transactions on Information Systems (TOIS)_ , 30(1):4:1–4:34, 2012.
  * Carterette et al. [2008] B. A. Carterette, V. Pavlu, E. Kanoulas, J. A. Aslam, and J. Allan.  Evaluation over Thousands of Queries.  In T.-S. Chua, M.-K. Leong, D. W. Oard, and F. Sebastiani, editors, _Proc. 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2008)_ , pages 651–658. ACM Press, New York, USA, 2008.
  * Chapelle et al. [2009] O. Chapelle, D. Metzler, Y. Zhang, and P. Grinspan.  Expected Reciprocal Rank for Graded Relevance.  In D. W.-L. Cheung, I.-Y. Song, W. W. Chu, X. Hu, and J. J. Lin, editors, _Proc. 18th International Conference on Information and Knowledge Management (CIKM 2009)_ , pages 621–630. ACM Press, New York, USA, 2009.
  * Cormack et al. [1998] G. Cormack, C. R. Palmer, and C. L. A. Clarke.  Efficient Construction of Large Test Collections.  In W. B. Croft, A. Moffat, C. J. van Rijsbergen, R. Wilkinson, and J. Zobel, editors, _Proc. 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 1998)_ , pages 282–289. ACM Press, New York, USA, 1998.
  * Cozby and Bates [2018] P. C. Cozby and S. C. Bates.  _Methods in Behavioral Research_.  McGraw-Hill Education, New York, USA, 13th edition, 2018.
  * Davey and Priestley [2002] B. A. Davey and H. A. Priestley.  _Introduction to Lattices and Order_.  Cambridge University Press, Cambridge, UK, 2nd edition, 2002.
  * European Commission [2009] European Commission.  Commission Regulation (EC) No 607/2009 of 14 July 2009 laying down certain detailed rules for the implementation of Council Regulation (EC) No 479/2008 as regards protected designations of origin and geographical indications, traditional terms, labelling and presentation of certain wine sector products.  _Official Journal of the European Union, OJ L 193, 24.7.2009_ , 52:60–139, July 2009.
  * European Commission [2019] European Commission.  Commission Delegated Regulation (EC) No 2019/33 of 17 October 2018 supplementing Regulation (EU) No 1308/2013 of the European Parliament and of the Council as regards applications for protection of designations of origin, geographical indications and traditional terms in the wine sector, the objection procedure, restrictions of use, amendments to product specifications, cancellation of protection, and labelling and presentation.  _Official Journal of the European Union, OJ L 9, 11.1.2019_ , 62:2–45, January 2019.
  * Falmagne and Narens [1983] J. C. Falmagne and L. Narens.  Scales and Meaningfulness of Quantitative Laws.  _Synthese_ , 55(3):287–325, June 1983.
  * Ferguson et al. [1940] A. Ferguson, C. S. Myers, R. J. Bartlett, H. Banister, F. C. Bartlett, W. Brown, N. R. Campbell, K. J. W. Craik, J. Drever, J. Guild, R. A. Houstoun, J. O. Irwin, G. W. C. Kaye, S. J. F. Philpott, L. F. Richardson, J. H. Shaxby, T. Smith, R. H. Thouless, and W. S. Tucker.  Quantitative estimates of sensory events: final report of the committee appointed to consider and report upon the possibility of quantitative estimates of sensory events.  _Advancement of Science_ , 2:331–349, 1940.
  * Ferrante et al. [2015] M. Ferrante, N. Ferro, and M. Maistro.  Towards a Formal Framework for Utility-oriented Measurements of Retrieval Effectiveness.  In J. Allan, W. B. Croft, A. P. de Vries, C. Zhai, N. Fuhr, and Y. Zhang, editors, _Proc. 1st ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR 2015)_ , pages 21–30. ACM Press, New York, USA, 2015.
  * Ferrante et al. [2017] M. Ferrante, N. Ferro, and S. Pontarollo.  Are IR Evaluation Measures on an Interval Scale?  In J. Kamps, E. Kanoulas, M. de Rijke, H. Fang, and E. Yilmaz, editors, _Proc. 3rd ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR 2017)_ , pages 67–74. ACM Press, New York, USA, 2017.
  * Ferrante et al. [2019] M. Ferrante, N. Ferro, and S. Pontarollo.  A General Theory of IR Evaluation Measures.  _IEEE Transactions on Knowledge and Data Engineering (TKDE)_ , 31(3):409–422, March 2019.
  * Ferrante et al. [2020] M. Ferrante, N. Ferro, and E. Losiouk.  How do interval scales help us with better understanding IR evaluation measures?  _Information Retrieval Journal_ , 23(3):289–317, June 2020.
  * Ferro [2017] N. Ferro.  What Does Affect the Correlation Among Evaluation Measures?  _ACM Transactions on Information Systems (TOIS)_ , 36(2):19:1–19:40, September 2017.
  * Ferro and Peters [2019] N. Ferro and C. Peters, editors.  _Information Retrieval Evaluation in a Changing World – Lessons Learned from 20 Years of CLEF_ , volume 41 of _The Information Retrieval Series_ , 2019. Springer International Publishing, Germany.
  * Fisher [1925] R. A. Fisher.  _Statistical Methods for Research Workers_.  Oliver & Boyd, Edinburgh, UK, 1925.
  * Friedman [1937] M. Friedman.  The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance.  _Journal of the American Statistical Association_ , 32(200):675–701, December 1937.
  * Friedman [1939] M. Friedman.  A Correction: The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance.  _Journal of the American Statistical Association_ , 34(205):109, March 1939.
  * Fuhr [2017] N. Fuhr.  Some Common Mistakes In IR Evaluation, And How They Can Be Avoided.  _SIGIR Forum_ , 51(3):32–41, December 2017\.
  * Gaito [1959] J. Gaito.  Non-Parametric Methods in Psychological Research.  _Psychological Reports_ , 5(1):115–125, March 1959.
  * Gaito [1980] J. Gaito.  Measurement Scales and Statistics: Resurgence of an Old Misconception.  _Psychological Bulletin_ , 87(3):564–567, 1980\.
  * Gardner [1975] P. L. Gardner.  Scales and Statistics.  _Review of Educational Research_ , 45(1):43–57, Winter 1975.
  * Gibbons and Chakraborti [2011] J. D. Gibbons and S. Chakraborti.  _Nonparametric Statistical Inference_.  Chapman & Hall/CRC, Taylor and Francis Group, Boca Raton (FL), USA, 5th edition, 2011.
  * Hand [1996] D. J. Hand.  Statistics and the Theory of Measurement.  _Journal of the Royal Statistical Society. Series A (Statistics in Society)_ , 159(3):445–492, 1996.
  * Harman and Voorhees [2005] D. K. Harman and E. M. Voorhees, editors.  _TREC. Experiment and Evaluation in Information Retrieval_ , 2005\. MIT Press, Cambridge (MA), USA.
  * Heine [1973] M. H. Heine.  Distance between sets as an objective measure of retrieval effectiveness.  _Information Storage and Retrieval_ , 9(3):181–198, March 1973.
  * Hochberg and Tamhane [1987] Y. Hochberg and A. C. Tamhane.  _Multiple Comparison Procedures_.  John Wiley & Sons, USA, 1987.
  * Hull [1993] D. A. Hull.  Using Statistical Testing in the Evaluation of Retrieval Experiments.  In R. Korfhage, E. Rasmussen, and P. Willett, editors, _Proc. 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 1993)_ , pages 329–338. ACM Press, New York, USA, 1993.
  * Järvelin and Kekäläinen [2002] K. Järvelin and J. Kekäläinen.  Cumulated Gain-Based Evaluation of IR Techniques.  _ACM Transactions on Information Systems (TOIS)_ , 20(4):422–446, October 2002.
  * Jiang and Allan [2016] J. Jiang and J. Allan.  Correlation Between System and User Metrics in a Session.  In D. Kelly, R. Capra, N. Belkin, J. Teevan, and P. Vakkari, editors, _Proc. 1st ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR 2016)_ , pages 285–288. ACM Press, New York, USA, 2016.
  * Kendall [1948] M. G. Kendall.  _Rank correlation methods_.  Griffin, Oxford, England, 1948.
  * Krantz et al. [1971] D. H. Krantz, R. D. Luce, P. Suppes, and A. Tversky.  _Foundations of Measurement. Additive and Polynomial Representations_ , volume 1.  Academic Press, New York, USA, 1971.
  * Kruskal and Wallis [1952] W. H. Kruskal and W. A. Wallis.  Use of Ranks in One-Criterion Variance Analysis.  _Journal of the American Statistical Association_ , 47(260):583–621, December 1952.
  * Kutner et al. [2005] M. H. Kutner, C. J. Nachtsheim, J. Neter, and W. Li.  _Applied Linear Statistical Models_.  McGraw-Hill/Irwin, New York, USA, 5th edition, 2005.
  * Lord [1953] F. M. Lord.  On the Statistical Treatment of Football Numbers.  _American Psychologist_ , 8(12):750–751, 1953\.
  * Losada et al. [2016] D. E. Losada, J. Parapar, and A. Barreiro.  Feeling Lucky? Multi-armed Bandits for Ordering Judgements in Pooling-based Evaluation.  In S. Ossowski, editor, _Proc. 2016 ACM Symposium on Applied Computing (SAC 2016)_ , pages 1027–1034. ACM Press, New York, USA, 2016.
  * Luce and Tukey [1964] R. D. Luce and J. W. Tukey.  Simultaneous Conjoint Measurement: A New Type of Fundamental Measurement.  _Journal of Mathematical Psychology_ , 1(1):1–27, January 1964.
  * Luce et al. [1990] R. D. Luce, D. H. Krantz, P. Suppes, and A. Tversky.  _Foundations of Measurement. Representation, Axiomatization, and Invariance_ , volume 3.  Academic Press, New York, USA, 1990.
  * Maddalena and Mizzaro [2014] E. Maddalena and S. Mizzaro.  Axiometrics: Axioms of Information Retrieval Effectiveness Metrics.  In S. Mizzaro and R. Song, editors, _Proc. 6th International Workshop on Evaluating Information Access (EVIA 2014)_ , pages 17–24. National Institute of Informatics, Tokyo, Japan, 2014.
  * Marcus-Roberts and Roberts [1987] H. M. Marcus-Roberts and F. S. Roberts.  Meaningless Statistics.  _Journal of Educational and Behavioral Statistics_ , 12(4):383–394, Winter 1987.
  * Michel [1986] J. Michel.  Measurement Scales and Statistics: A Clash of Paradigms.  _Psychological Bulletin_ , 100(3):398–407, 1986\.
  * Michel [1990] J. Michel.  _An Introduction to the Logic of Psychological Measurement_.  Lawrence Erlbaum Associates Inc., Mahwah (NJ), USA, 1990.
  * Moffat [2013] A. Moffat.  Seven Numeric Properties of Effectiveness Metrics.  In R. E. Banchs, F. Silvestri, T.-Y. Liu, M. Zhang, S. Gao, and J. Lang, editors, _Proc. 9th Asia Information Retrieval Societies Conference (AIRS 2013)_ , volume 8281, pages 1–12. Lecture Notes in Computer Science (LNCS) 8281, Springer, Heidelberg, Germany, 2013.
  * Moffat and Zobel [2008] A. Moffat and J. Zobel.  Rank-biased Precision for Measurement of Retrieval Effectiveness.  _ACM Transactions on Information Systems (TOIS)_ , 27(1):2:1–2:27, December 2008.
  * Moffat et al. [2013] A. Moffat, P. Thomas, and F. Scholer.  Users Versus Models: What Observation Tells Us About Effectiveness Metrics.  In A. Iyengar, Q. He, J. Pei, R. Rastogi, and W. Nejdl, editors, _Proc. 22nd International Conference on Information and Knowledge Management (CIKM 2013)_ , pages 659–668. ACM Press, New York, USA, 2013.
  * Narens [2002] L. Narens.  _Theories of Meaningfullness_.  Lawrence Erlbaum Associates, Mahwah (NJ), USA, 2002.
  * O’Brien [1985] R. M. O’Brien.  The Relationship Between Ordinal Measures and Their Underlying Values: Why All the Disagreement?  _Quality & Quantity_, 19(3):265–277, June 1985.
  * Roberts [1985] F. S. Roberts.  Applications of the Theory of Meaningfulness to Psychology.  _Journal of Mathematical Psychology_ , 29(3):311–332, September 1985.
  * Robertson [2006] S. E. Robertson.  On GMAP: and Other Transformations.  In P. S. Yu, V. Tsotras, E. A. Fox, and C.-B. Liu, editors, _Proc. 15th International Conference on Information and Knowledge Management (CIKM 2006)_ , pages 78–83. ACM Press, New York, USA, 2006.
  * Rowe et al. [2010] B. R. Rowe, D. W. Wood, A. L. Link, and D. A. Simoni.  _Economic Impact Assessment of NIST’s Text REtrieval Conference (TREC) Program_.  RTI Project Number 0211875, RTI International, USA. http://trec.nist.gov/pubs/2010.economic.impact.pdf, July 2010.
  * Sakai [2014] T. Sakai.  Statistical Reform in Information Retrieval?  _SIGIR Forum_ , 48(1):3–12, June 2014.
  * Sakai [2016a] T. Sakai.  A Simple and Effective Approach to Score Standardisation.  In B. A. Carterette, H. Fang, M. Lalmas, and J.-Y. Nie, editors, _Proc. 2nd ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR 2016)_ , pages 95–104. ACM Press, New York, USA, 2016a.
  * Sakai [2016b] T. Sakai.  Statistical Significance, Power, and Sample Sizes: A Systematic Review of SIGIR and TOIS, 2006-2015.  In R. Perego, F. Sebastiani, J. Aslam, I. Ruthven, and J. Zobel, editors, _Proc. 39th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2016)_ , pages 5–14. ACM Press, New York, USA, 2016b.
  * Sakai [2020] T. Sakai.  On Fuhr’s Guideline for IR Evaluation.  _SIGIR Forum_ , 54(1):p14:1–p14:8, June 2020\.
  * Sakai et al. [2021] T. Sakai, D. W. Oard, and N. Kando, editors.  _Evaluating Information Retrieval and Access Tasks – NTCIR’s Legacy of Research Impact_ , volume 43 of _The Information Retrieval Series_ , 2021. Springer International Publishing, Germany.
  * Salton and Lesk [1968] G. Salton and M. E. Lesk.  Computer Evaluation of Indexing and Text Processing.  _Journal of the ACM (JACM)_ , 15(1):8–36, January 1968.
  * Sauro and Lewis [2016] J. Sauro and J. R. Lewis.  _Quantifying the User Experience: Practical Statistics for User Research_.  Morgan Kaufmann Publisher, Inc., San Francisco, CA, USA, 2nd edition, 2016.
  * Savoy [1997] J. Savoy.  Statistical Inference in Retrieval Effectiveness Evaluation.  _Information Processing & Management_, 33(44):495–512, 1997.
  * Scholten and Borsboom [2009] A. Z. Scholten and D. Borsboom.  A reanalysis of Lord’s statistical treatment of football numbers.  _Journal of Mathematical Psychology_ , 53(2):69–75, April 2009.
  * Sebastiani [2020] F. Sebastiani.  Evaluation measures for quantification: an axiomatic approach.  _Information Retrieval Journal_ , 23(3):255–288, June 2020.
  * Senders [1958] V. L. Senders.  _Measurement and statistics: a basic text emphasizing behavioral science applications_.  Oxford University Press, New York, USA, 1958.
  * Siegel [1956] S. Siegel.  _Nonparametric Statistics: For the Behavioral Science_.  McGraw-Hill, New York, USA, 1956.
  * Singhal et al. [1997] A. Singhal, J. Choi, D. Hindle, and F. C. N. Pereira.  AT&T at TREC-6: SDR Track.  In E. M. Voorhees and D. K. Harman, editors, _The Sixth Text REtrieval Conference (TREC-6)_ , pages 227–232. National Institute of Standards and Technology (NIST), Special Publication 500-240, Washington, USA, 1997.
  * Stevens [1946] S. S. Stevens.  On the Theory of Scales of Measurement.  _Science, New Series_ , 103(2684):677–680, June 1946.
  * Student [1908] Student.  The Probable Error of a Mean.  _Biometrika_ , 6(1):1–25, March 1908.
  * Suppes et al. [1989] P. Suppes, D. H. Krantz, R. D. Luce, and A. Tversky.  _Foundations of Measurement. Geometrical, Threshold, and Probabilistic Representations_ , volume 2.  Academic Press, New York, USA, 1989.
  * Tague-Sutcliffe and Blustein [1994] J. M. Tague-Sutcliffe and J. Blustein.  A Statistical Analysis of the TREC-3 Data.  In D. K. Harman, editor, _The Third Text REtrieval Conference (TREC-3)_ , pages 385–398. National Institute of Standards and Technology (NIST), Special Publication 500-225, Washington, USA, 1994.
  * Townsend and Ashby [1984] J. T. Townsend and F. G. Ashby.  Measurement Scales and Statistics: The Misconception Misconceived.  _Psychological Bulletin_ , 96(2):394–401, 1984\.
  * Tukey [1949] J. W. Tukey.  Comparing Individual Means in the Analysis of Variance.  _Biometrics_ , 5(2):99–114, June 1949.
  * Urbano et al. [2019] J. Urbano, H. Lima, and A. Hanjalic.  A New Perspective on Score Standardization.  In B. Piwowarski, M. Chevalier, E. Gaussier, Y. Maarek, J.-Y. Nie, and F. Scholer, editors, _Proc. 42nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2019)_ , pages 1061–1064. ACM Press, New York, USA, 2019.
  * van Rijsbergen [1974] C. J. van Rijsbergen.  Foundations of Evaluation.  _Journal of Documentation_ , 30(4):365–373, 1974.
  * van Rijsbergen [1979] C. J. van Rijsbergen.  _Information Retrieval_.  Butterworths, London, England, 2nd edition, 1979.
  * van Rijsbergen [1981] C. J. van Rijsbergen.  Retrieval effectiveness.  In K. Spärck Jones, editor, _Information Retrieval Experiment_ , pages 32–43. Butterworths, London, United Kingdom, 1981.
  * Velleman and Wilkinson [1993] P. F. Velleman and L. Wilkinson.  Nominal, Ordinal, Interval, and Ratio Typologies Are Misleading.  _The American Statistician_ , 47(1):65–72, February 1993.
  * Voorhees [1998] E. M. Voorhees.  Variations in relevance judgments and the measurement of retrieval effectiveness.  In W. B. Croft, A. Moffat, C. J. van Rijsbergen, R. Wilkinson, and J. Zobel, editors, _Proc. 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 1998)_ , pages 315–323. ACM Press, New York, USA, 1998.
  * Voorhees [2000] E. M. Voorhees.  Variations in relevance judgments and the measurement of retrieval effectiveness.  _Information Processing & Management_, 36(5):697–716, September 2000.
  * Voorhees [2004] E. M. Voorhees.  Overview of the TREC 2004 Robust Track.  In E. M. Voorhees and L. P. Buckland, editors, _The Thirteenth Text REtrieval Conference Proceedings (TREC 2004)_. National Institute of Standards and Technology (NIST), Special Publication 500-261, Washington, USA, 2004.
  * Voorhees [2005] E. M. Voorhees.  Overview of the TREC 2005 Robust Retrieval Track.  In E. M. Voorhees and L. P. Buckland, editors, _The Fourteenth Text REtrieval Conference Proceedings (TREC 2005)_. National Institute of Standards and Technology (NIST), Special Publication 500-266, Washington, USA, 2005.
  * Voorhees [2018] E. M. Voorhees.  On Building Fair and Reusable Test Collections using Bandit Techniques.  In A. Cuzzocrea, J. Allan, N. W. Paton, D. Srivastava, R. Agrawal, A. Broder, M. J. Zaki, S. Candan, A. Labrinidis, A. Schuster, and H. Wang, editors, _Proc. 27th International Conference on Information and Knowledge Management (CIKM 2018)_ , pages 407–416. ACM Press, New York, USA, 2018.
  * Voorhees and Harman [1999] E. M. Voorhees and D. K. Harman.  Overview of the Eigth Text REtrieval Conference (TREC-8).  In E. M. Voorhees and D. K. Harman, editors, _The Eighth Text REtrieval Conference (TREC-8)_ , pages 1–24. National Institute of Standards and Technology (NIST), Special Publication 500-246, Washington, USA, 1999.
  * Ware and Benson [1975] W. B. Ware and J. Benson.  Appropriate Statistics and Measurement Scales.  _Science Education_ , 59(4):575–582, October/December 1975.
  * Webber et al. [2008] W. Webber, A. Moffat, and J. Zobel.  Score Standardization for Inter-Collection Comparison of Retrieval Systems.  In T.-S. Chua, M.-K. Leong, D. W. Oard, and F. Sebastiani, editors, _Proc. 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2008)_ , pages 51–58. ACM Press, New York, USA, 2008.
  * Wilcoxon [1945] F. Wilcoxon.  Individual Comparisons by Ranking Methods.  _Biometrics Bulletin_ , 1(6):80–83, December 1945.
  * Yilmaz et al. [2008] E. Yilmaz, J. A. Aslam, and S. E. Robertson.  A New Rank Correlation Coefficient for Information Retrieval.  In T.-S. Chua, M.-K. Leong, D. W. Oard, and F. Sebastiani, editors, _Proc. 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2008)_ , pages 587–594. ACM Press, New York, USA, 2008.
  * Zhang et al. [2017] F. Zhang, Y. Liu, X. Li, M. Zhang, Y. Xu, and S. Ma.  Evaluating Web Search with a Bejeweled Player Model.  In N. Kando, T. Sakai, H. Joho, H. Li, A. P. de Vries, and R. W. White, editors, _Proc. 40th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2017)_ , pages 425–434. ACM Press, New York, USA, 2017.
  * Zhang et al. [2020] F. Zhang, J. Mao, Y. Liu, X. Xie, W. Ma, M. Zhang, and S. Ma.  Models Versus Satisfaction: Towards a Better Understanding of Evaluation Metrics.  In Y. Chang, X. Cheng, J. Huang, Y. Lu, J. Kamps, V. Murdock, J.-R. Wen, A. Diriye, J. Guo, and O. Kurland, editors, _Proc. 43rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2020)_ , pages 379–388. ACM Press, New York, USA, 2020.

  *[IR]: Information Retrieval
  *[TREC]: Text REtrieval Conference
  *[CLEF]: Conference and Labs of the Evaluation Forum
  *[NTCIR]: NII Testbeds and Community for Information access Research
  *[FIRE]: Forum for Information Retrieval Evaluation
  *[AP]: Average Precision
  *[DCG]: Discounted Cumulative Gain
  *[nDCG]: Normalized Discounted Cumulative Gain
  *[RBP]: Rank-Biased Precision
  *[RR]: Reciprocal Rank
  *[ERS]: Empirical Relational System
  *[NRS]: Numerical Relational System
  *[ANOVA]: ANalysis Of VAriance
  *[MAP]: Mean Average Precision
  *[GMAP]: Geometric Mean Average Precision
  *[MRR]: Mean Reciprocal Rank
  *[gP]: Generalized Precision
  *[gR]: Generalized Recall
  *[gRBP]: Graded Rank-Biased Precision
  *[ERR]: Expected Reciprocal Rank
  *[RoS]: Rankings of Systems
  *[SBTO]: Set-Based Total Order
  *[RSV]: Retrieval Status Value
  *[RBTO]: Rank-Based Total Order
  *[HSD]: Honestly Significant Difference