# Optimal activity and battery scheduling algorithm using load and solar
generation forecasts

Yogesh Pipada Sunil Kumar  University of Adelaide
Adelaide, Australia
yogeshpipada.sunilkumar,  Rui Yuan  University of Adelaide
Adelaide, Australia
r.yuan,  Nam Trong Dinh  University of Adelaide
Adelaide, Australia
trongnam.dinh,  S. Ali Pourmousavi  University of Adelaide
Adelaide, Australia
<EMAIL_ADDRESS>
###### Abstract

Energy usage optimal scheduling has attracted great attention in the power
system community, where various methodologies have been proposed. However, in
real-world applications, the optimal scheduling problems require reliable
energy forecasting, which is scarcely discussed as a joint solution to the
scheduling problem. The 5th IEEE Computational Intelligence Society (IEEE-CIS)
competition raised a practical problem of decreasing the electricity bill by
scheduling building activities, where forecasting the solar energy generation
and building consumption is a necessity. To solve this problem, we propose a
technical sequence for tackling the solar PV and demand forecast and optimal
scheduling problems, where solar generation prediction methods and an optimal
university lectures scheduling algorithm are proposed.

###### Index Terms:

Forecasting, Refined motif, Optimisation, Valley-filling scheduling, Mixed-
integer linear programming (MILP)

##  I Introduction

###  I-A Motivation

Even though infrequent, peak load is a cause for increased capital and
operating expenses for power networks. The reason for this is twofold; the
higher need for grid reinforcements and the use of more expensive fossil fuel-
based generators to satisfy the peak loads (for a short duration). Thereby to
manage the extra costs due to peak demand, network operators include a peak
demand tariff (commensurate to the peak load) in the electricity bills of
commercial customers. This motivates large-scale commercial customers such as
universities and manufacturers to manage their demand better and invest in
assets such as solar photovoltaic (PV) panels and stationary batteries that
have the potential to shift their demand and reduce their electricity bills. A
secondary effect of this is the reduction in CO2 emissions because of the
lesser usage of fossil fuel generators, which has the potential to combat
climate change and aid the process of decarbonization of our power networks.

However, optimal scheduling of (schedulable) loads and batteries (to minimize
electricity costs) requires predictions of inflexible load (baseload), solar
PV generation and the spot price of electricity. The first two variables
majorly depend on weather conditions. Also, increasing renewable generation in
the generation mix has led to increasing price volatility in the Australian
national electricity market (NEM) which also creates a correlation between
electricity prices and the weather [1]. This makes designing such algorithms
challenging as most commercial activities are planned over the mid to long
term; hence require reliable predictions. Additionally, the scheduling
problems are generally mixed-integer linear programs (MILPs) which are NP-hard
and may be intractable depending on the formulation and problem size.

Surrounding this premise, the IEEE Computational Intelligence Society (IEEE-
CIS) partnered with Monash University (Victoria, Australia) to conduct a
competition seeking technical solutions to manage Monash’s micro-grid
containing rooftop solar panels and stationary batteries [2]. The main
challenge was to develop an optimal scheduling algorithm for Monash’s lecture
theatres and operation of batteries to minimize their electricity bill,
considering their baseload, solar generation and NEM electricity spot prices.
To this end, the contestants were provided with actual time series data of the
building loads without any lecture program (baseload) and solar generation
from the micro-grid. So the contestants were expected to predict the baseload
and solar generation by taking into account real-world weather data (also
provided) for one month in the future. Following this, the contestants had to
use actual electricity spot prices for the same duration along with these
predictions for the optimal scheduling algorithm.

###  I-B Related work

We developed separate algorithms for the solar and baseload forecasts based on
practical insights and the given data. Many approaches are available in the
literature for both data types, a relatively mature and crowded research
field. Therefore, the methods we have developed for this problem borrow themes
from this mature literature.

For solar predictions, the basic theme used is similar to the “clear sky”
models for forecasting solar irradiance and thereby estimating PV generation
[3]. The general idea here is to create a baseline model for PV generation,
assuming the sky is clear, meaning there is no cloud coverage or temperature
variance. This baseline is then modified based on actual or expected weather
conditions to estimate the actual PV generation. The literature around this
idea is based on physical models of irradiance calculations, where equations
are used to develop the baseline for a given geographical location [3]. Newer
methods use data-driven techniques such as time series forecasting and machine
learning models [4]. The data-driven methods are gaining popularity because of
their robustness, speed and geographic adaptability.

Intuitively, this method gives reliable solar forecasts because of solar
generation’s seasonal and diurnal nature. However, the main drawback is the
lack of data specifically related to “clear sky” days; Consequently, we use
the most commonly occurring day’s generation as baseline, which can be
discovered by our previous work [5].

For the baseload forecasting, the theme used is the application of ensemble
methods, as the current state of the art identifies these methods to be most
accurate when compared to stand-alone methods [6]. We use mainly a combination
of random forest (RF), gradient boost (GB), autoregressive integrated moving
average (ARIMA) and support vector machine (SVM), which are all well-studied
and standard methods for time series forecasting [7]. Specifically, we apply
different forecasting methods to different sub-series after disaggregating the
load profiles using Seasonal and Trend decomposition using Loess (STL) due to
their cyclic patterns [8].

The optimal scheduling of distributed energy resources (DER) and
controllable/uncontrollable loads for micro-grids comes under the class of
problems commonly referred to as energy management problems. However, these
classes of problems tend to be non-convex and non-linear in nature because of
the constraints associated with the scheduled devices, e.g., scheduling
uninterruptible loads such as lectures that cannot be stopped once started. As
reviewed by the authors in [9], the majority of these problems are modeled via
classical optimization approaches such as MILP [10] and mixed-integer non-
linear programs (MINLPs) [11]. Other approaches include dynamic programming
[12], rule-based optimization and meta-heuristic algorithms [13].

Since these problems tend to be non-convex and non-linear, the major challenge
associated with solving these problems is tractability and (in the case of
meta-heuristic methods) convergence to a global optimum. The mathematical
literature surrounding MILPs offers a variety of techniques to simplify these
problems before solving them, and the availability of solvers such as Gurobi®
for MILPs make them a very attractive option for solving these problems. Also,
unlike other models where the algorithms (may) converge to a local optimal
solution, modern MILP solvers can obtain globally optimal solutions for this
class of problems.

###  I-C Contributions

Based on the related work studied, this paper offers the following
contributions to the body of knowledge:

  * •

A solar forecasting algorithm using training data from refined motif (RM)
discovery technique and use of an over-parameterized 1D-convoluted neural
network (1D-CNN) implemented via residual networks (ResNet). To overcome the
lack of “clear sky” data, we incorporated the work done previously by Rui Yuan
et al. [5] to identify RMs in the given solar generation dataset. An RM is the
most repetitive pattern within a given time series, which can be extracted
along with the exogenous variables (e.g., weather information) associated with
this pattern. Using this as a baseline, we estimated solar generation by
training an over-parameterized 1D-CNN. Some studies have shown that over-
parameterization of CNNs can lead to better performance at the expense of
longer training time [14, 15, 16]. Therefore, a ResNet was implemented to
develop a deeper NN but with faster computation time.

  * •

An optimal micro-grid scheduling algorithm is solved based on real-world data
for a university-based application. To the best of our knowledge, there has
not yet been a study to co-optimize lectures schedule (and associated
resources) and battery operation (with PV panels). Therefore, in this paper,
we have developed and tested our algorithm using real-world data and practical
case instances provided by Monash University. Given the large problem size
(one-month schedule) and the presence of a quadratic term in the objective
function for the cost of peak demand, we proposed a two sub-problem approach
to formulate a tractable problem. The first sub-problem was used to limit the
peak demand throughout the month, eliminating the quadratic term from the
objective. Then, the peak demand was used to solve the second sub-problem to
minimize the total electricity costs.

The rest of the paper is structured as follows. Section II introduces the
data, information and problem requirements. Section III proposes the
forecasting and optimization methodologies. Section IV presents the numerical
results of the scheduling algorithm based on time series forecasting. Finally,
we conclude the paper section V.

##  II Background Information

This section describes the competition requirements and data provided by the
organizers.

###  II-A Problem statement

This section describes the scheduling constraints and objective function
provided by the competition organizers and has been adapted from [2]. We were
required to develop prediction algorithms for the baseload of six buildings in
Monash University and the solar generation of PV panels connected to them.
After this, we had to use these predictions to optimally schedule lecture
activities and battery operation while minimizing the electricity costs (i.e.,
electrical energy consumption cost plus peak demand charge). We were allowed
to consider the electricity prices as known parameters to simplify the problem
and focus on predicting solar generation and baseload.

For each lecture activity, we are provided with several small or large rooms
needed, the electrical power consumed per room and the duration of the
activities (in steps of 15 minutes). We are also provided with a list of
precedence activities, i.e., activities that must be performed at least one
day before the activity in question. For the batteries, we are provided with
the maximum energy rating, i.e., state of charge (SOC), the peak
charge/discharge power and charge/discharge efficiency.

The scheduling must start from the first Monday of the month, and each
activity must; 1) take place within office hours (9:00–17:00), 2) happen once
every week and recur on the same day and time every week. Also, the number of
rooms (associated with each activity) allocated at a given time interval must
not exceed the total number of rooms in the six buildings. The batteries can
operate in one of three states; charging, discharging or idle, which is
determined by the scheduling algorithm. However, the battery power must be at
the peak charge/discharge capacity once operated.

###  II-B Dataset information

The organizers provided us with datasets for the electricity consumption of
the six buildings and their connected solar panels [2]. About two years’ worth
of data was provided with a granularity of 15 minutes. However, the datasets
contained many missing points, especially the demand profiles with consecutive
weeks of unrecorded data, as shown in [17]. Information regarding COVID-19
lockdown and how it influenced the forecast and earlier timelines were also
provided to aid in incorporating the effect of COVID-19 pandemic on baseload
in our models. ERA5 climate data was also provided as exogenous variables for
the time series forecasting via a partnership with OikoLabs [2]. Also, we were
allowed to use electricity price information from the Australian NEM [18] as a
parameter for objective function calculation. The input parameters used for
solar and building forecasting are summarized in Table I.

TABLE I: Input data used for forecasting

Weather inputs (15mins) Forecast outputs (15mins) |  Building (15mins) |  Solar (15mins)
---|---|---
Temperature (∘C) | ✓ | ✓
Dewpoint temperature (∘C) | ✓ |
Wind speed (m/s) |  | ✓
Relative humidity (0-1) | ✓ | ✓
Surface solar radiation (W/$m^{2}$) | ✓ | ✓
Total cloud cover (0-1) |  | ✓
Occupancy (0-1) | ✓ |
Annual harmonics (0-1) |  | ✓

###  II-C Instance information

To test the algorithms each team developed, the competition organisers
provided ten problem instances. Each instance provided information regarding
the number of activities to be scheduled and the specifics of each activity;
duration of the activity in time steps, power consumed per room for each time
interval, number of large/small rooms required and a precedence list. The
precedence list contains the activities that must happen one day before the
specified task. Depending on the number of activities in each instance, they
were divided into two sub-categories; large (200 activities) and small (50
activities) instances. We report simulation results for one large and one
small instances in section IV.

##  III Methodology

This section describes the methodology developed to solve the problem
described in II-A. Mainly, we outline our data cleaning strategies,
forecasting methodology and scheduling algorithms used in the competition.

###  III-A Solar generation forecast

Figure 1: Block diagram of the proposed solar generation prediction model

The solar generation forecast methodology is depicted in Fig. 1. The solar
data was mainly clean, with no outliers and a few missing points, which were
removed during the cleaning process. Since solar time series data is
repetitive in nature, we used the method described in [5] to extract the RM
(most repetitive patterns) and its associated weather conditions. The
intuitive idea is to train a neural network (NN) to reshape this RM with
respect to the difference in weather conditions between the RM weather
conditions and the actual data weather conditions.

###  III-B Baseload forecast

Figure 2: Block diagram of the proposed baseload prediction model

The block diagram of the methodology used for baseload forecasting is shown in
Fig. 2, which demonstrates the use of different ensemble forecasting methods
for different buildings. We applied the same procedure for the data cleaning
process as in the PV generation forecast. However, since there were more
missing points in the demand profiles, which lasted for many weeks
consecutively, we used the average value from the previous week for the long-
period missing data. The choice of ensemble methods was achieved through the
use of the Python sklearn package’s voting regressor function [19]. For this
dataset, this resulted in a combination of RF, SVM, ARIMA and GB methods for
best accuracy. In terms of the length of training data, due to the highly
dynamic nature of demand profiles, which was exacerbated during the COVID-19
period, we only used 2 months of historical data for training.

Due to the seasonal/cyclic patterns demonstrated in building 1, 2 and 5, as
shown in [17], we utilized STL decomposition to capture these properties by
disaggregating the load profiles into three components, namely trend,
seasonality and residual. We then trained each profile with different methods
depending on the performance of the historical data. In the end, the
predictions from these three sub-series were combined to obtain the final
building demand forecast. For buildings 0 and 4 with no clear cyclic patterns,
we observed that either RF or GB (without STL decomposition) were sufficient
to capture their repetitive nature. Lastly, for building 3, historical data
showed that the load only took on a finite set of discrete values; thus, we
fixed the predictions to the median.

###  III-C Optimal scheduling algorithm

Figure 3: Block diagram of the proposed optimal scheduling algorithm

We developed a MIP-based optimal scheduling problem to represent all the
scheduling constraints, where the objective function was provided by the
organizers as follows:

$\displaystyle
O(P_{t})=\frac{0.25}{1000}\sum_{t\in\mathcal{T}}\,P_{t}\,\lambda^{\text{aemo}}_{t}+0.005\,\left(\max_{t\in\mathcal{T}}P_{t}\right)^{2}$
(1)

Here $P_{t}$ and $\lambda^{\text{aemo}}_{t}$ are the net demand and AEMO
electricity spot price at time $t$ over the time horizon $\mathcal{T}$
respectively. Therefore, it can be seen that the problem is a MIQP problem,
which is NP-hard and hence most likely intractable given the size of this
problem ($\|\mathcal{T}\|=2880$). Hence, to improve the tractability, we
reformulated the MIQP problem into two MILP sub-problems, as demonstrated in
Fig. 3. The first sub-problem is used to estimate an upper bound for the
maximum demand ($P^{\text{max}}_{1}=\max_{t\in\mathcal{T}}P_{t}$), which is
used to remove the quadratic term from the objective function in the second
sub-problem.

The scheduling is developed for a time duration $\Delta t$ with
$P^{\text{base}}_{t}\text{ and }P^{\text{solar}}_{t}$ being the baseload and
solar generation forecasts, respectively. For a set of batteries
$b\in\mathcal{B}$ over the time horizon $t\in\mathcal{T}$, we define
$E^{\text{bat}}_{b,t}$ as battery SOC in kWh, $P^{\text{bat}}_{b,t}$ is the
battery power in kW, $E^{\text{cap}}_{b}$ is the battery capacity in kWh,
$B^{\text{c}}_{b}/B^{\text{d}}_{b}$ is the battery charge and discharge power
in kW, $u^{\text{c}}_{b,t}/u^{\text{d}}_{b,t}$ are the battery
charge/discharge status, and $\eta^{\text{c}}_{b,t}/\eta^{\text{d}}_{b,t}$ are
the charging/discharging efficiencies. For a set of activities
$a\in\mathcal{A}$ over the same time horizon, we define $A^{\text{kW}}_{a}$ as
the power required per room for the activity in kW,
$R^{\text{large}}_{a}/R^{\text{small}}_{a}$ are the number of large/small
rooms needed for the activity, $P^{\text{sched}}_{t}$ be the total power
scheduled in kW, and $u^{\text{start}}_{a,t}/u^{\text{active}}_{a,t}$ are
binary variables which track the activity start time and active time,
respectively. $\mathcal{D}_{a}$ is the duration set for the activity, and
$\mathcal{P}_{a}$ is the set of activities that must precede activity $a$.
Additionally, we define $\mathcal{T}_{n}$ as the set of time intervals where
activities must not be active, $\mathcal{T}_{f}$ is the set of time intervals
for the first week and $L^{\text{day}}_{t}$ corresponds to the day of the time
interval $t$. We present the equations used for the two sub-problems in the
below sections.

####  III-C1 Sub-problem 1

$\displaystyle\min_{\psi_{1}}P^{\text{max}}_{1}$ (2a) Subject to,
$\displaystyle 0\leq P_{t}\leq
P^{\text{max}}_{1}\quad\forall\,t\in\mathcal{T}$ (2b) $\displaystyle 0\leq
E^{\text{bat}}_{b,t}\leq
E^{\text{cap}}_{b}\quad\forall\,t\in\mathcal{T},\,b\in\mathcal{B}$ (2c)
$\displaystyle u^{\text{c}}_{b,t}+u^{\text{d}}_{b,t}\leq
1\quad\forall\,t\in\mathcal{T},\,b\in\mathcal{B}$ (2d) $\displaystyle
P^{\text{bat}}_{b,t}=u^{\text{c}}_{b,t}\,\frac{B^{\text{c}}_{b}}{\sqrt{\eta_{b}^{\text{c}}}}-u^{\text{d}}_{b,t}\,B^{\text{d}}_{b}\,\sqrt{\eta_{b}^{\text{d}}}\quad\forall\,t\in\mathcal{T},\,b\in\mathcal{B}$
(2e) $\displaystyle
E^{\text{bat}}_{b,-1}=E^{\text{cap}}_{b}\quad\forall\,b\in\mathcal{B}$ (2f)
$\displaystyle
E^{\text{bat}}_{b,t}=E^{\text{bat}}_{b,t-1}+\Bigg{(}u^{\text{c}}_{b,t}\,B^{\text{c}}_{b}-u^{\text{d}}_{b,t}\,B^{\text{d}}_{b}\Bigg{)}\Delta
t\quad\forall\,t\in\mathcal{T},\,b\in\mathcal{B}$ (2g) $\displaystyle
P^{\text{sched}}_{t}=\sum_{a\in\mathcal{A}}u^{\text{active}}_{a,t}\,A^{kW}_{a}\,\Bigg{(}R^{\text{small}}_{a}+R^{\text{large}}_{a}\Bigg{)}\quad\forall\,t\in\mathcal{T}$
(2h) $\displaystyle
P_{t}=P^{\text{base}}_{t}-P^{\text{solar}}_{t}+P^{\text{sched}}_{t}+\sum_{b\in\mathcal{B}}P^{\text{bat}}_{b}\quad\forall\,t\in\mathcal{T}$
(2i)
$\displaystyle\sum_{a\in\mathcal{A}}u^{\text{active}}_{a,t}\,R^{\text{large}}_{a}\leq
H^{\text{large}};\sum_{a\in\mathcal{A}}u^{\text{active}}_{a,t}\,R^{\text{small}}_{a}\leq
H^{\text{small}}\;\forall t\in\mathcal{T}$ (2j) $\displaystyle
u^{\text{active}}_{a,t}=0;u^{\text{start}}_{a,t}=0\quad\forall\,t\in\mathcal{T}_{n},\,a\in\mathcal{A}$
(2k)
$\displaystyle\sum_{t\in\mathcal{T}_{f}}u^{\text{start}}_{a,t}=1\quad\forall\,a\in\mathcal{A}$
(2l) $\displaystyle u^{\text{start}}_{a,t}\leq
u^{\text{active}}_{a,t+k}\>\forall\,$ $\displaystyle
t\in\\{0,1,\dots,\left|\mathcal{T}\right|-\left|\mathcal{D}_{a}\right|-1\\},\,k\in\mathcal{D}_{a},a\in\mathcal{A}$
(2m)
$\displaystyle\sum_{t\in\mathcal{T}_{f}}u^{\text{active}}_{a,t}=\left|\mathcal{D}_{a}\right|\quad\forall\,a\in\mathcal{A}$
(2n)
$\displaystyle\sum_{t\in\mathcal{T}_{f}}\left(u^{\text{active}}_{a,t}\,L^{\text{day}}_{t}-u^{\text{active}}_{k,t}\,L^{\text{day}}_{t}\right)\geq
1\;\;\forall\,k\in\mathcal{P}_{a},a\in\mathcal{A}$ (2o) $\displaystyle
u^{\text{start}}_{a,t}=u^{\text{start}}_{a,t-\alpha}\quad\forall\,t\in\mathcal{T}_{f}^{c},\,a\in\mathcal{A}$
(2p)

where
$\psi_{1}=\\{P,P^{\text{bat}},P_{1}^{\text{max}},E^{\text{bat}},u^{\text{c}},u^{\text{d}},u^{\text{active}},u^{\text{start}}\\}$.
Equation (2b) helps find the peak demand of the schedule and prevents energy
export to the grid. Equations (2c)–(2g) represent battery related constraints.
Equations (2h)–(2i) are used to calculate the net energy imported from the
grid, and equation (2j) ensures the number of rooms engaged through activities
is within the limit. Equation (2k) ensures activities do not start outside
office hours and before the first Monday of the month, equation (2l) ensures
activities happen once a week, equations (2m)–(2n) ensure activities are
active for the necessary duration after starting, equation (2o) satisfies that
all precedence (prerequisite) activities happen a day before and equation (2p)
ensures activities recur at the same time every week. The objective here is to
minimize peak demand while satisfying all the scheduling constraints.

Since this problem is only used to solve for the upper bound maximum demand
$P^{\text{max}}_{1}$, it is not necessary to solve the full problem. We use
the property that the solution of the LP relaxation of a minimization-based
MILP problem is its lower bound ($y^{*}_{\text{MILP}}\geq y^{*}_{\text{LP}}$)
[20] to just solve the LP relaxation and use this for sub-problem 2, thereby
improving tractability of our method.

####  III-C2 Sub-problem 2

$\displaystyle\min_{\psi_{2}}\frac{0.25}{1000}\sum_{t\in\mathcal{T}}\,P_{t}\,\lambda^{\text{aemo}}_{t}$
(3a) $\displaystyle\textbf{Subject to, Equations~{}}\eqref{eqref:SOC
limit}\text{\\---}\eqref{eqref:Precedence constraints}$ $\displaystyle 0\leq
P_{t}\leq\beta^{\text{mul}}\,P^{\text{max}^{*}}_{1}$ (3b)

where $\psi_{2}=\left(\psi_{1}\setminus\\{P^{\text{max}}_{1}\\}\right)$. Using
equation (3b), we ensure that peak demand does not exceed
$\beta^{\text{mul}}P^{\text{max}^{*}}_{1}$ limiting the demand charge.
Intuitively, $\beta^{\text{mul}}\geq 1$ is a multiplier used to change the
tightness of the MILP, creating a trade-off between computation time and
electricity cost; higher values of $\beta^{\text{mul}}$ decreases computation
speed but increases the electricity cost (objective of the problem) and vice
versa.

((a)) Difference in the RM at and predicted generation weather conditions for
Nov $4^{\text{th}}$ 2020

((b)) RM vs. predicted solar generation for Nov $4^{\text{th}}$ 2020

((c)) Scheduled power, predicted net load and NEM spot price for Nov
$2^{\text{nd}}$ 2020

Figure 4: Prediction and optimization results

##  IV Results and Discussion

The simulations were run for November 2020 as per the competition stipulation
with $\Delta t=15\text{ min},\ \|\mathcal{T}\|=2880$ and other instance
information as specified in II-C. For the scheduling, two $\beta^{\text{mul}}$
values were chosen; 1.10 for small instances and 1.15 for large instances.
These values were obtained via trial and error. The prediction and
optimization algorithms were run in Python 3.8.8 using Gurobi® solver.

TABLE II: Baseload and PV generation prediction NRMSE Building # | Proposed | BI | Solar # | Proposed | BI
---|---|---|---|---|---
0 | 0.43 | 0.43 | 0 | 0.72 | 0.81
1 | 0.31 | 0.34 | 1 | 0.77 | 0.77
2 | 0.23 | 0.24 | 2 | 0.74 | 0.79
3 | 0.32 | 0.32 | 3 | 0.74 | 0.75
4 | 0.40 | 0.40 | 4 | 0.78 | 0.79
5 | 0.13 | 0.14 | 5 | 0.72 | 0.75

Since each of the twelve datasets has different scales, we decide to use the
normalised root mean square error (NRMSE), which has been used to facilitate
the comparison among time series with varying magnitudes [21]. The metric is
calculated as follows:

$\displaystyle\text{NRMSE}=\sqrt{\sum_{t}\frac{\hat{y}_{t}-y_{t}}{\|\mathcal{T}\|}}\times\frac{1}{\bar{y}}$
(4)

where $\hat{y}_{t}$ is the model predictions, $y_{t}$ is the model actual data
and $\bar{y}$ represents the mean of the actual data.

In order to demonstrate the superiority of our methods, we compare our
proposed techniques with the best individual (BI) prediction models that are
mentioned in section I-B. Please note that the BI models are picked from out-
of-sample validation on the actual datasets using the same inputs and training
length as the proposed methods. Table II displays the NRMSE of the actual data
in November 2020 between the proposed and the BI models. It can be seen that
the proposed models achieve lower errors in almost all the twelve profiles
except for the demand of buildings 0 and 4. As mentioned in section III-B,
these two profiles are predicted using either RF or GB due to their non-cyclic
behavior. Therefore, they have the highest errors among all the demand
profiles. For the PV generation time series, the proposed method also achieves
better prediction accuracy with quite consistent NRMSE due to the high
dependence of solar PV generation on weather data.

Fig. 4(a) and 4(b) are used to explain the intuition behind the RM method for
the PV generation; the former shows the difference between RM and a predicted
generation weather condition for November 4th, 2020 while the latter shows the
RM and the predicted generation profiles for the same day. We can see that as
the difference in temperature is positive, the predicted generation is lower
than the RM and vice versa. The ResNet is then trained with the changes in
weather conditions to learn how to reshape the RM to predict solar generation.

Fig. 4(c) shows the power scheduled for the small and large instances, the
predicted load and the NEM spot price profiles for Novemeber $2^{\text{nd}}$
2020\. It can be seen that during lower price periods (intervals 24–72), most
of the load is scheduled and the scheduled load is clipped to the value
obtained from sub-problem 1. During high price periods (intervals 72–88), the
scheduled load is lower than the predicted baseload due to battery operation.
This ensures that electricity cost is minimized.

TABLE III: Electricity cost from different forecasting results Forecast methods | Base case | BI | Proposed
---|---|---|---
Cost ($AUD) | 589356 | 466652 | 357210

The organizers provided a base schedule/electricity cost for the problem based
on no predictions and an $\epsilon$-greedy algorithm [2]. Table III compares
the electricity cost from three different forecast methods, where the base
case represents the electricity cost provided by the organizers; whereas BI
and the proposed cost represent the cost optimized on their respective
forecasts. Our methodology reduced this cost by roughly 23% and 40% compared
to the BI and base case, respectively. We were able to solve small instances
(158400 binaries) in an average time of 447 seconds per instance and large
instances (590400 binaries) in 1550 seconds per instance. It can be seen that
even though the problem size grew 3.73 times, the time to solve the problem
increased almost linearly and not exponentially with problem size as is the
case in MIPs. It further demonstrates the tractability and scalability of our
algorithm.

##  V Conclusion

In this paper, we developed an optimal scheduling solution for solving a real-
world electricity cost minimization problem for six buildings at Monash
University in Melbourne, Australia. A two-step solution was presented in this
paper, which included forecasting and an optimization step. Our solution with
energy forecast reduced the total cost of electricity roughly by 40%. We would
like to highlight that we placed fifth in the competition among 50
participants and were off from the best solution only by 4% with respect to
the objective value due to their better forecast. For future work, the
forecasting of baseload and solar generation can be improved by using modern
machine learning techniques, such as transformers and info-GANs. For the
optimization, a major improvement can be devising an algorithm to select
optimal $\beta^{\text{mul}}$ considering the trade-off between the tightness
(complexity) of the problem and cost minimization.

## References

  * [1] L. Downey, K. Green, J. Grice, D. McConnell, J. Steinert, and N. Twibill, “Untangling the nem: a policymaker’s guide to the national electricity market,” 2022.
  * [2] C. Bergmeir. (2022) Ieee cis predict + optimize competition. [Online]. Available: https://ieee-dataport.org/competitions/ieee-cis-technical-challenge-predictoptimize-renewable-energy-scheduling
  * [3] K. Palani, R. Kota, A. P. Azad, and V. Arya, “Blue skies: A methodology for data-driven clear sky modelling,” _IJCAI International Joint Conference on Artificial Intelligence_ , vol. 0, pp. 3777–3783, 2017.
  * [4] H. Long, Z. Zhang, and Y. Su, “Analysis of daily solar power prediction with data-driven approaches,” _Applied Energy_ , vol. 126, pp. 29–37, 2014.
  * [5] R. Yuan, S. A. Pourmousavi, W. L. Soong, G. Nguyen, and J. A. Liisberg, “Irmac: Interpretable refined motifs and binary classification for rooftops pv owners,” _arXiv preprint arXiv:2109.13732_ , 2021. [Online]. Available: https://arxiv.org/pdf/2109.13732.pdf
  * [6] E. Raviv, K. E. Bouwman, and D. Van Dijk, “Forecasting day-ahead electricity prices: Utilizing hourly prices,” _Energy Economics_ , vol. 50, pp. 227–239, 2015.
  * [7] T. Hong and S. Fan, “Probabilistic electric load forecasting: A tutorial review,” _International Journal of Forecasting_ , vol. 32, no. 3, pp. 914–938, 2016. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0169207015001508
  * [8] M. Theodosiou, “Forecasting monthly and quarterly time series using stl decomposition,” _International Journal of Forecasting_ , vol. 27, no. 4, pp. 1178–1195, 2011. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0169207011000070
  * [9] M. F. Zia, E. Elbouchikhi, and M. Benbouzid, “Microgrids energy management systems: A critical review on methods, solutions, and prospects,” _Applied Energy_ , vol. 222, pp. 1033–1055, 2018. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0306261918306676
  * [10] N. T. Dinh, S. A. Pourmousavi, S. Karimi-Arpanahi, Y. P. S. Kumar, M. Guo, D. Abbott, and J. A. R. Liisberg, “Optimal sizing and scheduling of community battery storage within a local market,” in _Proceedings of the Thirteenth ACM International Conference on Future Energy Systems_ , ser. e-Energy ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 34–46. [Online]. Available: https://doi.org/10.1145/3538637.3538837
  * [11] S. A. Helal, R. J. Najee, M. O. Hanna, M. F. Shaaban, A. H. Osman, and M. S. Hassan, “An energy management system for hybrid microgrids in remote communities,” in _2017 IEEE 30th Canadian Conference on Electrical and Computer Engineering (CCECE)_ , 2017, pp. 1–4.
  * [12] L. N. An and T. Quoc-Tuan, “Optimal energy management for grid connected microgrid by using dynamic programming method,” in _2015 IEEE Power & Energy Society General Meeting_, 2015, pp. 1–5.
  * [13] H. Li, C. Zang, P. Zeng, H. Yu, and Z. Li, “A genetic algorithm-based hybrid optimization approach for microgrid energy management,” in _2015 IEEE International Conference on Cyber Technology in Automation, Control, and Intelligent Systems (CYBER)_ , 2015, pp. 1474–1478.
  * [14] Y. Balaji, M. Sajedi, N. M. Kalibhat, M. Ding, M. Soltanolkotabi, S. Feizi, and L. Angeles, “UNDERSTANDING OVERPARAMETERIZATION IN GENERATIVE ADVERSARIAL NETWORKS,” 2021.
  * [15] S. Bubeck and M. Sellke, “A Universal Law of Robustness via Isoperimetry,” 2021\. [Online]. Available: http://arxiv.org/abs/2105.12806
  * [16] A. Power, Y. Burda, H. Edwards, I. Babuschkin, and V. Misra, “Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets,” 2021, pp. 1–9.
  * [17] FRESNO. (2022) Ieee-cis data challenge - fresno. [Online]. Available: https://gitlab.com/ryuan/ieee-cis-data-challenge-fresno/-/tree/main/data
  * [18] AEMO. (2022) Nem aggregated demand and price data. [Online]. Available: https://www.aemo.com.au/energy-systems/electricity/national-electricity-market-nem/data-nem/aggregated-data
  * [19] Sklearn. (2022) Voting Regressor. [Online]. Available: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.VotingRegressor.html
  * [20] J. P. Vielma, “Mixed integer linear programming formulation techniques,” _SIAM Review_ , vol. 57, no. 1, pp. 3–57, 2015.
  * [21] B. Das, B. Nair, V. K. Reddy, and P. Venkatesh, “Evaluation of multiple linear, neural network and penalised regression models for prediction of rice yield based on weather parameters for west coast of india,” _International journal of biometeorology_ , vol. 62, no. 10, pp. 1809–1822, 2018.


# Separating form and meaning: Using self-consistency to quantify task
understanding across multiple senses

Xenia OhmerΩ Elia BruniΩ111Shared senior authorship Dieuwke Hupkes∞111Shared
senior authorship
ΩOsnabrück University ∞ FAIR
{xenia.ohmer<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

At the staggering pace with which the capabilities of large language models
(LLMs) are increasing, creating future-proof evaluation sets to assess their
understanding becomes more and more challenging. In this paper, we propose a
novel paradigm for evaluating LLMs which leverages the idea that correct world
understanding should be consistent across different (Fregean) senses of the
same meaning. Accordingly, we measure understanding not in terms of
correctness but by evaluating consistency across multiple senses that are
generated by the model itself. We showcase our approach by instantiating a
test where the different senses are different languages, hence using
multilingual self-consistency as a litmus test for the model’s understanding
and simultaneously addressing the important topic of multilinguality. Taking
one of the latest versions of ChatGPT as our object of study, we evaluate
multilingual consistency for two different tasks across three different
languages. We show that its multilingual consistency is still lacking, and
that its task and world understanding are thus not language-independent. As
our approach does not require any static evaluation corpora in languages other
than English, it can easily and cheaply be extended to different languages and
tasks and could become an integral part of future benchmarking efforts.

##  1 Introduction

The staggering pace with which the capabilities of large language models
(LLMs) increased in the recent past comes with many questions related to what
kind of progress we are making on the road towards true machine intelligence
and human-level understanding. To assess such progress, practitioners often
rely on benchmarks that measure natural language understanding (e.g. Williams
et al., 2018; Nie et al., 2020), commonsense reasoning (e.g. Sap et al., 2019;
Bisk et al., 2020), or probe for factual knowledge (e.g. Hendrycks et al.,
2021), among other things. The extent to which such benchmarks can be used to
asesss whether LLMs “understand” language is widely debated (e.g. Mitchell and
Krakauer, 2023; Raji et al., 2021). Often mentioned concerns in this context
are that LLMs may learn specific lexical patterns rather than general
principles (e.g. Ray Choudhury et al., 2022) and, relatedly, that benchmark
scores may confuse competence in _form_ with competence in _meaning_ (e.g.
Heineman, ). More recently, the enormous amount of data that models are
trained on and the fact that this data is often not publically accessible
further increased the difficulty of assessing whether benchmarks really
quantify what they are meant to quantify. A benchmark always makes at least
some assumptions concerning what a model has seen in its training phase, and,
given the rapid changes on that front, it is difficult to design challenging
benchmarks that remain informative past training rounds of new models. This
process is further exacerbated by the fact that novel evaluation data may leak
into the training data of newly trained models111E.g. portions of the BIG-
Bench data (Srivastava et al., 2022) were inadvertently added to the GPT4
training corpus (OpenAI, 2023, footnote 5).– which even the most future-
proofed designed benchmarks may not withstand.

In this paper, we propose a novel approach to evaluate models’ task or world
understanding that aims to create some separation between form and meaning in
benchmarking and simultaneously mitigates the challenging evaluation-
contamination loop. Our method is grounded on the idea that language is used
to describe or act in the world (Wittgenstein, 1953) and that this world
functions as an anchor for diverse linguistic forms. Having a genuine
understanding of that world thus implies consistency among different
linguistic expressions that pertain to the same entities within the world. To
give an example, if you ask a colleague who is fluent in both French and
English if a particular statement is true, you expect their answer to be
invariant to the language (French or English) in which you ask this question.
We leverage this intuition to investigate whether models have a consistent
world model across different senses (in the case above: languages) and –
consequently – a consistent understanding of the tasks that they are asked to
execute. Crucially, rather than generating different senses ourselves, we ask
the model to create different versions of the same question. This ensures that
potential inconsistencies are really due to model-internal inconsistencies
rather than misinterpretations of the question. Additionally, the method is
(more) data-contamination proof: as the different senses are regenerated for
every evaluation, there is no data that can be contaminated in new training
rounds. Lastly, it can cheaply and easily be applied to any already available
benchmark and therefore reduces the burden on data generation.

Figure 1: Illustration of the basic mechanism of our paradigm: We use the
model to generate other senses of the original input. The model’s answers on
the original input and the alternative sense are used to evaluate its
consistency. In this example, the model is presented with the task of
paraphrase detection in English (sentences taken from PAWS-X) and generates
another sense by translating from English to German.

Our approach can be applied to a number of different senses. Here, we showcase
it focusing on the multilingual case described in the example above, by asking
whether models are consistent across different languages (see Figure 1). In
essence, we are thus using multilingual self-consistency as a litmus test for
their understanding, simultaneously addressing the important topic of
multilinguality. Taking one of the latest SOTA versions of
ChatGPT222https://openai.com/blog/chatgpt as our object of study, we evaluate
multilingual understanding for two different tasks – paraphrase detection and
natural language inference – across three different languages (Chinese, German
and English). It turns out that the model is inconsistent across all languages
and tasks, despite being able to perform the tasks reasonably well in English
and generating high-quality translations. Taken together, our analyses provide
strong evidence that the model’s task understanding is modulated by the
representational form of the task.

In sum, we make the following contributions:

  * i)

We propose a novel, cheap, data contamination-proof evaluation paradigm to
evaluate understanding through multisense consistency of LLMs;

  * ii)

We showcase this paradigm by implementing a particular instantiation that
utilises _multilinguality_ to create different senses;

  * iii)

We use this instantiation to evaluate a recent SOTA LLM, illustrating that its
multilingual consistency is still lacking.

  * iv)

With a range of ablation experiments (see Figure 2), we demonstrate that the
observed inconsistencies can be ascribed to a language-dependent task
understanding (rather than an inability to translate or perform the task)

With our work, we hope to not only present an interesting set of empirical
results on multilingual consistency but also propose a novel, easily
applicable method to generate many more challenging evaluation tests.

##  2 Related work

Existing benchmarks for evaluating language understanding in LLMs form the
foundation for our work. The main idea of our paradigm is to evaluate LLMs in
terms of their consistency across different forms of the same meaning (given
by the original datapoint) and is therefore related to other work on self-
consistency in LLMs. In creating multiple senses through translation, there is
also a close connection between our execution of this paradigm and
multilingual evaluation. In Appendix A, we provide a GenBench eval card
(Hupkes et al., 2023) that classifies our work in the context of
generalisation research.

#### Evaluating language understanding.

A wide range of benchmark tasks has been developed to evaluate specific
aspects of natural language understanding in LLMs. To evaluate general
language understanding across diverse tasks, genres, and datasets, several of
these tasks have been combined into multi-task benchmarks, such as GLUE Wang
et al. (2018) or SuperGLUE Wang et al. (2019a). These benchmarks focus on
English and evaluate, among others, paraphrase identification (e.g. PAWS;
Zhang et al., 2019), natural language inference (e.g. MNLI; Williams et al.,
2018), and commonsense reasoning (e.g. COPA; Roemmele et al., 2011). In
response to the rapid improvements of LLMs on these benchmarks, despite a
generally subhuman language understanding, other multi-task benchmarks have
been developed. MMLU, for example, assesses world knowledge and problem-
solving ability across a large number of subjects that humans learn, covering
STEM, humanities, social sciences, and more Hendrycks et al. (2021). In line
with the continuous development of novel multi-task benchmarks, our paradigm
makes an effort to find more appropriate evaluation methods. Notably, we do
not only assess performance but also evaluate the model’s ability to
consistently solve the tasks across multiple languages, thereby drawing
conclusions about its ability to abstract from specific representational
forms.

#### Self-consistency in LLMs.

Various studies have shown that inconsistencies are common in LLMs (and
suggested methods for improving consistency, which is not our focus). These
studies are mostly concerned with self-consistency in natural language
inference (NLI) (e.g. Minervini and Riedel, 2018; Wang et al., 2019b; Li et
al., 2019; Hosseini et al., 2021), question answering (e.g. Kassner and
Schütze, 2020; Alberti et al., 2019; Mitchell et al., 2022; Chen et al., 2021;
Elazar et al., 2021; Kassner et al., 2021; Asai and Hajishirzi, 2020; Hosseini
et al., 2021), as well as arithmetic and commonsense reasoning (e.g. Wang et
al., 2023). For example, Kassner et al. (2021) created a dataset to measure a
model’s consistency by evaluating its responses to sentence pairs that are
subject to certain constraints (e.g. if X is a dog is true, X has a tail must
also be true). More similar to our work, Elazar et al. (2021) studied whether
factual knowledge in masked language models is invariant to paraphrasing. To
this end, they created ParaRel, a dataset containing cloze-style English
paraphrases (e.g. Homeland originally aired on [MASK], Homeland premiered on
[MASK]). So in these two examples, consistency is either evaluated against a
network of logical relationships between beliefs or by generating different
forms of the same meaning through paraphrasing. BECEL (Jang et al., 2022) is a
benchmark that can be used to evaluate these two types of consistency (logical
and semantic) across various tasks. For each task, the benchmark provides an
alternative version (e.g. for semantic consistency the inputs are paraphrased)
to compare the model’s answers across task instances. This benchmark has
recently been used to evaluate ChatGPT, showing that it is more consistent for
negations than other PLMs, but still likely to generate different responses to
paraphrases of the same meaning (Jang and Lukasiewicz, 2023). Unlike previous
work – with the exception of (Jang and Lukasiewicz, 2023) – we focus on true
_self_ -consistency: Different forms of the same meaning are generated by the
model itself, rather than externally. Furthermore, we evaluate self-
consistency through a multilingual lens.

#### Multilingual evaluation.

The development of cross- and multi-lingual LLMs has spurred interest in
multilingual evaluation beyond translation. Several multilingual versions of
benchmark tasks have been generated, such as PAWS-X Yang et al. (2019), XCOPA
Ponti et al. (2020), and XNLI Conneau et al. (2018) – usually through expert
translations from the original task (for a more expansive summary, we refer to
Hupkes et al., 2023, Appendix D). In addition, multilingual tasks have been
combined to form multilingual multi-task benchmarks, including XTREME Hu et
al. (2020), XTREME-R Ruder et al. (2021), and XGLUE Liang et al. (2020). All
of these benchmarks reveal language-dependent differences in performance for
current multilingual LLMs. Our approach is different in that we aim to
evaluate self-consistency by detecting language-dependent changes in model
performance, relying on the model’s own translations instead of external
translations.

##  3 Methods

We now proceed with describing the model we use for our experiments (§ 3.1),
the benchmark data we use for our experiments (§ 3.2), and the procedure we
use for extracting translations from the model (§ 3.3).

###  3.1 Model and hyperparameters

We showcase our paradigm using gpt-3.5-turbo-0301, a specific snapshot of
gpt-3.5-turbo from March 1st, 2023. Working with the snapshot ensures that the
model parameters do not change between experiments. We use the default
parameters but set the temperature to $0.25$. We found a low temperature to
yield model responses that more closely match the template answers for
benchmarking, as well as model translations that better capture the meaning of
the source sentences. In addition, we set the maximal number of generated
tokens to $256$ for benchmarking, which requires only very short answers, and
$2048$ for translation.

###  3.2 Benchmarking

#### Tasks and languages.

We evaluate understanding using the multilingual benchmarks PAWS-X and XNLI
(test splits). While our paradigm does not require parallel multilingual
datasets, we use them here to evaluate translation quality, compare
translations between two languages in both directions, and analyse differences
that arise from using model-internal instead of model-external translations.
PAWS-X is an adversarial paraphrase identification task, consisting of
sentence pairs created by word-swapping, resulting in negative pairs that have
clearly distinct meanings, but a high lexical overlap (see, for instance, the
example in Figure 1). The English sentences are drawn from Wikipedia and Quora
and translated into six other languages to create a multilingual benchmark.
XNLI, on the other hand, is an NLI benchmark spanning 15 different languages.
The datasets contain sentence pairs where one sentence entails the other,
contradicts it, or neither of the two (neutral). The sentences are drawn from
various sources, including transcripted face-to-face and telephone
conversations. Importantly, on either task, the model’s judgment should not be
dependent on nuances in meaning that may be lost in translation. For our
experiments, we focus on the German, English, and Chinese partitions of the
respective benchmarks.

#### Instructions.

We design task instructions in English to evaluate the model’s zero-shot
performance. Given that the benchmarks are binary/ternary classification
problems, the instructions can be formulated such that the model’s responses
can be easily standardised and evaluated:

  * •

PAWS-X: Do the following sentences have the same meaning? Sentence 1:
“[sentence_1]” Sentence 2: “[sentence_2]” Please answer with “yes” or “no”.

  * •

XNLI: Given the following sentence pair, which one of the following is true:
(A) the first sentence entails the second sentence, (B) the first sentence
contradicts the second sentence, or (C) neither of the two? Sentence 1:
“[sentence_1]” Sentence 2: “[sentence_2]” Please answer with “A”, “B”, or “C”.

In addition, these instructions are translated into German and Chinese by
native speakers (see Appendix B), in sum giving us ground truth input data and
instructions in each language.

#### Evaluation.

We process each input in a separate request. We only accept model responses
matching the template answer (e.g. “yes”) or containing it (e.g. “Yes, the
sentences have the same meaning.”), ignoring casing.333Only a negligible
amount of responses do not fall into one of these two categories ($<1\%$).
These are mapped onto an additional label indicating invalidity. In the second
case, we apply a semi-automatic standardisation procedure: a function maps the
model’s responses to one of the template answers, and these mappings are
checked, and if necessary corrected, by hand. Using the standardised
responses, we can calculate the model’s accuracy on the task, as well as the
model’s consistency between different runs.

###  3.3 Model-internal translations

We experiment with translations from English to Chinese and German, and from
Chinese and German to English. The original English, Chinese, and German model
inputs serve as baselines for our simulations. Our main goal is to evaluate
the consistency between the model’s responses on these baselines and the
model’s responses on a model-internally generated translation, always
comparing the source language baseline to the translation from source to
target.

#### Translations procedure and notation.

We generate model-internal (zero-shot) translations of both input data and
instructions. The translation instructions, written by native speakers (see
Appendix C), are always given in the source language. For the task
instructions, the model translates the instruction prefix (e.g. Do the
following sentences have the same meaning?), the word for sentence (e.g.
sentence), and the instruction suffix (e.g. Please answer with “yes” or “no”.)
in separate requests, and we recompose these translations to an instruction in
the target language (see Appendix B). For the input data, the model translates
each sentence per input sentence pair in a separate request.

In what follows, we will denote the instruction of a task $T$ with $I$ and the
input to which it is applied with $X$. We annotate the language in which
either of those is given with a subscript, which also indicates whether or not
it is a model translation from another language. Thus, $T_{en}$ refers to the
scenario in which both the instruction and the input are given in English,
using the original benchmark data, while $T_{en\rightarrow de}$ denotes the
model’s translation of instruction and input sentences from English to German.
Further, $I_{en\rightarrow de}$ indicates an instruction in German, translated
from English by the model, and $I_{en}/X_{en\rightarrow de}$ denotes the
scenario in which the original English instruction is used, but the input data
is German translated from English by the model.

#### Evaluation.

The model’s translations of the task instructions were reviewed by native
speakers who found the translations to be appropriate, apart from slight
deviations in the translations from Chinese to English: For PAWS-X the
instructions mention a single sentence instead of a sentence pair (Does the
following sentence have the same meaning?) and for XNLI the word covers is
used for entails. To evaluate the quality of the model’s translations of the
actual input sentences, we employ BLEU scores Papineni et al. (2002)
calculated with SacreBLEU Post (2018), as well as ROUGE Lin (2004), and
COMET-22 444Model wmt22-comet-da: https://huggingface.co/Unbabel/wmt22-comet-
da Rei et al. (2022) scores (see Section 4.3).

Figure 2: An overview of our experiments and analyses. In the core experiment
(upper grey box) the model is used to translate the instruction ($I_{L1}$) and
the task input ($X_{L1}$) from the source language $L_{1}$ to the target
language $L_{2}$. We use additional experiments (lower grey box) to separately
study the effects of translating either the task data or the instruction. We
analyze the model’s consistency for the core experiment (§ 4.1) and the mixed
translations (§ 4.2), its translation ability (§ 4.3), and its task peformance
(§ 4.4).

##  4 Results

In this section, we discuss the results of our experiments (see Figure 2),
beginning with our primary experiment where we assess how consistent the
model’s task understanding is across languages (§ 4.1). We further examine
these findings through a series of experiments that investigate the individual
effects of translating the dataset or the instruction (§ 4.2). In subsequent
analyses, we ensure that inconsistencies do not arise due to inaccurate
translations (§ 4.3) or poor task performance (§ 4.4). Instead, we find that
the model’s model’s task understanding varies based on the linguistic form
(here, language) chosen to represent the same meaning.

###  4.1 Multilingual consistency

In our primary experiment, we assess the consistency of a model’s
understanding by comparing the model’s responses in a monolingual setting –
with the original input data and instruction language – with its responses
when using a model-internal translation of those. Crucially, as the used
translation is produced by the model itself rather than externally, we can
exclude the scenario in which the model produces different answers because it
assigns different interpretations to the sentences.

The results are reported in column T (_Task_) of Table 1. As we can see, there
is not a single case where the scores are near-maximal, indicating that the
task understanding of the model is not consistent across the evaluated
languages. Regarding the language pairs, consistencies tend to be higher when
translating between English and German compared to English and Chinese, with
an exception for translations to English on XNLI (bottom rows).

| Consistency
---|---
| Src$\rightarrow$Tgt | $T$ | $I$ | $X$
PAWS-X | en$\rightarrow$de | 0.84 | 0.93 | 0.85
en$\rightarrow$zh | 0.76 | 0.91 | 0.79
de$\rightarrow$en | 0.86 | 0.93 | 0.86
zh$\rightarrow$en | 0.70 | 0.87 | 0.75
XNLI | en$\rightarrow$de | 0.74 | 0.81 | 0.76
en$\rightarrow$zh | 0.67 | 0.77 | 0.71
de$\rightarrow$en | 0.63 | 0.69 | 0.81
zh$\rightarrow$en | 0.67 | 0.79 | 0.72
Table 1: Consistency between baseline (input data and instruction in source
language) and model-internal translation from source (src) to target (tgt)
language. The first column shows the consistency for translating input data
and instruction (T). The other two columns show the consistencies for
translating only the instructions (I) or only the input data (X), while the
respective other component remains unchanged from the original source.

###  4.2 Interpretation versus execution consistency

When an inconsistency in the model’s responses occurs when transitioning from
one language to another, we need to determine whether it is due to an
inadequate understanding of what it is asked to do in the target language or
an inability to perform what it is asked to do in that language.555To
exemplify this, compare the scenario in which you are asked to judge whether
one English sentence implies the other, but the request is made in a language
that you do not have a great command of with the scenario in which the
question is asked in English, but the sentences to be judged are in a language
you do not understand well. We differentiate these effects by assessing the
model’s consistency when translating only the instruction, while retaining the
original input sentences (e.g. comparing $T_{en}$ and $I_{en\rightarrow
de}/X_{en}$) and its consistency when translating only the input sentences,
while preserving the original instruction (e.g. comparing $T_{en}$ and
$I_{en}/X_{en\rightarrow de}$). We show the results shown in Table 1.

Neither consistencies for translating only the instructions (column I) nor
those for translating only the input sentences (column X) are at their
maximum, indicating that the model is inconsistent in both interpretation and
execution. Inconsistencies are consistently higher for PAWS-X than XNLI,
probably because PAWS-X is a binary and XNLI a ternary classification problem.
However, even translating a simple instruction, such as the one for PAWS-X,
leads to inconsistencies for all translations. Consistencies seem to decrease
more when translating the input sentences compared to the instructions (except
for German to English on XNLI) and even further when translating both (column
T). Thus, inconsistencies in complete translations seem to be driven by
differences in both task interpretation and execution, although differences in
execution are more pronounced.

| Src$\rightarrow$Tgt | BLEU | Rouge1 | Rouge2 | Rouge-l | COMET-22
---|---|---|---|---|---|---
PAWS-X | en$\rightarrow$de | 56.5 | 0.80 | 0.64 | 0.77 | 0.89
en$\rightarrow$zh | 49.2 | 0.68 | 0.42 | 0.62 | 0.86
de$\rightarrow$en | 60.0 | 0.87 | 0.72 | 0.83 | 0.88
| zh$\rightarrow$en | 37.6 | 0.73 | 0.49 | 0.66 | 0.85
XNLI | en$\rightarrow$de | 41.4 | 0.71 | 0.52 | 0.68 | 0.88
en$\rightarrow$zh | 43.5 | 0.66 | 0.39 | 0.62 | 0.87
de$\rightarrow$en | 45.8 | 0.76 | 0.57 | 0.74 | 0.89
| zh$\rightarrow$en | 28.0 | 0.61 | 0.37 | 0.57 | 0.86
Table 2: Evaluation of the model-internal translation of the input data.

###  4.3 Consistency and translation quality

The metric we propose in some way conflates monolingual task understanding and
translation quality: inconsistencies can be driven by misalignment in task
understanding, but also by poor translation quality.666The two are related but
distinct. While a bad task understanding can lead to a bad translation, a bad
translation might also arise from a generally low understanding of the target
language (due to a lack of sufficient training data), or an inability to
translate. While both are important, and the metric therefore favours models
that do well across the board, it is worth further investigating which of the
two drives the observed inconsistencies: the translation quality or the
alignment.

We start by considering the hypothesis that the model’s consistency is
suboptimal simply because it is not a good model of translation. If the
translation quality is poor, inconsistencies may arise from differences in
meaning between original and translated inputs, rather than a language-
dependent task understanding. To evaluate the model’s translation quality
specifically on the benchmark data, we examine the translations of the input
data for all languages and directions using commonly adapted metrics for
translation quality (see Table 2).

All metrics indicate that the model’s translations are of high quality across
tasks and languages, with the sole exception of translations from Chinese to
English. The scores are generally higher for PAWS-X than XNLI, which might be
due to the more challenging and diverse text sources used in generating XNLI.
The high scores thus suggest that, for most of the considered source-target
language combinations, inconsistencies cannot be ascribed to changes in
meaning induced by the translation.

|  | en$\rightarrow$de | en$\rightarrow$zh | de$\rightarrow$en | zh$\rightarrow$en
---|---|---|---|---|---
PAWS-X |  $\rho$(BLEU, consistency) | 0.02 | 0.07 | 0.06 | 0.03
$\rho$(BLEU, accuracy) | 0.08 | 0.07 | 0.07 | 0.04
XNLI |  $\rho$(BLEU, consistency) | 0.03 | 0.02 | 0.05 | 0.09
$\rho$(BLEU, accuracy) | 0.01 | 0.02 | 0.03 | 0.05
Table 3: Pearson correlation between translation quality (BLEU) and
consistency between the model’s responses on original and translated inputs,
and Pearson correlation between translation quality (BLEU) and accuracy on the
translated inputs. Here, we use model-internal translations of the input
sentences ($X_{source\rightarrow target}$, always paired with an instruction
in the original language ($I_{source})$.

To further substantiate this claim, we compute the Pearson correlations
between the BLEU score of the translation and the (binary) consistencies
between the model’s original responses and its responses on the translated
benchmark data (see Table 3, top row for each task). We focus on the
simulations with model-internal translations of the input sentences, keeping
the instruction in the source language (e.g. $I_{en}/X_{en\rightarrow de}$).
For these simulations, we can obtain a translation quality score per data
point, which is not confounded with the translation quality of the
instruction. The BLEU score for a given data point is calculated by averaging
the scores for the translation of each sentence from the sentence pair
provided by the data set. All correlations are positive, yet, the absolute
values are very low ($\leq 0.09$). These findings suggest that the observed
inconsistencies are largely independent of the translation quality, at least
in light of the generally high translation quality observed for this specific
model. Additional evidence is presented in Appendix D, revealing significant
inconsistencies even when exclusively using the best translations. Note that
correlations between translation quality and accuracy on the translated inputs
are also very low (see Table 3, bottom row for each task), further indicating
that translation quality is generally sufficient to not affect the model’s
performance on the target language.

###  4.4 Consistency and performance

In the previous subsection, we saw that, for the languages we consider,
inconsistencies are not driven by poor translation quality. What we have not
done so far, is look at the actual model accuracies for the tasks. Are the
consistencies low simply because the model is not able to perform well on this
task, and its answers are – to some extent – “random”?

#### Task accuracies

In Table 4 (column $T_{src}$), we report the monolingual task accuracies for
the model on both tasks, for all languages. Accuracies are generally higher
for PAWS-X (with only two class labels) than XNLI (with three class labels).
In particular, accuracies for German on XNLI are very low. Appendix F presents
the accuracies for various combinations of input data and instruction
languages, which reveals that the model struggles with the German instruction
for XNLI, rather than the German input data. Furthermore, the accuracies for
English are higher than for other languages. While this may not come as a
surprise given the predominantly English training data, it does raise an
intriguing point in the light of the high translation qualities: if a model is
able to perform a particular task (XNLI) in English, and it is able to
correctly translate the task into a different language, why is it not able to
perform the task at a similar level in that other language and what does that
entail?

| Accuracy
---|---
| Src | Tgt | $T_{\text{src}}$ | $T_{src\rightarrow tgt}$
PAWS-X | en | de | 0.77 | 0.76
en | zh | 0.66
de | en | 0.71 | 0.73
zh | en | 0.60 | 0.68
XNLI | en | de | 0.71 | 0.60
en | zh | 0.60
de | en | 0.48 | 0.65
zh | en | 0.56 | 0.61
Table 4: Accuracies on PAWS-X and XNLI for the original task $T_{src}$ (i.e.
$T_{en}$, $T_{de}$, $T_{zh}$), and model-internal translations
$T_{src\rightarrow tgt}$ from source (src) to target (tgt) language.

To further investigate this point, we now consider the accuracies of the model
on the task using the model’s own translation, which we report in Table 4
(column $T_{src\rightarrow tgt}$). The results confirm our earlier observation
that the model does not maintain consistent meaning representations across
languages: even though translations are generated by the model itself, and
thus should be meaning-preserving according to the model, they lead to
differences in performance (compared to the baselines in column $T_{src}$).

These differences in performance also have practical consequences. While
translations from English to German or Chinese lead to a decrease in accuracy,
translations from German or Chinese to English lead to an increase in
accuracy. E.g. translating from German to English or Chinese to English on
PAWS-X increases the respective accuracy from $0.71$ to $0.73$ and from $0.60$
to $0.68$, and the same pattern can be observed for XNLI (for details on the
respective effects of translating instructions and input sentences, see
Appendix E). Due to the model’s language-dependent task understanding,
performance on lower-resource languages may be increased by instructing the
model to first translate the input to English and then to perform the task.

#### Consistent correct vs incorrect

We investigate if there is a difference in consistency between examples for
which the model provides a correct answer and those for which it provides an
incorrect answer. First, we establish a baseline, by computing the consistency
between two runs with the same $T_{en}$ inputs (Table 5, first column, row 1
for each task, respectively). The overall consistencies for this baseline are
very high: $99\%$ for PAWS-X and $98\%$ for XNLI. Accordingly, when asked the
same question multiple times, the model usually gives the same response. In
the second and third row (per task, respectively), we further break down
consistency, and compute what percentage of the correct and incorrect examples
were consistent. As we can see, the baseline case has a high consistency for
incorrect responses ($98\%$ and $96\%$), implying that the model’s errors are
systematic and not due to random guessing.

Moving to the model-internal translations, we observe a general decrease in
consistency, which affects both correct and false responses but is much more
prevalent in the latter. This comparison is particularly interesting because
correct and incorrect consistent examples provide different levels of evidence
for the consistency of a model. When a model is consistently _incorrect_
across two examples, this points to an error in the model’s understanding, but
it provides stronger evidence for the consistency of the underlying
representations than examples that are consistently correct. Whereas the
latter are correct in both languages and could thus in theory have been
inferred independently from the data for those respective languages, it is
more unlikely that a model makes an identical but unrelated incorrect
inference in two different languages. In fact, for the model under scrutiny,
the inconsistency across incorrect examples is much lower than for correct
examples, suggesting at least some of the correct consistent examples have
been inferred independently in both languages, rather than being shared. In
conclusion, although the translation process is largely meaning-preserving, it
can change the model’s understanding of both correctly and incorrectly
executed tasks, which is particularly visible for incorrect examples.

| $T_{en}$ | $T_{en\rightarrow de}$ | $T_{en\rightarrow zh}$ | $T_{de\rightarrow en}$ | $T_{zh\rightarrow en}$
---|---|---|---|---|---
PAWS-X | consistency all | 0.99 | 0.84 | 0.76 | 0.86 | 0.70
consistency correct | 0.99 | 0.89 | 0.78 | 0.92 | 0.82
| consistency incorrect | 0.98 | 0.67 | 0.71 | 0.72 | 0.52
XNLI | consistency all | 0.98 | 0.74 | 0.67 | 0.63 | 0.67
consistency correct | 0.99 | 0.77 | 0.71 | 0.83 | 0.80
| consistency incorrect | 0.96 | 0.66 | 0.57 | 0.45 | 0.50
Table 5: Detailed consistencies for the core experiment: model-internal
translations of input data and instructions. In addition, consistencies for a
baseline between two different runs with $T_{en}$ are provided. Listed are the
consistency across all responses (consistency all), as well as the consistency
across all correct responses (consistency correct) and across all incorrect
responses (consistency incorrect) on the source data.

##  5 Conclusion

In this paper, we present a novel paradigm for evaluating language models,
which leverages consistency across different linguistic senses. Our method can
be used to assess generalisation ability beyond specific forms. It offers
affordability and applicability to different evaluation tasks, while also
mitigating the risk of evaluating on data that the model has already
encountered during training. As such, multisense evaluation is not an
alternative to current benchmarks but a complement. Reporting consistency next
to standard evaluation metrics like accuracy, BLEU or F1-scores will make
model evaluation more meaningful in providing an estimate of how well the
model understands a given task beyond its specific form. Therefore, we
encourage other researchers to treat multisense consistency as an essential
part of benchmarking.

To showcase the effectiveness of our paradigm, we conduct a multilingual
multisense consistency evaluation of ChatGPT (gpt-3.5-turbo), a SOTA LLM. The
results of this evaluation unveil significant inconsistencies across different
senses generated by the model itself through translation, suggesting a lack of
genuine, cross-sense task understanding. To ensure the validity of this
interpretation, we consider several alternative explanations. Our experiments
specifically rule out 1) model-subjective changes in meaning caused by the
translation process (by using model-internal translations), 2) poor
translation quality, 3) inadequate performance on the task, and 4) random
guesses leading to inconsistent outcomes across runs. Collectively, these
findings strongly indicate that ChatGPT exhibits a language- and therefore
sense-dependent task understanding, which might also affect other leading
LLMs.

While we focus on one LLM, two tasks (natural language inference and
paraphrase detection), and three languages (Chinese, English, and German), our
approach can be cheaply and easily expanded to include more languages, tasks,
models, and different notions of “sense”. As such, we believe that our set of
experiments can serve as a starting point for further investigations into
multisense consistency. Since our approach involves generating multiple senses
of the same meaning through translation, it is well-suited for evaluating
current and future models, given the growing trend towards multilingual models
with increasingly proficient translation abilities. Nevertheless, there are
numerous other conceivable multisense evaluation approaches. For instance,
instead of using model-internal translations, one could employ model-internal
paraphrases. Multiple senses could also be generated in different domains,
such as arithmetic (different formulas yielding the same result) or code
(different functions producing the same input-output mapping). Last but not
least, calculating consistency for various tasks may help disentangle
“unfounded” language-specific differences (forming the focus of our analysis)
from language-specific differences that reflect cultural differences and
biases.

In conclusion, multisense consistency can be applied as long as the model
under investigation can create different senses of a given task and has some
understanding of the task in its original sense. It offers the possibility of
evaluating the task understanding of a model detached from the specific
realization of this task, and we hope it will contribute to making standard
benchmark evaluations more meaningful.

## Limitations

While our method can certainly be extended to other tasks and models, some of
these extensions may prove more challenging than others. In particular,
evaluating consistency between model responses that are more variable than the
ones in our experiments is less straightforward, and requires an appropriate
definition of consistency. More variable responses may arise when working with
LLMs that have not been adapted to deal with instructions. For example, we
instruct ChatGPT to choose a response from a set of pre-defined responses
(Please answer with “yes” or “no”, Please answer with “A”, “B”, or “C”) and it
largely follows these instructions. A standard LLM may deviate from these
answer templates, leading to complications in calculating the consistency. In
addition, more variable responses may arise when dealing with tasks that do
not correspond to a classification problem. Even testing factual knowledge
with a question-answering task may lead to variations in responses. For
example, model responses like 5, 5 times, or five, may be consistent but are
different. Further, the model can generate responses that are only partially
overlapping, e.g. disastrous financial situation versus bad financial
situation, which might require a graded definition of consistency. Thus,
moving forward, it is important to develop appropriate definitions of
consistency as well as corresponding automatic evaluation procedures. Given
that judging whether the meaning of two answers is the same is much easier
than providing these answers, the consistency evaluation might even be
outsourced to the model under investigation.

## Acknowledgements

We would like to thank Xin Huang for her help with the Chinese translations,
and Marco Baroni for giving us feedback on the draft.

## References

  * Alberti et al. (2019) Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. 2019\.  Synthetic QA corpora generation with roundtrip consistency.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6168–6173, Florence, Italy. Association for Computational Linguistics.
  * Asai and Hajishirzi (2020) Akari Asai and Hannaneh Hajishirzi. 2020.  Logic-guided data augmentation and regularization for consistent question answering.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5642–5650, Online. Association for Computational Linguistics.
  * Bisk et al. (2020) Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020.  Piqa: Reasoning about physical commonsense in natural language.  In _Thirty-Fourth AAAI Conference on Artificial Intelligence_.
  * Chen et al. (2021) Jifan Chen, Eunsol Choi, and Greg Durrett. 2021.  Can NLI models verify QA systems’ predictions? In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 3841–3854, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  XNLI: Evaluating cross-lingual sentence representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.
  * Elazar et al. (2021) Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021.  Measuring and Improving Consistency in Pretrained Language Models.  _Transactions of the Association for Computational Linguistics_ , 9:1012–1031.
  * (7) David Heineman.  Rethinking reasoning evaluation with theories of intelligence.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021.  Measuring massive multitask language understanding.  In _International Conference on Learning Representations (ICLR)_.
  * Hosseini et al. (2021) Arian Hosseini, Siva Reddy, Dzmitry Bahdanau, R Devon Hjelm, Alessandro Sordoni, and Aaron Courville. 2021.  Understanding by understanding not: Modeling negation in language models.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1301–1312, Online. Association for Computational Linguistics.
  * Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020.  XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation.  In _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 4411–4421. PMLR.
  * Hupkes et al. (2023) Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Christos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schottmann, Khuyagbaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari, Maria Ryskina, Rita Frieske, Ryan Cotterell, and Zhijing Jin. 2023.  State-of-the-art generalisation research in NLP: A taxonomy and review.  _arXiv_ , arXiv:2210.03050.
  * Jang et al. (2022) Myeongjun Jang, Deuk Sin Kwon, and Thomas Lukasiewicz. 2022.  BECEL: Benchmark for consistency evaluation of language models.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 3680–3696, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Jang and Lukasiewicz (2023) Myeongjun Jang and Thomas Lukasiewicz. 2023.  Consistency analysis of chatgpt.  _arXiv_ , arXiv:2303.06273.
  * Kassner and Schütze (2020) Nora Kassner and Hinrich Schütze. 2020.  Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7811–7818, Online. Association for Computational Linguistics.
  * Kassner et al. (2021) Nora Kassner, Oyvind Tafjord, Hinrich Schütze, and Peter Clark. 2021.  BeliefBank: Adding memory to a pre-trained language model for a systematic notion of belief.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 8849–8861, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Li et al. (2019) Tao Li, Vivek Gupta, Maitrey Mehta, and Vivek Srikumar. 2019.  A logic-driven framework for consistency of neural models.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3924–3935, Hong Kong, China. Association for Computational Linguistics.
  * Liang et al. (2020) Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Ming Zhou. 2020.  XGLUE: A new benchmark datasetfor cross-lingual pre-training, understanding and generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6008–6018, Online. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Minervini and Riedel (2018) Pasquale Minervini and Sebastian Riedel. 2018.  Adversarially regularising neural NLI models to integrate logical background knowledge.  In _Proceedings of the 22nd Conference on Computational Natural Language Learning_ , pages 65–74, Brussels, Belgium. Association for Computational Linguistics.
  * Mitchell et al. (2022) Eric Mitchell, Joseph Noh, Siyan Li, Will Armstrong, Ananth Agarwal, Patrick Liu, Chelsea Finn, and Christopher Manning. 2022.  Enhancing self-consistency and performance of pre-trained language models through natural language inference.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 1754–1768, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Mitchell and Krakauer (2023) Melanie Mitchell and David C. Krakauer. 2023.  The debate over understanding in ai’s large language models.  _Proceedings of the National Academy of Sciences_ , 120(13):e2215907120.
  * Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020.  Adversarial NLI: A new benchmark for natural language understanding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4885–4901, Online. Association for Computational Linguistics.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.  _arXiv_ , arXiv:2303.08774.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Ponti et al. (2020) Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020.  XCOPA: A multilingual dataset for causal commonsense reasoning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2362–2376, Online. Association for Computational Linguistics.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Raji et al. (2021) Inioluwa Deborah Raji, Emily Denton, Emily M. Bender, Alex Hanna, and Amandalynne Paullada. 2021.  AI and the everything in the whole wide world benchmark.  In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual_.
  * Ray Choudhury et al. (2022) Sagnik Ray Choudhury, Anna Rogers, and Isabelle Augenstein. 2022.  Machine reading, fast and slow: When do models “understand” language? In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 78–93, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Rei et al. (2022) Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022.  COMET-22: Unbabel-IST 2022 submission for the metrics shared task.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Roemmele et al. (2011) Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. 2011.  Choice of plausible alternatives: An evaluation of commonsense causal reasoning.  In _2011 AAAI Spring Symposium Series_.
  * Ruder et al. (2021) Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. 2021\.  XTREME-R: Towards more challenging and nuanced multilingual evaluation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10215–10245, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Sap et al. (2019) Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019.  Social IQa: Commonsense reasoning about social interactions.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4463–4473, Hong Kong, China. Association for Computational Linguistics.
  * Srivastava et al. (2022) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  _CoRR_ , abs/2206.04615.
  * Wang et al. (2019a) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a.  Superglue: A stickier benchmark for general-purpose language understanding systems.  In _Proceedings of the 33rd International Conference on Neural Information Processing Systems (NeurIPS)_ , Red Hook, NY, USA. Curran Associates Inc.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Wang et al. (2019b) Haohan Wang, Da Sun, and Eric P. Xing. 2019b.  What if we simply swap the two text fragments? a straightforward yet effective way to test the robustness of methods to confounding signals in nature language inference tasks.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 7136–7143.
  * Wang et al. (2023) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023.  Self-consistency improves chain of thought reasoning in language models.  _arXiv_ , arXiv:2203.11171.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Wittgenstein (1953) Ludwig Wittgenstein. 1953.  _Philosophical investigations. Philosophische Untersuchungen._ Macmillan.
  * Yang et al. (2019) Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019.  PAWS-X: A cross-lingual adversarial dataset for paraphrase identification.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3687–3692, Hong Kong, China. Association for Computational Linguistics.
  * Zhang et al. (2019) Yuan Zhang, Jason Baldridge, and Luheng He. 2019.  PAWS: Paraphrase adversaries from word scrambling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1298–1308, Minneapolis, Minnesota. Association for Computational Linguistics.

##  Appendix A Genbench evaluation card

Our work uses generalisation across senses to assess task understanding in
LLMs. In Figure 3, we provide the GenBench eval card (Hupkes et al., 2023) of
our experiments.

Motivation
---
Practical Cognitive Intrinsic Fairness $\boxtimes$
Generalisation type
Compo- sitional Structural Cross Task Cross Language Cross Domain Robust- ness
$\boxtimes$ $\boxtimes$
Shift type
Covariate Label Full Assumed $\boxtimes$
Shift source
Naturally occuring Partitioned natural Generated shift Fully generated
$\boxtimes$
Shift locus
Train–test Finetune train–test Pretrain– train Pretrain– test $\boxtimes$
Figure 3: Our experiments assess cross-lingual generalisation for natural
corpora, in pretrained LLMs, to assess LLM task understanding.

##  Appendix B Task instructions

Table 6 shows the task instructions for both tasks, in all languages. The
table shows the original English, German, and Chinese instructions, as well as
the model-internal translations of these instructions.

Task | Language |  Instruction
---|---|---
PAWS-X | $I_{en}$ |  Do the following sentences have the same meaning? Sentence 1: “[sentence_1]” Sentence 2: “[sentence_2]” Please answer with “yes” or “no”.
$I_{de}$ |  Haben die folgenden Sätze die gleiche Bedeutung? Satz 1: “[sentence_1]” Satz 2: “[sentence_2]” Bitte antworte mit “ja” oder “nein”.
$I_{zh}$ |  下面的句子有着相同的含义吗？ 句子 1: “[sentence_1]” 句子 2: “[sentence_2]” 请用“是”或者“否”回答。777Note that we also accept “ 不是 ” instead of “否”.
$I_{en\rightarrow de}$ |  Haben die folgenden Sätze die gleiche Bedeutung? Satz 1: “[sentence_1]” Satz 2: “[sentence_2]” Bitte antworten Sie mit “Ja” oder “Nein”.
$I_{en\rightarrow zh}$ |  以下句子的意思相同吗？句子 1: “[sentence_1]” 句子 2: “[sentence_2]” 请用“是”或“不是”回答。
$I_{de\rightarrow en}$ |  Do the following sentences have the same meaning? Sentence 1: “[sentence_1]” Sentence 2: “[sentence_2]” Please respond with ‘yes’ or ‘no’.
$I_{zh\rightarrow en}$ |  Does the following sentence have the same meaning? Sentence 1: “[sentence_1]” Sentence 2: “[sentence_2]” Please answer with ‘yes’ or ‘no’.
XNLI | $I_{en}$ |  Given the following sentence pair, which one of the following is true: (A) the first sentence entails the second sentence, (B) the first sentence contradicts the second sentence, or (C) neither of the two? Sentence 1: “[sentence_1]” Sentence 2: “[sentence_2]” Please answer with “A”, “B”, or “C”.
$I_{de}$ |  Welche dieser Aussagen trifft auf das folgende Satzpaar zu: (A) der erste Satz impliziert den zweiten Satz, (B) der erste Satz widerspricht dem zweiten Satz, oder (C) keines von beiden? Satz 1: “[sentence_1]” Satz 2: “[sentence_2]” Bitte antworte mit “A”, “B” oder “C”.
$I_{zh}$ |  对于给出的一对句子，以下哪一个选项是正确的：（A）第一个句子涵盖了第二个句子（B）第一个句子与第二个句子矛盾 （C）两者都不？句子 1: “[sentence_1]” 句子 2: “[sentence_2]” 请用“A”、“B”或“C”来回答。
$I_{en\rightarrow de}$ |  Angesichts des folgenden Satzpaares, welche der folgenden Aussagen ist wahr: (A) Der erste Satz impliziert den zweiten Satz, (B) Der erste Satz widerspricht dem zweiten Satz oder (C) Keines von beiden? Satz 1: “[sentence_1]” Satz 2: “[sentence_2]” Bitte antworten Sie mit “A”, “B” oder “C”.
$I_{en\rightarrow zh}$ |  给定以下句子对，哪一个是正确的：（A）第一句蕴含第二句，（B）第一句与第二句相矛盾，还是（C）两者都不是？句子 1: “[sentence_1]” 句子 2: “[sentence_2]” 请用“A”、“B”或“C”回答。
$I_{de\rightarrow en}$ |  Which of these statements applies to the following pair of sentences: (A) the first sentence implies the second sentence, (B) the first sentence contradicts the second sentence, or (C) neither of the above? Sentence 1: “[sentence_1]” Sentence 2: “[sentence_2]” Please reply with “A”, “B”, or “C”.
$I_{zh\rightarrow en}$ |  For a given pair of sentences, which of the following options is correct: (A) The first sentence covers the second sentence. (B) The first sentence contradicts the second sentence. (C) Neither of them? Sentence 1: “[sentence_1]” Sentence 2: “[sentence_2]” Please answer with “A”, “B”, or “C”.
Table 6: Task instructions in different languages. The original instructions
in English, German, and Chinese are given by $I_{en}$, $I_{de}$, and $I_{zh}$.
The model-internal translations of these instructions are given by
$I_{scr\rightarrow tgt}$, with $src$ being the source language, and $tgt$ the
target language.

##  Appendix C Translation instructions

We used the following instructions for model-internal translations:

  * •

en$\rightarrow$de/zh:
Please translate the following text into
German/Chinese: “[text]”

  * •

de$\rightarrow$en:
Bitte übersetze den folgenden Text ins
Englische: “[text]”

  * •

zh$\rightarrow$en:
请将下面的文字翻译成英语: “[text]”

##  Appendix D Inconsistencies for very high quality translations

We extend the analyses from Section 4.3 by calculating the inconsistencies for
data points with a BLEU score of at least $50$. Our focus remains on
translations of the input data and the model is instructed in the original
(source) language. Table 7 shows the amount of data (%) included in the
analysis, along with the corresponding consistency. Importantly, the model’s
inconsistency persists, as it never achieves consistencies surpassing $0.87$.
Moreover, across the board, consistencies exhibit only a slight improvement
compared to the original values (see consistency orig, same as in Table 1,
column X). The only substantial increase in performance occurs for
translations from Chinese to English on XNLI, with consistency rising from
$0.72$ to $0.80$. This finding aligns with the observation that the
translations from Chinese to English are of significantly lower quality than
the other translations. Hence, bad translations may reduce consistency, but
this phenomenon is only observed in one specific case.

|  | en$\rightarrow$de | en$\rightarrow$zh | de$\rightarrow$en | zh$\rightarrow$en
---|---|---|---|---|---
PAWS-X | consistency orig | 0.85 | 0.79 | 0.86 | 0.75
consistency BLEU $>50$ | 0.86 | 0.82 | 0.87 | 0.78
| % included BLEU $>50$ | 56.6 | 40.1 | 67.1 | 20.6
XNLI | consistency (orig) | 0.76 | 0.71 | 0.81 | 0.72
consistency BLEU $>50$ | 0.77 | 0.72 | 0.82 | 0.80
| % included BLEU $>50$ | 35.6 | 32.3 | 39.6 | 10.5
Table 7: Only datapoints with BLEU scores of $>50$ were included in this
analysis. The table shows the percentage of included data points (% included
BLEU>50), and the consistency of the model for these selected translations
(consistency BLEU>50) compared to the original consistency (consistency orig)
repeated from Table 1 (column X).

##  Appendix E Task performance for model-internal translations

Table 8 shows the model’s accuracy for all internal translations: instruction
only (I), input sentences only (X), or both (T). Section 4.4 shows that
translating from German and Chinese to English increases the model’s accuracy
compared to the original $T_{de}$ and $T_{zh}$ inputs. The additional results
presented here indicate that this increase in accuracy is largely due to
translating the instruction, although a complete translation (T) leads to
further improvements.

| Accuracy
---|---
| Src$\rightarrow$Tgt | $T$ | $I$ | $X$
PAWS-X | en$\rightarrow$de | 0.76 | 0.77 | 0.77
en$\rightarrow$zh | 0.66 | 0.75 | 0.70
de$\rightarrow$en | 0.73 | 0.72 | 0.70
zh$\rightarrow$en | 0.68 | 0.67 | 0.63
XNLI | en$\rightarrow$de | 0.60 | 0.63 | 0.67
en$\rightarrow$zh | 0.60 | 0.63 | 0.62
de$\rightarrow$en | 0.65 | 0.64 | 0.49
zh$\rightarrow$en | 0.61 | 0.59 | 0.56
Table 8: Accuracies on the benchmark tasks for the model-internal translations
from source (src) to target (tgt) language. We consider translations of both
input data and instructions (T), instruction only (I), and input data only
(X).

##  Appendix F Performance on mixed languages for input data and instructions

We look at different ablations to understand the effect of using a language
other than English for input data or instruction. Table 9 shows the model’s
accuracy on different combinations of languages for input sentences and
instructions, always using the input sentences provided by the multilingual
benchmark, and the English, German, or Chinese instructions developed for us
by native speakers. Accuracy decreases when instruction or input data are
changed from English to German or Chinese. Changing the language for both at
the same time further decreases accuracy, as errors from each language change
accumulate. For PAWS-X there is a more substantial decrease when translating
instructions or input data to Chinese compared to German. For XNLI, especially
the use of the German instruction is detrimental, with accuracies dropping
from $0.71$ (en/en, see Table 4) to $0.50$ (en/de). Testing alternative German
instructions revealed that this effect does not only pertain to our specific
formulation. While a decrease in performance may be expected for non-English
inputs, the extent of this effect when changing only the task instruction is
surprising. Changing even very simple instructions (as the one for PAWS-X)
from English to Chinese can lead to a $10\%$ absolute decrease in accuracy.

| X / I
---|---
| en/de | en/zh | de/en | zh/en
PAWS-X | 0.75 | 0.67 | 0.73 | 0.68
XNLI | 0.50 | 0.60 | 0.65 | 0.59
Table 9: Accuracies on mixed-language combinations of original input data (X)
and instructions (I).