Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

SafeAI 2023, The AAAI Workshop on Artificial Intelligence Safety, Feb 13–14,
2023, Washington, D.C.

<EMAIL_ADDRESS>[1] [] [] []

# Active Reward Learning from Multiple Teachers

Peter Barnett  Rachel Freedman  Justin Svegliato  Stuart Russell  Center for
Human-Compatible AI, University of California, Berkeley, CA 94720, USA

(2022)

###### Abstract

Reward learning algorithms utilize human feedback to infer a reward function,
which is then used to train an AI system. This human feedback is often a
preference comparison, in which the human teacher compares several samples of
AI behavior and chooses which they believe best accomplishes the objective.
While reward learning typically assumes that all feedback comes from a single
teacher, in practice these systems often query multiple teachers to gather
sufficient training data. In this paper, we investigate this disparity, and
find that algorithmic evaluation of these different sources of feedback
facilitates more accurate and efficient reward learning. We formally analyze
the value of information (VOI) when reward learning from teachers with varying
levels of rationality, and define and evaluate an algorithm that utilizes this
VOI to actively select teachers to query for feedback. Surprisingly, we find
that it is often more informative to query comparatively irrational teachers.
By formalizing this problem and deriving an analytical solution, we hope to
facilitate improvement in reward learning approaches to aligning AI behavior
with human values.

###### keywords:

Reward Learning Active Learning Preference Learning Value of Information

##  1 Introduction

Standard AI and machine learning algorithms require the designer to specify a
cost or reward function. This objective incentivizes desired behavior and
penalizes mistakes, teaching the system how to perform the task. While such
objectives are easy to manually specify for problems with clear win
conditions, such as games [1, 2, 3] and tasks with clear goals, such as image
classification [4, 5], they can be challenging to formalize for more nuanced
tasks [6]. For example, Lee et al. [7] find that humans struggle to define an
objective that incentivizes bipedal locomotion, despite being experts in both
machine learning and walking. By incentivizing incorrect behavior,
misspecified objectives can lead to useless or even dangerous outcomes [8].
Ensuring that AI systems optimize objectives that align with our own is a
crucial part of building safe and beneficial AI.

Reward learning techniques enable AI systems to learn their objectives by
observing and interacting with humans instead of requiring their designers to
specify these objectives manually [9]. Humans can train reward learning
systems using a variety of feedback modalities, including demonstrations [10,
11, 12], pairwise comparisons [7, 13, 14], natural language [15], numeric
values [16], corrections [17], and proxy rewards [18, 19]. Reward learning
from pairwise comparisons in particular has proven remarkably effective across
a variety of tasks, including complex physical maneuvers for continuous
control systems [7, 14] and text summarization for language language models
[20, 21]. In the future, it may even be possible to use reward learning to
train AI systems to assist humans in researching safe AI [8, 22].

However, to infer reward functions from human feedback, reward learning
systems must model human decision-making, and incorrect human decision-making
models often leads to poor inference [23, 24, 25]. Moreover, reward learning
systems typically assume that all feedback comes from a single distribution or
teacher, despite querying multiple teachers to generate sufficient feedback.
However, humans often vary in their expertise, focus, and intelligence,
affecting the noisiness of their feedback. The practice of conflating all
feedback implicitly disregards the differences between different teachers,
increasing the likelihood of human model misspecification and the limitations
of reward learning [26].

In this work, we extend reward learning to take advantage of differences
between teachers. We develop a Bayesian reward learning algorithm that
actively selects which teacher to query based on the noisiness of their
feedback and the learner’s current belief. We find that querying a _less_
rational teacher can often be more informative than querying a _more_ rational
teacher, since teacher mistakes inform the agent of the relative values of
alternatives. For example, imagine that two teachers are comparing two
alternatives, $A$ and $B$. $A$ is worth more than $B$, but only slightly. If
the first teacher is perfectly rational, they will always select $A$ over $B$.
The learner can infer from this that $A$ is preferable to $B$, but has no way
to learn how significant the distinction is. However, assume that the second
teacher is somewhat less rational, and occasionally mixes up alternatives of
similar value. Then they will typically choose $A$, but sometimes choose $B$,
and this allows the learner to infer that the gap between $A$ and $B$ is
small. Section 3 formalizes this rationality model and inference procedure.

The rest of the paper is as follows. In Section 2, we discuss prior work on
reward learning, active learning, and human modeling. In Section 3, we
describe the mechanics of reward learning, including the model of human
rationality and the metrics that will be used to measure the value of
information (VOI) of teacher feedback. In Section 4, we propose a teacher
selection algorithm that selects which teacher to query for feedback at each
time step based on the modeled rationality of each teacher and the learner’s
belief distribution over the reward function. In Sections 5 and 6, we present
theoretical and empirical results, showing that the learner’s belief will
eventually converge to the true reward function under the teacher selection
algorithm, that querying less rational teachers can often be more informative,
and that our teacher selection method outperforms simple heuristics like
always querying the most rational teacher. By formalizing the problem of
learning from multiple teachers and deriving an analytical solution, we hope
to facilitate improvement in reward learning approaches to value alignment.

##  2 Related Work

### Reward Learning

Reward learning techniques allow AI systems to learn reward functions by
observing or interacting with humans. For example, inverse reinforcement
learning agents observe human behavior or policies, and then infer an
underlying reward function that the behavior optimizes [10, 11, 12]. Recent
advances in reward learning have focused on learning from preference
comparisons. Here, human teachers observe paired samples of system behavior,
then choose which sample they prefer out of each pair. The system learns a
reward model that maximizes the likelihood of these preferences, then uses
that model to generate a reward signal to guide its behavior. This technique
has been successfully applied to many domains, from continuous control [7, 14]
to language generation tasks [20, 21]. Reward learning can also use a variety
of other feedback modalities, including preference comparisons [7, 13, 14],
natural language [15], numeric values [16], corrections [17], and proxy
rewards [18, 19], but we focus on preference comparisons in this paper due to
its recent success.

### Active Reward Learning

Human feedback is expensive and time-consuming to generate, so reward learning
algorithms must learn efficiently from limited data. They do this in part by
actively selecting the queries that are sent to human teachers in order to
maximize the expected VOI of human feedback. Sadigh et al. [13] assume that
the system is a Bayesian learner, actively synthesizing queries that maximize
the expected volume removed from the learner’s posterior. Bıyık and Sadigh
[27] develop efficient approximations to this method and show how to integrate
active query selection and reward learning in practice. Lee et al. [7] take a
different approach, empirically evaluating various heuristic strategies for
query selection and finding that uncertainty-based sampling methods tend to
perform the best. However, all of this previous work focuses on choosing which
queries to send to the teachers. In this paper, we instead consider which
teachers to send these queries to.

### Human Modeling

To infer reward functions, AI systems must model the behavior of humans. Early
work on reward learning assumed that human behavior was perfectly rational and
that human teachers always chose the alternative that maximized their reward
[10]. Later work models human behavior as pedagogic [24], systematically
biased [28], and noisily or Boltzmann-rational [9, 12]. We will follow recent
work on learning from human preferences [7, 9, 12, 14] and model human
teachers as Boltzmann-rational, making choices according to a well-known
probability model specified later in the paper.

Figure 1: Our active reward learning approach.

##  3 Active Reward Learning

In this section, we formalize the problem of selecting the most informative
teacher to query in order to gradually learn the correct reward model. In
particular, we are interested in greedily selecting the teacher to query at
each time step such that the reward model of the agent efficiently converges
to the correct reward model.

At a high level, the teacher selection problem begins with a set of items or
trajectories to compare, along with a set of human teachers to evaluate those
comparisons. The human teachers each have a different level of rationality
that is known _a priori_ , meaning that the probability of a given human
teacher making a mistake by preferring a less valuable item over a more
valuable item is known in advance. During each time step of our approach
depicted in Figure 1, two items are sampled from the set of items (_Step 1_)
and then a human teacher is selected to be queried based on these items and
the current belief about the reward model (_Step 2_). The human teacher is
asked which of the two items they prefer (_Step 3_), and their preference is
used to update the reward model (_Step 4_). This process of selecting a query
and a teacher is repeated until the reward model converges to the correct
reward model.

_Query selection_ is the problem of choosing which items to present to the
teacher [7]. Some approaches to query selection include choosing the pair of
items for which the preference predictors are most uncertain [7, 14]. Other
approaches to query selection include selecting the pair of items that ensure
that the space of queries is well covered. Finally, there are more active
methods that actively synthesize queries in order learn more efficiently [13,
29]. Since our focus is on teacher selection rather than query selection, for
the purposes of our analysis we will assume that queries are sampled uniformly
at random. However, existing methods for query selection can be easily
combined with our teacher selection algorithm to further improve reward
learning.

To formalize the problem of teacher selection, this section proceeds as
follows. We (1) provide a representation of items and rewards, (2) apply a
well-known model of human rationality to our problem, (3) offer a method for
updating belief distributions that uses preference comparisons from a human
teacher, and (4) propose two metrics that measure the correctness of a belief
distribution.

### Representing Items and Rewards

Intuitively, each item can be represented as a set of features. For example, a
book could be described by the number of pages and the number of positive
reviews or a maneuver made by a self-driving car could be described by its
position and distance from other vehicles at each time step. Hence, each item
$i$ can formally be represented by a feature vector
$\phi_{i}\in\mathbb{R}^{d}$ where $d$ is the number of features that describe
the $i$th item.

Given this representation of an item, the reward $R(i)$ for an item $i$ can be
expressed as a dot product between the feature vector $\phi_{i}$ and the
weight vector $\mathbf{w}\in\mathbb{R}^{d}$ for the reward model that is being
learned:

$R(i)=\mathbf{w}^{\top}\phi_{i}.$ (1)

If the items cannot be expressed by a feature vector, this approach can still
be used by treating the feature vector $\phi_{i}$ as a one-hot vector: given
the $i$th item, the $i$th entry of the feature vector $\phi_{i}$ would be $1$
and every other entry would be $0$ while the $i$th entry of the weight vector
$\mathbf{w}$ would be the reward $R(i)$ for the $i$th item.

During reward learning, the human teacher is presented with two items and the
probability of the human choosing one item over another item depends on the
difference in reward between the two items at hand. We therefore express the
difference in the reward between two items $i$ and $j$ as the equation

$R(i)-R(j)=\mathbf{w}^{\top}(\phi_{i}-\phi_{j})=\mathbf{w}^{\top}\varphi_{ij},$
(2)

where $\varphi_{ij}=\phi_{i}-\phi_{j}$ is the difference in the feature
vectors of the two items.

### Modeling Human Rationality

Human teachers can be represented as Boltzmann-rational agents following a
large body of existing work on reward learning [7, 9, 12, 14, 30, 31, 32, 33,
34]. Moreover, we assume that each teacher has a different known rationality
parameter $\beta$ rather than assuming $\beta=1$ for all teachers. Boltzmann-
rational teachers are more likely to choose the higher reward item if they are
“more rational" (i.e., a higher $\beta$), or if the difference in reward
between the two items is greater. The probability that the teacher chooses an
item $i$ over and an item $j$ is given by

$P(i\succ j;\beta)=\frac{\exp(\beta R(i))}{\exp(\beta R(i))+\exp(\beta
R(j))}.$ (3)

We thus model the human choice probabilistically:

$P(I|\mathbf{w};\varphi_{ij},\beta)=\frac{1}{1+\exp(-I\beta\mathbf{w}^{\top}\varphi_{ij})},$
(4)

where $I=+1$ if the human prefers item $i$ over item $j$ and $I=-1$ if the
human prefers item $j$ over item $i$. This reflects the difference in value of
the two items but not their absolute value. Equation 4 is a logistic model of
the probability of the human preference $I$, where $\beta$ determines the
slope. As the difference in reward between the two items increases, the
probability that the teacher chooses the higher reward item approaches 1.

### Updating Belief Distributions

The goal of reward learning is to learn the weight vector $\mathbf{w}$ of the
reward model. Given the preference of a teacher $I$, the difference in feature
vectors $\varphi_{ij}$, and the teacher’s rationality parameter $\beta$, the
learner updates its belief over the weights of the reward model. That is, the
belief over the weights of the reward model is updated such that the reward
model now predicts that the item selected by the teacher is more valuable than
it was prior to the belief update. Formally, we begin with the current belief
distribution $P(\mathbf{w})$, which we treat as the prior distribution, and
update it according to Bayes’ theorem in the following way:

$P(\mathbf{w}|I;\varphi_{ij},\beta)=\frac{P(I|\mathbf{w};\varphi_{ij},\beta)P(\mathbf{w})}{\int
P(I|\mathbf{w}^{\prime};\varphi_{ij},\beta)P(\mathbf{w}^{\prime})d\mathbf{w}^{\prime}},$
(5)

where $P(I|\mathbf{w};\varphi_{ij},\beta)$ is given by Equation 4.

Table 1: The general form of an expected metric $\mathcal{M}$ along with the expected metrics for mean squared error (MSE) and log loss (LL). Expected Metric | Equation
---|---
$\operatorname*{\mathbb{E}}_{\begin{subarray}{c}\mathbf{w}\sim P_{\mathbf{w}}\\\ I\sim P_{I|\mathbf{w}}\end{subarray}}\left[\mathcal{M}(P_{\mathbf{w}|I},\mathbf{w};\varphi_{ij},\beta)\right]$ | $\int P_{\mathbf{w}}\sum_{I}P_{I|\mathbf{w}}\mathcal{M}(P_{\mathbf{w}|I},\mathbf{w})d\mathbf{w}$
$\operatorname*{\mathbb{E}}_{\begin{subarray}{c}\mathbf{w}\sim P_{\mathbf{w}}\\\ I\sim P_{I|\mathbf{w}}\end{subarray}}\left[\operatorname*{MSE}(P_{\mathbf{w}|I},\mathbf{w};\varphi_{ij},\beta)\right]$ | $2\sum_{I}\frac{2}{\int f_{I}(\mathbf{w})d\mathbf{w}}\times\left[\int f_{I}(\mathbf{w})d\mathbf{w}\int f_{I}(\mathbf{w})\left\lVert\mathbf{w}\right\rVert^{2}d\mathbf{w}-\left\lVert\int f_{I}(\mathbf{w})\mathbf{w}d\mathbf{w}\right\rVert^{2}\right]$
$\operatorname*{\mathbb{E}}_{\begin{subarray}{c}\mathbf{w}\sim P_{\mathbf{w}}\\\ I\sim P_{I|\mathbf{w}}\end{subarray}}\left[\operatorname*{LL}(P_{\mathbf{w}|I},\mathbf{w};\varphi_{ij},\beta)\right]$ | $-\sum_{I}\int f_{I}(\mathbf{w})\log\left(\frac{f_{I}(\mathbf{w})}{\int f_{I}(\mathbf{w}^{\prime})d\mathbf{w}^{\prime}}\right)d\mathbf{w}$

### Measuring Belief Distribution Error

After querying a teacher and updating the belief over the weights of the
reward model $\mathbf{w}$, the belief distribution can be evaluated on a
metric that measures the “correctness” or the distance of this belief
distribution to the true belief distribution. Here, we consider two such
metrics: the mean squared error ($\operatorname*{MSE}$) and the log loss
($\operatorname*{LL}$). The $\operatorname*{MSE}$ measure represents how “far
away” the belief distribution is from the true value while the
$\operatorname*{LL}$ measure represents the height of the belief distribution
at the true value. In both cases, a lower score indicates a more accurate
distribution. Using $Q(\mathbf{w})$ as the belief distribution over the weight
vector $\mathbf{w}$ and $\mathbf{w}_{\emph{\mbox{true}}}$ as the true weight
vector, the $\operatorname*{MSE}$ and $\operatorname*{LL}$ measures are given
as follows.

$\displaystyle\operatorname*{MSE}(Q(\mathbf{w}),\mathbf{w}_{\emph{\mbox{true}}})$
$\displaystyle=\int
Q(\mathbf{w})||\mathbf{w}-\mathbf{w}_{\emph{\mbox{true}}}||^{2}d\mathbf{w}$
(6)
$\displaystyle\operatorname*{LL}(Q(\mathbf{w}),\mathbf{w}_{\emph{\mbox{true}}})$
$\displaystyle=-\log(Q(\mathbf{w}_{\emph{\mbox{true}}}))$ (7)

Note that we will describe a greedy approach that selects the teacher that in
expectation leads to our belief distribution scoring the best on one of these
metrics after a single update in the next section.

Work on active learning from human preferences uses volume removal (i.e.,
removing as much of the integral of the unnormalized distribution as possible)
as a metric [13, 27, 33]. However, this may not be an appropriate metric for
teacher selection. This is because a larger Boltzmann rationality parameter
$\beta$ results in a larger volume of the belief distribution being removed
but may not necessarily lead to a more accurate belief distribution.

##  4 Teacher Selection

We propose a method for selecting and querying the teacher that produces the
best immediate improvement in the expectation of a given metric, which
approximates the expected VOI of the teacher feedback. The metrics evaluate
how similar the posterior belief is to the ground truth reward, so lower
scores indicate improvements in the learned reward model. The algorithm
considers uncertainty over two variables: the ground-truth parameterization of
the reward model and the item from the query that the teacher prefers. In
particular, the expectation of the metric must be taken over the current
belief distribution $P(\mathbf{w})$ and the probability
$P(I|\mathbf{w};\varphi_{ij},\beta)$ of the teacher preferring each item.
Formally, we express the expectation of a given metric $\mathcal{M}$ in Table
1. Note that we use the notation $P_{\mathbf{w}}=P(\mathbf{w})$,
$P_{I|\mathbf{w}}=P(I|\mathbf{w};\varphi_{ij},\beta)$, and
$P_{\mathbf{w}|I}=P(\mathbf{w}|I,\varphi_{ij},\beta)$ throughout this section.

Importantly, the expected value of a given metric only depends on the known
variables $\varphi_{ij}$ and $\beta$ along with the current belief
distribution $P_{\mathbf{w}}$ given a straightforward substitution of
Equations 4 and 5. This enables our method to calculate the expected value of
the metric for a given teacher with the rationality parameter $\beta$. This
will be used to find the teacher to query at each time step: the teacher with
the lowest metric in expectation should be selected as that would result in a
weight vector that is closest to the true weight vector in expectation.

Finally, given the general form of an expected metric, Table 1 defines the
expectations of the MSE and LL metrics using the function
$f_{I}(\mathbf{w})=P_{\mathbf{w}}/(1+\exp(-I\beta\mathbf{w}^{\top}\varphi_{ij}))$.

### Selecting a Teacher

To select the teacher to query, we first calculate the expected metric for
each teacher $\beta$ given the current belief distribution $P(\mathbf{w})$ and
then select the teacher that would result in the lowest expected metric score.
Formally, the rationality parameter $\beta^{*}$ that leads to the largest
reduction in the expectation of the metric is defined as follows:

$\beta^{\ast}=\operatorname*{argmin}_{\beta\in\boldsymbol{\beta}}\left[\operatorname*{\mathbb{E}}_{\begin{subarray}{c}\mathbf{w}\sim
P_{\mathbf{w}}\\\ I\sim
P_{I|\mathbf{w}}\end{subarray}}\left[\mathcal{M}(P_{\mathbf{w}|I},\mathbf{w};\varphi_{ij},\beta)\right]\right],$
(8)

where $\boldsymbol{\beta}$ is a vector of the $\beta$ values of the teachers.

### Learning a Reward Model

To learn the reward model, the learner begins with an initial belief
distribution $P_{\mathbf{w}}$ over the reward function parameterization and
then updates it according to Algorithm 1. First, the algorithm generates
queries of paired items and calculates $\beta^{\ast}$, which is the
rationality parameter that leads to the largest improvement in the expectation
over the correctness metric. The algorithm queries the teacher with this
rationality parameter, and the teacher responds with a preference indicating
which of the two items in the query they prefer. This preference is used to
update the belief distribution $P_{\mathbf{w}}$. The algorithm iterates until
convergence, which is when the entropy of the distribution $P_{\mathbf{w}}$
becomes lower than a specified threshold $\epsilon$.

1

2

3

Input: An initial belief distribution $P(\mathbf{w})$, a list of the teachers’
Boltzmann rationality parameters $\boldsymbol{\beta}$, an expected metric
function $\operatorname*{\mathbb{E}}[\mathcal{M}]$, and an entropy convergence
threshold $\epsilon$

Output: A posterior belief distribution $P(\mathbf{w})$

4

5$\emph{converged}\leftarrow\emph{False}$

6 while  _not converged_ do

7 $\phi_{i},\ \phi_{j}\leftarrow\textsc{GenerateQuery}()$

8 $\varphi_{ij}\leftarrow\phi_{i}-\phi_{j}$

9
$\beta^{\ast}\leftarrow\operatorname*{argmin}_{\beta\in\boldsymbol{\beta}}\operatorname*{\mathbb{E}}\left[\mathcal{M}(P(\mathbf{w}),\mathbf{w};\varphi_{ij},\beta)\right]$

10 $I\leftarrow\textsc{Teacher}(\beta^{*}).\textsc{Query}(\phi_{i},\phi_{j})$

11 $P(\mathbf{w})\leftarrow\textsc{Normalize}(P(\mathbf{w})\cdot
P(I|\mathbf{w},\varphi_{ij},\beta^{*}))$

12 $\emph{entropy}\leftarrow-\int P(\mathbf{w})\log P(\mathbf{w})d\mathbf{w}$

13 $\emph{converged}\leftarrow\emph{entropy}<\epsilon$

14return $P(\mathbf{w})$

Algorithm 1 LearnRewardModel$(\cdot)$

##  5 Theoretical Analysis

In this section, we first prove that the belief distribution will converge to
the true distribution and then show that, under certain conditions, querying a
less rational teacher can result in more informative feedback.

### Convergence

Algorithm 1 queries multiple teachers with different $\beta$ values until the
reward estimate convergences. Here, we show that this process will make the
belief distribution over $\mathbf{w}$ converge to the true value.

######  Theorem 1.

In the limit of $N\to\infty$ random queries to Boltzmann-rational teachers
with positive, finite $\beta$ values, the posterior distribution over
$\mathbf{w}$ converges to the true value.

###### Proof.

The likelihood of a sequence of human choices $\underline{I}\in[\pm 1]^{N}$
from humans with rationality parameters $\underline{\beta}$ is
$P(\underline{I}|\mathbf{w};\underline{\beta})=\prod_{i=1}^{N}P(\underline{I}_{i}|\mathbf{w};\underline{\beta}_{i})$.
The posterior distribution over $\mathbf{w}$ after a sequence of queries is

$P(\mathbf{w}|\underline{I};\underline{\beta})\propto\prod_{i}^{N}P(\underline{I}_{i}|\mathbf{w};\underline{\beta}_{i})P(\mathbf{w}).$

We will show that $P(\mathbf{w}|\underline{I};\underline{\beta})\to 0$ as
$N\to\infty$ for all $\mathbf{w}\neq\mathbf{w}_{\emph{\mbox{true}}}$. The
Bayes factor between $\mathbf{w}$ and $\mathbf{w}_{\emph{\mbox{true}}}$ is

$\displaystyle\operatorname*{BF}=\frac{P(\mathbf{w}|\underline{I};\underline{\beta})}{P(\mathbf{w}_{\emph{\mbox{true}}}|\underline{I};\underline{\beta})}=\frac{\prod_{i}^{N}P(\underline{I}_{i}|\mathbf{w};\underline{\beta}_{i})P(\mathbf{w})}{\prod_{i}^{N}P(\underline{I}_{i}|\mathbf{w}_{\emph{\mbox{true}}};\underline{\beta}_{i})P(\mathbf{w}_{\emph{\mbox{true}}})},$

where $P(\mathbf{w}_{\emph{\mbox{true}}}|\underline{I};\underline{\beta})$ is
the posterior distribution at $\mathbf{w}_{\emph{\mbox{true}}}$. We can show
that $\operatorname*{BF}\to 0$ as $N\to\infty$ except when
$\mathbf{w}=\mathbf{w}_{\emph{\mbox{true}}}$. This implies
$P(\mathbf{w}|\underline{I};\underline{\beta})\to 0$ except when
$\mathbf{w}=\mathbf{w}_{\emph{\mbox{true}}}$. We require
$P(\mathbf{w}_{\emph{\mbox{true}}})\neq 0$ as $\operatorname*{BF}$ is
undefined otherwise. Trivially, $\operatorname*{BF}=1$ when
$\mathbf{w}=\mathbf{w}_{\emph{\mbox{true}}}$.

We now consider $\mathbf{w}\neq\mathbf{w}_{\emph{\mbox{true}}}$. We can define
the negative logarithm of BF, which approaches $\infty$ as
$\operatorname*{BF}\to 0$:

$\displaystyle-$ $\displaystyle\log\left(\operatorname*{BF}\right)$
$\displaystyle=-\log\left(\frac{\prod_{i}^{N}P(\underline{I}_{i}|\mathbf{w};\underline{\beta}_{i})P(\mathbf{w})}{\prod_{i}^{N}P(\underline{I}_{i}|\mathbf{w}_{\emph{\mbox{true}}};\underline{\beta}_{i})P(\mathbf{w}_{\emph{\mbox{true}}})}\right)$
$\displaystyle=-\sum_{i}^{N}\log\left(\frac{P(\underline{I}_{i}|\mathbf{w};\underline{\beta}_{i})}{P(\underline{I}_{i}|\mathbf{w}_{\emph{\mbox{true}}};\underline{\beta}_{i})}\right)-\log\left(\frac{P(\mathbf{w})}{P(\mathbf{w}_{\emph{\mbox{true}}})}\right).$

The first term is the sum of many terms. If this term approaches $\infty$ as
$N\to\infty$ then $\operatorname*{BF}\to 0$. We now examine each term in the
sum and show that in expectation they are each positive. All of these terms
are independent as they are only depend on the likelihood and not on the
current distribution. Hence, they will not decay with additional steps, and so
the sum will diverge if the individual terms are positive in expectation. The
expected value for each term in the sum is

$\displaystyle\mathbb{E}$
$\displaystyle\left[-\log\left(\frac{P(\underline{I}_{i}|\mathbf{w};\underline{\beta}_{i})}{P(\underline{I}_{i}|\mathbf{w}_{\emph{\mbox{true}}};\underline{\beta}_{i})}\right)\right]$
$\displaystyle=-\sum_{\mathclap{\underline{I}_{i}\in+1,-1}}P(\underline{I}_{i}|\mathbf{w}_{\emph{\mbox{true}}};\underline{\beta}_{i})\log\left(\frac{P(\underline{I}_{i}|\mathbf{w};\underline{\beta}_{i})}{P(\underline{I}_{i}|\mathbf{w}_{\emph{\mbox{true}}};\underline{\beta}_{i})}\right).$

This is the KL divergence between
$P(\underline{I}_{i}|\mathbf{w}_{\emph{\mbox{true}}};\underline{\beta}_{i})$
and $P(\underline{I}_{i}|\mathbf{w};\underline{\beta}_{i})$. This is strictly
non-negative and only equal to zero when
$P(\underline{I}_{i}|\mathbf{w};\underline{\beta}_{i})=P(\underline{I}_{i}|\mathbf{w}_{\emph{\mbox{true}}};\underline{\beta}_{i})$.
When $\beta=0$, each of these terms equals $0$. As $\beta\to\infty$,
$P(\underline{I}_{i}|\mathbf{w};\underline{\beta}_{i})\to
H(I\mathbf{w}^{\top}\varphi)$, where $H(\cdot)$ is the Heaviside step
function. In this case, it holds that
$P(\underline{I}_{i}|\mathbf{w};\underline{\beta}_{i})=P(\underline{I}_{i}|\mathbf{w}_{\emph{\mbox{true}}};\underline{\beta}_{i})$
whenever the values $\mathbf{w}^{\top}\varphi$ and
$\mathbf{w}_{\emph{\mbox{true}}}^{\top}\varphi$ have the same sign.

Therefore, for positive, finite $\beta$ each of the terms in the sum is
positive, so the sum diverges, and so the
$P(\mathbf{w}|\underline{I};\underline{\beta})\to 0$ for all
$\mathbf{w}\neq\mathbf{w}_{\emph{\mbox{true}}}$. ∎

### Bigger $\beta$ isn’t always more informative

Querying a more rational teacher (with a larger $\beta$ value) does not always
lead to faster convergence to the true value, as measured by lower MSE or LL,
because the magnitude of $\mathbf{w}^{\top}\varphi_{ij}$ can be learned from
the teacher making mistakes.

We empirically observe this in Figure 2, where we demonstrate that if our
current belief distribution $P(\mathbf{w})$ is a normal distribution
characterized by $\mu$ and $\sigma$, a lower $\beta$ value is more informative
for certain values of $\mu$ and $\sigma$. Specifically, when the distribution
is symmetric ($\mu=0$) then a larger value of $\beta$ is better, and as the
distribution gets broader (larger $\sigma$) larger $\beta$ is also better. If
the distribution is very wide then a large $\beta$ allows us to quickly remove
a lot of probability mass, while if the distribution is narrow (and
asymmetric) then we learn about the value of $\mathbf{w}^{\top}\varphi_{ij}$
from the humans making mistakes, which requires the human to be less than
perfectly rational. For example, if $\mathbf{w}^{\top}\varphi_{ij}>0$ then a
perfectly rational human would always choose item $i$ over item $j$, and we
would not learn about the actual _value_ of $\mathbf{w}^{\top}\varphi_{ij}$.

Figure 2: For some prior beliefs over $\mathbf{w}$, querying a teacher with a
lower $\beta$ parameter is more informative. The plots show the most
informative $\beta$ value (according to the mean squared error and log loss
metrics, respectively) for a range of beliefs. Each belief is a Gaussian,
parameterized by $\mu$ (horizontal axis) and $\sigma$ (vertical axis). The
purple regions of the plots indicate beliefs where it is most informative to
query a teacher with a $\beta$ of approximately $1$.

##  6 Restaurant Recommendation

We now discuss how our method for reward learning using feedback from multiple
teachers can be applied to a simplified restaurant recommendation domain. In
this domain, the goal is to learn a reward function that can be used to
recommend restaurants to a user. This reward model must be learned from
feedback from multiple teachers, in this case by asking which of two
restaurants a human prefers. It is important to highlight that our approach is
compatible with a variety of popular recommendation tasks, including
entertainment [35, 36], news [37], and shopping [38] recommendations.

More formally, the problem of restaurant recommendation has a set of
restaurants $\rho=\\{\rho_{1},\rho_{2},\ldots,\rho_{n}\\}$ that can be
recommended to a user. Moreover, there is a set of users
$U=\\{U_{1},U_{2},\ldots,U_{m}\\}$ who can be queried about their restaurant
preferences. Each restaurant is expressed as a set of features
$F=\\{\mathrm{Cleanliness},\mathrm{Vegan},\mathrm{Spiciness}\\}$ where
$\mathrm{Cleanliness}\in[1,10]$ describes the cleanliness of the restaurant,
$\mathrm{Vegan}\in\\{0,1\\}$ describes whether the restaurant is vegan-
friendly, and $\mathrm{Spiciness}\in[1,10]$ describes the spiciness of the
food. The preference rating for each restaurant is denoted by
$\mathbf{w}^{\top}\rho_{i}$, where $\mathbf{w}\in\mathbb{R}^{3}$ is a weight
vector that parameterizes the reward model. The aim is to learn the weights
$\mathbf{w}$ using feedback from multiple users to provide useful restaurant
recommendations.

We can represent the restaurant recommendation domain using our approach. The
set of items ${\phi_{1},\phi_{2},\ldots,\phi_{n}}$ is the set of restaurants
$\rho$. The set of human users $U$ is the set of human teachers. The users are
modelled as Boltzmann-rational, and have known rationality parameters
$\beta_{1},\beta_{2},\ldots,\beta_{m}$. Beginning with an initial distribution
$P(\mathbf{w})$, we will use Algorithm 1 to converge to the weight values for
the reward function that represents the user preferences. First, we select a
pair of restaurants for a user to compare (in this case randomly selected) and
apply Equation 8 describing which user should be queried in order to achieve
the lowest metric score in expectation after a single update. Next, this user
is selected and then asked which of the two restaurants they prefer. Finally,
using the selected user’s preference, the reward model weights are updated
according to Equation 5 to generate a new belief distribution. The process is
repeated until the belief distribution converges.

##  7 Experiments

We now show that our approach method for selecting $\beta$ outperforms several
baseline methods, using the simple restaurant recommendation domain. In Figure
3, we compare: (1) selecting the largest $\beta$ value to see if the result
that larger $\beta$ is not always better is true in practice; (2) selecting
$\beta$ randomly to ensure that the advantage over selecting the largest
$\beta$ is not just due to the randomness of the selection; and (3) always
selecting $\beta=1$ because this is often what is assumed to be the
rationality parameter in other work.

In this experiment, the size of the weight vector is $d=3$ and the domain of
the weights is $W=[-10,10]^{3}$, which is discretized. The prior distribution
of the weights is a uniform distribution over this domain
$P(\mathbf{w})=\mathcal{U}(W)$ and the true weight
$\mathbf{w}_{\emph{\mbox{true}}}\in W$ is sampled from this prior. There are
21 teachers, with $\beta$ values uniformly spaced between 0 and 4. For 100
steps, two restaurant feature vectors
$\phi=\\{\mathrm{Cleanliness},\mathrm{Vegan},\mathrm{Spiciness}\\}$ are
generated randomly, where
$\mathrm{Cleanliness},\mathrm{Spiciness}\sim\mathcal{U}(1,10)$, and
$\mathrm{Vegan}$ are uniformly drawn from $\\{0,1\\}$. While we generate our
samples randomly in order to isolate the the effect of teacher selection, any
of the active query selection methods from previous work could be used here.
The teacher is selected and then queried using one of the various methods and
the belief distribution is updated based on the preference of that teacher.
The same $\phi$ vectors are used for each method, so that the only difference
between the methods is the selection of $\beta$. This procedure is repeated
100 times, each time sampling a new true weight vector
$\mathbf{w}_{\emph{\mbox{true}}}$.

Figure 3: Active teacher selection improves reward inference. These plots show
the expected mean squared error and expected log loss over the course of 100
iterations of reward inference using various teacher selection methods. The
solid line is the mean, and the shading is the standard deviation. Selecting
teacher $\beta$ w.r.t. mean square error most effectively minimizes mean
square error, while selecting $\beta$ w.r.t. log loss most effectively
minimizes log loss. In both cases, selecting teachers according to Equation 8
clearly outperforms the heuristic of always selecting the most rational
teacher (largest $\beta$) and the baselines (random $\beta$ and $\beta=1$).

Overall, we observe that the active teacher selection methods (MSE and LL)
outperform the baseline methods. Moreover, we examine how the most informative
value of $\beta$ changes with additional queries in Figure 4. As expected, the
optimal $\beta$ value decreases with additional queries, as the distribution
gets less broad. At beginning of training, our approach queries the teachers
with large $\beta$ values because this enables it to determine the sign of
$\mathbf{w}^{\top}\varphi_{ij}$, and then our approach queries the teachers
with smaller $\beta$ values to determine the magnitude of
$\mathbf{w}^{\top}\varphi_{ij}$ as it gets more information.

Figure 4: This plot shows the most informative values of $\beta$ during
training, averaged across 100 runs (given the expected mean squared error and
expected log loss respectively). The solid line is the mean and the shaded
area is the standard deviation. $\beta$ decreases over the course of training,
as the learner’s belief distribution over $\mathbf{w}$ becomes more confident.

##  8 Limitations and Future Work

For the sake of conceptual clarity and mathematical formalism, we have used
relatively simple human decision-making and reward models. Future work should
extend these results by increasing model complexity.

For example, this analysis assumes that humans are Boltzmann-rational
decision-makers with constant, known $\beta$ values. While more nuanced than
optimal models, Boltzmann-rational models fail to account for systematic
biases in human judgement [28, 39, 40]. This work could be improved by using
more complex, realistic models of human decision-making, for example by
allowing each human’s $\beta$ parameter to vary across the state space to
capture teacher specialization or by measuring and explicitly modeling
systematic cognitive biases. Moreover, this analysis assumes that the teacher
$\beta$ parameters are given, whereas in reality the agent may not have access
to this information. Future work should also examine ways of modeling this
part of human decision-making alongside learning the reward function.

Finally, future work could extend these results to nonlinear reward models,
such as ensembles of neural networks. Moreover, it could explore convergence
properties and optimal querying strategies for learning from teachers with
different reward functions. For example, variations in individual taste might
lead teachers to disagree on which restaurants are best. Future work should
explore the ramifications of such inter-teacher variance on teacher selection
and reward learning.

##  9 Conclusion

In this work, we motivated, specified, and evaluated an algorithm for
selecting which teacher to query during active reward learning with multiple
teachers. Our algorithm models the teachers as Boltzmann-rational with known
$\beta$ parameters. At each time step, it queries the teacher that will be
most informative in expectation. Interestingly, we find that the most
informative teacher is not always the most rational one. We prove and
demonstrate that the reward learner’s belief will eventually collapse to the
true reward function under our algorithm. Our hope is that this method and
analysis will improve reward learning in domains where feedback is gathered
from multiple teachers with varying levels of rationality.

## Acknowledgments

We thank the anonymous reviewers for their valuable comments. This work was
supported in part by a gift from the Open Philanthropy Foundation.

## References

  * Silver et al. [2016] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al.,  Mastering the game of Go with deep neural networks and tree search,  Nature 529 (2016) 484–489.
  * Silver et al. [2017] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al.,  Mastering the game of Go without human knowledge,  Nature 550 (2017) 354–359.
  * Berner et al. [2019] C. Berner, G. Brockman, B. Chan, V. Cheung, P. Dębiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse, et al.,  Dota 2 with large scale deep reinforcement learning,  arXiv preprint arXiv:1912.06680 (2019).
  * Krizhevsky et al. [2017] A. Krizhevsky, I. Sutskever, G. E. Hinton,  ImageNet classification with deep convolutional neural networks,  Communications of the ACM 60 (2017) 84–90.
  * Wang et al. [2017] F. Wang, M. Jiang, C. Qian, S. Yang, C. Li, H. Zhang, X. Wang, X. Tang,  Residual attention network for image classification,  in: IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 3156–3164.
  * Krakovna [2018] V. Krakovna, Specification gaming examples in AI, 2018.
  * Lee et al. [2021] K. Lee, L. M. Smith, P. Abbeel,  PEBBLE: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training,  in: 38th International Conference on Machine Learning, PMLR, 2021, pp. 6152–6163.
  * Leike et al. [2018] J. Leike, D. Krueger, T. Everitt, M. Martic, V. Maini, S. Legg,  Scalable agent alignment via reward modeling: A research direction,  arXiv preprint arXiv:1811.07871 (2018).
  * Jeon et al. [2020] H. J. Jeon, S. Milli, A. D. Dragan,  Reward-rational (implicit) choice: A unifying formalism for reward learning,  arXiv preprint arXiv:2002.04833 (2020).
  * Ng and Russell [2000] A. Y. Ng, S. J. Russell,  Algorithms for inverse reinforcement learning,  in: International Conference on Machine Learning, 2000, pp. 663–670.
  * Abbeel and Ng [2004] P. Abbeel, A. Y. Ng,  Apprenticeship learning via inverse reinforcement learning,  in: 21st International Conference on Machine Learning, 2004, p. 1.
  * Ziebart [2010] B. D. Ziebart, Modeling purposeful adaptive behavior with the principle of maximum causal entropy, Ph.D. thesis, Carnegie Mellon University, 2010.
  * Sadigh et al. [2017] D. Sadigh, A. Dragan, S. Sastry, S. Seshia,  Active preference-based learning of reward functions,  in: Robotics: Science and Systems XIII, 2017, pp. 53–63.
  * Christiano et al. [2017] P. F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, D. Amodei,  Deep reinforcement learning from human preferences,  Neural Information Processing Systems (2017) 4300–4308.
  * Goyal et al. [2019] P. Goyal, S. Niekum, R. J. Mooney,  Using natural language for reward shaping in reinforcement learning,  arXiv preprint arXiv:1903.02020 (2019).
  * Arumugam et al. [2019] D. Arumugam, J. K. Lee, S. Saskin, M. L. Littman,  Deep reinforcement learning from policy-dependent human feedback,  arXiv preprint arXiv:1902.04257 (2019).
  * Bajcsy et al. [2017] A. Bajcsy, D. P. Losey, M. K. O’Malley, A. D. Dragan,  Learning robot objectives from physical human interaction,  Machine Learning Research 78 (2017) 217–226.
  * Hadfield-Menell et al. [2017] D. Hadfield-Menell, S. Milli, P. Abbeel, S. J. Russell, A. Dragan,  Inverse reward design,  in: Neural Information Processing Systems, 2017, pp. 6765–6774.
  * Mindermann et al. [2018] S. Mindermann, R. Shah, A. Gleave, D. Hadfield-Menell,  Active inverse reward design,  arXiv preprint arXiv:1809.03060 (2018).
  * Stiennon et al. [2020] N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, P. F. Christiano,  Learning to summarize with human feedback,  Neural Information Processing Systems 33 (2020) 3008–3021.
  * Ziegler et al. [2019] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, G. Irving,  Fine-tuning language models from human preferences,  arXiv preprint arXiv:1909.08593 (2019).
  * Leike et al. [2022] J. Leike, J. Schulman, J. Wu, Our approach to alignment research, 2022\. URL: https://openai.com/blog/our-approach-to-alignment-research/.
  * Skalse and Abate [2022] J. Skalse, A. Abate,  Misspecification in inverse reinforcement learning,  arXiv preprint arXiv:2212.03201 (2022).
  * Milli and Dragan [2020] S. Milli, A. D. Dragan,  Literal or pedagogic human? Analyzing human model misspecification in objective learning,  in: Uncertainty in Artificial Intelligence, 2020, pp. 925–934.
  * Freedman et al. [2021] R. Freedman, R. Shah, A. Dragan,  Choice set misspecification in reward inference,  arXiv preprint arXiv:2101.07691 (2021).
  * Daniels-Koch and Freedman [2022] O. Daniels-Koch, R. Freedman,  The expertise problem: Learning from specialized feedback,  arXiv preprint arXiv:2211.06519 (2022).
  * Bıyık and Sadigh [2018] E. Bıyık, D. Sadigh,  Batch active preference-based learning of reward functions,  arXiv preprint arXiv:1810.04303 (2018).
  * Evans et al. [2016] O. Evans, A. Stuhlmüller, N. D. Goodman,  Learning the preferences of ignorant, inconsistent agents,  in: 30th AAAI Conference on Artificial Intelligence, 2016, pp. 323–329.
  * Bıyık et al. [2020] E. Bıyık, M. Palan, N. C. Landolfi, D. P. Losey, D. Sadigh,  Asking easy questions: A user-friendly approach to active reward learning,  in: Conference on Robot Learning, 2020, pp. 1177–1190.
  * Bradley and Terry [1952] R. A. Bradley, M. E. Terry,  Rank analysis of incomplete block designs: I. The method of paired comparisons,  Biometrika 39 (1952) 324–345.
  * Liang et al. [2022] X. Liang, K. Shu, K. Lee, P. Abbeel,  Reward uncertainty for exploration in preference-based reinforcement learning,  arXiv preprint arXiv:2205.12401 (2022).
  * Ramachandran and Amir [2007] D. Ramachandran, E. Amir,  Bayesian Inverse Reinforcement Learning.,  in: International Joint Conference on Artificial Intelligence, volume 7, 2007, pp. 2586–2591.
  * Palan et al. [2019] M. Palan, G. Shevchuk, N. Charles Landolfi, D. Sadigh,  Learning reward functions by integrating human demonstrations and preferences,  in: Robotics: Science and Systems XV, 2019, pp. 23–33.
  * Freedman et al. [2020] R. Freedman, J. S. Borg, W. Sinnott-Armstrong, J. P. Dickerson, V. Conitzer,  Adapting a kidney exchange algorithm to align with human values,  Artificial Intelligence 283 (2020) 103261.
  * Gomez-Uribe and Hunt [2015] C. A. Gomez-Uribe, N. Hunt,  The netflix recommender system: Algorithms, business value, and innovation,  ACM Transactions on Management Information Systems (TMIS) 6 (2015) 1–19.
  * Perano et al. [2021] M. Perano, G. L. Casali, Y. Liu, T. Abbate,  Professional reviews as service: A mix method approach to assess the value of recommender systems in the entertainment industry,  Technological Forecasting and Social Change 169 (2021) 120800.
  * Raza and Ding [2021] S. Raza, C. Ding,  News recommender system: A review of recent progress, challenges, and opportunities,  Artificial Intelligence Review (2021) 1–52.
  * Alamdari et al. [2020] P. M. Alamdari, N. J. Navimipour, M. Hosseinzadeh, A. A. Safaei, A. Darwesh,  A systematic study on the recommender systems in the E-commerce,  IEEE Access 8 (2020) 115694–115716.
  * Shah et al. [2019] R. Shah, N. Gundotra, P. Abbeel, A. Dragan,  On the feasibility of learning, rather than assuming, human biases for reward inference,  in: 36th International Conference on Machine Learning, PMLR, 2019, pp. 5670–5679.
  * Chan et al. [2021] L. Chan, A. Critch, A. Dragan,  Human irrationality: Both bad and good for reward inference,  arXiv preprint arXiv:2111.06956 (2021).


11institutetext: VinUni-Illinois Smart Health Center, VinUniversity, Hanoi,
Vietnam;
11email<EMAIL_ADDRESS>John von
Neumann Institute, University of Science, VNU-HCM, Vietnam;33institutetext:
College of Engineering & Computer Science, VinUniversity, Hanoi, Vietnam;
44institutetext: School of Information and Communication Technology, Hanoi
University of Science and Technology, Vietnam;
44email<EMAIL_ADDRESS>55institutetext: University of
Illinois at Urbana-Champaign, US;55email<EMAIL_ADDRESS>
Corresponding author

# Multi-stream Fusion for Class Incremental Learning in Pill Image
Classification

Trong-Tung Nguyen 11 2 2 Hieu H. Pham 11 3 3 * * Phi Le Nguyen 44 Thanh Hung
Nguyen 44 Minh Do 11 3 3 5 5

###### Abstract

Classifying pill categories from real-world images is crucial for various
smart healthcare applications. Although existing approaches in image
classification might achieve a good performance on fixed pill categories, they
fail to handle novel instances of pill categories that are frequently
presented to the learning algorithm. To this end, a trivial solution is to
train the model with novel classes. However, this may result in a phenomenon
known as catastrophic forgetting, in which the system forgets what it learned
in previous classes. In this paper, we address this challenge by introducing
the class incremental learning (CIL) ability to traditional pill image
classification systems. Specifically, we propose a novel incremental multi-
stream intermediate fusion framework enabling incorporation of an additional
guidance information stream that best matches the domain of the problem into
various state-of-the-art CIL methods. From this framework, we consider color-
specific information of pill images as a guidance stream and devise an
approach, namely “Color Guidance with Multi-stream intermediate fusion”(CG-
IMIF) for solving CIL pill image classification task. We conduct comprehensive
experiments on real-world incremental pill image classification dataset,
namely VAIPE-PCIL, and find that the CG-IMIF consistently outperforms several
state-of-the-art methods by a large margin in different task settings. Our
code, data, and trained model are available at https://github.com/vinuni-
vishc/CG-IMIF.

##  1 Introduction

Pill image recognition task has attracted various studies recently with the
aim to design high-quality algorithm for visual-based assistance system on
pill images. This can help the healthcare community automatically identify
unknown pill categories by taking several real-world pictures with mobile
devices. It is noteworthy that real-world scenarios of pill images are often
challenging due to the changing background as well as variances of pill
instances in terms of shape, color, and texture. There have been several works
that are developed to mitigate such challenges, most of them are based on
hand-crafted features [3, 5, 6, 10]. These works are then utilized by Ling et
al.[16] and combined with a two-stage training strategy to create a novel
framework for the pill recognition model in few-shot learning. Another
approach is to explore external knowledge from medical text data (e.g.
prescription) to improve the detection performance of visual-based models [18,
19]. However, existing models are often limited by novel instances of pill
categories which frequently arrive at a pill recognition system. This often
happens when a novel class of pill instance is introduced by images uploaded
from the end-user using mobile devices or from the healthcare community. A
report in [1] shows that there are roughly 40-50 novel drugs being approved
each year. In such a scenario, the core learning model of the system, which is
often deployed in a lightweight device (e.g, mobile phones), might need to
rewind the training process on the whole training data (in which novel
categories participate). This is not an effective strategy for many reasons.
Memory allocated for such extensively training data is often limited.
Acquiring novel knowledge while maintaining what the model has learned so far
requires the system to store a huge amount of samples for both old and new
classes, which is infeasible. Another solution for this is to provide an
initial training dataset for the model. The model is then fine-tuned on novel
categories to update the model’s knowledge about new pill instances. However,
this fine-tuning scheme suffers from a serious behavior of the learning system
which is widely known as catastrophic forgetting [8, 9] (degrading performance
on old tasks while accessing data of novel tasks). This system, therefore, is
in need of a flexible and effective strategy to handle the novel real-world
object categorization of pill image instances. In this way, it would be able
to incrementally learn from new classes without exhaustively storing old
category samples. This scenario is called class-incremental learning (CIL).

The progress of studies on class incremental learning (CIL) for visual tasks
has been developed significantly for many years. The general setting of CIL is
that the disjoint sets of different classes arrive at the learning algorithm
gradually. Many works such as [23, 13, 21, 4, 22] have proposed several
methods which employed available techniques to tackle the mutual challenge:
catastrophic forgetting. Knowledge distillation [12] is the most common
technique which is widely adopted to tackle catastrophic forgetting and was
first applied to the CIL setting by Li et al.[15]. After that, a derived
version [21] with additional usage of representation learning was proposed, in
which valuable herding exemplars are replayed frequently to keep track of the
old knowledge. The strategy of herding is to pick those neighbors which are
nearest to the mean sample of the class. Using this herding strategy, Castro
et al.[4] managed to build an end-to-end framework with an additionally
balanced fine-tuning strategy. On the other hand, Wu et al.[22] introduced a
bias correction approach by adding a bias correction layer. This is conducted
at the last layer of each incremental learning task to refine the overall
scores for the final prediction. Meanwhile, Hou et al.[13] identified the
imbalance between previous and new data as the main issue leading to
catastrophic forgetting. They tackled this imbalanced scenario by
incorporating three main components: cosine normalization, less-forget
constraint, and inter-class separation.

In this research, we aim to investigate the application of CIL methods in a
pill classification system. Fig.1 illustrates the effect of such a system with
and without class incremental learning capability. To the best of our
knowledge, we are the first to explore incremental learning on the pill
classification system. Existing single stream incremental learning methods
[23, 13, 21, 4, 22], when being applied to a domain of application for
practical usage, can be improved with the help of some domain-specific
knowledge. This serves as additional information which might collaborate well
with the original RGB image to alleviate catastrophic forgetting. The
introduction of a supplementary information stream requires a prudent strategy
to incorporate such information. Based on this motivation, we propose a novel
integration framework that serves as a plug-in technique for any available
class incremental learning algorithms. Our fusion framework enables the
incremental learning methods to receive additional information streams as
cues. This will then help to flexibly update corresponding feature
representations in an optimal way for each learning task through the
intermediate stage. To demonstrate the usage of such an integration framework,
we consider color information as additional stream and devise an approach,
named “Color Guidance with Multi-stream intermediate fusion”(CG-IMIF).
Experimental results on a real-world incremental pill image classification
dataset called VAIPE-PCIL show that the proposed learning framework
consistently surpasses most metric scores of various state-of-the-art methods
in different task settings.

Figure 1: The pipeline for a learning algorithm to acquire knowledge of pill
categories could be divided into two options: (a) feeding a fixed pill images
database to an off-the-shelf deep learning algorithm; (b) maintaining a few
samples of old categories as exemplars, combining with novel categories to
form a growing pill image dataset, and finally feeding into a growing deep
classification model.

Our contributions can be summarized in the following three aspects:

  1. 1.

We introduce CG-IMIF, a novel incremental learning framework based on multiple
streams for the task of pill classification from images. To the best of our
knowledge, we are the first to introduce the incremental learning capability
to this task and provide a new approach to tackle challenges in learning novel
pill classes.

  2. 2.

We conduct thorough experiments and in-depth ablation studies to demonstrate
the effectiveness of the proposed approach on a real-world incremental pill
image classification dataset. Experimental results show that the CG-IMIF
consistently outperforms previous state-of-the-art methods by a large margin.

The rest of this paper is organized as follows. We briefly formulate the
problem setting of pill CIL, which we aim to solve in Section 2. Details of
our proposed CG-IMIF framework are described in Section 3. Experimental
results and further analysis are presented in Section 4 and 5. Finally, we
conclude the paper with our discussion on strengths and limitations in Section
6, and 7.

##  2 Preliminaries

###  2.1 Problem Definition and Notation

Generally, the Class Incremental Learning (CIL) problem represented by $\tau$
consists of a sequence of $n$ image classification learning tasks

$\tau=[(C^{1},P_{train}^{1},P_{test}^{1}),(C^{2},P_{train}^{2},P_{test}^{2}),...,(C^{n},P_{train}^{n},P_{test}^{n})],$
(1)

where each tuple $(C^{t},P_{train}^{t},P_{test}^{t})$ depicts a task $t$.
$C^{t}$ is a set of $m^{t}$ categories, i.e.,
$C^{t}=\\{c_{1}^{t},c_{2}^{t},...,c_{m^{t}}^{t}\\}$, $P_{train}^{t}$ and
$P_{test}^{t}$ denote the training and testing data, respectively. To
represent the total number of classes up to the current task, we define
$M^{t}=\sum_{i=1}^{t}\left|C^{i}\right|$. The training, and testing data is
defined as $P^{t}=\\{(X^{t},Y^{t})\\}$ where $X^{t}$ and $Y^{t}$ denote the
training images and their corresponding labels, respectively. During the
training phase, the learning model at stage $t$ is presented with categories
set $C^{t}$, training samples $P_{train}^{t}$, and an exemplar set $K_{t}$. In
practice, $K_{t}$ is a fixed-size set acting as a support set which helps to
retain a partial set of images and the corresponding labels from previous
training data, i.e., $K_{t}\subseteq P_{train}^{1}\cup
P_{train}^{2}\cup...\cup P_{train}^{t-1}$. Therefore, a revised version of
training samples at stage $t$ can be obtained by combining $K_{t}$ and
$P_{train}^{t}$, $K_{t}\cup P_{train}^{t}=V_{train}^{t}$. It is also assumed
that categories of different learning tasks do not overlap (i.e $C^{i}\cap
C^{j}=\varnothing$ where $i\neq j$). At testing time, the performance of
learner $t$ is evaluated on all of the previous seen categories
$\bigcup_{i=1}^{t}C^{t}$ with samples from $\bigcup_{i=1}^{t}P_{test}^{t}$.

###  2.2 Conventional CIL Methods

Several CIL methods have been proposed which consider various properties of
CIL problem to tackle mutual challenge: catastrophic forgetting. Most CIL
works are divided into two branches: exemplar-based, and non-exemplar-based
approaches. While the latter is much more challenging, the former is more
practical since it is reasonable to maintain a few samples of old classes to
avoid performance degrading. Within the scope of this research, we aim to
exploit the capability of exemplar-based CIL methods by attaching these to our
proposed framework. Therefore, we first describe a few core components of
exemplar-based CIL approaches as follows.
Representative Memory is a set of samples from categories of old tasks and is
represented by $K_{t-1}$. It serves as an exemplar set to support model in
revisiting knowledge acquired from old tasks. In an exemplar-based approach,
the learning model can only access the previous category set $C^{t-1}$ through
$K_{t-1}$. The size of the support set is often limited and mainly divided
into two memory settings: 1) a constant number of exemplars per class, and 2)
and a limited capacity of S samples. In the first setting, the size of the
support set $K_{t}$ grows with the number of classes. In addition, the size of
$K_{t}$ in the second memory setting is constant over the time $t$. Samples
across categories are manipulated frequently with two main operations: new
sample selection and old sample removal. For each class, a sorted list of its
samples is maintained based on their distances to the class’ mean feature
vector. Hence, the most representative samples for each class are selected as
members in the next support set $K_{t+1}$. Meanwhile, the remaining samples
are ignored to reserve slots for novel samples from new classes.
Growing Deep Neural Networks in an exemplar-based method is constructed by two
main factors: the common feature extractor backbone, and a growing
classification layer module. At a specific learning stage $t$, new
classification head $CL_{t}$ is initiated to allocate corresponding parameters
$W_{t}$. Feature vectors, after being extracted by the feature extractor, are
fed into $CL_{t}$ to produce prediction logits for the current category set
$C^{t}$. The size of the logits after being input to $CL_{t}$ is equal to the
size of the category set $C^{t}$. The prediction vectors are then utilized to
compute the traditional cross-entropy loss which represents the training loss
on the set of pill images $P^{t}_{train}$ for the current task. On the other
hand, the old classification head $CL_{t-1}$ can be used to represent the old
knowledge of the model. Samples from support set $K_{t}$ can be passed through
a list of classification head module $CL_{i}$ from the first task to the
latest old task $t-1$ (i.e $i\in[0,t-1]$) in order to obtain prediction
logits.
Cross-Distillation Loss Function is common in most of exemplar-based methods.
This is constructed by combining cross-entropy, and distillation loss
function. The cross-entropy loss function helps minimize the overall empirical
errors when learning on new category set $C_{t}$ at task $t$. Meanwhile, the
distillation loss function plays a role in distilling the old model $M_{t-1}$
from previous tasks into the current model $M_{t}$ to avoid catastrophic
forgetting. Let’s consider the incremental learning model at a specific
learning stage $t$ where it has obtained $t-1$ numbers of classification
heads. New classification head $CL_{t}$, which is now added to learn on new
task $t$, produce the prediction logits as
$o(x)=[o_{1}(x),o_{2}(x),...,o_{t}(x)]$ for any input $x$. Similarly, output
logits which are produced by old classification head can be represented as
$\hat{o}(x)=[\hat{o}_{1}(x),\hat{o}_{2}(x),...,\hat{o}_{t-1}(x)]$. With these
representation, the distillation loss can be computed for all samples from
exemplar sets $K_{t-1}$ and from new classes $P^{t}_{train}$ (i.e.
$K_{t-1}\cup P^{t}_{train}=V_{train}^{t}$) as follows:

$\displaystyle L_{d}$ $\displaystyle=\sum_{x\in
V_{train}^{t}}\sum_{k=1}^{t-1}-\hat{\pi}_{k}(x)\log\left[\pi_{k}(x)\right],$
(2) $\displaystyle\hat{\pi}_{k}(x)$
$\displaystyle=\frac{e^{\hat{o}_{k}(x)/T}}{\sum_{j=1}^{t-1}e^{\hat{o}_{j}(x)/T}},\quad\pi_{k}(x)=\frac{e^{o_{k}(x)/T}}{\sum_{j=1}^{t-1}e^{o_{j}(x)/T}},$

where T plays as the temperature scaling factor. Meanwhile, the groundtruth
label for each sample $x$ (i.e. y(x)) for new category sets along with softmax
of logits of the $k$-th category set (i.e. $p_{k}=softmax(o_{k}(x))$ ) can be
used to compute the cross-entropy loss function as follows

$L_{c}=\sum_{x\in V_{train}^{t}}\sum_{k=1}^{t}-y(x)\log\left[p_{k}(x)\right],$
(3)

The final cross-distilled loss function; therefore, can be obtained by
combining distillation loss function in Eqn.2 and cross-entropy loss function
in Eqn.3

$L=\alpha L_{d}+(1-\alpha)L_{c},$ (4)

where the scalar value $\alpha$ controls the balance between the two
functions.

##  3 Methodology

The majority of current pill identification methods rely on RGB images.
Therefore, to the best of our knowledge, most existing systems fail to address
hard examples (e.g, pills with very similar shapes and colors) [16]. This
problem becomes more challenging in the context of the class incremental
learning. In this problem, we have to cope with two issues at the same time:
1) recognizing pill instances that belong to the novel classes, and 2) not
forgetting the previously learned knowledge of the old ones. We seek in this
study robust domain-specific knowledge, which could be in good companion to
traditional RGB image stream. However, the introduction of an additional
stream issue a different challenge; the significant need for a stream
integration method . To tackle such a challenge, we propose an Incremental
Multi-stream Intermediate Fusion framework (IMIF). The IMIF allows additional
information streams to be effectively propagated during the incremental
learning phase. In the following subsections, we briefly define the multi-
stream class incremental learning method and describe how it can be decomposed
into different components.

###  3.1 Multi-stream Class Incremental Learning Model

We define a multi-stream class incremental learning model M as a combination
of three key components: 1) a single stream base method X, 2) an additional
stream of information Y, and 3) a method of fusing stream Z.

M = Base method X \+ Feature stream Y \+ Fusion mechanism Z

At this point, the base method represents any method that follows the general
setting described in Sec.2.2. Y serves as a piece of additional domain
information that gives cues to the learning model apart from RGB images.
Normally, Y is specific to the domain of the task. Lastly, Z presents a fusion
mechanism that enables method M to incorporate additional information stream Y
into the incremental learning process. From this decomposition, our CG-IMIF
replaces: 1) the representative stream Y with color-specific information, and
2) the fusion technique Z with the proposed IMIF. In the following, we
describe our proposed Color histogram guidance stream and Incremental Multi-
stream Intermediate Fusion technique in Sec.3.2 and Sec.3.3, respectively.

Figure 2: Samples of pill images from VAIPE-PCIL dataset are shown on the
first row. The second row exhibits the corresponding color histogram
information for images on the upper row respectively.

###  3.2 Color Histogram Guidance Stream

Pill images compose of various features which can be used as discriminative
factors in classification problems. Among those is color distribution
information which can be approximated by the color histogram of pill images.
The color histogram represents the color distribution of the input image in
terms of three different channels: red, green, and blue. In detail, it is
often encoded into a single vector where the indexes of entries are mapped to
a set of all possible colors. In grey-scale images, values of vector entries
store the frequency that counts the total number of pixels having the
intensity (i.e., color value), which is hashed by the corresponding index of
the vector. However, in a three-dimensional image, color ranges for each
channel are associated across color channels to formulate a unique
combination. This combination accounts for those pixels of which color ranges
lie in three discrete ranges of value:
$[r_{i},r_{j}];[g_{k},g_{l}];[b_{m},b_{n}]$. The color feature vectors are
useful to make a distinction among pill instances that have similar shapes but
different colors. The color ranges for each channel of the RGB images are
divided into eight segments in our problem, where each segment represents 32
different consecutive color values. After that, the color histogram vector can
be obtained by accumulating the quantities of pixels assigned to a specific
color range for each channel to result in a vector with 8$\times$8$\times$8 =
512 elements. Fig. 2 illustrates a few examples of extracting the color
histogram stream for each corresponding cropped pill image.

Figure 3: The traditional training paradigm with only single-stream
information used by almost existing exemplar-based CIL methods.

###  3.3 Incremental Multi-stream Intermediate Fusion Technique

Traditional Early Fusion A naive fusion technique Z concatenates different
streams of information right after the feature extraction phase. Specifically,
feature vectors $f_{r}\in\mathbb{R}^{d_{r}}$, and $f_{Y}\in\mathbb{R}^{d_{Y}}$
are extracted from raw RGB images, and additional information stream Y,
respectively. Both of these features are then fed into separate projection
layers to project into the same latent space. In practice, the projection
layers are implemented by a single hidden layer controlled by parameters
$\Theta^{p}=[W^{p},b^{p}]$ as follows

$s_{r}=\sigma(W^{p}_{r}.f_{r}+b^{p}_{r}),\\\
s_{Y}=\sigma(W^{p}_{Y}.f_{Y}+b^{p}_{Y}),\\\ s_{g}=[s_{r},s_{Y}].$ (5)

The projected vectors are then concatenated to obtain the global feature
vector $s_{g}\in\mathbb{R}^{d_{g}}$. This global feature $s_{g}$ can be
considered as a single input into any traditional single stream CIL methods as
shown in Fig. 3.

Figure 4: Our proposed CG-IMIF architecture composes of: 1) color histogram
feature extraction (orange block), and 2) intermediate fusion framework
(purple block) to incorporate additional information stream.

Intermediate Fusion We observed that fusing information stream in an early
manner for a class incremental learning problem is not optimal. The global
feature $s_{g}$ is problematic since it is regularly updated at each
incremental task. As a result, the projection layer in the early phase can not
find good parameters that balance the performance of old and novel categories
in different tasks. Therefore, we propose to relocate the projection layer to
the intermediate phase instead (i.e, incremental learning phase) by initiating
an entirely novel projection layer in an incremental manner. When $C^{t}$ from
new task $t$ arrives at the system, a new classification layer $CL_{t}$ is
created . This layer accompanies by an attached projection layer specific to
the information stream of a specific task. Therefore, the parameters
controlling the projection layer for each information stream are different
from those defined in the early fusion.

$\Theta^{p^{t}}_{r}=[W^{p^{t}}_{r},b^{p^{t}}_{r}],\Theta^{p^{t}}_{Y}=[W^{p^{t}}_{Y},b^{p^{t}}_{Y}].$
(6)

##  4 Experiments

###  4.1 Dataset

We employ a real-world image dataset, namely VAIPE-Pill (VAIPE Pill
Identification) [2] to exploit CIL capability on the pill image classification
problem. This dataset is created to promote the research on recognizing
distinct types of medicines from mobile devices. The dataset contains 7,294
pill images of 262 categories taken in real-world scenarios. The
characteristics of VAIPE-Pill dataset are illustrated in Tab. 1.

To facilitate research of CIL in pill image classification tasks, we derive a
dataset version, namely VAIPE-PCIL (VAIPE Pill Class Incremental Learning)
dataset from the original VAIPE-Pill data. VAIPE-PCIL is obtained by cropping
pill instances from the original data. We only select those categories which
satisfy either of the following conditions: 1) the number of samples should
not be too small (i.e., and 2) larger than 10 samples), image size of samples
should be at least $64\times 64$. Samples of pill image from VAIPE-PCIL can be
found in Fig. 1. All of our experiments are conducted on the VAIPE-PCIL
dataset to study the performance of CG-IMIF.

Table 1: Statistics of VAIPE-Pill dataset on different characteristics. Characteristic | Training set | Testing set | Total
---|---|---|---
Number of images | 6,461 | 833 | 7,294
Number of pill categories | 262 | 262 | 262
Instances per category | $179.75$ | $23.56$ | $203.2$
Image size (pixel$\times$pixel, mean) | $3,311\times 3,276$ | $3,276\times 3,469$ | $3,300\times 3,400$
Instances per image | $7.28$ | $7.4$ | $7.3$
Number of bounding box annotations | 47,097 | 6,174 | 53,271
Number of categories per image | $5.18$ | $5.76$ | $5.32$

###  4.2 Experimental Protocol

Settings We follow the standard benchmark protocol proposed in [13]. We fix
class arrangements in random order. After each training stage $t$, the
resulting learner is evaluated on the testing data
$\bigcup_{i=1}^{t}P_{test}^{t}$ which represents for all of the testing data
up to the current task $t$. Since no test data from the previous learning
stage are hidden from the learner, it is guaranteed that no overfitting can
occur.

There are two commonly different task evaluation settings in class incremental
learning: task-awareness and task-agnostic. The first setting is much easier
for the algorithm since it has access to the task ID (i.e., ID or set of
categories) about the incoming test data. Therefore, it is reasonable to only
use the corresponding classification head in the incremental learning phase,
which is trained on that task-ID to evaluate the performance. This task
setting, however, is not practical in many real-world circumstances since
task-ID is not always available. We evaluate our performance in terms of task-
agnostic instead. In task-agnostic, the model is not given the task identities
of the test data. Hence, the evaluation results are achieved by taking the
results of all prediction logits, which are predicted by all of the
classification heads. In this way, the model has to learn to resolve the
confusion among classes from a different set of classes.

Evaluation Metrics We adopt two commonly used benchmark metrics from [13] for
CIL problems: average accuracy and average forgetting rate. The average
accuracy and forgetting rate records of performance for each incremental
learning phase are often if a single number is preferable. Meanwhile, the
average phase accuracy and forgetting rate would be used to observe learning
behaviors during incremental tasks for each method.

###  4.3 Implementation Details

All of our experiments and methods are implemented with Pytorch [20] and
trained on a single NVIDIA GeForce RTX 3090. We inherit the codebase from
FACIL [17]. They have already implemented various state-of-the-art methods for
CIL problems in a well-structured manner. Details of base models as well as
the implementation of our IMIF framework are discussed below. In all
experiments, we attach our IMIF framework to several state-of-the-art methods
in CIL: BiC[22], EEIL[4], and LUCIR[13]. Since these methods followed the
common prototype of exemplar-based methods, we discussed some of the general
settings of exemplar-based methods in our experiments before diving into
details about the setting of each base one. There are two common strategies to
reserve samples for old classes: 1) exemplar-management stores a constant
number of samples for each old class, or 2) it maintains a fixed capacity
(e.g, $R_{total}=2,000$ for CIFAR-100 [14] and $R_{total}=20,000$ for ImageNet
[7]). In our experiments, we follow the first setting since it is usually more
challenged than the second one. In addition, the exemplars are randomly
selected among different categories. For the training network, we made use of
50-layer ResNet [11] with no pre-trained weights as the backbone for feature
extraction module, which is applicable to the VAIPE-PCIL dataset. To fairly
compare the performance, we fixed the number of training epochs (200 epochs)
across different methods. The learning rate is initialized with 0.1 and is
divided by 1.5 if the loss function suffers from non-decreasing circumstances
for a specific number of attempts (e.g, $lr_{patience}=5$). The networks are
trained using stochastic gradient descent with mini-batches of 32 samples. The
training images are resized to the same shape of $256\times 256\times 3$ with
only one transformation (e.g, flipping). The class orders across different
methods are randomly fixed for a fair comparison.

In terms of configuration for base methods, we follow the same settings for
the original version BiC [22], and our improved version BiC-CG-IMIF. BiC[22]
proposed to integrate a bias correction layer attached to the end of each
classification head to adjust the classification score. The number of training
epochs for the bias correction layer is 200 epochs in our setting. Moreover,
we set 0.1 as the ratio of the number of exemplars that are used for the
validation. EEIL [4] performs an additional fine-tuning phase after each
official training phase to balance the performance between old and novel
categories. In our experiments, we fix 40 as the number of epochs for fine-
tuning and the learning rate fine-tuning factor as 0.01 across different
methods. We also adhere to the base setting of LUCIR[13] method where they
removed ReLU in the penultimate layer to take both positive and negative
values. For the IMIF framework, the projection layer implemented is
represented by a single hidden layer. Therefore, the output size for different
projection layers should be the same so that the transformed feature vectors,
then can be fused in the shared space. In terms of the color-guided
information, color ranges for each channel of the RGB images are divided into
8 segments where each segment represents 32 different consecutive pixel
values.

###  4.4 Experimental Results

Table 2:  Average accuracy $\bar{\mathcal{A}}$ (%) and forgetting rate $\mathcal{F}$ (%) of CG-IMIF compared to other state-of-the-art results in different task settings. Best scores are marked in bold for both evaluation metrics.  Metric | Method | _Task Settings_ |
---|---|---|---
$N$=5 |  $N$=10 |  $N$=15 |
|  EEIL[4] | 63.83 | 62.40 | 57.41 |
| EEIL-CG-IMIF | 70.80 | 64.85 | 60.93 |
_Average acc._ (%) $\uparrow$ |  BiC [22] | 53.83 | 55.75 | 53.77 |
$\bar{\mathcal{A}}=\frac{1}{n}\sum_{i=1}^{n}\mathcal{A}_{i}$ | BiC-CG-IMIF | 65.53 | 63.59 | 54.83 |
|  LUCIR [13] | 69.63 | 62.90 | 55.49 |
| LUCIR-CG-IMIF | 76.85 | 69.94 | 64.97 |
|  EEIL [4] | 49.82 | 45.46 | 48.27 |
| EEIL-CG-IMIF | 46.68 | 44.64 | 46.23 |
_Forgetting rate_. (%) $\downarrow$ |  BiC [22] | 20.05 | 30.50 | 26.93 |
$\bar{\mathcal{F}}=\frac{1}{n}\sum_{i=1}^{n}\mathcal{F}_{i}$ | BiC-CG-IMIF | 7.75 | 22.01 | 27.35 |
|  LUCIR [13] | 44.13 | 44.32 | 47.11 |
| LUCIR-CG-IMIF | 33.15 | 37.88 | 39.79 |
⋄ Using the similar exemplar settings and selection for fair comparison.

We evaluate our proposed CG-IMIF approach and report the overall performance
in comparison with several state-of-the-art approaches in Tab.2 Experimental
results show that most of the state-of-the-art approaches attached with our
proposed IMIF tool and color-specific information as additional stream help to
achieve consistent improvements over task settings. The setting consists of
three tasks in total where the number of categories is uniformly distributed
for 5, 10, and 15 tasks. It is noticeable that the lower score of forgetting
rate indicates that the model is more unlikely to forget about old knowledge.
In addition, average phase accuracy and forgetting rate is also illustrated in
Fig. 6,6 to inspect the learning behaviour of each method through incremental
phases. Dashed and solid lines with different colors are utilized to
differentiate the base ones (X) and our CG-IMIF, respectively. In terms of the
average phase accuracy, LUCIR-CG-IMIF obtains the highest performance where it
can consistently and significantly surpass other methods (also the base one-
LUCIR). On the other hand, BiC-CG-IMIF is better at mitigating the forgetting
constraint. However, it is not consistent over tasks and the curve fluctuates.
One possible explanation is that the bias layer inside the traditional BiC
method and BiC-CG-IMIF might cause the model to sacrifice the performance of
the current task to maintain the memory of the old ones.

|  |
---|---|---

(a) 5 tasks (b) 10 tasks

(c) 15 tasks

Figure 5: Incremental accuracy for different task settings among the original
version and our method CG-IMIF.

|  |
---|---|---

(a) 5 tasks (b) 10 tasks

(c) 15 tasks

Figure 6: Incremental forgetting rate for different task settings among the
original version and our method CG-IMIF.

##  5 Ablation Studies

To examine the effect of additional information stream usage and fusion
framework, we perform extensive ablation studies. This is aim to observe the
effect of different components in our proposed framework where LUCIR is chosen
as the base method. LUCIR is preferable because of the consistency and high
performance of LUCIR across task settings in the experimental results which
have been discussed in Sec. 4.4. In addition to color information, edge
signals might be a good candidate to discriminate different pill categories
based on their shape. To understand the importance of different stream usage
in our method, we compare 4 different settings: 1) RGB image only, 2) RGB and
edge images, 3) RGB and color histogram, and 4) a combination of all three
streams. Each separated row in Tab.3 refers to each scenario of information
stream usage with two different fusion techniques. Concretely, the setting
that combines RGB and color histogram streams achieves the highest score. One
possible explanation for this result is that the edge signal might not be
sufficiently strong to push the performance.

In addition, we implement the basic fusion technique where additional
information streams are fused in an early manner. Each separated row in Tab.3
illustrates the results of two different fusion mechanisms. Our fusion
technique (on the second line of each row) outperforms the traditional one in
various metrics and task settings. The best result is LUCIR-CG-IMIF which
integrates color histogram information into the traditional LUCIR method with
IMIF.

##  6 Discussions

Table 3:  Ablation performance to compare variants of combination which utilize different information streams as well as different fusion techniques. The Combination which achieves highest performance over different tasks is our CG-IMIF and is marked in green.  Variant of Combination |  _Average acc._ (%) $\uparrow$ |  |  _Forgetting rate._ (%) $\downarrow$
---|---|---|---
$N$=5 |  $N$=10 |  $N$=15 |  |  $N$=5 |  $N$=10 |  $N$=15
RGB only | 69.93 | 62.90 | 55.49 |  | 44.13 | 44.32 | 47.1
RGB-Edge \+ Early | 70.94 | 63.90 | 55.28 |  | 42.4 | 42.13 | 45.80
RGB-Edge \+ Intermediate | 72.58 | 68.38 | 62.90 |  | 38.78 | 38.19 | 41.02
RGB-Color \+ Early | 73.58 | 64.57 | 53.56 |  | 37.825 | 42.86 | 46.15
RGB-Color\+ Intermediate | 76.85 | 69.94 | 64.97 |  | 33.15 | 37.88 | 39.79
RGB-Edge-Color \+ Early | 69.99 | 63.33 | 56.34 |  | 42.35 | 44.17 | 46.24
RGB-Edge-Color\+ Intermediate | 73.65 | 68.32 | 62.15 |  | 36.30 | 38.48 | 40.58

Key Findings. To the best of our knowledge, this work is the first to tackle
the class incremental learning problem for the pill image domain, which is
crucial and applicable for real-world pill recognition systems. Also, we
empirically showed that the technique of intermediate fusion with the
additional stream is superior to the early fusion technique. One plausible
explanation for this effect is the flexibility of the fusion layer after it
has been relocated to the intermediate stage. This allows the additional
information to maintain its optimal performance for old tasks while learning
to adapt to new tasks.

Limitations. Though the proposed framework has superior performance over
several state-of-the-art methods in CIL, it contains some limitations in
different aspects. The new fusion layer at the intermediate phase might
enlarge the model’s size in terms of the number of parameters. Considering the
scenario when a massive amount of tasks are encountered in the learning
progress, the learning model could create sequences of abundant layers. This
might create a side effect when too much memory is reserved for storing the
model’s parameters. Such reservation is unreasonable in a real-world
deployment. Another restriction with the proposed framework is related to
additional stream utilization. Apart from the traditional RGB stream, another
information channel that is specific to the domain of usage might impose a
disagreement with the original RGB channel. This requires a careful study of a
different combination of streams accompanying the traditional stream to
observe its effect.

##  7 Conclusion

This paper introduces the incremental learning capability to the traditional
pill image classification systems. To this end, we propose a novel framework,
namely Incremental Multi-stream Intermediate Fusion (IMIF) which integrates an
additional stream of information to improve the performance of the single
stream CIL method. We then devise CG-IMIF which utilizes IMIF along with a
color histogram as guidance information. Our CG-IMIF is flexible and can be
attached to any exemplar-based approach to improve the performance of the base
ones. We experimentally show that CG-IMIF outperforms many existing state-of-
the-art methods on the VAIPE-PCIL dataset. We hope our work would lay the
foundation and could benefit several types of future research into the
continual learning ability of intelligent machines in smart health
applications.

Acknowledgements. This work was funded by Vingroup Joint Stock Company
(Vingroup JSC),Vingroup, and supported by Vingroup Innovation Foundation
(VINIF) under project code VINIF.2021.DA00128. Trong-Tung Nguyen was funded
and and supported by the Master, PhD Scholarship Programme/Post-Doctoral
Scholarship Programme of Vingroup Innovation Foundation (VINIF), Vingroup Big
Data Institute (VinBigdata), code VINIF.2021.ThS.JVN.07.

## References

  * [1] CDER’s New Molecular Entities and New Therapeutic Biological Products — fda.gov. https://www.fda.gov/drugs/development-approval-process-drugs/new-drugs-fda-cders-new-molecular-entities-and-new-therapeutic-biological-products, [Accessed 04-Jul-2022]
  * [2] Anh Duy, N., Dung Thuy, N., Thanh Hung, N., Phi Le, N., Hieu H., P., Minh N., D.: VAIPE: A Large-scale and Real-World Open Pill Image Dataset for Visual-based Medicine Inspection. https://vaipe.org/
  * [3] Caban, J.J., Rosebrock, A., Yoo, T.S.: Automatic identification of prescription drugs using shape distribution models. In: 2012 19th IEEE International Conference on Image Processing. pp. 1005–1008 (2012). https://doi.org/10.1109/ICIP.2012.6467032
  * [4] Castro, F.M., Marín-Jiménez, M.J., Guil, N., Schmid, C., Alahari, K.: End-to-end incremental learning. CoRR abs/1807.09536 (2018), http://arxiv.org/abs/1807.09536
  * [5] Chen, Z., Kamata, S.i.: A new accurate pill recognition system using imprint information 9067 (11 2013). https://doi.org/10.1117/12.2051168
  * [6] Chen, Z., Yu, J., Kamata, S.i., Yang, J.: Accurate system for automatic pill recognition using imprint information. IET Image Processing 9 (07 2015). https://doi.org/10.1049/iet-ipr.2014.1007
  * [7] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)
  * [8] French, R.: Catastrophic forgetting in connectionist networks. Trends in cognitive sciences 3, 128–135 (05 1999). https://doi.org/10.1016/S1364-6613(99)01294-2
  * [9] Goodfellow, I.J., Mirza, M., Xiao, D., Courville, A., Bengio, Y.: An empirical investigation of catastrophic forgetting in gradient-based neural networks (2013). https://doi.org/10.48550/ARXIV.1312.6211, https://arxiv.org/abs/1312.6211
  * [10] Hartl, A.: Computer-vision based pharmaceutical pill recognition on mobile phones (05 2012)
  * [11] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015), http://arxiv.org/abs/1512.03385
  * [12] Hinton, G.E., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. CoRR abs/1503.02531 (2015), http://arxiv.org/abs/1503.02531
  * [13] Hou, S., Pan, X., Loy, C.C., Wang, Z., Lin, D.: Learning a unified classifier incrementally via rebalancing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019)
  * [14] Krizhevsky, A., Nair, V., Hinton, G.: Cifar-100 (canadian institute for advanced research) http://www.cs.toronto.edu/~kriz/cifar.html
  * [15] Li, Z., Hoiem, D.: Learning without forgetting. CoRR abs/1606.09282 (2016), http://arxiv.org/abs/1606.09282
  * [16] Ling, S., Pastor, A., Li, J., Che, Z., Wang, J., Kim, J., Le Callet, P.: Few-shot pill recognition. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 9786–9795 (2020). https://doi.org/10.1109/CVPR42600.2020.00981
  * [17] Masana, M., Liu, X., Twardowski, B., Menta, M., Bagdanov, A.D., van de Weijer, J.: Class-incremental learning: survey and performance evaluation. arXiv preprint arXiv:2010.15277 (2020)
  * [18] Nguyen, A.D., Nguyen, T.D., Pham, H.H., Nguyen, T.H., Nguyen, P.L.: Image-based contextual pill recognition with medical knowledge graph assistance. arXiv preprint arXiv:2208.02432 (2022)
  * [19] Nguyen, T.T., Nguyen, H.D., Nguyen, T.H., Pham, H.H., Ide, I., Nguyen, P.L.: A novel approach for pill-prescription matching with gnn assistance and contrastive learning. arXiv preprint arXiv:2209.01152 (2022)
  * [20] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: Pytorch: An imperative style, high-performance deep learning library. In: Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32, pp. 8024–8035. Curran Associates, Inc. (2019), http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
  * [21] Rebuffi, S., Kolesnikov, A., Lampert, C.H.: icarl: Incremental classifier and representation learning. CoRR abs/1611.07725 (2016), http://arxiv.org/abs/1611.07725
  * [22] Wu, Y., Chen, Y., Wang, L., Ye, Y., Liu, Z., Guo, Y., Fu, Y.: Large scale incremental learning. CoRR abs/1905.13260 (2019), http://arxiv.org/abs/1905.13260
  * [23] Zhang, J., Zhang, J., Ghosh, S., Li, D., Tasci, S., Heck, L.P., Zhang, H., Kuo, C.J.: Class-incremental learning via deep model consolidation. CoRR abs/1903.07864 (2019), http://arxiv.org/abs/1903.07864