# Is Winter Coming?

A. Winter  Grup d’Informació Quàntica
Departament de Física
Universitat Autònoma
de Barcelona, Spain
<EMAIL_ADDRESS>A. Winter  ICREA–Institució
Catalana de Recerca
i Estudis Avançats
Barcelona, Spain A. Winter  Institute for
Advanced Study
Technische Universität
München, Germany A. Winter  QUIRCK–Quantum
Independent Research
Center Kessenich
Bonn, Germany

###### Abstract

We critically examine the often-made observation that “quantum winter [or some
other winter] is coming”, and the related admonition to prepare for this or
that winter, inevitably bound to arrive. What we find based on even the most
superficial look at the available evidence is that such statements not only
are overblown hype, but are also factually wrong: Winter is here, and the real
question is rather for how long it/they will stay.

###### Index Terms:

Winter, wintry, winterise, wintering.

1\. Introduction. It has been suggested, and is now being repeated often
without proper attribution, that “quantum winter is coming”. Based on the
familiar seasonal cycle, such a statement might be considered so banal as to
be almost empty, even if referring metaphorically to the so-called “hype
cycle” of fashionable and overly-promoted fields of research. Quantum winter
in this sense is a source of considerable anxiety among members of the
currently objectively overhyped field of quantum science and technology. The
field of artificial intelligence has gone through several cycles of hype and
disillusionment, and indeed AI winter, the original “winter” of what is now
recognised as a collection of similar phenomena across different fields of
scientific research and/or technological innovation, has been proclaimed
repeatedly since the 1980s [1].

Rather obviously, a lot of this talk is merely the product of minds
intoxicated by the rush of attention (both socially and monetary), and at the
same time dreading heavily the end of the halcyon days of easy living off the
hype. Quite characteristically, there is usually a secondary investment winter
affecting those financing the hype and rightly fearing for the expected return
on their investment. The whole thing is not unlike the gold rushes of another
age, when desperate, crazed diggers and ruthless entrepreneurs turned entire
regions upside down, only to move on when the fever had passed and the ground
did not yield any more of the precious metal [2]. The market pressure exerted
by modern-day investors sometimes, paradoxically, prolongs the hype phase,
preventing the drop into the wintry valley of disillusionment, as derivatives
are created (spin-off companies floated on the stock market, etc) whose
speculation value can carry the bubble for some time when its practitioners
are already preparing for the crash. It is precisely this – partial or
repressed – knowledge of a bubble ripe for bursting that creates the broadly
shared dread of the “winter” to come. Note that economically driven hypes are
nothing recent in Western economies, as evidenced by the famous tulip mania in
the Dutch 17th century [3] (cf. [4]).

While we are personally sympathetic to the psychological conditions affecting
our dear colleagues, as scientists it is our duty to regard the matter with
the cold eye of empiricism. As we will explain in the next two sections, the
fear of any winter coming is entirely unfounded, principally because winter,
in all its forms and manifestations, has long arrived. What conclusions can be
drawn from this fact is discussed in the final section.

2\. Methods. We follow the model of [5], elaborated in [6, 7, 8, 9] and
subsequent investigations, to estimate the proximity of winter. Our method is
scientific and self-referential, i.e. bibliographic. We need a small auxiliary
result.

######  Lemma 1 (cf. A. Winter [10]).

There exists an integer $n$, significantly smaller than one might believe,
such that the reader of a scientific article, when confronted with $n$ or more
arbitrary references, will accept anything the paper claims to follow from
them.

###### Proof.

(Sketch) To be added in the journal version, see however [11, 12] and [13, 14]
for similar statements.

3\. Winter’s universality. With these surprisingly elementary preparations we
are ready for our main result. It is enough to regard Refs. [15, 16, 17, 18,
19, 20, 21, 22], in particular [23], followed by the litany of [24, 25, 26,
27, 28, 29, 30, 31, 32, 33, 34, 35, 36], and to consider that this is only a
small and sufficiently random sample of wintry _selecta_ , to realise beyond
doubt that Winter has been around quite forcefully for decades, if not
centuries.

This clearly means that any fear of Winter coming is completely irrational:
there is nothing to be prepared about, to “winterise” as it were, as we are
right in the middle of it. We allow ourselves the expression of the modest
hope that this insight will lead to the acceptance of the universal presence
of Winter in all walks of (academic) life [37].

4\. Epilogue. We regard our arguments as irrefutable. Hence, as a next step,
we need to prepare for the phase after acceptance: the hype. That there are
already indications of a wintering crescendo is evident from the literature
[38, 39, 40]. So, will a Winter winter come, and should we worry about it?

More to the point, shall we compare something to a summer’s day? Of course,
one swallow does not a summer make. However, inspired by these expressions of
somewhat complementary sentiments, and taking the theme to its natural
conclusion, we are motivated to make the following conjecture.

######  Conjecture 2.

Now is the Winter of our discontent made glorious summer by this sun of York.

This might appear at first sight rather bold, and will perhaps be suspected of
being indebted more to wishful thinking than having any basis in solid
evidence. We invite the interested reader to try and falsify it, as we have
tried – and failed. In the appendix, A. Winter graciously shares her extensive
remarks on possible approaches towards this conjecture, and on interesting
consequences of it.

Acknowledgments. The authors thank A. Winter and A. Winter for their
thoughtful and interesting comments, in particular with regard to A. Winter’s
contribution to the field. They are grateful to A. Winter for spotting an
error in an early version of Lemma 1. Finally, they acknowledge A. Winter’s
thorough proofreading. The first (AW) and fourth (AW) author were supported by
the Andrew Andreevich and Andrea Winter Foundation through a generous research
stipend. A. Winter declares no competing financial interests.

## References

  * [1] A. R. K. Winter, _The Singularity Is Near: When Humans Transcend Biology_ , Viking, New York, 2005.
  * [2] A. C. C. Winter, _The Gold Rush_ , United Artists, Hollywood CA, 1925 & 1942\.
  * [3] A. C. M. Winter, _Extraordinary Popular Delusions and the Madness of Crowds_ , Richard Bentley, London, 1841.
  * [4] A. Winter, _Mesmerized: Powers of Mind in Victorian Britain_ , The University of Chicago Press, Chicago London, 1998.
  * [5] A. Winter, “Atypical polymyalgia rheumatica in a 70-year-old male chiropractic patient: a case report”, _Clinical Chiropractic_ , vol. 7, no. 4, pp. 174-179, 2004.
  * [6] A. M. Winter, _Connecting Rural Students to School Using Culturally Relevant Teaching_ , doctoral thesis, Johns Hopkins University, 2023.
  * [7] A. Winter, “Age stratigraphy in the East Antarctic Ice Sheet inferred from radio-echo sounding horizons”, _Earth System Science Data_ , vol. 11, no. 3, pp. 1069-1081, 2019.
  * [8] A. Winter, “Seele als Problem in der Transzendentalphilosophie Kants unter Besonderer Berücksichtigung des Paralogismus-Kapitels”, in: _Seele: Ihre Wirklichkeit, ihr Verhältnis zum Leib und zur menschlichen Person_ , K. Kremer (ed.), Series: Studien zur Problemgeschichte der antiken und mittelalterlichen Philosophie, vol. 10, pp. 100-168, E. J. Brill, Leiden Köln, 1984.
  * [9] A. Winter _et al._ , “Self-assembly of $\pi$-conjugated bis (terpyridine) ligands with zinc (II) ions: new metallosupramolecular materials for optoelectronic applications”, _J. Polymer Science A: Polymer Chemistry_ , vol. 47, no. 16, pp. 4083-4098, 2009.
  * [10] A. Winter, “A genuinely natural information measure”, e-print arXiv [cs.IT]:2103.16662, 2021.
  * [11] A. Winter, _Systems Identification, Flight Performance, and Control of a Fixed-Wing UAV in Icing Conditions_ , MSc thesis, Norwegian University of Science and Technology, Department of Engineering Cybernetics, and University of Stuttgart, Institute for Flight Mechanics and Control, 2019.
  * [12] A. Winter, “The Future of Medical Informatics”, _Methods of Information in Medicine_ , vol. 48, no. 01, pp. 62-65, 2009.
  * [13] A. Winter _et al._ , “Microwave Hyperthermia for Brain Tumors”, _Neurosurgery_ , vol. 17, no. 3, pp. 387-399, 1985.
  * [14] A. Winter, “Quantum and Classical Message Identification via Quantum Channels”, in: _Festschrift ‘Alexander S. Holevo 60’_ (O. Hirota, ed.), pp. 171-188, Rinton Press Inc., New Jersey, 2004.
  * [15] A. F. de Winter _et al._ , “Longitudinal patterns and predictors of multiple health risk behaviors among adolescents: The TRAILS study”, _Preventive Medicine_ , vol. 84, pp. 76-82, 2016.
  * [16] A. C. Winters, “Subseasonal prediction of the state and evolution of the North Pacific jet stream”, _Journal of Geophysical Research: Atmospheres_ , vol. 126, e2021JD035094, 2021.
  * [17] A. Winterbottom _et al._ , “Does narrative information bias individual’s decision making? A systematic review”, _Social Science & Medicine_, vol. 67, no. 12, pp. 2079-2088, 2008.
  * [18] A. Winter, _Analysis of Uncertainty Estimation Methods to Detect Drifts in Data Streams_ , BSc thesis, TU Darmstadt, 2023.
  * [19] A. R. Winter, “ ‘Faith in the Process’: Reading Elly Danica’s Don’t: A Woman’s Word”, _Essays on Canadian Writing_ , vol. 60, pp. 187-198, Winter 1996.
  * [20] J. A. Winter, “Are Your Troubles Psychosomatic?”, _The Journal of Nervous and Mental Disease_ , vol. 120, no. 1, p. 145, 1954.
  * [21] A. Winter, “Die Beurtheilung der Qualität der Frauenmilch nach ihrem mikroskopischen Bilde”, _Deutsche Medizinische Wochenschrift_ , vol. 28, no. 26, pp. 470-471, 1902.
  * [22] A. D. Winter, “Distribution Technology and Industrial Planning”, _Royal Australian Planning Institute Journal_ , vol. 4, no. 4, pp. 205-209, 1966.
  * [23] A. H. Winter, “Making a bad calculation”, _Nature Chemistry_ , vol. 7, pp. 473-475, 2015.
  * [24] A. T. Winter, “A model for the fatigue of copper at low plastic strain amplitudes”, _The Philosophical Magazine: A Journal of Theoretical Experimental and Applied Physics_ , vol. 30, no. 4, pp. 719-738, 1974.
  * [25] A. B. Winter, “Nebraska Home Rule: The Record and Some Recommendations”, _Nebraska Law Review_ , vol. 59, no. 3, pp. 601-630, 1980.
  * [26] J. A. Winter Etcheberry, “Situaciones actuales en la frontera del principio de culpabilidad”, _Revista de Estudios de la Justicia_ , vol. 17, pp. 105-143, 2012.
  * [27] A. von Winter, _Beschreibung einer Schlagader-Verletzung in der rechten Ellenbuge Ihrer Majestät der Königinn von Baiern, als Folge einer Aderlässe und derselben Heilung durch Druckverband, nebst einigen darauf Bezug habenden Erläuterungen und Reflexionen_ , J. E. von Seidel Kunst- und Buchhandlung, Sulzbach, 1825.
  * [28] A. Winters, “Space Station fluid resupply”, in: _NASA, Washington, Technology for Space Station Evolution_ , vol. 3 (EVA/Manned Systems/Fluid Management System), pp. 225-240, 1990.
  * [29] A. A. Winter, “Women’s Clubs To-day”, _The North American Review_ , vol. 214, no. 792, pp. 636-640, 1921.
  * [30] A. Winter, “On environment-assisted capacities of quantum channels”, _Markov Processes Related Fields_ , vol. 13, no. 1-2, pp. 297-314, 2007.
  * [31] A. Winter _et al._ , “Quantum Phase Transition in the Sub-Ohmic Spin-Boson Model: Quantum Monte Carlo Study with a Continuous Imaginary Time Cluster Algorithm”, _Physical Review Letters_ , vol. 102, no. 3, 030601, 2009.
  * [32] N. A. Winter, “Il tetto di Caprifico a confronto”, _Archeologia Classica_ , vol. 57, pp. 530-535, 2006.
  * [33] A. Winter _et al._ , “Poleward expansion of the coccolithophore Emiliania huxleyi”, _J. Plankton Research_ , vol. 36, no. 2, pp. 316-325, 2014.
  * [34] R. A. Winter, “Vertical control and price versus nonprice competition”, _The Quarterly Journal of Economics_ , vol. 108, no. 1, pp. 61-75, 1993.
  * [35] A. M. Winter, “Social services: effective practices in serving survivors of torture”, _Quarterly Journal on Rehabilitation of Torture Victims and Prevention of Torture_ , vol. 21, no. 1, pp. 48-55, 2011.
  * [36] A. McLean Winter, “The Laughing Doves of 1812 and the Satiric Endowment of Antiwar Rhetoric in the United States”, _Publications of the Modern Language Association of America_ , vol. 124, no. 5, pp. 1562-1581, 2009.
  * [37] A. F. D. Winter, _Justiz_ , Diogenes Verlag, Zürich, 1985.
  * [38] A. Winter Noble, “Em Tempos de Reformas, a Alfabética”, _Paralelo 31_ , vol. 2, no. 13, pp. 214-232, 2019.
  * [39] A. Winterer, _Verkehrte Sexualität – ein umstrittenes Pauluswort_ , doctoral dissertation, European University Studies, vol. 810, Frankfurt am Main, 2005.
  * [40] A. Winterest, “Sempre più bello”, _Bella Figura_ , vol. 1, no. 1, pp. 10-16, 2019.

Appendix — Remarks on Conjecture 2 (by A. Winter). This was to be something
profound reflecting my complex and nuanced opinions about the provocative work
of A. Winter _et al._ , but the present paragraph is too short to contain it.
For


# Multiple Attribute Fairness: Application to Fraud Detection

Meghanath M Y  AdobeSan JoseUSA<EMAIL_ADDRESS>,  Sriram Ravindran
AdobeSan JoseUSA<EMAIL_ADDRESS>,  Deepak Pai  AdobeSan JoseUSA
<EMAIL_ADDRESS>,  Anish Narang  AdobeSan JoseUSA<EMAIL_ADDRESS>and  Vijay
Srivastava  AdobeSan JoseUSA<EMAIL_ADDRESS>
(2022)

###### Abstract.

We propose a fairness measure relaxing the equality conditions in the popular
equal odds fairness regime (Hardt et al., 2016) for classification. We design
an iterative, model-agnostic, grid-based heuristic that calibrates the
outcomes per sensitive attribute value to conform to the measure. The
heuristic is designed to handle high arity attribute values and performs a per
attribute sanitization of outcomes across different protected attribute
values. We also extend our heuristic for multiple attributes. Highlighting our
motivating application, fraud detection, we show that the proposed heuristic
is able to achieve fairness across multiple values of a single protected
attribute, multiple protected attributes. When compared to current fairness
techniques, that focus on two groups, we achieve comparable performance across
several public data sets.

††journalyear: 2022††copyright: acmlicensed††conference: KDD ’22: ACM SIGKDD
International Conference on Knowledge Discovery & Data Mining; August 14–18,
2022; Washington, D.C. ††price: 15.00††doi: 10.1145/1122445.1122456††isbn:
978-1-4503-9999-9/18/06††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06 Figure 1.
Fraud Detection : False Positive Rates Pre (Left) and Post (Right)
Sanitization.

##  1\. Introduction

Machine learning algorithms are increasingly being used to make decisions in
critical areas such as access to credit, employment opportunities and
education. Recent research has highlighted concerns related to potential bias,
where ML techniques produce discriminatory outcomes (Chouldechova, 2017). For
instance, when Apple released its credit card, there were claims that women
were given a lower credit limit. Similarly, ProPublica analyzed a risk
assessment used by judges in the United States to predict recidivism risk for
an accused and concluded that the predictions are biased against black
defendants . From an industry standpoint, it is critical to sanitize ML
algorithms, ensure fairness and avoid ethical and legal concerns.

Detecting and mitigating bias in ML algorithms has gained traction in the last
few years both in academia and industry. However, existing approaches have
certain limitations, primarily around accounting for ground truth
distributions as well as type-1 and type-2 errors. In this paper, we use
online transaction fraud detection as a motivating example to discuss the
drawbacks of the current approach and to illustrate the value of the proposed
approach.

Motivating Application : As businesses continue to shift toward online
payments, there is a rising need to have an effective fraud detection solution
capable of real-time, actionable alerts and insights. This increasing need has
led to several solutions being offered by industry leaders (Gartner, 2021)
such as Amazon (AWS, 2021), Microsoft (Microsoft, 2021), ThreatMetrix (Nexis,
2021), Forter (Forter, 2021). These solutions can analyze hundreds of
parameters of an online transaction such as transaction amount, past
transaction trends, IP location of the transaction, transaction time, merchant
name etc. and output a score in real-time (usually from 0 to 100) indicating
the fraudulent nature of the transaction. While these solutions claim a high
predictive performance, biases in the training data and model inaccuracies can
lead to decisions that treat individuals unfavorably (unfairly) based on
characteristics such as transaction country and currency (sensitive
attributes). Ensuring that these biases are curtailed before deploying such
solutions is critical for the business.

In this work, we propose a fairness measure for classification and detail a
heuristic based on model calibration to ensure such fairness in the outcomes.
We showcase the efficacy of the proposed approach on several publicly
available data sets and a proprietary commercial data set that is comprised of
online transactions (Figure 1). For baselines, we consider Equalized Odds
(Hardt et al., 2016) and Calibrated Equalized Odds (Pleiss et al., 2017).

##  2\. Related Work

Anti-discrimination laws in the United States regulate two discriminatory
behaviors based on protected attributes. These doctrines of discrimination are
disparate treatment and disparate impact. Disparate treatment protects
individuals who are affected by algorithmic decision making against explicit
discrimination. The standard practice to conform to disparate treatment in
Machine learning techniques used for decision making is to exclude protected
attributes from inputs. This is easy to achieve in a fraud detection model –
one would just need to not use any sensitive attribute while training the
model. In contrast, disparate impact addresses outcome discrimination; it
recognizes liability for practices with uneven impacts on different protected
groups. While there is no single fairness measure that captures absolute
impact parity, equal opportunity, demographic parity, and equalized odds
(Beutel et al., 2017; Hardt et al., 2016; Chzhen et al., 2019; Pleiss et al.,
2017) have received considerable attention. Efforts from scholars revolve
around heuristics to ensure these fairness measures. While these techniques
are catered to ensure fairness for any Machine learning technique, they are
not always practical to implement in an industry setting. To expound, consider
demographic parity, which requires that the outcome rates across sensitive
attributes be the same. In the context of fraud detection, for a sensitive
attribute “country”, this would mean that acceptance or rejection rates of
online transactions for each country should be the same. This may not be ideal
since we observe that the ground truth fraud rates from different countries
are significantly different (Refer to Figure 2). This could possibly be
explained by the jurisdiction in each of these countries in handling online
fraud (stricter penalties would mean lower fraud rates and vice versa). If we
were to sanitize our fraud detection model to ensure demographic parity, one
would either end up allowing lots of fraudsters or declining lots of genuine
users both of which are not an ideal outcome for a firm.

Figure 2. Ground Truth Fraud Rates across Countries.

Another approach, equalized odds, aims to achieve the same false positive and
true positive rates across sensitive attributes. While this is better than the
demographic parity for fraud detection, the measure enforces equal importance
to both false positives and false negatives. This is again not ideal for a
firm given the cost of type-1 and type-2 errors are different for
electronic111The cost of declining a genuine customer is much higher than
allowing a fraudster. vs physical goods. Lastly, Equalized opportunity, is a
relaxed version of Equalized odds that only requires equal true positive rates
across sensitive attributes. This again is insufficient since we would not
want to deny a genuine customer and hence would like to balance false positive
rates as well. Furthermore, while getting to equal rates is possible in
theory, it is less practical with daily fluctuations in traffic and fraud
patterns. Additionally, from an implementation standpoint, another major
critique about the current techniques is that they require multiple
sanitizations of the ML model to achieve fairness. This largely arises since
majority of these techniques assume binary sensitive attributes. Such
constraints are not viable in a fraud detection system that usually deal with
high arity attributes and require real time inference. Further, to the best of
our knowledge, none of these methods perform sanitization of a machine
learning model in a post-hoc fashion across multiple protected attributes. We
tackle these drawbacks in our work by introducing a relaxed Equalized odds
fairness measure and a one-shot fairness heuristic to achieve the proposed
fairness measure. Further, we design the proposed heuristic to have the
flexibility to pick either or both constraints of the Equalized Odds
definition (similar FPR or similar TPR or both) (Hardt et al., 2016). Finally,
and importantly, we also extend the proposed fairness measure and heuristic to
handle multiple attributes.

##  3\. Method

The classical definition of Equality of Odds posits that true positive rates
and false positive rates across a protected attribute are the same. It
reflects a fundamental idea of fairness, that qualified individuals should be
given equal opportunity (true positive rate - TPR) to access a desirable
outcome while also requiring equal false positive rates (FPR, ensuring no bias
to the positive class) regardless of their demographics or any other sensitive
attributes. However, this strict requirement of equality in most settings
requires non-deterministic thresholds as pointed out in previous research
(Beutel et al., 2017; Hardt et al., 2016; Chzhen et al., 2019; Pleiss et al.,
2017) making it less appealing from a practical standpoint. Further, the
current heuristics to achieve such fairness involves multiple sanitizations –
one per each categorical attribute value (Ex: one for each country) of a
machine learning model to ensure the fairness criterion is met. We deal with
these issues by first, relaxing the notion of the standard Equalized odds.
Specifically, we relax the equality constraint of FPR, TPR and enforce a
weaker, yet practical constraint. Second, we propose a fairness heuristic that
sanitizes the outputs of a classification model to conform to the relaxed
equalized odds measure.
Note: We describe the details assuming a single attribute and later extend
this to multiple attributes in Section 3.3.

###  3.1. Relaxed Equalized Odds

Say $D=\\{d_{1},d_{2},…,d_{K}\\}$ is a protected attribute with arity $K$, we
say that a fraud detector model $F$222In general, $F$ can also be any
classification model that needs to be sanitized for bias. satisfies relaxed
equalized odds with respect to attribute $D$ if the false positive and true
positive rates of the respective attribute values
$fpr^{F}=\\{fpr_{1}^{F},fpr_{2}^{F},…,fpr_{K}^{F}\\}$,
$tpr^{F}=\\{tpr_{1}^{F},tpr_{2}^{F},…,tpr_{K}^{F}\\}$ lie within $n$ standard
deviations of their means i.e.,

(1) $\displaystyle\mu(fpr^{F})-n\sigma(fpr^{F})\leq
fpr_{i}^{F}\leq\mu(fpr^{F})+n\sigma(fpr^{F})$ (2)
$\displaystyle\mu(tpr^{F})-n\sigma(tpr^{F})\leq
tpr_{i}^{F}\leq\mu(tpr^{F})+n\sigma(tpr^{F})$

where $\mu$ and $\sigma$ are the average and standard deviation respectively
across the $K$ attribute values.

The above definition while weaker, captures the core philosophy of the
Equalized odds in a high arity($K$)333Note that mean and standard deviation
across the attributes would only hold a statistical meaning if the number of
attribute values are higher. This is traditionally the case in fraud detection
models where a sensitive attribute, say country or currency could take over
100 types of values.  setting – similar opportunity and similar false alarms
across a sensitive attribute.

###  3.2. Fairness Heuristic

Following a similar notation as above, say we would like to sanitize a fraud
detector $F$ with respect to the protected attribute $D$, our fairness
heuristic has the following key steps.

####  3.2.1. Choice of constraints

We provide an end user the capability to further relax the traditional
equalized odds definition to conform to either similar FPRs or TPRs or both
(i.e., Eq 1 or Eq 2 or both). This would be useful for a business where false
positives of a fraud detector are more costly compared to true positives. Said
differently, if it is costlier for a business to accept a fraudulent customer
compared to declining a genuine customer (false negative), then they might
choose to conform to similar FPRs, while also ensuring that the overall FPR is
low. In a similar way, a business might choose to have equal opportunity (TPR)
across a protected attribute. This choice decides the $selection_{metric}$ in
our heuristic. $selection_{metric}=F1$; if Eq (1) and Eq (2), $=F0.5;$, if
only Eq (1) and $=F2;$ if only Eq (2). The choice of the selection metric is
done based on the importance of the FPR and TPR to the end user. F1 treats
false positives and false negatives (true positives) equally while $F0.5$ and
$F2$ weight false positives and false negatives higher respectively.

####  3.2.2. Threshold grid initialization

The core idea of our heuristic is to calibrate the decision thresholds of the
model $F$ across the different attribute values D to conform to Eq (1) and/or
Eq (2). To do this, we initialize a linear grid of possible threshold values
denoted as $G_{thresh}$. A choice of $G_{thresh}$ could be
$\\{0.6,0.61,…,0.9\\}$. Note that depending on the threshold value, both FPR
and TPR of the model $F$ would change.

####  3.2.3. Performance computation

For all the values in $G_{thresh}$, per attribute value in $D$, we compute
model $F$’s performance metrics. These comprise of the metric that the end
user chooses to conform to in the fairness measure (say FPR in our running
example). At the end of this step, we would have
$fpr^{F_{g}}=\\{fpr_{1}^{F_{g}},fpr_{2}^{F_{g}},…,fpr_{K}^{F_{g}}\\}$
$\forall$ $g\in G_{thresh}$, where $fpr^{F_{g}}$ is the FPRs of the model $F$
at threshold $g$ across protected attribute values.

####  3.2.4. Pruning and Selection

Our next task is to iteratively prune the set of $\\{fpr^{F_{g}},g\in
G_{thresh}\\}$ to conform to Eq 1. and select custom thresholds from
$G_{thresh}$ for each attribute in $D$.
Prune : To do this, we compute $\mu(fpr^{F_{g}})$ and $\sigma(fpr^{F_{g}})$,
$\forall$ $g\in G_{thresh}$ and prune $fpr_{i}^{F_{g}}$ that are overly biased
compared to the mean, i.e., violate the constraint
$fpr_{i}^{F_{g}}\leq\mu(fpr^{F_{g}})+n\sigma(fpr^{F_{g}})$. This would mean
that for a protected attribute value $d_{i}$, $D=\\{d_{1},d_{2},…,d_{K}\\}$,
some of the threshold choices are pruned since their false positive rates are
statistically higher than the average. Denote such pruned set as
$\\{fpr^{F_{g}}\\}^{pruned}$.
Select : From $\\{fpr^{F_{g}}\\}^{pruned}$, we next select the threshold per
each attribute value $d_{i}$ where the $selection_{metric}$ (F-0.5 in our
running example) is maximized. This results in a choice of
$\\{(d_{1},g_{1}),...,(d_{K},g_{K})\\}$. Denote this choice of thresholds as
the model $F_{select}$
We check if $F_{select}$ conforms to the fairness constraints, i.e., Eq (1) in
our running example $\mu(fpr^{F_{select}})-n\sigma(fpr^{F_{select}})\leq
fpr_{i}^{F_{select}}\leq\mu(fpr^{F_{select}})+n\sigma({F_{select}})$. If not,
we repeat the pruning and selection step with the updated
$\\{fpr^{F_{g}}\\}^{pruned}$ until the constraint is satisfied. First, we note
that the final $F_{select}$ conforms to the fairness measure and preserves
model fidelity by selecting the best available model based on the selection
metric. Second, from Step 3 and Step 4, we note that our fairness heuristic
only requires aggregate information such as FPR, TPR, F-1 score about the
protected attribute $D$ and does not require information about the features
(protected or unprotected) or their feature values used to train the fraud
detector model. Hence, one could perform this sanitization in a differentially
private manner by adding appropriate noise to the aggregate information.
Third, the reliance of the output on solely the outcomes makes the fairness
heuristic model agnostic. Fourth, the proposed heuristic is a one-shot search
heuristic that conforms the model $F$ to the protected attribute $D$ rather
than performing $K$ sanitizations for each attribute of $D$. Finally, we note
that the grid heuristic discussed above is guaranteed to converge to a
solution if there is indeed a solution where the FPRs/TPRs are equal for the
sub-groups.

###  3.3. Multiple Attribute Extension

The fairness heuristic detailed above handles a single protected attribute and
conforms to equalized fairness in that attribute. However, for most practical
applications, we would require a machine learning model to conform to fairness
across multiple attributes. For instance, in the case of a fraud detection
model, one would like to sanitize the model to not be biased against a
country, currency that they use. The current state of the art fairness
heuristics deals with this issue while training (Foulds et al., 2020) or by
processing the features before learning (Zhang et al., 2021). Supplementing
this line of literature, we extend the proposed fairness heuristic which
performs sanitization post inference to handle multiple attributes.

A naive way to handle multiple attributes (consider two for example) say
$D^{1}=\\{d_{1}^{1},d_{2}^{1},…,d_{K^{1}}^{1}\\}$,
$D^{2}=\\{d_{1}^{2},d_{2}^{2},…,d_{K^{2}}^{2}\\}$ in our regime would be to
identify
$\\{((d_{1}^{1}d_{1}^{2}),g_{1}),...,(d_{K^{1}}^{1}d_{K^{2}}^{2}),g_{K^{1}\times
K^{2}})\\}$. Note that such calibration of thresholds would conform to a
strict notion of the Relaxed Equalized Odds ensuring the FPR and TPRs are
equal across every sub-population. While this is certainly an attractive
property to have, the computational complexity is exponential. A weaker notion
would be to ensure that the FPR and TPR values satisfy the constraints in Eq
(1) and (2) per attribute independently rather than all enumerations.

Expounding the two concepts with an example of two attributes country and
currency with attribute values US, IN and USD, INR respectively. In the
stronger notion, we want FPRs and TPRs across the eight sub populations
$\\{(US,USD),(US,INR),(IN,USD),\\\ (IN,INR),(US),(USD),(IN),(INR)\\}$ to
conform to Eq (1) and (2). In the weaker notion, we want them to conform to
four sub populations $\\{(US),(IN),(USD),(INR)\\}$. While the former is a
stronger fairness notion and scales exponentially with the number of
attributes, the latter is weaker and scales linearly. We propose heuristics
for both such notions of fairness giving the decision maker the flexibility to
opt to either of those depending on the number of attributes the model needs
to be sanitized on. We recommend the stronger notion for 10 or a smaller
number of attributes and the univariate, weaker notion for greater than that.
In both cases, we reduce this combinatorial computation by first pruning the
attribute space by identifying attributes that encompass similar sub
populations.

####  3.3.1. Attribute Pruning

If two attributes are dependent i.e., capture similar sub populations with
their attribute values (for instance country and currency), it is redundant to
calibrate the thresholds for both. We define dependence between attributes as
the Chi-square statistic between pairs of attributes. Based on the statistic,
we compute the $p$-value to infer if two attributes are statistically
independent of each other (p-value ¡= 0.01) and drop the attributes that are
highly dependent. This step prunes the space of protected attributes while
only leaving independent attributes to calibrate.

####  3.3.2. Strong Multiple Attribute Fairness : Attribute Value Pruning

In this regime, we want to ensure that the FPRs and TPRs of every sub-
population conform to Eq (1) and (2). While reducing the attribute space
partly addresses the computation issue discussed earlier, we are still left
with potentially multiple attributes with high arity. For instance, say we
have m protected attributes after pruning with cardinalities
$K_{1},K_{2},…,K_{m}$, we would be looking at possibly $K_{1}\texttimes
K_{2}\texttimes…\texttimes K_{m}$ subspaces to calibrate the thresholds on.
However, we note that to yield a reliable estimate of the performance metrics
fpr,tpr (Step 3 in the heuristic) or the selection metrics (Step 4.b.), as a
rule of thumb (Green, 1991), a subspace would at least need to have 100
transactions (data points). Hence, we prune away such subspaces using a
scalable implementation of the Frequent Pattern Tree (Han et al., 2000) data
structure. Let $C=\\{c_{1},c_{2},…,c_{P}\\}$ be the total number of subspaces
left with each at least having 100 transactions. An example of a $c_{1}$ would
be $\\{country=US,sex=Male,race=Hispanic\\}$. The resulting clusters $C$ are
then calibrated with the grid of thresholds to conform to the Relaxed
Equalized Odds fairness measure in a similar fashion to the single attribute
case where the performance table looks like $\\{fpr^{F_{g}}\\},g\in G$.

####  3.3.3. Weak Multiple Attribute Fairness

In this regime, we want to ensure that the FPRs and TPRs per attribute conform
to Eq (1) and Eq (2). Operationally, this can be achieved by changing the Step
3) in the fairness heuristic to compute the performance table across all
attributes and their attribute values. That is, for m protected attributes,
construct
$fpr^{F_{g}}=\\{fpr_{1}^{F_{g}},fpr_{2}^{F_{g}},…,fpr_{K_{1}}^{F_{g}},fpr_{1}^{F_{g}},fpr_{2}^{F_{g}},…,fpr_{K_{2}}^{F_{g}},\ldots\\\
fpr_{K_{1+K_{2}+\ldots K_{m}}}^{F_{g}}\\}$ $g\in G_{thresh}$, where
$fpr^{F_{g}}$ is the FPRs of the model $F$ at threshold g across all
univariate enumerations of the protected attribute values. This is followed
with the Pruning and Selection step as discussed earlier to find thresholds
per attribute value for all the m attributes.

##  4\. Experiments

Figure 3. Fraud Detection FPR (Pre = Red, Post = Green)

We showcase the efficacy of the proposed approach on several publicly
available data sets - criminal recidivism444Data source :
https://github.com/propublica/compas-analysis, income-prediction , health
prediction555Data source :
https://github.com/gpleiss/equalized_odds_and_calibration/, and a proprietary
commercial data set that is comprised of online transactions. For baselines,
we consider Equalized Odds (Hardt et al., 2016) and Calibrated Equalized Odds
(Pleiss et al., 2017). In the experiments, we set $n$ = 2 and investigate
single protected attribute with high arity, multiple protected attributes, and
also consider the case of 2 protected groups as investigated by earlier
studies.

Case Study : Fraud Detection We first discuss our motivating example, online
fraud detection666In all the experiments, positive refers to a fraudulent
transaction. Hence true positives would mean fraudulent transaction being
identified as fraud by the detector.. The aim of the prediction is to assess
whether an online transaction initiated by an individual is fraudulent. The
prediction model omits all protected attributes as input to conform to the
disparate treatment doctrine of fairness measure. However, as evident from 3
and 4 (red color bars) we observe that model exhibits a bias towards the
demographic attribute country. A similar trend is observed when we look at the
FNRs. As we motivate earlier, from a business perspective, it is useful to
have flexibility to sanitize these metrics across attributes

Figure 4. Fraud Detection FNR (Pre = Red, Post = Green)

We employ the proposed heuristic on the predictions made by the fraud detector
by conforming to both the constraints Eq (1) and Eq (2). From Figures 3, 4
(green color bars), we observe that the final $F_{select}$ with custom
thresholds per country considerably reduces the bias towards certain
countries. Further, we also note that the FPRs and FNRs across countries
conform to the relaxed equalized odds fairness measure – mean across the
countries lies withing two standard deviations.

Figure 5. Fraud Det. (XGBoost, Multiple attributes) FPR (Post)

Method —- Dataset | Proposed Heuristic | Equalized Odds | Calibrated Equalized Odds
---|---|---|---
| FPR
---
(Class 0)
| FPR
---
(Class 1)
| FNR
---
(Class 0)
| FNR
---
(Class 1)
| FPR
---
(Class 0)
| FPR
---
(Class 1)
| FNR
---
(Class 0)
| FNR
---
(Class 1)
| FPR
---
(Class 0)
| FPR
---
(Class 1)
| FNR
---
(Class 0)
| FNR
---
(Class 1)
| Online Fraud
---
(Country = 6)
0.001 | 0.0009 | 0.349 | 0.246 | 0.01 | 0.009 | 0.306 | 0.282 | 0.009 | 0.01 | 0.449 | 0.124
| Online Fraud
---
(Country = 22)
0.0007 | 0.0008 | 0.202 | 0.13 | 0.019 | 0.021 | 0.174 | 0.113 | 0.012 | 0.021 | 0.335 | 0.104
| Income
---
(AfricanAmerican)
0.42 | 0.351 | 0.61 | 0.77 | 0.13 | 0.12 | 0.463 | 0.348 | 0.884 | 0.345 | 0.642 | 0.705
| COMPAS
---
(AfricanAmerican)
0.387 | 0.401 | 0.521 | 0.506 | 0.435 | 0.430 | 0.492 | 0.491 | 0.315 | 0.449 | 0.561 | 0.372

Table 1. Comparison to Equalized and Calibrated Equalized Odds.

Next, we consider the exercise on the fraud detector model trained with
XGBoost to sanitize across with two protected attributes country and currency.
From Figure 5, we observe that we can achieve similar FPRs across the
attribute values of currency. From a separate experiment (not presented for
brevity), we also observe that sanitizing the model on currency inherently
sanitizes it against country given their high dependence.

Model benchmarks: Next, we compare the proposed heuristic to Equalized (Hardt
et al., 2016) and Calibrated Equalized Odds (Pleiss et al., 2017). These
techniques are aimed at achieving fairness across two groups (E.g. Country
is/not 1), unlike the proposed heuristic which aims to achieve fairness across
all sub groups with custom thresholds per country. In Table 1, rows 1 and 2,
we report FPR and FNR for Country is/not is Country 6777We pick the highest
(Country 6) and lowest frequency (Country 22) country based on the
transactions in our data set. Each country has at least 10,000 transactions
each. (Class 0 and Class 1). and Country is/not is Country 22 and observe that
we are able to achieve similar metrics compared to the benchmarks.

Income-Prediction This dataset from UCI Machine Learning Repository contains
14 demographic and occupational features for various people, with the goal of
predicting whether a person’s income is above $50, 000. In this scenario, we
seek to achieve predictions with equalized cost across genders (single
protected attributes, two groups - Male and Female). In this dataset, we
consider a scenario where the primary concern is ensuring equal generalized
false negative rates across genders, which would help job recruiters prevent
gender discrimination in salary estimates. Hence, we choose our fairness
constraint to require relaxed equalized false negative rates across groups. In
row 3, Table 1, we observe that the proposed heuristic achieves similar FNRs
for the two groups – 0.77 (African American) and 0.61 (not African American).
Also, note that we are able to achieve comparable (similar FNRs) or better
performance (better FPRs) compared to Calibrated Equalized Odds, designed
specifically to enforce equal FNRs across the two groups. Equalized Odds on
the other hand, ensures equal FPRs and FNRs across the two groups, trading off
the FPR for a lower FNR across groups (0.46 and 0.34) compared to the proposed
heuristic (0.61 and 0.77).

Criminal Recidivism Finally, we examine the proposed heuristic in t he context
of criminal recidivism. As noted by several studies earlier, in this dataset,
African Americans, receive a disproportionate number of false positive
predictions as compared with Caucasians when automated risk tools were used.
Hence, we aim to equalize the generalized false positive rate. In row 4, Table
1, we observe that the proposed heuristic achieve a) similar FPRs for the two
groups – 0.4 (African American) and 0.38 (not African American) b) similar
FNRs (0.60 and 0.59 respectively). Note that we are able to achieve comparable
(similar FNRs) and slightly better better FPRs compared to Equalized Odds,
designed to enforce equal FNRs and FPRs across the two groups. When compared
to the Calibrated Equalized Odds, where the weighted combination of error
rates are matched (equal FPRs and FNRs as discussed in (Pleiss et al., 2017)),
we observe that the proposed heuristic performs comparably in terms of both
FPRs and FNRs across two groups.

##  5\. Conclusion

We propose a fairness measure relaxing the FPR and TPR equality conditions in
the popular equal odds fairness regime (Hardt et al., 2016). To conform to the
proposed fairness, we design an iterative, model-agnostic, grid-based
heuristic that calibrates the outcomes per sensitive attribute value (for
e.g., different countries). Through a detailed case study of our motivating
application, fraud detection, we show that the proposed heuristic is able to
achieve fairness across multiple values of a single protected attribute,
multiple protected attributes. We compare our work to current fairness
techniques and show comparable performance across several public data sets.

## References

  * (1)
  * AWS (2021) Amazon AWS. 2021\.  Amazon Fraud Detector.  https://aws.amazon.com/fraud-detector/.
  * Beutel et al. (2017) Alex Beutel, Jilin Chen, Zhe Zhao, and Ed H Chi. 2017\.  Data decisions and theoretical implications when adversarially learning fair representations.  _arXiv preprint arXiv:1707.00075_ (2017).
  * Chouldechova (2017) Alexandra Chouldechova. 2017\.  Fair prediction with disparate impact: A study of bias in recidivism prediction instruments.  _Big data_ 5, 2 (2017), 153–163.
  * Chzhen et al. (2019) Evgenii Chzhen, Christophe Denis, Mohamed Hebiri, Luca Oneto, and Massimiliano Pontil. 2019.  Leveraging labeled and unlabeled data for consistent fair binary classification.  _Advances in Neural Information Processing Systems_ 32 (2019).
  * Forter (2021) Forter. 2021.  Maximize revenue with smarter fraud decisions.  https://www.forter.com/payment-protection/.
  * Foulds et al. (2020) James R Foulds, Rashidul Islam, Kamrun Naher Keya, and Shimei Pan. 2020.  An intersectional definition of fairness. In _2020 IEEE 36th International Conference on Data Engineering (ICDE)_. IEEE, 1918–1921.
  * Gartner (2021) Gartner. 2021.  Products in Online Fraud Detection (OFD) Market.  https://www.gartner.com/reviews/market/online-fraud-detection-systems.
  * Green (1991) Samuel B Green. 1991\.  How many subjects does it take to do a regression analysis.  _Multivariate behavioral research_ 26, 3 (1991), 499–510.
  * Han et al. (2000) Jiawei Han, Jian Pei, and Yiwen Yin. 2000.  Mining frequent patterns without candidate generation.  _ACM sigmod record_ 29, 2 (2000), 1–12.
  * Hardt et al. (2016) Moritz Hardt, Eric Price, and Nati Srebro. 2016.  Equality of opportunity in supervised learning.  _Advances in neural information processing systems_ 29 (2016).
  * Microsoft (2021) Azure Microsoft. 2021\.  Real-time fraud detection.  https://docs.microsoft.com/en-us/azure/architecture/example-scenario/data/fraud-detection.
  * Nexis (2021) Lexis Nexis. 2021\.  Comprehensive Fraud Prevention and Friction-less Customer Experience.  https://risk.lexisnexis.com/products/threatmetrix.
  * Pleiss et al. (2017) Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q Weinberger. 2017.  On fairness and calibration.  _Advances in neural information processing systems_ 30 (2017).
  * Zhang et al. (2021) Hantian Zhang, Xu Chu, Abolfazl Asudeh, and Shamkant B Navathe. 2021.  OmniFair: A Declarative System for Model-Agnostic Group Fairness in Machine Learning. In _Proceedings of the 2021 International Conference on Management of Data_. 2076–2088.