#  SurvSet: An open-source time-to-event dataset repository

Erik Drysdale<EMAIL_ADDRESS>
###### Abstract

Time-to-event (T2E) analysis is a branch of statistics that models the
duration of time it takes for an event to occur. Such events can include
outcomes like death, unemployment, or product failure. Most modern machine
learning (ML) algorithms, like decision trees and kernel methods, are
supported for T2E modelling with data science software (python and R). To
complement these developments, SurvSet is the first open-source T2E dataset
repository designed for a rapid benchmarking of ML algorithms and statistical
methods. The data in SurvSet have been consistently formatted so that a single
preprocessing method will work for all datasets. SurvSet currently has 76
datasets which vary in dimensionality, time dependency, and background (the
majority of which come from biomedicine). SurvSet is available on PyPI and can
be installed with pip install SurvSet. R users can download the data directly
from the corresponding git repository.

##  1 Introduction

Many disciplines study phenomena in which a certain amount of time must pass
before an event occurs. In statistics, this is referred to as time-to-event
(T2E) analysis. Such processes are usually characterized by two properties: i)
non-negative values, and ii) right-censoring. The latter refers to an
observation where the event has not yet occurred. Implicit in T2E processes is
that with sufficient measurement time, an event will inevitably occur. T2E
analysis frequently arises in fields like biomedicine (death or relapse),
economics (unemployment), e-commerce (customer churn), and engineering
(product failure). T2E methods are designed to account for both right-
censoring and non-negative values.

Due to the preponderance of biomedical research which studies end-points like
death, T2E methods are often referred to as survival analysis. Both terms will
be used interchangeably throughout this paper. Classical survival analysis is
a well-developed field with mature computational resources. More recently,
core machine learning (ML) algorithms have been adapted for T2E modelling,
with some examples shown below. For a comprehensive survey of survival methods
in ML see [1].

  * •

Regularized linear models (e.g. elastic net [2])

  * •

Decision trees [3]

  * •

Ensemble methods (e.g. random forests [4] and gradient boosters [5])

  * •

Kernel methods (e.g. SVMs [6])

  * •

Deep learning (e.g. feed-forward neural networks [7])

For most researchers and practitioners, the actual implementation of survival
methods (both classical and ML-based) is done with special packages in R and
python. The former includes packages like survival [8] and mlr3proba [9] while
the latter includes lifelines [10], pysurvival [11], and scikit-survival [12].
These packages will often have a handful of datasets to allow for testing and
benchmarking.

SurvSet seeks to complement these modelling packages by providing the first-
ever open-source T2E dataset repository for the benchmarking and evaluation of
ML algorithms and statistical methods. Benchmarking and contests have played
an important role in the development and refinement of ML algorithms [13].
SurvSet is comprised of 76 datasets that are structured in a consistent
format. This enables a single preprocessing method that works for all
datasets, and removes the need for researchers to spend time curating and
formatting datasets. The datasets also vary in dimensionality, background, and
whether the covariates vary across time.111The origins of this dataset were
for testing regularity conditions of the False Positive Control Lasso [14].

##  2 Overview & design

###  2.1 Usage

SurvSet is built around a single class, SurvLoader, which loads datasets
through the load_dataset method. This method is a convenient wrapper for
loading the underlying comma-separated files. As was previously mentioned,
these files can be accessed directly through github. The code block in 1
provides an example of how to load a dataset with SurvSet. The full list of
available datasets can be seen in the attribute df_ds (see Table
LABEL:tab:df_ds).

The load_dataset method returns a dictionary with keys df for the underlying
DataFrame (see Table LABEL:tab:df_ex) and a URL with a further description on
the columns.222See, for example, the ova reference. The columns of each
dataset are consistently structured. The observation ID, event, and survival
times are placed in the first three (or four) columns. The remaining feature
columns have a prefix to indicate whether they are categorical and can be one-
hot-encoded, or are numeric and can be normalized. The column order and names
are indicated below.

  1. 1.

pid: the unique observation identifier (relevant for datasets with time-
varying features)

  2. 2.

event: a binary event indicator (1=event has happened)

  3. 3.

time: time to event/censoring (or start time if time2 exists)

  4. 4.

time2: end time $[\texttt{time},\texttt{time2})$ if there are time-varying
features (non-existent otherwise)

  5. 5.

num_{}: prefix implies a continuous feature

  6. 6.

fac_{}: prefix implies a categorical feature

Listing 1: Example of loading data

⬇

1from SurvSet.data import SurvLoader

2loader = SurvLoader()

3# List of available datasets and meta-info

4print(loader.df_ds.head())

5# Load dataset and its reference

6df, ref = loader.load_dataset(ds_name=‘ova‘).values()

7print(df.head())

Table 1: Example of dataset list: df_ds ds_name | is_td | n | n_fac | n_ohe | n_num
---|---|---|---|---|---
hdfail | False | 52422 | 5 | 87 | 1
stagec | False | 146 | 4 | 15 | 3
veteran | False | 137 | 3 | 5 | 3
vdv | False | 78 | 0 | 0 | 4705
AML_Bull | False | 116 | 0 | 0 | 6283
Table 2: Example of ova dataset: df pid | event | time | num_karn | num_diam | fac_karn | fac_diam | fac_figo
---|---|---|---|---|---|---|---
1 | 1 | 7 | 1 | 3 | 1 | 3 | 1
2 | 1 | 8 | 2 | 2 | 2 | 2 | 1
3 | 1 | 9 | 4 | 4 | 4 | 4 | 1
4 | 1 | 10 | 4 | 4 | 4 | 4 | 1
5 | 1 | 13 | 2 | 4 | 2 | 4 | 1

###  2.2 Datasets

SurvSet currently has 76 datasets which vary in dimensionality (see Figure 1).
This includes high-dimensional genomics datasets ($p\gg n$) like gse1992, and
long and skinny datasets like hdfail ($n\gg p$). Most of these datasets come
from existing R packages, although not exclusively (see Table
LABEL:tab:ref_table). An initial empirical experiment suggests that fitting a
(regularized) linear model to each dataset produces a roughly uniform
distribution of concordance results (see Figure 2).

The construction of these datasets was necessarily subjective as decisions
needed to be made about which columns were relevant and which columns needed
to dropped (especially if they had information leakage). For the datasets with
time-varying features, time intervals were aggregated so that the minimum
number of rows were needed to capture all information about changes in
features. For example, if a measurement was made every 10 minutes, but the
first change of a feature occured at the 50 minute mark, then the first
interval would span [0,50) minutes. Users who are interested in how each
dataset was curated can explore the processing files in the _datagen folder
(which otherwise should not be used).

Figure 1: SurvSet datasets and dimensionality

##  3 Conclusion

SurvSet is a powerful resource to enable improvements for ML algorithms in T2E
analysis. Currently, novel algorithms seen in the literature are only ever
benchmarked against a handful of datasets. Furthermore, these datasets will
vary by project making it difficult to assess what constitutes state-of-the-
art performance. The hope is that SurvSet will become a well-known resource
for the ML community that focuses on survival analysis. This package can
easily be expanded with new datasets and interested parties are encouraged to
provide suggestions and contribute.

Figure 2: Concordance of a regularized linear model on SurvSet

Based on random 30% test set; uses CoxnetSurvivalAnalysis from scikit-survival
with default settings; concordance is equivalent to c-index [15]

Table 3: Dataset origin  Dataset | Package | Data source | Package
---|---|---|---
MCLcleaned | AdapEnetClass | [16] | [17]
rott2 | AF | [18] | [19]
hepatoCellular | asaur | [20] | [21]
pharmacoSmoking | asaur | [22] |
prostateSurvival | asaur | [23] |
chop | bujar | [24] | [25]
glioma | coin | [26] | [27]
ova | CoxRidge | [28, 29] | [30]
breast | coxphf | [31, 32] | [33]
UnempDur | ecdat | [34] | [35]
Unemployment | ecdat | [36] |
scania | eha | [37] | [38]
oldmort | eha | [39] |
hdfail | frailtySurv | [40] | [41]
uis | Hosmer | [42] | [42]
FRTCS | Hosmer | [42] |
smarto | hdnom | [43] | [44]
burn | iBST | [45] | [46]
d.oropha.rec | invGauss | [47] | [48]
aids | JM | [49] | [50]
heartvalve | joineR | [51] | [52]
epileptic | joineR | [53] |
dataOvarian1 | joint.Cox | [54] | [55]
aids2 | MASS | [56] | [57]
melanoma | MASS | [58, 59] |
grace | mlr3proba | [60, 42] | [61]
actg | mlr3proba | [62, 42] |
phpl04K8a | openml | [63] | [64]
zinc | NestedCohort | [65] | [66]
Pbc3 | pec | [67] | [68]
cost | pec | [69] |
GBSG2 | pec | [70] |
nki70 | penalized | [71] | [72]
micro.censure | plsRcox | [73] | [74]
divorce | princeton | [75] | [76]
follic | randomForestSRC | [77] | [78]
vdv | randomForestSRC | [79] |
Bergamaschi | RCASPAR | [80] | [81]
Dialysis | RcmdrPlugin.survival | [82] | [83]
Rossi | RcmdrPlugin.survival | [84] |
rdata | relsurv | [85] | [86]
NSBCD | Reddy | [87] | [88]
AML Bull | Reddy | [89] |
DBCD | Reddy | [90] |
DLBCL | Reddy | [91] |
DIVAT1 | RISCA | [92] | [93]
DIVAT2 | RISCA | [92, 94] |
DIVAT3 | RISCA | [92, 95] |
Z243 | RobustAFT | [96] | [97]
stagec | rpart | [98] | [99]
e1684 | smcure | [100] | [101]
whas500 | smoothHR | [42] | [102]
LeukSurv | spBayesSurv | [103] | [104]
cancer | survival | [105] | [106]
cgd | survival | [107, 108] |
colon | survival | [109] |
flchain | survival | [110, 111] |
heart | survival | [112] |
mgus | survival | [113] |
ovarian | survival | [114] |
pbc | survival | [115] |
retinopathy | survival | [116] |
veteran | survival | [117] |
nwtco | survival | [118] |
GSE4335 | survJamda.data | [119, 120] | [121]
GSE3143 | survJamda.data | [122] |
GSE1992 | survJamda.data | [123] |
wpbc | TH.data | [124] | [125]
TRACE | timereg | [126] | [127]
csl | timereg | [128] |
diabetes | timereg | [129] |
support2 | vanderbilt | [130] | [131]
prostate | vanderbilt | [132] |
Framingham | vanderbilt | [133] |
rhc | vanderbilt | [134] |
acath | vanderbilt | [135] |
wlbw | vanderbilt | [136] |

## References

  * [1] Ping Wang, Yan Li and Chandan K Reddy  “Machine learning for survival analysis: A survey”  In _ACM Computing Surveys (CSUR)_ 51.6 ACM New York, NY, USA, 2019, pp. 1–36
  * [2] Noah Simon, Jerome Friedman, Trevor Hastie and Rob Tibshirani  “Regularization paths for Cox’s proportional hazards model via coordinate descent”  In _Journal of statistical software_ 39.5 NIH Public Access, 2011, pp. 1
  * [3] A Ciampi, C-H Chang, S Hogg and S McKinney  “Recursive partition: A versatile method for exploratory-data analysis in biostatistics”  In _Biostatistics_ Springer, 1987, pp. 23–50
  * [4] Hemant Ishwaran, Udaya B Kogalur, Eugene H Blackstone and Michael S Lauer  “Random survival forests”  In _The annals of applied statistics_ 2.3 Institute of Mathematical Statistics, 2008, pp. 841–860
  * [5] Torsten Hothorn et al.  “Survival ensembles”  In _Biostatistics_ 7.3 Oxford University Press, 2006, pp. 355–373
  * [6] Vanya Van Belle, Kristiaan Pelckmans, Sabine Van Huffel and Johan AK Suykens  “Support vector methods for survival analysis: a comparison between ranking and regression approaches”  In _Artificial intelligence in medicine_ 53.2 Elsevier, 2011, pp. 107–118
  * [7] Håvard Kvamme, Ørnulf Borgan and Ida Scheel  “Time-to-event prediction with neural networks and Cox regression”  In _arXiv preprint arXiv:1907.00825_ , 2019
  * [8] Terry Therneau  “A package for survival analysis in S”  In _R package version_ 2.7, 2015
  * [9] Raphael Sonabend et al.  “mlr3proba: An R package for machine learning in survival analysis”  In _Bioinformatics_ 37.17 Oxford University Press, 2021, pp. 2789–2791
  * [10] Cameron Davidson-Pilon  “lifelines: survival analysis in Python”  In _Journal of Open Source Software_ 4.40, 2019, pp. 1317
  * [11] Stephane Fotso  “PySurvival: Open source package for Survival Analysis modeling”, 2019  URL: https://www.pysurvival.io/
  * [12] Sebastian Pölsterl  “scikit-survival: A Library for Time-to-Event Analysis Built on Top of scikit-learn.”  In _J. Mach. Learn. Res._ 21.212, 2020, pp. 1–6
  * [13] Rich Caruana and Alexandru Niculescu-Mizil  “An empirical comparison of supervised learning algorithms”  In _Proceedings of the 23rd international conference on Machine learning_ , 2006, pp. 161–168
  * [14] Erik Drysdale et al.  “The false positive control lasso”  In _arXiv preprint arXiv:1903.12584_ , 2019
  * [15] Frank E Harrell et al.  “Evaluating the yield of medical tests”  In _Jama_ 247.18 American Medical Association, 1982, pp. 2543–2546
  * [16] Andreas Rosenwald et al.  “The proliferation gene expression signature is a quantitative integrator of oncogenic events that predicts survival in mantle cell lymphoma”  In _Cancer cell_ 3.2 Elsevier, 2003, pp. 185–197
  * [17] Hasinur Rahaman Khan and Ewart Shaw  “AdapEnetClass: A Class of Adaptive Elastic Net Methods for Censored Data” R package version 1.2, 2015  URL: https://cran.r-project.org/package=AdapEnetClass
  * [18] Patrick Royston and Paul C. Lambert  “Flexible Parametric Survival Analysis Using Stata: Beyond the Cox Model”  StataCorp LP, 2011  URL: https://EconPapers.repec.org/RePEc:tsj:spbook:fpsaus
  * [19] Elisabeth Dahlqwist and Arvid Sjolander  “AF: Model-Based Estimation of Confounder-Adjusted Attributable Fractions” R package version 0.1.5, 2019  URL: https://cran.r-project.org/package=AF
  * [20] Li Li et al.  “CXCL17 expression predicts poor prognosis and correlates with adverse immune infiltration in hepatocellular carcinoma”  In _PloS one_ 9.10 Public Library of Science San Francisco, USA, 2014, pp. e110064
  * [21] Dirk F. Moore  “asaur: Data Sets for ”Applied Survival Analysis Using R”” R package version 0.50, 2016  URL: https://cran.r-project.org/package=asaur
  * [22] Michael B Steinberg et al.  “Triple-combination pharmacotherapy for medically ill smokers: a randomized trial”  In _Annals of internal medicine_ 150.7 American College of Physicians, 2009, pp. 447–454
  * [23] Grace L Lu-Yao et al.  “Outcomes of localized prostate cancer following conservative management”  In _Jama_ 302.11 American Medical Association, 2009, pp. 1202–1209
  * [24] G Lenz et al.  “Stromal gene signatures in large-B-cell lymphomas”  In _New England Journal of Medicine_ 359.22 Mass Medical Soc, 2008, pp. 2313–2323
  * [25] Zhu Wang and C.Y. Wang  “bujar: Buckley-James Regression for Survival Data with High-Dimensional Covariates” R package version 0.2-9, 2020  URL: https://cran.r-project.org/package=bujar
  * [26] C Grana et al.  “Pretargeted adjuvant radioimmunotherapy with yttrium-90-biotin in malignant glioma patients: a pilot study”  In _British journal of cancer_ 86.2 Nature Publishing Group, 2002, pp. 207–212
  * [27] Torsten Hothorn et al.  “coin: Conditional Inference Procedures in a Permutation Test Framework” R package version 1.42, 2021  URL: https://cran.r-project.org/package=coin
  * [28] JC Van Houwelingen et al.  “Predictability of the survival of patients with advanced ovarian cancer.”  In _Journal of Clinical Oncology_ 7.6, 1989, pp. 769–773
  * [29] Pierre JM Verweij and Hans C Van Houwelingen  “Cross-validation in survival analysis”  In _Statistics in medicine_ 12.24 Wiley Online Library, 1993, pp. 2305–2314
  * [30] Aris Perperoglou  “CoxRidge: Cox Models with Dynamic Ridge Penalties” R package version 0.9.2  URL: https://cran.r-project.org/package=CoxRidge
  * [31] A Lösch et al.  “Prognostic value of cathepsin D expression and association with histomorphological subtypes in breast cancer”  In _British journal of cancer_ 78.2 Nature Publishing Group, 1998, pp. 205–209
  * [32] Georg Heinze and Michael Schemper  “A solution to the problem of monotone likelihood in Cox regression”  In _Biometrics_ 57.1 Wiley Online Library, 2001, pp. 114–119
  * [33] Georg Heinze, Meinhard Ploner and Lena Jiricka  “coxphf: Cox Regression with Firth’s Penalized Likelihood” R package version 1.13.1, 2020  URL: https://cran.r-project.org/package=coxphf
  * [34] Brian P McCall  “Unemployment insurance rules, joblessness, and part-time work”  In _Econometrica: Journal of the Econometric Society_ JSTOR, 1996, pp. 647–682
  * [35] Yves Croissant and Spencer Graves  “Ecdat: Data Sets for Econometrics” R package version 0.3-9, 2020  URL: https://cran.r-project.org/package=ecdat
  * [36] Charles J Romeo  “Conducting inference in semiparametric duration models under inequality restrictions on the shape of the hazard implied by job search theory”  In _Journal of Applied Econometrics_ 14.6 Wiley Online Library, 1999, pp. 587–605
  * [37] Martin Dribe and Luciana Quaranta  “The Scanian Economic-Demographic Database (SEDD)”  In _Historical Life Course Studies_ , 2020
  * [38] Göran Broström and Jianming Jin  “eha: Event History Analysis” R package version 2.9.0, 2021  URL: https://cran.r-project.org/package=eha
  * [39] Sören Edvinsson  “The Demographic Data Base at Umeå University—a resource for historical studies”  In _Handbook of international historical microdata for population research_ Minnesota Population Center Minnesota, 2000, pp. 231–248
  * [40] “Hard drivedata and stats”  In _Backblaze Hard Drive Stats_ URL: https://www.backblaze.com/b2/hard-drive-test-data.html
  * [41] Vinnie Monaco, Malka Gorfine and Li Hsu  “frailtySurv: General Semiparametric Shared Frailty Model” R package version 1.3.7, 2021  URL: https://cran.r-project.org/package=frailtySurv
  * [42] David W Hosmer and Stanley Lemeshow  “Applied survival analysis: regression modelling of time to event data”  Wiley, 2002
  * [43] Petronella Cornelia Gerarda Simons et al.  “Second manifestations of ARTerial disease (SMART) study: rationale and design”  In _European journal of epidemiology_ 15.9 Springer, 1999, pp. 773–781
  * [44] Nan Xiao, Qing-Song Xu, Miao-Zhu Li and Frank Harrell  “hdnom: Benchmarking and Visualization Toolkit for Penalized Cox Models” R package version 6.0.0, 2019  URL: https://cran.r-project.org/package=hdnom
  * [45] JM Ichida, JT Wassell, MD Keller and LW Ayers  “Evaluation of protocol change in burn-care management using the Cox proportional hazards model with time-dependent covariates”  In _Statistics in Medicine_ 12.3-4 Wiley Online Library, 1993, pp. 301–310
  * [46] Cyprien Mbogning and Philippe Broet  “iBST: Improper Bagging Survival Tree” R package version 1.0, 2017  URL: https://cran.r-project.org/package=iBST
  * [47] John D Kalbfleisch and Ross L Prentice  “The statistical analysis of failure time data”  John Wiley & Sons, 2011
  * [48] Hakon K. Gjessing  “invGauss: Threshold regression that fits the (randomized drift) inverse Gaussian distribution to survival data” R package version 1.1, 2014  URL: https://cran.r-project.org/package=invGauss
  * [49] Anne Ipsen Goldman et al.  “Response of CD4 lymphocytes and clinical consequences of treatment using ddI or ddC in patients with advanced HIV infection.”  In _Journal of acquired immune deficiency syndromes and human retrovirology : official publication of the International Retrovirology Association_ 11 2, 1996, pp. 161–9
  * [50] Dimitris Rizopoulos  “JM: Joint Modeling of Longitudinal and Survival Data” R package version 1.5-1, 2022  URL: https://cran.r-project.org/package=JM
  * [51] Eric Lim et al.  “Longitudinal study of the profile and predictors of left ventricular mass regression after stentless aortic valve replacement”  In _The Annals of thoracic surgery_ 85.6 Elsevier, 2008, pp. 2026–2029
  * [52] Pete Philipson et al.  “joineR: Joint Modelling of Repeated Measurements and Time-to-Event Data” R package version 1.2.6, 2021  URL: https://cran.r-project.org/package=joineR
  * [53] Anthony G Marson et al.  “The SANAD study of effectiveness of carbamazepine, gabapentin, lamotrigine, oxcarbazepine, or topiramate for treatment of partial epilepsy: an unblinded randomised controlled trial”  In _The Lancet_ 369.9566 Elsevier, 2007, pp. 1000–1015
  * [54] Benjamin Frederick Ganzfried et al.  “curatedOvarianData: clinically annotated data for the ovarian cancer transcriptome”  In _Database_ 2013 Oxford Academic, 2013
  * [55] Takeshi Emura  “joint.Cox: Joint Frailty-Copula Models for Tumour Progression and Death in Meta-Analysis” R package version 3.1.6, 2022  URL: https://cran.r-project.org/package=joint.Cox
  * [56] Brian D Ripley and PJ Solomon  “A note on Australian AIDS survival”  Citeseer, 1994
  * [57] Brian Ripley, Bill Venables, Douglas M. Bates and Kurt Hornik  “MASS: Support Functions and Datasets for Venables and Ripley’s MASS” R package version 7.3-55, 2022  URL: https://cran.r-project.org/package=MASS
  * [58] KT Drzewiecki et al.  “Melanoma in Denmark: experience at the university hospital, Odense”  In _Cutaneous melanoma: Clinical management and treatment results worldwide. Philadelphia: JB Lippincott_ 469, 1985
  * [59] KT Drzewiecki, C Ladefoged and HE Christensen  “Biopsy and prognosis for cutaneous malignant melanomas in clinical stage I”  In _Scandinavian journal of plastic and reconstructive surgery_ 14.2 Taylor & Francis, 1980, pp. 141–144
  * [60] Eng Wei Tang, Cheuk-Kit Wong and Peter Herbison  “Global Registry of Acute Coronary Events (GRACE) hospital discharge risk score accurately predicts long-term mortality post acute coronary syndrome”  In _American heart journal_ 153.1 Elsevier, 2007, pp. 29–35
  * [61] Raphael Sonabend et al.  “mlr3proba: Probabilistic Supervised Learning for ’mlr3”’ R package version 0.4.4, 2022  URL: https://cran.r-project.org/package=mlr3proba
  * [62] Scott M Hammer et al.  “A controlled trial of two nucleoside analogues plus indinavir in persons with human immunodeficiency virus infection and CD4 cell counts of 200 per cubic millimeter or less”  In _New England Journal of Medicine_ 337.11 Mass Medical Soc, 1997, pp. 725–733
  * [63] Kerby Shedden et al.  “Gene expression-based survival prediction in lung adenocarcinoma: a multi-site, blinded validation study: Director’s Challenge Consortium for the molecular classification of lung adenocarcinoma”  In _Nature medicine_ 14.8 NIH Public Access, 2008, pp. 822
  * [64] Joaquin Vanschoren, Jan N. Rijn, Bernd Bischl and Luis Torgo  “OpenML: Networked Science in Machine Learning”  In _SIGKDD Explorations_ 15.2 New York, NY, USA: ACM, 2013, pp. 49–60  DOI: 10.1145/2641190.2641198
  * [65] CC Abnet et al.  “Zinc concentration in esophageal biopsies measured by X-ray fluorescence and cancer risk.”  In _CANCER EPIDEMIOLOGY BIOMARKERS & PREVENTION_ 13.11, 2004, pp. 1852S–1852S  AMER ASSOC CANCER RESEARCH 615 CHESTNUT ST, 17TH FLOOR, PHILADELPHIA, PA …
  * [66] Hormuzd A. Katki  “NestedCohort: Survival Analysis for Cohorts with Missing Covariate Information” R package version 1.1-3, 2020  URL: https://cran.r-project.org/package=NestedCohort
  * [67] Martin Lombard et al.  “Cyclosporin A treatment in primary biliary cirrhosis: results of a long-term placebo controlled trial”  In _Gastroenterology_ 104.2 Elsevier, 1993, pp. 519–526
  * [68] Thomas A. Gerds  “pec: Prediction Error Curves for Risk Prediction Models in Survival Analysis” R package version 2021.10.11, 2021  URL: https://cran.r-project.org/package=pec
  * [69] Henrik Stig Jørgensen et al.  “Acute stroke with atrial fibrillation: the Copenhagen Stroke Study”  In _Stroke_ 27.10 Am Heart Assoc, 1996, pp. 1765–1769
  * [70] M Schumacher et al.  “Randomized 2 x 2 trial evaluating hormonal treatment and the duration of chemotherapy in node-positive breast cancer patients. German Breast Cancer Study Group.”  In _Journal of Clinical Oncology_ 12.10, 1994, pp. 2086–2093
  * [71] Marc J Van De Vijver et al.  “A gene-expression signature as a predictor of survival in breast cancer”  In _New England Journal of Medicine_ 347.25 Mass Medical Soc, 2002, pp. 1999–2009
  * [72] Jelle Goeman, Rosa Meijer, Nimisha Chaturvedi and Matthew Lueder  “penalized: L1 (Lasso and Fused Lasso) and L2 (Ridge) Penalized Estimation in GLMs and in the Cox Model” R package version 0.9-51, 2018  URL: https://cran.r-project.org/package=penalized
  * [73] Benoit Romain et al.  “Allelotyping identification of genomic alterations in rectal chromosomally unstable tumors without preoperative treatment”  In _BMC cancer_ 10.1 BioMed Central, 2010, pp. 1–13
  * [74] Frederic Bertrand and Myriam Maumy-Bertrand  “plsRcox: Partial Least Squares Regression for Cox Models and Related Techniques” R package version 1.7.6, 2021  URL: https://cran.r-project.org/package=plsRcox
  * [75] Lee A Lillard and Constantijn WA Panis  “aML multilevel multiprocess statistical software”  In _Los Angeles: Econ Ware_ , 2003
  * [76] German Rodriguez  “Generalized Linear Models”  In _Princeton University_ The Trustees of Princeton University  URL: https://data.princeton.edu/wws509/datasets/#divorce
  * [77] PM Petersen et al.  “Long-term outcome in stage I and II follicular lymphoma following treatment with involved field radiation therapy alone”  In _Journal of Clinical Oncology_ 22.14_suppl American Society of Clinical Oncology, 2004, pp. 6521–6521
  * [78] Hemant Ishwaran and Udaya B. Kogalur  “randomForestSRC: Fast Unified Random Forests for Survival, Regression, and Classification (RF-SRC)” R package version 2.14.0, 2022  URL: https://cran.r-project.org/package=randomForestSRC
  * [79] Laura J Van’t Veer et al.  “Gene expression profiling predicts clinical outcome of breast cancer”  In _nature_ 415.6871 Nature Publishing Group, 2002, pp. 530–536
  * [80] Anna Bergamaschi et al.  “Distinct patterns of DNA copy number alteration are associated with different clinicopathological features and gene-expression subtypes of breast cancer”  In _Genes, Chromosomes and Cancer_ 45.11 Wiley Online Library, 2006, pp. 1033–1040
  * [81] Mugahid D and Kaderali L  “RCASPAR: A package for survival time prediction based on a piecewise baseline hazard Cox regression model” R package version 1.40.0, 2021  URL: https://www.bioconductor.org/packages/release/bioc/html/RCASPAR.html
  * [82] Marilia Sá Carvalho, Robin Henderson, Silvia Shimakura and Inês Pereira Silva Cunha Sousa  “Survival of hemodialysis patients: modeling differences in risk of dialysis centers”  In _International Journal for Quality in Health Care_ 15.3 Oxford University Press, 2003, pp. 189–196
  * [83] John Fox  “RcmdrPlugin.survival: R Commander Plug-in for the ’survival’ Package” R package version 1.2-2, 2022  URL: https://cran.r-project.org/package=RcmdrPlugin.survival
  * [84] Peter H Rossi, Richard A Berk and Kenneth J Lenihan  “Money, work and crime: some experimental results”  New York: Academic Press, 1980
  * [85] Maja Pohar and Janez Stare  “Relative survival analysis in R”  In _Computer methods and programs in biomedicine_ 81.3 Elsevier, 2006, pp. 272–278
  * [86] Maja Pohar Perme and Damjan Manevski  “relsurv: Relative Survival” R package version 2.2-6, 2021  URL: https://cran.r-project.org/package=relsurv
  * [87] Therese Sørlie et al.  “Repeated observation of breast tumor subtypes in independent gene expression data sets”  In _Proceedings of the national academy of sciences_ 100.14 National Acad Sciences, 2003, pp. 8418–8423
  * [88] Chandan Reddy  URL: https://people.cs.vt.edu/~reddy/
  * [89] Lars Bullinger et al.  “Use of gene-expression profiling to identify prognostic subclasses in adult acute myeloid leukemia”  In _New England Journal of Medicine_ 350.16 Mass Medical Soc, 2004, pp. 1605–1616
  * [90] Hans C Van Houwelingen et al.  “Cross-validated Cox regression on microarray gene expression data”  In _Statistics in medicine_ 25.18 Wiley Online Library, 2006, pp. 3201–3216
  * [91] Andreas Rosenwald et al.  “The use of molecular profiling to predict survival after chemotherapy for diffuse large-B-cell lymphoma”  In _New England Journal of Medicine_ 346.25 Mass Medical Soc, 2002, pp. 1937–1947
  * [92] “DIVAT”  In _DIVAT_ , 2022  URL: http://www.divat.fr/
  * [93] Yohann Foucher et al.  “RISCA: Causal Inference and Prediction in Cohort-Based Analyses” R package version 0.9, 2020  URL: https://cran.r-project.org/package=RISCA
  * [94] Florent Le Borgne et al.  “Comparisons of the performance of different statistical tests for time-to-event analysis with confounding factors: practical illustrations in kidney transplantation”  In _Statistics in Medicine_ 35.7 Wiley Online Library, 2016, pp. 1103–1116
  * [95] Florent Le Borgne et al.  “Standardized and weighted time-dependent receiver operating characteristic curves to evaluate the intrinsic prognostic capacities of a marker by taking into account confounding factors”  In _Statistical Methods in Medical Research_ 27.11 SAGE Publications Sage UK: London, England, 2018, pp. 3397–3410
  * [96] Alfio Marazzi  “Algorithms, routines, and S-functions for robust statistics”  CRC Press, 1993
  * [97] Alfio Marazzi and Jean-Luc Muralti  “RobustAFT: Truncated Maximum Likelihood Fit and Robust Accelerated Failure Time Regression for Gaussian and Log-Weibull Case” R package version 1.4-5, 2020  URL: https://cran.r-project.org/package=RobustAFT
  * [98] Leo Breiman, Jerome H Friedman, Richard A Olshen and Charles J Stone  “Classification and regression trees”  Routledge, 2017
  * [99] Terry Therneau, Beth Atkinson and Brian Ripley  “rpart: Recursive Partitioning and Regression Trees” R package version 4.1.16, 2022  URL: https://cran.r-project.org/package=rpart
  * [100] John M Kirkwood et al.  “Interferon alfa-2b adjuvant therapy of high-risk resected cutaneous melanoma: the Eastern Cooperative Oncology Group Trial EST 1684.”  In _Journal of clinical oncology_ 14.1, 1996, pp. 7–17
  * [101] Chao Cai, Yubo Zou, Yingwei Peng and Jiajia Zhang  “smcure: Fit Semiparametric Mixture Cure Models” R package version 2.0, 2012  URL: https://cran.r-project.org/package=smcure
  * [102] Artur Araujo and Luis Meira-Machado  “smoothHR: Smooth Hazard Ratio Curves Taking a Reference Value” R package version 1.0.3, 2021  URL: https://cran.r-project.org/package=smoothHR
  * [103] Robin Henderson, Silvia Shimakura and David Gorst  “Modeling spatial variation in leukemia survival data”  In _Journal of the American Statistical Association_ 97.460 Taylor & Francis, 2002, pp. 965–972
  * [104] Haiming Zhou and Timothy Hanson  “spBayesSurv: Bayesian Modeling and Analysis of Spatially Correlated Survival Data” R package version 1.1.5, 2021  URL: https://cran.r-project.org/package=spBayesSurv
  * [105] Charles Lawrence Loprinzi et al.  “Prospective evaluation of prognostic variables from patient-completed questionnaires. North Central Cancer Treatment Group.”  In _Journal of Clinical Oncology_ 12.3, 1994, pp. 601–607
  * [106] Terry M Therneau, Thomas Lumley and Crowson Cynthia  “survival: Survival Analysis” R package version 3.2-13, 2021  URL: https://cran.r-project.org/package=survival
  * [107] International Chronic Granulomatous Disease Cooperative Study Group*  “A controlled trial of interferon gamma to prevent infection in chronic granulomatous disease”  In _New England Journal of Medicine_ 324.8 Mass Medical Soc, 1991, pp. 509–516
  * [108] Thomas R Fleming and David P Harrington  “Counting processes and survival analysis”  John Wiley & Sons, 2011
  * [109] Charles G Moertel et al.  “Fluorouracil plus levamisole as effective adjuvant therapy after resection of stage III colon carcinoma: a final report”  In _Annals of internal medicine_ 122.5 American College of Physicians, 1995, pp. 321–326
  * [110] Robert A Kyle et al.  “Prevalence of monoclonal gammopathy of undetermined significance”  In _New England Journal of Medicine_ 354.13 Mass Medical Soc, 2006, pp. 1362–1369
  * [111] Angela Dispenzieri et al.  “Use of nonclonal serum immunoglobulin free light chains to predict overall survival in the general population”  In _Mayo Clinic Proceedings_ 87.6, 2012, pp. 517–523  Elsevier
  * [112] John Crowley and Marie Hu  “Covariance analysis of heart transplant survival data”  In _Journal of the American Statistical Association_ 72.357 Taylor & Francis, 1977, pp. 27–36
  * [113] Robert A Kyle  ““Benign” monoclonal gammopathy—after 20 to 35 years of follow-up”  In _Mayo Clinic Proceedings_ 68.1, 1993, pp. 26–36  Elsevier
  * [114] John H Edmonson et al.  “Prognosis in Advanced Ovarian Carcinoma Versus Minimal Residual”  In _Cancer treatment reports_ 63.2 US Department of Health, Education,Welfare, Public Health Service …, 1979, pp. 241–247
  * [115] E Rolland Dickson et al.  “Prognosis in primary biliary cirrhosis: model for decision making”  In _Hepatology_ 10.1 Wiley Online Library, 1989, pp. 1–7
  * [116] AL Blair et al.  “The 5-year prognosis for vision in diabetes.”  In _The Ulster medical journal_ 49.2 Ulster Medical Society, 1980, pp. 139
  * [117] Ross L Prentice  “Exponential survivals with censoring and explanatory variables”  In _Biometrika_ 60.2 Oxford University Press, 1973, pp. 279–288
  * [118] Norman E Breslow and Nilanjan Chatterjee  “Design and analysis of two-phase studies with binary outcome applied to Wilms tumour prognosis”  In _Journal of the Royal Statistical Society: Series C (Applied Statistics)_ 48.4 Wiley Online Library, 1999, pp. 457–468
  * [119] “Geo accession viewer”  In _National Center for Biotechnology Information_ U.S. National Library of Medicine  URL: https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE4335
  * [120] Haleh Yasrebi, Peter Sperisen, Viviane Praz and Philipp Bucher  “Can survival prediction be improved by merging gene expression data sets?”  In _PloS one_ 4.10 Public Library of Science San Francisco, USA, 2009, pp. e7431
  * [121] Haleh Yasrebi  “survJamda.data: Data for Package ’survJambda”’ R package version 1.0.2, 2015  URL: https://cran.r-project.org/package=survJamda.data
  * [122] “Geo accession viewer”  In _National Center for Biotechnology Information_ U.S. National Library of Medicine  URL: https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE3143
  * [123] “Geo accession viewer”  In _National Center for Biotechnology Information_ U.S. National Library of Medicine  URL: https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE1992
  * [124] W Nick Street, Olvi L Mangasarian and William H Wolberg  “An inductive learning approach to prognostic prediction”  In _Machine Learning Proceedings 1995_ Elsevier, 1995, pp. 522–530
  * [125] Torsten Hothorn  “TH.data: TH’s Data Archive” R package version 1.1-0, 2021  URL: https://cran.r-project.org/package=TH.data
  * [126] Gunnar Vagn Hagemann Jensen et al.  “Does in-hospital ventricular fibrillation affect prognosis after myocardial infarction?”  In _European heart journal_ 18.6 Oxford University Press, 1997, pp. 919–924
  * [127] Jeremy Silver Thomas Scheike Torben Martinussen and Klaus Holst  “timereg: Flexible Regression Models for Survival Data” R package version 2.0.1, 2021  URL: https://cran.r-project.org/package=timereg
  * [128] Poul Schlichting et al.  “Prognostic factors in cirrhosis identified by Cox’s regression model”  In _Hepatology_ 3.6 Wiley Online Library, 1983, pp. 889–895
  * [129] William J Huster, Ron Brookmeyer and Steven G Self  “Modelling paired survival data with covariates”  In _Biometrics_ JSTOR, 1989, pp. 145–156
  * [130] William A Knaus et al.  “The SUPPORT prognostic model: Objective estimates of survival for seriously ill hospitalized adults”  In _Annals of internal medicine_ 122.3 American College of Physicians, 1995, pp. 191–203
  * [131] Frank Harrell  “Datasets”  In _Vanderbilt Department of Biostatistics_ , 2022  URL: https://hbiostat.org/data/
  * [132] DF Andrews and AM Herzberg  “Prognostic variables for survival in a randomized comparison of treatments for prostatic cancer”  In _Data_ Springer, 1985, pp. 261–274
  * [133] Syed S Mahmood, Daniel Levy, Ramachandran S Vasan and Thomas J Wang  “The Framingham Heart Study and the epidemiology of cardiovascular disease: a historical perspective”  In _The lancet_ 383.9921 Elsevier, 2014, pp. 999–1008
  * [134] Alfred F Connors et al.  “The effectiveness of right heart catheterization in the initial care of critically III patients”  In _Jama_ 276.11 American Medical Association, 1996, pp. 889–897
  * [135] Discover Data@Duke  “Duke Databank for Cardiovascular Diseases Analysis”  URL: https://datacatalog.mclibrary.duke.edu/dataset/duke-databank-for-cardiovascular-diseases-analysis-datasets
  * [136] Michael O’Shea, David A Savitz, Marvin L Hage and KA Feinstein  “Prenatal events and the risk of subependymal/intraventricular haemorrhage in very low birthweight neonates”  In _Paediatric and perinatal epidemiology_ 6.3 Wiley Online Library, 1992, pp. 352–362


MIDL 2020Medical Imaging with Deep Learning 2020 MIDL 2020 – Short Paper
Leixin Zhou<EMAIL_ADDRESS>
Wenxiang Deng<EMAIL_ADDRESS>
Xiaodong Wu<EMAIL_ADDRESS>
Department of Electrical and Computer Engineering, University of Iowa, USA

# Robust Image Segmentation Quality Assessment

###### Abstract

Deep learning based image segmentation methods have achieved great success,
even having human-level accuracy in some applications. However, due to the
black box nature of deep learning, the best method may fail in some
situations. Thus predicting segmentation quality without ground truth would be
very crucial especially in clinical practice. Recently, people proposed to
train neural networks to estimate the quality score by regression. Although it
can achieve promising prediction accuracy, the network suffers robustness
problem, e.g. it is vulnerable to adversarial attacks. In this paper, we
propose to alleviate this problem by utilizing the difference between the
input image and the reconstructed image, which is conditioned on the
segmentation to be assessed, to lower the chance to overfit to the undesired
image features from the original input image, and thus to increase the
robustness. Results on ACDC17 dataset demonstrated our method is promising.

###### keywords:

Robust, Segmentation Quality, Deep Learning

##  1 Introduction

Segmentation quality assessment with the absence of ground truth, which
estimates segmentation accuracy without human or expert intervention, is of
high interest in medical imaging research and clinical fields. In many
applications, the deep learning based segmentation methods can even achieve
expert-level accuracy. However, in practice, deep learning methods may fail
due to many factors: such as domain shift [Patel et al.(2015)Patel, Gopalan,
Li, and Chellappa], adversarial noise, and low image quality. Therefore
predicting segmentation quality without ground truth would be very crucial and
of high interest for the downstream analysis.

One straightforward idea is to predict segmentation quality using a CNN
regression network, where the image and its segmentation are concatenated as
different channels to feed into the network [Robinson et al.(2018b)Robinson,
Oktay, Bai, Valindria, Sanghvi, Aung, Paiva, Zemrak, Fung, Lukaschuk, et al.,
Robinson et al.(2018a)Robinson, Oktay, Bai, Valindria, Sanghvi, Aung, Paiva,
Zemrak, Fung, Lukaschuk, et al.]. However, that state-of-the-art method
suffers the robustness problem if the input images have a different
distribution from that of those training datasets for the regress network.
This can be demonstrated with adversarial attacks, in which it involves adding
hand-crafted perturbations to the images drew from the distribution of
training data and leading to misbehave for deep neural networks.

Inspired by the work of representation learning and factorization [Mirza and
Osindero(2014), Chartsias et al.(2018)Chartsias, Joyce, Papanastasiou, Semple,
Williams, Newby, Dharmakumar, and Tsaftaris], we propose to improve the
prediction robustness by extracting features directly related to the
segmentation. More precisely, we propose to utilize the difference of the
original input image and the reconstructed image conditioned on the input
image and the input segmentation. Our work is most related to Kohlberger et
al.’s work [Kohlberger et al.(2012)Kohlberger, Singh, Alvino, Bahlmann, and
Grady], in which the quality assessment score is estimated by regression based
on numerous statistical and energy measures from segmentation algorithms. Our
method also shares merits of unsupervised lesion or outlier detection [Schlegl
et al.(2017)Schlegl, Seeböck, Waldstein, Schmidt-Erfurth, and Langs, Baur et
al.(2018)Baur, Wiestler, Albarqouni, and Navab, Chen and Konukoglu(2018),
Pawlowski et al.(2018)Pawlowski, Lee, Rajchl, McDonagh, Ferrante, Kamnitsas,
Cooke, Stevenson, Khetani, Newman, et al., Seeböck et al.(2016)Seeböck,
Waldstein, Klimscha, Gerendas, Donner, Schlegl, Schmidt-Erfurth, and Langs,
Alaverdyan et al.(2018)Alaverdyan, Jung, Bouet, and Lartizien], where only
normal data (ground truth segmentation in our scenario) is utilized in the
training of the reconstruction network.

Contributions: In this paper, we propose to make use of features directly
related to segmentation to improve the robustness of the quality regression
network for segmentation quality assessment. To achieve this goal, we have
developed two CNNs: one is a reconstruction network (REC-Net), which aims to
reconstruct the original image from the image masked by the provided
segmentation; the other is a quality regression network (REG-Net), which
predicts the segmentation quality based on the reconstruction difference image
and the provided segmentation. Our experiments on ACDC17 dataset
111https://www.creatis.insa-lyon.fr/Challenge/acdc/databases.html have
demonstrated highly promising performance of the proposed method.

##  2 Method

Assume the input image, its ground truth segmentation and the candidate
segmentation (to be assessed) are $\mathcal{I}_{in}\in\mathbb{R}^{n\times n}$,
$\mathcal{S}_{gt}\in\mathbb{Z}^{n\times n}$ and
$\mathcal{S}_{seg}\in\mathbb{Z}^{n\times n}$, respectively. It is trivial to
apply any metric functions (e.g. Dice, Jaccard scores) to the pair
$\mathcal{S}_{gt}$ and $\mathcal{S}_{seg}$ to get the ground truth
segmentation quality score, e.g. $\mathcal{GT}_{dice}$. However, the absence
of $\mathcal{S}_{gt}$ makes generating Dice prediction $\mathcal{P}_{dice}$
non-trivial.

Figure 1: The work flow of proposed segmentation quality assessment method.

The flow of the proposed method is demonstrated as in Fig. 1. We use
$\mathcal{I}_{in|\mathcal{S}}$ to represent the image with the segmented
target $\mathcal{S}$ being masked by zero, in which $\mathcal{S}_{ij}=1$ if
the corresponding pixel belongs to the target; otherwise $\mathcal{S}_{ij}=0$.
More specifically,
$\mathcal{I}_{in|\mathcal{S}}=\mathcal{I}_{in}\cdot(1-\mathcal{S})$. In other
words, all pixels that are labeled by $\mathcal{S}$ as the target object in
$\mathcal{I}_{in}$ are set to zero intensity. The reconstructed image using
the proposed reconstruction network (REC-Net) from
$\mathcal{I}_{in|\mathcal{S}}$, is denoted as $\mathcal{I}_{rec}$. The
difference image $\mathcal{I}_{dif}$, which serves as one input channel to the
quality regression network (REG-Net), is defined as:
$\mathcal{I}_{dif}=\mathcal{I}_{in}-\mathcal{I}_{rec}$. The output of REG-Net
$\mathcal{P}_{dice}$ is the predicted score for the segmentation quality.

During the training of REC-Net, only pairs of $\mathcal{I}_{in}$ and its
$\mathcal{S}_{gt}$ are utilized. The rationale behind is that the REC-Net is
trained to well recover the original input image only when $\mathcal{S}$ is a
good segmentation. However, during the training of REG-Net, segmentations of
different quality have to be used to teach the REG-Net the quality measure.
The REC-Net and the REG-Net have a U-net and Alex-net architectures,
respectively.

##  3 Experiments

###  3.1 Data

To validate the proposed segmentation quality assessment method, we utilize a
public dataset: Automated Cardiac Diagnosis Challenge (ACDC) MICCAI challenge
2017. For our experiments, only segmentation of left-ventricular myocardium
(LVM), which is very challenging, was considered. To train the REG-Net,
segmentations of different quality have to be generated first. In our
experiments, in contrast to random forests used in [Robinson et
al.(2018a)Robinson, Oktay, Bai, Valindria, Sanghvi, Aung, Paiva, Zemrak, Fung,
Lukaschuk, et al.], U-nets with different depths, different number of starting
filters and different training epochs, were applied to generate the simulated
segmentations with different quality.

###  3.2 Adversarial attacks generation

We compared the robustness of our proposed method with respect to adversarial
attacks against the state-of-the-art methods [Robinson et al.(2018a)Robinson,
Oktay, Bai, Valindria, Sanghvi, Aung, Paiva, Zemrak, Fung, Lukaschuk, et al.,
Robinson et al.(2017)Robinson, Valindria, Bai, Suzuki, Matthews, Page,
Rueckert, and Glocker, Robinson et al.(2018b)Robinson, Oktay, Bai, Valindria,
Sanghvi, Aung, Paiva, Zemrak, Fung, Lukaschuk, et al.]. We applied a simple
fast gradient sign method [Kurakin et al.(2016)Kurakin, Goodfellow, and
Bengio] to generate the adversarial images for REG-Net to conduct our
experiments. Only adversarial attacks on the original images
$\mathcal{I}_{in}$ and the difference image $\mathcal{I}_{dif}$ were
considered and no changes were made to $\mathcal{S}_{seg}$.

###  3.3 Performance comparison

The mean absolution error (MAE) of the Dice scores,
$\text{MAE}=\frac{\sum_{i=1}^{SN}|\mathcal{P}_{dice}^{i}-\mathcal{GT}_{dice}^{i}|}{SN}$,
was utilized as the metric, where $SN$ is total number of slices in the test
set. The results without adversarial attacks are shown in column $\epsilon=0$
in Table. 1. As can be seen, when there is no attack, the proposed method
works as well as Robinson et al.’s [Robinson et al.(2018a)Robinson, Oktay,
Bai, Valindria, Sanghvi, Aung, Paiva, Zemrak, Fung, Lukaschuk, et al.,
Robinson et al.(2017)Robinson, Valindria, Bai, Suzuki, Matthews, Page,
Rueckert, and Glocker, Robinson et al.(2018b)Robinson, Oktay, Bai, Valindria,
Sanghvi, Aung, Paiva, Zemrak, Fung, Lukaschuk, et al.]. The performance when
having attacks is demonstrated in the right most four columns in Table. 1.

Method | $\epsilon=0$ | $\epsilon=0.05$ | $\epsilon=0.1$ | $\epsilon=0.2$ | $\epsilon=0.3$
---|---|---|---|---|---
Robinson et al. | 0.04$\pm$0.05 | 0.08$\pm$0.06 | 0.11$\pm$0.07 | 0.14$\pm$0.08 | 0.16$\pm$0.09
proposed | 0.04$\pm$0.05 | 0.07$\pm$0.06 | 0.09$\pm$0.06 | 0.09$\pm$0.07 | 0.12$\pm$0.09
Table 1: Mean absolute errors of dice prediction under different levels of
adversarial attack.

It can be noticed that for both methods, the MAEs are monotonically non-
decreasing as the attack level increases. However, the proposed method has a
smaller increasing rate and works better than Robinson et al.’s.

##  4 Conclusion

In this paper, a robust method for segmentation quality assessment has been
proposed. We make use of the image difference between the input image and the
reconstructed image using our proposed image reconstruction network (REC-Net),
as the feature image for the quality score regression network (REG-Net).
Results on ACDC17 dataset verified our method is more robust.

## References

  * [Alaverdyan et al.(2018)Alaverdyan, Jung, Bouet, and Lartizien] Zara Alaverdyan, Julien Jung, Romain Bouet, and Carole Lartizien.  Regularized siamese neural network for unsupervised outlier detection on brain multiparametric magnetic resonance imaging: application to epilepsy lesion screening.  2018\.
  * [Baur et al.(2018)Baur, Wiestler, Albarqouni, and Navab] Christoph Baur, Benedikt Wiestler, Shadi Albarqouni, and Nassir Navab.  Deep autoencoding models for unsupervised anomaly segmentation in brain mr images.  _arXiv preprint arXiv:1804.04488_ , 2018.
  * [Chartsias et al.(2018)Chartsias, Joyce, Papanastasiou, Semple, Williams, Newby, Dharmakumar, and Tsaftaris] Agisilaos Chartsias, Thomas Joyce, Giorgos Papanastasiou, Scott Semple, Michelle Williams, David Newby, Rohan Dharmakumar, and Sotirios A Tsaftaris.  Factorised spatial representation learning: application in semi-supervised myocardial segmentation.  _arXiv preprint arXiv:1803.07031_ , 2018.
  * [Chen and Konukoglu(2018)] Xiaoran Chen and Ender Konukoglu.  Unsupervised detection of lesions in brain mri using constrained adversarial auto-encoders.  _arXiv preprint arXiv:1806.04972_ , 2018.
  * [Kohlberger et al.(2012)Kohlberger, Singh, Alvino, Bahlmann, and Grady] Timo Kohlberger, Vivek Singh, Chris Alvino, Claus Bahlmann, and Leo Grady.  Evaluating segmentation error without ground truth.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , pages 528–536. Springer, 2012.
  * [Kurakin et al.(2016)Kurakin, Goodfellow, and Bengio] Alexey Kurakin, Ian Goodfellow, and Samy Bengio.  Adversarial examples in the physical world.  _arXiv preprint arXiv:1607.02533_ , 2016.
  * [Mirza and Osindero(2014)] Mehdi Mirza and Simon Osindero.  Conditional generative adversarial nets.  _arXiv preprint arXiv:1411.1784_ , 2014.
  * [Patel et al.(2015)Patel, Gopalan, Li, and Chellappa] Vishal M Patel, Raghuraman Gopalan, Ruonan Li, and Rama Chellappa.  Visual domain adaptation: A survey of recent advances.  _IEEE signal processing magazine_ , 32(3):53–69, 2015.
  * [Pawlowski et al.(2018)Pawlowski, Lee, Rajchl, McDonagh, Ferrante, Kamnitsas, Cooke, Stevenson, Khetani, Newman, et al.] Nick Pawlowski, Matthew CH Lee, Martin Rajchl, Steven McDonagh, Enzo Ferrante, Konstantinos Kamnitsas, Sam Cooke, Susan Stevenson, Aneesh Khetani, Tom Newman, et al.  Unsupervised lesion detection in brain ct using bayesian convolutional autoencoders.  2018\.
  * [Robinson et al.(2017)Robinson, Valindria, Bai, Suzuki, Matthews, Page, Rueckert, and Glocker] Robert Robinson, Vanya V Valindria, Wenjia Bai, Hideaki Suzuki, Paul M Matthews, Chris Page, Daniel Rueckert, and Ben Glocker.  Automatic quality control of cardiac mri segmentation in large-scale population imaging.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , pages 720–727. Springer, 2017.
  * [Robinson et al.(2018a)Robinson, Oktay, Bai, Valindria, Sanghvi, Aung, Paiva, Zemrak, Fung, Lukaschuk, et al.] Robert Robinson, Ozan Oktay, Wenjia Bai, Vanya Valindria, Mihir Sanghvi, Nay Aung, José Paiva, Filip Zemrak, Kenneth Fung, Elena Lukaschuk, et al.  Real-time prediction of segmentation quality.  _arXiv preprint arXiv:1806.06244_ , 2018a.
  * [Robinson et al.(2018b)Robinson, Oktay, Bai, Valindria, Sanghvi, Aung, Paiva, Zemrak, Fung, Lukaschuk, et al.] Robert Robinson, Ozan Oktay, Wenjia Bai, Vanya V Valindria, Mihir M Sanghvi, Nay Aung, José Miguel Paiva, Filip Zemrak, Kenneth Fung, Elena Lukaschuk, et al.  Subject-level prediction of segmentation failure using real-time convolutional neural nets.  2018b.
  * [Schlegl et al.(2017)Schlegl, Seeböck, Waldstein, Schmidt-Erfurth, and Langs] Thomas Schlegl, Philipp Seeböck, Sebastian M Waldstein, Ursula Schmidt-Erfurth, and Georg Langs.  Unsupervised anomaly detection with generative adversarial networks to guide marker discovery.  In _International Conference on Information Processing in Medical Imaging_ , pages 146–157. Springer, 2017.
  * [Seeböck et al.(2016)Seeböck, Waldstein, Klimscha, Gerendas, Donner, Schlegl, Schmidt-Erfurth, and Langs] Philipp Seeböck, Sebastian Waldstein, Sophie Klimscha, Bianca S Gerendas, René Donner, Thomas Schlegl, Ursula Schmidt-Erfurth, and Georg Langs.  Identifying and categorizing anomalies in retinal imaging data.  _arXiv preprint arXiv:1612.00686_ , 2016.