# Discovering Significant Topics from Legal Decisions with Selective Inference

JERROLD SOH TSIN HOWE Yong Pung How School of Law, Singapore Management
University<EMAIL_ADDRESS>https://orcid.org/0000-0003-0270-5015).
This is an accepted manuscript of work forthcoming in PhilTrans A; please cite
the publisher’s version only. I thank Georgios Georgiou, James Greiner, Alma
Cohen, Lucian Bebchuk, William Hubbard, Michael Livermore, and Kevin Ashley,
and participants of the Complexity in Law & Governance Works in Progress
Workshop for constructive comments, including on very early versions of this
work. All errors are mine.

###### Abstract

We propose and evaluate an automated pipeline for discovering significant
topics from legal decision texts by passing features synthesized with topic
models through penalised regressions and post-selection significance tests.
The method identifies case topics significantly correlated with outcomes,
topic-word distributions which can be manually-interpreted to gain insights
about significant topics, and case-topic weights which can be used to identify
representative cases for each topic. We demonstrate the method on a new
dataset of domain name disputes and a canonical dataset of European Court of
Human Rights violation cases. Topic models based on latent semantic analysis
as well as language model embeddings are evaluated. We show that topics
derived by the pipeline are consistent with legal doctrines in both areas and
can be useful in other related legal analysis tasks.

Keywords: Legal Language Processing, Topic Models, Text-as-Data, Domain Name
Disputes, European Court of Human Rights

## ntroduction

Most legal information is stored exclusively in natural language texts. The
complexity of language means extracting such information is typically a
labour-intensive exercise primarily performed by specially-trained persons
(“lawyers”). This poses significant barriers to computational representations
and analysis of law [50, 59]. Researchers have increasingly sought to develop
automated processes for converting unstructured legal texts to structured
variables [33, 2]. Depending on the texts involved and variables required,
these have included term frequency counts [16], regular expressions [71],
topic models [1, 25, 10, 23, 66], word embeddings [20, 39], and language
models [12, 62, 48]. Given their centrality in legal analysis, court decisions
in particular have attracted significant scholarly attention. Many studies
have attempted to identify, categorise, or forecast case outcomes using
decision texts, often relying on opaque algorithms such as support vector
machines and neural networks [1, 49, 11, 55, 54]. Other researchers have
prioritised more explainable methods over end-to-end prediction. Typically,
algorithms are first developed to automatically extract case attributes and
other legally-relevant variables before using these variables to model
outcomes [3, 25, 8, 30, 31]. The goal is not necessarily predictive accuracy
alone, but also to identify and explain what motivates legal decisions.

In this work, we propose and evaluate a new automated pipeline for discovering
significant topics from decision texts, a task we define more formally in part
2.1. The pipeline takes decision texts and case outcomes as inputs and returns
estimates for statistically significant decision topics as well as the cases,
words, and phrases most strongly associated those topics. This allows
researchers to quickly identify potential variables, patterns, and cases of
interest in unfamiliar areas of law. The pipeline comprises four steps: pre-
processing and masking (part 2.2.1), topic modelling (part 2.2.2), selective
regression and inference (part 2.2.3), and topic evaluation (part 2.2.4). We
demonstrate and evaluate the pipeline on a new dataset of cases resolved under
the Uniform Domain Name Dispute Resolution Policy (“UDRP”). To explore how the
pipeline generalises, we further test it on a canonical dataset of European
Convention of Human Rights (“ECHR”) cases. For both datasets, we experiment
with Latent Semantic Analysis (“LSA”) [46] as well as two BERTopic (“BTO”)
models [34] primed with general and legally-finetuned embeddings respectively.

We show that topics discovered by the pipeline contain interpretable and
legally-sound information on topics correlated with legal outcomes (part 3).
Along the way, we identify several interesting patterns and case archetypes in
UDRP and ECHR case law. Thus, our key contributions are as follows. First, we
extend prior work analyzing legal outcomes from a topic modelling perspective
[1, 25, 66]. To be sure, the notion that topics synthesized from case
decisions could carry meaningful information about legal outcomes is not new.
Neither do we propose entirely new algorithms for, say, legal topic modelling.
Our incremental contribution lies in integrating several existing techniques
(e.g. masking [74], topic modelling [1, 66], and selective inference [76, 75])
into a pipeline that can be adapted to study other legal areas. Second, we
demonstrate the utility of selective inference techniques in the legal domain.
This has not, to our best knowledge, been studied in prior work. Finally, we
add to legal knowledge on UDRP and ECHR cases.

## ethods

###<EMAIL_ADDRESS>Discovering Significant Topics

This work relates to existing literature on the automated extraction of legal
factors from legal cases [25, 30, 31]. Legal factors are generally seen as
“stereotypical patterns of fact” [25] or more abstract “intermediate concepts”
[9] which influence case outcomes. However, as used in that literature, the
concept of legal factors has a specific meaning which does not overlap
perfectly with our present focus. We thus use the term “predictors” here to
refer broadly to variables which predict case outcomes. Drawing inspiration
from [14], suppose a legal outcome $Y$ is given by $Y=f(X,W)$, where $X$ is a
matrix of legal predictors, $W$ a matrix of non-legal predictors (e.g.
political ideologies [65]), and $f$ some adjudication function that maps cases
to outcomes. To identify individual predictors, we might collect data on
hypothesised variables $\hat{X},\hat{W}$ (i.e. approximations of $X$ and $W$),
and estimate the model $\hat{Y}=\hat{f}(\hat{X},\hat{W})$. Weights computed
for each $\hat{x},\hat{w}$ would capture the strength and polarity of their
correlation with outcomes. Variables assigned significant, non-zero weights
can be understood as potential legal (or non-legal) predictors. They may
further be seen as causal predictors, if the model is causally-identified, or
correlative predictors otherwise.

The challenge with legal applications is that the $x$s and $w$s are not
available as structured data but found only in some natural language corpus
$D$. Typically these are decision texts written to state and justify outcomes
for each case $i$, though other documents including submissions, affidavits,
and procedural records may also be relevant. We must apply a “codebook
function” $g:D\mapsto Q,Q\in\mathbb{R}^{n\times m}$ that maps $n$ texts to $m$
variables [33]. Where the variables desired are known ex ante based on legal
domain knowledge, the researcher’s aim is to extract observations of
$\hat{x}_{i}$. But in unfamiliar legal areas where candidate predictors are
not already known, the goal shifts from filling observations or estimating
coefficients to discovering such predictors to begin with. There are therefore
three different tasks related to legal predictors (table
LABEL:tab:task_names). Notably, these are not mutually exclusive and must
often be performed in tandem to answer the research question. Suppose as in
[72] that we want to know if case origin influences the probability of a
certiorari grant by the US Supreme Court. The variable of interest is known
but potential confounders remain to be identified. We would need to extract
observations for case origin, discover (and thereafter extract observations
for) potential confounders, and finally analyse coefficients for case origin
while controlling for these confounders. This work is chiefly concerned with
the discovery task, though extraction and analysis are by-products of the
proposed method.

Given Inputs |  Goal |  Task Label(s)
---|---|---
$\hat{X},\hat{W}$ |  Estimate weights |  Analysis
$D$, candidate $\hat{X},\hat{W}$s known |  Extract observations of $\hat{X},\hat{W}$ |  Identification/Extraction
$D$, candidate $\hat{X},\hat{W}$s unknown |  Discover candidate $\hat{X},\hat{W}$s |  Discovery

###<EMAIL_ADDRESS>Proposed Pipeline

####<EMAIL_ADDRESS>Pre-processing and Masking

We begin with a text corpus $D$ and structured categorical outcomes $Y$ for
$n$ cases in some legal area of interest. In theory, any corpus with
sufficient case information, such as case briefs and affidavits, could be
used. In practice, most legal analysis is based solely on decision texts.
Other legal documents are usually not accessible at scale. Thus, we tailor the
approach assuming $D$ is a decision corpus. The use of decision texts has
important implications for the kind of analyses possible and the pre-
processing steps necessary. Specifically, fitting legal outcome models on
decisions is problematic because decisions are written by judges, after
observing case facts, to justify case outcomes [72]. Extracted features could
therefore contain both post-treatment and post-outcome information, making
them “bad controls” [17]. Formally, suppose decisions are generated by the
process $D=t(Y,X,W,J)$, where $J$ accounts for the judges’ individual writing
styles and $t$ some text-generation function. Substituting this into the model
$\hat{Y}=g(D)$ gives $\hat{Y}=g(t(Y,X,W,J))$. Since we are indirectly
modelling $Y$ on itself, we should expect the model to produce large,
significant estimates for features still containing hints of $Y$ after the
transformations $g$ and $t$ instead of unbiased estimates for $x$s and $w$s.

As we do not control $t$, the natural solution, other than switching to some
pre-outcome corpus, is to build into $g$ processes for masking information on
$Y$. We follow standard steps from the legal prediction literature in masking
outcome-revealing sections of and phrases in the text from the model by
deleting them entirely at the start of the pipeline [1, 74]. This may over-
inclusively remove otherwise informative words, but is however taken as a
necessary and non-fatal trade-off [74]. It also may not remove all outcome
information from $D$. Since decisions are written to justify case outcomes,
even seemingly innocuous sections such as “Case Facts” could be arranged in a
way that favours the writer’s preferred disposition. Indeed, lawyers are
typically taught to present facts persuasively [64]. This pertains especially
to case briefs, but we cannot preclude its occurrence in decisions. As such,
we emphasise that predictors discovered by our method should be interpreted as
correlative.

Where required, we then pre-process the masked corpus in standard fashion by
lowercasing, stopping, and lemmatisation. This applies mainly to LSA as BTO is
trained on raw texts.111https://maartengr.github.io/BERTopic/faq.html#how-do-
i-remove-stop-words

####<EMAIL_ADDRESS>Topic Modelling

Topic models are suitable codebooks because of their readability: each $q\in
Q$ can be manually interpreted based on representative n-grams, and documents
with higher $q$ weights can be read as being more heavily or likely ‘about’
$q$. Of the numerous topic models in the literature, here we experiment with
one hot encoding (i.e. indicators for each n-gram in the corpus’ overall
vocabulary) (“OHE”), LSA, and BTO to cover a range of traditional and emerging
approaches. As topic models are well-documented elsewhere, below we provide a
condensed description of those we test.

LSA first computes a term-frequency/inverse-document-frequency (“TFIDF”)
encoding [40, 67, 21]. The TFIDF matrix is compressed into $m$ desired topics
(explained below) by applying Singular Value Decomposition (“SVD”) and keeping
only features corresponding to the largest $m$ singular values. The SVD of a
matrix $W=U_{m}S_{m}\Lambda_{m}^{T}$ where $rank(W)=m,m\leqslant rank(W)$ [5].
When $W$ is a TF-IDF matrix, $U_{m}$ corresponds to n-gram vectors, $S_{m}$ to
singular values of $W$, and $\Lambda_{m}$ to document vectors [21]. The corpus
is thus represented through $\Lambda_{m}$ as a distribution of $m$ topics
across $n$ documents [46]. These “topics” are represented in $U_{m}$ as
distributions across n-grams. For intuition, observe that an optimal
compression of term frequency matrices should squeeze co-informative terms
together, forming said topics. We use LSA here because of its prominence in
[1]’s influential work on legal outcome prediction for ECHR cases as well as
subsequent related work.

BTO [34] is modular framework which starts with paragraph embeddings typically
derived from a language model. Depending on the LM’s context window, longer
documents may be partitioned into smaller chunks if necessary [70]. Chunk
embeddings undergo dimensionality reduction via a standard algorithm such as
UMAP [52] (the default) or principal components analysis before clustering via
another algorithm such as HDBSCAN [51] or k-Means. Topics are extracted from
these clusters using a bag-of-words vectorizer followed by a “class-based”
TFIDF implementation given by $cTFIDF(c)=||tf_{w,c}||\times
log(1+\frac{A}{f_{w}})$ where $tf_{w,c}$ is the frequency of n-gram $x$ in
cluster $c$, $f_{w}$ is $w$’s’ frequency across all clusters, and $A$ is the
average number of tokens per cluster. This produces an arbitrary number of
topics which can be iteratively merged based on topic frequency and $cTFIDF$
similarity until a desired number remains. The resulting chunk-topic matrix
can then be re-constituted into document-level topics in several ways. For
instance, by assigning a document to the one topic which contains the largest
number of its chunks (i.e. max-pooling). Following [70], we take chunk-topic
counts normalised at document level. We test two BTO models primed with chunk
embeddings from (1) all-MiniLM-L6-v2 [38], a sentence transformer based on
[78] and recommended by [34] (“BTOM”), and (2) legalBERT [12], a BERT [19]
extension fine-tuned on UK, EU, and US legal documents (“BTOL”). Inspiration
for using BTO in the legal context comes from [66] which used a multilingual
MiniLM-embedded BTO model to study Canadian housing law court decisions
written in French.

Here we generate topics comprising $1,2,3$-grams for all topic models. For
LSA, we generally take only the 2500 most frequent n-grams at the TFIDF step
before reducing the matrix to a desired topic number based on corpus size. As
context, [1]’s best predictive models for ECHR cases generally used LSA topics
creating with the 2000 top $1,2,3,4$-grams. However, in our (unreported)
exploratory tests, we noted that 4-grams do not add new interpretable
information as they usually repeated terms already seen in $1,2,3$-grams. We
also set minimum document frequency cutoffs of 5 or 10 (depending on dataset
and topic model) in LSA’s TFIDF and BTO’s cTFIDF steps to limit computational
and memory overheads. Other parameters follow recommendations and defaults
from the sklearn [69] and bertopic [34] libraries.

####<EMAIL_ADDRESS>LASSO Regression and Selective Inference

We use a LASSO [76] regression model to associate topics with outcomes. The
LASSO uses the coefficient vector’s L1-norm as a penalty term when optimising
the model, such that the objective function becomes
$L(\beta)^{*}=L(\beta)-\lambda\lVert\beta_{j}\rVert$ where $\lambda$ is a
user-specified “shrinkage parameter” that controls penalisation magnitude, and
$j>0$ (the intercept is not penalised). The LASSO is suitable for legal
outcome models in three ways. First, as the goal is to discover interpretable
legal topics rather than inexplicably predict legal outcomes, regression
models are preferable to more opaque approaches like neural networks. Second,
the LASSO overcomes two common, related problems with legal outcome models.
First, as text feature matrices are typically large and sparse, and legal
corpora often yield few observations, legal outcome models are prone to the
$k\gg{}n$ problem [82, 35]: as $k$ approaches and eventually exceeds $n$
standard regression models relying on maximum likelihood estimation are liable
to produce biased estimates or failing to converge entirely. Second, legal
areas often present highly imbalanced response classes, forcing us to estimate
“rare events” [81, 4, 63]. For instance, in our UDRP dataset,
$\scriptstyle\mathtt{\sim}$90 percent of the cases are decided in the
complainant’s favour (table LABEL:tab:udrp_stats). Coupled with $k\gg{}n$,
legal outcome models could be perfectly separated — outcomes can be perfectly
predicted with a subset of features — preventing model convergence.

Penalised regressions are one standard countermeasure to both problems [26,
37, 81, 36, 35]. In bioinformatics and chemometrics, LASSO regressions have
been successfully deployed in studies involving large feature matrices and
rare events [79, 80]. Third, the LASSO lets us exploit emerging methods for
selective inference. Conventionally, significance tests are not done with
penalised regressions since regularisation means estimates are biased toward
zero and not consistent [35]. Nonetheless, LASSO regressions were demonstrably
capable of selecting the most significant regressors, particularly in a
$k\gg{}n$ setting (see [35]). More recently, [47] devised a method for
conducting valid post-selection significance tests which [75] extend to the
LASSO. P-values are computed after de-biasing the model post-selection [47,
75]. Coefficient estimates must still be interpreted in light of the penalty,
but p-values and standard errors remain valid and have been shown to be more
reliable than non-adjusted values from subset-selected models [47, 35].
Notably, if as cautioned above we confine ourselves to discovery correlative
rather than causal predictors, significance test validity is less of a
concern. We use [75]’s R package selectiveInference [77] and following their
documentation estimate the LASSO with glmnet [27].

####<EMAIL_ADDRESS>Evaluation

We test several model specifications for the primary UDRP dataset, varying
whether topic features are included and the topic model used (see part 2.3).
Each specification is also evaluated on standard measures of fit including the
area under the receiver operating characteristic curve (“AUROC”) and the
median deviance ratio (“MDR”). The latter summarises all deviance ratios
reported by glmnet along the $\lambda$ fitting path and can be interpreted as
the pseudo-$R^{2}$ [27]. We manually evaluate selected specifications by
delving into topics with the largest positive or negative coefficients and the
smallest p-values for those specifications. The author, who is legally-
trained, then studied the topics’ n-gram distributions and the cases most
strongly associated with them to see how far they corresponded with topics
known to be significant in legal doctrine. Notice that even if they do not,
topics discovered this way could point to some yet unknown X or W driving
legal outcomes. This step should therefore be informed by legal theory. To be
sure, we do not suggest it can be fully-automated, nor that the method is
sufficient to identify all legally-significant topics.

Other than evaluation, the method requires structured data in only two
respects. First, labelled case outcomes are needed. While not considered in
this work, existing methods for automated legal outcome extraction (e.g. [1,
41, 11, 54]) could be incorporated at an earlier pipeline step. Second,
tailored pre-processing work is necessary to sectionise documents and to mask
outcome-leaking information. Other than in these two areas, topics correlated
with legal outcomes are automatically synthesized from the corpus, selected by
the LASSO, and surfaced by post-selection significance tests. Prior domain
knowledge of potential legal predictors within the given legal area is neither
assumed nor required, though it would certainly be a bonus. Likewise, while
structured case metadata is not strictly needed, any available variables can
easily be included as additional covariates at the regression stage.

###<EMAIL_ADDRESS>Datasets

####<EMAIL_ADDRESS>Domain Name Disputes

The UDRP is a mandatory policy instituted in 1999 by the Internet Corporation
for Assigned Names and Numbers (“ICANN”) for resolving disputes over generic
top-level domains (“GTLD”s). Several countries have adopted similar policies
for their country-coded top-level domains (“CCTLDs”) [15]. Disputes are
administered by ICANN-appointed Dispute Resolution Providers (“DRP”s). The
largest DRP by disputes resolved is the World Intellectual Property
Organisation (“WIPO”). Under the Rules for Uniform Domain Name Dispute
Resolution Policy, a case begins when a trademark holder files a complaint
with a DRP. The DRP will ask the respondent for a written response, and
thereafter assemble an adjudication panel of 1 or 3 panellists, depending on
the parties’ preferences. Under UDRP Article 4a, the complainant must show
that (1) the contested domain is “identical or confusingly similar” to the
complainant’s trade or service mark; (2) the respondent does not have any
“rights or legitimate interests” in the contested domain; _and_ the contested
domain was “registered and used in bad faith”. While parties may be
represented by lawyers, all procedures are written and there are no physical
hearings. If a complaint succeeds, the panel may order the domain to be
transferred to the complainant or be cancelled altogether. Decisions are
communicated to and enforced by the relevant domain name registrar [58].

We obtained from WIPO’s online
database222http://www.wipo.int/amc/en/domains/search/ decision texts for WIPO-
administered UDRP disputes decided on and between 1999 and 2016. Regular
expressions were developed, by iterative testing on randomly-sampled
decisions, to partition the texts into 8 archetypal sections. Case outcomes
are typically stated in a final section titled “The Decision”, and
occasionally in a preceding section generally titled “Discussion and
Findings”. The latter details the panel’s legal reasoning and analysis. Both
sections were masked. Outcome labels “transfer”, “cancel”, and “deny” and
linguistic variants thereof were also removed. This left only sections on case
facts, parties involved, procedural history, and arguments presented for
downstream processing. Decisions where fewer than all 8 sections could be
detected, either because they were not in English or because of exceptional or
missing headers, were excluded. This reduced the initially downloaded set of
27,634 raw cases into 22,653 usable observations.333Most of the 4,981 cases
dropped here were non-English cases. This is consistent with official WIPO
statistics, available at https://www.wipo.int/amc/en/domains/statistics, which
show that around 85%–88% of cases each year are in English.

Labelled outcomes and other structured variables were extracted from case
summary tables on WIPO’s website. Each table contains case number, decision
date, the domains, parties, and panellists involved, and outcome. While only
three outcomes (i.e. transfer, cancellation, or complaint denied) are possible
per domain, cases with multiple domains could present mixtures (e.g.
“Complaint denied, transfer in part with dissenting opinion”). Nonetheless,
the vast majority (98.87%) of cases involved singular outcomes. By studying
the data we found that outcome statements start with the outcome assigned to a
majority of the contested domains (i.e. in the example above most domains
would not have been transferred). We thus binarised outcomes by recording 1
when the outcome statement begins with “Transfer” or “Cancellation”, and 0
when it begins with “Complaint denied”. Basic string methods were used to
extract other variables from the tables, including the number of panellists,
complainants, respondents, and domain names involved, whether the case
involved GTLDs or CCTLDs, and year and month indicators. We also created
indicators for repeat complainants (respondents) appearing in >100 (30) cases.

Identity indicators were also created for all panellists. We use this to
demonstrate how the method could be instrumental for studying how judge
identity influences legal outcomes, a staple in “judicial behaviour” research
[24]. Legal scholars have debated the UDRP’s merits [7], with critics alleging
structural pro-complainant biases in the UDRP procedural rules [57, 28, 56,
42]. Proponents [22, 60, 45] countered that critics fail to account for
specific case attributes. Empirical analyses have offered different
explanations for high complainant success rates. [43] argued that case
resolution efficiency was as important as apparent bias in determining
provider choice, while [44]’s used an alternative linear regression
methodology on [43]’s dataset of 2000–2001 cases.

| Complainant Won | Complainant Lost | Overall
---|---|---|---
No. comp’ts | 1.108 | 1.095 | 1.106
| (0.393) | (0.392) | (0.393)
No. resp’ts | 1.094 | 1.077 | 1.092
| (1.301) | (0.349) | (1.232)
No. domain names | 2.009 | 1.407 | 1.942
| (14.749) | (2.188) | (13.921)
Raw word count | 2,812.439 | 3,560.197 | 2,895.986
| (1,309.640) | (1,680.681) | (1,376.408)
Processed word count | 1,293.598 | 1,630.689 | 1,331.261
| (600.045) | (772.365) | (630.659)
GTLD cases (%) | 94.32 | 94.94 | 94.39
Three-member Panel Cases (%) | 3.21 | 24.50 | 5.59
Comp’t Won (%) | 100.00 | 0.00 | 88.83
N | 20122 | 2531 | 22653

Table LABEL:tab:udrp_stats summarises the dataset. It contains information on
significantly more cases and variables than an earlier UDRP corpus compiled by
[8]. On this data we run the penalised logit regression: complainantwon_i =
panelistidentity_i + panelsize_i + textfeatures_i + controls_i + ϵ_i where
$complainantwon_{i}$ is a indicator for complaint success,
$panelistidentity_{i}$ an indicator matrix for panellist involvement,
$panelsize_{i}$ indicates if the case involved three panellists or one,
$textfeatures_{i}$ is either an OHE, LSA, or BTO document-topic matrix.
$controls_{i}$ are indicators for year and month, repeat player involvement,
and whether the case involved GTLDs or CCTLDs. As indirect controls for
dispute complexity, we also included the raw and processed word counts of the
relevant decision, as well as the number of complainants, respondents, and
domain names involved.

To investigate the topic models’ impact, we estimate regressions with/without
topic features across three settings: (A) only 1-panellist GTLD cases, (B) all
GTLD cases, and (C) all cases. We partition the data by panel size and domain
type because these give rise to qualitatively different case types. To
evaluate models in the same regression setting on similar bases, we extract
exactly 250 topics with each topic model. We chose 250 after some iterative
testing with LSA because it represented a 90% compression of the original
TFIDF matrix (recall that the top 2,500 n-grams were used) but, as computed by
the SVD, explained about 61% of the variance in the same. Around the 250 mark,
reducing (increasing) the number of topics led to more (less) than
proportionate losses (gains) in variance explained. We used LSA rather than
BTO models to experiment with topic number because re-estimating BTO models
requires significantly more compute. There is some inevitable arbitrariness
here as identifying the appropriate number of topics is a known challenge in
topic modelling [32]. Future work could study how emerging techniques for
doing so (e.g. [29, 68]) could be incorporated into our method.

All topic models are trained using only decisions within the relevant
partition. This except for BTO chunk embeddings (only the first step) which
are pre-computed only once on the entire corpus and used across all settings,
as the embedding process is computationally expensive. We also pre-computed
the shrinkage parameter $\lambda$ to be used using specifications _without_
text features following the guideline suggested in [61, 1] to set
$\lambda=2\mathbf{E}[\lVert X^{T}\epsilon\rVert_{\infty}]$ where $\epsilon\sim
N(0,\hat{\sigma}^{2})$ and $\hat{\sigma}^{2}$ is the residual sum of squares
from a simple linear regression of $y$ on all regressors. The same $\lambda$s
were then used for mirror specifications _with_ text features. As a further
baseline, we also tested specifications with white noise placebos [53].

####<EMAIL_ADDRESS>European Convention on Human Rights violations

The ECHR establishes fundamental human rights for signatory jurisdictions,
including the prohibition of torture (Article 3), right to a fair trial
(Article 6), and right to respect for private and family life (Article 8). The
European Court of Human Rights (“ECtHR”) adjudicates complaints. The court
publishes decision texts and “case detail” tables on its “HUDOC”
database.444http://hudoc.echr.coe.int/ ECHR cases have been studied in several
prior works [1, 55] and included in the benchmark LexGLUE [13]. While LexGLUE
provides a large number of processed ECHR texts and outcomes, that dataset is
not linked to case identifiers, making topic interpretation challenging. Here
we use [55]’s dataset and replicate their pre-processing steps with their
published code. We limit the analysis to training set cases with clear
violation/non-violation outcomes (i.e. not filed in the dataset as “both”).
Below we focus on Articles 3, 6, and 8 which have the largest number of cases
in this dataset. Following [55], we use only text from the Procedure,
Circumstances, and Relevant Law sections. Table LABEL:tab:echr_stats
summarises the dataset. As our aim was to demonstrate generalisability, unlike
with the UDRP we did not further extract new case variables. The main
specification tested is $violation_{i}=textfeatures_{i}+\epsilon_{i}$ with
$textfeatures_{i}$ being 100 topics synthesised using the above topic models.

Article Contested: | Art 3 | Art 6 | Art 8
---|---|---|---
| Violation | No Violation | Violation | No Violation | Violation | No Violation
Raw Word Count | 4313.676 | 5135.169 | 1768.104 | 3894.557 | 4063.601 | 4909.45
| (3579.273) | (4046.314) | (2054.486) | (2894.177) | (4003.777) | (2897.937)
Proc’d Word Count | 2020.648 | 2424.053 | 821.394 | 1800.873 | 1891.228 | 2277.022
| (1680.586) | (1916.545) | (943.129) | (1344.95) | (1893.032) | (1353.649)
N | 284 | 284 | 454 | 449 | 228 | 229

## esults

###<EMAIL_ADDRESS>UDRP results

Table LABEL:tab:udrp_res summarises our primary results on the UDRP dataset.
Columns 1–3 report baseline estimates computed without any text features for
three main regression settings. Around 50 panellists are significant at
$\alpha=0.05$ across these baselines even with several controls included
(column 3), suggesting an association between their involvement and complaint
outcomes. The association is notably weaker in the corresponding topic
regressions with OHE, LSA, BTOM, and BTOL features added (columns 4, 5–7,
8–10, and 11–13). The topic regressions consistently yield fewer significant
panellists, smaller panellist effects, and better model fits. Statistical
significance can be observed shifting towards the topics instead. This can
already be observed with simple OHE, but is clearest with the LSA regressions,
where few panelists remain significant (9, 13, 8 in columns 5–7 versus 53, 50,
49 in columns 1–3). Across all regression settings, LSA consistently produces
the largest number of significant topics and the highest fit scores. Column
LABEL:tab:udrp_res.7 in particular yields 32 significant topics but only 8
significant panellists at $\alpha=0.05$ and the highest MDR (0.431) and AUROC
(0.914). BTOL and BTOM yield more significant panellists and fewer significant
topics, but are nonetheless superior to the non-text and white noise (column
LABEL:tab:udrp_res.14) baselines, suggesting that these topic models also
capture information on case features. The legally-finetuned BTOL performs
slightly better than BTOM (MDR=0.295, AUROC=0.849 in column 13 in versus
MDR=0.275, AUROC=0.838 in column 10), suggesting that domain adaptation helps.

Topic Model: | None | OHE | LSA
---|---|---|---
Y: Complaint Success (binary) | (1) | (2) | (3) | (4) | (5) | (6) | (7)
Panellists |  |  |  |  |  |  |
Median Coef | -0.0611 | -0.0308 | -0.0306 | -0.037 | -0.0397 | -0.0222 | -0.0241
Median PV | 0.0677 | 0.236 | 0.179 | 0.401 | 0.336 | 0.376 | 0.409
No. Sig. ($\alpha=0.05$) | 53 | 50 | 49 | 19 | 9 | 13 | 8
No. Sig. ($\alpha=0.01$) | 41 | 22 | 27 | 18 | 5 | 6 | 0
No. Sig. ($\alpha=0.001$) | 12 | 8 | 19 | 1 | 1 | 1 | 0
No. Selected | 117 | 159 | 151 | 117 | 68 | 101 | 102
Total No. | 462 | 501 | 510 | 510 | 462 | 501 | 510
Panel Size |  | -0.154 | -0.146 | -0.0755 |  | -0.138 | -0.128*
|  | (0.024) | (0.023) | (0.023) |  | (0.0332) | (0.0236)
Topics |  |  |  |  |  |  |
Median Coef |  |  |  | 0.0374 | 0.0382 | -0.0352 | -0.0363
Median PV |  |  |  | 0.269 | 0.389 | 0.183 | 0.167
No. Sig. ($\alpha=0.05$) |  |  |  | 7 | 24 | 31 | 32
No. Sig. ($\alpha=0.01$) |  |  |  | 2 | 16 | 20 | 14
No. Sig. ($\alpha=0.001$) |  |  |  | 1 | 9 | 8 | 6
No. Selected: |  |  |  | 37 | 92 | 103 | 105
Settinga | A | B | C | C | A | B | C
Median Dev. Ratio | 0.148 | 0.227 | 0.224 | 0.328 | 0.389 | 0.443 | 0.431
AUROC | 0.742 | 0.824 | 0.822 | 0.874 | 0.903 | 0.914 | 0.914
| BTOM | BTOL | Noise
| (8) | (9) | (10) | (11) | (12) | (13) | (14)
Panellists |  |  |  |  |  |  |
Median Coef | -0.0534 | -0.0288 | -0.0298 | -0.0615 | -0.0328 | -0.0363 | -0.0293
Median PV | 0.12 | 0.24 | 0.265 | 0.219 | 0.297 | 0.236 | 0.166
No. Sig. ($\alpha=0.05$) | 39 | 40 | 47 | 25 | 42 | 8 | 52
No. Sig. ($\alpha=0.01$) | 32 | 30 | 26 | 17 | 13 | 5 | 33
No. Sig. ($\alpha=0.001$) | 22 | 16 | 9 | 9 | 2 | 4 | 24
No. Selected | 97 | 141 | 150 | 95 | 143 | 132 | 151
Total No. | 462 | 501 | 510 | 462 | 501 | 510 | 510
Panel Size |  | -0.185 | -0.167 |  | -0.125 | -0.113 | -0.143
|  | (0.0239) | (0.0233) |  | (0.0245) | (0.0234) | (0.023)
Topics |  |  |  |  |  |  |
Median Coef | -0.0267 | 0.027 | -0.0114 | 0.032 | 0.0444 | 0.0295 | 0.0352
Median PV | 0.0854 | 0.351 | 0.317 | 0.206 | 0.654 | 0.348 | 0.58
No. Sig. ($\alpha=0.05$) | 27 | 11 | 11 | 20 | 17 | 15 | 1
No. Sig. ($\alpha=0.01$) | 15 | 4 | 4 | 10 | 10 | 9 | 0
No. Sig. ($\alpha=0.001$) | 6 | 0 | 0 | 4 | 4 | 6 | 0
No. Selected: | 69 | 97 | 96 | 67 | 79 | 85 | 30
Settinga | A | B | C | A | B | C | C
Median Dev. Ratio | 0.222 | 0.274 | 0.274 | 0.215 | 0.29 | 0.295 | 0.237
AUROC | 0.789 | 0.836 | 0.838 | 0.798 | 0.846 | 0.849 | 0.825

These results are relevant to legal debates on whether UDRP processes exhibit
pro-complainant bias. While our correlative models cannot establish the
absence of bias, our findings are consistent with [60, 45]’s argument that
high complaint success rates are better explained by case facts than
structural pro-complainant biases. More importantly, our results suggest that
the pipeline can automatically discover correlative legal predictors from
decision texts. This becomes clearer when inspecting the discovered topics.
The 5 LSA, BTOM, and BTOL topics with smallest p-values in columns 7, 10, and
13 respectively are presented in table LABEL:tab:udrp_words). Some topics are
intuitive. For example, the negative effect associated with LSA 17, a topic
populated by n-gram variations on “administratively deficient”, suggests
logically that “administratively deficient” complaints correlate to worse
complainant outcomes. Manual evaluation revealed that cases with the strongest
weights for this topic indeed involved deficient complaints.555WIPO Case Nos.
D2009-0021, D2011-1484, D2014-2277, D2014-1901, and D2016-0102. 3 of these
complaints were denied. Likewise, the top LSA 3 cases involved situations
where the complainant provided incorrect “contact information” for the domain
registrant and was asked to amend the complaint accordingly.666See WIPO Case
Nos. D2010-0593, D2012-1002, D2012-1761, D2014-1333, and D2016-0341.

Model | Topic | Coefficient |  Representative N-grams
---|---|---|---
LSA | 19 | 0.6069*** |  trade mark, lego, trade, famous, world, wipo case, amendment complaint, amendment, brand
28 | 0.1724*** |  asserts, trade mark, complainant asserts, argues, complainant argues, alleges, complainant alleges, alleges respondent, trade
3 | -0.3724*** |  contact information, registrant contact information, registrant contact, information, amended complaint, amended, amendment, amendment complaint, disclosed
17 | -0.4683*** |  trade mark, trade, amendment, deficient, administratively deficient, administratively, complaint administratively, complaint administratively deficient, amendment complaint
2 | -0.7685*** |  administrative, copy, e-mail, received, icann, notification, administrative panel, registrar domain, registrar domain name
BTOM | 5 | 0.1681** |  armani, ikea, boss, bmw, reg, classes, elite, hugo, hugo boss, international trademark
6 | 0.1517** |  pharmaceutical, sanofi, pfizer, sanofiaventis, aventis, 100 countries, prescription, drug, treatment, weight
11 | 0.1132** |  chase, cme, barclays, financial services, bank, nasdaq, financial, insurance, investment, banking
199 | -0.044* |  videos, sports, action, complaint exhibit, jeff, complaint exhibit respondent, january 31 2000, complainant action, skiing, omit
57 | -0.0747** |  qatar, al, emirates, arabic, discover, project, uae, abu, brothers, dubai
BTOL | 1 | 0.2028*** |  rules paragraph factual, paragraph factual background, paragraph factual, factual background complainant, background complainant, panel submitted statement, ensure compliance rules, impartiality independence required, independence required, ensure compliance
66 | 0.1911*** |  remedy transfer, support case, registered subsequently used, registered subsequently, elements complainant, subsequently used, complainant support, respondent transferred, remedy, policy domain
21 | 0.1645*** |  publicdomainregistrycom, dba publicdomainregistrycom, pvt dba, directi internet solutions, directi internet, internet solutions, directi, internet solutions pvt, solutions pvt, pvt
204 | -0.0672*** |  oy, page displayed, banners, english version, portal, illegally, marks owned, domain names redirect, names redirect, alex
69 | -0.0754*** |  domain hijacking, reverse domain hijacking, hijacking, reverse domain, respondent requests, reverse, finding reverse, respondent requests panel, finding reverse domain, domain hijacking complainant

Other topics are less readable, but their underlying logic can be identified
on closer inspection. For instance, LSA 19 and BTOM 5 are populated by
references to famous trademarks and brands. These topics feature most strongly
in complaints filed by large corporations which owned these and other famous
marks, and typically against individuals who had registered variations on
their brand names.777For LSA see WIPO Case Nos. D2009-1392, D2013-1265,
D2011-0391, D2010-1878, D2015-1445. For BTOM see WIPO Case Nos. D2002-0760,
D2006-0297, D2008-0416, D2011-0022, D2015-1936.. For example, 3 of the top 5
LSA 19 cases involved the “lego” company suing for domains such as
“legosets101.com” and “legowolds.com”. Panels typically found evidence of bad
faith in how respondents could not have registered these domains without
knowing of the complainants’ well-known marks. 9 of the top 10 complaints
succeeded. The exception was a complaint filed by “Hugo Boss” for “boss-
watch.com” and “boss-world.com”. This was denied because the respondent had
been selling watches under the “BOSS” mark in Hong Kong since the 1970s,
before the complainant’s mark was established.888WIPO Case No. D2015-1936.

Consider also BTOL 69, which associates references to “reverse domain name
hijacking” (“RDNH”) with lower complaint success rates. UDRP Rule 1 defines
RDNH as “using the Policy in bad faith to attempt to deprive a registered
domain name holder of a domain name”. When RDNH is found, the complaint fails.
Recall however that the masked texts used in topic modelling exclude the
“Discussion and Findings” and “Decision” sections, so the model should not
have information on whether RDNH occurred. Inspecting the cases here reveals
that RDNH n-grams feature strongly in the included “Contentions” section when
respondents actively defend the claim and raise the RDNH issue. In the usual
case where respondents default, neither panellists nor complainants have
incentives or need to discuss it. Thus while RDNH was not ultimately found in
any of the top 5 BTOL 69 cases, all were rare cases involving active
respondents. This explains the topic’s negative association with complaint
success.

Not every topic can be easily understood. For instance, BTOM 57, represented
by n-grams referencing Middle Eastern countries, indeed involved complainants
from this region.999WIPO Case Nos. D2005-0309, D2008-0835, D2008-0895,
D2009-0133, and D2015-0798. Of these, 2 also involved Middle Eastern
respondents. All 5 complaints were denied, but for differing reasons. In 3
cases the complainant failed to show bad faith because the domain had been
registered before the complainant’s mark was established. Whether complaints
from Middle Eastern parties are properly associated with these facts and with
lower success rates is however unclear. Likewise, BTOL 1 is populated by
n-grams tracking a typical portion in the “Procedural History” section which
states that “the Panel has submitted the Statement of Acceptance and
Declaration of Impartiality and Independence, as required by the Center to
ensure compliance with the Rules, paragraph 7”.101010e.g. WIPO Case Nos.
D2006-0874, D2006-1054, D2011-1122. When this sentence occurs immediately
before the next section header, “Factual Background”, the topic’s n-grams
arise (after stopword removal). Why this correlates with better complainant
outcomes is not clear. It may signal the lack of other procedural issues, such
that panellists can move directly to the next section.111111e.g. WIPO Case No.
D2010-0593., but more qualitative evaluation is needed to ascertain this.

###<EMAIL_ADDRESS>ECHR Results

Table LABEL:tab:echr_res presents results for LSA, BTOM, and BTOL regressions
fit on ECHR case. As with the UDRP, LSA tends to produce higher model fits and
the largest number of significant topics. This especially for Article 3, where
15 LSA topics are significant at ($\alpha=0.05$) compared to 3 BTOM and 4 BTOL
topics. This is notable given that LegalBERT was finetuned on ECHR cases [12].
It is thus not surprising that BTOL again produces higher fit measures than
BTOM, especially for Article 8 (AUROC=0.711 versus 0.638). However, both BTO
models produce broadly similar numbers of selected and significant topics.

Y: Violation Found (binary) | Article 3 | Article 6 | Article 8
---|---|---|---
Topic Model: | LSA | BTOM | BTOL | LSA | BTOM | BTOL | LSA | BTOM | BTOL
Median Coef | -0.164 | -0.161 | -0.144 | 0.153 | -0.185 | -0.119 | -0.005 | 0.041 | -0.129
Median PV | 0.02 | 0.177 | 0.088 | 0.521 | 0.171 | 0.324 | 0.211 | 0.257 | 0.451
No. Sig. ($\alpha=0.05$) | 15 | 3 | 4 | 5 | 2 | 3 | 3 | 1 | 0
No. Sig. ($\alpha=0.01$) | 13 | 1 | 4 | 3 | 1 | 1 | 2 | 0 | 0
No. Sig. ($\alpha=0.001$) | 10 | 0 | 2 | 1 | 1 | 1 | 1 | 0 | 0
No. Selected | 28 | 13 | 9 | 17 | 5 | 21 | 10 | 4 | 16
N | 568 | 568 | 568 | 903 | 916 | 916 | 457 | 458 | 458
$\lambda$ | 15.279 | 24.603 | 23.029 | 22.087 | 42.502 | 22.136 | 20.09 | 25.035 | 17.599
Median Dev. Ratio | 0.508 | 0.327 | 0.326 | 0.384 | 0.223 | 0.266 | 0.344 | 0.253 | 0.286
AUROC | 0.89 | 0.73 | 0.739 | 0.855 | 0.746 | 0.784 | 0.763 | 0.638 | 0.711
Art | Topic | Coefficient |  Representative n-grams (by descending weight)
---|---|---|---
3 | LSA 2 | 0.561*** |  servicemen, abduction, district prosecutor office, district prosecutor, prosecutor office, chechen, chechnya, military, prosecutor
BTOL 1 | 0.4994*** |  abduction, military, prosecutors office, men, relatives, prosecutors, criminal case, identify, forwarded, armed
BTOM 7 | 0.4069* |  athe, bthe, governments, account, version, 1the, events, applicants detention, cthe, 2the
BTOL 21 | -0.4696*** |  unhcr, december 2010, groups, sri, violations, 1951, lanka, sri lanka, international, refugees
LSA 10 | -0.7249*** |  regional, regional court, burned, villages, villagers, village, houses, pkk, appeal
LSA 3 | -1.3226*** |  asylum, country, refugee, board, unhcr, kingdom, forces, united, members
6 | LSA 2 | 1.5781*** |  court, applicant, enforcement, russian, bailiffs, ukraine, ukrainian, uah, enforcement proceedings
LSA 6 | 0.3544* |  ukraine, uah, ukrainian, bailiffs, bailiffs service, debtor, turkish, v., law relevant
LSA 11 | 0.2906** |  applicants, tenant, land, italian, cell, property, detention, possession, treatment
BTOL 79 | -0.2929* |  greek, territory, entry, called, witnesses, level0, level0 arabic, seq level0, seq level0 arabic, arabic
BTOL 0 | -0.3873*** |  3the applicant, alleged, 3the, detention, access, complained, custody, statements, particular, torture
BTOM 0 | -0.5878*** |  imprisonment, years, expert, village, tax, damage, social, seq, medical, regional court
8 | LSA 2 | 0.822*** |  detention, prosecutor, criminal, remand, applicant detention, criminal proceedings, regional, regional court, applicant
LSA 10 | 0.3846** |  hague convention, russian, poland, gypsies,α polish, gypsy,α hague, retention, sites
BTOM 70 | 0.3753* |  correspondence, prohibition, mean, content, shown, february 2003, ask, world, avoid, 9the
LSA 7 | -0.3084* |  prison, detention, expulsion, aliens, residence permit, detained, visits, asylum, cell

Table LABEL:tab:echr_words presents representative n-grams for significant
ECHR topics chosen based on smallest p-value and largest coefficient sizes.
For Article 3 (prohibition torture or inhuman or degrading treatment), LSA 2
and BTOL 1 correctly discover and assign positive effects to what the ECtHR
has described as “a whole series of cases concerning allegations of
disappearances in the Chechen Republic”.121212HUDOC Case No. 001-140017.
Applicants were typically Chechen individuals whose close relatives were
allegedly abducted by state military servicemen. Despite multiple complaints
to and visits from the state’s district prosecutor’s office, the applicants
hear nothing of their relatives for years. The ECtHR has “found on many
occasions” that the distress caused by their relatives’ disappearance and the
state’s indifference to their plight violates Article 3. The top cases for
these topics all involved similar fact patterns131313HUDOC Case Nos.
001-140017, 001-112097, 001-95882, 001-95457, 001-92119, 001-93121,
001-150311, 001-146390, and 001-70853. BTOL 21 captures cases involving
rejected asylum seekers who argued that they faced real risks of being
subjected to treatment violating Article 3 if they were sent home. This
allegedly because of their previous membership in military organisations that
had clashed with their countries’ current governments. “December 2010” is a
significant n-gram because the United Nations High Commissioner for Refugees
had issued updated eligibility guidelines for Afghan asylum seekers then. The
top 5 cases were all complaints from ex-Afghan security service personnel. As
the negative coefficient suggests, these claims were typically denied because,
among other reasons, these guidelines did not include them in their risk
profiles for rights violations.141414HUDOC Case Nos. 001-113328, 001-57451,
001-60924, 001-146372, 001-69022. Notably, there is a similar line of
unsuccessful complaints involving failed asylum seekers who previously served
in the Sri Lankan Tamil Tigers.151515HUDOC Case Nos. 001-102949, 001-102955,
001-102947, 001-102957, 001-104956. These were also picked up by LSA 12 (not
tabulated in table LABEL:tab:echr_words), which is represented by n-grams
including “sri lanka”, “ltte”, and “colombo”. LSA 3 also identifies cases
involving unsuccessful asylum seekers, but includes more varied claims from
individuals originally from Somalia, Iraq, and Libya.161616001-145018,
001-118339, 001-145789, 001-126027, 001-141949. These complaints tended to
fail because the court did not find a sufficiently real risk of treatment
contrary to Article 3.

For Article 6 (right to fair trial), LSA 2 and 6 surface a collection of cases
where Ukrainian individuals awarded compensation judgments against certain
(often state-linked) companies were forced to wait for years before receiving
due payment. They argued that the state bailiff had inordinately delayed
enforcement proceedings. Decisions for such cases are worded very similarly,
and typically reiterate how the ECtHR has “already” or “frequently found”
violations in like cases.171717HUDOC Case Nos. 001-91393, 001-71592,
001-93886, 001-75842, 001-78383, 001-78528, 001-78530, 001-75842, 001-78397,
and 001-70357. Notably, a Ukrainian government judicial enforcement reform
effort acknowledges Article 6 as its
motivation.181818https://www.kmu.gov.ua/en/reformi/verhovenstvo-prava-ta-
borotba-z-korupciyeyu/reformuvannya-sistemi-vikonannya-sudovih-rishen BTOL 79
is diluted by markup n-grams like “level0” but nonetheless identifies several
cases involving Cypriot individuals who had participated in a 1989 anti-
Turkish demonstration in disputed territory arising out of the 1974 Turkish
intervention in North Cyprus.191919HUDOC Case Nos. 001-169203, 001-61582,
001-139903, 001-113876, 001-139995. They were charged and convicted in the
Turkish courts for entering Turkish territory without permission. Typically
they argued that their Article 6 rights had been violated because the legal
proceedings were generally in Turkish, not Greek which they understood. The
ECtHR generally rejected these claims because the applicants had reasonable
access to interpreters and other legal assistance.

For Article 8 (right to respect for family and private life, home, and
correspondence), LSA 2 points to a line of cases filed against the Polish
authorities by individuals detained in criminal remand over the authorities’
standard practice of reading and re-sealing correspondence sent by these
individuals to the courts and stamping the envelopes with a “censored” label.
The ECtHR has noted how it has “held on many occasions” that the label forces
the court to assume an interference with correspondence that, unless
justified, violates Article 8.202020HUDOC Case No. 001-93604 at paragraph 94.
2 of the top 5 BTOM 70 cases have similar facts, but the topic also seems to
cover other kinds of interferences to correspondence. Relatedly, the top 5 LSA
10 cases all involve complaints filed by Romani persons against the British
government and its consistent refusal to grant them planning permission to
develop land they owned into caravan sites. After the ECtHR found this to be a
violation in 1996,212121HUDOC Case No. 001-58076. several similar and
ultimately successful cases were raised in which the ECtHR expressly “recalls
that it has already examined” such complaints and found violations.222222HUDOC
Case Nos. 001-59156, 001-59154, 001-59158, 001-59157.

## iscussion

We proposed and evaluated an automated pipeline for discovering significant
topics from cases by performing penalised regression and selective inference
on features synthesized from decision texts using topic models. We show that
significant topics discovered through this process capture relevant
information on factual patterns correlated with case outcomes. On a large (by
legal standards) dataset of UDRP cases, legal outcome models fit with decision
text topics consistently produce higher fit scores compared with models fit
without (table LABEL:tab:udrp_res). The LASSO also tends to select the topics
as significant predictors over other structured case attributes of potential
interest, such as judge identities. Coefficients and p-value estimates also
change noticeably. This holds across several regression settings and topic
modeling approaches. Using a canonical dataset of ECHR cases, we show that the
method generalises relatively easily, without the need for additional feature
engineering or pre-processing. Only structured outcome information and
unstructured decision texts are required, though additional variables can be
added at the regression step. Running similar procedures on the existing
dataset, albeit with corpus-tailored hyperparameters (such as topic number and
$\lambda$), yields significant topics consistent with ECHR case law.

Our experiments show that LSA is a useful, even if dated, codebook for
decision texts. Across all experiment settings, LSA produced higher fit scores
and a higher number of significant topics than both BTO models. LSA is also
computationally cheaper. This may appear counter-intuitive since BTO is a
significantly more sophisticated model which exploits recent advances like
word embeddings and language modelling. As noted by [73] in the context of
legal topic classification, the length of legal decision texts may offer one
explanation for LSA out-performing newer approaches in the legal domain. BTO’s
superiority over traditional topic models has mainly been demonstrated on
shorter texts like tweets and news articles [34, 18]. For longer documents,
our present approach of reconstituting document-level topics by normalising
chunk-level topic counts is, while standard in the literature [70], unlikely
to be the optimal way to deploy BTO. Using LMs with larger context windows
than those tested here could significantly improve BTO’s performance. Thus, we
do not suggest that LSA is necessarily better-suited for this task. Further,
since each topic model yields different topics which may provide different
insights on the cases, there is no clear metric for “better” in this context.
Further, BTO’s lower fit scores may be a methodological artefact since we did
not conduct hyperparameter optimisation, but chose similar parameters across
all topic models to establish a baseline comparison. A fully-optimised BTO
model may outperform a fully-optimised LSA. Hyperparameter tuning was not done
because, unlike typical machine learning settings, our task focuses on
explanation rather than classification and does not yield any clear
performance metric (e.g. F1 score) for evaluating a grid search. BTO’s
performance here should be interpreted in this light.

More importantly, qualitative evaluation of significant topic n-grams
demonstrates that the discovered topics rest on sound and interpretable legal
bases. For UDRP cases, the topics shed correlative light on how
administratively deficient complaints are less likely to win, how famous
trademark owners can be associated with higher success rates, and how
respondents who actively defend their domains are less likely to lose. For
Arts 3, 6, and 8 ECHR, the topics identify archetypal cases involving abducted
Chechen relatives, Afghan asylum seekers, Ukrainian judgment enforcers,
Cypriot demonstrators, Polish detainees, and Romani land owners. These are
correctly associated with their usual case outcomes. Essentially, unique case
features prompt judges into writing decisions with a higher preponderance of
correspondingly unique n-grams, producing signals which the topic models are
capable of detecting. As legal decisions are written with close reference to
case facts and relevant laws, and judges would generally not write about
irrelevant matters and non-issues, we theorise that the decision text
generation function accords with the standard topic modelling assumption that
texts are generated by sampling n-grams from latent topics [6].

To be sure, not every discovered topic made sense. This may point to
limitations in our evaluation process, since we only sampled the top 5 cases
associated with the most significant topics. We may also have been unable to
detect known patterns that the topics were in fact referencing. Further, there
is no reason why each topic should capture exactly one predictor. Certain
topics may have had n-gram distributions amalgamating several archetypal case
features. Individually insignificant topics could have been jointly
significant with others. Future work could examine this further by modeling
interaction terms and conducting joint tests, though this may make
interpreting the topics and coefficients more challenging for evaluators.
Besides human limitations, the automated process is also imperfect. It can
produce false positives (e.g. significant topics which do not actually capture
legal predictors; high document-topic weights for a case not actually on
topic) and false negatives (not attaching significance to topics which do; not
synthesising a related topic to begin with).

At a more abstract level, the four pipeline steps can be understood as a
series of dimensionality reduction steps, starting with a large decision
corpus, and resulting in numerical associations between decisions and topics
(document-topic weights), topics and outcomes (regression coefficients), and
topics and words (n-gram distributions). These mappings can be analysed
transitively to assist with the discovery, extraction, and analysis tasks
identified in part 2.1. To illustrate, after observing UDRP LSA topic 19
(table LABEL:tab:udrp_words), researchers could create and extract
observations for an indicator variable for whether the complainant owned a
famous mark. Decision-topic weights produced by the method could guide the
extraction process. After several such variables are extracted, a regression
(not necessarily the LASSO) could then be run on the reduced dataset. Notably,
given the increasing popularity of large language models, the pipeline’s
ability to reduce large legal corpora to smaller components could prove useful
in fitting legal texts into limited context windows.

## onclusion

We proposed and assessed an automated method for discovering significant
topics given only decision texts and case outcomes, building on prior work
examining how topic models can be used to predict and explain case outcomes
[1, 25, 66]. The task of legal topic discovery was formally defined and
distinguished from related identification and analysis tasks. We developed and
demonstrated pre-processing, topic modelling, regression and inference steps
tailored to this task and its legal context. The method shows promise in its
ability to discover archetypal case features and patterns consistent with the
jurisprudence of the UDRP and ECHR datasets tested, and could generalise to
other areas. It is however not perfect, and should be applied bearing the
possibility of false positives and negatives in mind. There are two extensions
we hope to pursue in future work. First, to conduct more rigorous experiments
and hyper-parameter search with BERTopic and its variations. Notably, BERTopic
is a modular framework involving six steps that accept several different
algorithms and (optional) parameters. Second, a more robust yet ideally less
manual method for evaluating and interpreting topics could be developed.

## References

  * [1] Nikolaos Aletras, Dimitrios Tsarapatsanis, Daniel Preoţiuc-Pietro, and Vasileios Lampos.  Predicting judicial decisions of the european court of human rights: A natural language processing perspective.  PeerJ Computer Science, 2, 2016.
  * [2] Wolfgang Alschner, Joost Pauwelyn, and Sergio Puig.  The data-driven future of international economic law.  Journal of International Economic Law, 20(2):217–231, 2017.
  * [3] Kevin D. Ashley and Stefanie Brüninghaus.  Predicting outcomes of case based legal arguments.  Artificial Intelligence and Law, 17:125–165, 2009.
  * [4] Concha Bielza, Víctor Robles, and Pedro Larrañaga.  Regularized logistic regression without a penalty term: An application to cancer classification with microarray data.  Expert Systems with Applications, 38(5):5110–5118, 2011.
  * [5] Christopher Bishop.  Pattern Recognition and Machine Learning.  Springer Verlag, 2006.
  * [6] David M. Blei, Andrew Y. Ng, and Michael I. Jordan.  Latent dirichlet allocation.  Journal of Machine Learning Research, 3:993–1022, Mar 2003.
  * [7] Colm Brannigan.  The UDRP: How do you spell success?  Digital Technology Law Journal, 5(1), 2004.
  * [8] Karl Branting, Craig Pfeifer, Bradford Brown, Lisa Ferro, John Aberdeen, Brandy Weiss, Mark Pfaff, and Bill Liao.  Scalable and explainable legal prediction.  Artificial Intelligence and Law, 29:213–238, 2021.
  * [9] Ilaria Canavotto and John Horty.  Reasoning with hierachies of open-textured predicates.  In Proceedings of the International Conference on Artificial Intelligence and Law, pages 52–61, June 2023.
  * [10] David J Carter, James J. Brown, and Adel Rahmani.  Reading the high court at a distance: Topic modelling the legal subject matter and judicial activity of the high court of australia, 1903-2015.  University of New South Wales Law Journal, 39(4):1300–1354, 2016\.
  * [11] Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras.  Neural legal judgment prediction in English.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4317–4323, Florence, Italy, July 2019. Association for Computational Linguistics.
  * [12] Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos.  LEGAL-BERT: The muppets straight out of law school.  In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2898–2904, Online, November 2020. Association for Computational Linguistics.
  * [13] Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, and Nikolaos Aletras.  LexGLUE: A benchmark dataset for legal language understanding in English.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4310–4330, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  * [14] Daniel L. Chen.  Judicial analytics and the great transformation of american law.  Artificial Intelligence and Law, pages 1–28, forthcoming.
  * [15] W. B. Chik.  Lord of your domain, but master of none: The need to harmonize and recalibrate the domain name regime of ownership and control.  International Journal of Law and Information Technology, 16(1):8–72, 2007.
  * [16] Jonathan Choi.  An empirical study of statutory interpretation in tax law.  New York University Law Review, 95:363–441, 2020.
  * [17] Carlos Cinelli, Andrew Forney, and Judea Pearl.  A crash course in good and bad controls.  Sociological Methods & Research, page 004912412210995, May 2022\.
  * [18] Muriël de Groot, Mohammad Aliannejadi, and Marcel R. Haas.  Experiments on generalizability of bertopic on multi-domain short text, 2022.
  * [19] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [20] J. Dhanani, R. Mehta, and D. Rana.  Effective and scalable legal judgment recommendation using pre-learned word embedding.  Complex and Intelligent Systems, 8:3199––3213, 2022.
  * [21] Sandor Dominich.  Mathematical Foundations of Information Retrieval.  Springer Science Business Media Dordrecht, 2001.
  * [22] M. Scott Donahey.  The udrp - fundamentally fair, but far from perfect.  Electronic Commerce & Law Reports, 6(34), August 2001.
  * [23] Arthur Dyevre, Monika Glavina, and Michal Ovádek.  The voices of european law: Legislators, judges and law professors.  German Law Journal, 22(6):956–982, 2021.
  * [24] Lee Epstein and Keren Weinshall.  The Strategic Analysis of Judicial Behavior.  Cambridge University Press, May 2021.
  * [25] Mohammad Hassan Falakmasir and Kevin D. Ashley.  Utilizing vector space models for identifying legal factors from text.  In Frontiers in Artificial Intelligence and Applications: Legal Knowledge and Information Systems, volume 302, pages 183–192, 2017.
  * [26] David Firth.  Bias reduction of maximum likelihood estimates.  Biometrika, 80(1):27–38, 1993.
  * [27] Jerome Friedman, Trevor Hastie, and Robert Tibshirani.  Regularization paths for generalized linear models via coordinate descent.  Journal of Statistical Software, 33(1):1–22, 2010.
  * [28] Michael A. Geist.  Fair.com?: An examination of the allegations of systemic unfairness in the icann udrp.  Brookings Journal of International LAw, 27, 2002.
  * [29] Martin Gerlach, Tiago P. Peixoto, and Eduardo G. Altmann.  A network approach to topic models.  Science Advances, 4(7), July 2018.
  * [30] Morgan Gray, Jaromir Savelka, Wesley Oliver, and Kevin Ashley.  Toward automatically identifying legally relevant factors.  In Frontiers in Artificial Intelligence and Applications: Legal Knowledge and Information Systems, volume 302, pages 53–62. IOS Press, December 2022.
  * [31] Morgan Alexander Gray, Jaromir Savelka, Wesley Oliver, and Kevin Ashley.  Automatic identification and empirical analysis of legally relevant factors.  In Proceedings of the International Conference on Artificial Intelligence and Law, pages 101–110, June 2023.
  * [32] Derek Greene, Derek O’Callaghan, and Pádraig Cunningham.  How Many Topics? Stability Analysis for Topic Models.  In ECML/PKDD, volume 8724, pages 498–513, Berlin, Heidelberg, 2014. Springer Berlin Heidelberg.  Series Title: Lecture Notes in Computer Science.
  * [33] Justin Grimmer and Brandon M. Stewart.  Text as data: The promise and pitfalls of automatic content analysis methods for political texts.  Political Analysis, 21(03):267–297, 2013.
  * [34] Maarten Grootendorst.  Bertopic: Neural topic modeling with a class-based tf-idf procedure, 2022\.
  * [35] Trevor Hastie, Robert Tibshirani, and Jerome H. Friedman.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction.  Springer, 2017.
  * [36] Trevor Hastie, Robert Tibshirani, and Martin Wainwright.  Statistical Learning with Sparsity: The Lasso and Generalizations.  CRC Press, Taylor & Francis Group, 2015.
  * [37] G. Heinze.  The application of firth’s procedure to cox and logistic regression.  Technical Report 10, 1999.  Section of Clinical Biometrics, Department of Medical Computer Sciences, Medical University of Vienna, Vienna, Austria.
  * [38] HuggingFace.  all-minilm-l6-v2, 2021.  https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2.
  * [39] Sahan Jayasinghe, Lakith Rambukkanage, Ashan Silva, Nisansa de Silva, Shehan Perera, and Madhavi Perera.  Learning sentence embeddings in the legal domain with low resource settings.  In Proceedings of the 36th Pacific Asia Conference on Language, Information and Computation, pages 494–502, Manila, Philippines, October 2022\. De La Salle University.
  * [40] Karen Sparck Jones.  A statistical interpretation of term specificity and its application in retrieval.  Journal of Documentation, 28(1):11–21, 1972.
  * [41] Daniel Martin Katz, Michael J. Bommarito, and Josh Blackman.  A general approach for predicting the behavior of the supreme court of the united states.  PLoS One, 12(4), Dec 2017.
  * [42] Patrick D. Kelley.  Emerging patterns in arbitration under the uniform domain-name dispute-resolution policy.  Berkeley Technology Law Journal, 17(1):181–204, Jan 2002.
  * [43] Jay P. Kesan and Andres A. Gallo.  The market for private dispute resolution services - an empirical re-assessment of icann-udrp performance.  Michigan Telecommunications & Technology Law Review, 11:285–380, 2005.
  * [44] Daniel Klerman.  Forum selling and domain-name disputes.  Loyola University Chicago Law Journal, 48:561–584, 2017.
  * [45] Annette Kur.  UDRP.  Max-Planck-Institute for Foreign and International Patent, Copyright and Competition Law, Munich, 2002.
  * [46] T. K. Landauer, P. W. Foltz, and D. Laham.  Introduction to latent semantic analysis.  Discourse Processes, 25:259–284, 1998.
  * [47] Jason D. Lee, Dennis L. Sun, Yuekai Sun, and Jonathan E. Taylor.  Exact post-selection inference, with application to the lasso.  The Annals of Statistics, 44(3):907–927, 2016.
  * [48] Daniele Licari, Praveen Bushipaka, Gabriele Marino, Giovanni Comandè, and Tommaso Cucinotta.  Legal holding extraction from italian case documents using italian legal-bert text summarization.  In Proceedings of the International Conference on Artificial Intelligence and Law, pages 148–156, June 2023.
  * [49] Zhenyu Liu and Huanhuan Chen.  A predictive performance comparison of machine learning models for judicial cases.  IEEE Symposium Series on Computational Intelligence, pages 1–6, 2017.
  * [50] L. Thorne McCarty.  Deep semantic interpretations of legal texts.  In Proceedings of the International Conference on Artificial Intelligence and Law, pages 217–224, 2007.
  * [51] Leland McInnes, John Healy, and Steve Astels.  hdbscan: Hierarchical density based clustering.  The Journal of Open Source Software, 2(11), mar 2017.
  * [52] Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger.  Umap: Uniform manifold approximation and projection.  Journal of Open Source Software, 3(29):861, 2018.
  * [53] Blakeley B. McShane and Abraham J. Wyner.  A statistical analysis of multiple temperature proxies: Are reconstructions of surface temperatures over the last 1000 years reliable?  Ann. Appl. Stat., 5(1):5–44, 03 2011.
  * [54] Masha Medvedeva, Michel Vols, and Martijn Wieling.  Using machine learning to predict decisions of the european court of human rights.  Artificial Intelligence and Law, 28:237–266, 2020.
  * [55] Masha Medvedeva, Martijn Wieling, and Michel Vols.  Rethinking the field of automatic prediction of court decisions.  Artificial Intelligence and Law, 31:195–212, 2023.
  * [56] M. Mueller.  Success by Default: A New Profile of Domain Name Trademark Disputes Under ICANN’s UDRP.  Syracuse University School of Information Studies, 2002.
  * [57] Milton Mueller.  Rough justice: A statistical assessment of icanns uniform dispute resolution policy.  The Information Society, 17(3):151–163, Jan 2001.
  * [58] Milton Mueller.  Ruling the Root: Internet Governance and the Taming of Cyberspace.  MIT Press, 2002.
  * [59] Adeline Nazarenko, François Lévy, and Adam Wyner.  A Pragmatic Approach to Semantic Annotation for Search of Legal Texts – An Experiment on GDPR.  In Proceedings of JURIX, pages 23–32. IOS Press, December 2021\.
  * [60] Branthover Ned.  UDRP — A Success Story: A Rebuttal to the Analysis and Conclusions of Professor Milton Mueller in “Rough Justice”.  International Trademark Association, 2002.
  * [61] Sahand Negahban, Bin Yu, Martin J Wainwright, and Pradeep K. Ravikumar.  A unified framework for high-dimensional analysis of m-estimators with decomposable regularizers.  In Y. Bengio, D. Schuurmans, J. D. Lafferty, C. K. I. Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems 22, pages 1348–1356. Curran Associates, Inc., 2009.
  * [62] Shounak Paul, Arpan Mandal, Pawan Goyal, and Saptarshi Ghosh.  Pre-trained language models for the legal domain: A case study on indian law.  In Proceedings of the International Conference on Artificial Intelligence and Law, pages 187–196, June 2023.
  * [63] Menelaos Pavlou, Gareth Ambler, Shaun Seaman, Maria De Iorio, and Rumana Z Omar.  Review and evaluation of penalised regression methods for risk prediction in low-dimensional data with few events.  Statistics in Medicine, 35(7), 2015.
  * [64] Brooke Rowland.  Writing a Statement of Facts in an Appellate Brief, 2014.  Retrieved 30 June 2023 at https://www.law.georgetown.edu/wp-content/uploads/2018/07/StatementofFactsinaBriefFinal.pdf.
  * [65] Theodore W. Ruger, Pauline T. Kim, Andrew D. Martin, and Kevin M. Quinn.  The supreme court forecasting project: Legal and political science approaches to predicting supreme court decisionmaking.  Columbia Law Review, 104(4):1150, 2004.
  * [66] Olivier Salaün, Fabrizio Gotti, Philippe Langlais, and Karim Benyekhlef.  Why Do Tenants Sue Their Landlords? Answers from a Topic Model.  In Frontiers in Artificial Intelligence and Applications: Legal Knowledge and Information Systems. IOS Press, December 2022.
  * [67] Gerard Salton and Christopher Buckley.  Term-weighting approaches in automatic text retrieval.  Information Processing & Management, 24(5):513–523, 1988.
  * [68] Stefano Sbalchiero and Maciej Eder.  Topic modeling, long texts and the best number of topics. Some Problems and solutions.  Quality & Quantity, 54(4):1095–1108, August 2020.
  * [69] Scikit-Learn.  Truncated Singular Value Decomposition and Latent Semantic Analysis, 2017.  Retrieved 30 June 2023 at http://scikit-learn.org/stable/modules/decomposition.html#lsa.
  * [70] Raquel Silveira, Carlos G. O. Fernandes, João A. Monteiro Neto, Vasco Furtado, and José Ernesto Pimentel Filho.  Topic modelling of legal documents via legal-bert.  In CEUR Workshop Proceedings, volume 2896, 2021.
  * [71] Jerrold Soh.  A network analysis of the singapore court of appeal’s citations to precedent.  Singapore Academy of Law Journal, 31:246–284, 2019.
  * [72] Jerrold Soh.  Causal Inference with Legal Texts.  MIT Computational Law Report, dec 7 2021.  https://law.mit.edu/pub/causalinferencewithlegaltexts.
  * [73] Jerrold Soh, How Khang Lim, and Ian Ernst Chai.  Legal Area Classification: A Comparative Study of Text Classifiers on Singapore Supreme Court Judgments.  In Proceedings of the Natural Legal Language Processing Workshop 2019, pages 67–77, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [74] Octavia-Maria Sulea, Marcos Zampieri, Mihaela Vela, and Josef Van Genabith.  Predicting the law area and decisions of french supreme court cases.  Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, pages 716–722, Oct 2017.
  * [75] Jonathan Taylor and Robert Tibshirani.  Post-selection inference for l1-penalized likelihood models.  Canadian Journal of Statistics, 46(1):41–61, Jun 2017.
  * [76] Robert Tibshirani.  Regression shrinkage and selection via the lasso.  Journal of the Royal Statistical Society, Series B, 58:267–288, 1994.
  * [77] Ryan Tibshirani, Rob Tibshirani, Jonathan Taylor, Joshua Loftus, and Stephen Reid.  selectiveInference: Tools for Post-Selection Inference, 2017.  R package version 1.2.4.
  * [78] Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou.  MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.  In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA, 2020. Curran Associates Inc.
  * [79] Tong Tong Wu, Yi Fang Chen, Trevor Hastie, Eric Sobel, and Kenneth Lange.  Genome-wide association analysis by lasso penalized logistic regression.  Bioinformatics, 25(6):714–721, 2009.
  * [80] Ying Zhu, Tuck Lee Tan, and Wai Kwong Cheang.  Penalized logistic regression for classification and feature selection with its application to detection of two official species of ganoderma.  Chemometrics and Intelligent Laboratory Systems, 171:55–64, 2017\.
  * [81] Christopher Zorn.  A solution to separation in binary response models.  Political Analysis, 13:157–170, 2005.
  * [82] Hui Zou and Trevor Hastie.  Regularization and variable selection via the elastic net.  Journal of the Royal Statistical Society. Series B (Statistical Methodology), 67(2):301–320, 2005.