lxyuan/LegalBenchRAG-Ettin-150M-Reranker

LegalBench-RAG Ettin 150M Reranker

This is a 150M-parameter cross-encoder for reranking evidence passages from contracts and privacy policies.

Use it after a first-stage retriever such as BM25 or an embedding model. The retriever finds candidate passages, then this model scores and reorders them so the most relevant legal evidence appears first.

The model was fine-tuned from cross-encoder/ettin-reranker-150m-v1 using the complete LegalBench-RAG dataset.

Results

Evaluation used 710 held-out questions. Documents in the test set were not used during training.

Model	NDCG@10	MRR@10	Hit@5	Character recall@5
BM25	0.3057	0.2582	0.4563	0.3965
Base Ettin model	0.3853	0.3577	0.5930	0.5035
Fine-tuned model	0.7424	0.8191	0.8676	0.8041

Fine-tuning improved NDCG@10 by 0.3570 over the base model. Performance improved in all four LegalBench-RAG domains.

Intended use

This model is designed for the second stage of a legal retrieval workflow:

1. Split legal documents into passages.
2. Retrieve an initial set of candidates with BM25 or an embedding model.
3. Give the query and candidate passages to this model.
4. Use the reordered passages for review, search results, or grounded generation.

The model returns a relevance score for each query-passage pair. It does not search the document collection itself.

It also does not answer legal questions, execute actions, or provide legal advice.

Usage

from sentence_transformers import CrossEncoder

model = CrossEncoder(
    "lxyuan/LegalBenchRAG-Ettin-150M-Reranker"
)

query = "When may either party terminate the agreement?"

passages = [
    "Either party may terminate this Agreement with thirty days written notice.",
    "Confidential information must be protected for five years.",
    "Invoices are payable within sixty days after receipt.",
]

ranked = model.rank(
    query,
    passages,
    return_documents=True,
)

for result in ranked:
    print(float(result["score"]), result["text"])

ranked is ordered from the most relevant passage to the least relevant passage.

The scores are useful for comparing passages for the same query. They are raw ranking scores, not calibrated probabilities.

Dataset

The model was trained using the complete LegalBench-RAG release, not the smaller 776-question mini benchmark.

The complete dataset contains:

• 6,889 questions
• 714 source documents
• 10,928 expert-annotated evidence spans
• 4 legal-document domains

Domain	Questions	Documents
ContractNLI	977	95
CUAD	4,042	462
MAUD	1,676	150
PrivacyQA	194	7

Preventing document leakage

The dataset was split by source document.

Questions about the same contract or privacy policy always remain in the same split. This prevents questions about one document from appearing in both training and evaluation.

Every annotated evidence span was also checked against the corresponding text in the source document.

The model repository does not redistribute the source contracts, privacy policies, or passage text.

Data preparation

A LegalBench-RAG example contains a question and one or more evidence spans inside a source document.

A simplified source row looks like this:

{
  "query": "What law governs this agreement?",
  "snippets": [
    {
      "file_path": "cuad/example.txt",
      "span": [
        120,
        184
      ],
      "answer": "This Agreement is governed by New York law."
    }
  ]
}

Source documents were divided into overlapping passages:

• Passage size: 384 tokens
• Passage overlap: 96 tokens
• Maximum query-passage length: 512 tokens

For every evidence span, the passage with the highest character overlap was used as a positive example.

BM25 selected up to four high-ranking passages from the same document that did not overlap the evidence. These became hard negative examples.

A prepared training row looks like this:

{
  "query": "What law governs this agreement?",
  "passage": "This Agreement is governed by New York law.",
  "label": 1.0
}

A hard negative has the same structure with "label": 0.0.

For evaluation, BM25 retrieved the top 32 passages for each question. Missing positive passages were not added to the candidate list, so the evaluation reflects a realistic retrieve-and-rerank workflow.

Model and training objective

This is a standalone cross-encoder model, not a LoRA adapter.

The query and passage are processed together. The model produces one relevance logit for each pair.

Training used BinaryCrossEntropyLoss, also known as binary cross-entropy with logits:

• Positive query-passage pairs were trained toward label 1.0.
• Hard negative pairs were trained toward label 0.0.
• Every labeled training pair contributed to the loss.

FP16 reduced memory and computation requirements. It did not change which examples contributed to the loss.

Training configuration

Setting	Value
Base model	cross-encoder/ettin-reranker-150m-v1
Base revision	025501c4e0f9bbeb4c5b198318e0089ff061cc14
Model size	150M parameters
Training type	Full fine-tuning
Training epochs	3
Best epoch	2
Maximum pair length	512 tokens
Passage size	384 tokens
Passage overlap	96 tokens
Training batch size	8
Gradient accumulation	4
Effective batch size	32
Learning rate	2e-05
Warmup	10%
Optimizer	AdamW
Learning-rate schedule	Linear
Precision	FP16
Hardware	NVIDIA T4
Seed	42

The recorded 26.3-minute runtime covers the final resumed training segment. It does not represent the total end-to-end runtime, which also included data preparation, baseline evaluation, earlier epochs, final evaluation, and model publication.

Choosing the best epoch

The model was trained for three epochs. Epoch 2 produced the best validation NDCG@10, so those weights were restored and published.

Epoch	Validation loss	Validation NDCG@10
1	0.0765	0.7825
2	0.0722	0.7861
3	0.1004	0.7820

The validation loss increased during epoch 3 while ranking quality decreased slightly.

We did not test the exact cause. The result only shows that the third epoch did not improve performance on the validation set. Publishing the epoch 2 checkpoint avoids using the weaker final checkpoint.

Evaluation

All systems were evaluated on the same held-out documents and the same BM25 candidate lists.

The main metrics are:

• NDCG@10: measures whether relevant passages appear near the top.
• MRR@10: measures the rank of the first relevant passage.
• Hit@5: measures whether the top five results contain relevant evidence.
• Character recall@5: measures how much annotated evidence is covered by the top five passages.

Results by domain

Domain	Base NDCG@10	Fine-tuned NDCG@10	Improvement
ContractNLI	0.7519	0.9767	+0.2249
CUAD	0.3875	0.8356	+0.4480
MAUD	0.1686	0.3895	+0.2209
PrivacyQA	0.4452	0.7324	+0.2872

The experiment's success rule required:

• An overall NDCG@10 improvement of at least 0.02
• No domain regression worse than -0.02

The model passed both conditions.

Limitations

• The model supports English text only.
• Training data focuses on NDAs, commercial contracts, merger and acquisition agreements, and consumer privacy policies.
• A reranker cannot recover relevant evidence that the first-stage retriever failed to retrieve.
• Benchmark questions may differ from the language used by real users.
• Relevance scores are not calibrated confidence values.
• A high score does not mean that a passage is legally correct.
• The model should be evaluated on the intended document types, jurisdictions, and retrieval system before production use.
• Human legal review is still required for legal decisions.

Reproducing the experiment

The complete training script is available here:

finetune_legalbenchrag_ettin_reranker.py

Run the full experiment with:

uv run --script scripts/finetune_legalbenchrag_ettin_reranker.py

The default configuration is designed for an NVIDIA T4 or a comparable CUDA GPU.

Tested environment

Package	Version
accelerate	1.14.0
datasets	5.0.1
huggingface-hub	1.30.0
numpy	2.3.3
rank-bm25	0.2.2
sentence-transformers	6.0.1
tensorboard	2.21.0
torch	2.14.0
transformers	5.16.1

Data and license

LegalBench-RAG builds on ContractNLI, CUAD, MAUD, and PrivacyQA.

ContractNLI, CUAD, and MAUD are distributed under CC BY 4.0. Review the original dataset licenses and the LegalBench-RAG repository before reusing the data.

The published model weights use the Apache 2.0 license, matching the base model.