mjbommar/magic-bert-50m-roformer-mlm

Magic-BERT 50M RoFormer MLM

A RoFormer-based transformer model trained for binary file understanding using masked language modeling (MLM). This model uses Rotary Position Embeddings (RoPE) for position encoding, which provides better relative position modeling than absolute embeddings.

Why Not Just Use libmagic?

For intact files starting at byte 0, libmagic works well. But libmagic matches signatures at fixed offsets. Magic-BERT learns structural patterns throughout the file, enabling use cases where you don't have clean file boundaries:

• Network streams: Classifying packet payloads mid-connection, before headers arrive
• Disk forensics: Identifying file types during carving, when scanning raw disk images without filesystem metadata
• Fragment analysis: Working with partial files, slack space, or corrupted data
• Adversarial contexts: Detecting file types when magic bytes are stripped, spoofed, or deliberately misleading

Model Description

This model uses the HuggingFace RoFormer architecture with a byte-level BPE tokenizer trained on binary file data. RoPE encodes position information directly in the attention computation through rotation matrices.

Property	Value
Parameters	42.3M
Hidden Size	512
Layers	8
Attention Heads	8
Max Sequence Length	512 tokens
Vocabulary Size	32,768 (byte-level BPE)
Position Encoding	RoPE (Rotary Position Embeddings)

Tokenizer

The tokenizer uses the Binary BPE methodology introduced in Bommarito (2025). The original Binary BPE tokenizers (available at mjbommar/binary-tokenizer-001-64k) were trained exclusively on executable binaries (ELF, PE, Mach-O). This tokenizer uses the same BPE training approach but was trained on a diverse corpus spanning 106 file types.

Intended Uses

Primary use cases:

• Fill-mask: Predicting missing bytes in binary files
• Magic byte and file signature recognition
• Feature extraction for downstream classification
• Research on binary file structure with relative position encoding

Example tasks:

• Completing partial file headers
• Identifying file type from structure
• Exploring position-independent file patterns

Detailed Use Cases

Network Traffic Analysis

When inspecting packet payloads, you often see file data mid-stream—TCP reassembly may give you bytes 1500-3000 of a PDF before you ever see byte 0. Traditional signature matching fails here. Structural understanding can identify file types from interior content.

Disk Forensics & File Carving

During disk image analysis, you scan raw bytes looking for file boundaries. Tools like Scalpel rely on header/footer signatures, but many files lack clear footers. This model can score byte ranges for file type probability, helping identify carved fragments or validate carving results.

Incident Response

Malware often strips or modifies magic bytes to evade detection. Polyglot files (valid as multiple types) exploit signature-based tools. Learning structural patterns provides a second opinion that doesn't rely solely on the first few bytes.

Embedded Content Detection

Files within files (email attachments, archive contents, OLE streams) may appear at arbitrary offsets. Embeddings enable similarity search: "find all chunks that look structurally like JPEG data" regardless of where they appear.

Architecture: RoPE vs Absolute Position Embeddings

This model uses Rotary Position Embeddings (RoPE), which encode position through rotation matrices applied to query and key vectors in attention. This differs from absolute position embeddings (used by standard BERT), which add learned position vectors to token embeddings.

Why RoPE?

RoPE was designed for better relative position modeling:

• Encodes relative position naturally through the rotation angle
• Theoretically supports length extrapolation (though not tested beyond 512)
• 28% fewer parameters than absolute position variant (42M vs 59M)

Position Bias: Key Finding

RoPE does not solve position bias for file type modeling. Both RoPE (this model) and absolute position embeddings (Magic-BERT) show nearly identical position bias (~47% accuracy drop at offset 1000).

This occurs because position bias is learned from the training data distribution, where files naturally start at offset 0. The position encoding mechanism doesn't affect this learned correlation. Solutions should focus on data augmentation rather than architecture changes.

Aspect	RoFormer (this)	Magic-BERT
Position Encoding	RoPE (rotary)	Absolute (learned)
Parameters	42.3M	59M
Perplexity	1.13	1.05
Fill-mask Top-1	61.8%	58.9%
Probing Accuracy	85.0%	87.0%
Position Bias @ 1000	~46% drop	~48% drop

RoFormer achieves better fill-mask accuracy with fewer parameters. Magic-BERT achieves better perplexity and probing accuracy.

MLM vs Classification: Two-Phase Training

This is the Phase 1 (MLM) model. The training pipeline has two phases:

Phase	Model	Task	Purpose
Phase 1	This model	Masked Language Modeling	Learn byte-level patterns and file structure
Phase 2	magic-bert-50m-roformer-classification	Contrastive Learning	Optimize embeddings for file type discrimination

When to use each:

• Use this model (MLM) for: fill-mask tasks, research, or as a base for custom fine-tuning
• Use classification model for: file type detection, similarity search, production classification

Training

Data

Trained on a diverse corpus of binary files spanning 106 MIME types, including documents, images, audio/video, archives, executables, and more.

Procedure

Phase	Steps	Learning Rate	Batch Size	Objective
MLM Pre-training	100,000	1e-4	240	Masked LM (15% masking)

Data augmentation: 50% of samples use random byte offset to reduce position bias.

Evaluation Results

Perplexity by Region

Region	Perplexity
Magic Bytes (0-9)	1.12
Header (10-49)	1.15
Body (50+)	1.13
Overall	1.13

Fill-Mask Accuracy

Metric	Value
Top-1 Accuracy	61.8%
Top-5 Accuracy	75.1%
Mean Reciprocal Rank	0.682

Representation Quality

Metric	Value
Linear Probe Accuracy	85.0%
Silhouette Score	0.328
Separation Ratio	2.65

How to Use

from transformers import RoFormerForMaskedLM, AutoTokenizer
import torch

model = RoFormerForMaskedLM.from_pretrained("mjbommar/magic-bert-50m-roformer-mlm")
tokenizer = AutoTokenizer.from_pretrained("mjbommar/magic-bert-50m-roformer-mlm")

model.eval()

# Read a binary file
with open("example.pdf", "rb") as f:
    data = f.read(512)

# Decode bytes to string using latin-1 (preserves all byte values 0-255)
text = data.decode("latin-1")

# Tokenize and mask a position
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
mask_pos = 1  # Mask second token (first is [CLS])
inputs["input_ids"][0, mask_pos] = tokenizer.mask_token_id

# Predict masked token
with torch.no_grad():
    outputs = model(**inputs)
    predictions = outputs.logits[0, mask_pos].topk(5)

print("Top-5 predictions:", tokenizer.convert_ids_to_tokens(predictions.indices))

Getting Embeddings

from transformers import RoFormerModel

model = RoFormerModel.from_pretrained("mjbommar/magic-bert-50m-roformer-mlm")

with torch.no_grad():
    outputs = model(**inputs)
    # CLS token embedding
    cls_embedding = outputs.last_hidden_state[:, 0, :]  # [batch_size, 512]
    # Mean pooling
    mean_embedding = outputs.last_hidden_state.mean(dim=1)  # [batch_size, 512]

Limitations

1. Position bias: Despite using RoPE, the model shows similar position bias to absolute position models (~46% accuracy drop at offset 1000). This is due to training data distribution, not architecture.

2. Sequence length: Limited to 512 tokens. While RoPE theoretically supports length extrapolation, this model was not trained for longer sequences.

3. HuggingFace integration: Uses standard RoFormer architecture, making it easy to deploy but less customizable than the Magic-BERT variant.

Model Selection Guide

Use Case	Recommended Model	Reason
Fill-mask / byte prediction	This model	Best fill-mask accuracy (61.8%) with fewer params
Research baseline	magic-bert-50m-mlm	Established BERT architecture, best perplexity
Classification + fill-mask	magic-bert-50m-classification	Retains 41.8% fill-mask capability
Production classification	magic-bert-50m-roformer-classification	Highest accuracy (93.7%), efficient (42M params)

Related Models

• magic-bert-50m-roformer-classification: Same architecture fine-tuned for classification (93.7% accuracy, recommended for production)
• magic-bert-50m-mlm: Absolute position embedding variant
• magic-bert-50m-classification: Absolute position variant fine-tuned for classification (89.7% accuracy)

Related Work

This model builds on the Binary BPE tokenization approach:

• Binary BPE Paper: Bommarito (2025) introduced byte-level BPE tokenization for binary analysis, demonstrating 2-3x compression over raw bytes for executable content.
• Binary BPE Tokenizers: Pre-trained tokenizers for executables are available at mjbommar/binary-tokenizer-001-64k.

Key difference: The original Binary BPE work focused on executable binaries (ELF, PE, Mach-O). Magic-BERT extends this to general file type understanding across 106 diverse formats, using a tokenizer trained on the broader dataset.

Citation

A paper describing Magic-BERT, the training methodology, and the dataset is forthcoming.

@article{bommarito2025binarybpe,
  title={Binary BPE: A Family of Cross-Platform Tokenizers for Binary Analysis},
  author={Bommarito, Michael J., II},
  journal={arXiv preprint arXiv:2511.17573},
  year={2025}
}