---
language:
  - en
license: apache-2.0
library_name: transformers
tags:
  - binary-analysis
  - file-type-detection
  - byte-level
  - fill-mask
  - mlm
  - roformer
  - rope
  - magic-bytes
  - security
pipeline_tag: fill-mask
model-index:
  - name: magic-bert-50m-roformer-mlm
    results:
      - task:
          type: fill-mask
          name: Masked Language Modeling
        metrics:
          - name: Perplexity
            type: perplexity
            value: 1.13
          - name: Fill-mask Top-1 Accuracy
            type: accuracy
            value: 61.8
          - name: Fill-mask Top-5 Accuracy
            type: accuracy
            value: 75.1
          - name: Probing Classification Accuracy
            type: accuracy
            value: 85.0
---

# Magic-BERT 50M RoFormer MLM

A RoFormer-based transformer model trained for binary file understanding using masked language modeling (MLM). This model uses Rotary Position Embeddings (RoPE) for position encoding, which provides better relative position modeling than absolute embeddings.

## Why Not Just Use libmagic?

For intact files starting at byte 0, libmagic works well. But libmagic matches *signatures at fixed offsets*. Magic-BERT learns *structural patterns* throughout the file, enabling use cases where you don't have clean file boundaries:

- **Network streams**: Classifying packet payloads mid-connection, before headers arrive
- **Disk forensics**: Identifying file types during carving, when scanning raw disk images without filesystem metadata
- **Fragment analysis**: Working with partial files, slack space, or corrupted data
- **Adversarial contexts**: Detecting file types when magic bytes are stripped, spoofed, or deliberately misleading

## Model Description

This model uses the HuggingFace RoFormer architecture with a byte-level BPE tokenizer trained on binary file data. RoPE encodes position information directly in the attention computation through rotation matrices.

| Property | Value |
|----------|-------|
| Parameters | 42.3M |
| Hidden Size | 512 |
| Layers | 8 |
| Attention Heads | 8 |
| Max Sequence Length | 512 tokens |
| Vocabulary Size | 32,768 (byte-level BPE) |
| Position Encoding | RoPE (Rotary Position Embeddings) |

### Tokenizer

The tokenizer uses the Binary BPE methodology introduced in [Bommarito (2025)](https://arxiv.org/abs/2511.17573). The original Binary BPE tokenizers (available at [mjbommar/binary-tokenizer-001-64k](https://huggingface.co/mjbommar/binary-tokenizer-001-64k)) were trained exclusively on executable binaries (ELF, PE, Mach-O). This tokenizer uses the same BPE training approach but was trained on a diverse corpus spanning 106 file types.

## Intended Uses

**Primary use cases:**
- Fill-mask: Predicting missing bytes in binary files
- Magic byte and file signature recognition
- Feature extraction for downstream classification
- Research on binary file structure with relative position encoding

**Example tasks:**
- Completing partial file headers
- Identifying file type from structure
- Exploring position-independent file patterns

## Detailed Use Cases

### Network Traffic Analysis
When inspecting packet payloads, you often see file data mid-stream—TCP reassembly may give you bytes 1500-3000 of a PDF before you ever see byte 0. Traditional signature matching fails here. Structural understanding can identify file types from interior content.

### Disk Forensics & File Carving
During disk image analysis, you scan raw bytes looking for file boundaries. Tools like Scalpel rely on header/footer signatures, but many files lack clear footers. This model can score byte ranges for file type probability, helping identify carved fragments or validate carving results.

### Incident Response
Malware often strips or modifies magic bytes to evade detection. Polyglot files (valid as multiple types) exploit signature-based tools. Learning structural patterns provides a second opinion that doesn't rely solely on the first few bytes.

### Embedded Content Detection
Files within files (email attachments, archive contents, OLE streams) may appear at arbitrary offsets. Embeddings enable similarity search: "find all chunks that look structurally like JPEG data" regardless of where they appear.

## Architecture: RoPE vs Absolute Position Embeddings

This model uses **Rotary Position Embeddings (RoPE)**, which encode position through rotation matrices applied to query and key vectors in attention. This differs from **absolute position embeddings** (used by standard BERT), which add learned position vectors to token embeddings.

### Why RoPE?

RoPE was designed for better relative position modeling:
- Encodes relative position naturally through the rotation angle
- Theoretically supports length extrapolation (though not tested beyond 512)
- 28% fewer parameters than absolute position variant (42M vs 59M)

### Position Bias: Key Finding

**RoPE does not solve position bias for file type modeling.** Both RoPE (this model) and absolute position embeddings (Magic-BERT) show nearly identical position bias (~47% accuracy drop at offset 1000).

This occurs because position bias is learned from the training data distribution, where files naturally start at offset 0. The position encoding mechanism doesn't affect this learned correlation. Solutions should focus on data augmentation rather than architecture changes.

| Aspect | RoFormer (this) | Magic-BERT |
|--------|-----------------|------------|
| Position Encoding | RoPE (rotary) | Absolute (learned) |
| Parameters | **42.3M** | 59M |
| Perplexity | 1.13 | **1.05** |
| Fill-mask Top-1 | **61.8%** | 58.9% |
| Probing Accuracy | 85.0% | **87.0%** |
| Position Bias @ 1000 | ~46% drop | ~48% drop |

RoFormer achieves better fill-mask accuracy with fewer parameters. Magic-BERT achieves better perplexity and probing accuracy.

## MLM vs Classification: Two-Phase Training

This is the **Phase 1 (MLM)** model. The training pipeline has two phases:

| Phase | Model | Task | Purpose |
|-------|-------|------|---------|
| **Phase 1** | **This model** | Masked Language Modeling | Learn byte-level patterns and file structure |
| Phase 2 | magic-bert-50m-roformer-classification | Contrastive Learning | Optimize embeddings for file type discrimination |

**When to use each:**
- Use **this model (MLM)** for: fill-mask tasks, research, or as a base for custom fine-tuning
- Use **classification model** for: file type detection, similarity search, production classification

## Training

### Data
Trained on a diverse corpus of binary files spanning 106 MIME types, including documents, images, audio/video, archives, executables, and more.

### Procedure

| Phase | Steps | Learning Rate | Batch Size | Objective |
|-------|-------|---------------|------------|-----------|
| MLM Pre-training | 100,000 | 1e-4 | 240 | Masked LM (15% masking) |

**Data augmentation:** 50% of samples use random byte offset to reduce position bias.

## Evaluation Results

### Perplexity by Region
| Region | Perplexity |
|--------|------------|
| Magic Bytes (0-9) | 1.12 |
| Header (10-49) | 1.15 |
| Body (50+) | 1.13 |
| **Overall** | **1.13** |

### Fill-Mask Accuracy
| Metric | Value |
|--------|-------|
| Top-1 Accuracy | 61.8% |
| Top-5 Accuracy | 75.1% |
| Mean Reciprocal Rank | 0.682 |

### Representation Quality
| Metric | Value |
|--------|-------|
| Linear Probe Accuracy | 85.0% |
| Silhouette Score | 0.328 |
| Separation Ratio | 2.65 |

## How to Use

```python
from transformers import RoFormerForMaskedLM, AutoTokenizer
import torch

model = RoFormerForMaskedLM.from_pretrained("mjbommar/magic-bert-50m-roformer-mlm")
tokenizer = AutoTokenizer.from_pretrained("mjbommar/magic-bert-50m-roformer-mlm")

model.eval()

# Read a binary file
with open("example.pdf", "rb") as f:
    data = f.read(512)

# Decode bytes to string using latin-1 (preserves all byte values 0-255)
text = data.decode("latin-1")

# Tokenize and mask a position
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
mask_pos = 1  # Mask second token (first is [CLS])
inputs["input_ids"][0, mask_pos] = tokenizer.mask_token_id

# Predict masked token
with torch.no_grad():
    outputs = model(**inputs)
    predictions = outputs.logits[0, mask_pos].topk(5)

print("Top-5 predictions:", tokenizer.convert_ids_to_tokens(predictions.indices))
```

### Getting Embeddings

```python
from transformers import RoFormerModel

model = RoFormerModel.from_pretrained("mjbommar/magic-bert-50m-roformer-mlm")

with torch.no_grad():
    outputs = model(**inputs)
    # CLS token embedding
    cls_embedding = outputs.last_hidden_state[:, 0, :]  # [batch_size, 512]
    # Mean pooling
    mean_embedding = outputs.last_hidden_state.mean(dim=1)  # [batch_size, 512]
```

## Limitations

1. **Position bias:** Despite using RoPE, the model shows similar position bias to absolute position models (~46% accuracy drop at offset 1000). This is due to training data distribution, not architecture.

2. **Sequence length:** Limited to 512 tokens. While RoPE theoretically supports length extrapolation, this model was not trained for longer sequences.

3. **HuggingFace integration:** Uses standard RoFormer architecture, making it easy to deploy but less customizable than the Magic-BERT variant.

## Model Selection Guide

| Use Case | Recommended Model | Reason |
|----------|-------------------|--------|
| Fill-mask / byte prediction | **This model** | Best fill-mask accuracy (61.8%) with fewer params |
| Research baseline | magic-bert-50m-mlm | Established BERT architecture, best perplexity |
| Classification + fill-mask | magic-bert-50m-classification | Retains 41.8% fill-mask capability |
| **Production classification** | **magic-bert-50m-roformer-classification** | Highest accuracy (93.7%), efficient (42M params) |

## Related Models

- **[magic-bert-50m-roformer-classification](https://huggingface.co/mjbommar/magic-bert-50m-roformer-classification)**: Same architecture fine-tuned for classification (93.7% accuracy, recommended for production)
- **[magic-bert-50m-mlm](https://huggingface.co/mjbommar/magic-bert-50m-mlm)**: Absolute position embedding variant
- **[magic-bert-50m-classification](https://huggingface.co/mjbommar/magic-bert-50m-classification)**: Absolute position variant fine-tuned for classification (89.7% accuracy)

## Related Work

This model builds on the Binary BPE tokenization approach:

- **Binary BPE Paper**: [Bommarito (2025)](https://arxiv.org/abs/2511.17573) introduced byte-level BPE tokenization for binary analysis, demonstrating 2-3x compression over raw bytes for executable content.
- **Binary BPE Tokenizers**: Pre-trained tokenizers for executables are available at [mjbommar/binary-tokenizer-001-64k](https://huggingface.co/mjbommar/binary-tokenizer-001-64k).

**Key difference**: The original Binary BPE work focused on executable binaries (ELF, PE, Mach-O). Magic-BERT extends this to general file type understanding across 106 diverse formats, using a tokenizer trained on the broader dataset.

## Citation

A paper describing Magic-BERT, the training methodology, and the dataset is forthcoming.

```bibtex
@article{bommarito2025binarybpe,
  title={Binary BPE: A Family of Cross-Platform Tokenizers for Binary Analysis},
  author={Bommarito, Michael J., II},
  journal={arXiv preprint arXiv:2511.17573},
  year={2025}
}
```