Instructions to use datalab-to/chandra with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use datalab-to/chandra with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="datalab-to/chandra")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("datalab-to/chandra")
model = AutoModelForMultimodalLM.from_pretrained("datalab-to/chandra", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use datalab-to/chandra with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "datalab-to/chandra"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "datalab-to/chandra",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/datalab-to/chandra

SGLang

How to use datalab-to/chandra with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "datalab-to/chandra" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "datalab-to/chandra",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "datalab-to/chandra" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "datalab-to/chandra",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use datalab-to/chandra with Docker Model Runner:
```
docker model run hf.co/datalab-to/chandra
```

chandra / README.md

tarun-menta

Add new model version metadata (#10)

bd40c21 6 months ago

preview code

Raw

History Blame Contribute Delete

6.19 kB

	---
	license: openrail
	library_name: transformers
	tags:
	- ocr
	- vlm
	new_version: datalab-to/chandra-ocr-2
	---

	# Chandra

	Chandra is an OCR model that outputs markdown, HTML, and JSON. It is highly accurate at extracting text from images and PDFs, while preserving layout information.

	You can try Chandra in the free playground [here](https://www.datalab.to/playground), or at a hosted API [here](https://www.datalab.to/).

	## Features

	- Convert documents to markdown, html, or json with detailed layout information
	- Good handwriting support
	- Reconstructs forms accurately, including checkboxes
	- Good support for tables, math, and complex layouts
	- Extracts images and diagrams, with captions and structured data
	- Support for 40+ languages

	## Quickstart

	The easiest way to start is with the CLI tools:

	```shell
	pip install chandra-ocr

	# With VLLM
	chandra_vllm
	chandra input.pdf ./output

	# With HuggingFace
	chandra input.pdf ./output --method hf

	# Interactive streamlit app
	chandra_app
	```

	## Benchmarks

	We used the olmocr benchmark, which seems to be the most reliable current OCR benchmark in our testing.

	<img src="bench.png" width="600px"/>

	\| Model \| ArXiv \| Old Scans Math \| Tables \| Old Scans \| Headers and Footers \| Multi column \| Long tiny text \| Base \| Overall \| Source \|
	\|:----------\|:--------:\|:--------------:\|:--------:\|:---------:\|:-------------------:\|:------------:\|:--------------:\|:----:\|:--------------:\|:------:\|
	\| Datalab Chandra v0.1.0 \| 82.2 \| 80.3 \| 88.0 \| 50.4 \| 90.8 \| 81.2 \| 92.3 \| 99.9 \| 83.1 ± 0.9 \| Own benchmarks \|
	\| Datalab Marker v1.10.0 \| 83.8 \| 69.7 \| 74.8 \| 32.3 \| 86.6 \| 79.4 \| 85.7 \| 99.6 \| 76.5 ± 1.0 \| Own benchmarks \|
	\| Mistral OCR API \| 77.2 \| 67.5 \| 60.6 \| 29.3 \| 93.6 \| 71.3 \| 77.1 \| 99.4 \| 72.0 ± 1.1 \| olmocr repo \|
	\| Deepseek OCR \| 75.2 \| 72.3 \| 79.7 \| 33.3 \| 96.1 \| 66.7 \| 80.1 \| 99.7 \| 75.4 ± 1.0 \| Own benchmarks \|
	\| GPT-4o (Anchored) \| 53.5 \| 74.5 \| 70.0 \| 40.7 \| 93.8 \| 69.3 \| 60.6 \| 96.8 \| 69.9 ± 1.1 \| olmocr repo \|
	\| Gemini Flash 2 (Anchored) \| 54.5 \| 56.1 \| 72.1 \| 34.2 \| 64.7 \| 61.5 \| 71.5 \| 95.6 \| 63.8 ± 1.2 \| olmocr repo \|
	\| Qwen 3 VL \| 70.2 \| 75.1 \| 45.6 \| 37.5 \| 89.1 \| 62.1 \| 43.0 \| 94.3 \| 64.6 ± 1.1 \| Own benchmarks \|
	\| olmOCR v0.3.0 \| 78.6 \| 79.9 \| 72.9 \| 43.9 \| 95.1 \| 77.3 \| 81.2 \| 98.9 \| 78.5 ± 1.1 \| olmocr repo \|
	\| dots.ocr \| 82.1 \| 64.2 \| 88.3 \| 40.9 \| 94.1 \| 82.4 \| 81.2 \| 99.5 \| 79.1 ± 1.0 \| dots.ocr repo \|

	## Examples

	<img src="handwritten_form.png" width="600px"/>

	\| Type \| Name \| Link \|
	\|------\|------\|------\|
	\| Tables \| Water Damage Form \| [View](https://github.com/datalab-to/chandra/blob/master/assets/examples/tables/water_damage.png) \|
	\| Tables \| 10K Filing \| [View](https://github.com/datalab-to/chandra/blob/master/assets/examples/tables/10k.png) \|
	\| Forms \| Handwritten Form \| [View](https://github.com/datalab-to/chandra/blob/master/assets/examples/forms/handwritten_form.png) \|
	\| Forms \| Lease Agreement \| [View](https://github.com/datalab-to/chandra/blob/master/assets/examples/forms/lease.png) \|
	\| Handwriting \| Doctor Note \| [View](https://github.com/datalab-to/chandra/blob/master/assets/examples/handwriting/doctor_note.png) \|
	\| Handwriting \| Math Homework \| [View](https://github.com/datalab-to/chandra/blob/master/assets/examples/handwriting/math_hw.png) \|
	\| Books \| Geography Textbook \| [View](https://github.com/datalab-to/chandra/blob/master/assets/examples/books/geo_textbook_page.png) \|
	\| Books \| Exercise Problems \| [View](https://github.com/datalab-to/chandra/blob/master/assets/examples/books/exercises.png) \|
	\| Math \| Attention Diagram \| [View](https://github.com/datalab-to/chandra/blob/master/assets/examples/math/attn_all.png) \|
	\| Math \| Worksheet \| [View](https://github.com/datalab-to/chandra/blob/master/assets/examples/math/worksheet.png) \|
	\| Math \| EGA Page \| [View](https://github.com/datalab-to/chandra/blob/master/assets/examples/math/ega.png) \|
	\| Newspapers \| New York Times \| [View](https://github.com/datalab-to/chandra/blob/master/assets/examples/newspapers/nyt.png) \|
	\| Newspapers \| LA Times \| [View](https://github.com/datalab-to/chandra/blob/master/assets/examples/newspapers/la_times.png) \|
	\| Other \| Transcript \| [View](https://github.com/datalab-to/chandra/blob/master/assets/examples/other/transcript.png) \|
	\| Other \| Flowchart \| [View](https://github.com/datalab-to/chandra/blob/master/assets/examples/other/flowchart.png) \|

	## Usage

	### Installation

	```shell
	pip install chandra-ocr
	```

	### From code

	```python

	from chandra.model import InferenceManager
	from chandra.model.schema import BatchInputItem

	# Run chandra_vllm to start a vLLM server first if you pass vllm, else pass hf
	# you can also start your own vllm server with the datalab-to/chandra model
	manager = InferenceManager(method="vllm")
	batch = [
	BatchInputItem(
	image=PIL_IMAGE,
	prompt_type="ocr_layout"
	)
	]
	result = manager.generate(batch)[0]
	print(result.markdown)
	```

	### With transformers

	```python
	from transformers import AutoModel, AutoProcessor
	from chandra.model.hf import generate_hf
	from chandra.model.schema import BatchInputItem
	from chandra.output import parse_markdown

	model = AutoModel.from_pretrained("datalab-to/chandra").cuda()
	model.processor = AutoProcessor.from_pretrained("datalab-to/chandra")

	batch = [
	BatchInputItem(
	image=PIL_IMAGE,
	prompt_type="ocr_layout"
	)
	]

	result = generate_hf(batch, model)[0]
	markdown = parse_markdown(result.raw)
	```

	# Credits

	Thank you to the following open source projects:

	- [Huggingface Transformers](https://github.com/huggingface/transformers)
	- [VLLM](https://github.com/vllm-project/vllm)
	- [olmocr](github.com/allenai/olmocr)
	- [Qwen 3 VL](https://github.com/QwenLM/Qwen3)