Add UnityVideo 5B model card
Browse files
README.md
CHANGED
|
@@ -1,3 +1,82 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
+
library_name: unityvideo
|
| 4 |
+
pipeline_tag: text-to-video
|
| 5 |
+
base_model: Wan-AI/Wan2.2-TI2V-5B
|
| 6 |
+
tags:
|
| 7 |
+
- video-generation
|
| 8 |
+
- multi-modal
|
| 9 |
+
- depth-estimation
|
| 10 |
+
- optical-flow
|
| 11 |
+
- controllable-generation
|
| 12 |
---
|
| 13 |
+
|
| 14 |
+
# UnityVideo Wan2.2-TI2V-5B
|
| 15 |
+
|
| 16 |
+
This repository contains the five-modality, three-task UnityVideo checkpoint
|
| 17 |
+
built on Wan2.2-TI2V-5B. Source code and usage instructions are available at
|
| 18 |
+
[JIA-Lab-research/UnityVideo](https://github.com/JIA-Lab-research/UnityVideo).
|
| 19 |
+
|
| 20 |
+
## Capabilities
|
| 21 |
+
|
| 22 |
+
- Modalities: depth, DensePose, optical flow (RAFT), segmentation, skeleton.
|
| 23 |
+
- Tasks: text-to-RGB+modality, video-to-modality, modality-to-video.
|
| 24 |
+
- Architecture: joint RGB/modality self-attention, split text cross-attention,
|
| 25 |
+
modality identity embeddings, and separate RGB/modality output heads.
|
| 26 |
+
|
| 27 |
+
## Files
|
| 28 |
+
|
| 29 |
+
| File | Size | SHA-256 | Use |
|
| 30 |
+
| --- | ---: | --- | --- |
|
| 31 |
+
| `checkpoints/unityvideo_wan22_ti2v_5b_step15000_ema.safetensors` | 10,020,954,352 bytes | `0df3909e312526c46f68097958afa055868f73354fe4276d693f7ebc398e6a39` | Inference |
|
| 32 |
+
| `checkpoints/unityvideo_wan22_ti2v_5b_step15000.safetensors` | 10,020,954,352 bytes | `4ee83a43ffdbaee90e7ff6a25fe108d8d65d255cd0803e3972bbbfb5b01db48c` | Fine-tuning |
|
| 33 |
+
|
| 34 |
+
The checkpoints contain the full two-stream DiT. Download the VAE, text encoder,
|
| 35 |
+
tokenizer, and base configuration from
|
| 36 |
+
[Wan-AI/Wan2.2-TI2V-5B](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B).
|
| 37 |
+
|
| 38 |
+
## Quick start
|
| 39 |
+
|
| 40 |
+
```bash
|
| 41 |
+
git clone https://github.com/JIA-Lab-research/UnityVideo.git
|
| 42 |
+
cd UnityVideo
|
| 43 |
+
pip install -e .
|
| 44 |
+
|
| 45 |
+
unityvideo-infer \
|
| 46 |
+
--task video2flow \
|
| 47 |
+
--modality depth \
|
| 48 |
+
--rgb-video input.mp4 \
|
| 49 |
+
--output depth.mp4
|
| 50 |
+
```
|
| 51 |
+
|
| 52 |
+
The CLI downloads this checkpoint and the Wan2.2 base model on first use. See
|
| 53 |
+
the GitHub README for all inference modes, local download commands, metadata
|
| 54 |
+
format, and distributed training.
|
| 55 |
+
|
| 56 |
+
## Training recipe
|
| 57 |
+
|
| 58 |
+
- Base: Wan2.2-TI2V-5B.
|
| 59 |
+
- Resolution: 256 x 256, 33 frames.
|
| 60 |
+
- Modalities: depth, DensePose, RAFT, segmentation, skeleton.
|
| 61 |
+
- Modality sampling: 0.2, 0.2, 0.2, 0.4, 0.4.
|
| 62 |
+
- Task sampling (`text2all`, `video2flow`, `flow2video`): 0.5, 0.25, 0.25.
|
| 63 |
+
- EMA decay: 0.9999.
|
| 64 |
+
- Effective released step: 15,000.
|
| 65 |
+
|
| 66 |
+
## Limitations
|
| 67 |
+
|
| 68 |
+
The released model was evaluated at 256 x 256 with 33 frames. Higher
|
| 69 |
+
resolutions, longer videos, unseen modality encodings, and safety-critical uses
|
| 70 |
+
require independent validation. The historical A14B dual-expert experiments
|
| 71 |
+
used an RGB+depth-only recipe and are not represented by this checkpoint.
|
| 72 |
+
|
| 73 |
+
## Citation
|
| 74 |
+
|
| 75 |
+
```bibtex
|
| 76 |
+
@article{huang2025unityvideo,
|
| 77 |
+
title={UnityVideo: Unified Multi-Modal Multi-Task Learning for Enhancing World-Aware Video Generation},
|
| 78 |
+
author={Huang, Jiehui and Zhang, Yuechen and He, Xu and Gao, Yuan and Cen, Zhi and Xia, Bin and Zhou, Yan and Tao, Xin and Wan, Pengfei and Jia, Jiaya},
|
| 79 |
+
journal={arXiv preprint arXiv:2512.07831},
|
| 80 |
+
year={2025}
|
| 81 |
+
}
|
| 82 |
+
```
|