JackAILab commited on
Commit
5136bba
·
verified ·
1 Parent(s): b5ac7c6

Add UnityVideo 5B model card

Browse files
Files changed (1) hide show
  1. README.md +79 -0
README.md CHANGED
@@ -1,3 +1,82 @@
1
  ---
2
  license: apache-2.0
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
+ library_name: unityvideo
4
+ pipeline_tag: text-to-video
5
+ base_model: Wan-AI/Wan2.2-TI2V-5B
6
+ tags:
7
+ - video-generation
8
+ - multi-modal
9
+ - depth-estimation
10
+ - optical-flow
11
+ - controllable-generation
12
  ---
13
+
14
+ # UnityVideo Wan2.2-TI2V-5B
15
+
16
+ This repository contains the five-modality, three-task UnityVideo checkpoint
17
+ built on Wan2.2-TI2V-5B. Source code and usage instructions are available at
18
+ [JIA-Lab-research/UnityVideo](https://github.com/JIA-Lab-research/UnityVideo).
19
+
20
+ ## Capabilities
21
+
22
+ - Modalities: depth, DensePose, optical flow (RAFT), segmentation, skeleton.
23
+ - Tasks: text-to-RGB+modality, video-to-modality, modality-to-video.
24
+ - Architecture: joint RGB/modality self-attention, split text cross-attention,
25
+ modality identity embeddings, and separate RGB/modality output heads.
26
+
27
+ ## Files
28
+
29
+ | File | Size | SHA-256 | Use |
30
+ | --- | ---: | --- | --- |
31
+ | `checkpoints/unityvideo_wan22_ti2v_5b_step15000_ema.safetensors` | 10,020,954,352 bytes | `0df3909e312526c46f68097958afa055868f73354fe4276d693f7ebc398e6a39` | Inference |
32
+ | `checkpoints/unityvideo_wan22_ti2v_5b_step15000.safetensors` | 10,020,954,352 bytes | `4ee83a43ffdbaee90e7ff6a25fe108d8d65d255cd0803e3972bbbfb5b01db48c` | Fine-tuning |
33
+
34
+ The checkpoints contain the full two-stream DiT. Download the VAE, text encoder,
35
+ tokenizer, and base configuration from
36
+ [Wan-AI/Wan2.2-TI2V-5B](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B).
37
+
38
+ ## Quick start
39
+
40
+ ```bash
41
+ git clone https://github.com/JIA-Lab-research/UnityVideo.git
42
+ cd UnityVideo
43
+ pip install -e .
44
+
45
+ unityvideo-infer \
46
+ --task video2flow \
47
+ --modality depth \
48
+ --rgb-video input.mp4 \
49
+ --output depth.mp4
50
+ ```
51
+
52
+ The CLI downloads this checkpoint and the Wan2.2 base model on first use. See
53
+ the GitHub README for all inference modes, local download commands, metadata
54
+ format, and distributed training.
55
+
56
+ ## Training recipe
57
+
58
+ - Base: Wan2.2-TI2V-5B.
59
+ - Resolution: 256 x 256, 33 frames.
60
+ - Modalities: depth, DensePose, RAFT, segmentation, skeleton.
61
+ - Modality sampling: 0.2, 0.2, 0.2, 0.4, 0.4.
62
+ - Task sampling (`text2all`, `video2flow`, `flow2video`): 0.5, 0.25, 0.25.
63
+ - EMA decay: 0.9999.
64
+ - Effective released step: 15,000.
65
+
66
+ ## Limitations
67
+
68
+ The released model was evaluated at 256 x 256 with 33 frames. Higher
69
+ resolutions, longer videos, unseen modality encodings, and safety-critical uses
70
+ require independent validation. The historical A14B dual-expert experiments
71
+ used an RGB+depth-only recipe and are not represented by this checkpoint.
72
+
73
+ ## Citation
74
+
75
+ ```bibtex
76
+ @article{huang2025unityvideo,
77
+ title={UnityVideo: Unified Multi-Modal Multi-Task Learning for Enhancing World-Aware Video Generation},
78
+ author={Huang, Jiehui and Zhang, Yuechen and He, Xu and Gao, Yuan and Cen, Zhi and Xia, Bin and Zhou, Yan and Tao, Xin and Wan, Pengfei and Jia, Jiaya},
79
+ journal={arXiv preprint arXiv:2512.07831},
80
+ year={2025}
81
+ }
82
+ ```