Some feedback
I've been playing with your model today since the comfy support is there. Beside known issues listed here https://www.reddit.com/r/comfyui/comments/1w9e5i4/viggleanimate_comfyui/
I observed that the model is unstable in several respects:
- If the background is different than original video the model keep the image background for few sec then person drift to video background id changes slightly and some deformities appears
- Black background on ref image works (to some extent)
- My longest video tested was 15 sec, in some movement even simple like hand moving while dancing the hand was deforming or glitching as some limbs are detached from the rest of the body just like it would be still few previous frames (not sure if the video length matter)
- Clothes changes for some reason - brown tank top on reference image but output video has green dress. Seed dependent.
All my tests was up to 0.8 megapixel
All my tests was on realistic videos since it's my use case. Hopefully this will help you improve. Can't wait for future releases.
Thanks, this is useful — and a couple of these are real limits we know about.
The first two seem an input problem, so let me separate them.
The reference must be a frame of your video, repainted. Not a character photo, not a cut-out on black. Pull a frame, open it in GPT-Image / Gemini / Grok / Nano Banana, and change only the character — same background, same framing, same pose, same lighting (not that strict, but should be almost same background). The model treats the reference as "this is what frame N of the video looks like after the edit" and propagates that edit.
Clothes changing (brown tank top → green dress): if the reference was a standalone character image, this is the same failure — the model has no clean binding between your character and the person in the video, so the seed decides. If it happens with a proper repainted frame, that's a real bug and I'd like to see it: please share the driving clip, the reference.
15 s clips: valid. The model is trained and evaluated at 124 frames (~5 s at 24 fps) and holds well to about 10 s. Past that, a single reference image stops anchoring identity and you get ghosting and limbs that lag behind the body — what you described. Right now the fix is to cut at shot boundaries and render each shot with its own repainted frame; we're working on longer holds.
Resolution: 480P at 5s would have the best quality. You may upscale it with FlashVSR later.
One thing to note: the ComfyUI integration isn't ours, so we don't know what defaults it ships with. you may also try it here https://huggingface.co/spaces/Viggle/viggle-animate
Feel free to share more of your thoughts, happy to answer questions and collect feedback.
Thanks, makes sense!
One more thing I'm curious about - do you think future versions could support refs that don't have to match the pose of a specific source frame?
For example, if the person only enters the frame after a few seconds, I'd like to give the model their reference image but still keep them invisible until they actually appear in the source video.
Or if they start with their back to the camera, I'd like to use a clear frontal ref even though the pose doesn't match.
Multiple image refs of the same character (front/side/back) could also be useful here. SCAIL-2 has an experimental multi-ref mode that works fairly well in some cases.
Basically: source video = motion/pose/timing, refs = identity/appearance. Is that something you're considering?
the reference doesn't have to be frame 0 (it can be even outside the reference video's time range sometimes). Pick any frame where the person is clearly visible and facing camera, repaint that one, and the model propagates it in both directions in time.
The repainted frame is something I plan to keep, though. Dropping it means a mask plus a text prompt to say who gets replaced and how they sit in the shot, and I found that unstable. Image models already do this on one frame very well, so we let them.
Extra front/side/back refs on top of the repainted frame is being considered, for exactly your case: today the reference only defines the views it contains, unseen sides come from the model's prior.
Hello, I can add the face expressions need some working. I know you guys pointed the lip-sync limitations but for me it's far more than that. On some videos "simple face expressions" like smile or blink can be completely ignored from reference video.