Kirin: Animal Motion Generation from In-the-Wild Video

Zhao, Brian Nlong; Pan, Zhuoyang; Rehg, James M.; Wu, Jiajun; Wu, Shangzhe

doi:10.1007/978-3-032-37261-1_1

ECCV 2026

Kirin: Animal Motion Generation from In-the-Wild Video

Brian Nlong Zhao¹^*, Zhuoyang Pan²^*, James M. Rehg¹, Jiajun Wu³, and Shangzhe Wu⁴

¹ University of Illinois Urbana-Champaign · ² University of Pennsylvania · ³ Stanford University · ⁴ University of Cambridge

^* Equal contribution

Kirin learns 3D quadruped motion directly from internet video, and turns a single image plus a text prompt into a fully rigged, animated mesh.

Paper PDF Code Dataset Watch

Dataset ready AiM3D is now available to download. Code release is still in progress — check back shortly.

Abstract

Understanding animal motion is fundamental to modeling animal behavior and biomechanics, yet progress in this area lags far behind human motion research due to the scarcity of high-quality motion data. While human motion can be captured in controlled environments, it is impractical for most animal species, resulting in small, domain-limited datasets that restrict downstream applications such as animation.

To address this challenge, we introduce Kirin, a framework that reconstructs motion from video, learns motion priors at scale, and generates realistic motion that can be directly applied to animated assets. Using large collections of in-the-wild animal videos, we reconstruct 3D motion sequences and pair them with captions to create AiM3D, the first large-scale dataset offering aligned video-text-motion tuples for quadruped animals. Building on this dataset, we develop a visual-guided motion generation model that conditions on both text and image to guide the generation of realistic motion across diverse animal species. Finally, by leveraging an off-the-shelf image-to-3D model, we automatically rig and animate 3D meshes using generated motion, producing ready-to-render animated animals. Together, our dataset and framework establish a new foundation for large-scale, text and image conditioned animal motion generation and animation.

Contribution 01

AiM3D — a large-scale animal motion dataset with aligned text, video, and 3D motion.

Contribution 02

The first animal motion generation model conditioned on both text and image input.

Contribution 03

A fully automatic 2D-image → 3D animated mesh system, beating prior 4D animation methods.

Method

A two-stage pipeline. A diffusion-based motion model conditioned on text and image generates the 3D motion; an automatic rigging module applies it to a T-posed mesh produced from the same image.

Fig. 02 · Pipeline

Left — generation pipeline. A text description and an image are provided as inputs. Both are used for motion generation, while the image is also used to generate a T-posed mesh. The animation module rigs the generated motion onto the mesh to produce the final animated 3D model.

Right — motion generation architecture. Text features are extracted with a frozen DistilBERT encoder; image features with a frozen DINOv3 encoder. Text, image, and denoising-step embeddings are combined and fed into a transformer decoder with cross-attention to produce the motion sequence.

Reconstruction backbone

Fig. 03 · Reconstruction

Overview of motion reconstruction. We leverage off-the-shelf 3D quadruped reconstruction and 3D tracking methods to infer articulation and global translation separately, then combine them to produce the final per-frame 3D motion that populates AiM3D.

Dataset

A large-scale animal motion dataset with aligned video, text, and 3D motion. Reconstructions cover diverse species, behaviours, and environments far beyond what controlled capture can offer.

AiM3D downloads → 23 species · 876 GB total

AiM3D is distributed as one archive per animal category, so you can take only the species you need rather than the whole corpus. Each archive holds per-sequence new_joints.npy and new_joint_vecs.npy reconstructions; Mean.npy, Std.npy, train.txt and test.txt provide the normalisation statistics and the official split.

bear boar cat cougar cow deer dog elephant fox goat hippo horse leopard moose panther pig rabbit racoon rhino sheep tiger wolf zebra

Archives range from 2 GB (panther) to 113 GB (cow). The server supports range requests, so downloads are resumable — we recommend the command line over a browser. wget -c https://download.cs.stanford.edu/viscam/AiM/AiM_full/horse.zip Or fetch everything, metadata included: wget -c -r -np -nH --cut-dirs=3 -R "index.html*" https://download.cs.stanford.edu/viscam/AiM/AiM_full/

AiM3D data examples — captions, video frames and reconstructed motion

Fig. 04 · AiM3D samples

Examples from AiM3D. Each row shows a real-world video sequence aligned with two VLM-generated captions describing the animal's motion, and the corresponding reconstructed 3D motion (visualized as keypoints). Reconstructions span species, gait, and behaviour — rhinos grazing, foxes sniffing and resting, rabbits hopping, horses trotting under saddle.

Global trajectories

Comparisons

Side-by-side qualitative comparisons of generated mesh and skeletal animation against the strongest published baseline. Drag to inspect; click pause to hold a frame.

Generated mesh animation

Cmp.01 "A moose walks straight ahead." 01 / 02

Reference input

Puppeteer baseline

Kirin ours

Cmp.02 "A horse trots forward." 02 / 02

Reference input

Puppeteer baseline

Kirin ours

Left-click drag to rotate

Right-click drag, or WASD to move

Scroll to zoom

Click

to pause

Generated skeletal motion

Fig. 05 · Skeleton

Generated skeletal motion comparison. Sequences from our model versus the strongest published baseline at matched frames. Kirin produces gait cycles with cleaner foot contact and more plausible global trajectory.

Animated mesh — gallery comparison

Mesh animation comparison against Puppeteer baseline

Fig. 06 · Mesh gallery

Comparison against Puppeteer*. A reproduction of Puppeteer's setup (no public code at the time of writing) generates static T-posed meshes that do not animate the prompted action. Kirin produces full motion sequences faithful to the input image and prompt. * Reproduced from the paper.

Results

Across motion-generation metrics and 4D animation benchmarks, Kirin outperforms or matches prior methods. Best results in color; second-best underlined.

Tab. 01 · Text-conditioned animal motion generation Higher R-Precision and Diversity is better · lower FID, MM-Dist is better

Method	R-Prec. Top-1 ↑	R-Prec. Top-3 ↑	FID ↓	MM-Dist ↓	Diversity ↑
Real motion	0.512±.004	0.823±.003	0.002±.001	2.974±.008	9.503±.065
MDM	0.298±.005	0.612±.004	1.842±.040	4.918±.011	8.612±.078
MotionDiffuse	0.321±.004	0.658±.005	1.514±.034	4.671±.012	8.804±.080
T2M-GPT	0.354	0.701	1.288±.030	4.402	9.014
MoMask	0.346±.005	0.689±.004	1.196	4.477±.010	8.961±.071
Kirin (text-only)	0.392±.004	0.748±.003	0.974±.026	4.142±.009	9.182±.069

Even without image conditioning, Kirin's architecture trained on AiM3D outperforms the strongest text-to-motion baselines across all five standard metrics. Numbers reported on the AiM3D test split with 95% confidence intervals.

Tab. 02 · Image + text-conditioned animal motion generation Adding image conditioning further improves identity preservation and motion plausibility.

Method	R-Prec. Top-1 ↑	R-Prec. Top-3 ↑	FID ↓	MM-Dist ↓	Image-CLIP ↑
Real motion	0.512±.004	0.823±.003	0.002±.001	2.974±.008	0.298±.003
MDM + img	0.331±.005	0.671±.004	1.487±.038	4.604±.011	0.221±.003
MoMask + img	0.378	0.722	1.041	4.291	0.247
Kirin (full)	0.421±.004	0.776±.003	0.812±.022	3.978±.010	0.281±.003

Conditioning on the input image alongside the text prompt — through a frozen DINOv3 encoder fused with text and time embeddings — yields the largest gains on FID and Image-CLIP, indicating that motion better matches the appearance and identity of the depicted animal.

Tab. 03 · 4D animation from a single image Mesh-level evaluation against published 4D animation methods.

Method	CLIP-T ↑	CLIP-I ↑	Temporal Cons. ↑	User Pref. ↑
Animate124	0.218±.004	0.704±.005	0.812±.006	11.4 %
4D-fy	0.231±.004	0.728±.005	0.847±.005	14.8 %
Puppeteer*	0.262	0.781	0.881	21.7 %
Kirin (ours)	0.291±.003	0.812±.004	0.924±.004	52.1 %

A user study with 38 participants comparing 24 prompt × image pairs from a held-out evaluation set; participants picked the most realistic animation per prompt. Kirin wins a majority of comparisons. * Reproduced; no public code at submission.

Citation

If you find this work useful, please consider citing our paper.

BibTeX

@inproceedings{zhao2026kirin,
  title     = {Kirin: Animal Motion Generation from In-the-Wild Video},
  author    = {Zhao, Brian Nlong and Pan, Zhuoyang and Rehg, James M. and Wu, Jiajun and Wu, Shangzhe},
  booktitle = {Computer Vision -- ECCV 2026},
  series    = {Lecture Notes in Computer Science},
  pages     = {1--19},
  publisher = {Springer},
  year      = {2026},
  doi       = {10.1007/978-3-032-37261-1_1}
}