C
O
L
L
A
G
E
Collaborative Human-Agent Interaction Generation using Hierarchical Latent Diffusion and Language Models
Divyanshu Daiya1, Damon Conover2, Aniket Bera1
1Department of Computer Science, Purdue University
2DEVCOM Army Research Laboratory
You can view our paper on arXiv.
Reach out to
divyanshu@purdue.edu for information on code and weights release.
Abstract
We propose a novel framework C O L L A G E for generating collaborative agent-object-agent interactions by leveraging large language models (LLMs) and hierarchical motion-specific vector-quantized variational autoencoders (VQ-VAEs). Our model addresses the lack of rich datasets in this domain by incorporating the knowledge and reasoning abilities of LLMs to guide a generative diffusion model. The hierarchical VQ-VAE architecture captures different motion-specific characteristics at multiple levels of abstraction, avoiding redundant concepts and enabling efficient multi-resolution representation. We introduce a diffusion model that operates in the latent space and incorporates LLM-generated motion planning cues to guide the denoising process, resulting in prompt-specific motion generation with greater control and diversity. Experimental results on the CORE-4D, and InterHuman datasets demonstrate the effectiveness of our approach in generating realistic and diverse collaborative human-object-human interactions, outperforming state-of-the-art methods. Our work opens up new possibilities for modeling complex interactions in various domains, such as robotics, graphics and computer vision.
Text-based Human-Object-Human Generation comparisons
Text-based Single Human Generation comparisons
Action-based Generation
"Throw"
"Dance"
"Squat"
"Jump"
Acknowledgements
This material is based upon work supported in part by the DEVCOM Army Research Laboratory under cooperative agreement W911NF2020221.
The website template was adapted from GRAM.