Towards Generalist Robots: A Promising Paradigm via Generative Simulation

Zhou Xian, Theophile Gervet, Zhenjia Xu, Tsun-Hsuan Wang, Yian Wang, Yufei Wang

Paper

• The purpose of this document is to share the excitement of the authors with the community and highlight a promising research direction in robotics and AI. The authors believe the proposed paradigm is a feasible path towards accomplishing the longstanding goal of robotics research: deploying robots, or embodied AI agents more broadly, in various non-factory real-world settings to perform diverse tasks.

ROBOGEN: TOWARDS UNLEASHING INFINITE DATA FOR AUTOMATED ROBOT LEARNING VIA GENERATIVE SIMULATION

Yufei Wang*, Zhou Xian*, Feng Chen*, Tsun-Hsuan Wang, Yian Wang, Katerina Fragkiadaki, Zackory Erickson, David Held, Chuang Gan

Paper Project Code

• RoboGen leverages the latest advancements in foundation and generative models. Instead of directly using or adapting these models to produce policies or low-level actions, we advocate for a generative scheme, which uses these models to automatically generate diversified tasks, scenes, and training supervisions, thereby scaling up robotic skill learning with minimal human supervision.

Thin-Shell Object Manipulations With Differentiable Physics Simulations

Yian Wang*, Juntian Zheng*, Zhehuan Chen, Zhou Xian, Gu Zhang, Chao Liu, Chuang Gan

Paper Project

• We introduce ThinShellLab - a fully differentiable simulation platform tailored for robotic interactions with diverse thin-shell materials possessing varying material properties, enabling flexible thin-shell manipulation skill learning and evaluation.

Articulated Object Manipulation with Coarse-to-fine Affordance for Mitigating the Effect of Point Cloud Noise

Suhan Ling*, Yian Wang*, Shiguang Wu, Yuzheng Zhuang, Tianyi Xu, Yu Li, Chang Liu, Hao Dong

Paper Project

• We leverage the property of real-world scanned point cloud that, the point cloud becomes less noisy when the camera is closer to the object. Therefore, we propose a novel coarse-to-fine affordance learning pipeline to mitigate the effect of point cloud noise in two stages.

MIXUP-AUGMENTED META-LEARNING FOR SAMPLE-EFFICIENT FINE-TUNING OF PROTEIN SIMULATORS

Jingbang Chen*, Yian Wang*, Xingwei Qv, ShuangJia Zheng, Yaodong Yang, Hao Dong, Jie Fu

Paper Code

• We apply soft prompt-based learning to molecular dynamics simulations, achieving strong generalization across various conditions with limited training data. The framework includes pre-training with data mixing and prompts, followed by meta-learning-based fine-tuning.

AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactions

Yian Wang*, Ruihai Wu*, Kaichun Mo*, Jiaqi Ke, Qingnan Fan, Leonidas J. Guibas, Hao Dong

Paper Project Code

• We propose a novel framework that learns to perform very few test-time interactions for quickly adapting the affordance priors to more accurate instance-specific posteriors by eliminating the dynamic and kinematic uncertainties.

VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objects

Ruihai Wu*, Yan Zhao*, Kaichun Mo*, Zizheng Guo, Yian Wang, Tianhao Wu, Qingnan Fan, Xuelin Chen, Leonidas J. Guibas, Hao Dong

Paper Project Code

• We propose an interaction-for-perception framework to predict dense geometry-aware, interaction-aware, and task-aware visual action affordance for 3D articulated objects.