Mechanical and Civil Engineering Seminar

Mechanical and Civil Engineering Seminar Series

Title: Perception-Rich Robot Autonomy with 3D World Models

Abstract: Recent advances in computer vision have led to the rise of highly expressive 3D scene models such as NeRFs and GSplats. More than just rendering lifelike images, these models allow robots to ground visual, semantic, physical, and affordance properties in a common 3D model, to rearrange objects in the scene and even simulate physical interactions. In this talk I will describe our efforts to build new robot autonomy features around these models, while preserving safety, modularity, and interpretability. I will present navigation algorithms for robots to safely maneuver through their environment using NeRFs and GSplats, while training the model online in a SLAM-like fashion. I will describe methods to embed semantic and affordance information into radiance fields, giving robots a 3D grounding for understanding and executing tasks from natural language commands. Finally, I will describe using these models as high-fidelity training environments for learning end-to-end visuo-motor policies. I will demonstrate such a policy for navigating a drone through an obstacle-rich environment while being robust to significant visual distractors and dynamic perturbations. I will conclude with future opportunities and challenges in 3D world models for robot autonomy.

Bio: Mac Schwager is an Associate Professor of Aeronautics and Astronautics and Computer Science (by courtesy) at Stanford University. He directs the Multi-robot Systems Lab (MSL) where he studies robot autonomy. He is interested in modular and end-to-end autonomy for UAVs, manipulators, and robotic vehicles, 3D mapping and SLAM, algorithmic and analytical tools for verifiable safety in learning-based autonomy, and collaborative intelligence in groups of robots and animals. He obtained his BS degree from Stanford, and his MS and PhD degrees from MIT. He was a postdoctoral researcher at the University of Pennsylvania and MIT. He received the NSF CAREER award in 2014, the DARPA YFA in 2018, and has received numerous best paper awards in conferences and journals including the IEEE Transactions on Robotics best paper award in 2016, the Best Paper Award in Robot Manipulation in ICRA 2018, and the Best Paper Award in Multi-Robot Systems in ICRA 2020.

NOTE: At this time, in-person Mechanical and Civil Engineering Lectures are open to all Caltech students/staff/faculty/visitors.