April 22, 2021 - Olga Russakovsky, Zhiwei Deng, Princeton COS

Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigation

Abstract: VisualAI lab focuses on bringing together the fields of computer vision, machine learning, human-machine interaction as well as fairness, accountability and transparency. In this talk, we will introduce the general goal of the lab, and how to build an agent that can understand and follow human’s language to perform tasks.

The ability to perform effective planning is crucial for building an instruction-following agent. When navigating through a new environment, an agent is challenged with (1) connecting the natural language instructions with its progressively growing knowledge of the world; and (2) performing long-range planning and decision making in the form of effective exploration and error correction. Current methods are still limited on both fronts despite extensive efforts. We introduce the Evolving Graphical Planner (EGP), a model that performs global planning for navigation based on raw sensory input. The model dynamically constructs a graphical representation, generalizes the action space to allow for more flexible decision making, and performs efficient planning on a proxy graph representation.  We demonstrate our model on a challenging Vision-and-Language Navigation task with photorealistic images and achieve superior performance compared to previous navigation architectures.

Bio: Dr. Olga Russakovsky is an Assistant Professor in the Computer Science Department at Princeton University. Her research is in computer vision, closely integrated with the fields of machine learning, human-computer interaction and fairness, accountability and transparency. She has been awarded the AnitaB.org's Emerging Leader Abie Award in honor of Denice Denton in 2020, the CRA-WP Anita Borg Early Career Award in 2020, the MIT Technology Review's 35-under-35 Innovator award in 2017, the PAMI Everingham Prize in 2016 and Foreign Policy Magazine's 100 Leading Global Thinkers award in 2015. In addition to her research, she co-founded and continues to serve on the Board of Directors of the AI4ALL foundation dedicated to increasing diversity and inclusion in Artificial Intelligence (AI). She completed her PhD at Stanford University in 2015 and her postdoctoral fellowship at Carnegie Mellon University in 2017. 

Dr. Zhiwei Deng is a postdoctoral researcher in the Computer Science Department at Princeton University. He completed his PhD in Computing Science in Simon Fraser University in 2019. His research interests mainly lie in computer vision, machine learning, and the integration with robotics. He is particularly interested in structured and compositional models, generative models and autonomous navigation. He has previously held research internship positions at Facebook Reality Labs and Disney Research.

-----------------------------------------------------------------------------------------
May 6, 2021 - Bartolomeo Stellato, Princeton ORFE