Thema der Dissertation:
Self-Supervised multi-modal learning for Dense Visual Perception Thema der Disputation:
From Open-Loop Learning to Closed-Loop Driving: Bridging the Gap in Autonomous Driving
Self-Supervised multi-modal learning for Dense Visual Perception Thema der Disputation:
From Open-Loop Learning to Closed-Loop Driving: Bridging the Gap in Autonomous Driving
Abstract: Modern autonomous-driving systems are widely trained using large-scale datasets recorded with human-driven vehicles. In this open-loop setting, autonomous driving models learn by regressing the predicted actions of the model at recorded sequences of observations with their respective ground-truth actions performed by the driver. This setup enables the usage of large amount of real-word data while avoiding the safety and cost constraints associated with training directly through interactions with a vehicle.
However, the ultimate goal of an autonomous driving system is inherently closed loop. In such real-world settings, the model’s actions influence the vehicle’s future state and the behavior of other traffic participants, which alters the future observations it receives. This produces a fundamental gap between the conditions under which models are trained / validated and those under which they are ultimately deployed. Small prediction errors can accumulate over time causing the vehicle to increasingly deviate from the distribution of states present in the training data, causing closed-loop drift if not handled properly. Furthermore, conventional imitation learning struggles with inherently multimodal driving decisions, where multiple actions may be valid but imitation losses treat the human-driven action as the unique correct solution. These effects also hinder evaluation, where open-loop metrics are not enough to properly assess the driving performance of an autonomous driving system.
This presentation will examine the origins and consequences of the gap between open-loop and closed-loop settings and discuss why directly training on closed-loop real-world data remains challenging due to safety, cost and scalability constraints. We will review emerging approaches that attempt to bridge this gap while keeping the benefits of large-scale open-loop datasets, including the introduction of closed-loop learning objectives into open-loop training, as well as approaches based on simulation and photorealistic scene generation that enable closed-loop interaction during training. Finally, we will discuss the advantages, limitations, and remaining challenges of the existing approaches towards reliable real-world deployment of autonomous driving systems.
However, the ultimate goal of an autonomous driving system is inherently closed loop. In such real-world settings, the model’s actions influence the vehicle’s future state and the behavior of other traffic participants, which alters the future observations it receives. This produces a fundamental gap between the conditions under which models are trained / validated and those under which they are ultimately deployed. Small prediction errors can accumulate over time causing the vehicle to increasingly deviate from the distribution of states present in the training data, causing closed-loop drift if not handled properly. Furthermore, conventional imitation learning struggles with inherently multimodal driving decisions, where multiple actions may be valid but imitation losses treat the human-driven action as the unique correct solution. These effects also hinder evaluation, where open-loop metrics are not enough to properly assess the driving performance of an autonomous driving system.
This presentation will examine the origins and consequences of the gap between open-loop and closed-loop settings and discuss why directly training on closed-loop real-world data remains challenging due to safety, cost and scalability constraints. We will review emerging approaches that attempt to bridge this gap while keeping the benefits of large-scale open-loop datasets, including the introduction of closed-loop learning objectives into open-loop training, as well as approaches based on simulation and photorealistic scene generation that enable closed-loop interaction during training. Finally, we will discuss the advantages, limitations, and remaining challenges of the existing approaches towards reliable real-world deployment of autonomous driving systems.
Zeit & Ort
15.09.2026 | 11:00
Seminarraum E006
(Fachbereich Mathematik und Informatik, Königin-Luise-Str. 24-26, 14195 Berlin)