Thema der Dissertation:
On Spatial Inductive Bias for Deep Learning in Medical Imaging Thema der Disputation:
Does Scaling Neural Networks Lead to Understanding?
On Spatial Inductive Bias for Deep Learning in Medical Imaging Thema der Disputation:
Does Scaling Neural Networks Lead to Understanding?
Abstract: Scaling neural networks trained on next-token prediction has produced models with increasingly broad and seemingly sophisticated knowledge of the world [1, 2]. But does improved prediction through scale eventually lead to an understanding of the underlying mechanisms that generate the observed data?
Controlled game-playing systems provide a setting in which this question can be studied empirically. Sequence-prediction neural networks trained only to predict legal moves have been shown to develop internal, linearly decodable representations of the underlying game state [3, 4]. These results suggest that sufficiently capable predictors can recover latent structure without being explicitly trained to represent it. Whether this phenomenon extends beyond constrained environments is less clear. In particular, can scaling lead models to recover general underlying laws that support extrapolation, or do learned representations remain dominated by task-specific correlations and local heuristics? In this talk, I examine this question through results on emergent world representations in game-playing systems [3] and recent evidence on the limits of physical extrapolation and inductive biases under scaling [5, 6]. References
[1] Jared Kaplan et al. Scaling Laws for Neural Language Models. arXiv:2001.08361 [cs]. Jan. 2020. doi: 10.48550/arXiv.2001.08361. url: http://arxiv.org/abs/2001.08361 (visited on 07/07/2025).
[2] Tom Brown et al. “Language models are few-shot learners”. In: Advances in neural information processing systems 33 (2020), pp. 1877–1901.
[3] Kenneth Li et al. “Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task”. In: International Conference on Learning Representations. 2023. url: https://openreview.net/forum?id=DeG07_TcZvT .
[4] Neel Nanda. Actually, Othello-GPT Has a Linear Emergent World Representation.Personal research write-up (not peer reviewed). Mar. 2023. url:https://www.neelnanda.io/mechanistic-interpretability/othello (visited on 08/31/2026).
[5] Bingyi Kang et al. “How Far Is Video Generation from World Model: A Physical Law Perspective”. In: Forty-second International Conference on Machine Learning. 2025. url: https://openreview.net/forum?id =DLlVjZQ7vD
[6] Keyon Vafa et al. “What Has a Foundation Model Found? Using Inductive Bias to Probe for World Models”. In: Proceedings of the 42nd International Conference on Machine Learning. Vol. 267. Proceedings of Machine Learning Research. 2025. arXiv: 2507.06952. url: https://proceedings.mlr.press/v267/vafa25a.html
Controlled game-playing systems provide a setting in which this question can be studied empirically. Sequence-prediction neural networks trained only to predict legal moves have been shown to develop internal, linearly decodable representations of the underlying game state [3, 4]. These results suggest that sufficiently capable predictors can recover latent structure without being explicitly trained to represent it. Whether this phenomenon extends beyond constrained environments is less clear. In particular, can scaling lead models to recover general underlying laws that support extrapolation, or do learned representations remain dominated by task-specific correlations and local heuristics? In this talk, I examine this question through results on emergent world representations in game-playing systems [3] and recent evidence on the limits of physical extrapolation and inductive biases under scaling [5, 6]. References
[1] Jared Kaplan et al. Scaling Laws for Neural Language Models. arXiv:2001.08361 [cs]. Jan. 2020. doi: 10.48550/arXiv.2001.08361. url: http://arxiv.org/abs/2001.08361 (visited on 07/07/2025).
[2] Tom Brown et al. “Language models are few-shot learners”. In: Advances in neural information processing systems 33 (2020), pp. 1877–1901.
[3] Kenneth Li et al. “Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task”. In: International Conference on Learning Representations. 2023. url: https://openreview.net/forum?id=DeG07_TcZvT .
[4] Neel Nanda. Actually, Othello-GPT Has a Linear Emergent World Representation.Personal research write-up (not peer reviewed). Mar. 2023. url:https://www.neelnanda.io/mechanistic-interpretability/othello (visited on 08/31/2026).
[5] Bingyi Kang et al. “How Far Is Video Generation from World Model: A Physical Law Perspective”. In: Forty-second International Conference on Machine Learning. 2025. url: https://openreview.net/forum?id =DLlVjZQ7vD
[6] Keyon Vafa et al. “What Has a Foundation Model Found? Using Inductive Bias to Probe for World Models”. In: Proceedings of the 42nd International Conference on Machine Learning. Vol. 267. Proceedings of Machine Learning Research. 2025. arXiv: 2507.06952. url: https://proceedings.mlr.press/v267/vafa25a.html
Zeit & Ort
07.10.2026 | 10:00
Seminarraum 051
(Fachbereich Mathematik und Informatik, Takustr.9, 14195 Berlin)
&
WebEx