Special Year Learning Seminar
A Conformal Symplectic Perspective on Large Language Models
The transformer architecture, introduced in the 2017 paper Attention Is All You Need, forms the backbone of modern large language models (LLMs)—and gives ChatGPT its “T.” Despite the remarkable capabilities of these models, many aspects of their mathematical foundations remain poorly understood.
In this talk, we explore transformers through the lens of infinite-dimensional damped Hamiltonian systems and their conformal symplectic structure. We introduce a transformer architecture based on these systems, discuss the connection between tokens and the underlying infinite-dimensional dynamics, and present numerical results illustrating the approach.
The talk assumes no prior knowledge of LLMs, attention mechanisms, or infinite-dimensional Hamiltonian systems. The aim is to provide a self-contained introduction to this perspective and a starting point for further discussion.
This talk is based on ongoing joint work with W. Li, G. Steidl, and V. Stein.