Biology Seminar

How the structure of data in the external world gets represented and used in the internal states of neural networks is a fundamental question in both AI and neuroscience. This question has a long history in computational neuroscience, and more recently, researchers are starting to make progress on this question in modern AI models. In this talk, I will focus on the temporal structure of information in natural language and how this is processed by large language models. We find that predictive information has a multiscale structure in long text documents — i.e. there is useful predictive information at all length scales that we can measure. This implies that the underlying data generating process must have an approximately self-similar form at all scales, strongly constraining hierarchical generative models of language. This result also grounds “scaling laws” observed in training language models where the performance keeps improving with more data and training; the multiscale structure of information implies there is always something more to learn.