LADE Seminar

Emily Shana Cheng, Universitat Pompeu Fabra - Barcelona

Title: “Emergence of a High-Dimensional Abstraction Phase in Language Transformers”


A language model (LM) is a mapping from a linguistic context to an output token. However, much remains to be known about this mapping, including how its geometric properties relate to its function. We take a high-level geometric approach to its analysis, observing, across five pre-trained transformer-based LMs and three input datasets, a distinct phase characterized by high intrinsic dimensionality. During this phase, representations (1) correspond to the first full linguistic abstraction of the input; (2) are the first to viably transfer to downstream tasks; (3) predict each other across different LMs. Moreover, we find that an earlier onset of the phase strongly predicts better language modelling performance. In short, our results suggest that a central high-dimensionality phase underlies core linguistic processing in many common LM architectures.

Jun 19, 2024 11:30 AM — 12:30 PM
LADE Seminar
Meeting Room S-20, Building C, Area Science Park
Località Padriciano 99, Trieste, 34149
Area Science Park - RIT
Area Science Park - RIT
Research Institute

The Institute of Research and Innovation Technology (RIT) at Area Science Park carries out cutting-edge research and provides services and consulting to public and private-sector users through its three laboratories equipped with state-of-the-art technology.