LADE Seminar

Sebastian Goldt (SISSA)

Title: The Gaussian world is not enough – how training data shapes neural representations

Abstract

What do neural networks learn from their data? We discuss this question in two learning paradigms: supervised classification with feed-forward networks, and masked language modelling with transformers. First, we give analytical and experimental evidence for a “distributional simplicity bias”, whereby neural networks learn increasingly complex distributions of their inputs. We then show that neural networks learn from the higher-order cumulants (HOCs) more efficiently than lazy methods, and show how HOCs shape the learnt features. We finally characterise the distributions that are learnt by single- and multi-layer transformers, and show a similar distributional simplicity bias for masked language modelling.

Date
Nov 29, 2023 11:30 AM — 12:30 PM
Event
LADE Seminar
Location
TBD, Centro Congressi, Area Science Park
Località Padriciano 99,, Trieste, 34149
Area Science Park - RIT
Area Science Park - RIT
Research Institute

The Institute of Research and Innovation Technology (RIT) at Area Science Park carries out cutting-edge research and provides services and consulting to public and private-sector users through its three laboratories equipped with state-of-the-art technology.