Data Science Seminar

In-Context Learning: A Case Study of Simple Function Classes

Shivam Garg

In-Context Learning: A Case Study of Simple Function Classes

When	Wednesday, February 14, 2024, 1:30 PM – 2:30 PM (MT)

Abstract

In-context learning refers to the ability of a model to learn new tasks from a sequence of input-output pairs given in a prompt. Crucially, this learning happens at inference time without any parameter updates to the model. I will discuss our empirical efforts that shed light on some basic aspects of in-context learning: To what extent can Transformers, or other models such as LSTMs be efficiently trained to in-context learn fundamental function classes, such as linear functions, sparse linear functions, and small decision trees? How can one evaluate in-context learning algorithms? And what are the qualitative differences between these architectures with respect to their ability to be trained to perform in-context learning? This is based on joint work with Dimitris Tsipras, Percy Liang, and Greg Valiant.

Speaker

Shivam Garg

Harvard

Shivam Garg is a postdoctoral fellow at Harvard University, hosted by Seth Neel and Boaz Barak. Before this, he completed his PhD from Stanford University, advised by Gregory Valiant. He is interested in the foundations of intelligence, in both artificial and natural systems.

Tags: algorithms & theory deep learning large language models

Part of the Data Science Seminar. Something wrong on this page? Edit _data/talks/2024-02-14-shivam-garg.toml.