From classical statistics to modern deep learning

<<< All talks

From classical statistics to modern deep learning

When	Friday, November 12, 2021, 2:00 PM – 3:00 PM (MT)

Abstract

Recent empirical successes of deep learning have exposed significant gaps in our fundamental understanding of learning and optimization mechanisms. Modern best practices for model selection are in direct contradiction to the methodologies suggested by classical analyses. Similarly, the efficiency of SGD-based local methods used in training modern models, appeared at odds with the standard intuitions on optimization.

First, I will present evidence, empirical and mathematical, that necessitates revisiting classical statistical notions, such as over-fitting. I will continue to discuss the emerging understanding of generalization, and, in particular, the “double descent” risk curve, which extends the classical U-shaped generalization curve beyond the point of interpolation.

Second, I will discuss why the landscapes of over-parameterized neural networks are generically never convex, even locally. Instead they satisfy the Polyak-Lojasiewicz (PL) condition across most of the parameter space instead, presents an powerful framework for optimization in general over-parameterized models and allows SGD-type methods to converge to a global minimum.

While our understanding has significantly grown in the last few years, a key piece of the puzzle remains – how does optimization align with statistics to form the complete mathematical picture of modern ML?

Speaker

Tags: deep learning machine learning optimization statistics

---

Part of the Data Science Seminar. Something wrong on this page? Edit _data/talks/2021-11-12-mikhail-belkin.toml.