Neural Networks Expressivity through the lens of Dynamical Systems

<<< All talks

Neural Networks Expressivity through the lens of Dynamical Systems

When	Friday, February 11, 2022, 3:00 PM – 4:00 PM (MT)

Abstract

Given a target function f, how large must a neural network be in order to approximate f? Understanding the representational power of Deep Neural Networks (DNNs) and how their structural properties (e.g., depth, width, type of activation unit) affect the functions they can compute, has been an important yet challenging question in approximation theory and deep learning even in the early days of AI.

In this talk, I want to tell you about some recent progress on this topic that uses ideas from dynamical systems. The main results are exponential depth-width trade-offs for DNNs representing certain families of functions. Our techniques rely on a generalized notion of fixed points, called periodic points that have played a major role in chaos theory (Li-Yorke chaos and Sharkovsky’s theorem).

Based on three recent works:

• with Ioannis Panageas, Sai Ganesh Nagarajan and Xiao Wang from ICLR’20 (spotlight): https://arxiv.org/abs/1912.04378 (https://arxiv.org/abs/1912.04378)
• with Ioannis Panageas and Sai Ganesh Nagarajan from ICML’20: https://arxiv.org/abs/2003.00777 (https://arxiv.org/abs/2003.00777)
• with Clayton Sanford from AISTATS’22: https://arxiv.org/abs/2110.10295 (https://arxiv.org/abs/2110.10295)

Speaker

Tags: deep learning

---

Part of the Data Science Seminar. Something wrong on this page? Edit _data/talks/2022-02-11-vaggos-chatziafratis.toml.