How Differential Equations and Random Graph Insights Benefit Deep Learning

<<< All talks

How Differential Equations and Random Graph Insights Benefit Deep Learning

When	Friday, October 29, 2021, 2:00 PM – 3:00 PM (MT)
Where	MEB 3147

Abstract

We will present recent results on developing new deep learning algorithms leveraging differential equations and random graph insights. First, we will present a new class of continuous-depth deep neural networks that were motivated by the ODE limit of the classical momentum method, named heavy-ball neural ODEs (HBNODEs). HBNODEs enjoy two properties that imply practical advantages over NODEs: (i) The adjoint state of an HBNODE also satisfies an HBNODE, accelerating both forward and backward ODE solvers, thus significantly accelerate learning and improve the utility of the trained models. (ii) The spectrum of HBNODEs is well structured, enabling effective learning of long-term dependencies from complex sequential data. Second, we will extend HBNODE to graph learning leveraging diffusion on graphs, resulting in new algorithms for deep graph learning. The new algorithms are more accurate than existing deep graph learning algorithms and more scalable to deep architectures, and also suitable for learning at low labeling rate regimes. Moreover, we will present a fast multipole method-based efficient attention mechanism for modeling graph nodes interactions. Third, if time permits, we will discuss building an efficient and reliable overlay network for decentralized federated learning based on the random graph theory.

Speaker

Tags: algorithms & theory deep learning networks & graphs

---

Part of the Data Science Seminar. Something wrong on this page? Edit _data/talks/2021-10-29-bao-wang.toml.