Lecture 1: Professor Demba Ba, Harvard University
Title: Deeply-Sparse Signal Representations, Artificial Neural Networks and Hierarchical Processing in the Brain
Abstract: Two important problems in neuroscience are to understand 1) how the brain represents sensory signals hierarchically and 2) how populations of neurons encode stimuli and how this encoding is related to behavior. My talk will focus on the tools I have developed to answer the first question. First, because they provide theoretical insights as to the complexity of learning deep neural networks. Second, because the framework behind these tools has implications on the principles of hierarchical processing the brain. I will show a strong parallel between deep neural network architectures and sparse recovery and estimation, namely that a deep neural network architecture with ReLU nonlinearities arises from a finite sequence of cascaded sparse coding models, the outputs of which, except for the last element in the cascade, are sparse and unobservable. I have shown that if the measurement matrices in the cascaded sparse coding model (a) satisfy RIP and (b) all have sparse columns except for the last, they can be recovered with high probability in the absence of noise using a sequential alternating-optimization algorithm. The method of choice in deep learning to solve this problem is by training a deep auto-encoder. My main result states that the complexity of learning this deep sparse coding model is given by the product of the number of active neurons (sparsity) in the deepest layer and its embedding dimension (of the sparse vector). More importantly, the theory gives a practical prescription for how, starting from the number of hidden units at the first layer, to pick the number of hidden units in all layers. I will demonstrate the usefulness of these ideas by showing that one can train auto-encoders to learn interpretable convolutional dictionaries in two applications, namely deconvolution of electrophysiology data and image denoising.
Bio: Demba Ba received the B.Sc. degree in electrical engineering from the University of Maryland, College Park, MD, USA, in 2004, and the M.Sci. and Ph.D. degrees in electrical engineering and computer science with a minor in mathematics from the Massachusetts Institute of Technology, Cambridge, MA, USA, in 2006 and 2011, respectively. In 2006 and 2009, he was a Summer Research Intern with the Communication and Collaboration Systems Group, Microsoft Research, Redmond, WA, USA. From 2011 to 2014, he was a Postdoctoral Associate with the MIT/Harvard Neuroscience Statistics Research Laboratory, where he developed theory and efficient algorithms to assess synchrony among large assemblies of neurons. He is currently an Assistant Professor of electrical engineering and bioengineering with Harvard University, where he directs the CRISP group. His research interests lie at the intersection of high-dimensional statistics, optimization and dynamic modeling, with applications to neuroscience and multimedia signal processing. Recently, he has taken a keen interest in the connection between neural networks, sparse signal processing, and hierarchical representations of sensory signals in the brain, as well as the implications of this connection on the design of data-adaptive digital signal processing hardware. In 2016, he was the recipient of a Research Fellowship in Neuroscience from the Alfred P. Sloan Foundation.
Reading:
- Convolutional neural networks via convolutional sparse coding
http://www.jmlr.org/papers/volume18/16-505/16-505.pdf
- Deeply-sparse Signal Representations
Recent Comments