Analysis and Applications of a Correlation-based Model for Neural Training

Neural networks have become a fundamental tool for most signal processing tasks, and data processing in general. The training process of these models, composed of millions of parameters, is typically based on a stochastic gradient decent variant, which is computationally intensive. Modeling and understanding the network dynamics throughout the training process has a growing interest, as larger and deeper networks are used. A recent approach for modeling the training dynamics is correlation mode decomposition (CMD) algorithm. This method aims to partition the network parameters into several clusters, where each cluster is characterized by a single common dynamic. CMD achieves substantial dimensionality reduction, with improved results, on a variety of networks. However, CMD has several limitations, including the fact that it is an offline algorithm, which performers post-training, and suffers from very high memory consumption. In this research we analyze the CMD algorithm from different perspectives. We introduce new observations regarding this method that help acquire a better understanding of the SGD training process. We propose an iterative approach that can work during the training process, mitigating the memory consumption issue. Additionally, an algorithm for gradual reduction in gradient computation is introduced. Moreover, we present how loss or accuracy landscapes can be visualized using CMD. Our proposed method is useful for modeling, accelerating, and optimizing SGD based training.

M.Sc. student under the supervision of Prof. Guy Gilboa.