סמינר: Graduate Seminar
The Knowledge Within: Low-Rank Gradient Approximation for Memory-Efficient Deep Learning
| Modern deep neural networks face growing costs in computation, memory, and communication. Approximation is a standard response; this seminar examines cases where model- and task-specific structure makes it more effective, briefly drawing on earlier work in data-free compression and out-of-distribution detection.
The main focus is memory-efficient optimization. Existing low-rank training methods exploit low-dimensional structure, but often emphasize reconstruction, deterministic subspaces, or systems efficiency rather than the statistical quality of the gradient estimator. Deterministic top-(k) methods are biased, while uninformed randomized estimators can have high variance. I present PLUMAGE, which uses spectrum-informed randomized sampling to control this tradeoff. For a fixed target gradient, it is unbiased and minimum-variance within the stated exact-(k) singular-component class. In practice, spectral information is reused between refreshes; experiments show that stale basis and singular-value information still reduce variance relative to uniform sampling and improve optimization over deterministic top-(k) alternatives. Ph.D. student Under the supervision of Prof. Daniel Soudry. Join Zoom Meeting Meeting agenda Meeting ID: 989 6295 3356 — Join by SIP Join instructions |