Renjie Liao, Simon Kornblith, Geoffrey Hinton, Mengye Ren
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Forward gradient learning computes a noisy directional gradient and is a biologically plausible alternative to backprop for learning deep neural networks. However, the standard forward gradient algorithm, when applied naively, suffers from high variance when the number of parameters to be learned is large. In this paper, we propose a series of architectural and algorithmic modifications that together make forward gradient learning practical for standard deep learning benchmark tasks. We show that it is possible to substantially reduce the variance of the forward gradient estimator by applying perturbations to activations rather than weights. We further improve the scalability of forward gradient by introducing a large number of local greedy loss functions, each of which involves only a small number of learnable parameters, and a new MLPMixer-inspired architecture, LocalMixer, that is more suitable for local learning. Our approach matches backprop on MNIST and CIFAR-10 and significantly outperforms previously proposed backprop-free algorithms on Im-ageNet. Code is released at https://github.com/google-research/ google-research/tree/master/local_forward_gradient.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2210.03310")
get_code_for_paper("2210.03310")
have("2210.03310")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2210.03310.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2210.03310)
Connect an agent — have() is free.