We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
The impact of gradient noise on training deep models is widely acknowledged but not well understood. In this context, we study the distribution of gradients during training. We introduce a method, Gradient Clustering, to minimize the variance of average mini-batch gradient with stratified sampling. We prove that the variance of average mini-batch gradient is minimized if the elements are sampled from a weighted clustering in the gradient space. We measure the gradient variance on common deep learning benchmarks and observe that, contrary to common assumptions, gradient variance increases during training, and smaller learning rates coincide with higher variance. In addition, we introduce normalized gradient variance as a statistic that better correlates with the speed of convergence compared to gradient variance.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2007.04532")
get_code_for_paper("2007.04532")
have("2007.04532")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2007.04532.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2007.04532)
Connect an agent — have() is free.