We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Deep learning optimizers are often motivated through a mix of convex and approximate second-order theory. We select three such methods -- Adam, Shampoo and Prodigy -- and argue that each method can instead be understood as a squarely first-order method without convexity assumptions. In fact, after switching off exponential moving averages, each method is equivalent to steepest descent under a particular norm. By generalizing this observation, we chart a new design space for training algorithms. Different operator norms should be assigned to different tensors based on the role that the tensor plays within the network. For example, while linear and embedding layers may have the same weight space of $\mathbb{R}^{m\times n}$, these layers play different roles and should be assigned different norms. We hope that this idea of carefully metrizing the neural architecture might lead to more stable, scalable and indeed faster training.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2409.20325")
get_code_for_paper("2409.20325")
have("2409.20325")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2409.20325.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2409.20325)
Connect an agent — have() is free.