Laurence Aitchison, Johannes Von Oswald, Nicolas Zucchet, Simon Schug, Frederik Benzing, Robert Meier, Yassir Akram, Angelika Steger, © Benzing
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Neural networks trained with stochastic gradient descent (SGD) starting from different random initialisations typically find functionally very similar solutions, raising the question of whether there are meaningful differences between different SGD solutions. Entezari et al. recently conjectured that despite different initialisations, the solutions found by SGD lie in the same loss valley after taking into account the permutation invariance of neural networks. Concretely, they hypothesise that any two solutions found by SGD can be permuted such that the linear interpolation between their parameters forms a path without significant increases in loss. Here, we use a simple but powerful algorithm to find such permutations that allows us to obtain direct empirical evidence that the hypothesis is true in fully connected networks. Strikingly, we find that two networks already live in the same loss valley at the time of initialisation and averaging their random, but suitably permuted initialisation performs significantly above chance. In contrast, for convolutional architectures, our evidence suggests that the hypothesis does not hold. Especially in a large learning rate regime, SGD seems to discover diverse modes. 1 .
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2209.07509")
get_code_for_paper("2209.07509")
have("2209.07509")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2209.07509.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2209.07509)
Connect an agent — have() is free.