SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2303.01215 · ICLR · 2023

Why (and When) does Local SGD Generalize Better than SGD?

Longbo Huang, Sanjeev Arora, Kaifeng Lyu, Xinran Gu

arXiv · PDF · Open in the Atlas

Code that ran

8 of 8 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 2 honoured the contract we drafted, 0 violated it and 5 have no contract verdict; 1 failure of our instrument; 0 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2303.01215.json · badge: /api/ran/2303.01215.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
hmgxr128/local-sgd canonical (classifier label)
classifier basis not recorded on the link
8 ran · 0 no recorded run
FunctionStatusWhere it lives
Estimate_BN_single Ran
ran on a synthesized input
hmgxr128/local-sgd/train_imagenet/train_utils.py
code served (permissive licence) · get_code("98ab004b1ad7d3de")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
import tempfile
import os

torch.manual_seed(0)

# Create a minimal model with BatchNorm2d layers
class MinimalModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 4, kernel_size=3, padding=1)
        self.bn1 = nn.BatchNorm2d(4)
        self.conv2 = nn.Conv2d(4, 8, kernel_size=3, padding=1)
        self.bn2 = nn.BatchNorm2d(8)
    
    def forward(self, x):
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.conv2(x)
        x = self.bn2(x)
        return x

# Create model
model = MinimalModel()

# Create a minimal dataset (list of (images, labels) tuples)
trainset = [
    (torch.randn(2, 3, 8, 8), torch.randint(0, 10, (2,)))
    for _ in range(4)
]

# Create a sequential sampler
sampler = torch.utils.data.SequentialSampler(trainset)

# Create args object with required attributes
class ArgsConfig:
    def __init__(self):
        self.nw = 0  # num_workers
        self.device = 'cpu'
        self.rank = 0
        # Create temporary directory for saving
        self.temp_dir = tempfile.mkdtemp()
        self.bn_param_pth = self.temp_dir + '/'

args = ArgsConfig()

# Ensure the directory exists
os.makedirs(args.bn_param_pth, exist_ok=True)

ARGS = {
    'model_list': [model],
    'trainset': trainset,
    'sampler': sampler,
    'args': args
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list — a list output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
Split_list Ran
ran on a synthesized input
hmgxr128/local-sgd/train_imagenet/train_utils.py
code served (permissive licence) · get_code("16e1aa6244b65b44")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Construct arguments for Split_list function
# The function takes a list and splits it into num_splits parts using np.array_split

# Create a simple list to split
lst = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

# Number of splits
num_splits = 3

# Define ARGS dict with all parameters
ARGS = {
    "lst": lst,
    "num_splits": num_splits
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list — a list output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_flat_buffers_from Ran
ran, honoured the contract we drafted
hmgxr128/local-sgd/train_imagenet/distributed_utils.py
code served (permissive licence) · get_code("555bde0b9b97c6ec")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 1, dtype family float, shape (14,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

class SimpleModelWithBuffers(nn.Module):
    def __init__(self):
        super().__init__()
        # Register buffers with names ending in 'mean' and 'var'
        self.register_buffer('bn_mean', torch.randn(4))
        self.register_buffer('bn_var', torch.randn(3))
        self.register_buffer('layer_mean', torch.randn(2))
        self.register_buffer('layer_var', torch.randn(5))
        # Also register a buffer that doesn't match the pattern
        self.register_buffer('other_buffer', torch.randn(6))

model = SimpleModelWithBuffers()

ARGS = {
    'model': model
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 1, dtype family float, shape (14,) — shape (14,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_flat_params_from Ran
ran on a synthesized input
hmgxr128/local-sgd/train_imagenet/distributed_utils.py
code served (permissive licence) · get_code("0ff68f73548c6eac")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 1, dtype family float, shape (23,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear1 = nn.Linear(4, 3)
        self.linear2 = nn.Linear(3, 2)
    
    def forward(self, x):
        x = self.linear1(x)
        x = self.linear2(x)
        return x

model = SimpleModel()

ARGS = {
    "model": model
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 1, dtype family float, shape (23,) — shape (23,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_flat_tensor_from_tensor_sequence Ran
ran, honoured the contract we drafted
hmgxr128/local-sgd/train_cifar/distributed_utils.py
code served (permissive licence) · get_code("cfe82136874645f0")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 1, dtype family float, shape (29,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# The function takes a sequence of tensors and flattens them
# It iterates through seq, calls view(-1) on each tensor, and concatenates them
# So we need a sequence (list, tuple, etc.) of tensors

seq = [
    torch.randn(2, 4),
    torch.randn(3, 5),
    torch.randn(1, 6)
]

ARGS = {
    "seq": seq
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 1, dtype family float, shape (29,) — shape (29,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_mean_flat_tensor_from_tensor_sequences Ran
ran; our contract draft was wrong, not the code
hmgxr128/local-sgd/train_cifar/distributed_utils.py
code served (permissive licence) · get_code("faa36bfafa971e5b")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 1, dtype family float, shape (12,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# The function takes a single parameter: seqs
# It expects seqs to be an iterable of sequences
# Each sequence should contain tensors that can be stacked together
# The function zips across sequences and stacks corresponding tensors

# Create a few tensor sequences
# Each sequence is a list of tensors
# Let's make 2 sequences, each with 3 tensors of shape (4,) to give variety

seq1 = [
    torch.randn(4),
    torch.randn(4),
    torch.randn(4),
]

seq2 = [
    torch.randn(4),
    torch.randn(4),
    torch.randn(4),
]

seqs = [seq1, seq2]

ARGS = {
    "seqs": seqs
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 1, dtype family float, shape (12,) — shape (12,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
is_bn Ran
ran on a synthesized input
hmgxr128/local-sgd/train_imagenet/train_utils.py
code served (permissive licence) · get_code("0ed7845068a97028")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# The function is_bn checks if an object m is an instance of BatchNorm1d, BatchNorm2d, or BatchNorm3d
# We need to provide a valid argument for parameter 'm'

# Test with a BatchNorm1d instance
bn1d = nn.BatchNorm1d(4)

ARGS = {
    "m": bn1d
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
reduce_value Ran
ran on a synthesized input
hmgxr128/local-sgd/train_imagenet/distributed_utils.py
code served (permissive licence) · get_code("e73752eb3a5d5a4b")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 1, dtype family float, shape (4,) as a later run recorded it. The image it ran in was not recorded. The code mentions cuda, and the harness aliases cuda to the CPU for every sample; whether this run needed that is not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.distributed as dist

torch.manual_seed(0)

# Create a small tensor for the value parameter
# The function expects a tensor that supports:
# - torch.cuda.synchronize() (will be called)
# - dist.all_reduce() (will be called)
# - division by world_size (value /= world_size)
value_tensor = torch.tensor([1.5, 2.5, 3.5, 4.5])

ARGS = {
    "value": value_tensor,
    "average": True
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 1, dtype family float, shape (4,) — shape (4,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block

Also present in other papers' code

Of 8 samples lifted from this paper's linked repositories, 8 carry a function identity and 0 do not. Of the 8 distinct functions with an identity, 4 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 4 later) and 4 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
get_flat_tensor_from_tensor_sequence 2 other papers
2 later
2306.05862 (2023-06)
Lessons from Generalization Error Analysis of Federated Learning : You May Communicate Less Often!
2310.14423 (2023-10)
A Quadratic Synchronization Rule for Distributed Deep Learning
get_mean_flat_tensor_from_tensor_sequences 2 other papers
2 later
2306.05862 (2023-06)
Lessons from Generalization Error Analysis of Federated Learning : You May Communicate Less Often!
2310.14423 (2023-10)
A Quadratic Synchronization Rule for Distributed Deep Learning
get_flat_buffers_from 1 other paper
1 later
2310.14423 (2023-10)
A Quadratic Synchronization Rule for Distributed Deep Learning
reduce_value 1 other paper
1 later
2306.05862 (2023-06)
Lessons from Generalization Error Analysis of Federated Learning : You May Communicate Less Often!

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Local SGD is a communication-efficient variant of SGD for large-scale training, where multiple GPUs perform SGD independently and average the model parameters periodically. It has been recently observed that Local SGD can not only achieve the design goal of reducing the communication overhead but also lead to higher test accuracy than the corresponding SGD baseline (Lin et al., 2020b), though the training regimes for this to happen are still in debate (Ortiz et al., 2021). This paper aims to understand why (and when) Local SGD generalizes better based on Stochastic Differential Equation (SDE) approximation. The main contributions of this paper include (i) the derivation of an SDE that captures the long-term behavior of Local SGD in the small learning rate regime, showing how noise drives the iterate to drift and diffuse after it has reached close to the manifold of local minima, (ii) a comparison between the SDEs of Local SGD and SGD, showing that Local SGD induces a stronger drift term that can result in a stronger effect of regularization, e.g., a faster reduction of sharpness, and (iii) empirical evidence validating that having a small learning rate and long enough training time enables the generalization improvement over SGD but removing either of the two conditions leads to no improvement.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2303.01215")
get_code_for_paper("2303.01215")
have("2303.01215")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2303.01215.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2303.01215.svg)](https://syntology.ai/paper/2303.01215)

Connect an agent — have() is free.