We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. We have no repository linked to it either. That is a statement about our graph, not about the paper.
Pipeline parallelism (PP) when training neural networks enables larger models to be partitioned spatially, leading to both lower network communication and overall higher hardware utilization. Unfortunately, to preserve the statistical efficiency of sequential training, existing PP techniques sacrifice hardware efficiency by decreasing pipeline utilization or incurring extra memory costs. In this paper, we investigate to what extent these sacrifices are necessary. We devise PipeMare, a simple yet robust training method that tolerates asynchronous updates during PP execution without sacrificing utilization or memory, which allows efficient use of fine-grained pipeline parallelism. Concretely, when tested on ResNet and Transformer networks, asynchrony enables PipeMare to use up to $2.7\times$ less memory or get $4.3\times$ higher pipeline utilization, with similar model quality, when compared to state-of-the-art synchronous PP training techniques.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("1910.05124")
get_code_for_paper("1910.05124")
have("1910.05124")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/1910.05124.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/1910.05124)
Connect an agent — have() is free.