Zhongzhan Huang, Liang Lin, Pan Zhou, Shuicheng Yan
We lifted 13 functions out of this paper's own repositories and ran 13 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| copy not recorded | — | 6 of 6 |
| YichengDWu/tinyedm | — | 5 of 5 |
| sail-sg/scalelong | — | 2 of 2 |
| Function | Status | Where it lives |
|---|---|---|
| Attention | Ran | sail-sg/scalelong/libs/uvit.py pointer only (licence: NONE) · get_code("a21d05625a5f984b") |
| Block | Ran | sail-sg/scalelong/libs/uvit.py pointer only (licence: NONE) · get_code("b4ecd978b7e2b6d9") |
| Conv2d | Ran | YichengDWu/tinyedm/src/tinyedm/networks.py code served (permissive licence) · get_code("2c4d769802a65492") |
| ScaleLong | Ran | YichengDWu/tinyedm/src/tinyedm/networks.py code served (permissive licence) · get_code("6c9210c75f7702f5") |
| mp_silu | Ran | YichengDWu/tinyedm/src/tinyedm/networks.py code served (permissive licence) · get_code("10b4cd4bf2697ea3") |
| normalize | Ran | YichengDWu/tinyedm/src/tinyedm/networks.py code served (permissive licence) · get_code("2add1f3e88daf919") |
| patchify | Ran | this paper's copy was not recorded; identical code first harvested from baofff/U-ViT pointer only · get_code("7a7de2a8e080f047") |
| patchify | Ran | this paper's copy was not recorded; identical code first harvested from baofff/u-vit pointer only · get_code("9cf076a2ae913f94") |
| pixel_norm | Ran | YichengDWu/tinyedm/src/tinyedm/networks.py code served (permissive licence) · get_code("029abee41f338892") |
| timestep_embedding | Ran | this paper's copy was not recorded; identical code first harvested from baofff/U-ViT pointer only · get_code("262b18d8278dcf70") |
| timestep_embedding | Ran | this paper's copy was not recorded; identical code first harvested from baofff/u-vit pointer only · get_code("b494dd297ded5960") |
| unpatchify | Ran | this paper's copy was not recorded; identical code first harvested from baofff/U-ViT pointer only · get_code("82e1ad8a9b00554e") |
| unpatchify | Ran | this paper's copy was not recorded; identical code first harvested from baofff/u-vit pointer only · get_code("f9244c08c1011341") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
In diffusion models, UNet is the most popular network backbone, since its long skip connects (LSCs) to connect distant network blocks can aggregate long-distant information and alleviate vanishing gradient. Unfortunately, UNet often suffers from unstable training in diffusion models which can be alleviated by scaling its LSC coefficients smaller. However, theoretical understandings of the instability of UNet in diffusion models and also the performance improvement of LSC scaling remain absent yet. To solve this issue, we theoretically show that the coefficients of LSCs in UNet have big effects on the stableness of the forward and backward propagation and robustness of UNet. Specifically, the hidden feature and gradient of UNet at any layer can oscillate and their oscillation ranges are actually large which explains the instability of UNet training. Moreover, UNet is also provably sensitive to perturbed input, and predicts an output distant from the desired output, yielding oscillatory loss and thus oscillatory gradient. Besides, we also observe the theoretical benefits of the LSC coefficient scaling of UNet in the stableness of hidden features and gradient and also robustness. Finally, inspired by our theory, we propose an effective coefficient scaling framework ScaleLong that scales the coefficients of LSC in UNet and better improve the training stability of UNet. Experimental results on four famous datasets show that our methods are superior to stabilize training, and yield about 1.5× training acceleration on different diffusion models with UNet or UViT backbones. Click here for Code.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2310.13545")
get_code_for_paper("2310.13545")
have("2310.13545")
Connect an agent — have() is free.