We lifted 24 functions out of this paper's own repositories and ran 23 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| yuyangshu/retinavit | pwc_unofficial | 17 of 18 |
| copy not recorded | — | 4 of 4 |
| osiriszjq/impulse_init | extension | 1 of 1 |
| conceptofmind/Simple-ViT-flax | reimplementation | 1 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| Model | Ran | yuyangshu/retinavit/big_vision/models/mlp_mixer.py code served (permissive licence) · get_code("0a92f4de53a489b4") |
| SpatialConv2d_Linear_init | Ran | this paper's copy was not recorded; identical code first harvested from osiriszjq/impulse_init pointer only · get_code("0440e95034d883bd") |
| SpatialConv2d_init | Ran | this paper's copy was not recorded; identical code first harvested from osiriszjq/impulse_init pointer only · get_code("b04417d600247aaa") |
| dot_product_attention | Ran | yuyangshu/retinavit/big_vision/models/attn_override.py code served (permissive licence) · get_code("2c4ed3f75ab6ee14") |
| dot_product_attention_weights | Ran | yuyangshu/retinavit/big_vision/models/attn_override.py code served (permissive licence) · get_code("c8927d5d53a840b6") |
| fix_old_checkpoints | Ran | yuyangshu/retinavit/big_vision/models/bit.py code served (permissive licence) · get_code("75f9a36fc8351778") |
| get | Ran | yuyangshu/retinavit/big_vision/datasets/core.py code served (permissive licence) · get_code("f40ba1fc76a56621") |
| get_block_desc | Ran | yuyangshu/retinavit/big_vision/models/bit.py code served (permissive licence) · get_code("4f697aeaa3747cc2") |
| get_count | Ran | yuyangshu/retinavit/big_vision/optax.py code served (permissive licence) · get_code("5c44a598f86dadfc") |
| get_posemb | Ran | yuyangshu/retinavit/big_vision/models/vit.py code served (permissive licence) · get_code("ec1bf067066bb434") |
| impulse_init | Ran | osiriszjq/impulse_init/vit.py code served (permissive licence) · get_code("ce9f577538e842e0") |
| magnitude | Ran | yuyangshu/retinavit/big_vision/models/attn_override.py code served (permissive licence) · get_code("9d78a212d314b83e") |
| make_for_inference | Ran | yuyangshu/retinavit/big_vision/input_pipeline.py code served (permissive licence) · get_code("a0fec96972fe8326") |
| npload | Ran | yuyangshu/retinavit/big_vision/utils.py code served (permissive licence) · get_code("2b3549774154825a") |
| onehot | Ran | yuyangshu/retinavit/big_vision/utils.py code served (permissive licence) · get_code("7b17682564107576") |
| pad_shard_unpad | Ran | yuyangshu/retinavit/big_vision/utils.py code served (permissive licence) · get_code("5ad1f39431340fbb") |
| pair | Ran | this paper's copy was not recorded; identical code first harvested from LINs-lab/DynMoE pointer only · get_code("6ba8cee9f5daea41") |
| posemb_conv | Ran | yuyangshu/retinavit/big_vision/models/vit.py code served (permissive licence) · get_code("4f5dd3aec9da6ede") |
| posemb_sincos_2d | Ran | this paper's copy was not recorded; identical code first harvested from lucidrains/vit-pytorch pointer only · get_code("cfe18b9cc8afbfe9") |
| posemb_sincos_2d | Ran | conceptofmind/Simple-ViT-flax/simple_vit.py code served (permissive licence) · get_code("b848f8a0a583e58b") |
| posemb_sincos_2d | Ran | yuyangshu/retinavit/big_vision/models/vit.py code served (permissive licence) · get_code("2d67a8a756fe7cf8") |
| standardize | Ran | yuyangshu/retinavit/big_vision/models/bit_paper.py code served (permissive licence) · get_code("c272f024d815d59d") |
| weight_standardize | Ran | yuyangshu/retinavit/big_vision/models/bit.py code served (permissive licence) · get_code("37b141b05f87de19") |
| find_states | Not yet run | yuyangshu/retinavit/big_vision/optax.py code served (permissive licence) · get_code("1f64c95d3b75c46b") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
It is commonly accepted that the Vision Transformer model requires sophisticated regularization techniques to excel at ImageNet-1k scale data. Surprisingly, we find this is not the case and standard data augmentation is sufficient. This note presents a few minor modifications to the original Vision Transformer (ViT) vanilla training setting that dramatically improve the performance of plain ViT models. Notably, 90 epochs of training surpass 76% top-1 accuracy in under seven hours on a TPUv3-8, similar to the classic ResNet50 baseline, and 300 epochs of training reach 80% in less than one day.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2205.01580")
get_code_for_paper("2205.01580")
have("2205.01580")
Connect an agent — have() is free.