MLP Can Be A Good Transformer Learner

Xiaojun Chang, Xiaodan Liang, Tao Tang, Dongrui Liu, Andy Song, Sihao Lin, Pumeng Lyu

Code that ran

We lifted 1 functions out of this paper's own repositories and ran 1 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repository	Role	Ran
sihaoevery/lambda_vit	—	1 of 1

Function	Status	Where it lives
Layer_scale_init_Block_paralx2	Ran	sihaoevery/lambda_vit/models_v2.py pointer only (licence: NOASSERTION) · get_code("97b9971c8c8372d6")

Repositories linked to this paper

sihaoevery/lambda_vit — marked official in Papers with Code; named in the paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

we propose to integrate the uninformative attention layers into their subsequent counterparts by degenerating them into identical mapping, yielding only MLP in certain transformer blocks. Experimental results on ImageNet-1k show that the proposed method can remove 40% attention layer of DeiT-B, improving throughput and memory bound without performance compromise.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2404.05657")
get_code_for_paper("2404.05657")
have("2404.05657")

Connect an agent — have() is free.