Alexander Rush, Victor Sanh, Ella Charlaix, François Lagunas
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Pre-training has improved model accuracy for both classification and generation tasks at the cost of introducing much larger and slower models. Pruning methods have proven to be an effective way of reducing model size, whereas distillation methods are proven for speeding up inference. We introduce a block pruning approach targeting both small and fast models. Our approach extends structured methods by considering blocks of any size and integrates this structure into the movement pruning paradigm for fine-tuning. We find that this approach learns to prune out full components of the underlying model, such as attention heads. Experiments consider classification and generation tasks, yielding among other results a pruned model that is a 2.4x faster, 74% smaller BERT on SQuAD v1, with a 1% drop on F1, competitive both with distilled models in speed and pruned models in size.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2109.04838")
get_code_for_paper("2109.04838")
have("2109.04838")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2109.04838.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2109.04838)
Connect an agent — have() is free.