We lifted 7 functions out of this paper's own repositories and ran 2 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| yzh119/BPT | canonical | 2 of 7 |
| Function | Status | Where it lives |
|---|---|---|
| to_contiguous | Ran | yzh119/BPT/modules/op.py code served (permissive licence) · get_code("355eda071b2a6ed3") |
| unpack_params | Ran | yzh119/BPT/utils.py code served (permissive licence) · get_code("82c86b7826716ef2") |
| average_model | Not yet run | yzh119/BPT/utils.py code served (permissive licence) · get_code("1887f5eadaadc3aa") |
| masked_mm | Not yet run | yzh119/BPT/modules/op.py code served (permissive licence) · get_code("18d1d5a8e178b8d3") |
| move_to_device | Not yet run | yzh119/BPT/modules/beam.py code served (permissive licence) · get_code("6f00f4844df2c25b") |
| sparse_softmax | Not yet run | yzh119/BPT/modules/op.py code served (permissive licence) · get_code("281d17936edd3b56") |
| tensor_to_sequence | Not yet run | yzh119/BPT/utils.py code served (permissive licence) · get_code("257a29f25914a6ac") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
The Transformer model is widely successful on many natural language processing tasks. However, the quadratic complexity of self-attention limit its application on long text. In this paper, adopting a fine-to-coarse attention mechanism on multi-scale spans via binary partitioning (BP), we propose BP-Transformer (BPT for short). BPT yields $O(k\cdot n\log (n/k))$ connections where $k$ is a hyperparameter to control the density of attention. BPT has a good balance between computation complexity and model capacity. A series of experiments on text classification, machine translation and language modeling shows BPT has a superior performance for long text than previous self-attention models. Our code, hyperparameters and CUDA kernels for sparse attention are available in PyTorch.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("1911.04070")
get_code_for_paper("1911.04070")
have("1911.04070")
Connect an agent — have() is free.