Catherine Arnett, Ivan Yamshchikov, Pavel Chizhov, Elizaveta Korotkova
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Language models can greatly benefit from efficient tokenization. However, they still mostly utilize the classical Byte-Pair Encoding (BPE) algorithm, a simple and reliable method. BPE has been shown to cause such issues as undertrained tokens and sub-optimal compression that may affect the downstream performance. We introduce PickyBPE, a modified BPE algorithm that carries out vocabulary refinement during tokenizer training by removing merges that leave intermediate "junk" tokens. Our method improves vocabulary efficiency, eliminates under-trained tokens, and does not compromise text compression. Our experiments show that this method either improves downstream performance or does not harm it.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2409.04599")
get_code_for_paper("2409.04599")
have("2409.04599")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2409.04599.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2409.04599)
Connect an agent — have() is free.