Sho Takase, Tatsuya Hiraoka, Kei Uchiumi, Atsushi Keyaki, Naoaki Okazaki
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Since traditional tokenizers are isolated from a downstream task and model, they cannot output an appropriate tokenization depending on the task and model, although recent studies imply that the appropriate tokenization improves the performance. In this paper, we propose a novel method to find an appropriate tokenization to a given downstream model by jointly optimizing a tokenizer and the model. The proposed method has no restriction except for using loss values computed by the downstream model to train the tokenizer, and thus, we can apply the proposed method to any NLP task. Moreover, the proposed method can be used to explore the appropriate tokenization for an already trained model as post-processing. Therefore, the proposed method is applicable to various situations. We evaluated whether our method contributes to improving performance on text classification in three languages and machine translation in eight language pairs. Experimental results show that our proposed method improves the performance by determining appropriate tokenizations.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2105.12410")
get_code_for_paper("2105.12410")
have("2105.12410")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2105.12410.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2105.12410)
Connect an agent — have() is free.