Shafiq Joty, Lav Varshney, Min-Yen Kan, Samson Tan
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Inflectional variation is a common feature of World Englishes such as Colloquial Singapore English and African American Vernacular English. Although comprehension by human readers is usually unimpaired by nonstandard inflections, current NLP systems are not yet robust. We propose Base-Inflection Encoding (BITE), a method to tokenize English text by reducing inflected words to their base forms before reinjecting the grammatical information as special symbols. Fine-tuning pretrained NLP models for downstream tasks using our encoding defends against inflectional adversaries while maintaining performance on clean data. Models using BITE generalize better to dialects with non-standard inflections without explicit training and translation models converge faster when trained with BITE. Finally, we show that our encoding improves the vocabulary efficiency of popular data-driven subword tokenizers. Since there has been no prior work on quantitatively evaluating vocabulary efficiency, we propose metrics to do so. 1
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2004.14870")
get_code_for_paper("2004.14870")
have("2004.14870")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2004.14870.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2004.14870)
Connect an agent — have() is free.