SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2108.00801 · ACL Findings · 2021

LICHEE: Improving Language Model Pre-training with Multi-grained Tokenization

Di Niu, Zhenyang Li, Weidong Guo, Mingjun Zhao, Lusheng Zhang, Jinwen Luo, Zhenhua Liu, Jianbo Tang, Content Group

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Language model pre-training based on large corpora has achieved tremendous success in terms of constructing enriched contextual representations and has led to significant performance gains on a diverse range of Natural Language Understanding (NLU) tasks. Despite the success, most current pre-trained language models, such as BERT, are trained based on single-grained tokenization, usually with fine-grained characters or sub-words, making it hard for them to learn the precise meaning of coarse-grained words and phrases. In this paper, we propose a simple yet effective pretraining method named LICHEE to efficiently incorporate multi-grained information of input text. Our method can be applied to various pretrained language models and improve their representation capability. Extensive experiments conducted on CLUE and SuperGLUE demonstrate that our method achieves comprehensive improvements on a wide variety of NLU tasks in both Chinese and English with little extra inference cost incurred, and that our best ensemble model achieves the state-of-the-art performance on CLUE benchmark competition. * * Equal contribution.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2108.00801")
get_code_for_paper("2108.00801")
have("2108.00801")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2108.00801.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2108.00801.svg)](https://syntology.ai/paper/2108.00801)

Connect an agent — have() is free.