SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2104.07204 · NAACL · 2021

Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models

Yansong Feng, Dongyan Zhao, Songfang Huang, Yuxuan Lai, Yijia Liu

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Chinese pre-trained language models usually process text as a sequence of characters, while ignoring more coarse granularity, e.g., words. In this work, we propose a novel pre-training paradigm for Chinese -Lattice-BERT, which explicitly incorporates word representations along with characters, thus can model a sentence in a multi-granularity manner. Specifically, we construct a lattice graph from the characters and words in a sentence and feed all these text units into transformers. We design a lattice position attention mechanism to exploit the lattice structures in self-attention layers. We further propose a masked segment prediction task to push the model to learn from rich but redundant information inherent in lattices, while avoiding learning unexpected tricks. Experiments on 11 Chinese natural language understanding tasks show that our model can bring an average increase of 1.5% under the 12-layer setting, which achieves new state-of-the-art among base-size models on the CLUE benchmarks. Further analysis shows that Lattice-BERT can harness the lattice structures, and the improvement comes from the exploration of redundant information and multigranularity representations. 1

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2104.07204")
get_code_for_paper("2104.07204")
have("2104.07204")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2104.07204.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2104.07204.svg)](https://syntology.ai/paper/2104.07204)

Connect an agent — have() is free.