SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2109.04312 · EMNLP · 2021

MATE: Multi-view Attention for Table Transformer Efficiency

William W. Cohen, Thomas Müller, Maharshi Gor, Julian Martin Eisenschlos

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

This work presents a sparse-attention Transformer architecture for modeling documents that contain large tables. Tables are ubiquitous on the web, and are rich in information. However, more than 20% of relational tables on the web have 20 or more rows (Cafarella et al., 2008), and these large tables present a challenge for current Transformer models, which are typically limited to 512 tokens. Here we propose MATE, a novel Transformer architecture designed to model the structure of web tables. MATE uses sparse attention in a way that allows heads to efficiently attend to either rows or columns in a table. This architecture scales linearly with respect to speed and memory, and can handle documents containing more than 8000 tokens with current accelerators. MATE also has a more appropriate inductive bias for tabular data, and sets a new state-of-the-art for three table reasoning datasets. For HY-BRIDQA (Chen et al., 2020b), a dataset that involves large documents containing tables, we improve the best prior result by 19 points. * Work done at Google Research. 1 The term "semi-structured text" refers to text that has structure that does not reflect a known data schema. Typically semi-structured text is organized as an HTML tree or variable length lists and tables.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2109.04312")
get_code_for_paper("2109.04312")
have("2109.04312")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2109.04312.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2109.04312.svg)](https://syntology.ai/paper/2109.04312)

Connect an agent — have() is free.