SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 1608.04465 · 2016

Fast, Small and Exact: Infinite-order Language Modelling with Compressed Suffix Trees

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Efficient methods for storing and querying are critical for scaling high-order n-gram language models to large corpora. We propose a language model based on compressed suffix trees, a representation that is highly compact and can be easily held in memory, while supporting queries needed in computing language model probabilities on-the-fly. We present several optimisations which improve query runtimes up to 2500x, despite only incurring a modest increase in construction time and memory usage. For large corpora and high Markov orders, our method is highly competitive with the state-of-the-art KenLM package. It imposes much lower memory requirements, often by orders of magnitude, and has runtimes that are either similar (for training) or comparable (for querying).

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("1608.04465")
get_code_for_paper("1608.04465")
have("1608.04465")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/1608.04465.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/1608.04465.svg)](https://syntology.ai/paper/1608.04465)

Connect an agent — have() is free.