SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2305.13009 · NeurIPS · 2023

Textually Pretrained Speech Language Models

Yossi Adi, Felix Kreuk, Itai Gat, Tal Remez, Roy Schwartz, Gabriel Synnaeve, Emmanuel Dupoux, Jade Copet, Michael Hassid, Tu Nguyen, Alexis Conneau, Alexandre Defossez

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Speech language models (SpeechLMs) process and generate acoustic data only, without textual supervision. In this work, we propose TWIST, a method for training SpeechLMs using a warm-start from a pretrained textual language models. We show using both automatic and human evaluations that TWIST outperforms a cold-start SpeechLM across the board. We empirically analyze the effect of different model design choices such as the speech tokenizer, the pretrained textual model, and the dataset size. We find that model and dataset scale both play an important role in constructing better-performing SpeechLMs. Based on our observations, we present the largest (to the best of our knowledge) SpeechLM both in terms of number of parameters and training data. We additionally introduce two spoken versions of the StoryCloze textual benchmark to further improve model evaluation and advance future research in the field. We make speech samples, code and models publicly available. 2 * Core Contributor. ♢ Work done while working at Meta. 2 https://pages.cs.huji.ac.il/adiyoss-lab/twist/ 3 E.g., podcasts, local radio, and video games. See https://www.insiderintelligence.com/content/ look-us-digital-audio-market-2022-how-big-who-s-listening-what-they-listening. 37th Conference on Neural Information Processing Systems (NeurIPS 2023).

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2305.13009")
get_code_for_paper("2305.13009")
have("2305.13009")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2305.13009.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2305.13009.svg)](https://syntology.ai/paper/2305.13009)

Connect an agent — have() is free.