SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2502.02199 · 2025

When Dimensionality Hurts: The Role of LLM Embedding Compression for Noisy Regression Tasks

Janet Pierrehumbert, Felix Drinkall, Stefan Zohren

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Large language models (LLMs) have shown remarkable success in language modelling due to scaling laws found in model size and the hidden dimension of the model's text representation. Yet, we demonstrate that compressed representations of text can yield better performance in LLM-based regression tasks. In this paper, we compare the relative performance of embedding compression in three different signal-to-noise contexts: financial return prediction, writing quality assessment and review scoring. Our results show that compressing embeddings, in a minimally supervised manner using an autoencoder's hidden representation, can mitigate overfitting and improve performance on noisy tasks, such as financial return prediction; but that compression reduces performance on tasks that have high causal dependencies between the input and target data. Our results suggest that the success of interpretable compressed representations such as sentiment may be due to a regularising effect.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2502.02199")
get_code_for_paper("2502.02199")
have("2502.02199")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2502.02199.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2502.02199.svg)](https://syntology.ai/paper/2502.02199)

Connect an agent — have() is free.