One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversity

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 3 functions out of this paper's own repositories and ran 3 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• skmur/onefish-twofish — marked official in Papers with Code; named in the paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Researchers in social science and psychology have recently proposed using large language models (LLMs) as replacements for humans in behavioral research. In addition to arguments about whether LLMs accurately capture population-level patterns, this has raised questions about whether LLMs capture human-like conceptual diversity. Separately, it is debated whether post-training alignment (RLHF or RLAIF) affects models' internal diversity. Inspired by human studies, we use a new way of measuring the conceptual diversity of synthetically-generated LLM "populations" by relating the internal variability of simulated individuals to the population-level variability. We use this approach to evaluate non-aligned and aligned LLMs on two domains with rich human behavioral data. While no model reaches human-like diversity, aligned models generally display less diversity than their instruction fine-tuned counterparts. Our findings highlight potential trade-offs between increasing models' value alignment and decreasing the diversity of their conceptual representations.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2411.04427")
get_code_for_paper("2411.04427")
have("2411.04427")

Connect an agent — have() is free.