Marzena Karpinska, Mohit Iyyer, Minh Pham, Alisha Srivastava, Kaan Korukluoglu, Minh Nhat, Le Duyen, Tran Chau, Lewis Carroll
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Large language models (LLMs) are known to memorize and recall English text from their pretraining data. However, the extent to which this ability generalizes to other languages or transfers across languages remains unclear. This paper investigates multilingual and cross-lingual memorization in LLMs, probing whether memorized content in one language (e.g., English) can be recalled when presented in a different language. To do so, we introduce OWL, a dataset of 31.5K aligned excerpts from 20 books in ten languages, including original English texts, official translations (Vietnamese, Spanish, Turkish), and new translations in six low-resource languages (Sesotho, Yoruba, Maithili, Malagasy, Setswana, Tahitian). We evaluate memorization across model families and sizes through three tasks: (1) direct probing, which asks the model to identify a book's title and author; (2) name cloze, which requires predicting masked character names; and (3) prefix probing, which involves generating continuations. We find that some LLMs consistently recall content across languages, even for texts without existing translation. GPT-4o, for example, identifies authors and titles 69.4% of the time and masked entities 6.3% of the time in newly translated excerpts. While perturbations (e.g., masking characters, shuffling words) reduce accuracy, the model's performance remains above chance level. Our results highlight the extent of cross-lingual memorization and provide insights on the differences between the models.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2505.22945")
get_code_for_paper("2505.22945")
have("2505.22945")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2505.22945.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2505.22945)
Connect an agent — have() is free.