Siyuan Huang, Yixuan Wang, Xinbing Wang, Zhouhan Lin, Boyi Zeng, Zhiyu Li, He Li, Ziwei He, Shixiang Song
We lifted 6 functions out of this paper's own repositories and ran 6 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| lumia-group/ponderinglm | canonical | 3 of 3 |
| copy not recorded | — | 3 of 3 |
| Function | Status | Where it lives |
|---|---|---|
| repeat_kv | Ran | this paper's copy was not recorded; identical code first harvested from fe1ixxu/ALMA pointer only · get_code("30d7eec482ebf6b1") |
| apply_rotary_pos_emb | Ran | this paper's copy was not recorded; identical code first harvested from fe1ixxu/ALMA pointer only · get_code("bac65c3dafaec040") |
| create_score_evaluation_response | Ran | lumia-group/ponderinglm/src/llamafactory/api/chat.py code served (permissive licence) · get_code("dbea5358be6b5ad0") |
| dictify | Ran | lumia-group/ponderinglm/src/llamafactory/api/common.py code served (permissive licence) · get_code("52a9ebcf2e9dee87") |
| jsonify | Ran | lumia-group/ponderinglm/src/llamafactory/api/common.py code served (permissive licence) · get_code("57583cf880d95444") |
| rotate_half | Ran | this paper's copy was not recorded; identical code first harvested from fe1ixxu/ALMA pointer only · get_code("b99eea6376d1e212") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Humans ponder before articulating complex sentence elements, enabling deeper cognitive processing through focused effort. In this work, we introduce this pondering process into language models by repeatedly invoking the forward process within a single token generation step. During pondering, instead of generating an actual token sampled from the prediction distribution, the model ponders by yielding a weighted sum of all token embeddings according to the predicted token distribution. The generated embedding is then fed back as input for another forward pass. We show that the model can learn to ponder in this way through self-supervised learning, without any human annotations. Experiments across three widely used open-source architectures-GPT-2, Pythia, and LLaMA-and extensive downstream task evaluations demonstrate the effectiveness and generality of our method. On 9 downstream benchmarks, our pondering-enhanced Pythia models significantly outperform the official Pythia models. Notably, our PonderPythia models demonstrate remarkable effectiveness: PonderPythia-
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2505.20674")
get_code_for_paper("2505.20674")
have("2505.20674")
Connect an agent — have() is free.