Skill Neologisms: Towards Skill-based Continual Learning

Mihaela Van Der Schaar, Antonin Berthon, Nicolas Astorga

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 5 functions out of this paper's own repositories and ran 3 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• antoninbrthn/skill-neologisms

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Modern LLMs show mastery over an evergrowing range of skills, as well as the ability to compose them flexibly. However, extending model capabilities to new skills in a scalable manner is an open problem: fine-tuning and parameter-efficient variants risk catastrophic forgetting, while context-based approaches have limited expressiveness and are constrained by the model's effective context. We explore skill neologisms-soft tokens integrated in the model's vocabulary and optimized to improve capabilities over a specific skill-as a way to selectively acquire new skills without weight updates. We first observe that pretrained LLMs already exhibit tokens associated with procedural knowledge. We then show on a controlled synthetic task that skill neologisms can be learned to improve model capabilities on specific skills while being composable with out-of-distribution skills, and that independently trained skill neologisms can be composed zero-shot. Finally, we validate zero-shot composition of independently learned skill neologisms on the more realistic natural language setting of the Skill-Mix benchmark (Yu et al., 2024). These results suggest that skill neologisms may provide a scalable path towards skill-based continual learning.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2605.04970")
get_code_for_paper("2605.04970")
have("2605.04970")

Connect an agent — have() is free.