We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. We have no repository linked to it either. That is a statement about our graph, not about the paper.
Large language models are versatile tools but are not suitable for small inference budgets. Small models have more efficient inference, but their lower capacity means that their performance can be good only if one limits their scope to a specialized domain. This paper explores how to get good specialized small language models using a large, generic, pretraining set and a limited amount of specialized data. We consider two scenarios, depending on whether (i) one can afford pretraining a model for each specialization task, or (ii) one wants to cheaply adapt a single pretrained model for each task. In the first scenario, we propose an effective solution based on importance sampling: we resample the pretraining set to imitate the specialization data and train a small model on it. In the second scenario, we propose a novel architecture, projected networks (PN). PN is a large network whose parameters can be linearly projected into a small network for specialization. For both scenarios, we demonstrate the empirical effectiveness of our solutions across various domains, training set sizes, and training budgets.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2402.01093")
get_code_for_paper("2402.01093")
have("2402.01093")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2402.01093.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2402.01093)
Connect an agent — have() is free.