SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2602.15902 · ICML · 2026

Doc-to-LoRA: Learning to Instantly Internalize Contexts

Edoardo Cetin, Robert Lange, Rujikorn Charakorn, Shinnosuke Uesaka

arXiv · PDF · Open in the Atlas

Code that ran

18 of 32 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 2 honoured the contract we drafted, 0 violated it and 5 have no contract verdict; 11 failures of our instrument; 14 no recorded run), measured 2026-11-22; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2602.15902.json · badge: /api/ran/2602.15902.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
SakanaAI/doc-to-lora community
Papers with Code archive
18 ran · 14 no recorded run
FunctionStatusWhere it lives
DiscreteOneHotTaskEncoder Ran
ran on a synthesized input
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("21cbf833527e000d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, class construct-then-call prompt, model not recorded) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from torch import nn

class DiscreteOneHotTaskEncoder(nn.Module):

    def __init__(self, task_emb_size: int, n_classes: int, n_embs: int):
        super().__init__()
        self.n_classes = n_classes
        self.n_embs = n_embs
        self.encoded_task_emb_size = n_classes * n_embs
        self.mlp = nn.Sequential(nn.Linear(task_emb_size, self.encoded_task_emb_size * 4), nn.SiLU(), nn.Dropout(0.05), nn.Linear(self.encoded_task_emb_size * 4, self.encoded_task_emb_size))

    def forward(self, x: torch.Tensor) -> dict[str, torch.Tensor]:
        logits = self.mlp(x)
        logits = logits.reshape(*logits.shape[:-1], self.n_embs, self.n_classes)
        if self.training:
            dist = torch.distributions.one_hot_categorical.OneHotCategorical(logits=logits)
            one_hot = dist.sample() + dist.probs - dist.probs.detach()
        else:
            one_hot = torch.eye(self.n_classes, device=x.device)[torch.argmax(logits, dim=-1)]
        return {'encoded_task_emb': torch.flatten(one_hot, start_dim=-2, end_dim=-1)}


INIT_ARGS = {
    'task_emb_size': 4,
    'n_classes': 3,
    'n_embs': 2,
}

ARGS = {
    'x': torch.randn(2, 4),
}

if __name__ == '__main__':
    model = DiscreteOneHotTaskEncoder(**INIT_ARGS)
    model.eval()
    output = model(**ARGS)
    print(output)
Setup block drafted by
a language model, class construct-then-call prompt, model not recorded
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: torch and numpy seeded to 0 by the harness immediately before the class was constructed from INIT_ARGS (fixture_check.py)
MLPResidualBlock Ran
ran on a synthesized input
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("96865de164c308ad")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, class construct-then-call prompt, model not recorded) and returned Tensor, ndim 2, dtype family float, shape (2, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from torch import nn

class MLPResidualBlock(nn.Module):

    def __init__(self, input_size, hidden_size, output_size, pre_layer_norm, post_dropout):
        super().__init__()
        layers = []
        if pre_layer_norm:
            layers.append(nn.LayerNorm(input_size))
        layers += [nn.Linear(input_size, hidden_size), nn.SiLU(), nn.Dropout(0.05), nn.Linear(hidden_size, output_size), nn.SiLU()]
        if post_dropout:
            layers.append(nn.Dropout(0.05))
        self.mlp = nn.Sequential(*layers)

    def forward(self, x):
        return x + self.mlp(x)


INIT_ARGS = {
    'input_size': 4,
    'hidden_size': 8,
    'output_size': 4,
    'pre_layer_norm': True,
    'post_dropout': True,
}

ARGS = {
    'x': torch.randn(2, 4),
}

if __name__ == '__main__':
    block = MLPResidualBlock(**INIT_ARGS)
    output = block(**ARGS)
    print(f"Input shape: {ARGS['x'].shape}")
    print(f"Output shape: {output.shape}")
    print(f"Output:\n{output}")
Setup block drafted by
a language model, class construct-then-call prompt, model not recorded
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 2, dtype family float, shape (2, 4) — shape (2, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: torch and numpy seeded to 0 by the harness immediately before the class was constructed from INIT_ARGS (fixture_check.py)
SoftmaxTaskEncoder Ran
ran on a synthesized input
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("aeca7914797f1f3a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, class construct-then-call prompt, model not recorded) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from torch import nn
import torch.nn.functional as F

class SoftmaxTaskEncoder(nn.Module):

    def __init__(self, task_emb_size: int, code_dim: int, n_embs: int):
        super().__init__()
        self.n_embs = n_embs
        self.code_dim = code_dim
        self.mlp = nn.Sequential(nn.Linear(task_emb_size, code_dim * 4), nn.SiLU(), nn.Dropout(0.05), nn.Linear(code_dim * 4, n_embs))
        self.embed = nn.Parameter(torch.randn(n_embs, code_dim), requires_grad=True)
        self.ln = nn.LayerNorm(code_dim)

    def forward(self, x: torch.Tensor) -> dict[str, torch.Tensor]:
        logits = self.mlp(x).reshape(-1, self.n_embs)
        probs = F.softmax(logits, dim=-1)
        emb = probs.unsqueeze(-1) * self.embed.unsqueeze(0)
        emb = emb.sum(1)
        emb = self.ln(emb)
        return {'encoded_task_emb': emb}


INIT_ARGS = {
    'task_emb_size': 4,
    'code_dim': 3,
    'n_embs': 5,
}

ARGS = {
    'x': torch.randn(2, 4),
}

if __name__ == '__main__':
    model = SoftmaxTaskEncoder(**INIT_ARGS)
    output = model(**ARGS)
    print(output)
    print(output['encoded_task_emb'].shape)
Setup block drafted by
a language model, class construct-then-call prompt, model not recorded
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: torch and numpy seeded to 0 by the harness immediately before the class was constructed from INIT_ARGS (fixture_check.py)
TaskEncoder Ran
ran on a synthesized input
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("656b1d9383fc1abf")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, class construct-then-call prompt, model not recorded) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from torch import nn

class TaskEncoder(nn.Module):

    def __init__(self, task_emb_size: int, encoded_task_emb_size: int):
        super().__init__()
        self.encoded_task_emb_size = encoded_task_emb_size
        self.mlp = nn.Sequential(nn.Linear(task_emb_size, encoded_task_emb_size), nn.LayerNorm(encoded_task_emb_size))

    def get_one_hot_task_emb(self, num_tasks: int, task_idx: torch.Tensor) -> torch.Tensor:
        return torch.eye(num_tasks, device=task_idx.device)[task_idx]

    def forward(self, x: torch.Tensor) -> dict[str, torch.Tensor]:
        return {'encoded_task_emb': self.mlp(x)}


INIT_ARGS = {
    'task_emb_size': 4,
    'encoded_task_emb_size': 3,
}

ARGS = {
    'x': torch.randn(2, 4),
}

if __name__ == '__main__':
    model = TaskEncoder(**INIT_ARGS)
    output = model(**ARGS)
    print(output)
    print(output['encoded_task_emb'].shape)
Setup block drafted by
a language model, class construct-then-call prompt, model not recorded
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: torch and numpy seeded to 0 by the harness immediately before the class was constructed from INIT_ARGS (fixture_check.py)
VQTaskEncoder Ran
ran on a synthesized input
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("2c3f7d0ccd979162")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, class construct-then-call prompt, model not recorded) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from torch import nn
import torch.nn.functional as F

class VQTaskEncoder(nn.Module):

    def __init__(self, task_emb_size: int, code_dim: int, n_embs: int, decay: float=0.99):
        super().__init__()
        self.n_embs = n_embs
        self.code_dim = code_dim
        self.decay = decay
        self.mlp = nn.Sequential(nn.Linear(task_emb_size, code_dim * 4), nn.SiLU(), nn.Dropout(0.05), nn.Linear(code_dim * 4, code_dim))
        self.ln = nn.LayerNorm(code_dim)
        self.register_buffer('embed', torch.randn(code_dim, n_embs))
        self.register_buffer('cluster_size', torch.zeros(n_embs))
        self.register_buffer('embed_avg', self.embed.clone())

    def embed_code(self, embed_id):
        return F.embedding(embed_id, self.embed.transpose(0, 1))

    def forward(self, x: torch.Tensor) -> dict[str, torch.Tensor]:
        z_e = self.mlp(x).reshape(-1, self.code_dim)
        dist = z_e.pow(2).sum(1, keepdim=True) - 2 * z_e @ self.embed + self.embed.pow(2).sum(0, keepdim=True)
        indices = torch.argmin(dist, dim=-1)
        one_hot = F.one_hot(indices, self.n_embs).to(z_e.dtype)
        indices = indices.view(*x.shape[:-1])
        quantize = self.embed_code(indices)
        if self.training:
            embed_onehot_sum = one_hot.sum(0)
            embed_sum = z_e.transpose(0, 1) @ one_hot
            self.cluster_size.data.mul_(self.decay).add_(embed_onehot_sum, alpha=1 - self.decay)
            self.embed_avg.data.mul_(self.decay).add_(embed_sum, alpha=1 - self.decay)
            n = self.cluster_size.sum()
            cluster_size = (self.cluster_size + 1e-05) / (n + self.n_embs * 1e-05) * n
            embed_normalized = self.embed_avg / cluster_size.unsqueeze(0)
            self.embed.data.copy_(embed_normalized)
        commitment_cost = 0.01
        diff = commitment_cost * (quantize.detach() - z_e).pow(2).mean()
        quantize = self.ln(z_e + (quantize - z_e).detach())
        return {'encoded_task_emb': quantize, 'loss': diff}


INIT_ARGS = {
    'task_emb_size': 5,
    'code_dim': 4,
    'n_embs': 3,
    'decay': 0.99
}

ARGS = {
    'x': torch.randn(2, 5)
}

if __name__ == '__main__':
    model = VQTaskEncoder(**INIT_ARGS)
    model.train()
    output = model(**ARGS)
    print("Output keys:", output.keys())
    print("Encoded task emb shape:", output['encoded_task_emb'].shape)
    print("Loss:", output['loss'].item())
Setup block drafted by
a language model, class construct-then-call prompt, model not recorded
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: torch and numpy seeded to 0 by the harness immediately before the class was constructed from INIT_ARGS (fixture_check.py)
add_full_stop Ran
ran; our contract draft was wrong, not the code
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("e6ae5d7b6ce64ac3")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

def add_full_stop(s):
    s = s.strip()
    if s[-1].isalpha():
        s += '.'
    return s

ARGS = {
    's': 'hello world'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
apply_rotary_pos_emb Ran
ran; our contract draft was wrong, not the code
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("22f75dc2bea7b46e")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned Tensor, ndim 2, dtype family float, shape (2, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Create small tensors for the function
batch_size = 2
hidden_dim = 4

# x: input tensor with shape (batch_size, hidden_dim)
x = torch.randn(batch_size, hidden_dim)

# cos: cosine positional embedding, same shape as x
cos = torch.randn(batch_size, hidden_dim)

# sin: sine positional embedding, same shape as x
sin = torch.randn(batch_size, hidden_dim)

ARGS = {
    "x": x,
    "cos": cos,
    "sin": sin,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 2, dtype family float, shape (2, 4) — shape (2, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
cls_pool Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("ed175170e2628076")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 2, dtype family float, shape (2, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# The function expects:
# 1. outputs: dict with 'last_hidden_state' key containing a tensor
# 2. attention_mask: tensor where we check if padding is right-aligned

# Create a small batch with batch_size=2, sequence_length=3, hidden_dim=4
batch_size = 2
seq_length = 3
hidden_dim = 4

# outputs dict with last_hidden_state tensor (batch_size, seq_length, hidden_dim)
outputs = {
    'last_hidden_state': torch.randn(batch_size, seq_length, hidden_dim)
}

# attention_mask: (batch_size, seq_length)
# For right padding, all tokens should be attended to (1s followed by 0s if any)
# The check is: attention_mask[:, 0].sum() == attention_mask.shape[0]
# This means all first tokens should be 1 (padding_side='right')
attention_mask = torch.ones(batch_size, seq_length, dtype=torch.long)

ARGS = {
    'outputs': outputs,
    'attention_mask': attention_mask
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 2, dtype family float, shape (2, 4) — shape (2, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
embed_texts Ran
ran, honoured the contract we drafted
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("895bcd8257169039")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 2, dtype family float, shape (2, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
from math import sqrt

torch.manual_seed(0)

# Small embedding model
class TinyEmbeddingModel(nn.Module):
    def __init__(self, hidden_dim=4):
        super().__init__()
        self.hidden_dim = hidden_dim
        self.embedding = nn.Embedding(100, hidden_dim)
        self.linear = nn.Linear(hidden_dim, hidden_dim)
    
    def forward(self, input_ids, attention_mask, output_hidden_states=True):
        embedded = self.embedding(input_ids)
        output = self.linear(embedded)
        return type('Output', (), {
            'last_hidden_state': output,
            'hidden_states': [output]
        })()

# Small tokenizer stand-in
class TinyTokenizer:
    def __call__(self, texts, truncation=True, padding=True, max_length=8192, return_tensors='pt'):
        batch_size = len(texts)
        seq_length = 3
        
        input_ids = torch.randint(1, 100, (batch_size, seq_length))
        attention_mask = torch.ones(batch_size, seq_length)
        
        return {
            'input_ids': input_ids,
            'attention_mask': attention_mask
        }

# Simple pooling function
def pooling_fn(outputs, attention_mask):
    hidden_states = outputs.last_hidden_state
    mask_expanded = attention_mask.unsqueeze(-1).expand(hidden_states.size())
    sum_hidden = (hidden_states * mask_expanded).sum(dim=1)
    sum_mask = mask_expanded.sum(dim=1)
    pooled = sum_hidden / sum_mask.clamp(min=1e-9)
    return pooled

# Task description format function
def task_desc_format_fn(text):
    return f"Task: {text}"

# Create instances
emb_model = TinyEmbeddingModel(hidden_dim=4)
emb_tokenizer = TinyTokenizer()
device = 'cpu'

# Sample texts
texts = ["text one", "text two"]

ARGS = {
    'texts': texts,
    'emb_model': emb_model,
    'emb_tokenizer': emb_tokenizer,
    'task_desc_format_fn': task_desc_format_fn,
    'pooling_fn': pooling_fn,
    'device': device,
    'batch_size': None
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 2, dtype family float, shape (2, 4) — shape (2, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
embed_tokens Ran
ran, honoured the contract we drafted
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("464adbe41baffcdb")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 2, dtype family float, shape (2, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
from math import sqrt

torch.manual_seed(0)

# Build a minimal embedding model that supports **kwargs and output_hidden_states
class MinimalEmbeddingModel(nn.Module):
    def __init__(self, hidden_size=4):
        super().__init__()
        self.hidden_size = hidden_size
        self.embedding = nn.Embedding(100, hidden_size)
        self.linear = nn.Linear(hidden_size, hidden_size)
    
    def forward(self, input_ids, attention_mask, output_hidden_states=False):
        # input_ids: [batch_size, seq_len]
        embedded = self.embedding(input_ids)  # [batch_size, seq_len, hidden_size]
        output = self.linear(embedded)  # [batch_size, seq_len, hidden_size]
        
        # Return an object with last_hidden_state attribute
        class ModelOutput:
            def __init__(self, last_hidden_state):
                self.last_hidden_state = last_hidden_state
        
        return ModelOutput(last_hidden_state=output)

# Build a minimal pooling function that takes outputs and attention_mask
def pooling_fn(outputs, attention_mask):
    # outputs has last_hidden_state: [batch_size, seq_len, hidden_size]
    # attention_mask: [batch_size, seq_len]
    last_hidden = outputs.last_hidden_state  # [batch_size, seq_len, hidden_size]
    
    # Mean pooling: average over sequence dimension
    # Expand mask to match hidden dimension
    mask_expanded = attention_mask.unsqueeze(-1).expand(last_hidden.size()).float()
    sum_hidden = (last_hidden * mask_expanded).sum(dim=1)  # [batch_size, hidden_size]
    sum_mask = mask_expanded.sum(dim=1)  # [batch_size, hidden_size]
    
    # Avoid division by zero
    mean_pooled = sum_hidden / (sum_mask + 1e-9)  # [batch_size, hidden_size]
    return mean_pooled

# Create tokenized texts with batch size 2, sequence length 3
batch_size = 2
seq_len = 3
hidden_size = 4

tokenized_texts = {
    'input_ids': torch.randint(0, 100, (batch_size, seq_len)),
    'attention_mask': torch.ones((batch_size, seq_len), dtype=torch.long)
}

emb_model = MinimalEmbeddingModel(hidden_size=hidden_size)
device = 'cpu'
batch_size_param = 2

ARGS = {
    'tokenized_texts': tokenized_texts,
    'emb_model': emb_model,
    'pooling_fn': pooling_fn,
    'device': device,
    'batch_size': batch_size_param
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 2, dtype family float, shape (2, 4) — shape (2, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_emb_model_and_fns Ran
ran; our contract draft was wrong, not the code
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("a449079623eb8c90")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
from unittest.mock import Mock, MagicMock

torch.manual_seed(0)

# Create a minimal AutoModel mock
class MockAutoModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(4, 4)
    
    def forward(self, input_ids, attention_mask):
        batch_size = input_ids.shape[0]
        last_hidden_state = torch.randn(batch_size, 8, 4)
        return {
            'last_hidden_state': last_hidden_state,
            'hidden_states': [torch.randn(batch_size, 8, 4) for _ in range(12)]
        }
    
    def eval(self):
        return super().eval()

# Create a minimal tokenizer mock
class MockTokenizer:
    def __init__(self):
        self.pad_token_id = 0
        self.eos_token_id = 1
        self.padding_side = "right"

# Mock AutoModel.from_pretrained
def mock_from_pretrained(model_name, device_map=None, torch_dtype=None, trust_remote_code=False):
    model = MockAutoModel()
    return model

# Mock AutoTokenizer.from_pretrained
def mock_tokenizer_from_pretrained(model_name):
    return MockTokenizer()

# Patch the imports at module level
import transformers
original_automodel = transformers.AutoModel
original_autotokenizer = transformers.AutoTokenizer

transformers.AutoModel.from_pretrained = staticmethod(mock_from_pretrained)
transformers.AutoTokenizer.from_pretrained = staticmethod(mock_tokenizer_from_pretrained)

ARGS = {
    'emb_model_name': 'mock-gte-model',
    'device': 'cpu'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch, transformers — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_in_out_features Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("8cbf94593c76c7d4")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
from peft import PeftConfig, PeftModel
from peft.tuners.tuners_utils import BaseTunerLayer
from unittest.mock import MagicMock, patch

torch.manual_seed(0)

# Create a mock PeftConfig
peft_config = MagicMock(spec=PeftConfig)
peft_config.target_modules = ['q_proj', 'v_proj']

# Create a mock BaseTunerLayer with base_layer as nn.Linear
class MockTunerLayer(nn.Module, BaseTunerLayer):
    def __init__(self, name, in_feat, out_feat):
        super().__init__()
        self.name = name
        self.in_features = in_feat
        self.out_features = out_feat
        self.base_layer = nn.Linear(in_feat, out_feat)
    
    def forward(self, x):
        return self.base_layer(x)

# Create a mock model with peft_config and named_modules
class MockPeftModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.peft_config = {'default': peft_config}
        
        # Add tuner layers with distinct sizes
        self.layer1_q_proj = MockTunerLayer('q_proj', 4, 5)
        self.layer1_v_proj = MockTunerLayer('v_proj', 4, 6)
        self.layer2_q_proj = MockTunerLayer('q_proj', 4, 5)
        
        # Add a non-tuner layer that should be skipped
        self.layer1_linear = nn.Linear(4, 5)
    
    def named_modules(self):
        yield 'layer1.q_proj', self.layer1_q_proj
        yield 'layer1.v_proj', self.layer1_v_proj
        yield 'layer2.q_proj', self.layer2_q_proj
        yield 'layer1.linear', self.layer1_linear

model = MockPeftModel()

# Patch check_target_module_exists to return True for tuner layers
def mock_check_target_module_exists(config, module_name):
    return 'q_proj' in module_name or 'v_proj' in module_name

with patch('peft.tuners.tuners_utils.check_target_module_exists', side_effect=mock_check_target_module_exists):
    ARGS = {
        'model': model,
        'peft_config': None,
    }
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
peft, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_layers Ran
ran; our contract draft was wrong, not the code
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("eb1fa1a3b994be5d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned Sequential. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# The function checks if model has a 'model' attribute, and if so, recurses.
# Otherwise it returns model.layers
# So we need to create a model that has a .layers attribute

class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.layers = nn.Sequential(
            nn.Linear(4, 3),
            nn.ReLU(),
            nn.Linear(3, 2)
        )
    
    def forward(self, x):
        return self.layers(x)

# Create a model instance
model = SimpleModel()

ARGS = {
    'model': model
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Sequential — a Sequential output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a Sequential output the recorded ndim 1 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_pooling_fn Ran
ran; our contract draft was wrong, not the code
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("f6bfe8f5c075f8d0")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned function. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from typing import Literal

torch.manual_seed(0)

# Test with 'last_token' pooling type
ARGS = {
    'pooling_type': 'last_token'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type function — a function output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a function output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
last_token_pool Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("b6c4cb90c7bfeae1")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned Tensor, ndim 2, dtype family float, shape (2, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Create a batch of 2 sequences with hidden dimension 4
batch_size = 2
seq_length = 3
hidden_dim = 4

# Create last_hidden_state tensor
last_hidden_state = torch.randn(batch_size, seq_length, hidden_dim)

# Create outputs dict with last_hidden_state
outputs = {
    'last_hidden_state': last_hidden_state
}

# Create attention_mask - shape (batch_size, seq_length)
# Using right-padding scenario (not all 1s in last column)
attention_mask = torch.tensor([
    [1, 1, 1],
    [1, 1, 0]
], dtype=torch.long)

ARGS = {
    'outputs': outputs,
    'attention_mask': attention_mask
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 2, dtype family float, shape (2, 4) — shape (2, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
lora_state_dict_to_tensor_dict Ran
ran; our contract draft was wrong, not the code
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("4eb352ce41bff2cc")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Create a minimal LoRA state dict with the expected structure
# The function looks for keys containing 'layers.{layer_idx}' and 'lora_A' or 'lora_B'
lora_sd = {
    'model.layers.0.self_attn.q_proj.lora_A.default': torch.randn(4, 3),
    'model.layers.0.self_attn.q_proj.lora_B.default': torch.randn(3, 4),
    'model.layers.1.self_attn.q_proj.lora_A.default': torch.randn(4, 3),
    'model.layers.1.self_attn.q_proj.lora_B.default': torch.randn(3, 4),
    'model.layers.0.mlp.up_proj.lora_A.default': torch.randn(4, 3),
    'model.layers.0.mlp.up_proj.lora_B.default': torch.randn(3, 4),
    'model.layers.1.mlp.up_proj.lora_A.default': torch.randn(4, 3),
    'model.layers.1.mlp.up_proj.lora_B.default': torch.randn(3, 4),
}

# Target modules to look for in the state dict keys
target_modules = ['q_proj', 'up_proj']

# Layer indices to extract
layer_indices = [0, 1]

# Device (CPU only)
device = 'cpu'

ARGS = {
    'lora_sd': lora_sd,
    'target_modules': target_modules,
    'layer_indices': layer_indices,
    'device': device,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
lora_tensor_dict_to_param_dict Ran
ran; our contract draft was wrong, not the code
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("1fc5218833c6c6ea")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned ParameterDict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Construct lora_tensor_dict with nested structure matching what the function expects
# The function expects a dict with keys 'A' and 'B', each containing dicts of tensors
lora_tensor_dict = {
    'A': {
        'layer1': torch.randn(4, 3),
        'layer2': torch.randn(5, 2),
    },
    'B': {
        'layer1': torch.randn(3, 4),
        'layer2': torch.randn(2, 5),
    }
}

requires_grad = True

ARGS = {
    'lora_tensor_dict': lora_tensor_dict,
    'requires_grad': requires_grad,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ParameterDict — a ParameterDict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a ParameterDict output the recorded ndim 1 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
zero_lora_param_dict Ran
ran; our contract draft was wrong, not the code
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("4731a23d7470dc77")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ParameterDict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from torch import nn

torch.manual_seed(0)

# Define the arguments for zero_lora_param_dict
target_modules = ['q_proj', 'v_proj']
n_layers = 2
rank = 3
in_features = {'q_proj': 4, 'v_proj': 5}
out_features = {'q_proj': 6, 'v_proj': 7}

ARGS = {
    'target_modules': target_modules,
    'n_layers': n_layers,
    'rank': rank,
    'in_features': in_features,
    'out_features': out_features
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ParameterDict — a ParameterDict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a ParameterDict output the recorded ndim 1 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
HyperModulator Not yet run
unverified (harvested, no recorded run)
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("c9a3604b537bfc0e")
How it ran

no invocation recorded.

TextToLoRA Not yet run
unverified (harvested, no recorded run)
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("81a8f2d7a29f3811")
How it ran

no invocation recorded.

_embed_tokens_single_batch Not yet run
unverified (harvested, no recorded run)
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("9a22c2784acf8fe5")
How it ran

no invocation recorded.

apply_lora_to_layers Not yet run
unverified (harvested, no recorded run)
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("937d38a0e976d855")
How it ran

no invocation recorded.

create_hypermod Not yet run
unverified (harvested, no recorded run)
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("6e23d59c5480daea")
How it ran

no invocation recorded.

get_init_peft_weights Not yet run
unverified (harvested, no recorded run)
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("b0fb60a036b8e6ee")
How it ran

no invocation recorded.

get_lora_module_names Not yet run
unverified (harvested, no recorded run)
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("7572df21036527e9")
How it ran

no invocation recorded.

get_model Not yet run
unverified (harvested, no recorded run)
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("122a565cbe1f2d83")
How it ran

no invocation recorded.

get_model_and_tokenizer Not yet run
unverified (harvested, no recorded run)
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("d38e50d952d93469")
How it ran

no invocation recorded.

get_peft_modules Not yet run
unverified (harvested, no recorded run)
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("4b6e8a0361945762")
How it ran

no invocation recorded.

get_tokenizer Not yet run
unverified (harvested, no recorded run)
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("f3fb84a5a6d3b065")
How it ran

no invocation recorded.

load_hypermod Not yet run
unverified (harvested, no recorded run)
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("177a916196a3d9b2")
How it ran

no invocation recorded.

load_hypermod_checkpoint Not yet run
unverified (harvested, no recorded run)
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("4dc184dbfb1660e4")
How it ran

no invocation recorded.

lora_forward Not yet run
unverified (harvested, no recorded run)
SakanaAI/doc-to-lora/src/ctx_to_lora/modeling/text_to_lora.py
code served (permissive licence) · get_code("f8caf753f4a6cb6a")
How it ran

no invocation recorded.

Also present in other papers' code

Of 32 samples lifted from this paper's linked repositories, 32 carry a function identity and 0 do not. Of the 32 distinct functions with an identity, 0 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 0 later) and 32 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Long input sequences are central to in-context learning, document understanding, and multi-step reasoning of Large Language Models (LLMs). However, the quadratic attention cost of Transformers makes inference memory-intensive and slow. While context distillation (CD) can transfer information into model parameters, per-prompt distillation is impractical due to training costs and latency. To address these limitations, we propose Doc-to-LoRA (D2L), a lightweight hypernetwork that meta-learns to perform approximate CD within a single forward pass. Given an unseen prompt, D2L generates a LoRA adapter for a target LLM, enabling subsequent queries to be answered without re-consuming the original context, reducing latency and KV-cache memory consumption during inference of the target LLM. On a long-context needle-in-a-haystack task, D2L successfully learns to map contexts into adapters that store the needle information, achieving nearperfect zero-shot accuracy at sequence lengths exceeding the target LLM's native context window by more than 4×. On real-world QA datasets with limited compute, D2L outperforms standard CD while significantly reducing peak memory consumption and update latency. We envision that D2L can facilitate rapid adaptation of LLMs, opening up the possibility of frequent knowledge updates and personalized chat behavior. Checkout our project at github.com/SakanaAI/doc-to-lora. *S. Uesaka contributed while he was an intern at Sakana AI.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2602.15902")
get_code_for_paper("2602.15902")
have("2602.15902")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2602.15902.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2602.15902.svg)](https://syntology.ai/paper/2602.15902)

Connect an agent — have() is free.