SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2402.03660 · 2024

On the Emergence of Cross-Task Linearity in the Pretraining-Finetuning Paradigm

arXiv · PDF · Open in the Atlas

Code that ran

6 of 7 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 0 honoured the contract we drafted, 0 violated it and 1 has no contract verdict; 5 failures of our instrument; 1 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2402.03660.json · badge: /api/ran/2402.03660.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
zzp1012/cross-task-linearity canonical (classifier label)
classifier basis: census
6 ran · 1 no recorded run
FunctionStatusWhere it lives
get_evaluate_fn Ran
ran; our contract draft was wrong, not the code
zzp1012/cross-task-linearity/task_arithemetic_t5/data_utils.py
code served (permissive licence) · get_code("4adc94f247d34feb")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned function. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# Define the evaluation functions that would be imported
def eval_imdb():
    return "imdb_eval"

def eval_race():
    return "race_eval"

def eval_qasc():
    return "qasc_eval"

def eval_multi_news():
    return "multi_news_eval"

def eval_squad():
    return "squad_eval"

def eval_allenai_common_gen():
    return "allenai/common_gen_eval"

# Make these functions available in the eval namespace
import sys
current_module = sys.modules[__name__]
setattr(current_module, 'eval_imdb', eval_imdb)
setattr(current_module, 'eval_race', eval_race)
setattr(current_module, 'eval_qasc', eval_qasc)
setattr(current_module, 'eval_multi_news', eval_multi_news)
setattr(current_module, 'eval_squad', eval_squad)
setattr(current_module, 'eval_allenai/common_gen', eval_allenai_common_gen)

# Valid dataset names from the function's dataset_names list
ARGS = {
    'dataset_name': 'imdb'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type function — a function output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a function output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_logger Ran
ran; our contract draft was wrong, not the code
zzp1012/cross-task-linearity/task_arithemetic_t5/utils.py
code served (permissive licence) · get_code("edd9c782220bb4e8")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Logger. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import logging

ARGS = {
    "name": "test_logger",
    "verbose": True
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Logger — a Logger output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a Logger output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_t5_featuremaps Ran
ran; our contract draft was wrong, not the code
zzp1012/cross-task-linearity/task_arithemetic_t5/featuremap.py
code served (permissive licence) · get_code("599b1212c12f5eb3")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned defaultdict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from collections import defaultdict

torch.manual_seed(0)

# Construct encoder_hidden_states: list of tensors, one per layer
# Layer 0 is embedding, layers 1+ are T5Blocks
# Each tensor shape: (batch_size, seq_len, hidden_dim)
encoder_hidden_states = [
    torch.randn(2, 5, 4),  # layer 0: embedding
    torch.randn(2, 5, 4),  # layer 1: T5Block.0
    torch.randn(2, 5, 4),  # layer 2: T5Block.1
]

# Construct decoder_hidden_states: list containing one element (a list of tensors per layer)
# decoder_hidden_states[0][layer_id] gives the hidden state for layer_id
# Layer 0 is embedding, layers 1+ are T5Blocks
decoder_hidden_states = [
    [
        torch.randn(2, 5, 4),  # layer 0: embedding
        torch.randn(2, 5, 4),  # layer 1: T5Block.0
        torch.randn(2, 5, 4),  # layer 2: T5Block.1
    ]
]

# Construct featuremaps: dict mapping layer names to lists
featuremaps = defaultdict(list)

ARGS = {
    "featuremaps": featuremaps,
    "encoder_hidden_states": encoder_hidden_states,
    "decoder_hidden_states": decoder_hidden_states,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type defaultdict — a defaultdict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a defaultdict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
interpolate_weights Ran
ran; our contract draft was wrong, not the code
zzp1012/cross-task-linearity/task_arithemetic_t5/utils.py
code served (permissive licence) · get_code("471e85c48d4ec0ef")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned OrderedDict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from collections import OrderedDict

torch.manual_seed(0)

# Create two weight dictionaries with matching keys
# These simulate model state_dicts
A = OrderedDict()
A['layer1.weight'] = torch.randn(4, 3)
A['layer1.bias'] = torch.randn(4)
A['layer2.weight'] = torch.randn(5, 4)
A['layer2.bias'] = torch.randn(5)

B = OrderedDict()
B['layer1.weight'] = torch.randn(4, 3)
B['layer1.bias'] = torch.randn(4)
B['layer2.weight'] = torch.randn(5, 4)
B['layer2.bias'] = torch.randn(5)

alpha = 0.6
beta = 0.4

ARGS = {
    'A': A,
    'B': B,
    'alpha': alpha,
    'beta': beta
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type OrderedDict — a OrderedDict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a OrderedDict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
process_hf_dataset_to_torch_dataset Ran
ran; our contract draft was wrong, not the code
zzp1012/cross-task-linearity/task_arithemetic_t5/data_utils.py
code served (permissive licence) · get_code("e0fa6eddf2fbbbaf")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Dataset. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np
from datasets import Dataset

torch.manual_seed(0)

# Create a minimal mock HuggingFace dataset structure
def create_mock_hf_dataset(dataset_name):
    # Create sample data with different structures based on dataset_name
    if dataset_name == 'imdb':
        data = {
            'text': ['This is a great movie!', 'Bad film.'],
            'label': [1, 0]
        }
    elif dataset_name == 'race':
        data = {
            'passage': ['A passage about history', 'Another passage'],
            'question': ['What happened?', 'When?'],
            'answer': ['1952', '1960']
        }
    elif dataset_name == 'qasc':
        data = {
            'question': ['What is X?', 'Where is Y?'],
            'choices': [['A', 'B'], ['C', 'D']],
            'answerKey': ['A', 'C']
        }
    elif dataset_name == 'multi_news':
        data = {
            'document': ['Doc 1 text', 'Doc 2 text'],
            'summary': ['Summary 1', 'Summary 2']
        }
    elif dataset_name == 'squad':
        data = {
            'id': ['1', '2'],
            'question': ['Who?', 'What?'],
            'answers': [{'text': ['answer1'], 'answer_start': [0]}, {'text': ['answer2'], 'answer_start': [0]}]
        }
    elif dataset_name == 'common_gen':
        data = {
            'concepts': [['dog', 'cat'], ['tree', 'forest']],
            'target': ['A dog and cat play', 'Forest has many trees']
        }
    else:
        data = {'text': ['sample1', 'sample2']}
    
    # Create a dataset and wrap it in a dict-like structure
    hf_dataset = Dataset.from_dict(data)
    
    # Return a dict-like object with splits
    splits_dict = {
        'test': hf_dataset,
        'validation': hf_dataset,
        'train': hf_dataset
    }
    
    return splits_dict

# Build ARGS with valid arguments
ARGS = {
    'hf_ds': create_mock_hf_dataset('imdb'),
    'dataset_name': 'imdb'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
datasets, numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Dataset — a Dataset output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a Dataset output the recorded ndim 1 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
set_device Ran
ran on a synthesized input
zzp1012/cross-task-linearity/task_arithemetic_t5/utils.py
code served (permissive licence) · get_code("3e7cb9bdaf2ede2d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned device. The image it ran in was not recorded. The code mentions cuda, and the harness aliases cuda to the CPU for every sample; whether this run needed that is not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import logging

torch.manual_seed(0)

# The function set_device requires a device parameter (int)
# Since we're CPU-only and the function will error on exit(1) if CUDA is not available,
# we need to mock or bypass this. We'll provide a valid device number that the function
# can process, understanding it will fail at runtime due to no CUDA.
# However, to make valid setup code that could theoretically execute, we provide device=0

ARGS = {
    "device": 0
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type device — a device output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a device output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_hf_dataset Not yet run
unverified (harvested, no recorded run)
zzp1012/cross-task-linearity/task_arithemetic_t5/data_utils.py
code served (permissive licence) · get_code("598fb7842545bd8e")
How it ran

no invocation recorded.

Also present in other papers' code

Of 7 samples lifted from this paper's linked repositories, 7 carry a function identity and 0 do not. Of the 7 distinct functions with an identity, 3 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 3 later) and 4 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
get_logger 1 other paper
1 later
2505.18909 (2025-05)
On the Role of Label Noise in the Feature Learning Process
interpolate_weights 1 other paper
1 later
2505.18909 (2025-05)
On the Role of Label Noise in the Feature Learning Process
set_device 1 other paper
1 later
2505.18909 (2025-05)
On the Role of Label Noise in the Feature Learning Process

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

The pretraining-finetuning paradigm has become the prevailing trend in modern deep learning. In this work, we discover an intriguing linear phenomenon in models that are initialized from a common pretrained checkpoint and finetuned on different tasks, termed as Cross-Task Linearity (CTL). Specifically, we show that if we linearly interpolate the weights of two finetuned models, the features in the weight-interpolated model are often approximately equal to the linear interpolation of features in two finetuned models at each layer. We provide comprehensive empirical evidence supporting that CTL consistently occurs for finetuned models that start from the same pretrained checkpoint. We conjecture that in the pretraining-finetuning paradigm, neural networks approximately function as linear maps, mapping from the parameter space to the feature space. Based on this viewpoint, our study unveils novel insights into explaining model merging/editing, particularly by translating operations from the parameter space to the feature space. Furthermore, we delve deeper into the root cause for the emergence of CTL, highlighting the role of pretraining.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2402.03660")
get_code_for_paper("2402.03660")
have("2402.03660")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2402.03660.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2402.03660.svg)](https://syntology.ai/paper/2402.03660)

Connect an agent — have() is free.