SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2402.13991 · ACL · 2024

Analysing The Impact of Sequence Composition on Language Model Pre-Training

Pasquale Minervini, Wei Liu, Yuxiang Wu, Piotr Miłoś, Szymon Tworkowski, Konrad Staniszewski, Yu Zhao, Yuanbin Qu

arXiv · PDF · Open in the Atlas

Code that ran

21 of 21 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 5 honoured the contract we drafted, 1 violated it and 0 have no contract verdict; 15 failures of our instrument; 0 no recorded run); 1 of them was also run on the host, not in the jail, by the peer-agreement lane that set its level, measured 2026-11-22; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2402.13991.json · badge: /api/ran/2402.13991.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
yuzhaouoe/pretraining-data-packing canonical (classifier label)
classifier basis: census
21 ran · 0 no recorded run
FunctionStatusWhere it lives
repeat_kv Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
yuzhaouoe/pretraining-data-packing/llama_modeling.py
code served (permissive licence) · get_code("30d7eec482ebf6b1")
How it ran

Ran in Syntology's jail on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned Tensor, ndim 4, dtype family float, shape (2, 6, 5, 4) as a later run recorded it. The image it ran in was not recorded. The comparison behind this sample's level is a different run: the peer-agreement lane ran it again on the host, not in the jail (report dated 2026-09-04T23:09:20Z). If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Create a tensor with shape (batch, num_key_value_heads, seqlen, head_dim)
# Using distinct sizes: batch=2, num_key_value_heads=3, seqlen=5, head_dim=4
hidden_states = torch.randn(2, 3, 5, 4)

# n_rep is the repetition factor
n_rep = 2

ARGS = {
    "hidden_states": hidden_states,
    "n_rep": n_rep,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 4, dtype family float, shape (2, 6, 5, 4) — shape (2, 6, 5, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
apply_rotary_pos_emb Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
yuzhaouoe/pretraining-data-packing/llama_modeling.py
code served (permissive licence) · get_code("f725bc2d76076485")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned tuple, ndim 5, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Create small tensors for q and k
# Shape: (batch_size=2, seq_len=3, hidden_dim=4)
q = torch.randn(2, 3, 4)
k = torch.randn(2, 3, 4)

# Create cos and sin tensors
# The function squeezes dimensions 1 and 0, then indexes by position_ids
# So we need shape that allows this: (1, 1, seq_len, hidden_dim)
cos = torch.randn(1, 1, 5, 4)
sin = torch.randn(1, 1, 5, 4)

# Create position_ids
# Shape: (batch_size=2, seq_len=3)
position_ids = torch.tensor([[0, 1, 2], [1, 2, 3]])

ARGS = {
    "q": q,
    "k": k,
    "cos": cos,
    "sin": sin,
    "position_ids": position_ids,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 5, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
data_collator Ran
ran; our contract draft was wrong, not the code
yuzhaouoe/pretraining-data-packing/packing_dataset.py
code served (permissive licence) · get_code("874a9519ebeb3c53")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Create example data structures that match what the function expects
examples = [
    {
        'input_ids': [1, 2, 3, 4],
        'labels': [5, 6, 7, 8],
        'fragment_lens': [2, 3],
        'fragment_nums': 2
    },
    {
        'input_ids': [9, 10, 11, 12],
        'labels': [13, 14, 15, 16],
        'fragment_lens': [1, 4],
        'fragment_nums': 2
    }
]

ARGS = {
    'examples': examples,
    'max_num_fragments_in_chunk': 65
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
estimate_total_batch_num Ran
ran, honoured the contract we drafted
yuzhaouoe/pretraining-data-packing/save_offline_dataset.py
code served (permissive licence) · get_code("eb90b0d7f1e2c692")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import tempfile
import shutil

# Create temporary directory structure for testing
temp_base = tempfile.mkdtemp()

# Create split directories with chunk subdirectories and batch files
split_dir_1 = os.path.join(temp_base, "split_1")
split_dir_2 = os.path.join(temp_base, "split_2")

os.makedirs(split_dir_1)
os.makedirs(split_dir_2)

# Create chunk directories in split_1
chunk_1_dir = os.path.join(split_dir_1, "chunk_0")
chunk_2_dir = os.path.join(split_dir_1, "chunk_1")
os.makedirs(chunk_1_dir)
os.makedirs(chunk_2_dir)

# Create batch files in chunks
open(os.path.join(chunk_1_dir, "batch_0.pt"), 'w').close()
open(os.path.join(chunk_1_dir, "batch_1.pt"), 'w').close()
open(os.path.join(chunk_1_dir, "batch_2.pt"), 'w').close()
open(os.path.join(chunk_2_dir, "batch_0.pt"), 'w').close()

# Create chunk directories in split_2
chunk_3_dir = os.path.join(split_dir_2, "chunk_0")
os.makedirs(chunk_3_dir)

# Create batch files in chunk
open(os.path.join(chunk_3_dir, "batch_0.pt"), 'w').close()
open(os.path.join(chunk_3_dir, "batch_1.pt"), 'w').close()

ARGS = {
    "cur_all_split_dirs": [split_dir_1, split_dir_2]
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
eval_generation_em Ran
ran, honoured the contract we drafted
yuzhaouoe/pretraining-data-packing/evaluation/eval_utils.py
code served (permissive licence) · get_code("d9243c078ceacce8")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re
import string
import unicodedata

def normalize_answer(s):
    """Normalize answer."""
    s = unicodedata.normalize('NFD', s)

    def remove_articles(text):
        return re.sub('\\b(a|an|the)\\b', ' ', text)

    def white_space_fix(text):
        return ' '.join(text.split())

    def remove_punc(text):
        exclude = set(string.punctuation)
        return ''.join((ch for ch in text if ch not in exclude))

    def lower(text):
        return text.lower()
    return white_space_fix(remove_articles(remove_punc(lower(s))))

def exact_match_score(prediction, ground_truth):
    return normalize_answer(prediction) == normalize_answer(ground_truth)

def metric_max_over_ground_truths(metric_fn, prediction, ground_truths):
    scores_for_ground_truths = []
    for ground_truth in ground_truths:
        score = metric_fn(prediction, ground_truth)
        scores_for_ground_truths.append(score)
    return max(scores_for_ground_truths)

def eval_generation_em(refs, preds):
    scores = []
    for ref, pred in zip(refs, preds):
        ref_answer = ref['answer']
        em = metric_max_over_ground_truths(exact_match_score, pred, ref_answer)
        scores.append(em)
    avg_score = sum(scores) / len(scores)
    return avg_score

# Setup code
refs = [
    {'answer': ['The quick brown fox', 'a quick fox']},
    {'answer': ['Paris is the capital', 'Paris, capital city']}
]

preds = [
    'the quick brown fox',
    'paris is the capital'
]

ARGS = {
    'refs': refs,
    'preds': preds
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type float, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
exact_match_score Ran
ran, violated the contract we drafted
yuzhaouoe/pretraining-data-packing/evaluation/eval_utils.py
code served (permissive licence) · get_code("efa3643614e9c5f5")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re
import string
import unicodedata

def normalize_answer(s):
    """Normalize answer."""
    s = unicodedata.normalize('NFD', s)

    def remove_articles(text):
        return re.sub('\\b(a|an|the)\\b', ' ', text)

    def white_space_fix(text):
        return ' '.join(text.split())

    def remove_punc(text):
        exclude = set(string.punctuation)
        return ''.join((ch for ch in text if ch not in exclude))

    def lower(text):
        return text.lower()
    return white_space_fix(remove_articles(remove_punc(lower(s))))

def exact_match_score(prediction, ground_truth):
    return normalize_answer(prediction) == normalize_answer(ground_truth)

ARGS = {
    'prediction': 'The quick brown fox jumps over the lazy dog.',
    'ground_truth': 'the quick brown fox jumps over the lazy dog'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_batch_dir_and_path Ran
ran; our contract draft was wrong, not the code
yuzhaouoe/pretraining-data-packing/analysis/burstiness.py
code served (permissive licence) · get_code("bb142953e9e84a14")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import tempfile

# Create a temporary directory to serve as data_dir
data_dir = tempfile.mkdtemp()

# Define arguments for the function
ARGS = {
    'data_dir': data_dir,
    'batch_idx': 2048
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_fragment_lens Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
yuzhaouoe/pretraining-data-packing/packing_dataset.py
code served (permissive licence) · get_code("1bc433883369de22")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# The function takes a chunk parameter
# Looking at the function body:
# - chunk needs to support len()
# - chunk needs to be iterable
# - chunk contains tokens that are compared to the integer 2
# - tokens are indexed

# Create a simple list/array of tokens where some tokens equal 2
chunk = [0, 1, 2, 0, 1, 2, 0]

ARGS = {
    "chunk": chunk
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_offset_path Ran
ran; our contract draft was wrong, not the code
yuzhaouoe/pretraining-data-packing/utils.py
code served (permissive licence) · get_code("da105da73281cce5")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import tempfile

# Create a temporary directory for testing
temp_dir = tempfile.mkdtemp()

# Create a sample jsonl file path
jsonl_path = os.path.join(temp_dir, "data.jsonl")

# Define arguments for get_offset_path
ARGS = {
    "jsonl_path": jsonl_path
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_worker_id_and_iter_indices Ran
ran; our contract draft was wrong, not the code
yuzhaouoe/pretraining-data-packing/packing_dataset.py
code served (permissive licence) · get_code("2a6a0d112f2ef000")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np
import logging

# Set up logging
logging.basicConfig(level=logging.DEBUG)

# Create a stand-in class for worker_info
class WorkerInfo:
    def __init__(self, num_workers, worker_id):
        self.num_workers = num_workers
        self.id = worker_id

# Test case 1: with multiple workers
worker_info_multi = WorkerInfo(num_workers=4, worker_id=1)
indices_multi = np.array([10, 20, 30, 40, 50, 60, 70, 80, 90, 100])

# Test case 2: with single worker (worker_info is None)
indices_single = np.array([1, 2, 3, 4, 5])
worker_info_single = None

ARGS = {
    "indices": indices_multi,
    "worker_info": worker_info_multi,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_json Ran
ran; our contract draft was wrong, not the code
yuzhaouoe/pretraining-data-packing/evaluation/fewshot.py
pointer only (licence: MIT); the stored copy get_code serves, from gstoica27/Re-TACRED, is pointer only (licence: NONE) · get_code("813da56adc0dd787")
How it ran

The body of this sample is withheld: this paper's link allows the body inline (MIT), but get_code serves the stored copy, first harvested from gstoica27/Re-TACRED, and GitHub detected no licence in that repository when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

load_jsonl Ran
ran; our contract draft was wrong, not the code
yuzhaouoe/pretraining-data-packing/evaluation/fewshot.py
code served (permissive licence) · get_code("ad0a5228eb3d81f4")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (3,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import numpy as np
import tempfile
import os

# Create a temporary JSONL file for testing
temp_dir = tempfile.mkdtemp()
temp_path = os.path.join(temp_dir, "test_data.jsonl")

# Write some test data to the JSONL file
test_data = [
    {"id": 1, "text": "hello world"},
    {"id": 2, "text": "foo bar"},
    {"id": 3, "text": "baz qux"},
    {"id": 4, "text": "test data"},
    {"id": 5, "text": "another line"},
]

with open(temp_path, 'w', encoding='utf-8') as f:
    for item in test_data:
        f.write(json.dumps(item) + '\n')

# Define ARGS
ARGS = {
    'path': temp_path,
    'max_line': 3
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 1, dtype family float, shape (3,) — shape (3,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_offset Ran
ran, honoured the contract we drafted
yuzhaouoe/pretraining-data-packing/utils.py
code served (permissive licence) · get_code("11cde6e6bca04ca4")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family int; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import pickle
import tempfile
import shutil

# Create a temporary directory for our test pickle files
temp_dir = tempfile.mkdtemp()

# Create a sample offset data structure (typically a list or dict of byte offsets)
sample_offset_data = [0, 100, 250, 500, 1000]

# Create a sample JSONL file path
jsonl_filename = "test_data.jsonl"
jsonl_path = os.path.join(temp_dir, jsonl_filename)

# Create the actual JSONL file (empty is fine for this test)
with open(jsonl_path, 'w') as f:
    f.write('{"test": "data"}\n')

# Create the offset pickle file
offset_pkl_path = os.path.join(temp_dir, jsonl_filename + '.offset.pkl')
with open(offset_pkl_path, 'wb') as f:
    pickle.dump(sample_offset_data, f)

# Define ARGS dictionary with the jsonl_path parameter
ARGS = {
    'jsonl_path': jsonl_path
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family int — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_pickle Ran
ran; our contract draft was wrong, not the code
yuzhaouoe/pretraining-data-packing/utils.py
code served (permissive licence) · get_code("b678e0cd3681e4cd")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import pickle
import tempfile
import logging

logging.basicConfig(level=logging.INFO)

# Create a temporary directory and a valid pickle file
temp_dir = tempfile.mkdtemp()
test_data = {'key': 'value', 'number': 42, 'list': [1, 2, 3]}
test_pickle_path = os.path.join(temp_dir, 'test.pkl')

with open(test_pickle_path, 'wb') as f:
    pickle.dump(test_data, f)

ARGS = {
    'path': test_pickle_path
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
metric_max_over_ground_truths Ran
ran; our contract draft was wrong, not the code
yuzhaouoe/pretraining-data-packing/evaluation/eval_utils.py
code served (permissive licence) · get_code("6829ceb72569fa12")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned float, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

def simple_metric_fn(prediction, ground_truth):
    """A simple metric function that computes similarity between two tensors."""
    if isinstance(prediction, torch.Tensor) and isinstance(ground_truth, torch.Tensor):
        return float(torch.nn.functional.cosine_similarity(
            prediction.unsqueeze(0), 
            ground_truth.unsqueeze(0)
        ).item())
    return 0.0

prediction = torch.randn(4)
ground_truths = [torch.randn(4), torch.randn(4), torch.randn(4)]

ARGS = {
    "metric_fn": simple_metric_fn,
    "prediction": prediction,
    "ground_truths": ground_truths,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type float, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
normalise_pred Ran
ran; our contract draft was wrong, not the code
yuzhaouoe/pretraining-data-packing/evaluation/fewshot.py
code served (permissive licence) · get_code("9afac6122e0fd74e")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# The function normalise_pred takes a single parameter 'pred'
# and calls .strip(), .split('\n'), and indexes [0], then .strip() again
# This means pred must be a string

pred = "hello world\nsecond line\nthird line"

ARGS = {
    "pred": pred
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
process_batch_attn Ran
ran; our contract draft was wrong, not the code
yuzhaouoe/pretraining-data-packing/analysis/distraction.py
code served (permissive licence) · get_code("137756758ba95a6a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 4, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# The function expects `attn` to be a list/sequence of tensors
# where attn[layer_idx][item_idx] exists and has a mean(dim=0) method
# It iterates over len(attn[0]) items and len(attn) layers
# Each attention matrix should be a tensor that supports .mean(dim=0)

# Create a batch of attention matrices across layers
# Structure: list of layers, each containing a batch of attention tensors
# attn[layer_idx][item_idx] should be a tensor with shape (heads, seq_len, seq_len)
# After mean(dim=0), it becomes (seq_len, seq_len)

num_layers = 3
batch_size = 2
num_heads = 4
seq_len = 5

# Build attn as a list of layers, where each layer contains batch items
attn = []
for layer_idx in range(num_layers):
    layer_batch = []
    for item_idx in range(batch_size):
        # Attention tensor: (num_heads, seq_len, seq_len)
        attention_tensor = torch.randn(num_heads, seq_len, seq_len)
        layer_batch.append(attention_tensor)
    attn.append(layer_batch)

ARGS = {
    "attn": attn
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 4, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
rotate_half Ran
ran; our contract draft was wrong, not the code
yuzhaouoe/pretraining-data-packing/llama_modeling.py
code served (permissive licence) · get_code("b99eea6376d1e212")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned Tensor, ndim 2, dtype family float, shape (2, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# The function rotate_half takes a single parameter 'x' which is a tensor
# It accesses x.shape[-1] to get the last dimension
# It uses indexing with [..., :] slicing notation
# It calls torch.cat() to concatenate tensors

# Create a small tensor with shape (2, 4) - batch size 2, hidden dim 4
x = torch.randn(2, 4)

ARGS = {
    "x": x
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 2, dtype family float, shape (2, 4) — shape (2, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
save_data_collator Ran
ran; our contract draft was wrong, not the code
yuzhaouoe/pretraining-data-packing/save_offline_dataset.py
code served (permissive licence) · get_code("efbd9b87adf49321")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# The function expects a list of dicts, where each dict has at least 'input_ids' key
# It extracts only the 'input_ids' from each example
examples = [
    {'input_ids': torch.tensor([1, 2, 3, 4], dtype=torch.long)},
    {'input_ids': torch.tensor([5, 6, 7, 8], dtype=torch.long)}
]

ARGS = {
    'examples': examples
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
tokenize_one_item Ran
ran, honoured the contract we drafted
yuzhaouoe/pretraining-data-packing/analysis/distraction.py
code served (permissive licence) · get_code("b494dc24033bda53")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family int; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

class SimpleTokenizer:
    """Minimal tokenizer that implements the methods used by tokenize_one_item"""
    def encode(self, text, add_special_tokens=False, truncation=False):
        # Simple implementation: split text into words and return token IDs
        words = text.split()
        # Convert each word to a simple integer ID
        token_ids = [hash(word) % 1000 for word in words]
        return token_ids

torch.manual_seed(0)

# Create minimal instances
tokenizer = SimpleTokenizer()

item = {
    'text': 'hello world this is a test'
}

ARGS = {
    'tokenizer': tokenizer,
    'item': item
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family int — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
zipfian_fit Ran
ran, honoured the contract we drafted
yuzhaouoe/pretraining-data-packing/analysis/burstiness.py
code served (permissive licence) · get_code("92ed24e518eb48dc")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float64, ndim 0, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np
from scipy.stats import zipf
from scipy.optimize import minimize_scalar

# Create sample data that follows a Zipfian distribution
# The function expects an array of positive integers
np.random.seed(0)
data = np.random.zipf(a=1.5, size=100)
# Ensure all values are at least 1 (zipf.pmf requires positive integers)
data = np.maximum(data, 1)

ARGS = {
    "data": data
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, scipy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type float64, ndim 0, dtype family float — no output shape is stated: observe_signature.py recorded float ndim 0, which disagrees with the fixture run
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block

Also present in other papers' code

Of 21 samples lifted from this paper's linked repositories, 21 carry a function identity and 0 do not. Of the 21 distinct functions with an identity, 6 are also present in other papers whose code we hold (6 with at least one paper dated earlier, 5 the same month, 5 later) and 15 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
rotate_half 360 other papers
68 dated earlier, 25 the same month, 258 later, 6 same year, order not determinable, 3 undated · 30 only as an assembly brick
1901.02860 (2019-01)
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
2203.08913 (2022-03)
Memorizing Transformers
2302.13971 (2023-02)
LLaMA: Open and Efficient Foundation Language Models
and 357 more not named here
repeat_kv 243 other papers
33 dated earlier, 19 the same month, 187 later, 2 same year, order not determinable, 2 undated · 22 only as an assembly brick
1901.02860 (2019-01)
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
2203.08913 (2022-03)
Memorizing Transformers
2305.17888 (2023-05)
LLM-QAT: Data-Free Quantization Aware Training for Large Language Models
and 240 more not named here
apply_rotary_pos_emb 83 other papers
19 dated earlier, 11 the same month, 51 later, 2 same year, order not determinable · 6 only as an assembly brick
2306.15595 (2023-06)
Extending Context Window of Large Language Models via Positional Interpolation
2307.05695 (2023-07)
ReLoRA: High-Rank Training Through Low-Rank Updates
2307.06945 (2023-07)
In-context Autoencoder for Context Compression in a Large Language Model
and 80 more not named here
exact_match_score 56 other papers
43 dated earlier, 3 the same month, 10 later
1508.06615 (2015-08)
Character-Aware Neural Language Models
1606.05250 (2016-06)
SQuAD: 100,000+ Questions for Machine Comprehension of Text
1608.07905 (2016-08)
Machine Comprehension Using Match-LSTM and Answer Pointer
and 53 more not named here
metric_max_over_ground_truths 6 other papers
3 dated earlier, 1 the same month, 2 later
2104.08773 (2021-04)
Cross-Task Generalization via Natural Language Crowdsourcing Instructions
2210.10861 (2022-10)
QA Domain Adaptation using Hidden Space Augmentation and Self-Supervised Contrastive Adaptation
2310.11511 (2023-10)
Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
and 3 more not named here
load_json 1 other paper
1 dated earlier
2104.08398 (2021-04)
Re-TACRED: Addressing Shortcomings of the TACRED Dataset

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Most language model pre-training frameworks concatenate multiple documents into fixedlength sequences and use causal masking to compute the likelihood of each token given its context; this strategy is widely adopted due to its simplicity and efficiency. However, to this day, the influence of the pre-training sequence composition strategy on the generalisation properties of the model remains underexplored. In this work, we find that applying causal masking can lead to the inclusion of distracting information from previous documents during pre-training, which negatively impacts the performance of the models on language modelling and downstream tasks. In intra-document causal masking, the likelihood of each token is only conditioned on the previous tokens in the same document, eliminating potential distracting information from previous documents and significantly improving performance. Furthermore, we find that concatenating related documents can reduce some potential distractions during pre-training, and our proposed efficient retrieval-based sequence construction method, BM25Chunk, can improve incontext learning (+11.6%), knowledge memorisation (+9.8%), and context utilisation (+7.2%) abilities of language models without sacrificing efficiency.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2402.13991")
get_code_for_paper("2402.13991")
have("2402.13991")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2402.13991.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2402.13991.svg)](https://syntology.ai/paper/2402.13991)

Connect an agent — have() is free.