SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2406.18522 · NeurIPS · 2024

ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generation

Li Yuan, Jiebo Luo, Yujun Shi, Jinfa Huang, Shenghai Yuan, Shaofeng Zhang, Yongqi Xu, Yaoyang Liu, Ruijie Zhu, Xinhua Cheng

arXiv · PDF · Open in the Atlas

Code that ran

24 of 27 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 2 honoured the contract we drafted, 3 violated it and 0 have no contract verdict; 19 failures of our instrument; 3 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2406.18522.json · badge: /api/ran/2406.18522.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
pku-yuangroup/magictime canonical (classifier label)
classifier basis: census
13 ran · 0 no recorded run
PKU-YuanGroup/ChronoMagic-Bench canonical (classifier label)
classifier basis: census
7 ran · 2 no recorded run
pku-yuangroup/chronomagic-bench canonical (classifier label)
classifier basis: classifier_2025
1 ran · 1 no recorded run
pku-yuangroup/open-sora-plan reimplementation (classifier label)
classifier basis: classifier_2025
3 ran · 0 no recorded run
FunctionStatusWhere it lives
calculate_average_score Ran
ran, violated the contract we drafted
pku-yuangroup/chronomagic-bench/MTScore/step0-get_MTScore.py
code served (permissive licence) · get_code("8204634e93f03f47")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# The function expects 'scores' to be an iterable of tuples/sequences
# where each score has at least 2 elements: score[0] (general_score) and score[1] (metamorphic_score)
# Both should be numeric values that can be summed and divided

scores = [
    (0.85, 0.92),
    (0.78, 0.88),
    (0.91, 0.95),
]

ARGS = {
    "scores": scores
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
calculate_average_scores Ran
ran; our contract draft was wrong, not the code
PKU-YuanGroup/ChronoMagic-Bench/UMT/step3_get_merge_umt_scores.py
code served (permissive licence) · get_code("5ce38342e526b34f")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import tempfile
import shutil

# Create a temporary directory structure with scores.txt files
temp_dir = tempfile.mkdtemp()

# Create directory structure: model_name/version/scores.txt
model1_v1_dir = os.path.join(temp_dir, "model_a", "v1")
model1_v2_dir = os.path.join(temp_dir, "model_a", "v2")
model2_v1_dir = os.path.join(temp_dir, "model_b", "v1")

os.makedirs(model1_v1_dir, exist_ok=True)
os.makedirs(model1_v2_dir, exist_ok=True)
os.makedirs(model2_v1_dir, exist_ok=True)

# Create scores.txt files with UMTScore values
with open(os.path.join(model1_v1_dir, "scores.txt"), "w") as f:
    f.write("Results: {'UMTScore': 85.5}")

with open(os.path.join(model1_v2_dir, "scores.txt"), "w") as f:
    f.write("Results: {'UMTScore': 90.3}")

with open(os.path.join(model2_v1_dir, "scores.txt"), "w") as f:
    f.write("Results: {'UMTScore': 78.2}")

ARGS = {
    "input_path": temp_dir
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
check_unused_params Ran
ran; our contract draft was wrong, not the code
pku-yuangroup/open-sora-plan/opensora/train/train_causalvae.py
code served (permissive licence) · get_code("82a0c7ca20f4f943")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear1 = nn.Linear(4, 3)
        self.linear2 = nn.Linear(3, 2)
    
    def forward(self, x):
        x = self.linear1(x)
        x = self.linear2(x)
        return x

model = SimpleModel()

# Create a simple forward pass and backward to populate gradients for some params
x = torch.randn(2, 4)
y = torch.randn(2, 2)
output = model(x)
loss = nn.MSELoss()(output, y)
loss.backward()

# Now linear1 parameters have gradients, but we'll manually set linear2 to have no gradients
# by not using it in the backward pass. We can do this by creating a fresh model
# where we only compute gradients for some parameters.

model2 = SimpleModel()
x = torch.randn(2, 4)
y = torch.randn(2, 2)
output = model2.linear1(x)
loss = nn.MSELoss()(output, torch.randn(2, 3))
loss.backward()
# Now linear2 parameters have no gradients

ARGS = {
    "model": model2
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
create_prompts Ran
ran; our contract draft was wrong, not the code
pku-yuangroup/magictime/data_preprocess/step3_1_GPT4V_video_caption_concise.py
code served (permissive licence) · get_code("be4269f1f4abf08e")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from torch import nn
from tqdm import tqdm

txt_prompt = '\nImagine you\'re an expert data annotator with a specialization in summarizing time-lapse videos. You will be supplied with "Video_Reasoning", "8_Key-Frames_Reasoning", and "8_Key-Frames_Captioning" from a video, your task is to craft a concise summary for the given time-lapse video.\n\nSince only textual information is given, you can employ logical deductions to bridge any informational gaps if necessary. For guidance on the expected output format and content length (no more than 70 words), refer to the provided examples:\n\n"Video_Summary": Time-lapse of a ciplukan fruit growing from a small bud to a mature, rounded form among leaves, gradually enlarging and smoothing out by the video\'s end.\n\n"Video_Summary": Time-lapse of red onion bulbs sprouting and growing over 10 days: starting dormant, developing shoots and roots by Day 2, significant growth by Day 6, and full development by Day 10.\n\n"Video_Summary": "{Video Summary}"\n\nAttention: Do not reply outside the example template! The process of reasoning and thinking should not be included in the {Video Summary}! Do not use words similar to by frame or at frame! Below are the Video, Video_Reasoning, Frame_Reasoning and Frame_Captioning.\n'

data = {
    'video_001': {
        'Video_Reasoning': 'A plant grows from seedling to mature flowering stage over 30 days with visible leaf expansion and stem elongation.',
        'Frame_Reasoning': [
            'Frame 1: Seedling just sprouted with two cotyledons',
            'Frame 2: First true leaves beginning to emerge',
            'Frame 3: Stem height increasing noticeably',
            'Frame 4: Multiple leaf pairs visible and expanding',
            'Frame 5: Plant reaching mid-growth stage with dense foliage',
            'Frame 6: Flower buds beginning to form at stem tips',
            'Frame 7: Flowers opening with vibrant colors',
            'Frame 8: Mature flowering plant with full canopy'
        ],
        'Frame_Captioning': [
            'Tiny green seedling with oval cotyledons on soil',
            'Young plant with emerging compound leaves',
            'Rapidly growing stem with alternating leaves',
            'Bushy plant with well-developed leaf structure',
            'Dense foliage with visible leaf veins and texture',
            'Plant developing small green buds at branch tips',
            'Colorful flowers blooming throughout the canopy',
            'Fully mature flowering plant in full bloom'
        ]
    },
    'video_002': {
        'Video_Reasoning': 'Crystallization process showing dissolved salt forming crystal structures over 24 hours in evaporating water.',
        'Frame_Reasoning': [
            'Frame 1: Clear salt solution with no visible crystals',
            'Frame 2: First microscopic crystals beginning to form',
            'Frame 3: Small cubic structures appearing throughout solution',
            'Frame 4: Crystal size increasing with more nucleation sites',
            'Frame 5: Larger crystals forming regular geometric patterns',
            'Frame 6: Crystal growth accelerating as water evaporates',
            'Frame 7: Large interconnected crystal formations visible',
            'Frame 8: Fully crystallized structure with no remaining liquid'
        ],
        'Frame_Captioning': [
            'Transparent liquid solution with uniform appearance',
            'Tiny sparkles emerging in the solution',
            'Small glittering cubic crystals scattered throughout',
            'Medium-sized crystals with defined edges and faces',
            'Growing crystal network forming geometric arrangements',
            'Large crystal formations becoming dominant in container',
            'Extensive crystalline structure nearly filling the space',
            'Dense crystalline mass completely filling the container'
        ]
    }
}

ARGS = {
    'txt_prompt': txt_prompt,
    'data': data
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch, tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
create_prompts Ran
ran; our contract draft was wrong, not the code
pku-yuangroup/magictime/data_preprocess/step3_1_GPT4V_video_caption_detail.py
code served (permissive licence) · get_code("c744aa94b5baa1ab")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np
from tqdm import tqdm

torch.manual_seed(0)

txt_prompt = '\nImagine you are a data annotator, specialized in generating summaries for time-lapse videos. You will be supplied with "Video_Reasoning", "8_Key-Frames_Reasoning", and "8_Key-Frames_Captioning" from a video, your task is to craft a succinct and precise summary for the given time-lapse video. Note: The summary should efficiently encapsulate all discernible elements, particularly emphasizing the primary subject. It is important to indicate whether the video pertains to a forward or reverse sequence. Additionally, integrate any time-related aspects from the video into the summary.\n\nSince only textual information is given, you can employ logical deductions to bridge any informational gaps if necessary. For guidance on the expected output format, refer to the provided examples:\n\n"Video_Summary": "The time-lapse video showcases the growth and ripening process of strawberries in a forward sequence. The video starts with fully bloomed strawberry flowers, which then wilt slightly. As the video progresses, the yellow stamens recede, and the flowers continue to wilt. The white petals disappear, and the green immature strawberries become more prominent. The strawberries then grow in size, displaying a green color with some red hues. As the video continues, the strawberries gradually ripen, turning from green to a deep red color."\n\n"Video_Summary": "This time-lapse video succinctly documents the 50-day decomposition process of a pear in a forward sequence, from its fresh, ripe state on day 1 to a shrunken, moldy, and rotten form by day 50. Throughout the video, the pear\'s gradual deterioration is evident through increasing browning, the development of mold patches, and significant changes in color, texture, and structure."\n\n"Video_Summary": "The time-lapse video showcases a Halloween pumpkin\'s decomposition process in reverse. The video starts with a pumpkin in a highly decomposed state at 92 days post-carving and then counts down the days, reversing the process. The pumpkin gradually re-inflates, reducing the signs of wrinkling and drying, until it appears freshly carved at 1 day post-carving."\n\n"Video_Summary": "{Video Summary}"\n\nAttention: Do not reply outside the example template! The process of reasoning and thinking should not be included in the {Video Summary}! Do not use words similar to by frame or at frame! Below are the Video, Video_Reasoning, Frame_Reasoning and Frame_Captioning.\n'

data = {
    'video_001': {
        'Video_Reasoning': 'This video shows a strawberry plant growing over time in a forward sequence.',
        'Frame_Reasoning': [
            'Frame 1: Initial flower blooming stage',
            'Frame 2: Flower in mid-bloom',
            'Frame 3: Early fruit formation',
            'Frame 4: Green strawberry developing',
            'Frame 5: Strawberry growing larger',
            'Frame 6: Color changing to light red',
            'Frame 7: Deep red color emerging',
            'Frame 8: Fully ripe strawberry'
        ],
        'Frame_Captioning': [
            'White strawberry flower with yellow stamens',
            'Pink and white flower petals open',
            'Sepals replacing flower petals',
            'Small green fruit forming',
            'Medium-sized green strawberry',
            'Strawberry turning pinkish-red',
            'Mostly red strawberry with some green',
            'Fully ripe deep red strawberry'
        ]
    },
    'video_002': {
        'Video_Reasoning': 'This video shows a pear decomposing over 50 days in a forward sequence.',
        'Frame_Reasoning': [
            'Frame 1: Fresh ripe pear',
            'Frame 2: Minor browning starts',
            'Frame 3: More browning visible',
            'Frame 4: Mold begins to develop',
            'Frame 5: Significant mold patches',
            'Frame 6: Pear shrinking with mold',
            'Frame 7: Advanced decomposition state',
            'Frame 8: Highly decomposed pear'
        ],
        'Frame_Captioning': [
            'Fresh green and yellow pear',
            'Small brown spots appearing',
            'Brown discoloration spreading',
            'White mold patches visible',
            'Mold covering significant area',
            'Pear deflated with dark coloring',
            'Mushy texture with extensive mold',
            'Severely decomposed remains'
        ]
    }
}

ARGS = {
    'txt_prompt': txt_prompt,
    'data': data
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch, tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
extract_score Ran
ran, honoured the contract we drafted
PKU-YuanGroup/ChronoMagic-Bench/UMT/step3_get_merge_umt_scores.py
code served (permissive licence) · get_code("690f98d4b6d870df")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def extract_score(score_str):
    match = re.search("UMTScore': (-?\\d+\\.\\d+)", score_str)
    if match:
        return float(match.group(1))
    return None

# Setup code
score_str_with_match = "Some text UMTScore': 42.5 more text"

ARGS = {
    "score_str": score_str_with_match
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type float, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_image_filenames Ran
ran; our contract draft was wrong, not the code
PKU-YuanGroup/ChronoMagic-Bench/GPT4o_MTScore/step1-get_temp_results.py
code served (permissive licence) · get_code("88a50570825e6b77")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import tempfile
import shutil

# Create a temporary directory with some test image files
temp_dir = tempfile.mkdtemp()

# Create some dummy image files
image_files = ['test1.jpg', 'test2.png', 'test3.jpeg', 'test4.bmp', 'test5.gif']
for img_file in image_files:
    open(os.path.join(temp_dir, img_file), 'w').close()

# Also create some non-image files to ensure filtering works
non_image_files = ['readme.txt', 'data.csv', 'script.py']
for non_img_file in non_image_files:
    open(os.path.join(temp_dir, non_img_file), 'w').close()

ARGS = {
    'directory': temp_dir
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
has_been_processed Ran
ran, violated the contract we drafted
pku-yuangroup/magictime/data_preprocess/step3_1_GPT4V_video_caption_concise.py
code served (permissive licence) · get_code("bf832b7c97d32bc2")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import json
import tempfile
from threading import Lock

# Create a temporary directory and file for testing
temp_dir = tempfile.mkdtemp()
output_file = os.path.join(temp_dir, "test_output.json")

# Create a sample output file with some processed video IDs
sample_data = {
    "video_001": {"status": "processed"},
    "video_002": {"status": "processed"}
}

with open(output_file, 'w') as f:
    json.dump(sample_data, f)

# Define arguments for the function
ARGS = {
    "video_id": "video_001",
    "output_file": output_file
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
image_b64 Ran
ran; our contract draft was wrong, not the code
PKU-YuanGroup/ChronoMagic-Bench/GPT4o_MTScore/step1-get_temp_results.py
code served (permissive licence) · get_code("8fec9877e0724b67")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import base64
import tempfile
import os

# Create a temporary image file for testing
with tempfile.NamedTemporaryFile(mode='wb', suffix='.png', delete=False) as f:
    # Write a minimal valid PNG file (1x1 transparent pixel)
    png_data = (
        b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01\x00\x00\x00\x01'
        b'\x08\x06\x00\x00\x00\x1f\x15\xc4\x89\x00\x00\x00\nIDATx\x9cc\x00\x01'
        b'\x00\x00\x05\x00\x01\r\n-\xb4\x00\x00\x00\x00IEND\xaeB`\x82'
    )
    f.write(png_data)
    temp_image_path = f.name

try:
    ARGS = {
        'image_path': temp_image_path
    }
finally:
    pass
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
image_b64 Ran
ran; our contract draft was wrong, not the code
pku-yuangroup/magictime/data_preprocess/step2_1_GPT4V_frame_caption.py
code served (permissive licence) · get_code("f3426404967f1cdf")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import base64
import tempfile
import os

# Create a temporary image file for testing
temp_dir = tempfile.mkdtemp()
temp_image_path = os.path.join(temp_dir, "test_image.png")

# Write a minimal valid PNG file (1x1 pixel)
png_bytes = (
    b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01\x00\x00\x00\x01'
    b'\x08\x02\x00\x00\x00\x90wS\xde\x00\x00\x00\x0cIDATx\x9cc\xf8\x0f\x00'
    b'\x00\x01\x01\x00\x05\xcc\xfb\xef\x00\x00\x00\x00IEND\xaeB`\x82'
)

with open(temp_image_path, 'wb') as f:
    f.write(png_bytes)

ARGS = {
    'image_path': temp_image_path
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
is_disordered Ran
ran, violated the contract we drafted
pku-yuangroup/magictime/data_preprocess/step2_2_preprocess_frame_caption.py
code served (permissive licence) · get_code("8433f9cddcd55ccd")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

def is_disordered(section):
    frames = []
    for entry in section:
        try:
            frame_num = int(entry.split(':')[0].split(' ')[1])
            frames.append(frame_num)
        except ValueError:
            continue
    return not all((earlier <= later for earlier, later in zip(frames, frames[1:])))

ARGS = {
    "section": [
        "Frame 1: data",
        "Frame 2: data",
        "Frame 3: data"
    ]
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_existing_results Ran
ran; our contract draft was wrong, not the code
pku-yuangroup/magictime/data_preprocess/step3_1_GPT4V_video_caption_concise.py
code served (permissive licence) · get_code("0a479b0595cd2868")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import json
import tempfile

# Create a temporary directory and file for testing
temp_dir = tempfile.mkdtemp()
file_path = os.path.join(temp_dir, "results.json")

ARGS = {
    "file_path": file_path
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_json Ran
ran; our contract draft was wrong, not the code
pku-yuangroup/magictime/data_preprocess/step2_2_preprocess_frame_caption.py
code served (permissive licence) · get_code("8d5c8c1fcb16fbe2")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import tempfile
import os

# Create a temporary JSON file with sample content
temp_dir = tempfile.mkdtemp()
json_file_path = os.path.join(temp_dir, "test.json")

# Write a sample JSON file
sample_data = {
    "name": "test",
    "values": [1, 2, 3],
    "nested": {"key": "value"}
}

with open(json_file_path, 'w') as f:
    json.dump(sample_data, f)

ARGS = {
    "file_path": json_file_path
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
merge_json_files_with_transmit_status Ran
ran; our contract draft was wrong, not the code
pku-yuangroup/magictime/data_preprocess/step4_1_create_webvid_format.py
code served (permissive licence) · get_code("84c5f662c19b0d36")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import pandas as pd
import tempfile
import os

# Create temporary files for testing
temp_dir = tempfile.mkdtemp()
caption_file = os.path.join(temp_dir, 'captions.json')
output_file = os.path.join(temp_dir, 'output.csv')

# Create a valid caption JSON file
caption_data = {
    'video_1': {'Video_GPT4_Caption': 'A cat sitting on a mat'},
    'video_2': {'Video_GPT4_Caption': 'A dog running in the park'},
    'video_3': {'Video_GPT4_Caption': 'A bird flying in the sky'}
}

with open(caption_file, 'w', encoding='utf-8') as f:
    json.dump(caption_data, f)

ARGS = {
    'caption_file': caption_file,
    'output_file': output_file
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
pandas — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
parse_entry Ran
ran; our contract draft was wrong, not the code
PKU-YuanGroup/ChronoMagic-Bench/GPT4o_MTScore/step2-get_GPT4o-MTScore.py
code served (permissive licence) · get_code("64ef6f61d121b16a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

# Analyze the function:
# - parse_entry takes a single parameter: `value`
# - It expects `value` to be a string
# - It calls rsplit('\n', 1) to split on newline
# - It expects the first part to contain 'Brief Reasoning Statement:'
# - It expects the second part to contain '"Score":' and curly braces
# - Returns a tuple of (brief_reasoning_statement, score)

# Construct a valid input string that matches the expected format
value = 'Brief Reasoning Statement: This is a test reasoning statement\n{"Score": 8.5}'

ARGS = {
    'value': value
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
parse_video_id Ran
ran; our contract draft was wrong, not the code
PKU-YuanGroup/ChronoMagic-Bench/GPT4o_MTScore/step1-get_temp_results.py
code served (permissive licence) · get_code("98e1a3659e986ade")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def parse_video_id(filename):
    match = re.match('(.+)_frame\\d+\\.png', filename)
    return match.group(1) if match else None

ARGS = {
    'filename': 'video_001_frame42.png'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
process_frame_caption Ran
ran; our contract draft was wrong, not the code
pku-yuangroup/magictime/data_preprocess/step2_2_preprocess_frame_caption.py
code served (permissive licence) · get_code("741f5ea273d405d5")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re
import json
import tempfile
import os

# Create a temporary JSON file with test data
temp_dir = tempfile.mkdtemp()
test_file = os.path.join(temp_dir, "test_captions.json")

# Create sample data that matches the expected format
sample_data = {
    "frame_1": """Brief Reasoning Statement: This frame shows a person walking in the park

"Reasoning": ["Person is moving", "Outdoor setting", "Daytime"]

"Captioning": ["A person walks", "Park environment", "Sunny weather"]
""",
    "frame_2": """Brief Reasoning Statement: The scene depicts a car parked

"Reasoning": ["Vehicle is stationary", "Urban location"]

"Captioning": ["Red car", "Street parking"]
""",
    "frame_3": """Some unmatched content without proper structure"""
}

# Write the sample data to the temp file
with open(test_file, 'w') as f:
    json.dump(sample_data, f)

# Define ARGS
ARGS = {
    "file_path": test_file
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
process_json Ran
ran; our contract draft was wrong, not the code
pku-yuangroup/magictime/data_preprocess/step3_2_preprocess_video_caption.py
code served (permissive licence) · get_code("f877f7eeafa5e4b6")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import tempfile
import os

# Create a temporary JSON file with test data
temp_dir = tempfile.mkdtemp()
test_file = os.path.join(temp_dir, "test_data.json")

# Create test data with some entries that match the pattern and some that don't
test_data = {
    "entry1": '"Video_Summary": "This is a test summary about the video"',
    "entry2": '"Other_Field": "Some other value"',
    "entry3": '"Video_Summary": "Another video summary with content"',
    "entry4": '"Description": "Just a description without video summary"'
}

# Write the test data to the temporary file
with open(test_file, 'w') as f:
    json.dump(test_data, f)

ARGS = {
    "file_path": test_file
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
read_json_file Ran
ran; our contract draft was wrong, not the code
pku-yuangroup/magictime/data_preprocess/step3_2_preprocess_video_caption.py
code served (permissive licence) · get_code("36c10de5d93aa54f")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import tempfile
import os

# Create a temporary JSON file with valid content
temp_dir = tempfile.mkdtemp()
test_file = os.path.join(temp_dir, "test.json")

test_data = {
    "key1": "value1",
    "key2": 42,
    "key3": [1, 2, 3],
    "key4": {"nested": "object"}
}

with open(test_file, 'w') as f:
    json.dump(test_data, f)

ARGS = {
    "file_path": test_file
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
remove_by_Frame Ran
ran; our contract draft was wrong, not the code
pku-yuangroup/magictime/data_preprocess/step3_2_preprocess_video_caption.py
code served (permissive licence) · get_code("ea99ef524e653f5f")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def remove_by_Frame(data):
    to_exclude = {}
    to_keep = {}
    pattern = re.compile('(by|at|in|on) Frame \\d+', re.IGNORECASE)
    for key, value in data.items():
        video_summary = value.get('Video_GPT4_Caption', '')
        if pattern.search(video_summary):
            to_exclude[key] = value
        else:
            to_keep[key] = value
    return (to_keep, to_exclude)


# Construct valid test data
data = {
    'video_1': {
        'Video_GPT4_Caption': 'A person walking in the park',
        'duration': 10
    },
    'video_2': {
        'Video_GPT4_Caption': 'Action at Frame 5 shows a collision',
        'duration': 15
    },
    'video_3': {
        'Video_GPT4_Caption': 'Bird flying by Frame 12 during sunset',
        'duration': 8
    },
    'video_4': {
        'Video_GPT4_Caption': 'Car on Frame 3 entering the garage',
        'duration': 12
    },
    'video_5': {
        'Video_GPT4_Caption': 'Simple scene with no frame references',
        'duration': 20
    }
}

ARGS = {
    'data': data
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
resize_frame Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
PKU-YuanGroup/ChronoMagic-Bench/GPT4o_MTScore/step0-extract_video_frames.py
code served (permissive licence) · get_code("f665b668e1ed740a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 3, dtype family int, shape (256, 341, 3) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import cv2
import numpy as np

# Create a sample frame - a simple RGB image with distinct height and width
# Using shape (480, 640, 3) to represent a typical video frame
frame = np.random.randint(0, 256, (480, 640, 3), dtype=np.uint8)

ARGS = {
    "frame": frame,
    "short_edge": 256
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
cv2, numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 3, dtype family int, shape (256, 341, 3) — shape (256, 341, 3) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
setup_logger Ran
ran; our contract draft was wrong, not the code
pku-yuangroup/open-sora-plan/opensora/train/train_causalvae.py
code served (permissive licence) · get_code("4d57397fa521749f")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned RootLogger. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import logging
from colorlog import ColoredFormatter

# The function takes a single parameter: rank
# rank is used in an f-string to format the logger name prefix
# It should be an integer or something that can be converted to string

ARGS = {
    "rank": 0
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
colorlog — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type RootLogger — a RootLogger output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a RootLogger output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
total_params Ran
ran, honoured the contract we drafted
pku-yuangroup/open-sora-plan/opensora/train/train_causalvae.py
code served (permissive licence) · get_code("e52f61a4ca40419d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear1 = nn.Linear(4, 3)
        self.linear2 = nn.Linear(3, 2)
    
    def forward(self, x):
        x = self.linear1(x)
        x = self.linear2(x)
        return x

model = SimpleModel()

ARGS = {
    "model": model
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
zero_module Ran
ran; our contract draft was wrong, not the code
pku-yuangroup/magictime/utils/unet_blocks.py
code served (permissive licence) · get_code("4719c763c53be3fe")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned SimpleModule. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Create a simple nn.Module with some parameters
class SimpleModule(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(4, 3)
        self.conv = nn.Conv1d(2, 5, 3)
    
    def forward(self, x):
        return x

module = SimpleModule()

ARGS = {
    "module": module
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type SimpleModule — a SimpleModule output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a SimpleModule output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_movement_vector Not yet run
unverified (harvested, no recorded run)
pku-yuangroup/chronomagic-bench/CHScore/step0-get_CHScore.py
code served (permissive licence) · get_code("f066b30259edc2d3")
How it ran

no invocation recorded.

process_video Not yet run
unverified (harvested, no recorded run)
PKU-YuanGroup/ChronoMagic-Bench/CHScore/step0-get_CHScore.py
code served (permissive licence) · get_code("77225e720da62c78")
How it ran

no invocation recorded.

read_video_from_path Not yet run
unverified (harvested, no recorded run)
PKU-YuanGroup/ChronoMagic-Bench/CHScore/step0-get_CHScore.py
code served (permissive licence) · get_code("f4e0669c270a5ee9")
How it ran

no invocation recorded.

Also present in other papers' code

Of 27 samples lifted from this paper's linked repositories, 27 carry a function identity and 0 do not. Of the 26 distinct functions with an identity, 7 are also present in other papers whose code we hold (2 with at least one paper dated earlier, 1 the same month, 7 later) and 19 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
zero_module 196 other papers
133 dated earlier, 3 the same month, 59 later, 1 same year, order not determinable · 49 only as an assembly brick
2006.11239 (2020-06)
Denoising Diffusion Probabilistic Models
2010.02502 (2020-10)
Denoising Diffusion Implicit Models
2105.05233 (2021-05)
Diffusion Models Beat GANs on Image Synthesis
and 193 more not named here
load_json 5 other papers
2 dated earlier, 3 later
2401.14490 (2024-01)
LongHealth: A Question Answering Benchmark with Long Clinical Documents
2405.19360 (2024-05)
ART: Automatic Red-teaming for Text-to-Image Models to Protect Benign Users
2407.15720 (2024-07)
Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalability
and 2 more not named here
check_unused_params 2 other papers
2 later
2412.00131 (2024-12)
Open-Sora Plan: Open-Source Large Video Generation Model
Li_WF-VAE_Enhancing_Video_VAE_by_Wavelet-Driven_Energy_Flow_for_Latent_CVPR_2025_paper (2025 (year only))
WF-VAE: Enhancing Video VAE by Wavelet-Driven Energy Flow for Latent Video Diffusion Model
total_params 2 other papers
2 later
2412.00131 (2024-12)
Open-Sora Plan: Open-Source Large Video Generation Model
Li_WF-VAE_Enhancing_Video_VAE_by_Wavelet-Driven_Energy_Flow_for_Latent_CVPR_2025_paper (2025 (year only))
WF-VAE: Enhancing Video VAE by Wavelet-Driven Energy Flow for Latent Video Diffusion Model
calculate_average_score 1 other paper
1 later
2410.05363 (2024-10)
Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation
read_json_file 1 other paper
1 later
2506.13326 (2025-06)
VIS-Shepherd: Constructing Critic for LLM-based Data Visualization Generation
setup_logger 1 other paper
1 later
2412.00131 (2024-12)
Open-Sora Plan: Open-Source Large Video Generation Model

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

We propose a novel text-to-video (T2V) generation benchmark, ChronoMagic-Bench 2 , to evaluate the temporal and metamorphic knowledge skills in time-lapse video generation of the T2V models (e.g. Sora [9] and Lumiere [4]). Compared to existing benchmarks that focus on visual quality and text relevance of generated videos, ChronoMagic-Bench focuses on the models' ability to generate time-lapse videos with significant metamorphic amplitude and temporal coherence. The benchmark probes T2V models for their physics, biology, and chemistry capabilities, in a free-form text control. For these purposes, ChronoMagic-Bench introduces 1,649 prompts and real-world videos as references, categorized into four major types of time-lapse videos: biological, human creation, meteorological, and physical phenomena, which are further divided into 75 subcategories. This categorization ensures a comprehensive evaluation of the models' capacity to handle diverse and complex transformations. To accurately align human preference on the benchmark, we introduce two new automatic metrics, MTScore and CHScore, to evaluate the videos' metamorphic attributes and temporal coherence. MTScore measures the metamorphic amplitude, reflecting the degree of change over time, while CHScore assesses the temporal coherence, ensuring the generated videos maintain logical progression and continuity. Based on the ChronoMagic-Bench, we conduct comprehensive manual evaluations of ten representative T2V models, revealing their strengths and weaknesses across different categories of prompts, providing a thorough evaluation framework that addresses current gaps in video generation research. More encouragingly, we create a large-scale ChronoMagic-Pro dataset, containing 460k high-quality pairs of 720p time-lapse videos and detailed captions. Each caption ensures high physical content and large metamorphic amplitude, which have a far-reaching impact on the video generation community. 3 * Corresponding author 2 "Chrono" derives from the Greek word "chronos", which means "time". 3 The source data and code are publicly available on https://pku-yuangroup.github.io/ChronoMagic-Bench. 38th Conference on Neural Information Processing Systems (NeurIPS 2024) Track on Datasets and Benchmarks.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2406.18522")
get_code_for_paper("2406.18522")
have("2406.18522")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2406.18522.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2406.18522.svg)](https://syntology.ai/paper/2406.18522)

Connect an agent — have() is free.