SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2410.18693 · 2024

Unleashing LLM Reasoning Capability via Scalable Question Synthesis from Scratch

arXiv · PDF · Open in the Atlas

Code that ran

6 of 11 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 0 honoured the contract we drafted, 2 violated it and 0 have no contract verdict; 4 failures of our instrument; 5 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2410.18693.json · badge: /api/ran/2410.18693.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
yyding1/scalequest canonical (classifier label)
classifier basis: census
6 ran · 5 no recorded run
FunctionStatusWhere it lives
createDifficultyPrompt Ran
ran; our contract draft was wrong, not the code
yyding1/scalequest/src/train_question_generator/question_optim/prompts.py
code served (permissive licence) · get_code("5b44e74560a2f7e1")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

# Setup code for createDifficultyPrompt function

ARGS = {
    "problem": "A store sells apples for $2 each and oranges for $3 each. If someone buys 5 apples and 3 oranges, how much do they spend?"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
createSolvabilityPrompt Ran
ran; our contract draft was wrong, not the code
yyding1/scalequest/src/train_question_generator/question_optim/prompts.py
code served (permissive licence) · get_code("e35afc303e481325")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

# Define ARGS dict with all parameters for createSolvabilityPrompt
ARGS = {
    'problem': 'A store sells apples for $2 each. How many apples can you buy with $10?'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
filter_answer Ran
ran, violated the contract we drafted
yyding1/scalequest/src/data_generation/question_filtering/language_solvability_check.py
code served (permissive licence) · get_code("b65ce61293fa5c19")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# The function accesses line['generation_for_solvability_check'][0]
# and calls .lower() on it, so it expects a string.
# We need a dict with key 'generation_for_solvability_check' 
# that maps to a list/sequence with at least one element (a string).

line = {
    'generation_for_solvability_check': ['Yes, this is the answer']
}

ARGS = {
    'line': line
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
filter_query Ran
ran, violated the contract we drafted
yyding1/scalequest/src/data_generation/question_filtering/language_solvability_check.py
code served (permissive licence) · get_code("7d6fdd2ac2e21fdd")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

# Construct valid arguments for filter_query
# The function expects a dict with a 'query' key containing a string

ARGS = {
    'line': {
        'query': 'hello'
    }
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
process_data_on_device Ran
ran; our contract draft was wrong, not the code
yyding1/scalequest/src/data_generation/reward_filtering/rm_score.py
code served (permissive licence) · get_code("747dcdb948f1521d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. The code mentions cuda, and the harness aliases cuda to the CPU for every sample; whether this run needed that is not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
from unittest.mock import MagicMock

torch.manual_seed(0)

# Create a mock reward model that mimics internlm2-7b-reward behavior
class MockRewardModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(4, 1)
    
    def get_score(self, tokenizer, chat_messages):
        # Return a mock score (float)
        return 0.75

# Create a mock tokenizer
class MockTokenizer:
    def __call__(self, text):
        return {"input_ids": [1, 2, 3], "attention_mask": [1, 1, 1]}
    
    def encode(self, text):
        return [1, 2, 3]

# Create mock dataset - list of dictionaries with query and response
dataset = [
    {
        'query': 'What is machine learning?',
        'response': 'Machine learning is a subset of AI that enables systems to learn from data.'
    },
    {
        'query': 'Explain neural networks',
        'response': 'Neural networks are computational models inspired by biological neurons.'
    }
]

# Create mock model path
score_model_path = 'internlm2-7b-reward'

# Create mock tokenizer
score_tokenizer = MockTokenizer()

# GPU ID
gpu_id = 0

# Patch AutoModel.from_pretrained to return our mock model
import transformers
original_from_pretrained = transformers.AutoModel.from_pretrained

def mock_from_pretrained(*args, **kwargs):
    return MockRewardModel()

transformers.AutoModel.from_pretrained = mock_from_pretrained

ARGS = {
    'dataset': dataset,
    'score_model_path': score_model_path,
    'score_tokenizer': score_tokenizer,
    'gpu_id': gpu_id
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch, tqdm, transformers — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
run_openai_inference_async Ran
ran; our contract draft was wrong, not the code
yyding1/scalequest/src/train_question_generator/question_optim/openai_gen.py
code served (permissive licence) · get_code("d99b1fd214fb4ca1")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned coroutine. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import numpy as np

# Set a dummy OpenAI API key for testing (won't actually be used since we're not making real requests)
os.environ['OPENAI_KEY'] = 'sk-test-dummy-key-for-setup'

# Create sample request data
requests = [
    {
        'messages': [
            {'role': 'user', 'content': 'What is 2+2?'}
        ],
        'metadata': 'request_1'
    },
    {
        'messages': [
            {'role': 'user', 'content': 'What is the capital of France?'}
        ],
        'metadata': 'request_2'
    }
]

# Define ARGS dictionary with all parameters for run_openai_inference_async
ARGS = {
    'requests': requests,
    'model': 'gpt-3.5-turbo',
    'max_tokens': 100,
    'num_generations': 1,
    'temperature': 0.7,
    'top_p': 0.9,
    'openai_timeout': 45
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, openai, tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type coroutine — a coroutine output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a coroutine output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
generate_score Not yet run
unverified (harvested, no recorded run)
yyding1/scalequest/src/data_generation/question_filtering/generate_difficulty_score.py
code served (permissive licence) · get_code("686a5f1d92047af3")
How it ran

no invocation recorded.

generate_score_parallel Not yet run
unverified (harvested, no recorded run)
yyding1/scalequest/src/data_generation/reward_filtering/rm_score.py
code served (permissive licence) · get_code("10e724226473950f")
How it ran

no invocation recorded.

process_data_on_device Not yet run
unverified (harvested, no recorded run)
yyding1/scalequest/src/data_generation/question_filtering/generate_difficulty_score.py
code served (permissive licence) · get_code("5086e0bcf250b990")
How it ran

no invocation recorded.

process_wrapper Not yet run
unverified (harvested, no recorded run)
yyding1/scalequest/src/data_generation/question_filtering/generate_difficulty_score.py
code served (permissive licence) · get_code("fb330c46d149f244")
How it ran

no invocation recorded.

run_openai_inference Not yet run
unverified (harvested, no recorded run)
yyding1/scalequest/src/train_question_generator/question_optim/openai_gen.py
code served (permissive licence) · get_code("c7214b71033ab87a")
How it ran

no invocation recorded.

Also present in other papers' code

Of 11 samples lifted from this paper's linked repositories, 11 carry a function identity and 0 do not. Of the 11 distinct functions with an identity, 0 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 0 later) and 11 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Improving the mathematical reasoning capabilities of Large Language Models (LLMs) is critical for advancing artificial intelligence. However, access to extensive, diverse, and high-quality reasoning datasets remains a significant challenge, particularly for the open-source community. In this paper, we propose ScaleQuest, a novel, scalable, and cost-effective data synthesis method that enables the generation of large-scale mathematical reasoning datasets using lightweight 7B-scale models. ScaleQuest introduces a two-stage question-tuning process comprising Question Fine-Tuning (QFT) and Question Preference Optimization (QPO) to unlock the question generation capabilities of problem-solving models. By generating diverse questions from scratch -- without relying on powerful proprietary models or seed data -- we produce a dataset of 1 million problem-solution pairs. Our experiments demonstrate that models trained on our data outperform existing open-source datasets in both in-domain and out-of-domain evaluations. Furthermore, our approach shows continued performance improvement as the volume of training data increases, highlighting its potential for ongoing data scaling. The extensive improvements observed in code reasoning tasks demonstrate the generalization capabilities of our proposed method. Our work provides the open-source community with a practical solution to enhance the mathematical reasoning abilities of LLMs.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2410.18693")
get_code_for_paper("2410.18693")
have("2410.18693")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2410.18693.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2410.18693.svg)](https://syntology.ai/paper/2410.18693)

Connect an agent — have() is free.