SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2408.07930 · 2024

MAG-SQL: Multi-Agent Generative Approach with Soft Schema Linking and Iterative Sub-SQL Refinement for Text-to-SQL

arXiv · PDF · Open in the Atlas

Code that ran

21 of 25 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 3 honoured the contract we drafted, 9 violated it and 0 have no contract verdict; 9 failures of our instrument; 4 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2408.07930.json · badge: /api/ran/2408.07930.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
LancelotXWX/MAG-SQL canonical (classifier label)
classifier basis: census
21 ran · 4 no recorded run
FunctionStatusWhere it lives
condition_has_like Ran
ran, violated the contract we drafted
LancelotXWX/MAG-SQL/evaluation/evaluation_spider.py
code served (permissive licence) · get_code("8231e3fdffa9c8ee")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Sonnet 4.5, prompt python_fixture_v1) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

WHERE_OPS = ('not', 'between', '=', '>', '<', '>=', '<=', '!=', 'in', 'like', 'is', 'exists')

# The function expects conds to be a list where every other element (starting at 0, i.e., conds[::2])
# is a cond_unit. Each cond_unit must be indexable with [1] to get an operation index.
# The operation index should be an integer that can be checked against WHERE_OPS.index('like').

# WHERE_OPS.index('like') = 9

# Create a valid conds structure
# conds[::2] gives us every other element starting from index 0
# Each cond_unit needs to be a sequence where cond_unit[1] is an integer representing the operation

# Example: conds with 4 elements, so conds[::2] will give us elements at indices 0 and 2
# Each cond_unit can be a tuple/list like (col_id, op_id, value)

conds = [
    (0, 2, 'value1'),  # index 0: operation '=' (index 2)
    'separator',        # index 1: some separator (not accessed by [::2])
    (1, 9, 'pattern'), # index 2: operation 'like' (index 9)
    'separator'         # index 3: another separator (not accessed by [::2])
]

ARGS = {'conds': conds}
Setup block drafted by
a language model, Claude Sonnet 4.5, prompt python_fixture_v1
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
condition_has_or Ran
ran, violated the contract we drafted
LancelotXWX/MAG-SQL/evaluation/evaluation_spider.py
code served (permissive licence) · get_code("f2d76234528d3b57")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Sonnet 4.5, prompt python_fixture_v1) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

# The function checks if 'or' is in conds[1::2], which means it's looking at every other element starting from index 1
# This suggests conds is a list where odd indices might contain operators like 'or', 'and', etc.
# A typical structure would be: [condition1, 'or', condition2, 'and', condition3, ...]

# Example 1: conds with 'or' in odd positions
conds_with_or = ['condition1', 'or', 'condition2', 'and', 'condition3']

# Example 2: conds without 'or' in odd positions
conds_without_or = ['condition1', 'and', 'condition2', 'and', 'condition3']

# We'll use the first example as the argument
ARGS = {
    'conds': conds_with_or
}
Setup block drafted by
a language model, Claude Sonnet 4.5, prompt python_fixture_v1
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
condition_has_sql Ran
ran, violated the contract we drafted
LancelotXWX/MAG-SQL/evaluation/evaluation_spider.py
code served (permissive licence) · get_code("954c76e256cc7220")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Sonnet 4.5, prompt python_fixture_v1) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

# The function checks if any condition unit in conds (at even indices) 
# has val1 (index 3) or val2 (index 4) that is a dict.
# conds is a list where every other element (starting at 0) is a cond_unit
# Each cond_unit must be indexable with indices 3 and 4

# Case 1: conds with no SQL (no dicts in positions 3 or 4)
conds_no_sql = [
    [0, 1, 2, None, None],  # cond_unit at index 0
    "separator",             # odd index (skipped)
    [0, 1, 2, "value", 5],  # cond_unit at index 2
    "separator",             # odd index (skipped)
]

# Case 2: conds with SQL (dict in position 3)
conds_with_sql_val1 = [
    [0, 1, 2, {"nested": "sql"}, None],  # cond_unit with dict at index 3
    "separator",
]

# Case 3: conds with SQL (dict in position 4)
conds_with_sql_val2 = [
    [0, 1, 2, None, {"nested": "query"}],  # cond_unit with dict at index 4
]

# Use one of these for the test
ARGS = {
    "conds": conds_with_sql_val1
}
Setup block drafted by
a language model, Claude Sonnet 4.5, prompt python_fixture_v1
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
execute_sql Ran
ran, honoured the contract we drafted
LancelotXWX/MAG-SQL/evaluation/evaluation_bird_ex.py
pointer only (licence: MIT); the stored copy get_code serves, from HasanAlpCaferoglu/E-SQL, is pointer only (licence: NONE) · get_code("9c222961369b8d61")
How it ran

The body of this sample is withheld: this paper's link allows the body inline (MIT), but get_code serves the stored copy, first harvested from HasanAlpCaferoglu/E-SQL, and GitHub detected no licence in that repository when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

execute_sql Ran
ran, honoured the contract we drafted
LancelotXWX/MAG-SQL/evaluation/evaluation_bird_ves.py
code served (permissive licence) · get_code("9a2dafca24381ab4")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import sqlite3
import tempfile
import os

# Create a temporary database file
temp_dir = tempfile.gettempdir()
db_path = os.path.join(temp_dir, "test_db.sqlite")

# Clean up if it exists
if os.path.exists(db_path):
    os.remove(db_path)

# Create a simple database with a test table
conn = sqlite3.connect(db_path)
cursor = conn.cursor()
cursor.execute("CREATE TABLE test_table (id INTEGER PRIMARY KEY, name TEXT)")
cursor.execute("INSERT INTO test_table (name) VALUES ('test1')")
cursor.execute("INSERT INTO test_table (name) VALUES ('test2')")
conn.commit()
conn.close()

# Define the SQL query to execute
sql = "SELECT * FROM test_table"

# Build the arguments dictionary
ARGS = {
    "sql": sql,
    "db_path": db_path
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type float, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_schema Ran
ran; our contract draft was wrong, not the code
LancelotXWX/MAG-SQL/evaluation/process_sql.py
code served (permissive licence) · get_code("0992d2a47ae733fa")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import sqlite3
import tempfile
import os

# Create a temporary SQLite database with some test tables
temp_dir = tempfile.mkdtemp()
db_path = os.path.join(temp_dir, "test.db")

conn = sqlite3.connect(db_path)
cursor = conn.cursor()

# Create a sample table
cursor.execute("""
    CREATE TABLE users (
        id INTEGER PRIMARY KEY,
        name TEXT,
        email TEXT
    )
""")

# Create another sample table
cursor.execute("""
    CREATE TABLE products (
        id INTEGER PRIMARY KEY,
        title TEXT,
        price REAL,
        stock INTEGER
    )
""")

conn.commit()
conn.close()

ARGS = {
    "db": db_path
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_schema_from_json Ran
ran; our contract draft was wrong, not the code
LancelotXWX/MAG-SQL/evaluation/process_sql.py
pointer only (licence: MIT); the stored copy get_code serves, from ygan/NatSQL, is pointer only (licence: NONE) · get_code("c4dcaf29b663a364")
How it ran

The body of this sample is withheld: this paper's link allows the body inline (MIT), but get_code serves the stored copy, first harvested from ygan/NatSQL, and GitHub detected no licence in that repository when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

is_commonword Ran
ran, violated the contract we drafted
LancelotXWX/MAG-SQL/main_scripts/bridge_content_encoder.py
code served (permissive licence) · get_code("b724cc2d4066a429")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# The function is_commonword takes a single string parameter 's'
# and checks if its stripped version is in the _commonwords set
# No complex objects needed - just a simple string

ARGS = {
    's': 'yes'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
is_number Ran
ran, violated the contract we drafted
LancelotXWX/MAG-SQL/main_scripts/bridge_content_encoder.py
code served (permissive licence) · get_code("f39fdc1ce6cffe08")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# The function is_number takes a single string parameter 's'
# It tries to convert the string to a float after removing commas
# We need to provide valid test cases

ARGS = {
    "s": "123.45"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
is_stopword Ran
ran, violated the contract we drafted
LancelotXWX/MAG-SQL/main_scripts/bridge_content_encoder.py
code served (permissive licence) · get_code("d56a1f9221823a48")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

# The function is_stopword takes a single parameter: s (a string)
# It checks if s.strip() is in the _stopwords set
# So we need to provide a string argument

ARGS = {
    "s": "hello"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
is_valid_date Ran
ran, violated the contract we drafted
LancelotXWX/MAG-SQL/main_scripts/utils.py
code served (permissive licence) · get_code("e683610fb710931f")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def is_valid_date(date_str):
    if not isinstance(date_str, str):
        return False
    date_str = date_str.split()[0]
    if len(date_str) != 10:
        return False
    pattern = '^\\d{4}-\\d{2}-\\d{2}$'
    if re.match(pattern, date_str):
        year, month, day = map(int, date_str.split('-'))
        if year < 1 or month < 1 or month > 12 or (day < 1) or (day > 31):
            return False
        else:
            return True
    else:
        return False

ARGS = {
    'date_str': '2023-12-25'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
is_valid_date_column Ran
ran, violated the contract we drafted
LancelotXWX/MAG-SQL/main_scripts/utils.py
code served (permissive licence) · get_code("6e6792ab3f04a268")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def is_valid_date(date_str):
    if not isinstance(date_str, str):
        return False
    date_str = date_str.split()[0]
    if len(date_str) != 10:
        return False
    pattern = '^\d{4}-\d{2}-\d{2}$'
    if re.match(pattern, date_str):
        year, month, day = map(int, date_str.split('-'))
        if year < 1 or month < 1 or month > 12 or (day < 1) or (day > 31):
            return False
        else:
            return True
    else:
        return False

def is_valid_date_column(col_value_lst):
    for col_value in col_value_lst:
        if not is_valid_date(col_value):
            return False
    return True

# Setup code to construct valid arguments
col_value_lst = [
    "2023-01-15",
    "2024-12-25",
    "2000-06-30"
]

ARGS = {
    "col_value_lst": col_value_lst
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
iterated_execute_sql Ran
ran, honoured the contract we drafted
LancelotXWX/MAG-SQL/evaluation/evaluation_bird_ves.py
code served (permissive licence) · get_code("2b57439bdcb8e43d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float64, ndim 0, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np
import sqlite3
import tempfile
import os

# Create a temporary database with sample data
temp_db_fd, temp_db_path = tempfile.mkstemp(suffix='.db')
os.close(temp_db_fd)

conn = sqlite3.connect(temp_db_path)
cursor = conn.cursor()

# Create a simple test table
cursor.execute('''
    CREATE TABLE test_table (
        id INTEGER PRIMARY KEY,
        value INTEGER
    )
''')

# Insert some test data
for i in range(10):
    cursor.execute('INSERT INTO test_table (value) VALUES (?)', (i * 10,))

conn.commit()
conn.close()

# Define the arguments
ARGS = {
    'predicted_sql': 'SELECT * FROM test_table WHERE value > 20',
    'ground_truth': 'SELECT * FROM test_table WHERE value > 20',
    'db_path': temp_db_path,
    'iterate_num': 3
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type float64, ndim 0, dtype family float — no output shape is stated: observe_signature.py recorded float ndim 0, which disagrees with the fixture run
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
join_tokens Ran
ran; our contract draft was wrong, not the code
LancelotXWX/MAG-SQL/evaluation/parse.py
code served (permissive licence) · get_code("2178b95c919ae420")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

from typing import List, Tuple, Set, Iterator, Dict, Any, Union
from collections import namedtuple

Token = namedtuple('Token', ['ttype', 'value'])

# Create a list of Token objects for testing
tokens = [
    Token(ttype='WORD', value='hello'),
    Token(ttype='SPACE', value='  '),
    Token(ttype='WORD', value='world'),
    Token(ttype='SPACE', value=' '),
    Token(ttype='WORD', value='test'),
]

ARGS = {
    'tokens': tokens
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_json Ran
ran; our contract draft was wrong, not the code
LancelotXWX/MAG-SQL/evaluation/evaluation_bird_ex.py
pointer only (licence: MIT); the stored copy get_code serves, from wbbeyourself/mac-sql, is pointer only (licence: NONE) · get_code("72eccdcbaaff9c8d")
How it ran

The body of this sample is withheld: this paper's link allows the body inline (MIT), but get_code serves the stored copy, first harvested from wbbeyourself/mac-sql, and GitHub detected no licence in that repository when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

permute_tuple Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
LancelotXWX/MAG-SQL/evaluation/exec_eval.py
code served (permissive licence) · get_code("f23d6ff3772963bc")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# The function permute_tuple takes:
# 1. element: a Tuple - can be any tuple
# 2. perm: a Tuple - should be a permutation of indices matching element length

# Create a simple tuple with 4 elements of different types
element = (1, "hello", 3.14, torch.tensor([1, 2, 3]))

# Create a permutation tuple that reorders the indices
# For a 4-element tuple, a valid permutation could be (3, 1, 0, 2)
perm = (3, 1, 0, 2)

ARGS = {
    "element": element,
    "perm": perm,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
postprocess Ran
ran; our contract draft was wrong, not the code
LancelotXWX/MAG-SQL/evaluation/parse.py
code served (permissive licence) · get_code("58e472cff9780d63")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# The function postprocess takes a single string parameter 'query'
# and performs string replacements on it.
# No complex objects needed - just a simple string.

ARGS = {
    'query': '> = test < = and ! = check'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
quick_rej Ran
ran, violated the contract we drafted
LancelotXWX/MAG-SQL/evaluation/exec_eval.py
code served (permissive licence) · get_code("a40510ba6a37f798")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Analyze the function:
# - result1: List[Tuple] - list of tuples
# - result2: List[Tuple] - list of tuples
# - order_matters: bool - determines comparison mode

# The function processes tuples through unorder_row which sorts elements
# by their string representation and type. Then it either compares lists
# directly (order_matters=True) or as sets (order_matters=False).

# Create test data with various tuple types
result1 = [
    (1, "a", 2.5),
    (10, "b", 3.5),
    (5, "c", 1.5),
]

result2 = [
    (5, "c", 1.5),
    (1, "a", 2.5),
    (10, "b", 3.5),
]

order_matters = False

ARGS = {
    "result1": result1,
    "result2": result2,
    "order_matters": order_matters,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
rename_file Ran
ran; our contract draft was wrong, not the code
LancelotXWX/MAG-SQL/main_scripts/utils.py
code served (permissive licence) · get_code("009030dd773225d2")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import tempfile
import shutil

# Create a temporary directory for testing
temp_dir = tempfile.mkdtemp()

# Create a temporary test file
test_file_path = os.path.join(temp_dir, "test.txt")
with open(test_file_path, "w") as f:
    f.write("test content")

# Define arguments for the rename_file function
ARGS = {
    "file_path": test_file_path,
    "new_name": "backup"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
replace_multiple_spaces Ran
ran; our contract draft was wrong, not the code
LancelotXWX/MAG-SQL/evaluation/evaluation_bird_ex.py
pointer only (licence: MIT); the stored copy get_code serves, from wbbeyourself/mac-sql, is pointer only (licence: NONE) · get_code("13917dfdddd70f5e")
How it ran

The body of this sample is withheld: this paper's link allows the body inline (MIT), but get_code serves the stored copy, first harvested from wbbeyourself/mac-sql, and GitHub detected no licence in that repository when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

unorder_row Ran
ran; our contract draft was wrong, not the code
LancelotXWX/MAG-SQL/evaluation/exec_eval.py
code served (permissive licence) · get_code("afc8ef89576ef2aa")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np
from typing import Tuple, Any, List, Set

torch.manual_seed(0)

# Construct a tuple with mixed types to test the sorting logic
row = (
    "zebra",
    42,
    3.14,
    torch.tensor([1, 2, 3]),
    {"key": "value"},
    [1, 2, 3],
    None,
)

ARGS = {
    "row": row,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
api_func Not yet run
unverified (harvested, no recorded run)
LancelotXWX/MAG-SQL/main_scripts/llm.py
code served (permissive licence) · get_code("61ca5e74d54e4c09")
How it ran

no invocation recorded.

clean_abnormal Not yet run
unverified (harvested, no recorded run)
LancelotXWX/MAG-SQL/evaluation/evaluation_bird_ves.py
code served (permissive licence) · get_code("3553bf8903f0b994")
How it ran

no invocation recorded.

safe_call_llm Not yet run
unverified (harvested, no recorded run)
LancelotXWX/MAG-SQL/main_scripts/llm.py
code served (permissive licence) · get_code("a0e3cdde26258b5d")
How it ran

no invocation recorded.

tokenize Not yet run
unverified (harvested, no recorded run)
LancelotXWX/MAG-SQL/evaluation/process_sql.py
code served (permissive licence) · get_code("a906f5de1f5971b0")
How it ran

no invocation recorded.

Also present in other papers' code

Of 25 samples lifted from this paper's linked repositories, 25 carry a function identity and 0 do not. Of the 25 distinct functions with an identity, 19 are also present in other papers whose code we hold (19 with at least one paper dated earlier, 0 the same month, 12 later) and 6 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
condition_has_like 14 other papers
10 dated earlier, 3 later, 1 same year, order not determinable
1809.08887 (2018-09)
Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task
1906.02285 (2019-06)
SParC: Cross-Domain Semantic Parsing in Context
2020.emnlp-main.564 (2020 (year only))
Re-examining the Role of Schema Linking in Text-to-SQL
and 11 more not named here
condition_has_or 14 other papers
10 dated earlier, 3 later, 1 same year, order not determinable
1809.08887 (2018-09)
Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task
1906.02285 (2019-06)
SParC: Cross-Domain Semantic Parsing in Context
2020.emnlp-main.564 (2020 (year only))
Re-examining the Role of Schema Linking in Text-to-SQL
and 11 more not named here
get_schema_from_json 13 other papers
8 dated earlier, 5 later
1911.04942 (2019-11)
RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers
2020.emnlp-main.564 (2020 (year only))
Re-examining the Role of Schema Linking in Text-to-SQL
2106.01093 (2021-06)
LGESQL: Line Graph Enhanced Text-to-SQL Model with Mixed Local and Non-Local Relations
and 10 more not named here
condition_has_sql 11 other papers
9 dated earlier, 1 later, 1 same year, order not determinable
1809.08887 (2018-09)
Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task
1906.02285 (2019-06)
SParC: Cross-Domain Semantic Parsing in Context
2020.emnlp-main.564 (2020 (year only))
Re-examining the Role of Schema Linking in Text-to-SQL
and 8 more not named here
get_schema 11 other papers
8 dated earlier, 3 later
1911.04942 (2019-11)
RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers
2020.emnlp-main.564 (2020 (year only))
Re-examining the Role of Schema Linking in Text-to-SQL
2106.01093 (2021-06)
LGESQL: Line Graph Enhanced Text-to-SQL Model with Mixed Local and Non-Local Relations
and 8 more not named here
tokenize 10 other papers
7 dated earlier, 3 later
1911.04942 (2019-11)
RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers
2020.emnlp-main.564 (2020 (year only))
Re-examining the Role of Schema Linking in Text-to-SQL
2106.01093 (2021-06)
LGESQL: Line Graph Enhanced Text-to-SQL Model with Mixed Local and Non-Local Relations
and 7 more not named here
permute_tuple 6 other papers
3 dated earlier, 3 later
2302.08468 (2023-02)
LEVER: Learning to Verify Language-to-Code Generation with Execution
2310.17342 (2023-10)
ACT-SQL: In-Context Learning for Text-to-SQL with Automatically-Generated Chain-of-Thought
2406.12288 (2024-06)
An Investigation of Neuron Activation as a Unified Lens to Explain Chain-of-Thought Eliciting Arithmetic Reasoning of LLMs
and 3 more not named here
quick_rej 6 other papers
3 dated earlier, 3 later
2302.08468 (2023-02)
LEVER: Learning to Verify Language-to-Code Generation with Execution
2310.17342 (2023-10)
ACT-SQL: In-Context Learning for Text-to-SQL with Automatically-Generated Chain-of-Thought
2406.12288 (2024-06)
An Investigation of Neuron Activation as a Unified Lens to Explain Chain-of-Thought Eliciting Arithmetic Reasoning of LLMs
and 3 more not named here
unorder_row 6 other papers
3 dated earlier, 3 later
2302.08468 (2023-02)
LEVER: Learning to Verify Language-to-Code Generation with Execution
2310.17342 (2023-10)
ACT-SQL: In-Context Learning for Text-to-SQL with Automatically-Generated Chain-of-Thought
2406.12288 (2024-06)
An Investigation of Neuron Activation as a Unified Lens to Explain Chain-of-Thought Eliciting Arithmetic Reasoning of LLMs
and 3 more not named here
is_commonword 5 other papers
5 dated earlier
2109.05093 (2021-09)
PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models
2212.09741 (2022-12)
One Embedder, Any Task: Instruction-Finetuned Text Embeddings
2306.08891 (2023-06)
Interleaving Pre-Trained Language Models and Large Language Models for Zero-Shot NL2SQL Generation
and 2 more not named here
is_number 5 other papers
5 dated earlier
2109.05093 (2021-09)
PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models
2212.09741 (2022-12)
One Embedder, Any Task: Instruction-Finetuned Text Embeddings
2306.08891 (2023-06)
Interleaving Pre-Trained Language Models and Large Language Models for Zero-Shot NL2SQL Generation
and 2 more not named here
is_stopword 5 other papers
5 dated earlier
2109.05093 (2021-09)
PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models
2212.09741 (2022-12)
One Embedder, Any Task: Instruction-Finetuned Text Embeddings
2306.08891 (2023-06)
Interleaving Pre-Trained Language Models and Large Language Models for Zero-Shot NL2SQL Generation
and 2 more not named here
join_tokens 5 other papers
3 dated earlier, 2 later
2302.08468 (2023-02)
LEVER: Learning to Verify Language-to-Code Generation with Execution
2310.17342 (2023-10)
ACT-SQL: In-Context Learning for Text-to-SQL with Automatically-Generated Chain-of-Thought
2406.12288 (2024-06)
An Investigation of Neuron Activation as a Unified Lens to Explain Chain-of-Thought Eliciting Arithmetic Reasoning of LLMs
and 2 more not named here
postprocess 5 other papers
3 dated earlier, 2 later
2302.08468 (2023-02)
LEVER: Learning to Verify Language-to-Code Generation with Execution
2310.17342 (2023-10)
ACT-SQL: In-Context Learning for Text-to-SQL with Automatically-Generated Chain-of-Thought
2406.12288 (2024-06)
An Investigation of Neuron Activation as a Unified Lens to Explain Chain-of-Thought Eliciting Arithmetic Reasoning of LLMs
and 2 more not named here
execute_sql 3 other papers
2 dated earlier, 1 later
2312.11242 (2023-12)
MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQL
2405.15307 (2024-05)
Before Generation, Align it! A Novel and Effective Strategy for Mitigating Hallucinations in Text-to-SQL Generation
2409.16751 (2024-09)
E-SQL: Direct Schema Linking via Question Enrichment in Text-to-SQL
load_json 2 other papers
2 dated earlier
2312.11242 (2023-12)
MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQL
2405.15307 (2024-05)
Before Generation, Align it! A Novel and Effective Strategy for Mitigating Hallucinations in Text-to-SQL Generation
clean_abnormal 1 other paper
1 dated earlier
2405.15307 (2024-05)
Before Generation, Align it! A Novel and Effective Strategy for Mitigating Hallucinations in Text-to-SQL Generation
iterated_execute_sql 1 other paper
1 dated earlier
2405.15307 (2024-05)
Before Generation, Align it! A Novel and Effective Strategy for Mitigating Hallucinations in Text-to-SQL Generation
replace_multiple_spaces 1 other paper
1 dated earlier
2312.11242 (2023-12)
MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQL

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Recent In-Context Learning based methods have achieved remarkable success in Text-to-SQL task. However, there is still a large gap between the performance of these models and human performance on datasets with complex database schema and difficult questions, such as BIRD. Besides, existing work has neglected to supervise intermediate steps when solving questions iteratively with question decomposition methods, and the schema linking methods used in these works are very rudimentary. To address these issues, we propose MAG-SQL, a multi-agent generative approach with soft schema linking and iterative Sub-SQL refinement. In our framework, an entity-based method with tables' summary is used to select the columns in database, and a novel targets-conditions decomposition method is introduced to decompose those complex questions. Additionally, we build a iterative generating module which includes a Sub-SQL Generator and Sub-SQL Refiner, introducing external oversight for each step of generation. Through a series of ablation studies, the effectiveness of each agent in our framework has been demonstrated. When evaluated on the BIRD benchmark with GPT-4, MAG-SQL achieves an execution accuracy of 61.08%, compared to the baseline accuracy of 46.35% for vanilla GPT-4 and the baseline accuracy of 57.56% for MAC-SQL. Besides, our approach makes similar progress on Spider. The codes are available at https://github.com/LancelotXWX/MAG-SQL.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2408.07930")
get_code_for_paper("2408.07930")
have("2408.07930")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2408.07930.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2408.07930.svg)](https://syntology.ai/paper/2408.07930)

Connect an agent — have() is free.