SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 1909.05378 · 2019

CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases

arXiv · PDF · Open in the Atlas

Code that ran

8 of 11 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 0 honoured the contract we drafted, 0 violated it and 7 have no contract verdict; 1 failure of our instrument; 3 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/1909.05378.json · badge: /api/ran/1909.05378.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
amolk/editsql community (classifier label)
classifier basis not recorded on the link
8 ran · 3 no recorded run
FunctionStatusWhere it lives
compute_loss Ran
ran on a synthesized input
amolk/editsql/model/torch_utils.py
code served (permissive licence) · get_code("7e2d038f758acc4a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# gold_seq: list of str - sequence of gold tokens
gold_seq = ["hello", "world"]

# scores: list of torch.Tensor - score tensors for each position
# Each score tensor should be a probability distribution over vocabulary
scores = [
    torch.softmax(torch.randn(7), dim=0),  # 7-dim probability dist for position 0
    torch.softmax(torch.randn(7), dim=0),  # 7-dim probability dist for position 1
]

# index_to_token_maps: list of dict str->list of int
# Maps from token string to indices in the probability distribution where that token occurs
index_to_token_maps = [
    {"hello": [0, 1], "world": [2, 3], "other": [4, 5, 6]},
    {"hello": [0, 1], "world": [2, 3], "other": [4, 5, 6]},
]

# gold_tok_to_id: callable (str, dict)->list of int
# Maps from gold token and token_map dict to the indices where the gold token appears
def gold_tok_to_id(token, token_map):
    """Returns list of indices for the given token from the token_map."""
    if token in token_map:
        return token_map[token]
    return []

# noise: float, optional
noise = 1e-08

ARGS = {
    "gold_seq": gold_seq,
    "scores": scores,
    "index_to_token_maps": index_to_token_maps,
    "gold_tok_to_id": gold_tok_to_id,
    "noise": noise,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
encode_snippets_with_states Ran
ran; our contract draft was wrong, not the code
amolk/editsql/model/model.py
code served (permissive licence) · get_code("0bff7fccef337b38")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

class Snippet:
    def __init__(self, startpos, endpos):
        self.startpos = startpos
        self.endpos = endpos
        self.embedding = None
    
    def set_embedding(self, embedding):
        self.embedding = embedding

snippets = [
    Snippet(startpos=0, endpos=1),
    Snippet(startpos=1, endpos=2),
]

states = [
    torch.randn(4),
    torch.randn(4),
    torch.randn(4),
]

ARGS = {
    "snippets": snippets,
    "states": states,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
gen_l_hpu Ran
ran on a synthesized input
amolk/editsql/model/utils_bert.py
code served (permissive licence) · get_code("8a55b7ef1c467cea")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family int; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

# Analyzing the function: gen_l_hpu expects i_hds to be an iterable of tuples/sequences
# Each element in i_hds should be iterable (the for loop iterates over i_hds1)
# Each element in i_hds1 (i_hds11) should support subtraction (i_hds11[1] - i_hds11[0])
# This means i_hds11 should be indexable with [0] and [1], suggesting tuples of two numbers

# Based on the docstring example: i_hds = [(17, 18), (19, 21), (22, 23), (24, 25), (26, 29), (30, 34)]
# But the code iterates twice: for i_hds1 in i_hds, then for i_hds11 in i_hds1
# This means each element should itself be iterable of tuples

# The structure should be: list of lists of tuples, where each tuple has 2 elements
i_hds = [
    [(0, 5), (10, 13)],
    [(20, 22), (30, 35)],
    [(40, 42)]
]

ARGS = {
    'i_hds': i_hds
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family int — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
generate_inputs Ran
ran on a synthesized input
amolk/editsql/model/utils_bert.py
code served (permissive licence) · get_code("0ef533abd86a2b26")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

class SimpleTokenizer:
    def tokenize(self, text):
        """Tokenize a word into subword tokens."""
        # Simple tokenizer that splits words into character-level tokens
        return list(text)

tokenizer = SimpleTokenizer()

nlu1_tok = ["what", "is", "the"]

hds1 = ["name", "value", "type"]

ARGS = {
    "tokenizer": tokenizer,
    "nlu1_tok": nlu1_tok,
    "hds1": hds1
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_seq_from_scores Ran
ran on a synthesized input
amolk/editsql/model/torch_utils.py
code served (permissive licence) · get_code("03149eb9ca360505")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# Create mock scores (list of tensors)
# Each score is a tensor with shape (vocab_size,) representing logits/scores
scores = [
    torch.randn(4),  # First position: 4 possible tokens
    torch.randn(5),  # Second position: 5 possible tokens
    torch.randn(3),  # Third position: 3 possible tokens
]

# Create index_to_token_maps (list of lists of strings)
# Each inner list maps indices to token strings
index_to_token_maps = [
    ["token_a", "token_b", "token_c", "token_d"],  # Maps to first scores tensor
    ["token_e", "token_f", "token_g", "token_h", "token_i"],  # Maps to second scores tensor
    ["token_j", "token_k", "token_l"],  # Maps to third scores tensor
]

ARGS = {
    "scores": scores,
    "index_to_token_maps": index_to_token_maps,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
linear_layer Ran
ran on a synthesized input
amolk/editsql/model/torch_utils.py
code served (permissive licence) · get_code("4f3f1f5678f97059")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 2, dtype family float, shape (2, 3) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Create tensors with distinct dimensions for clarity
# exp: batch_size=2, input_dim=4
exp = torch.randn(2, 4)

# weights: input_dim=4, output_dim=3
weights = torch.randn(4, 3)

# biases: output_dim=3
biases = torch.randn(3)

ARGS = {
    "exp": exp,
    "weights": weights,
    "biases": biases,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 2, dtype family float, shape (2, 3) — shape (2, 3) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
score_query_tokens Ran
ran on a synthesized input
amolk/editsql/model/token_predictor.py
code served (permissive licence) · get_code("004b8e1db7db5c18")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Construct the arguments for score_query_tokens

# previous_query: a list of query tokens (can be strings, ints, or token ids)
previous_query = ["token1", "token2"]

# previous_query_states: a tensor of shape (num_tokens, state_dim)
# num_tokens must match len(previous_query) = 2
# state_dim = 4
previous_query_states = torch.randn(2, 4)

# scorer: a tensor that will be used in torch.mm(torch.t(scorer), previous_query_states)
# scorer needs to be shaped such that torch.t(scorer) @ previous_query_states works
# torch.t(scorer) has shape (state_dim, something)
# previous_query_states has shape (num_tokens, state_dim) = (2, 4)
# For matrix multiplication: (state_dim, something) @ (num_tokens, state_dim)
# This requires: something == num_tokens
# So scorer should be (num_tokens, state_dim) or we need scorer to be (something, state_dim)
# Let's make scorer shape (3, 4) so torch.t(scorer) is (4, 3)
# Then (4, 3) @ (2, 4) fails because 3 != 2
# Actually, looking at the code: torch.t(scorer) @ previous_query_states
# If scorer is (a, b), torch.t(scorer) is (b, a)
# previous_query_states is (2, 4)
# So (b, a) @ (2, 4) requires a == 2
# Result is (b, 4)
# Then torch.t of result is (4, b)
# And scores.size()[0] is 4, which is checked against len(previous_query) = 2
# Wait, that's wrong. Let me re-read.
# scores = torch.t(torch.mm(torch.t(scorer), previous_query_states))
# Let scorer be (a, b):
#   torch.t(scorer) = (b, a)
#   previous_query_states = (2, 4)
#   torch.mm((b, a), (2, 4)) requires a == 2, result is (b, 4)
#   torch.t((b, 4)) = (4, b)
# Then scores.size()[0] should equal len(previous_query) = 2
# So 4 should equal 2? That won't work.
# Let me reconsider: previous_query_states is (num_tokens, state_dim)
# If num_tokens = 2, state_dim = 4
# Let scorer = (state_dim, hidden) = (4, 3)
#   torch.t(scorer) = (3, 4)
#   torch.mm((3, 4), (2, 4)) - dimensions don't match (4 != 2)
# 
# Actually: torch.mm((b, a), (c, d)) requires a == c
# scorer shape (a, b), torch.t(scorer) = (b, a)
# previous_query_states = (num_tokens, state_dim)
# mm requires: a == num_tokens
# result: (b, state_dim)
# torch.t(result) = (state_dim, b)
# scores.size()[0] = state_dim, should equal len(previous_query) = num_tokens
# So state_dim == num_tokens
# Let both be 2:

previous_query = ["token1", "token2"]
previous_query_states = torch.randn(2, 2)  # (num_tokens=2, state_dim=2)
scorer = torch.randn(2, 3)  # (num_tokens=2, hidden=3)
# torch.t(scorer) = (3, 2)
# mm((3, 2), (2, 2)) = (3, 2)
# torch.t((3, 2)) = (2, 3)
# scores.size()[0] = 2, len(previous_query) = 2 ✓

ARGS = {
    "previous_query": previous_query,
    "previous_query_states": previous_query_states,
    "scorer": scorer
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
score_schema_tokens Ran
ran on a synthesized input
amolk/editsql/model/token_predictor.py
code served (permissive licence) · get_code("c3efa7d8a524d46d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Create a small stand-in class for input_schema
class InputSchema:
    def __init__(self, num_tokens):
        self.column_names_surface_form = "test_column"
        self._num_tokens = num_tokens
    
    def __len__(self):
        return self._num_tokens

# Define arguments
num_schema_tokens = 2
schema_state_dim = 4
scorer_dim = 3

# input_schema: needs len() and column_names_surface_form attribute
input_schema = InputSchema(num_schema_tokens)

# schema_states: shape (schema_state_dim, num_schema_tokens)
# After torch.t(scorer) @ schema_states, result should have shape (num_schema_tokens,)
# torch.t(scorer) has shape (num_schema_tokens, schema_state_dim) if scorer is (schema_state_dim, num_schema_tokens)
# So schema_states should be (schema_state_dim, num_schema_tokens)
schema_states = torch.randn(schema_state_dim, num_schema_tokens)

# scorer: shape (schema_state_dim, num_schema_tokens)
# torch.t(scorer) @ schema_states gives shape (num_schema_tokens, num_schema_tokens)
# Then torch.t() of that gives (num_schema_tokens, num_schema_tokens)
# We want final scores to have shape (num_schema_tokens,) or (num_schema_tokens, something)
# The check is scores.size()[0] == len(input_schema) == num_schema_tokens
# So scorer should be (schema_state_dim, num_schema_tokens)
scorer = torch.randn(schema_state_dim, num_schema_tokens)

ARGS = {
    "input_schema": input_schema,
    "schema_states": schema_states,
    "scorer": scorer
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
flatten_distribution Not yet run
unverified (harvested, no recorded run)
amolk/editsql/model/decoder.py
code served (permissive licence) · get_code("c59f3f02b15d3770")
How it ran

no invocation recorded.

get_bert Not yet run
unverified (harvested, no recorded run)
amolk/editsql/model/utils_bert.py
code served (permissive licence) · get_code("9ac04740fba8c9cf")
How it ran

no invocation recorded.

score_snippets Not yet run
unverified (harvested, no recorded run)
amolk/editsql/model/token_predictor.py
code served (permissive licence) · get_code("26a5daacb4d2f144")
How it ran

no invocation recorded.

Also present in other papers' code

Of 11 samples lifted from this paper's linked repositories, 11 carry a function identity and 0 do not. Of the 11 distinct functions with an identity, 2 are also present in other papers whose code we hold (2 with at least one paper dated earlier, 0 the same month, 0 later) and 9 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
encode_snippets_with_states 1 other paper
1 dated earlier
1906.02285 (2019-06)
SParC: Cross-Domain Semantic Parsing in Context
flatten_distribution 1 other paper
1 dated earlier
1804.06868 (2018-04)
Learning to Map Context-Dependent Sentences to Executable Formal Queries

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

We present CoSQL, a corpus for building cross-domain, general-purpose database (DB) querying dialogue systems. It consists of 30k+ turns plus 10k+ annotated SQL queries, obtained from a Wizard-of-Oz (WOZ) collection of 3k dialogues querying 200 complex DBs spanning 138 domains. Each dialogue simulates a real-world DB query scenario with a crowd worker as a user exploring the DB and a SQL expert retrieving answers with SQL, clarifying ambiguous questions, or otherwise informing of unanswerable questions. When user questions are answerable by SQL, the expert describes the SQL and execution results to the user, hence maintaining a natural interaction flow. CoSQL introduces new challenges compared to existing task-oriented dialogue datasets:(1) the dialogue states are grounded in SQL, a domain-independent executable representation, instead of domain-specific slot-value pairs, and (2) because testing is done on unseen databases, success requires generalizing to new domains. CoSQL includes three tasks: SQL-grounded dialogue state tracking, response generation from query results, and user dialogue act prediction. We evaluate a set of strong baselines for each task and show that CoSQL presents significant challenges for future research. The dataset, baselines, and leaderboard will be released at https://yale-lily.github.io/cosql.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("1909.05378")
get_code_for_paper("1909.05378")
have("1909.05378")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/1909.05378.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/1909.05378.svg)](https://syntology.ai/paper/1909.05378)

Connect an agent — have() is free.