| PUMS_to_model_generated |
Ran ran on a synthesized input |
socialfoundations/surveying-language-models/surveying_llms/process_acs.py code served (permissive licence) · get_code("0a582fdbf690b3ab") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataFrame. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
import pandas as pd
# Create a minimal PUMS dataset with the required columns
var_of_interest = ['SEX', 'AGEP', 'HISP', 'RAC1P', 'NATIVITY', 'CIT', 'SCH', 'SCHL', 'LANX', 'ENG', 'HICOV', 'DEAR', 'DEYE', 'MAR', 'FER', 'GCL', 'MIL', 'WRK', 'ESR', 'JWTRNS', 'WKL', 'WKWN', 'WKHP', 'COW', 'PINCP', 'ST', 'PWGTP']
# Create a small sample dataset with realistic values
np.random.seed(0)
n_rows = 10
dataset_dict = {
'SEX': np.random.randint(1, 3, n_rows),
'AGEP': np.random.randint(0, 100, n_rows),
'HISP': np.random.randint(1, 25, n_rows),
'RAC1P': np.random.randint(1, 10, n_rows),
'NATIVITY': np.random.randint(1, 3, n_rows),
'CIT': np.random.randint(1, 6, n_rows),
'SCH': np.random.randint(1, 3, n_rows),
'SCHL': np.random.randint(1, 25, n_rows),
'LANX': np.random.randint(1, 3, n_rows),
'ENG': np.random.randint(1, 5, n_rows),
'HICOV': np.random.randint(1, 3, n_rows),
'DEAR': np.random.randint(1, 3, n_rows),
'DEYE': np.random.randint(1, 3, n_rows),
'MAR': np.random.randint(1, 6, n_rows),
'FER': np.random.randint(0, 2, n_rows),
'GCL': np.random.randint(1, 3, n_rows),
'MIL': np.random.randint(1, 3, n_rows),
'WRK': np.random.randint(0, 2, n_rows),
'ESR': np.random.randint(1, 7, n_rows),
'JWTRNS': np.random.randint(1, 12, n_rows),
'WKL': np.random.randint(1, 3, n_rows),
'WKWN': np.random.randint(1, 53, n_rows),
'WKHP': np.random.randint(0, 100, n_rows),
'COW': np.random.randint(1, 10, n_rows),
'PINCP': np.random.randint(-10000, 300000, n_rows),
'ST': np.random.randint(1, 57, n_rows),
'PWGTP': np.random.randint(1, 100, n_rows),
}
dataset = pd.DataFrame(dataset_dict)
# Introduce some NaN values to make it more realistic
dataset.loc[0, 'MIL'] = np.nan
dataset.loc[1, 'ENG'] = np.nan
ARGS = {
'dataset': dataset
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, pandas — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type DataFrame — a DataFrame output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataFrame output the recorded ndim 2 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| col2freq |
Ran ran on a synthesized input |
socialfoundations/surveying-language-models/surveying_llms/load_responses.py code served (permissive licence) · get_code("c1a982017d824fbb") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (3,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import pandas as pd
import numpy as np
# Create a small pandas Series with categorical data (choices 1-3)
col = pd.Series([1, 2, 1, 3, 2, 1], name='category')
# Define parameters
n_cats = 3
assign_nan = None
normalize = True
weight = None
# Construct ARGS dictionary
ARGS = {
'col': col,
'n_cats': n_cats,
'assign_nan': assign_nan,
'normalize': normalize,
'weight': weight
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, pandas — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 1, dtype family float, shape (3,) — shape (3,) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_choice_logprobs |
Ran ran on a synthesized input |
socialfoundations/surveying-language-models/surveying_llms/fill_openai.py code served (permissive licence) · get_code("a1e7e5128d3218a0") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Analyze the function:
# - top_tokens: used with .index(option) and "in" operator, so it's a list of strings
# - top_logprobs: indexed by integer from top_tokens.index(), so it's a list of numbers
# - n_options: used in range(n_options), so it's an int
# - client: checked with "is not None" to determine option format
# Create test data
top_tokens = [' A', ' B', ' C', ' X'] # List of token strings, some matching options
top_logprobs = [-0.5, -1.2, -2.3, -3.1] # Corresponding log probabilities
n_options = 3 # We want options ' A', ' B', ' C'
client = None # When None, options are prefixed with space
ARGS = {
'top_tokens': top_tokens,
'top_logprobs': top_logprobs,
'n_options': n_options,
'client': client
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_openai_logprobs |
Ran ran on a synthesized input |
socialfoundations/surveying-language-models/surveying_llms/fill_openai.py code served (permissive licence) · get_code("e2560a0c5cd234dc") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 2, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import openai
from unittest.mock import Mock, MagicMock
# Create a mock client that simulates the OpenAI API response
mock_client = Mock()
# Create mock logprob objects
mock_logprob_1 = Mock()
mock_logprob_1.token = "A"
mock_logprob_1.logprob = -0.5
mock_logprob_2 = Mock()
mock_logprob_2.token = "B"
mock_logprob_2.logprob = -1.2
mock_logprob_3 = Mock()
mock_logprob_3.token = "C"
mock_logprob_3.logprob = -2.1
# Create mock completion response structure
mock_completion = Mock()
mock_choice = Mock()
mock_logprobs_obj = Mock()
mock_content_logprobs = Mock()
mock_content_logprobs.top_logprobs = [mock_logprob_1, mock_logprob_2, mock_logprob_3]
mock_logprobs_obj.content = [mock_content_logprobs]
mock_choice.logprobs = mock_logprobs_obj
mock_completion.choices = [mock_choice]
mock_client.chat.completions.create.return_value = mock_completion
ARGS = {
"model": "gpt-3.5-turbo",
"prompt": "What is the first letter of the alphabet?",
"client": mock_client
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- openai — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 2, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| move_tmp |
Ran ran on a synthesized input |
socialfoundations/surveying-language-models/surveying_llms/utils.py code served (permissive licence) · get_code("62fb9e3940a61e1c") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import os
import tempfile
import shutil
# Create a temporary directory structure to use as model_dir
temp_base = tempfile.mkdtemp()
model_dir = os.path.join(temp_base, 'test_model')
os.makedirs(model_dir, exist_ok=True)
# Create some dummy files in the model directory to simulate a model
with open(os.path.join(model_dir, 'model.pt'), 'w') as f:
f.write('dummy model file')
with open(os.path.join(model_dir, 'config.json'), 'w') as f:
f.write('{"dummy": "config"}')
ARGS = {
'model_dir': model_dir
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| openai_upper_bound |
Ran ran on a synthesized input |
socialfoundations/surveying-language-models/surveying_llms/load_responses.py code served (permissive licence) · get_code("ec0ed2a7a17e301d") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (6,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Construct valid arguments for openai_upper_bound
# The function takes:
# - logp: array of log probabilities (with -inf for unseen entries)
# - psum: sum of seen probabilities
# - minlogp: minimum log probability for unseen entries
# Create a small example with 6 vocabulary items
# Some entries are seen (have log probs), others are not (-inf)
logp = np.array([
np.log(0.5), # seen
np.log(0.3), # seen
-np.inf, # not seen
-np.inf, # not seen
np.log(0.1), # seen
-np.inf # not seen
])
# psum is the sum of probabilities for seen entries
psum = 0.5 + 0.3 + 0.1 # = 0.9
# minlogp is the minimum log probability threshold for unseen entries
minlogp = np.log(0.01)
ARGS = {
"logp": logp,
"psum": psum,
"minlogp": minlogp
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 1, dtype family float, shape (6,) — shape (6,) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| process_naive |
Ran ran on a synthesized input |
socialfoundations/surveying-language-models/surveying_llms/load_responses.py code served (permissive licence) · get_code("e53485caa3cf5ad4") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
import pandas as pd
def openai_upper_bound(logp, psum, minlogp):
""" Since the OpenAI API only gives top-k logits, fill the missing entries with an upper bound """
not_seen = logp == -np.inf
n_not_seen = not_seen.sum()
p = np.exp(logp)
if n_not_seen > 0:
pleft = (1.0 - psum) / n_not_seen
pmin = min(pleft, np.exp(minlogp))
p[not_seen] = pmin
return p / p.sum()
def process_naive(dataset, n_categories, isopenai=False):
"""
Inputs: pd.DataFrame output by naively filling the form
Returns: dictionary, (survey question: response distribution)
"""
distributions = {}
for _, row in dataset.iterrows():
var = row['var']
if var not in n_categories:
continue
p = np.array([row[str(i + 1)] for i in range(n_categories[var])])
if isopenai:
p = openai_upper_bound(p, row['sp'], row['mlogp'])
distributions[var] = p / p.sum()
return distributions
# Create a small pandas DataFrame that matches the expected structure
data = {
'var': ['question_1', 'question_2', 'question_3'],
'1': [0.2, -np.inf, 0.1],
'2': [0.5, 0.3, -np.inf],
'3': [0.1, 0.4, 0.2],
'sp': [0.8, 0.7, 0.5],
'mlogp': [-2.0, -1.5, -2.5]
}
dataset = pd.DataFrame(data)
# n_categories maps variable names to number of categories
n_categories = {
'question_1': 3,
'question_2': 3,
'question_3': 3
}
ARGS = {
'dataset': dataset,
'n_categories': n_categories,
'isopenai': False
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, pandas — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| query_model_batch |
Ran ran on a synthesized input |
socialfoundations/surveying-language-models/surveying_llms/fill.py code served (permissive licence) · get_code("7cd6faa6ecc2e154") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 2, dtype family float, shape (2, 100) as a later run recorded it. The image it ran in was not recorded. The code mentions cuda, and the harness aliases cuda to the CPU for every sample; whether this run needed that is not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
import numpy as np
torch.manual_seed(0)
# Stand-in tokenizer class
class MockTokenizer:
def __init__(self, vocab_size=100, pad_token_id=0):
self.vocab_size = vocab_size
self.pad_token_id = pad_token_id
def encode(self, text, return_tensors='pt'):
# Return a simple encoded tensor based on text length
tokens = torch.tensor([1, 2, 3, 4, 5][:len(text)], dtype=torch.long)
if return_tensors == 'pt':
return tokens.unsqueeze(0)
return tokens
# Minimal model that mimics transformer behavior
class MockModel(nn.Module):
def __init__(self, vocab_size=100, hidden_size=8):
super().__init__()
self.vocab_size = vocab_size
self.hidden_size = hidden_size
self.embedding = nn.Embedding(vocab_size, hidden_size, padding_idx=0)
self.linear = nn.Linear(hidden_size, vocab_size)
def forward(self, input_ids, attention_mask=None):
# input_ids shape: (batch_size, seq_len)
embeddings = self.embedding(input_ids) # (batch_size, seq_len, hidden_size)
logits = self.linear(embeddings) # (batch_size, seq_len, vocab_size)
# Return object with logits attribute
class Output:
pass
output = Output()
output.logits = logits
return output
# Create instances
tokenizer = MockTokenizer(vocab_size=100, pad_token_id=0)
model = MockModel(vocab_size=100, hidden_size=8)
text_inputs = ["hello world", "test"]
context_size = 10
ARGS = {
"text_inputs": text_inputs,
"tokenizer": tokenizer,
"model": model,
"context_size": context_size,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 2, dtype family float, shape (2, 100) — shape (2, 100) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_tokenizer_model |
Not yet run unverified (harvested, no recorded run) |
socialfoundations/surveying-language-models/surveying_llms/utils.py code served (permissive licence) · get_code("093daf2e93f4d661") How it ran |