SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2205.06262 · EMNLP · 2022

FETA: A Benchmark for Few-Sample Task Transfer in Open-Domain Dialogue

William Wang, Yi-Lin Tuan, Connor Pryor, Lise Getoor, Alon Albalak, Deepak Ramachandran, Jay Pujara, Pegah Jandaghi, Luke Yoffe

arXiv · PDF · Open in the Atlas

Code that ran

9 of 9 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 0 honoured the contract we drafted, 0 violated it and 9 have no contract verdict; 0 failures of our instrument; 0 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2205.06262.json · badge: /api/ran/2205.06262.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
alon-albalak/tlidb canonical (classifier label)
classifier basis not recorded on the link
9 ran · 0 no recorded run
FunctionStatusWhere it lives
get_DD_subset_from_DDpp_context Ran
ran on a synthesized input
alon-albalak/tlidb/dataset_preprocessing/DailyDialog/add_DDpp_annotations.py
code served (permissive licence) · get_code("6513af62bcecc6c7")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Construct DDpp_context - a string or collection of strings to search within
DDpp_context = "hello world this is a test string with some content for matching purposes"

# Construct DD_datum - a dict with 'dialogue' key containing a list of turn dicts
DD_datum = {
    'dialogue': [
        {
            'utterance': 'hello world this is the first turn in the conversation'
        },
        {
            'utterance': 'this is another turn that does not match well'
        },
        {
            'utterance': 'yet another message for testing purposes here'
        },
        {
            'utterance': 'matching purposes and some content for the final turn'
        }
    ]
}

ARGS = {
    'DDpp_context': DDpp_context,
    'DD_datum': DD_datum
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_DD_turn_subset_from_DDpp_start Ran
ran on a synthesized input
alon-albalak/tlidb/dataset_preprocessing/DailyDialog/add_DDpp_annotations.py
code served (permissive licence) · get_code("5f11d3669eee492b")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family int; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

# Create a minimal DD_datum structure that matches what the function expects
DD_datum = {
    'dialogue': [
        {
            'turn_id': 0,
            'utterance': "Hello, how are you today?"
        },
        {
            'turn_id': 1,
            'utterance': "I'm doing well, thank you for asking."
        },
        {
            'turn_id': 2,
            'utterance': "That's great to hear. Let's discuss something."
        },
        {
            'turn_id': 3,
            'utterance': "Sure, I'd be happy to talk about it."
        },
        {
            'turn_id': 4,
            'utterance': "What do you think about this topic?"
        }
    ]
}

# Create DDpp_start string that will match part of an utterance in the dialogue
# Using the first 20 characters of "Hello, how are you today?" which is "hello, how are you "
DDpp_start = "Hello, how are you today?"

# Number of turns to retrieve
DDpp_len = 3

ARGS = {
    'DDpp_start': DDpp_start,
    'DDpp_len': DDpp_len,
    'DD_datum': DD_datum
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family int — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_dataloader Ran
ran on a synthesized input
alon-albalak/tlidb/tlidb/TLiDB/data_loaders/data_loaders.py
code served (permissive licence) · get_code("e74b4dc394b98cfc")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataLoader. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from torch.utils.data import Dataset

torch.manual_seed(0)

class TLiDBDataset(Dataset):
    def __init__(self, task_type='classification', num_choices=4):
        self.task_metadata = {
            'type': task_type,
            'num_choices': num_choices
        }
        self.data = [(torch.randn(4), torch.randint(0, 2, (1,))) for _ in range(8)]
    
    def __len__(self):
        return len(self.data)
    
    def __getitem__(self, idx):
        return self.data[idx]

dataset = TLiDBDataset(task_type='classification', num_choices=4)

ARGS = {
    'split': 'train',
    'dataset': dataset,
    'batch_size': 2,
    'model_type': 'Encoder',
    'num_workers': 0,
    'pin_memory': False
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type DataLoader — a DataLoader output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataLoader output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_eval_loader Ran
ran on a synthesized input
alon-albalak/tlidb/tlidb/TLiDB/data_loaders/data_loaders.py
code served (permissive licence) · get_code("1d86103d939c1b2c")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataLoader. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader

torch.manual_seed(0)

# Create a minimal dataset class that mimics TLiDBDataset
class MinimalDataset(Dataset):
    def __init__(self, task_metadata=None):
        self.task_metadata = task_metadata or {
            'type': 'classification',
            'num_choices': 1
        }
        self.data = [(torch.randn(4), torch.tensor(0)) for _ in range(8)]
    
    def __len__(self):
        return len(self.data)
    
    def __getitem__(self, idx):
        return self.data[idx]

# Create a minimal model class
class MinimalEncoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(4, 2)
    
    def forward(self, x):
        return self.linear(x)

# Build the arguments
dataset = MinimalDataset(task_metadata={
    'type': 'classification',
    'num_choices': 1
})

batch_size = 2

model_type = 'Encoder'

loader_kwargs = {
    'num_workers': 0,
    'pin_memory': False
}

ARGS = {
    'dataset': dataset,
    'batch_size': batch_size,
    'model_type': model_type,
    **loader_kwargs
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type DataLoader — a DataLoader output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataLoader output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_train_loader Ran
ran on a synthesized input
alon-albalak/tlidb/tlidb/TLiDB/data_loaders/data_loaders.py
code served (permissive licence) · get_code("ed08ccd15847eb99")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataLoader. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
from torch.utils.data import Dataset

torch.manual_seed(0)

class TLiDBDataset(Dataset):
    def __init__(self, task_type, num_choices=None):
        self.task_metadata = {
            'type': task_type,
        }
        if num_choices is not None:
            self.task_metadata['num_choices'] = num_choices
        self.data = [(torch.randn(4), torch.tensor(1)) for _ in range(8)]
    
    def __len__(self):
        return len(self.data)
    
    def __getitem__(self, idx):
        return self.data[idx]

dataset = TLiDBDataset(task_type='multiple_choice', num_choices=4)
batch_size = 8
model_type = 'Encoder'
loader_kwargs = {
    'num_workers': 2,
    'drop_last': False
}

ARGS = {
    'dataset': dataset,
    'batch_size': batch_size,
    'model_type': model_type,
    **loader_kwargs
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type DataLoader — a DataLoader output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataLoader output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_dataset Ran
ran on a synthesized input
alon-albalak/tlidb/tlidb/TLiDB/datasets/TLiDB_dataset.py
code served (permissive licence) · get_code("56f5d5806ea068f8")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import json
import tempfile
from pathlib import Path

# Create a temporary directory for testing
temp_dir = tempfile.mkdtemp()

# Create the dataset structure that load_dataset_local expects
dataset_name = "test_dataset"
dataset_subfolder = os.path.join(temp_dir, f"TLiDB_{dataset_name}")
os.makedirs(dataset_subfolder, exist_ok=True)

# Create a sample JSON file in the dataset folder
sample_data = {
    "key1": "value1",
    "key2": [1, 2, 3],
    "key3": {"nested": "data"}
}

json_file_path = os.path.join(dataset_subfolder, "data.json")
with open(json_file_path, 'w') as f:
    json.dump(sample_data, f)

# Define ARGS for the load_dataset function
ARGS = {
    "name": dataset_name,
    "dataset_folder": temp_dir,
    "url": None  # url is optional when the dataset already exists locally
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_dataset_local Ran
ran on a synthesized input
alon-albalak/tlidb/tlidb/TLiDB/datasets/TLiDB_dataset.py
code served (permissive licence) · get_code("ad8920c648d3320a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import json
import tempfile
import shutil

# Create a temporary directory structure with test JSON files
temp_dir = tempfile.mkdtemp()
dataset_name = "test_dataset"
dataset_path = os.path.join(temp_dir, f"TLiDB_{dataset_name}")
os.makedirs(dataset_path)

# Create test JSON files
test_data_1 = {"key1": "value1", "key2": "value2"}
test_data_2 = {"key3": "value3", "key4": "value4"}

with open(os.path.join(dataset_path, "file1.json"), "w") as f:
    json.dump(test_data_1, f)

with open(os.path.join(dataset_path, "file2.json"), "w") as f:
    json.dump(test_data_2, f)

# Also create sample_format.json which should be skipped
with open(os.path.join(dataset_path, "sample_format.json"), "w") as f:
    json.dump({"sample": "data"}, f)

ARGS = {
    "name": "test_dataset",
    "dataset_folder": temp_dir
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_split_ids Ran
ran on a synthesized input
alon-albalak/tlidb/tlidb/TLiDB/datasets/TLiDB_dataset.py
code served (permissive licence) · get_code("43a3c2167ee18528")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import tempfile
import shutil

# Create a temporary directory structure for the dataset
temp_dir = tempfile.mkdtemp()
dataset_name = "test_dataset"
dataset_folder = temp_dir

# Create the dataset folder structure
dataset_path = os.path.join(dataset_folder, f'TLiDB_{dataset_name}')
os.makedirs(dataset_path, exist_ok=True)

# Create sample ID files
split_types = ['train', 'val', 'test']
for split in split_types:
    # Create regular split file
    ids_file = os.path.join(dataset_path, f'TTiDB_{split}_ids.txt')
    with open(ids_file, 'w') as f:
        f.write('\n'.join([f'id_{i}' for i in range(10)]))
    
    # Create few-shot split files
    for percent in [5, 10]:
        few_shot_file = os.path.join(dataset_path, f'TTiDB_{percent}_percent_few_shot_{split}_ids.txt')
        with open(few_shot_file, 'w') as f:
            f.write('\n'.join([f'few_shot_id_{i}' for i in range(5)]))

ARGS = {
    'name': 'test_dataset',
    'dataset_folder': dataset_folder,
    'split': 'train',
    'few_shot_percent': None
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
numel Ran
ran on a synthesized input
alon-albalak/tlidb/tlidb/TLiDB/utils/utils.py
code served (permissive licence) · get_code("cbf7ee125a81ce54")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# The function 'numel' takes a single parameter 'obj' and:
# - If it's a tensor, returns obj.numel()
# - If it's a list, returns len(obj)
# - Otherwise raises TypeError

# Test with a tensor
tensor_obj = torch.randn(2, 4)

# Alternative: test with a list
list_obj = [1, 2, 3, 4, 5]

# Using tensor as the primary test case
ARGS = {
    'obj': tensor_obj
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block

Also present in other papers' code

Of 9 samples lifted from this paper's linked repositories, 9 carry a function identity and 0 do not. Of the 9 distinct functions with an identity, 0 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 0 later) and 9 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Task transfer, transferring knowledge contained in related tasks, holds the promise of reducing the quantity of labeled data required to finetune language models. Dialogue understanding encompasses many diverse tasks, yet task transfer has not been thoroughly studied in conversational AI. This work explores conversational task transfer by introducing FETA: a benchmark for FEw-sample TAsk transfer in open-domain dialogue. FETA contains two underlying sets of conversations upon which there are 10 and 7 tasks annotated, enabling the study of intra-dataset task transfer; task transfer without domain adaptation. We utilize three popular language models and three learning algorithms to analyze the transferability between 132 source-target task pairs and create a baseline for future work. We run experiments in the single-and multi-source settings and report valuable findings, e.g., most performance trends are model-specific, and span extraction and multiple-choice tasks benefit the most from task transfer. In addition to task transfer, FETA can be a valuable resource for future research into the efficiency and generalizability of pre-training datasets and model architectures, as well as for learning settings such as continual and multitask learning.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2205.06262")
get_code_for_paper("2205.06262")
have("2205.06262")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2205.06262.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2205.06262.svg)](https://syntology.ai/paper/2205.06262)

Connect an agent — have() is free.