SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2202.00449 · ICML · 2022

A Consistent and Efficient Evaluation Strategy for Attribution Methods

Gjergji Kasneci, Yao Rong, Tobias Leemann, Vadim Borisov, Enkelejda Kasneci, Rank, Morf

arXiv · PDF · Open in the Atlas

Code that ran

10 of 16 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (7 of them constructed an object rather than computing a result; 1 honoured the contract we drafted, 1 violated it and 8 have no contract verdict; 0 failures of our instrument; 6 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2202.00449.json · badge: /api/ran/2202.00449.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
tleemann/road_evaluation canonical (classifier label)
classifier basis: classifier_2025
10 ran · 6 no recorded run
FunctionStatusWhere it lives
ChannelMeanImputer Ran
ran on a synthesized input · constructed an object; no result was computed
tleemann/road_evaluation/road/road.py
code served (permissive licence) · get_code("8d39eb0a1bab6a0a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, ChannelMeanImputer: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Create sample tensors for testing
# img: (C, H, W) tensor
C, H, W = 3, 4, 5
img = torch.randn(C, H, W)

# mask: (H, W) tensor with binary values (0 or 1)
mask = torch.randint(0, 2, (H, W)).float()

ARGS = {
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ChannelMeanImputer — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a ChannelMeanImputer output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
ConvBlock_ResNet Ran
ran on a synthesized input · constructed an object; no result was computed
tleemann/road_evaluation/road/road.py
code served (permissive licence) · get_code("a5a2de6a98b65ad1")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, ConvBlock_ResNet: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

ARGS = {
    'n_channels': 4
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ConvBlock_ResNet — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a ConvBlock_ResNet output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
GNet_ResNet Ran
ran on a synthesized input · constructed an object; no result was computed
tleemann/road_evaluation/road/road.py
code served (permissive licence) · get_code("5e433e65a3f46c52")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, GNet_ResNet: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

ARGS = {
    'in_channels': 3,
    'n_downsampling': 2,
    'n_blocks': 4,
    'attention': False
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type GNet_ResNet — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a GNet_ResNet output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
ImputedDataset Ran
ran on a synthesized input · constructed an object; no result was computed
tleemann/road_evaluation/road/road.py
code served (permissive licence) · get_code("a75a12951c70b941")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, ImputedDataset: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np
from torch.utils.data import Dataset

torch.manual_seed(0)
np.random.seed(0)

# Create a minimal base dataset
class MinimalBaseDataset(Dataset):
    def __init__(self, num_samples=2):
        self.num_samples = num_samples
    
    def __len__(self):
        return self.num_samples
    
    def __getitem__(self, index):
        # Return (image, label) tuple
        # Image: (C, H, W) = (3, 4, 5)
        img = torch.randn(3, 4, 5)
        label = torch.tensor(index % 2)
        return img, label

# Create a minimal mask dataset
class MinimalMaskDataset(Dataset):
    def __init__(self, num_samples=2):
        self.num_samples = num_samples
    
    def __len__(self):
        return self.num_samples
    
    def __getitem__(self, index):
        # Return binary mask (H, W) matching image dimensions
        mask = torch.randint(0, 2, (4, 5)).float()
        return mask

# Alternatively, mask can be a list
def create_mask_list(num_samples=2):
    return [torch.randint(0, 2, (4, 5)).float() for _ in range(num_samples)]

# Create the base imputation module
class ChannelMeanImputer:
    def __call__(self, img, mask):
        for c in range(img.shape[0]):
            mean_c = img[c, :, :].mean()
            img[c, :, :][mask == 0] = mean_c
        return img
    
    def batched_call(self, img, mask):
        channel_mean_tensor = img.view(img.shape[0], img.shape[1], -1).mean(axis=2)
        c_shape = channel_mean_tensor.shape
        channel_mean_tensor = channel_mean_tensor.unsqueeze(2).unsqueeze(3).expand(c_shape[0], c_shape[1], img.shape[2], img.shape[3])
        return channel_mean_tensor * (1.0 - mask.unsqueeze(1)) + img * mask.unsqueeze(1)

# Create base dataset and mask
base_dataset = MinimalBaseDataset(num_samples=2)
mask_list = create_mask_list(num_samples=2)

# Create predictions if needed
predictions = [torch.tensor([0.8, 0.2]), torch.tensor([0.3, 0.7])]

# Define transforms (optional, can be None or a simple callable)
def simple_transform(img):
    return img

def simple_target_transform(target):
    return target

ARGS = {
    "base_dataset": base_dataset,
    "mask": mask_list,
    "th_p": 0.5,
    "remove": True,
    "imputation": ChannelMeanImputer(),
    "transform": simple_transform,
    "target_transform": simple_target_transform,
    "prediction": predictions,
    "use_cache": False,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ImputedDataset — the run constructed an instance of its own class and did not call it; no shape is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
ImputedDatasetMasksOnly Ran
ran on a synthesized input · constructed an object; no result was computed
tleemann/road_evaluation/road/road.py
code served (permissive licence) · get_code("11860240064c7dd0")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, ImputedDatasetMasksOnly: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)
np.random.seed(0)

# Create a minimal base dataset that returns (image, label) tuples
class MinimalBaseDataset(torch.utils.data.Dataset):
    def __init__(self, num_samples=2):
        self.num_samples = num_samples
    
    def __getitem__(self, index):
        # Return (image, label) where image is 3x4x5 (channels x height x width)
        img = torch.randn(3, 4, 5)
        label = torch.tensor(index % 2)
        return (img, label)
    
    def __len__(self):
        return self.num_samples

# Create a minimal mask dataset
class MinimalMaskDataset:
    def __init__(self, num_samples=2):
        self.num_samples = num_samples
    
    def __getitem__(self, index):
        # Return explanation map with shape (4, 5, 1) to match base_dataset image spatial dims
        # rescale_channel expects axis=-1 to sum over
        mask = np.random.rand(4, 5, 1).astype(np.float32)
        return mask
    
    def __len__(self):
        return self.num_samples

# Instantiate base components
base_dataset = MinimalBaseDataset(num_samples=2)
mask_dataset = MinimalMaskDataset(num_samples=2)

# Build ARGS dictionary with all required parameters
ARGS = {
    'base_dataset': base_dataset,
    'mask': mask_dataset,
    'th_p': 0.5,
    'remove': True,
    'prediction': [0, 1],
    'use_cache': False
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ImputedDatasetMasksOnly — the run constructed an instance of its own class and did not call it; no shape is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
ImputingDataLoaderWrapper Ran
ran on a synthesized input · constructed an object; no result was computed
tleemann/road_evaluation/road/road.py
code served (permissive licence) · get_code("47503ad265daf120")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, ImputingDataLoaderWrapper: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from torch.utils.data import DataLoader

torch.manual_seed(0)

class SimpleImputer:
    def batched_call(self, img: torch.Tensor, mask: torch.Tensor) -> torch.Tensor:
        # Simple imputation: just return the image as is
        return img

class SimpleDataset:
    def __init__(self):
        self.data = [
            (torch.randn(3, 4, 5), torch.tensor([0]), torch.tensor([1]), torch.ones(4, 5)),
            (torch.randn(3, 4, 5), torch.tensor([1]), torch.tensor([0]), torch.ones(4, 5)),
        ]
    
    def __len__(self):
        return len(self.data)
    
    def __getitem__(self, idx):
        return self.data[idx]

def simple_image_transform(x):
    return x * 0.5

def simple_target_transform(x):
    return x + 1

dataset = SimpleDataset()
base_loader = DataLoader(dataset, batch_size=2)
imputer = SimpleImputer()
image_transform = simple_image_transform
target_transform = simple_target_transform

ARGS = {
    "org_data_loader": base_loader,
    "imputer": imputer,
    "image_transform": image_transform,
    "target_transform": target_transform,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ImputingDataLoaderWrapper — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a ImputingDataLoaderWrapper output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
NoisyLinearImputer Ran
ran on a synthesized input · constructed an object; no result was computed
tleemann/road_evaluation/road/road.py
code served (permissive licence) · get_code("629765eba65036f5")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, NoisyLinearImputer: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# Create a small image tensor (C, H, W)
img = torch.randn(3, 4, 5)

# Create a small mask tensor (H, W) with binary values
# 1 indicates present pixels, 0 indicates absent pixels
mask = torch.ones(4, 5)
# Mark some pixels as absent
mask[1, 2] = 0
mask[2, 3] = 0
mask[3, 1] = 0

# Define the neighbors weights (this is already defined in the function)
neighbors_weights = [
    ((1, 1), 1 / 12), 
    ((0, 1), 1 / 6), 
    ((-1, 1), 1 / 12), 
    ((1, -1), 1 / 12), 
    ((0, -1), 1 / 6), 
    ((-1, -1), 1 / 12), 
    ((1, 0), 1 / 6), 
    ((-1, 0), 1 / 6)
]

ARGS = {
    'noise': 0.01,
    'weighting': neighbors_weights
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, scipy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type NoisyLinearImputer — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a NoisyLinearImputer output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
ZeroImputer Ran
ran on a synthesized input
tleemann/road_evaluation/road/road.py
code served (permissive licence) · get_code("428d991a1dcea84e")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, class construct-then-call prompt, model not recorded) and returned Tensor, ndim 3, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

class BaseImputer:

    def __call__(self, img: torch.Tensor, mask: torch.Tensor) -> torch.Tensor:
        """ Call the Imputation function to fill the masked pixels in an image.
            :param img: original image (C,H,W)-tensor
            :param mask: (H,W)-tensor with a binary mask. 0 indicates pixels absent, 1 indicates pixels present.
            :returns: a (C,H,W) tensor, where the original values are kept, if the mask for the pixels is 1 or imputed otherwise.
            The return tensor is copied to cpu()
        """
        raise NotImplementedError('Please implement an imputation function or use an existing imputor.')

    def batched_call(self, img: torch.Tensor, mask: torch.Tensor) -> torch.Tensor:
        """ Call the Imputation function to fill the masked pixels in an image. However, in this version,
            and entire batch of images will be processed, which can results in considerable speedup.
            :param img: B original images (B, C, H, W)-tensor
            :param mask: (B, H, W)-tensor with binary masks. 0 indicates pixels absent, 1 indicates pixels present.
            :returns: a (B, C, H, W) tensor, where the original values are kept, if the mask for the pixels is 1 or imputed otherwise.
            The returned tensor is left on the device that this dataloader is instructed to use (may not be CPU).
        """
        raise NotImplementedError('This imputer does not support the bached interface.')

class ZeroImputer(BaseImputer):

    def __call__(self, img: torch.Tensor, mask: torch.Tensor):
        return img * mask.unsqueeze(0)

    def batched_call(self, img: torch.Tensor, mask: torch.Tensor):
        assert img.device == mask.device
        return img * mask.unsqueeze(1)


INIT_ARGS = {}

ARGS = {
    'img': torch.randn(3, 4, 5),
    'mask': torch.randint(0, 2, (4, 5)).float()
}

METHOD = '__call__'

if __name__ == '__main__':
    imputer = ZeroImputer(**INIT_ARGS)
    result = imputer(**ARGS)
    print(f"Result shape: {result.shape}")
    print(f"Result:\n{result}")
    
    batched_img = torch.randn(2, 3, 4, 5)
    batched_mask = torch.randint(0, 2, (2, 4, 5)).float()
    batched_result = imputer.batched_call(batched_img, batched_mask)
    print(f"Batched result shape: {batched_result.shape}")
    print(f"Batched result:\n{batched_result}")
Setup block drafted by
a language model, class construct-then-call prompt, model not recorded
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 3, dtype family float — no output shape is stated: observe_signature.py observed this class with its constructor sizes canonicalised where it could, which the node does not record, so its shape is not stated as this setup block's
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: torch and numpy seeded to 0 by the harness immediately before the class was constructed from INIT_ARGS (fixture_check.py)
normalize_map Ran
ran, honoured the contract we drafted
tleemann/road_evaluation/road/road.py
code served (permissive licence) · get_code("b0613e306198627a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 2, dtype family float, shape (2, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Create a simple test array with some variation
test_array = np.array([
    [1.0, 2.0, 3.0, 4.0],
    [5.0, 6.0, 7.0, 8.0]
], dtype=np.float32)

ARGS = {
    "s": test_array
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 2, dtype family float, shape (2, 4) — shape (2, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
rescale_channel Ran
ran, violated the contract we drafted
tleemann/road_evaluation/road/road.py
code served (permissive licence) · get_code("1d09d53a4f0a9512")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 2, dtype family float, shape (2, 3) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Create a small test array with shape (2, 3, 4)
# This represents a batch of 2 samples, with spatial dimensions 3x4
exp = np.random.randn(2, 3, 4)

ARGS = {
    "exp": exp
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 2, dtype family float, shape (2, 3) — shape (2, 3) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
BaseImputer Not yet run
unverified (harvested, no recorded run)
tleemann/road_evaluation/road/road.py
code served (permissive licence) · get_code("a9eaec23fad20830")
How it ran

no invocation recorded.

GAINImputer Not yet run
unverified (harvested, no recorded run)
tleemann/road_evaluation/road/road.py
code served (permissive licence) · get_code("c676b8a8d65d31fe")
How it ran

no invocation recorded.

SelfAttention Not yet run
unverified (harvested, no recorded run)
tleemann/road_evaluation/road/road.py
code served (permissive licence) · get_code("b28b45f46cb768f9")
How it ran

no invocation recorded.

_from_str Not yet run
unverified (harvested, no recorded run)
tleemann/road_evaluation/road/road.py
code served (permissive licence) · get_code("dd12bbba39b07ee5")
How it ran

no invocation recorded.

road_eval Not yet run
unverified (harvested, no recorded run)
tleemann/road_evaluation/road/road.py
code served (permissive licence) · get_code("44de25a587d6939c")
How it ran

no invocation recorded.

run_road Not yet run
unverified (harvested, no recorded run)
tleemann/road_evaluation/road/road.py
code served (permissive licence) · get_code("5c9f538fc7ee4d08")
How it ran

no invocation recorded.

Also present in other papers' code

Of 16 samples lifted from this paper's linked repositories, 16 carry a function identity and 0 do not. Of the 16 distinct functions with an identity, 0 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 0 later) and 16 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

With a variety of local feature attribution methods being proposed in recent years, follow-up work suggested several evaluation strategies. To assess the attribution quality across different attribution techniques, the most popular among these evaluation strategies in the image domain use pixel perturbations. However, recent advances discovered that different evaluation strategies produce conflicting rankings of attribution methods and can be prohibitively expensive to compute. In this work, we present an informationtheoretic analysis of evaluation strategies based on pixel perturbations. Our findings reveal that the results are strongly affected by information leakage through the shape of the removed pixels as opposed to their actual values. Using our theoretical insights, we propose a novel evaluation framework termed Remove and Debias (ROAD) which offers two contributions: First, it mitigates the impact of the confounders, which entails higher consistency among evaluation strategies. Second, ROAD does not require the computationally expensive retraining step and saves up to 99 % in computational costs compared to the state-of-the-art. We release our source code at https://github.com/ tleemann/road_evaluation.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2202.00449")
get_code_for_paper("2202.00449")
have("2202.00449")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2202.00449.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2202.00449.svg)](https://syntology.ai/paper/2202.00449)

Connect an agent — have() is free.