SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 1902.00751 · 2019

Parameter-Efficient Transfer Learning for NLP

Andrea Gesmundo, Sylvain Gelly, Neil Houlsby, Andrei Giurgiu, Bruna Morrone, Quentin De Laroussilhe, Mona Attariyan, Stanisław Jastrze ¸bski

arXiv · PDF · Open in the Atlas

Code that ran

14 of 22 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (3 of them constructed an object rather than computing a result; 0 honoured the contract we drafted, 0 violated it and 13 have no contract verdict; 1 failure of our instrument; 8 no recorded run), measured 2026-11-21; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/1902.00751.json · badge: /api/ran/1902.00751.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
CyberZHG/keras-bert canonical (classifier label)
classifier basis: census
6 ran · 4 no recorded run
AsaCooperStickland/Bert-n-Pals community
Papers with Code archive
3 ran · 0 no recorded run
prrao87/fine-grained-sentiment community
Papers with Code archive
2 ran · 0 no recorded run
osu-mlb/petl_vision community
Papers with Code archive
1 ran · 1 no recorded run
zphang/bert_on_stilts community
Papers with Code archive
1 ran · 0 no recorded run
Davidzhangyuanhan/NOAH community
Papers with Code archive
1 ran · 0 no recorded run
krypticmouse/Adapter-BERT community
Papers with Code archive
0 ran · 1 no recorded run
cs-mshah/Adapter-Bert community
Papers with Code archive
0 ran · 1 no recorded run
google-research/adapter-bert marked official in Papers with Code
Papers with Code archive
0 ran · 1 no recorded run
FunctionStatusWhere it lives
Adapter Ran
ran on a synthesized input
osu-mlb/petl_vision/model/adapter.py
pointer only (licence: NONE) · get_code("7b012e4d16a33bf8")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, osu-mlb/petl_vision when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

Adapter Ran
ran on a synthesized input
zphang/bert_on_stilts/pytorch_pretrained_bert/modeling.py
code served (permissive licence) · get_code("69eca16a33f5afcb")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, class construct-then-call prompt, model not recorded) and returned Tensor, ndim 3, dtype family float, shape (2, 4, 8) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import math
import torch
from torch import nn

def gelu(x):
    """Implementation of the gelu activation function.
        For information: OpenAI GPT's gelu is slightly different (and gives slightly different results):
        0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
        Also see https://arxiv.org/abs/1606.08415
    """
    return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))

def swish(x):
    return x * torch.sigmoid(x)

ACT2FN = {'gelu': gelu, 'relu': torch.nn.functional.relu, 'swish': swish}

class Adapter(nn.Module):

    def __init__(self, config):
        super(Adapter, self).__init__()
        self.down_project = nn.Linear(config.hidden_size, config.adapter_size)
        self.activation = ACT2FN[config.adapter_act] if isinstance(config.adapter_act, str) else config.adapter_act
        self.up_project = nn.Linear(config.adapter_size, config.hidden_size)
        self.init_weights(config)

    def forward(self, hidden_states):
        down_projected = self.down_project(hidden_states)
        activated = self.activation(down_projected)
        up_projected = self.up_project(activated)
        return hidden_states + up_projected

    def init_weights(self, config):
        self.down_project.weight.data.normal_(mean=0.0, std=config.adapter_initializer_range)
        self.down_project.bias.data.zero_()
        self.up_project.weight.data.normal_(mean=0.0, std=config.adapter_initializer_range)
        self.up_project.bias.data.zero_()


class Config:
    def __init__(self, hidden_size, adapter_size, adapter_act, adapter_initializer_range):
        self.hidden_size = hidden_size
        self.adapter_size = adapter_size
        self.adapter_act = adapter_act
        self.adapter_initializer_range = adapter_initializer_range


INIT_ARGS = {
    'config': Config(
        hidden_size=8,
        adapter_size=3,
        adapter_act='gelu',
        adapter_initializer_range=0.02
    )
}

ARGS = {
    'hidden_states': torch.randn(2, 4, 8)
}
Setup block drafted by
a language model, class construct-then-call prompt, model not recorded
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 3, dtype family float, shape (2, 4, 8) — shape (2, 4, 8) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: torch and numpy seeded to 0 by the harness immediately before the class was constructed from INIT_ARGS (fixture_check.py)
AdapterSuper Ran
ran on a synthesized input · constructed an object; no result was computed
Davidzhangyuanhan/NOAH/model/module/adapter_super.py
code served (permissive licence) · get_code("7a291b0c4ae38f59")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, AdapterSuper: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# AdapterSuper expects:
# - embed_dims: int - the embedding dimension
# - reduction_dims: int - the reduction dimension for the adapter
# - drop_rate_adapter: float (optional) - dropout rate

ARGS = {
    'embed_dims': 8,
    'reduction_dims': 4,
    'drop_rate_adapter': 0.1
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type AdapterSuper — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a AdapterSuper output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
BERTLayerNorm Ran
ran on a synthesized input
AsaCooperStickland/Bert-n-Pals/modeling.py
code served (permissive licence) · get_code("fcde41b66078c9e5")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, class construct-then-call prompt, model not recorded) and returned Tensor, ndim 3, dtype family float, shape (2, 3, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

class BERTLayerNorm(nn.Module):

    def __init__(self, config, multi_params=None, variance_epsilon=1e-12):
        """Construct a layernorm module in the TF style (epsilon inside the square root).
        """
        super(BERTLayerNorm, self).__init__()
        if multi_params is not None:
            self.gamma = nn.Parameter(torch.ones(config.hidden_size_aug))
            self.beta = nn.Parameter(torch.zeros(config.hidden_size_aug))
        else:
            self.gamma = nn.Parameter(torch.ones(config.hidden_size))
            self.beta = nn.Parameter(torch.zeros(config.hidden_size))
        self.variance_epsilon = variance_epsilon

    def forward(self, x):
        u = x.mean(-1, keepdim=True)
        s = (x - u).pow(2).mean(-1, keepdim=True)
        x = (x - u) / torch.sqrt(s + self.variance_epsilon)
        return self.gamma * x + self.beta


class Config:
    def __init__(self, hidden_size, hidden_size_aug=None):
        self.hidden_size = hidden_size
        self.hidden_size_aug = hidden_size_aug if hidden_size_aug is not None else hidden_size


INIT_ARGS = {
    'config': Config(hidden_size=4),
    'multi_params': None,
    'variance_epsilon': 1e-12
}

ARGS = {
    'x': torch.randn(2, 3, 4)
}
Setup block drafted by
a language model, class construct-then-call prompt, model not recorded
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 3, dtype family float, shape (2, 3, 4) — shape (2, 3, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: torch and numpy seeded to 0 by the harness immediately before the class was constructed from INIT_ARGS (fixture_check.py)
BERTLowRank Ran
ran on a synthesized input
AsaCooperStickland/Bert-n-Pals/modeling.py
code served (permissive licence) · get_code("4fcb1d49ea7c6f51")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, class construct-then-call prompt, model not recorded) and returned Tensor, ndim 3, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

class Config:
    def __init__(self):
        self.hidden_size = 8
        self.hidden_size_aug = 6
        self.extra_dim = None

config = Config()

INIT_ARGS = {
    'config': config,
    'extra_dim': None
}

ARGS = {
    'hidden_states': torch.randn(2, 5, 8),  # (batch_size, seq_len, hidden_size)
    'attention_mask': None
}
Setup block drafted by
a language model, class construct-then-call prompt, model not recorded
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 3, dtype family float — no output shape is stated: observe_signature.py observed this class with its constructor sizes canonicalised where it could, which the node does not record, so its shape is not stated as this setup block's
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: torch and numpy seeded to 0 by the harness immediately before the class was constructed from INIT_ARGS (fixture_check.py)
BERTSelfOutput Ran
ran on a synthesized input
AsaCooperStickland/Bert-n-Pals/modeling.py
code served (permissive licence) · get_code("c97326dd9a8f8a78")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, class construct-then-call prompt, model not recorded) and returned Tensor, ndim 2, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import copy
import math
import torch
import torch.nn as nn

def gelu(x):
    """Implementation of the gelu activation function."""
    return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))

class BERTLayerNorm(nn.Module):
    def __init__(self, config, multi_params=None, variance_epsilon=1e-12):
        """Construct a layernorm module in the TF style (epsilon inside the square root)."""
        super(BERTLayerNorm, self).__init__()
        if multi_params is not None:
            self.gamma = nn.Parameter(torch.ones(config.hidden_size_aug))
            self.beta = nn.Parameter(torch.zeros(config.hidden_size_aug))
        else:
            self.gamma = nn.Parameter(torch.ones(config.hidden_size))
            self.beta = nn.Parameter(torch.zeros(config.hidden_size))
        self.variance_epsilon = variance_epsilon

    def forward(self, x):
        u = x.mean(-1, keepdim=True)
        s = (x - u).pow(2).mean(-1, keepdim=True)
        x = (x - u) / torch.sqrt(s + self.variance_epsilon)
        return self.gamma * x + self.beta

class BERTLowRank(nn.Module):
    def __init__(self, config, extra_dim=None):
        super(BERTLowRank, self).__init__()
        if config.extra_dim:
            self.aug_dense = nn.Linear(config.hidden_size, config.extra_dim)
            self.aug_dense2 = nn.Linear(config.extra_dim, config.hidden_size)
        else:
            self.aug_dense = nn.Linear(config.hidden_size, config.hidden_size_aug)
            self.aug_dense2 = nn.Linear(config.hidden_size_aug, config.hidden_size)
        self.config = config
        self.hidden_act_fn = gelu

    def forward(self, hidden_states, attention_mask=None):
        hidden_states_aug = self.aug_dense(hidden_states)
        hidden_states_aug = self.hidden_act_fn(hidden_states_aug)
        hidden_states = self.aug_dense2(hidden_states_aug)
        return hidden_states

class BERTSelfOutput(nn.Module):
    def __init__(self, config, multi_params=None, houlsby=False):
        super(BERTSelfOutput, self).__init__()
        if houlsby:
            multi = BERTLowRank(config)
            self.multi_layers = nn.ModuleList([copy.deepcopy(multi) for _ in range(config.num_tasks)])
        if multi_params is not None:
            self.dense = nn.Linear(config.hidden_size_aug, config.hidden_size_aug)
        else:
            self.dense = nn.Linear(config.hidden_size, config.hidden_size)
        self.LayerNorm = BERTLayerNorm(config, multi_params)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)
        self.houlsby = houlsby

    def forward(self, hidden_states, input_tensor, attention_mask=None, i=0):
        hidden_states = self.dense(hidden_states)
        hidden_states = self.dropout(hidden_states)
        if self.houlsby:
            hidden_states = hidden_states + self.multi_layers[i](hidden_states, attention_mask)
        hidden_states = self.LayerNorm(hidden_states + input_tensor)
        return hidden_states

class Config:
    def __init__(self, hidden_size=4, hidden_size_aug=3, hidden_dropout_prob=0.1, 
                 num_tasks=2, extra_dim=None):
        self.hidden_size = hidden_size
        self.hidden_size_aug = hidden_size_aug
        self.hidden_dropout_prob = hidden_dropout_prob
        self.num_tasks = num_tasks
        self.extra_dim = extra_dim

config = Config(hidden_size=4, hidden_size_aug=3, hidden_dropout_prob=0.1, 
                num_tasks=2, extra_dim=None)

INIT_ARGS = {
    'config': config,
    'multi_params': None,
    'houlsby': False
}

ARGS = {
    'hidden_states': torch.randn(2, 4),
    'input_tensor': torch.randn(2, 4),
    'attention_mask': None,
    'i': 0
}

if __name__ == '__main__':
    model = BERTSelfOutput(**INIT_ARGS)
    output = model(**ARGS)
    print("Output shape:", output.shape)
    print("Test passed!")
Setup block drafted by
a language model, class construct-then-call prompt, model not recorded
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 2, dtype family float — no output shape is stated: observe_signature.py observed this class with its constructor sizes canonicalised where it could, which the node does not record, so its shape is not stated as this setup block's
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: torch and numpy seeded to 0 by the harness immediately before the class was constructed from INIT_ARGS (fixture_check.py)
Transformer Ran
ran on a synthesized input · constructed an object; no result was computed
prrao87/fine-grained-sentiment/training/transformer_utils/model.py
code served (permissive licence) · get_code("96aac0e3241b82e0")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, Transformer: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Parameters for the Transformer
embed_dim = 8
hidden_dim = 16
num_embeddings = 32
num_max_positions = 64
num_heads = 2
num_layers = 2
dropout = 0.1
causal = True

ARGS = {
    "embed_dim": embed_dim,
    "hidden_dim": hidden_dim,
    "num_embeddings": num_embeddings,
    "num_max_positions": num_max_positions,
    "num_heads": num_heads,
    "num_layers": num_layers,
    "dropout": dropout,
    "causal": causal,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Transformer — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a Transformer output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
TransformerWithAdapters Ran
ran on a synthesized input · constructed an object; no result was computed
prrao87/fine-grained-sentiment/training/transformer_utils/model.py
code served (permissive licence) · get_code("22acb9946a708133")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, TransformerWithAdapters: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

ARGS = {
    "adapters_dim": 8,
    "embed_dim": 16,
    "hidden_dim": 32,
    "num_embeddings": 64,
    "num_max_positions": 20,
    "num_heads": 2,
    "num_layers": 2,
    "dropout": 0.1,
    "causal": True,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type TransformerWithAdapters — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a TransformerWithAdapters output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
checkpoint_loader Ran
ran on a synthesized input
CyberZHG/keras-bert/keras_bert/loader.py
code served (permissive licence) · get_code("81518ffac57afb91")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned function. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import tensorflow as tf
import tempfile
import os

# Create a temporary checkpoint file with TensorFlow 1.x checkpoint format
temp_dir = tempfile.mkdtemp()
checkpoint_file = os.path.join(temp_dir, "test_checkpoint")

# Create a simple checkpoint using TensorFlow 1.x API
with tf.compat.v1.Session() as sess:
    # Create some variables to save
    var1 = tf.Variable([[1.0, 2.0], [3.0, 4.0]], name="test_var_1")
    var2 = tf.Variable([5.0, 6.0], name="test_var_2")
    
    sess.run(tf.compat.v1.global_variables_initializer())
    
    # Save the checkpoint
    saver = tf.compat.v1.train.Saver()
    saver.save(sess, checkpoint_file)

ARGS = {
    "checkpoint_file": checkpoint_file
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
tensorflow — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type function — a function output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a function output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
extract_embeddings_generator Ran
ran on a synthesized input
CyberZHG/keras-bert/keras_bert/util.py
code served (permissive licence) · get_code("6afe8c057023f35c")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned generator. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import backend as K
from collections import namedtuple

# Create mock classes for required components

class MockTokenizer:
    def __init__(self, vocabs, cased=False):
        self.vocabs = vocabs
        self.cased = cased
    
    def encode(self, text1, text2=None, max_len=None):
        # Return token ids and segment ids
        if isinstance(text1, str):
            token_len = min(5, max_len) if max_len else 5
        else:
            token_len = min(5, max_len) if max_len else 5
        
        tokens = [101] + [self.vocabs.get(chr(i), 1) for i in range(token_len - 2)] + [102]
        segments = [0] * len(tokens)
        
        if max_len and len(tokens) < max_len:
            tokens.extend([0] * (max_len - len(tokens)))
            segments.extend([0] * (max_len - len(segments)))
        
        return tokens[:max_len] if max_len else tokens, segments[:max_len] if max_len else segments

class Extract(keras.layers.Layer):
    def __init__(self, index=0, name=None, **kwargs):
        super().__init__(name=name, **kwargs)
        self.index = index
    
    def call(self, inputs):
        return inputs[:, self.index:self.index+1, :]

class MaskedGlobalMaxPool1D(keras.layers.Layer):
    def __init__(self, name=None, **kwargs):
        super().__init__(name=name, **kwargs)
    
    def call(self, inputs):
        return keras.layers.GlobalMaxPooling1D()(inputs)

# Build a minimal mock BERT model
def build_mock_model():
    seq_len = 8
    vocab_size = 100
    embed_dim = 4
    
    input_ids = keras.Input(shape=(seq_len,), dtype=tf.int32, name='input_ids')
    segment_ids = keras.Input(shape=(seq_len,), dtype=tf.int32, name='segment_ids')
    
    # Simple embedding and output
    x = keras.layers.Embedding(vocab_size, embed_dim)(input_ids)
    outputs = keras.layers.Dense(embed_dim)(x)
    
    model = keras.Model(inputs=[input_ids, segment_ids], outputs=outputs)
    return model

# Create mock vocabulary
mock_vocab = {
    '[CLS]': 101,
    '[SEP]': 102,
    '[PAD]': 0,
    'hello': 1,
    'world': 2,
    'test': 3,
}

# Patch the required functions
import sys
from types import ModuleType

# Mock module for missing functions
class MockModule:
    @staticmethod
    def load_trained_model_from_checkpoint(config_file, checkpoint_file, output_layer_num=1):
        return build_mock_model()
    
    @staticmethod
    def load_vocabulary(vocab_path):
        return mock_vocab
    
    @staticmethod
    def Tokenizer(vocabs, cased=False):
        return MockTokenizer(vocabs, cased=cased)

# Inject mocks into the function's namespace
POOL_NSP = 'POOL_NSP'
POOL_MAX = 'POOL_MAX'
POOL_AVE = 'POOL_AVE'

def load_trained_model_from_checkpoint(config_file, checkpoint_file, output_layer_num=1):
    return build_mock_model()

def load_vocabulary(vocab_path):
    return mock_vocab

def Tokenizer(vocabs, cased=False):
    return MockTokenizer(vocabs, cased=cased)

class Extract(keras.layers.Layer):
    def __init__(self, index=0, name=None, **kwargs):
        super().__init__(name=name, **kwargs)
        self.index = index
    
    def call(self, inputs):
        return inputs[:, self.index:self.index+1, :]

class MaskedGlobalMaxPool1D(keras.layers.Layer):
    def __init__(self, name=None, **kwargs):
        super().__init__(name=name, **kwargs)
    
    def call(self, inputs):
        return keras.layers.GlobalMaxPooling1D()(inputs)

# Create sample texts
sample_texts = [
    "hello world",
    "test sentence",
]

# Build ARGS dictionary
ARGS = {
    'model': 'mock_model_path',
    'texts': sample_texts,
    'poolings': None,
    'vocabs': mock_vocab,
    'cased': False,
    'batch_size': 2,
    'cut_embed': True,
    'output_layer_num': 1,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, tensorflow — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type generator — a generator output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a generator output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
gen_batch_inputs Ran
ran on a synthesized input
CyberZHG/keras-bert/keras_bert/bert.py
code served (permissive licence) · get_code("501f37049301636e")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Set seed for reproducibility
np.random.seed(0)

# Define special tokens
TOKEN_PAD = ''
TOKEN_UNK = '[UNK]'
TOKEN_CLS = '[CLS]'
TOKEN_SEP = '[SEP]'
TOKEN_MASK = '[MASK]'

def get_base_dict():
    """Get basic dictionary containing special tokens."""
    return {TOKEN_PAD: 0, TOKEN_UNK: 1, TOKEN_CLS: 2, TOKEN_SEP: 3, TOKEN_MASK: 4}

# Build token_dict with special tokens and some regular tokens
base_dict = get_base_dict()
token_dict = base_dict.copy()
token_dict.update({
    'hello': 5,
    'world': 6,
    'foo': 7,
    'bar': 8,
    'baz': 9,
})

# Build token_list - list of all available tokens (used for random replacement)
token_list = ['hello', 'world', 'foo', 'bar', 'baz']

# Build sentence_pairs - list of pairs, each pair contains [first_sentence, second_sentence]
# Each sentence is a list of tokens
sentence_pairs = [
    [['hello', 'world'], ['foo', 'bar']],
    [['baz', 'hello'], ['world', 'foo']],
]

ARGS = {
    'sentence_pairs': sentence_pairs,
    'token_dict': token_dict,
    'token_list': token_list,
    'seq_len': 16,
    'mask_rate': 0.15,
    'mask_mask_rate': 0.8,
    'mask_random_rate': 0.1,
    'swap_sentence_rate': 0.5,
    'force_mask': True,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_checkpoint_paths Ran
ran; our contract draft was wrong, not the code
CyberZHG/keras-bert/keras_bert/util.py
code served (permissive licence) · get_code("13d48a14384b395e")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned CheckpointPaths. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import tempfile

def get_checkpoint_paths(model_path):
    from collections import namedtuple
    CheckpointPaths = namedtuple('CheckpointPaths', ['config', 'checkpoint', 'vocab'])
    config_path = os.path.join(model_path, 'bert_config.json')
    checkpoint_path = os.path.join(model_path, 'bert_model.ckpt')
    vocab_path = os.path.join(model_path, 'vocab.txt')
    return CheckpointPaths(config_path, checkpoint_path, vocab_path)

# Create a temporary directory to simulate a model_path
temp_dir = tempfile.mkdtemp()

# The function expects a model_path which is a string representing a directory
ARGS = {
    'model_path': temp_dir
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type CheckpointPaths — a CheckpointPaths output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a CheckpointPaths output the recorded ndim 1 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_inputs Ran
ran on a synthesized input
CyberZHG/keras-bert/keras_bert/layers/inputs.py
code served (permissive licence) · get_code("9cbbfb6d9b9dc99e")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import tensorflow as tf

ARGS = {
    'seq_len': 128
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
tensorflow — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list — a list output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_token_embedding Ran
ran on a synthesized input
CyberZHG/keras-bert/keras_bert/bert.py
code served (permissive licence) · get_code("2276095f620fd668")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Parameter, ndim 2, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

class EmbeddingLayer(nn.Module):
    def __init__(self):
        super().__init__()
        self.embedding = nn.Embedding(10, 4)
        # output is a tuple where index 1 is the embedding weights
        self.output = (torch.randn(2, 4), self.embedding.weight)
    
    def forward(self, x):
        return self.embedding(x)

class MockModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.embedding_token = EmbeddingLayer()
        self._layers = {'Embedding-Token': self.embedding_token}
    
    def get_layer(self, name):
        return self._layers[name]

model = MockModel()

ARGS = {
    'model': model
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Parameter, ndim 2, dtype family float — no output shape is stated: observe_signature.py recorded Tensor ndim 2, which disagrees with the fixture run
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
AdapterModule Not yet run
unverified (harvested, no recorded run)
krypticmouse/Adapter-BERT/src/model/adapter.py
pointer only (licence: NONE) · get_code("cbac3bf75f2ee9c8")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, krypticmouse/Adapter-BERT when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

AdapterModule Not yet run
unverified (harvested, no recorded run)
cs-mshah/Adapter-Bert/adapter-bert/model/adapter.py
code served (permissive licence) · get_code("2de542d0225f66f0")
How it ran

no invocation recorded.

build_model_from_config Not yet run
unverified (harvested, no recorded run)
CyberZHG/keras-bert/keras_bert/loader.py
code served (permissive licence) · get_code("604dea8db99819dc")
How it ran

no invocation recorded.

extract_embeddings Not yet run
unverified (harvested, no recorded run)
CyberZHG/keras-bert/keras_bert/util.py
code served (permissive licence) · get_code("a8a537c8d7277f7e")
How it ran

no invocation recorded.

feedforward_adapter Not yet run
unverified (harvested, no recorded run)
google-research/adapter-bert/modeling.py
code served (permissive licence) · get_code("b42f7bbad5547ecf")
How it ran

no invocation recorded.

get_pretrained Not yet run
unverified (harvested, no recorded run)
CyberZHG/keras-bert/keras_bert/datasets/pretrained.py
code served (permissive licence) · get_code("02f4590e932e79ef")
How it ran

no invocation recorded.

init_weight Not yet run
unverified (harvested, no recorded run)
osu-mlb/petl_vision/model/adapter.py
pointer only (licence: NONE) · get_code("332f932a85b80a1b")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, osu-mlb/petl_vision when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

load_trained_model_from_checkpoint Not yet run
unverified (harvested, no recorded run)
CyberZHG/keras-bert/keras_bert/loader.py
code served (permissive licence) · get_code("315e40f9424312ef")
How it ran

no invocation recorded.

Also present in other papers' code

Of 22 samples lifted from this paper's linked repositories, 22 carry a function identity and 0 do not. Of the 22 distinct functions with an identity, 2 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 2 later) and 20 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
feedforward_adapter
brick of this paper's assembly: entry
1 other paper
1 later
2005.11787 (2020-05)
Common Sense or World Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformers
get_checkpoint_paths 1 other paper
1 later
2003.08813 (2020-03)
Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentation

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-21.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Fine-tuning large pre-trained models is an effective transfer mechanism in NLP. However, in the presence of many downstream tasks, fine-tuning is parameter inefficient: an entire new model is required for every task. As an alternative, we propose transfer with adapter modules. Adapter modules yield a compact and extensible model; they add only a few trainable parameters per task, and new tasks can be added without revisiting previous ones. The parameters of the original network remain fixed, yielding a high degree of parameter sharing. To demonstrate adapter's effectiveness, we transfer the recently proposed BERT Transformer model to 26 diverse text classification tasks, including the GLUE benchmark. Adapters attain near state-of-the-art performance, whilst adding only a few parameters per task. On GLUE, we attain within 0.4% of the performance of full fine-tuning, adding only 3.6% parameters per task. By contrast, fine-tuning trains 100% of the parameters per task. 1

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("1902.00751")
get_code_for_paper("1902.00751")
have("1902.00751")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/1902.00751.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/1902.00751.svg)](https://syntology.ai/paper/1902.00751)

Connect an agent — have() is free.