SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2404.02098 · 2024

BRAVEn: Improving Self-Supervised Pre-training for Visual and Auditory Speech Recognition

arXiv · PDF · Open in the Atlas

Code that ran

10 of 14 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 5 honoured the contract we drafted, 3 violated it and 0 have no contract verdict; 2 failures of our instrument; 4 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2404.02098.json · badge: /api/ran/2404.02098.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
ahaliassos/raven canonical (classifier label)
classifier basis: census
10 ran · 4 no recorded run
FunctionStatusWhere it lives
average_checkpoints Ran
ran; our contract draft was wrong, not the code
ahaliassos/raven/utils.py
code served (permissive licence) · get_code("96ab9eba7ed25bc3")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import tempfile
import os

torch.manual_seed(0)

# Create temporary checkpoint files with the expected structure
temp_dir = tempfile.mkdtemp()

checkpoint_paths = []
for i in range(3):
    checkpoint_path = os.path.join(temp_dir, f"checkpoint_{i}.pt")
    
    # Create a state_dict with 'model.' prefix keys
    state_dict = {
        'model.layer1.weight': torch.randn(4, 3),
        'model.layer1.bias': torch.randn(4),
        'model.layer2.weight': torch.randn(5, 4),
    }
    
    checkpoint = {'state_dict': state_dict}
    torch.save(checkpoint, checkpoint_path)
    checkpoint_paths.append(checkpoint_path)

ARGS = {
    'last': checkpoint_paths,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
end_detect Ran
ran, violated the contract we drafted
ahaliassos/raven/espnet/nets/e2e_asr_common.py
code served (permissive licence) · get_code("08b8264f1a3a1591")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np
import six

# Construct ended_hyps: list of hypothesis dicts with 'score' and 'yseq' keys
ended_hyps = [
    {
        'score': -5.2,
        'yseq': [1, 2, 3]
    },
    {
        'score': -4.8,
        'yseq': [1, 2]
    },
    {
        'score': -6.1,
        'yseq': [1, 2, 3, 4]
    },
    {
        'score': -5.5,
        'yseq': [1, 2, 3]
    }
]

# i: an integer representing current position/step
i = 5

# M: number of iterations to check (default 3)
M = 3

# D_end: threshold for score difference (default is log(1 * exp(-10)))
D_end = np.log(1 * np.exp(-10))

ARGS = {
    'ended_hyps': ended_hyps,
    'i': i,
    'M': M,
    'D_end': D_end
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, six — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_cer Ran
ran, honoured the contract we drafted
ahaliassos/raven/metrics.py
code served (permissive licence) · get_code("15461a4fffed58aa")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float64, ndim 0, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# get_cer takes two parameters: s and ref
# Both should be strings (or string-like objects that can be converted to lists)
# The function converts them to lists of characters and calls get_er on them

s = "hello"
ref = "hallo"

ARGS = {
    "s": s,
    "ref": ref
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type float64, ndim 0, dtype family float — no output shape is stated: observe_signature.py recorded float ndim 0, which disagrees with the fixture run
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_er Ran
ran, honoured the contract we drafted
ahaliassos/raven/metrics.py
code served (permissive licence) · get_code("4f158e2aa3d314f9")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float64, ndim 0, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Construct arguments for get_er function
# The function computes normalized Levenshtein distance between two sequences

# s: list of words/char in sentence to measure
s = ['hello', 'world', 'test']

# ref: list of words/char in reference
ref = ['hello', 'world', 'example']

ARGS = {
    's': s,
    'ref': ref
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type float64, ndim 0, dtype family float — no output shape is stated: observe_signature.py recorded float ndim 0, which disagrees with the fixture run
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_vgg2l_odim Ran
ran, honoured the contract we drafted
ahaliassos/raven/espnet/nets/e2e_asr_common.py
code served (permissive licence) · get_code("02eb6bc0a23198cb")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Construct arguments for get_vgg2l_odim
# The function takes:
# - idim: the input dimension (will be divided by in_channel, then by 2 twice)
# - in_channel: input channel size (default 3)
# - out_channel: output channel size (default 128)

# Example values:
# idim should be a number that represents total input dimension
# Let's use a value that makes sense for audio/signal processing
# If we have 3 channels and want a reasonable output, let's use 1200

ARGS = {
    "idim": 1200,
    "in_channel": 3,
    "out_channel": 128
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_wer Ran
ran, honoured the contract we drafted
ahaliassos/raven/metrics.py
code served (permissive licence) · get_code("c0ed4f13d5339d56")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float64, ndim 0, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# The function get_wer takes two string parameters: s and ref
# It splits both strings by whitespace and passes the resulting word lists to get_er
# get_er expects list-like objects that support len() and indexing

# Create two distinct sentences for testing
s = "hello world this is a test"
ref = "hello world this is a test sentence"

ARGS = {
    "s": s,
    "ref": ref
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type float64, ndim 0, dtype family float — no output shape is stated: observe_signature.py recorded float ndim 0, which disagrees with the fixture run
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
ids_to_str Ran
ran; our contract draft was wrong, not the code
ahaliassos/raven/utils.py
code served (permissive licence) · get_code("995e1b53bc619c6a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# token_ids: an iterable of integers that can be mapped to int
# char_list: a list/sequence where indices map to characters
token_ids = [0, 1, 2, 3, 1, 0]

char_list = ['h', 'e', 'l', 'o', '<space>', 'w']

ARGS = {
    'token_ids': token_ids,
    'char_list': char_list
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
is_scipy_wav_style Ran
ran, violated the contract we drafted
ahaliassos/raven/espnet/utils/cli_utils.py
code served (permissive licence) · get_code("a6b0233b308d11ee")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy
from collections.abc import Sequence

# Create a valid scipy wav-style tuple: (sample_rate as int, audio_data as numpy.ndarray)
sample_rate = 44100
audio_data = numpy.array([0.1, 0.2, 0.3, 0.4], dtype=numpy.float32)
valid_wav_style = (sample_rate, audio_data)

ARGS = {
    "value": valid_wav_style
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
label_smoothing_dist Ran
ran, honoured the contract we drafted
ahaliassos/raven/espnet/nets/e2e_asr_common.py
code served (permissive licence) · get_code("076c0dd1245be79b")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (5,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import numpy as np
import tempfile
import os

# Create a temporary transcript file with the expected JSON structure
transcript_data = {
    'utts': {
        'utt1': {
            'output': [
                {
                    'tokenid': '1 2 3 1 2'
                }
            ]
        },
        'utt2': {
            'output': [
                {
                    'tokenid': '2 3 4 2'
                }
            ]
        },
        'utt3': {
            'output': [
                {
                    'tokenid': '1 4 3'
                }
            ]
        }
    }
}

# Write to a temporary file
temp_file = tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False)
json.dump(transcript_data, temp_file)
temp_file.close()
transcript_path = temp_file.name

ARGS = {
    'odim': 5,
    'lsm_type': 'unigram',
    'transcript': transcript_path,
    'blank': 0
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 1, dtype family float, shape (5,) — shape (5,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
strtobool Ran
ran, violated the contract we drafted
ahaliassos/raven/espnet/utils/cli_utils.py
code served (permissive licence) · get_code("a322f4a71dff8cd9")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from distutils.util import strtobool as dist_strtobool

def strtobool(x):
    return bool(dist_strtobool(x))

# The function strtobool takes a single parameter 'x' which should be a string
# that can be converted to a boolean value. Valid inputs include:
# 'y', 'yes', 't', 'true', 'on', '1' -> True
# 'n', 'no', 'f', 'false', 'off', '0' -> False

ARGS = {
    'x': 'true'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
distutils, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
adadelta_eps_decay Not yet run
unverified (harvested, no recorded run)
ahaliassos/raven/espnet/asr/asr_utils.py
code served (permissive licence) · get_code("bb3665ddb6b8dd2d")
How it ran

no invocation recorded.

adam_lr_decay Not yet run
unverified (harvested, no recorded run)
ahaliassos/raven/espnet/asr/asr_utils.py
code served (permissive licence) · get_code("1fa5ca494ed9750a")
How it ran

no invocation recorded.

get_param_groups Not yet run
unverified (harvested, no recorded run)
ahaliassos/raven/utils.py
code served (permissive licence) · get_code("8b81ecb31057c782")
How it ran

no invocation recorded.

restore_snapshot Not yet run
unverified (harvested, no recorded run)
ahaliassos/raven/espnet/asr/asr_utils.py
code served (permissive licence) · get_code("96081a60369c59e0")
How it ran

no invocation recorded.

Also present in other papers' code

Of 14 samples lifted from this paper's linked repositories, 14 carry a function identity and 0 do not. Of the 14 distinct functions with an identity, 9 are also present in other papers whose code we hold (8 with at least one paper dated earlier, 0 the same month, 6 later) and 5 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
adadelta_eps_decay 7 other papers
5 dated earlier, 2 later
2007.09127 (2020-07)
CTC-Segmentation of Large Corpora for German End-to-end Speech Recognition
2010.11148 (2020-10)
FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization
2022.findings-emnlp.402 (2022 (year only))
BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model
and 4 more not named here
adam_lr_decay 7 other papers
5 dated earlier, 2 later
2007.09127 (2020-07)
CTC-Segmentation of Large Corpora for German End-to-end Speech Recognition
2010.11148 (2020-10)
FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization
2022.findings-emnlp.402 (2022 (year only))
BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model
and 4 more not named here
restore_snapshot 7 other papers
5 dated earlier, 2 later
2007.09127 (2020-07)
CTC-Segmentation of Large Corpora for German End-to-end Speech Recognition
2010.11148 (2020-10)
FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization
2022.findings-emnlp.402 (2022 (year only))
BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model
and 4 more not named here
end_detect 2 other papers
2 dated earlier
1805.03294 (2018-05)
Improved training of end-to-end attention models for speech recognition
2005.14327 (2020-05)
On the Comparison of Popular End-to-End Models for Large Scale Speech Recognition
get_er 2 other papers
1 dated earlier, 1 later
2308.06112 (2023-08)
Lip2Vec: Efficient and Robust Visual Speech Recognition via Latent-to-Latent Visual to Audio Representation Mapping
2411.02256 (2024-11)
Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs
get_wer 2 other papers
1 dated earlier, 1 later
2308.06112 (2023-08)
Lip2Vec: Efficient and Robust Visual Speech Recognition via Latent-to-Latent Visual to Audio Representation Mapping
2411.02256 (2024-11)
Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs
label_smoothing_dist 2 other papers
2 dated earlier
1805.03294 (2018-05)
Improved training of end-to-end attention models for speech recognition
2005.14327 (2020-05)
On the Comparison of Popular End-to-End Models for Large Scale Speech Recognition
get_cer 1 other paper
1 later
2411.02256 (2024-11)
Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs
get_vgg2l_odim 1 other paper
1 dated earlier
1805.03294 (2018-05)
Improved training of end-to-end attention models for speech recognition

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Self-supervision has recently shown great promise for learning visual and auditory speech representations from unlabelled data. In this work, we propose BRAVEn, an extension to the recent RAVEn method, which learns speech representations entirely from raw audio-visual data. Our modifications to RAVEn enable BRAVEn to achieve state-of-the-art results among self-supervised methods in various settings. Moreover, we observe favourable scaling behaviour by increasing the amount of unlabelled data well beyond other self-supervised works. In particular, we achieve 20.0% / 1.7% word error rate for VSR / ASR on the LRS3 test set, with only 30 hours of labelled data and no external ASR models. Our results suggest that readily available unlabelled audio-visual data can largely replace costly transcribed data.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2404.02098")
get_code_for_paper("2404.02098")
have("2404.02098")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2404.02098.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2404.02098.svg)](https://syntology.ai/paper/2404.02098)

Connect an agent — have() is free.