Alexander Rush, Sanchit Gandhi, Patrick Von Platen
We lifted 11 functions out of this paper's own repositories and ran 11 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| huggingface/open_asr_leaderboard | pwc_unofficial | 11 of 11 |
| Function | Status | Where it lives |
|---|---|---|
| build_asr_keep_token_ids | Ran | huggingface/open_asr_leaderboard/ark_asr/run_eval.py code served (permissive licence) · get_code("95565023b8f5c27e") |
| build_eos_token_ids | Ran | huggingface/open_asr_leaderboard/ark_asr/run_eval.py code served (permissive licence) · get_code("34e38edcaaae0414") |
| normalize | Ran | huggingface/open_asr_leaderboard/benchmark_fitting/score_voxpopuli_ref_errors.py code served (permissive licence) · get_code("251be1ea1d702e3c") |
| normalize_token_ids | Ran | huggingface/open_asr_leaderboard/ark_asr/run_eval.py code served (permissive licence) · get_code("585c0c7f730e6d1f") |
| parse_hyp | Ran | huggingface/open_asr_leaderboard/nemo_asr/run_eval_salm.py code served (permissive licence) · get_code("a41e5c4f1618068b") |
| resolve_device | Ran | huggingface/open_asr_leaderboard/audio8_asr/run_eval.py code served (permissive licence) · get_code("e82aa9a9f589056d") |
| resolve_dtype | Ran | huggingface/open_asr_leaderboard/audio8_asr/run_eval.py code served (permissive licence) · get_code("573d7aef30a8051e") |
| to_hub_id | Ran | huggingface/open_asr_leaderboard/normalizer/model_ids.py code served (permissive licence) · get_code("20e94432951733ce") |
| to_hub_ids | Ran | huggingface/open_asr_leaderboard/normalizer/model_ids.py code served (permissive licence) · get_code("bb9a452edf61892e") |
| transcribe | Ran | huggingface/open_asr_leaderboard/nemo_asr/run_eval_salm.py code served (permissive licence) · get_code("465981d7412a5bb3") |
| unique_token_ids | Ran | huggingface/open_asr_leaderboard/audio8_asr/run_eval.py code served (permissive licence) · get_code("288177fe23420283") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Speech recognition applications cover a range of different audio and text distributions, with different speaking styles, background noise, transcription punctuation and character casing. However, many speech recognition systems require dataset-specific tuning (audio filtering, punctuation removal and normalisation of casing), therefore assuming a-priori knowledge of both the audio and text distributions. This tuning requirement can lead to systems failing to generalise to other datasets and domains. To promote the development of multi-domain speech systems, we introduce the End-to-end Speech Benchmark (ESB) for evaluating the performance of a single automatic speech recognition (ASR) system across a broad set of speech datasets. Benchmarked systems must use the same data preand post-processing algorithm across datasets -assuming the audio and text data distributions are a-priori unknown. We compare a series of state-of-the-art (SoTA) end-to-end (E2E) systems on this benchmark, demonstrating how a single speech system can be applied and evaluated on a wide range of data distributions. We find E2E systems to be effective across datasets: in a fair comparison, E2E systems achieve within 2.6% of SoTA systems tuned to a specific dataset. Our analysis reveals that transcription artefacts, such as punctuation and casing, pose difficulties for ASR systems and should be included in evaluation. We believe E2E benchmarking over a range of datasets promotes the research of multi-domain speech recognition systems. ESB is available at https://huggingface.co/esb.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2210.13352")
get_code_for_paper("2210.13352")
have("2210.13352")
Connect an agent — have() is free.