Shu-Wen Yang, Andy Liu, Hung-Yi Lee, Shinji Watanabe, Kushal Lakhotia, Abdelrahman Mohamed, Jiatong Shi, Shang-Wen Li, Jeff Cheng-I, Xuankai Chang, Shuyan Dong, Hsiang-Sheng Tsai, and 6 more
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Transfer learning has proven to be crucial in advancing the state of speech and natural language processing research in recent years. In speech, a model pre-trained by self-supervised learning transfers remarkably well on multiple tasks. However, the lack of a consistent evaluation methodology is limiting towards a holistic understanding of the efficacy of such models. SUPERB was a step towards introducing a common benchmark to evaluate pretrained models across various speech tasks. In this paper, we introduce SUPERB-SG, a new benchmark focused on evaluating the semantic and generative capabilities of pre-trained models by increasing task diversity and difficulty over SUPERB. We use a lightweight methodology to test the robustness of representations learned by pre-trained models under shifts in data domain and quality across different types of tasks. It entails freezing pretrained model parameters, only using simple task-specific trainable heads. The goal is to be inclusive of all researchers, and encourage efficient use of computational resources. We also show that the task diversity of SUPERB-SG coupled with limited task supervision is an effective recipe for evaluating the generalizability of model representation. ˚Equal contribution.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2203.06849")
get_code_for_paper("2203.06849")
have("2203.06849")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2203.06849.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2203.06849)
Connect an agent — have() is free.