Zhou Zhao, Yichong Leng, Jin Xu, Chang Zhou, Jingren Zhou, Qian Yang, Ziyue Jiang, Yunfei Chu, Xiaohuan Zhou, Wenrui Liu, Yuanjun Lv
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Recently, instruction-following audio-language models have received broad attention for human-audio interaction. However, the absence of benchmarks capable of evaluating audio-centric interaction capabilities has impeded advancements in this field. Previous models primarily focus on assessing different fundamental tasks, such as automatic speech recognition, and lack an assessment of the openended generative capabilities centered around audio. Thus, it is challenging to track the progression in the Large Audio-Language Models (LALMs) domain and to provide guidance for future improvement. In this paper, we introduce AIR-Bench (Audio InstRuction Benchmark), the first benchmark designed to evaluate the ability of LALMs to understand various types of audio signals (including human speech, natural sounds, and music), and furthermore, to interact with humans in the textual format. AIR-Bench encompasses two dimensions: foundation and chat benchmarks. The former consists of 19 tasks with approximately 19k singlechoice questions, intending to inspect the basic single-task ability of LALMs. The latter one contains 2k instances of open-ended questionand-answer data, directly assessing the comprehension of the model on complex audio and its capacity to follow instructions. Both benchmarks require the model to generate hypotheses directly. We design a unified framework that leverages advanced language models, such as GPT-4, to evaluate the scores of generated hypotheses given the meta-information of the audio. Experimental results demonstrate a high level of consistency between GPT-4-based evaluation and human evaluation. By revealing the limitations of existing LALMs through evaluation results, AIR-Bench can provide insights into the direction of future research.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2402.07729")
get_code_for_paper("2402.07729")
have("2402.07729")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2402.07729.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2402.07729)
Connect an agent — have() is free.