We lifted 3 functions out of this paper's own repositories and ran 3 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| Felixgithub2017/MMCU | canonical | 2 of 2 |
| copy not recorded | — | 1 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| find_valid_substrings | Ran | Felixgithub2017/MMCU/TestBloomz.py pointer only (licence: NONE) · get_code("5dda29b6436a1765") |
| format_example | Ran | Felixgithub2017/MMCU/TestBloomz.py pointer only (licence: NONE) · get_code("a4f2a52c39dd4a34") |
| format_subject | Ran | this paper's copy was not recorded; identical code first harvested from pjlab-sys4nlp/llama-moe pointer only · get_code("6ab745408cb8648b") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
The development of large-scale Chinese language models is flourishing, yet there is a lack of corresponding capability assessments. Therefore, we propose a test to measure the multitask accuracy of large Chinese language models. This test encompasses four major domains, including medicine, law, psychology, and education, with 15 subtasks in medicine and 8 subtasks in education. We found that the best-performing models in the zero-shot setting outperformed the worst-performing models by nearly 18.6 percentage points on average. Across the four major domains, the highest average zero-shot accuracy of all models is 0.512. In the subdomains, only the GPT-3.5-turbo model achieved a zero-shot accuracy of 0.693 in clinical medicine, which was the highest accuracy among all models across all subtasks. All models performed poorly in the legal domain, with the highest zero-shot accuracy reaching only 0.239. By comprehensively evaluating the breadth and depth of knowledge across multiple disciplines, this test can more accurately identify the shortcomings of the models.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2304.12986")
get_code_for_paper("2304.12986")
have("2304.12986")
Connect an agent — have() is free.