Tatiana Shavrina, Valentin Malykh, Maria Tikhonova, Alena Fenogenova, Andrey Chertok, Denis Shevelev, Ekaterina Artemova, Vladislav Mikhailov, Anton Emelyanov, Andrey Evlampiev
We lifted 1 functions out of this paper's own repositories and ran 1 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| RussianNLP/RussianSuperGLUE | canonical | 1 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| parse_write_preds_arg | Ran | RussianNLP/RussianSuperGLUE/jiant-russian-v2/evaluate.py code served (permissive licence) · get_code("85e4a5947d2004ea") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
In this paper, we introduce an advanced Russian general language understanding evaluation benchmark -RussianGLUE. Recent advances in the field of universal language models and transformers require the development of a methodology for their broad diagnostics and testing for general intellectual skills -detection of natural language inference, commonsense reasoning, ability to perform simple logical operations regardless of text subject or lexicon. For the first time, a benchmark of nine tasks, collected and organized analogically to the SuperGLUE methodology (Wang et al., 2019), was developed from scratch for the Russian language. We provide baselines, human level evaluation, an opensource framework for evaluating models and an overall leaderboard of transformer models for the Russian language. Besides, we present the first results of comparing multilingual models in the adapted diagnostic test set and offer the first steps to further expanding or assessing state-of-the-art models independently of language.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2010.15925")
get_code_for_paper("2010.15925")
have("2010.15925")
Connect an agent — have() is free.