Jamin Shin, Hangyeol Yu, Seunghyun Lee, Hyeongdon Moon, Seungtaek Choi, Juneyoung Park, Myeongho Jeong, Minsam Kim, Yoonseok Yang
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
The automatic generation of Multiple Choice Questions (MCQ) has the potential to reduce the time educators spend on student assessment significantly. However, existing evaluation metrics for MCQ generation, such as BLEU, ROUGE, and METEOR, focus on the n-gram based similarity of the generated MCQ to the gold sample in the dataset and disregard their educational value. They fail to evaluate the MCQ's ability to assess the student's knowledge of the corresponding target fact. To tackle this issue, we propose a novel automatic evaluation metric, coined Knowledge Dependent Answerability (KDA), which measures the MCQ's answerability given knowledge of the target fact. Specifically, we first show how to measure KDA based on student responses from a human survey. Then, we propose two automatic evaluation metrics, KDA disc and KDA cont , that approximate KDA by leveraging pre-trained language models to imitate students' problem-solving behavior. Through our human studies, we show that KDA disc and KDA cont have strong correlations with both (1) KDA and (2) usability in an actual classroom setting, labeled by experts. Furthermore, when combined with ngram based similarity metrics, KDA disc and KDA cont are shown to have a strong predictive power for various expert-labeled MCQ quality measures. 1
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2211.11902")
get_code_for_paper("2211.11902")
have("2211.11902")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2211.11902.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2211.11902)
Connect an agent — have() is free.