Xiaobin Wang, Anna Korhonen, Ivan Vuli'c, Songbo Hu, Moy Yuan
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
We present DIALIGHT, a toolkit for developing and evaluating multilingual Task-Oriented Dialogue (TOD) systems which facilitates systematic evaluations and comparisons between TOD systems using fine-tuning of Pretrained Language Models (PLMs) and those utilising the zero-shot and in-context learning capabilities of Large Language Models (LLMs). In addition to automatic evaluation, this toolkit features (i) a secure, user-friendly web interface for fine-grained human evaluation at both local utterance level and global dialogue level, and (ii) a microservice-based backend, improving efficiency and scalability. Our evaluations reveal that while PLM fine-tuning leads to higher accuracy and coherence, LLM-based systems excel in producing diverse and likeable responses. However, we also identify significant challenges of LLMs in adherence to task-specific instructions and generating outputs in multiple languages, highlighting areas for future research. We hope this open-sourced toolkit will serve as a valuable resource for researchers aiming to develop and properly evaluate multilingual TOD systems and will lower, currently still high, entry barriers in this field. 1 Disclaimer: Here, we (coarsely) differentiate between PLMs and LLMs in terms of their dependency on fine-tuning with task-specific datasets for achieving optimal performance. LLMs, such as LLaMA (Touvron et al., 2023) and GPT-4 (OpenAI, 2023), are characterised by their extensive training on a broad spectrum of data. This approach enables LLMs to adapt to diverse tasks with minimal reliance on task-specific data. Empirical evidence suggests that LLMs demonstrate remarkable capabilities, nearing or 'surpassing' human-level performance on NLP benchmarks such as SuperGLUE (Wang et al., 2019) and BIG-Bench (Srivastava et al., 2023).
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2401.02208")
get_code_for_paper("2401.02208")
have("2401.02208")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2401.02208.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2401.02208)
Connect an agent — have() is free.