We lifted 3 functions out of this paper's own repositories and ran 2 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| x-plug/roleinteract | canonical | 2 of 3 |
| Function | Status | Where it lives |
|---|---|---|
| format_name | Ran | x-plug/roleinteract/dataset.py pointer only (licence: NONE) · get_code("9e20b00486307eb3") |
| make_group_profiles | Ran | x-plug/roleinteract/dataset.py pointer only (licence: NONE) · get_code("cfde14b173792b93") |
| json_load | Not yet run | x-plug/roleinteract/dataset.py pointer only (licence: NONE) · get_code("62d67ba64ad1a05c") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Large language models (LLMs) have advanced the development of various AI conversational agents, including role-playing conversational agents that mimic diverse characters and human behaviors. While prior research has predominantly focused on enhancing the conversational capability, role-specific knowledge, and stylistic attributes of these agents, there has been a noticeable gap in assessing their social intelligence. In this paper, we introduce SocialBench, the first benchmark designed to systematically evaluate the sociality of role-playing conversational agents at both individual and group levels of social interactions. The benchmark is constructed from a variety of sources and covers a wide range of 500 characters and over 6,000 question prompts and 30,800 multi-turn role-playing utterances. We conduct comprehensive evaluations on this benchmark using mainstream open-source and closed-source LLMs. We find that agents excelling in individual level does not imply their proficiency in group level. Moreover, the behavior of individuals may drift as a result of the influence exerted by other agents within the group. Experimental results on SocialBench confirm its significance as a testbed for assessing the social interaction of role-playing conversational agents. The benchmark is publicly accessible at https://github.com/X-PLUG/SocialBench.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2403.13679")
get_code_for_paper("2403.13679")
have("2403.13679")
Connect an agent — have() is free.