SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2412.13682 · ICLR · 2024

ChinaTravel: An Open-Ended Travel Planning Benchmark with Compositional Constraint Validation for Language Agents

Yu-Feng Li, Lan-Zhe Guo, Jie-Jing Shao, Zhenhua Dong, Bo-Wen Zhang, Xiao-Wen Yang, Bai-Zhi Chen, Si-Yu Han, Jing-Hao Pang, Wen-Da Wei, Guohao Cai

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 15 functions out of this paper's own repositories and ran 0 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

RepositoryRoleRan
LAMDASZ-ML/ChinaTravel canonical 0 of 15
FunctionStatusWhere it lives
calc_time_delta Not yet run LAMDASZ-ML/ChinaTravel/chinatravel/evaluation/preference.py
code served (permissive licence) · get_code("2ab68efe91d4bf24")
city_lookup Not yet run LAMDASZ-ML/ChinaTravel/chinatravel/environment/language.py
code served (permissive licence) · get_code("4a50071d3c3731aa")
city_names Not yet run LAMDASZ-ML/ChinaTravel/chinatravel/environment/language.py
code served (permissive licence) · get_code("83dd28641f911ace")
convenient_restaurant Not yet run LAMDASZ-ML/ChinaTravel/chinatravel/evaluation/preference.py
code served (permissive licence) · get_code("23eac122637ede76")
convenient_transport Not yet run LAMDASZ-ML/ChinaTravel/chinatravel/evaluation/preference.py
code served (permissive licence) · get_code("0abf81e73b933293")
decode_numpy_dict Not yet run LAMDASZ-ML/ChinaTravel/chinatravel/agent/utils.py
code served (permissive licence) · get_code("9d3045c07c4343e4")
get_funcname_by_preference Not yet run LAMDASZ-ML/ChinaTravel/chinatravel/evaluation/rank.py
code served (permissive licence) · get_code("1eb3756d467b8dbc")
get_rank_with_value Not yet run LAMDASZ-ML/ChinaTravel/chinatravel/evaluation/rank.py
code served (permissive licence) · get_code("9da02eac7da26244")
is_jsonable Not yet run LAMDASZ-ML/ChinaTravel/chinatravel/agent/base.py
code served (permissive licence) · get_code("e41eb14033474ba5")
load_json_file Not yet run LAMDASZ-ML/ChinaTravel/chinatravel/agent/utils.py
code served (permissive licence) · get_code("71db62c521e435b8")
load_query Not yet run LAMDASZ-ML/ChinaTravel/chinatravel/evaluation/rank.py
code served (permissive licence) · get_code("8331cd223183af27")
normalize_lang Not yet run LAMDASZ-ML/ChinaTravel/chinatravel/environment/language.py
code served (permissive licence) · get_code("5a627af2661e5732")
normalize_run_name Not yet run LAMDASZ-ML/ChinaTravel/chinatravel/agent/load_model.py
code served (permissive licence) · get_code("b79e07c34e67d9bf")
resolve_agent_llm_name Not yet run LAMDASZ-ML/ChinaTravel/chinatravel/agent/load_model.py
code served (permissive licence) · get_code("f52571c17760df0c")
resolve_llm_name Not yet run LAMDASZ-ML/ChinaTravel/chinatravel/agent/load_model.py
code served (permissive licence) · get_code("bb6cbda96b51d163")

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Travel planning stands out among real-world applications of Language Agents because it couples significant practical demand with a rigorous constraint-satisfaction challenge. However, existing benchmarks primarily operate on a slot-filling paradigm, restricting agents to synthetic queries with pre-defined constraint menus, which fails to capture the open-ended nature of natural language interaction, where user requirements are compositional, diverse, and often implicitly expressed. To address this gap, we introduce ChinaTravel, with four key contributions: 1) a practical sandbox aligned with the multi-day, multi-POI travel planning, 2) a compositionally generalizable domain-specific language (DSL) for scalable evaluation, covering feasibility, constraint satisfaction, and preference comparison 3) an openended dataset that integrates diverse travel requirements and implicit intent from 1154 human participants, and 4) fine-grained analysis reveal the potential of neurosymbolic agents in travel planning, achieving a 37.0% constraint satisfaction rate on human queries, a 10× improvement over purely neural models, yet highlighting significant challenges in compositional generalization. Overall, ChinaTravel provides a foundation for advancing language agents through compositional constraint validation in complex, real-world planning scenarios.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2412.13682")
get_code_for_paper("2412.13682")
have("2412.13682")

Connect an agent — have() is free.