Eyhab Al-Masri
We lifted 3 functions out of this paper's own repositories and ran 3 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| aeris-lab/llmrank | canonical | 3 of 3 |
| Function | Status | Where it lives |
|---|---|---|
| average_overlap | Ran | aeris-lab/llmrank/aaai_lamas_study.ipynb.py pointer only (licence: NONE) · get_code("2b7f07565648511c") |
| jaccard_similarity | Ran | aeris-lab/llmrank/aaai_lamas_study.ipynb.py pointer only (licence: NONE) · get_code("9ffccdf789b0e482") |
| rbo_score | Ran | aeris-lab/llmrank/aaai_lamas_study.ipynb.py pointer only (licence: NONE) · get_code("fb7901043d51dd5a") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Large language models (LLMs) increasingly operate as autonomous agents that reason over external APIs to perform complex tasks. However, their reliability and agreement remain poorly characterized. We present a unified benchmarking framework to quantify inter-LLM divergence-the extent to which models differ in API discovery and ranking under identical tasks. Across 15 canonical API domains and 5 major model families, we measure pairwise and group-level agreement using set-, rank-, and consensus-based metrics: Average Overlap, Jaccard, Rank-Biased Overlap, Kendall's τ/W, and Cronbach's α. Results show moderate overall alignment (AO ≈ 0.50, τ ≈ 0.45) but strong domain dependence: structured tasks (Weather, Speech-to-Text) are stable, while open-ended ones (Sentiment Analysis) diverge sharply. Volatility and consensus analyses reveal that coherence clusters around data-bound domains and degrades for abstract reasoning. These insights enable reliability-aware orchestration in multiagent systems, where consensus weighting can improve coordination among heterogeneous LLMs. Beyond performance benchmarking, our results reveal systematic failure modes in multi-agent LLM coordination, where apparent agreement masks instability in action-relevant rankings. This hidden divergence poses a pre-deployment safety risk, motivating diagnostic benchmarks for early detection.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2604.22760")
get_code_for_paper("2604.22760")
have("2604.22760")
Connect an agent — have() is free.