Harsh Sharma, Prashant Devadiga, Chandra Bhushan, Keyur Doshi, Nitin Kukreja, Suvradip Paul, Yatharth Dedhia
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Banking assistants must use account-specific information to answer requests and, in many cases, take actions through tools. Evaluating only the final response misses important errors. An assistant may ask for information it already has, rely on stale context, select the wrong account, or write an invalid value after stating the correct one. We introduce IndicBankBench, a 799-case benchmark for Indian retail banking spanning five operational domains, a capability/refusal domain, and twenty primary axes. Cases are evaluated at four stages: safety, action and tool use, response adequacy, and advisory quality. Tool use and most safety checks are deterministic. A narrow resolver handles only ambiguous confirmation-before-write cases, while a separate LLM judge evaluates semantic response adequacy. We run every case three times and report strict pass^3, which requires success on all trials. Across the eleven evaluated models, strict reliability ranges from 43.7% to 58.2%, whereas at-least-once success ranges from 60% to 74%. This gap shows that at-least-once success can overstate dependable banking behavior. The case-level diagnostics also distinguish systems that ask unnecessary questions from those that act but fail to reconcile customer context or fully resolve the request. We release the cases, mock environment, and evaluation harness.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2609.29167")
get_code_for_paper("2609.29167")
have("2609.29167")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2609.29167.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2609.29167)
Connect an agent — have() is free.