We lifted 3 functions out of this paper's own repositories and ran 2 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| zzxslp/mm-navigator | canonical | 2 of 3 |
| Function | Status | Where it lives |
|---|---|---|
| load_image | Ran | zzxslp/mm-navigator/call_gpt4.py pointer only (licence: NONE) · get_code("4fc4c9e17e2de6f3") |
| load_screen | Ran | zzxslp/mm-navigator/call_gpt4.py pointer only (licence: NONE) · get_code("e283f80f5bfb7f7a") |
| run_api | Not yet run | zzxslp/mm-navigator/call_gpt4.py pointer only (licence: NONE) · get_code("7a18b1e110b39399") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
We present MM-Navigator, a GPT-4V-based agent for the smartphone graphical user interface (GUI) navigation task. MM-Navigator can interact with a smartphone screen as human users, and determine subsequent actions to fulfill given instructions. Our findings demonstrate that large multimodal models (LMMs), specifically GPT-4V, excel in zero-shot GUI navigation through its advanced screen interpretation, action reasoning, and precise action localization capabilities. We first benchmark MM-Navigator on our collected iOS screen dataset. According to human assessments, the system exhibited a 91\% accuracy rate in generating reasonable action descriptions and a 75\% accuracy rate in executing the correct actions for single-step instructions on iOS. Additionally, we evaluate the model on a subset of an Android screen navigation dataset, where the model outperforms previous GUI navigators in a zero-shot fashion. Our benchmark and detailed analyses aim to lay a robust groundwork for future research into the GUI navigation task. The project page is at https://github.com/zzxslp/MM-Navigator.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2311.07562")
get_code_for_paper("2311.07562")
have("2311.07562")
Connect an agent — have() is free.