We lifted 4 functions out of this paper's own repositories and ran 4 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| mobilellm/autodroid | pwc_unofficial | 4 of 4 |
| Function | Status | Where it lives |
|---|---|---|
| get_id_from_view_desc | Ran | mobilellm/autodroid/tools.py code served (permissive licence) · get_code("2915bdcc0dd313fc") |
| get_view_without_id | Ran | mobilellm/autodroid/tools.py code served (permissive licence) · get_code("eaff9d242f5e9ce7") |
| insert_id_into_view | Ran | mobilellm/autodroid/tools.py code served (permissive licence) · get_code("874a827043f77340") |
| load_yaml_files | Ran | mobilellm/autodroid/walk.py code served (permissive licence) · get_code("dfad80aced080c6d") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Mobile task automation is an attractive technique that aims to enable voice-based hands-free user interaction with smartphones. However, existing approaches suffer from poor scalability due to the limited language understanding ability and the non-trivial manual efforts required from developers or end-users. The recent advance of large language models (LLMs) in language understanding and reasoning inspires us to rethink the problem from a model-centric perspective, where task preparation, comprehension, and execution are handled by a unified language model. In this work, we introduce AutoDroid, a mobile task automation system capable of handling arbitrary tasks on any Android application without manual efforts. The key insight is to combine the commonsense knowledge of LLMs and domain-specific knowledge of apps through automated dynamic analysis. The main components include a functionality-aware UI representation method that bridges the UI with the LLM, exploration-based memory injection techniques that augment the app-specific domain knowledge of LLM, and a multi-granularity query optimization module that reduces the cost of model inference. We integrate AutoDroid with off-the-shelf LLMs including online GPT-4/GPT-3.5 and on-device Vicuna, and evaluate its performance on a new benchmark for memory-augmented Android task automation with 158 common tasks. The results demonstrated that AutoDroid is able to precisely generate actions with an accuracy of 90.9%, and complete tasks with a success rate of 71.3%, outperforming the GPT-4-powered baselines by 36.4% and 39.7%. The demo, benchmark suites, and source code of AutoDroid will be released at url{https://autodroid-sys.github.io/}.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2308.15272")
get_code_for_paper("2308.15272")
have("2308.15272")
Connect an agent — have() is free.