We lifted 18 functions out of this paper's own repositories and ran 13 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| xiaomi/mobilevlm | canonical | 13 of 18 |
| Function | Status | Where it lives |
|---|---|---|
| calculate_iou | Ran | xiaomi/mobilevlm/test_515/count_aver_iou.py pointer only (licence: NOASSERTION) · get_code("3aac5dfd1627a360") |
| dict2dict | Ran | xiaomi/mobilevlm/corpus/list_prefix.py pointer only (licence: NOASSERTION) · get_code("e4b5d64062a91b7f") |
| dict2txt | Ran | xiaomi/mobilevlm/corpus/list_prefix.py pointer only (licence: NOASSERTION) · get_code("48c980f8b78480c1") |
| dictcount | Ran | xiaomi/mobilevlm/corpus/list_prefix.py pointer only (licence: NOASSERTION) · get_code("b2e1feaa54352576") |
| gpt_usage | Ran | xiaomi/mobilevlm/corpus/chatgpt.py pointer only (licence: NOASSERTION) · get_code("e479638f07e3d5f2") |
| load_dict | Ran | xiaomi/mobilevlm/corpus/bm25.py pointer only (licence: NOASSERTION) · get_code("e0f0dbdcd40e0123") |
| load_json | Ran | xiaomi/mobilevlm/corpus/bm25.py pointer only (licence: NOASSERTION) · get_code("1c84aeb1de479bcc") |
| load_txt | Ran | xiaomi/mobilevlm/corpus/bm25.py pointer only (licence: NOASSERTION) · get_code("1aa5d2a1084ca8a3") |
| local_image_to_data_url | Ran | xiaomi/mobilevlm/test_515/gpt.py pointer only (licence: NOASSERTION) · get_code("829b3a29017bf6be") |
| preprocess | Ran | xiaomi/mobilevlm/finetune.py pointer only (licence: NONE) · get_code("8b733c9fc590e4c1") |
| process_html | Ran | xiaomi/mobilevlm/test_515/build_test_inout_put.py pointer only (licence: NOASSERTION) · get_code("d4b2c3b54965a829") |
| read_json_file | Ran | xiaomi/mobilevlm/corpus/xml_test_22.py pointer only (licence: NOASSERTION) · get_code("707a0ecb9f3c6f2d") |
| scale_coordinates | Ran | xiaomi/mobilevlm/test_515/count_aver_iou.py pointer only (licence: NOASSERTION) · get_code("a8686802123b33d2") |
| any_tree_to_html | Not yet run | xiaomi/mobilevlm/xml_to_html.py pointer only (licence: NOASSERTION) · get_code("2f9f1293699146bc") |
| chatgpt | Not yet run | xiaomi/mobilevlm/corpus/chatgpt.py pointer only (licence: NOASSERTION) · get_code("ff85b6fede884ce7") |
| extract_coordinates | Not yet run | xiaomi/mobilevlm/test_515/count_aver_iou.py pointer only (licence: NOASSERTION) · get_code("7261ce7a4e26085e") |
| gpt_4v_actionspace | Not yet run | xiaomi/mobilevlm/test_515/gpt.py pointer only (licence: NOASSERTION) · get_code("b6659c22b1479358") |
| gpt_4v_ref | Not yet run | xiaomi/mobilevlm/test_515/gpt.py pointer only (licence: NOASSERTION) · get_code("e513b28efe0be636") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Recently, mobile AI agents based on VLMs have been gaining increasing attention. These works typically utilize VLM as a foundation, fine-tuning it with instruction-based mobile datasets. However, these VLMs are typically pre-trained on general-domain data, which often results in a lack of fundamental capabilities specific to the mobile domain. Therefore, they may struggle to recognize specific UI elements and understand intra-UI fine-grained information. In addition, the current fine-tuning task focuses on interacting with the most relevant element for the given instruction. These fine-tuned VLMs may still ignore the relationships between UI pages, neglect the roles of elements in page transitions and lack inter-UI understanding. To address issues, we propose a VLM called MobileVLM, which includes two additional pre-training stages to enhance both intra- and inter-UI understanding. We defined four UI-based pre-training tasks, enabling the model to better perceive fine-grained elements and capture page transition actions. To address the lack of mobile pre-training data, we built a large Chinese mobile dataset Mobile3M from scratch, which contains 3 million UI pages, and real-world transition actions, forming a directed graph structure. Experimental results show MobileVLM excels on both our test set and public mobile benchmarks, outperforming existing VLMs.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2409.14818")
get_code_for_paper("2409.14818")
have("2409.14818")
Connect an agent — have() is free.