We lifted 12 functions out of this paper's own repositories and ran 9 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| yxuansu/pandagpt | pwc_unofficial | 9 of 12 |
| Function | Status | Where it lives |
|---|---|---|
| apply_rotary_pos_emb | Ran | yxuansu/pandagpt/code/model/modeling_llama.py code served (permissive licence) · get_code("9b4dff79d5e6102c") |
| build_one_instance | Ran | yxuansu/pandagpt/code/model/openllama.py code served (permissive licence) · get_code("31c4012c092a79ed") |
| cast_if_src_dtype | Ran | yxuansu/pandagpt/code/model/ImageBind/models/helpers.py code served (permissive licence) · get_code("751b1c8ccbe160c4") |
| get_clip_timepoints | Ran | yxuansu/pandagpt/code/model/ImageBind/data.py code served (permissive licence) · get_code("432c312da80537d6") |
| get_sinusoid_encoding_table | Ran | yxuansu/pandagpt/code/model/ImageBind/models/multimodal_preprocessors.py code served (permissive licence) · get_code("35a743e50e17e66d") |
| load_and_transform_vision_data | Ran | yxuansu/pandagpt/code/model/ImageBind/data.py code served (permissive licence) · get_code("e1e69d9ba9ffcb24") |
| parse_text | Ran | yxuansu/pandagpt/code/web_demo.py code served (permissive licence) · get_code("826ceb601fb31250") |
| process_batch_instance | Ran | yxuansu/pandagpt/code/model/openllama.py code served (permissive licence) · get_code("bf50c12b15a26bba") |
| rotate_half | Ran | yxuansu/pandagpt/code/model/modeling_llama.py code served (permissive licence) · get_code("b99eea6376d1e212") |
| imagebind_huge | Not yet run | yxuansu/pandagpt/code/model/ImageBind/models/imagebind_model.py code served (permissive licence) · get_code("a22759b6e81ab6c6") |
| interpolate_pos_encoding | Not yet run | yxuansu/pandagpt/code/model/ImageBind/models/multimodal_preprocessors.py code served (permissive licence) · get_code("71227efe6a7df1d2") |
| interpolate_pos_encoding_2d | Not yet run | yxuansu/pandagpt/code/model/ImageBind/models/multimodal_preprocessors.py code served (permissive licence) · get_code("ddd2e52e6b0956b5") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
We present PandaGPT, an approach to emPower large lANguage moDels with visual and Auditory instruction-following capabilities. Our pilot experiments show that PandaGPT can perform complex tasks such as detailed image description generation, writing stories inspired by videos, and answering questions about audios. More interestingly, PandaGPT can take multimodal inputs simultaneously and compose their semantics naturally. For example, PandaGPT can connect how objects look in an image/video and how they sound in an audio. To do so, PandaGPT combines the multimodal encoders from ImageBind and the large language models from Vicuna. Notably, only aligned image-text pairs are required for the training of PandaGPT. Thanks to the strong capability of ImageBind in embedding data from different modalities into the same space, PandaGPT displays emergent, i.e. zero-shot, cross-modal behaviors for data other than image and text (e.g., video, audio, depth, thermal, and IMU). We hope that PandaGPT serves as an initial step toward building AGI that can perceive and understand inputs in different modalities holistically, as we humans do. Our project page is at https://panda-gpt.github.io/.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2305.16355")
get_code_for_paper("2305.16355")
have("2305.16355")
Connect an agent — have() is free.