Agrim Gupta, Keshigeyan Chandrasegaran, Manling Li, Jiajun Li, Zane Durante, Lea Hadzic, Taran Kota, Jimming He, Cristobal Eyzaguirre, Fei-Fei Hourvideo Edu
We lifted 16 functions out of this paper's own repositories and ran 1 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| keshik6/HourVideo | canonical | 1 of 16 |
| Function | Status | Where it lives |
|---|---|---|
| encode_image | Ran | keshik6/HourVideo/hourvideo/gpt4_utils.py code served (permissive licence) · get_code("f41cb1a19b154297") |
| check_caption_completion_status | Not yet run | keshik6/HourVideo/hourvideo/form_world_state_history.py code served (permissive licence) · get_code("d31b2cfa1a2eabc8") |
| convert_seconds_to_hms | Not yet run | keshik6/HourVideo/hourvideo/llm_utils.py code served (permissive licence) · get_code("aca03fd411a5b059") |
| correct_keys_in_list | Not yet run | keshik6/HourVideo/hourvideo/gpt4_utils.py code served (permissive licence) · get_code("8c8de8a6584771d2") |
| create_index_lists | Not yet run | keshik6/HourVideo/hourvideo/gpt4_captioner.py code served (permissive licence) · get_code("e06becb29bf68207") |
| extract_first_letter | Not yet run | keshik6/HourVideo/hourvideo/gemini_utils.py code served (permissive licence) · get_code("1a60d839f3eb71b4") |
| filter_intervals | Not yet run | keshik6/HourVideo/hourvideo/form_world_state_history.py code served (permissive licence) · get_code("65d9bfa83974ba96") |
| get_color | Not yet run | keshik6/HourVideo/hourvideo/hv_utils.py code served (permissive licence) · get_code("132f9da86c87ad0d") |
| get_openai_completion | Not yet run | keshik6/HourVideo/hourvideo/gpt4_captioner.py code served (permissive licence) · get_code("d37eb61fd6663794") |
| get_world_state_history | Not yet run | keshik6/HourVideo/hourvideo/form_world_state_history.py code served (permissive licence) · get_code("d21c0d0cbe00da28") |
| load_json | Not yet run | keshik6/HourVideo/hourvideo/hv_utils.py code served (permissive licence) · get_code("8ddb08051f41a1c4") |
| load_openai_client | Not yet run | keshik6/HourVideo/hourvideo/gpt4_utils.py code served (permissive licence) · get_code("8b1c14fc5481eb57") |
| load_yaml_files | Not yet run | keshik6/HourVideo/hourvideo/hv_utils.py code served (permissive licence) · get_code("6483832049cc55d0") |
| num_tokens_from_string | Not yet run | keshik6/HourVideo/hourvideo/llm_utils.py code served (permissive licence) · get_code("dfc109b0a442de2e") |
| preprocess_llm_output | Not yet run | keshik6/HourVideo/hourvideo/llm_utils.py code served (permissive licence) · get_code("5bf0c932374156bb") |
| timestamp_to_seconds | Not yet run | keshik6/HourVideo/hourvideo/gemini_utils.py code served (permissive licence) · get_code("6ad55351c4f42d8f") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
We present HourVideo, a benchmark dataset for hour-long video-language understanding. Our dataset consists of a novel task suite comprising summarization, perception (recall, tracking), visual reasoning (spatial, temporal, predictive, causal, counterfactual), and navigation (room-to-room, object retrieval) tasks. HourVideo includes 500 manually curated egocentric videos from the Ego4D dataset, spanning durations of 20 to 120 minutes, and features 12,976 high-quality, five-way multiple-choice questions. Benchmarking results reveal that multimodal models, including GPT-4 and LLaVA-NeXT, achieve marginal improvements over random chance. In stark contrast, human experts significantly outperform the state-of-the-art long-context multimodal model, Gemini Pro 1.5 (85.0% vs. 37.3%), highlighting a substantial gap in multimodal capabilities. Our benchmark, evaluation toolkit, prompts, and documentation are available at hourvideo.stanford.edu. In this work, we introduce HourVideo-a benchmark dataset designed for long-form video-language understanding. To design tasks that require long-term comprehension, we first propose a novel task Correspondence to
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2411.04998")
get_code_for_paper("2411.04998")
have("2411.04998")
Connect an agent — have() is free.