Cees Snoek, Piyush Bagad, Makarand Tapaswi
We lifted 10 functions out of this paper's own repositories and ran 10 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| bpiyush/TestOfTime | canonical | 9 of 9 |
| jayleicn/singularity | — | 1 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| active_event_description_combiner | Ran | bpiyush/TestOfTime/package/datasets/synthetic.py code served (permissive licence) · get_code("d9fa3afc6393fd8c") |
| average_precision | Ran | bpiyush/TestOfTime/package/evaluators/videoclip_action_retrieval.py code served (permissive licence) · get_code("deacd11293b2686f") |
| check_args | Ran | bpiyush/TestOfTime/postpretrain.py code served (permissive licence) · get_code("f2d84c0542d17a04") |
| compute_metrics | Ran | bpiyush/TestOfTime/package/metrics/retrieval.py code served (permissive licence) · get_code("b717eef2896560b3") |
| freeze_required_layers | Ran | bpiyush/TestOfTime/postpretrain.py code served (permissive licence) · get_code("5c70504364bdc3ac") |
| interpolate_temporal_pos_embed | Ran | jayleicn/singularity/models/utils.py code served (permissive licence) · get_code("f64c56c357505158") |
| mean_average_precision | Ran | bpiyush/TestOfTime/package/evaluators/videoclip_action_retrieval.py code served (permissive licence) · get_code("0196aa2f7fc22400") |
| passive_event_description_combiner | Ran | bpiyush/TestOfTime/package/datasets/synthetic.py code served (permissive licence) · get_code("053b2f9dfa05e107") |
| remove_multiple_spaces | Ran | bpiyush/TestOfTime/package/datasets/synthetic.py code served (permissive licence) · get_code("9fbb5b9290a438a0") |
| update_config | Ran | bpiyush/TestOfTime/postpretrain.py code served (permissive licence) · get_code("149908a60cbc536a") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Modelling and understanding time remains a challenge in contemporary video understanding models. With language emerging as a key driver towards powerful generalization, it is imperative for foundational video-language models to have a sense of time. In this paper, we consider a specific aspect of temporal understanding: consistency of time order as elicited by before/after relations. We establish that seven existing video-language models struggle to understand even such simple temporal relations. We then question whether it is feasible to equip these foundational models with temporal awareness without re-training them from scratch. Towards this, we propose a temporal adaptation recipe on top of one such model, VideoCLIP, based on post-pretraining on a small amount of video-text data. We conduct a zero-shot evaluation of the adapted models on six datasets for three downstream tasks which require varying degrees of time awareness. We observe encouraging performance gains especially when the task needs higher time awareness. Our work serves as a first step towards probing and instilling a sense of time in existing videolanguage models without the need for data and computeintense training from scratch.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2301.02074")
get_code_for_paper("2301.02074")
have("2301.02074")
Connect an agent — have() is free.