Mohit Bansal, Jie Lei, Tamara Berg
We lifted 2 functions out of this paper's own repositories and ran 2 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| jayleicn/moment_detr | canonical | 1 of 1 |
| houzhijian/cone | — | 1 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| CONE | Ran | houzhijian/cone/cone/model.py pointer only (licence: NONE) · get_code("ab831e4d334aed83") |
| MomentDETR | Ran | jayleicn/moment_detr/moment_detr/model.py code served (permissive licence) · get_code("ed43d511a292c720") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Detecting customized moments and highlights from videos given natural language (NL) user queries is an important but under-studied topic. One of the challenges in pursuing this direction is the lack of annotated data. To address this issue, we present the Query-based Video Highlights (QVHIGHLIGHTS) dataset. It consists of over 10,000 YouTube videos, covering a wide range of topics, from everyday activities and travel in lifestyle vlog videos to social and political activities in news videos. Each video in the dataset is annotated with: (1) a human-written free-form NL query, (2) relevant moments in the video w.r.t. the query, and (3) five-point scale saliency scores for all query-relevant clips. This comprehensive annotation enables us to develop and evaluate systems that detect relevant moments as well as salient highlights for diverse, flexible user queries. We also present a strong baseline for this task, Moment-DETR, a transformer encoder-decoder model that views moment retrieval as a direct set prediction problem, taking extracted video and query representations as inputs and predicting moment coordinates and saliency scores end-to-end. While our model does not utilize any human prior, we show that it performs competitively when compared to well-engineered architectures. With weakly supervised pretraining using ASR captions, Moment-DETR substantially outperforms previous methods. Lastly, we present several ablations and visualizations of Moment-DETR. Data and code is publicly available at https://github.com/jayleicn/moment_detr.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2107.09609")
get_code_for_paper("2107.09609")
have("2107.09609")
Connect an agent — have() is free.