Shangchen Zhou, Ziwei Liu, Yunhai Tong, Kai Chen, Chen Loy, Xiangtai Li, Jingkang Yang, Yining Li, Jiangning Zhang, Jianzong Wu, Chenyang Si
We lifted 10 functions out of this paper's own repositories and ran 7 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| jianzongwu/Language-Driven-Video-Inpainting | canonical | 7 of 10 |
| Function | Status | Where it lives |
|---|---|---|
| create_1d_absolute_sin_cos_embedding | Ran | jianzongwu/Language-Driven-Video-Inpainting/rovi/models/utils.py pointer only (licence: NONE) · get_code("2eb56328ee222a96") |
| get_cosine_scheduler | Ran | jianzongwu/Language-Driven-Video-Inpainting/rovi/util.py pointer only (licence: NONE) · get_code("d38dd5fa6084deeb") |
| get_mask_loss_weight_scale_quadratic | Ran | jianzongwu/Language-Driven-Video-Inpainting/rovi/util.py pointer only (licence: NONE) · get_code("a37c74cff74be0d0") |
| get_unet_constant_deocder_cosine_scheduler | Ran | jianzongwu/Language-Driven-Video-Inpainting/rovi/util.py pointer only (licence: NONE) · get_code("5ca301a24eda4247") |
| inflated_interpolate_3d | Ran | jianzongwu/Language-Driven-Video-Inpainting/rovi/models/mask_decoder.py pointer only (licence: NONE) · get_code("1b457a7bd99a3de9") |
| load_video | Ran | jianzongwu/Language-Driven-Video-Inpainting/inference_interactive.py pointer only (licence: NONE) · get_code("a612419f9fdb01ba") |
| transform_first_frame | Ran | jianzongwu/Language-Driven-Video-Inpainting/inference_interactive.py pointer only (licence: NONE) · get_code("0b8d172f39477909") |
| get_down_block | Not yet run | jianzongwu/Language-Driven-Video-Inpainting/rovi/models/unet_blocks.py pointer only (licence: NONE) · get_code("f6cbe1f5a30557f3") |
| get_mask_decoder | Not yet run | jianzongwu/Language-Driven-Video-Inpainting/rovi/models/mask_decoder.py pointer only (licence: NONE) · get_code("45965320324d4cff") |
| get_up_block | Not yet run | jianzongwu/Language-Driven-Video-Inpainting/rovi/models/unet_blocks.py pointer only (licence: NONE) · get_code("b371191d08e93feb") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
We introduce a new task -language-driven video inpainting, which uses natural language instructions to guide the inpainting process. This approach overcomes the limitations of traditional video inpainting methods that depend on manually labeled binary masks, a process often tedious and labor-intensive. We present the Remove Objects from Videos by Instructions (ROVI) dataset, containing 5,650 videos and 9,091 inpainting results, to support training and evaluation for this task. We also propose a novel diffusion-based language-driven video inpainting framework, the first end-to-end baseline for this task, integrating Multimodal Large Language Models to understand and execute complex language-based inpainting requests effectively. Our comprehensive results showcase the dataset's versatility and the model's effectiveness in various language-instructed inpainting scenarios. We have made datasets, code, and models publicly available at https://github.com/jianzongwu/Language-Driven-Video-Inpainting.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2401.10226")
get_code_for_paper("2401.10226")
have("2401.10226")
Connect an agent — have() is free.