SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2303.01237 · CVPR · 2023

FlowFormer++: Masked Cost Volume Autoencoding for Pretraining Optical Flow Estimation

Hongsheng Li, Jifeng Dai, Hongwei Qin, Zhaoyang Huang, Simon See, Ka Cheung, Xiaoyu Shi, Manyuan Zhang, Dasong Li

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

FlowFormer [24] introduces a transformer architecture into optical flow estimation and achieves state-of-the-art performance. The core component of FlowFormer is the transformer-based cost-volume encoder. Inspired by the recent success of masked autoencoding (MAE) pretraining in unleashing transformers' capacity of encoding visual representation, we propose Masked Cost Volume Autoencoding (MCVA) to enhance FlowFormer by pretraining the cost-volume encoder with a novel MAE scheme. Firstly, we introduce a block-sharing masking strategy to prevent masked information leakage, as the cost maps of neighboring source pixels are highly correlated. Secondly, we propose a novel pre-text reconstruction task, which encourages the cost-volume encoder to aggregate long-range information and ensures pretraining-finetuning consistency. We also show how to modify the FlowFormer architecture to accommodate masks during pretraining. Pretrained with MCVA, FlowFormer++ ranks 1st among published methods on both Sintel and KITTI-2015 benchmarks. Specifically, FlowFormer++ achieves 1.07 and 1.94 average endpoint error (AEPE) on the clean and final pass of Sintel benchmark, leading to 7.76% and 7.18% error reductions from FlowFormer. FlowFormer++ obtains 4.52 F1-all on the KITTI-2015 test set, improving FlowFormer by 0.16. * Xiaoyu Shi and Zhaoyang Huang assert equal contributions. Encoder Flow Head Recon. Head Pretraining dataflow Finetuning dataflow Transformer Cost-Volume Encoder Block-sharing Masks Reconstructed Cost Patch Pretrained modules Dropped in finetuning Frozen in pretraining Cost Volume

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2303.01237")
get_code_for_paper("2303.01237")
have("2303.01237")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2303.01237.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2303.01237.svg)](https://syntology.ai/paper/2303.01237)

Connect an agent — have() is free.