SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2207.11075 · ECCV · 2022

RealFlow: EM-based Realistic Optical Flow Dataset Generation from Videos

Ao Luo, Haoqiang Fan, Kunming Luo, Shuaicheng Liu, Yunhui Han, Jiangyu Liu, Guiming Luo

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Obtaining the ground truth labels from a video is challenging since the manual annotation of pixel-wise flow labels is prohibitively expensive and laborious. Besides, existing approaches try to adapt the trained model on synthetic datasets to authentic videos, which inevitably suffers from domain discrepancy and hinders the performance for realworld applications. To solve these problems, we propose RealFlow, an Expectation-Maximization based framework that can create large-scale optical flow datasets directly from any unlabeled realistic videos. Specifically, we first estimate optical flow between a pair of video frames, and then synthesize a new image from this pair based on the predicted flow. Thus the new image pairs and their corresponding flows can be regarded as a new training set. Besides, we design a Realistic Image Pair Rendering (RIPR) module that adopts softmax splatting and bi-directional hole filling techniques to alleviate the artifacts of the image synthesis. In the E-step, RIPR renders new images to create a large quantity of training data. In the M-step, we utilize the generated training data to train an optical flow network, which can be used to estimate optical flows in the next E-step. During the iterative learning steps, the capability of the flow network is gradually improved, so is the accuracy of the flow, as well as the quality of the synthesized dataset. Experimental results show that RealFlow outperforms previous dataset generation methods by a considerably large margin. Moreover, based on the generated dataset, our approach achieves state-of-the-art performance on two standard benchmarks compared with both supervised and unsupervised optical flow methods. Our code and dataset are available at https://github.com/megvii-research/RealFlow. Generate dataset using video sequences and a pre-trained CNN Video Sequence Realistic Image Pair Rendering Estimation Training using the generated dataset CNN Network Network Training Train CNN using the generated dataset Generate dataset using video sequences Loss

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2207.11075")
get_code_for_paper("2207.11075")
have("2207.11075")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2207.11075.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2207.11075.svg)](https://syntology.ai/paper/2207.11075)

Connect an agent — have() is free.