Aliaksandr Siarohin, Sergey Tulyakov, Ivan Skorokhodov, Kaust, Willi Menapace
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. We have no repository linked to it either. That is a statement about our graph, not about the paper.
Diffusion models have demonstrated remarkable performance in image and video synthesis. However, scaling them to high-resolution inputs is challenging and requires restructuring the diffusion pipeline into multiple independent components, limiting scalability and complicating downstream applications. In this work, we study patch diffusion models (PDMs) -a diffusion paradigm which models the distribution of patches, rather than whole inputs, keeping up to ⇡0.7% of the original pixels. This makes it very efficient during training and unlocks end-to-end optimization on high-resolution videos. We improve PDMs in two principled ways. First, to enforce consistency between patches, we develop deep context fusion -an architectural technique that propagates the context information from low-scale to high-scale patches in a hierarchical manner. Second, to accelerate training and inference, we propose adaptive computation, which allocates more network capacity and computation towards coarse image details. The resulting model sets a new stateof-the-art FVD score of 66.32 and Inception Score of 87.68 in class-conditional video generation on UCF-101 256 2 , surpassing recent methods by more than 100%. Then, we show that it can be rapidly fine-tuned from a base 36 ⇥ 64 low-resolution generator for high-resolution 64⇥288⇥512 text-to-video synthesis. To the best of our knowledge, our model is the first diffusion-based architecture which is trained on such high resolutions entirely end-to-end. Project webpage: https://snap-research.github.io/hpdm.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2406.07792")
get_code_for_paper("2406.07792")
have("2406.07792")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2406.07792.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2406.07792)
Connect an agent — have() is free.