Video Diffusion Models
07 Apr 2022
TL;DR: The authors proposed a diffusion model for video generation, which is a natural extension of the standard image diffusion architecture and enables jointly training from image and video data, which they find to reduce the variance of minibatch gradients and speed up optimization.
read more
Abstract: Generating temporally coherent high fidelity video is an important milestone in generative modeling research. We make progress towards this milestone by proposing a diffusion model for video generation that shows very promising initial results. Our model is a natural extension of the standard image diffusion architecture, and it enables jointly training from image and video data, which we find to reduce the variance of minibatch gradients and speed up optimization. To generate long and higher resolution videos we introduce a new conditional sampling technique for spatial and temporal video extension that performs better than previously proposed methods. We present the first results on a large text-conditioned video generation task, as well as state-of-the-art results on established benchmarks for video prediction and unconditional video generation. Supplementary material is available at https://video-diffusion.github.io/
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
Diffusion Models in Vision: A Survey
TL;DR: Denoising diffusion models represent a recent emerging topic in computer vision, demonstrating remarkable results in the area of generative modeling as discussed by the authors , and are widely appreciated for the quality and diversity of the generated samples, despite their known computational burdens.
Diffusion Models: A Comprehensive Survey of Methods and Applications
Lu Yang,Zhilong Zhang,Yang Song,Shenda Hong,Runsheng Xu,Yang Zhao,Wentao Zhang,Bin Cui,Ming–Hsuan Yang +8 more
TL;DR: Diffusion models are a powerful family of generative models for image, video, and molecule generation with record-breaking performance. This survey categorizes the research into sampling, likelihood estimation, and data handling with special structures. It also discusses potential combinations with other generative models and applications in various fields.
478
InstructPix2Pix: Learning to Follow Image Editing Instructions
Tim Brooks,Aleksander Holynski,Alexei A. Efros +2 more
- 01 Jun 2023
TL;DR: InstructPix2Pix learns to edit images from human instructions by generating a large dataset of image editing examples and training a conditional diffusion model on it.
458
Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models
Andreas Blattmann,Robin Rombach,Huan Ling,Tim Dockhorn,Seung Wook Kim,Sanja Fidler,Karsten Kreis +6 more
- 01 Jun 2023
TL;DR: High-resolution video synthesis with latent diffusion models enables high-quality video generation while reducing compute demands.
226
Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation
Jay Zhangjie Wu,Yixiao Ge,Xintao Wang,Stan Weixian Lei,Yuchao Gu,Yufei Shi,Wynne Hsu,Ying Shi,Xiaohu Qie,Mike Zheng Shou +9 more
- 01 Oct 2023
TL;DR: Tune-A-Video enables one-shot tuning of image diffusion models for T2V generation, leveraging pre-trained T2I models and introducing a tailored spatio-temporal attention mechanism.
176
References
Scaling Autoregressive Video Models
Dirk Weissenborn,Oscar Täckström,Jakob Uszkoreit +2 more
TL;DR: This study presents a conceptually simple autoregressive video generation model using 3D self-attention, achieving competitive results on benchmark datasets with high fidelity and realism, and demonstrating potential for modeling complex phenomena in large-scale datasets like Kinetics.
Deep Unsupervised Learning Using Nonequilibrium Thermodynamics
Jascha Sohl-Dickstein,Eric A. Weiss,Niru Maheswaranathan,Surya Ganguli +3 more
TL;DR: Researchers develop a deep unsupervised learning approach inspired by non-equilibrium statistical physics, enabling flexible and tractable generative models with thousands of layers, rapid learning, sampling, and evaluation, and open-source implementation.
Diffusion Models Beat GANs on Image Synthesis
Prafulla Dhariwal,Alexander Quinn Nichol +1 more
TL;DR: Diffusion models outperform GANs in image synthesis, achieving superior sample quality through architecture ablation and classifier guidance, with FID scores of 2.97-7.72 on ImageNet, and matching BigGAN-deep with fewer forward passes.
DiffWave: A Versatile Diffusion Model for Audio Synthesis.
Zhifeng Kong,Wei Ping,Jiaji Huang,Kexin Zhao,Bryan Catanzaro +4 more
TL;DR: DiffWave is a non-autoregressive diffusion model for audio synthesis, producing high-fidelity audios in various tasks, outperforming WaveNet vocoder in speech quality and outperforming autoregressive and GAN-based models in unconditional generation.
Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling.
Jacob Menick,Nal Kalchbrenner +1 more
TL;DR: Researchers propose Subscale Pixel Networks (SPNs) and Multidimensional Upscaling to generate high-fidelity images, addressing challenges in encoding context and preserving detail. They achieve state-of-the-art results on CelebAHQ and ImageNet datasets, setting new benchmarks in unconditional image generation.
Related Papers (5)
Chi-Farn Chen,Min-Hsin Chen +1 more
- 10 Dec 2006
Kamlesh Vrma,Debashis Ghosh,Avnish Kumar +2 more
- 01 Nov 2018