Video Diffusion Models
07 Apr 2022
TL;DR: The authors proposed a diffusion model for video generation, which is a natural extension of the standard image diffusion architecture and enables jointly training from image and video data, which they find to reduce the variance of minibatch gradients and speed up optimization.
read more
Abstract: Generating temporally coherent high fidelity video is an important milestone in generative modeling research. We make progress towards this milestone by proposing a diffusion model for video generation that shows very promising initial results. Our model is a natural extension of the standard image diffusion architecture, and it enables jointly training from image and video data, which we find to reduce the variance of minibatch gradients and speed up optimization. To generate long and higher resolution videos we introduce a new conditional sampling technique for spatial and temporal video extension that performs better than previously proposed methods. We present the first results on a large text-conditioned video generation task, as well as state-of-the-art results on established benchmarks for video prediction and unconditional video generation. Supplementary material is available at https://video-diffusion.github.io/
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
DiffusionDet: Diffusion Model for Object Detection
Shoufa Chen,Peize Sun,Yibing Song,Ping Luo +3 more
- 01 Oct 2023
TL;DR: DiffusionDet is a novel object detection framework based on a diffusion process, achieving competitive performance with flexibility in the number of boxes and iterations.
176
A Survey on Generative Diffusion Models
Hanqun Cao,Cheng Tan,Zhan Gao,Yilun Xu,Guangyong Chen,Pheng-Ann Heng,Stan Z. Li +6 more
TL;DR: This survey comprehensively elucidates the fundamental formulation of diffusion, algorithmic enhancements, and the manifold applications of diffusion from three distinct angles: the fundamental formulation of diffusion, algorithmic enhancements, and the manifold applications of diffusion.
145
All are Worth Words: A ViT Backbone for Diffusion Models
Fan Bao,Shu Nie,Xue Ke-min,Yue Cao,Chongxuan Li,Hua Su,Jing Zhu +6 more
- 01 Jun 2023
TL;DR: U-ViT is a ViT-based architecture for diffusion models that achieves comparable or superior performance to a CNN-based U-Net. It utilizes long skip connections between shallow and deep layers and treats all inputs as tokens.
66
DiffRF: Rendering-Guided 3D Radiance Field Diffusion
Norman Müller,Yawar Siddiqui,Lorenzo Porzi,Samuel Rota Bulò,Peter Kontschieder,Matthias Nießner +5 more
- 01 Jun 2023
TL;DR: DiffRF is a novel approach for 3D radiance field synthesis based on denoising diffusion probabilistic models. It directly generates volumetric radiance fields from posed images, learns multi-view consistent priors, and enables free-view synthesis and accurate shape generation.
63
Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Services
Minrui Xu,Hongyang Du,Dusit Niyato,Jiawen Kang,Zehui Xiong,Shiwen Mao,Zhu Han,Abbas Jamalipour,Dong In Kim,Xuemin Shen,Victor C. M. Leung,H. Vincent Poor +11 more
TL;DR: AIGC services deployed at mobile edge networks provide personalized and customized AIGC services in real time while maintaining user privacy.
61
References
PixelSNAIL: an Improved Autoregressive Generative Model.
Xi Chen,Nikhil Mishra,Mostafa Rohaninejad,Pieter Abbeel +3 more
TL;DR: Researchers introduce PixelSNAIL, an improved autoregressive generative model combining causal convolutions with self-attention, achieving state-of-the-art log-likelihood results on CIFAR-10 and ImageNet, outperforming previous models with 2.85 and 3.80 bits per dim, respectively.
Adversarial Video Generation on Complex Datasets
Aidan Clark,Jeff Donahue,Karen Simonyan +2 more
TL;DR: This study presents a large-scale Generative Adversarial Network (GAN) model, DVD-GAN, that generates high-fidelity video samples on complex datasets, achieving state-of-the-art results in video synthesis and prediction tasks, particularly on Kinetics-600 and UCF-101 datasets.
Latent Neural Differential Equations for Video Generation
Cade Gordon,Natalie Parde +1 more
TL;DR: This study introduces Latent Neural Differential Equations for video generation, leveraging their continuous time representation to improve quality and efficiency, achieving a new state-of-the-art Inception Score of 15.20 in 64x64 pixel unconditional video generation.
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models.
Alex Nichol,Prafulla Dhariwal,Aditya Ramesh,Pranav Shyam,Pamela Mishkin,Bob McGrew,Ilya Sutskever,Mark Chen +7 more
TL;DR: Researchers develop GLIDE, a text-guided diffusion model for photorealistic image generation and editing, outperforming DALL-E in human evaluations and demonstrating fine-tuning capabilities for image inpainting and text-driven editing.
UCF101: A Dataset of 101 Human Actions Classes from Videos in the Wild
Khurram Soomro,Amir Roshan Zamir,Mubarak Shah +2 more
TL;DR: UCF101 is a large-scale human action dataset with 101 classes, 13k clips, and 27 hours of video data, featuring realistic user-uploaded videos with camera motion and cluttered backgrounds, posing a challenging task for action recognition.
Related Papers (5)
Chi-Farn Chen,Min-Hsin Chen +1 more
- 10 Dec 2006
Kamlesh Vrma,Debashis Ghosh,Avnish Kumar +2 more
- 01 Nov 2018