Video synthesis has recently made remarkable strides benefiting from the rapid development of diffusion models. However, it still encounters challenges in terms of semantic accuracy, clarity and spatio-temporal continuity. They primarily arise from the scarcity of well-aligned text-video data and the complex inherent structure of videos, making it difficult for the model to simultaneously ensure semantic and qualitative excellence. In this report, we propose a cascaded I2VGen-XL approach that enhances model performance by decoupling these two factors and ensures the alignment of the input data by utilizing static images as a form of crucial guidance. I2VGen-XL consists of two stages: i) the base stage guarantees coherent semantics and preserves content from input images by using two hierarchical encoders, and ii) the refinement stage enhances the video's details by incorporating an additional brief text and improves the resolution to 1280$\times$720. To improve the diversity, we collect around 35 million single-shot text-video pairs and 6 billion text-image pairs to optimize the model. By this means, I2VGen-XL can simultaneously enhance the semantic accuracy, continuity of details and clarity of generated videos. Through extensive experiments, we have investigated the underlying principles of I2VGen-XL and compared it with current top methods, which can demonstrate its effectiveness on diverse data. The source code and models will be publicly available at \url{https://i2vgen-xl.github.io}.

I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models

Knowledge graph completion (KGC) can predict missing links and is crucial for real-world knowledge graphs, which widely suffer from incompleteness. KGC methods assume a knowledge graph is static, but that may lead to inaccurate prediction results because many facts in the knowledge graphs change over time. Recently, emerging methods have shown improved predictive results by further incorporating the timestamps of facts; namely, temporal knowledge graph completion (TKGC). With this temporal information, TKGC methods can learn the dynamic evolution of the knowledge graph that KGC methods fail to capture. In this paper, for the first time, we summarize the recent advances in TKGC research. First, we detail the background of TKGC, including the problem definition, benchmark datasets, and evaluation metrics. Then, we summarize existing TKGC methods based on how timestamps of facts are used to capture the temporal dynamics. Finally, we conclude the paper and present future research directions of TKGC.

Temporal Knowledge Graph Completion: A Survey

The recent wave of AI-generated content (AIGC) has witnessed substantial success in computer vision, with the diffusion model playing a crucial role in this achievement. Due to their impressive generative capabilities, diffusion models are gradually superseding methods based on GANs and auto-regressive Transformers, demonstrating exceptional performance not only in image generation and editing, but also in the realm of video-related research. However, existing surveys mainly focus on diffusion models in the context of image generation, with few up-to-date reviews on their application in the video domain. To address this gap, this paper presents a comprehensive review of video diffusion models in the AIGC era. Specifically, we begin with a concise introduction to the fundamentals and evolution of diffusion models. Subsequently, we present an overview of research on diffusion models in the video domain, categorizing the work into three key areas: video generation, video editing, and other video understanding tasks. We conduct a thorough review of the literature in these three key areas, including further categorization and practical contributions in the field. Finally, we discuss the challenges faced by research in this domain and outline potential future developmental trends. A comprehensive list of video diffusion models studied in this survey is available at https://github.com/ChenHsing/Awesome-Video-Diffusion-Models.

pdf/a-survey-on-video-diffusion-models-3ttxdju9ej.pdf

A Survey on Video Diffusion Models

GraIL and its variants have shown their promising capacities for inductive relation reasoning on knowledge graphs. However, the uni-directional message-passing mechanism hinders such models from exploiting hidden mutual relations between entities in directed graphs. Besides, the enclosing subgraph extraction in most GraIL-based models restricts the model from extracting enough discriminative information for reasoning. Consequently, the expressive ability of these models is limited. To address the problems, we propose a novel GraIL-based framework, termed MINES, by introducing a Message Intercommunication mechanism on the Neighbor-Enhanced Subgraph. Concretely, the message intercommunication mechanism is designed to capture the omitted hidden mutual information. It introduces bi-directed information interactions between connected entities by inserting an undirected/bi-directed GCN layer between uni-directed RGCN layers. Moreover, inspired by the success of involving more neighbors in other graph-based tasks, we extend the neighborhood area beyond the enclosing subgraph to enhance the information collection for inductive relation reasoning. Extensive experiments prove the promising capacity of the proposed MINES from various aspects, especially for the superiority, effectiveness, and transfer ability. 

MINES: Message Intercommunication for Inductive Relation Reasoning over Neighbor-Enhanced Subgraphs

Recently, POI (Point-of-interest) recommendation for groups has become a critical challenge when helping groups to discover potentially interesting new places, and some effective recommendation models have been proposed to address this issue. However, most existing research focuses on POI recommendation for fixed groups, few studies have been conducted on POI recommendation for occasional groups. To tackle this issue, we propose a POI recommendation model for occasional groups based on Hybrid Graph Neural Networks (termed as PROG-HGNN) which combines excellent graph neural networks models. Firstly, PROG-HGNN generates the fitted representation of the occasional group based on the Node Influence Indicator (INF) method and Graph Attention Networks (GAT) model. Then, PROG-HGNN learns POIs’ representations containing members’ POI interaction preferences and members’ POI transfer preferences with the Signed Bipartite Graph Neural Networks (SBGNN) model and the Session-based Graph Neural Networks (SRGNN) model, respectively. Finally, PROG-HGNN recommends the potential POIs for the occasional group based on the fitted representation of the occasional group and the learned representations of POIs. We verify our proposed model on three public benchmark datasets (Foursquare, Gowalla and Yelp), which contains 124,933 to 860,888 POI check-in records. The comparison between our proposed model and the twelve baseline models demonstrates the outstanding performance of PROG-HGNN. In terms of Precision@K and Recall@K, our model achieves about 32.92% and 19.67% improvement compared with the best baseline models on the three benchmark datasets averagely. Adequate ablation experiments prove the effectiveness of the members’ POI interaction preferences learning module and POI transfer preferences learning module. 

POI recommendation for occasional groups Based on hybrid graph neural networks

Next Point-of-Interest (POI) recommendation has shown great value for both users and businesses in the field of location-based services. Many spatial–temporal inferring methods have been developed to perform this task, but the data sparsity of POI trajectories greatly hinders the recommendation performance. Knowledge graphs (KGs) have been demonstrated as an effective way to tackle data sparsity in the general recommendation field by leveraging the valuable information of entities and relations. Yet, few studies have explored applying KGs for the next POI recommendation task because of the following challenges: (1) how to represent the dynamic mobility behaviors of users with the static entities and relations in KGs; and (2) how to utilize the different types of entities and relations in KGs to capture long- and short-term preferences of users. In this work, we investigate building a spatial–temporal KG (STKG) from check-in sequences of users to promote the next POI recommendation, without introducing any external attributes of users and POIs. In STKG, we design a novel spatial–temporal transfer relation to intuitively capture users’ transition patterns between neighboring POIs. Then, based on the STKG, we propose an innovative model, named STKGRec, for the next POI recommendation, which explicitly models long- and short-term preferences of users in an end-to-end manner. In particular, STKGRec learns both the spatial–temporal correlation of consecutive and nonconsecutive visits in the current check-in sequence to comprehensively capture the short-term preferences of users. Extensive experiments on four real-world datasets demonstrate the superiority of STKGRec against the state-of-the-art baseline methods. The code of our proposed model is available at https://github.com/WeiChen3690/STKGRec . 

Building and exploiting spatial-temporal knowledge graph for next POI recommendation

In this study, we present an efficient and effective approach for achieving temporally consistent synthetic-to-real video translation in videos of varying lengths. Our method leverages off-the-shelf conditional image diffusion models, allowing us to perform multiple synthetic-to-real image generations in parallel. By utilizing the available optical flow information from the synthetic videos, our approach seamlessly enforces temporal consistency among corresponding pixels across frames. This is achieved through joint noise optimization, effectively minimizing spatial and temporal discrepancies. To the best of our knowledge, our proposed method is the first to accomplish diverse and temporally consistent synthetic-to-real video translation using conditional image diffusion models. Furthermore, our approach does not require any training or fine-tuning of the diffusion models. Extensive experiments conducted on various benchmarks for synthetic-to-real video translation demonstrate the effectiveness of our approach, both quantitatively and qualitatively. Finally, we show that our method outperforms other baseline methods in terms of both temporal consistency and visual quality.

Video ControlNet: Towards Temporally Consistent Synthetic-to-Real Video Translation Using Conditional Image Diffusion Models

This study introduces an efficient and effective method, MeDM, that utilizes pre-trained image Diffusion Models for video-to-video translation with consistent temporal flow. The proposed framework can render videos from scene position information, such as a normal G-buffer, or perform text-guided editing on videos captured in real-world scenarios. We employ explicit optical flows to construct a practical coding that enforces physical constraints on generated frames and mediates independent frame-wise scores. By leveraging this coding, maintaining temporal consistency in the generated videos can be framed as an optimization problem with a closed-form solution. To ensure compatibility with Stable Diffusion, we also suggest a workaround for modifying observed-space scores in latent-space Diffusion Models. Notably, MeDM does not require fine-tuning or test-time optimization of the Diffusion Models. Through extensive qualitative, quantitative, and subjective experiments on various benchmarks, the study demonstrates the effectiveness and superiority of the proposed approach. Project page can be found at https://medm2023.github.io

pdf/medm-mediating-image-diffusion-models-for-video-to-video-4b8ro02bcm.pdf

MeDM: Mediating Image Diffusion Models for Video-to-Video Translation with Temporal Correspondence Guidance

Many physical adversarial patch generation methods are widely proposed to protect personal privacy from malicious monitoring using object detectors. However, they usually fail to generate satisfactory patch images in terms of both stealthiness and attack performance without making huge efforts on careful hyperparameter tuning. To address this issue, we propose a novel naturalistic adversarial patch generation method based on the diffusion models (DM). Through sampling the optimal image from the DM model pretrained upon natural images, it allows us to stably craft high-quality and naturalistic physical adversarial patches to humans without suffering from serious mode collapse problems as other deep generative models. To the best of our knowledge, we are the first to propose DM-based naturalistic adversarial patch generation for object detectors. With extensive quantitative, qualitative, and subjective experiments, the results demonstrate the effectiveness of the proposed approach to generate better-quality and more naturalistic adversarial patches while achieving acceptable attack performance than other state-of-the-art patch generation methods. We also show various generation trade-offs under different conditions.

Diffusion to Confusion: Naturalistic Adversarial Patch Generation Based on Diffusion Model for Object Detector

Although the rapid development of deep generative models (DGM) enables diverse applications of content creation, increasing illegal uses of the technologies also severely threaten the privacy and security of personal information, especially for faces. Several previous works have been proposed to leverage adversarial attacks to fight against these malicious manipulations by adding an imperceptible perturbation to each input image to disrupt the output. In addition, to improve its scalability, a sequential cross-model universal perturbation attack has been proposed to learn a common adversarial perturbation to defend the images from the manipulation of multiple DGMs. However, we find that the order of DGMs for the adversarial perturbation generation does matter and influence the final defense performance. To address this issue, we propose to generate the universal perturbation through joint optimization of multiple DGMs. From the extensive experimental results, we find that the universal perturbation generated by the proposed method can successfully disrupt the output faces of multiple DGMs at the same time and achieves higher attack success rates than the previous state-of-the-art method based on the sequential generation, even under the situations where the model robustness of DGMs are enhanced by random perturbations.

A Comparative Study of Cross-Model Universal Adversarial Perturbation for Face Forgery

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

Shuohao Lin

Author Tools

Chat about Author