How do humans recognize the action "opening a book" ? We argue that there are two important cues: modeling temporal shape dynamics and modeling functional relationships between humans and objects. In this paper, we propose to represent videos as space-time region graphs which capture these two important cues. Our graph nodes are defined by the object region proposals from different frames in a long range video. These nodes are connected by two types of relations: (i) similarity relations capturing the long range dependencies between correlated objects and (ii) spatial-temporal relations capturing the interactions between nearby objects. We perform reasoning on this graph representation via Graph Convolutional Networks. We achieve state-of-the-art results on both Charades and Something-Something datasets. Especially for Charades, we obtain a huge 4.4% gain when our model is applied in complex environments.

Videos as Space-Time Region Graphs.

Anomaly detection with weakly supervised video-level labels is typically formulated as a multiple instance learning (MIL) problem, in which we aim to identify snippets containing abnormal events, with each video represented as a bag of video snippets. Although current methods show effective detection performance, their recognition of the positive instances, i.e., rare abnormal snippets in the abnormal videos, is largely biased by the dominant negative instances, especially when the abnormal events are subtle anomalies that exhibit only small differences compared with normal events. This issue is exacerbated in many methods that ignore important video temporal dependencies. To address this issue, we introduce a novel and theoretically sound method, named Robust Temporal Feature Magnitude learning (RTFM), which trains a feature magnitude learning function to effectively recognise the positive instances, substantially improving the robustness of the MIL approach to the negative instances from abnormal videos. RTFM also adapts dilated convolutions and self-attention mechanisms to capture long- and short-range temporal dependencies to learn the feature magnitude more faithfully. Extensive experiments show that the RTFM-enabled MIL model (i) outperforms several state-of-the-art methods by a large margin on four benchmark data sets (ShanghaiTech, UCF-Crime, XD-Violence and UCSD-Peds) and (ii) achieves significantly improved subtle anomaly discriminability and sample efficiency. Code is available at this https URL.

Weakly-supervised Video Anomaly Detection with Robust Temporal Feature Magnitude Learning

Video-based human action recognition is one of the most important and challenging areas of research in the field of computer vision. Human action recognition has found many pragmatic applications in video surveillance, human–computer interaction, entertainment, autonomous driving, etc. Owing to the recent development of deep learning methods for human action recognition, the performance of action recognition has significantly enhanced for challenging datasets. Deep learning techniques are mainly used for recognizing actions in images and videos comprising of Euclidean data. A recent development in deep learning methods is the extension of these techniques to non-Euclidean data or graph data with many nodes and edges. Human body skeleton resembles a graph, therefore, the graph convolutional network (GCN) is applicable to the non-Euclidean body skeleton. In the past few years, GCN has emerged as an important tool for skeleton-based action recognition. Therefore, we conduct a survey using GCN methods for action recognition. Herein, we present a comprehensive overview of recent GCN techniques for action recognition, propose a taxonomy for the categorization of GCN techniques for action recognition, carry out a detailed study of the benchmark datasets, enlist relevant resources and open-source codes, and finally provide an outline for future research directions and trends. To the best of authors’ knowledge, this is the first survey for action recognition using GCN techniques. Impact Statement— Graph convolutional neural networks have marked a great progress in recent years. There is a similarity between body skeleton and a graph; therefore, GCNs have been widely used for skeleton-based action recognition. In this article, we summarize recent graph-based action recognition techniques, provide a deeper insight of these methods, list source-codes and available resources. This article will help the researchers to develop a basic understanding of graph convolutional methods for action recognition, benefit from useful resources, and think about future directions.

Graph Convolutional Neural Network for Human Action Recognition: A Comprehensive Survey

Artificial Intelligence of Things-assisted two-stream neural network for anomaly detection in surveillance Big Video Data

The introduction of depth sensors such as Microsoft Kinect have driven research in human action recognition. Human skeletal data collected from depth sensors convey a significant amount of information for action recognition. While there has been considerable progress in action recognition, most existing skeleton-based approaches neglect the fact that not all human body parts move during many actions, and they fail to consider the ordinal positions of body joints. Here, and motivated by the fact that an action's category is determined by local joint movements, we propose a cuboid model for skeleton-based action recognition. Specifically, a cuboid arranging strategy is developed to organize the pairwise displacements between all body joints to obtain a cuboid action representation. Such a representation is well structured and allows deep CNN models to focus analyses on actions. Moreover, an attention mechanism is exploited in the deep model, such that the most relevant features are extracted. Extensive experiments on our new Yunnan University-Chinese Academy of Sciences-Multimodal Human Action Dataset (CAS-YNU MHAD), the NTU RGB+D dataset, the UTD-MHAD dataset, and the UTKinect-Action3D dataset demonstrate the effectiveness of our method compared to the current state-of-the-art.

A Cuboid CNN Model With an Attention Mechanism for Skeleton-Based Action Recognition

Real-world anomalous events are complicated, diverse, and rarely occurred. The main challenge to anomaly detection is to learn normal and anomalous patterns accurately. In this work, we propose discriminative clip mining for anomaly detection and classification: firstly, by introducing clip-level class activation mapping, an efficient Discriminative Anomalous Clip Miner (DACM) is developed to mine discriminative anomalous clips from a large number of normal ones; secondly, with the mined discriminative clips, an attentive ranking loss is designed to increase the anomalous instances hit rate of the traditional Multiple Instance Learning (MIL) model. Furthermore, by integrating the DACM and attentive MIL, one novel anomaly detection framework is proposed to learn more contrastive anomalous and normal patterns, and thus higher recognition performance can be achieved. Our experimental results on the widely-used UCF-Crime dataset show that, as compared to the state-of-the-art approaches, the proposed method achieves competitive performance both in anomaly detection and anomalous activity classification.

Discriminative Clip Mining for Video Anomaly Detection

Recent human action recognition methods mainly model a two-stream or 3D convolution deep learning network, with which humans spatial-temporal features can be exploited and utilized effectively. However, due to the ignoring of interaction exploiting, most of these methods cannot get good enough performance. In this paper, we propose a novel action recognition framework with Graph Convolutional Network (GCN) based Interaction Reasoning: Objects and discriminative scene patches are detected using an object detector and class active mapping (CAM), respectively; and then a GCN is introduced to model the interaction among the detected objects and scene patches. Evaluation of two widely used video action benchmarks shows that the proposed work can achieve comparable performance: the accuracy up to 43.6% at EPIC Kitchen, and 47.0% at VLOG benchmark without using optical flow, respectively.

Improving Action Recognition with the Graph-Neural-Network-based Interaction Reasoning

Recent human action recognition models mainly focus on exploiting human features, such as pose or skeleton features. However, most of these methods do not pay enough attention to action-related backgrounds. In this work we propose a novel multi-stream features fusion framework based on discriminative patch exploiting. Unlike existing part-based or attention-based multi-stream methods, our work improves the recognition accuracy by 1) Paying more attention to exploiting of global and local discriminative patches, which include not only the acting human but also the interactive scenes. 2) Proposing an effective multi-stream feature pooling and fusion mechanism: 2D and 3D features from RGB frames and discriminative patches are combined to enhance spatial-temporal feature representation ability. Our framework is evaluated on two widely used video action benchmarks, where it outperforms other state-of-the-art methods: the accuracy up to 87.8% at HMDB51, and 98.8% at UCF101.

Global and Local Discriminative Patches Exploiting for Action Recognition

Recent human action recognition methods mainly model a two-stream or multi-stream deep neural network, with which human spatiotemporal features can be exploited effectively. However, due to the ignoring of interactive scenes, most of these methods cannot achieve impressive performance. In this paper, we propose a novel multi-stream fusion framework based on discriminative scene patches and motion patches. Unlike existing two-stream or multi-stream methods, our work improves the accuracy by 1) Attaching more attention to the exploiting of discriminative scene patches and motion patches. 2) Proposing a novel 2D+3D multi-stream feature aggregation mechanism: 2D features from RGB images and 3D features of valued patches are combined to improve the representation of spatiotemporal features. Our framework is evaluated on three widely used video action benchmarks, where it outperforms other state-of-the-art recognition approaches by a significant margin: the accuracy up to 85.7% at JHMDB, 87.7% at HMDB51, and 98.6% at UCF101, respectively.

Improving Action Recognition with Valued Patches Exploiting

Trajectories and spatiotemporal attention model have been successfully used in skeleton-based action recognition. Most existing methods focus more attention on temporal structure mining. However, only a few local joints and their position features (e.g., critical position changes of hand, head, leg etc.) are responsible for the action label. In this work, we introduce a novel action recognition framework using Key Joints Selection and Spatiotemporal Mining, which can identify both key joints and their position & velocity histogram as well as trajectory features for action classification. First, histogram of human joints position and velocity are developed to enhance the spatiotemporal structure representation of existing trajectory-based methods. Second, the key joints are selected according to their information gains, and then their position & velocity histograms are weighted and composed with trajectory features to form one richer representation for final action classification. Experiments on two widely-tested benchmark datasets show that by combining the strength of both richer features and key joints selecting, our method can achieve state-of-the-art or competitive performance compared with existing results using sophisticated models such as deep learning, with advantages regarding the recognition accuracy and robustness.

Key Joints Selection and Spatiotemporal Mining for Skeleton-Based Action Recognition

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

Wu Luo

Author Tools

Chat about Author