The neuromorphic event cameras can efficiently sense the latent geometric structures and motion clues of a scene by generating asynchronous and sparse event signals. Due to the irregular layout of the event signals, how to leverage their plentiful spatio-temporal information for recognition tasks remains a significant challenge. Existing methods tend to treat events as dense image-like or point-serie representations. However, they either suffer from severe destruction on the sparsity of event data or fail to encode robust spatial cues. To fully exploit their inherent sparsity with reconciling the spatio-temporal information, we introduce a compact event representation, namely 2D-1T event cloud sequence (2D-1T ECS). We couple this representation with a novel light-weight spatio-temporal learning framework (ECSNet) that accommodates both object classification and action recognition tasks. The core of our framework is a hierarchical spatial relation module. Equipped with specially designed surface-event-based sampling unit and local event normalization unit to enhance the inter-event relation encoding, this module learns robust geometric features from the 2D event clouds. And we propose a motion attention module for efficiently capturing long-term temporal context evolving with the 1T cloud sequence. Empirically, the experiments show that our framework achieves par or even better state-of-the-art performance. Importantly, our approach cooperates well with the sparsity of event data without any sophisticated operations, hence leading to low computational costs and prominent inference speeds. 

ECSNet: Spatio-Temporal Feature Learning for Event Camera

The neuromorphic event cameras can efficiently sense the latent geometric structures and motion clues of a scene by generating asynchronous and sparse event signals. Due to the irregular layout of the event signals, how to leverage their plentiful spatio-temporal information for recognition tasks remains a significant challenge. Existing methods tend to treat events as dense image-like or point-serie representations. However, they either suffer from severe destruction on the sparsity of event data or fail to encode robust spatial cues. To fully exploit their inherent sparsity with reconciling the spatio-temporal information, we introduce a compact event representation, namely 2D-1T event cloud sequence (2D-1T ECS). We couple this representation with a novel light-weight spatio-temporal learning framework (ECSNet) that accommodates both object classification and action recognition tasks. The core of our framework is a hierarchical spatial relation module. Equipped with specially designed surface-event-based sampling unit and local event normalization unit to enhance the inter-event relation encoding, this module learns robust geometric features from the 2D event clouds. And we propose a motion attention module for efficiently capturing long-term temporal context evolving with the 1T cloud sequence. Empirically, the experiments show that our framework achieves par or even better state-of-the-art performance. Importantly, our approach cooperates well with the sparsity of event data without any sophisticated operations, hence leading to low computational costs and prominent inference speeds.

The inherent spike-based and event-driven computation makes spiking neural networks (SNNs) naturally suitable to provide efficient and low-latency solution in neuromorphic vision processing. In this work we propose an address-event representation (AER) object recognition system which consists of a noise filter, an event stream segmentation module and a hierarchical convolutional SNN. In order to improve the feature learning capacity of bio-inspired shallow hierarchical SNNs for AER object recognition, we apply a convolutional SNN with R-STDP learning rule and first-spike coding to further extract features after primary feature learning using Gabor filter-based event-driven convolution. The R-STDP learning rule enables the network to adjust weights by combining local STDP learning rule with a global reward signal and to make classification without using an external classifier. Experimental results show that our method significantly outperforms the existing hierarchical SNN-based recognition methods on four popular AER datasets. Moreover, our method has much better recognition ability when using very short input event stream, and can effectively learn from small size training sets. In addition, the robustness of the network to input noise is improved. Our method is very beneficial for developing neuromorphic-vision object recognition algorithm in resource-constrained applications when event streams are short and training samples are limited.

A Bio-Inspired Hierarchical Spiking Neural Network With Reward-Modulated STDP Learning Rule for AER Object Recognition

Recognizing and comprehending human actions and gestures is a crucial perception requirement for robots to interact with humans and carry out tasks in diverse domains, including service robotics, healthcare, and manufacturing. Event cameras, with their ability to capture fast-moving objects at a high temporal resolution, offer new opportunities compared to standard action recognition in RGB videos. However, previous research on event camera action recognition has primarily focused on sensor-specific network architectures and image encoding, which may not be suitable for new sensors and limit the use of recent advancement in transformer-based architectures. In this study, we employ using a computationally efficient model, namely the video transformer network (VTN), which initially acquires spatial embeddings per event-frame and then utilizes a temporal self-attention mechanism. This approach separates the spatial and temporal operations, resulting in VTN being more computationally efficient than other video transformers that process spatio-temporal volumes directly. In order to better adopt the VTN for the sparse and finegrained nature of event data, we design Event-Contrastive Loss <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\left(\mathscr{L}_{E C}\right)$</tex> and event specific augmentations. Proposed <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\left(\mathscr{L}_{E C}\right)$</tex> promotes learning fine-grained spatial cues in the spatial backbone of VTN by contrasting temporally misaligned frames. We evaluate our method on real-world action recognition of N-EPIC Kitchens dataset, and achieve state-of-the-art results on both protocols - testing in seen kitchen (74.9% accuracy) and testing in unseen kitchens (42.43% and 46.66% Accuracy). Our approach also takes less computation time compared to competitive prior approaches. We also evaluate our method on the standard DVS Gesture recognition dataset, achieving a competitive accuracy of 97.9% compared to prior work that uses dedicated architectures and image-encoding for the DVS dataset. These results demonstrate the potential of our framework EventTransAct for real-world applications of event-camera based action recognition. Project Page: https://tristandb8.github.io/EventTransAct_webpage/ 

EventTransAct: A Video Transformer-Based Framework for Event-Camera Based Action Recognition

Recently, new sensors with active pixels were brought to market. These sensors export local variations of light intensity in the form of asynchronous events with low latency. Since the data output format is a stream of addressable events and not a complete image of light intensities, new algorithms are required for known problems in the field of Computer Vision, such as segmentation, VO, SLAM, object, and scene recognition. There are some proposed methodologies for object recognition using conventional methods, convolutional neural networks, and third-generation neural networks based on spikes. However, convolutional neural networks and spike neural networks require specific hardware for processing, hard to miniaturize. Also, several traditional Computer Vision operators and feature descriptors have been neglected in the context of event sensors and could contribute to lighter methodologies in object recognition. This paper proposes an algorithm for local binary pattern extraction in sparse structures, typically found in this context. This paper also proposes two methodologies using local binary patterns to captures with event-based sensors for object recognition. The first methodology exploits the known motion performed by the sensor, while the second is motion agnostic. It is demonstrated experimentally that the LBP operator is a fast and light alternative that enables variable reduction using PCA in some cases. The experiments also show that it is possible to reduce the final feature vector for classification by up to 99,73% when compared to conventional methods considered state-of-the-art while maintaining comparable accuracy. 

A Sparse Local Binary Pattern extraction algorithm applied to event sensor data for object classification

This paper introduces an event-based categorization model to explore neuromorphic computing for object recognition. The proposed method makes full use of the precise timing information inherently present in the output of a bio-inspired vision sensor and utilizes event-driven processing to keep the form of address event representation (AER). In this model, a hierarchy of event-based time-surface is used to extract the spatio-temporal features of the AER data, and then these extracted features are classified by spiking neural network (SNN) with event-driven Tempotron rule. To demonstrate the effectiveness of the presented method, we conducted a series of experiments on some accessible AER datasets and compared our method to other developed event-based categorization models. Additional pooling operation in feature extraction can reduce the number of parameters and computing time. Experimental results show that our method has better classification performance and real-time computation. That is, the proposed model can extract useful features and consumes much less simulation time while still maintaining competitive accuracy.

An Event-based Categorization Model Using Spatio-temporal Features in a Spiking Neural Network

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

Junwei Lu

Author Tools

Chat about Author

Papers

An Event-based Categorization Model Using Spatio-temporal Features in a Spiking Neural Network