Robust detection and tracking of objects is crucial for the deployment of autonomous vehicle technology. Image based benchmark datasets have driven development in computer vision tasks such as object detection, tracking and segmentation of agents in the environment. Most autonomous vehicles, however, carry a combination of cameras and range sensors such as lidar and radar. As machine learning based methods for detection and tracking become more prevalent, there is a need to train and evaluate such methods on datasets containing range sensor data along with images. In this work we present nuTonomy scenes (nuScenes), the first dataset to carry the full autonomous vehicle sensor suite: 6 cameras, 5 radars and 1 lidar, all with full 360 degree field of view. nuScenes comprises 1000 scenes, each 20s long and fully annotated with 3D bounding boxes for 23 classes and 8 attributes. It has 7x as many annotations and 100x as many images as the pioneering KITTI dataset. We define novel 3D detection and tracking metrics. We also provide careful dataset analysis as well as baselines for lidar and image based detection and tracking. Data, development kit and more information are available online.

/pdf/nuscenes-a-multimodal-dataset-for-autonomous-driving-3znmpkqyuu.pdf

nuScenes: A Multimodal Dataset for Autonomous Driving

nuScenes: A multimodal dataset for autonomous driving

As the foundation of intelligent systems, machine vision perceives the surrounding environment and provides a basis for decision-making. Object detection is the core task in machine vision. 3D object detection can provide object steric size and location information. Compared with the 2D object detection widely studied in image coordinates, it can provide more applications of detection systems. Accurate LiDAR data has a stronger spatial capture capability and is insensitive to natural light, which makes LiDAR a potential sensor for 3D detection. Recently, deep neural network has been developed to learn powerful object features from sensor data. However, the sparsity of LiDAR point cloud data poses challenges to the network processing. Plenty of emerged efforts have been made to address this difficulty, but a comprehensive review literature is still lacking. The purpose of this article is to review the challenges and methodologies of 3D object detection networks using LiDAR data. On this account, we first give an outline of 3D detection task and LiDAR sensing techniques. Then we unfold the review of deep 3D detection networks with three kinds of LiDAR point cloud representations and their challenges. We next summarize evaluation metrics and performance of algorithms on three authoritative 3D detection benchmarks. Finally, we provide valuable insights of challenges and open issues.

Deep 3D Object Detection Networks Using LiDAR Data: A Review

Accurate 3D object detection in LiDAR based point clouds suffers from the challenges of data sparsity and irregularities. Existing methods strive to organize the points regularly, e.g. voxelize, pass them through a designed 2D/3D neural network, and then define object-level anchors that predict offsets of 3D bounding boxes using collective evidences from all the points on the objects of interest. Contrary to the state-of-the-art anchor-based methods, based on the very nature of data sparsity, we observe that even points on an individual object part are informative about semantic information of the object. We thus argue in this paper for an approach opposite to existing methods using object-level anchors. Inspired by compositional models, which represent an object as parts and their spatial relations, we propose to represent an object as composition of its interior non-empty voxels, termed hotspots, and the spatial relations of hotspots. This gives rise to the representation of Object as Hotspots (OHS). Based on OHS, we further propose an anchor-free detection head with a novel ground truth assignment strategy that deals with inter-object point-sparsity imbalance to prevent the network from biasing towards objects with more points. Experimental results show that our proposed method works remarkably well on objects with a small number of points. Notably, our approach ranked \(1^{st}\) on KITTI 3D Detection Benchmark for cyclist and pedestrian detection, and achieved state-of-the-art performance on NuScenes 3D Detection Benchmark.

Object as Hotspots: An Anchor-Free 3D Object Detection Approach via Firing of Hotspots

Autonomous driving, in recent years, has been receiving increasing attention for its potential to relieve drivers’ burdens and improve the safety of driving. In modern autonomous driving pipelines, the perception system is an indispensable component, aiming to accurately estimate the status of surrounding environments and provide reliable observations for prediction and planning. 3D object detection, which aims to predict the locations, sizes, and categories of the 3D objects near an autonomous vehicle, is an important part of a perception system. This paper reviews the advances in 3D object detection for autonomous driving. First, we introduce the background of 3D object detection and discuss the challenges in this task. Second, we conduct a comprehensive survey of the progress in 3D object detection from the aspects of models and sensory inputs, including LiDAR-based, camera-based, and multi-modal detection approaches. We also provide an in-depth analysis of the potentials and challenges in each category of methods. Additionally, we systematically investigate the applications of 3D object detection in driving systems. Finally, we conduct a performance analysis of the 3D object detection approaches, and we further summarize the research trends over the years and prospect the future directions of this area. 

3D Object Detection for Autonomous Driving: A Comprehensive Survey

SARPNET: Shape attention regional proposal network for liDAR-based 3D object detection

The invention belongs to the field of computer deep reinforcement learning and pattern recognition, particularly relates to a three-dimensional visual inspection method, system and device based on a shape attention mechanism, and aims to solve the problems that the precision of a single-stage detector is lower than that of a two-stage detector, and the two-stage detector consumes a large amount oftime and is not suitable for a real-time system. The method comprises the steps of representing point cloud data through three-dimensional grid voxels; extracting features and encoding a spatial sparse feature map; after projecting to a top view, extracting different scale features; combining the features by adopting a deconvolution layer; extracting a shape attention feature map through the attention weight and the convolutional coding layer; and obtaining a target category, a target position, a target size and a target direction through the target classification network and the regression positioning network. According to the method, a sampling strategy based on distance constraint and an attention mechanism based on shape prior are used, instability caused by uneven data distribution is relieved, the problem that a single-stage detector lacks shape prior is solved, and the method is high in precision, short in consumed time, high in real-time performance and good in robustness.

Three-dimensional visual inspection method, system and device based on shape attention mechanism

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

Ye Yangyang

Author Tools

Chat about Author

Papers

SARPNET: Shape attention regional proposal network for liDAR-based 3D object detection

Three-dimensional visual inspection method, system and device based on shape attention mechanism