A brief survey on RGB-D semantic segmentation using deep learning

Efficient RGB-D semantic segmentation has received considerable attention in mobile robots, which plays a vital role in analyzing and recognizing environmental information. According to previous studies, depth information can provide corresponding geometric relationships for objects and scenes, but actual depth data usually exist as noise. To avoid unfavorable effects on segmentation accuracy and computation, it is necessary to design an efficient framework to leverage cross-modal correlations and complementary cues. In this article, we propose an efficient lightweight encoder–decoder network that reduces the computational parameters and guarantees the robustness of the algorithm. Working with channel and spatial fusion attention modules, our network effectively captures multi-level RGB-D features. A globally guided local affinity context module is proposed to obtain sufficient high-level context information. The decoder uses a lightweight residual unit (LRU) that combines short- and long-distance information with a few redundant computations. Experimental results on the NYUv2, SUN RGB-D, and Cityscapes datasets show that our method achieves a better tradeoff among segmentation accuracy, inference time, and parameters than the state-of-the-art (SOTA) methods. 

Spatial Information-Guided Adaptive Context-Aware Network for Efficient RGB-D Semantic Segmentation

The trade-off between feature representation capability and spatial positioning accuracy is crucial to dense classification or semantic segmentation of remote sensing images. In order to better balance the low-level spatial details in the shallow network and the high-level abstract semantics in the deep network, the bilateral attention refinement lightweight network BARNet is introduced. In this way, we can use the fine-grained features in the shallow layer to further supplement and capture the deeper information of the high-level semantic features. The network employs an asymmetric encoder decoder architecture for the task of real-time semantic segmentation. Encoder part proposes a lightweight network residual unit with the split, concatenate and split bottleneck structure to achieve more light weighted, effificient and powerful feature extraction. In the decoding section, we propose an adaptive method to enhance feature representation in local attention enhancement module. In addition, the global context embedding module is introduced to divide the high-level features into two branches. One branch gets the weight vector to guide the low-level learning, and the other branch will get a semantic vector, which is used to calculate the multi-label category loss and further introduce into the overall loss function to regulate the training process better. The effectiveness and efficiency of the network are verified on ISPRS Potsdam data set and CCF data set, respectively. The results show that the models using these strategies outperform the baseline network on MIoU, PA and F1, which increase by 18.86%, 16.21% and 15.64% on the Potsdam dataset; 10.51%, 6.53% and 8.19% on the CCF dataset.

/pdf/real-time-semantic-segmentation-of-remote-sensing-images-1mjhketrod.pdf

Real-Time Semantic Segmentation of Remote Sensing Images Based on Bilateral Attention Refined Network

Foundation models have indeed made a profound impact on various fields, emerging as pivotal components that significantly shape the capabilities of intelligent systems. In the context of intelligent vehicles, leveraging the power of foundation models has proven to be transformative, offering notable advancements in visual understanding. Equipped with multi-modal and multi-task learning capabilities, multi-modal multi-task visual understanding foundation models (MM-VUFMs) effectively process and fuse data from diverse modalities and simultaneously handle various driving-related tasks with powerful adaptability, contributing to a more holistic understanding of the surrounding scene. In this survey, we present a systematic analysis of MM-VUFMs specifically designed for road scenes. Our objective is not only to provide a comprehensive overview of common practices, referring to task-specific models, unified multi-modal models, unified multi-task models, and foundation model prompting techniques, but also to highlight their advanced capabilities in diverse learning paradigms. These paradigms include open-world understanding, efficient transfer for road scenes, continual learning, interactive and generative capability. Moreover, we provide insights into key challenges and future trends, such as closed-loop driving systems, interpretability, embodied driving agents, and world models. To facilitate researchers in staying abreast of the latest developments in MM-VUFMs for road scenes, we have established a continuously updated repository at https://github.com/rolsheng/MM-VUFM4DS

Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectives

Enhancing Real-time Inference Performance for Time-Critical Software-Defined Vehicles

In this paper, we address the problem of real-time image semantic segmentation for indoor scene. As for scene parsing, both accuracy and speed are equally important. However, most of existing methods mainly focus on improving accuracy rather than speed. How to find a balance between accuracy and speed is crucial for real-time semantic segmentation tasks. To tackle this problem, we propose a lightweight framework with depthwise dilation residual module and multi-scale information integration module. A single DSNet yields the performance of mIoU accuracy 32.12% on SUN RGB-D dataset and accuracy 26.32% on ADE20K dataset, which is the most challenge scene parsing dataset. Besides, our system yields real-time inference on a single NVIDIA GPU.

DSNET: Accelerate Indoor Scene Semantic Segmentation

LiDAR and camera are two critical sensors for multi-modal 3D semantic segmentation and are supposed to be fused efficiently and robustly to promise safety in various real-world scenarios. However, existing multi-modal methods face two key challenges: 1) difficulty with efficient deployment and real-time execution; and 2) drastic performance degradation under weak calibration between LiDAR and cameras. To address these challenges, we propose CPGNet-LCF, a new multi-modal fusion framework extending the LiDAR-only CPGNet. CPGNet-LCF solves the first challenge by inheriting the easy deployment and real-time capabilities of CPGNet. For the second challenge, we introduce a novel weak calibration knowledge distillation strategy during training to improve the robustness against the weak calibration. CPGNet-LCF achieves state-of-the-art performance on the nuScenes and SemanticKITTI benchmarks. Remarkably, it can be easily deployed to run in 20ms per frame on a single Tesla V100 GPU using TensorRT TF16 mode. Furthermore, we benchmark performance over four weak calibration levels, demonstrating the robustness of our proposed approach.

pdf/revisiting-multi-modal-3d-semantic-segmentation-in-real-d2rfeyxu72.pdf

Revisiting Multi-modal 3D Semantic Segmentation in Real-world Autonomous Driving

LiDAR point cloud segmentation is one of the most fundamental tasks for autonomous driving scene understanding. However, it is difficult for existing models to achieve both high inference speed and accuracy simultaneously. For example, voxel-based methods perform well in accuracy, while Bird's-Eye-View (BEV)-based methods can achieve real-time inference. To overcome this issue, we develop an effective 3D-to-BEV knowledge distillation method that transfers rich knowledge from 3D voxel-based models to BEV-based models. Our framework mainly consists of two modules: the voxel-to-pillar distillation module and the label-weight distillation module. Voxel-to-pillar distillation distills sparse 3D features to BEV features for middle layers to make the BEV-based model aware of more structural and geometric information. Label-weight distillation helps the model pay more attention to regions with more height information. Finally, we conduct experiments on the SemanticKITTI dataset and Paris-Lille-3D. The results on SemanticKITTI show more than 5% improvement on the test set, especially for classes such as motorcycle and person, with more than 15% improvement. The code can be accessed at https://github.com/fengjiang5/Knowledge-Distillation-from-Cylinder3D-to-PolarNet.

pdf/knowledge-distillation-from-3d-to-bird-s-eye-view-for-lidar-3v3j78yk.pdf

Knowledge Distillation from 3D to Bird's-Eye-View for LiDAR Semantic Segmentation

3D point cloud semantic segmentation is one of the fundamental tasks for environmental understanding. Although significant progress has been made in recent years, the performance of classes with few examples or few points is still far from satisfactory. In this paper, we propose a novel multi-to-single knowledge distillation framework for the 3D point cloud semantic segmentation task to boost the performance of those hard classes. Instead of fusing all the points of multi-scans directly, only the instances that belong to the previously defined hard classes are fused. To effectively and sufficiently distill valuable knowledge from multi-scans, we leverage a multilevel distillation framework, i.e., feature representation distillation, logit distillation, and affinity distillation. We further develop a novel instance-aware affinity distillation algorithm for capturing high-level structural knowledge to enhance the distillation efficacy for hard classes. Finally, we conduct experiments on the SemanticKITTI dataset, and the results on both the validation and test sets demonstrate that our method yields substantial improvements compared with the baseline method. The code is available at https://github.com/skyshoumeng/M2SKD.

Multi-to-Single Knowledge Distillation for Point Cloud Semantic Segmentation

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

Feng Jiang

Author Tools

Chat about Author

Papers

DSNET: Accelerate Indoor Scene Semantic Segmentation

Revisiting Multi-modal 3D Semantic Segmentation in Real-world Autonomous Driving

Knowledge Distillation from 3D to Bird's-Eye-View for LiDAR Semantic Segmentation

Multi-to-Single Knowledge Distillation for Point Cloud Semantic Segmentation