As deep learning inference applications are increasing in embedded devices, an embedded device tends to equip neural processing units (NPUs) in addition to a multi-core CPU and a GPU. NVIDIA Jetson AGX Xavier is an example. For fast and efficient development of deep learning applications, TensorRT is provided as the SDK for high-performance inference, including an optimizer and runtime that delivers low latency and high throughput for deep learning inference applications. Like most deep learning frameworks, TensorRT assumes that the inference is executed on a single processing element, GPU or NPU, not both. In this article, we present a TensorRT-based framework supporting various optimization parameters to accelerate a deep learning application targeted on an NVIDIA Jetson embedded platform with heterogeneous processors, including multi-threading, pipelining, buffer assignment, and network duplication. Since the design space of allocating layers to diverse processing elements and optimizing other parameters is huge, we devise a parameter optimization methodology that consists of a heuristic for balancing pipeline stages among heterogeneous processors and fine-tuning the process for optimizing parameters. With nine real-life benchmarks, we could achieve 101%~680% performance improvement and up to 55% energy reduction over the baseline inference using a GPU only.

TensorRT-Based Framework and Optimization Methodology for Deep Learning Inference on Jetson Boards

Most deep-learning-based target detection methods have high computational complexity and memory consumption, and they are difficult to deploy on edge devices with limited computing resources and memory. To tackle this problem, this article proposes to learn a lightweight detector named Light-YOLOv4, and it is obtained from YOLOv4 through model compression. To this end, first, we perform sparsity training by applying L1 regularization to the channel scaling factors, so the less important channels and layers can be found. Second, channel pruning and layer pruning are enforced on the network to prune the less important parts, which could significantly reduce network's width and depth. Third, the pruned model is retrained with a knowledge distillation method to improve the detection accuracy. Fourth, the model is quantized from FP32 to FP16, and it could further accelerate the model with almost no loss of detection accuracy. Finally, to evaluate Light-YOLOv4's performance on edge devices, Light-YOLOv4 is deployed on NVIDIA Jetson TX2. Experiments on the SAR ship detection dataset (SSDD) demonstrate that the model size, parameter size, and FLOPs of Light-YOLOv4 have been reduced by 98.63%, 98.66%, and 91.30% compared with YOLOv4, and the detection speed has been increased to 4.2×. While the detection accuracy of Light-YOLOv4 is only slightly reduced, for example, the mAP has only reduced by 0.013. Besides, experiments on the Gaofen Airplane dataset also prove the feasibility of Light-YOLOv4. Moreover, compared with other state-of-the-art methods, such as SSD and FPN, Light-YOLOv4 is more suitable for edge devices.

/pdf/light-yolov4-an-edge-device-oriented-target-detection-method-xf7rksyo9o.pdf

Light-YOLOv4: An Edge-Device Oriented Target Detection Method for Remote Sensing Images

Due to the workforce shortage caused by the declining birth rate and aging population, robotics is one of the solutions to replace humans and overcome this urgent problem. This paper introduces a deep learning-based object detection algorithm for empty-dish recycling robots to automatically recycle dishes in restaurants and canteens, etc. In detail, a lightweight object detection model YOLO-GD (Ghost Net and Depthwise convolution) is proposed for detecting dishes in images such as cups, chopsticks, bowls, towels, etc., and an image processing-based catch point calculation is designed for extracting the catch point coordinates of the different-type dishes. The coordinates are used to recycle the target dishes by controlling the robot arm. Jetson Nano is equipped on the robot as a computer module, and the YOLO-GD model is also quantized by TensorRT for improving the performance. The experimental results demonstrate that the YOLO-GD model is only 1/5 size of the state-of-the-art model YOLOv4, and the mAP of YOLO-GD achieves 97.38%, 3.41% higher than YOLOv4. After quantization, the YOLO-GD model decreases the inference time per image from 207.92 ms to 32.75 ms, and the mAP is 97.42%, which is slightly higher than the model without quantization. Through the proposed image processing method, the catch points of various types of dishes are effectively extracted. The functions of empty-dish recycling are realized and will lead to further development toward practical use.

/pdf/yolo-gd-a-deep-learning-based-object-detection-algorithm-for-1j3ntct6.pdf

YOLO-GD: A Deep Learning-Based Object Detection Algorithm for Empty-Dish Recycling Robots

Automatic defogging, deblurring, and real-time segmentation system for sewer pipeline defects

Abstract Deep neural networks (DNNs) have made significant achievements in a wide variety of domains. For the deep learning tasks, multiple excellent hardware platforms provide efficient solutions, including graphics processing units (GPUs), central processing units (CPUs), field programmable gate arrays (FPGAs), and application-specific integrated circuit (ASIC). Nonetheless, CPUs outperform other solutions including GPUs in many cases for the inference workload of DNNs with the support of various techniques, such as the high-performance libraries being the basic building blocks for DNNs. Thus, CPUs have been a preferred choice for DNN inference applications, particularly in the low-latency demand scenarios. However, the DNN inference efficiency remains a critical issue, especially when low latency is required under conditions with limited hardware resources, such as embedded systems. At the same time, the hardware features have not been fully exploited for DNNs and there is much room for improvement. To this end, this paper conducts a series of experiments to make a thorough study for the inference workload of prominent state-of-the-art DNN architectures on a single-instruction-multiple-data (SIMD) CPU platform, as well as with widely applicable scopes for multiple hardware platforms. The study goes into depth in DNNs: the CPU kernel-instruction level performance characteristics of DNNs including branches, branch prediction misses, cache misses, etc, and the underlying convolutional computing mechanism at the SIMD level; The thorough layer-wise time consumption details with potential time-cost bottlenecks; And the exhaustive dynamic activation sparsity with exact details on the redundancy of DNNs. The research provides researchers with comprehensive and insightful details, as well as crucial target areas for optimising and improving the efficiency of DNNs at both the hardware and software levels. 

/pdf/an-architecture-level-analysis-on-deep-learning-models-for-5jthl0q2.pdf

An architecture-level analysis on deep learning models for low-impact computations

Parallel scheduling of multiple real-time applications onto heterogeneous processors is needed in the emerging embedded systems such as self-driving cars, smart cameras, and smartphones. Assuming that an embedded application is specified as a synchronous dataflow (SDF) graph or its extension, we propose a novel parallel scheduling methodology based on an evolutionary algorithm where the mapping of tasks onto processors is evolved to optimize a given objective function in an iterative fashion. In each iteration, we use an existing worst-case response time (WCRT) analysis tool to check if all applications satisfy their real-time requirements by translating each SDF graph into a directed acyclic graph (DAG) that is assumed in the WCRT analysis tool. Since the WCRT analysis must be performed in each iteration of evolution, we propose a clustering technique to reduce drastically the analysis time that depends on the number of nodes and their dependency. We formally prove that the proposed clustering technique does not change the estimated WCRT of each application. The effectiveness of the proposed scheduling methodology with the clustering technique is verified with extensive experiments using real-life benchmarks, randomly generated graphs, and the comparison with the existing technique.

/pdf/parallel-scheduling-of-multiple-sdf-graphs-onto-2yg4y2z11x.pdf

Parallel Scheduling of Multiple SDF Graphs Onto Heterogeneous Processors

IoT systems can be regarded as distributed systems that are composed of heterogeneous smart devices with different performances and functionalities. Many IoT applications with various resources and real-time requirements will run concurrently in an IoT system. In addition, non-functional properties such as power consumption and device lifetime are considered important. Since the IoT services can be added or removed anytime and the device status may change at runtime, the system configuration is unprecedentedly dynamic, which brings up a challenging scheduling problem of IoT services onto the devices. We assume that an IoT application is specified by a directed graph, called service graph, where nodes represent services while the directed edges represent the dependencies between them. To tackle this problem, we propose a novel adaptive approach that consists of two scheduling techniques, incremental and global. The incremental heuristic method is proposed to provide fast responsiveness to the dynamically changing configuration while satisfying the real-time constraints. During steady state operation, GA-based global rescheduling is periodically applied. We use the acceptance ratio of new services and energy consumption as two performance metrics of the proposed scheduling method. The viability of the proposed approach is verified by extensive simulations with random IoT service graphs.

Adaptive run-time scheduling of dependent services for service-oriented IoT systems

SSD has become an indispensable element in today’s computer systems, and their architecture is constantly evolving with new host interfaces for higher performance and larger storage capacities thanks to incessant flash technology development. As the complexity of SSD architecture increases, it is necessary to use a systematic methodology for architecture design. In this paper, we propose a novel methodology to explore the design space of an SSD based on a genetic algorithm at the early design stage. The key technical challenge in the design space exploration (DSE) is fast and accurate performance estimation or fitness evaluation in the genetic algorithm. To tackle this challenge, we propose two performance estimation methods. One is based on the scheduling of the task graph abstracted from the firmware and the other one is based on a neural network (NN) regression model. While the NN-based method is faster, the accuracy of the NN-based method depends on the training data set that consists of hardware configurations and performance. The scheduling-based performance estimator is used to generate the training data set fast. The viability of the proposed methodology is confirmed by comparison with a state-of-the-art SSD simulator in terms of accuracy and speed for design space exploration.

Fast Performance Estimation and Design Space Exploration of SSD Using AI Techniques

As deep learning inference applications are increasing, an embedded device tends to equip neural processing units (NPUs) in addition to a CPU and a GPU. For fast and efficient development of deep learning applications, TensorRT is provided as the SDK for NVIDIA hardware platform, including optimizer and runtime that delivers low latency and high-throughput for deep learning inference. Like most deep learning frameworks, TensorRT assumes that the inference is executed on a single processing element, GPU or NPU, not both. In this paper, we propose a parallelization methodology to maximize the throughput of a single deep learning application using both GPU and NPU by exploiting various types of parallelism on TensorRT. With six real-life benchmarks, we could achieve 81% 391% throughput improvement over the baseline inference using GPU only.

Deep Learning Inference Parallelization on Heterogeneous Processors with TensorRT

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

Jangryul Kim

Author Tools

Chat about Author

Papers

Parallel Scheduling of Multiple SDF Graphs Onto Heterogeneous Processors

Adaptive run-time scheduling of dependent services for service-oriented IoT systems

Fast Performance Estimation and Design Space Exploration of SSD Using AI Techniques

Deep Learning Inference Parallelization on Heterogeneous Processors with TensorRT