Open AccessProceedings Article10.1109/IPDPSW.2018.00098

Exploring the Vision Processing Unit as Co-Processor for Inference

- 21 May 2018

- pp 589-598

TL;DR: This work considers the integration of co-processors in high-performance computing (HPC) to enable low-power, seamless computation offloading of certain operations, and explores the so-called Vision Processing Unit (VPU), a highly-parallel vector processor with a power envelope of less than 1W.

Abstract: The success of the exascale supercomputer is largely debated to remain dependent on novel breakthroughs in technology that effectively reduce the power consumption and thermal dissipation requirements. In this work, we consider the integration of co-processors in high-performance computing (HPC) to enable low-power, seamless computation offloading of certain operations. In particular, we explore the so-called Vision Processing Unit (VPU), a highly-parallel vector processor with a power envelope of less than 1W. We evaluate this chip during inference using a pre-trained GoogLeNet convolutional network model and a large image dataset from the ImageNet ILSVRC challenge. Preliminary results indicate that a multi-VPU configuration provides similar performance compared to reference CPU and GPU implementations, while reducing the thermal-design power (TDP) up to 8x in comparison.

AI Agents for this Paper

Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps

Most frequently asked questions

1. What are the contributions mentioned in the paper "Exploring the vision processing unit as co-processor for inference" ?

In this work, the authors consider the integration of co-processors in high-performance computing ( HPC ) to enable low-power, seamless computation offloading of certain operations.. The authors evaluate this chip during inference using a pre-trained GoogLeNet convolutional network model and a large image dataset from the ImageNet ILSVRC challenge.

2. What have the authors stated for future works in "Exploring the vision processing unit as co-processor for inference" ?

As future work, the authors expect to conduct a thorough study of the possibilities of the Myriad 2 VPU as co-processor for task offloading on HPC.

Figure 6: (a) Inference performance (per subset) of the ILSVRC 2012 Validation dataset using batch-mode on the CPU, GPU, and multi-VPU configurations. (b) Normalized performance scaling per batch size relative to the baseline of a single input for each device type. Note that, in both figures, the number of active VPU chips is coupled with the input size (1-8).

Figure 7: (a) Top-1 prediction error (per subset) of the ILSVRC 2012 Validation dataset using the CPU (FP32) and VPU (FP16) implementations. (b) Absolute confidence difference error after filtering the top-1 miss-predictions between the CPU and VPU implementations.

Figure 1: High-level representation of one of the SHAVE vector processors featured on the Myriad 2 VPU [14]. The Connection Matrix (CMX) enables seamless interaction between the vector processors and other hardware components.

Figure 2: Approximate implementation of the Myriad 2 VPU used within the Neural Compute Stick (NCS) platform [16]. The Neural Compute API allows us to coordinate the execution on the VPU of one or more NCS devices [18].

Figure 8: (a) Throughput performance comparison per Watt using the CPU, GPU, and multi-VPU configurations. (b) Inference performance per batch size on a subset of the ILSVRC 2012 Validation dataset using the CPU, GPU, and multi-VPU configurations. The dashed lines represent the projected value of the multi-VPU configuration if the scaling continues. Note that, in both figures, the number of active VPU chips is coupled with the input size (1-16).

Figure 3: Class diagram specification of the NCSw framework. The simple modular design allows us to provide implementations for new kind of devices (e.g., FPGA).

Citations

•Journal Article•10.1109/JPROC.2019.2921977

Deep Learning With Edge Computing: A Review

Jiasi Chen, +1 more

- 15 Jul 2019

TL;DR: This paper will provide an overview of applications where deep learning is used at the network edge, discuss various approaches for quickly executing deep learning inference across a combination of end devices, edge servers, and the cloud, and describe the methods for training deep learning models across multiple edge devices.

...read moreread less

1.3K

•Journal Article•10.3390/RS12142205

CloudScout: A Deep Neural Network for On-Board Cloud Detection on Hyperspectral Images

Gianluca Giuffrida, +6 more

- 10 Jul 2020

- Remote Sensing

TL;DR: A custom Convolutional Neural Network deployed for a nanosatellite payload to select images eligible for transmission to ground, called CloudScout, which should enable a new era of edge applications and enhance remote sensing applications directly on-board satellite.

...read moreread less

154

•Journal Article•10.1145/3524500

Neural Architecture Search Survey: A Hardware Perspective

Krishna Teja Chitty-Venkata, +1 more

- 13 Apr 2022

- ACM Computing Surveys

TL;DR: Hardware-Aware Neural Architecture Search (HW-NAS) automates the architectural design process of DNNs to alleviate human effort, and generate efficient models accomplishing acceptable accuracy-performance tradeoffs.

...read moreread less

Journal Article•10.1109/jproc.2022.3153408

A Survey of Deep Learning on Mobile Devices: Applications, Optimizations, Challenges, and Research Opportunities

01 Mar 2022

- Proceedings of the IEEE

TL;DR: In this article , the authors provide a comprehensive survey for the current studies of adopting and deploying deep learning (DL) on mobile devices and provide insights into potential research opportunities for developing DL for mobile devices.

...read moreread less

Journal Article•10.1016/j.eswa.2023.122220

A new framework for deep learning video based Human Action Recognition on the edge

Antonio Carlos Cob-Parro, +4 more

- 01 Oct 2023

- Expert Systems With Applications

TL;DR: This paper proposes a deep learning-based framework for real-time human action recognition on edge devices, leveraging a lightweight feature vector and LSTM network, achieving state-of-the-art accuracy in five datasets, including a novel one recorded in the wild.

...read moreread less

...

Expand

References

•Proceedings Article•10.1109/CVPR.2015.7298594

Going deeper with convolutions

Christian Szegedy, +8 more

- 07 Jun 2015

TL;DR: Inception as mentioned in this paper is a deep convolutional neural network architecture that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC14).

...read moreread less

56.6K

•Proceedings Article

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Sergey Ioffe, +1 more

- 06 Jul 2015

TL;DR: Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin.

...read moreread less

43.7K

•Proceedings Article•10.1109/CVPR.2015.7298965

Fully convolutional networks for semantic segmentation

Jonathan Long, +2 more

- 07 Jun 2015

TL;DR: The key insight is to build “fully convolutional” networks that take input of arbitrary size and produce correspondingly-sized output with efficient inference and learning.

...read moreread less

42.6K

•Journal Article•10.1007/S11263-015-0816-Y

ImageNet Large Scale Visual Recognition Challenge

Olga Russakovsky, +11 more

- 01 Dec 2015

- International Journal of Computer Vision

TL;DR: The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) as mentioned in this paper is a benchmark in object category classification and detection on hundreds of object categories and millions of images, which has been run annually from 2010 to present, attracting participation from more than fifty institutions.

...read moreread less

41.6K

•Book

Data Mining: Practical Machine Learning Tools and Techniques

Ian H. Witten, +2 more

- 25 Oct 1999

TL;DR: This highly anticipated third edition of the most acclaimed work on data mining and machine learning will teach you everything you need to know about preparing inputs, interpreting outputs, evaluating results, and the algorithmic methods at the heart of successful data mining.

...read moreread less

25.4K

...

Expand

Exploring the Vision Processing Unit as Co-Processor for Inference

Chat with Paper

AI Agents for this Paper

Most frequently asked questions

1. What are the contributions mentioned in the paper "Exploring the vision processing unit as co-processor for inference" ?

2. What have the authors stated for future works in "Exploring the vision processing unit as co-processor for inference" ?

Figures

Citations

Deep Learning With Edge Computing: A Review

CloudScout: A Deep Neural Network for On-Board Cloud Detection on Hyperspectral Images

Neural Architecture Search Survey: A Hardware Perspective

A Survey of Deep Learning on Mobile Devices: Applications, Optimizations, Challenges, and Research Opportunities

A new framework for deep learning video based Human Action Recognition on the edge

References

Going deeper with convolutions

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Fully convolutional networks for semantic segmentation

ImageNet Large Scale Visual Recognition Challenge

Data Mining: Practical Machine Learning Tools and Techniques

Related Papers (5)

Memory Controller for Vector Processor

Energy-efficient hardware acceleration through computing in the memory

A 275mW heterogeneous multimedia processor for IC-stacking on Si-interposer

A comparative analysis of microarchitecture effects on CPU and GPU memory system behavior

A Media-Enhanced Vector Architecture for Embedded Memory Systems