Enhancing Event-based Structured Light Imaging with a Single Frame

TL;DR: A Multi-Modal Feature Fusion Network (MFFN) consisting of a feature fusion module and an upscale module to simultaneously fuse events and a single intensity frame, suppress event perturbations, and reconstruct a high-quality depth surface is proposed.

Abstract: Benefiting from the extremely low latency, events have been used for Structured Light Imaging (SLI) to predict the depth surface. However, existing methods only focus on improving scanning speeds but neglect perturbations from event noise and timestamp jittering for depth estimation. In this paper, we build a hybrid SLI system equipped with an event camera, a high-resolution frame camera, and a digital light projector, where a single intensity frame is adopted as a guidance to enhance the event-based SLI quality. To achieve this end, we propose a Multi-Modal Feature Fusion Network (MFFN) consisting of a feature fusion module and an upscale module to simultaneously fuse events and a single intensity frame, suppress event perturbations, and reconstruct a high-quality depth surface. Further, for training MFFN, we build a new Structured Light Imaging based on Event and Frame cameras (EF-SLI) dataset collected from the hybrid SLI system, containing paired inputs composed of a set of synchronized events and one single corresponding frame, and ground-truth references obtained by a high-quality SLI approach. Experiments demonstrate that our proposed MFFN outperforms state-of-the-art event-based SLI approaches in terms of accuracy at different scanning speeds.