The ability to form reconstructions beyond line-of-sight view could be transformative in a variety of fields, including search and rescue, autonomous vehicle navigation, and reconnaissance. Most existing active non-line-of-sight (NLOS) imaging methods use data collection steps in which a pulsed laser is directed at several points on a relay surface, one at a time. The prevailing approaches include raster scanning of a rectangular grid on a vertical wall opposite the volume of interest to generate a collection of confocal measurements. These and a recent method that uses a horizontal relay surface are inherently limited by the need for laser scanning. Methods that avoid laser scanning to operate in a snapshot mode are limited to treating the hidden scene of interest as one or two point targets. In this work, based on more complete optical response modeling yet still without multiple illumination positions, we demonstrate accurate reconstructions of foreground objects while also introducing the capability of mapping the stationary scenery behind moving objects. The ability to count, localize, and characterize the sizes of hidden objects, combined with mapping of the stationary hidden scene, could greatly improve indoor situational awareness in a variety of applications. 

https://www.nature.com/articles/s41467-023-39327-2.pdf

Non-line-of-sight snapshots and background mapping with an active corner camera

Abstract We introduce an approach for three-dimensional full-colour non-line-of-sight imaging with an ordinary camera that relies on a complementary combination of a new measurement acquisition strategy, scene representation model, and tailored reconstruction method. From an ordinary photograph of a matte line-of-sight surface illuminated by the hidden scene, our approach reconstructs a three-dimensional image of the scene hidden behind an occluding structure by exploiting two orthogonal edges of the structure for transverse resolution along azimuth and elevation angles and an information orthogonal scene representation for accurate range resolution. Prior demonstrations beyond two-dimensional reconstructions used expensive, specialized optical systems to gather information about the hidden scene. Here, we achieve accurate three-dimensional imaging using inexpensive, and ubiquitous hardware, without requiring a calibration image. Thus, our system may find use in indoor situations like reconnaissance and search-and-rescue. 

Two-edge-resolved three-dimensional non-line-of-sight imaging with an ordinary camera.

Non-line-of-sight (NLOS) imaging systems involve the measurement of an optical signal at a diffuse surface. A forward model encodes the physics of these measurements mathematically and can be inverted to generate a reconstruction of the hidden scene. Some existing NLOS imaging techniques rely on illuminating the diffuse surface and measuring the photon time of flight (ToF) of multi-bounce light paths. Alternatively, some methods depend on measuring high-frequency variations caused by shadows cast by occluders in the hidden scene. While forward models for ToF-NLOS and Shadow-NLOS have been developed separately, there has been limited work on unifying these two imaging modalities. Dove et al introduced a unified mathematical framework capable of modeling both imaging techniques [Opt. Express27, 18016 (2019)10.1364/OE.27.018016]. The authors utilize this general forward model, known as the two frequency spatial Wigner distribution (TFSWD), to discuss the implications of reconstruction resolution for combining the two modalities but only when the occluder geometry is known a priori. In this work, we develop a graphical representation of the TFSWD forward model and apply it to novel experimental setups with potential applications in NLOS imaging. Furthermore, we use this unified framework to explore the potential of combining these two imaging modalities in situations where the occluder geometry is not known in advance.

Towards a More Accurate Light Transport Model for Non-Line-Of-Sight Imaging

Passive non-line-of-sight (NLOS) imaging has potential applications in autonomous driving and search and rescue, but current deep learning approaches often produce suboptimal images due to sparse and homogeneous projection features, leading to an ill-posed reconstruction process. To address this, we propose the Hyperspectral Fusion NLOS imaging technique (HFN-Net), which first leverages high-dimensional features from multiple spectra and incorporates spatial-spectral attention from a hyperspectral full-color auto-encoder. This method improves color fidelity and structural details by fully utilizing the limited information and increasing feature diversity. Additionally, we developed the Hyperspectral NLOS dataset (HS-NLOS) for training and evaluation. Experimental results show that HFN-Net offers performance improvements over traditional passive NLOS 2D imaging techniques, emphasizing the importance of multi-spectral information. 

Hyper-NLOS: Hyperspectral Passive Non-Line-of-Sight Imaging

Non-line-of-sight imaging (NLOS) is the process of estimating information about a scene that is hidden from the direct line of sight of the camera. NLOS imaging typically requires time-resolved detectors and a laser source for illumination, which are both expensive and computationally intensive to handle. In this paper, we propose an NLOS-based localization and posture classification technique that uses an off-the-shelf projector and camera system. We leverage a message-passing neural network to learn a visible scene geometry and predict the best position to be spotlighted by the projector that can maximize the NLOS signal. The neural network is trained end-to-end and the network parameters are optimized to maximize the NLOS performance. Unlike prior deep-learning-based NLOS techniques that assume planar relay walls, our system allows us to handle line-of-sight scenes where scene geometries are more arbitrary. Our method demonstrates state-of-the-art performance in object localization and position classification using both synthetic and real scenes.

Learning-based Spotlight Position Optimization for Non-Line-of-Sight Human Localization and Posture Classification

In a built environment, wanting to see without direct line of sight is often due to being outside a doorway. The two vertical edges of the doorway provide occlusions that can be exploited for non-line-of-sight imaging by forming corner cameras. While each corner camera can separately yield a robust 1D reconstruction, joint processing suggests novelties in both forward modeling and inversion. The resulting doorway camera provides accurate and robust 2D reconstructions of the hidden scene. This work provides a novel inversion algorithm to jointly estimate two views of change in the hidden scene, using the temporal difference between photographs acquired on the visible side of the doorway. Successful reconstruction is demonstrated in a variety of real and rendered scenarios, including different hidden scenes and lighting conditions. A Cramer-Rao bound analysis is used to demonstrate the 2D resolving power of the doorway camera over other passive acquisition strategies and to motivate the novel biangular reconstruction grid.

Double Your Corners, Double Your Fun: The Doorway Camera

Focused ion beam microscopes require stable beam current for accurate micrograph formation and sample milling. In practice, the beam current emitted by a gas field ion source may fluctuate away from the desired setting due to contamination. The neon beam microscope in particular has been less widely adopted due to poor temporal stability and shorter source lifetime. In a neon beam microscope, beam current variations may be modeled as jumping among a set of values, as shown for a set of two values in Figure 1(a) [1]. When a sample is raster scanned horizontally, these variations give rise to horizontal stripe artifacts in the micrograph, as shown in Figure 1(d). Existing microscopes do not measure the beam current, however mitigation techniques have been developed to remove horizontal content from micrographs post-facto [2]. These methods require tuning and can remove horizontal sample features in addition to the targeted artifacts. In [3,4], we showed that the time-resolved (TR) processing methods introduced in [5,6] greatly reduce the artifacts that arise due to imperfectly known beam current. In this work, we demonstrate online estimation of the unknown neon beam current from the same secondary electron count data used to form the micrograph [7]. Our beam current estimate further improves the micrograph and could also be used to prevent sample damage, improve milling accuracy, and for instrument diagnostics.

Addressing Neon Gas Field Ion Source Instability Through Online Beam Current Estimation

The ability to form non-line-of-sight (NLOS) images of changing scenes could be transformative in a variety of fields, including search and rescue, autonomous vehicle navigation, and reconnaissance. Most existing active NLOS methods illuminate the hidden scene using a pulsed laser directed at a relay surface and collect time-resolved measurements of returning light [1]. The prevailing approaches include raster scanning of a rectangular grid on a vertical wall opposite the volume of interest to generate a collection of confocal measurements [2–5]. These and a recent method that uses a horizontal relay surface [6] are inherently limited by the need for laser scanning. Methods that avoid laser scanning track the moving parts of the hidden scene as one or two point targets [7, 8]. In this work, based on more complete optical response modeling yet still without multiple illumination positions, we demonstrate accurate reconstructions of objects in motion and a ‘map’ of the stationary scenery behind them. The ability to count, localize, and characterize the sizes of hidden objects in motion, combined with mapping of the stationary hidden scene, could greatly improve indoor situational awareness in a variety of applications. The challenge of both active and passive NLOS imaging techniques is that measured light returns to the sensor after multiple diffuse bounces. With each bounce, light is scattered in all directions, eliminating directional information, and attenuating light by a factor proportional to the inverse-square of the path length. Particularly in the passive setting, where no illumination is introduced, occluding structures that limit possible light paths have been used to help separate light originating from different directions in the hidden scene [9–17]. Useful structures include the aperture formed by an open window [9] or the inverse pinhole [18] created when a once-present object moves between measurement frames. Unlike other occluding structures, whose shapes must be estimated or somehow known [11, 12, 14, 15, 19], vertical wall edges have a known shape and are often present when NLOS vision is desired. An edge occluder blocks light as a function of its azimuthal incident angle around the corner and, as a result, enables computational recovery of azimuthal information about the hidden scene. This was first demonstrated in the passive setting [10, 20], where 1D (in azimuthal angle) reconstructions of the hidden scene were formed from photographs of the floor adjacent to the occluding edge; 2D reconstruction was demonstrated in a controlled static environment, although the longitudinal information present in the passive measurement was found to be weak [21]. In the active setting, most of the approaches proposed to date scan a pulsed laser over a set of points on a planar Lambertian relay wall and perform time-resolved sensing with a single-photon detector to collect transient information [2–5, 22–27]. To reconstruct large-scale scenes, these approaches generally require scanning a large area of the relay wall and thus a large opening into the hidden volume. To partially alleviate these weaknesses, edge-resolved transient imaging (ERTI) [6] combines the use of an edge occluder from passive NLOS imaging with the transient measurement abilities of active systems. ERTI scans a laser on the floor along an arc around a vertical edge, incrementally illuminating more of the hidden scene with each scan position. Differences between measurements at consecutive scan positions are processed together to reconstruct a large-scale stationary hidden scene. However, the laser scanning requirement is still a limiting constraint. An earlier work using the floor as a relay surface shortens acquisition time by using a 32× 32 pixel SPAD array in conjunction with a stationary laser [8]. Simultaneous measurements from the 1024 pixels and background subtraction are used to track the horizontal position of a hidden object in motion, modeled as a point reflector. In this work, we use similar hardware as in [8] and also use a floor as a relay surface. As illustrated in Figure 1A, our

Non-Line-of-Sight Tracking and Mapping with an Active Corner Camera

In a particle beam microscope, a raster-scanned focused beam of particles interacts with a sample to generate a secondary electron (SE) signal pixel by pixel. Conventionally formed micrographs are noisy because of limitations on acquisition time and dose. Recent work has shown that estimation methods applicable to a time-resolved measurement paradigm can greatly reduce noise, but these methods apply pixel by pixel without exploiting image structure. Raw SE count data can be modeled with a compound Poisson (Neyman Type A) likelihood, which implies data variance that is signal-dependent and greater than the variation in the underlying particle–sample interaction. These statistical properties make methods that assume additive white Gaussian noise ineffective. This article introduces methods for particle beam micrograph denoising that use the plug-and-play framework to exploit image structure while being applicable to the unusual data likelihoods of this modality. Approximations of the data likelihood that vary in accuracy and computational complexity are combined with denoising by total variation regularization, BM3D, and DnCNN. Methods are provided for both conventional and time-resolved measurements, assuming SE counts are available. In simulations representative of helium ion microscopy and scanning electron microscopy, significant improvements in root mean-squared error (RMSE), structural similarity index measure (SSIM), and qualitative appearance are obtained. Average reductions in RMSE are by factors ranging from 2.24 to 4.11.

Denoising Particle Beam Micrographs With Plug-and-Play Methods

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

Christopher Yu

Author Tools

Chat about Author