Abstract: A multispectral image camera captures image data within specific wavelength ranges in narrow wavelength bands across the electromagnetic spectrum. Images from a multispectral camera can extract a additional information that the human eye or a normal camera fails to capture and thus may have important applications in precision agriculture, forestry, medicine, and object identification. Conventional multispectral cameras are made up of multiple image sensors each fitted with a narrow passband wavelength filter and optics, which makes them heavy, bulky, power hungry, and very expensive. The multiple optics also create an image co-registration problem. Here, we demonstrate a single sensor based three band multispectral camera using a narrow spectral band red–green–blue color mosaic in a Bayer pattern integrated on a monochrome CMOS sensor. The narrow band color mosaic is made of a hybrid combination of plasmonic color filters and a heterostructured dielectric multilayer. The demonstrated camera technology has reduced cost, weight, size, and power by almost n times (where n is the number of bands) compared to a conventional multispectral camera.

Abstract: Image demosaicing and super-resolution are two important tasks in color imaging pipeline. So far they have been mostly independently studied in the open literature of deep learning; little is known about the potential benefit of formulating a joint demosaicing and super-resolution (JDSR) problem. In this article, we propose an end-to-end optimization solution to the JDSR problem and demonstrate its practical significance in computational imaging. Our technical contributions are mainly two-fold. On network design, we have developed a Residual-Dense Squeeze-and-Excitation Networks (RDSEN) supported by a pre-demosaicing network (PDNet) as the pre-processing step. We address the issue of spatio-spectral attention for color-filter-array (CFA) data and discuss how to achieve better information flow by concatenating Residue-Dense Squeeze-and-Excitation Blocks (RDSEBs) for JDSR. Experimental results have shown that significant PSNR/SSIM gain can be achieved by RDSEN over previous network architectures including state-of-the-art RCAN. On perceptual optimization, we propose to leverage the latest ideas including relativistic discriminator and pre-excitation perceptual loss function to further improve the visual quality of textured regions in reconstructed images. Our extensive experiment results have shown that Texture-enhanced Relativistic average Generative Adversarial Network (TRaGAN) can produce both subjectively more pleasant images and objectively lower perceptual distortion scores than standard GAN for JDSR. Finally, we have verified the benefit of JDSR to high-quality image reconstruction from real-world Bayer pattern data collected by NASA Mars Curiosity.

Abstract: As digital cameras become more enhanced and small, CCD sensors can relate to only one color of a pixel. This color mosaic pattern is called as Bayer Pattern(BP) which requires processing to obtain a color image with a higher resolution. Each image pixel that undergoes interpolation has a full color spectrum based on surrounding pixel colors. Here we introduce Adaptive CFA(ACFA) interpolation model. For normal image regions hue technique is used while edge regions adapt the new technique. It is proposed to apply fuzzy logic and fuzzy rule which is based on Genetic Algorithm that uses random local search to enhance the PSNR. Medical image reconstruction by this proposed fuzzy based method outperforms the other medical image reconstruction methods.

Abstract: A multispectral image camera captures image data within specific wavelength ranges in narrow wavelength bands across the electromagnetic spectrum. Images from a multispectral camera can extract additional information that the human eye or a normal camera fails to capture and thus may have important applications in precision agriculture, forestry, medicine and object identification. Conventional multispectral cameras are made up of multiple image sensors each fitted with a narrow passband wavelength filter and optics, which makes them heavy, bulky, power hungry and very expensive. The multiple optics also create image co-registration problem. Here, we demonstrate a single sensor based three band multispectral camera using a narrow spectral band RGB colour mosaic in a Bayer pattern integrated on a monochrome CMOS sensor. The narrow band colour mosaic is made of a hybrid combination of plasmonic colour filters and heterostructured dielectric multilayer. The demonstrated camera technology has reduced cost, weight, size and power by almost n times (where n is the number of bands) compared to a conventional multispectral camera.

Abstract: Low lighting images usually contain Poisson noise, which is pixel amplitude-dependent More panchromatic or white pixels in a color filter array (CFA) are believed to help the demosaicing performance in dark environments In this paper, we first introduce a CFA pattern known as CFA 30 that has 75% white pixels, 125% green pixels, and 625% of red and blue pixels We then present algorithms to demosaic this CFA, and demonstrate its performance for normal and low lighting images In addition, a comparative study was performed to evaluate the demosaicing performance of three CFAs, namely the Bayer pattern (CFA 10), the Kodak CFA 20, and the proposed CFA 30 Using a clean Kodak dataset with 12 images, we emulated low lighting conditions by introducing Poisson noise into the clean images In our experiments, normal and low lighting images were used For the low lighting conditions, images with signal-to-noise (SNR) of 10 dBs and 20 dBs were studied We observed that the demosaicing performance in low lighting conditions was improved when there are more white pixels Moreover, denoising can further enhance the demosaicing performance for all CFAs The most important finding is that CFA 30 performs better than CFA 10, but is slightly inferior to CFA 20, in low lighting images

Abstract: In this paper, the impact of demosaicing on gradient extraction is studied and a gradient-based feature extraction pipeline based on raw Bayer pattern images is proposed. It is shown both theoretically and experimentally that the Bayer pattern images are applicable to the central difference gradient-based feature extraction algorithms without performance degradation, or even with superior performance in some cases. The color difference constancy assumption, which is widely used in various demosaicing algorithms, is applied in the proposed Bayer pattern image-based gradient extraction pipeline. Experimental results show that the gradients extracted from Bayer pattern images are robust enough to be used in histogram of oriented gradients (HOG)-based pedestrian detection algorithms and shift-invariant feature transform (SIFT)-based matching algorithms.

Abstract: Spectral image reconstruction from RGB images has emerged as a hot topic in the computer vision community due to easy-access and low-cost acquisition of the latter. The goal is to learn a non-linear mapping from 3-RGB bands to L spectral bands. With the growth of the available spectral datasets, this mapping has been learned using deep convolutional representations. However, these methods demand a large number of spectral images to train the net to obtain a good recovery. In contrast, the proposed process consists of a pre-training step where the weights of a convolutional neural network fit with a large amount of available RGB datasets without spectral mapping, taking into account the RGB system acquisition as a layer. Then, some layers of this pre-trained network are frozen to retrain it with the available spectral dataset to generate a spectral image with L bands. The proposed training scheme can be used with any pre-existing deep network that maps RGB to spectral images and it is here evaluated with a “U-net” architecture, and the RGB sensing is based on the Bayer filter pattern. The simulated and experimental data demonstrate the effectiveness of the proposed approach compared to training without transfer learning, showing a gain of up to 4 dB, with less spectral data.

Abstract: High-throughput measurement of thermal deformation and determination of coefficient of thermal expansion (CTE) using a high-resolution digital single lens reflex (DSLR) camera and digital image correlation (DIC) is described. To mitigate the mosaic effect caused by the Bayer filter of DSLR cameras, a color image pre-processing method, which adjusts the brightness and equalizes the color channels of the raw image, is carried out. In addition, a Gaussian pre-filtering step is adopted for denoising the images captured with DSLR cameras to enhance the subpixel registration accuracy. Then, by processing the recorded images using the state-of-the-art DIC algorithm, full-field displacements and strains can be determined. Compared with conventional industrial cameras, a DSLR camera offers not only portability, compactness, and economy but also much higher resolution of recorded images, allowing CTE characterization with higher throughput. Real experiments, including a verification experiment of the color image pre-processing technique, a benchmark CTE determination of Al alloy, and a high-throughput CTE determination of 15 samples of three different metals, validated the feasibility and accuracy of the proposed technique. The proposed method is cost-effective and time-saving, showing great potential in the high-throughput CTE measurement and other high-throughput strain measurement scenarios.

Abstract: In CFA 2.0, there are white pixels in a color filter array (CFA) that has proven to help the demosaicing performance for images collected in low light conditions. Here, we evaluate the performance of demosaicing for images collected in low light conditions using an RGBW pattern with 75% white pixels. We term this CFA the CFA 3.0. Using a data set containing 10 images collected in low light conditions, we performed extensive experiments. Both objective and subjective evaluations were used in our experiments.

Abstract: It is commonly believed that having more white pixels in a color filter array (CFA) will help the demosaicing performance for images collected in low lighting conditions. We present a comparative study to evaluate the performance of demosaicing for images collected in realistic low lighting conditions using two CFAs: the standard Bayer pattern (aka CFA 1.0) and the Kodak CFA 2.0 (RGBW pattern with 50% white pixels). Using a data set containing 10 images collected in low lighting conditions, we observe that having more white pixels does help the demosaicing performance. However, some cautions are needed in quantifying the performance.

Abstract: An algorithm for correcting color contamination is proposed for two-color, double-exposure particle tracking velocimetry (PTV). A two-color PTV system used in this study consists of blue and green laser diodes and a consumer digital camera, where a laser pulser with an avalanche transistor is developed for achieving optical pulses of 50 ns for application in airflows. In the PTV, the camera captures images of tracer particles illuminated by a sequence of green and blue light pulses with a certain time interval. Because of spectral characteristics of a Bayer filter, camera sensors in a blue channel respond to green light scattered by the particles. This color contamination results in pseudo-particles in the blue channel. Pixels occupied by the pseudo-particles have very high correlation of light intensity between green and blue channels. In the proposed method, the pseudo-particles caused by the color contamination are detected and removed based on the high correlation. The present color contamination correction hardly affects real particles illuminated by the blue laser diodes. Measurements of an airflow induced by DC fans confirm that the proposed system with the color contamination correction works well for PTV.

Abstract: In modern digital cameras, the Bayer color filter array (CFA) has been widely used. It is also widely known as CFA 1.0. However, Bayer pattern is inferior to the red-green-blue-white (RGBW) pattern, which is also known as CFA 2.0, in low lighting conditions in which Poisson noise is present. It is well known that demosaicing algorithms cannot effectively deal with Poisson noise and additional denoising is needed in order to improve the image quality. In this paper, we propose to evaluate various conventional and deep learning based denoising algorithms for CFA 2.0 in low lighting conditions. We will also investigate the impact of the location of denoising, which refers to whether the denoising is done before or after a critical step of demosaicing. Extensive experiments show that some denoising algorithms can indeed improve the image quality in low lighting conditions. We also noticed that the location of denoising plays an important role in the overall demosaicing performance.

Abstract: In this paper, we introduce a novel color pixel reconstruction algorithm for a RGBZ CMOS imager. A RGBZ imager is a Vision System on Chip (VSoC) sensor that captures simultaneously color and depth information with a hybrid pixel matrix. The imager pattern is based on a Bayer filter array where the Z pixel covers an equivalent area of a 2x2-color-pixel size, and one quarter of the total color information is missing. We propose an original adaptive algorithm to reconstruct the color information using a new operator called semi-gradient. The results show improvements on edges, corners, and narrow lines reconstruction, and a reduction of color and structural artefacts compared to classical reconstruction algorithms.

Abstract: Spectral image reconstruction from RGB images has emerged as a hot topic in the computer vision community due to easy-access and low-cost acquisition compared with traditional spectral imaging acquisition methods. With the growth of the available spectral data-sets, this reconstruction problem has been effectively addressed using deep convolutional neural networks (CNN). The goal is to learn a non-linear mapping from 3-RGB bands to L spectral bands. However, these methods demand many spectral images to train the CNN to obtain a good recovery. In contrast, the proposed process consists of a pre-training step where the weights of a convolutional neural network fit with a large number of RGB image data sets available without its corresponding ground-truth spectral images, taking into account the RGB spectral response of the camera which is modeled as a non-trainable layer. Then, some layers of this pre-trained network are frozen to retrain it with the available spectral data-set to generate a spectral image with L bands. The proposed training scheme can be used with any pre-existing deep network that maps RGB to spectral images, and it is here evaluated with a “U-net” architecture. The RGB sensing is based on the Bayer filter pattern from a Nikon D90 DSLR camera. The simulated and experimental data demonstrate the effectiveness of the proposed approach compared to training without transfer learning, showing a gain of up to 4 dB, with less spectral data.

Abstract: Most modern color digital cameras are equipped with a single image sensor with a color filter array (CFA). One of the most important stages of preprocessing is noise reduction. Most research related to this topic ignores the problem associated with the actual color image acquisition process and assumes that we are processing the image in the sRGB space. In the presented paper, the real process of developing raw images obtained from the CFA sensor was analyzed. As part of the work, a diverse database of test images in the form of a digital negative and its reference version was prepared. The main problem posed in the work was the location of the denoising and demosaicing algorithms in the entire raw image processing pipeline. For this purpose, all stages of processing the digital negative are reproduced. The process of noise generation in the image sensors was also simulated, parameterizing it with ISO sensitivity for a specific CMOS sensor. In this work, we tested commonly used algorithms based on the idea of non-local means, such as NLM or BM3D, in combination with various techniques of interpolation of CFA sensor data. Our experiments have shown that the use of noise reduction methods directly on the raw sensor data, improves the final result only in the case of highly disturbed images, which corresponds to the process of image acquisition in difficult lighting conditions.

Abstract: The article deals with theoretical aspects of CMOS sensor cross-links effect as a main drawback of the dispersion method implementation with CMOS video camera and RGB optical radiation source. The paper is concerned with usage of diffraction grating to solve the problem of images overlapping on CMOS sensor with Bayer pattern. It is shown that the diffraction distribution allows to determine the energy centers of different images of the same RGB optical radiation source independently at the same time without overlapping. The text gives mathematical description of how diffraction image is formed and how to use it in dispersion method. The article is of interest to people who deals with optical radiation propagation through the air tract with vertical temperature gradient.

Abstract: To measure object wave in digital holography, the phase-shifting technique is popular by changing the phase of the reference wave. A piezo actuator (PZT) is usually used as a device to shift the phase. In a case to obtain the phase information of color object with multiple wavelengths, four-step phase-shifting algorithm with quarter wavelength shift is not convenient since the amount of the phase shift is difference according to each wavelength. In this paper, the generalized four-step phase-shifting method is proposed and experimentally verified to obtain color digital hologram using an image sensor with a Bayer pattern for capturing interference pattern shifted phase by one PZT.

Abstract: We have developed one-shot RGB-spectroscopic full-field OCT (FF-OCT). In this system, red (R), green (G), and blue (B) lights emitted from LED light sources are synthesized into the light incident to a Michelson interferometer. The interference image at the detecting plane are separated into RGB images by a Bayer filter on a single-panel CMOS color camera.We show the possibility of RGB-spectroscopic OCT imaging in the one-shot operation in this study.

Abstract: A novel approach for image demosaicking based on adaptive lattice-aware filter (ALF) and global refinement unit (GRU) is proposed in this work. We generate ALFs dynamically, which are adaptive to positions of pixels within color lattices in a color filter array, to obtain a locally demosaicked image. We then refine the locally demosaicked image using GRU to exploit global information, as well as local information. To extend the receptive fields efficiently, we adopt dilated convolutions in GRU. Experimental results demonstrate that the proposed algorithm provides the state-of-the-art performances in standard demosaicking datasets.

Abstract: A number of works using Unmanned Aerial Vehicles (UAV) have been taking advantage of the ADC Lite multispectral camera by Tetracam. The images acquired by the camera have been qualitatively and quantitatively analyzed by many authors without previously correcting radiometric distortions, which is mandatory for practical and academic purposes. In this paper we made it clear that distortions such as the read noise, pixels non-uniform response to the incoming radiance and the vignetting effect are present in the ADC images. Beside these distortions, the weighting values in the Color Processing File (CPF), provided by Tetracam and used to interpolate the Bayer filter, delivers wrong bright values in a number of situations, causing information loss, particularly in the Red band, producing very high frequencies (~105) of black saturated pixels. The correction of these effects and remaining noise artifacts in the image frame (caused by the color interpolation) is mandatory when working with them, which is clearly demonstrated in this paper when we calculated a normalized band ratio from the raw and corrected images.

Abstract: Embodiments relate to directional bilateral filtering of a raw image. For each pixel in the image, a block of pixels surrounding that pixel is used for filtering. When the block of pixels in a Bayer pattern have directionality, directional filter coefficients are used instead of default filter coefficients. To obtain a directional tap, a directional filter coefficient is attenuated by an attenuation factor that differs based at least on the location of the pixels in the pixel block. The directional taps are blended with non-directional taps derived from the default filter coefficients using a weight representing confidence on the directionality. The filtered pixel values are then obtained by multiplying pixel values with corresponding taps.

Abstract: A recent image sensor in a camera contains millions of pixels. The pixel itself records only the intensity of a single light signal of red, green, or blue in the Bayer patterned sensor. When the camera moves position slightly, however, the different kinds of wavelength light signals can be recorded in a different pixel. In other words, we can complement the missing color signals in the Bayer pattern by the captured ray from the moved camera. Consequently, a full-color image can be generated without demosaicing, but the capturing method may cause a false-color. In our proposed method, we take more than 50 images to complement the pixels because the camera position is moved artificially rather than mechanically. Besides, we use the dense optical flow to track the movement of all pixels between the reference image and the other images. All images are converted from Bayer images to grayscale images to calculate the optical flow. Then, the pixels are linked according to the optical flow, the missing color pixel values in reference Bayer image are complemented from the other corresponding Bayer images. In our experiments, we generated sharper images than the demosaiced method.

Abstract: The present invention relates to a Bayesian based image restoration method for a camera which comprises the steps of: generating a Bayer pattern for an image; calculating an optimal Bayesian estimator (OBE); reconstructing a green channel by using the calculated OBE with regard to the Bayer pattern; reconstructing red/blue channels based on the reconstructed green channel; and refining the reconstructed green channel. Therefore, the present invention can improve CPSNR, S-CIELAB, FSIM, and zipper effect measurement while maintaining high efficiency compared to a conventional demosaicking method.

Abstract: A method of chromatically-corrected image demosaicing for a sensor with a colour filter array (CFA) such as a Bayer filter array. The demosaicing makes use of local chromatic aberration correction data to determine chromatically-corrected pixel positions for spatial interpolation. In implementations a chromatically-corrected quadratic adjustment is also applied.

Abstract: An image sensing apparatus including a pixel array comprising two or more photo-sensor elements, a first optical filter disposed on a first photo-sensor element of the pixel array and a second optical filter disposed on a second photo-sensor element of the pixel array. The first optical filter is configured such that a spectral response of the first optical filter includes a first passband in a first wavelength range and a second passband in a second wavelength range, where the first passband and the second passband are separated by a first stop band. The second optical filter is configured such that a spectral response of the second optical filter includes a third passband in the first wavelength range and a fourth passband in the second wavelength range, where the third passband and the fourth passband are separated by a second stop band.

Abstract: The present invention is to provide an image signal processing method, an image signal processor, and an image sensor chip for performing stabilization on a Bayer pattern image using a gyro sensor. The image signal processing method according to an embodiment of the present invention includes the steps of: generating Bayer order status information indicating whether a Bayer order of a Bayer pattern image is changed based on parallel translation information of gyro information; performing parallel translation correction on the Bayer pattern image by using the parallel translation information; and performing interpolation on the Bayer pattern image on which the parallel translation correction has been performed based on the Bayer order status.

Abstract: A video camera or video camera module comprises: an RGB, Bayer filter, single chip image sensor, to generate raw video data; a computer processor to perform image processing on the raw RGB (colour) video data, comprising at least one pre-processing step, followed by post processing video compression; a detector to determine an excess of light in at least one RGB colour channel compared to the other RGB colour channel(s), consistent with laser dazzle from at least one type of laser, and to output to the processor a determination of at least one specified RGB colour channel as being subject to such excess of light, either based on video data from the RGB sensor, or based on data from a light spectrum sensor or laser detector. The computer processor is adapted to respond to the determination of excess light by reducing values of the specified RGB colour channel(s) as at least part of the at least one pre-processing step in advance of the post processing compression. Thus, in the event of laser dazzle that is specific to a colour channel, the compressed video data generated will retain more detail from the other colour channels.

Abstract: A high dynamic range sensing device is disclosed. The device includes an array of Bayer pattern units. Each of the Bayer pattern units comprises a plurality of pixels and each of the plurality of pixels comprises a plurality of photodiodes. At least one of the plurality of photodiodes in each pixel is configured to detect near infrared (NIR) light and at least one of the plurality of photodiodes in each of the plurality of pixels is configured to detect visible light.