Abstract: In this paper, we present new data pre-processing and augmentation techniques for DNN-based raw image denoising. Compared with traditional RGB image denoising, performing this task on direct camera sensor readings presents new challenges such as how to effectively handle various Bayer patterns from different data sources, and subsequently how to perform valid data augmentation with raw images. To address the first problem, we propose a Bayer pattern unification (BayerUnify) method to unify different Bayer patterns. This allows us to fully utilize a heterogeneous dataset to train a single denoising model instead of training one model for each pattern. Furthermore, while it is essential to augment the dataset to improve model generalization and performance, we discovered that it is error-prone to modify raw images by adapting augmentation methods designed for RGB images. Towards this end, we present a Bayer preserving augmentation (BayerAug) method as an effective approach for raw image augmentation. Combining these data processing technqiues with a modified U-Net, our method achieves a PSNR of 52.11 and a SSIM of 0.9969 in NTIRE 2019 Real Image Denoising Challenge, demonstrating the state-of-the-art performance. Our code is available at this https URL.

Abstract: We propose a Bayer ME algorithm which is used to improve the performance of Motion-Compensated Frame Interpolation (MCFI). The core of the proposed algorithm is a predictive model designed from the alternate arrangement of Bayer pattern. According to the predictive model, the Motion Vector Field (MVF) of the interpolated frame is first split into basic blocks and absent blocks, and then an improved Bilateral Motion Estimation (BME) is proposed to compute the MVs of basic blocks. Finally, with the local stationary statistics of MVF, the MV of an absent block is predicted from the MVs of its neighboring basic blocks. Experimental results show that the proposed Bayer ME algorithm can improve both objective and subjective quality of the interpolated frame with a low computational complexity.

Abstract: The security of multimedia documents becomes an urgent need, especially with the increasing image falsifications provided by the easy access and use of image manipulation tools. Hence, usage of image authentication techniques fulfills this need. In this paper, we propose an effective self-embedding fragile watermarking scheme for color images tamper detection and restoration. To decrease the capacity of insertion, a Bayer pattern is used to reduce the color host image into a gray-level watermark, to further improve the security Torus Automorphism permutation is used to scramble the gray-level watermark. In our algorithm, three copies of the watermark are inserted over three components (R, G, and B channels) of the color host image, providing a high probability of detection accuracy and recovery if one copy is destroyed. In the tamper detection process, a majority voting technique is used to determine the legitimacy of the image and recover the tampered regions after interpolating the extracted gray-level watermark. Using our proposed method, tampering rate can achieve 25% with a high visual quality of recovered image and PSNR values greater than 34 (dB). Experimental results demonstrate that the proposed method affords three major properties: the high quality of watermarked image, the sensitive tamper detection and high localization accuracy besides the high-quality of recovered image.

Abstract: Bayer pattern filters have been used in many commercial digital cameras. In National Aeronautics and Space Administration’s (NASA) mast camera (Mastcam) imaging system, onboard the Mars Science Laboratory (MSL) rover Curiosity, a Bayer pattern filter is being used to capture the RGB (red, green, and blue) color of scenes on Mars. The Mastcam has two cameras: left and right. The right camera has three times better resolution than that of the left. It is well known that demosaicing introduces color and zipper artifacts. Here, we present a comparative study of demosaicing results using conventional and deep learning algorithms. Sixteen left and 15 right Mastcam images were used in our experiments. Due to a lack of ground truth images for Mastcam data from Mars, we compared the various algorithms using a blind image quality assessment model. It was observed that no one algorithm can work the best for all images. In particular, a deep learning-based algorithm worked the best for the right Mastcam images and a conventional algorithm achieved the best results for the left Mastcam images. Moreover, subjective evaluation of five demosaiced Mastcam images was also used to compare the various algorithms.

Abstract: It is commonly believed that having more white pixels in a color filter array (CFA) will help the demosaicing performance for images collected in low lighting conditions. However, to the best of our knowledge, a systematic study to demonstrate the above statement does not exist. We present a comparative study to systematically and thoroughly evaluate the performance of demosaicing for low lighting images using two CFAs: the standard Bayer pattern (aka CFA 1.0) and the Kodak CFA 2.0 (RGBW pattern with 50% white pixels). Using the clean Kodak dataset containing 12 images, we first emulated low lighting images by injecting Poisson noise at two signal-to-noise (SNR) levels: 10 dBs and 20 dBs. We then created CFA 1.0 and CFA 2.0 images for the noisy images. After that, we applied more than 15 conventional and deep learning based demosaicing algorithms to demosaic the CFA patterns. Using both objectives with five performance metrics and subjective visualization, we observe that having more white pixels indeed helps the demosaicing performance in low lighting conditions. This thorough comparative study is our first contribution. With denoising, we observed that the demosaicing performance of both CFAs has been improved by several dBs. This can be considered as our second contribution. Moreover, we noticed that denoising before demosaicing is more effective than denoising after demosaicing. Answering the question of where denoising should be applied is our third contribution. We also noticed that denoising plays a slightly more important role in 10 dBs signal-to-noise ratio (SNR) as compared to 20 dBs SNR. Some discussions on the following phenomena are also included: (1) why CFA 2.0 performed better than CFA 1.0; (2) why denoising was more effective before demosaicing than after demosaicing; and (3) why denoising helped more at low SNRs than at high SNRs.

Abstract: Image sensors in digital cameras use a technology called "Bayer Patterns" allowing color photography with a planar arrangements of photosensitive elements. Alternating arrangements of red, green and blue filter masks on top of a rectangular grid of such elements allow capturing of color information, but also require a de-mosaicing algorithm to reconstruct a full-resolution color image from the sensor data. In high-speed applications, or applications where the system design requires low-latency, low-complexity compression, the JPEG XS standard of the JPEG committee offers an elegant solution to compress Bayer pattern images close to the sensor, and to transmit the compressed data over a lower bandwidth connection while maintaining visually lossless quality. This paper presents contributions to JPEG XS that are currently under discussion in the JPEG committee (SC29WG1) as parts of an amendment for Bayer pattern compression.

Abstract: The most widely used color filter array (CFA) pattern in commercial digital color cameras is the Bayer pattern, and the captured image is called the Bayer CFA image, in which each pixel contains only one color value and each image consists of 25% red, 50% green, and 25% blue color values. The chroma 4:2:2 or 4:2:0 subsampling of Bayer CFA images is a necessary process prior to compression. According to the block-distortion minimization principle, in this paper, we propose an effective gradient descent-based chroma subsampling (GDCS) method for Bayer CFA images. Based on the test Bayer CFA images collected from the Kodak and IMAX datasets, experimental results demonstrated that in high efficiency video coding, our GDCS method has better quality and quality-bitrate tradeoff performance of the reconstructed images when compared with the existing chroma subsampling methods.

Abstract: Bayer pattern is a low cost approach to generating RGB images in commercial digital cameras. In NASA's mast camera (Mastcams) onboard the Mars rover Curiosity, Bayer pattern has also been used in capturing the RGB bands. It is well known that debayering (also known as demosaicing) introduces color and zipper artifacts. Currently, NASA is using a demosaicing algorithm developed in early 2000’s. It is probably the right time to assess some state-of-the-art algorithms and recommend a more recent and powerful approach to NASA for its future missions. In this paper, we present results of a comparative study on the use of conventional and deep learning algorithms for demosaicing Mastcam images. Due to lack of ground truth, subjective evaluation has been used in our study.

Abstract: Sensors in digital cameras use a technology called "Bayer Patterns'" allowing color photography with a planar arrangements of sensor elements. Alternating arrangements of red, green and blue filter masks on top of a rectangular grid of sensor elements allow capturing of color information, but also require a de-mosaicing algorithm to reconstruct a full-resolution color image from the sensor data. In high-speed applications, or applications where the system design requires low-latency, low-complexity compression, the JPEG~XS standard of the JPEG committee offers an elegant solution to compress Bayer pattern images close to the sensor, and to transmit the compressed data over a lower bandwidth connection while maintaining visually lossless quality. This paper discusses algorithms, and in particular modifications to the rate allocation of JPEG XS to enable compression of Bayer pattern sensor data. The main contribution of this work is a modified algorithm to compute band gains and band priorities that ensures PSNR-optimality of the demosaiced image rather the Bayer pattern itself.

Abstract: 12 Mpix color commercial-of-the-shelf (COTS) image sensor from CMOSIS was tested with proton radiation. The target mission required an irradiation with protons of energy of 50 MeV and fluences up to 1·1012 p/cm2. Several intermediate steps were introduced to check the behavior of the image sensor. A low-cost test camera was developed to control the image sensors, acquire the images, and monitor the currents and voltages during the tests. Each color was characterized separately according to the EMVA 1288 standard. Such treatment allowed also analysis of the bayer filter deposited on the image sensor surface. Post-radiation characterization revealed that a significant deterioration in the parameter performance was found independently of the pixel color. The most affected parameters were dark current and dark signal nonuniformities (DSNU) which have increased from about one to two orders of magnitude.

Abstract: Systems and methods spectrally and radiometrically calibrate an optical spectrum detected with a color-image sensor of an optical spectrometer. When the color-image sensor includes a Bayer filter, the red-peaked, green-peaked, and blue-peaked spectral responses of the color filters forming the Bayer filter may be used to identify unique spectral signatures in the red, green, and blue color channels. These spectral signatures may be used to associate calibration wavelengths to the pixel locations of the color-image sensor where the spectral signatures are observed. A fitted model may then be used to associate a wavelength to each pixel location of the color-image sensor. These systems and methods account for translational shifts of the optical spectrum on the color-image sensor induced by optical image stabilization, and thus may aid optical spectrometry utilizing a digital camera in a smartphone or tablet computer.

Abstract: In this paper, we propose a weights-based image demosaicking algorithm which is based on the Bayer pattern color filter array (CFA). When reconstructing the missing G components, the proposed algorithm uses weights based on posteriori gradients to mitigate color artifacts and distortions. Furthermore, the proposed algorithm makes full use of the correlation of R–B channels in high frequency when interpolating R/B values at B/R positions. Experimental results show that the proposed algorithm is superior to previous similar algorithms in composite peak signal-to-noise ratio (CPSNR) and subjective visual effect. The biggest advantage of the proposed algorithm is the use of posteriori gradients and the correlation of R–B channels in high frequency.

Abstract: Image segmentation is a challenging task in computer vision fields. In this paper, we aim to distinguish green peppers from large amounts of green leaves by using hyperspectral information. Our key aim is to design a novel optical filter to identify the bands where peppers differ substantially from green leaves. We design an optical filter as a learnable weight in front of an RGB filter with a fixed weight, and classify green peppers in an end-to-end manner. Our work consists of two stages. In the first stage, we obtain the optical filter parameters by training an optical filter and a small neural network simultaneously at the pixel level of hyperspectral data. In the second stage, we apply the learned optical filter and an RGB filter in a successive manner to a hyperspectral image to obtain an RGB image. Then we use a SegNet-based network to obtain better segmentation results at the image level. Our experimental results demonstrate that this two-stage method performs well for a small dataset and the optical filter helps to improve segmentation accuracy.

Abstract: Image demosaicking is a method of reconstructing an RGB image from a Bayer pattern, which is required when color information is lacking because a single charge-coupled device is used during the image extraction process of a digital camera. There are many restrictions on the reconstruction of a Bayer image to an RGB image. Given that each pixel contains only one-color information, artifacts, such as false color or the zipper effect, may occur at the edges, which can arise as a result of significant differences in brightness and color change. We propose a demosaicking method for adaptively selecting the reference range of color difference to obtain reliable information from texture regions and reconstructing into the RGB image. In particular, we determine the adaptive weight and reference range for four directions, east (E), west (W), south (S), and north (N), to improve the reliability of the color pixel obtained by a color difference estimation using guided filtering applied on residuals. In our experiment, we compare the results of the proposed method for the Kodak and IMAX datasets with those of nine demosaicking methods. The proposed method shows similar or improved results in terms of the color peak signal-to-noise ratio. In addition, compared to other methods, the visual quality improved by reducing residual artifacts.

Abstract: . The main idea of this particular study was to validate if the new FOVEON technology implemented by sigma cameras can provide better overall results and outperform the traditional Bayer pattern sensor cameras regarding the radiometric information that records as well as the photogrammetric point cloud quality that can provide. Based on that, the scope of this paper is separated into two evaluations. First task is to evaluate the quality of information reconstructed during de-mosaicking step for Bayer pattern cameras by detecting potential additional colour distortion added during the de-mosaicking step, and second task is the geometric comparisons of point clouds generated by the photos by Bayer and FOVEON sensors against a reference point cloud. The first phase of the study is done using various de-mosaicking algorithms to process various artificial Bayern pattern images and then compare them with reference FOVEON images. The second phase of the study is carried on by reconstructing 3D point clouds of the same objects captured by a Bayer and a FOVEON sensor respectively and then comparing the various point clouds with a reference one, generated by a structured light hand-held scanner. The comparison is separated into two parts, where initially we evaluate five separate point clouds (RGB, Gray, Red, Green, Blue) for each camera sensor per site and then a second comparison is evaluated on colour classified RGB point cloud segments.

Abstract: The present invention relates to an image signal processor (ISP) for controlling a total shutter image sensor module on a stroboscope. According to the present invention, an ISP (112), which processes image signal processes for a total shutter image sensor module (111) obtaining images to be captured by using a total shutter scheme with mosaic and Bayer patterns, includes: a black level adjustment module (112a) which adjusts the black level of each pixel included in the Bayer pattern images of the total shutter scheme obtained by the total shutter image sensor module (111) and obtains the pixel information of each pixel with the adjusted black level; and a noise reduction module (112b) which executes boundary highlighting and noise elimination processes in order to clearly sense the boundaries in the images and eliminate the noise occurring in the images with the pixel information of each pixel with the adjusted black level and improve the quality of the images. While observing the larynx images displayed on a display monitor by signal processing on the total shutter image sensor module (111), the ISP can prevent the distortion, shaking, and partial exposure in order to provide excellent images similar to the images captured with the naked eye, thereby ensuring excellent diagnostic and inspection functions and enabling differentiation from other related existing equipment in the market for enhanced economic value.

Abstract: The image registration by digital still cameras and video cameras requires color filters to be posed onto the photosensitive sensors (CCD or CMOS). The filters are arranged in patterns across the face of the image sensing array. The conventional color filter array (CFA) capture only one color component at each image pixel. The missing colors in the raw sensor data are interpolated by a process called CFA interpolation or demosaicing. Quality of the full-color reconstruction process is mostly relied on demosaicing method applied. Most of the current demosaicing methods are computationally expensive and often too slow for real-time scenarios. Many industrial applications require real-time and high quality demosaicing solutions, and quite often slow image reconstruction process is a real bottleneck. The purpose of this research is to present a comparative performance study of demosaicing algorithms on general-purpose GPUs. The experimental results of CUDA-based implementations of two state-of-the-art and widely applied in practice CFA algorithms are presented. The performance efficiency is assessed and analyzed by experimental studies on a set of real photographic test images on two general-purpose graphic cards. The obtained results demonstrated the benefit of exploiting the contemporary GPUs in speeding up the demosaicing process, especially for practical applications that need to meet real-time and high-speed video processing requirements combined with high quality of the full-color image reconstruction.

Abstract: An improved image intensified, low light level sensor to enable true color images in real-time from a prior art image intensified, low light level device. The device comprises a Bayer pattern in front of the photocathode element and a CCD or CMOS imager that observes or replaces the phosphor layer of the traditional low light level, intensified imager.

Abstract: We design and experimentally demonstrate micron- and submicron-sized color filters in the visible region using metal–dielectric–metal (MDM) Fabry–Perot cavity arrays. Large-area MDM filters (150 μm × 150 μm) of varying dielectric thicknesses show polarization-independent bandpass filtering of primary and secondary colors. The lateral dimensions of each cavity and the spacing between adjacent cavities are systematically reduced to study the possibility of using these color filters as Bayer filter arrays for color cameras. Up to micron-sized dimensions with micron-sized spacing between the adjacent MDM cavities, it is observed that the dominant mechanism for color filtering is Fabry–Perot effect. However, as the size and the pitch of the arrays of MDM are reduced further to the submicron length scale, polarization-dependent transmission is observed, and the dominant effect for color filtering is observed to be the leaked radiation from the grating-coupled plasmons. By keeping the pitch and lateral dimensions of MDM arrays fixed for plasmon resonance, the dielectric spacer thickness is systematically varied. Our electromagnetic simulations and microscopy-based imaging reveal that there exists a critical dielectric thickness for a specific cavity period, where the Fabry–Perot resonance and the Plasmons couple to each other and exhibit a high transmittivity. However, at other dielectric thicknesses, two weaker transmission peaks corresponding to decoupled modes; one corresponding to Fabry–Perot effect and the other corresponding to the signature of grating-coupled plasmons are present. Such MDM filter arrays can act as polarization-insensitive Bayer filters, as well as current-drawing contacts of multispectral imaging sensors with pixel sizes and separations, each up to a micron. With the pixel sizes in submicron regime, the integrated color filters with photodiodes may be useful for polarization-sensitive color imaging and surface sensors.

Abstract: An image sensor may include a pixel array having an N×M array of merged pixels arranged in a Bayer pattern, each merged pixel including an k*l matrix of unit pixels of a same color, where k and l are integers greater than two; and an image signal processor to process signals output by the pixel array in accordance with a normal mode or a zoom-in mode. In the zoom-in mode, signals from the pixel array may be remosaiced such that signals corresponding to the unit pixels are arranged in a p*q matrix of unit pixels of a same color, wherein p is a non-negative integer that is less than k and q is a non-negative integer less than 1, the p*q matrixes being arranged in a Bayer pattern.

Abstract: Image demosaicing and super-resolution are two important tasks in color imaging pipeline. So far they have been mostly independently studied in the open literature of deep learning; little is known about the potential benefit of formulating a joint demosaicing and super-resolution (JDSR) problem. In this paper, we propose an end-to-end optimization solution to the JDSR problem and demonstrate its practical significance in computational imaging. Our technical contributions are mainly two-fold. On network design, we have developed a Residual-Dense Squeeze-and-Excitation Networks (RDSEN) supported by a pre-demosaicing network (PDNet) as the pre-processing step. We address the issue of spatio-spectral attention for color-filter-array (CFA) data and discuss how to achieve better information flow by concatenating Residue-Dense Squeeze-and-Excitation Blocks (RDSEBs) for JDSR. Experimental results have shown that significant PSNR/SSIM gain can be achieved by RDSEN over previous network architectures including state-of-the-art RCAN. On perceptual optimization, we propose to leverage the latest ideas including relativistic discriminator and pre-excitation perceptual loss function to further improve the visual quality of textured regions in reconstructed images. Our extensive experiment results have shown that Texture-enhanced Relativistic average Generative Adversarial Network (TRaGAN) can produce both subjectively more pleasant images and objectively lower perceptual distortion scores than standard GAN for JDSR. Finally, we have verified the benefit of JDSR to high-quality image reconstruction from real-world Bayer pattern data collected by NASA Mars Curiosity.

Abstract: The RGBW color filter arrays (CFA), also known as CFA2.0, contains R, G, B, and white (W) pixels. It is a 4 × 4 pattern that has 8 white pixels, 4 green pixels, 2 red pixels, and 2 blue pixels. The pattern repeats itself over the whole image. In an earlier conference paper, we cast the demosaicing process for CFA2.0 as a pansharpening problem. That formulation is modular and allows us to insert different pansharpening algorithms for demosaicing. New algorithms in interpolation and demosaicing can also be used. In this paper, we propose a new enhancement of our earlier approach by integrating a deep learning-based algorithm into the framework. Extensive experiments using IMAX and Kodak images clearly demonstrated that the new approach improved the demosaicing performance even further.

Abstract: Digital color imaging relies on spectral filters on top of a pixelated sensor, such as a CMOS image sensor. An important parameter of imaging devices is their resolution, which depends on the size of the pixels. For many applications, a high resolution is desirable, consequently requiring small spectral filters. Dielectric nanostructures, due to their resonant behavior and its tunability, offer the possibility to be assembled into flexible and miniature spectral filters, which could potentially replace conventional pigmented and dye-based color filters. In this paper, we demonstrate the generation of transmissive structural colors based on uniform-height amorphous silicon nanostructures. We optimize the structures for the primary RGB colors and report the construction of submicrometer RGB filter arrays for a pixel size down to 0.5 ?m.

Abstract: In this paper we deal with the demosaicing problem when the Bayer pattern is used. We propose a fast heuristic algorithm, consisting of three parts. In the first one, we initialize the green channel by means of an edge-directed and weighted average technique. In the second part, the red and blue channels are updated, thanks to an equality constraint on the second derivatives. The third part consists of a constant-hue-based interpolation. We show experimentally how the proposed algorithm gives in mean better reconstructions than more computationally expensive algorithms.

Abstract: Capturing images at high ISO modes will introduce much realistic noise, which is difficult to be removed by traditional denoising methods. In this paper, we propose a novel denoising method for high ISO JPEG images via deep fusion of collaborative and convolutional filtering. Collaborative filtering explores the non-local similarity of natural images, while convolutional filtering takes advantage of the large capacity of convolutional neural networks (CNNs) to infer noise from noisy images. We observe that the noise variance map of a high ISO JPEG image is spatial-dependent and has a Bayer-like pattern. Therefore, we introduce the Bayer pattern prior in our noise estimation and collaborative filtering stages. Since collaborative filtering is good at recovering repeatable structures and convolutional filtering is good at recovering irregular patterns and removing noise in flat regions, we propose to fuse the strengths of the two methods via deep CNN. The experimental results demonstrate that our method outperforms the state-of-the-art realistic noise removal methods for a wide variety of testing images in both subjective and objective measurements. In addition, we construct a dataset with noisy and clean image pairs for high ISO JPEG images to facilitate research on this topic.

Abstract: We developed a series of multispectral imaging cameras based on the custom Bayer filter principle and made by hybridization onto commercial CMOS sensors. These cameras are based on 1.3Mpx and 4.2Mpx monochrome sensors. They provide 8 color channels plus a panchromatic one. The Bayer principle allows to get compact, robust and low-cost instruments. Apart from these off-the-shelf cameras, the hybrid approach allows a large flexibility in terms of filter/commercial sensor choice to customize any new multispectral camera.

Abstract: Due to the latest progress in image sensor manufacturing technology, the emergence of a sensor equipped with an RGGB Bayer filter and a directional polarizing filter has brought significant advantages to computer vision tasks where RGB and polarization information is required. In this regard, joint chromatic and polarimetric image demosaicing is indispensable. However, as a new type of array pattern, there is no dedicated method for this challenging task. In this Letter, we collect, to the best of our knowledge, the first chromatic-polarization dataset and propose a chromatic-polarization demosaicing network (CPDNet) to address this joint chromatic and polarimetric image demosaicing issue. The proposed CPDNet is composed of the residual block and the multi-task structure with the costumed loss function. The experimental results show that our proposed methods are capable of faithfully recovering full 12-channel chromatic and polarimetric information for each pixel from a single mosaic image in terms of quantitative measures and visual quality.