Abstract: Recent progress on many imaging and vision tasks has been driven by the use of deep feed-forward neural networks, which are trained by propagating gradients of a loss defined on the final output, back through the network up to the first layer that operates directly on the image. We propose back-propagating one step further---to learn camera sensor designs jointly with networks that carry out inference on the images they capture. In this paper, we specifically consider the design and inference problems in a typical color camera---where the sensor is able to measure only one color channel at each pixel location, and computational inference is required to reconstruct a full color image. We learn the camera sensor's color multiplexing pattern by encoding it as layer whose learnable weights determine which color channel, from among a fixed set, will be measured at each location. These weights are jointly trained with those of a reconstruction network that operates on the corresponding sensor measurements to produce a full color image. Our network achieves significant improvements in accuracy over the traditional Bayer pattern used in most color cameras. It automatically learns to employ a sparse color measurement approach similar to that of a recent design, and moreover, improves upon that design by learning an optimal layout for these measurements.

Abstract: An image sensor includes a pixel array including first to fourth pixels having an R, G, and B Bayer pattern. An analog-to-digital converter (ADC) block converts a pixel signal of each of the first to fourth pixels into a digital pixel signal. At least one among the first to fourth pixels includes two photo diodes separated in a first direction, and at least one other of the first to fourth pixels includes two photo diodes separated in a second direction which is different from the first direction.

Abstract: Devices using single sensors to capture colour images are cheaper due to high cost of Charge Couple Device (CCD) sensors or Complementary Metal-Oxide Semiconductor (CMOS) sensors. Single sensor devices use Colour Filter Array (CFA) to sample one colour band at every pixel location. Demosaicking process is applied to interpolate the two missing colours from the surrounding. Typically compression is done on the demosaicked images which may not be efficient due to the individual compression of the different colour space. This work investigated compression of raw data before demosaicking and performs demosaicking to reconstruct the R, G, B bands later. A novel Vector Quantization (VQ) technique for encoding the wavelet decomposed image using Modified Artificial Bee Colony (ABC) optimization algorithm is proposed. The proposed technique is compared with Genetic Algorithm based VQ and ABC based quantization and with standard LBG and Lloyd algorithm. Results show higher Peak Signal-to-Noise Ratio (PSNR) indicating better reconstruction.

Abstract: This paper describes a novel image filtering method that removes random-valued impulse noise superimposed on a natural color image. In impulse noise removal, it is essential to employ a switching-type filtering method, as used in the well-known switching median filter, to preserve the detail of an original image with good quality. In color image filtering, it is generally preferable to deal with the red (R), green (G), and blue (B) components of each pixel of a color image as elements of a vectorized signal, as in the well-known vector median filter, rather than as component-wise signals to prevent a color shift after filtering. By taking these fundamentals into consideration, we propose a switching-type vector median filter with non-local processing that mainly consists of a noise detector and a noise removal filter. Concretely, we propose a noise detector that proactively detects noise-corrupted pixels by focusing attention on the isolation tendencies of pixels of interest not in an input image but in difference images between RGB components. Furthermore, as the noise removal filter, we propose an extended version of the non-local median filter, we proposed previously for grayscale image processing, named the non-local vector median filter, which is designed for color image processing. The proposed method realizes a superior balance between the preservation of detail and impulse noise removal by proactive noise detection and non-local switching vector median filtering, respectively. The effectiveness and validity of the proposed method are verified in a series of experiments using natural color images.

Abstract: This paper proposes an image compression algorithm for wireless capsule endoscopy with high compression rate, low computation complexity and high quality. By analyzing the WCE images properties, the algorithm proposes a color space transform to de-correlate the R-G1-G2-B color components in the Bayer pattern image. Then a 4×4 integer Discrete Cosine Transformation is adopted to reduce the spatial redundancy in the image with the consideration of its low complexity. For increasing compression rate furthermore, run-length encoding on four adjacent blocks is proposed. The compression rate and peak signal-to-noise of the image compression algorithm are 93.6% and 40.9dB respectively. Its corresponding pipeline hardware architecture with distributed storage is also proposed for minimizing the memory size and reducing power consumption. The ASIC design of the compressor has been implemented in a 0.18 μm CMOS process, whose area is 1.5mm × 2.0mm and power consumption is 1.03mW at 2 fps with 480×480 resolution.

Abstract: The optimal strategy of chroma 422 subsampling for Bayer pattern in H.264 video coding is proposed. The state-of-the-art work uses a predetermined conversion method from colour filter array to YCbCr 422. Derived from this structure, first, 64 subsampling methods are designed to exploit more possibilities. Secondly, among which, the four optimal methods are found by the pre-screening process, which show their better performances than the state-of-the-art method by up to 2.57 dB in CPSNR averaged for different videos, demonstrating the practicability of the proposed methods. Finally, for the experiment consisting of the whole conversion and H.264 video encoding/decoding procedures, the proposed optimal methods perform better than the state-of-the-art method under all different operating bitrates and different tested videos, by up to 2.25 dB in BDPSNR.

Abstract: In this paper, an image fusion algorithm is proposed for a multi-aperture camera. Such camera is a feasible alternative to traditional Bayer filter camera in terms of image quality, camera size and camera features. The camera consists of several camera units, each having dedicated optics and color filter. The main challenge of a multi-aperture camera arises from the fact that each camera unit has a slightly different viewpoint. Our image fusion algorithm corrects the parallax error between the sub-images using a disparity map, which is estimated from the single-spectral images. We improve the disparity estimation by combining matching costs over multiple views using trifocal tensors. Images are matched using two alternative matching costs, mutual information and Census transform. We also compare two different disparity estimation methods, graph cuts and semi-global matching. The results show that the overall quality of the fused images is near the reference images.

Abstract: Modern cameras have an image sensor with millions of pixels, among which some of them are inevitably defective (known as hot pixels). Moreover, the number of hot pixels will increase with ageing. colour filter array (CFA) demosaicing will result in colour artifacts caused by these hot pixels. Such artifacts are virtually impossible to remove after demosaicing. In this paper, we propose two effective filter structures for removing low and high density hot pixels within CFA demosaicing for efficient implementation on FGPA for real-time processing. It has been shown that our proposed filter structures are able to remove hot pixels effectively with minimal colour artifacts while preserving image details.

Abstract: Image forgery detection algorithms are used to verify the authenticity of digital images. In literature, different algorithms have been proposed to detect the authenticity of an image, for example, methods based on copy-move forgery detection, histogram analysis, methods based on color filter array, etc. The objective of this paper is to present an image forgery detection algorithm based on color filter array (CFA) interpolated images. Proposed algorithm has been applied on different images in-order to identify in which bayer pattern and map filtering parameter the forged part is clearly visible.

Abstract: Modern industrial cameras mainly use the Bayer pattern as color filter array (CFA). However, this filtering limits the resolution of the color space. As interpolation methods cannot reconstruct the original image perfectly, they have to be optimized for a specific application. Therefore, the interpolation should match the purpose of the image processing system. Many standard algorithms are optimized for the subjective impression of a human observer which is not necessarily ideal for industrial applications. This paper introduces a new methodology for evaluating interpolation algorithms in industrial applications. For this purpose, a new dataset based on printed circuit boards as representatives for objects in industrial application is introduced. It shows many distinct features which are typical for industrial scenarios such as high contrast object edges, highly reflective materials and low variety of surface colors. Furthermore, two new error measures based on edge accuracy are presented which are tailored to many measurement tasks which employ edges. It can be shown in an evaluation of common CFA interpolation algorithms that these new error measures are better suited to identify the best interpolation algorithm for retaining edge accuracy than conventional error measures.

Abstract: The efficiency and accuracy of keypoint detection can potentially be improved using filters which operate directly on the raw Bayer-pattern image for preprocessing such as Histogram of Oriented Gradients (HOG). In applications where the goal is detection and identification of image content rather than the production of an image for human viewing, demosaicing would be unnecessary. For focal plane processing, filter structures are mapped to the color filter arrangement of the Bayer-pattern subsampling. The results show that the proposed filters can work well for most natural image edge detection and orientation, which is used for determining the HOG histograms.

Abstract: Demosaicing, or color filter array (CFA) interpolation, estimates missing color channels of raw mosaiced images from a CFA to reproduce full‐color images. It is an essential process for single‐sensor digital cameras with CFAs. In this paper, a new demosaicing method for digital cameras with Bayer‐like W‐RGB CFAs is proposed. To preserve the edge structure when reproducing full‐color images, we propose an edge direction–adaptive method using color difference estimation between different channels, which can be applied to practical digital camera use. To evaluate the performance of the proposed method in terms of CPSNR, FSIM, and S‐CIELAB color distance measures, we perform simulations on sets of mosaiced images captured by an actual prototype digital camera with a Bayer‐like W‐RGB CFA. The simulation results show that the proposed method demosaics better than a conventional one by approximately +22.4% CPSNR, +0.9% FSIM, and +36.7% S‐CIELAB distance.

Abstract: Simultaneous acquisition of visible and near-infrared images is useful for various image enhancement techniques. In this paper, we study a method for visible and near-infrared image separation from an image captured by a single sensor with cyan, magenta, yellow, and green color filter array. The proposed separation method is mathematically simple, and its hardware can be also implemented by only slight modification of conventional camera. Experimental results observed that a single CMYG-NIR mixed image can be separated into the RGB and MR images.

Abstract: Provided are a method and a device for converting a White-Red-Green-Blue (WRGB) color filter array into a Red-Green-Blue (RGB) color filter array in order to be easily applied to a commercial digital camera. The method includes (a) correcting a color of a White-Red-Green-Blue (WRGB) color filter array, (b) converting the WRGB color filter array into a Red-Green-Blue (RGB) color filter array, and (c) correcting a green of the RGB color filter array by using multichannel color difference value.

Abstract: Image Compression is very important tool to reduce the complexity and the power consumption for applications that have limited size capacity such as capsule endoscopy. To reduce the power required to code and transmite the resulting data of the capsule images, research has mainly focused on reducing the complexity of the design of image compression system. This was either by utlizing interpolation technique to convert the resulting Bayer images into full colour images or by modifing colour transformation with structure conversion to dedicate the Bayer images to the available image compression systems. Both methods are in high power consumption, requiring long processing time and on-chip memory. Thus, this study aims to develop a nearly lossless image compression system with a novel method of subtraction schemefollowed by a linear prediction and Golomb-Rice. The results of this study shows an excellent compression ratio of 93% and high quality reconstructed images with PSNR of 42dB. These results and the high mean structure similarity index matching between the original and the decompressed images confirm the validity of the proposed image compression system. Since this new method has overcome the need for either colour transformation or structure separation units which were necessary units in the compression systems for capsule endoscopy in the existing literature, the power and the processing time which was required to run those units has been eliminated by our new method.

Abstract: The invention discloses an image processing chip and a caching method for image data in the chip. The method comprises the following steps that the image processing chip applies a sliding window of 3*3 pixels to traverse a pixel matrix in a Bayer pattern, after the sliding window moves for one grid, if only one pixel with the unknown basic color component enters the sliding window, the pixel is located at the second row, and a CFA (Color Filter Array) interpolation module computes the unknown basic color component of the pixel at this time; if two pixels with the unknown basic color components enter the sliding window, the two pixels are respectively located at the first row and the third row, the unknown basic color component of the pixel at the first row is read from the cache at this time, the CFA interpolation module computes the unknown basic color component of the pixel at the third row temporarily, and the known basic color component of the center pixel of the sliding window is written into the cache, so that the area of the image processing chip is saved and the hardware cost of the image processing chip is reduced.

Abstract: The present disclose provides a Bayer color filter array based high dynamic range video recording method and device. The method includes configuring different photosensitive times for exposure according to odd-numbered dual columns and even-numbered dual columns, and obtaining an image frame with different exposure values of the odd-numbered dual columns and even-numbered dual columns; decomposing the image frame into an underexposure image frame and an overexposure image frame, where underexposure dual columns and missing dual columns are alternatingly distributed in the underexposure image frame, and overexposure dual columns and missing dual columns are alternatingly distributed in the overexposure image frame. The method also includes acquiring recovered pixel values of pixel points of the missing dual columns in the underexposure image frame and the overexposure image frame; and merging the overexposure image frame and the underexposure image frame to obtain a high dynamic range frame.

Abstract: A critical step in creating an image using a Bayer pattern sampled color camera is demosaicing, the process of combining the individual color channels using a post-processing algorithm to produce the final displayed image. The demosaicing process can introduce degradations which reduce the quality of the final image. These degradations must be accounted for in order to accurately predict the performance of color imaging systems. In this paper, we present analytical derivations of transfer functions to allow description of the effects of demosaicing on the overall system blur and noise. The effects of color balancing and the creation of the luminance channel image are also explored. The methods presented are validated through Monte Carlo simulations, which can also be utilized to determine the transfer functions of non-linear demosaicing methods. Together with this new treatment of demosaicing, the framework behind the color detector component in NV-IPM is discussed.

Abstract: An image sensor including a color filter and a method of manufacturing the image sensor are provided The image sensor includes a light-sensing layer configured to detect incident light, and convert the incident light to an electrical signal The image sensor further includes a color filter layer disposed on the light-sensing layer, the color filter layer including color filters, each of the color filters being configured to transmit, among the incident light, light in a wavelength band to the light-sensing layer The image sensor further includes an isolation layer disposed between the color filters, the isolation layer being configured to optically isolate the color filters from each other An upper portion of each of the color filters has a cylindrical shape, and a lower portion of each of the color filters has a hemispherical shape

Abstract: The present invention relates to a multi-directional weighted interpolation method for demosaicing a Bayer pattern Color Filter Array (CFA). The multi-directional weighted interpolation method for demosaicing a Bayer pattern CFA according to the present invention comprises the steps of: (a) interpolating omitted green elements in 8 directions including vertical, horizontal, and diagonal directions by using a green interpolation unit; (b) interpolating red and blue elements based on relationships in an interpolated green plane by using red and blue interpolation units; and (c) improving the green plane, a red plane, and a blue plane by using differences between color values and a post-processing unit.

Abstract: A method for forming a color filter array includes a step of exposing a photosensitive color filter film, a step of forming a color filter array from the color filter film by developing the color filter film using a developer, and a step of cleaning the color filter array while rotating the color filter array and moving a nozzle for spraying fluid containing liquid and gas above the color filter array in a direction intersecting with an axis of the rotation. The method reduces variation in thickness of a color filter that is generated in the cleaning step.

Abstract: The present invention relates to a skin imaging method and a skin imaging apparatus, in which one white light source instead of R, G, and B light sources are used and the use of an RGB filter is avoided accordingly, so that a complicated rotation control of the RGB filter is unnecessary. An imaging according to a penetration depth with respect to a skin subject surface is realized by using a single white light source, a polarizer, and a Semrock filter. According to the present invention, unlike an existing skin imaging apparatus requiring a plurality of light sources and an RGB filter for irradiating a multi-wavelength, a low-cost skin imaging apparatus having a simple structure can be realized only with one white light source, a general polarizer, and a Semrock filter.

Abstract: Digital cameras acquire color images using a single sensor with Color filter Arrays. A single color component per pixel is acquired using color filter arrays and the remaining two components are obtained using demosaicing techniques. The conventional demosaicing techniques existent induce artifacts in resultant images effecting reconstruction quality. To overcome this drawback a frequency based demosaicing technique is proposed. The luminance and chrominance components extracted from the frequency domain of the image are interpolated to produce intermediate demosaiced images. A novel Neural Network Based Image Reconstruction Algorithm is applied to the intermediate demosaiced image to obtain resultant demosaiced images. The results presented in the paper prove the proposed demosaicing technique exhibits the best performance and is applicable to a wide variety of images.

Abstract: For the charge coupled device camera that adopts Bayer color filter array can only record single channel information of gray value from red, green or blue; it has to estimate the others by using the method of interpolation to achieve the process of color reduction. In order to improve the quality of images, it needs to modify the bad pixels before color reduction. In this article, it proposed an interpolation method based on two-step approach model for bad pixels in Bayer color image. It firstly modified the four neighboring pixels which in the same channel with the bad pixel, and then calculate the gray value of bad pixel by using bilinear interpolation method. Through this method, it made a better use of the information of pixels near the bad pixel, and effectively improved the accuracy of estimation for bad pixels.

Abstract: This paper introduces the pentagraph image fusion (PIF) scheme for motion-related blur prevention in images. The PIF algorithm processes five monochromatic images into a single, low-noise, no-blur color image. The images are acquired using a new photography scheme, sequential filter photography (SFP), where instead of using a stationary Bayer pattern color filter array in front of the image sensor, a tunable color filter array is used. Using this approach, several monochromatic images are captured one by one and later fused into one color image. The SFP introduces various advantages such as higher resolution, better SNR, and the ability to control both exposure time and color filter separately for each image. The PIF algorithm harnesses all the advantages of SFP for the first time, to the best of our knowledge, in the field of blur-free image acquisition. Five images are taken with controllable exposure time and color filter, three images for the color bands, and two high-signal panchromatic images. These images are fused together to be a single, low-noise, no-blur color image. The algorithm presents a generic approach of dealing with both local and global motion blur and does not require any user intervention.

Abstract: Embodiments of the present disclosure relate to a sensor interface circuit that performs scaling of image data in a Bayer pattern without spreading defective pixels across multiple pixels. The sensor interface circuit may include a register circuit storing operating parameters of the sensor interface circuit. The sensor interface circuit includes a scaling circuit with a first defect pixel detection circuit to detect a first defective pixel in an input image by analyzing pixels in a line of an input image data along a first direction. A first scaling circuit is coupled to the first defect pixel detection circuit and generates a scaled line of pixels representing the line of the input image scaled along the first direction according to the operating parameters.

Abstract: In this paper, we present new applications of the Spectral Edge image fusion method. The Spectral Edge image fusion algorithm creates a result which combines details from any number of multispectral input images with natural color information from a visible spectrum image. Spectral Edge image fusion is a derivative–based technique, which creates an output fused image with gradients which are an ideal combination of those of the multispectral input images and the input visible color image. This produces both maximum detail and natural colors. We present two new applications of Spectral Edge image fusion. Firstly, we fuse RGB–NIR information from a sensor with a modified Bayer pattern, which captures visible and near–infrared image information on a single CCD. We also present an example of RGB–thermal image fusion, using a thermal camera attached to a smartphone, which captures both visible and low–resolution thermal images. These new results may be useful for computational photography and surveillance applications.