WO2012063265A2

WO2012063265A2 - Method and apparatus for detecting the bad pixels in sensor array and concealing the error

Info

Publication number: WO2012063265A2
Application number: PCT/IN2011/000775
Authority: WO
Inventors: Sudipta Mukhopadhyay; Abhishek Kumar Tripathi
Original assignee: Indian Institute Of Technology, Kharagpur
Priority date: 2010-11-12
Filing date: 2011-11-11
Publication date: 2012-05-18
Also published as: IL226318A0; WO2012063265A9; WO2012063265A4; KR101559724B1; WO2012063265A3; KR20130096293A

Abstract

A method of detection of bad pixel in sensor array and correction of detected bad pixels by inpainting and also to image sensor devices involving such sensor arrays adapted for identification and correction of the bad pixels. Importantly, image sensors which are provided to convert optical images to electrical signals would benefit from such possible detection and correction of pixel errors thereby providing for better quality of output. The method and the device enables providing for better quality cameras in particular digital cameras involving CCD image sensors or CMOS sensors. It monitors the health of the sensor arrays and increase its life, without involving any additional hardware cost. The invention targets all possible varieties of defective pixels including dead pixels, stuck pixels, hot and cold pixels and therefore is adapted to serve as a complete solution to such art of handling image corruption even for high numbers of bad pixels.

Description

TITLE: METHOD AND APPARATUS FOR DETECTING THE BAD PIXELS IN SENSOR ARRAY AND CONCEALING THE ERROR .

FIELD OF THE INVENTION

The present invention relates to sensor arrays and in particular to the method of detection of bad pixel in sensor array and correction of detected bad pixels by inpainting and also to image sensor devices involving such sensor arrays adapted for identification and correction of the bad pixels. Importantly, image sensors which are provided to convert optical images to electrical signals would benefit from such possible detection and correction of pixel errors thereby providing for better quality of output. Advantageously, the invention would enable providing for better quality cameras in particular digital cameras involving CCD image sensors or CMOS sensors which also accomplish the task of capturing light and converting into electrical signals. The invention is targeted at removing defective pixels occurring in image sensors such as in digital cameras. The method of the invention has significant economic advantage in production and usage of sensors, as it helps monitoring the health of the sensor arrays and also increase the usable life of it, without involving any additional hardware cost. Importantly, the invention targets all possible varieties of defective pixels including dead pixels, stuck pixels, hot and cold pixels and therefore is adapted to serve as a complete solution to such art of handling image corruption even for high numbers of bad pixels including the various varieties of bad pixels discussed hereinbefore the invention would favour accurate detection of defective sensor arrays and ascertain how much the image sensor is impaired and more importantly would favour possible improving the acquired image quality by inpainting. Moreover, when error concealment by inpainting is not applied after detection of bad pixels, the error pixels map generated can be used as a signature of that particular camera to verify the authenticity of any image captured using that camera.

BACKGROUND ART

It is well known in the art that sensors almost always have some pixels that don't have the full dynamic range that they should. Most of these pixels can be corrected with the flat filed correction parameters, also implemented in the camera. However, the pixels that cannot be properly corrected with flat filed correction are considered defective pixels. These dead or hot pixels, usually stuck dark or bright, are corrected with the defective pixel correction routine. Such a routine corrects the defective pixel with interpolated values based on neighboring pixels. Typical correction methods include averaging with immediate neighbors. Correction can span multiple pixels if by some chance many consecutive pixels in a row are defective. Boundary conditions also exist to handle defective pixels at the edges of the image array. Defective pixels occur on image sensors in digital cameras. These pixel fail to sense light correctly. There are many variations of these defective pixels[Filip Hroch, The robust detection of stars on CCD images", Journal of Experimental Astronomy, Springer Netherlands, Vol. 9, No. 4, pp. 251-259, Dec. 1999.] viz. dead pixels, stuck pixels, hot and cold pixels. Dead pixel is a pixel that always reads zero on all exposures. Stuck pixel is a pixel that always read maximum value on all exposures. Dead and stuck pixels are complementary of each other. Hot pixel is one that reads high on longer exposures. Obviously stuck pixel is an extreme case of a hot pixel. Similarly there can be some pixels which are not exactly to zero rather they are close to zero. Those pixels are identified herein as cold pixels. In all the cases, the pixels report these values irrespective of what the original scene was. Dead pixel runs a charge across the liquid crystal material thus no light ever passes through it. However, a stuck pixel allows all light to pass through because its transistor are not working properly[Ref. IEEE Transaction on Intrumentation and Measurement, VOL. 47, NO. 1, Feb. 1998 and Uwe EWERT, Uwe ZSCHERPEL, Klaus BAVENDIEK, Strategies for Film Replacement in Radiography - Films and Digital Detectors in Comparison", 17th World Conference on Nondestructive Testing, Shanghai, China, 25-28 Oct 2008.].

Bad or defective pixels are often transistors that are permanently dead and appears black dot or are stuck on and show up as white or colored pixels. Sometimes, it is possible to repair a bad pixel by gently rubbing the screen with a cloth or running a pixel repair computer program over the area for several hours, the latter repetitively flashing colors on and off to attempt to dislodge a stuck pixel. Dead pixels are much less likely to correct themselves overtime or be repaired through any methods. There are several methods to fix stuck pixels. These effects usually happen with the exposures and are, among other things, related to the temperature of the camera. Usually the bad pixels look like little random spots all over the image. Some cameras have a noise reduction setting that can remove the same.

A common mistake in testing for bad pixels is to cover the lens with the lens cap, set the camera to AUTO. Setting to Auto will cause the camera to lower the shutter speed thus result in taking a long exposure. This produces some red, greenish and sometimes white pixels. This is a normal state and it is referred to as Christmas Tree artifact. For testing of bad pixels, it is usual to take some shots of normal indoor environment such as a portrait and then comparing the two shot using a photo editor such as Photoshop or Nikon Editor where it allows to compare two images side by side. In case of a bright pixel occurring on one image, it is important to do the comparison on the other image, both should have the bright pixel on the same location. If it is on the same location it can be consider as a HOT pixel. In addition, a dead pixel is dead all the time and would not show up with this test since the pixel is black. Stuck and dead pixels are generally not considered as a problem for a photographer.

Modern DSLRs such as the Canon 5D mk2 have built in defective pixel remapping. Most RAW processing software (eg Lightroom and Bibble Labs) have a feature to automatically fix defective pixels. However, when the picture is converted to JPEG it gets a little bit more complicated since the strong discontinuity of a hot pixel causes JPEG compression artifacts in the surrounding pixels.

US Patent 5047863 is a defect correction apparatus for solid state imaging devices including inoperative pixel detection employs a frame buffer responsively coupled to the imaging device for storing digital values of shuttered dark pixel data from image locations of the imaging device. A register responsively coupled to the imaging device and having an output coupled to the buffer sequentially clocks digital values of image pixel data of the imaging device into the frame buffer. A comparator having an output operatively coupled to the register and being responsively coupled to the dark pixel data of the frame buffer produces enable and inhibit outputs. When any selected element of dark pixel data is less than a threshold, indicative of an operative pixel element in the imaging device, data stored in the register corresponding to the pixel is stored in the frame buffer in response to the enable signal. When the value of the dark pixel data in the frame buffer is greater than the threshold the comparator produces an inhibit signal. The value of image data of a previous operative pixel element is thereby entered into the register and frame buffer.

US States Patent 5499114 is a digital image scanning apparatus with pixel data compensation for bad photosites, where pixel data from bad photosites of a linear imaging device is suppressed and replaced with pixel data from the next available good photosite. The loss of pixel data blocks from the beginning of the scan line is made up at the end of the scan line by continuing to write to memory the end pixel data a requisite number of times needed to complete the full scan line data.

US Patent 6618084 is a pixel correction system and method for CMOS imager which provides a fault tolerant radiation imager such as a CMOS imager. Such image sensor includes circuitry for masking and/or correcting defective pixels during image generation.

US Patent 7522200 is an on-chip dead pixel correction in a CMOS imaging sensor which disclosed a method for correcting for dead pixels in a CMOS image sensor. However, it would be clearly apparent from the above that all such available methods of pixel correction involve complexities and cannot serve as the much required accurate and simple manner of bad pixel detection and rectification by inpainting required in the art in particular for the online monitoring of the health of the sensor and in turn increase the usable life span of the sensor.

OBJECTS OF THE INVENTION The basic object of the present invention is thus directed to developing a method for detection of bad pixel in sensor array and error correction which would enable detection of bad pixel in a simple manner and carry out the inpainting required to enhance image quality by image sensors of digital camera and essentially also favour the online monitoring of the health of the sensor and in turn increase the usable life span of the sensors.

Another object of the present invention is to provide for new method of detection of bad pixels including dead, stuck, hot and cold pixels which would be adapted to favour better utilization of image sensors and like devices provided to convert optical image to electrical signals such as those used in digital cameras and other imaging devices.

Another object of the present invention is directed to development of an advanced technique of handling image corruption due to bad pixels in sensor arrays and the like even for high number of bad pixels. Another object of the present invention is directed to advancement in detection and correction of bad pixels whereby the bad pixel detection error probability can be reduced to zero by involving use of multiple images following the technique of the present invention. A further object of the present invention is directed to providing new method of detection of bad pixel and inpainting of bad pixels to correct pixel errors which would make it possible to apply the inpainting procedure independent of the method of detection of the bad pixel. Another object of the present invention is directed to provide for method to detect bad pixel online or offline depending on application demand wherein the online detection would favorably reduced the correction delay while the offline detection would be adapted to reduce the complexity and power expenditure. Yet further object of the present invention is directed to providing bad pixel detection and inpainting which would be adapted to reduce the impact of bad pixel and thereby rejection rate at the time of production and increase the effective life of the sensor arrays. Another object of the present invention is directed to a method of detection of bad pixels which would be adapted to reduce the false and missed detection involving the number of independent acquisition and further reduced by selection of ROI in the image.

Yet further object of the present invention is directed to providing sensor arrays which would facilitate accurate detection of bad pixels and it's required inpainting for better output and performance of sensor arrays and like imaging devices.

A further object of the present invention is directed to favour provision of method whereby it would be possible as to determining how much the image sensor is impaired and till replacement possibilities of improving the image quality by inpainting.

Another object of the present invention is directed to the development of bad pixel determination methodology involving powerful scheme directed to achieve high PSNR. A still further objective of the present invention is directed to the authentication of images taken from a particular camera with bad pixel(s), wherein the error pixels map generated using the detected bad pixels can be used as a signature of that particular camera, when the inpainting is not applied for bad pixels.

SUMMARY OF THE INVENTION

The basic aspect of the present invention is thus directed to a method for detecting the bad pixels in sensor array and concealing of the error comprising generating error/noise map involving intensity based detection of location information of bad pixels of sensors using local statistics of the digital image involving values signifying the location of bad pixel and normal pixel respectively.

A further aspect of the present invention is directed to said method for detecting the bad pixels in sensor array comprising generating said error/noise map comprising providing a noise map involving intensity based detection of location information of bad pixels of sensors using local statistics of the digital image and subjecting the location of the thus identified bad pixels to further evaluation of its local statistics to confirm that the pixels are bad or not and thereby generate a final noise map with 1 and 0 values which signify the location of bad pixel and normal pixel respectively.

A still further aspect of the present invention is directed to a method for detecting the bad pixels in sensor array comprising step of authentication of the images captured by a camera by comparing with the error/noise map thus generated with images taken from a camera.

Advantageously, said method for detecting the bad pixels in sensor array according to the present invention comprising generating error map over a several subsequent images and involving combinations thereof to further confirm error detection.

A still further aspect of the present invention is directed to said method for detecting the bad pixels in sensor array and concealing of the error comprising (i) generating noise map involving intensity based detection of location information of bad pixels of sensors using local statistics of the digital image involving values signifying the location of bad pixel and normal pixel respectively;

(ii) carrying out an adaptive inpainting procedure for correcting only the detected bad pixels in said noise map in relation to a noisy image comprising the step of replacing the bad pixels by the median value (intensity) of normal pixels within a specified window around said detected bad pixel.

A still further aspect of the present invention is directed to said method for detecting the bad pixels in sensor array and concealing of the error comprising

(i) generating said noise map comprises providing a noise map involving intensity based detection of location information of bad pixels of sensors using local statistics of the digital image and subjecting the location of the thus identified bad pixels to further evaluation of its local statistics to confirm that the pixels are bad or not and thereby generate a final noise map with 1 and 0 values which signify the location of bad pixel and normal pixel respectively;

(ii) an adaptive inpainting procedure for correcting only the detected bad pixels in said final noise map in relation to a noisy image comprising replacing the bad pixels by the median value(intensity) of normal pixels within a specified window around said detected bad pixel.

According to an aspect of the present invention directed to said method for detecting the bad pixels in sensor array and concealing of the error wherein said bad pixels include dead, stuck, hot and cold pixels.

A still further aspect of the present invention directed to said method wherein carrying out the step of detection of said dead pixel comprises

(I) imposing a Wi X W₂ window for each noisy image and centering the same on the current pixel to determine the maximum value (s_max) and minimum value (s_min) within the window, thereafter generating the first stage noise map Y following : if iSi -S^^ anclis, max

r(i, j)=

0 otherwise wherein s_i(j is the intensity value of the pixel at location (i,j) and S_min is the global minimum intensity value of the image and wherein in this first stage noise map, r(i,j) = 1 means corresponding pixel in noisy image is probably dead and r(i,j) = 0 means corresponding pixel in noisy image is normal; followed by

(II) generating a final noise map Y comprising

Level 1) imposing a w_x x w₂ window around the dead pixel in noisy image, such that Wi, w₂ are much smaller than previous window size; if more than w_min normal pixels such as when pixels which have corresponding values in first stage noise map 0 are found, thereafter calculating the distance measure d between the normal pixels and the central dead pixel and going to Level 3; if the number of normal pixels is not more than w_min then going to Level 2; where it is assumed for simulation that d is mean of absolute distance (MAD);

Level 2) setting the w, = W| + p_if 1= 1,2, pj are small positive integers and going to

Level 1 and increasing the window up to a pre determined fixed size;

Level 3) if the corresponding d is less than the threshold δ (δ is a small positive value), replacing the corresponding pixel in temporary noise map r to 0 else leave it and move to next dead pixel and go to Level 1.

According to yet another aspect of the present invention directed to said method, wherein carrying out the step of detection of stuck pixels comprises (I) imposing a W_a X W₂ window for each noisy image and centering the same on the current pixel to determine the maximum value (s_max) and minimum value (s_min) within the window, thereafter generating the first stage noise map Y following :

wherein s_i(j is the intensity value of the pixel at location (i,j) and S_max is the global maximum intensity value of the image and in this first stage noise map, r(i,j) = 1 means corresponding pixel in noisy image is probably stuck and r(ij) = 0 means corresponding pixel in noisy image is normal followed by

(II) generating a final noise map Y comprising

Level 1 : Imposing a Wi x w₂ window around the stuck pixel in noisy image, such that wj, w₂ are much smaller than previous window size and if more than w_min normal pixels are found, then calculating distance measure d between the normal pixels and the central stuck pixel and go to Level 3 and if the number of normal pixels are not more than w_min then go to Level 2. Here it is assumed that d is MAD.

Level 2 : Setting the w_f = w, + pj, i=l,2, p, are small positive integers and going to Level 1, Increasing the window up to a pre determined fixed size.

Level 3 : If the corresponding d is less than the threshold δ (<5 is a small positive), replacing the corresponding pixel in temporary noise map V to 0 else leave it and move to next stuck pixel and going to Level 1, whereby the noise map V is final •noise map.

A steel further aspect of the present invention is directed to said method wherein carrying out the step of detection of said hot and cold pixels comprises (I) imposing a Wx X W₂ window for each noisy image and centering the same on the current pixel, thereafter generating the first stage noise map Y following :

where s_ifj is the intensity value of the pixel at location (i,j). Sm_ax and Sm_in are the global maximum and minimum intensity value respectively. In first stage noise map, r(i,j) = 1 means corresponding pixel in noisy image is probably hot or cold and r(i,j) = 0 means corresponding pixel in noisy image is normal. (II) generating a final noise map Y comprising

Level 1 : Imposing a w_x x w₂ window around the hot (cold) pixel in noisy image, such that w_x, w₂ are much smaller than previous window size and if more than w_m!n normal pixels are found, then calculating distance measure d between the normal pixels and the central hot (cold) pixel and going to Level 3. If the number of normal pixels are not more than w_mln then going to Level 2 wherein it is assumed that the distance measure d is MAD.

Level 2 : Setting the w, = Wj + p,, 1=1,2, p, are small positive integers and going to Level 1. Increasing the window up to a predetermined fixed size.

Level 3 : taking minimum of distance of the central hot (cold) pixel from both end preferably such that if image is 8 bit then T₃ = min(Sj,_j , 255 - S ). and If the corresponding d is less than the threshold qT3, replacing the corresponding pixel in temporary noise map V to 0 else leaving it and moving to next hot(cold) pixel and going to Level Iwith q being a constant (0 < q < 1) to thereby generate the final noise map V.

Advantageously, said method of the present invention is adapted to be applied for images of any bit depth.

A still further aspect of the present invention directed to said method, wherein the said step of correcting the pixel error by inpainting comprising the steps of :

(1) imposing a w₃ x w₄ window around the bad pixel in noisy image to find out a set S having all those noisy image pixels under current window for which corresponding value in noisy map are 0; if S is not a null set then going to Step2, otherwise going to Step3,

(2) replacing the current pixel with median of the set S and proceeding to the next bad pixel and going to Stepl,

(3) setting the Wi = Wi + AWj, i=3,4 Aw, are small positive integers and going to Stepl and increasing the window up to a pre determined maximum size. Importantly, also in said method according to the present invention, said bad pixel detection procedure followed by bad pixel inpainting is adapted to achieve high PSNR even with image corruption for high number of bad pixels. In said method , said bad pixel detection is performed online or offline and once detected the inpainting may be applied online/offline depending on application demand.

According to yet another aspect of the present invention is directed to an image sensor system comprising a sensor array adapted to detect bad pixels and carry out inpainting of bad pixels comprising :

(i) means adapted for generating noise map involving intensity based detection of location information of bad pixels of sensors using local statistics of the digital image involving values signifying the location of bad pixel and normal pixel respectively;

(ii) means adapted for carrying out an adaptive inpainting procedure for correcting only the detected bad pixels in said noise map in relation to a noisy image comprising the step of replacing the bad pixels by the median value (intensity) of normal pixels within a specified window around said detected bad pixel.

Importantly in said image sensor system , said sensor array is adapted to detect variety of bad pixels selected from dead pixel, stuck pixel, cold and hot pixels following any of the aforesaid methods.

According to yet another aspect of the present invention directed to said image sensor system, which is adapted to convert optical/thermal any other signal including thermal , micro-wave, ultrasound, x-ray to electrical signals.

The details of the invention, its objects and advantages are explained hereunder in greater detail in relation to non-limiting exemplary illustrations as per the following accompanying figures: BRIEF DESCRIPTION OF THE ACOMPANING FIGURE

Figure 1: is the schematic block diagram of the sensor based imaging system according to the present invention showing the steps of bad pixel detection and inpainting.

Figure 2: First row shows the corrupted " Chemical plant' images. Images are corrupted by the random dead lines and/or random dead pixels. Second row shows the corresponding restored image from the above by the proposed formula.

Figure 3: First row shows the corrupted " Utrecht' images. Images are corrupted by the random dead lines and/or random dead pixels. Second row shows the corresponding restored image from the above by the proposed formula.

Figure 4: First row shows the corrupted " Outdoor¹ images. Images are corrupted by the random dead lines and/or random dead pixels. Second row shows the corresponding restored image from the above by the proposed formula.

Figure 5: First row shows the corrupted ^" Bridge' images. Images are corrupted by the random dead lines and/or random dead pixels. Second row shows the corresponding restored image from the above by the proposed formula.

Figure 6: First row shows the corrupted ^" Chemical plant" images. Images are corrupted by the random stuck lines and/or random stuck pixels. Second row shows the corresponding restored image from the above by the proposed formula.

Figure 7: First row shows the corrupted " Utrecht" images. Images are corrupted by the random stuck lines and/or random stuck pixels. Second row shows the corresponding restored image from the above by the proposed formula.

Figure 8: First row shows the corrupted " Outdoor' images. Images are corrupted by the random stuck lines and/or random stuck pixels. Second row shows the corresponding restored image from the above by the proposed formula. Figure 9: First row shows the corrupted ^' Bridge' images. Images are corrupted by the random stuck lines and/or random stuck pixels. Second row shows the corresponding restored image from the above by the proposed formula.

Figure 10: First row shows the corrupted images. Images are corrupted by the random hot and cold pixels. Second row shows the corresponding restored image from the above by the proposed formula.

Figure ll:(a) Original 16 bit image having hot pixels. Two crop regions having hot pixels are shown in red rectangle. Zoomed view of (b) crop region 1 and (c) crop region 2.

Figure 12:(a) Output of image shown in Figure 11, Corresponding crop regions are shown in red rectangle. Zoomed view of corresponding (b) crop region 1 and (c) crop region 2.

DETAILED DESCRIPTION OF THE INVENTION WITH REFERENCE TO THE ACCOMPANING FIGURES

As discussed hereinbefore, the present invention involves a fully automatic dead, stuck, hot and cold pixel detection and inpainting method. The invention would favour enlarging the operating life of sensors in a sensor array. Basically, the proposed detection method comprises of two stages. In the first stage of detection method local characteristics of the noisy image are used. The second stage is fashioned in a different way for better performance. Proposed detection formulas can handle image corruption up to higher level of noise. Furthermore, for the restoration of the corrupted image after detection of the bad pixels, adaptive bad pixels inpainting protocol is followed. Proposed method has adaptability for real time performance which will save lot of hardware cost.

More importantly, the proposed detection method for bad pixel detection comprises of two stages. In the first stage all pixels examined to prepare a first stage noise map, which keeps the location information of bad pixels. This stage makes use of local statistics of the image. In the second stage only those pixels marked bad in first stage are examined using local statistics to confirm whether they are bad or not. In the second stage the first stage noise map is modified accordingly to get final noise map. In the final noise map 1 and 0 values signify the location of bad pixel and normal pixel respectively.

Reference is now invited to the accompanying Figure 1 that schematically illustrate the block diagram of the sensor imaging system according to the present invention showing the steps of detection and inpainting of the bad pixels in sensor array.

Steps of proposed dead pixels detection method are as follows:

To illustrate the method it is assumed the images are for 8-bit deep, however it is no way a limitation of this method and it can be applied for images of any bit depth.

Step 1 : For each pixel in noisy image impose a Wl x W2 window, which is centered on the current pixel, and find out the maximum value (s_max) and minimum value (s_min) within the window. In this stage the pixels are very conservatively labeled as normal. Use Equation (1) and prepare a first stage noise map Y.

where Sy is the intensity value of the pixel at location (i,j) and S_min is the global minimum intensity value of the image. In first stage noise map, r(i,j) = 1 means corresponding pixel in noisy image is probably dead and r(i,j) = 0 means corresponding pixel in noisy image is normal.

Step 2 : For each dead pixel in image (having corresponding value 1 in first stage noise map) following actions are performed :

Level 1 : Impose a Wi x w₂ window around the dead pixel in noisy image, such that w_l7 w₂ are much smaller than previous window size. If more than w_min normal pixels (i.e. pixels which have corresponding values in first stage noise map 0) are found, then calculate the distance measure d between the normal pixels and the central dead pixel and go to Level 3. If the number of normal pixels is not more than w_min then go to Level 2. Here for simulation it is assumed that distance measure d is mean of absolute distance (MAD). Level 2 : Set the w, = w, + p i= l,2, Pi are small positive integers and go to Level 1. Increase the window up to a pre determined fixed size.

Level 3 : If the corresponding d is less than the threshold δ (δ is a small positive value), replace the corresponding pixel in temporary noise map r to 0 else leave it and move to next dead pixel and go to Level 1.

Now this noise map is called as final noise map. This final noise map has values 0 and 1, where 0 and 1 denote the corresponding pixel is normal and dead respectively.

Steps of proposed stuck pixels detection procedure are as follows:

Step 1 : For each pixel in noisy image impose a Wl x W2 window, which is centered on the current pixel, and find out the maximum value (s_max) and minimum value (s_min) within the window.

Use Equation (2) and prepare a first stage noise map V.

where S is the intensity value of the pixel at location (i,j) and S_max is the global maximum intensity value of the image. In first stage noise map, r(i,j) = 1 means corresponding pixel in noisy image is probably stuck and r(i,j) = 0 means corresponding pixel in noisy image is normal.

Step 2 : For each stuck pixel in image following actions are performed :

Level 1 : Impose a w_x x w₂ window around the stuck pixel in noisy image, such that w_if w₂ are much smaller than previous window size. If more than w_min normal pixels are found, then calculate distance measure d between the normal pixels and the central stuck pixel and go to Level 3. If the numbers of normal pixels are not more than w_min then go to Level 2. Here it is assumed d is MAD. Level 2 : Set the w, = w, + p_i( i= l,2, p, are small positive integers and go to Level 1. Increase the window up to a predetermined fixed size.

Level 3 : If the corresponding d is less than the threshold δ (δ is a small positive value), replace the corresponding pixel in temporary noise map Y to 0 else leave it and move to next stuck pixel and go to Level 1.

Now this noise map Y is final noise map.

Steps of proposed hot and cold pixels detection method are discussed as follows:

Step 1 : For each pixel in noisy image, use Equation(3) and prepare a first stage noise map ^" r'.

1 if (S_max - Sij) < T₄ OF (Sij - S_min) < T₅

0 otherwise

(3) where S is the intensity value of the pixel at location (i,j). S_max and S_mjn are the global maximum and minimum intensity value respectively. In first stage noise map, r(i,j) = 1 means corresponding pixel in noisy image is probably hot or cold and r(i,j) = 0 means corresponding pixel in noisy image is normal.

Step 2 : For each hot (cold) pixel in image following actions are performed:

Level 1 : Impose a Wi x w₂ window around the hot (cold) pixel in noisy image, such that w_lf w₂ are much smaller than previous window size. If more than w_min normal pixels are found, then calculate distance measure d between the normal pixels and the central hot (cold) pixel and go to Level 3. If the numbers of normal pixels are not more than w_min then go to Level 2. Here for simulation it is assumed that MAD is the distance measure d.

Level 2 : Set the w, = w, + p_if i=l,2, p, are small positive integers and go to Level 1. Increase the window up to a pre determined fixed size.

Level 3 : Minimum of distance of the central hot (cold) pixel from both end is taken as T₃. If image is 8 bit then T₃ = min(s_i(j , 255 - Sy) . If the corresponding d is less than the threshold qT₃, replace the corresponding pixel in temporary noise map Y to 0 else leave it and move to next hot (cold) pixel and go to Level 1. Here q is a constant (0 < q < 1).

Now this noise map Y is final noise map.

Accurate noise map (zero miss detection and zero false detection) can be obtained by running the test on multiple different images acquired by the sensor. Noise maps of these noisy images are calculated by the proposed detection procedure. Accurate noise map are detected by the highest voting of the each pixel in all the different images. Next time, when an image is tested there is no need to calculate the noise map because the spatial positions of the bad lines and/or bad pixels are already known. Thus inpainting procedure is applied directly.

Above detection procedure gives a noise map Y with values 0 and 1 for the normal and bad (dead, stuck, hot and cold) pixels respectively. Here an adaptive bad pixels inpainting procedure is proposed, which is the modification of the adaptive switching median filter (ASMF) according to the property of the sensor noise. Adaptive filtering involves the correction of only those pixels in noisy image which have corresponding value 1 in noise map Y. These bad pixels are replaced by the median value of those pixels of noisy image within the specified window w₃ x w₄ for which corresponding value in noise map are 0. If there are no such pixels in the current window then window size, w_f is increased up to a pre determined maximum size, w_fmax for processing, beyond which retain the original pixels as such.

Steps of proposed inpainting method for 8-bit images are as follows:

Step 1 : Impose a w₃ x w₄ window around the bad pixel in noisy image. Find out a set S having all those noisy image pixels under current window for which corresponding value in noisy map are 0. If S is not a null set then go to Step 2 otherwise go to Step 3.

Step 2 : Replace the current pixel with median of the set S. Now proceed to the next bad pixel and go to Stepl. Step 3 : Set the Wj = w, + Δνν,, i=3,4 AWj are small positive integers and go to Stepl. Increase the window up to a predetermined maximum size.

To verify the performance of the proposed method , simulations were carried out on the various 8 bit gray scale images. These images are corrupted by the dead (stuck) pixels and/or two dead (stuck) lines. Dead (stuck) pixels are generated by replacing randomly selected (uniform distribution) image pixels with zero (white). Dead (stuck) lines are generated by replacing two or three consecutive rows or columns of the image with zeros (extreme value i.e. 255 for 8 bit image). Here one bad line covers whole image width and another line covers half image width. Direction (row or column) and positions of bad lines are uniformly distributed. Images corrupted by hot (cold) pixels are generated by replacing randomly selected (uniform distribution) image pixels with value close to extreme value (zero). Performance of the proposed detection procedure is analyzed in terms of the average miss and false detection. Miss detection means that a bad pixel is detected as a normal pixel. False detection means that a normal pixel is detected as a bad pixel. Thus, as the value of miss-detection and/or false-detection increases, the performance of the detector decreases. Results show that proposed bad pixel detection procedure ensure zero miss detection and very low false detection.

Results for the accurate noise map detection by the various combinations of the multiple images are found experimentally. Results show that three images are sufficient to obtain the accurate positions of the bad pixels. It is necessary to periodically monitor the sensor. This regular monitoring of sensor provides the exact locations of the defected sensor arrays. Once the number of the defected sensor arrays are obtained it is easy to find that how much the image sensor is impaired and acquired image quality can be improved by inpainting till replacement of sensor array.

Quantitative performance of restoration process [8] is evaluated in terms of the average Peak Signal-to-Noise Ratio (PSNR). If MAX is the maximum intensity value of the image then for the restored image Z of size M x N, the PSNR is

(MAX)²

PSNR = lO lo jo - where mean square error (MSE) is

M x N (5) with respect to the noise free original image

Average of PSNR is calculated over 100 different ensembles of randomly corrupted same image. The above method clearly reveals that, a novel, accurate and simple bad pixels detection and inpainting could be accomplished by way of the present invention. Importantly, the proposed method provides the online monitoring of the health of the sensor and helps to increase the usable life span of the sensor. This method reduces the rejection rate at the time of sensor production by using light weight real time software with no additional hardware cost. The proposed detection method is of two stages. The first stage makes use of local image statistics. The next stage helps to get rid of the false detections, giving rise to the performance close to the ideal detector. Extensive simulation results further confirmed that proposed detection method could achieve zero miss detection and very low false detection. Accurate detection can be achieved by applying the bad pixels detection procedure on multiple different images. Thus advantageously, adaptive bad pixels inpainting method can be applied directly for any image. Qualitative results for dead pixels and dead lines are shown in Fig.2, 3, 4, and 5. Qualitative results for stuck pixels and stuck lines are shown in Fig. 6, 7, 8 and 9. Results for hot and cold pixels are shown in Fig.10. Simulation is also performed on 16 bit original medical image. Results of the medical image are shown in Fig.11 and 12. Proposed bad (dead, stuck, hot and cold) pixels detection and inpainting procedure is capable to serve as a powerful scheme because it achieves high PSNR, in spite of image corruption up to higher level of noise in a senor array/imaging device.

It is thus possible by way of the present invention to providing a method for the detection of the bad pixels and rectification by inpainting of sensor arrays of an imaging device such as a digital camera. The method is adapted to ensure zero miss detection and very low false detection and thus ensuring improvement of image quality even with higher proportion of bad pixels in sensor array. The method and device of the invention thus enable monitoring the health of sensors online and enhance the useful life of sensor array by correcting the bad pixels in such device favoring significant economic advantage.

Claims

WE CLAIM:

1. A method for detecting the bad pixels in sensor array comprising generating error/noise map involving intensity based detection of location information of bad pixels of sensors using local statistics of the digital image involving values signifying the location of bad pixel and normal pixel respectively.

2. A method for detecting the bad pixels in sensor array as claimed in claim 1 comprising generating said error/noise map comprising providing a noise map involving intensity based detection of location information of bad pixels of sensors using local statistics of the digital image and subjecting the location of the thus identified bad pixels to further evaluation of its local statistics to confirm that the pixels are bad or not and thereby generate a final noise map with 1 and 0 values which signify the location of bad pixel and normal pixel respectively.

3. A method for detecting the bad pixels in sensor array as claimed in anyone of claims 1 to 2 comprising assessing number of bad pixels and its subgroups to determine health of a sensor array and/or generating error map over a several subsequent images and involving combinations thereof to further confirm error detection.

4. A method for detecting the bad pixels in sensor array as claimed in anyone of claims 1 or 3 comprising step of authentication of the images captured by a camera by comparing with the error/noise map thus generated with images taken from a camera.

5. A method for detecting the bad pixels in sensor array and concealing of the error comprising

(i) generating noise map involving intensity based detection of location information of bad pixels of sensors using local statistics of the digital image involving values signifying the location of bad pixel and normal pixel respectively;

6. A method for detecting the bad pixels in sensor array and concealing of the error comprising

7. A method for detecting the bad pixels in sensor array and concealing of the error as claimed in anyone of claims 5 or 6 wherein said bad pixels include dead, stuck, hot and cold pixels.

8. A method as claimed in claim 7 wherein carrying out the step of detection of said dead pixel comprises (I) imposing a W_x X W₂ window for each noisy image and centering the same on the current pixel to determine the maximum value (s_max) and minimum value (s_min) within the window, thereafter generating the first stage noise map V following :

wherein Sy is the intensity value of the pixel at location (i,j) and S_min is the global minimum intensity value of the image and wherein in this first stage noise map, r(i,j) = 1 means corresponding pixel in noisy image is probably dead and r(i,j) = 0 means corresponding pixel in noisy image is normal; followed by

(II) generating a final noise map V comprising

Level 1) imposing a w_x x w₂ window around the dead pixel in noisy image, such that w_lr w₂ are much smaller than previous window size; if more than w_min normal pixels such as when pixels which have corresponding values in first stage noise map 0 are found, thereafter calculating the distance measure d between the normal pixels and the central dead pixel and going to Level 3; if the number of normal pixels is not more than w_min then going to Level 2; where it is assumed for simulation that d is mean of absolute distance (MAD);

Level 2) setting the w, = Wj + p,, i= l,2, p, are small positive integers and going to

Level 1 and increasing the window up to a pre determined fixed size;

9. A method as claimed in claim 7 , wherein carrying out the step of detection of stuck pixels comprises (I) imposing a Wj X W₂ window for each noisy image and centering the same on the current pixel to determine the maximum value (s_max) and minimum value (s_m,„) within the window, thereafter generating the first stage noise map V following :

wherein s_i(j is the intensity value of the pixel at location (i,j) and Smax is the global maximum intensity value of the image and in this first stage noise map, r(i,j) = 1 means corresponding pixel in noisy image is probably stuck and r(i,j) = 0 means corresponding pixel in noisy image is normal followed by

(II) generating a final noise map V comprising Level 1 ; Imposing a w_x x w₂ window around the stuck pixel in noisy image, such that Wi, w₂ are much smaller than previous window size and if more than w_min normal pixels are found, then calculating distance measure d between the normal pixels and the central stuck pixel and go to Level 3 and if the number of normal pixels are not more than w_min then go to Level 2. Here it is assumed that d is MAD.

Level 2 : Setting the w, = Wj + p_if i=l,2, Pi are small positive integers and going to Level 1, Increasing the window up to a pre determined fixed size.

Level 3 : If the corresponding d is less than the threshold δ (δ is a small positive), replacing the corresponding pixel in temporary noise map V to 0 else leave it and move to next stuck pixel and going to Level 1, whereby the noise map V is final noise map.

A method as claimed in claim 7 wherein carrying out the step of detection of said hot and cold pixels comprises (I) imposing a Wj X W₂ window for each noisy image and centering the same on the current pixel, thereafter generating the first stage noise map V following :

if (S_max - s_{i j}) < T₄ or (,¾ - S_min) < T₅

otherwise

where S is the intensity value of the pixel at location (i,j). S_max and S_mm are the global maximum and minimum intensity value respectively. In first stage noise map, r(i,j) = 1 means corresponding pixel in noisy image is probably hot or cold and r(i,j) = 0 means corresponding pixel in noisy image is normal.

(II) generating a final noise map Y comprising

Level 1 : Imposing a Wi x w₂ window around the hot (cold) pixel in noisy image, such that Wi, w₂ are much smaller than previous window size and if more than w_min normal pixels are found, then calculating distance measure d between the normal pixels and the central hot (cold) pixel and going to Level 3. If the number of normal pixels are not more than w_min then going to Level 2 wherein it is assumed that the distance measure d is MAD.

Level 2 : Setting the w, = w, + p,, i= l,2, p, are small positive integers and going to Level 1. Increasing the window up to a predetermined fixed size.

Level 3 : taking minimum of distance of the central hot (cold) pixel from both end preferably such that if image is 8 bit then T₃ = min(s_i(j , 255 - Sjj). and If the corresponding d is less than the threshold qJ3, replacing the corresponding pixel in temporary noise map V to 0 else leaving it and moving to next hot(cold) pixel and going to Level iwith q being a constant (0 < q < 1) to thereby generate the final noise map V.

11. A method as claimed in anyone of claims 5 to 10 wherein said method is adapted to be applied for images of any bit depth.

12. A method as claimed in anyone of claims 5 to 11, wherein the said step of correcting the pixel error by inpainting comprising the steps of :

(1) imposing a w₃ x w₄ window around the bad pixel in noisy image to find out a set S having all those noisy image pixels under current window for which corresponding value in noisy map are 0; if S is not a null set then going to Step 2, otherwise going to Step 3,

(2) replacing the current pixel with median of the set S and proceeding to the next bad pixel and going to Step 1,

(3) setting the w, = W| + Aw,, i=3,4 Aw, are small positive integers and going to Stepl and increasing the window up to a pre determined maximum size.

13. A method as claimed in anyone of claims 5 to 12, wherein said bad pixel detection procedure followed by bad pixel inpainting is adapted to achieve high PSNR even with image corruption for high number of bad pixels.

14. A method as claimed in anyone of claims 5 to 13, wherein in said bad pixel detection is performed online or offline and once detected the inpainting may be applied online/offline depending on application demand.

15. An image sensor system comprising a sensor array adapted to detect bad pixels and carry out inpainting of bad pixels comprising :

16. An image sensor system as claimed in claim 15 wherein said sensor array is adapted to detect variety of bad pixels selected from dead pixel, stuck pixel, cold and hot pixels following any of the methods as claimed in claims 5 to 15.

17. An image sensor system as claimed in claim 15 or 16, which is adapted to convert optical/thermal any other signal including thermal , micro-wave, ultrasound, x-ray to electrical signals.

18. A method to detect the bad pixels in sensor array and correction of detected error of bad pixels by inpainting and an image sensor adapted to carry out such method substantially as herein described and illustrated with reference to the accompanying figures.