WO1994018801A1

WO1994018801A1 - Color wide dynamic range camera using a charge coupled device with mosaic filter

Info

Publication number: WO1994018801A1
Application number: PCT/US1994/001358
Authority: WO
Inventors: Ran Ginosar; Tamar Genossar; Ofra Zinaty; Noam Sorek; Daniel J. Kligler; Yehoshua Y. Zeevi; Arkadi Neyshtadt; Dov Avni
Original assignee: I Sight, Inc.
Priority date: 1993-02-08
Filing date: 1994-02-07
Publication date: 1994-08-18
Also published as: EP0739571A1

Abstract

The apparatus (10) is a color wide dynamic range apparatus which includes a filter (12) interposed immediately in front of reoccurring color elements so that each pixel represents a given color element for the scene. At least two exposure levels are taken of the scene and the pixel outputs are decoded to generate the video luminance and chrominance signals. The images of the at least two exposure levels are combined to form a final image.

Description

COLOR WIDE DYNAMIC RANGE CAMERA

USING A CHARGE COUPLED DEVICE WITH MOSAIC FILTER

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. patent application Serial No. 07/795,350 filed November 20,

1991 entitled "Color Wide Dynamic Range Camera", which is, in turn, a continuation-in-part of U.S. patent application

Serial No. 07/388,547, filed August 23, 1989, now U.S.

Patent No. 5,114,442. Additionally, this application is related to U.S. Patent No. 4,858,014 and currently pending

U.S. patent application Serial No. 07/805,512, filed December 11, 1991. The disclosures of all of the above- identified U.S. patents and patent applications are incorporated herein by reference.

BACKGROUND OF THE INVENTION

Field of the Invention

This invention pertains to video imagery and more particularly to apparatuses and techniques for providing enhancement of video color images. In particular, the present invention uses a four-color mosaic filter with a single chip CCD in conjunction with color wide dynamic range algorithms. It is also applicable, however, to other types of mosaic filters known in the art. Description of the Prior Art

Various types of video enhancement apparatuses and techniques have been proposed. Prior implementations of color wide dynamic range cameras, such as those disclosed in the above-identified parent applications hereto, have used a plurality of CCD chips to generate the image data for subsequent processing. The use of multiple CCD chips, however, adds to the complexity and cost of the instrument.

Moreover, current consumer video cameras, i.e., camcorders, almost universally use a single CCD chip. Therefore, a single CCD implementation is required to use dynamic range enhancement algorithms in a camcorder. Single CCD chip implementations are similarly preferred for endoscopic applications.

OBJECTS AND SUMMARY OF THE INVENTION

It is therefore an object of this invention to provide a color wide dynamic range camera implemented with a single CCD chip.

It is also an object of this invention to provide a color wide dynamic range camera which is adapted for use with a camcorder.

It is a further object of this invention to provide a color wide dynamic range camera which is adapted for use with a conventional endoscope.

These and other objects of the invention will be more apparent from the discussion below.

SUMMARY OF THE INVENTION

There is thus provided in accordance with the preferred embodiment of the present invention, video imaging apparatus including means for providing a plurality of video color images of a scene at different exposure levels using a single CCD chip, each color image being separated into several (e.g., four in the preferred embodiment) different components prior to sensing by the CCD chip by way of a multiple color mosaic filter in front of the CCD chip. The pixel outputs are then decoded — subjected to specific mathematical operations by the processing electronics following the CCD output — to generate the video luminance and chrominance signals.

The present invention integrates the digital processing of the mosaic color CCD data with ADAPTIVE SENSITIVITY™ dynamic range enhancement. This integration provides for a substantial savings in total system processing hardware chip count and cost. It also permits better control of the color and detail production of the camera's video output. The mosaic storage format also provides for a unique video image compression technique.

BRIEF DESCRIPTION OF THE DRAWINGS

Further objects and advantages of the invention will become apparent from the following description and claims, and from the accompanying drawings, wherein:

Figure 1 is a general block diagram of the present invention.

Figure 2 is a representative illustration of the data image elements, with the size of the data image elements exaggerated.

Figure 3 is a general block diagram of the long and short processing of the present invention.

Figure 4 is a block diagram of the color path of the present invention.

Figure 5 is a block diagram of the intensity path of the present invention.

Figure 6 is a block diagram of the look-up table processing of the present invention.

Figure 7 is a block diagram of the joint operations of the present invention. Figure 8 is a block diagram of the differential color, intensity result block of the present invention.

Figure 9 is a block diagram of the color suppression factor block of the present invention.

Figure 10 is a block diagram of the color conversion block of the present invention.

Figure 11 is a block diagram of the mosaic generation block of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring now to the drawings in detail wherein like numerals indicate like elements throughout the several views, one sees that Figure 1 is a block diagram of the apparatus 10 of the present invention.

Apparatus 10 includes a mosaic filter 12 which is bonded to the front of CCD 14 (preferably a single chip), generally as part of the CCD integrated circuit manufacturing process.

The alternating mosaic filter elements are cyan, magenta, yellow and green (wherein C "cyan' = G 'green' + B 'blue'; M 'magenta' = R 'red' + B; and Ye 'yellow' = R+G). When the CCD 14 charge output is read out, the photoelectric charges from vertically adjacent sensor elements of CCD 14 are combined in the analog shift register (not shown). The on-chip addition gives rise to α, β, γ and δ elements as described below and as described in the Sony CCD 1992 Data Book (Sony part number ICX038AK), as well as earlier editions.

As shown in Figure 2, the mosaic complementary additive color image comprises alternating first and second rows of image data elements 18_A — 18_D wherein first rows include alternating α and γ data elements (18_A and 18_B, respectively), and wherein the second rows include alternating β and δ data elements (18_C and 18_D, respectively). The α image data elements 18_A are an equal mixture of cyan plus green (i.e., C + G = B 'blue' + 2G 'green'). The γ image data elements 18_B are an equal mixture of magenta plus yellow (i.e., M + Ye = 2R 'red' + B + G). Similarly, the β image data elements 18_C are an equal mixture of cyan plus magenta (i.e., C + M = 2B + G + R) and the δ image data elements 18_D are an equal mixture of green plus yellow (i.e. , G + Ye = 2G + R).

Those skilled in the art will recognize that: Y (i.e., intensity) =

from which the definition of intensity (Y) in the red, green, blue (RGB) system may be derived:

Y = R + 1.5G + B

Of course, those skilled in the art will realize that other color combinations are equally applicable. Each mosaic element of filter 12 covers the sum of two adjacent pixel sensors of CCD 14 so that each pixel output of CCD 14 is representative of one of the above color combinations given for the various image data elements 18. Four different monochromatic images, each representative of one color combination chosen from the colors of image data elements 18_A - 18_D of a given scene, are therefore generated by CCD 14.

As can be further seen from Figure 1, apparatus 10 includes four major functions as summarized below:

1. Long/short exposure processing:

The first stages of the algorithm are performed on the two exposures (long/short) separately. The processing of each exposure is divided into two paths:

a. Color path processing - evaluates color component for each pixel.

b. Intensity (Y) path processing - handles intensity information for each pixel. This includes point ("DC") intensity and edge information. 2. Point processing:

Each of the long/short exposure length processing functions (typically implemented on separate chips) outputs its point intensity information, obtained from the Y path processing, to four look-up tables (LUTs). These tables determine the point intensity result of the two exposures, the normalized color weighting or color selection function and the saturation color suppression factor. This information serves the joint operation processing stage. The four LUTs are programmable, thus enabling operation with different functions when necessary. In an alternative embodiment, these LUTs may be replaced by a programmable, piecewise linear (PWL) or other digital function generator.

3. Joint operations processing:

Joint operations processing joins results produced by the long and short processing blocks, and results obtained from the functions implemented in the table processing LUTs, and evaluates the final output of the algorithm. The processing is divided into:

a. Color components and Y result calculation — evaluates the final result of the color components and the intensity of each pixel.

b. Color suppression factor calculation— evaluates the color suppression factor for each pixel, based on both edges and saturation information. c. Color conversion processing — converts mosaic differential color space to RGB color space and produces RGB and Y/Cr/Cb outputs for each pixel.

4. Generate mosaic processing.

Generate Mosaic processing converts RGB color space back to mosaic color space for each pixel. The mosaic information generated enables economical hardware storage of processed images. This information can be retrieved and replayed through the algorithm — in Replay Mode— to produce RGB or Y/Cr/Cb output of the stored result.

Referring now to Figure 1, similar to U.S. Patent No. 5,144,442 and parent U.S. patent application Serial No. 07/795,350 (the disclosures of which, again, along with U.S. patent application Serial No. 07/805,512 and U.S. Patent No. 4,858,014 are incorporated herein by reference), apparatus 10 includes long/short processing as implemented by mosaic long exposure field block 20 and mosaic short exposure field block 22 which obtain, respectively, a long and a short exposure from CCD 14 in order to allow subsequent processing by long exposure processing block 24 and short exposure processing block 26. The terms "long" and "short" exposures are used here generally to denote two image inputs to apparatus 10. In general, "long" is used to mean an input with a higher exposure level, and "short", a lower exposure level. The higher exposure may be generated in several ways, including longer integration time, typically obtained by controlling the "electronic shutter" of the CCD chip; higher gain in the analog amplifiers preceding digitization; or a larger mechanical iris opening or other external gating means.

These two image inputs are usually generated by a single CCD chip, but may also be generated simultaneously by two separate, boresighted CCD chips, as disclosed in the aforementioned earlier applications. For the more common case in which the two inputs are generated by a single CCD chip, they may be generated either sequentially (as in the case of the first method above— integration time control) or concurrently (by using two input channels with different gain levels). When a sequential method is used, field memories are required at the input to apparatus 10 (in blocks 20 and 22) to synchronize the data coming from the two sequential fields or frames. These memories are not needed in concurrent modes, except for purposes of "freezing" the image for electronic, digital storage. Switching logic incorporated in blocks 20 and 22 controls the data flow into and out of these field memories, depending on which mode (sequential or concurrent) is used. Of course, this implementation could be expanded to more than two exposure levels. Blocks 24 and 26 may typically be provided on separate processing chips or incorporated together in a single chip. The processing for each exposure is divided into two paths:

1. Color path processing— handles color information for each pixel (see color path block 28 in Figure 3 and, in more detail, in Figure 4); and

2. Intensity (Y) path processing— handles intensity information for each pixel (see Y path block 30 in Figure 3 and, in more detail, in Figure 5).

Additionally, as shown in more detail in Figure 3, long/short exposure processing blocks 24, 26 include mosaic white balance block 32.

Mosaic white balance block 32 receives the following field of information from long/short exposure field blocks 20, 22:

α γ α γ α γ . . . . ..

β δ β δ β δ . . . . ..

α γ α γ α γ . . . . ..

β δ β δ β δ . . . . ..

α γ α γ α γ . . . . ..

β δ β δ β δ . . . . ..

. . .

That is, the information from CCD 14 with the mosaic data order intact is received.

After processing, the mosaic white balance block 32 outputs color-corrected data values: α_wb γ_wb α_wb γ_wb α _wb α_wb ....

β_wb δ_wb β_wb δ_wb β_wb δ_wb ....

α_wb γ_wb α_wb γ_wb α _wb α_wb ....

β_wb δ_wb β_wb δ_wb β_wb δ_wb ....

α_wb γ_wb α_wb γ_wb α _wb α_wb ....

β_wb δ_wb β_wb δ_wb β_wb δ_wb ....

. . .

Mosaic white balance block 32 contains mosaic color balance functions. These functions may typically be implemented as eight mosaic white balance LUTs (look-up tables). That is, for each exposure there is a set of four LUTs, one for each mosaic data type: α, β , γ, and δ . Independent calculation of white balance correction factors is performed for each exposure. This enables white balancing scenes where the observable parts of the two exposures are at different color temperatures. The LUTs may contain multiplicative correction factors which are evaluated as follows:

From the definitions of α, β , γ, δ and Y it follows, that for a white image (where by definition R*=G=B), the following relations should hold:

Based on these relations correction factors can be calculated by enforcing these relations on the average of a white image:

where Y denotes a selective average over Y in the given white image and

and

are the respective average values of α, β, γ and δ . Saturated or cutoff pixels are excluded from this average. Since by definition,

the equations for the correction factors are:

The LUT values are calculated by simple multiplication of each mosaic data type by its respective correction factor: α_wb = α * C_α

β_wb = β * c_β

γ_wb = 7 * C_γ

δ_wb = δ * C_δ

In an alternative embodiment, these LUTs are replaced by digital multipliers. Furthermore, the LUTs may also be loaded with correction functions other than simple linear multiplicative factors. Alternatively, the mosaic balance correction factors can be computed based on four average signals, namely, α, β, γ and δ , instead of merely two of them as above. This alternative yields improved uniformity of color balance under difficult conditions. Alternatively, the white balance function may be done on the RGB color components in the color conversion block 78 (described below).

Referring now to Figure 4, color path block 28 is shown in more detail. As previously stated, the input to color path block 28 is the image data α_wb, β_wb, γ_wb, δ_wb after processing by mosaic white balance block 32.

The initial processing of color path block 28 is performed by color difference evaluation block 34 which receives data α_wb, β_wb, γ_wb, δ_wb from mosaic white balance block 32 and calculates color difference components dr, db for each pixel in the array: dr dr dr . . . . = (γ-α) (γ-a) (γ-α) . . . . db db db . . . . = ( δ-β) ( δ-β) ( δ-β) . . . . dr dr dr . . . . = (γ-a) (γ-a) (γ-α) . . . . db db db . . . . = ( δ-β) ( δ-β) ( δ-β) . . . . wherein:

dr Ξ γ - α , the differences between successive readings in even lines . db ≡ δ - β , the differences between successive readings in odd lines.

The correct evaluation of dr and db requires horizontal interpolation as described in the following equations:

For each pixel (i,j) in even lines i:

where j is the pixel index along the line (here and henceforth).

For each pixel (i,j) in odd lines i: db(j_odd) =δ_wb(J_odd)

db (j_even) β_wb(j_even)

Color difference components dr, db are thereafter received by low-pass color component block 36 which calculates a low-pass color component dr_lp or db_lp for each pixel: dr_lpdr_lpdr_lp . . . . = (γ-α)_lp (7-0-)_lp . . . .

db_lpdb_lpdb_lp . . . . = (δ-β)_lp (δ-β)_lp . . . .

dr_lpdr_lpdr_lp . . . . = (γ-α)_lp (γ-α)_lp . . . .

db_lpdb_lpdb_lp . . . . = (δ-β)_lp (δ-β)_lp . . . .

Block 36 performs horizontal low-pass filtering on dr_hp and db_hp calculated in block 34. This reduces color artifacts caused by interpolation. In a preferred embodiment, the low-pass filter width is five pixels and its coefficients are ¼, ¼, ¼,

The equations follow: For pixels (i,j) in even lines i: dr(j-2) + 2*dr(j-1) + 2*dr(j) + 2*dr(j+1) + dr(j+2) dr_lp(j) _________________________________________________________

8

For pixels (i,j) in odd lines i: db(j-2) + 2*db(j-1) + 2*db(j) + 2*db(j+1) + db(j+2) db_lp (J)= _________________________________________________________

8

Delay buffer 38 receives the output from low-pass color component block 36 and directs db_lp(i_even -1), dr_lp (i_odd - 1), db_lp(i_even +1) and dr_lp (i_odd+1) to vertical interpolation block 40 and dr(i_even) and db(i_odd) to multiplexer 42.

Vertical interpolation block 40 receives the low-pass color components as described above and generates interpolated low-pass color components dr_lp in the odd numbered lines and dbL in the even numbered lines: db_lp db_lp db_lp db_lp . . . . .

db_lp db_lp db_lp db_lp . . . . .

The equations follow:

For even lines i:

db_lp(i-1,j) + db_lp(i+1,j)

db_lp (i,j)= _________________________

2

For odd lines i:

dr_lp(i-1,j) + d_lp (i+1,j)

dr_lp (i,j)= _____________________________

2 The interpolated low-pass color components dr_lp db_lp are multiplexed with the original low-pass components dr_lp, db_lp to give the color path output values dr and dp for each pixel. This function is performed by multiplexer 42, which separates the output received from delay buffer block 38 and vertical interpolator block 40 into a first path including db_lp (i_even) and db_lp(i_odd) and a second path including dr_lp (i_even) and dr_lp(i_odd).

Referring n

igure 5, which discloses in more detail the intensity (Y) processing block 30 shown in Figure 3, one sees that the input to intensity (Y) processing block 30 from mosaic white balance block 32 (Figure 3) is received by intensity evaluation block 44 which outputs computed intensity Y for each pixel.

Since only one of the four data types (α, β, γ, δ) is present at any given pixel, the intensity evaluation block 44 calculation is performed as follows (based on the prior definition of Y) :

For pixels (i,j) in even lines i:

For pixels (i , j ) in odd lines i :

The output from intensity evaluation block 44 is received by delay buffer 46, generate output intensity block 48 and limit block 50.

Delay buffer 46 is a delay line of two horizontal lines, required for the 3x3 and 1x3 matrix transformations in Y path block 30. Together with the color path delay buffer 38 and with Y path delay buffer 54, it may be implemented in a preferred embodiment in mosaic data space, operating on the input α, β, y, δ data before the intensity (Y) evaluation block 44 and color difference evaluation block 34. It is shown here schematically for clarity.

Vertical low-pass filter 52 receives intensity (Y) signals from the intensity evaluation block 44 as delayed by delay buffer 46. Block 52 generates the vertical low- pass intensity Y_vlp defined as: γ_vlp(i,j) = Y(i-1,j) + 2Y(i,j) + Y(i+1,j)

4

The unfiltered intensity (Y) input will sometimes exhibit horizontal stripes, one pixel high in each field, in areas of transition to saturation. These stripes stem from the different color spectra of the a , β, y, and δ pixels, as a result of which the α+7 value of Y(i(even),j) may, for instance, reach saturation at a lower level of optical intensity than the β+δ value of the vertically adjacent Y(i+1(odd),j). Y_vlp averages these values to obtain a function that is smooth over the transition area.

Generate output intensity block 48 receives intensity (Y) information from intensity evaluation block 44 and vertical low-pass intensity (Y_vlp) information from vertical low-pass filter 52. The output of block 48 is output intensity (Y_out) to point processing LUT block 62 (see Figure

1).

Block 48 replaces the original luminance Y, computed by the intensity evaluation block 44, with Y_vlp when Y_vlp approaches saturation, in order to prevent the appearance of horizontal stripes as explained above. Block 48 implements the function: Y if Y_vlp < Y_threahold

Y_out =

Y_vlp if Y_vlp ≥ Y_threahold

The value of Y_thrahold is typically equal to approximately 220 on an 8-bit scale of 0-255. As values of Y approach saturation, image detail is lost in any event, so that substituting Y_vlp in the high range does not adversely affect the perceived resolution. Y_vlp is used as the selecting input in order to ensure a smooth transition.

Limit block 50 receives intensity (Y) signals from intensity evaluation block 44 and generates limited luminance Y_limit. Limit block 50 cuts off the upper range of intensity (Y) values that are to be input to edge detection block 56, in order to prevent detection of false edges or horizontal stripes that can arise in areas of transition to saturation. Limit block 50 implements the function: Y_limit = min {Y, Y_lim} the value of Y_lim is typically equal to approximately 220.

The output of limit block 50 (i.e., Y_limit) is delayed by delay buffer 54 and received by edge detection block 56 which outputs edge information for each pixel.

Edge detector block 56 convolves the Y_limit value and its 8 immediate neighbors, with a high-pass or edge detecting kernel.

In one embodiment, the 3x3 Laplacian operator may be used:

Alternatively, to accommodate the geometric characteristics of the CCD raster and to give greater emphasis to the vertical edges, the following kernel may be used:

In an alternative embodiment, the edge detector block 56 could be implemented as separate horizontal and vertical convolution operations (such as a 1 x 3 or 3 x 1 matrix), with additional logic to avoid overemphasis of diagonal edges. This alternative embodiment is less hardware intensive and gives improved picture quality in some circumstances.

Edge suppress block 58 receives the vertical low-pass intensity (Y_vlp) signals from vertical low-pass filter 52 and outputs edge suppression function f_edge to edge multiplier 60.

The edge suppression function varies between 0 and 1 in the long exposure processing block 24 only. In the short exposure processing block 26, the function is set to 1, i.e., no edge suppression at this point. The function is typically implemented in block 24 in a piecewise linear fashion as follows: if Y_vlp < LOWSAT

if LOWSAT ≤ Y_vlp < DEEPSAT if DEEPSAT≤ Y_lp

Typically LOWSAT is set to approximately 190 and

DEEPSAT to approximately 220.

Edge multiplier 60 receives input from blocks 56, 58 and generates suppressed edge ed_supp to intensity (Y) result calculation.

Edge multiplier 60 multiplies the edge output of the edge detector block 56 by the edge suppression function f_edge from block 58 to generate an output value ed_supp to joint operations block 64 (see Figure 1). The purpose of this multiplication is to suppress distorted large edges that may appear in the long exposure at intensity (Y) values near saturation, at the same time as they appear in the short exposure at lower values of intensity (Y). The double appearance of such edges was found empirically to cause the resulting displayed edges to be overemphasized and sometimes smeared on account of blooming in the long exposure. The long exposure edge is suppressed so that only the short exposure edge will pass through to the output image. The edge suppress function may also be used to reduce the amplitude of edges from the long exposure which may be otherwise exaggerated due to the higher gain of the long exposure relative to the short exposure.

Additionally, as shown in phantom in Figure 5, an optional multiplier or LUT (block 57) may be added to multiply the output of block 56 times the ratio of exposure times (duration of long exposure/duration of short exposure) or the corresponding gain ratio, or some function of the exposure and/or gain ratio. This reflects the ratio of scales of these two values.

In the above manner, Y path block 30 outputs processed luminance Y_out, edge, and edge_supp to point processing block 62 and joint operations block 64.

Referring now to Figure 6, one sees that point processing block 62 includes four point processing functions, all of which receive output intensity (Y_out) values from the long and short exposure processing blocks 24, 26 (see Figure 1). These functions may typically be implemented as LUTs in RAM or ROM memory. Point processing block 62 generates arbitrary function values for input to the joint operations block 64 (Figure 1). The four tables of block 62 are:

1. The intensity (DC result) block 66 which generates a LUT value of intensity (Y_lut) for the joint operations block 64.

Block 66 controls the amount of point ("DC") luminance that is summed with the edge information in generating the output luminance, Y_result. In its most general formulation,

where f is an arbitrary function. It has been found that a quasilogarithmic or fractional power dependence of Y_lut on the inputs gives the best output image appearance, and the general function above can generally be reduced to a more compact LUT or piecewise linear implementation.

One simple possible computation of Y_lut is as follows:

a) Y_short is multiplied by the exposure ratio, so that it is on the same scale as Y_long. That is, if a certain pixel x is acquired within the active (linear) sensitivity region of both the short and long exposures, then Y_long(x) = R*Y_short(x), where R is the exposure ratio, R=long exposure time/short exposure time (or any other ratio representing the two sensitivities).

b) Subsequently, Y_long and R*Y_short are linearly combined, so that the sum of their relative weights is always 1. That is, Y_wdr=a*Y_long + b*R *Y_short, 1≥a≥0, 1≥b≥0, and a+b=1 (the wdr index stands for 'wide dynamic range'). The common practice is to set a=1 in the region where the short exposure is cut-off (too dark), b=1 in the region where the long exposure is saturated (too bright), and a>0, b>0 in the region where both exposures carry meaningful information. However, this does not cover all cases, e.g. when neither exposure carries any information (long is saturated and short is cut-off, or both saturated, or both cut-off).

c) Finally, the dynamic range of Y_wdr is reduced (yielding Y_lut) by either a logarithmic function, or by multiplying it by a small fraction, or by using any empirically found mapping which resembles the log function or a similar contraction, e.g., the square root.

Other possible values for Y_lut comprise empirical modifications of the function described above.

2. & 3. Color weight normalize blocks 68, 70 for long and short exposures, respectively, which generate normalizing color weights w₁/Y₁ and w₅/Y₅. Blocks 68 and 70 control the proportions of mixing the color values, dr and db, from the long and short exposures, respectively, that will be used to generate the output color values, dr_result and Y_result. Generally, w₁ and w₅ are chosen so as to give predominant weight at each pixel to the color values taken from the exposure in which the intensity (Y) luminance values are in the linear portion of the range, and to give a smooth transition over luminance gradient regions of the image. For the most part, W₁ and w₅ are determined on the basis of Y_out(long) alone, except for cases where the long exposure is near saturation while the short is near cutoff, so that neither gives a linear reading.

The weighting values are complementary, i.e., w₁ = 1 - w₅, and w₁, w₅ > 0. The outputs of blocks 68 and 70 are normalized by division by the corresponding values of Y_out for the long and short exposures. Preferably, a floating point representation for the output values of blocks 68, 70 is used so as to maintain sufficient accuracy to prevent noticeable quantization in the output image.

Alternatively, instead of weighted addition of the normalized color from the two exposures, simple selection of the normalized color from one exposure or the other, with an ordered dither

(alternation) of the color selection in areas of transition may be used. In this case, when W₁ = 1, the color is selected from the long exposure; when w₅ = 1, it is selected from the long exposure, and when neither w factor is 1, color values are taken alternately from long and short, according to a pseudo-random probability distribution in which the long and short color value probabilities are equal to the w₁ and w₅ values. A normalized color value that is an average of the long and short values may also be mixed into the dither, in order to give a smoother color transition.

4. Saturation color suppression factor block 72 generates the color suppression factor Wht that reduces chroma saturation (adds white to the image) in areas of luminance saturation of the input image. An additional edge color suppression factor, Z_ed, is computed in the joint operations block (as will be described hereinafter). The minimum of Wht and Z_ed, both of which vary from 1 to 0, multiplies the chroma components at the output stage of color conversion. Thus, as Wht approaches zero, so does the color saturation of the output image. The purpose of the saturation color suppression function is to reduce the appearance of color artifacts that arise due to CCD saturation. The linear relationships between the α, β, γ, and δ CCD outputs and the true RGB colors break down as the CCD 14 approaches saturation. As non-linear deviations cannot be readily corrected, suspected distorted colors are "whitewashed". Similar techniques are used in the analog domain in conventional CCD cameras.

As shown in Figure 6:

Wht = w₁ + w₅*z⁵ w₁ and w₅ are identical to the above color weighting values. The variable Z₅ is a function of Y_out(short), varying between 0 and 1, as shown schematically in the lower right corner of Figure 6. It tends to zero in areas where the short exposure luminance approaches either saturation or cutoff. This function will give Wht = 0 at the saturation end (where generally w₅ = 1 while w₁ = 0). At the cutoff end, normally w₁ ≈ 1 as long as there is adequate overlap between the long and short exposures, so that in this range the function will usually give Wht ≈ 1.

In normal mode, in which only one input channel is operative (see explanation below), Wht = 1 from Y=0 up to the low saturation threshold (typically 190). From this threshold up to the deep saturation limit of Y (typically 220), Wht drops linearly to its saturation value of 0. In replay mode (see below), there is no saturation color suppression.

Figure 7 discloses the joint operations block 64 (also see Figure 1). Joint operations block 64 combines the chrominance and luminance data from the long and short exposure processing blocks 24, 26, together with data from point processing block 62, to generate a combined Y/dr/db result. Block 64 then converts this result to output in standard RGB or Y/Cr/Cb (luminance, chrominance (red) and chrominance (blue)) color space. A color suppression factor Z is computed and applied to the chrominance outputs in order to reduce color artifacts (by reducing chroma saturation) around edges and areas of luminance signal saturation.

Joint operations block 64 includes:

1. The dr, db, Y block 74 (recalling that dr

and db are the differences between successive readings in even and odd lines, respectively) which receives dr, db values from the color path outputs of long and short exposure processing blocks 24, 26 respectively; ed_supp from the intensity (Y) path output of long exposure processing block 24 and edge data from the intensity

(Y) path output of short exposure processing block 26; and Y_LUT, W₁/Y₁ and W₅/Y₅ from table processing (LUT) block 62. Block 74 generates combined intensity Y/dr/db results to color conversion block 78 (to be discussed). Block 74 will be discussed in greater detail hereinafter.

2. The color suppression factor block 76 which

receives ed_long and ed_short from edge detector block 56 and saturation color suppression factor (Wht) from point processing block 62 and generates chroma suppression factor Z for color conversion block 78. Block 76 will be discussed in greater detail hereinafter. 3. The color conversion block 78 which receives Y_result, dr_result, db_result from block 74 and Z, the color suppression factor from block 76 and generates R_out, G_out, and B_out and Cr and Cb.

Block 78 will be discussed in greater detail hereinafter.

The dr, db, Y block 74 is shown in further detail in Figure 8.

Block 74 includes an intensity (Y) calculation which is performed by adders 79, 80 and edge limiting block 81.

Adder 79 receives ed_supp (long) data from long exposure processing block 24, and ed_short from short exposure processing block 26. These two inputs are added to give edge_result, which is then input to the edge limiting block 81. Edge limiting is implemented as a piecewise linear function with 6 inflection points (A₁...A₆) and 4 slopes (S₁...S₄), as shown in the upper right inset of Figure 8. Generally the inflection points and slopes are chosen so as to enhance the smaller edges (i.e., S₂ and S₃ ≥ 1), while large edges (edge > A₅ or < A₂) are suppressed. Since these large edges come through strongly in the Y_LUT contribution anyway, the output image has a more pleasing appearance if they are not additionally enhanced. A₃ and A₄ may be set to 0, but it is sometimes desirable to set them to small non-zero values in order to suppress false edges due to noise. The best results appear to be obtained with |A₁| and |A₆| values of

50 to 60. The best values of the slopes |S_i| are typically in the range 0.5 to 2, but the hardware allows a greater range.

The edge_Umitcd output is then summed by adder 80 with the

Y_LUT output of block 62 to obtain the output luminance value

Y_result. Additionally, as shown in phantom in Figure 10, adder

80 may be removed from its location in Figure 8 and placed so thac the output of block 81 is not added to Y_result until just before being added into block 113_A-C, that is, as late as possible.

Block 74 further includes a dr, db calculation which is performed by the remaining sections of block 74. The dr, db calculation receives low-pass color components dr, db from the color paths of long and short exposure processing blocks 24, 26; w₁/Y₁ and w₅/Y₅ from block 62; and Y_result as calculated by adder 80. The dr, db calculation outputs dr_result and db_result.

The long and short values of dr and db are multiplied by the respective normalized color weights, w₁/Y₁ and w₅/Y₅ by multipliers 82, 84, 86, 88. These normalized, weighted color values from the two exposures are summed together by adders 90, 92 and then multiplied by Y_result by multipliers 94, 96 to give the scaled values:

dr_result = Y_result * db_result = Y_result *

Alternatively, dr_result and db_result may be generated by selection between the long and short normalized dr and db inputs (and possibly their long/short average values).

The color suppression factor block 76 of Figure 7 is shown in more detail in Figure 9.

Maximum value block 100 selects the higher of the two absolute values of ed_long and ed_short as calculated by absolute value blocks 98, 99. The result of the calculation of block 100, ed_max, is input to edge chroma suppression factor block 102 to calculate Z_ed. The calculation of Z_ed is implemented as a piecewise linear function, shown in the upper right corner of Figure 9. As can be seen in Figure 9, Z_ed receives a value between Th and 1, given by: if ed_max<E1 Z_ed = ( 1 - Th) if E1≤ed_max<E2

if E2≤ed_max

Typically, E1=10 and E2=27 have been found to give good results. The minimum value of Z_ed, Th, is ordinarily set to zero, to give complete chroma suppression at very strong edges. Th ≠ 0 is used only in replay of images stored in mosaic format (see generate mosaic block 120 described hereinafter), in which case Z_ed serves to suppress color anomalies resulting from the reinterpolation of the pixel values.

Thereafter, as shown in Figure 9, minimum value block 104 selects the minimum of the two color suppression factors, Z_ed and Wht, thereby determining the edge criterion or saturation criterion that should be used to provide the required degree of chroma suppression at the given pixel.

Referring now to Figure 10, which discloses in detail color conversion block 78 of Figure 7, one sees that color conversion block 78 receives Y_result dr_result, and db_result from block 74 and Z from block 76 and generates outputs in both the RGB and Y/Cr/Cb formulations.

In other words, block 78 takes the interim dynamic range enhancement results Y/dr/db, and converts them into conventional color components for system output.

Block 78 includes horizontal low-pass filter 106 which receives Y_result and calculates Y_result (1p) for the color matrix block 108.

Horizontal low-pass filter 106 is identical to the low-pass color component block 36 in the color path block

28 (see Figures 3 and 4). Since the dr and db inputs to the color matrix 108 have already been low-pass filtered by this low-pass filter operatar, it is necessary to filter the intensity (Y) value an well in order to prevent color artifacts.

Color matrix block 108 receives Y_result (lp) from horizontal low-pass filter 106 and dr_result and db_result from block 74 and generates low-pass RGB color component outputs.

If one recalls the derivation of Y, dr and db from the original α , β, γ and δ values of the mosaic CCD input:

Y ≡

dr ≡ γ - α

db ≡ δ - β

together with the RGB equivalencies of α, β, γ and δ , one obtains the following relationships between RGB and the Y/dr/db values:

R 0.2 0.4 0.1 Y

G = 3.5 * 0.4 -0.2 0.2 dr

B 0.2 -0.1 -0.4 db

The factor of 3.5 is required for normalization of the relation Y = R + 1.5G + B. Due to hardware implementation considerations, the color conversion matrix is calculated as follows:

R 1.0 2.0 0.5 Y

G = 0.7 * 2.0 -1.0 1.0 dr

B

1.0 -0.5 -2.0

db

In this way the matrix multiplication is performed by a series of shift/add operations. The multiplicative factor 0.7 is combined (by multiplication) with externally programmed RGB white balance correction factors as is described hereinafter. RGB white balance multipliers 109_A, 109_B, 109_C receive low-pass RGB signals from color matrix block 108 and generate normalized low-pass RGB signals.

Multipliers 109_A, 109_B, 109_C multiply each of the RGB low-pass values by a pre-computed white balance correction factor, adjusted by the normalization factor 0.7 required by the color matrix calculation. Although conventional RGB white balancing uses only two multiplicative factors, correcting R and B while G is held constant, this "short cut" does not preserve constant Y achromatic luminance. This loss of normalization may lead to the appearance of artifacts and incorrect luminance in the output. It is necessary, therefore, to use three multiplicative factors, normalized to preserve constant luminance Y.

The calculation of the correction factors is performed off-line by capturing a white image and selectively computing average values

and

excluding pixels near saturation or cutoff. From the definition Y = R + 1.5G + B, it follows that for a corrected pixel in the white image, it should be found that:

R = G = B = Y

From this relationship one derives the correction factors to be used in multipliers 109_A, 109_B, 109_C:

R factor = G factor =

B factor =

Output signal enhancement block 110 (which includes chroma suppression and RGB output functions) receives corrected low-pass RGB color component signals from color matrix block 108 via multipliers 109_A, 109_B, 109_C; Y_result from block 74; Y_result (lp) from block 106; and chroma suppression factor Z from block 76.

As noted above, the RGB values output from color matrix block 108 are low-pass values. High-frequency image information is "re-injected" into RGB according to the following equation (given here only for the R component, since the treatment of G and B is identical):

R_hp = R_lp + K*Y_result - K*Y

_result

K is an arbitrary constant between 0 and 1, chosen according to the degree of high-frequency enhancement required. Values in the range 0.4 < K < 0.8 are typically used.

The addition and subtraction of Y_result values to the RGB components can alter the original values of R/G and B/G, with the result that the correct hue of the image is not preserved. Therefore, in an alternative embodiment, R, G and B are multiplied by a high-pass enhancement function.

Since the RGB color component values contain both luminance and chrominance information, the chroma suppression factor, Z, is best applied to chrominance-only components, by adders 113_A, 113_B, 113_C: Cr = R - Y_result

Cg = G - Y_result

Cb = B - Y_result

Combining these equations with the previous ones for "high frequency re-injection", one obtains the following formula, which is implemented as shown in Figure 10 (including arithmetic element blocks 114, 115, 116) to obtain Cr from R_lp: Cr = R_lp + K* (Y_result - Y

) - Y_result and likewise for Cg and Cb. These Cr/Cg/Cb values are multiplied by Z by multipliers 112_A, 112_B, 112_C. At this point, Y/Cr/Cb output is available directly (using Y = Y_result), though it is preferable to add a bias of +128 to the signed digital outputs Cr and Cb in order to convert them to positive values for D/A conversion. In the alternative, Y_result can be added back into the chroma-suppressed Cr/Cg/Cb values (by adders 113_A-113_C) to obtain the final R_out/G_out/B_out.

Referring now to Figure 11, which discloses generate mosaic block 120 of Figure 1 in more detail, one sees that the input of generate mosaic block 120 is R_out/G_out/B_out from color conversion block 78 of joint operations block 64. The output of block 120 is the equivalent a, β , y, δ values in the format: α_eq γ_eq α_eq γ_eq . . . . .

β_eq δ_eq β_eq δ_eq . . . . ..

α_eq γ_eq α_eq γ_eq . . . . ..

β_eq δ_eq β_eq δ_eq . . . . ..

In order to reduce memory requirements for image storage and to allow stored images to be replayed through the apparatus 10 for display, the final RGB values from the processed image are used to generate equivalent, simulated mosaic values of α, β, γ, and δ . In this way, only eight bits per pixel of information must be stored, rather than the 24 bits of full output information. These mosaic values can later be replayed to regenerate the stored image.

The simulated mosaic values are generated by the following matrix in matrix block 122, based on the color equivalencies given hereinabove.

The factor of ¼ that multiplies the matrix is used for reasons of hardware convenience— in order to ensure that α, β, γ, and δ do not overflow the range 0-255 of 8 bits. To maintain the normalization relations given hereinabove, the factor should actually be 1/3.5. Therefore, in replay mode, mosaic white balance block 32 is used to multiply the a , β, γ, and δ values back by 4/3.5 (=8/7) before reprocessing. Finally, multiplexer 124 selects which one of the four mosaic values to output for each pixel according to the table:

Pixel _i,j

Apparatus 10 has three modes of operation: normal, adaptive sensitivity (AS), and replay.

1. Normal mode emulates the performance of a mosaic color CCD camera without adaptive sensitivity. In this mode only the long exposure portion of the pipeline operates. The processing functions are limited to decoding the mosaic input into conventional color components: Y/Cr/Cb or RGB, while additionally performing filtering operations for anti-aliasing, detail (edge) enhancement and chroma suppression where required. 2. Adaptive sensitivity mode uses all the resources of the processing pipeline to generate wide dynamic range images as described hereinabove.

3. Replay mode is required for displaying images that have been stored in RAM or disk. Apparatus 10 stores these images in a regenerated mosaic format in order to save on storage memory requirements. Replay mode is similar to normal mode, except that most of the enhancement operations are not performed: since the stored data have already been filtered once, it is for the most part not desirable to filter them again.

The preceding specific embodiments are illustrative of the practice of the invention. It is to be understood, however, that other expedients known to those skilled in the art or disclosed herein, may be employed without departing from the spirit of the invention or the scope of the appended claims.

Claims

CLAIMS What is Claimed is:

1. A color wide dynamic range video imaging apparatus comprising:

sensor means for providing a plurality of color video images of a scene at different exposure levels;

means for dividing each color video image into components; and

means for processing said components of each of said plurality of video images to produce a combined color video image including image information from said components of each of said plurality of color video images by applying neighborhood transforms to at least one of said components of each of said plurality of video images,

wherein said means for processing includes means for calculating point intensity data for said each of said plurality of said video images.

2. The color wide dynamic range video imaging apparatus of Claim 1, wherein said means for processing includes means for calculating color weighting factors.

3. The color wide dynamic range video imaging apparatus of Claim 2, wherein said means for processing includes means for calculating saturation color suppression factors.

4. The color wide dynamic range video imaging apparatus of Claim 1, wherein said means for dividing each color video image into components includes a filter means in front of said sensor means and said filter means includes filter elements of a plurality of colors, said plurality of colors corresponding to said components.

5. The color wide dynamic range video imaging apparatus of Claim 4, wherein said sensor means includes a plurality of pixel sensing elements and

wherein said filter elements are arranged in a regular repeating pattern with each filter element in front of a single pixel element of said sensing means.

6. The color wide dynamic range video imaging apparatus of Claim 5, wherein said processing means includes means for evaluating an intensity of each color component;

means for evaluating color suppression factors for each pixel; and

means for converting said components into RGB color space.

7. The color wide dynamic range video imaging apparatus of Claim 6, wherein said means for evaluating an intensity of each color component communicates with a means for substituting luminance values when luminance otherwise approaches saturation.

8. The color wide dynamic range video imaging apparatus of Claim 6, wherein said means for evaluating an intensity of each color component communicates with a means for limiting luminance values.

9. The color wide dynamic range video imaging apparatus of Claim 5, wherein said processing means includes white balancing means which calculates correction factors for said components based upon the intensity of said color video images.

10. The color wide dynamic range video imaging apparatus of Claim 9, wherein white balancing means calculates an average intensity of said color video images excluding saturated pixels and cut-off pixels.

11. The color wide dynamic range video imaging apparatus of Claim 9, including vertical low-pass filter means receiving intensity and false edge suppression means, wherein said false edge suppression means receives intensity data from said vertical low-pass filter means and calculates false edge suppression factors which are reduced when said intensity data exceeds a pre-selected saturation value.

12. The color wide dynamic range video imaging apparatus of Claim 11, wherein said false edge suppression means calculates said false edge suppression factors for an image of a longest of said different exposure levels.

13. The color wide dynamic range video imaging apparatus of Claim 12, further including edge detection means and wherein an output of said edge detection means is multiplied times said false edge suppression factors.

14. The color wide dynamic range video imaging apparatus of Claim 5, wherein color components are calculated by multiplying an intensity times a sum of products of prior color components and output of said means for calculating said color weighting factors.

15. The color wide dynamic range video imaging apparatus of Claim 6, including means for calculating white balance correction factors for said components converted into RGB color space, wherein said white balance correction factors for a given respective RGB component are calculated by dividing average overall luminance by a multiple of an average value of said respective given RGB component.

16. The color wide dynamic range video imaging apparatus of Claim 15, wherein said average value of said respective given RGB component is calculated excluding pixels which are substantially near saturation or substantially near cut-off.

17. The color wide dynamic range video imaging apparatus of Claim 6, including means for converting said RGB color space components into mosaic color components corresponding to colors of said filter elements of said filter means.

18. The color wide dynamic range video imaging apparatus of Claim 1, wherein said means for processing is implemented on a single chip.

19. An imaging apparatus comprising:

means for dividing each color video image into components; and

wherein said means for processing includes means for calculating point intensity data for said each of said plurality of said video images, and

wherein said processing means includes white balancing means which calculates correction factors for said components based upon the intensity of said color video images.

20. A color wide dynamic range video processing chip comprising:

means for processing components of each of a plurality of video images of a scene at different exposure levels to produce a combined color video image including image information from said components of each of said plurality of color video images by applying neighborhood transforms to at least one of said components of each of said plurality of video images,