DE102011100350A1 - Image sensor with adjustable resolution - Google Patents

Abstract

The invention relates to an image sensor (1) and a method for operating an image sensor (1), the image sensor (1) having a matrix of sensor elements (2) to which color filters (R; G; B) are assigned and which are grouped (3 ) are divided. The image sensor (1) is operated in such a way that it normally works with a resolution that is lower than the number of sensor elements (2) and only when required with an increased to maximum spatial resolution of the image sensor (1).

Description

The invention relates to an image sensor and to a method for operating an image sensor, wherein the image sensor has a multiplicity of sensor elements arranged in matrix form, to which color filters are assigned and which are divided into groups.

In vehicles, driver assistance systems with sensor devices are increasingly being used today. The driver assistance systems often use a camera system as environment sensor, for example, to detect a variety of objects in the environment of the vehicle. In the case of the camera systems used, an image sensor (for example a CMOS or CCD image chip) is usually used in a camera housing behind an imaging system, for example a camera. B. a lens is arranged.

A camera system of this kind is for example from the DE 30 21 470 C2 known. The system described therein comprises an image sensor having a plurality of picture elements which are arranged in rows and column direction and detect electromagnetic radiation which is projected by means of a taking lens on the image sensor.

In addition, image sensors are already known that are able to recognize colors, eg. B. by demosaicing / demosaicing in an overlaid with a mosaic color filter image sensor. Such an image sensor is used for example in the DE 10 2005 047 127 A1 specified. The CMOS image sensor described therein includes a sub-layer or substrate having photodiode areas and metal lines formed in a semiconductor substrate to generate charges of incident light, a dielectric interlayer formed on the entire surface of the sub-layer, color filter layers of Red, green and blue formed on the dielectric interlayer to respectively transmit light of particular wavelengths to the photodiode regions, a planarization layer formed on the color filter layers, and convex microlenses having a certain curvature and formed on the planarization layer are to converge light on their respective color filter in the color filter layers, and thereby converge color separated light on the photodiode areas.

In order to meet future requirements for applications of camera systems, especially for driver assistance functions - z. As for the detection of traffic signs or traffic signs - are increasingly image sensors with color detection (colorimager) and with a resolution (number of pixels) of significantly larger than one megapixel needed.

For image processing, however, image sizes of more than one megapixel are currently difficult to handle because, on the one hand, in particular in parallel processing for several applications, the required computational power is not available and, on the other hand, the generated heat loss unduly heats the camera system.

In known image sensors, which are usually provided with a Bayer color filter (Bayer pattern), so-called "binning" several pixels - typically 2 or 4 pixels -, z. B. by adding the brightness values, either analogously by charge addition or charge transport or digitally by adding the digitized values, connected together to form a pixel and thus evaluated summarized. As a result, the photosensitivity of the image sensor increases, the resolution and at the same time reduces the required computing power and u. a. the noise ratio can be improved. Such image sensors, which allow a variable resolution, are usually operated completely in high-resolution mode (without binning) or in low-resolution mode (with binning). The disadvantage here is that either in high-resolution mode constantly a greatly increased computing power for image processing is required or in low-resolution mode, the color information is lost.

In one "IEEE Workshop on Charge-Coupled Devices and Advanced Image Sensors, 2001" was described by K. Hara et al. proposed a solution to obtain the color information when reducing CIF format to QCIF. For this, however, a relatively complicated circuit is required and the sensitivity of the image sensor is reduced by the process as a whole.

In addition, today, for example, standardized digital cameras also allow a reduction of the resolution, but this is usually done by a mathematical scaling of an initially high-resolution captured image. Thus, the processing of a high-resolution image is also required and there is thus basically no reduction in the required computing power.

It is thus common to all known systems that they hitherto offer no suitable solution in order to achieve both the increased image resolution required for future requirements and at the same time to provide color information without excessively high or unnecessary computing power and the required electrical circuits required for image acquisition and image processing . complex.

The invention is therefore an object of the invention to provide an image sensor and a method for its operation to meet the future requirements of camera systems, especially for use as sensor systems in vehicles, the described problems and disadvantages of known systems are avoided.

This object is achieved by an image sensor having the features of claim 1 and by a method having the features of claim 7. Advantageous embodiments and further developments are the subject of dependent claims, wherein combinations and developments of individual features are conceivable with each other.

An essential idea of the invention is to provide a high resolution image sensor, i. H. an image sensor comprising a plurality of photosensitive sensor elements to configure with a color filter such that each arrays of preferably 2 × 2 sensor elements with identical color filters are adjacent to each other. Depending on requirements, the image sensor is operated in such a way that in normal operation the sensor elements are interconnected in groups (arrays of adjacent sensor elements with identical filter properties) and the image sensor thus has a reduced resolution compared to the absolute number of sensor elements. For arrays with 2 × 2 sensor elements, the image sensor consequently has a 4-fold reduced resolution. As a result, color resolution can be achieved by means of demosaicing, as in known image sensors, and at the same time computing power can be saved if high resolution is not required. Is it, however, for example, required for the detection of object details, z. As in the detection of additional traffic signs that a higher spatial resolution is achieved for the entire image sensor or for certain areas, the sensor elements of all groups or selected groups are evaluated individually (without binning). The image sensor is thus operated in particular in such a way that, in the normal case, the resolution is reduced compared with the number of sensor elements, and the image sensor is only used if required with an increased to maximum spatial resolution. The color information remains intact in both cases.

The image sensor according to the invention comprises a multiplicity of sensor elements which form pixels or pixels and are sensitive to electromagnetic radiation, which are arranged next to one another in the form of a matrix, in particular in columns and in rows. By way of example, the sensor elements can be designed in such a way that they convert electromagnetic or optical radiation into electrical energy and / or exhibit an electrical resistance which depends on the incident electromagnetic radiation. The term electromagnetic radiation refers to both visible light, as well as invisible infrared light and ultraviolet radiation. The image sensor can be designed, for example, as a CCD image sensor (CCD: Charge-Coupled Device) or as a CMOS image sensor (CMOS: Complementary Metal Oxide Semiconductor), in particular as an active pixel sensor (Active Pixel Sensor, APS), with light-sensitive photodiodes as sensor elements. The sensor elements of the image sensor are associated with color filters, so that each sensor element detects electromagnetic radiation from a specific wavelength range. The color filters can be designed, for example, as absorbing color masks, in particular in the primary colors red, green and blue and / or in the colors yellow, magenta and cyan, which are preferably arranged directly over the sensor elements of the image sensor. Each sensor element provides in this case only information / signals for a single color component of the light or for electromagnetic radiation from the corresponding wavelength range. The color filter can in particular also be designed as a thin-layer color filter, with a carrier which is arranged above the sensor elements of the image sensor and on which a color filter is applied, for example as a mixture of organic color pigments and inorganic materials, eg. B. halides. The image sensor can also be configured with a microlens color filter system, wherein a color filter layer is formed between the sensor elements and microlenses arranged above them. Alternatively or additionally, it is also possible to use a system that utilizes the different penetration depths of light, in particular of red and blue light, in silicon (Foveon X3 sensor). The sensor elements of the image sensor are divided into groups. Each group comprises an array of n × m sensor elements, with n ≥ 2 and m ≥ 2. The image sensor preferably comprises at least four groups of sensor elements, each group having the same number and the same ratio of n × m sensor elements. According to the invention, the sensor elements of a group are assigned the same or the same color filter, ie each group supplies only information or signals for a single color component or for incident electromagnetic radiation from a specific wavelength range. The sensor elements of a group can be configured either with separate (ie with the same) color filters having identical filter properties, or it can be configured for all sensor elements of a group a common (ie the same) color filter. In the interconnection (binning) of the sensor elements of a group thus the image sensor according to the invention functions as known image sensors and the color detection (demosaicing) can in particular as be performed in a normal manner in a normal image sensor. The spatial resolution and the color resolution of the image sensor are reduced in this case compared to the total number of sensor elements and thus over the maximum spatial resolution of the image sensor.

In a preferred embodiment, each group comprises an array of n × n sensor elements. Each group is designed with the same number of sensor elements with n ≥ 2. Preferably, each group comprises exactly 2 × 2 sensor elements.

In a further preferred embodiment, the groups are arranged next to each other and arranged in the form of a matrix in the row and column direction.

In an advantageous embodiment, the sensor elements of adjacent groups are assigned different color filters. Preferably, the color filters of the sensor elements alternate in groups in a row of adjacent groups, for example in the sequence red-green or green-blue.

In a further advantageous embodiment, color filters are assigned to the sensor elements in such a way that the image sensor, due to the matrix-like juxtaposed groups which in turn each comprise at least 2 × 2 sensor elements, wherein the sensor elements of a group are assigned the same or the same color filter, a Bayer matrix (Bayer pattern). The image sensor is similar to a chess board, with a regular color filter array (coated, which preferably consists of 50% green and 25% red and blue color.) The color filters are based on the groups, for example, in odd rows in the sequence red-green and arranged in straight lines in the sequence green-blue.

In a particular embodiment, the image sensor comprises at least two operating modes or the image sensor can be operated in at least two ways. In one operating mode, the sensor elements are preferably interconnected or evaluated in groups. This process has already been described in the introduction and is generally referred to as binning, in particular also as on-chip binning, wherein adjacent sensor elements (in this case all sensor elements of a group) are combined or interconnected and whereby, in particular, the resolution capability is changed. The combination of several sensor elements takes place, for example, by addition of the detected brightness values, either analogously by charge addition by charge transport or digitally by addition of digitized brightness values. In another mode, the sensor elements are evaluated individually, d. H. In particular, there is no interconnection of adjacent sensor elements by means of binning. The individual evaluation can also be made only for selected groups, the sensor elements of the remaining groups are combined or not evaluated. By the individual evaluation of sensor elements of selected groups or of all groups, a higher spatial resolution can be achieved for the affected areas of the image sensor.

The inventive method relates to the operation of an image sensor, preferably an image sensor according to the invention, according to one of the preceding embodiments. The method thus relates in particular to the operation of a high-resolution image sensor before or before its electromagnetic radiation sensitive sensor elements a color filter array is configured so that a plurality of adjacent sensor elements have the same filter characteristics and evaluated in groups in groups in normal operation or in an output mode. Preferably, the groups of sensor elements with identical filter properties are each the same size, d. H. comprise the same number of n × m and n × n sensor elements with n ≥ 2 and m ≥ 2. An embodiment with 2 × 2 or 3 × 3 sensor elements per group is preferred. The groups form in particular a matrix-shaped filter array in the row and column direction, wherein the sensor elements of the groups are preferably configured with color filters such that a Bayer matrix (Bayer pattern) is spanned by the groups. In normal operation or in an output mode, the image sensor thus has due to the grouping or by binning a relation to its absolute number of sensor elements reduced resolution, for example in groups of 2 × 2 sensor elements, a 4-fold reduced resolution. The reduction of the resolution relates in particular to the spatial resolution of the image sensor.

In an advantageous embodiment of the method, in particular in the event that a higher spatial resolution is to be achieved, an operating mode is selected in which all sensor elements of the image sensor are evaluated individually (without binning), ie. H. in which the division into groups, in particular for the spatial representation, is resolved.

In this case, the color resolution of the image sensor remains unchanged, whereas the spatial resolution with respect to the normal operation / output operation multiplies, in particular to the number of total existing sensor elements.

In a further advantageous embodiment of the method, in particular for the case in which a higher spatial resolution is to be achieved, an operating mode is selected in which the individual sensor elements of the selected groups are evaluated individually (without binning) with regard to selected groups. The sensor elements of the remaining, ie the non-selected groups, are further grouped according to groups (evaluated with binning). As a result, for the area / areas of the image sensor in which the selected groups are located, a higher spatial resolution compared to the normal operation / output operation and compared with the remaining groups is achieved, which are further interconnected, in particular by means of binning.

In a further advantageous embodiment of the method, in particular for the case in which a higher spatial resolution is to be achieved, an operating mode is selected in which the individual sensor elements of the selected groups are evaluated individually (without binning) with regard to selected groups and with respect to the sensor elements of the remaining ones , d. H. the non-selected groups, no evaluation at all. As a result, a higher spatial resolution compared to the normal operation / output operation is achieved for the area / areas of the image sensor in which the selected groups are located. Particularly advantageous in this embodiment is that due to the fact that the remaining groups are not evaluated, computing power can be saved in the image processing with increased resolution.

In an advantageous embodiment of the method, the proportion of selected groups which are operated with increased resolution (without binning) corresponds to at most a quarter or a ninth relative to the total quantity / number of groups of the image sensor, in which case the sensor elements of the remaining Groups are not evaluated. Preferably, exactly one fourth of the groups is selected, in particular if the groups each comprise 2 × 2 sensor elements. In this case, the read timing can be maintained from the normal operation / output operation since each case, i. H. both in normal operation and in operation with increased resolution, the same number of sensor elements must be evaluated. Analogously, when configuring the image sensor with groups of 3 × 3 sensor elements, exactly one ninth of the groups are selected for the operating mode with increased spatial resolution, etc.

The inventive method and the image sensor according to the invention are preferably used in vehicles, in particular in conjunction with camera devices and image processing systems, for example for detecting the interior and / or the environment of the vehicle. The use in the context of functions of one or more driver assistance systems which use an image-processing sensor unit preferably takes place here. Known driver assistance systems are, for example, traction control or traction control such as ABS (Antilock Braking System), ESP (Electronic Stability Program), EDS (Electronic Differential Lock), as well as adaptive cornering lights, headlights and dipped beam assist, night vision systems (English : night vision), Cruise Control, Park Assist, Brake Assist, Adaptive Cruise Control (ACC), Distance Alert, Turn-off Assistant, Traffic Jam Assistant, Lane Detection System, Lane Keeping Assist, Lane Keeping Assist, Lane Change Assist, Intelligent Speed Adaptation, ANB (Automatic Braking), Curve Assist, Tire pressure monitoring system, driver condition detection, traffic sign recognition, platooning.

Further advantages and optional embodiments will become apparent from the description and the drawings. Embodiments are shown in simplified form in the drawing and explained in more detail in the following description.

In 1 is an embodiment of the image sensor according to the invention 1 shown. The representation is greatly simplified and, in principle, only schematically shows the color filter arrangement via photosensitive sensor elements 2 of the image sensor 1 For example, as an image pickup element in a camera device behind an imaging system, for. B. a lens with one or more lenses, can be arranged on a circuit board. The image sensor 1 comprises a multiplicity of sensor elements arranged next to one another in a matrix and sensitive to electromagnetic radiation 2 , z. As photosensitive photodiodes, which color filters R, G, B are assigned. The color filters in this case correspond to the three primary colors red (R), green (G) and blue (B). Each of the sensor elements 2 Accordingly, only detected electromagnetic radiation from the wavelength range corresponding to the respective color component. The image sensor 1 out 1 For example, it may be constructed as a CMOS image chip, with a bottom layer of photodiodes arranged in matrix form and light-sensitive, an overlying intermediate layer with individual color filters assigned to the photodiodes and an overlying microlens structure. In principle, any possible and known structure of an image sensor, in particular a CCD or CMOS image chip is possible, the image sensor having a color filter arrangement corresponding to 1 and with an arbitrary resolution (number of sensor elements 2 ) can be configured. The sensor elements 2 of the image sensor 1 are in groups 3 assigned. Every group 3 consists of one Array of 2 × 2 sensor elements 2 , The sensor elements 2 a group 3 in each case the same color filter R, G, B is assigned. This can be done either for each sensor element 2 a separate color filter R, G, B or it may be for a group 3 of sensor elements 2 a common color filter R, G, B may be arranged. In the embodiment of the image sensor according to the invention 1 out 1 is through the groups 3 of the image sensor 1 , a Bayer matrix spanned. Where, based on the groups 3 , in odd rows alternate the color filters in the sequence red (R) -Green (G) and in even lines the color filters in the sequence green (G) -Blue (B). According to the invention, the image sensor 1 out 1 operated in normal operation or in an output mode such that the sensor elements 2 according to groups 3 be evaluated in summary, in particular by means of binning. This results in a resolution of the image sensor reduced by a factor of four 1 compared to its maximum resolution with respect to the absolute number of sensor elements 2 , Require, for example, certain image processing tasks, holistic or certain areas of the image sensor 1 concerning, a higher spatial resolution, can be used for all groups 3 or for selected groups 3 the binning picked up and the sensor elements 2 of these groups 3 be evaluated individually. When arranging the image sensor 1 in a camera in a vehicle for monitoring the vehicle ahead in the direction of travel environment, such as traffic sign recognition, if necessary, especially if objects such. B. traffic signs and their additional characters were detected, according to groups 3 summarized evaluation of the sensor elements 2 be lifted, thereby increasing the spatial resolution. The color resolution must, in particular in the case of traffic sign recognition, usually not be increased because only a higher spatial resolution is required for the detection of traffic signs and their additional characters. By a corresponding 1 designed image sensor 1 Thus, the computing power required for the image processing or for the recognition of objects can be optimized in which a cancellation of the binning and associated increased computing power is limited to times when actually relevant or interesting objects are detected in the image, the details of which are increased Resolution are recognizable. According to a further embodiment of the image sensor 1 and a preferred embodiment of the method according to the invention for operating the image sensor 1 , the cancellation of the binning can be applied to the area of interest of the image sensor 1 ie on selected groups 3 , which again results in a considerable reduction in the number of sensor elements to be processed 2 or reach pixel. In a particularly advantageous embodiment, the area of interest by removing the binning, has a quarter of the size of the image sensor 1 or it includes the proportion of selected groups 3 a quarter of all groups 3 especially if each group 3 exactly 2 × 2 sensor elements 2 includes. The remaining groups 3 will not be evaluated temporarily in this case. In this case, the reading timing relative to the operation of the image sensor 1 be maintained in normal operation. As a result, the electronic circuit can be made easier and the image processing can go through with the same timing, as in low-resolution image, ie as in normal operation.

LIST OF REFERENCE NUMBERS

1

image sensor

2

sensor element

3

group

R

red

G

green

B

blue

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• DE 3021470 C2 [0003]
• DE 102005047127 A1 [0004]

Cited non-patent literature

• "IEEE Workshop on Charge-Coupled Devices and Advanced Image Sensors, 2001" was described by K. Hara et al. [0008]

Claims (12)

Image sensor ( 1 ) comprising a multiplicity of sensor elements which are arranged next to one another in a matrix and which are sensitive to electromagnetic radiation (US Pat. 2 ), wherein the sensor elements ( 2 ) Color filters (R; G; B) are assigned, so that each sensor element (R 2 ) detects electromagnetic radiation from a specific wavelength range, characterized in that the sensor elements ( 2 ) in groups ( 3 ), each group ( 3 ) each an array of n × m sensor elements ( 2 ), with n ≥ 2 and m ≥ 2, and the sensor elements ( 2 ) of a group ( 3 ) is associated with the same or the same color filter (R; G; B). Image sensor ( 1 ) according to claim 1, wherein each group ( 3 ) each an array of n × n sensor elements ( 2 ), Image sensor ( 1 ) according to any one of the preceding claims, characterized in that the groups ( 3 ) are arranged next to each other in matrix form. Image sensor ( 1 ) according to one of the preceding claims, characterized in that the sensor elements ( 2 ) of contiguous groups ( 3 ) are each assigned different color filters (R; G; B). Image sensor ( 1 ) according to one of the preceding claims, characterized in that the sensor elements ( 2 ) Color filters (R; G; B) are assigned such that the image sensor (R; 1 ) due to the matrix-like juxtaposed groups ( 3 ) each having at least 2 × 2 sensor elements ( 2 ) having the same or the same color filters (R; G; B) has a Bayer matrix. Image sensor ( 1 ) according to one of the preceding claims, characterized in that at least two operating modes of the image sensor ( 1 ) are provided, wherein in one mode, the sensor elements ( 2 ) by groups ( 3 ) and in another operating mode the sensor elements ( 2 ) of each group ( 3 ) or selected groups ( 3 ) are evaluated individually. Method for operating an image sensor ( 1 ), which is designed according to one of the preceding claims, characterized in that the output mode is an operating mode of the image sensor ( 1 ), in which the sensor elements ( 2 ) by groups ( 3 ) are evaluated together. A method according to claim 7, characterized in that in the event that a higher spatial resolution is to be achieved, an operating mode of the image sensor ( 1 ), at which all sensor elements ( 2 ) of the image sensor ( 1 ) are evaluated individually. A method according to claim 7, characterized in that in the event that a higher spatial resolution is to be achieved, an operating mode of the image sensor ( 1 ) in the case of selected groups ( 3 ) the sensor elements ( 2 ) individually and concerning the remaining groups ( 3 ) the sensor elements ( 2 ) by groups ( 3 ) are summarized. A method according to claim 7, characterized in that in the event that a higher spatial resolution is to be achieved, an operating mode of the image sensor ( 1 ) in the case of selected groups ( 3 ) the sensor elements ( 2 ) and the remaining groups ( 3 ) no evaluation of the sensor elements ( 2 ) he follows. Method according to claim 9 or 10, characterized in that the amount of selected groups ( 3 ) a share of i. a quarter of the total amount of groups ( 3 ) of the image sensor ( 1 ), if each group ( 3 ) exactly 2 × 2 sensor elements ( 2 ). ii. a ninth of the total amount of groups ( 3 ) of the image sensor ( 1 ), if each group ( 3 ) exactly 3 × 3 sensor elements ( 2 ). Method according to one of claims 7 to 11, which is used in the context of one or more functions of one or more driver assistance systems of a vehicle, in particular in the context of image processing tasks for detecting the vehicle interior and / or the vehicle environment.

Images