CN102314683A - Computational imaging method and imaging system based on nonplanar image sensor - Google Patents

Abstract

The invention discloses a computational imaging method and an imaging system based on a nonplanar image sensor, and the light-receiving surface of the image sensor is arranged by adopting a nonplanar mode. The imaging method includes the following steps: extracting the image information which is acquired by the nonplanarly arranged light-receiving surface in single exposure, and generating an image focused at different depths according to the image information; in combination with the method for estimating depth from focus (DFF) and the method of estimating depth from defocus (DFD), carrying out depth estimation on the image focused at different depths to obtain a depth map; generating a full-focused image on the basis of the image focused at different depths and the depth map, and generating refocused images focused at specific positions and provided with specific depths of field on the basis of the obtained depth map and the full-focused image. The method and the system can use single exposure to effectively acquire the information of the depth of field, and can use a single image to restore and control the depth of field.

Description

A kind of calculating formation method and imaging device of nonplanar graph image-position sensor

Technical field

The present invention relates to calculate shooting field, relate in particular to the calculating sensory field, specifically, relate to being calculated to be of a kind of nonplanar graph image-position sensor as method and system.

Background technology

Traditional digital photography comes down to obtain the projection of scene on two dimensional surface in the three-dimensional world, uses resulting single image can accurately not recover the degree of depth and the structural information of three-dimensional scenic, can't bring depth perception, stereoscopic sensation to human eye.Traditional camera imaging model need be weighed between the depth of field and signal to noise ratio (S/N ratio), adjusts the depth of field through aperture, and when aperture is big more, the depth of field is more little, and aperture is more little, and then the depth of field is bigger but signal to noise ratio (S/N ratio) is very low.Just need grow exposure if increase signal to noise ratio (S/N ratio), and long exposure can be introduced the motion blur of image, therefore, traditional camera imaging model can't satisfy the demand of the depth of field and signal to noise ratio (S/N ratio) simultaneously.In addition, traditional camera also can't carry out big depth field imaging.To sum up, traditional single exposure digital photography can't be obtained the full appreciation to scene.

In computer vision field, utilize many images to come the degree of depth and the structure of restoration scenario usually, method commonly used comprises: use image from various visual angles, utilize the different cameral parameter that the image that is obtained is set, beat modes such as active light to scene.But adopt these methods to need the various visual angles camera usually, increase extra active light source or need the one camera multiexposure, multiple exposure.

The calculating shooting of rise in recent years is learned through designing novel collection mechanism to gather more visual information.Theoretical based on light field; The optical field acquisition of design one camera; Light field camera like classics can be gathered the angle information that traditional camera is lost through sacrificing spatial resolution in single exposure; Utilize the light field that is collected to carry out refocused, depth of field expansion, and can infer the degree of depth and the structure of scene.In addition, also received widely based on single exposure depth of field control of alternate manner and to have paid close attention to, made the fuzzy core of different depth have more differentiation property like mode, thereby can effectively carry out depth of field control and carry out rough estimation of Depth through the coding aperture.

But present acquisition mode all can't effectively collect the depth information of scene.The acquisition system theoretical based on light field in fact only collected the angle information of scene, and really do not collect the depth information of scene, and the angle information that usage space resolution exchanges for exists very big redundancy.Other single exposure acquisition system also can only be inferred the depth information of scene through the degree of depth clue in the image that collects, and the accurately degree of depth and the structural information of effective restoration scenario.Therefore, need to explore and use single exposure effectively to carry out the method that depth information is gathered, and then carry out accurate estimation of Depth and depth of field control.

Summary of the invention

The object of the invention provides being calculated to be as method and system of a kind of nonplanar graph image-position sensor; Solve traditional imaging and can't effectively gather the problem of scene depth information; And can decoupling zero under the situation of single exposure go out to focus on the image sequence of scene different depth, thereby effectively the degree of depth of restoration scenario with carry out depth of field control.

In order to solve the problems of the technologies described above; The invention provides a kind of calculating formation method based on imageing sensor; It is characterized in that the daylighting surface of said imageing sensor adopts the on-plane surface mode to arrange, said formation method may further comprise the steps: the image information acquisition step; The image information that gather through single exposure on the daylighting surface that extraction on-plane surface mode is arranged forms the image that focuses on different depth according to said image information; Depth map generates step, unites from the method for focusing estimating depth with from defocusing the method for estimating depth, and the said image that focuses on different depth is carried out estimation of Depth to obtain depth map; The image depth controlled step generates a width of cloth total focus image based on said image and the said depth map that focuses on different depth, focuses on the refocused image that ad-hoc location has particular depth of view based on depth map that is obtained and the generation of total focus image.

Further, this method also comprises: said daylighting surface adopts the on-plane surface arrangement mode to be: said daylighting surface is sensor pixel, and the pixel of said imageing sensor is arranged on the different height rank.

Further; This method also comprises: said daylighting surface adopts the on-plane surface arrangement mode to be: fibre faceplate is set guides light to arrive sensor plane, said fibre faceplate is set to the on-plane surface by the distribution of differing heights rank back to the end in said sensor plane.

Further; This method also comprises: said daylighting surface is divided into a plurality of regional areas; Comprise all height level in each regional area, each height level is corresponding to a focal plane of scene, and the image information of on each height level, being gathered is formed the image that focuses on a degree of depth.

Further; This method also comprises: generate in the step at said depth map; Through obtaining ID information from the mode that focuses on estimating depth, with said ID information as primary data, through drawing said depth map from the mode that defocuses estimating depth from the mode that defocuses estimating depth.

Further; This method also comprises: generate in the step at said depth map; Through obtaining ID information from the mode that defocuses estimating depth, with said ID information as primary data, through drawing said depth map from the mode that focuses on estimating depth from the mode that focuses on estimating depth.

Further; This method also comprises: in said image depth controlled step; With in the image sequence between respective pixel relatively the image information in the pixel of clear focusing take out and form a width of cloth total focus image, perhaps adopt the mode of deconvolution to obtain the image of clear total focus.

Further, this method also comprises: in said image depth controlled step, calculate the inconsistent fuzzy core of refocused image overall, draw the refocused image according to said fuzzy core and said total focus image.

The present invention also provides a kind of calculating imaging device based on imageing sensor; It is characterized in that; Comprise with lower unit: image sensor cell; Its daylighting surface adopts the on-plane surface mode to arrange, and the image information that gather through single exposure on the daylighting surface that extraction on-plane surface mode is arranged forms the image that focuses on different depth according to said image information; The depth map generation unit, it is united from the method that focuses on estimating depth with from defocusing the method for estimating depth, and the said image that focuses on different depth is carried out estimation of Depth to obtain depth map; The image depth control module, it generates a width of cloth total focus image based on said image and the said depth map that focuses on different depth, focuses on the refocused image that ad-hoc location has particular depth of view based on depth map that is obtained and the generation of total focus image.

Further, this system also comprises: said daylighting surface adopts the on-plane surface arrangement mode to be: said daylighting surface is sensor pixel, and the pixel of said imageing sensor is arranged on the different height rank.

Further, this system also comprises: said daylighting surface adopts the on-plane surface arrangement mode to be: fibre faceplate is set comes guiding fiber to pixel planes, said fibre faceplate is set to the on-plane surface by the distribution of differing heights rank back to the end in said pixel planes.

Further; This system also comprises: said daylighting surface is divided into a plurality of regional areas; Comprise all height level in each regional area, each height level is corresponding to a focal plane of scene, and the image information of on each height level, being gathered is formed the image of a degree of depth.

Compared with prior art, the present invention has the following advantages:

The present invention has realized a kind of calculating formation method and imaging device of nonplanar graph image-position sensor, uses single exposure to gather scene depth information effectively, for using single image to come the restoration scenario degree of depth and carry out depth of field control algolithm to have great importance.Compare the optical field acquisition method that spatial resolution is exchanged for the angular resolution and then the deduction degree of depth, utilize nonplanar sensor to sacrifice spatial resolution and come directly to exchange for depth resolution, make the utilization of spatial resolution more effective.Specific on-plane surface daylighting surface pattern of rows and columns of sensor designs; Through the resulting image sequence of single image decoupling zero that obtains by this on-plane surface sensor imaging device collection; Employing is from the method that defocuses estimating depth with from focusing on the method associating estimating depth of estimating depth; Thereby it is more accurate to obtain to carry out depth estimation result than traditional single image, thereby can carry out accurate more depth of field control.

Further, only just can obtain the image that focuses on different depth, make traditional from focusing on estimating depth DFF and can being applied to dynamic scene from defocusing estimating depth DFD algorithm with single exposure.

Further, the on-plane surface fibre faceplate that under the situation that does not change original camera hardware system, before sensor, adds made obtains the effect on on-plane surface sensor daylighting surface, can practice thrift cost largely.And estimation of Depth and depth of field control algolithm can realize on hardware systems such as ordinary PC or workstation, and be easy to use, flexible.

Fibre faceplate used in the present invention is widely used in military affairs, criminal investigation, monitoring, space flight, navigation, mining industry, the CCD coupling in fields such as medical treatment, figure image intensifying coupling, and high-definition television imaging and advanced aspects such as office equipment image applications.Utilize an end face to be non-plane surface optical fiber panel and sensors coupled, the on-plane surface sensor imaging device that is obtained, the field that can be applicable to need big depth field imaging field, need carry out depth of field control field, need the scene three-dimensional structure to recover.Therefore, the present invention has application fields.

Other features and advantages of the present invention will be set forth in instructions subsequently, and, partly from instructions, become obvious, perhaps understand through embodiment of the present invention.The object of the invention can be realized through the structure that in instructions, claims and accompanying drawing, is particularly pointed out and obtained with other advantages.

Description of drawings

Accompanying drawing is used to provide further understanding of the present invention, and constitutes the part of instructions, is used to explain the present invention with embodiments of the invention, is not construed as limiting the invention.In the accompanying drawings:

Fig. 1 is the process flow diagram based on the calculating formation method of imageing sensor according to the embodiment of the invention one;

Fig. 2 is other sensor surface pixel example alignment of branch differing heights level according to the embodiment of the invention one;

Fig. 3 is according to the on-plane surface fibre faceplate instance of the embodiment of the invention one and utilizes it and the synoptic diagram of sensors coupled;

Fig. 4 is the on-plane surface fibre faceplate synoptic diagram according to the alternate manner of the embodiment of the invention one;

Fig. 5 is being used for from defocusing estimating depth and the imaging model synoptic diagram that focuses on estimating depth according to the embodiment of the invention one;

Fig. 6 is the imaging model synoptic diagram that is used to form the refocused image according to the embodiment of the invention one;

Fig. 7 is the structural representation based on the calculating imaging device of imageing sensor according to the embodiment of the invention two.

Embodiment

Below will combine accompanying drawing and embodiment to specify embodiment of the present invention, how the application technology means solve technical matters to the present invention whereby, and the implementation procedure of reaching technique effect can make much of and implement according to this.Need to prove that only otherwise constitute conflict, each embodiment among the present invention and each characteristic among each embodiment can mutually combine, formed technical scheme is all within protection scope of the present invention.

In addition; Can in computer system, carry out in the step shown in the process flow diagram of accompanying drawing such as a set of computer-executable instructions, and, though logical order has been shown in process flow diagram; But in some cases, can carry out step shown or that describe with the order that is different from here.

Embodiment one

Fig. 1 is the process flow diagram based on the calculating formation method of imageing sensor according to the embodiment of the invention one, specifies the step of this method below in conjunction with Fig. 1.

Step S110 extracts the surperficial image information of being gathered of daylighting that on-plane surface is arranged, and forms the image that focuses on different depth according to this image information.

The imageing sensor that relates in the calculating formation method of present embodiment has adopted nonplanar daylighting surface arrangement mode.Generally speaking; The daylighting surface of sensor is a sensor pixel; Obtain the imaging results of scene under different focusing surfaces for being implemented in single exposure collection; Be preferably the height level of branch specific quantity for the pixel arrangement mode of on-plane surface sensor surface, choose the number of the different focusing surfaces of the corresponding scene of other number of different height level, also can confirm according to the precision of the required scene depth resolution that exchanges for.Preferably; Imageing sensor on average is divided into a plurality of regional areas with behavior unit; The pixel that all comprises predefined all height level in each regional area; Corresponding delegation of each height level or multirow pixel, other pixel of differing heights level that sets in each regional area is thought the same position of corresponding three-dimensional scene.Fig. 2 is the example schematic of other sensor surface pixel arrangement of differing heights level in a kind of minute; The image sensor surface pixel is divided into regional area 1,2,3...; Each regional area comprises 3 row pixels, and pixel column 1.1,2.1,3.1... are on a height, and pixel column 1.2,2.2,3.2... are on a height; Pixel column 1.3,2.3,3.3... are on a height; It is three different depths of corresponding scene respectively, and think local pixel capable 1.1,1.2,1.3 corresponding the same point in the scene, same corresponding the same point of scene of local pixel capable 2.1,2.2,2.3 and local pixel capable 3.1,3.2,3.3.In this manner, when imaging, only need single exposure, can come the scene on the different focussing planes is carried out to picture according to other pixel of differing heights level, to obtain the image of the different depth of field.

Because it is higher directly to make the sensor cost with the arrangement of differing heights rank pixel.Preferably; Place the daylighting surface of fibre faceplate through being close to the sensor pixel plane as sensor; And make on-plane surface as shown in Figure 2 on the end face of pixel planes at fibre faceplate, therefore fibre faceplate guiding light just be equivalent to and obtained nonplanar sensor to sensor.This fibre faceplate can be set to a series of neat compact arranged optical fiber, and preferred, the centre distance minimum of fiber can reach 3um, and numerical aperture can be accomplished more than or equal to 1.

Preferably; Make the fibre diameter fibre faceplate identical with pixel size; The on-plane surface optical fiber arrangements precision of making on fibre faceplate surface can reach optical fiber one by one, then a certain on-plane surface fibre faceplate instance and utilize it and the synoptic diagram of sensors coupled as shown in Figure 3.The optical fiber cross section is on imaging plane among Fig. 3, and the guiding visible light is to the sensor pixel plane, and the optical fiber cross section is equivalent to original sensor plane.Among the figure label be 1 optical fiber cross section in one plane, a focussing plane a in its corresponding scene.Label be 2 and 3 optical fiber cross section on two other plane, b and c on corresponding two other focussing plane.

In when imaging, recover the different image sequences that focus on, all are labeled as 1 pixel and take out and form piece image and just can obtain a width of cloth and focus on the image on a plane among Fig. 3.In like manner, with all are labeled as 2 and 3 pixel and take out the composition piece image respectively and just can obtain the image that focuses on b and the c plane respectively among the figure.Can obtain the image sequence that focuses on the scene different depth of a series of low resolution thus.As preferably, the different depth rank number of on-plane surface fibre faceplate and the different arrangement mode of optical fiber also can be set, as shown in Figure 4.Equally, the pixel surface of on-plane surface arrangement also can be used mode shown in Figure 4.

(wherein f is a focal length according to imaging model 1/f=1/v+1/u; U, v are respectively object distance and image distance); When focal length is the millimeter rank; The subtle change of image distance will cause the bigger variation of object distance, and for present embodiment, the change in depth that the on-plane surface sensor surface is small can be corresponding to the bigger variation of focussing plane.Suppose that focal length is 9mm, infinite if the focussing plane of scene is moved to by 1m, then the displacement of respective sensor only needs 81.7um; If it is infinite that the focussing plane of scene is moved to by 0.5m, then the displacement of respective sensor only needs 164.9um.Therefore, according to the imaging model theory, on the size of sensor, can guarantee the feasibility of present embodiment.

Step S120 unites from the method for focusing estimating depth DFF with from defocusing the method for estimating depth DFD, and the said image that focuses on different depth is carried out estimation of Depth to obtain depth map.

After obtaining to focus on the image sequence different depth under through step S110, in this step, adopt from focusing on estimating depth (DFF) and from defocusing two kinds of methods estimations of estimating depth (DFD) scene depth (depth map).As shown in Figure 5, suppose that the on-plane surface sensing obtains 5 equivalent imaging planes 1,2,3,4,5, resulting 5 images focus on scene 1 ', 2 ', 3 ', 4 ', 5 ' respectively and locate.

Use is obtained ID figure from the method that focuses on estimating depth (DFF), and detailed process is following:

From focusing on estimating depth mainly is the degree of depth of inferring scene through the focus level of measurement image sequence.For each width of cloth figure that obtains, do the preliminary measurement that difference obtains the focus level of image pixel with original image with through the original image behind the local non-mean filter; Use partitioning algorithm (like mean shift) that every image is cut apart then and obtain image block, the internal focus degree value of each image block is averaged as the focus level value of this piece interior pixels.Obtain after the focus level value of each pixel of all images sequence; The focus level of respective pixel between the image is carried out match about the variation of image distance with Gaussian function; Obtain the Gaussian function peak value, pairing image distance size when promptly each pixel focus level is maximum, for example the image distance size v of certain specific pixel when focus point d among Fig. 5; The object distance that can obtain scene corresponding point d ' according to the lens imaging model is big or small, has promptly obtained the degree of depth s of scene corresponding point.

In the practical application, if depth map not high to the result's of estimation of Depth accuracy requirement then that can directly use DFF to estimate.Obtain more high accuracy depth figure if desired, under the less situation of sensor daylighting surface elevation rank, the depth map that can DFF be estimated is as ID, and adopts and defocus being optimized of estimating depth (DFD).

Use is optimized depth map from the method that defocuses estimating depth (DFD), and detailed process is following:

From defocusing estimating depth mainly is the degree of depth of inferring scene through the relative fog-level between the measurement image sequence.

Suppose whole scene all the image of clear focusing (total focus) be I, the out-of-focus image that then focuses on particular depth of view under a certain degree of depth is:

Wherein, x and y all represent specific two-dimensional pixel coordinate,

The span of remarked pixel coordinate x, h _σ(y, x) can be approximate with Gauss model for fuzzy core:

$h_{\mathrm{\σ}}(y,x)=\frac{1}{2{\mathrm{\π\σ}}^2\left(y\right)}\exp (-\frac{{\left|\right|y-x\left|\right|}_2}{2{\mathrm{\σ}}^2\left(y\right)})---\left(2\right)$

Wherein σ (y) is the respective pixel y fuzzy quantity relevant with the degree of depth:

$\mathrm{\σ}\left(y\right)=\mathrm{\γ}\frac{\mathrm{Dv}}{2}|\frac{1}{F}-\frac{1}{v}-\frac{1}{s\left(y\right)}|---\left(3\right)$

Wherein F is a focal length, and D is a diaphragm diameter, and v is an image distance, and s is object distance (degree of depth), and γ is a calibration parameter.

Utilize above-mentioned convolution model can obtain to focus on the image I of different depth for two width of cloth ₁, I ₂Between fuzzy relatively convolution model be:

During as

, relatively fuzzy

relevant with the degree of depth

$I_1\left(y\right)=\∫h_{{\mathrm{\σ}}_1}(y,x)I\left(x\right)\mathrm{dx}\≈{\hat{I}}_1\left(y\right)=\∫h_{\mathrm{\Δ}{\mathrm{\σ}}_{12}}(y,x)I_2\left(x\right)\mathrm{dx}---\left(4\right)$

During as

, relatively fuzzy

relevant with the degree of depth

$I_2\left(y\right)=\∫h_{{\mathrm{\σ}}_2}(y,x)I\left(x\right)\mathrm{dx}\≈{\hat{I}}_2\left(y\right)=\∫h_{\mathrm{\Δ}{\mathrm{\σ}}_{12}}(y,x)I_1\left(x\right)\mathrm{dx}---\left(5\right)$

The fuzzy relatively relation with scene depth is:

$s\left(y\right)={(\frac{1}{F}-\frac{1}{v_2-v_1}-\frac{1}{|v_2-{\mathrm{<figure-callout\; id="1"\; label="v"\; filenames="HDA0000076217190000012.png,HDA0000076217190000021.png"\; state="\{\{state\}\}">v</figure-callout>}}_1|}\sqrt{1+\frac{4\mathrm{\&Delta;}{\mathrm{\&sigma;}}_{12}\left(y\right)\left|\mathrm{\&Delta;}{\mathrm{\&sigma;}}_{12}\left(y\right)\right|}{{\mathrm{\&gamma;}}^2{D}^2}\frac{v_2-v_1}{{\mathrm{<figure-callout\; id="2"\; label="v"\; filenames="HDA0000076217190000012.png,HDA0000076217190000021.png"\; state="\{\{state\}\}">v</figure-callout>}}_2+v_1}})}^{-1}---\left(6\right)$

From defocus estimating depth the energy term that will optimize be:

$\hat{s}=\arg \underset{s}{\min }E=\arg \underset{s}{\min }(E_d+\mathrm{\&alpha;}{E}_m)---\left(7\right)$

Wherein α is the regular terms coefficient, E _m(s) be level and smooth, it can have multiple choices, as gets l ₁Norm, promptly

If focus on the image I of different depth for two width of cloth ₁, I ₂, data item:

$E_d=E_{12}=\&Integral;H\left(\mathrm{\&Delta;}{\mathrm{\&sigma;}}_{12}\left(y\right)\right){\left|\right|I_1\left(y\right)-{\hat{I}}_1\left(y\right)\left|\right|}_2\mathrm{dy}+\&Integral;(1-H\left(\mathrm{\&Delta;}{\mathrm{\&sigma;}}_{12}\left(y\right)\right)){\left|\right|I_2\left(y\right)-{\hat{I}}_2\left|\right|}_2\mathrm{dy}---\left(8\right)$

Wherein H () is a step function.

For obtained by the on-plane surface imaging process many different images that focus on, adopt the constraint of many images to make and to obtain more superior result from what defocus estimating depth.Like I among the figure ₁, I ₂, I ₃, I ₄And I ₅Adopt and estimate fuzzy relatively mode in twos, promptly data item is:

E _d＝E ₁₂+E ₂₃+E ₃₄+E ₄₅ (9)

The depth map that can DFF be estimated during iteration optimization minimizes formula (7) and obtains final depth estimation result as ID, obtains depth map.

In practical application; If it is more that the daylighting surface of sensor comprises other number of plies of differing heights level, the degree of depth that the degree of depth that then obtains through DFD does not have DFF to obtain is accurate, therefore; As preferably; Obtain ID information through the mode from DFD, with the primary data of ID information as the mode of DFF, the mode through DFF draws depth map.

Step S130 generates a width of cloth total focus image based on image that focuses on different depth and depth map, focuses on the image that ad-hoc location has particular depth of view based on depth map that is obtained and the generation of total focus image.

The detailed process of obtaining (to the expansion of the scene depth of field) of total focus image is following:

The method of obtaining of total focus image has two kinds in the present embodiment:

A kind of method be with in the image sequence between respective pixel relatively the image information in the pixel of clear focusing take out and form a width of cloth total focus image.After the focus level of each pixel of acquisition all images sequence in step S120 from focus on estimating depth (DFF) method, a certain position pixel in the total focus image is taken from the highest pairing pixel of image of focus level in this position pixel of image sequence.Adopt some zone in the resulting total focus image of this mode possibly still exist the phenomenon of defocusing blurring.

Another kind method is the image that the mode of employing deconvolution obtains clear total focus:

Owing to can regard the convolution (formula (1)) of the inconsistent fuzzy core of image and space of a clear total focus as, can note by abridging and be I for the image of a limited depth of field _b=I*h, I are picture rich in detail, and h is a fuzzy core.Wherein to the fuzzy core h at any one pixel y place (y, shape x) is identical with the shape of aperture, generally uses Gaussian function or gate function approximate, thinks known; Its size is (formula (2) and (3)) of being departed from the degree decision of focussing plane by its corresponding fields sight spot.If corresponding scene point focuses on imaging plane, then (y x) can regard unit impulse response as to fuzzy core h.To depart from focussing plane far away more when the point of scene, and then the size of fuzzy core is just big more.Try to achieve the degree of depth of scene through step S120 after, just can obtain each the corresponding scene point of each pixel of image and the distance of focussing plane, thereby can confirm the size of the fuzzy core of each pixel for given focussing plane.Therefore can obtain the inconsistent fuzzy core of the pairing overall situation for resulting each width of cloth image that focuses on the image sequence under the different depth.

Equally, suppose that the resulting image sequence that focuses under the different depth is I ₁, I ₂, I ₃, I ₄, I ₅, can try to achieve its corresponding overall inconsistent fuzzy core is h ₁, h ₂, h ₃, h ₄, h ₅, the image of corresponding potential total focus is I.Can obtain the total focus image through the minimization of energy function so, concrete formula is following:

$\hat{I}=\arg \underset{I}{\min }({\mathrm{<figure-callout\; id="1"\; label="E"\; filenames="HDA0000076217190000012.png,HDA0000076217190000021.png"\; state="\{\{state\}\}">E</figure-callout>}}_1+{\mathrm{<figure-callout\; id="2"\; label="E"\; filenames="HDA0000076217190000012.png,HDA0000076217190000021.png"\; state="\{\{state\}\}">E</figure-callout>}}_2+{\mathrm{<figure-callout\; id="3"\; label="E"\; filenames="HDA0000076217190000012.png,HDA0000076217190000021.png"\; state="\{\{state\}\}">E</figure-callout>}}_3+{\mathrm{<figure-callout\; id="4"\; label="E"\; filenames="HDA0000076217190000031.png"\; state="\{\{state\}\}">E</figure-callout>}}_4+{\mathrm{<figure-callout\; id="5"\; label="E"\; filenames="HDA0000076217190000031.png"\; state="\{\{state\}\}">E</figure-callout>}}_5+\mathrm{\&alpha;}{E}_m)$

Wherein, α is the regular terms coefficient, E _i=|| I*h _i-I _i|| ₂In like manner, a wherein level and smooth E _m(I) multiple choices can be arranged, as get l ₁Norm, promptly

Adopt that some zone may exist ringing in the resulting total focus image of this mode.

The refocused image to obtain detailed process following:

Though in step S110, can obtain many images that focus under the different depth, utilize the depth map of resulting scene and the image of total focus can carry out scene depth of field control flexibly.

As shown in Figure 6, suppose to obtain focussing plane and be positioned at the big or small [s of being of the s place depth of field ₁, s ₂] the refocused image, its corresponding image distance setting is respectively v, [v ₁, v ₂], then have

$v=\frac{2v_1{v}_2}{{\mathrm{<figure-callout\; id="1"\; label="v"\; filenames="HDA0000076217190000012.png,HDA0000076217190000021.png"\; state="\{\{state\}\}">v</figure-callout>}}_1+{\mathrm{<figure-callout\; id="2"\; label="v"\; filenames="HDA0000076217190000012.png,HDA0000076217190000021.png"\; state="\{\{state\}\}">v</figure-callout>}}_2}$

Synthetic diaphragm diameter be

wherein c be the acceptable fuzzy diameter of maximum.Because the degree of depth of scene is known, then can calculate the inconsistent fuzzy core h ' of refocused image overall according to formula (2) and (3).

Therefore can be according to the image I of total focus image I and fuzzy core h ' acquisition refocused _b, formula is following:

I _b＝I*h′

Embodiment two

Fig. 7 is the structural representation according to the calculating imaging device of the embodiment of the invention two, specifies the composition of this calculating imaging device below in conjunction with Fig. 7.

This calculating imaging device comprises following each unit:

Image sensor cell, its surface adopts the on-plane surface mode to arrange, and the image information that gather through single exposure on the surface that extraction on-plane surface mode is arranged forms the image that focuses on different depth according to the image information of gathering.

Nonplanar arrangement mode has been adopted on the surface of image sensor cell.Generally speaking; The surface of sensor is the pixel of sensor; Obtain the imaging results of scene under different focusing surfaces for being implemented in single exposure collection; Be preferably the height level of branch specific quantity for the pixel arrangement mode of on-plane surface sensor surface, choose the number of the different focusing surfaces of the corresponding scene of other number of different height level, also can confirm according to the precision of the required scene depth resolution that exchanges for.Preferably; Imageing sensor on average is divided into a plurality of regional areas with behavior unit; The pixel that all comprises predefined all height level in each regional area; Corresponding delegation of each height level or multirow pixel, other pixel of differing heights level that sets in each regional area is thought the same position of corresponding three-dimensional scene.Fig. 2 is the example schematic of other sensor surface pixel arrangement of differing heights level in a kind of minute; The image sensor surface pixel is divided into regional area 1,2,3...; Each regional area comprises 3 row pixels, and pixel column 1.1,2.1,3.1... are on a height, and pixel column 1.2,2.2,3.2... are on a height; Pixel column 1.3,2.3,3.3... are on a height; It is three different depths of corresponding scene respectively, and think local pixel capable 1.1,1.2,1.3 corresponding the same point in the scene, same corresponding the same point of scene of local pixel capable 2.1,2.2,2.3 and local pixel capable 3.1,3.2,3.3.In this manner, when imaging, only need single exposure, can come the scene on the different focussing planes is carried out to picture according to other pixel of differing heights level, to obtain the image of the different depth of field.

Because it is higher directly to make the sensor cost with the arrangement of differing heights rank pixel; Preferably; Configuration image sensor unit as follows: place the surface of fibre faceplate as sensor through the pixel planes of being close to sensor; And fibre faceplate make on-plane surface as shown in Figure 2 on the end face of pixel planes, therefore fibre faceplate guiding light just be equivalent to and obtained nonplanar sensor to sensor.This fibre faceplate can be set to a series of neat compact arranged optical fiber, and preferred, the centre distance minimum of fiber can reach 3um, and numerical aperture can be accomplished more than or equal to 1.Preferably; Make the fibre diameter fibre faceplate identical with pixel size; The on-plane surface optical fiber arrangements precision of making on fibre faceplate surface can reach optical fiber one by one, then a certain on-plane surface fibre faceplate instance and utilize it and the synoptic diagram of sensors coupled as shown in Figure 3.The optical fiber cross section is on imaging plane among Fig. 3, and the guiding visible light is to the sensor pixel plane, and the optical fiber cross section is equivalent to original sensor plane.Among the figure label be 1 optical fiber cross section in one plane, a focussing plane a in its corresponding scene.Label be 2 and 3 optical fiber cross section on two other plane, b and c on corresponding two other focussing plane.

In when imaging, recover the different image sequences that focus on, all are labeled as 1 pixel and take out and form piece image and just can obtain a width of cloth and focus on the image on a plane among Fig. 3.In like manner, with all are labeled as 2 and 3 pixel and take out the composition piece image respectively and just can obtain the image that focuses on b and the c plane respectively among the figure.Can obtain the image sequence that focuses on the scene different depth of a series of low resolution thus.As preferably, the different depth rank number of on-plane surface fibre faceplate and the different arrangement mode of optical fiber also can be set, as shown in Figure 4.Equally, the pixel surface of on-plane surface arrangement also can be used mode shown in Figure 4.

The depth map generation unit, it is united from the method that focuses on estimating depth DFF with from defocusing the method for estimating depth DFD, and the said image that focuses on different depth is carried out estimation of Depth to obtain depth map.

In the depth map generation unit, at first utilize from the mode that focuses on estimating depth DFF and estimate ID, thereby draw the depth map of estimation as ID from the mode that defocuses estimating depth DFD.Also can be with the degree of depth of estimating from the mode that defocuses estimating depth DFD as ID, through drawing the depth map of estimation from the mode that focuses on estimating depth DFF from the mode that focuses on estimating depth DFF.Step S120 is identical among detailed process and the embodiment one, is not described further at this.

Image depth control The unit, it generates a width of cloth total focus image based on said image and the said depth map that focuses on different depth, focuses on the refocused image that ad-hoc location has particular depth of view based on depth map that is obtained and the generation of total focus image.

In the image depth control module, through with in the image sequence between respective pixel relatively the image information in the pixel of clear focusing take out and form a width of cloth total focus image, perhaps adopt the mode of deconvolution to obtain the image of clear total focus.Utilize the depth map of resulting scene and the image of total focus can carry out scene depth of field control flexibly, draw and focus on the refocused image that ad-hoc location has particular depth of view.Step S130 is identical among detailed process and the embodiment one, is not described further at this.

Each unit in the present embodiment can be used for realizing the preferred version of each step that embodiment one is corresponding equally, also is not described further at this.

The above is merely embodiments of the invention, and is in order to restriction the present invention, not all within spirit of the present invention and principle, any modification of being done, is equal to replacement, improvement etc., all should be included within protection scope of the present invention.

Those skilled in the art should be understood that; Above-mentioned each module of the present invention or each step can realize that they can concentrate on the single calculation element with the general calculation device, perhaps are distributed on the network that a plurality of calculation element forms; Alternatively; They can realize with the executable program code of calculation element, thereby, can they be stored in the memory storage and carry out by calculation element; Perhaps they are made into each integrated circuit modules respectively, perhaps a plurality of modules in them or step are made into the single integrated circuit module and realize.Like this, the present invention is not restricted to any specific hardware and software combination.

Though the embodiment that the present invention disclosed as above, the embodiment that described content just adopts for the ease of understanding the present invention is not in order to limit the present invention.Technician under any the present invention in the technical field; Under the prerequisite of spirit that does not break away from the present invention and disclosed and scope; Can do any modification and variation what implement in form and on the details; But scope of patent protection of the present invention still must be as the criterion with the scope that appending claims was defined.

Claims (12)

1. the calculating formation method based on imageing sensor is characterized in that, the daylighting surface of said imageing sensor adopts the on-plane surface mode to arrange, and said formation method may further comprise the steps:

The image information acquisition step, the image information that gather through single exposure on the daylighting surface that extraction on-plane surface mode is arranged forms the image that focuses on different depth according to said image information;

Depth map generates step, unites from the method for focusing estimating depth with from defocusing the method for estimating depth, and the said image that focuses on different depth is carried out estimation of Depth to obtain depth map;

The image depth controlled step generates a width of cloth total focus image based on said image and the said depth map that focuses on different depth, focuses on the refocused image that ad-hoc location has particular depth of view based on depth map that is obtained and the generation of total focus image.

2. formation method according to claim 1 is characterized in that, said daylighting surface adopts the on-plane surface arrangement mode to be: said daylighting surface is sensor pixel, and the pixel of said imageing sensor is arranged on the different height rank.

3. formation method according to claim 1; It is characterized in that; Said daylighting surface adopts the on-plane surface arrangement mode to be: fibre faceplate is set guides light to arrive sensor plane, said fibre faceplate is set to the on-plane surface by the distribution of differing heights rank back to the end in said sensor plane.

4. according to claim 2 or 3 described formation methods; It is characterized in that; Said daylighting surface is divided into a plurality of regional areas; Comprise all height level in each regional area, each height level is corresponding to a focal plane of scene, and the image information of on each height level, being gathered is formed the image that focuses on a degree of depth.

5. formation method according to claim 1; It is characterized in that; Generate in the step at said depth map; Through obtaining ID information from the mode that focuses on estimating depth, with said ID information as primary data, through drawing said depth map from the mode that defocuses estimating depth from the mode that defocuses estimating depth.

6. formation method according to claim 1; It is characterized in that; Generate in the step at said depth map; Through obtaining ID information from the mode that defocuses estimating depth, with said ID information as primary data, through drawing said depth map from the mode that focuses on estimating depth from the mode that focuses on estimating depth.

7. formation method according to claim 1; It is characterized in that; In said image depth controlled step; With in the image sequence between respective pixel relatively the image information in the pixel of clear focusing take out and form a width of cloth total focus image, perhaps adopt the mode of deconvolution to obtain the image of clear total focus.

8. formation method according to claim 7 is characterized in that, in said image depth controlled step, calculates the inconsistent fuzzy core of refocused image overall, draws the refocused image according to said fuzzy core and said total focus image.

9. a calculating imaging device is characterized in that, comprises with lower unit:

Image sensor cell, its daylighting surface adopts the on-plane surface mode to arrange, and the image information that gather through single exposure on the daylighting surface that extraction on-plane surface mode is arranged forms the image that focuses on different depth according to said image information;

The depth map generation unit, it is united from the method that focuses on estimating depth with from defocusing the method for estimating depth, and the said image that focuses on different depth is carried out estimation of Depth to obtain depth map;

The image depth control module, it generates a width of cloth total focus image based on said image and the said depth map that focuses on different depth, focuses on the refocused image that ad-hoc location has particular depth of view based on depth map that is obtained and the generation of total focus image.

10. imaging device according to claim 9 is characterized in that, said daylighting surface adopts the on-plane surface arrangement mode to be: said daylighting surface is sensor pixel, and the pixel of said imageing sensor is arranged on the different height rank.

11. imaging device according to claim 9; It is characterized in that; Said daylighting surface adopts the on-plane surface arrangement mode to be: fibre faceplate is set comes guiding fiber to pixel planes, said fibre faceplate is set to the on-plane surface by the distribution of differing heights rank back to the end in said pixel planes.

12. according to claim 10 or 11 described imaging devices; It is characterized in that; Said daylighting surface is divided into a plurality of regional areas; Comprise all height level in each regional area, each height level is formed the image that focuses on a degree of depth corresponding to a focal plane of scene by the image information of being gathered on each height level.

Images