CN100534378C - 3D positioning system and method in endoscopic main body in medical use - Google Patents

Abstract

A 3D locating system for the main body of medical endoscope is composed of a PC, a multi-channel image acquisition card, two CCD cameras installed to rigid metallic frame, a calibrating board of camera, an endoscope, and a platform attached to said endoscope and with 3 locating markers. Its method is also disclosed.

Description

Endoscopic main body in medical use 3 D positioning system and method

Technical field

The present invention relates to a kind of endoscopic main body in medical use 3 D positioning system and method, relate in particular to a kind of endoscopic main body in medical use 3 D positioning system and method based on computer binocular vision.

Background technology

Wicresoft and noinvasive surgical technic are one of topmost development trends of contemporary clinical medicine, and it can improve the operation quality greatly, alleviates patient's misery, shorten rehabilitation duration, become an international research focus.Traditional endoscope diagnosis art is the important means of minimally invasive surgery, and its typical equipment is a medical endoscope, and it has important role to the diagnosis and the treatment of focus.

Because the not visible deficiency that waits of the complexity of working environment and self shape, situations such as conduit winding, unexpected knot button loop can take place in the conventional endoscope conduit in intervention procedure, bring misery and danger to patient.Based on urgent clinical demand and good market prospect, the conventional endoscope system is improved and innovation research worldwide expansion with intelligent sensor technology, automatic technology and visualization technique.Wherein, novel endoscope's mirror shape sensory perceptual system is a research focus in this field.

If can show endoscope's mirror body, can prevent the generation that the intestinal mirror twines effectively in the intravital shape of people.Endoscope's mirror shape is rebuild research and is mainly contained following several mode both at home and abroad at present: endoscope's mirror body magnetic field space navigation system, ultrasound positioning system, optical fibre grating three-dimensional navigation system etc.

Endoscope's mirror body magnetic field space navigation system is that the people such as Bladen of Britain proposed the earliest in 1993 the earliest, system is based on Faradic electricity magnetic induction theorem, electromagnetism generation equipment is installed to produce the space magnetic field of known structure outside endoscope, by imbedding the magnetic field intensity signal of the magnetic field sensor acquisition in endoscope's mirror body, and, calculate the space coordinates position and the attitude angle of this point according to the known emission magnetic field space regularity of distribution.Imbed several discrete Magnetic Sensors in endoscopic catheters inside, the real-time sampling data of pick off reflect the variation in continuous magnetic field, can obtain the locus of each pick off as calculated, then these discrete points are fitted to successive curve, just can carry out three dimensional display on computers, realize the spatial form perception of endoscope's mirror body.This technology is relatively ripe through the research and development in surplus ten years, and OLYMPUS company has produced the magnetic field shape sensory perceptual system that is used for the endoscopic catheters clinical examination.Domestic University of Fuzhou is also carrying out certain research and inquirement aspect this.

The magnetic field localization method be a kind of safe, do not have radiation and real-time performance endoscope's mirror shape cognitive method preferably, but still have the following disadvantages: because the principle restriction, sensor-based system needs the magnetic field generator of several low frequencies to be used for producing magnetic field, and the relevant external equipment that system needs is many; In order to prevent electromagnetic interference, nonmetallic operating board and examinating couch also are necessary; In order to obtain the positional information of more discrete points on endoscope's mirror body, must imbed sensor coil in a plurality of positions, certainly will introduce more lead, to the certain influence of structural behaviour generation of mirror body; This technology is transformed endoscope's catheter interior structure, can not supportingly use with endoscopic system in existing the use, in case and coil or lead go wrong, safeguard comparatively difficulty.

Osaka, Japan university medical college has been developed a kind of human body ultrasonic probe inside navigation system based on existing ultrasonic endoscopic and external optical positioning system, is used for directly measuring and showing the locus and the movement locus of ultrasonic endoscopic probe.Place the sign sonac of an emission ultrasonic pulse at human body surface.In checking process, the ultrasonic endoscopic probe sends the ultrasound wave that is used for scanogram, and receives echo.By index transducer emission and the synchronous ultrasound wave of scanning ultrasonic signal, on the hyperacoustic echo of scanning that is added to after the reception of being popped one's head in.Observe the position of index transducer in scanogram, the direction and the distance of the relative index transducer that just can obtain popping one's head in.By obtain the locus of external index transducer based on the optical positioning system of binocular vision, try to achieve the position of the probe of ultrasonic endoscopic then according to the position relation of external tag pick off and body ultrasonic probe inside.

This localization method only needs to increase the external tag pick off and just can realize on the basis of existing ultrasonic endoscopic.Analyze after deliberation, there is following weak point in system: based on this system, can only determine the position of single ultrasonic endoscopic probe, real-time endoscope's mirror body global shape can not be provided; This navigation system can not be used with the fibre optic endoscopes and the fujinon electronic video endoscope of extensive use in the hospital based on ultrasonic endoscopic.

Shanghai University's medical robot and Computer Aided Surgery research group are from the needs of intelligent endoscope inspection, having made the endoscope's mirror shape sensory perceptual system based on the multiple spot fiber grating that can detect the endoscopic catheters spatial form in real time, mainly is to utilize the cylindrical curvature in tiny space to change the ultimate principle that is directly proportional with the echo centre wavelength of surface grating.Utilize many equidistant optical fiber that are carved with some gratings, optical fiber is sticked on the base material at a certain angle, set up a sensing network.Utilize this sensing network to gather the curvature data of each point, reconstruct the spatial form of whole endoscope mirror body then according to certain algorithm match, it is better to rebuild effect, real-time reconstruction and demonstration that can the implementation space curve.But owing to utilize the curvature information that obtains of fiber grating to be based on the relative quantity of the locating piece of endoscopic main body, therefore endoscope's mirror body 3D shape of rebuilding is the spatial form of endoscope's mirror body based on locating piece, and can't obtain the real space attitude of endoscope's mirror body.

Summary of the invention

Purpose of the present invention is to provide a kind of endoscopic main body in medical use 3 D positioning system and method, utilizes the method for computer binocular vision, by index point is set near endoscopic main body, tries to achieve the spatial attitude of endoscopic main body.Then can be applicable to obtain the spatial attitude of endoscope's mirror body reality in conjunction with endoscopic catheters 3D shape by the fiber-optic grating sensor reconstruction.

In order to reach the foregoing invention purpose, design of the present invention is:

The present invention by the method for the binocular vision that uses a computer, by near external endoscopic main body calibrating block index point being set, tries to achieve the spatial attitude of endoscopic main body on the basis that rebuilds based on the endoscopic catheters 3D shape of fiber-optic grating sensor.Then can be applicable to obtain the spatial attitude of endoscopic catheters reality in conjunction with endoscopic catheters 3D shape by the fiber-optic grating sensor reconstruction.

According to the foregoing invention design, the present invention adopts following technical proposals:

A kind of endoscopic main body in medical use 3 D positioning system, comprise a personal computer, a Multiplexing Image Grab Card, two ccd video cameras and an endoscopic main body, it is characterized in that described personal computer links to each other with two ccd video cameras through described Multiplexing Image Grab Card respectively, the attached platform of endoscope that witness marker point is arranged is installed on described endoscopic main body, witness marker point on the attached platform of endoscope is three the location LED index points that are equipped with of its upper surface, remain on the same plane, space, by obtaining the position of these three index points, calculate the spatial attitude of endoscopic main body, and the camera calibration plate that two ccd video cameras can photograph is set near endoscopic main body; Described personal computer is finished functions such as camera calibration, mark point recognition, the demonstration of endoscopic main body spatial attitude; Described Multiplexing Image Grab Card is gathered the image that ccd video camera transmits, and image transfer is arrived computer; Described ccd video camera is fixed on the metal rack, the image of the attached platform of acquisition camera scaling board and endoscope, and be delivered in the personal computer through Multiplexing Image Grab Card, calculate by personal computer and try to achieve the endoscopic main body spatial attitude.

In the above-mentioned camera calibration plate gridiron pattern pattern is arranged, this gridiron pattern pattern is made up of the square of identical size, is used for the parameter calibration of ccd video camera.

The terminal 3-D positioning method of a kind of medical endoscope adopts above-mentioned endoscopic main body in medical use 3 D positioning system to position operation, it is characterized in that its positioning action step:

1. calibrating parameters is set: before the operation beginning, at first calibrating parameters will be set, i.e. tessellated size and number in the input calibration plate;

2. ccd video camera is demarcated: gather the scaling board image of some different angles by Multiplexing Image Grab Card, personal computer is demarcated two ccd video cameras, obtains the camera parameters of ccd video camera;

3. the attached platform of endoscope is installed: the attached platform of endoscope is contained in the terminal position of endoscope, and opens the LED index point on the attached platform of endoscope;

4. the locus of calculation flag point: personal computer is discerned index point, in conjunction with described camera parameters, calculates the locus of index point, and finds the solution the transformation relation of camera coordinate system and index point coordinate system;

Above-mentioned step 2. in, described camera marking method is the calibration algorithm based on the Zhang Zhengyou two-step method.

The present invention makes it compared with prior art owing to adopted above technology, has following tangible advantage and characteristics:

1. the present invention does not need endoscopic main body and conduit are transformed, and all devices is external, and whole system is formed simple, and is easy to use.

2. the endoscopic main body in medical use 3-D positioning method that uses of the present invention, simple to operate, speed is fast, the precision height.This technology can real-time response be calculated the spatial attitude of external endoscopic main body.

3. system and method for the present invention can follow the endoscopic catheters shape sensory perceptual system based on the multiple spot fiber grating to combine, can in the hope of endoscopic catheters real space attitude.

The specific embodiment

A preferred embodiment of the present invention is described as follows:

This endoscopic main body in medical use 3 D positioning system, comprise a personal computer, a Multiplexing Image Grab Card, two ccd video cameras, a camera calibration plate, an endoscope, attached platform of endoscope, a stylobate are in the SHAPE DETECTION instrument of fiber grating.At first Multiplexing Image Grab Card is gathered the image of scaling board from ccd video camera, image transfer in personal computer; Personal computer can come out the parameter calibration of ccd video camera based on Zhang Zhengyou camera calibration algorithm; Multiplexing Image Grab Card is installed in the image of the attached platform of endoscope of endoscopic main body from the ccd video camera collection, image transfer in personal computer; Personal computer is obtained the locus of fixed point come in conjunction with the parameter of asking video camera.

The endoscopic main body 3-D positioning method of present embodiment adopts above-mentioned endoscopic main body 3 D positioning system operation, and its step is as follows:

1. calibrating parameters is set: before operation beginning, at first calibrating parameters will be set, the size of input calibration piece and calibrating parameters is set: before doing beginning, at first calibrating parameters will be set, i.e. tessellated size and number in the input calibration plate;

2. ccd video camera is demarcated: gather the scaling board image of some different angles by Multiplexing Image Grab Card, individual personal computer is demarcated two ccd video cameras, obtains the camera parameters of ccd video camera;

3. the attached platform of endoscope is installed: the attached platform of endoscope is contained on the body of endoscope, and the LED index point on the attached platform of unlatching endoscope;

4. the locus of calculation flag point: personal computer carries out discerning of index point, in conjunction with described camera parameters, calculates the locus of index point, and finds the solution the transformation relation of camera coordinate system and index point coordinate system;

Camera marking method in the present embodiment is a kind of camera calibration algorithm based on the Zhang Zhengyou two-step method.Its principle is as follows:

The point of a two-dimensional space can be used m=[u, v] ^TRepresent.And a three-dimensional point can be used M=[X, Y, Z] ^TWe use increases by one 1 spatial point vector $\tilde{m}={[u,v,1]}^T$ With $\tilde{M}={[X,Y,Z,1]}^T.$ Common pinhole camera modeling can be represented with formula 1:

$s\tilde{m}=A\left[\begin{array}{cc}R& t\end{array}\right]\tilde{M}---\left(1\right)$

Wherein s is any zoom factor, and (R t) is the video camera external parameter, is rotation and the translation transformation matrix that is tied to camera coordinate system from world coordinates, and A is an intrinsic parameters of the camera, can use

$A=\left[\begin{array}{ccc}\mathrm{\α}& \mathrm{\γ}& u_0\\ 0& \mathrm{\β}& v_0\\ 0& 0& 1\end{array}\right]$

Represent, wherein (u ₀, v ₀) be the coordinate of original point, α and β are the zoom factors of image u and v axle, γ is the deflection parameter of image coordinate axle.

At first the hypothetical world coordinate is the body surface of z=0, and r is decomposed into [r spin matrix ₁r ₂r ₃], formula 1 can be write as like this:

$s\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=A\left[\begin{array}{cccc}{r}_1& r_2& r_3& t\end{array}\right]\left[\begin{array}{c}X\\ Y\\ 0\\ 1\end{array}\right]=A\left[\begin{array}{ccc}{r}_1& r_2& t\end{array}\right]\left[\begin{array}{c}X\\ Y\\ 1\end{array}\right]---\left(2\right)$

Therefore, M is a point of body surface, because Z equals 0 always, so, M=[X, Y] T.Equally, $\tilde{M}={[X,Y,1]}^T.$ Relation between object point M and the picture point m is decided by matrix H:

$s\tilde{m}=H\tilde{M}$ Wherein $H=A\left[\begin{array}{ccc}{r}_1& r_2& t\end{array}\right]$

1 separates intrinsic parameters of the camera

Matrix H can be estimated by image.Decomposing H is H=[h ₁h ₂h ₃], draw from formula 2:

[h ₁?h ₂?h ₃]＝λA[r ₁?r ₂?t]

Wherein λ is any scalar.Because r1 and r2 quadrature, so:

$h_1^T{A}^{-T}{A}^{-1}{h}_2=0---\left(3\right)$

$h_1^T{A}^{-T}{A}^{-1}{h}_1=h_2^T{A}^{-T}{A}^{-1}{h}_2---\left(4\right)$

More than be two basic constraint functions of built-in variable

2 groups of solving an equation

Order

$B=A^{-T}{A}^{-1}=\left[\begin{array}{ccc}{B}_{11}& B_{12}& B_{13}\\ B_{12}& B_{22}& B_{23}\\ B_{13}& B_{23}& B_{33}\end{array}\right]=\left[\begin{array}{ccc}\frac{1}{{\mathrm{\α}}^2}& -\frac{\mathrm{\γ}}{{\mathrm{\α}}^2\mathrm{\β}}& \frac{v_0\mathrm{\γ}-u_0\mathrm{\β}}{{\mathrm{\α}}^2\mathrm{\β}}\\ -\frac{\mathrm{\γ}}{{\mathrm{\α}}^2\mathrm{\β}}& \frac{{\mathrm{\γ}}^2}{{\mathrm{\α}}^2{\mathrm{\β}}^2}+\frac{1}{{\mathrm{\β}}^2}& -\frac{\mathrm{\γ}(v_0\mathrm{\γ}-u_0\mathrm{\β})}{{\mathrm{\α}}^2{\mathrm{\β}}^2}-\frac{v_0}{{\mathrm{\β}}^2}\\ \frac{v_0\mathrm{\γ}-u_0\mathrm{\β}}{{\mathrm{\α}}^2\mathrm{\β}}& -\frac{\mathrm{\γ}(v_0\mathrm{\γ}-u_0\mathrm{\β})}{{\mathrm{\α}}^2{\mathrm{\β}}^2}-\frac{v_0}{{\mathrm{\β}}^2}& \frac{{(v_0\mathrm{\γ}-u_0\mathrm{\β})}^2}{{\mathrm{\α}}^2{\mathrm{\β}}^2}+\frac{v_0^2}{{\mathrm{\β}}^2}+1\end{array}\right]---\left(5\right)$

Because B is symmetric, definition:

b＝[B ₁₁，B ₁₂，B ₂₂，B ₁₃，B ₂₃，B ₃₃] ^T (6)

The i column vector of definition H is h _i=[h _I1, h _I2, h _I3] T, can obtain:

$h_i^T{\mathrm{Bh}}_j=V_{\mathrm{ij}}^{T}b---\left(7\right)$

V wherein _Ij=[h _I1h _J1, h _I1h _J2+ h _I2h _J1, h _I2h _J2, h _I3h _J1+ h _I1h _J3, h _I3h _J2+ h _I2h _J3, h _I3h _J3] ^T

Therefore, formula 3 and formula 4 can be written as:

$\left[\begin{array}{c}{V}_{12}^T\\ {(V_{11}-V_{12})}^T\end{array}\right]b=0---\left(8\right)$

If n object plane, then equation are provided

Vb＝0 (9)

Wherein V is the matrix of 2n*6, if n, then has only a unique b value more than or equal to 3.If n equals 2, we can be provided with deflection parameter γ is 0, solves an equation.

If the b value is estimated that all inner parameter matrix A also can obtain.Equally, the external parameter of each image also can be obtained.

r ₁＝λA ^-1h ₁

r ₂＝λA ^-1h ₂

r ₃＝r ₁×r ₂

t＝λA ^-1h ₃

3 least-squares estimation methods

Obtain after the parameter initial value, we come accurately these parameter values by the least-squares estimation estimation technique.By m point of n body surface, method of least square draws: $\underset{i=1}{\overset{n}{\mathrm{\Σ}}}\underset{j=1}{\overset{m}{\mathrm{\Σ}}}{\left|\right|m_{\mathrm{ij}}-\stackrel{)}{m}(A,R_i,t_i,M_j)\left|\right|}^2$

This is a non-linear least square problem, can find the solution with the Levenberg-Marquardt method.

The solution of 4 radial distortions

The radial distortion function is as follows:

$\stackrel{(}{x}=x+x[k_1(x^2+y^2)+k_2{(x^2+y^2)}^2]$

$\stackrel{(}{y}=y+y[k_1(x^2+y^2)+k_2{(x^2+y^2)}^2]$

K wherein ₁And k ₂It is coefficient of radial distortion.

Therefore

$\stackrel{(}{u}=u+(u-u_0)[k_1(x^2+y^2)+k_2{(x^2+y^2)}^2]$

$\stackrel{(}{v}=v+(v-v_0)[k_1(x^2+y^2)+k_2{(x^2+y^2)}^2]$

Can be write as

$\left[\begin{array}{cc}(u-u_0)(x^2+y^2)& (u-u_0){(x^2+y^2)}^2\\ (v-v_0)(x^2+y^2)& (v-v_0){(x^2+y^2)}^2\end{array}\right]\left[\begin{array}{c}{k}_1\\ k_2\end{array}\right]=\left[\begin{array}{c}\stackrel{(}{u}-u\\ \stackrel{(}{v}-v\end{array}\right]$

M point by n image obtains 2mn equation, and the solution of this linear equation is as follows:

k＝(D ^TD) ^-1D ^Td

If the radial distortion parameter is estimated, complete method of least square is as follows:

$\underset{i=1}{\overset{n}{\mathrm{\Σ}}}\underset{j=1}{\overset{m}{\mathrm{\Σ}}}{\left|\right|m_{\mathrm{ij}}-\stackrel{)}{m}(A,k_1,k_2{R}_i,t_i,M_j)\left|\right|}^2$

The algorithm of two-step method is as follows:

(1) scaling board is set before video camera, obtains several punctuate images;

(2) obtain all fixed points;

(3) estimate all inner parameter and external parameter by solving an equation;

(4) estimate the radial distortion parameter;

(5) method of least square is optimized all parameters.

Spatial point coordinate Calculation formula in the present embodiment is as follows:

The space coordinate of an index point on left camera review is (u _L, v _L), the coordinate on right camera review is (u _R, v _R), its space coordinates at left camera coordinate system is (x _L, y _L, z _L), be (x in the space coordinates of right camera coordinate system _R, y _R, z _R).According to formula 1, can in the hope of, this index point is to the straight line L1 parametric equation at the center of left video camera under left camera coordinate system

$\left\{\begin{array}{c}{x}_L=f_x^L\left(t\right)\\ y_L=f_y^L\left(t\right)\\ z_L=f_z^L\left(t\right)\end{array}\right.$

This index point is to the straight line L2 parametric equation at the center of right video camera under right camera coordinate system

$\left\{\begin{array}{c}{x}_R=f_x^R\left(t\right)\\ y_R=f_y^R\left(t\right)\\ z_R=f_z^R\left(t\right)\end{array}\right.$

By the outer parameter of the left and right cameras of trying to achieve, can obtain being tied to the transformation matrix of left camera coordinate system from right camera coordinates

$H={\left[\begin{array}{cc}{T}_L& R_L\\ 0& 1\end{array}\right]}^{\′}\left[\begin{array}{cc}{T}_R& R_R\\ 0& 1\end{array}\right]$

Like this, can be transforming under the left camera coordinate system at straight line L2 under the right camera coordinate system:

f ^L＝Hf ^R

By asking the intersection point of two rays, just can obtain the space coordinates of index point under left camera coordinate system come at last.

Generally, and since reasons of error, two rays and non-intersect, therefore, the locus that we as a token of put the mid point of the common vertical line section of two rays.

Pass through 3 index point { S in the present embodiment ₁, S ₂, S ₃To ask terminal coordinate to be tied to the transformation for mula of left camera coordinate system as follows for coordinate:

We are provided with index point S ₃To straight line S ₁S ₂Vertical point be fixed point coordinate system { the initial point O of A} _A, ray S ₁S ₂Be the x direction of principal axis, plane { S ₁S ₂S ₃It is the xoy plane.Fixed point coordinate system { A} and calibrating block coordinate system { the transformation relation H of B} (based on the frame of reference of optical fibre grating three-dimensional reconstruction) _B-〉ABe that we are provided with:

A＝H _B->AB

{ { translation matrix of C} is A} to left camera coordinate system from the index point coordinate system

T _A->C＝p(O _CO _A)

Spin matrix R _A-〉CCan pass through ray S ₁S ₂The x axle of rotation and left camera coordinate system overlaps, ray O _AS ₃The y axle of rotation and left camera coordinate system overlaps and obtains.

{ { transformation relation of C} is B} the calibrating block coordinate system to left camera coordinate system like this

$C=\left[\begin{array}{cc}{R}_{A->C}& T_{A->C}\\ 0& 1\end{array}\right]H_{B->A}B.$

Claims (2)

1. endoscopic main body in medical use 3 D positioning system, comprise a personal computer, a Multiplexing Image Grab Card, two ccd video cameras and an endoscopic main body, it is characterized in that described personal computer links to each other with two ccd video cameras through described Multiplexing Image Grab Card respectively, the attached platform of endoscope that witness marker point is arranged is installed on described endoscopic main body, witness marker point on the attached platform of endoscope is three the location LED index points that are equipped with of its upper surface, remain on the same plane, space, by obtaining the position of these three index points, calculate the attitude of endoscopic main body, and the camera calibration plate that two ccd video cameras can photograph is set near endoscopic main body; Described personal computer is finished camera calibration, mark point recognition, endoscopic main body spatial attitude Presentation Function; Described Multiplexing Image Grab Card is gathered the image that ccd video camera transmits, and image transfer is arrived computer; Described ccd video camera is fixed on the metal rack, the image of the attached platform of acquisition camera scaling board and endoscope, and be delivered in the personal computer through Multiplexing Image Grab Card, calculate by personal computer and try to achieve the endoscopic main body spatial attitude.

2. endoscopic main body in medical use 3 D positioning system according to claim 1 is characterized in that in the described camera calibration plate gridiron pattern pattern being arranged, and this gridiron pattern pattern is made up of the square of identical size, is used for the parameter calibration of ccd video camera.