WO1996034365A1 - Apparatus and method for recreating and manipulating a 3d object based on a 2d projection thereof - Google Patents

Abstract

A method for generating information regarding a 3D object from at least one 2D projection thereof. The method includes providing at least one 2D projection (40) of a 3D object, generating an array of numbers (50, 60) described by: aijk = vi'bik - vj''ajk (i,j,k = 1,2,3), where ajk and bjk are elements of matrices A and B respectively and vi' and vi'' are elements of vectors v' and v'' respectively, wherein the matrices (50) and vectors (60) together describe camera parameters of three views (102) of the 3D object and employing the array to generate information regarding the 3D object (70).

Description

APPARATUS AND METHOD FOR RECREATING AND MANIPULATING

A 3D OBJECT BASED ON A 2D PROJECTION THEREOF

FIELD OF THE INVENTION

The present invention relates to apparatus and methods for processing 2D proj ections of 3D obj ects and particularly to apparatus and methods for geometric analysis of a 2D proj ection image .

BACKGROUND OF THE INVENTION

Publications describing state of the art methods for processing 2D proj ections of 3D obj ects , and technologies relevant thereto , are described in references cited hereinbelow .

The disclosures of all publications mentioned in the specification and of the publications cited therein are hereby incorporated by reference .

SUMMARY OF THE INVENTION

The present invention seeks to provide image transfer apparatus and methods which are useful for generating a novel view of a 3D scene from first and second reference views thereof .

The present invention also seeks to provide 3D scene reconstruction methods and apparatus for generating a 3D representation of a 3D scene from first , second and third views thereof .

The present invention also seeks to provide improved apparatus and methods for processing 2D proj ections of 3D obj ects .

The present invention also seeks to provide methods for reconstruction of a 3D obj ect based on a trilinear tensor defined on three views of the 3D obj ect .

The present invention additionally seeks to provide methods for image transfer for a 3D obj ect based on a trilinear tensor defined on three views of the 3D obj ect .

The present invention also seeks to provide an image transfer method for generating a novel view of a 3D scene from f irst and second reference views thereof , the method including providing f irst and second reference views of a 3D scene , employing geometric information regarding the first reference view , second reference view and novel view , respectively , to generate a tril inear tensor representing the geometric relationship between the f irst , second and novel views and generating the novel view by computing a multiplicity of novel view locations each corresponding to dif ferent first and second corresponding locations in the first and second reference views respectively based on the f irst and second corresponding locations and the tril inear tensor .

The step of providing may , for example , comprise scanning in the f irst and second reference images . The step of employing may , for example , include the step of generating a set of first , second and third corresponding locations in the f irst reference view , second reference view and novel view , respectively .

There is thus provided , in accordance with a preferred embodiment of the present invention , an image transfer method for generating a novel view of a 3D scene from first and second reference views thereof , the method including providing first and second reference views of a 3D scene , employing geometric information regarding the first reference view, second reference view and novel view, respectively, to generate a trilinear tensor representing the geometric relationship between the first , second and novel views , and generating the novel view by computing a multipl icity of novel view locations each corresponding to different first and second corresponding locations in the first and second reference views respectively based on the f irst and second corresponding locations and the trilinear tensor .

Further in accordance with a preferred embodiment of the present invention , the step of providing includes scanning in the first and second reference images .

Still further in accordance with a preferred embodiment of the present invention , the step of employing includes the step of generating a set of first, second and third corresponding locations in the f irst reference view, second reference view and novel view , respectively .

There is also provided , in accordance with a preferred embodiment of the present invention , a 3D scene reconstruction method for generating a 3 D representation of a 3 D scene from first , second and third views thereof , the method including providing first , second and third views of a 3D scene , employing geometric information regarding the first , second and third views to generate a tril inear tensor representing the geometric relationship between the first , second and third views , and generating a 3D representation of the 3D scene from the trilinear tensor .

Further in accordance with a preferred embodiment of the present invention , the step of generating a 3D representation includes computing an epipolar geometric representation of the f irst and second views from the trilinear tensor , and generating the 3D representation from the epipolar geometric representation .

Also provided , in accordance with another preferred embodiment of the present invention , is image transfer apparatus for generating a novel view of a 3D scene from f irst and second reference views thereof , the apparatus including apparatus for providing first and second reference views of a 3D scene , a trilinear tensor generator operative to employ geometric information regarding the f irst reference view , second reference view and novel view , respectively , to generate a trilinear tensor representing the geometric relationship between the first , second and novel views , and a novel view generator operative to generate the novel view by computing a multiplicity of novel view locations each corresponding to dif ferent f irst and second corresponding locations in the first and second reference views respectively based on the f irst and second corresponding locations and the trilinear tensor .

Further provided , in accordance with another preferred embodiment of the present invention , is 3D scene reconstruction apparatus for generating a 3D representation of a 3D scene from first , second and third views thereof , the apparatus including appparatus for providing f irst , second and third views of a 3D scene , a trilinear tensor generator operative to employ geometric information regarding the f irst , second and third views to generate a trilinear tensor representing the geometric relationship between the f irst , second and third views , and a 3D scene representation generator operative to generate a 3D representation of the 3D scene from the tril inear tensor .

Also provided , in accordance with another preferred embodiment of the present invention , is a method for generating information regarding a 3D obj ect from at least one and preferably three 2D proj ections thereof , the method including providing at least one and preferably three 2D proj ections of a 3D obj ect , generating an array of numbers described by : α_ijk = v_i' b_jk - v_j" a_ik (i,j,k = 1,2,3) ,

where a_ij and b_jk are elements of matrices A and B respectively and v_i' and v_i" are elements of vectors v' and v" respectively, wherein the matrices and vectors together describe camera parameters of three views of the 3D object, and employing the array to generate information regarding the 3D object.

Also provided, in accordance with a preferred embodiment of the present invention, is a method for generating a new view of a 3D object from first and second existing views thereof having n corresponding points p_i and p_i' (i = 1 ... n) , the method including generating a tensor α_ijk , and plugging the values of points p_i and p_i' (i = 1 ... n) into trilinear forms, thereby to extract an x", y" value representing a location in the new view, and generating the new view on the basis of the result of the plugging in step.

Further provided, in accordance with a preferred embodiment of the present invention, is a method for reconstructing a 3D object from at least one and preferably three 2D projections thereof, the method including providing at least one and preferably three 2D projections of a 3D object, generating an array of numbers described by:

α_ijk = v_i' b_jk - v_j" a_ik (i,j,k = 1,2,3) ,

where a_ij and b_jk are elements of matrices A and B respectively and v_i' and v_i " are elements of vectors v' and v" respectively, wherein the matrices and vectors together describe camera parameters of three views of the 3D object, permuting the array into three homography matrices associated with three corresponding planes in 3D space, and employing the three homography matrices to reconstruct the 3D object.

Also provided, in accordance with a preferred embodiment of the present invention, is a visual recognition method including providing three perspective views of a 3D object between which a trilinear relationships exists, and employing the trilinear relationship between the views in order to perform visual recognition by alignment.

Further in accordance with a preferred embodiment of the present invention, the method also includes reprojecting the 3D object.

Still further in accordance with a preferred embodiment of the present invention, the information regarding the 3D object includes a reconstruction of the 3D object.

Additionally in accordance with a preferred embodiment of the present invention, the information regarding the 3D object includes at least one new view of the 3D object generated without reconstructing the 3D object.

Further in accordance with a preferred embodiment of the present invention, the at least one and preferably three 2D projections includes at least one aerial photograph.

Still further in accordance with a preferred embodiment of the present invention, the at least one and preferably three 2D projections includes at least one satellite photograph.

Further in accordance with a preferred embodiment of the present invention, the information regarding the 3D object comprises at least one coordinate of the 3D object.

Further in accordance with a preferred embodiment of the present invention, the 3D object includes an aerospace object.

Still further in accordance with a preferred embodiment of the present invention, the 3D object includes a large object such as a ship.

Further in accordance with a preferred embodiment of the present invention, the 3D object includes a nonexistent object.

Also provided, in accordance with a preferred embodiment of the present invention, is apparatus for generating information regarding a 3D object from at least one and preferably three 2D projections thereof, the apparatus including apparatus for providing at least one and preferably three 2D projections of a 3D object, an array generator operative to generate an .array of numbers described by:

α_ijk = v_i' b_jk - v_j" a_ik (i,j,k = 1,2,3) , where a_ij and b_jk are elements of matrices A and B respectively and v_i' and v_i" are elements of vectors v' and v" respectively, wherein the matrices and vectors together describe camera parameters of three views of the 3D object, and an array analyzer employing the array to generate information regarding the 3D object.

Also provided, in accordance with a preferred embodiment of the present invention, is apparatus for generating a new view of a 3D object from first and second existing views thereof having n corresponding points p_i and p_i' (i = 1 ... n) , the apparatus including apparatus for generating a tensor α_ijk , and apparatus for plugging the values of points p_i and p_i' (i = 1 ... n) into trilinear forms, thereby to extract an x", y" value representing a location in the new view, and apparatus for generating the new view on the basis of the result of the plugging in step.

Also provided, in accordance with a preferred embodiment of the present invention, is apparatus for reconstructing a 3D object from at least one and preferably three 2D projections thereof, the apparatus including apparatus for providing at least one and preferably three 2D projections of a 3D object, an array generator operative to generate an array of numbers described by:

α_ijk = v_i' b_jk - v_j" a_ik (i,j,k = 1,2,3) ,

where a_ij and b_jk are elements of matrices A and B respectively and v_i' and v_i" are elements of vectors v' and v" respectively, wherein the matrices and vectors together describe camera parameters of three views of the 3D object, an array permutator operative to permute the array into three homography matrices associated with three corresponding planes in 3D space, and a 3D object reconstructor operative to employ the three homography matrices to reconstruct the 3D object.

Further provided, in accordance with a preferred embodiment of the present invention, is visual recognition apparatus including apparatus for providing three perspective views of a 3D object between which a trilinear relationships exists, and apparatus for employing the trilinear relationship between the views in order to perform visual recognition by alignment. Further in accordance with a preferred embodiment of the present invention, at least one result of performing the above method is employed in order to perform at least one of the following applications: map making from aerial and satellite photographs and coordinate measurements in aerospace and shipyard assembly plants, coordinate measurements of industrial parts (CMM), automated optical based inspection of industrial parts, robotic cell alignment, robotic trajectory identification, 3D robotic feedback, 3D modelling of scenes, 3D modelling of objects, reverse engineering, and 3D digitizing.

Algebraic computation processes useful for recognition are described below with reference to Figs. 1 - 5.

BRIEF DESCRIPTION OF THE DRAWINGS AND APPENDICES

The present invention will be understood and appreciated from the following detailed description, taken in conjunction with the drawings in which:

Fig. 1 is an illustration of two graphs comparing the performance of an epipolar intersection method, shown in dotted line, with the performance of a trilinear functions method, shown in dashed line, in the presence of image noise;

Fig. 2 is a pictorial illustration of two model views of a three-dimensional scene as well as a third reprojected view thereof;

Fig. 3 is a pictorial illustration of a reprojection using a trilinear result;

Fig. 4 is a pictorial illustration of a reprojection using intersection of epipolar lines;

Fig. 5 is a pictorial illustration of a reprojection using a linear combination of views method;

Fig. 6 is a simplified functional block diagram of 3D scene reconstruction apparatus, constructed and operative in accordance with a preferred embodiment of the present invention, which is operative to generate a 3D representation of a 3D scene from at least three views thereof;

Fig. 7 is a simplified flowchart illustration of a preferred 3D scene reconstruction method, operative in accordance with a preferred embodiment of the present invention, which is useful in conjunction with the apparatus of Fig. 6;

Fig. 8 is a simplified functional block diagram of image transfer apparatus, constructed and operative in accordance with a preferred embodiment of the present invention, which is operative to generate a novel view of a 3D scene from at least two reference views thereof; and

Fig. 9 is a simplified flowchart illustration of a preferred image transfer method, operative in accordance with a preferred embodiment of the present invention, which is useful in conjunction with the apparatus of Fig. 8;

Fig. 10 is an illustration of a 3D reconstruction of a shoe which is also illustrated in Fig. 2;

Fig. 11 is a simplified block diagram illustration of a preferred method and apparatus for quality assurance of workpieces; and

Fig. 12 is a simplified block diagram illustration of a preferred method and apparatus for generating a digital terrain map.

Also attached herewith are appendices which aid in the understanding and appreciation of one preferred embodiment of the invention shown and described herein, namely:

Appendix A, which is a listing, in "C" language, of a preferred implementation of trilinear computation unit unit 50 and epipolar geometry generation unit 60 of Fig. 6; and

Appendix B, which is a listing of a preferred software implementation of 3D reconstruction apparatus constructed and operative in accordance with a preferred embodiment of the present invention.

Detailed Description of preferred Embodiments

A pin-hole camera, like 35mm still camera or Video recorder, produces a two-dimensional projection (2D) of the viewed three-dimensional (3D) world. The resulting image can be analyzed on a geometric and photometric level. The geometric level means the geometric relation between the locations of features (points, lines) in 3D and their respective location in the 2D image. The photometric level means the radiometric (reflectance properties of the surface, the spectral properties of the light sources illuminating the scene, etc.) relation between the scene and the luminosity (pixel grey values) in the image.

The 3D world is modeled as a cloud of points and so is the 2D world. In other words, we identify point features in the image that correspond to point features in 3D. There is a geometric relation between the two sets of points. When the camera moves in 3D we get more images of the same 3D world (i.e., of the same cloud of 3D points) and the question of interest is the geometrical relation between the corresponding set of 2D image points and the set of 3D points. In other words, if P_i denotes a set of 3D points, and

are three sets of 2D points (across three images) arranged such that points with same index i correspond to the same 3D point P_i , then the image sets alone reveal much information about the 3D set.

There are two questions of interest: (i) what is necessary to measure in the 2D images (how many points across how many images) in order to recover the location of the 3D points (reconstruction), (ii) given the image correspondences can we synthesis new views (the problem of image transfer) without explicitly reconstructing the object. The patent deals with both questions.

The general material of 3D-from-2D is an active area in Computer Vision fstructure-from- motion and stereopsis), and is the sole engine in the industry of photogrammetry (map making from areal and satellite photographs, coordinate measurements in aerospace and shipyard assembly plants, etc.).

The relationship between 3D points and 2D points can be represented as a 3 x 4 transformation matrix:

where p = (x, y, 1 )^T represents the 2D point in homogeneous coordinates (i.e., the image plane is the 2D projective plane), where x, y are the horizontal and vertical components of the point's location in the image (with respect to some arbitrary origin - say the geometric center of the image plane); P is represented by a 4-vector, which means that we view the object space as embedded in the three-dimensional projective space. The matrix M is 3 x 4 and represents the camera parameters (for example, the vector Q defined by MQ=0 is the camera's center of projection— the point where all the optical rays converge to). The sign denotes equality up to scale.

Because we are embedding the image and the object into projective spaces (which means that the world can undergo rigid motion and be stretched and sheared) we can transform the object space such that the camera transformation associated with the first view is [I, 0], where I is the identity matrix. For three views we have:

where P = (x, y, l, k)^T , and the image points are p = (x, y, 1)^T, p' = (x', y', 1)^T and p" = (x", y", 1)^T . Note: what we measure in the images are the locations of p,p', p", the location of P (i.e., the value of k) is not known. A and B are some 3 x 3 matrices (not independent of each other) and v', v" are some 3-vectors together describing the camera parameters of the three views. Note: the camera parameters A, B, v', v" are unknown. In other words, given three views with corresponding points there exists

a set of numbers k_i and parameters A, B, v', v" such that the equations above hold for all i.

I. TRILINEAR TENSOR

The array of 27 numbers described by:

a_ijk = v'_ib_jk - v"_j a_ik. i, j, k = 1, 2, 3

where α_rs, b_τs are the elements of A, B, and v_r' , v_r" are the elements of v', v", is referred to as the "trilinear tensor" . These numbers are invariant to the particular projective representation of the 3D and 2D worlds, i.e., they are intrinsic to the three views (this is one of the general properties of tensors that they do not depend on the choice of basis for representation).

NOTE: the array α_ijk provides the foundation for numerous applications. A preferred method for producing the array from image measurements p, p' , p" will be described next. A . recovering α_ijk from image measurements

A corresponding triplet p, p', p" satisfies a number of trilinear relationships. First notation: We identify vectors and matrices by fixing some of the indices while varying others. For example, α_ijk is a set of scalars, a_ij. is a set of 9 vectors (k varies while i,j remain fixed): α_i.. is a set of 3 matrices (α₁.., α₂.. and α₃..), and so forth.

For a given order (i.e., we do not permute the views), there are 9 trilinear functions, that come in sets of 4 linearly independent forms. The coefficients of these forms are elements of the array α_{ij k} For example, the following four forms are linearly independent invariant functions of three views:

The remaining functions are described in equations 5-11 presented herein.

Since every corresponding triplet p, p', p" contributes four linearly independent equations, then seven corresponding points across the three views uniquely determine (up to scale) the tensor α_ijk .

NOTE 1 : what this means is that if we identify 7 triplets of corresponding points across the three views— say the object is a human face and we track the corner of the nose across three views: its location in the first view will be denoted by (x, y), the location of the nose in the second view will be denoted by (x', y') and the location of the nose in the third view is (x", y")— then each such triplet provides 4 linearly independent equations for the 27 parameters we seek to recover. Thus, we obtain 28 equations which is more than sufficient to solve for α_ijk.

NOTE 2: There are many ways to solve for linear systems, especially when more equations than unknowns are provided (i.e., when more than 7 corresponding triplets are given), for example least-squares technique, and robust methods for removing outliers within least- squares techniques. II . RAN K 4 CO NSTRA I NT : TENSO RIAL CON NECTIONS ACROSS M O RE THAN 3 V I EWS

Given a set of m > 3 views, the various trilinear tensors of subsets of 3 views out of entire set of views are not independent, and in fact are linearly related to each other. This relation, described below, we call "the rank 4 constraint" .

We arrange a tensor of three views a tensor in a 9 x 3 matrix denoted by G, where column j = 1, 2, 3 contains the elements α_{1,j,1 ,} α_{1,j,2 ,} ... ,α _3,j,3.

Let the the arbitrary set of m views be arranged as ψ₁, ψ₂, ψ_j, j = 3, ..., m. Consider the m— 2 tensors of views < ψ₁, ψ₂, ψ_j > and let the correspondence G matrices be denoted by G_j . Then the concatenation of the G_j matrices into a 9 x 3(m— 2) matrix has rank 4 regardless of m (in general, if the G₂ matrices were independent, the rank of the concatenation would have been the smaller dimension, i.e., 9 when m≥ 5).

The rank-4-constraint implies that the four largest principle components of the concatenation matrix represents the geometry between views ψ ₁ and ψ₂, in a way that is statically optimal.

A. Application 1: Image Transfer

Say we have two views with corresponding points p_{i ,} p'_i, i = 1, ..., n. We wish to create a new view of the same object. Given the tensor α_ijk we plug into the trilinear forms the values of p_i, p'_i and extract the value of x", y", regardless of the manner in which α_ijk is obtained.

The tensor a_{ij k} can be obtained, for example, if we know the locations of p" for 7 points, e.g.. i = 1, ..., 7. Then for every 8th point we can recover p" from the pair p, p' and the tensor.

B. Application 2: Reconstruction

A tensor of three views, or the four largest principle components of the concatenation matrix of m > 3 views (see Section II) can be used to recover the epipolar geometry between two views, and from there to be used for reconstructing a 3D model of the scene.

Let E_j = α._j. be three 3 x 3 matrices, i.e., E₃ = α_1,j,1, α_1,j,2, ..., α_3,j,3. Then this particular permutation of the array of 27 numbers into three matrices yields the following result:

The three matrices E₁ , E₂, E₃ are projective transformations (homography matrices) associated with some three planes in 3D space What this means is that there exists three planes π_j in space, such that if P is coplanar with π ₁ (for example) and p, p' are the image locations in the first two views, then E_1p

p'. Similarly for j = 2, 3.

The "fundamental matrix" F (or "coplanarity constraint" as used in photogrammetrical literature) between the first two views satisfies:

a relation that provides a linear system of equations for solving for F.

Similarly, the four largest principle components of the concatenation matrix (Section II) for m > 3 views can be used to solve for F, as follows. Let A_k, k = 1, ..., 4 be the four largest principle components of the concatenation matrix, each arranged in a 3 x 3 matrix (row-wise). Then,

which provides a statically optimal way to linearly recover F from m > 3 views.

The "epipolar" point v' can be recovered, either by the relation F^τv' = 0, or directly from the tensor as follows:

The cross-product of corresponding columns of each two of the matrices out of the three matrices E₁ , E₂, E₃ provides an epipolar line, i.e., a line that is coincident with v'. Similarly, if we denote by α₁ , α₂, α₃ the columns of one E matrix, and by b₁, b₂, b₃ the columns of another E matrix, then

α_i x b_j— b_i x α_j is an epipolar line for i j. Taken together the tensor gives rise to 18 epipolar lines whose

intersection defines the epipole v'. The point of intersection could be found by, for example, a least-squares method.

Similarly, the epipolar point v' can be recovered from A₁ , ..., A₄, the four largest principle components concatenation matrix, as follows, let α₁ , α₂, α₃ be the the columns of matrix A₁ , and by b₁ , b₂, b₃ the columns of A₂, then α_i x b_j— b_i x α_j is an epipolnr line for i j , and

α_i X b_i

provides another set of three epipolar lines. By selecting in this way two out of the four matrices A₁ , ..., A₄ we get 24 epipolar lines which provide a redundant set of equations for solving for v'.

The projective model of the 3D scene follows from F and v', for example: the 3 x 4 matrix [[v']F, v'] is a camera transformation from 3D coordinates to the image coordinates of the second view. In other words, for every matching pair p, p' of points in first and second view, respectively, we have:

thus the coordinate vector [x, y, l, k) provides a 3D (projective) model of scene.

NOTE 1: the permutation α_i.. yields also three homography matrices W₁, W ₂, W₃ of three other planes. The homography matrices are from first two third view, i.e., W _1p

p" for p,p" coming from P coplanar with the plane associated with W₁. Thus, analogous to the use of the matrices E₁, E₂, E₃, we can recover the fundamental matrix F" and epipolar point v" associated with the epipolar geometry between the first and third view.

ACHIEVING IMAGE CORRESPONDENCES ACROSS 3 VI EWS

One of the important steps in manipulating 3D scenes from 2D imagery is the step of obtaining a reliable set (as dense as possible) of matching points across the views at hand. The current state-of-the-art use correlation techniques across the image intensities of two views together with geometric information, such as the epipolar geometry.

With the tensor one can extend the correspondence method to take advantage of three views together and without assuming camera calibration. Given an initial set of corresponding points across three views one recovers the trilinear tensor. Then, the trilinear tensor provides a constraint for using correlation methods to accurately locate the matching points. This can be done as follows.

Let E₁ , E₂, E₃ denote the arrangement α._j. of the tensor and W₁ , W₂, W₃ denote the arrangement α_i.. of the tensor. We first align all three images such that the displacements between matching points become only along epipolar lines. This is done as follows.

Let E =∑_j ρ_jE_j where the coefficients ρ_j are the solution of the linear least-squares condition E_pk = p'_k for all initial matching points p_k, p'_k in first and second image, respectively. Similarly, let W = ∑_i μ_iW_i where the coefficients μ_i are the solution of the linear least- squares condition Wp_k

for all initial matching points p_k,

in first and third image, respectively. The process of "warping" image two with the transformation E^{- 1} and image three with the transformation W^-1 produces two new images denote by 2' and 3', such that the matching points across images 1, 2' are along epipolar lines and the matching points across images 1 , 3' are along epipolar lines.

The epipolar lines can be derived from the tensor as presented herein (Section II. B in detailed description of preferred embodiment). Let w' and w" be the direction vectors of the epipolar lines between 1 , 2' and 1, 3', respectively. Hence, if p = (x, y), p' = (x', y'), p" = (x", y") are matching points in images 1 , 2', 3' respectively, then p' = p+αw' and p" = p+βw" for some coefficients α, β. Note that w', w", a, β are quantities that change with the location

(x, y ) .

The trilinear equations, (3)-(6) presented herein, provide a geometric constraint on the coefficients α, β, i.e., for each arbitrary value of α, the value of β is then uniquely determined, and vice versa. In order to recover both α, β we use the image intensities of 1. 2', 3' as follows. The coefficients α, β satisfy the "constant brightness equation" :

where ( I_x, I_y) is the gradient vector at location p = (x, y) in the first image, and is

the temporal derivative between images 1, 2', and is the temporal derivative between

images 1, 3' (all derivatives are discrete approximations). From the trilinear equations, (3)- (6) presented herein, we can obtain f(α, β) = 0 where the coefficients of the function f() are elements of the tensor. Thus one obtains a least-squares solution for a and β that satisfies both geometric an photometric constraints.

The entire procedure can be done hierarchically along a Laplacian Pyramid as described in "J .R. Bergen and R. Hingorani, Hierarchical motion-based frame rate conversion" ; Technical report. David Sarnoff Research Center, 1990.

Reference is now made to Fig. 6 which is a simplified functional block diagram of 3D scene reconstruction apparatus, constructed and operative in accordance with a preferred embodiment of the present invention, which is operative to generate a 3D representation of a 3D scene from at least three views thereof.

The apparatus of Fig. 6 includes apparatus for providing at least 3 digital images of a 3D scene or object from at least 3 respective viewpoints, such as a CCD camera 10 which operates from 3 different viewpoints, or such as a film camera 20, which may be airborne, associated with a scanner 30 (such as a Zeiss PS-1 which is operative to digitize the images generated by film camera 20.

The at least 3 digital views are fed to a matching point finder 40 which is operative to identify at least 7 and preferably a multiplicity of triplets of matching points from the 3 digital views. "Matching points" from different views are points or locations which correspond to a single location in the real, 3D world. These points may, for example, be identified manually. For aerial photographs, commercially available matching point software may be employed such as the Match-T package marketed by Inpho in Stuttgart, which performs a correspondence function.

Conventional methods for finding matching points are described in Wang, H. and Brady, J. M, "Corner detection: some new results", IEEE colloguium Digest of Systems Aspects of Machine Perception and vision", Londone 1992, pp. 1.1 - 1.4.

A trilinear tensor computation unit 50 receives the matching point triplets and computes a trilinear tensor representing the geometric relationship between the three views.

The trilinear tensor is then employed to generate a 3D representation of a 3D scene. According to one embodiment of the present invention, the trilinear tensor is employed by an epipolar geometry generation unit 60 to compute an epipolar geometric representation of two of the three views. Subsequently, 3D representation generation unit 70 generates the 3D representation of the scene or object from the epipolar geometric representation output of unit 60, as described in more detail in:

a. Faugeras, O. D. "What can be seen in 3 dimensions with an uncalibrated stereo rig?", Proceedings of the European Conference on Computer Vision, pages 563 - 578, Santa Margherita Ligure, Italy, June 1992.

b. Hartley, R. et al, "Stereo from uncalibrated cameras." Proceedings IEEE Conf. on Computer Vision and Pattern Recognition, 761 - 764, Champaign, IL, June 1992.

c. Shashua, A. "Projective structure from uncalibrated images: structure from motion and recognition. IEEE Transactions on PAMI, 16(8) : 778 - 790, 1994.

A preferred implementation of units 50 and 60 is described, in "C" computer language form, in Appendix A. A preferred implementation of unit 50 is described from page 1 of Appendix A until toward the end of page 4 thereof. A preferred implementation of unit 60 is described from the end of page 4 of Appendix A until the middle of page 8. Subroutines and statistical procedures which are useful in understanding the above material appear from page 8 of Appendix A onward.

The 3D representation, including 3D information representating at least a portion or an aspect of the 3D scene or object, may be employed by a laser computer printer to generate a new view of the object or scene. Alternatively, conventional CAD (computer aided design) software in conjunction with a conventional plotter may be employed to generate a new view of the object or scene. The CAD software may also be operative to compare 2 CAD files in quality assurance applications.

Reference is now made of Fig. 7 which is a simplified flowchart illustration of a preferred 3D scene reconstruction method, operative in accordance with a preferred embodiment of the present invention, which is useful in conjunction with the apparatus of Fig. 6. Fig. 7 is generally self-explanatory. The 3 views of the scene or object can be digital, if they are accessed from a digital archive or generated, e.g. by a digital camera. If they are not digital, they are scanned or otherwise digitized. Any suitable conventional or other formats may be employed . For example , the Sil icon Graphics ' s Inventor format may initially be employed for the 3D representation . The Inventor format may be converted into Postscript in order to print a new view of the 3D representation .

As shown , the 3D representation of the scene is useful in performing a wide range of activities such as 3D measurements of the scene or object , generation , e. g. as a printout , of a new view of the scene or obj ect , and quality assurance comparisons in which the generated 3D representation of the obj ect or scene is compared to a desired obj ect or scene or a desired 3D representation thereof , using conventional methods .

Reference is now made to Fig . 8 which is a simplified functional block diagram of image transfer apparatus , constructed and operative in accordance with a preferred embodiment of the present invention , which is operative to generate a novel view of a 3D scene from at least two reference views thereof . The apparatus of Fig . 8 is similar to the apparatus of Fig . 6 . However , in Fig . 8 , a novel view of a 3D scene is directly generated from only at least two reference views thereof , preferably without generating a 3D representation intermediate .

In Fig. 8 , geometric information regarding the two reference views and the desired novel view is employed to generate a trilinear tensor representing the geometric relationship between the three views . Typically, at least 7 triplets of locations in the novel view may be identified , e . g . manually , which correspond to 7 locations respectively in each of the at least two reference views . Typically, at least some infromation regarding the novel view is available . For example , if it is desired to update a GIS (geographic information system) year-old view , based on at least two new reference views of the same area . It is typically possible to identify at least 7 locations in the year- old view which correspond to 7 locations in the two reference views and which can be assumed will still exist in the soon-to-be-generated current version of the year-old view .

The novel view is typical ly generated by computing a multiplicity of novel view locations each corresponding to different first and second corresponding locations in said first and second reference views respectively based on said first and second corresponding locations and said trilinear tensor. For example, the matching point finder 140 may generate a multiplicity of pairs of matching points from the two reference views, say 1000 such pairs. For the first 7 pairs, a user may manually indicate 7 matching points in the novel view.

Once the trilinear tensor is generated by trilinear tensor computation unit 150, the coordinates of Matching Point Pairs 8 - 1000 may be plugged into the trilinear tensor, as shown in Fig. 9, in order to generate coordinates of matching points 8 - 1000 in the novel view.

The novel view thus generated may, for example, be compared to the same view as seen a year before, in order to identify differences in the scene which took place in the course of the year.

Reference is now made to Fig. 9 which is a simplified flowchart illustration of a preferred image transfer method, operative in accordance with a preferred embodiment of the present invention, which is useful in conjunction with the apparatus of Fig. 8. Fig. 9 is generally self-explanatory.

In any of the embodiments described herein, if more than 3 views are provided, "intermediate" tensors may be computed for each 3 views. Then, a representative tensor may be computed based on the relationships between these "intermediate" tensors.

Fig. 10 is an illustration of a 3D reconstruction of a shoe which is also illustrated in Fig. 2. Fig. 10 was generated oy finding matching points and reconstructing their 3D locations. Next the coordinates are processed by CAD software to generate the surface shown in Fig. 10.

Fig. 11 is a simplified block diagram illustration of a preferred method and apparatus for quality assurance of workpieces. Fig. 11 includes an array 200 of 3 CCD cameras 210 which are aimed at a single location, so as to yield three perspectives of that location. The CCD cameras are attached to a robot arm 212 and therefore can move relative to a workpiece 214 arriving along a conveyor belt 220 in accordance with suitable instructions from a controller 224.

When a workpiece enters the field of view of the CCD cameras 210, the conveyor belt 220 typically pauses to allow substantially the entirety of the surface area of the workpiece to be imaged by the cameras 210. The controller 224, via robot arm 212, moves the camera array 200 around the object such that, typically, almost its entire surface area is imaged. For example, the camera array 200 may be moved through 10 different positions around the object, and at each position, each of the 3 CCD cameras images the workpiece. The number of positions employed depends on the complexity of the workpiece.

This process yields a plurality of image triplets, each image triplet including three digital images of the same portion of the workpiece, from 3 respective perspectives. For each image triplet, the corresponding position of the array 200 and of each of the cameras 210 may be computed, based on the robot arm's location, which is known, and using hand-eye calibration.

Each image triplet is processed by units 240, 250, 260, 270 and 290 which may be similar to units 40, 50, 60, 70 and 90, respectively, of Fig. 6.

The CAD model information generated by CAD S/W 290 from each image triplet (each probe location) is stored in a suitable memory 300. A computation unit 310 is operative to integrate the multiplicity of probe locations corresponding to the multiplicity of positions of CCD camera array 200, into a single coordinate system. The necessary coordinate transformations are computed by inverting the transformations which define the CCD camera array's motion.

A computational unit 320 compares the output of unit 310 to a reference CAD model and computes differences therebetween. These differences are compared, in a computational unit 330, to accepted tolerances.

The apparatus of Fig. 11 is also suitable for 3D digitization applications for reverse engineering or CAD (computer aided design purposes) .

Fig. 12 is a simplified block diagram illustration of a preferred method and apparatus for generating a digital terrain map, e.g. in order to update municipal maps, to detect illegal structures, to serve a car navigation system, or even to map a microscopic object such as an integrated circuit.

An airborne CCD camera 334 is flown over a scene for which it is desired to generate a digital terrain map. The camera 334 generates 3 digital images of the scene from 3 respective perspectives. The 3 images are processed by units 340, 350, 360 and 370 which may be similar to units 40, 50, 60 and 70 of Fig. 6.

A surface interpolation procedure is performed by a surface interpolation unit 380, on the output of 3D representation generation unit 370. A suitable surface interpolation method is described in Grimson, W. E. L, "A computational theory of visual surface interpolation", Proc. of the Royal Soc. of London

B, 298, pp. 395 - 427, 1982.

Finally, perspective distortions are removed from the output of surface interpolation unit 380 by an orthophoto generation unit 390, resulting in an orthophoto (planimetric map) . A suitable method of operation for unit 390 is described in Slama,

C. C., "Manual of Photogrammetry", American Society of Photogrammetry, 1980.

The present invention is also useful for visualization of a 3D object, particularly for entertainment, animation or educational applications. A camera array such as array 200 of Fig. 11 circles around an object to be visualized and images substantially the entire surface area which it is desired to display to a user. For example, the camera array may image the object from each of 200 positions surrounding the object.

Synthetic images may then be generated for positions other than the 200 above-mentioned positions. A desired position may be indicated by a user, e.g. by means of a joystick. The apparatus of the present invention may be used to generate a synthetic image for that position. Conventional driving simulation games employ synthetic backgrounds, however, the present invention may be employed to provide a driving simulation game with a real background. To do this, an array of at least 3, and preferably 5 - 10 cameras is moved within a desired scene such that substantially the entirety of the scenes can be captured by the camera array from at least 3 and preferably more different perspectives. For example, the scene may be captured from each of approximately 1000 positions of the camera array. New views are then generated, in accordance with the present invention, in order to accomodate a user's need for new views as indicated, e.g. by a joystick.

Algebraic functions useful for recognition are now described, based on an article entitled "Algebraic functions for recognition", to be published in IEEE, Transactions on PAMI.

Algebraic Functions For Recognition

I. INTRODUCTION

We establish a general result about algebraic connections across three perspective views of a 3D scene and demonstrate its application to visual recognition via alignment. We show- that , in general, any three perspective views of a scene satisfy a pair of trilinear functions of image coordinates. In the limiting case, when all three views are orthographic, these functions become linear and reduce to the form discovered by [38]. Using the trilinear result one can manipulate views of an object (such as generate novel views from two model views) wit hout recovering scene structure (metric or non-metric) , camera transformation, or even the epipolar geometry. Moreover, the trilinear functions can be recovered by linear methods with a minimal configuration of seven points. The latter is shown to be new lower bound on the minimal configuration that is required for a general linear solution to the problem of re-projecting a 3D scene onto an arbitrary novel view given corresponding points across two reference views. Previous solutions rely on recovering the epipolar geometry which, in turn, requires a minimal configuration of eight points for a linear solution.

The central results are contained in Theorems 1, 2 and 3. The first theorem states that the variety of views ψ of a fixed 3D object obtained by an uncalibrated pin-hole camera satisfy a relation of the sort F(ψ, ψ₁, ψ₂) = 0, where ψ₁, ψ₂ are two arbitrary views of the object, and F has a special trilinear form. The coefficients of F can be recovered linearly without establishing first the epipolar geometry, 3D structure of the object, or camera motion. The auxiliary Lemmas required for the proof of Theorem 1 may be of interest on their own as theyestablish certain regularities across projective transformations of the plane and introduce new view invariants (Lemma 4).

Theorem 2 addresses the problem of recovering the coefficients of the trilinear functions in the most economical way. It is shown that among all possible trilinear functions across three views, there exists at most four linearly independent such functions. As a consequence, the coefficients of these functions can be recovered linearly from seven corresponding points across three views.

Theorem 3 is an obvious corollary of Theorem 1 but contains a significant practical aspect. It is shown that if the views ψ₁ , ψ₂ are obtained by parallel projection, then F reduces to a special bilinear form— or. equivalently. that any perspective view ψ can be obtained by a rat ional linear function of two orthographic views. The reduction to a bilinear form implies that simpler recognition schemes are possible if the two reference views (model views) stored in memory are orthographic.

These results may have several applications (discussed in Section VI), but the one emphasized throughout this manuscript is for the task of recognition of 3D objects via alignment. The alignment approach for recognition ([37, 16], and references therein) is based on the result that the equivalence class of views of an object (ignoring self occlusions) undergoing 3D rigid, affine or projective transformations can be captured by storing a 3D model of the object, or simply by storing at least two arbitrary "model" views of the object— assuming that the correspondence problem between the model views can somehow be solved (cf.

[27. 5. 33]). During recognition a small number of corresponding points between the novel input view and the model views of a particular candidate object are sufficient to "re-project" the model onto the novel viewing position. Recognition is achieved if the re-projected image is successfully matched against the input image. We refer to the problem of predicting a novel view from a set of model views using a limited number of corresponding points, as the problem of re-projection.

The problem of re-projection can in principal be dealt with via 3D reconstruction of shape and camera motion. This includes classical structure from motion methods for recovering rigid camera motion parameters and metric shape [36, 18, 35, 14, 15], and more recent methods for recovering non-metric structure, i.e., assuming the objects undergo 3D affine or projective transformations, or equivalently, that the cameras are uncalibrated [17, 25, 39. 10. 13. 30]. The classic approaches for perspective views are known to be unstable under errors in image measurements, narrow field of view, and internal camera calibration [3. 9, 12] . and therefore, are unlikely to be of practical use for purposes of re-projection. The non-metric approaches, as a general concept, have not been fully tested on real images, but the methods proposed so far rely on recovering first the epipolar geometry— a process that is also known to be unstable in the presence of noise.

It is also known that the epipolar geometry alone is sufficient to achieve re-projection by means of intersecting epipolar lines [24, 6, 8, 26, 23, 11] using at least eight corresponding points across the three views. This, however, is possible only if the centers of the three cameras are non-collinear— which can lead to numerical instability unless the centers are far from collinear— and any object point on the tri-focal plane cannot be re-projected as well . Furthermore, as with the non-metric reconstruction methods, obtaining the epipolar geometry is at best a sensitive process even when dozens of corresponding points are used and with the state of the art methods (see Section V for more details and comparative analysis with simulated and real images).

For purposes of stability, therefore, it is worthwhile exploring more direct tools for achieving re-projection. For instance, instead of reconstruction of shape and invariants we would like to establish a direct connection between views expressed as a functions of image coordinates alone— which we call "algebraic functions of views". Such a result was established in the orthographic case by [38]. There it was shown that any three orthographic views of an object satisfy a linear function of the corresponding image coordinates— this we will show here is simply a limiting case of larger set of algebraic functions, that in general have a trilinear form. With these functions one can manipulate views of an object, such as create new views, without the need to recover shape or camera geometry as an intermediate step — all what is needed is to appropriately combine the image coordinates of two reference views. Also, with these functions, the epipolar geometries are intertwined, leading not only to absence of singularities, and a lower bound on the minimal configuration of points, but as we shall see in the experimental section to more accurate performance in the presence of errors in image measurements. Part of this work (Theorem 1 only) was presented in concise form in [31] .

II. NOTATIONS

We consider object space to be the three-dimensional projective space p³. and image space to be the two-dimensional projective space P². Let Φ C P³ be a set of points standing for a 3D object, and let ψ_i C P² denote views (arbitrary), indexed by i. of Φ. Given two cameras with centers located at O, O'∈ p³, respectively, the epipoles are defined to be at the intersection of the line

with both image planes. Because the image plane is finite, we can assign, without loss of generality, the value 1 as the third homogeneous coordinate to every observed image point. That is, if (x, y) are the observed image coordinates of some point (with respect to some arbitrary origin— say the geometric center of the image), then p = ( x. y. 1 ) denotes the homogeneous coordinates of the image plane. Note that this convention ignores special views in which a point in Φ is at infinity in those views— these singular cases are not modeled here. Since we will be working with at most three v iews at a time, we denote the relevant epipoles as follows: let v∈ ψ₁ and v'∈ ψ ₂ be the corresponding epipoles between views ψ₁ , ψ₂, and let ∈ ψ₁ and v"∈ ψ₃ the corresponding epipoles between views ψ₁ , ψ₃. Likewise, corresponding image points across three views will be denoted by p = (x, y, 1), p' = (x', y', 1 ) and p" = (x", y", 1). The term "image coordinates" will denote the non-homogeneous coordinate representation of P², e.g., (x, y), (x', y'), (x", y") for the three corresponding points.

Planes will be denoted by π _i , indexed by i, and just π if only one plane is discussed. All planes are assumed to be arbitrary and distinct from one another. The symbol = denotes equality up to a scale, GL_n stands for the group of n x n matrices, and PGL_n is the group defined up to a scale.

III. THE TRILINEAR FORM

The central results of this manuscript are presented in the following two theorems. The remaining of the section is devoted to the proof of this result and its implications.

Theorem 1 (Trilinearity) Let ψ₁, ψ₂, ψ₃ be three arbitrary perspective views of some object, modeled by a set of points in 3D. The image coordinates (x, y) ∈ ψ₁ , (x', y') ∈ ψ ₂ and (x", y")∈ ψ₃ of three corresponding points across three views satisfy a pair of trilinear equations of the following form:

and o

where the coefficients α_j , β_j , j = 1, ..., 12, are fixed for all points, are uniquely defined up to an overall scale, and α_j = β_j , j = 1 , ..., 6.

The following auxiliary propositions are used as part of the proof.

Lemma 1 (Auxiliary - Existence) Let A∈ PGL₃ be the projective mapping (homography) ψ₁→ ψ ₂ due to some plane π . Let A be scaled to satisfy p'_o

Ap₀ + v'. where p_o∈ ψ ₁ and p'_o∈ ψ ₂ are corresponding points coming from an arbitrary point P_o∉ π . Then, for any corresponding pair p∈ ψ ₁ and p'∈ ψ ₂ coming from an arbitrary point P∈ P³, we have p' Ap + kv'.

The coefficient k is independent of ψ₂ , i. e. , is invariant to the choice of the second view.

The lemma, its proof and its theoretical and practical implications are discussed in detail in [28. 32]. Note that the particular case where the homography A is affine, and the epipole v' is on the line at infinity, corresponds to the construction of affine structure from two ort hographic views [17]. In a nutshell, a representation R ₀ of P³ (tetrad of coordinates) can always be chosen such that an arbitrary plane π is the plane at infinity. Then, a general uncalibrated camera motion generates representations R which can be shown to be related to R ₀ by an element of the affine group. Thus, the scalar k is an affine invariant within a projective framework, and is called a relative affine invariant. A ratio of two such invariants, each corresponding to a different reference plane, is a projective invariant [32]. For our purposes, there is no need to discuss the methods for recovering k— all we need is to use the existence of a relative affine invariant k associated with some arbitrary reference plane π which, in turn, gives rise to a homography A.

Definition 1 Homographies A_i∈ PGL₃ from ψ₁→ ψ_i due to the same plane π, are said to be scale-compatible if they are scaled to satisfy Lemma 1, i. e., for any point P∈ Φ projecting onto p∈ ψ₁ and pⁱ∈ ψ_i , there exists a scalar k that satisfies pⁱ

A_ip + kvⁱ , for any view ψ_i, where vⁱ∈ ψ₁ is the epipole with ψ₁ (scaled arbitrarily).

Lemma 2 (Auxiliary— Uniqueness) Let A, A'∈ PGL₃ be two homographies of ψ₁ → v ₂ due to planes π₁, π₂ , respectively. Then, there exists a scalar s , that satisfies the equation:

A - sA' = [αv', βv', γ v'], for some coefficients α, β , γ .

Proof: Let q∈ ψ₁ be any point in the first view. There exists a scalar s_q that satisfies v ' A _q - s_qA'q. Let H = A - s_qA', and we have Hq

v'. But, as shown in [29], Av

v' for any homography ψ₁ → ψ₂ due to any plane. Therefore. Hv

v' as well. The mapping of two distinct points q. v onto the same point v' could happen only if H is the homography due to the meridian plane (coplanar with the projection center O), thus Hp

v' for all p∈ ψ₁ , and s_q is a fixed scalar s. The latter, in turn, implies that H is a matrix whose columns are multiples of v'. []

Lemma 3 (Auxiliary for Lemma 4) Let A, A'∈ PGL₃ be homographies from ψ₁ → ψ₂ due to distinct planes π₁, π₂, respectively, and B, B'∈ PGL₃ be homographies from ψ₁ → ψ₃ due to π₁, π₂, respectively. Then, A' = AT for some T∈ PGL₃, and B = BCTC^-1, where Cv

Proof: Let A where A₁,A₂ are homographies from ψ₁,ψ₂ onto π₁. respectively.

Similarly

where B₁,B₂ are homographies from ψ₁,ψ₃ onto π₁, respectively. Let = (c₁.c₂,C₃)^T, and let

diag(c₁,c₂,c₃)A₁. Then, B₁

A₁ C^-1, and thus, we have

Note that the only difference between A₁ and B₁ is due to the different

location of the epipoles v,

, which is compensated by

Let E₁∈ PGL₃ be the homography from ψ₁ to π₂, and E₂∈ PGL₃ the homography from π₂ to π₁. Then with proper scaling of E₁ and E₂ we have

and with proper scaling of C we have,

[]

Lemma 4 (Auxiliary— Uniqueness) For scale-compatible homographies. the scalars s,α,β,γ of Lemma 2 are invariants indexed by ψ₁, π₁, π₂. That is. given an arbitrary third view ψ_3. let B,B' be the homographies from ψ₁ → ψ₃ due to π₁, π₂. respectively. Let B be scale-compatible with A, and B' be scale-compatible with A' . Then,

Proof: We show first that 5 is invariant, i.e., that B— sB' is a matrix whose columns are multiples of v". From Lemma 2, and Lemma 3 there exists a matrix H, whose columns are multiples of v'. a matrix T that satisfies A' = AT, and a scalar s such that I— sT = A^-1 H . After multiplying both sides by BC, and then pre-multiplying by C^-1 we obtain

B - sBCTC^-1 = BCA^-1HC^-1. f rom Lemma 3. we have B' = BCTC^{- 1}. The matrix A^{- 1} H has columns which are multiples of γ (because A^{- 1} v'

v), C A^{- 1}H is a matrix whose columns are multiple of

and BC A^{- 1}H is a matrix whose columns are multiples of v". Pre-multiplying BC A^-1H by C^-1 does not change its form because every column of BC A^-1HC^-1 is simply a linear combination of the columns of BCA^{- 1}H. As a result, B— sB' is a matrix whose columns are multiples of v".

Let H— A— sA' and H = B— sB'. Since the homographies are scale compatible, we have from Lemma 1 the existence of invariants k, k' associated with an arbitrary p∈ ψ₁ , where k is due to π₁, and k' is due to π₂: p'

Ap + kv'

A'p+ k'v' and p"

Bp+ kv"

B'p + k'v".

Then from Lemma 2 we have Hp = (sk'— k)v' and Hp = (sk'— k)v". Since p is arbitrary. this could happen only if the coefficients of the multiples of v' in H and the coefficients of the multiples of v" in

coincide. []

Proof of Theorem: Lemma 1 provides the existence part of theorem, as follows. Since Lemma 1 holds for any plane, choose a plane π₁ and let A, B be the scale-compatible homographies ψ₁ → ψ₂ and ψ₁ → ψ₃, respectively. Then, for every point p ∈ ψ₁ , with corresponding points p'∈ ψ₂, p"∈ ψ ₃, there exists a scalar k that satisfies: p' = Ap + kv', and p" = Bp + kv". We can isolate k from both equations and obtain:

where b₁, b₂, b₃ and α₁, α₂, α₃ are the row vectors of A and B and v' = =

. Because of the invariance of k we can equate terms of ( 1) with terms of (2) and

obtain trilinear functions of image coordinates across three views. For example, by equating the first two terms in each of the equations, we obtain:

In a similar fashion, after equating the first term of (1 ) with the second term of (2). we obtain:

Both equations are of the desired form, with the first six coefficients identical across both equat ions. The question of uniqueness arises because Lemma 1 holds for any plane. If w e choose a different plane, say π₂. with homographies A', B', then we raust show that the new homo- gr aphies give rise to the same coefficients (up to an overall scale). The parenthesized terms in (3) and (4) have the general form: , for some i and j . Thus, we need to show-

that there exists a scalar s that satisfies [ _:

This, however, follows directly from Lemmas 2 and 4. []

The direct implication of the theorem is that one can generate a novel view (ψ₃) by simply combining two model views (ψ₁ , ψ₂). The coefficients α_j and β_j of the combination can be recovered together as a solution of a linear system of 17 equations (24— 6— 1 ) given nine corresponding points across the three views (more than nine points can be used for a least-squares solution).

In the next theorem we obtain the lower bound on the number of points required for solving for the coefficients of the trilinear functions. The existence part of the proof of Theorem 1 indicates that there exists nine trilinear functions of that type, with coefficients having the general form Thus, we have at most 27 distinct coefficients (up to a uniform

scale), and thus, if more than two of the nine trilinear functions are linearly independent, we may solve for the coefficients using less than nine points. The next theorem shows that at most four of the trilinear functions are linearly independent and consequently seven points are sufficient to solve for the coefficients.

Theorem 2 There exists nine distinct trilinear forms of the type described in Theorem 1 of which at most four are linearly independent. The coefficients of the four trilinear forms can be recovered linearly with seven corresponding points across the three views.

Proof: The existence of nine trilinear forms follow directly from (1 ) and (2). Let α_ij =

The nine forms are given below (the first two are (3) and (4) repeated for

convenience):

For a given triplet the first four functions on the list produce a 4 x 27 matrix. The

rank of the matrix is four because it contains four orthogonal columns (columns associated with α₁₁ , α ₁₂, α ₂₁ and α₂₂), therefore these functions are linearly independent. Since we have 27 coefficients, and each triplet p, p', p" contributes four linear equations, then seven corresponding points across the three views provide a sufficient number of equations for a linear solution for the coefficients (given that the system is determined up to a common scale, seven points produce two extra equations which can be used for consistency checking or for obtaining a least squares solution).

The remaining trilinear forms are linearly spanned by the first four, as follows:

where the numbers in parenthesis represent the equation numbers of the various trilinear functions. []

Taken together, both theorems provide a constructive means for solving for the positions x", y" in a novel view given the correspondences p, p' across two model views. This process of generating a novel view can be easily accomplished without the need to explicitly recover structure, camera transformation, or even just the epipolar geometry— and requires fewer corresponding points than any other known alternative.

The solution for x", y" is unique without constraints on the allowed camera transformations. There are, however, certain camera configurations that require a different set of four trilinear functions from the one suggested in the proof of Theorem 2. For example, the set of equat ions (5), (6), (9) and ( 10) are also linearly independent. Thus, for example, in case and vanish simultaneously , i.e. , v' (0, 1 , 0), then that set. should be used inst ead. Similarly, equations (3), (4). (9) and ( 10) are linearly independent, and should be used in case X = ( 1 , 0, 0). Similar situations arise with v"

( 1. 0, 0) and v"

(0. 1 , 0) which can be dealt by choosing the appropriate basis of four functions from the six discussed above. Note that we have not addressed the problem of singular configurations of seven points. For example, its clear that if the seven points are coplanar, then their correspondences across the three views could not possibly yield a unique solution to the problem of recovering the coefficients. The matter of singular surfaces has been studied for the eight-point case necessary for recovering the epipolar geometry [19, 14, 22]. The same matter concerning the results presented in this manuscript is an open problem.

Moving away from the need to recover the epipolar geometry carries distinct and significant advantages. To get a better idea of these advantages, we consider briefly the process of reprojection using epipolar geometry. The epipolar intersection method can be described succinctly (see [11]) as follows. Let E₁₃ and E₂₃ be the matrices ("fundamental" matrices in recent terminology [10]) that satisfy p"F₁₃p = 0, and p"F₂₃p' = 0. Then, by incidence of p" with its epipolar line, we have:

Therefore, eight corresponding points across the three views are sufficient for a linear solution of the two fundamental matrices, and then all other object points can be re-projected onto the third view. Equation ( 12) is also a trilinear form, but not of the type introduced in Theorem 1. The differences include (i) epipolar intersection requires the correspondences coming from eight points, rather than seven, (ii) the position of p" is solved by a line intersection process which is singular in the case the three camera centers are collinear: in the trilinearity result the components of p" are solved separately and the situation of three collinear cameras is admissible, (iii) the epipolar intersection process is decomposable, i.e., only two views are used at a time; whereas the epipolar geometries in the trilinearity result are intertwined and are not recoverable separately. The latter implies a better numerically behaved method in the presence of noise as well, and as will be shown later, the performance, even using the minimal number of required points, far exceeds the performance of epipolar intersection using many more points. In other words, by avoiding the need to recover the epipolar geometry we obtain a significant practical advantage as well, since the epipolar geometry is the most error-sensitive component when working with perspective views. The connection between the general result of trilinear functions of views and the "linear combination of views" result [38] for orthographic views, can easily be seen by setting A and B to be affine in P², and = 0. For example. (3) reduces to

which is of the form

As in the perspective case, each point contributes four equations, but here there is no advantage for using all four of them to recover the coefficients, therefore we may use only two out of the four equations, and require four corresponding points to recover the coefficients. Thus, in the case where all three views are orthographic, x" (y") is expressed as a linear combination of image coordinates of the two other views— as discovered by [38].

IV. THE BILINEAR FORM

Consider the case for which the two reference (model) views of an object are taken or- thographically (using a tele lens would provide a reasonable approximation), but during recognition any perspective view of the object is allowed. It can easily be shown that the three views are then connected via bilinear functions (instead of trilinear):

Theorem 3 (Bilinearity) Within the conditions of Theorem 1, in case the views ψ₁ and ψ₂ are obtained by parallel projection, then the pair of trilinear forms of Theorem 1 reduce to the following pair of bilinear equations:

and

/

where α_j = 3j , j = 1 , ... , 4.

Proof: Under these conditions we have from Lemma 1 that A is affine in P² and v₃' = 0. therefore (3 ) reduces to:

Similarly. (4 ) reduces to:

Bot h equations are of the desired form, with the first tour coefficients identical across both equations. []

The remaining trilinear forms undergo a similar reduction, and Theorem 2 still holds, i.e.. we still have four linearly independent bilinear forms. Consequently, we have 21 coefficients up to a common scale (instead of 27) and four equations per point, thus five corresponding points (instead of seven) are sufficient for a linear solution.

A bilinear function of three views has two advantages over the general trilinear function. First, as mentioned above, only five corresponding points (instead of seven) across three views are required for solving for the coefficients. Second, the lower the degree of the algebraic function, the less sensitive the solution may be in the presence of errors in measuring correspondences. In other words, it is likely (though not necessary) that the higher order terms, such as the term x"x'x in Equation 3, will have a higher contribution to the overall error sensitivity of the system.

Compared to the case when all views are assumed orthographic, this case is much less of an approximation. Since the model views are taken only once, it is not unreasonable to require that they be taken in a special way, namely, with a tele lens (assuming we are dealing w ith object recognition, rather than scene recognition). If this requirement is satisfied, then the recognition task is general since we allow any perspective view to be taken during the recognition process.

V. EXPERIMENTAL DATA

The experiments described in this section were done in order to evaluate the practical aspect of using the trilinear result for re-projection compared to using epipolar intersection and the linear combination result of [38] (the latter we have shown is simply a limiting case of the trilinear result).

The epipolar intersection method was implemented as described in Section III by recovering first the fundamental matrices. Although eight corresponding points are sufficient for a linear solution, in practice one would use more than eight points for recovering the fundamental matrices in a linear or non-linear squares method. Since linear least squares methods are still sensitive to image noise, we used the implementation of a non-linear method described in [20] which was kindly provided by T. Luong and L. Quan (these were two implementations of the method proposed in [20]— in each case, the implementation that provided the better results was adopted).

The first experiment is with simulation data showing that even when the epipolar geometry is recovered accurately, it is still significantly better to use the trilinear result which avoids the process of line intersection. The second experiment is done on a real set of images, comparing the performance of the various methods and the number of corresponding points that are needed in practice to achieve reasonable re-projection results. 4. Computer Simulations

We used an object of 46 points placed randomly with z coordinates between 100 units and 120 units, and x, y coordinates ranging randomly between -125 and +125. Focal length was of 50 units and the first view was obtained by fx/z, fy/z. The second view (ψ₂) was generated by a rotation around the point (0, 0, 100) with axis (0.14, 0.7, 0.7) and by an angle of 0.3 radians. The third view (ψ ₃) was generated by a rotation around an axis (0, 1, 0) with the same translation and angle. Various amounts of random noise was applied to all points that were to be re-projected onto a third view, but not to the eight or seven points that were used for recovering the parameters (fundamental matrices, or trilinear coefficients). The noise was random, added separately to each coordinate and with varying levels from 0.5 to 2.5 pixel error. We have done 1000 trials as follows: 20 random objects were created, and for each degree of error the simulation was ran 10 times per object. We collected the maximal re-projection error (in pixels) and the average re-projection error (averaged of all the points that were re-projected). These numbers were collected separately for each degree of error by averaging over all trials (200 of them) and recording the standard deviation as well. Since no error were added to the eight or seven points that were used to determine the epipolar geometry and the trilinear coefficients, we simply solved the associated linear systems of equations required to obtain the fundamental matrices or the trilinear coefficients.

The results are shown in Figure 1A-1B. The graph on the left shows the performance of both algorithms for each level of image noise by measuring the maximal re-projection error. We see that under all noise levels, the trilinear method is significantly better and also has a smaller standard deviation. Similarly for the average re-projection error shown in the graph on the right .

This difference in performance is expected, as the trilinear method takes all three views together, rather than every pair separately, and thus avoids line intersections. B. Experiments On Real Images

Figure 2A-2C shows three views of the object we selected for the experiment. The object is a sports shoe with added texture to facilitate the correspondence process. This object was chosen because of its complexity, i.e. , it has a shape of a natural object and cannot easily be described parametrically (as a collection of planes or algebraic surfaces). Note that the situation depicted here is challenging because the re-projected view is not in-between the two model views, i.e.. one should expect a larger sensitivity to image noise than in-between situations. A set of 34 points were manually selected on one of the frames. ψ₁ , and their correspondences were automatically obtained along all other frames used in this experiment . The correspondence process is based on an implementation of a coarse- to- fine optical-flow algorithm described in [7]. To achieve accurate correspondences across distant views, intermediate in-between frames were taken and the displacements across consecutive frames were added. The overall displacement field was then used to push ( "warp" ) the first frame towards the target frame and thus create a synthetic image. Optical-flow was applied again between the synthetic frame and the target frame and the resulting displacement was added to the overall displacement obtained earlier. This process provides a dense displacement field which is then sampled to obtain the correspondences of the 34 points initially chosen in the first frame. The results of this process are shown in Figure 2A-2C by displaying squares centered around the computed locations of the corresponding points. One can see that the correspondences obtained in this manner are reasonable, and in most cases to sub-pixel accuracy. One can readily automate further this process by selecting points in the first frame for which the Hessian matrix of spatial derivatives is well conditioned— similar to the confidence values suggested in the implementations of [4. 7, 34]— however, the intention here was not so much as to build a complete system but to test the performance of the trilinear re-projection method and compare it to the performance of epipolar intersection and the linear combination methods.

The trilinear method requires at least seven corresponding points across the three views ( we need 26 equation, and seven points provide 28 equations), whereas epipolar intersection can be done (in principle) with eight points. The question we are about to address is what is t he number of points that are required in practice (due to errors in correspondence, lens distortions and other effects that are not adequately modeled by the pin-hole camera model ) t o achieve reasonable performance? The trilinear result was first applied with the minimal number of points (seven) for solving for the coefficients, and then applied with 8,9, and 10 points using a linear least-squares solution ( note that in general, better solutions may be obtained by using SVD or Jacobi methods instead of linear least-squares, but that was not attempted here). The results are shown in Figure 3A-3B. Seven points provide a re-projection with maximal error of 3.3 pixels and average error of 0.98 pixels. The solution using 10 points provided an improvement with maximal error of 1.44 and average error of 0.44 pixels. The performance using eight and nine points was reasonably in-between the performances above. Using more points did not improve significantly the results; for example, when all 34 points were used the maximal error went down to 1.14 pixels and average error stayed at 0.42 pixels.

Next the epipolar intersection method was applied. We used two methods for recovering the fundamental matrices. One method is by using the implementation of [20]. and the other is by taking advantage that four of the corresponding points are coming from a plane (the ground plane). In the former case, much more than eight points were required in order to achieve reasonable results. For example, when using all the 34 points, the maximal error was 43.4 pixels and the average error was 9.58 pixels. In the latter case, we recovered first the homography B due to the ground plane and then the epipole v" using two additional points (those on the film cartridges). It is then known (see [28, 21 , 32]) that F₁₃ = [v")B, where [v"] is the anti-symmetric matrix of v" . A similar procedure was used to recover F₂₃. Therefore, only six points were used for re-projection, but nevertheless, the results were slightly better: maximal error of 25.7 pixels and average error of 7.7 pixels. Figure 4A-4B shows these results.

Finally, we tested the performance of re-projection using the linear combination method. Since the linear combination method holds only for orthographic views, we are actually test ing the orthographic assumption under a perspective situation, or in other words, whether the higher (bilinear and trilinear) order terms of the trilinear equations are significant or not. The linear combination method requires at least four corresponding points across the three views. We applied the method with four, 10 (for comparison with the trilinear case shown in Figure 3A-3B). and all 34 points (the latter two using linear least squares). The results are displayed in Figure 5A-5C. The performance in all cases are significantly poorer than when using the trilinear functions, but better than the epipolar intersection method. VI. DISCUSSION

We have seen that any view of a fixed 3D object can be expressed as a trilinear function with two reference views in the general case, or as a bilinear function when the reference views are created by means of parallel projection. These functions provide alternative, much simpler, means for manipulating views of a scene than other methods. Moreover, they require fewer corresponding points in theory, and much fewer in practice. Experimental results show that the trilinear functions are also useful in practice yielding performance that is significantly better than epipolar intersection or the linear combination method.

In general two views admit a "fundamental" matrix (cf. [10]) representing the epipolar geometry between the two views, and whose elements are subject to a cubic constraint (rank of the matrix is 2). The trilinearity results (Theorems 1 ,2) imply that three views admit a tensor with 27 distinct elements. We have seen that the tensor does not fail in cases where the epipolar constraint fails, such as when the three cameras are along a straight line (not an uncommon situation). The issue of singular configurations of 7 points (besides the obvious singular configuration of 7 coplanar points) was not addressed in this manuscript. However, the robustness of the re-projection results may indicate that either such configurations are very rare or do not exist. It would be, thus, important to investigate this issue as it is widely believed that the numerical instability of the epipolar constraint lies in the existence of such critical surfaces. The notion of the "trilinear" tensor, its properties, relation to the geometry of three views, and applications to 3D reconstruction from multiple views, constitutes an important future direction.

The application that was emphasized throughout the manuscript is visual recognition via alignment. Reasonable performance was obtained with the minimal number of required points (seven) with the novel view (ψ₃)— which may be too many if the image to model matching is done by trying all possible combinations of point matches. The existence of bilinear functions in the special case where the model is orthographic, but the novel view is perspective, is more encouraging from the standpoint of counting points. Here we have the result that only five corresponding points are required to obtain recognition of perspective views ( provided we can satisfy the requirement that the model is orthographic). We have not experimented with bilinear functions to see how many points would be needed in practice, but plan to do that in the future. Because of their simplicity, one may speculate that these algebraic functions will find uses in tasks other than visual recognition— some of those are discussed below.

There may exist other applications where simplicity is of major importance, whereas the number of points is less of a concern. Consider for example, the application of model-based compression. With the trilinear functions we need 17 parameters to represent a view as a function of two reference views in full correspondence (recall, 27 coefficients were used in order to reduce the number of corresponding points from nine to seven). Assume both the sender and the receiver have the two reference views and apply the same algorithm for obtaining correspondences between the two views. To send a third view (ignoring problems of self occlusions that may be dealt with separately) the sender can solve for the 17 parameters using many points, but eventually send only the 17 parameters. The receiver then simply combines the two reference views in a "trilinear way" given the received parameters. This is clearly a domain where the number of points is not a major concern, whereas simplicity, and robustness (as shown above) due to the short-cut in the computations, is of great importance.

Related to image coding, an approach of image decomposition into "layers" was recently proposed by [1, 2]. In this approach, a sequence of views is divided up into regions, whose motion of each is described approximately by a 2D affine transformation. The sender sends the first image followed only by the six affine parameters for each region for each subsequent frame. The use of algebraic functions of views can potentially make this approach more powerful because instead of dividing up the scene into planes one can attempt to divide the scene into objects, each carries the 17 parameters describing its displacement onto the subsequent frame.

Another area of application may be in computer graphics. Re-projection techniques provide a short-cut for image rendering. Given two fully rendered views of some 3D object, other views (again ignoring self-occlusions) can be rendered by simply "combining" the reference views. Again, the number of corresponding points is less of a concern here.

REFERENCES

[1] E.H. Adelson. Layered representations for image coding. Technical Report 181, Media Laboratory, Massachusetts Institute of Technology, 1991.

[2] E.H. Adelson and J.Y.A. Wang. Layered representation for motion analysis. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition, pages 361-366. New York. NY. June 1993. [3] G. Adi v. Inherent ambiguities in recovering 3D motion and structure from a noisy flow field. IEEE Transactions on Pattern Analysts and Machine Intelligence. PAMI- 11 (5):477-4S9, 19S9.

[4] P. Anandan. A unified perspective on computational techniques for the measurement of visual motion. In Proceedings Image Understanding Workshop, pages 219-230, Los Angeles, CA, February 1987. Morgan Kaufmann, San Mateo, CA.

[5] LA. Bachelder and S. Ullman. Contour matching using local affine transformations. In Proceedings Image Understanding Workshop. Morgan Kaufmann, San Mateo, CA. 1992.

[6] E.B. Barrett, M.H. Brill. N.N. Haag, and P.M. Payton. Invariant linear methods in photogrammetry and model-matching. In J.L. Mundy and A. Zisserman. editors. Applications of invariances in computer vision. MIT Press, 1992.

[7] J.R. Bergen and R. Hingorani. Hierarchical motion-based frame rate conversion. Technical report, David Sarnoff Research Center, 1990.

[8] S. Demey, A. Zisserman, and P. Beardsley. Affine and projective structure from motion.

In Proceedings of the British Machine Vision Conference, October 1992.

[9] R. Dutta and M.A. Synder. Robustness of correspondence based structure from motion.

In Proceedings of the International Conference on Computer Vision, pages 106-110, Osaka, Japan, December 1990.

[10] O.D. Faugeras. What can be seen in three dimensions with an uncalibrated stereo rig? In Proceedings of the European Conference on Computer Vision, pages 563-57S. Santa Margherita Ligure. Italy. June 1992.

[11] O.D. Faugeras and L. Robert. What can two images tell us about a third one? Technical

Report INRIA. France. 1993.

[12] W.E.L. Grimson. Why stereo vision is not always about 3D reconstruction. A.I. Memo No. 1435. Artificial Intelligence Laboratory, Massachusetts Institute of Technology. July 1993.

[13] R. Hartley, R. Gupta, and T. Chang. Stereo from uncalibrated cameras. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition, pages 761-764. Champaign, IL.. June 1992.

[14] B. K.P. Horn. Relative orientation. International Journal of Computer V ision. 4:59-78,

1990.

[1 5] B.K. P. Horn. Relative orientation revisited. Journal of the Optical Society of A merica. 8: 1630-1638, 1991.

[16] D.P. Huttenlocher and S. Ullman. Recognizing solid objects by alignment with an image.

International Journal of Computer Vision, 5(2): 195-212, 1990.

[17] J.J. Koenderink and A.J. Van Doom. Affine structure from motion. Journal of the Optical Society of America, 8:377-385, 1991.

[18] H.C. Longuet-Higgins. A computer algorithm for reconstructing a scene from two projections. Nature, 293:133-135. 1981.

[19] H.C. Longuet-Higgins. the reconstruction of a scene from two projections - configurations that defeat the 8-point algorithm. In Proc. of the 1st Conf. Al applications, pages 395-397, Denver, December 1984.

[20] Q.T. Luong, R. Deriche, O.D. Faugeras, and T. Papadopoulo. On determining the fundamental matrix: Analysis of different methods and experimental results. Technical Report INRIA, France, 1993.

[21] Q.T. Luong and T. Vieville. Canonical representations for the geometries of multiple projective views. Technical Report INRIA, France, fall 1993.

[22] S.J. Maybank. The projective geometry of ambiguous surfaces. Proceedings of the Royal Society of London, 332:1-47, 1990.

[23] J. Mundy and A. Zisserman. Appendix— projective geometry for machine vision. In J. Mundy and A. Zisserman, editors, Geometric invariances in computer vision. MIT Press, Cambridge. 1992.

[24] J.L. Mundy, R.P. Welty, M.H. Brill. P.M. Payton, and E.B. Barrett. 3-D model alignment without computing pose. In Proceedings Image Understanding Workshop, pages 727-735. Morgan Kaufmann. San Mateo, CA, January 1992.

[25] A. Shashua. Correspondence and affine shape from two orthographic views: Motion and Recognition. A.I. Memo No. 1327, Artificial Intelligence Laboratory, M assachusetts Institute of Technology, December 1991.

[26] A. Shashua. Geometry and Photometry in 3D visual recognition. PhD thesis. M.I.T Artificial Intelligence Laboratory, AI-TR-1401, November 1992.

[27] A. Shashua. Illumination and view position in 3D visual recognition. In S.J. Hanson J.E. Moody and R.P. Lippmann, editors. Advances in Neural Information Processing Systems 4. pages 404-411. San Mateo, CA: Morgan Kaufmann Publishers. 1992. Proceedings of the fourth annual conference NIPS, Dec. 1991 , Denver. CO. [28] A. Shashua. On geometric and algebraic aspects of 3D affine and projective structures from perspective 2D views. In The 2nd European Workshop on In variants. Ponta Delagada. Azores, October 1993. Also in MIT Al memo No. 1405, July 1993.

[29] A. Shashua. Projective depth: A geometric invariant for 3D reconstruction from two perspective/orthographic views and for visual recognition. In Proceedings of the International Conference on Computer Vision, pages 583-590, Berlin, Germany. May 1993.

[30] A. Shashua. Projective structure from uncalibrated images: structure from motion and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1994. in press.

[31] A. Shashua. Trilinearity in visual recognition by alignment. In Proceedings of the European Conference on Computer Vision, Stockholm, Sweden. May 1994.

[32] A. Shashua and N. Navab. Relative affine structure: Theory and application to 3d reconstruction from perspective views. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition, Seattle, Washington, 1994.

[33] A. Shashua and S. Toelg. The quadric reference surface: Applications in registering views of complex 3d objects. In Proceedings of the European Conference on Computer Vision, Stockholm, Sweden, May 1994.

[34] C. Tomasi and T. Kanade. Factoring image sequences into shape and motion. In IEEE Workshop on Visual Motion, pages 21-29, Princeton, NJ, September 1991.

[35] R.Y. Tsai and T.S. Huang. Uniqueness and estimation of three-dimensional motion parameters of rigid objects with curved surface. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-6:13-26. 1984.

[36] S. U llman. The Interpretation of Visual Motion. MIT Press, Cambridge and London, 1979.

[37] S. U llman. Aligning pictorial descriptions: an approach to object recognition. Cognition, 32:193-254, 1989. Also: in MIT Al Memo 931, Dec. 1986.

[38] S. Ullman and R. Basri. Recognition by linear combination of models. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-13:992— 1006. 1991. Also in M.I.T Al Memo 1052, 1989.

[39] D. Weinshall. Model based invariants for 3-D vision. International Journal of Computer Vision, 10( 1 ):27-42, 1993. It is appreciated that some or all of the image processing operations shown and described herein may be implemented in software. Alternatively, some or all of the image processing operations shown and described herein may be implemented in ROM (read-only memory) form. The software components may, alternatively, be implemented in hardware, if desired, using conventional techniques.

The present invention has very broad applications and specifically is applicable in all fields in which 3D from 2D techniques are known to be useful. Applications of the present invention include at least the following: photogrammetry applications comprising map making from aerial and satellite photographs and coordinate measurements in aerospace and shipyard assembly plants, coordinate measurements of industrial parts (CMM) , automated optical based inspection of industrial parts, robotic cell alignment, robotic trajectory identification, 3D robotic feedback, 3D modelling of scenes, 3D modelling of objects, reverse engineering, and 3D digitizing.

Appendix B is a listing of a preferred software implementation of 3D reconstruction apparatus constructed and operative in accordance with a preferred embodiment of the present invention. To implement the 3D reconstruction apparatus based on Appendix B, Maple2 software, commercially available from Math- Soft, may be employed in conjunction with an IBM PCT or SUN workstation. The following procedure may be employed:

a. Programs generated from the listing of Appendix B may be loaded into Maple2 using the OPEN FILE command.

b. The subroutines may be run by bringing the cursor to the row on which appears the WITH (LINALG) command.

c. The RETURN key is then processed until the cursor reaches the end of the file.

d. The cursor is then returned to the beginning of the file.

e. The simulation is run by pressing RETURN until the following row is reached, inclusive:

EVALN (PCOMP) The system then prints on the screen a comparison between the given 3D world and the reconstructed 3D world .

It is appreciated that the listing of Appendix B is also useful in implementing the image transfer apparatus and method shown and described herein . The image transfer embodiment may , for example , be based on the listing of Appendix B used in conj unction with Equations 3 and 4 of the above section entitled Algebraic Function for Recognition .

It is appreciated that the particular embodiments described in the Appendices is intended only to provide an extremely detailed disclosure of the present invention and is not intended to be limiting .

It is appreciated that various features of the invention which are , for clarity , described in the contexts of separate embodiments may also be provided in combination in a single embodiment . Conversely , various features of the invention which are , for brevity , described in the context of a single embodiment may also be provided separately or in any suitable subcombination .

It will be appreciated by persons skilled in the art that the present invention is not limited to what has been particularly shown and described hereinabove . Rather , the scope of the present invention is defined only by the claims that follow :

APPENDIX A

/

APPENDIX B

Claims

1. A method for generating information regarding a 3D object from at least one 2D projection thereof, the method comprising:

providing at least one 2D projection of a 3D object; generating an array of numbers described by:

α_ijk = v_i' b_jk - v_j" a_ik (i,j,k = 1,2,3), where a_ij and b_jk are elements of matrices A and B respectively and v_i' and v_i" are elements of vectors v' and v" respectively, wherein said matrices and vectors together describe camera parameters of three views of the 3D object; and

employing the array to generate information regarding the 3D object.

2. A method for generating a new view of a 3D object from first and second existing views thereof having n corresponding points p_i and p_i' (i = 1 ... n), the method comprising:

generating a tensor α_ijk; and

plugging the values of points p_i and p_i' (i = 1 ... n) into trilinear forms, thereby to extract an x", y" value representing a location in the new view; and

generating the new view on the basis of the result of the plugging in step.

3. A method for reconstructing a 3D object from at least one 2D projection thereof, the method comprising:

providing at least one 2D projection of a 3D object; generating an array of numbers described by:

α_ijk = v_i' b_jk - v_j" a_ik (i,j,k = 1,2,3), where a_ij and b_jk are elements of matrices A and B respectively and v_i' and v_i" are elements of vectors v' and v" respectively, wherein said matrices and vectors together describe camera parameters of three views of the 3D object;

permuting the array into three homography matrices associated with three corresponding planes in 3D space; and

employing the three homography matrices to reconstruct the 3D object.

4. A visual recognition method comprising:

providing three perspective views of a 3D object between which a trilinear relationships exists; and

employing the trilinear relationship between the views in order to perform visual recognition by alignment.

5. A method according to claim 4 and also comprising reprojecting the 3D object.

6. A method according to claim 1 wherein said information regarding the 3D object comprises a reconstruction of the 3D object.

7. A method according to claim 1 wherein said information regarding the 3D object comprises at least one new view of the 3D object generated without reconstructing the 3D object.

8. A method according to claim 1 wherein said at least one 2D projection comprises at least one aerial photograph.

9. A method according to claim 1 wherein said at least one 2D projection comprises at least one satellite photograph.

10. A method according to claim 1 wherein said information regarding the 3D object comprises at least one coordinate of the 3D object.

11. A method according to claim 1 wherein the 3D object comprises an aerospace object.

12. A method according to claim 1 wherein the 3D object comprises a large object such as a ship.

13. A method according to claim 1 wherein the 3D object comprises a nonexistent object.

14. Apparatus for generating information regarding a 3D object from at least one 2D projection thereof, the apparatus comprising:

apparatus for providing at least one 2D projection of a 3D object;

an array generator operative to generate an array of numbers described by:

α_ijk = v_i' b_jk - v_j" a_ik (i,j,k = 1,2,3), where a_ij and b_jk, are elements of matrices A and B respectively and v_i' and v_i" are elements of vectors v' and v" respectively, wherein said matrices and vectors together describe camera parameters of three views of the 3D object; and

an array analyzer employing the array to generate information regarding the 3D object.

15. Apparatus for generating a new view of a 3D object from first and second existing views thereof having n corresponding points p_i and p_i' (i = 1 ... n), the apparatus comprising:

apparatus for generating a tensor a_ijk; and

apparatus for plugging the values of points p_i and p_i' (i = 1 ... n) into trilinear forms, thereby to extract an x", y" value representing a location in the new view; and

apparatus for generating the new view on the basis of the result of the plugging in step.

16. Apparatus for reconstructing a 3D object from at least one 2D projection thereof, the apparatus comprising:

apparatus for providing at least one 2D projection of a

3D object;

an array generator operative to generate an array of numbers described by:

α_ijk = v_i' b_jk - v_j" a_ik (i,j,k = 1,2,3), where a_ij and b_jk are elements of matrices A and B respectively and v_i' and v_i" are elements of vectors v' and v" respectively, wherein said matrices and vectors together describe camera parameters of three views of the 3D object;

an array permutator operative to permute the array into three homography matrices associated with three corresponding planes in 3D space; and

a 3D object reconstructor operative to employ the three homography matrices to reconstruct the 3D object.

17. Visual recognition apparatus comprising:

apparatus for providing three perspective views of a 3D object between which a trilinear relationships exists; and

apparatus for employing the trilinear relationship between the views in order to perform visual recognition by alignment.

18. A method according to claim 1 wherein said at least one 2D projection comprises three 2D projections.

19. Apparatus according to claim 14 wherein said at least one 2D projection comprises three 2D projections.

20. A method according to claim 1 and also comprising employing a result of said method in order to perform coordinate measurement of industrial parts.

21. A method according to claim 1 and also comprising employing a result of said method in order to perform automated optical based inspection of industrial parts.

22. A method according to claim 1 and also comprising employing a result of said method in order to perform robotic cell alignment.

23. A method according to claim 1 and also comprising employing a result of said method in order to perform robotic trajectory identification.

24. A method according to claim 1 and also comprising employing a result of said method in order to perform 3D robotic feedback.

25. A method according to claim 1 and also comprising employing a result of said method in order to perform 3D modelling of scenes.

26. A method according to claim 1 and also comprising employing a result of said method in order to perform 3D modelling of objects.

27. A method according to claim 1 and also comprising employing a result of said method in order to perform reverse engineering.

28. A method according to claim 1 and also comprising employing a result of said method in order to perform 3D digitizing.

29. A 3D scene reconstruction method for generating a 3D representation of a 3D scene from first, second and third views thereof, the method comprising:

providing first, second and third views of a 3D scene; employing geometric information regarding said first, second and third views to generate a trilinear tensor representing the geometric relationship between the first, second and third views; and

generating a 3D representation of the 3D scene from said trilinear tensor.

30. A method according to claim 29 wherein said step of generating a 3D representation comprises:

computing an epipolar geometric representation of the first and second views from the trilinear tensor; and

generating said 3D representation from said epipolar geometric representation.

31. Image transfer apparatus for generating a novel view of a 3D scene from first and second reference views thereof, the apparatus comprising:

apparatus for providing first and second reference views of a 3D scene;

a trilinear tensor generator operative to employ geometric information regarding said first reference view, second reference view and novel view, respectively, to generate a trilinear tensor representing the geometric relationship between the first, second and novel views; and

a novel view generator operative to generate said novel view by computing a multiplicity of novel view locations each corresponding to different first and second corresponding locations in said first and second reference views respectively based on said first and second corresponding locations and said trilinear tensor.

32. 3D scene reconstruction apparatus for generating a 3D representation of a 3D scene from first, second and third views thereof, the apparatus comprising:

appparatus for providing first, second and third views of a 3D scene;

a trilinear tensor generator operative to employ geometric information regarding said first, second and third views to generate a trilinear tensor representing the geometric relationship between the first, second and third views; and

a 3D scene representation generator operative to generate a 3D representation of the 3D scene from said trilinear tensor.

33. A method according to claim 29 wherein said providing step comprises providing m > 3 views of the 3D scene and wherein said employing step is performed for each of a plurality of subsets of 3 views from among the m views, thereby to generate a plurality of intermediate tensors, the method also comprising, prior to said step of generating, combining the plurality of intermediate tensors into a final trlinear tensor and wherein said step of generating comprises generating a 3D representation of the 3D scene from said final trlinear tensor.

34. A method according to claim 33 wherein said employing step is performed for the first and second views in combination with each of the remaining views, thereby to generate m-2 intermediate tensors.

35. A method according to claim 33 wherein said step of combining comprises:

arranging each of the intermediate tensors within a corresponding 9 x (3 (m-2)) matrix; and

defining the final trilinear tensor as the four largest principal components of said matrix.

36. Matching point finding apparatus for finding a multiplicity of matching locations across three views, the apparatus comprising:

providing an initial set of matching locations across three views;

generating a trilinear tensor representing the relationships between said three views, based on said initial set; and

employing the trilinear tensor to generate a final set of matching locations.

37. Apparatus according to claim 36 wherein said step of employing comprises:

for each of a multiplicity of locations within a first of the three views:

generating first and second corresponding epipolar lines from the tensor which are associated with the remaining two views respectively;

selecting a first candidate matching location along the first epipolar line and computing a second candidate matching location along the second epipolar line based on the first matching location; and

repeating the selecting step until the two candidate matching locations and the first view location match.

38. A method according to claim 29 and also comprising:

performing surface interpolation on the 3D representation, thereby to generate a surface interpolated output; and

generating an orthophoto from the surface interpolated output.

39. A method according to claim 29 wherein said step of providing comprises sequentially positioning at least three imaging devices at a sequence of positions within the scene and imaging at least a portion of the scene, using each of the imaging devices, at each of the positions within the sequence.

40. A method according to claim 29 and also comprising comparing the 3D representation of the 3D scene to a stored model of the 3D scene.