WO1998045004A1

WO1998045004A1 - Method for generating 3d computer games, and computer games using the method

Info

Publication number: WO1998045004A1
Application number: PCT/IL1998/000137
Authority: WO
Inventors: Shmuel Banitt
Original assignee: Shmuel Banitt
Priority date: 1997-04-09
Filing date: 1998-03-26
Publication date: 1998-10-15
Also published as: IL120647A0

Abstract

There is provided a method for generating a composite, three-dimensional-like computer game, by interactive assembly and processing of two or more separate image sequences. A bank containing a plurality of main sequences of true-life origin is provided. These sequences being shot separately, or extracted from one source. Secondary sequences are shot separately or extracted from one source, whereas their contents are at least roughly related to the contents of the main sequence, and stored in another bank. The secondary sequences are selected as a function of the selected main sequences. The positioned relationship of the secondary sequence and the main sequence is selected, and each secondary sequence is processed. The player selects the main sequence which is displayed together with its related secondary sequence(s). The main and secondary sequences are processed on-line according to inputs provided by the player. Optionally, computer-generated images are displayed with the main or secondary images.

Description

METHOD FOR GENERATING 3D COMPUTER GAMES.AND COMPUTER GAMES USING THE METHOD

Field of the Invention

The present invention relates to the field of image processing. More

particularly, the invention relates to a method for generating computer

games, to computer games using the method, and to display means for

playing the game.

Background of the Invention

Computer games are widespread nowadays. They typically employ a bank

of computer-generated images and of sounds, which are either recorded or

computer generated, and which are combined according to predetermined

rules, to produce a sequence of actions related to actions taken by the

player. Thus, interactive games employ the power of the computer to cause

a specific occurrence, or a specific series of occurrences, to take place as a

result of input provided by the player.

The technology employed in the art suffers from several severe drawbacks,

the most important of which being: 1) that there is only a limited feehng of

reality, no matter how high the quality of the image; and 2) more

importantly, the cost of generating computer images is very high. Even in some instances, where computer-generated images have been

substituted by real shots, the feehng of reality is minimal, because no

depth of the scene is obtained, and in this respect the situation is even

worse than with computer generated images, where 3D-like images can be

produced. Furthermore, only a small number of shots can be used, because

of image processing hmitations, which affects the feeling of continuity of

the game. Above all, however, the existing computer games employing

real-life shots are uninteresting, because they lack reality and 3D feehng,

and because they are made of a narrow choice of repetitive sequences of

events.

In a copending patent apphcation (WO 95/33340), the description of which

is incorporated herein by reference, several considerations pertaining in

general to the generation of 3D-like sequences of images are detailed. These

are further briefly discussed below, for ease of reference.

There are several known methods to perceive depth. Examples are

stereoscopic perception of depth, psychological perception of depth using

visual cues and the use of changing focus to perceive depth. A viewer

viewing a scene in which the focal point changes will perceive that the scene

has depth. Stereoscopic depth perception requires that the viewer views a scene from

two different locations at the same time. In real life, this means that he

sees the same scene using both his left eye and right eye, each eye

perceiving a somewhat different image. When two dimensional visual

images are substituted for real life, it requires the use of two images taken

from different right eye and left eye locations respectively.

On the other hand, a viewer can perceive depth by virtue of a number of

psychological depth perception cues which are present in both real life and

in two dimensional visual images. Common cues include effects where the

change of coloring or softening of the shapes gives a feeling of distance to far

objects, distribution of light and shade on an object, overlapping of contours

as when a near object hides part of a more distant object, geometrical

perspective for example in the case of the converging of the parallel rails of

a railroad track as they extend in the distance, movement of an object in a

visual image either towards or away from the viewer or audience, and

movement for stimulating the Pulfrich effect

This is true of all the various devices in which two dimensional visual

images are projected onto screens. Such devices include televisions, motion

picture theaters, training simulators, multi-media systems and so on having

applications in the areas of entertainment, work, education, exhibitions,

training, etc. A number of devices had been proposed to permit a viewer or an audience to

view three dimensional visual images or at least the illusion of three

dimensional visual images when viewing two dimensional visual images.

These broadly fall into three groups: those which provide stereoscopic

viewing, those which help create the psychological illusion of depth and

those which display images on a three-dimensional display system, such as

a surrounding screen.

Devices providing stereoscopic viewing generally require the projection of

two pictures of the same view, taken simultaneously by more than one

camera from shghtly different angles, to emulate the binocular vision of

normal eyes. Several examples are shown in U.S. Patent Nos. 4,420,230 to

McElveen, 4,714,319 to Zeevi, and 5,225,861 to Marks. The use of a

vibrating mirror or screen is shown in U.S. Patent Nos. 4,130,832 to Sher

and 3,814,513 to Nims et al. Other systems require special spectacles to be

worn by the viewer or other devices which direct the proper view to the

proper eye. All these systems are cumbersome and costly to use. Also, they

tend to cause an uneasy feeling in the observer often resulting in headaches

that may become severe. Thus these systems have failed to become popular.

The second approach is shown in U.S. Patents 4,651,219 to Rickert,

3,582,961 to Shindo and 4,000,367 to Field wherein depth perception is provided by isolating the image to be viewed. In this way, the stereoscopic

cues of the space surrounding the image are suppressed so that the inherent

psychological depth indicators in the picture can be sensed by the observer

who perceives a two dimensional visual image as a three dimensional scene.

Other devices which produce a similar result are described in U.S. Patent

No. 3,820,873 to Redington et al., U.S. Patent No. 4,154,514 to Harvey

which utilize a curved screen, and U.S. Patent No. 4,941,041 to Kenyon

which utilizes the Pulfrich illusion.

Other devices for providing three dimensional viewing are also described in

the following U.S. Patents:

U.S Patent No. 2,468,046 to de los Monteros discloses an apparatus for

projecting and viewing images with a depth effect through the use of at

least one mirror to reflect the image from a projector onto a screen such that

a degree of displacement between the images produces the illusion of

stereoscopy.

U.S. Patent 3,514,871 to Tucker discloses a wide angle visual display for a

training simulator which eliminates distracting discontinuities found in

many wide angle displays. It does so by providing three virtual image lenses

between the viewing point and the two dimensional displays and by

providing substantial duphcation of the projected computer generated image on the marginal portions of the displays. This creates binocular

vision at the corners of the displays which gives the illusion of three

dimensionality.

U.S. Patent 5,274,405 to Webster describes a device into which the head is

placed wherein images are projected both to the area of visual attention and

the area of peripheral vision of each eye. The images are focused on the

screens to provide sharp stereoscopic viewing throughout the field of view.

Another technique which is also known in the art is to take pictures of an

object, e.g., a room, from different points of view, and to generate therefrom

sets of spatial parameters, or database for novel view synthesis. Such a

method is described, e.g., in "S. Avidan and A. Shashua, Novel View

Synthesis in Tensor Space, Hebrew University Technical Report, 1996" (also

available from the World Wide Web at http://www.cs.huji.ac.il). This method

permits to generate three-dimensional images of the object, and to generate

new images extrapolating from the parameters so obtained. This method

(called hereinafter "the parameters method", for the sake of brevity) is a

suitable method also for generating games according to the present

invention, as will be understood from the description to follow. It suffers,

however, from the drawback that substantial computational power is

required to carry it out. Nevertheless, as less expensive computing power is made available to the public constantly, the aforesaid parameters method is

a suitable candidate for the purposes of the invention.

Other methods for producing three-dimensional moving images are based

on computer generated images. An example is U.S. Patent 5,184,956 to

Langlais et al., where computer generated data is used to produce a three-

dimensional display for the purposes of driving training. It is easy to

appreciate by any one skilled in this field that it is not difficult to adapt

computer generated imaging techniques to many types of display systems

such as stereoscopic display, wide angle display, and also three-

dimensional display. This is so because the entire information required of

each part of the image is known to the computer's program - including all

the three-dimensional structure of each object, its surface color, texture, and

so on. This seeming advantage is also the major disadvantage of such

systems: the preparation of a realistic image requires painful and elaborate

design of each and every detail. Thus the display is either non-realistic or

else extremely expensive and laborious.

The method for generating a composite, three-dimensional-like image

sequence according to WO 95/33340, by the assembly of two or more

separate image sequences, comprises the steps of: (a) providing a main sequence of images;

(b) providing at least one secondary sequence of images the contents

of which are at least roughly related to the contents of said main

sequence;

(c) selecting the positioned relationship of each of said at least one

secondary sequence of images and of said main sequence of

images;

(d) processing each of said at least one secondary sequence of images

so as to optimize its matching with said main sequence in the

selected positioned relationship with it; and

(e) displaying said main sequence of images and said secondary

sequences of images at the same time and in substantially the

same positioned relationship as selected in step (c) above.

The term "three-dimensional-like" as used herein (also referred to herein as

"three dimensional", for the sake of brevity), is meant to indicate a sequence

of images which is displayed in more than one plane, and which thereby

evokes the feeling of real action taking place around the spectator. This

feehng, more fully described herein, gives the viewer the feeling of being "in

the movie", and provides a sense of reahty unattainable by regular movies

which are displayed in one plane only, as hereinafter defined. The three-

dimensional-like image sequence of the invention is "composite", inasmuch

as, in order to obtain the said effect, more than one image is displayed, as said, on different planes, the resulting whole image being composed of said

different displayed images.

The term "plane", used herein in relation to the display mean on which a

sequence of images is viewed, is to be understood to include display surfaces

in general, even if they are curved to some extent, and the term "plane"

should not be interpreted as meaning that concave or convex surfaces are

excluded. Similarly, when reference is made to an image which hes

substantially on one plane, this should be understood to include also an

image which is displayed on a curved display, as explained above.

Furthermore, it should be understood that it is possible to use curved

displays essentially made of one piece, onto which a main sequence of

images is displayed, along with one or more secondary sequences of images,

each sequence of images occupying a different portion of the curved display

or screen. As will be apparent to a skilled person, such a single, curved

display is equivalent to a plurality of single displays, each used to display a

single sequence of images. Furthermore, such a single curved display would

necessarily be made of sections which substantially he in different "planes", as defined above, when using curved displays.

The main sequence of images is preferably, but not limitatively, located in

the central part of the composite displayed image. This is because the main

image sequence is the standard by which the secondary image sequence is processed, and it will further be, in many instances, of higher quahty (in

terms of resolution, number of frames, etc.) than the secondary image

sequences. However, it is permissible to position the display displaying the

main image sequence at some other position relative to the center of the

picture, if technical or aesthetic requirements demand it.

Once the position of the main image sequence is selected, it is necessary to

select the location of the secondary image sequences within the composite

picture. There is no theoretical hmitation to the location of the secondary

image sequences, and some of them can be located, e.g., above or below the

display which displays the main image sequence. (These alternative

positions are not shown in the appended drawings, for the sake of brevity, since they are self evident from the description provided for the illustrative

geometries). Furthermore, a secondary image sequence may border with a

second secondary image sequence, and not with the main image sequence,

whereby said second secondary image sequence borders, e.g., with the main

image sequence. However, for most apphcations, when it is desired to

simplify and reduce the cost of both the process of creating the three-

dimensional image sequence, and of displaying it, it is preferred to position

the secondary sequences beside the main sequence.

As stated, in order to provide a three-dimensional-like image, it is necessary

that different image sequences be displayed on different planes. Typically, -li¬

the display on which the main sequence is viewed is positioned at an angle

with the one or more displays on which the at least one secondary sequence

is viewed. The term "positioned at an angle", however, should be interpreted

here to indicate that at least a major portion of the secondary image

sequences hes on planes (as hereinbefore defined) which intersect the plane

on which the main image sequence is displayed, no matter what the actual

interface between the display means of the secondary image sequences and

of the main image sequence are. Thus, for instance, curved displays may be

used, if desired, where the transition from the display of the main image

sequence to that of the secondary image sequences is smoother than when

using two fully planar displays, positioned at an angle with respect to one

another.

As stated, the secondary image sequences are processed in order to provide

a matching with the main image sequence. Therefore, according to one

preferred embodiment of the invention one or more of the secondary

sequences of images has been recorded at a time different from the time at

which the main sequence of images has been recorded.

One or more of the secondary sequences of images may have been recorded

at a location different from the location at which the main sequence of

images has been recorded. Furthermore, one or more of the secondary sequences of images may be

computer generated. It is also possible to mix images of different origin.

Thus, for instance, in one particular embodiment of the invention some of

the images in a sequence of images are computer generated, and some are

images acquired by movie or NCR equipment.

According to a preferred embodiment of WO 95/33340, the main sequence of

images and the secondary sequences of images are recorded on different

media. Typically, but non limitatively, the media on which the sequences of

images are recorded is selected from among film, video and digital data.

In many cases, and particularly when different recording and/or displaying

methods are used for the main image sequence and for the secondary image

sequences, the number of images in a first sequence, for a given sequence

display time, may not match the number of images in a second sequence to

be displayed together with said first sequence, for the same display time.

According to a preferred embodiment of WO 95/33340, therefore, the

method further comprises matching the image sequences to be displayed

together so as to reach essentially identical starting and ending times of

display for all sequences.

As stated, the secondary image sequences are processed to optimize their

matching to the main sequence of images. According to a preferred embodiment of WO 95/33340, the processing of a secondary sequence of

images comprises matching the field of view of said secondary sequence to

that of the main sequence. According to another preferred embodiment of

WO 95/33340, the processing of a secondary sequence of images to optimize

its matching to the main sequence of images comprises matching the

apparent motion of said secondary sequence to that of the main sequence.

As will be apparent to a person skilled in the art, the term "apparent

motion" embraces two different types of movements: the first, is the

movement that the camera appears to make relative to the scene being

recorded, as viewed by the observer. For instance, if the camera rotates 0.3°

per frame in the main sequence, to concentrate on an object that was

previously at the edge of its field of view, the images in the neighboring

secondary sequences must likewise rotate, at the same angular rate.

The second type of apparent motion is that of an object within the image

sequence. If, for instance, there is a road extending from the main sequence

into a secondary sequence, and a car travels on that road, it cannot appear

to change speed abruptly when it reaches the edge of the main sequence,

but its movement must continue smoothly on the display of the secondary

image sequence. However, in some cases less accuracy of transitions from

main to secondary images may be sufficient. For example, for scenes which

comprise many objects (more than 100), accuracy and smooth transition is required for the ensemble of aU objects in general, but less accuracy may be sufficient for every individual object of said ensemble appearing in the

scene.

In yet another preferred embodiment of WO 95/33340 the processing of a

secondary sequence of images to optimize its matching to the main sequence

of images comprises matching the displayed action of said secondary

sequence to that of the main sequence.

As will be apparent to a person skilled in the art, other parameters of the

image can be processed to improve the matching of the image sequences.

For instance, in a preferred embodiment of WO 95/33340 the processing of a

secondary sequence of images to optimize its matching to the main sequence

of images comprises matching the colors and shades of said secondary

sequence to that of the main sequence.

One advantage of the invention described in WO 95/33340 is that it is not

necessary to provide in all cases image sequences of high quahty and

resolution. According to one preferred embodiment, for instance, the

resolution of the images in a secondary sequence of images may be lower

than the resolution of images in the main sequence of images.

It has now been found, and this is an object of the invention, that it is

possible to generate computer games which exploit the three-dimensional like effect obtained by the method described above, and which lead to

substantially improved visual quahty and lower costs.

It is an object of the invention to provide a method for generating 3D-like

computer games, which employ real shots.

It is another object of the invention to provide computer games which

employ the method of the invention.

It is a further object of the invention to provide means for playing the

computer games generated according to the invention.

Other objects and advantages of the invention will become apparent as the

description proceeds.

SUMMARY OF THE INVENTION

The method for generating a composite, three-dimensional-like image

sequence, by the interactive assembly and processing of two or more

separate image sequences comprises the steps of:

(a) providing a bank containing a plurality of individually selectable

main images or main sequences of images, said images being

images of true-life origin and said images or sequences being shot

separately, or extracted from one source; (b) providing at least one bank of secondary images or sequences of

images, said images or sequences being shot separately or

extracted from one source, the contents of which are at least

roughly related to the contents of said main sequence, said

secondary sequences being selectable as a function of the selected

main images or sequences of images;

(c) selecting the positioned relationship of each of said at least one

secondary images or sequences of images and of said main images

or sequences of images;

(d) pre-processing each of said at least one secondary image or

secondary sequence of images so as to provide a desired processed

image;

(e) providing selecting means to select a main image or a main

sequence of images, as the result of an input provided by a player,

wherein said selected main image or main sequence of images is

displayed together with its related secondary image(s) or

sequence(s) of images, at the same time and in substantially the

same positioned relationship as selected in step (c) above;

(f) providing means for on-line processing said main image and said

secondary image, as a function of an input provided by a player;

(g) optionally, playing a sound related to said main sequence of

images and said secondary sequences of images, and/or playing or generating additional outputs related to such sequences of

images; and

(h) optionally, displaying at the same time non-real life, computer-

generated images in superimposed positioned relationship with

the main or secondary images.

The pre-processing referred to in (d) above may be effected to obtain many

different results. For instance, pre-processing can be aimed at optimizing

the matching of the image with said main image or main sequence of

images in the selected positioned relationship with it, or to create a

database of image parameters, or to change one or more of its

characteristics, etc.

By "images of true-life origin" is meant to indicate images which have been

recorded with a camera or the like recording means, and which are not

drawings or computer-generated images, but which may have been

processed, or combined with other images, or drawn on. The bank of images

contains shots taken by the various cameras, which are organized in

sequences, but which can also be utilized singularly. Thus, for instance,

when it is desired to generate an image on a side screen, or as a result of a

joystick movement, e.g., zooming in, it is possible to utilize an out-of-

sequence shot, taken from the bank, whenever this is suitable to create the

desired result. Various combinations of shots and sequences are of course possible, as will be appreciated by the skilled person, and such options are

therefore not reviewed herein in detail, for the sake of brevity. Thus, each

image stored in the image bank has, associated with it, a set of parameters

which define it, in terms which are related to the type of game being played,

and which permit easily and quickly to locate it, when needed for a given

purpose, within the image bank. The number and nature of the parameters

used to characterize an image may vary from one game to another, and is

solely dictated by the needs of the specific game. Thus, the selecting means

used to select a given image of sequence of images, may comprise database

managing software, used to locate and retrieve images efficiently.

By "extracted from one source" it is meant to indicate a procedure whereby

a broad view movie, which has been shot with a wide lens, or which is a

combination of two or more movies matched together using image

processing and/or mosaic artwork techniques, is used to generate therefrom

separate images or sequences of images. In some cases matching may be

implemented without image processing.

According to a preferred embodiment of the invention, the main image and

the secondary images are processed, as a function of an input provided by a

player, so as to zoom in or out of the original image stored in the image

bank. According to another preferred embodiment of the invention, the

main image and the secondary images are processed, as a function of an input provided by a player, so as to simulate the turning of an observer's head.

Other types of image processing are of course also possible and desirable.

For instance, in one preferred embodiment of the invention the main image

and the secondary image are processed, as a function of an input provided

by a player, so as to change the speed of movement of the observer.

Other types of suitable processing includes the generation of images by the

parameters method described above. Thus, according to this method,

whenever relevant parameters such as location, view angle, zoom, etc.

(referred herein sometime also as "the observer") change, a new image is

generated using the parameters previously made available by the analysis

of the real life images taken.

The additional outputs related to such sequences of images, which can be

played or generated, include any kind of stimuli that it is desired to

associate with the game, for instance, shaking of a seat, smell, smoke, etc.

According to a preferred embodiment of the invention, the sound played is

not necessarily associated with the main image being displayed, but may be

associated with the value of the input provided by the player. Thus, the

same image or set of images may be accompanied by different sounds if it

has been displayed as a result of different inputs from the player. Thus, as will be apparent to a person skilled in the art, a single image may be used to

generate different sequences, and different sequences may be used to

generate different scenes, depending on the development of the game.

The mixing of computer-generated and of real-life images can be extensively

exploited, because the human mind tends to allow them to blend without

finding disturbance in them. Thus, in some instances it may be more

efficient and quick, if the input from the player required too heavy an image

processing of the image available from the image bank, to generate a

suitable image, or a part thereof, through the computer, rather than

utilizing a real-life image. This, for instance, is convenient when a game

includes gun shots, explosions, or other occurrences requiring special effects.

Typically, however, the main image and the secondary image are processed,

as a function of an input provided by a player, so as to create an image

corresponding to that seen by an observer from a different point of view, or

so as to zoom in or out of the original image stored in the image bank,

similarly, to the eyes of the player moving closer of farther away from the

scene.

The invention is also directed to a bank of selectable, composite, three-

dimensional-like image sequences, which are generated by the method of

the invention. Thus, in one aspect, the invention is d rected to a bank of

selectable secondary sequences of images, suitable to be used as a

component of a computer game, comprising a main image or a main sequence of images and at least one secondary image or sequence of images

selectable together with said main image or sequence of images, said

secondary image or sequence of images having a content that at least

roughly matches the contents of the main image of said composite, three-

dimensional-like sequence of images, and said secondary image or sequence

of images further having been optimized for display together with said main

sequence of images.

Without wishing to limit the invention in any way, it should be mentioned

that in many cases it is preferred to provide images bigger than the screen

by 25% to 100%, to allow flexibility in the image processing. In such a case,

the image displayed is only a fraction of the total image, so that portions of the image which are not displayed for one desired viewing angle will be

displayed when the input of the player corresponds to another angle.

The invention also encompasses an interactive computer game comprising:

(a) a bank containing a plurality of individually selectable main

images or main sequences of images, said images being images of

true-life origin;

(b) at least one bank of secondary images or sequences of images, the

contents of which are at least roughly related to the contents of

said main sequence, said secondary sequences being selectable as

a function of the selected main images or sequences of images, and each of said at least one secondary image or secondary

sequence of images having been optionally pre-processed so as to

optimize its matching with said main image or main sequence of

images in the selected positioned relationship with it, or to obtain

other results, as discussed above;

(c) means for selecting the positioned relationship of each of said at

least one secondary images or sequences of images and of said

main images or sequences of images;

(d) interaction and processing means to select a main image or a

main sequence of images, as the result of an input provided by a

player, wherein said selected main image or main sequence of

images is d splayed together with its related secondary image(s)

or sequence(s) of images, at the same time and in substantially

the same positioned relationship as selected in step (c) above;

(e) display means to display said main image or sequence of images,

together with said at least one secondary image or sequence of

secondary images, so as to provide a 3D-like feeling to the player;

(f) means for on-line processing said main image and said secondary

image, as a function of an input provided by a player;

(g) means for playing a sound related to said main sequence of

images and said secondary sequences of images; and (h) optional means for displaying at the same time non-real life,

computer-generated images in superimposed positioned

relationship with the main or secondary images.

According to a preferred embodiment of the invention, the interactive

computer game comprises means for controlling the speed of display of the

frames on the display means. According to another preferred embodiment of

the invention, the means for processing the main image and the secondary

image, as a function of an input provided by a player, comprise image

processing means suitable to generate from the image available from the

image bank an image corresponding to that seen by an observer from a

different point of view.

The computer game of the invention preferably, but non-limitatively,

further comprises means for processing the main image and the secondary

image, as a function of an input provided by a player, so as to simulate the

turning of an observer's head. Further suitable means for processing the

main image and the secondary image, as a function of an input provided by

a player, comprise image processing means suitable to zoom in or out of the

original image stored in the image bank.

The invention is further directed to visual display systems for playing three-

dimensional-like computer games, comprising display means for displaying a main image or main sequence of images, and at least one additional

display means for displaying at least one secondary image or secondary

sequence of images, said display means for displaying a main image or

sequence of images, and said at least one additional display means for

displaying at least one secondary image or sequence of images being

positioned in substantially the same positioned relationship selected for the

generation of the composite three-dimensional-like image.

Brief Description of the Drawings

The above and other characteristics and advantages of the invention will

be more readily apparent through the following detailed description of

preferred embodiments thereof, with reference to the appended drawings,

wherein:

Fig. 1 schematically illustrates the overlap between three movies to

be displayed simultaneously;

Fig. 2 (2A and 2B) illustrates the operation of a game, according to

two different preferred embodiments of the invention;

Fig. 3 illustrates the result of a joystick movement simulating the

turning of an observer's head; and Fig. 4 schematically illustrates the operation involved in image

processing with head turning, in the case where a larger image than

displayed is available.

Detailed Description of Preferred Embodiments

In order to display an interactive 3d movie, three movies need to be

prepared as shown in Fig. 1. This means that three films have been taken

so that the three fields of view overlap to a certain extent, for example

25% of the field is overlapped in each side. Also, the captured images are

vertically larger than required, typically in the same amount, say 25% on

each side. It is very important that the three focal points of the cameras

coincide, as otherwise the effect of tilting the three cameras together will

look unnatural and the desired psychophysical effect will not be achieved.

All this can be done either by using a gig of three rigidly mounted, fixed,

calibrated cameras, or else, by using the preferred technique of filming the

three sequences separately and then, using appropriate processing means

and software, processing them to be fit for display on a 3D-like screen. The

output of this stage is three image sequences, given on a digital video cassette, or, preferably, on a random access data device such as a

computer hard disk.

According to a preferred embodiment of the present invention, the three

separate movies of Fig. 1 above are merged into one "large" (broad view) movie, using mosaic techniques. Thus, the resulting large movie is

equivalent to a movie taken by using the same camera. Similarly, the

large movie may be stored on a digital video cassette, or, preferably, on a

random access data device such as a computer hard disk, or on any other

suitable media.

Once this is ready, an interactive 3d movie can be presented in the

following way. Reference is now made to Figure 2A. In the preferred

embodiment shown in this figure, we have three image sequences residing

in three computer disks, 211, 212, 213. The main computer 230 pulls

sequentially the three sequences, and prepares them to be displayed on

the three corresponding display means 241, 242, 243, which can be video

screens, video projectors, LCD screens, or any other display means. It is

important the display means will reproduce the geometry of the three

cameras, as far as the geometry of the viewing eyes in relation to the

displays will be similar (in the geometrical sense) to the geometry of the

three (real or virtual) cameras. It should be kept in mind that the original

image sequences have been taken with a field of view which is larger than

that required to be properly displayed on a 3d-like display means. A

computer interaction means 220 is provided, such as a mouse or a joystick,

to allow interaction of the user with the system. Fig. 2B illustrates the operation of a game according to a preferred

embodiment of the present invention, using one large movie. The large

movie is stored in a computer disk 280. The main computer 270 extracts

the large movie sequence information and generates three sub -sequences,

which size and relative location are a function of the desired position,

which may be the center or any other position selected by the player. The

main computer manipulates the side movie sequence(s) to represent the

correct angles of the camera. Similar computer interaction means 260 is

provided, such as a mouse or a joystick, to allow interaction of the user

with the system.

This control mechanism provides parameters that effect the way that the

images are displayed on the screen. In this particular embodiment the

parameters are horizontal and vertical tilt (viewing angle), camera zoom,

and motion speed. It is clear to anyone skilled in the art that many other

parameters and parameter combinations can also be controlled or coupled

to input devices, such as sound volume, chair wobble, mechanical effects,

smell, digital video effects such as fog and many more. The following is a

detailed description of how the image is prepared to be displayed, using

the controls given by the interaction means, in this case horizontal tilt

theta, vertical tilt phi (given in radians), zoom z (given as percentage of

default image which is, say 70% of full image), and motion speed t (given

as percentage of standard speed - 25 or 30 frames per second). First the speed is controlled by standard video techniques, such as dropping frames

and/or generating virtual frames to be displayed with or without original

frames. For instance, virtual frames are generated in case when speed

reduction is required, e.g., creating an intermediate frame interpolating

between two existing frames (e.g., frame No. 1.5, interpolating midway

between frames No. 1 and No.2). Zoom is controlled by clipping from the

original image a larger or smaller sub-image and enlarging it to the

required image size. In addition, zoom may induce compensations like, for

instance, translation on secondary sequences. Rotation is done by the

standard application of projective transformations, where a horizontal

rotation of an image by theta is computed by the transformation:

x^v = ( x cos(theta) - sin(theta) ) / (x sin(theta) + cos (theta) )

y = y / (x sin(theta) + cos (theta) )

while vertical rotation by phi is given by the transformation:

x = x / (y sin(phi) + cos (phi) )

y = ( y cos(phi) - sin(phi) ) / (y sin(phi) + cos (phi) )

All in all, the final transformation involving zoom, vertical, and horizontal

rotation is obtained by multiplying together the separate transformations,

and then applying the result to the image. This can be done either by a standard computer utilizing a fast CPU module such as a "Pentium Pro"

which is sold by Intel Corporation, or by a computer assisted by

speciahzed graphics hardware accelerators (video texture mapping

module), such as an 02 workstation marketed by Silicon Graphics

Limited, California, U.S.A..

Looking now at Fig. 3, four sequences of images are given (Fig. 3A through

Fig. 3D), illustrating consecutive stages in the on-line image processing

carried out during the game, as the result of an input by the player,

corresponding to the turning of the head of the player to the left. Three

images are shown, which, for illustration purposes are shown on a plane,

but which should be understood to be positioned at an angle, e.g., as

schematically shown in Fig. 1. The images are numbered 1-3, from right to

left.

Each of the images 1, 2 and 3 , recovered from the data bank, contain

more information than displayed. However, each image is masked so that

only a portion thereof is shown. Thus, for instance, image (1) of Fig. 3A is

taken with a broader field of view, which includes the details not seen in

Fig. 3A, but which are seen in image (1) if Fig. 3D. Similarly, the details

seen in the right hand side of image (1) of Fig. 3D are contained in the

image (1) of Fig. 3A, but they are masked and therefore not seen. In this

way, it is possible to provide an effect of change and movement, while moving with the mask within the same image, without making any

further processing of the image, except the masking. Before being cut or

masked, the images are rotated according to the aforementioned (or

alternative) formulae, to provide the effect of turning.

This procedure is further schematically illustrated in Fig. 4. The whole

image available from the date bank is seen, together with the mask (the

inner square) which shows the part of the image which is being displayed.

In Fig. 4(a) the leftmost part of the building is displayed, and the portions

of the building found outside the mask are not seen on the display means.

A the head of the observer moves to the left (as a result of an input given,

e.g., by a keyboard, a joystick, or other means) the mask also rotates to the

left, showing portions of the building formerly not displayed, and hiding

the earlier displayed leftmost parts of the building.

Turning again to Fig. 3, it should be remembered that while the

movement of the head is effected, the game continues and the frames

continue to be displayed, at the required speed. This is seen, for instance,

by looking at image (3). In Fig. 3A the moving car is not only much more

to the right than in Fig. 3D, but is also not as close to the observer. While

the observer's head moved, the car has continued to advance. Of course, this example and the above description has been provided only

for the purpose of illustration, and are not intended to limit the invention

in any way. As will be appreciated by the skilled person, the invention can

be carried out in a great variety of ways, employing many different image

processing techniques and apparatus, and using different images and

images obtained and generated in different ways, all without exceeding

the scope of the invention.

Claims

1. A method for generating a composite, three-dimensional-like image

sequence, by the interactive assembly and processing of two or more

separate image sequences, comprising the steps of:

(a) providing a bank containing a plurahty of individually selectable

main images or main sequences of images, said images being

images of true-life origin, and said images or sequences being

shot separately, or extracted from one source;

(b) providing at least one bank of secondary images or sequences of

images, said images or sequences being shot separately or

extracted from one source, the contents of which are at least

roughly related to the contents of said main sequence, said

secondary sequences being selectable as a function of the selected

main images or sequences of images;

(c) selecting the positioned relationship of each of said at least one

secondary images or sequences of images and of said main images

or sequences of images;

(d) pre-processing each of said at least one secondary image or

secondary sequence of images so as to produce a desired

processed image;

(e) providing selecting means to select a main image or a main

sequence of images, as the result of an input provided by a player, wherein said selected main image or main sequence of images is

displayed together with its related secondary image(s) or

sequence(s) of images, at the same time and in substantially the

same positioned relationship as selected in step (c) above;

(f) providing means for on-line processing said main image and said

secondary image, as a function of an input provided by a player;

(g) optionally, playing a sound related to said main sequence of

images and said secondary sequences of images, and/or playing or

generating additional outputs related to such sequences of

images; and

(h) optionally, displaying at the same time non-real life, computer-

generated images in superimposed positioned relationship with

the main or secondary images.

2. A method according to claim 1, wherein the main image and the

secondary image are processed, as a function of an input provided by a

player, so as to create an image corresponding to that seen by an observer

from a different point of view.

3. A method according to claim 1, wherein the main image and the

secondary image are processed, as a function of an input provided by a

player, so as to zoom in or out of the original image stored in the image

bank.

4. A method according to claim 1, wherein the main image and the

secondary image are processed, as a function of an input provided by a

player, so as to simulate the turning of an observer's head.

5. A method according to claim 1, wherein the main image and the

secondary image are processed, as a function of an input provided by a

player, so as to change the speed of movement of the observer.

6. A method according to claim 1, wherein the sound played is not

associated with the main image being displayed, but is associated with

the value of the input provided by the player.

7. A method according to claim 4, wherein the image displayed is only a

fraction of the total image, so that portions of the image which are not

displayed at one time are displayed when the input of the player

corresponds to a turning of the observer's head.

8. A method according to claim 7, wherein the size of the image is 25% to

100% larger than the displayed image.

9. A bank of selectable, composite, three-dimensional-like image sequences,

whenever generated by the method of claim 1.

10. A bank of selectable secondary sequences of images, suitable to be used

as a component of a computer game, comprising a main image or a main

sequence of images and at least one secondary image or sequence of

images selectable together with said main image or sequence of images,

said secondary image or sequence of images having a content that at

least roughly matches the contents of the main image of said composite,

three-dimensional-like sequence of images, and said secondary image or

sequence of images further having been optimized for display together

with said main sequence of images.

11. An interactive computer game comprising:

(a) a bank containing a plurality of individuaUy selectable main

images or main sequences of images, and said images or

sequences being shot separately, or extracted from one source,

said images being images of true-life origin;

(b) at least one bank of secondary images or sequences of images, the

contents of which are at least roughly related to the contents of

said main sequence, and said images or sequences being shot

separately, or extracted from one source, said secondary

sequences being selectable as a function of the selected main

images or sequences of images, and each of said at least one secondary image or secondary sequence of images having

optionally been pre-processed;

(c) means for selecting the positioned relationship of each of said at

least one secondary images or sequences of images and of said

main images or sequences of images;

(d) interaction and processing means to select a main image or a

main sequence of images, as the result of an input provided by a

player, wherein said selected main image or main sequence of

images is displayed together with its related secondary image(s)

or sequence(s) of images, at the same time and in substantially

the same positioned relationship as selected in step (c) above;

(e) display means to display said main image or sequence of images,

together with said at least one secondary image or sequence of

secondary images, so as to provide a 3D-like feeling to the player;

(f) means for processing said main image and said secondary image,

as a function of an input provided by a player;

(g) means for playing a sound related to said main sequence of

images and said secondary sequences of images; and

(h) optional means for displaying at the same time non-real life,

computer-generated images in superimposed positioned

relationship with the main or secondary images.

12. An interactive computer game according to claim 11, comprising means

for controlling the speed of display of the frames on the display means.

13. A computer game as claimed in claim 11, wherein the means for

processing the main image and the secondary image, as a function of an

input provided by a player, comprise image processing means suitable

to generate from the image available from the image bank an image

corresponding to that seen by an observer from a different point of view.

14. A computer game according to claim 11, comprising means for

processing the main image and the secondary image, as a function of an

input provided by a player, so as to simulate the turning of an observer's

head.

15. A computer game as claimed in claim 11, wherein the means for

processing the main image and the secondary image, as a function of an

input provided by a player, comprise image processing means suitable

to zoom in or out of the original image stored in the image bank.

16. A visual display system for playing three-dimensional-like computer

games, comprising display means for displaying a main image or main

sequence of images, and at least one additional display means for

displaying at least one secondary image or secondary sequence of images, said display means for displaying a main image or sequence of

images, and said at least one additional display means for displaying at

least one secondary image or sequence of images being positioned in

substantially the same positioned relationship selected for the

generation of the composite three-dimensional-like image.

17. A method for generating a composite, three-dimensional-like image

sequence, by the interactive assembly and processing of two or more

separate image sequences, essentially as described and illustrated.

18. An interactive computer game, essentially as described and illustrated.