CA2420390A1

CA2420390A1 - Computerized image system

Info

Publication number: CA2420390A1
Application number: CA002420390A
Authority: CA
Inventors: Jason A. Stowe; Bryan M. Vandrovec; Douglas B. Clayton
Original assignee: Individual
Current assignee: Immersive Technologies LLC
Priority date: 2000-08-24
Filing date: 2001-08-24
Publication date: 2002-02-28
Also published as: US7034825B2; US20040100465A1; WO2002017044A3; EP1323013A2; WO2002017044A2; AU2001292559A1

Abstract

This system and process for providing a rendered image of a virtual 3-D scene comprises an image generating system which stores therein data defining the 3-D scene. Responses to receiving control data such as viewing parameters, editing commands, or instructions to a software application working with the scene data, the visualization systems (5) renders a pixel data image of the scene and transmits the pixel data over a data network (21), such as the Internet, to a user computer (13) to be viewed on a display device (31), such as a monitor.

Description

Patent Application COMPUTERIZED IMAGE SYSTEM
Field of the Invention This invention relates to the field of electronic transmission of computer generated images, and, more particularly, to systems and methods by which computers interact over a computer data network to transfer data for displaying computer-rendered images for viewing by a user.
Background of the Invention Computer data networks, and particularly the Internet, allow users to access software on separate computers located either nearby or at great distances from the user. These remotely-accessed software applications sometimes involve display to the user of computer-rendered images which represent views of virtual three-dimensional scenes.
A number of systems have been developed which provide for fairly quick turnaround time for preparation and display of rendered images of virtual scenes. The rate of image rendering can even be fast enough that images can be rendered in sequence at a rate that can approximate or achieve real-time graphics interaction of the user with the remote application.
The most common of these high-speed graphics rendering systems are based on dedicated "graphics pipeline" hardware in the user's computer. In these graphics pipeline systems, the remote application server transmits simple scene data over the network to the user computer. This scene data typically comprises data defining a group of triangles, or "primitives", that make up the objects of the three-dimensional scene. At the user computer, the scene data is sent to a separate graphics pipeline circuit board of the user computer, such as Open GL graphics H/W sold by Silicon Graphics or NVIDIA.
The graphics hardware renders a rasterized image representing a view of the virtual scene from the scene data by a series of discrete and independent computation steps. The circuit performs these computation steps like an assembly line, or "pipeline", or parallel calculations in which several images are simultaneously proceeding through the circuitry in varying states of completion. This circuit provides for rendering of serial images separated by close increments of time, which provides a flow of images that is fast enough to approximate sequential frames of video.
Unfortunately, although images can be rendered fairly quickly and close together by graphics pipeline hardware, the resulting images are very coarse and unrealistic. For example, usually the image does not contain any shadows, and the surfaces have an unrealistically uniform quality.
At the same time, increasing demand for more complex scenes is increasing the amount of scene data needed for a 3-D scene, which places even greater demands on the limited power of pipeline graphics cards. Improvements in communications protocols have increased data transmission rates, but the amount of scene data that is desired for transfer is becoming prohibitively large, despite these increases.
As an example, the actual image pixel data for frames of the film "Star Wars: the Phantom Menace" each represented about 25 megabytes of data, while the scene data that defined the scenes ran as high as approximately 10 to 15 gigabytes of data, i.e., hundreds of times larger.

Summary of the Invention It is therefore an object of the present invention to provide a method and a system that provides for interactive rendering of images for three-dimensional scenes over data networks, preferably with turnaround times that are suitable for real-time interactivity.
According to the methods and systems of the present invention, a user computer transmits control data over a computer data network, such as the Internet, to an image generating computer system. The image generating system automatically uses the control data to influence scene data defining a three dimensional scene and/or optical parameters for viewing the defined scene.
Instead of transmitting the scene data to the user computer for rendering an image using the user's system, as has been done in the pipeline graphics systems of the prior art, the image generating system renders pixel data at a location remote from the user, which data represents a view of the three-dimensional scene based on the scene data and the viewing parameters. The image generating system transmits the pixel data of the image over the data network to the user computer, and the user computer then displays the pixel data to the user as a viewable image.
The pixel data may be for a single image, transmitted as raw pixel data, or in a compressed form, such as in JPEG format. In the preferred embodiment, however, images are rendered by the image generating system in a continuing stream and transmitted for viewing as real-time interactive video, preferably in MPEG
format.
This system is particularly advantageous where the size of the scene data needed to render the pixel image exceeds the size of the data package needed to define the pixel image, and where the computational power required to render the pixel image within an acceptable turnaround time exceeds the computational capacity of the user system.
Preferably, a large number of users are connected with the image generating system, and these users may be widely distributed geographically. To improve real-time operation, the image generating system preferably includes a discrete geometric system that maintains the scene data. The geometric system is connected by very high-speed, high volume data transmission lines to a number of separate, geographically distributed visualization systems. Each visualization system communicates with a respective user computer over the data network, and is preferably located close in the network to the user computer to reduce latency in the network communications links.
To further improve access of the visualization systems to the scene data in a widely distributed system, the geometric system may be made up of a plurality of geographically distributed computer systems connected by high speed data links.
It is further an object of the invention to provide an image generating system with improved rendering techniques and systems.
It is also an object of the invention to provide a computer-accessible storage device or memory storing a database containing the scene data organized in a way which facilitates the determination of what visual objects are encountered along a path through the scene or in the neighborhood of a location therein.
It is also an object of the invention to provide a method for determining the objects in a scene that lie in a path using the aforesaid stored database.
Other objects and advantages of the invention will become apparent from the specification herein.

Brief Description of the Drawings Fig. 1 is a schematic view of the mufti-user image generating system of the invention.
Fig. 2A is a detailed schematic showing one of the visualization systems and its link to a user computer.
Fig. 2B is a schematic showing an alternate embodiment of an image generating system in which single-frame rendering is provided at slower turnaround times.
Fig. 3 is a perspective view diagram showing a three-dimensional virtual scene with visual objects therein.
Fig. 4 is a front elevational view of the scene of Fig. 3, to which multiple uniform subdivision partitioning has been applied.
Fig. 5 is a diagram of a hash table used to store scene data.
Fig. 6 is a diagram of a scene divided into cells.
Fig. 7 is a diagram as in Fig. 6 with a further subdivision.
Fig. ~ is a diagram as in Fig. 7, showing multiple, virtual uniform subdivision ("MVUS") Fig. 9 is a diagram of the use of the hash table entry for accessing a record of a complex visual object which is repeated in various scenes.
Fig. 10 is a diagram of the use of the hash table entry to identify the visual objects in a cell.
Fig. 11 is a diagram of a rotated nested sub-cell in a larger MVUS grid.
Fig. 12A is a flowchart of the algorithm for tracing a ray through the space defined by a hash table as shown in Fig. 5.

Fig. 12B is a detail flowchart showing the traversal algorithm used in the flowchart of Fig. 12A.
Fig. 13 is a diagram of a virtual scene, illustrating the image rendering process of the invention.
Fig. 14 is a diagram of the image generating system in which graphics pipeline circuitry is part of the image rendering system.
Fig. 15 shows a raster silhouette, the first step in the pipeline calculation for determining illumination of a point from the light source in Fig. 13.
Fig. 16 shows a second step of the calculation, wherein an illumination texture is mapped on the light source silhouette of Fig. 14.
Fig. 17 shows a further step, wherein the effect of transparent objects is combined with the light source silhouette of Fig. 15.
Fig. 18 shows the light source of Fig. 16, with occluding opaque intervening objects as viewed incorporated into the silhouette.
Fig. 19 illustrates the BRDF model of viewing of a reflected light or a visual object.
Fig. 20 is a diagram of reflected light intensity as a function of varying viewpoint.
Fig. 21A shows the viewed intensity of reflected light as a function of varying angles of incident light from the light source, mapped onto a sphere surrounding the point of reflection.
Fig. 21B shows the viewed intensity of reflected light as in Fig. 21A, but mapped onto a hemi-cube around the point of reflection.
Fig. 22A shows the hemi-cube of Fig. 21B partially unfolded into a plane.

Fig. 22B shows the unfolded hemi-cube of Fig. 22A organized as a texture field for transmission to a graphics card.
Fig. 23 shows a result of a rendering process according to the invention wherein the rasterized light pattern is prepared with the raster frame in the same orientation during all light source calculations.
Fig. 24 shows a result of a rendering process according to the invention wherein the rasterized light pattern is rotated randomly before the raster processing.
Fig. 25 shows the combination of the light source raster pattern of Fig. 17 with the reflection texture of the illuminated point, to yield the light for the point rendered.
Fig. 26 shows a image rendered according to the invention from the scene of Fig. 13.
Detailed Description In the systems and methods here described, an interactive image generating system communicates over a network with users to provide them with images rendered for scene data defining a three dimensional virtual "scene." The image generating system stores the scene data so as to make rendering more efficient, and the rendering process used is also improved by incorporating pipeline-style steps in the software or hardware without sacrificing image quality.
a. The Interactive System As best seen in Figure 1, a computer image generating and displaying system according to the invention comprises a geometry control system generally indicated at 3 connected with a number of geographically distributed visualization systems 5.
Each visualization system communicates over a link 15 via a data network, such as the Internet, with at least one user computer 13.
_ 7_ The geometric control system 3 comprises a number of individual geometry computer systems 7, shown in Figure 1 to be three systems, but potentially, and preferably much greater in number. The geometry control system 3 maintains and processes scene data which defines one or more three-dimensional scenes from which computer rendered images can be prepared and displayed to the various users 13.
The scene data may be distributed among the geometry computer systems 7 of the geometry control system 3 in a variety of ways. Preferably, each geometry system 7 has scene data for separate scenes, and each geometry computer 7 has access to every scene on the various geometry computer systems 7 via high-speed communications lines 9. When one of the geometry computer systems 7 alters the scene data, it advises the other geometry computer systems 7 of the changes to the scene data being made over communications lines 9, and each geometry system 7 is thus kept up to date.
The high speed data links 9 are preferably dedicated, hard-wired data links that transfer data at an extremely high rate, either on single channels or in parallel, as is well-known in the art. The data links connect all of the geometry systems 7, with each other, either directly or through other geometry systems 7.
Alternatively, the geometry system 7 can be in the same network location as a connected visualization computer 5.
The individual geometry computer systems 7 each communicate with the visualization computer systems 5 using further high speed data links 11. Data links 11 are preferably also dedicated hard-wired links which allow for a very high rate of data transmission, preferably on the order of at least about 10 megabits/sec, and more preferably above about a gigabidsec. The visualization systems 5 may be thousands of miles apart, either within a single country or across international boundaries, or , _ g_ may be clustered together, even in a single installation, where a group of users are close together. As a result of the distance and number of network hops potentially present, latency even in high-speed communication may be an issue when the communication is via a data network or some other link with an inherent time delay, e.g., a synchronous satellite link. To reduce latency, the individual geometry computers 7 are therefore preferably distributed geographically and located on a data network to be close thereon to the associated visualization computer systems 5, i.e., within at most 2 or 3 network hops of each other.
The visualization computers 5, in turn, are each connected with a respective user computer 13 by computer data network links 15. The linkage 15 may be through individual local area networks (LANs) but are most preferably through the Internet or cellular phone networks. The links 15 may also be telephone or cable TV links between the user computers 13 and the visualization computer 5. As a result, the links 15 to the various user computers 13 may not be completely separate links, but may overlap with each other or share some communication pathways, such as where all of the links 15 are Internet communications. Similarly, a number of visualization systems 5 may be tied in to the same local area network and communicate with user systems 13 on the network.
The individual visualization computer systems 5 each receives scene data from the geometry system 3 and produces an electronic image therefrom. The image represents a view of at least a portion of the scene defined by the scene data, prepared pursuant to data defining a point of view that corresponds to the specific user, and viewing parameter data for rendering the scene. The electronic image is preferably pixel data, but may be any other viewable image format, including analog images, such as cable TV signals, layered depth images, or any 2D or 3D image format.

The image which is produced by the visualization system 5 is transmitted over the data link 15 to the associated user computer 13. As best shown in Figure 2A, the visualization system 5 includes a communications connection or other hardware data linkage 17 communicating via high-speed data link 11 to one of the geomeriy computers 7 of the geometry system 3. Visualization system 5 also functions as a server, having communication connection 19 which links the visualization system 5 to a computer data network, generally indicated at 21. The data network preferably is the Internet, a wireless network, (e.g., cellular), or a broadband network, e.g., a cable network or DSL, but may also be a local area network ("LAN") or other type of computer data transmission network.
Each user computer 13 comprises the computer components familiar in the art, such as a computer CPU 23 and input devices for introducing instructions into the computer CPU, such as keyboard 27 and mouse 29. A display 31 is connected with the computer CPU 23 and displays images transmitted thereto in a form which is viewable by the human user thereof. The computer 13 may also be provided with a microphone input and/or an audio output system.
The user computer 13 also includes a communication connection 25 which is connected with the data network 21 for communicating with the associated visualization system 5. The nature of the network and the connection is preferably such that streaming serial images, e.g., VHS- or HDTV-quality video, can be transmitted to the user computer 13.
To decrease latency in the network link between the user and the visualization computer, the visualization computer 5 is desirably located as close as feasible in network proximity to the user computer 13. Although the system may advantageously be used with much larger network separation, a connection of less than 2 or 3 network hops is preferred for a real-time interactive system with streaming .
images. Ideally, however, one or no network hops are present between the user computer 13 and the visualization computer 5. Such a system would exist where the visualization computer 5 is located at the Internet Service Provider (ISP) for the user computer. The user there communicates with the ISP server and visualization computer over a very short link of a data network, such as local telephone line or a cable TV line, linking the user computer 13 with a visualization computer physically in or adjacent to, and wired to, the ISP server.
As one example of such a system, the user computer 13 comprises a very simple computer system that has the limited function of processing input from a joystick control thereon and transmitting instruction data derived therefrom over cable TV wiring to a server.
The server contains or is wired directly to a visualization computer which renders images from the joystick input data and from scene data, and then returns rendered images as real-time cable TV video over the cable TV line.
According to the preferred embodiment of the invention, the computer is equipped with a software package which allows the user to send control data to create or edit the scene without the need to transmit scene data itself. In other words, the user computer software receives input from one of the input devices thereon, and interprets the input as instructions or data configured to interface with software or hardware of the visualization computer. This interface results in a change in the scene data. The user computer 13 transmits control data, the interfacing data or instructions, to the image generating system via data communication connection 25 with the link 15. This control data is configured to influence the ultimate rendered image, and comprises data that is used to control or influence either the scene data itself or the viewing parameter data used by the image generating system to render the image.
Although it is less desirable from the standpoint of reducing data traffic load and the size of the application running on the user computer, the control data may include partial scene data to be incorporated into the scene data on the visualization computer.
The control data may also be one or more instructions to a software application operating on the visualization computer 5.
Where the control data is instructions to a software application running on the visualization system, the instructions may cause any of a variety of changes to the scene data, to the point-of-view data, or to the viewing parameters data. For example, the instruction may be a direct edit command for a visual object in the scene, such as to change its color, size, orientation or position. The instruction may similarly direct the visualization computer software to make a change in the point of view or the viewing parameter data. The instruction also can be a higher level instruction to a specialized application, such as to game software running on the visualization system, which would cause a substantive change in the scene, such as, for example, opening a door, pushing a rock off a cliff, or any other imaginable action in the virtual scene.
The instruction can also have a combined effect on the scene and viewing data, as, for example, where the instruction is to change the point of view, and this results in the movement of an avatar of the user in the scene which can be seen by the user himself or another user.
Because the user needs only the limited software package to communicate with the image system, there is no need for the user to acquire or download a large software package or to obtain special hardware such as graphics cards to perform the tasks performed by the image generating system to which the user is linked.

Preferably all users on the system can access a virtual scene in the same area thereof and make non-conflicting changes while the other users observe.
Conflicts in user-directed changes are processed by conventional methods of task sharing of data, which are well known in the art. In addition, a lockout function in the geometry system software prevents two users simultaneously changing the same object in the scene.
The scene data and the viewing parameter data are stored in the image generating system both in the visualization systems 7 and the geometric computer systems. However, since each visualization system 7 generates electronic images for only its user(s), the visualization system does not require the scene data in its entirety, but only that portion of the scene data that is relevant to the point of view of the associated user(s). The geometric system 3 therefore preferably transmits only relevant up-to-date scene data to each visualization computer 7 at a high frequency, and ensures that the subsets of scene data that are sent to the visualization computer 7 are the most current versions of those subsets, as the scene data is modified by the action of the many users of the system. Because the voluminous scene data is stored only in the image generating part of the overall system, the user computers need not support the large amount of memory or storage space needed to accommodate complex scene data.
The viewing parameter data defines how the scene is viewed, and includes, at a minimum, data that identifies a point of view of the user which the visualization computer 5 uses in rendering the electronic image. Subsequent changes in the point of view can also be entered by the data input devices, which may be the keyboard 27, the mouse 29, a joystick (not shown), or any of a variety of other input devices well known in the art. The viewing parameters may also include parameters that define the optics of a camera, which is simulated in the rendering of the view of the scene. Such parameters can be focal distance, depth of field, distortions due to camera lenses, or other general parameters such as graininess, filtering, etc.
Each visualization computer may support one or more user computers over the associated links 15. Consequently, the computational demands on a given visualization computer may be quite high while another visualization computer linked thereto by data link 15 is idle or less heavily loaded. Accordingly, the visualization systems are provided with load-sharing software that allows visualization systems to share the computational effort of rendering one or more images for a given user. The data processed and the resulting data are transferred between visualization computers by direct or indirect links between the visualization computers, or possibly via the high-speed links 9 of the geometric system 3. Algorithms for this type of load sharing are known in the prior art, and generally rely on running assessments of individual computer computational load with protocols for balancing the computation.
Similar load balancing of computation is also provided among the various geometry computer systems 7 to efficiently distribute computations involving the scene data. Since the geometry computers 7 are fully connected, directly or indirectly, to all other geometry computers by links 9, computational load can be readily distributed and balanced throughout geometry control system 3.
Streaming video images may be generated together with streaming audio. The audio portion of such a signal is generated by a sound engine hardware or software on the visualization computer. The resulting sound is produced based on the scene data, and possibly other sound recording files and is made audible over an audio system of the computer. The sound engine preferably provides a realistic acoustic effect, such as reverberation, echoes, etc., based on the scene data.

Similarly, the user computer may be fitted with tactile output devices that simulate the feel of an object in the scene. Devices of this type are currently available on the market, and these devices can simulate a variety of textures, such as sandpaper or ice.
In an alternate embodiment, shown in Figure 2B, the image generating system may provide rendered images to user computers 13 as a single frame at a time, with a turnaround time which is slower than a real-time interactive system. The user computers 13 are similar to those of the preferred embodiment, although slower communication software and hardware may be used therewith. If a change to the , scene is made by any of the user computers 13, the control data reflecting these changes is transmitted to the image generating system 30 over links 32 via a computer data network 34. The image generating system 30 automatically returns the rendered electronic image to the user computer 13 automatically responsive to the control data.
In such a system, a turnaround time for returning an image can be as much as 5 to 10 seconds, so the latency issue addressed in the real-time interactive system above with its geographical distribution does not arise. Consequently, the image generating system 30 of the alternate embodiment preferably combines both the geometric and visualization systems 3 and 7 in a single location and can be located at any distance from the users, preferably on a server computer system connected directly to the data network, which is preferably the Internet. Many users can access the visualization system, and their tasks are simply run in a queue.
The method by which any of the image generating systems renders an electronic image can be virtually any rendering system compatible with the turnaround time needed for the given application. For example, the visualization system can use graphics pipeline circuitry, such as Open GL cards, to create the image and then transmit the image to the user.
Particularly preferred, however, is the improved rendering system disclosed herein, which provides especially fast rendering, particularly beneficial in a real-time interactive system.
b. Organization of the Scene Data One of the features of the present system that is preferably used to enhance the speed of rendering is the organization of the scene data in a database stored in a computer data storage device or memory that allows rapid identification of the visual objects that are contacted by a ray or beam tracing through the scene space.
The scene data defines the light sources and objects in the three-dimensional scene. Light sources are defined as either parallel-ray directional lighting (such as sunlight or moonlight), point lighting, or as area light sources, which are two- or three-dimensional surfaces defined by one or more triangles or quadrilaterals.
The objects in the scene are defined as visual objects meaning anything that reflects or alters light that contacts it, including transparent objects.
Visual objects may be defined by any protocol known in the art. Visual objects are generally triangles (also referred to as primitives), tesselated surfaces (i.e., assemblies of triangles), or higher order objects, e.g., spheres, ellipsoids, or more geometrically complex sheets, such as NLTRBS surfaces or patches, which are non-planar surfaces defined by a function or a combination of functions. The triangular primitives are identified by the coordinates of their three vertex points.
Higher complexity objects are defined by the function that describes them, together with whatever point or points are appropriate. Also, where a visual object is a complex surface, the definition of the object also includes a normal vector to the surface of the object, defined either numerically or as a function.
Each visual object also has a set of additional parameters which further define its visual characteristics, e.g., a color, a transparency coefficient, a specular coefficient, a texture definition, and a reflectance identifier. The visual object may also be a layered object having a plurality of these parameters each representing a layer thereof.
The color of the object is preferably defined according to a common computer color definition in which three numbers represent, respectively, the red, green, and blue intensities of the color. The transparency coefficient defines how much light can pass through the object. This transparency coefficient ranges from 0 to 1, with 0 corresponding to complete transparency, and 1 corresponding to total opacity.
The specular coefficient is also a number from 0 to 1 which defines the specular quality of the visual\object, i.e., how mirror-like the surface is, with 0 indicating a completely diffuse surface and 1 indicating a mirror surface.
The texture is preferably defined by a code or tag pointing to a stored bitmap or image that can be mapped onto the object to give it an appearance of concrete, marble, sand, etc. The textures may be one-, two- or three-dimensional arrays.
Alternatively, the texture may be defined as procedural texturing, wherein a formula or algorithm is identified or defined which, when calculated or performed, generates the desired texture for the object.
The reflectance identifier is a code or tag that specifies which bi-directional reflectance distribution function ("BRDF") should be used for the surface of the object. The specific BRDF defines how the surface of the visual object scatters light as a function of the angle of the incoming light and the angle of viewing. The BRDF
will be discussed in greater detail below with regard to rendering.
The scene data may be created in a variety of ways. For one, it may be created by a user through one of the user computer systems, in which case, the user computer and/or the visualization computer is equipped with a communication software package which enables the user to generate and edit visual objects in the scene data stored in the image system. A number of appropriate interactive programs exist on the market which offer this function, including 3D Studio Max from AutoDesk, Inc.
Alternatively, the scene data can be created by an application developer on one of the visualization computers, on one of the geometry computers, or on a separate system and downloaded into the geometry control system.
The defined visual objects may be downloaded from the user computer to be placed in the scene, or they may be transmitted in more compact format. A
particularly effective reduction in the size of the transferred visual objects during creation or editing thereof is obtained by defining the texture of an object procedurally, i.e., defined as a function as opposed to a bit map. When the user defines a texture by a function, this function is transmitted to the image generating system instead of transmitting the bit map for the texture. The data defining a function for a texture such as marble or wood grain is generally small in comparison with the data for a color bit map of the actual texture. The transmission of the selected texture of an object by its functional definition therefore represents a substantial reduction in the amount of data that must be transferred from the user computer to the image system.
During or after creation of a scene by defining the various scene elements, it is advantageous according to the present invention to organize these scene elements in data storage so as to make it easier to render an image from the scene data.
This is accomplished using a database having a data structure that organizes the scene for more efficient storage and transmission, and, more importantly, for more efficient calculation of relevant scene elements to reduce the computation time needed for rendering the scene.
The pre-rendering organizing process is, illustrated by an example shown in Figs. 3 and 4. In Fig. 3, a very simple scene is shown, comprising a set of visible articles 35 composed of visual objects and distributed in the scene space 33.
Fig. 4 shows the same scene in a front elevational view.
To organize the scene data for the scene database, the entire scene volume, sometimes referred to as "the world" or the "the universe", is initially partitioned into a grid of cells, which are preferably cubic or rectangular boxes with sides at right angles to each other. These cells are preferably identified by a Cartesian-type coordinate system, with each cell identified by three non-negative integer coordinates.
The world space can be divided into as many grid cells as are desired.
For simplicity's sake, in the example of Fig. 3, the world or scene 33 is partitioned by midpoint bisecting cuts A and B into eight equal subdivisions or cells or boxes, four of which, 37a, 37b, 37c, and 37d, are seen in Fig. 4. It will be understood that another bisecting cut is made to partition the front half of the scene 39 from the back, although that cut and the four rearward cells are not visible in the view of Fig. 4.
After the scene is divided into the grid of cells, each of the cells is then analyzed by counting the number of visual objects that are found in it. If the cell is empty, or if it contains fewer than a subdivision threshold number of visual objects, usually a number between 10 and 100, no further analysis or subdivision is performed on that cell. If, however, the cell has more than the subdivision threshold number of visual objects therein, it is subdivided further by division into two along each dimension. Alternatively, subdivision into a number of cells greater than 2 in each dimension, i.e., a subdivision number of 3 or more, may also be used, instead of subdivisions by 2's. Subdivision in 2's is preferred, however, because it gives a computational advantage in a binary digital computer, where multiplication and division by 2 is a bit-shift operation.
Furthermore, where the individual box contains a large number of visual objects, e.g., more than 10,000, and these are distributed through the box, it is most probable that this box will be subdivided several more times before the count of visual objects in each sub-box is less than the threshold. Recording the repeated subdivision of the boxes is a waste of memory, and also makes processing of the scene data inefficient. Accordingly, when a cell contains a number of visual objects that is larger than a multiple subdivision threshold number, that cell is subdivided not by the subdivision number (i.e., 2) of partition in each dimension, but by an integer power of the subdivision number, here 2.
This process is illustrated in Figure 4. The space 33 is subdivided once into cells or boxes 37a, 37b, 37c, and 37d, and four boxes behind them that cannot be seen. Box 37b contains no geometry, so it is not subdivided any further. Boxes 37a and 37d contain some geometry; they are subdivided by two partitions in each dimension. Assuming for the sake of example that each of the sub-boxes 39a to 39h has fewer than the subdivision threshold number of visual objects, they are not subdivided further.
Sub-box 37c, however, contains a large number of visual objects which exceeds the multiple subdivision threshold, and these are distributed throughout the box 37c. This sub-box 37c is therefore subdivided in eight (8) sub-cells in each dimension, yielding ~3 (i.e., 512) sub-cells 41. Assuming, again, for the sake of the example, each of these sub-cells 41 contains fewer than the subdivision threshold number of visual objects, these cells 41 are not further subdivided. However, if the number of objects in any cell 41 exceeds the subdivision threshold number, subdivision of that cell 41 is repeated until no sub-cell therein contains more than the subdivision threshold number of objects.
The number of multiple subdivisions taken may be any number which is an integer power of the usual subdivision number. In the preferred embodiment, the subdivision number is 2, so the multiple subdivisions are 2M in each dimension, yielding 23M subcells. °The number M of multiple subdivision levels preferably depends on the number of visual objects in the box being subdivided.
Preferably, the multiple subdivision number M is calculated as directly proportional to the quantity:
logq (N/T) where N is the number of objects in the cell, T is the single subdivision threshold (the minimum number of objects in a cell to trigger a single subdivision), and q =
s3, where s is the usual subdivision number which in the preferred embodiment is 2.
Where a large number of objects are present, if the objects are bunched in a small region of the box, only single subdivision, not multiple subdivision, is used.
This type of scene data structure is referred to as a multiresolution virtual uniform subdivision (MVUS) data structure. The virtual subdivisions are called uniform because, due to the multiple subdivision number being an integral power of the subdivision number, the planes of subdivision of coarser levels of resolution always line up with a plane of subdivision of those cells that are divided in a finer grid. This can be seen in Figure 4, where partition plane C in box 39c aligns with partition plane D in multiply subdivided box 37c. Similarly, plane E in box 37d aligns with partition plane F. The spatial result of this is that each face of each box faces an integral number of adjacent box faces, which is advantageous for tracing rays or beams therein, as will become herein apparent.
As best shown in Figure 5, for efficient storage, the scene data that has been organized in an MVUS structure is loaded into a hash table 43. The hash table has m positions, each of which may have one or more records 45 therein, or no record. Only those boxes or cells that contain geometry are stored in the hash table 43, so if a cell is not found in the hash table 43, that cell is empty.
The individual record 45 contains a number of packets of information. At the very least, the record of a cell that has geometry in it contains coordinate fields i, j, and k which identify the box by its location in a Cartesian coordinate system of the MVUS
structure. The field l contains a number identifying the level of subdivision of the cell.
For example, in Figure 4, for box 37b, l=0. For boxes 39a to 39h, l=1. For boxes 41, l=3.
Data field n stands for the number of further subdivisions of the particular cell.
Where n is zero, this means that the box is not subdivided further. Where n is 1, as it is for boxes 37a and 37d of Fig. 3, there is one further level of subdivision.
For box 37c, n is 3.
The Cartesian coordinate system for the MVUS sub-cells is best illustrated in a two-dimensional scheme in Figures 6 to 8. Those of ordinary skill in the art will be able easily to appreciate the extension of the simplified 2-D example to a 3-D
space.

Each box is identified by the coordinates of the corners of the box which,are lowest in value. At each level of subdivision, the coordinates are multiplied by the subdivision. Thus, in Figure 6, the four level-zero cells are identified by indices i and j as (0,0), (0,1), (1,0) and (1,1). It will be understood that for a more complex scene, the level-zero grid usually contains a much larger number of grid cells, and the set of coordinate values will be from (0,0,0) to the number of cells in each dimension, e.g., (1000, 800, 100).
'At a single further level of subdivision, as best seen in Fig. 7, the coordinate scale of the lower-resolution level is multiplied by 2 to yield the coordinates of the smaller higher-resolution sub-cells so created. This multiplication by 2 is particularly efficient in a digital system because, in binary, this is accomplished by a simple bit shift to the left one place. In the example shown, the coordinates of the level 1 boxes are (2,2), (2,3), (3,2) and (3,3). For multiply-subdivided boxes at level 3, as seen in Figure 8, the coordinates are multiplied by 24 = 16. The indices of the level 3 boxes range from (8,0) to (15,7).
The location of the stored data record in the hash table for a given box identified by its three spatial coordinates i, j, k, and by its level number l is determined by the formula:
hash address = (cii+c~j+ckk+cl l) modm where .-- C~ ~ Ck -- CI -- m to loo looo lo,ooo The resulting hash table allows for ready access to the scene data in a given cell.

The scene data is obtained using the data as shown in figures 9 and 10. The hash table record 45 of an undivided cell, i.e., an end node of the MVUS
subdivision, comprises the i, j, k, l, and n fields, as discussed above, with n being zero.
After the n data field, the record contains data identifying the visual objects in the cell.
In the simplest type of scene data arrangement, the data identifying the objects is a pointer identifying a location in a separate stored database which is a list of visual objects organized by cell. In the preferred embodiment, however, several ways of defining the contents of a cell are available, including pointers to objects in a visual object list, but also data identifying oriented nested objects formed of many visual objects.
Accordingly, in the preferred embodiment, the hash table record 45 includes a scene graph pointer field 47, a series of object pointer fields 49, and a nested MVUS
pointer field 51.
Abject pointer fields 49 each contain a pointer to a location in a separate visual object database 53 of visual objects in the scene. The location in the visual object database 53 holds a record 55 defining a visual object which is defined as has been mentioned previously.
The scene graph pointer field 47 contains a pointer that identifies a node of a scene graph data structure 57 (see Fig. 9) of a stored database of visual objects. The scene graph 57 has a number of nodes 59 each of which holds a record. 'The node record 59 includes transform matrix data 61 and a virtual instance pointer field 63.
The term "virtual instance" refers to an occurrence in the scene of an object which is a fairly complicated assembly of visual objects that occur repeatedly in a scene, such as, for example, a seat in a stadium. Rather than store many copies of the data defining the visual objects that make up the repeated object, a single copy of the data defining the object is stored in a library of objects 67, and this is used whenever there is a virtual instance of the object. The transform matrix is used to transform the coordinates of the object as defined in the library into the actual coordinates of the particular virtual occurrence in the cell.
The light list 65 identifies the light sources in the database of visual objects 53 that are relevant, i.e., that shine light onto objects in the cell. The light list also identifies any parallel beam light in the scene by the vector of its direction of propagation.
The nested cell pointer field of the hash table record contains a pointer to the a beginning of a segment of a nested cell structure database. A nested cell is used when a complex object comprising a substantial number of visual objects occupies only a small portion of the box it is in. In such a situation, illustrated in Fig.
11, most of the cell 69 is empty, except for the object 71. A box or boundary volume 73 is defined by an algorithm that identifies the object 71 and circumscribes the object 71 with a box, which is then rotated and shrunk in to fit closely around the object 71. This boundary volume 73 is then parsed using the MVUS procedure, with subdivision partitions applied as to a larger scene as was discussed above. The nested MVUS hash table is stored coupled with data defining a transform matrix that expresses the orientation of the nested box relative to the larger box and that is used to transform a ray or beam in the larger box into the internal coordinates of the nested boundary volume 73.
Once the ray or beam is converted to the coordinate system of the nested MVUS, the nested MVUS itself is traversed exactly as the larger surrounding MVUS, and a nested MVUS may contain further nested MVUS boxes with respective transformation matrices which axe processed similarly. ' c. Rendering the electronic image In the preferred embodiment, the image generating system creates pixel data which constitutes a view of the scene defined by the scene data, in the MVUS
data structure as seen from the point of view of the user as specified by viewing parameters data. The MVUS data structure allows for very fast determinations of ray tracing, and also beam tracing, to identify what visual objects are visible from a given point of view during rendering.
To illustrate the rendering method of the preferred embodiment, a view of a three-dimensional scene to be rendered is shown in Figure 13. It will be understood that the illustration is only a graphical representation of the computations performed by the rendering system, as the objects of the scene are in reality mathematically defined by electronically stored data.
The scene space 77 encloses a number of opaque and transparent visual objects. An opaque chair 79 is adjacent a transparent glass table 81 on which an opaque magazine 83 is resting. The scene is illuminated by a single rectangular light source 85 in the ceiling 87. The image to be rendered is from point of view 89, with the direction of viewing defined by normal vector 91.
The image for which pixels are to be derived is delineated by a frame 93 which defines therein a grid of pixels of the desired resolution for the image. For each pixel, at least one ray and preferably between 4 and 64 rays are extrapolated from the point of view 89 through the pixel and into the scene space 77. The directions of these rays are governed by the viewing parameters data, and their purpose is to determine which point of which visual object in the scene is seen from the point of view 89 through the specific pixel. Once each ray's contact point is determined, the rendering process determines what color it will appear to have, and then assigns the pixel that color, or, if more than one ray is used, combines the colors of the several rays through the pixel, and assigns the averaged color to the pixel.
The process of tracing a ray in an MVUS data structure is illustrated in Figs.
12A and 12B .
The first step 94 in a ray trace is to initialize the data areas used in the trace.
The priority queue is emptied, and the nearest contacted object is set to infinity, meaning that any object contacted by the ray will replace it. If the ray does not strike the "world MVUS", i.e., the largest MVUS of the scene (decision 95), the process moves to A (finished). If a collision object has been found at finish, that object is returned as the object struck by the ray. Otherwise, a zero (black) or the MVUS
background color is the returned value. The background color value may also be a portion of a bit map of the image of the wall of the MVUS grid which the ray hits, such as a view of mountains in the distance or a flat representation of an adjacent scene.
If the ray does hit the world MVUS, then the first cell of contact is calculated and the world MVUS is listed in the priority queue. Control then passes to the decision block 97 for queue processing.
At the first pass, the queue holds only the world MVUS. In subsequent passes, the queue is loaded by loop iteration 9~ with all the MVUS's in the current cell. In either case, the queue is checked for entries. If empty, the process goes to A
and the currently identified nearest collision object is returned as the contacted object. If the queue is not empty, the nearest MVUS in the queue is compared with the collision object. If the nearest collision object is nearer (decision 99), the currently identified nearest collision object is returned as the object struck by the ray.

If the MVUS is nearer, it is traversed by a traversal subroutine 100, which is detailed in the flowchart of Fig. 12B.
In the subroutine 100, the first contacted cell is identified, and the i, j, k, and l coordinate values of the contacted box are then used to calculate the hash address for the MVUS hash table. If the hash table does not have an entry for the specified cell (decision 101), and the lower resolution cell is not known (decision 102), then the traversal pulls back to a lower resolution of the MVUS grid, and continues to pull back until a cell in the hash table is found or the lowest resolution of the MVUS is reached (a simple bit-shifting operation where the subdivision number is 2).
If the lowest resolution cell is reached, decision 103 checks if the ray has left the MVUS
and decision 104 checks if a preset distance limit on the distance of travel for the ray has been reached. If either is the case, the traversal subroutine returns. If neither is the case, the ray is extended to the next cell of the MVUS, and that cell is checked for presence in the hash table (decision 101).
If the current cell is in the hash table, it is checked whether its geometry is at a higher resolution. If so, the process shifts to a higher resolution, i.e., to smaller cells, with the current cell coordinates adjusted by the next level of subdivision of the current cell (again a simple bit-shifting operation where the MVUS subdivision number is 2), and the hash table is checked (101). If the geometry is at the present resolution, the visual objects are put in a list to be returned to the main routine (105).
The ray is checked for whether it has reached the end of the MVUS or whether the predetermined maximum distance has been reached. If not, the next cell that the ray contacts is set as the current cell, and the subroutine returns. Otherwise, the subroutine 100 returns with the same cell as current cell.
_ 28_ On return to the main routine, the ray is checked for whether it has left the nearest MVUS (decision 106). If it has, that MVUS is removed from the queue (107).
Then, after B, if geometry was returned by the traversal operation, a collision check is performed (108). If the ray collides with an object (decision 109), that object is made the nearest collision object if it is closer than the presently identified nearest object (110). Control then proceeds to C, where the current cell is scanned for any nested MVUS's that the ray hits (iteration 98).
The result of the process of the flowchart is the identification of the visual object in the scene that is struck by the ray.
The determined contacted object is then checked for transparency. If the object is transparent, its light filtering attributes are incorporated in a queue of transparent objects through which the ray passes, and the ray is traced and extended through the visual object as before, until it strikes another object, which is in turn examined for opacity and specularity, and processed as was the first.
If the point struck by the ray is on an opaque object, the color of that point is the color of the ray for determining the pixel image data. The color of a point is derived as the sum of two color values, (1) the color of light reflected from the point from direct illumination from light sources in the scene, plus (2) the color of light that is indirectly illuminating from the point from other illuminated objects.
The color of the reflected direct illumination is derived from the indicated BRDF for the visual object, the angle of viewing of the visual object at the point, and the angle of incident light at the point.
The example of Fig. 13 illustrates the process for determining the color of reflected direct illumination for a point on an opaque visual object. One of the rays used in the rendering, ray 95, is shown extending through a pixel in frame 93 and contacting the seat of chair 79 at point 97. The sole source of illumination for point 97 is light source 85 in the ceiling 87 of the scene 77. To assess how much light arrives at point 97 from this light source 85, a beam trace is made using a pyramidal frustrum beam generally indicated at 99, starting at point 97. This beam trace is used, in combination with graphics pipeline circuitry which is "tricked" into doing the calculations, to generate a raster pattern which represents the light source 85 as seen from point 97. The pipeline graphics hardware in the disclosed embodiment is preferably Open GL accelerator chips made by NVIDIA, but more efficient systems can be envisioned which are specifically designed and dedicated to this particular process.
The frustrum 99 is defined by four expanding edge lines 111, 113, 115 and 117 starting at point 97. The frustrum 99 is pyramidal, with a quadrilateral, preferably square, cross-section, as indicated by cross-section 119. The length of the side of the square of the cross-section of frustrum 99 in the area of the light source 85 is at least the maximum diagonal dimension of the quadrilateral light source 85 as seen from point 97.
As the frustrum is being defined, random angular orientation is imparted to the frustrum 99. The benefit of this random rotation may be seen in Figures 23 and 24.
Figure 23 shows a rendering wherein the light sources are calculated using a 16x16 rasterized light source calculation. With a uniform orientation of the frustrum, the rasterized representation of the light source intensity results in aliasing of the light intensity, which is ultimately visible as a rippling in the shadows. In contrast, Figure 24 shows an image rendered using a 32x32 rasterization of the light source with a random rotation applied to the frustrum, resulting in a uniform shadow without ripples.

The rasterizing of each light source preferably makes use of graphics pipeline circuitry, taking advantage of certain computational steps that these circuits can perform very efficiently. Pipeline graphics circuits perform basically two types of functions very rapidly: (1) combining a raster image with a bit map, a process sometimes referred to as texture mapping; and (2) converting mathematical definitions of triangles into pixels (usually referred to as scan conversion).
The present invention takes advantage of both these special capabilities.
As best seen in Figure 14, the hardware of the visualization system includes a digital portion 121 which receives the control data from, and transmits the image to, the user computer, as has been discussed above. The digital portion 121 also stores the 3D scene data indicated schematically at 123. The digital portion 121 is also wired to a hardware circuit 125, which is preferably a circuit comprising a set of pipeline graphics components, arranged to parallel-process data received from the digital portion 121.
The frustrum 99 is defined as described above by software running on the computer portion 121 of the visualization system. The computer portion 121 projects the frustrum as a beam through the MVUS scene space to ascertain what visual objects lie within the frustrum. The process for extending a beam is similar to that of extending a ray through the MVUS space. For a beam, however, some additional computation is required to identify all objects in its volume.
From the starting point, the apex of the frustrum, the dominant direction of the beam, i.e., the x, y, or z direction in which the beam proceeds most quickly is determined. The cross-section of the beam perpendicular to the dominant direction becomes the advancing front of the beam as it extends through the MVUS space.
Every cell of the MVUS that falls wholly or partly in the beam is opened until the terminal sub-cell is reached, and a list of intersected cells containing geometry is kept as the beam goes through the MVUS structure.
When the list of cells falling partly or completely in the beam is obtained, the visual objects therein are collected and assembled into a "mini-scene"
consisting only of the visual objects in the frustrum 99 and the light source being beam-traced.
Each light source is in the list of visual objects, and is defined as one or more triangles or quadrilaterals with three or four vertex points, an illumination intensity value, and, optionally, a pointer to an emission distribution function ("EDF") that defines the light source light emission pattern. The EDF is preferably a bit map that defines the pattern of the light source emissions. For example, an incandescent light bulb is brighter at its center where the filament is glowing. Similarly, fluorescent light panels usually show bands of brightness over the fluorescent bulbs therein. The EDF defines a bit map that expresses this variation in intensity.
The digital portion 121 transmits this mini-scene data to the hardware pipeline 125, together with the light source definition and its EDF, and a definition of the reflection point, i.e., the point 97 for which the amount of incident light is being determined and its reflective parameters (i.e., its BRDF). From this mini-scene data, the hardware 125 generates a raster light pattern, or, alternatively, a single luminance value, which is transmitted back to the digital portion 121. This raster which represents the direct illumination from light source 85 that is reflected from point 97 along the ray 95.
In the pipeline circuit process performed by the hardware 125, the first step is defining a raster pattern which is a silhouette of the light source 85 in the frustrum 99 in a pure black and white pattern, as shown in Figure 15. The raster pattern is preferably fairly coarse, e.g., 16 x 16 or 32 x 32 pixels. Greater raster resolution is preferably used, however, where the light source is at a very oblique angle relative to the cross section of the frustrum.
After the black and white silhouette is defined, the pipeline circuit combines the black and white silhouette (Figure 15) with the relevant EDF, which is a bitmap that is part of the mini-scene data, and which is mapped onto the visual object of the light source. The combination of the EDF texture and the silhouette results in a raster pattern, such as is shown in Figure 16, which represents the view of the light 85 from point 97 without any intervening obstructions.
The next step in the creation of the light-source raster pattern is that any transparent objects which intervene between the light source 85 and the point 97 are incorporated in the raster pattern. In the present example, this means the green glass top 81 of the table. The transparency of the object is defined as a fraction of the red, green, and blue components of light passing therethrough. Each of the occluded pixels of the raster pattern by this transparency fraction data is multiplied to yield the raster pattern shown in Figure 17. Alternatively, the transparency may be a more complex transparency texture, defined procedurally or as a bitmap, mapped against the raster pattern.
The next step in the formulation of the light-source raster pattern is determining the occlusion of the light source by any opaque objects, and blacking out any occluded pixels in the raster pattern. Such a raster pattern is shown in Figure 18, wherein the occlusion of the opaque magazine 83 is blacked out of the raster pattern.
The resulting raster pattern represents the view of the EDF textured light source, as viewed from the reflective point through any transparent objects, andlor blocked by any intervening opaque objects. This raster represents the light incident at the reflective point from the light source.

Once the incident light raster is created, the pipeline hardware 125 combines the incident light raster with the BRDF for the visual object to determine the light diffusely reflected from the reflective point.
The BRDF functions are derived and used based on a classical model of diffuse reflection. Classically, reflection is modeled using a polar coordinate system, illustrated in Fig. 19. According to this model, a visual object 130 has a reflecting point generally indicated at 131, and light L from a modeled light source 133 bounces off this point 131 to the viewed line of sight to point 131, such as, e.g., line R. The intensity and color of the reflected light is expressed as a reflectance value which is the fraction of incident light that is reflected along the line of sight.
Some materials are anisotropic, such as, e.g., velvet, meaning that they have reflectance qualities that change depending on the angle that the material is rotated about a normal axis N. Isotropic materials on the other hand reflect the same intensity regardless of how they are rotated. As best seen in Fig. 19, to model reflectance of an anisotropic material, a direction reference vector D is defined for the material, and the rotative angle 9L;gnc of the incoming light vector L and the rotative angle Aview of the reflectance viewing vector R are used to determine the intensity of the reflection.
Modeling of isotropic material is simpler. Effectively, 6L;ghc is considered to be zero degrees at all times, and 6vlew is the angle between the incident light vector and the viewing vector.
In either type of material though, the azimuth angle cp Light of the incoming light vector L and the azimuth angle cp,,;ew of the reflected light being viewed are used to model the reflection of light.
Based on the angles cp Light, BView~ and cp v;ew (and 6L;ghc for anisotropic materials), the BRDF derives a reflectance value ranging from zero (no reflection) to one (100% reflection). An illustration of a usual distribution of reflectance values for a given incident light vector L is provided in Fig. 20. The shape of the surface shown is defined by the magnitude of the reflectance vector R over all the polar coordinate directions. Normally, maximum reflectance is centered around a specular angle or mirror angle M, where cp Light = ~P view and ~Light = eview - 180°.
This maximum reflectance is seen as a bulge in reflectance values KR generally indicated at 135.
There is also usually a bulge in values called backscatter or retroreflection generally indicated at 137, where 6Lignt = evieW ~d ~p view ~ ~P Lignt~ For most angles, though, the graph is a generally hemispherical middle portion 139, representing a generally uniform scattering of light, as in a Lambertian surface.
The BRDF function as used in the present invention follows this general model, but is preferably defined as data in the opposite direction, i.e., as a function of reflectance values for a given line of sight which vary over the field of various values Of eLight ~d ~ Light~
This function is illustrated in Fig. 21A. A single point of view 141 sees a point on visual object 143 along line of sight V. A set of polar coordinate data is developed for reflection of light to the viewpoint dependent on the 9Light and cp Light for the incident ray. This data is illustrated as hemisphere 145. Each point on the hemisphere 145 represents a color value which is the fraction of white light passing through that point which would be reflected off the reflective point at the center of the hemisphere to be seen at the point of view 141.
For example, light arriving along ray L passes through point 147 of hemisphere 145. This point 147 is in a general region of the hemisphere 145 in which diffuse reflection is higher, i.e., around the mirror angle to the angle V of viewing.
The ray therefore falls in a more light-colored spot of the hemisphere, indicating greater reflection of this particular ray. Rays striking at different points would be reflected at lesser degrees, which is indicated by the darker color of the hemisphere at those points. For example, a low angle incident Iight ray passing through point 147 contacts a darker point on henusphere 145, meaning that the reflection of light of that ray to point of view 141 would be minimal.
This model also allows for modeling of tinted reflection, since each point on hemisphere 145 is a color value.
To make the hemisphere 145 more readily usable in the present method, especially where graphics pipeline circuitry is used, the BRDF data for the point of view is converted not into a hemisphere, but into a hemicube 151 around the reflective point 153 on visual object 154 as illustrated in Figure 21B. As with the hemisphere 145, each colored point of hemicube 153 represents the fraction of a ray of white light extending through that point and reflecting off point 153 that would be reflected to be viewed along line V.
In the rendering process, the hemicube 151 is used in the preparation of the rasterized data for illumination of a point by a light source. The BRDF
identifier for the visual object struck by the rendering ray, such as the seat of the chain 79 in Fig.
13, identifies a set of hemicube texture maps, each representing roughly one degree of azimuth of the point of view relative to the visual object viewed. The angle of viewing the visual object and the BRDF identifier are used to select the proper BRDF
hemicube texture map, and the resulting texture bitmap sheet is included as part of the mini-scene. When an anisotropic BRDF is required, both the azimuth and rotative angle of viewing are used to index into a set of textured BRDF hemicube maps to obtain the proper bitmap.

Figs. 22A and 22B show how this sheet is formed. Fig. 22A is the surface of the hemicube of Fig. 21B unfolded to lie in a plane. Fig.~22B is the organization of the parts of the unfolded hemicube as they are transmitted to the pipeline graphics circuitry. The format herein is particularly suitable for OpenGL graphics cards.
In the miniscene data, the hemicube is assembled around the reflecting point, as in Fig. 21B. The frustrum 99 for the raster image of the light source extends through the hemicube 151, defining a relevant portion thereof, generally indicated at 155. The graphics circuitry combines the texture of this part of the hemicube with the assembled raster pattern of all other components of the miniscene (the raster pattern of Fig. 18) to yield the reflected raster pattern of Fig. 25.
This BRDF combined raster pattern represents the direct light that is reflected from the point 97. This raster may be returned to digital portion 121 as pixel data from the pipeline circuitry 125, or it may be returned as a single data value derived as the sum of all of the raster pixels.
In addition to reflection of direct light, the rendering process also takes into account reflection of other objects in the scene. The calculation of light reflection from these objects is performed by the digital portion 121 of the visualization computer. This calculation is preferably performed based generally on the Phong reflection model, which is well known in the prior art, and is discussed in, for example Phong, "Illumination for Computer Generated Pictures", Communications of the ACM (CACM), Volume 18, Number 6, pp. 311-317, June 1975, which is herein incorporated by reference. The result is combined with the result of the light source rasterization to yield the color returned by the ray. If the object is specular (i.e.
specular value not zero), the ray is extended in the mirror angle direction relative to the incoming ray striking the object. This specular ray is then extended onward through the MVUS grid to see if it strikes any other objects, either in the current box or outside it. If the ray actually leaves the box, the MVUS method discussed above is used to determine whether the ray enters another box containing visual objects. If the reflected ray hits an object either in the current box or another box, that object is also checked for transparency and specularity, and the process repeats.
If the object that the reflected ray strikes also is specular, the ray is again reflected at a specular angle and extended to the next object that it encounters. This specular reflection and tracing back continues for a limited number of reflections, e.g., four or five, because these reflections generally are of reducing intensity and brightness, and consequently have diminishing impact on the illumination of the first visual object struck by the ray. Each specular reflection is recorded as the color fraction of the specular light reflected and is kept as a running multiple of the reflections to this point. This reflected color fraction is derived from the BRDF field of the specular object, where the color of the relevant point on the relevant hemicube is recorded as the specular reflection fraction for the object.
Ultimately, the ray will hit a non-specular object, will hit a light source, will leave the scene space, or the number limit on reflections will be reached, and the ray will stop. If the ray hits an object, the color of the object is determined based on the method discussed above for determining reflection for light from light sources, and this resulting color is multiplied by the running multiple of the reflections, with the resulting product being the determined light color for the specular reflection.
Similarly, where a light source is struck, the color of the point of the light source is multiplied by the running reflection multiple, and the result is returned as the reflection at the point of other objects. Otherwise, a zero (black) is returned, which has no effect on the color of the point.

The color from reflected objects is added to the color of reflected light sources, and the result is the color for the ray, which is then assigned as the pixel image data for the pixel in question, and combined (usually averaged) with other rays through the pixel to provide the color.
The ray tracing and rendering process is repeated for all pixels of the desired image, resulting in a realistic image, such as is shown in Fig. 26, the rendered image for the scene of Fig. 13.
It will be understood that, according to an aspect of the invention, the rendering of images is preferably done by the above rendering process and using MVUS scene data. However, the visualization system of the invention may also use a different rendering method. For example, an existing graphics pipeline system could be used to render the image transmitted to the user. Similarly, a conventional organization of scene data, such as a voxel grid or octree structure, can be used.
In addition, although MVUS scene data databases have herein been shown used with the foregoing rendering process, it should be understood that computer-accessible data storage devices storing MVUS scene databases can be used advantageously with other image generation processes and systems. Generally, the MVUS design can be used advantageously in any image generating system wherein the computer or computers extend a path (e.g., a ray or beam) through a scene and determine if it contacts any visual objects in the scene, or wherein the computer or computers locate objects within some spatial neighborhood around a point or location in the scene space.
For example, some rendering methods rely on a global illumination process to render images of the scene. As an example of such a process, a point viewed in the rendered image is identified and then its illumination is determined by casting 360 degrees by 90 degrees of rays or beams around the point and determining what visual objects or light sources those rays encounter. If the ray or beam hits another visual object, the illumination of that point is determined by the same 360 by 90 degrees casting of rays or beams. The process is repeated for any visual objects that those rays or beams hit, until each traced-back illumination ray or beam meets a light source or leaves the scene, whereupon the total illumination of the viewed point is calculated.
The casting of the rays or beams and the identification of the visual objects that lie therein in such a method can be efficiently accomplished using an MVUS scene database.
Also, a process relying on forward tracing may rely on an MVUS scene database. In forward tracing, each light source for the scene is identified, and the light projected therefrom is traced along a path through the scene. When the path contacts an object, reflected light is projected from the contacted point, and by repeating this process for all points struck by the illumination from the light sources the illumination of all visual objects in the scene is determined. Again, the MVUS database structure improves the efficiency of identifying the visual objects in the scene that lie in the paths of the light.
Still other rendering processes not mentioned here can make advantageous use of an MVUS scene database.
The MVUS database of the preferred embodiment is stored on a computer-accessible data storage device, as is well known to those of skill in the art.
Preferred for this data storage is random access computer memory (RAM), since it allows for rapid recovery of data by the rendering computer system. The RAM may be any such computer memory devices, including SRAM or DRAM, and the database may be stored in a single computer system, or in two or more linked computer systems.

In the preferred embodiment, the MVUS database is stored using RAM
memory, and access to this is accelerated using a content addressable memory (CAM) operating as a high-speed cache lookup of scene data without need to access the hash table. The CAM memory is generally smaller than the data stored in the hash table 43, so the CAM stores only the most recently accessed entries in the hash table, operating as a cache, i.e., less accessed data is dropped from the CAM as it fills up.
The CAM is accessed by a content key made up of the values i, j, k and 1 for the MVUS
partition being addressed. Such a CAM enhances the speed of the access to the scene data by an order of magnitude.
As a further option, where the computer system has a CAM memory that is large enough to store all the entries of the MVUS hash table 43, then all of the hash table entries may be loaded into CAM memory. In that case, the rendering application program will rely exclusively on the CAM to look up the MVUS database scene graph pointers and other pointers in the hash table entry, and the hash table 43 itself may be dispensed with.
Alternatively, because the MVUS data structure reduces the time needed to access the scene data, MVUS databases can be used advantageously in slower data storage hardware, such as, e.g., computer disk drives.
It should also be understood that the terms used herein should be considered terms of description rather than limitation, as those with this specification before them will be able to make variations and modifications to the disclosed methods and systems without departing from the spirit of the invention.

Claims

WHAT IS CLAIMED IS:

1. A dynamic media system comprising:
an image generating computer system having electronic storage storing scene data for a three dimensional scene;
a user computer connected with the image generating computer system over a data communications link and transmitting control data to said image generating computer system over said link;
the image generating computer system receiving the control data and automatically rendering electronic image data from the control data and the scene data, said electronic image data representing an image of the scene seen from a point of view, the image generating computer system transmitting the pixel image data to the user over the communications link; and the user computer receiving the electronic image data and displaying the electronic image data at the user computer as a visual image viewable by a user.

2. The system of claim 1, wherein the communication link is a computer data network.

3. The system of claim 2, wherein the computer data network is the Internet.

4. The system of claim 2, wherein the computer data network is a local area network.

5. The system of claim 2, wherein the computer data network comprises a wireless cell

6. The system of claim 1, wherein the communication link is a cable TV link.

7. The system of claim 1, wherein the communication link is a telephone connection.

8. The system of claim 1, wherein the image generating computer system comprises a geometric system storing the scene data.

9. The system of claim 8, wherein the geometric system comprises a plurality of separate geometric computer systems each storing therein a respective version of the scene data.

10. The system of claim 8, wherein the image generating computer system further comprises a visualization system connected by a communication link to the geometric system and receiving the scene data therefrom , said visualization system rendering the image from the scene data received from the geometric system and transmitting the rendered image to the user system.

11. The system of claim 10, wherein the communication link between the geometric system and the visualization system is a dedicated data link.

12. The system of claim 1, wherein the control data comprises an instruction selected from the group consisting of instructions directing a change of the point of view, instructions directing a change in an optical parameter of viewing the scene, instructions to modify the scene data, and instructions to application software controlling the scene data.

13. The system of claim 10, wherein the control data comprises an instruction selected from the group consisting of instructions directing a change of the point of view, instructions directing a change in an optical parameter of viewing the scene, instructions to modify the scene data, and instructions to application software controlling the scene data.

14. The system of claim 13, wherein the visualization system transmits the instruction to the geometric system.

15. The system of claim 14, wherein, where the instruction is an instruction directing a change of the point of view or of the optical parameter of viewing, the visualization system does not transmit the instruction to the geometric system.

16. The system of claim 1, wherein the image data is pixel data.

17. The system of claim 1, wherein the image data is transmitted to the user computer as compressed pixel data.

18. The system of claim 17, wherein the pixel data is compressed using a JPEG
protocol.

19. The system of claim 1, wherein the image data is transmitted as a video stream.

20. The system of claim 19, wherein the communications link is a cable TV link carrying the video stream.

21. The system of claim 19, wherein the video stream is in a raw video format.

22. The system of claim 19, wherein the video stream is compressed video.

23. The system of claim 22, wherein the video is compressed using an MPEG
protocol.

24. The system of claim 19, and said video stream including audio data configured to be converted to audible sound by the user computer.

25. The system of claim 24, wherein the audio data is derived from the scene data.

26. The system of claim 1, wherein the user computer has software therein providing an interactive application to the user for modifying the scene data.

27. A dynamic media system comprising:
a geometric control system having data storage, said data storage holding scene data representative of visual objects of a three dimensional scene;
a plurality of visualization systems connected with and communicating with the geometric control system; and a plurality of user computers each communicating with a respective visualization system via a link through a computer data network;
each of said user computers having a display for displaying images and a data entry device for entering data to the user computer;
the geometric control system transmitting the scene data to the visualization system;
the user computers each transmitting control data to the visualization system associated therewith;
the visualization systems each rendering from the scene data a respective electronic image representing a frame view of the scene from a point of view specific to the associated user computer; and the visualization systems transmitting the image to the associated user computer, said user computer displaying the image.

28. The system of claim 28, and said visualization systems being separated from one another or the network by at least one node, and each being connected with the respective user computer with less than three network nodes.

29. The system of claim 28, and said visualization systems each being connected with the geometric control system by a dedicated data link.

30. The system of claim 29, wherein each of the visualization systems communicates with at least two of the user computers.

31. The system of claim 27, wherein the geometric system comprises a plurality of geometric host computer systems connected with each other, each of the geometric host computer systems communicating with at least one of the visualization systems.

32. The system of claim 31, wherein the geometric host computer systems each store therein a respective version of the scene data, said version comprising a data portion of recent scene data and a data portion of older scene data.

33. The system of claim 32, wherein the recent scene data for the geometric host computer system corresponds to an area in the scene proximate the point of view of the user computers associated with the visualization system communicating therewith.

34. The system of claim 27, wherein the computer data network is the Internet.

35. The system of claim 27, wherein the computer data network is a local area network.

36. The system of claim 28, wherein the computer data network is a wireless cell communication network.

37. A dynamic media system comprising:
a user computer having a connection to a data communication link;
a data entry device connected with the user computer and providing for a user the capability to enter data to the user computer; and a display device connected with the user computer and displaying images therefrom;
the user computer receiving input data from the data entry device and forming control data from the input data, said control data being configured to cause a change in data selected from the group consisting of scene data defining a 3D scene from which an image can be desired, and data defining optical parameters used in deriving said image;
the user computer transmitting the control data over the data communication link;
said user computer receiving electronic image data over the data communications network, said electronic image data being produced dependent on said scene data and said control data and representing a view of the scene according to the viewing parameters; and the user computer causing the display device to display an output image derived from the electronic image data thereon.

38. The system of claim 37, wherein the data communication linlc is the Internet.

39. The system of claim 37, wherein the data communication link is a local area network.

40. The system of claim 37, wherein the data communication network is a wireless cell system.

41. The system of claim 37, wherein the data communication network is a cable TV
link.

42. The system of claim 37, wherein the data communication network is a telephone connection.

43. The system of claim 37, wherein the control data comprises electronic instructions configured to be received and acted on by a software application operating in a computer image system connected with the data communication link.

44. The system of claim 37, wherein the control data is configured to cause a change in position or visual attributes of at least one visual object which is part of the scene data.

45. The system of claim 37, wherein the control data comprises data configured to cause alteration in data representing a point of view for the view of the scene.

46. The system of claim 37, wherein the control data comprises data configured to cause alteration in data representing an optical parameter for the view of the scene.

47. The system of claim 46, wherein the optical parameter is selected from the group consisting of focal length, depth of field, camera lens distortion optics and atmospheric transparency.

48. The system of claim 37, wherein the data entry device is a keyboard, a mouse, a joystick or a touch sensitive pad.

49. The system of claim 37, wherein the electronic image data is pixel data.

50. The system of claim 37, wherein the electronic image data is compressed pixel data.

51. The system of claim 50, wherein the image data is compressed according to a JPEG protocol.

52. The system of claim 37, wherein the image data is transmitted a part of a video stream.

53. The system of claim 52, wherein the communication link is a cable TV link.

54. The system of claim 52, wherein the video stream is in a raw video format.

55. The system of claim 53, wherein the video stream is compressed video.

56. The system of claim 55, wherein the video is compressed using an MPEG
protocol.

57. The system of claim 52, and said video stream including audio data configured to be converted to audible sound by the user computer.

58. The system of claim 57, wherein the audio data is derived from the scene data.

59. The system of claim 37, and said user computer receiving the data input, transmitting the control data, and displaying the output image at points in time in close enough proximity for real-time interactive access to the scene data from the user.

60. The system of claim 37, wherein the user computer receives the data input, transmits the control data, receives the image pixel data, and displays the output image within less than about 15 seconds.

61. The system of claim 37, wherein the user computer receives the data input, transmits the control data, receives the image pixel data, and displays the output image within less than about .05 seconds.

62. A dynamic media system comprising:
an image computer system storing scene data representing a three dimensional scene and viewing data representing at least one viewing parameter for viewing the scene, the viewing data including data defining a point of view; and a communication connection linking the image computer system to a data communication link;
the image computer system receiving control data through said communication link from a remote source, said control data being configured to interact with the image computer system so as to cause alteration in the scene data or so as to cause formulation or modification of the viewing data;
the image computer system automatically rendering an electronic image from the scene data responsive to receiving the control data, said image data representing a view of the scene dependent upon the viewing data;
the image computer system transmitting the electronic image over the communication link in a format effective to reach the source of the control data and in a form such that a viewable image can be derived from the electronic image and displayed.

63. The system of claim 62, wherein the data communication link is the Internet.

64. The system of claim 62, wherein the data communication link is a local area network.

65. The system of claim 62, wherein the data communication link is a wireless cell communication network.

66. The system of claim 62, wherein the data communication link is a cable TV
link.

67. The system of claim 62, wherein the control data comprises an instruction directing a change in the point of view data.

68. The system of claim 62, wherein the control data comprises an instruction selected from the group consisting of instructions directing a change of the point of view, instructions directing a change in an optical parameter of viewing the scene, instructions to modify the scene data, and instructions to application software controlling the scene data.

69. The system of claim 62, wherein the image computer system is connected with a second data link and receives the scene data over the second data link.

70. The system of claim 69, wherein the control data comprises an instruction selected from the group consisting of instructions directing a change of the point of view, instructions directing a change in an optical parameter of viewing the scene, instructions to modify the scene data, and instructions to application software controlling the scene data.

71. The system of claim 70, wherein the image computer system transmits the instruction over the second data link in a format configured to be used to affect the scene data received by the image computer system.

72. The system of claim 69, wherein the first and second data links comprise a connection through the Internet.

73. The system of claim 69, wherein the second data link is a dedicated data link.

74. The system of claim 62, wherein the electronic image is transmitted to the user computer as pixel data.

75. The system of claim 62, wherein the electronic image is transmitted to the user computer as compressed pixel data.

76. The system of claim 75, wherein the pixel data is compressed using a JPEG
protocol.

77. The system of claim 62, wherein the electronic image is transmitted as part of a video stream.

78. The system of claim 77, wherein the data communication link is a cable TV
connection.

79. The system of claim 77, wherein the video stream is in a raw video format.

80. The system of claim 77, wherein the video stream is compressed video.

81. The system of claim 80, wherein the video is compressed using an MPEG
protocol.

82. The system of claim 77, wherein the video stream includes audio data configured to be converted to audible sound.

83. The system of claim 82, wherein the image computer includes a sound engine which derives the audio data from the scene data and the point of view data.

84. The system of claim 62, wherein the image computer system operates so that between the receiving of the control data and the transmitting of the electronic image rendered therefrom, a period of time of less than about 15 seconds elapses.

85. The system of claim 62, wherein the image computer system operates so that between the receiving of the control data and the transmitting of the electronic image rendered therefrom, a period of time of less than about .05 seconds elapses.

86. The system of claim 62, and the image computer system storing at least some of the scene data organized in a multiresolution virtual uniform subdivision data structure, the image computer system using the multiresolution virtual uniform subdivision data structure to render the electronic image.

87. The system of claim 62, and the scene data including data defining a light source and a visual object;
said image computer system rendering the pixel image data, said rendering including beam tracing the light source from a location on the visual object in the scene.

88. The system of claim 87, wherein the image computer system further includes graphics pipeline circuitry;
said graphics pipeline circuitry generating rasterized illumination data corresponding to said beam tracing.

89. A method of preparing an electronic image, said method comprising:
creating scene data defining a three dimensional scene, said scene data including data defining at least one area light source and at least one visual object;
receiving control data over a data communication link;
rendering, responsive to the control data, an electronic image from scene data, said electronic image representing a view of the scene from a point of view;
and transmitting the electronic image over the data communication link in a form such that the electronic image can be received and displayed as a viewable image.

90. The method of claim 89, wherein the rendering being based on optical parameter data representing optical parameters for the view of the scene, the optical parameters including data defining the point of view; and the control data comprising data configured to cause a change in the scene data or to cause a change in the optical parameter data.

91. The method of claim 90, and the control data comprising data configured to cause a change in the scene data.

92. The method of claim 91, wherein the control data is an instruction to a software application generating or modifying the scene data.

93. The method of claim 92, wherein the software application is run on a first computer system and the rendering step is run on a second computer system.

94. The method of claim 93 wherein the control data is received at a second computer system, and the method further comprises transmitting the control data from the second computer to the first computer.

95. The method of claim 94, and further comprising receiving the scene data from the first computer at the second computer.

96. The method of claim 89, and further comprising receiving at a computer system data transmitted from a separate geometric system, the rendering step being performed at least in part by the computer system.

97. The method according to claim 96, wherein the control data comprises data configured to cause a change in the optical parameter data.

98. The method of claim 97, wherein the optical parameter data is stored on the computer system, and the data changing the optical parameter data is used by an application running on the computer system to change the optical parameter data.

99. The method of claim 89, wherein said rendering includes tracing a beam from at least one point on the visual object and deriving for said point rasterized data representing the area light source as seen from the point.

100. The method of claim 99, wherein said beam tracing includes defining a beam frustrum enclosing the area light source.

101. The method of claim 100, wherein said defining of the frame further includes applying a rotational orientation to the frustrum.

102. The method of claim 101, wherein the rotational orientation is a random angular displacement.

103. The method of claim 99, wherein the rasterized data is derived using dedicated raster-generating circuitry.

104. The method of claim 103, wherein the raster-generating circuitry performs a plurality of independent operations each on a respective portion of the circuitry.

105. The method of claim 104, wherein, while the rendering is being done by performing one of the independent operations, other calculations are performed using the other independent operations.

106. The method of claim 105, wherein the other calculations are part of a process of deriving rasterized data from a second light source, or for a different point on the same visual object or another visual object.

107. The method of claim 99, and the beam trace including forming subscene data representing a subscene including the area light source and at least some of the visual objects.

108. The method of claim 107, and the rasterized area light data is derived using dedicated raster-generating circuitry.

109. The method of claim 108, and said deriving rasterized data further comprising providing the subscene data to the dedicated circuitry.

110. The method of claim 109, wherein the raster-generating circuitry performs a plurality of independent operations each on a respective portion of the circuitry.

111. The method of claim 110, wherein, while the rendering is being done by performing one of the independent operations, other calculations are performed using the other independent operations.

112. The method of claim 111, wherein the other calculations are part of a process of deriving rasterized data from a second light source, or for a different point on the same visual object or another visual object.

113. The method of claim 89, and said creating of scene data including organizing data representing the area light source and visual object in a multiresolution virtual uniform subdivision data structure.

114. The method of claim 113, and said rendering including deriving for at least one point on the visual object rasterized data representing the area light source as seen from the point; and the point on the visual object being identified by tracing a ray from the point of view using the multiresolution virtual uniform subdivision data structure.

115. The method according to claim 99, and directional reflective coefficients of the visual object being stored as reflection texture data;
said rendering step including combining the rasterized data with an appropriate portion of the reflection texture data.

116. The method according to claim 115, wherein the reflection texture data is stored as planar texture data representing a surface of a box with planar walls around the point.

117. The method according to claim 107, the scene data being organized at least in part in a multiresolution virtual uniform subdivision (MVUS) data structure;
said subscene data being derived using the MVUS data structure for rays proceeding from the point to the area light source.

118. A method for rendering an electronic image representing a view from a point of view of a virtual scene defined by scene data comprising a plurality of visual objects and at least one area light source, said method comprising:
identifying one of the visual objects, or a portion thereof, which is visible from the point of view, and tracing a beam from the visual object or portion, said beams enclosing therein the light source;
deriving from said beam tracing illumination data defining the amount of light reaching the object or portion thereof; and determining from the illumination data an amount of light from the area light source that is reflected from the visual object or portion thereof.

119. The method of claim 118, wherein the scene data includes a second area light source, and the method further comprises tracing a beam enclosing the second light source and deriving illumination therefrom.

120. The method of claim 118, wherein the scene data is organized in an MVUS
data structure and the visual object is identified by traversing the MVUS data structure.

121. The method of claim 118, wherein said beam tracing includes defining a frustrum that encloses the area light source.

122. The method of claim 121, wherein said frustrum has an angular rotational displacement incorporated therein prior to deriving the illumination data.

123. The method of claim 121, and further comprising creating mini-scene data defining visual objects in the frustrum.

124. The method of claim 123, wherein the deriving of illumination data is performed by pipeline graphics circuitry, and the mini-scene data is transmitted thereto.

125. The method of claim 118, wherein the deriving of the illumination data is performed by pipeline graphics circuitry.

126. The method of claim 125, wherein the illumination data represents a color value.

127. The method of claim 118, wherein the illumination data represents a color value.

128. The method of claim 118, wherein the step of determining of reflected light is performed as part of the derivation of illumination data.

129. The method of one of claims 118 to 128, wherein the determining of reflected light is performed using a reflectance function that varies dependent on an angle of incidence and an angle of viewing at the point of light being modeled.

130. The method of claim 129, wherein the reflectance function combines a reflectance texture map representing varying reflectance at the point based on said angles with the illumination data to produce the determination of the reflected light.

131. The method of claim 130, wherein the reflectance texture map is stored electronically in a form corresponding to a mapping of the varying reflectance of light on a surface of a box structure around the object or portion.

132. The method of one of claims 118 to 131, wherein the amount of reflected light from the light source is combined with a color value for other light reflected off the object.

133. The method of one of claims 118 to 132, wherein the electronic image is a pixel data image.

134. The method of one of claims 118 to 133, wherein an illumination texture is mapped onto the area light source.

135. A method of rendering an electronic image representing a view from a point of view of a virtual scene defined by scene data comprising a plurality of visual objects, said method comprising:
identifying one of the visual objects or a portion thereof which is visible from the point of view;
deriving for the object an angle of viewing thereof;
deriving illumination data for light illuminating said object or portion, said illumination data including directional data indicating an angle of incidence of the light on the object and an intensity value;

combining the intensity value with a reflection coefficient derived from the angle of incidence and the angle of viewing of the visual object.

136. The method of claim 135, wherein the reflection coefficient is part of a texture map.

137. The method of claim 136, wherein the texture map is structured as a box surrounding a reflective portion of a visual object.

138. The method of claim 137 wherein each discrete point of the box represents the amount of light passing through said discrete point that would be reflected along a line of sight.

139. The method of one of claims 135 to 138, wherein the texture map is combined with the illumination data by graphics pipeline circuitry.

140. The method of one of claims 135 to 139, wherein the illumination data includes a raster pattern.

141. A method for rendering an electronic image representing a view of a virtual scene as viewed from a point of view, said method comprising:
providing scene data defining the virtual scene in a grid format for efficient tracing of the passage of light therethrough; and tracing a path for light in said virtual scene using said scene data.

142. The method of claim 141, and said path of light that is traced being a beam.

143. The method of claim 141, and said beam being a generally pyramidal frustrum.

144. The method of claim 141, and said path being a ray.

145. The method of claim 140, and said grid format comprising an MVUS data structure.

146. The method of claim 145, and said MVUS data structure including a subdivision of the virtual scene, said cell subdivision containing at least partially thereon a pointer to a second MVUS structure.

147. The method of claim 146, and said second MVUS structure having a transformation function associated therewith based on a relative orientation of the second MVUS to the first MVUS.

148. The method of claim 146 or 147, wherein the second MVUS is in a library of objects for placement in virtual instances.

149. The method of one of claims 141 to 146, said grid structure containing cells being subdivided into subcells in each of three dimensions by a subdivision number where the cell contains a number of visual objects that exceeds a first subdivision threshold value.

150. The method of one of claims 141 to 149, wherein the grid structure has cells which are subdivided into subcells in each of three dimensions in a number which is an integer power of the subdivision number where the cell contains a number of visual objects that exceeds a second multiple subdivision threshold value.

151. The method of one of claims 141 to 150, wherein cells of the grid structure are recorded in a hash table, said hash table containing said cells based on addresses calculated from grid coordinates of the cell or subcell and a level number representing the level of subdivision thereof.

152. The method of one of claims 141 to 151, wherein said tracing is performed using an algorithm as disclosed herein.

153. An apparatus performing the method of one of claims 118 to 151, said apparatus including a digital computer.

154. The apparatus of claim 153, said apparatus further comprising pipeline graphics circuitry.

155. A computer-readable storage device storing thereon instructions causing a computer to perform the method of one of claims 118 to 151.

156. A method for storing in a computer-readable database scene data comprising electronic visual object data defining visual objects each having associated therewith electronic data identifying a location of the visual object in a virtual space, said method comprising:
a) determining from the electronic visual object data a number representing a count of the how many of the visual objects have locations within a cell in the virtual space;
b) comparing said number of visual objects with a predetermined threshold subdivision value and/or a predetermined threshold multiple subdivision value;
c) responsive to a determination that the number of visual objects in the cell exceeds the predetermined threshold subdivision value but not the predetermined threshold multiple subdivision value, performing a subdivision operation representing a partitioning of the cell in at least one dimension thereof into an integer subdivision number s of sub-cells that are equal in size to each other and together constitute the cell;
d) responsive to a determination that the number of visual objects in the cell exceeds the predetermined threshold multiple subdivision value, performing a multiple subdivision operation representing a partitioning of the cell in at least one dimension thereof into a number of sub-cells, which number is an integral power, s n, of subdivision number s, said sub-cells being equal in size to each other and together constituting the cell;
e) responsive to a determination that the number of visual objects in the cell does not exceed the predetermined threshold subdivision value, storing data identifying the visual objects in the cell in the database so that the visual object data of said visual objects can be accessed by a computer based on coordinates identifying said cell; and f) repeating steps (a) to (d) for each of the sub-cells created in steps (c) or (d), wherein each sub-cell becomes the cell in step (a) for which the number of visual objects is counted.

157. A method according to claim 156, wherein the operations representing a partitioning of the cell are of partitioning the cell in at least two dimensions thereof into s or s n sub-cells that are equal in size to each other and together constitute the cell.

158. A method according to claim 156, wherein the operations representing a partitioning of the cell are of partitioning the cell in three dimensions thereof into s3 or s3n sub-cells that are equal in size to each other and together constitute the cell.

159. A method according to claim 156, wherein the subdivision number is 2.

160. A method according to claim 157, wherein the subdivision number is 2.

161. A method according to claim 158, wherein the subdivision number is 2, and where the subdivision operation yields eight sub-cells constituting the cell, and wherein the multiple subdivision operation yields 2 3n sub-cells of the cell.

162. A method according to claim 156, wherein said cell is identified by integer coordinates thereof, and each sub-cell thereof is identified by at least one coordinate derived by multiplying at least one of the coordinates of the cell by the subdivision number to obtain a product, and adding a non-negative integer number less than the subdivision number to the product as an indication of displacement from an edge of the cell.

163. A method according to claim 157, wherein said cell is identified by integer coordinates thereof, and each sub-cell thereof is identified by at least two coordinates derived by multiplying at least two of the coordinates of the cell by the subdivision number to obtain products, and adding a respective non-negative integer number less than the subdivision number to each product as an indication of displacement of the sub-cell from edges of the cell.

164. A method according to claim 158, wherein said cell is identified by three integer coordinates thereof, and each sub-cell thereof is identified by three coordinates derived by multiplying the coordinates of the cell by the subdivision number to obtain three products, and adding a respective non-negative integer number less than the subdivision number to each product as an indication of displacement of the sub-cell from edges of the cell.

165. A method according to claim 156, wherein the value of n in the multiple subdivision operation varies dependent on the number of visual objects in the cell.

166. A method according to claim 164, wherein the value of n in the multiple subdivision operation is calculated as directly proportional to the quantity:
where N is the number of objects in the cell, T is the subdivision threshold value, and q = s3.

167. A method according to claim 162, 163 or 164, wherein the data identifying the visual objects is stored so as to be accessed by the coordinates of the cell and a level number representing the number of subdivisions by the subdivision number between the cell and the first cell to which step (a) was applied in the method.

168. A method according to claim 167, wherein the data identifying visual objects is stored in a hash table structure accessed based on the coordinates and level number of the cell.

169. A method according to claim 167, wherein the data identifying visual objects comprises a stored pointer data value that points to another location in electronic data storage at which the visual object data can be located.

170. A method according to claim 167, wherein the data identifying visual objects is accessed by a content access memory using a key based on the coordinates and level number of the cell.

171. A method according to claim 156, wherein the visual object data of said visual objects is stored in a scene graph stored in the data storage device, and the data identifying the visual objects is a stored pointer value identifying at least one location in the scene graph.

172. A method according to claim 156, wherein the visual object data of said visual objects is stored in RAM memory.

173. A method according to claim 156, wherein one of the visual objects stored is identified by a pointer to another database of visual objects prepared according to the present method couple with data identifying a transform matrix that spatially orients cells of the second database of visual objects relative to the coordinates of the cell.

174. A method according to claim 156, wherein, when the operation of multiple subdivision is performed, a record is stored that represents the integral power n of the multiple subdivision.

175. A computer system for storing in a computer-readable data storage device a database of scene data comprising electronic visual object data defining visual objects each having associated therewith electronic data identifying a location of the visual object in a virtual space, said system comprising:
means for determining from the electronic visual object data a number representing a count of the how many of the visual objects have locations within a cell in the virtual space;
means for comparing said number of visual objects with a predetermined threshold subdivision value and/or a predetermined threshold multiple subdivision value;
means for performing, responsive to a determination that the number of visual objects in the cell exceeds the predetermined threshold subdivision value but not the predetermined threshold multiple subdivision value, a subdivision operation representing a partitioning of the cell in at least one dimension thereof into an integer subdivision number s of sub-cells that are equal in size to each other and together constitute the cell;
means for performing, responsive to a determination that the number of visual objects in the cell exceeds the predetermined threshold multiple subdivision value, a multiple subdivision operation representing a partitioning of the cell in at least one dimension thereof into a number of sub-cells, which number is an integral power, s n, of subdivision number s, said sub-cells being equal in size to each other and together constituting the cell;
means for storing data identifying the visual objects in the cell in the database so that the visual object data of said visual objects can be accessed by a computer based on coordinates identifying said cell, said means for storing storing said data responsive to a determination that the number of visual objects in the cell does not exceed the predetermined threshold subdivision value, and means for providing to the means for determining, means for comparing, means for performing and means for storing each of the sub-cells created, said means for determining, means for comparing, means for performing and means for storing processing each sub-cell as the cell for which the number of visual objects is counted.

176. A system according to claim 175, wherein the operations representing a partitioning of the cell are of partitioning the cell in at least two dimensions thereof into s or s n sub-cells that are equal in size to each other and together constitute the cell.

177. A system according to claim 175, wherein the operations representing a partitioning of the cell are of partitioning the cell in three dimensions thereof into s3 or s 3n sub-cells that are equal in size to each other and together constitute the cell.

178. A system according to claim 175, wherein the subdivision number is 2.

179. A system according to claim 176, wherein the subdivision number is 2.

180. A system according to claim 177, wherein the subdivision number is 2, and where the subdivision operation yields eight sub-cells constituting the cell, and wherein the multiple subdivision operation yields 2 3n sub-cells of the cell.

181. A system according to claim 175, wherein said cell is identified by integer coordinates thereof, and each sub-cell thereof is identified by at least one coordinate derived by multiplying at least one of the coordinates of the cell by the subdivision number to obtain a product, and adding a non-negative integer number less than the subdivision number to the product as an indication of displacement from an edge of the cell.

182. A system according to claim 176, wherein said cell is identified by integer coordinates thereof, and each sub-cell thereof is identified by at least two coordinates derived by multiplying at least two of the coordinates of the cell by the subdivision number to obtain products, and adding a respective non-negative integer number less than the subdivision number to each product as an indication of displacement of the sub-cell from edges of the cell.

183. A system according to claim 177, wherein said cell is identified by three integer coordinates thereof, and each sub-cell thereof is identified by three coordinates derived by multiplying the coordinates of the cell by the subdivision number to obtain three products, and adding a respective non-negative integer number less than the subdivision number to each product as an indication of displacement of the sub-cell from edges of the cell.

184. A system according to claim 175, wherein the value of n in the multiple subdivision operation varies dependent on the number of visual objects in the cell.

185. A system according to claim 177, wherein the value of n in the multiple subdivision operation is calculated as directly proportional to the quantity:

~log q(N/T)~
where N is the number of objects in the cell, T is the subdivision threshold value, and q = s3.

186. A system according to claim 181, 182 or 183, wherein the means for storing stores the data identifying the visual objects so as to be accessed based on the coordinates of the cell and a level number representing the number of subdivisions by the subdivision number between the cell and the first cell to which subdivision or multiple subdivision was applied by the system.

187. A system according to claim 186, wherein means for storing includes data storage comprising a hash table structure in which the data identifying visual objects is stored so as to be accessed based on the coordinates and level number of the cell.

188. A system according to claim 186, wherein the data identifying visual objects comprises a stored pointer data value that points to another location in electronic data storage at which the visual object data can be located.

189. A system according to claim 186, wherein the means for storing stores the data identifying visual objects so as to be accessed by a content access memory using a key based on the coordinates and level number of the cell.

190. A system according to claim 175, wherein the means for storing stores the visual object data of said visual objects in a scene graph stored in the data storage device, and the data identifying the visual objects is a stored pointer value identifying at least one location in the scene graph.

191. A system according to claim 175, wherein the means for storing includes RAM
memory storing the visual object data of said visual objects.

192. A system according to claim 175, wherein one of the visual objects stored is identified by a pointer to another database of visual objects coupled with data identifying a transform matrix that spatially orients cells of the second database of visual objects relative to the coordinates of the cell.

193. A system according to claim 175, and further comprising means for storing for the multiply subdivided cell a record that represents the integral power n of the multiple subdivision when the operation of multiple subdivision is performed.

194. A memory for storing scene data for access by a computer program being executed on a computer system, said scene data comprising data defining a plurality of visual objects each having a respective virtual location in a scene space, said memory comprising:
a plurality of cell data entries stored in said memory, each of said cell data entries corresponding to respective cell of the scene space, said cell being defined by coordinates thereof and a subdivision level value;
the cell data entries being stored in said memory so that each cell data entry is accessed based on the coordinates and level value of the associated cell;

said cell data entries each comprising data identifying the visual object or objects having a virtual location within the cell;
said cells being configured such that no more than a threshold subdivision number of visual objects have locations in any cell.

195. A memory according to claim 194, wherein the cells are defined as volumes of the scene space defined by integer boundaries of a coordinate grid, or as subdivisions derived therefrom, said subdivisions being derived by portioning a larger cell in at least one spatial dimension thereof by a positive integer subdivision number s of equally spaced partitions.

196. A memory according to claim 195, wherein the subdivisions are derived by partitioning in at least two dimensions.

197. A memory according to claim 195, wherein the subdivisions are derived by partitioning in three dimensions.

198. A memory according to claim 195, wherein the subdivision number s is 2.

199. A memory according to claim 196, wherein the subdivision number s is 2.

200. A memory according to claim 197, wherein the subdivision number s is 2.

201. A memory according to claim 195, wherein cells that have therein a number of visual objects that exceeds a multiple subdivision threshold number of visual objects are multiply subdivided by partitioning the cell in at least one dimension thereof into a number of sub-cells equal in size to each other, said number being an integral power, s n , of subdivision number s.

202. A memory according to claim 201, wherein the integer power n of multiple subdivision varies dependent upon the number of visual objects in the cell.

203. A memory according to claim 201, wherein the cell entry for a cell that is multiply subdivided includes data recording the value of the integer power n of multiple subdivision thereof.

204. A memory according to claim 194, wherein the cells are defined as volumes of the scene space defined by integer boundaries of a coordinate grid, or as subdivisions derived therefrom such that:
cells that have therein a number of visual objects that exceeds a subdivision threshold number of visual objects, being subdivided by partitioning a larger cell in each of three spatial dimensions thereof by a positive integer subdivision number s, or an integral power (s n) thereof, of equally spaced partitions so as to yield s3 or s3n subdivisions thereof, and cells that have therein a number of visual objects that exceeds a multiple subdivision threshold number of visual objects are multiply subdivided by partitioning the cell in three spatial dimensions thereof into a number of subdivisions equal in size to each other, said number being an integral power of subdivision number s.

205. A memory according to claim 204, wherein the subdivision number s is 2.

206. A memory according to claim 204, wherein the coordinates of each cell are:
a) coordinates of the associated grid position thereof; or b) coordinates derived from the coordinates of a cell that was subdivided to yield the cell.

207. A memory according to claim 206, wherein the coordinates are derived by multiplying the coordinates of the subdivided cell by the number of cells that the subdivided cell was subdivided in each dimension.

208. A memory according to claim 195, wherein the level number of each cell represents the number of subdivisions into s partitions needed to reach the cell from the grid coordinate level, which is subdivision level zero.

209. A computer system having a memory according to one of claims 195 to 208, said computer system further comprising a tracing application program executable for identifying any of the visual objects of the scene data that lie in a defined region of scene space.

210. A computer system according to claim 209, wherein said defined region of scene space is a path therethough.

211. A computer system according to claim 209, wherein said defined region of scene space is a generally pyramidal beam extending therethough.

212. A computer system according to claim 209, wherein said defined region of scene space is a generally linear ray extending therethough.

213. A method of rendering an image from scene data, said method comprising:
providing to a computer system access to a memory according to one of claims 195 to 208; and accessing said memory so as to identify any of the visual objects of the scene data that lie in a defined region of scene space.

214. A method according to claim 213, wherein said defined region of scene space is a path therethough.

215. A method according to claim 213, wherein said defined region of scene space is a generally pyramidal beam extending therethough.

216. A method according to claim 213, wherein said defined region of scene space is a generally linear ray extending therethough.