WO2002080773A1

WO2002080773A1 - Augmentet reality apparatus and ct method

Info

Publication number: WO2002080773A1
Application number: PCT/US2002/010663
Authority: WO
Inventors: Gabor Fichtinger; Ken Masamune; Andrew M. Bzostek; Russell H. Taylor
Original assignee: Johns Hopkins University
Priority date: 2001-04-05
Filing date: 2002-04-05
Publication date: 2002-10-17

Abstract

An image overlay system to assist surgical interventions inside a conventional CT scanner is disclosed. The overlay system is mounted non-invasively on the gantry of the CT scanner. The system includes a flat LCD overlay display, a half-mirror, a mechanism to aid the placement of surgical devices, and an adjustable passive holding attachment mounted on the CT scanner that supports the LCD display and half-mirror. In a pre-operative calibration process, the display, half-mirror and imaging plane of the scanner are spatially registered by imaging a calibration object. Following the calibration, the patient is brought into the scanner, and slice images are acquired and sent to the overlay display. Looking at the patient through the half-mirror, the CT slice image appears to be floating inside the patient in correct size and position. This vision enables the physician to see both the surface and the inside of the patient at the same time, which can be used in guiding a surgical intervention.

Description

AUGMENTET REALITY APPARATUS AND CT METHOD

FIELD OF THE INVENTION

The present invention relates to surgical instruments, and, more particularly, to an image overlay system to assist surgical interventions inside conventional cross-sectional imaging systems.

BACKGROUND OF THE INVENTION

Multiple solutions have been developed to track surgical devices and combine auxiliary information with medical image data on a computer screen to assist surgical interventions. Since the end of the 1980's, significant research has been devoted to three-dimensional ("3D") medical images for surgical simulation and planning. Ordinarily, these 3D images are shown on a flat two- dimensional ("2D") display on the wall of an operating room. The fundamental drawback of these systems is that the operating surgeon must look away from the operation field, observe the display, then turn his or her attention back to the operation field and use hand-eye coordination to navigate a surgical device. In this arrangement, the surgeon must mentally register the computer generated 2D image with the anatomy of the patient seen in the surgical field, and use hand-eye coordination to execute the surgical plan. Therefore, the actions of the surgeon are not truly objective, primarily because the images and the patient are not accurately registered together.

To achieve better comprehension of 3D medical images, augmented reality displays are becoming more popular in the medical field. For example, one research group named the Roberts' Group has developed a frameless, image-overlay system for a stereoscopic microscope for neurosurgery. Another research group named the Fuchs' Group has developed an augmented reality system under ultrasound image guidance therapy. In these systems, medical images are visualized using a stereoscopic display, e.g., a polarized shutter system with two small monitors located on the user's head. However, there are potential errors in alignment and spatial registration that can occur in using this binocular system approach.

Other known systems have different problems that diminish their ability to be used with medical image data to assist surgical interventions.

DiGioia and others have developed a system that requires cumbersome calibration and both the patient and operating surgeon to be instrumented with tracking devices to ensure proper registration of image, patient, and viewer at all times.

Masamune and others have developed a stand-alone 2D image overlay mechanism for CT and MRI images, but with no physical attachment to the imaging device. The perceived location of the overlay image is independent of the viewer's location. Masumune's system is not intra-operative and it requires post-imaging registration between the patient and the image overlay device. The system uses multiple image slices and modalities, but it does not contain a surgical placement mechanism.

Stetten and others have developed a 2D image overlay system coupled with an ultrasonic transducer. The ultrasonic transducer is rigidly fixed with respect to the image overlay device to preserve registration. The system provides real-time imagery, but it does not allow for multiple image directions and locations, and it does not include a surgical instrument placement mechanism.

Each of the currently known systems is affected by one or more of the following problems: (1) the patient and surgeon are both instrumented with tracking devices; (2) there is a lack of ability to match intra-operative images into the physical patient; (3) they require cumbersome calibration and quality control; (4) they have a lack of an ability to use arbitrary image planes; and (5)they lack a surgical device placement mechanism.

SUMMARY OF THE INVENTION

The present invention solves the foregoing problems by displaying patient slice images so that they appear to be physically positioned with respect to the patient at the point at which they were taken and by facilitating the placement of surgical devices into the patient via a surgical instrument placement mechanism.

Accordingly, it is an object of the present invention to provide an intra- operative visualization system that allows simultaneous viewing of the surgical field as seen by the surgeon and the patient's internal anatomy. It is also an object of the present invention to provide an image overlay system that provides accurate depth perception and overlay of image slices on the patient's body. It is a further object of the present invention to provide a system that makes a CT image slice "appear" in the exact same location where it was acquired, while the patient is still in the CT scanner.

The present invention is a simple, safe and inexpensive image overlay system to assist surgical interventions inside a conventional CT scanner or other cross-sectional imaging system, such as a magnetic resonance imaging ("MRI") system, positron emission tomography ("PET"), or single photon emission computed tomography ("SPECT") system. The overlay system of the present invention is mounted non-invasively on the gantry of the CT scanner. The image overlay system includes a flat liquid crystal display ("LCD"), a half- mirror, and a passive holding attachment that supports the display and mirror and provides seven degrees-of-freedom for proper placement of the display and mirror in the CT scanner or other cross-sectional imaging system scanner. The image overlay system also includes a mechanism to facilitate placement and insertion into a patient of a surgical device.

A pre-operative calibration process is used to spatially register the display, half-mirror, and imaging plane of the scanner using a rigid calibration object. Following the calibration, the patient is brought into the scanner, and images acquired are sent to the overlay display. Looking at the patient through the half-mirror, the CT slice image appears to be floating inside the patient in correct size and position. This vision enables the surgeon to see both the surface and the inside of the patient at the same time, and can be used by the surgeon in guiding a surgical intervention. Surgical devices are navigated in the visual field of view. Invasive devices are visible up to the point they enter the body. If the physician wishes to see the portion of the device that has been inserted into the body, then a new image slice needs to be acquired and displayed on the overlay, provided the given imaging modality is able to visualize the surgical device. (For example, CT scanner can visualize metal needles.) The perceived location of the overlay image is independent of the viewer's location. Thus, the surgeon and patient are not instrumented with any type of tracking devices.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGURE 1 is a schematic drawing of the basic physical layout of the image overlay system of the present invention.

FIGURE 2 is a schematic drawing of the image overlay system of the present invention in which multiple CT slices are obtained by vertical translation of the display to compensate for organs that are larger than one slice.

FIGURE 3 is a schematic drawing of the image overlay system of the present invention in which multiple CT slices are obtained by horizontal translation of the mirror to compensate for organs that are larger than one slice.

FIGURE 4 is a schematic drawing of the image overlay system of the present invention in which image panning by horizontal translation of the display is achieved to compensate for a body that is too large to fit on the display.

FIGURE 5 is a schematic drawing of the image overlay system of the present invention in which patient interference with the system is resolved by a coordinated tilting of the mirror and the display.

FIGURE 6 is a schematic drawing of the image overlay system of the present invention in which tilted CT slice images are obtained using a tilting of the CT gantry.

FIGURE 7 is a schematic drawing of the image overlay system of the present invention in which an overlay of synthetic (re-sampled) angled CT slices are obtained by tilting the display.

FIGURE 8 is a schematic drawing of the image overlay system of the present invention in which an overlay of synthetic (re-sampled) angled CT slices are obtained by tilting the mirror.

FIGURE 9 is a schematic drawing of the image overlay system of the present invention in which a robot surgical device placement mechanism is remotely controlled by a joystick control.

FIGURE 10 is a schematic drawing of the image overlay system of the present invention in which a robot surgical device placement mechanism is remotely controlled by a personal computer. FIGURE 11 is a schematic drawing of the image overlay system of the present invention in which the surgical device placement mechanism is a mechanical device that is manually actuated by a surgeon.

FIGURE 12 is a schematic drawing of the image overlay system of the present invention in which the surgical device is placed by the surgeons free hand.

FIGURE 13 is a schematic drawing of the image overlay system of the present invention in which a tracking device is used to assist the surgeon in placement of the surgical device by free hand.

FIGURE 14 is a schematic drawing of the image overlay system of the present invention in which a tracking device is used with a mechanical surgical device placement mechanism that is manually actuated by the surgeon.

FIGURE 15 is a schematic drawing of the image overlay system of the present invention in which a tracking device transmits positioning information to be visually superimposed on the virtual overlay image to assist the surgeon in placing a surgical device by free hand.

FIGURE 16 is a schematic drawing of the image overlay system of the present invention in which a tracking device transmits positioning information to be visually superimposed on the virtual overlay image to assist the surgeon in placing a surgical device using a mechanical placement device.

FIGURE 17 is a schematic drawing of the image overlay system of the present invention in which data from an encoded instrument placement mechanism is visually superimposed on the virtual overlay image to assist the surgeon in placing a surgical device using the placement mechanism.

FIGURE 18 is a schematic drawing of the image overlay system of the present invention in which data from the computed tomography scanner is registered with multi-modal pre-operative images so that a composite image is sent to the image overlay system's display.

FIGURE 19 is a schematic drawing of the image overlay system of the present invention showing a first step in an image overlay calibration procedure.

FIGURE 20 is a schematic drawing of the image overlay system of the present invention showing a second step in an image overlay calibration procedure.

FIGURE 21 is a schematic drawing of the image overlay system of the present invention showing a third step in an image overlay calibration procedure.

FIGURE 22 is a schematic drawing of the image overlay system of the present invention using passive calibration for placement of the surgical device mechanism.

FIGURE 23 is a schematic drawing of the image overlay system of the present invention using active calibration for placement of the surgical device mechanism.

FIGURE 24 is a schematic drawing of the basic physical layout of the image overlay system of the present invention shown in FIGURE 1, except that the image display has an orientation other than the horizontal orientation shown in FIGURE 1.

FIGURE 25 is a perspective view of the image overlay system of the present invention shown in FIGURE 24.

FIGURE 26 is a perspective view of a holding attachment with seven degrees-of-freedom used to mount the image display in the arrangement of the image overlay system shown in FIGURE 24. FIGURE 27 is a perspective view of a triangle marker used in a system registration procedure positioned on the scanner bed of the CT scanner and adjusted to the laser of the CT scanner.

FIGURE 28 is a computer generated image of a slice of the triangle marker taken by the CT scanner.

FIGURE 29 is a perspective view of the triangle marker used in the system registration procedure with the overlay image of the slice of the triangle marker taken by the CT scanner being viewed through the half transparent mirror.

DESCRIPTION OF THE PREFERRED EMBODIMENT

FIGURE 1 is a schematic drawing of the basic layout of the image overlay system 10 of the present invention. The system 10 is designed to be used with a computed tomography ("CT") scanner system 11 or other cross- sectional imaging system, such as a magnetic resonance imaging ("MRI") system, positron emission tomography ("PET"), or single photon emission computed tomography ("SPECT") system. The system can also be used with ultrasound and thermal imaging systems. By way of example, the present invention will be described with respect to a CT system, which typically includes a CT gantry 12 and a CT table 14. The system of the present invention allows a surgeon 18 to more precisely place a surgical device 32 in a patient 16 laying on CT table 14 using a CT slice image of patient 16 for guidance.

Image overlay system 10 includes a display 22 for displaying the slice image of patient 16 obtained by CT scanner system 11, a half transparent mirror 24 for reflecting the slice image shown on display 22, a surgical device placement mechanism 30 to assist in the placement of surgical device 32 in patient 16, and a mounting device 26 mounted on top of CT gantry 12. Mounting device 26 is used to support display 22, mirror 24 and placement mechanism 30. Half- mirror 24 is preferably either a half silver mirror or a light brown acrylic plate.

CT scanner system 11 has a CT image plane 20 in which slice images of patient 16 are taken. In the embodiment of the image overlay system 10 shown in FIGURE 1 , mirror 24 is oriented with respect to image plane 20 at an angle 25 that is nominally 45°. Display 22 is then oriented with respect to mirror 24 at an angle 23 that is the same as angle 25, and thus, nominally 45°.

Surgical device placement mechanism 30 shown in the embodiment of FIGURE 1 is a mechanical device that constrains surgical device 32 in a plane that coincides with the laser marker 21 of CT gantry 12. Preferably, placement mechanism 30 is designed to allow surgical device 32 to be moved in the x and y directions of the constrained plane and rotated within the same plane so that it can be properly inserted in patient 16. Preferably, placement mechanism 30 includes a plurality of arms that are under tension for support and that are interconnected by a plurality of pivot joints. One example of such a mechanism is a surgical tool support used by dentists, but that is constrained to move in a selected plane. Surgical instrument 32 can be a needle or other type of surgical instrument, such as a drill, trocar, biopsy needle, injection device or tissue ablator, by way of example.

FIGURE 24 is another drawing of the basic layout of image overlay system 10 of the present invention, except that in FIGURE 24 the slice image display 22 has a tilted orientation that is different from the generally horizontal orientation of display 22 shown in FIGURE 1. The half transparent mirror 24 also shown in FIGURE 24 thus has a different orientation and position from the position and orientation of mirror 24 shown in FIGURE 1 to accommodate the change in display 22's orientation. FIGURE 25 is a perspective view of the embodiment of image overlay system 10 shown in FIGURE 24. The relative angle between display 22 and mirror 24 is not the nominal 45 degrees. The display 22 and mirror 24 were tilted, in order to give sufficient room for the patient and sufficient workspace for the surgeon. This solution is further explained later in Figure 5. In the embodiment of image overlay system 10 shown in FIGURES 24 and 25, mounting device 26 is detachably placed over CT gantry 12 as in the embodiment shown in FIGURE 1. In addition, attached to the mounting device 26 shown in FIGURES 24 and 25 is a holding attachment 78 on which are mounted display 22 and half-mirror 24.

As seen in FIGURE 26, holding attachment 78 includes a u-shaped bracket 80 that is positioned over a cross brace 81 of mounting device 26. Canterlevered from bracket 80 is a shaft 82 on which a parallel link 84 is rotatably mounted at a first pivot joint 86. At the opposite end of parallel link 84 is a second pivot joint 88 to which is connected a third pivot joint 92 supporting a mounting plate 94 on which display 22 is mounted. Included in mounting plate 94 is a circular groove 90 in which is rotatably mounted a cross arm 96 forming part of a frame 76 that supports display 22 and half-mirror 24. Mounted at the ends of cross arm 96 are two vertically oriented arms 97 on which mirror 24 is rotatably mounted. As can be seen in FIGURE 26, holding attachment 78 provides seven degrees of freedom (indicated by arrows) for the placement and orientation of display 22 and half-mirror 24 with respect to a patient 16 lying on CT table 14 of CT scanner system 11.

In current CT imaging systems, an image slice is typically taken at the center of the gantry. Holding attachment 78 allows mirror 24 of the overlay system 10 to be accurately positioned on the center of CT gantry 12, without making alterations to CT scanner 11 in violation of safety regulations. Holding attachment 78 also allows the overlay system 10 to be easily installed on a variety of CT gantries of different shapes.

Registration of the CT image taken by CT scanner system 11 and the patient 16 is critical to proper operation of the image overlay system 10. FIGURES 27 to 29 describe the preferred registration method used with the present invention. Referring first to FIGURE 27, a triangle-shaped fiducial marker 100 is placed on CT table 14 and adjusted to the laser marker 21 of CT scanner system 11 which shows the plane of acquisition of system 11. As can be seen in FIGURE 27, triangle marker 100 is made from a clear material and has a longitudinal dimension that allows a slice image of marker 100 to be taken by scanner 11. After a slice image of triangle marker 100 is taken, the image is transferred, preferably via DICOM, to a personal computer 50 (FIGURE 24) used for image processing. The resolution of display 22 is then pre-calibrated to obtain a full sized image, after which the slice image is transferred to computer 50 and adjusted to its full size. The image is then sent to display 22 and adjusted while looking through mirror 24 to the real image of triangle marker 100 using the edges of triangle marker 100 and surgeon 18's perception. Upon completing the registration procedure, it is possible to observe the slice image of marker 100 in an accurate position. The calibration object triangle marker 100 is then replaced with patient 16, while making sure that mirror 24 and display 22 are not moved.

Referring now to FIGURE 25, the image overlay system 10 is designed to make a CT slice image 75 appear in the exact same location where it was acquired from patient 16, while patient 16 is still in CT scanner 11. By superimposing image 75 on patient 16, the physician 18 can see the surface of patient 16 and the inside of the body of patient 16 at the same time. This helps the physician 18 to find the optimal access to a target within patient 16, while minimizing collateral damage to normal, healthy structures around the target. Slice image 75 is not projected into the surgical field. It is merely made to appear as though it is there. Thus, the central piece of system 10 is half-mirror 24. Half-mirror 24 allows the scene in the surgical field behind it to be seen at the same time as reflections in half-mirror 24. The reflections are fainter than they would be in a full mirror and the transparent surgical field scene is also clearer through clear glass, but with proper lighting, a reasonable compromise is achieved between the two. Optimal lighting can be determined experimentally, by using a simple continuous light dimmer attached to the light switch of the room. In operation, CT scanner 11 acquires a CT slice image of patient 16. Slice image 75 is rendered on display 22 which preferably is a distortion-free flat panel display. Display 22 is positioned above half-mirror 24, so that the reflection of slice image 75 coincides with patient 16's body seen behind mirror 24. Given the previous registration of the real view and the slice image 75 in mirror 24 using triangle 100, the result is an optically stable alignment of the real view and the slice image 75 in mirror 24, provided that mirror 24, display 22, and scanner 11 have been properly aligned.

Referring now to FIGURES 2 and 3, depicted therein are two schemes for using multiple CT slice images to view organs that are larger than a single CT slice image without physically moving patient 16. In FIGURE 2, display 22 is moved some vertical distance d from its original position, either toward or away from patient 16, to a new position. As a consequence, another overlay image 28A is displayed a horizontal distance d away (commensurate with the change in position of display 22) from the position at which the original overlay image 28 was displayed. This allows a second CT slice image of patient 16 taken at a position different from where the original image slice was taken to be displayed at the proper location with respect to patient 16. In FIGURE 3, half-mirror 24 is moved some horizontal distance d from its original position, either toward or away from patient 16, to a new position. As a consequence, second overlay image 28A is displayed at a new position a horizontal distance d away (commensurate with the change in position of half- mirror 24) from the position at which original overlay image 28 was displayed. Here again, this allows the second CT slice image of patient 16 to be displayed at the proper location with respect to patient 16.

FIGURE 4 discloses a scheme for image panning by parallel translation of display 22 or by moving the slice image on display 22 via software control to accommodate a body that is too large to fit on display 22. In this arrangement, display 22 is translated in a horizontal plane a distance d. The result is that overlay image 28 is translated in a vertical plane by the same distance d as display 22 is moved.

FIGURE 5 shows an arrangement for resolving interference of the image overlay system 10 with a patient 16. In this arrangement, half-mirror 24 is tilted so as to form with respect to image plane 20 an angle 25A that is equal to some angle °. When half-mirror 24 is tilted to eliminate interference with patient 16, the angular relationship between half-mirror 24 and display 22 must be maintained. Thus, the angular relationship between display 22 and half- mirror 24 must also be an angle 23A that is equal to α°.

FIGURE 6 discloses an arrangement for using tilted CT slice images that are obtained from patient 16 by tilting CT gantry 12. Tilting the CT gantry is achieved by a standard mechanism commonly available on contemporary CT scanners. This feature is not typically available in other cross sectional imaging modalities. Such an arrangement could be used, for example, to obtain a different perspective on a patient's spine which is typically curved, and which may be further curved due to a disease such as kyphosis or scoliosis. Using the tilted CT slice images allows physician 18 to follow the curvature of a patient's spine while obtaining a single CT slice images of the spine. In the embodiment shown in FIGURE 6, the angular relationship between display 22 and half- mirror 24 is maintained.

FIGURES 7 and 8 show embodiments of image overlay system 10 in which tilted CT slice images are obtained from synthetic (or re-sampled) CT slice images that are oblique cuts of a computer-generated 3D image or digital representation of some volume within patient 16. In FIGURE 7, display 22 is tilted at some angle 44 equal to α° to achieve a rotation of overlay image 28 with respect to image plane 20 at an angle 45 that is also equal to α°, but in the opposite direction. In FIGURE 8, half-mirror 24 is tilted so as to produce a different angle 23B with respect to display 22 that is equal to some angle α°. The result is an angle 25B between half-mirror 24 and overlay image 28. Both of the embodiments shown in FIGURES 7 and 8 use a computer-generated 3D image to form a new cross-section that is the overlay image 28.

FIGURES 9 and 10 show embodiments of image overlay system 10 in which the surgical device placement mechanism is a robot device 30A that is remotely controlled. Currently, there is no such small imager-compatible surgical robot is available commercially. Several prominent research groups are developing such devices that are expected to enter the marketplace in the near future. In the embodiment of FIGURE 9, the surgeon 18 uses a joystick control unit 48 that is connected to robot placement mechanism 30A through a cable 56. In the embodiment of FIGURE 10, surgeon 18 uses a desktop computer 50 that is connected to robot placement mechanism 30A through a cable 57. In these embodiments, robot placement mechanism 30A can be a device that is, for example, electromechanical, pneumatic, ultrasonic, or some other control arrangement. As with the placement mechanism 30 used in the embodiment of FIGURE 1, robot placement mechanism 30A is designed to allow surgical instrument 32 to be moved in the X,Y directions of a constrained plane and rotated within that same plane so that it can be properly inserted in patient 16.

FIGURE 11 shows an embodiment of image overlay system 10 in which a passive mechanism 30 for the placement of a surgical device 32 is used. As in the embodiment of FIGURE 1 , placement mechanism 30 is a mechanical device that is designed to allow surgical instrument 32 to be moved in the X and Y directions of a constrained plane and rotated within that same plane so that it can be properly inserted in patient 16. Placement mechanism 30 is positioned by surgeon 18's hand 52 into the proper location and orientation for insertion of surgical instrument 32 into patient 16.

FIGURE 12 shows image overlay system 10 without placement mechanism 30. In the embodiment of FIGURE 12, physician 18 holds surgical instrument 32 in his hand 52 and positions instrument 32 for insertion into patient 16 using the laser marker 21 of the CT scanner 11 as a visual guide for proper placement and orientation of instrument 32 prior to insertion.

FIGURE 13 also shows image overlay system 10 without placement mechanism 30. In the embodiment of FIGURE 13, physician 18 again holds surgical instrument 32 in his hand 52 and positions instrument 32 for insertion into patient 16. In this embodiment, however, the physician uses a tracking system that includes a base unit 54 attached to mounting device 26 and a transmission unit 56 attached to surgical instrument 32. Several commercially available tracking devices can be used for this purpose, such as Polaris (Northern Digital, Canada) , Flashpoint (Image Guided Technologies, Boulder, CO). Tracking information generated by units 54 and 56 is available to surgeon 18 on personal computer 50 that is connected to base unit 54 through a cable 57 running between base unit 54 and computer 50. In this embodiment, surgeon 18 uses the tracking data available on computer 50 as a guide for proper positioning and orientation of instrument 32 prior to insertion in patient 16. Typically, the position data is numerical data reflecting the coordinate location of the surgical instrument 32 with respect to the patient 16.

FIGURE 14 again shows the image overlay system 10 of the present invention with a tracking system that includes a base unit 54 attached to mounting device 26 and a transmission unit 56 attached to surgical instrument 32. In the embodiment of FIGURE 14, system 10 also includes a surgical device placement mechanism 30 for holding and positioning surgical instrument 32. Tracking information generated by units 54 and 56 is again available to surgeon 18 on computer 50, which is connected to base unit 54 through cable 57. Here, surgeon 18 uses the tracking data available on computer 50 as a guide for positioning and orientation of placement mechanism 30 and instrument 32 held by mechanism 30 prior to insertion in patient 16. Here again, the position data generated by tracking device units 54 and 56 is numerical data reflecting the coordinate location of the surgical instrument 32 with respect to the patient 16.

FIGURE 15 shows image overlay system 10 without the placement mechanism 30, but with tracking units 54 and 56. In the embodiment of FIGURE 15, physician 18 again holds surgical instrument 32 in his hand 52 and positions instrument 32 for insertion into patient 16 using tracking base unit 54 attached to mounting device 26 and transmission unit 56 attached to surgical instrument 32. Tracking information generated by units 54 and 56 is again displayed on personal computer 50, but also overlaid on overlay image 28. This is accomplished by a transfer of such information from computer 50 to display 22 through cable 58 connecting computer 50 to display 22. The placement of the tracking data on overlay image 28 readily allows surgeon 18 to use the tracking data as a guide for proper positioning and orientation of instrument 32 prior to insertion in patient 16. Here again, the position data is numerical data reflecting the coordinate location of the surgical instrument 32 with respect to the patient 16.

The embodiment of image overlay system 10 shown in FIGURE 16 again uses a tracking system that includes base unit 54 attached to mounting device 26 and transmission unit 56 attached to surgical instrument 32. In the embodiment of FIGURE 16, there is also surgical device placement mechanism 30 for holding and positioning surgical instrument 32. Tracking information generated by units 54 and 56 is again available to surgeon 18 on computer 50 and overlaid on overlay image 28. This is accomplished by a transfer of such information from computer 50 to display 22 through cable 58 connecting computer 50 to display 22. Here again, the placement of the tracking data on overlay image 28 readily allows surgeon 18 to use the tracking data as a guide for proper positioning and orientation of placement mechanism 30 and instrument 32, held by mechanism 30, prior to insertion in patient 16. As in the prior embodiments, the position data generated by tracking device units 54 and 56 is numerical data reflecting the coordinate location of the surgical instrument 32 with respect to the patient 16.

FIGURE 17 shows an embodiment of the image overlay system of the present invention in which coordinate data from an encoded instrument placement mechanism 30B is visually superimposed on virtual overlay image 28 to assist surgeon 18 in the placement of surgical device 32 in patient 16. Here again, encoded placement mechanism 30B constrains surgical device 32 in a plane that coincides with the laser marker 21 of CT scanner 11. Placement mechanism 30B is also designed to allow surgical device 32 to be moved in the X, Y directions of the constrained plane and rotate within the same plane so that device 32 can be properly inserted in patient 16. In this embodiment of placement mechanism 3 OB, there is also included an encoding mechanism for providing the X, Y coordinates of device 32 and coordinates corresponding to its rotational orientation. Encoded instrument placement mechanism 30B can be a passive placement device or robot of some sort, not specified further here. The coordinate data from encoded instrument placement mechanism 3 OB is sent to personal computer 50, and then transmitted to display 22 so that it can be visually superimposed on virtual overlay image 28.

FIGURE 18 shows an embodiment of image overlay system 10 in which slice images 60 from CT scanner 11 are registered with multi-modal pre- operative slice images 62 and 64 from, for example, MRI and PET scanners, respectively, so that a composite, more comprehensive slice image is sent to display 22 for overlay image 28C . Composite overlay images are desirable because different types of scans have different advantages. For example, CT scans are good for hard tissue, such as bones and ligaments. In contrast, soft tissue differentiation in CT scans is relatively poor, whereas it is good in MRI scans. Typically, the auxiliary scans are done pre-operatively, then spatially registered to the coordinate space of the intra-operative CT images, by some standard means medical image registration not specified further in this document. FIGURES 19, 20 and 21 show a robust calibration procedure without the use of a calibration object. Step 1 of the calibration procedure is shown in FIGURE 19. Step 1 is to confirm that display 22 is horizontal, CT gantry 12 is vertical, and half-mirror 24 is oriented at 45° with respect to image plane 20. Preferably, a bubble level 66, as shown in FIGURE 19, can be used for this purpose. Step 2 of the calibration procedure, which is shown in FIGURE 20, is to translate monitor 22 or half-mirror 24 off-plane to align the current image plane with CT scanner 1 l's laser marker 21. Preferably, a CT slice image of patient 16 is taken and the height of display 22 is adjusted some distance d until the plane of overlay image 28 coincides with the plane of the axial laser marker 21 of scanner 11.

Step 3 of the image overlay calibration procedure can be explained in FIGURE 21. In step 3, display 22 is translated in plane, or the image on display 22 is translated, until the overlay image 28 is aligned with patient 16's body. Preferably, the contour of patient 16's body is highlighted in the image on display 22 using image processing. Alternatively, fiducial marks (not shown) can be applied to the body of patient 16 prior to taking CT images. . The fiducials are then highlighted using the imaging processor of display 22. Device placement mechanism 30 may also be instrumented with similar fiducials. Display 22 is then adjusted within plane motion or the image on display 22 is adjusted with panning via software control without moving display 22 until overlay image 28 matches the contour of patient 16's body and all selected fiducials and anatomical landmarks. FIGURE 22 shows passive surgical device placement mechanism calibration. In this procedure, CT gantry 12 is brought to a vertical position. Bubble level 66 (as shown in Figure 19) is used to confirm and insure the vertical position of CT gantry 12. The laser light of scanner 1 l's laser marker 21 in the axial image plane must be cast in the central line of surgical device placement mechanism 30. FIGURE 23 shows a calibration method applicable to encoded surgical device placement mechanism, either passive or active robotic mechanism. The method employs rigid body fiducial based stereotactic registration system attached to the surgical device placement mechanism. The rigid fiducial frame is assembled from several rods, in a known geometrical pattern. Upon imaging, the rods are cut across the CT slice and the mark of each rod is detected in the CT image. Based on the relative position of these marks and the a priori known relative spatial position of the rods, one can calculate the spatial location of the fiducial frame with respect to the CT image. This completes the registration process.

Although the present invention has been described in terms of particular embodiments, it is not intended that the invention be limited to those embodiments. Modifications of the disclosed embodiments within the spirit of the invention will be apparent to those skilled in the art. The scope of the present invention is defined by the claims that follow.

Claims

WHAT IS CLAIMED IS:

1. An image overlay system to assist surgical interventions by a surgeon on a patient inside a cross-sectional imaging system including an imaging gantry for taking slice images of the patient, the image overlay system comprising: a half-mirror positioned between the surgeon and the patient, a display positioned with respect to the half -mirror so that slice images shown on the display are projected onto the half- mirror so that the slice images appear to be floating inside the patient in correct size and position, and a holding attachment connected to the imaging gantry for supporting the half-mirror and display.

2. The image overlay system as recited in claim 1, further comprising a mechanism for placement of a surgical device with respect to the patient.

3. The image overlay system as recited in claim 1, wherein the holding attachment supporting the display allows the display to be off-plane translated so that multiple slice images can be displayed with respect to the patient at the respective locations at which the slice images were taken by the cross-sectional imaging system.

4. The image overlay system as recited in claim 1, wherein the holding attachment supporting the half-mirror allows the half- mirror to be translated off-plane so that multiple slice images can be displayed with respect to the patient at the respective locations at which the slice images were taken by the cross-sectional imaging system.

5. The image overlay system as recited in claim 1, wherein the holding attachment supporting the display allows the display to be in-plane translated so that an image could be panned inside the plane of imaging, at which the slice images were taken by the cross-sectional imaging system.

6. The image overlay system as recited in claim 1 further comprising a computer connected to the display, wherein the computer is programmed to allow the display's image to be horizontally translated so that an image could be panned inside the plane of imaging at which the slice images were taken by the cross-sectional imaging system.

7. The image overlay system as recited in claim 1, wherein the holding attachment supporting the display and half-mirror includes a connection that allows the half-mirror and display to be simultaneously tilted to avoid interference with the patient while in the cross-sectional imaging system.

8. The image overlay system as recited in claim 7, wherein the connection is a rotational joint.

9. The image overlay system as recited in claim 1, wherein the imaging gantry is capable of being tilted so that tilted slice images of the patient can be obtained and displayed using the image overlay system.

10. The image overlay system as recited in claim 1, wherein the holding attachment supporting the display allows the display to be tilted so that tilted slice images can be displayed with respect to the patient.

11. The image overlay system as recited in claim 1 further comprising a computer connected to the display, wherein tilted slice images are formed by the computer using a three-dimensional image of the patient formed from a plurality of slice images of the patient taken by the cross-sectional imaging system.

12. The image overlay system as recited in claim 1, wherein the holding attachment supporting the half-mirror allows the half- mirror to be tilted so that tilted slice images can be displayed with respect to the patient.

13. The image overlay system as recited in claim 12 further comprising a computer connected to the display, wherein tilted slice images are formed by the computer using a three-dimensional image of the patient formed from a plurality of slice images of the patient taken by the cross-sectional imaging system.

14. The image overlay system as recited in claim 2, wherein the placement mechanism is a robot mechanism, and wherein the image overlay system further comprises a joystick for controlling the robot placement mechanism.

15. The image overlay system as recited in claim 2, wherein the placement mechanism is a robot mechanism, and wherein the image overlay system further comprises a computer for controlling the robot placement mechanism.

16. The image overlay system as recited in claim 2, wherein the placement mechanism is a passive mechanical mechanism manually actuated by the surgeon.

17. The image overlay system as recited in claim 16, wherein the placement mechanism allows the surgical device to be moved in X and Y directions of a constrained plane and rotated within the plane, whereby the surgical instrument can be manually positioned and oriented by the surgeon with respect to the patient.

18. The image overlay system as recited in claim 2, wherein the placement mechanism allows the surgical device to be moved in X and Y directions of a constrained plane and rotated within the plane, whereby the surgical instrument can be manually positioned and oriented by the surgeon with respect to the patient.

19. The image overlay system as recited in claim 1, wherein the cross-sectional imaging system includes a laser marker that is used by the surgeon to position a surgical device with respect to the patient free-hand.

20. The image overlay system as recited in claim 1 further comprising a tracking device for providing tracking data that is used by the surgeon to position a surgical device with respect to the patient.

21. The image overlay system as recited in claim 20, wherein the surgeon positions the surgical device with respect to the patient free-hand.

22. The image overlay system as recited in claim 20, wherein the tracking device includes a base unit mounted on the holding attachment and a transmitting unit mounted on the surgical device.

23. The image overlay system as recited in claim 2 further comprising a tracking device for providing tracking data that is used by the surgeon to position a surgical device with respect to the patient.

24. The image overlay system as recited in claim 23, wherein the tracking data is used by the surgeon to position the placement mechanism and thereby the surgical device with respect to the patient.

25. The image overlay system as recited in claim 23, wherein the tracking device includes a base unit mounted on the holding attachment and a transmitting unit mounted on the surgical device.

26. The image overlay system as recited in claim 20 further comprising a computer for visually superimposing on the slice images shown on the display the tracking data obtained from the tracking device to assist the surgeon in positioning the surgical device with respect to the patient free-hand.

27. The image overlay system as recited in claim 23 further comprising a computer for visually superimposing on the slice images shown on the display the tracking data obtained from the tracking device to assist the surgeon in positioning the placement mechanism and thereby the surgical device with respect to the patient.

28. The image overlay system recited in claim 1, wherein the cross-sectional imaging system is a computed tomography imaging system.

29. The image overlay system recited in claim 28, wherein the cross-sectional imaging system further comprises a computer for registering slice images taken by the computed tomography imaging system with slice images from cross-sectional imaging systems of a type different from the computed tomography imaging system.

30. The image overlay system recited in claim 29, wherein the different type of imaging system is a magnetic resonance imaging system.

31. The image overlay system recited in claim 29, wherein the different type of imaging system is a position emission tomography imaging system.

32. The image overlay system recited in claim 29, wherein the different type of imaging system is a single photon emission computed tomography imaging system.

33. The image overlay system as recited in claim 1, wherein the holding attachment includes a plurality of arms and rotational joints that allow a plurality of degrees-of-freedom in movement and positioning of the holding attachment.

34. The image overlay system as recited in claim 33, wherein the holding attachment has seven degrees-of-freedom.

35. An image overlay system for use by a physician with a patient inside a computed tomography ("CT") scanning system for taking slice images of the patient inside a CT gantry, the image overlay system comprising: a half-mirror position between the physician and the patient, an overlay display positioned with respect to the half-mirror so that slice images shown on the overlay display are projected onto the half-mirror, and a holding attachment supporting the half-mirror and display and being connected to the CT gantry.

36. The image overlay system as recited in claim 35 further comprising a mechanism for placement by the physician of a surgical device with respect to the patient.

37. The image overlay system as recited in claim 35, wherein the holding attachment allows the overlay display to be off-plane translated and/or the half-mirror to be off-plane translated so that multiple slice images can be displayed with respect to the patient at the respective locations at which the slice images were taken by the CT scanning system. Do not we miss here one? In-plane motion of the display to achieve panning? (See claim 5.)

38. The image overlay system as recited in claim 35, wherein the holding attachment supporting the display allows the display to be in-plane translated so that an image could be panned inside the plane of imaging, at which the slice images were taken by the cross-sectional imaging system.

39. The image overlay system as recited in claim 35, wherein the holding attachment allows the half-mirror to be tilted to avoid interference with the patient while in the CT scanning system or to allow tilted slice images to be displayed with respect to the patient.

40. The image overlay system as recited in claim 35, wherein the CT gantry is capable of being tilted so that tilted slice images of the patient can be obtained and displayed with respect to the patient.

41. The image overlay system as recited in claim 35 further comprising a computer that is programmed to allow the overlay display's image to be horizontally translated an image could be panned inside the plane of imaging at which the slice images were taken by the cross-sectional imaging system.

42. The image overlay system as recited in claim 36, wherein the placement mechanism is a robot mechanism, and wherein the image overlay system further comprises a joystick for controlling the robot placement mechanism.

43. The image overlay system as recited in claim 36, wherein the placement mechanism is a robot mechanism, and wherein the image overlay system further comprises a computer for controlling the robot placement mechanism.

44. The image overlay system as recited in claim 36, wherein the placement mechanism is a mechanical mechanism manually actuated by the physician.

45. The image overlay system as recited in claim 44, wherein the placement mechanism allows the surgical device to be moved in X and Y directions of a constrained plane and rotated within the plane so that the surgical instrument can be positioned and oriented with respect to the patient.

46. The image overlay system as recited in claim 44, wherein the CT scanning system includes a laser marker that is used by the physician to position the surgical device with respect to the patient free-hand.

47. The image overlay system as recited in claim 35 further comprising a tracking device for providing tracking data that is used by the physician to position a surgical device with respect to the patient.

48. The image overlay system as recited in claim 36 further comprising a tracking device for providing tracking data that is used by the physician to position the placement mechanism and thereby the surgical device with respect to the patient.

49. The image overlay system as recited in claim 35 further comprising a tracking device for providing data that is used to position a surgical device with respect to the patient and a computer for visually superimposing on the slice images shown on the overlay display tracking data obtained from the tracking device to assist in the placement of the surgical device.

50. The image overlay system as recited in claim 35 further comprising a computer for registering slice images taken by the computed tomography scanning system with slice images from cross-sectional imaging systems of a type different from the computed tomography scanning system.

51. The image overlay system as recited in claim 35 further comprising a calibration object for use in registering the half- mirror, overlay display and the CT scanning system.

52. The image overlay system as recited in claim 51, wherein the calibration object is a three-dimensional prism.

53. The image overlay system as recited in claim 1 further comprising a calibration object for use in registering the half- mirror, overlay display and the cross-sectional imaging system.

54. The image overlay system as recited in claim 53, wherein the calibration object is a three-dimensional prism

55. The image overlay system as recited in claim 52 wherein the three-dimensional prism is triangularly-shaped.

56. The image overlay system as recited in claim 54 wherein the three-dimensional prism is triangularly-shaped.

57. A method for registering the image overlay system recited in claim 35 comprising the steps of: placing on the CT gantry a fiducial marker, adjusting the calibration object to a laser marker of the CT scanning system, obtaining a slice image of the fiducial marker, precalibrating the overlay display and displaying on the overlay display and adjusting to full size the slice image of the fiducial marker, while looking through the half-mirror, adjusting the slice image of the calibration object to the actual calibration object on the CT gantry.

58. The method recited in claim 57, wherein the calibration object is a three-dimensional prism.

59. A method for registering the image overlay system recited in claim 1 comprising the steps of: placing on the imaging gantry a fiducial marker, adjusting the calibration object to a laser marker of the cross-sectional imaging system, obtaining a slice image of the fiducial marker, precalibrating the display and displaying on the display and adjusting to full size the slice image of the fiducial marker, while looking through the half-mirror, adjusting the slice image of the calibration object to the actual calibration object on the imaging gantry.

60. The method recited in claim 59, wherein the calibration object is a three-dimensional prism.

61. A method for registering the image overlay system recited in claim 1 comprising the steps of: confirming that the display is horizontal, the imaging gantry is vertical, and the half-mirror is oriented at a selected angle with respect to an image plane of the cross-sectional imaging system, obtaining a slice image of a patient on the imaging gantry and adjusting the height of the display until the plane of the overlay image coincides with the plane of an axial laser marker of the cross-sectional scanner, translating the display in plane or the slice image displayed on the display until the slice image is aligned with the patient's body.

62. The method as recited in claim 61, wherein the step of confirming that the display is horizontal, the imaging gantry vertical, and the half-mirror oriented at a selected angle is performed using a bubble level.

63. The method as recited in claim 61, wherein the selected angle is 45 ^e .

64. A method of calibrating the surgical device placement mechanism recited in claim 2 comprising the steps of: bringing the imaging gantry to a vertical position, and ensuring that a laser marker of the cross-sectional imaging system is in a central line of the surgical device placement mechanism.

65. The method of claim 64 further comprising the step of: confirming the vertical position of the imaging gantry using a bubble level.

66. A method for performing a surgical intervention on a patient inside a cross-sectional imaging scanner system including an imaging gantry, the method comprising the steps of: obtaining a slice image of the patient using the scanner system, displaying the slice image on an overlay display, projecting the slice image displayed on the display onto a half -mirror so that the slice image is reflected from the half-mirror, viewing the patient through the half-mirror with the slice image reflected therefrom, whereby the slice image appears to be overlaying and floating inside the patient's body in correct size and position, using the overlaying slice image to position a surgical device with respect to the patient, and performing the surgical intervention using the positioned surgical device.

67. The method as recited in claim 66 further comprising the step of positioning the surgical device with respect to the patient using a placement mechanism that holds the surgical device

68. The method as recited in claim 66 further comprising the step of translating the display off-plane so that multiple slice images can be displayed with respect to the patient at the respective locations at which the slice images were taken.

69. The method as recited in claim 66 further comprising the step of translating the half-mirror off-plane so that multiple slice images can be displayed with respect to the patient at the respective locations at which the slice images were taken.

70. The method as recited in claim 66 further comprising the step of translating the display in-plane so that an image could be panned inside the plane of imaging at which the slice images were taken.

71. The method as recited in claim 66 further comprising the step of translating the slice image displayed on the display so an image could be panned inside the plane of imaging at which the slice images were taken.

72. The method as recited in claim 66 further comprising the step of simultaneously tilting the half-mirror and display to avoid interference with the patient while the patient is in the scanner system.

73. The method as recited in claim 66 further comprising the step of tilting the imaging gantry so that tilted slice images of the patient can be obtained and displayed.

74. The method as recited in claim 66 further comprising the step of generating tilted slice images using a computer- generated three-dimensional image of the patient formed from a plurality of slice images of the patient taken by the scanner system.

75. The method as recited in claim 66 further comprising the step of tilting the half-mirror so that tilted slice images can be displayed with respect to the patient.

76. The method as recited in claim 67 further comprising the step of positioning the surgical device with respect to the patient using a robot placement mechanism that is controlled by a joystick.

77. The method as recited in claim 67 further comprising the step of positioning the surgical device with respect to the patient using a robot placement mechanism that is controlled by a personal computer.

78. The method as recited in claim 66 further comprising the step of placing a surgical device with respect to the patient free-hand.

79. The method as recited in claim 66 further comprising the step of placing a surgical device with respect to the patient using tracking data that is obtained from a tracking device.

80. The method as recited in claim 79, wherein the tracking data is numerical data.

81. The method as recited in claim 80, wherein the tracking data is visually displayed on the slice image reflected from the half-mirror.

82. The method as recited in claim 66, wherein the scanning system is a computed tomography scanning system.

83. The method as recited in claim 66, wherein the scanning system is a magnetic resonance imaging system.

84. The method as recited in claim 66, wherein the scanning system is a position emission tomography scanning system.

85. The method as recited in claim 66, wherein the scanning system is a single photon emission computed tomography scanning system.