WO1996028800A1

WO1996028800A1 - Computer based medical procedure simulation system

Info

Publication number: WO1996028800A1
Application number: PCT/US1996/002317
Authority: WO
Inventors: Jonathan R. Merril; Alan S. Millman
Original assignee: High Techsplanations, Inc.
Priority date: 1995-03-10
Filing date: 1996-03-08
Publication date: 1996-09-19
Also published as: AU5025396A; CA2144505A1

Abstract

A virtual reality medical simulation system (1) provides a highly realistic simulation of a variety of medical procedures, particularly catheter based procedures such as balloon angioplasty. A catheter interface device (3) tracks a catheter wire (7) in translation and rotation and outputs a signal to a computer program that displays movement of a virtual catheter within a virtual arterial tree including an occluded region. The catheter tip is navigated through the arterial tree and into the occluded region, whereupon the computer program generates a tactile feedback force that is output to the catheter interface device (3) to simulate increased resistance. Once the catheter (7) is properly positioned within the occlusion, a virtual balloon catheter can be inflated to remove the occlusion. A variety of special effect enhancements are included in the simulation, including 3-D imaging, simulated radiopaque dye infusion, bleeding and atrial pulsation in synchronism with an on screen EKG display.

Description

COMPUTER BASED MEDICAL PROCEDURE SIMULATION SYSTEM

BACKGROUND OF THE INVENTION

The present invention relates to computer-based simulation of medical procedures, e.g., surgical procedures. A particular application of the present invention concerns a system for providing a realistic "virtual reality" simulation of a cardiac catheterization procedure.

Continuing public debate concerning rising health care costs has pin-pointed continuing medical education (CME) and graduate medical education (GME) as major expenses to address in the health reform effort. More than $6 billion is spent on CME and GME annually. Much of this cost is attributed to expensive training, resources and material. The use of virtual reality (VR) in medical education and training can reduce the expenses associated with changes in medical technology, as well as provide enhanced physician training.

VR technology has been used in a wide variety of fields, including military, architecture, chemistry and biomedicine. The application of VR technology to education and training, however, is just beginning to be exploited. VR technologies are becoming more applicable in medicine, in part because of the high costs of traditional training resources (lab animals, physican's time, etc.).

The value of computer-assisted instruction has long been recognized. During the past decade, numerous medical centers around the country have designed and used software for medical education. The software in use is primarily designed to run on IBM PC compatible and Macintosh computers. These instructional programs also have many advantages over traditional lecture-oriented learning, e.g., they are interactive and allow for self-paced instruction. Many of the programs also incorporate simulations which require the user to make patient management decisions. If incorrect decisions are made, text describing why the choice was incorrect is provided.

More recently, medical education programs have been created using personal computer graphics for the instruction of anatomy. This type of program provides increased educational value because it gives students the opportunity to quickly access anatomic images and related textual data. Rather than spending hours in the gross anatomy lab

SDBSπiUIE SHEET (RULE 26) dissecting cadavers (and being exposed to toxic chemical preservatives), these programs allow students to work in the library - or at home - and spend as much time as needed to learn the material. The programs also provide textbook references so students can immediately read about the clinically relevant aspects.

Although these computer-assisted programs offer many advantages in medical education, there are some inherent limitations and drawbacks. For example, these programs are confined to two dimensional (2-D) imaging. This limits the scope of depicting anatomical orientation, particularly with respect to showing the relation of superficial and deeper structures. These programs are similarly limited by lacking the ability to rotate in space so that anterior and posterior relationships can be seen. Additionally, while highly valuable for anatomic instruction, these programs do not allow for practice of actual medical procedures under realistic conditions.

A system capable of providing a realistic real-time simulation of surgical procedures would help reduce the operative risk associated with new technologies in medicine, allowing better transference of skills from training to the operating room. A tremendous demand exists for enhancing the way that physicians learn new invasive procedures. Appropriate education in new medical and surgical procedures is often outpaced by the desire of physicians to incorporate a procedure into their practice.

As one example, until a few years ago, the treatment for coronary artery disease was open heart surgery, an expensive and very difficult surgical procedure. As an alternative to open heart surgery, percutaneous transluminal coronary angioplasty (PTCA) is now generally considered safe and effective to relieve the same conditions that heart surgery did just a few years ago. With the recent availability of this new procedure, the training of physicians presents a considerable problem. Currently, physician experience depends on the availability of a weekend course and practicing the procedure in an animal lab. Unfortunately, two days rarely provides enough practice time. Additionally, animal anatomy is quite different than human anatomy. A realistic virtual reality simulation could offer the physician the ability to practice hundreds of procedures in a realistic environment, and with realistic anatomy, prior to patient contact.

Computerized surgical simulations have the potential for improving surgical morbidity and mortality. Studies have shown that for a wide range of diagnostic and therapeutic procedures, doctors doing their first few to several dozen cases are much more

- 2 -

SUBSTITUTE SHEET (RUU 26) likely to make a greater number of errors. This phenomenon has been referred to as the "learning curve. " Adequate proctoring of learners by experienced surgeons is cumbersome, as there are few surgeons experienced enough to proctor their colleagues. It is difficult for physicians, particularly those in rural areas, to travel to larger medical centers for training. The requirement also places a burden on experts who, already busy with their clinical responsibilities, could become overwhelmed with proctoring requests.

Another potential advantage of VR surgical training is in the standardization of surgical training. Such standardization would help insure, e.g., that all board-certified physicians are top-notch. Providing training with computer programming would allow major medical centers to directly interact with students and physicians in a rural setting. By so doing, the credentialing and granting of privileges for traditional procedures, as well as newer procedures like laparoscopic surgery, could be objectively provided based on standardized skills assessment, regardless of where a surgeon is located. The challenge has been to create a surgical simulation that is sufficiently realistic that the motor skills and anatomic recognition acquired in the simulation are readily transferred to performance of the actual procedure.

In addition to realistically simulating the involved anatomy, i.e., body parts, a realistic computer-based surgical simulator must also provide a realistic surgical environment. A key in this regard is to provide the involved proxy medical instrument(s) with a realistic look and feel. This can be accomplished through the use of computer peripheral devices that mimic the actual instrument(s) used by a surgeon to perform an actual procedure. In this respect, the closer the proxy surgical instrument is to the actual surgical instrument, the more realistic the surgical simulation.

One computer interface device that has been used in surgical simulations is the Immersion PROBE produced by Immersion Corporation of Palo Alto, California. The Immersion PROBE comprises a pen-like stylus supported on a light-weight mechanical linkage. The stylus is held between the fingers like a medical instrument and is wielded dexterously. The Immersion PROBE reports the position and orientation of the stylus to a host computer via a standard serial port interface. The Immersion PROBE linkages provide six degrees of freedom. Sensors provided at the linkage joints convey spatial coordinates (X,Y,Z) and orientation (roll, pitch, yaw) of the stylus to the host computer.

While the Immersion PROBE is a useful general purpose 3-D computer/user interface tool, it does not closely resemble in feel or appearance an actual surgical or medical instrument. For example, it's pen-like stylus and six degrees of freedom bear little resemblance to the type of elongated flexible member used in a catheterization procedure. An additional shortcoming of the Immersion PROBE is that it has no provision for providing tactile feedback to the user. Tactile feedback is very important to create a realistic feel.

Hon U.S. Patent No. 4,907,973 discloses a computer-based surgical simulation system including a peripheral device designed to simulate a surgical instrument, as well as the involved anatomy, i.e., body parts. In particular, the Hon patent discloses a system for computer simulation of a balloon-angioplasty operation. Therein, a catheter-like device is negotiated through a mock arterial path of a physical internal body part modeling device. Sensors on the catheter track the progress of the catheter through the mock arterial path. In one embodiment, a vessel constricting simulator within the mock arterial path provides a fixed resistance to the progress of the catheter.

While the Hon system has the potential for providing a more realistic feel of a catheter device as compared with the Immersion PROBE, it lacks flexibility for easily altering the involved anatomy and procedures. The simulation shown on the computer monitor appears to correspond to the structure of the physical model. Providing different anatomies, patients conditions and procedures would require additional physical models for each.

Another shortcoming of the Hon system is its lack of "special effects" for realistically simulating changes in the virtual environment responsive to operator manipulations and performance. For example, in the catheter insertion simulation of Hon, no provision is made for (1) a realistic simulation of bleeding, e.g., in the event an arterial wall is punctured, (2) radio-opaque dye infusion to render the arteries visible under a fluoroscope, or (3) catheter balloon inflation and consequent removal of an inclusion. Moreover, the Hon system is over-simplified and less than realistic, since no provision is made for tracking and displaying catheter rotation in the virtual environment. Importantly, in an actual catheterization procedure, rotation of the catheter wire is used, e.g., to direct the catheter in the proper direction. SUMMARY OF THE TNVENTION

In view of the foregoing, a principal object of the present invention is to provide for the performance of a simulated medical procedure under highly realistic conditions.

It is a further object of the invention to provide a high degree of flexibility, thereby allowing performance of a variety of medical procedures under different patient conditions, including actual patient conditions.

Yet another object of the present invention is to combine with a realistic simulation experience, tutorial aspects such as expert assistance and evaluation.

These and other objects are achieved by a computer based medical procedure simulation system in accordance with the present invention. In one such system, first memory means are provided for storing image data representative of a virtual internal body environment. Second memory means are provided for storing image data representing a virtual medical instrument extending within the internal body environment. Display means are provided for displaying images corresponding to the image data stored in the first and second memory means. A first computer peripheral device tracks the movement of a physical member representing the virtual medical instrument, and outputs a signal based thereon. First input means are provided for receiving the signal from the peripheral device. First calculating means are provided for calculating position data representing the position of the virtual medical instrument within the virtual internal body environment, based on the signal from the peripheral device. First updating means are provided for updating the image data representing the virtual medical instrument extending within the virtual internal body environment, based on said calculated position data.

An exemplary embodiment of the present invention is a cardiac catheterization simulator used to train interventional cardiologists to perform specific catheterization procedures such as angioplasty and stent placement. The cardiologist works with the system in the same way that he she would perform an actual catheterization, i.e., by manipulating a proxy catheter. The location and movement of the catheter is monitored by watching a simulated fluoroscopic view, and the cardiologist can feel the interaction between the catheter and the artery by the force required to move the catheter.

These and other objects, features and advantages of the present invention will be readily apparent and fully understood from the following detailed description of preferred embodiments, taken in connection with the appended drawings.

- 5 -

SUBSTlTuTE SHEET (RULE 26) BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is a schematic view of an overall medical procedure simulation system in accordance with the present invention.

Figure 2 is a schematic view illustrating the operating principles of a catheter interface device used in the system shown in Fig. 1.

Figures 3-17 are logic flow diagrams for a program used in the simulation system of Fig. 1.

Figures 18-29 are representative screen displays produced in die simulation system of the present invention.

Figure 30 is a front-left perspective view of the catheter interface device used in the simulation system shown in Fig. 1.

Figure 31 is a rear-right perspective view of the catheter interface device shown in Fig. 30, with the cover removed.

Figure 32 is a front elevational view of the internal componentry of the catheter interface device, with the electrical connections omitted for visual clarity.

Figure 33 is a rear elevational view of the internal componentry of the catheter interface device, with the electrical connections omitted for visual clarity.

Figure 34 is a right-side elevational view of the internal componentry of the catheter interface device, with the electrical connections omitted for visual clarity.

Figure 35 is a left-side elevational view of the internal componentry of the catheter interface device, with the electrical connections omitted for visual clarity.

Figure 36 is a top-plan view of the internal componentry of the catheter interface device.

Figure 37 is a cross-sectional view taken along a line 37-37 in Fig. 36. DFTATT.ITO DESCRIPTION OF THE PREFERRED EMBODIMENTS

An overall simulation system 1 in accordance with the present invention is illustrated in Fig. 1. The system comprises two primary parts: a computer/human interface device 3 and a computer 5 storing a software program which works in conjunction with the device. In the exemplary embodiment described in detail herein, the system simulates a cardiac catheterization procedure, and interface device 3 is a catheter interface device. Catheter interface device 3 tracks the translational and rotational movement of a proxy catheter 7 provided in the form of a wire or like elongated flexible object. Catheter interface device 3 also provides selective tactile feedback resistance to the motion of the proxy catheter. The translation and rotation of the wire are converted into digital signals which are sent over line 9 to the simulation program. The virtual reality simulation is presented on a high resolution computer monitor 10. The feedback resistance is controlled by digital signals 11 sent by the program back to the catheter interface device over line 11.

In addition to the catheter interface device, the system comprises a switching mechanism, e.g., an ON-OFF foot switch 13. When activated in one mode, the switch triggers the software to simulate a release of radio-opaque contrast material. This allows for visualization of the coronary arteries, including the area in the artery that is obstructed, as in an actual procedure. In a second mode encountered when the virtual catheter enters an occluded portion of the virtual artery, the switch serves to simulate a syringe operated to selectively inflate and deflate a virtual balloon catheter. Randomly generated or user selected patient determinative data will determine how many successive inflations and deflations of the virtual balloon catheter are required to successfully remove the occlusion. In place of a simple footswitch providing a dual function, separate devices (e.g., proxy syringes) may be employed to actuate dye infusion and balloon inflation. Other devices that may be incorporated into the various types of catheterization procedures may be employed as well.

For enhanced effect, the screen simulations can be presented in 3-D. This is accomplished through the use of a stereographic system, e.g., CrystalEyes available from Stereo Graphics, Inc. of San Rafael, California. This product uses eyeglasses 15 having active liquid crystal shutters which open and close sixty times per second. The shutters are synchronized to monitor 10 via an infrared emitter 17 mounted on the front of the monitor, and a receiver 19 mounted on the glasses. Although not used in an actual angioplasty procedure, 3-D imagery provides a more sophisticated examination of the cardiac anatomy for these types of procedures. A surgeon's ability to see organs and tissues in 3-D, in laparoscopic procedures, and similar angioplasty procedures, will give physicians an enhanced ability to view and manipulate those organs and tissues.

Digital, compact-disc quality multi-media audio can be used to provide an instructional narration in the inventive system, as well as to provide soothing background music as is often provided during actual surgery. For this purpose, the illustrated system includes a CD-ROM drive 21, an audio mixer 23 (for mixing background music from the CD-ROM drive and a narrative "voice-over" stored on the computer harddrive), and a pair of amplified speakers 25.

The operating principles of the catheter interface device tracking mechanism are schematically illustrated in Fig. 2. To obtain translation tracking, a friction wheel 27 presses against the catheter wire 7. This causes the wheel to rotate when the wire is pushed or pulled.

An optical encoder 29 is attached to the friction wheel for measuring the rotation. Rotation measurement is obtained by passing the catheter wire through a semi-tight grommet 31. When the wire is rotated, a wheel 33 attached to grommet 31 rotates. An optical sensor 35 mounted adjacent to the wheel measures the rotation of wheel 33 and outputs a corresponding signal. At the same time, grommet 31 allows the catheter wire to slide translationaUy therethrough.

A feedback resistance force is achieved by having wire 7 slide through a wire pressing mechanism 37 actuated by a solenoid 39. When the solenoid is activated, it pushes a plunger 41 against one side of a sleeve 43, pressing the catheter wire 7 and thereby making it more difficult to move the same.

An exemplary catheter based simulation is the simulation of PTCA. As mentioned above, PTCA is a technique for relieving partial blockage of arteries in the heart. In the actual procedure, a wire-like guiding catheter is threaded through an artery in the leg, up to the blocked area in the heart. The surgeon can see where the wire is going by injecting radio-opaque contrast material into the artery, thereby rendering the artery visible on a fluoroscopic (X-ray) display. The physician directs the catheter's placement by twisting and pushing as it is guided through the artery. Once the end of the catheter is in the proper position, the surgeon can thread a balloon catheter onto the guiding catheter. Once the balloon catheter is properly placed, the surgeon can inflate the balloon to open the occluded portion of the artery so that regular blood flow is restored. More recently, a self-guiding balloon catheter have been used to perform the procedure, thus eliminating the preliminary step of placing a separate guiding catheter.

The exemplary simulation system shown and described in detail herein proceeds very much like an actual PTCA procedure, particularly one using a self-guiding balloon catheter. Additionally, a number of simulation enhancements are provided, as described in detail herein below. It will be appreciated that the principles of the invention could be

- 8 -

SUBSπWfE SHEET (RULE 26) applied in order to create a double catheter insertion simulating more closely the more conventional technique of first inserting a guiding catheter and then inserting a separate balloon catheter thereover.

The software monitors the translation and rotation of catheter wire, as the cardiologist manipulates it. Using this information, the program tracks the progress of the catheter through a simulated arterial tree forming part of a 3-dimensional internal body environment. The simulation program uses geometry data which can be created for each specific procedure (and virtual patient). The location and orientation of catheter 7 is converted into an image by the program, which image is displayed on computer monitor 10. The image has the same fluoroscopic appearance as a cardiologist sees during an actual procedure (but with the optional enhancement of 3-D imaging) (see the representative screen displays of Figs. 22-26). In addition, other views which are not possible during a procedure can be provided by the simulation program to enhance the instructional value of the simulation (see Figs. 27-29). Depending on the position and orientation of the virtual catheter within the virtual arterial tree, the simulation program may calculate and output to catheter interface device 3 a resistance force signal so that the cardiologist can experience the feel of the catheter insertion, just as he she would during the actual procedure. In particular, in the preferred embodiment, a tactile feedback force is generated when the virtual catheter enters an occluded portion of the virtual artery.

The system can add additional realism by displaying at the outset patient history and laboratory data screens (see Figs. 18-20). The system can also incorporate variations in complications corresponding to different patient conditions, the presentation of which can be randomly generated by the computer, or selected by the user.

In order to provide additional realism, the system can simulate procedural error conditions. For example, if the catheter wire is moved too abruptly, this triggers the simulation of a puncture in the arterial wall, and associated bleeding (see Fig. 25). If the virtual balloon catheter is inflated to an excessive amount, a realistic simulation of balloon rupture occurs.

As seen in Figs. 21-29, a simulated electrocardiogram (EKG) waveform can also be displayed on the monitor 10. Provision is made for changing the waveform from "abnormal" to "normal" during the course of the simulation. Provision can also be made for the random occurrence of abnormalities such as atrial fibrillation. Additional realism is provided by simulating pulsation of the arterial tree (and the catheter therein) in synchronism with the EKG.

Additional abilities that can be provided by the simulation system include, among others, interactive control of the point of view, recording of the procedure, and monitoring by an expert assistant. Changing the point of view is important. For example, viewing the arterial tree from different perspectives allows the cardiologist or student to better learn the 3-dimensional relationships of the arterial branches. This is difficult to master in practice since only a 2-dimensional projection of the 3-dimensional geometry is available in the fluoroscopic view used during an actual angioplasty procedure. The progress of the cardiologist can be recorded and played back for review and evaluation. A virtual "expert assistant" can be included to monitor the actions of the cardiologist during the simulation and can provide information such as warnings and explanations.

Many different catheter-based procedures can be simulated with the inventive system. The specific procedure is created by the data used in the interaction aspects of the simulation program. Thus, the same catheter interface device can be used with many different procedures while the simulation program is customized for each procedure. This enables the catheterization simulation system to be used to create training environments for many different clinical procedures and patient conditions. The patient conditions can be simulated or actual. With respect to actual conditions, those conditions could be taken from representative past patients for use in general training, or could be taken from a current patient for use in a pre-operative surgical rehearsal.

The logic flow of the exemplary PTCA simulation program (and representative screen displays thereof) are now described in detail, with reference to Figs. 3-29.

Figure 3 depicts three primary modules of the main program: error checking 45, case presentation 47 and calibration 49. In error checking module 45, features and function of the system are diagnosed for presence and functionality. As previously mentioned, a stereographic 3-D viewing system can be utilized by the system to enhance the simulation. To determine if the stereographic transmitter is present, a call to the device is made at step 49. If the transmitter is present, the video is adjusted for three dimensional perception at step 51. If the transmitter is not present, the stereographic viewing function is disabled at step 53 and the program proceeds without the visual enhancement. The system may

- 10 -

SUBSimiTE SHEET (RULE 26) alternatively utilize other types of visual perception enhancement devices, such as headmounted displays, and projection devices (including holographic projection devices).

As previously mentioned, the system utilizes a catheter interface device 3 that provides for direct human interaction with the computer simulation. The presence and functionality of catheter interface device 7 is determined at step 55. If the device is not detected or if it fails a diagnostic routine, the error is reported via the display and the simulator program will exit to the host operating system (step 57).

Given that catheterization interface device 3 passes the previous error checking routines, the program next proceeds to the detection of footswitch 13 used for dye infusion and balloon inflation (step 59). If the system fails to detect the footswitch, an error is visually reported and the program exits to the host operating system (step 61).

To synchronize the program polling rate, simulation timing functions, and threshold arrays, the program implements a routine to assess the speed of the host computer system (step 63). All synchronization variables are output to global data buffers to be utilized by time dependent subroutines, array determinant threshold subroutines, and polling count determinate subroutines within the simulation program. Next, the program proceeds to a case presentation module 47.

In case presentation module 47, a patient case and its relative data are retrieved (step 65) via a random selection subroutine 67, from an on-line database 69. This stage may be replaced or supplemented by a routine allowing the user to select a specific patient case from the case files, or to input data from a selected patient case not in the stored files. The latter would be particularly useful for conducting a preoperative rehearsal. The case specific data which is stored in memory in the preferred embodiment can include EKG data (normal and abnormal), occlusion location data and angioplasty success data, i.e., the required number of successive balloon inflation cycles required to remove the occlusion. (The angioplasty success data is artificial since in an actual procedure the number of balloon inflations required to remove an occlusion typically is not known in advance.) The case determinant variables are set in step 71.

Patient information is delivered via a multimedia presentation at step 12. This may include a patient history screen as shown in Fig. 18, an arteriogram as shown in Fig. 19, and an indication of hemodynamics as seen in Fig. 20. In addition to displaying patient information on the screen, corresponding or supplemental information is conveyed by a digitally recorded voice provided through the multi-media audio system. If footswitch 13 is activated at step 75, the presentation will proceed ("fast forward") to the next topic (step 77). So long as the program polling does not detect an activated footswitch, the program will proceed in a linear fashion through each topic, until the last topic is reached (step 79).

Before the simulation actually begins, the catheter interface device is calibrated within calibration module 49 (step 81). To calibrate catheter interface device 3, the catheter wire 7 is synchronized with the program by the user. The user is instructed by the program to pull the guide wire out while a status bar on the screen display of monitor 10 decreases to indicate the approach of the catheter starting point. This calibration is required for the program to ascertain the point of origin for screen representation and translation tracking. Rotation degree is also set to a starting position at this time.

Referring now to Fig. 4, upon completion of the calibration routine, the program will proceed to a catheter placement simulation module 83. At this stage, the screen displays a simulated fluoroscopic (X-ray) view (see Fig. 22), wherein the patient's rib-cage 84 is clearly seen. The arterial tree remains invisible until a subsequent dye infusion routine is executed. Once the dye infusion routine is executed, the arterial tree 86 including an occluded region 88, and the virtual catheter 90, become clearly visible (see Figs. 23-29). The background image can be created by digitizing an actual fluoroscopic image and, if desired for clarity, by modifying the digitized image data to remove unwanted structures and "noise" . In the present embodiment, this background remains inanimate throughout the simulation (as it generally would during an actual procedure). As also seen in Figs. 21-29, the screen presents along its bottom border a moving EKG waveform 92, based on the stored patient data.

In catheter placement simulation module 83, a timer subroutine 85 simulates real time by counting units that are determined by the polling rate time adjustment step 63. Referring to Fig. 5 showing the timer subroutine, the timer unit is defined as an integer with a value of 1. If the current sum timer value is greater than 1 at step 85a, then the value is increased by 1 unit at step 85b. The value is set to 1 upon initiation of the timer, at step 85c.

Referring back to Fig. 4, the program proceeds to an EKG data determination subroutine 87. Referring to Fig. 6, if an occlusion (blockage) is indicated in the case specific arterial dataset (blockage=l) at step 87a then the program will check to see that

- 12 -

SUBSππFTE SHEET RULE 26) the abnormal EKG data is stored in the program memory buffer (step 87b). If it is not yet stored, the case determinant data is retrieved at step 87c from the EKG database 87d. The EKG data base 87d can provide sinus rhythm patterns as well as atrial fibrillation data, and other case determinate cardiac events. The buffer space is given a local definition, i.e. CASE_ABNORMAL at step 87e, and the EKG data is placed in a common buffer (step 87f). Alternately if the correct buffer name is found at step 87b, then the program will proceed out of the subroutine.

Conversely, if it is determined that a blockage does not exist at step 87a, then the program will check to see that the EKG data is set to CASE_NORMAL at step 87g. If it is, the program will proceed out of the subroutine. Otherwise, the CASE_NORMAL data is retrieved at step 87h from the EKG database 87d. The buffer name is set to CASE_NORMAL at step 87i, and the data in the EKG data buffer is overwritten at step 87f. The program provides the onscreen EKG simulation based on this data in the EKG Buffer, which is derived from the stored case data.

Referring momentarily back to Fig. 4, the onscreen EKG display is produced and updated within subroutine 89. Referring to Fig. 7, the update EKG subroutine will check to see that the EKG data is translated into a usable array at step 89a. If an array is not detected, the EKG data is retrieved at step 89b from the EKG data buffer 89c and sorted into an array. The array is processed at step 89d by translating the value from the EKG buffer array 89c, to a floating-point number between 0 and 1. A value equal to 75% to 80% of the highest value in the translated array is set as an audible beep threshold value at step 89e. The adjusted array is next defined as the spline deformation array at step 89f . If it is determined at step 89a that the spline deformation array has already been set, the subroutine will bypass the array definition process.

Each time the program polls update EKG screen subroutine 89, the spline deformation array is advanced one place holder (step 89g). A range from the array is then used to deform the EKG screen spline at step 89h. The real patient data may be plotted and the EKG screen decay can be simulated. If it is determined at step 89i that the apex of the spline is at or above the defined beep threshold value, then an audio beep is generated by the sound system 89j, at step 89k.

The simulation can include, as a simulated complication, randomly generated periods of EKG irregularity, e.g., atrial fibrillation. This is illustrated in the screen display of Fig. 21, wherein the surgeon is instructed to stand-by until normal sinus rhythm resumes. At this point in an actual procedure, a drug would be administered to the patient. Such a step could be included in the present simulation, e.g., using footswitch 13 as a trigger mechanism. Alternatively, the simulation can be simplified by simply notifying the surgeon that the drug has been administered. Another feature that can be provided is a graphic time clock 94 (as shown in Fig. 21) notifying the surgeon of the time remaining before the drug takes effect (and sinus rhythm is resumed).

Referring again to the main program of Fig. 4, at subroutine 91 an arterial pulse is realistically simulated by synchronizing arterial and catheter movement with the EKG pulse data. Referring to Fig. 8, subroutine 91 will retrieve a spline deformation array at step 91a, from a spline deformation buffer 91b. The spline deformation array is the data used to represent the virtual arterial tree that will appear upon dye infusion. The configuration of the arterial tree can be adapted from commercially available 3-D graphics packages (such as from ViewPoint Data Labs of Orem, Utah) or can be independently generated based upon physical models or actual anatomies. Offset data points are extrapolated from the array data at step 91c. The geometry is offset (redrawn) and the screen is updated at step 91d. The program then proceeds to step 93 of the main program (Fig. 4), where the status of footswitch 13 is checked. At this stage, footswitch 13 is used to initiate the infusion of contrasting dye (subroutine 95). The default position of the switch is OFF and the program will simulate rapid dissipation of the dye into the bloodstream (as would occur in an actual procedure) when the footswitch is released. At step 93, a check is made to see whether the footswitch has been depressed. If not, dye infusion subroutine 95 is by-passed.

When the footswitch is depressed, the program proceeds to the dye infusion subroutine 95. Referring to Fig. 9, dye infusion is activated at step 95a, whereupon a ramp constrained, random flow of values is streamed into an infusion value definition (see 95b) used to determine the shade or concentration of the dye. By altering the value of the dye concentration at various points, the shade variations simulate the mixing of blood and dye. By ramping the value stream, simulated dye that is infused behind dye infused earlier will appear less diffused, thereby looking to be of higher concentration. By randomizing the shade determinative value stream as it is ramped upwardly, it is possible to create a strikingly realistic simulation of one fluid (in this case dye) diffusing into another fluid (in this case blood). The constraints on the ramped random value stream 95b can be adjusted to provide the appearance of different fluid viscosities and rates of diffusion. Alternatively, a particle system or procedural material map may be used.

The path of the dye is mapped (at step 95c) in accordance with the artery data set 95d. The flow of the dye is mapped progressively from the top of the artery down through the branches. When the footswitch is released, the mapped contrasting dye is reverse mapped to simulate diffusion in a downward direction, as if blood were flowing from above and "washing" the dye from the site.

If it has been determined at step 95e that a puncture in the arterial wall has been mapped on the arterial geometry, and footswitch 13 is in the ON position, the puncture origin 95f is mapped at step 95g, to simulate bleeding from the puncture into the surrounding body cavity. Such bleeding is shown at 96 in the screen display of Fig. 25. If no origin is detected, the program will proceed. As will be described, a puncture is caused to be mapped onto the geometry when the user moves the catheter wire 7 of the catheter interface device 3 too abruptly.

The next step in the main program is a catheter movement sampling subroutine 97. Catheter interface device 7 includes an electronic interface having a software driver for interpreting the signals produced by the catheter wire tracking sensors. As shown in Fig. 10, the data that is provided by the driver is sampled at step 97a, at a rate that has been determined by the program at step 63. Longitudinal translation 97b and rotation 97c of the wire through the catheter interface device are processed to determine the wire location relative to the calibration point position established in step 81. The Z-axis (corresponding to the catheter insertion depth) and the rotation data are placed in an array buffer in step 97d.

The program then proceeds to subroutine 99 for determining the location and orientation of the virtual catheter. As shown in Fig. 11, this is computed at step 99a from the Z-axis and rotation value array stored in array buffer 97d, a geometry offset value 99b, and the constraints provided by the artery data set 99b. By taking account of a geometry offset value (which is a product of the arterial geometry data and the EKG data), a simulated flutter or vibration of the catheter in synchronism with the arterial pulse is achieved. The catheter location and orientation data are placed in a data buffer at step 99d for use in a subsequent catheter update subroutine.

- 15 With the arterial tree illuminated by simulated fluoroscopic dye, the catheter is navigated through the arteries toward the occlusion region. Figs. 23-26 show the progression of virtual catheter 90 through arterial tree 86 and into occluded region 88. As can be seen in these screen displays, virtual catheter 90 has an angled or bent end portion 98. This allows for steering the catheter down different pathways, e.g., branches in the virtual arterial tree, based upon the rotational orientation of the catheter wire.

To determine whether the catheter has entered occluded region 88, within subroutine 101 it is determined if the spline origin (representing the catheter end) is within the blockage range stored as part of the case specific data. Referring to Fig. 12, the blockage range data 101a, and catheter location definition 101b are compared at step 101c. A local value of 1 is output at step lOld if the comparison determines that the catheter location is within the blockage range. Conversely, if the comparison indicates that the catheter is not within the occlusion region, a value of 0 is passed at step lOle.

If subroutine 101 returns a value of 1, then the program will output a signal to the catheter interface device to impart force feedback (a.k.a. haptic feedback, resistive feedback) to the catheter wire (at step 103). The force feedback signal can be provided at a single preset level, or could be provided in gradations dependent on the location of the catheter and the cross-sectional dimensions of the occluded region. Then, the program proceeds to a balloon inflation simulation module 105. If subroutine 101 returns a value of 0, the program will proceed to subroutine 107 for checking whether the catheter has punctured the arterial wall.

Puncture simulation in the illustrated program is based on a detected velocity of catheter wire 7 within the catheter interface device 3. This provides a realistic simulation in that during an actual procedure an abrupt movement of the catheter can often result in an arterial puncture. Likewise, in the simulation, if the catheter wire is moved abruptly by the user, the velocity of the catheter exceeds a preset level and thereby triggers the simulation of a puncture.

Referring to Fig. 13, in puncture detecting subroutine 107, the program will first check at step 107a for the presence of a previously defined puncture origin 107b. If a puncture origin 107b does exist, then the program checks at step 107c whether an audio response (e.g., verbal announcement of the error) through the multimedia sound system 107d is completed. During the audio response, the program continues to loop, resulting

- 16 -

SUBSTITUTE SHEET (RULE 26, in the simulation of bleeding from the puncture origin (see 96 in Fig. 25), due to execution of dye infusion subroutine 95. At this time, a flashing screen message may also appear to alert the user of the procedural mistake. Following completion of the audio response, the simulation proceeds to termination step 107e, whereupon the screen display can be frozen to preserve the unsuccessful result for evaluation.

If a puncture origin is not detected at step 107a, the presence of a previously logged velocity variable is determined at step 107f. If none is detected, the current time 107g is placed in a local variable definition (buffer A) at step 107h and the program proceeds. If a previously recorded value is detected in step 107f, then that variable is moved to a second buffer (buffer B), in step 107i. The current time 107g is placed in the primary buffer (A) at step 107j. The time between events is then calculated at step 107k. If the calculated elapsed time is determined to be below a preset value at step 107e, this indicates a velocity in excess of the velocity constraint. Under this condition, the puncture origin 107b is set to the current catheter location (step 107m). The program will then activate the audio response to the puncture event (step 107n), and program control returns to the main program. If the time between events calculated in step 107k is found (in step 107e) to be within the defined constraints, the program will proceeds to the next stage without a puncture origin having been defined.

An alternative method of detecting a puncture would be to utilize a collision detecting component, whereby a puncture would be simulated upon an impact of the virtual catheter tip against a wall of the virtual artery.

In the next main program step (subroutine 111), the catheter position and orientation are updated. The catheter is represented by a spline generated from data points. Referring to Fig. 15, to determine the length of the spline, the starting point (i.e. catheter origin Ilia), and the current location of the catheter 111b, are retrieved and processed in step 111c. In step Hid, the spline length definition computed in step 111c is combined with geometry offset data 11 le. The curvature of the spline (i.e. , spline deformation) is derived from the geometry offset data 11 Id, which is a product of the artery geometry and the EKG pulse information. If a puncture origin is present (11 If), the path of the spline will not be constrained by the geometry map at that point, thus allowing the catheter to pass through the puncture, e.g., as seen at 96 in the screen display of Fig. 25.

17 The rotation of the catheter wire is reflected by the position of the bent or angled wire end portion of the virtual catheter. The catheter is pointed or navigated on the path chosen by the user by rotating the wire (and bent end portion thereof). If it is determined in step lllg that the end of the catheter is located at a branch in the arterial data set lllh (as seen at 100 in the screen display of Fig. 24), the program will determine the direction of travel or trajectory at step llli based upon retrieved rotation data lllj. Finally, the spline is deformed and redrawn in step 111k, and program control proceeds.

Referring back to Fig. 4, when the catheter is determined to be within the occlusion range at step 101, and after force-feedback is activated at step 103, the program proceeds to balloon inflation module 105. Therein, the program will first magnify the occlusion region and change the viewing perspective (step 113), as seen in the screen displays of Figs. 27-29, in order to enhance visualization of the arterial structure and the effect of the procedure thereon. Contrasting dye is automatically activated, the visual components are adjusted to the new perspective, and the polling cycle is redirected to loop within the balloon inflation module. The required program components are replicated within this stage, i.e., catheter update 111, timer 85, EKG data 87, arterial pulse update 91, and footswitch status check 93. Within the balloon inflation module, footswitch 13 becomes operative to activate balloon inflation rather than dye infusion. If step 93 determines that the footswitch is activated, the program proceeds to a balloon inflation subroutine 115.

Referring to Fig. 14 showing balloon inflation subroutine 115, if the footswitch is depressed so that a value of 1 is detected at step 115a, then the program proceeds to step 115b wherein it is determined whether an inflation status buffer 115c has been previously established. If the buffer has been established, the value within the buffer is increased by one unit, at step 115d. The value from the inflation status buffer is then used to deform (inflate) the balloon tip 102 of the virtual catheter (step 115e) and then to correspondingly deform the occlusion geometry outwardly (step 1150- These effects are seen clearly in the screen display of Fig. 28.

If it is determined at step 115b that an inflation status buffer has not yet been previously established, then one is created at step 115g. The value is set to one unit at step 115h, and the value from inflation status buffer is then used to deform (inflate) the balloon tip of the catheter (step 115e), and then to correspondingly deform the occlusion geometry outwardly (step 115f). If the footswitch value is not equal to 1 at the time step 115a is polled, then the program will determine at step 115i if an inflation status buffer 115j has been previously established. If it has not, the program proceeds to loop within the main balloon inflation module 105. If it has, the buffer is reduced by 1 unit at step 115k, thus serving to decrementally deflate the balloon catheter. If the buffer is reduced to a value of 0 (step 1151), effectively deflating the balloon (as seen in Fig. 29), the buffer is released from memory at step 115m. If the condition exists that the buffer has been reduced by 1 unit (step 115k) but has not been reduced to a value of 0, then the decremented value from the inflation status buffer is then used to deform (in this case deflate) the balloon tip of the catheter, and then to correspondingly deform the occlusion geometry, at steps 115e and 115f .

On exiting balloon inflation subroutine 115, program control proceeds to a balloon rupture subroutine 117. A threshold value for inflation is set upon the case specific arterial dataset retrieval. This threshold should be equal to the value of units required to fully deform the blocked region of the artery so that it is of healthy dimensions. It is possible also to introduce die threshold at a lower value to provide for a risk management educational element to the program. Referring to Fig. 16, if the value retrieved from the inflation status buffer 117a is determined to exceed the threshold value at step 117b, then the balloon volume is set to 0 units (step 117c), abruptly deflating the balloon (as seen in Fig. 29) and thereby simulating balloon rupture. This action can be buffered over time to deflate at a visually correct rate. Various events such as fragmentation and vascular damage may be added to this scenario to enhance the program's realism. When a rupture has occurred, the rupture audio response 117d is activated at step 117e, and the program then proceeds to a final summation of the simulation step 117f.

The desired outcome (objective) of the simulation is clearance of the occlusion and restoration of normal sinus rhythm. If balloon rupture is avoided in subroutine 117, process control proceeds to subroutine 119 for determining when the occlusion has been successfully cleared. Referring to Fig. 17, the program will detect whether the balloon has been allowed to inflate sufficiently to cause complete deformation of the occlusion range cross-sections to the proper (corrected) diameter. If so, an occlusion value of 0 will be detected at step 119a. If the cross-sections have not been fully corrected, then the program will continue to loop.

- 19 -

UBSIIIUTE SHEET RULE 26) In an actual procedure, removal of a blockage typically requires repeated balloon inflation cycles. Similarly, in the simulation, repeated inflation cycles are required to successfully clear the occlusion. The required number of successive inflation cycles is determined by a success value definition which is part of the retrieved case specific data. For each polling cycle wherein it is determined that the blockage has been fully deformed by the balloon to the corrected cross-section, a count value is incremented at step 119c and a comparison is made at step 119d to see if the success value definition has been reached. If so, the program proceeds to a successful concluding summation at step 119e. The concluding summation may, e.g., include an announcement of the successful result, and provide a display of the elapsed time of the procedure. If the success value has not been reached, the program continues through the polling loop. Importantly, because of the inflation time limitation imposed by the balloon rupture subroutine 117, a user cannot achieve a successful clearance of the occlusion by simply maintaining the footswitch in the depressed (ON) position. Rather, it is necessary for the user to successively inflate the balloon and allow it to deflate (via the decrementing function of step 115k) at least several times, as in an actual procedure.

In the preferred embodiment, the software is programmed using the industry standard language C++ and OpenGL. The use of these two languages has several advantages:

• ability to port d e software to different hardware platforms. OpenGL has been licensed by many different workstation manufacturers including Windows NT, ATT, DEC, and DOS. OpenGL is an open, documented standard for describing 3- dimensional interactions with objects.

• ability of software to adapt to upcoming hardware innovations. OpenGL handles most graphics routines, and its use is scaled to the particular hardware that is available on the computer.

• compatibility with Nintendo/Silicon Graphics, Inc. hardware arrangements, as well as set-top boxes intended for interactive television. The availability of inexpensive platforms for VR will open opportunities for use in third world countries, rural settings and individual homes and offices.

In order to maintain real-time frame rates (30 fps), it is important to limit picture detail to the essential portions of the screen. The limited resource is the graphics rendering

- 20 -

SUBST1TUTE SHEET (RULE 26) speed (the number of polygons per second), and the software must know where to best allocate this resource to determine what features being displayed should have the most detail. In the exemplary cardiac catheterization simulation described herein, the real time graphics are limited to the arterial tree and the catheter therein; the remaining background of the simulated fluoroscopic view is inanimate (as it would be in an actual procedure). With this arrangement, real time rates are readily attainable.

Specific attention will now be given to the preferred system hardware for the above- described cardiac catheterization simulation system. There are a number of different hardware platforms that can be utilized for VR applications. Current configurations include:

High end systems. VR can be run on systems such as the Silicon Graphics, Inc. RealityEngine. These are costly and logistically the most difficult to transport due to their large size. The Silicon Graphics Indigo 2 Extreme workstation has the graphics and computer power necessary for many VR applications, and is less costly.

Personal computer systems. Although the current generation of personal computers could be used for some limited VR applications, they lack the specialized hardware found in graphics workstations which facilitates the production of real-time complex 3-D manipulation of objects. Personal computers are optimized for 2-D graphics, and as a result, they are ideal for playback of digitized linear sequences which can be obtained from 3-D graphics systems (VR). Personal computers are also well suited for the display of 2-D representations of 3-D phenomena.

The angioplasty application of the present invention was developed on a Silicon Graphics Indigo 2 Extreme workstation. A complete hardware specification for this application is set forth below:

• Silicon Graphics Indigo 2 Extreme graphic hardware (MIPS R4000/100 Mhz, 64MB RAM, 1GB hard drive, 19" SGI monitor, CD ROM drive, DAT tape drive.

• stereographic equipment (e.g., CrystalEyes eyeglasses and emitter + cable).

• audio equipment (one pair of stereo power speakers, Mackie 1202 (or compatible) mixer or another pair of power speakers.

• audio CD for providing continuous background music (narrative "voice- over" can be recorded onto hard drive).

- 21 -

SUBSΪITUTE SHEET (RULE 26) • miscellaneous (two stereo mini-male to RCA male cables, six RCA female to 1/4" male phone jack, power strip).

• catheter interface device (as described in detail herembelow).

• foot pedal switch peripheral, e.g., as is available from Immersion Corporation.

In the preferred embodiment, me catheterization simulation system of the present invention employs a virtual catheter interface produced for use in applicants' system by Immersion Corporation. This device is now described in detail with references to Figs. 30- 37.

Catheter interface device 3 tracks the movement of wire 7 in a 3-D space, with the movement of the wire constrained to two degrees of freedom. This provides a realistic simulation of an actual catheter insertion procedure, wherein once the catheter is inserted into a patient, it is limited to two degrees of freedom. More particularly, within a patient's body, the catheter wire is constrained to translation (in-out motion) and rotation.

The catheter wire may be formed of a variety of known materials, e.g., metal or nylon. Generally, the gauge should not be bigger than 050 (0.050" diameter), and the insertion length should be at least 14". The catheter wire can be an actual catheter wire, such as is commercially available, e.g., from Cordis Corporation of Miami Lakes, Florida. An actual catheter should be modified such that any tool at the end (such as any cutting edges or balloon) is removed leaving only the handle and the shaft.

Catheter wire 7 can include an end mounted handle or "grip" portion 121. The "grip" portion can be any conventional device used to manipulate a catheter. Alternatively, the catheter shaft could be gripped directly. While the particular embodiment described herein comprises a wire, it should be understood that other catheters and catheter-like members could be used as well, including particularly elongated flexible tubular catheters.

An electronic interface 123 serves to electrically couple the force feedback actuator and motion sensors contained in catheter interface device 7 to the computer 5. An electronic interface that is particularly adapted for the present invention is commercially available from Immersion Corporation. This is basically the same electronic interface designed for use with Immersion's PROBE. Unlike the PROBE setup, a serial interface is configured to be bi-directional, so as to allow for transmission of a tactile feedback signal from the computer to the force feedback actuator. The electronic interface has six channels corresponding to the six degrees of freedom of the Immersion PROBE. However, with the present invention, the electronic interface requires use of only two channels for the tracking of catheter wire 7, since the catheter wire is constrained to two degrees of freedom. A third channel is used for transmission of the tactile feedback signal.

The electronic interface includes a software driver. The update speed of the device is between 500Hz and 2kHz - this speed cannot be exceeded with a single serial port. If necessary to achieve a cleaner "feel," the device can be connected with two serial ports running simultaneously, whereby a 2X speed improvement is obtained by doubling the data transmission capability. Other methods of improving interface speeds between the catheter interface device and the computer include standard and high speed ethemet connections, use of SCSI or direct memory access parts (DMA), or a variety of more rapid standard interfaces. In addition, dedicated hardware to facilitate input/output can be used, such as boards for rapid data acquisition.

The operative components of catheter interface device 3 are mounted on top of the housing of electronic interface 123 within a separate metal housing 125 having a removable cover 127. A rubber-like tubular member 129 is used to assure free passage of wire 7 through the housing cover 127. Housing 125 could instead be modeled after the external bodypart(s) located within the vicinity of the catheter insertion location, e.g., a leg or arm, in order to render the simulation components more "transparent" to the user.

The operative components of catheter interface device 3 comprise a block assembly 131 forming a through-passage 133 (see Fig. 37) and mounting locations for a first input transducer (sensor) 135 that senses the translational movement of wire 7 and produces a corresponding electrical signal for input to computer 5, an output transducer (actuator) 137 that imparts a resistance force to wire 7 responsive to a tactile feedback signal received from computer 5, and a second sensor - a rotational transducer — 139 for tracking the rotation of wire 7.

Block assembly 131 comprises a plurality of blocks formed of aluminum or other lightweight material, such as plastic. The blocks may be cast, molded, and/or machined as a monoblock members, with the aforementioned sensors and actuator attached thereto and/or incorporated therein.

A variety of transducers, readily available in the commercial market are suitable for use in the present invention. For example, the input transducers (sensors) can include

- 23 - encoded wheel transducers, potentiometers, optical encoders, eta. The output transducer (actuator) could be a stepper motor, servo motor, magnetic particle brake, friction brake, pneumatic actuator, etc. Hybrid or bi-directional transducers that pair input and output transducers together could also be used in place of separate sensor and actuator devices. A simple bi-directional transducer that could be used is a permanent magnet electric motor/generator.

It should also be noted that the present invention can utilize both absolute and relative sensors. An absolute sensor is one in which the angle of the sensor is known in absolute terms, such as with an analog potentiometer. Relative sensors only provide relative angle information, and thus require an initial calibration step, as is provided in the present simulation system.

In the preferred embodiment, actuator 137 is a solenoid capable of producing a force output of at least 5 ounces for imparting tactile feedback in the form of passive resistance (frictional drag) on wire 7. As best seen in Figs. 35 and 37, a solenoid plunger 141 has attached to its end a lower pressing plate 143. Lower pressing plate 143 engages wire 7 against an upper pressing plate 145 by applying a force in a direction substantially perpendicular to the translation direction of wire 7. This produces a frictional force along the translation direction of wire 7. Such solenoids, controllable to a thousand or more selectable resistance force values, are available commercially from, e.g., Guardian Electric of Woodstock, Illinois. It will be appreciated that other actuator devices may be employed in the present invention. For example, the actuator could be a pneumatic or hydraulic device which applies a force to the wire. It will be appreciated by those skilled in the art that the choice of electromechanical, electromagnetic, pneumatic or hydraulic actuation will depend largely on the required response time, and the cost and complexity of the device. Electromechanical/electromagnetic transducers are generally preferred as they typically have a fast response time, are low cost, and are simpler and smaller than hydraulic and pneumatic devices.

One or both of upper and lower press plates 143, 145 can comprise a resilient friction (brake) pad 147 (shown attached to upper press plate 145), for providing a smooth braking action to wire 7. Materials appropriate for the resilient brake pad will be apparent to those of ordinary skill in the art. As best seen in Figs. 36 and 37, the translation sensor incorporates a friction wheel 149, which wheel is mounted on a rotatable shaft 151. The friction wheel engages wire 7 with a normal force such that translation of wire 7 causes rotation of shaft 151. As best seen in Fig. 35, shaft 151 is coupled to a disk 153 made of clear plastic material and having a number of dark radial bands 155 formed on its circumference. As best seen in Fig. 36, a photodetector pair 157a,b including a light source and a detector is positioned on opposite sides of disk 153 in alignment with bands 155. As disk 153 rotates about its central axis, bands 155 alternatingly cause light to impinge and not impinge upon the detector, thereby creating an electronic signal which is passed to electronic interface device 123 over a bus wire 157. In the illustrated preferred embodiment of catheter interface device 3, translational movement of the wire is tracked with a resolution of at least 0.002". One suitable type of sensor is an optical encoder wheel type input transducer (Model SI) marketed by U.S. Digital of Vancouver, Washington. The same basic type of sensor can be used for rotation sensor 139.

In the preferred embodiment, the rotation sensor 139 includes a disk 159 rotatably coupled with wire 7. Disk 159 is preferably made from a clear plastic material and has a number of dark radial bands 160 formed on its circumference. A photodetector pair 161 including a light source and a detector is positioned on opposite sides of disk 159 in alignment with bands 160. As disk 159 rotates about its central axis, the bands alternatingly allow light emanating from the light source to impinge and not impinge upon the detector, thus producing an electronic signal which is also passed to electronic interface device 123 over bus wire 157. A second photodetector could be provided so as to allow determination of the rotation direction, as is well known to those skilled in the art.

The engagement between wire 7 and the central aperture of disk 159 is preferably accomplished by the formation of a friction seal. This can be provided by a resilient (e.g. , rubber or plastic) grommet 163 mounted within the hub of disk 159. Grommet 163 receives wire 7 with a semi-tight fit causing disk 159 to rotate coaxially with the wire, while at the same time allowing relatively free wire translation. Preferably, the mass of the rotating transducer parts is kept very small so that it only takes a small amount of friction to ensure coaxial rotation of wire 7 and disk 159 without slippage. Because the level of friction is so small, it does not substantially impede translational motion (i.e., in- out motion) of wire 7. Alternatively, wire 7 could include a lengthwise extending rib, slot

- 25 -

SUB$TΠUTESHEΠG«IU26) or flat, which is engaged with a corresponding key- way provided on disk 159. While more costly and difficult to manufacture, the use of a non-circular wire and corresponding key- way would ensure a slip-free connection between the wire and the disk, without introducing translational friction.

The principles of the present invention can also be applied to create an intravenous (IV) line insertion simulator. In the IV insertion simulator, a hardware device and software work together to reproduce the visual and physical experience of inserting an IV needle into an artery. The simulator would work by having the user grasp a proxy needle connected to a position monitoring device. Preferably, a custom IV hardware interface device would be designed for this purpose. Alternatively, the Immersion PROBE could be used for performing a general simulation (without tactile feedback). The insertion would be carried out by interacting with a simulation of the activity on a computer generated display. The display would show the needle as the user moves it along the part of the body into which the needle is to be inserted. As the needle is pushed into the skin, the simulation software would detect the contact and display an appropriate view of the process. For instance, if the needle were properly inserted into an artery, then a small amount of bleeding could be displayed along with a change in the shape of the skin where the needle has passed under. In addition, a force mimicking that encountered by the user during an actual IV insertion could be generated by the IV interface device, in response to commands from me simulation program, similar to the arrangement cardiac catheterization simulator described in detail herein.

The concepts of the present invention are applicable to provide a realistic simulation of a wide range of other medical procedures, including particularly invasive surgical procedures wherein a device is inserted into an internal body passageway, or directly into body tissues. The procedures which can be simulated using the present inventive techniques span every specialty of medicine, including but not limited to cardiology, invasive radiology, neurosurgery, neurology, anesthesiology and, of course, general surgery. Procedures that can be simulated with the inventive system include, but are not limited to:

• placement and/or use of therapeutic mechanical devices, e.g., angioplasty, stent placement and coil placement for aneurism treatment.

• intravenous (IV) line and central line placement, e.g., jugular, femoral. • introduction into the body of materials or devices to enhance the ability to discern internal body parts, e.g., angiography, intra-coronary ultrasound devices, and fiber-optic scopes.

• placement of devices to obtain tissue and fluid samples from the body, e.g., lumbar puncture, and liver and kidney biopsy.

• needle placement, e.g., epidural, phlebotomy simulation, deployment of therapeutic agents at the cite of injury/disease (e.g., use of a catheter for precise placement of chemotherapeutic agents at the cite of an intravascular tumor).

• placement and manipulation of devices to remove fluids or tissues, e.g., urinary catheters, and pleural fluid removal.

• devices for trauma management, e.g., chest tube placement.

• devices for airway management, e.g., fiber-optic laryngoscope, rigid laryngoscope, and intubation procedures.

• devices for assessing disease and pathology, e.g., fiber-optic scope manipulation (e.g., cystoscopes, bronchoscopy, flexible sigmoidoscopy, colonoscopy, and endoscopy).

• devices for minimally invasive surgery, e.g., laparoscopic instrumentation. The present invention has been described in terms of preferred and exemplary embodiments thereof. Other embodiments, modifications and variations within the scope and spirit of the appended claims will, given the benefit of this disclosure, occur to persons of ordinary skill in the art.

Claims

1. A computer based catheterization simulation system, comprising: first memory means for storing image data representative of a virtual internal body environment; second memory means for storing image data representing a virtual catheter extending wititin said internal body environment; display means for displaying images corresponding to said image data stored in said first and second memory means; first input means for receiving an output signal from a first computer peripheral device that tracks the translational movement of an elongated relatively flexible member, representing the virtual catheter, without a physical model corresponding to said virtual internal body environment; first calculating means for calculating position data representing the position of the virtual catheter within said virtual internal body environment, based on die signal from the peripheral device; and first updating means for updating the image data representing the virtual catheter extending within said virtual internal body environment, based on said calculated position data.

2. A catheterization simulation system according to claim 1, wherein: the end of said virtual catheter is angled; said peripheral device tracks the rotational movement of the elongated relatively flexible member and outputs to the first input means a signal based thereon; said first calculating means calculates orientation data representing an orientation of the angled end of the virtual catheter, based on d e signal from the first peripheral device; and said updating means updates the image data representing the virtual catheter based on the calculated orientation data.

3. A catheterization simulation system according to claim 1 , further comprising: determining means for determining the location of the virtual medical instrument within the virtual internal body environment; and output means for outputting a tactile feedback response signal to said first computer peripheral device based on said location of the virtual catheter, said tactile

- 28 -

T TE SHUT RULE 26) feedback response signal serving to cause a mechanism of said first computer peripheral device to impart to said elongated relatively flexible member a tactile feedback force.

4. A catheterization simulation system according to claim 3, wherein said tactile feedback force is a force resisting movement of said elongated relatively flexible member.

5. A catheterization simulation system according to claim 1 , further comprising: second calculating means for calculating the velocity of the virtual catheter within the virtual internal body environment, based upon the signal from the peripheral device; and second updating means for modifying the image data representative of the virtual internal body environment, based on the calculated velocity data.

6. A catheterization simulation system according to claim 5, wherein the second updating means modifies the image data representative of the virtual internal body environment when the calculated velocity of the virtual catheter exceeds a preset velocity.

7. A surgical catheterization simulation system according to claim 6, wherein me second updating means modifies the image data representative of d e virtual internal body environment to simulate a puncture of an internal body passageway, and associated bleeding.

8. A catheterization simulation system according to claim 1, further comprising third memory means for storing image data representing a simulated real-time wave form of a signal representing a physiological parameter, and wherein said display means displays said simulated real-time wave form together with said images corresponding to said image data stored in said first and second memory means.

9. A catheterization simulation system according to claim 8, wherein said simulated real-time waveform is an electrocardiogram (EKG) waveform.

10. A catheterization simulation system according to claim 9, further comprising second updating means for modifying me image data representative of the virtual internal body environment, based on the image data representing the EKG waveform.

11. A catheterization simulation system according to claim 10, wherein the virtual internal body environment comprises a vascular passageway, and the image data modification based on the simulated real-time waveform simulates a pulsation of the vascular passageway in synchronism with the EKG waveform.

12. A catheterization simulation system according to claim 1 , further comprising:

- 29 - a second input means for receiving a signal from a second computer peripheral device; and second updating means for modifying me image data representative of the virtual internal body environment, based on the signal from said second computer peripheral device.

13. A catheterization simulation system according to claim 12, wherein the virtual internal body environment comprises a vascular passageway, and said image data modification based on the signal from the second computer peripheral device simulates the release and flow of contrasting dye through the vascular passageway.

14. A cadieterization simulation system according to claim 1 , further comprising: a second input means for receiving a signal from a second computer peripheral device; and second updating means for modifying the image data representing the virtual catheter extending witivin said virtual internal body environment based on me signal from said second computer peripheral device.

15. A cadieterization simulation system according to claim 14, wherein the virtual internal body environment comprises a vascular passageway having an occlusion therein, and said image data modification based on the signal from the second computer peripheral device simulates the inflation of a balloon catheter within said occlusion.

16. A catheterization simulation system according to claim 15, wherein an inflation amount of the simulated balloon catheter increases with time while a switch of the second computer peripheral device is held in an ON position.

17. A catheterization simulation system according to claim 16, further comprising third updating means for modifying the image data representative of the virtual vascular passageway, based on the inflation amount of the simulated balloon catheter.

18. A catheterization simulation system according to claim 17, wherein me image data modification based on die inflation amount of d e simulated balloon catheter simulates the clearing of the occlusion.

19. A catheterization simulation system according to claim 16, wherein the image data modification based on the signal from the second computer peripheral device simulates the rupture of d e simulated balloon catheter in the event the switch is held in die ON position for a time period exceeding a preset time limit.

20. A catheterization simulation system according to claim 1 , wherein said image data representative of a virtual internal body environment, and said image data representing a virtual catiieter, comprise 3-D image data.

21. A cadieterization simulation system according to claim 11, wherein said vascular passageway comprises an arterial tree.

22. A catheterization simulation system according to claim 13, wherein said vascular passageway comprises an arterial tree.

23. A cadieterization simulation system according to claim 15, wherein said vascular passageway comprises an arterial tree.

24. A computer based medical procedure simulation system, comprising: first memory means for storing image data representative of a virtual internal body environment; second memory means for storing image data representing a virtual medical instrument extending within said internal body environment; display means for displaying images corresponding to said image data stored in said first and second memory means; first input means for receiving an output signal from a first computer peripheral device tiiat tracks me movement of a physical member representing me virtual medical instrument; first calculating means for calculating position data representing the position of die virtual medical instrument within said virtual internal body environment, based on d e signal from me peripheral device; first updating means for updating the image data representing me virtual medical instrument extending within said virtual internal body environment, based on said calculated position data; second calculating means for calculating the velocity of the virtual medical instrument witiiin the virtual internal body environment, based upon the signal from the peripheral device; and second updating means for modifying the image data representative of the virtual internal body environment, based on the calculated velocity data.

25. A medical procedure simulation system according to claim 24, wherein the second updating means modifies the image data representative of d e virtual internal body

- 31 -

4UBSΠΠΠESHEEΓ (RULE26) environment when the calculated velocity of die virtual medical instrument exceeds a preset velocity.

26. A medical procedure simulation system according to claim 25, wherein the second updating means modifies the image data representative of die virtual internal body environment to simulate a puncture of an internal body passageway, and associated bleeding.

27. A computer based medical procedure simulation system, comprising: first memory means for storing image data representative of a virtual internal body environment; second memory means for storing image data representing a virtual medical instrument extending witiiin said internal body environment; display means for displaying images corresponding to said image data stored in said first and second memory means; first input means for receiving an output signal from a first computer peripheral device tivat tracks the movement of a physical member representing me virtual medical instrument; first calculating means for calculating position data representing the position of die virtual medical instrument within said virtual internal body environment, based on die signal from me peripheral device; first updating means for updating die image data representing the virtual medical instrument extending within said virtual internal body environment, based on said calculated position data; and second updating means for modifying the image data representative of the virtual internal body environment, based on d e signal from said peripheral device, to simulate a puncture of an internal body passageway, and associated bleeding.

28. A computer based medical procedure simulation system, comprising: first memory means for storing image data representative of a virtual internal body environment; second memory means for storing image data representing a virtual medical instrument extending within said internal body environment; display means for displaying images corresponding to said image data stored in said first and second memory means;

- 32 - first input means for receiving an output signal from a first computer peripheral device tiiat tracks the movement of a physical member representing die virtual medical instrument; first calculating means for calculating position data representing the position of the virtual medical instrument within said virtual internal body environment, based on me signal from me peripheral device; first updating means for updating die image data representing die virtual medical instrument extending within said virtual internal body environment, based on said calculated position data; and third memory means for storing image data representing a simulated real-time wave form of a signal representing a physiological parameter; wherein said display means displays said simulated real-time wave form together with said images corresponding to said image data stored in said first and second memory means.

29. A medical procedure simulation system according to claim 28, wherein said simulated real-time waveform is an electrocardiogram (EKG) waveform.

30. A medical procedure simulation system according to claim 28, further comprising second updating means for modifying the image data representative of die virtual internal body environment, based on me image data representing d e simulated real¬ time waveform.

31. A medical procedure simulation system according to claim 29, further comprising second updating means for modifying the image data representative of die virtual internal body environment, based on the image data representing the simulated EKG waveform.

32. A medical procedure simulation system according to claim 31, wherein e virtual internal body environment comprises a vascular passageway, and die image data modification based on d e simulated real-time waveform simulates a pulsation of d e vascular passageway in synchronism with the EKG waveform.

33. A medical procedure simulation system according to claim 32, wherein said vascular passageway comprises an arterial tree.

33

SUBSTITUTE

34. A medical procedure simulation system according to claim 29, wherein me image data representing the EKG wave form includes EKG irregularity data for generating periods of EKG irregularity.

35. A medical procedure simulation system according to claim 34, wherein said EKG irregularity is randomly generated.

36. A medical procedure simulation system according to claim 28, further comprising second updating means for modifying d e image data representing die simulated real-time waveform based upon a change in die image data representative of die virtual internal body environment.

37. A medical procedure simulation system according to claim 29, further comprising second updating means for modifying the image data representing die EKG waveform based upon me existence or non-existence of data representing a simulated vascular abnormality in the image data representative of the virtual internal body environment.

38. A medical procedure simulation system according to claim 37, wherein e simulated vascular abnormality comprises a vascular occlusion.

39. A computer based medical procedure simulation system, comprising: first memory means for storing image data representative of a virtual internal body environment; second memory means for storing image data representing a virtual medical instrument extending within said internal body environment; display means for displaying images corresponding to said image data stored in said first and second memory means; first input means for receiving an output signal from a first computer peripheral device tiiat tracks the movement of a physical member representing the virtual medical instrument; first calculating means for calculating position data representing the position of e virtual medical instrument within said virtual internal body environment, based on me signal from the peripheral device; first updating means for updating the image data representing die virtual medical instrument extending within said virtual internal body environment, based on said calculated position data; and

- 34 -

SUBSIIIUIE SHEET RU second updating means for modifying me image data representative of die virtual internal body environment so as to simulate die infusion of one fluid into another.

40. A medical procedure simulation system according to claim 39, wherein the simulated infusion of one fluid into another comprises simulated bleeding based on a calculated velocity of die virtual medical instrument.

41. A medical procedure simulation system according to claim 40, wherein die velocity of the virtual medical instrument is calculated from said calculated position data.

42. A computer based medical procedure simulation system, comprising: first memory means for storing image data representative of a virtual internal body environment; display means for displaying images corresponding to said image data stored in said first memory means; and updating means for modifying die image data representative of the virtual internal body environment so as to simulate the infusion of one fluid into another.

43. A medical procedure simulation system according to claim 42, wherein me simulated infusion of one fluid into another comprises a simulated infusion of contrasting dye into a bloodstream.

44. A medical procedure simulation system according to claim 42, wherein me simulated infusion of one fluid into another is accomplished by streaming into a stored infusion value definition a ramp constrained random flow of values, to tiiereby vary the concentration of said one fluid at various points.

45. A medical procedure simulation system according to claim 43, wherein me simulated infusion of contrasting dye into a bloodstream is accomplished by streaming into a stored infusion value definition a ramp constrained random flow of values, to thereby vary die concentration of die dye at various points.

46. A computer based catheterization simulation system, comprising: first memory means for storing image data representative of a virtual internal body environment; second memory means for storing image data representing a virtual catheter extending witiiin said internal body environment; display means for displaying images corresponding to said image data stored in said first and second memory means;

- 35 -

SUBSmUΪE SHEET (RULE 26) first input means for receiving an output signal from a first computer peripheral device tiiat tracks the translational movement of an elongated relatively flexible member representing die virtual catheter; first calculating means for calculating position data representing die position of die virtual catheter witiiin said virtual internal body environment, based on die signal from the peripheral device; first updating means for updating the image data representing die virtual catheter extending within said virtual internal body environment, based on said calculated position data; a second input means for receiving a signal from a second computer peripheral device; and second updating means for modifying me image data representing die virtual catheter extending witiiin said virtual internal body environment based on die signal from said second computer peripheral device; wherein the virtual internal body environment comprises a vascular passageway having an occlusion therein, and said image data modification based on d e signal from the second computer peripheral device simulates d e inflation of a balloon catiieter witiiin said occlusion.

47. A cadieterization simulation system according to claim 46, wherein an inflation amount of the simulated balloon catheter increases with time while a switch of me second computer peripheral device is held in an ON position.

48. A cadieterization simulation system according to claim 47, further comprising tiiird updating means for modifying die image data representative of die virtual vascular passageway, based on die inflation amount of the simulated balloon catheter.

49. A catheterization simulation system according to claim 48, wherein d e image data modification based on the inflation amount of the simulated balloon catiieter simulates the clearing of die occlusion.

50. A catheterization simulation system according to claim 47, wherein e image data modification based on the signal from the second computer peripheral device simulates me rupture of die simulated balloon catiieter in die event die switch is held in die ON position for a time period exceeding a preset time limit.