WO1993014454A1

WO1993014454A1 - A sensory integrated data interface

Info

Publication number: WO1993014454A1
Application number: PCT/US1993/000163
Authority: WO
Inventors: Bruce Nappi; William Schroeder
Original assignee: Foster-Miller, Inc.
Priority date: 1992-01-10
Filing date: 1993-01-08
Publication date: 1993-07-22
Also published as: US5751260A

Abstract

A system for interfacing between a user and a computer includes a microvideo display monitor, which may be mounted in a position which is substantially fixed relative to a given eye of the user, and a video input device which obtains an image of the given eye. The video input device is maintained in a fixed spacial relationship with the microvideo display monitor. From the image of the user's eye obtained by the video input device, the location on the display on which this eye is focused may be determined. A computer which controls the generation of selected displays on the microvideo display monitor responds to the eye focus location to perform predetermined functions. A voice or speech recognition unit may also be used to provide inputs to the computer. The computer may then perform predetermined functions in response to any combination of speech, and eye location data. The system may also be used for identification of the user and to determine whether the user is impaired.

Description

A SENSORY INTEGRATED DATA INTERFACE

Field of the Invention

The present invention is related to man-machine interfaces. More particularly, the present invention is related to interfaces between a computer operator and a computer. The present invention is particularly- applicable for providing a hands-free input interface to portable computers.

Background of the Invention

Although computer technology has advanced significantly since its first inception, the development of the interface between a human user and the computer has been relatively stagnant.

Most advanced man-machine interfaces are limited to keyboard, mouse, or trackball control in connection with some advanced graphics packages on large area video displays. Such interfaces are sometimes impractical in applications where portable and mobile data entry stations are required. The interfaces are particularly problematic where work surfaces are limited or where hands-free operation of a computer is required. For example, maintenance of in-service •aircraft is often done on the flight line, in engine compartments or wiring panels. The lack of panel and work surface space in these areas restricts the ability of an operator to use keyboards and large computer displays.

Current data interfaces also fail to make full use of all types of means for providing input to a computer beyond touch at a keyboard or other device. Though many types of inputs are known and used, they have yet to be collectively and successfully integrated into a data interface.

Accordingly, it is an object of the present invention to provide an improved man-machine data interface.

Summary of the Invention

To accomplish the foregoing and other objects of the invention, there is provided a system for interfacing between a user and a computer which includes a microvideo display monitor and a device for mounting this monitor in a position which is substantially fixed relative to a given eye of the user. A video input device is used to obtain an image of the given eye. This device is maintained in a fixed spatial relationship with the microvideo display monitor after it has been calibrated with this display.

With such a system, a microvideo display may be used to generate selected displays under control of the computer. From the image obtained of the given eye of the user, an output to the computer is provided, which output is indicative of the location on the monitor where the given eye is focused. Such a function is also known as an eyetracker. The eyetracker may be used by the computer to control a cursor location on the microvideo display monitor. The computer also may perform other predetermined functions in response to the eyetracker and the location on the monitor where the given eye is focused,

The system of the present invention may also include an audio receiver, such as a microphone, to receive inputs from the user to the computer and may generate inputs to the computer which are interpretations of the audio signals received from the user. The inputs generated as interpretations of the audio signals from the user may include commands generated by a speech recognizer to instruct the computer to perform different functions for different recognized speech.

The computer may also perform functions in response to both recognized speech and the location on the display on which the given eye of the user is focused. Even more functions may be performed in response to the location on which the given eye is focused, and the duration of time during which the eye is focused on that location.

The system of the present invention may also be used to perform identification of the user by sensing personal characteristics from the speech recognizer and the eyetracker, for example. Similarly, such characteristics could also be used to determine whether the user is impaired.

In another aspect of the present invention, the computer is at a location remote from the user, and the system includes a transceiver which allows bilateral communication with the computer.

Other aspects and advantages of the present invention will be more completely understood in view of the detailed description below, taken in conjunction with the accompanying drawing.

Brief Description of the Drawing

Figure 1 is an isometric view of a data interface in accordance with the present invention;

Figure 2 is a top, plan, partially cross-sectional view of the embodiment of Figure 1;

Figure 3 is a flow chart describing the operation of an eyetracker driver;

Figure 4 is a flow chart describing the opeation of a voice recognition unit driver;

Figure 5 is a flow chart describing how the operating system of a computer handles interrupts from the voice recognition unit and the eyetracker;

Figure 6 is a schematic block diagram of a portable unit for use in connection with the present invention.

Detailed Description

Figure l illustrates a preferred embodiment of a data interface in accordance with the present invention. In Figure 1, an integrated microvideo display and eyetracker unit 10 is attached to a headband 12 which may be worn by a use . Many types of headbands or other mounting devices may be used to hold the unit 10 in a position which is fixed relative to a given eye of the user. An eyetracker determines the location on a video display at which the computer operator is looking and controls a cursor on the display to coincide with the determined location. The integration of an eyetracker with a microvideo display as in unit 10, provides hands-free control of the cursor location on the microvideo display. Moreover, the microvideo display is not bulky and does not require substantial counterspace or work area. A communication device 16, e.g. a wire or cable, is provided to transmit information between the integrated microvideo display and eyetracker unit 10 to a computer 18 (not shown to scale) . The computer 18 performs the various computations required by these devices to move the cursor.

An interface such as in Figure 1 allows complete portable computer interaction without a keyboard or tabletop monitor. The computer 18 may be portable, and carried, for instance, using a backpack. To minimize the amount of computer hardware which needs to be transported, the computer 18 may include a portable unit including the hardware elements for the videodisplay and eye tracking hardware which communicates with a main computer which can be considered a remote base station. This embodiment will be described in more detail below in connection with Fig. 6.

A description of the integrated microvideo display and eyetracker unit 10 will now be provided. Referring to Figure 2, the integrated unit 10 includes a housing 20 which supports a microvideo display 22 and hardware for an eyetracker unit in a fixed spatial relationship so as to maintain a fixed relationship of the display coordinate system and the eyetracking coordinate system. The eyetracker hardware obtains an image of the eye and provides a signal which is indicative of this image. One embodiment includes an infrared illuminator 24 which illuminates the eye, and a lens 26 for focusing reflections from the eye onto a video chip 28. The video chip 28 is a photodiode array similar to the light-sensitive portions of common cameras. The photodiode array either is infra—red sensitive or is provided with an infra-red transmitting filter. The video chip 28 sends its signals through the communication device 16 to the computer 18 for processing to determine the eye location relative to objects on the microvideo display 22. If necessary, the housing also supports a dichroic mirror 30 which allows the operator to look directly at the microvideo display 22 without interference from the eyetracking hardware. The housing 20 may also support a lens 32 between the dichroic mirror 30 and microvideo display 22 in order to enhance the picture obtained from the microvideo display 22.

Microvideo displays, such as display 22 in Figure 2, and eyetracking hardware, for instance, elements 24, 26 and 28 in Figure 2, are well-known, commercially—available devices. However, their combination as a computer interface was not known.

An eyetracker suitable for use in the present invention is available from Iscan, Inc. of Cambridge, Mass. The product includes a circuit board for performing high speed image processing, eyetracking hardware which may be integrated such as in unit 10 of Figure 1 and a computer application program to be run by a host computer such as computer 18. The Iscan eyetracker may be connected to an IBM-type personal computer running the DOS operating system. Such a computer could be used as the computer 18 of Figure 1. The Iscan system is designed to be connected to a video display unit. The application program is provided to control the cursor location on the video display in response to the eyetracker and to calibrate the eyetracker with the X-Y coordinate system of the display unit. Calibration includes the steps of displaying two or more points on the display unit and instructing the operator to look at the points. Until the present invention, this system has not successfully been adapted for use with a microvideo display.

Microvideo displays which are suitable for use with the present invention are well known in the art. An example of such a product which is commercially available is the PRIVATE EYE, manufactured by Reflection Technology, Inc. of Waltham, Mass. This product is described in U.S. Patent 4,902,083 and 4,934,773. A similar device is also described in U.S. Patent 5,003,300. Such devices are often used on videoca displays.

The microvideo display and eyetracker may be integrated in a unit 10 as shown in Figure 1, or may be separate but having a fixed relationship therebetween after calibration. A description of the computer 18 to which the eyetracker and microvideo display unit 10 are connected will now be provided.

The computer 18 may be any type of general purpose computer, but preferably a portable personal computer or a specialized hardware package. Such products are well-known and commercially-available, such as the IBM-PC AT. Many other types of computers could also be used. The commercially available eyetracker and microvideo display units, as described above, or similar customized devices, often have hardware units which may be connected to the computer 18 and include application programs to be run on the computer 18. Although the individual control elements (i.e., their application programs) for the microvideo display and eyetracker are well-known, they may not be compatible with each other when running simultaneously on the computer 18. Thus, in some cases, special application programs need to be provided to make the commercial application programs compatible. In some cases, the commercial programs may be rewritten. Such rewriting steps are well within the ordinary skill of a programmer, and merely involve standard debugging practices.

On computer 18, the application programs for the eyetracker and microvideo display operate in a manner similar to standard peripheral device drivers, such as a mouse, keyboard, printer or video display. The eyetracker, acting as in input device, is analogous to a mouse. Thus, it provides commands, known as interrupts, to computer _8. The computer 18 responds to these interrupts to change the cursor location on the video display. Sometimes, additional input besides cursor location and movement is used to indicate that an item displayed on the microvideo display is selected. Using a button, such as button 19 in Figure 1, or other type of input from a keyboard or mouse is a common practice for such selection and may also be used in combination with the eyetracker and videodisplay unit. Such a combination would be made in a manner substantially similar to the combination of a mouse unit with a button.

The selection of a displayed object more advantageously may be provided in other ways in accordance with the present invention. For instance, the attention span of an operator on a specific displayed item may be determined by an eyetracker. The computer may determine the duration of time during which the user is focused on a given location. After a predetermined amount of time, the computer could perform a function. For example, if the computer display included menu items, a user could select a menu item simply by focusing his attention on that item for the predetermined amount of time. In some cases, however, time delay should not be used to cause the computer to perform a function, especially when undesirable, or irreversible results may occur due to that function.

Audio input and output may also be used in connection with the eyetracker microvideo display. Audio output control systems, such as speech generators or synthesizers, are well known in the art. Such systems are commonly available on most computer systems. A speech generation unit could be used to generate voice prompts which may be used, for instance, to provide instructions to a user. For audio input, a voice or speech recognition unit may be used. For these purposes, the headband 12 of Figure 1 may also support an earphone 14 and microphone 15. More than one earphone may be used if desired. The microphone 15 (Fig. 1) is preferably mounted in the housing 20 (Figure 2) of the integrated unit 10. Many such audio input and output devices are available in the art and may be combined on a computer 18 in a manner well—known to those skilled in the art. This combination may also require some debugging of the application programs controlling the input and output devices as described above.

Using voice or speech recognition technology, a spoken word, or even a simple audio click, received by the computer 18 from microphone 15 is translated into intelligent computer commands. Most tasks require only a small vocabulary of input commands, regardless of the capacity of the voice recognition unit for input data. The voice or speech recognition unit also may be used to provide a system with a unique voice input stamp based on the operator's vocal patterns, to identify the user, thereby improving system security.

The selection of an appropriate voice recognition system depends on the application. A speaker dependent system is trained to recognize only a single operator's voice with a high percent of accuracy. A speaker independent system typically may recognize any person's voice, but with greater errors. Some systems also distinguish between discrete speech and continuous speech. Discrete speech isolates each word and requires a certain amount of pause between each spoken word in order for the computer to interpret it correctly. Continuous speech systems allow the speaker to speak more normally, without needing to pause between words. Other trade offs include speed, accuracy, capability and complexity, and vocabulary development.

A suitable, commercially-available product is Dragon Dictate, available from Dragon Systems of Newton, Massachusetts. This system is capable of adapting to the speech of individual speakers, and learns to understand each individual. It also maintains personal speech pattern files for each user. Furthermore, it is capable of distinguishing between similar sounding words by applying English usage rules.

Flowcharts illustrating the operation of the eyetracker and voice recognition unit in connection with the computer 18 will now be described in connection with Figures 3-5. Figure 3 is a flowchart describing an eyetracker driver for the computer 18. The driver begins with step 70 of retrieving the X-Y coordinates provided by the eyetracker hardware. In Step 72, the new coordinates obtained in Step 70 are compared to the current coordinates. If the coordinates have not changed, the driver returns to Step 70. Otherwise, the coodinates indicate a new eye position and therefore new coordinate position, and the cursor location is updated in Step 74. An interrupt signal is then sent in Step 76. The interrupt is handled in a manner described below in connection with Figure 5.

Figure 4 is a flowchart illustrating the operation of a driver of a voice recognition unit. This driver is similar to a typical driver for a keyboard and provides interrupts similar to well—known keyboard interrupts. The driver begins with Step 80 of examining the signal for a level above a predetermined threshold indicating the start of audio input. This signal is then decoded in Step 82 to determine whether the voice signal indicates a word. If the determined word is not found in the system dictionary in Step 84, the driver looks again for a new signal threshold in Step 80. Otherwise, the voice signal indicates a word found in the dictionary, and that word is written to a buffer, which is similar to a keyboard buffer in well-known systems (Step 86). An interrupt signal is then sent in Step 88 to be handled by the operating system of computer 18.

Handling of interrupts by the operating system of computer 18 will now be described in connection with Figure 5. Interrupt handling is a well-known process in computers. In general, an application program being run by a computer such as 18, and as indicated in Step 90, is interrupted by the operating system when the operating system receives interrupt signals from other processes and programs. The operating system then decides to process or to ignore those signals. The system steps through the known types of interrupts that it may receive to determine if the received interrupt should be handled. For example, the system may first determine whether a voice interrupt has been received in Step 94. If a voice interrupt has been received, the system retrieves the contents of the buffer in Step 96 and interprets those contents as a command in Step 98. After the command is executed, execution of the application program resumes with Step 90.

If the interrupt was determined not to be a voice interrupt in Step 94, it is next determined whether the interrupt signal indicates a new cursor position in Step 100. Upon receiving a cursor interrupt, the system obtains the new cursor position in Step 102 and updates the cursor pointers for the video display in Step 104. The execution of the application program then resumes in Step 90. An operating system typically may handle other interrupts which are not relevant to this discussion.

Beyond the advantage of providing complete portable computer interaction, the present invention also allows simultaneous multi-level computer input from a variety of sensory inputs, and also allows machine detection of user condition and actions. For example, the eyetracker allows the human eye to move a cursor on a screen in a manner similar to the way a mouse moves a cursor, except that the eye is faster. In practice, every mouse-guided cursor move is actually preceded by an eye fixation to locate the target, a forced gaze, and peripheral vision monitoring to observe the cursor moving across the screen and a final tracking maneuver to position the cursor at the target. These activities are eliminated by using an eyetracker instead of a mouse. When combined with voice input, a variety of commands may be provided for each selected cursor location, rather than a single selection command. Thus, the number of selections a user needs to make may be reduced.

The system of the present invention also allows machine detection of user condition and actions. It should be evident that, while the operator provide inputs to the system, the system observes the operator. For example, since eye motions are strongly connected to physical health, the eyetracker may determine if the operator is impaired due to illness, exhaustion, emotional distraction, drugs, etc. Also, such feedback could be used in training exercises. For example, if an operator was being trained in map reading, the system could determine if the operator is looking at the right features on the display, or even if the operator is looking in the right vicinity on the display. As another example, the voice patterns, detected by the voice recognition unit, and the eye image, obtained by the eyetracker, may also be used to verify the identity of the operator.

It should also be evident that the present invention may also be used in situations where standard computer interfaces may not be used. For example, when an operator is physically handicapped, but still has use of his voice and eyes, the present invention may be used to provide an interface for them to a computer.

The computation hardware, due to its small size for the eyetracker and voice recognition units could be provided in a small portable package in connection with the head mounted unit which communicates with a remote computer. Such a system is shown schematically in Figure 6.

The portable unit 40 may include the computation hardware elements which are typically used for the speech generation unit 42 (for use in providing voice prompts to the user), the voice recognition unit 44, the microvideo display unit 46 and the eyetracker unit 48. The operation of these units and communication with the computer 18 is controlled by the microprocessor 50 and radio transceiver 52. The microprocessor is merely used to route data between the computer 18, a memory 54 and the hardware units 42-48. The portable unit 40 may be powered by a battery 56. The computer 18 also has a corresponding radio transreceiver (not shown) for communication with this portable unit.

Other input devices could also be combined with the eyetracker microvideo display unit. Such devices include oscilloscope probes, pressure sensors, temperature sensors, bar code readers and other types of specialized sensory probes. These probes could be used to provide further input data to the computer. This optional input device may also be attached to the portable unit 50 as shown at 58 in Figures 1 and 6 to provide data to the computer 18.

The present invention could also be made using two eyetracking devices which would allow for three-dimensional graphic displays and depth perception in eyetracking.

Having now provided a detailed description of the present invention, it should be apparent to those skilled in the art that the foregoing is illustrative only and not limiting, having been presented by way of example only. Numerous other embodiments and modifications thereof are contemplated as falling within the scope of the present invention as defined by the appended claims and equivalents thereto.

What is claimed is:

Claims

1. A system for interfacing between a user and a computer, comprising: a microvideo display monitor; means for mounting said monitor in a position which is substantially fixed relative to a given eye of the user; and means for obtaining an image of said given eye, said means being in a fixed spatial relationship with said monitor.

2. A system as in claim 1, said microvideo display monitor connected to generate selected displays under control of said computer; means responsive to said means for obtaining an image for providing an input to said computer indicative of the location on the monitor where said given eye is focused; said means for obtaining and said means for providing defining an eyetracker; and said computer performing predetermined functions in response to inputs received from said eyetracker.

3. A system as in claim 2 further comprising: means for receiving audio inputs from the user; and means responsive to said means for receiving for generating outputs to said computer which are indicative of the audio input from the user.

4. A system as in claim 3 wherein said means for generating includes means for recognizing speech, said computer responding differently to different recognized speech from the speech recognizing means.

5. A system as in claim 4 wherein said computer responds to an eyetracker input in response to an audio click from said speech recognizing means, the functions performed by the computer being determined both by the display locations on which said given eye is focused and the recognized speech from the user.

6. A system as in claim 2 including means for determining a duration of time during which said given eye is focused on a location and for generating an output if said duration exceeds a predetermined threshold, and means responsive to said threshold output for applying an eyetracker output indicative of said location to the computer.

7. A system as in claim 3 including means responsive to an audio click from said means for generating when the given eye is focused on a location for applying an eyetracker output indicative of said location to the computer.

8. A system as in claim 2 including a speech synthesizer responsive to said computer for generating voice prompts to the use .

9. A system as in claim 2 including sensor means positionable by the user for generating additional inputs to said computer.

10. A system as in claim 3 including means for identifying the user responsive to said means for receiving audio inputs.

11. A system as in claim 2 including means for identifying the user, responsive to said eyetracker input.

12. A system as in claim 2 including means responsive to said eyetracker input for determining if the user is impaired.

13. A system as in claim 2 wherein said computer is at a location remote from the user, and wherein said system includes transceiver means for bilateral communication with said computer .

14. A system as in claim 1, further comprising a housing for supporting the microvideo display monitor and the means for obtaining the image of said given eye, said housing maintaining said fixed spatial relationship between the monitor and the means for obtaining an image of said given eye.

15. The system of claim 14 further comprising a dichroic mirror supported within said housing for enabling an operator to look at the microvideo display without interference from the means for obtaining an image of said given eye.

16. The system of claim l wherein the means for obtaining an image of said given eye comprises: an infared illuminator for illuminating the eye, light sensitive means for providing a signal indicative of the eye; and a lens for focusing reflections from the infared illuminator from the eye onto the light sensitive means.

17. A system as in claim 2, including means for providing an input from the user, said computer performing further predetermined functions in response to said means for providing an input.

18. A system for interfacing between user and a computer, comprising: a microvideo display monitor; means for mounting said monitor in a position which is substantially fixed relative to a given eye of the user; means for obtaining an image of said given eye; means for calibrating the means for obtaining to said microvideo monitor, said means for obtaining an image being maintained in a fixed spatial relationship with said monitor after calibration by said means for calibrating.