US20080150899A1 - Virtual workstation - Google Patents

Virtual workstation Download PDF

Info

Publication number
US20080150899A1
US20080150899A1 US11/944,226 US94422607A US2008150899A1 US 20080150899 A1 US20080150899 A1 US 20080150899A1 US 94422607 A US94422607 A US 94422607A US 2008150899 A1 US2008150899 A1 US 2008150899A1
Authority
US
United States
Prior art keywords
virtual
user
keyboard
present
display
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US11/944,226
Inventor
Julius Lin
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Intelligentek Corp
Original Assignee
Intelligentek Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Intelligentek Corp filed Critical Intelligentek Corp
Priority to US11/944,226 priority Critical patent/US20080150899A1/en
Assigned to INTELLIGENTEK CORP. reassignment INTELLIGENTEK CORP. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: LIN, JULIUS
Publication of US20080150899A1 publication Critical patent/US20080150899A1/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/014Hand-worn input/output arrangements, e.g. data gloves
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011Arrangements for interaction with the human body, e.g. for user immersion in virtual reality

Definitions

  • the present invention is generally related to methods and devices for producing a virtual computing environment. More specifically, the present invention is related to methods and devices that in combination mimic the functionality and behavior of a physical computer system in a virtual environment.
  • the methods and devices in accordance with the present invention includes a portable computing system wherein the system is configured to provide the user with a virtual hardware environment that may be utilized for computing, gaming or other uses.
  • keyboard which has a plurality of depressible keys each corresponding to a particular alphanumeric character, symbol, or computer function. While computer keyboards are widely accepted and quite suitable in many situations, keyboards are not always the most efficient, convenient, or easy to use devices.
  • keyboards include up to 110 individually marked keys mounted to a base with as many switches. All of these components must be produced and assembled, which accounts for considerable expense. Since keyboards are mechanical, are also more prone to failure than solid-state devices, additionally, due to the likelihood of failure, broken keyboards additionally present disposal problems. Further, the conventional keyboard cannot be quickly changed to a new keyboard layout, such as might be desired by those who have learned a keyboard layout other than the somewhat inefficient but traditional QWERTY layout.
  • keyboards are built generally in one size for all users. As a result, users with relatively small or large hands must adapt to a keyboard size that is not optimal for their particular hand size. A person with smaller hands must stretch further to strike a key some distance from a home row of keys, whereas a person with larger hands will have a harder time accurately striking any desired key. Keyboard size that is optimized for a particular use may lead to decreased hand fatigue. However, keyboard manufacturers have determined an ergonomically acceptable compromise, which is a compromise nevertheless. Since keyboards are produced having only one size forces a user to type with his hands close together in an unnatural manner.
  • split keyboards which are split into a separate keyboard for each hand, are more comfortable for the user and produce a slightly faster typing speed as a result. Additionally, as computers become more common in the workplace, a greater number of injuries have been reported due to utilizing a keyboard.
  • keyboards are produced by www.vkb.co.il, www.canasta.com, and www.virtualdevices.net. These types of keyboards utilize an infrared projection system, wherein a keyboard is projected onto a surface and a sensor detects the position of a finger on top of the projected keyboard image.
  • a problem with they types of keyboards is that you can only utilize the system on a smooth clean non- transparent steady surface, additionally, if you rest your hands within the projected keyboard the sensor may interpret this motion as keystrokes, thereby resulting in errors. Further still, since the keyboard is projected onto the surface, the user may experience light interference from surrounding light sources.
  • each of these systems are a novel approach to overcoming the dependence on a physical keyboard, there are still shortcomings. Namely, each of the systems require the user to either be physically tethered to a computer, where pressure sensitive devices that must be depressed on a surface or find a smooth surface to set up a virtual keyboard. Additionally, these devices do not allow a user to customize the layout of the keyboard or adapt the keyboard to a users specific style of typing. In addition to being tethered to a computer, the use of physical input devices may cause injury to the user. For example, many claims are filed every year for repetitive stress injuries incurred from keyboard usage. Examples of common injuries are carpal tunnel syndrome, eye fatigue, neck and back strain, many of which are attributed to usage of a personal computer.
  • voice recognition systems Another shortcoming of voice recognition systems is that it is difficult to use voice recognition for editing, browsing the Internet, graphic design and similar input intensive programs. Additionally, constant talking may fatigue the user's voice, wherein the user's pitch and tone may change, thereby leading to additional input errors because the voice recognition software no longer recognizes the user's voice. Further still, voice recognition systems cannot be utilized in cubicle type work environments or similar “open” type environment where noise interference from other voices may confuse the voice recognition software.
  • Additional input devices may also be utilized in conjunction with keyboards.
  • pointing devices such as “mouse” pointing devices and so called “track ball” devices are also popular computer interfaces.
  • these types of devices provide velocity information, in both an X direction and an orthogonal Y direction, to the computer, as well as signals from one or more momentary contact push buttons.
  • a pointing icon or other “tool” on a computer monitor responds to such velocity input by corresponding X and Y movement on the computer monitor.
  • Graphics tablets are another type of “pointing” input device that provide the computer with X and Y positional information, as opposed to velocity information, which is used in much the same manner by the computer.
  • Such devices are well suited for pointing to various software “push button” options on the screen, selecting portions of text or a group of software “objects,” freehand on-screen drawing, positioning a typing cursor location, and similar functions.
  • pointing devices are remarkably ill suited for text data input.
  • the present invention provides systems and methods for interaction within a virtual environment. Wherein the methods of the present invention may be utilized to virtually interact with a computer system through the use of virtual input devices, wherein the methods and systems allow a user to input data into a computing system without physical limitations.
  • a method for controlling a microprocessor based system in a virtual environment comprising: loading a computer program into a memory space; transmitting at least one signal from at least one transmitter; displaying virtual input devices on a display device, wherein the virtual input devices initially have pre-determined coordinates; receiving data generated from movement of at least one sensor; calculating coordinates of the sensor movement; comparing calculated coordinates to the pre-determined coordinates of each virtual input device; calculating desired input from coordinates generated by the sensor movement; and displaying input on the display device and transmitting the input to the operating system.
  • a method for controlling a microprocessor controlled device in a virtual environment comprising: loading a computer program into a memory space; loading an operating system into a memory space; transmitting at least one signal from a transmitting device displaying a virtual keyboard and a virtual input device on a display device; initializing coordinates defining individual keys of the keyboard; initializing coordinates defining a location of the input device in relation to the keyboard; receiving data at least one sensor wherein the data received is converted into coordinate information and stored in a memory space; determining if coordinated derived from movement correlates to a key location of the virtual keyboard or movement of the virtual input device; and displaying sensor movement on the display device.
  • a method of generating and controlling a virtual workstation comprising: initializing an operating system and a controller system in a microprocessor based computer system; displaying virtual hands, a virtual keyboard, a virtual pointing device and a virtual workstation environment on a display device; monitoring movement of sensors disposed on a user's hands for movement; displaying movement of the virtual hands in response to movement of the sensors; and determining if movement of at least one sensor passes a user defined threshold.
  • a method of generating a virtual gaming system comprising: initializing an operating system and a controller system in a microprocessor based computer system; loading a game program into a memory space; displaying a virtual player on a display device; monitoring movement of sensors disposed on a user for movement; and displaying movement of the virtual player in response to movement of the sensors.
  • a system for virtually controlling a microprocessor based system comprising: a microprocessor based computer system having an operating system configured to be run thereon; a display device, at least one tracking device; at least one sensor, wherein said tracking device is configured to track movement of said sensor and determine coordinates of said sensor and time components of sensor movement within a pre-defined area; and a software component, wherein said software is stored in a computer readable medium, wherein said software is in communication with said tracker and said display device, wherein said software determines vector movement and acceleration of said sensor and displays said sensor movement on said display device.
  • a system for implementing a virtual reality environment comprising: a display device associated with a user, the display device being responsive to image data for generating and displaying an image simulating a physical computer system, including a physical keyboard, a physical input device, and physical representation of the user's hands, wherein each of the simulated components appear to the user to be in space independent of actual physical objects; an output device for transmitting a signal; an input device for generating data in response to interaction with the signal; a processor connected to the input and output device and the display device and operating a virtual environment manager program and a multi-dimensional basic input and output program for generating a virtual keyboard, a virtual input device, and virtual hands, the processor being responsive to data generated from the input device, for generating motion image data corresponding to the input device data; and wherein the display device is responsive to motion image data for generating a second image simulating physical motion of at least one virtual component.
  • a system for implementing a virtual reality (VR) computing environment comprising: VR display device including at least one display and worn by a user the one display viewable by the user, with the VR display, responsive to first image data, for generating and displaying a first VR image simulating a physical computer system including a virtual keyboard having a plurality of physical keys, a virtual mouse having at least one physical button, with the first VR image representing the VR keyboard and VR mouse, the VR keyboard and VR mouse having a first appearance corresponding to the first image data; an input and an output device for generating motion-representative data corresponding to motion of a user's body part; and a processor connected to the VR display device and operating a virtual environment manager (VEM) and multi-dimensional basic input/output subsystem(MD-BIOS) program, wherein the VEM and MD-BIOS provide the first image data to the VR display device, the processor being responsive to the motion-representative data generated from the input device, for generating motion image data
  • VEM virtual environment manager
  • MD-BIOS
  • FIG. 1 is a block diagram illustrating exemplary mechanical devices that may be utilized with the methods in accordance with the present invention.
  • FIG. 2 is an exemplary embodiment of a computer system in accordance with the methods of the present invention.
  • FIG. 3 is an exemplary block diagram of the tracker system in accordance with the present invention.
  • FIG. 4 is an exemplary embodiment of the system in accordance with the present invention.
  • FIG. 5 is an exemplary embodiment illustrating an alternative embodiment of a display device in accordance with the present invention.
  • FIG. 6 an exemplary embodiment of an alternative display device in accordance with the present invention.
  • FIG. 7 an exemplary embodiment of a user's hand illustrating the wireless communication of the tracker system.
  • FIG. 8 is a functional flow diagram illustrating the method in accordance with the present invention.
  • FIG. 9 is a functional flow diagram illustrating the interaction between the software and hardware components of the present invention in accordance with the methods of the present invention.
  • FIG. 10 is a block diagram illustrating an exemplary embodiment of the memory structure of the multi-dimensional basic input/output system in accordance with the present invention.
  • FIG. 11 is a block diagram illustrating the threshold values and calculation methods of the virtual workstation manager in accordance with the present invention.
  • FIG. 12 is a functional flow diagram illustrating the method steps in accordance with the present invention.
  • FIG. 13 is an exemplary embodiment of a virtual workstation environment as seen from a user's perspective, wherein a virtual keyboard is shown disposed three dimensionally over a virtual mouse.
  • FIG. 14 is an exemplary embodiment of a virtual workstation environment as seen from a user's perspective, wherein the virtual mouse is shown disposed three dimensionally over the virtual keyboard.
  • FIG. 15 is a functional flow chart illustrating hardware components of a virtual car control center.
  • FIG. 16 is an exemplary embodiment of the present invention wherein the virtual environment manager has been configured as a virtual car control system.
  • FIG. 17 is an exemplary embodiment of the high-level virtual buttons in accordance with the virtual car control system.
  • FIG. 18 is an exemplary embodiment illustrating the high-level virtual buttons and the second level of virtual buttons in accordance with the virtual car control system.
  • FIG. 19 is an exemplary embodiment illustrating the high-level virtual buttons, the second level, and third level of virtual buttons in accordance with the virtual car control system.
  • FIG. 20 illustrates exemplary embodiments of additional second level virtual buttons after selection of the high-level virtual button.
  • keyboard is defined herein to include alphanumeric keyboards, subsets of alphanumeric keyboards, keypads including numerical keypads, telephone and DTMF keypads, security access input devices using buttons and labels, etc., and so it not limited to QWERTY alphanumeric keyboards. Accordingly, it is understood that the use of the term “keyboard” and the depiction in any of the figures of a keyboard such as a QWERTY alphanumeric keyboard typically used with personal computers and the like is only an example of a keyboard for use, interaction, and operation by a user for any application of keyboards for input and/or output devices.
  • keyboard is more than a plurality of keys, since a keyboard includes a layout of the plurality of keys as well as keys, with the layout typically being predetermined.
  • the keys may be associated with symbols such as alphabetical, numerical, mathematical, or other representations, and the keys may include associated pictorial or symbolic representations thereupon. Accordingly, a keyboard is not identical to a set of buttons but may be a plurality of buttons having a layout and a set of symbols associated with each key or button.
  • VR virtual reality
  • a graphical representation of a switch may be changed to represent the flicking or switching of the switch, which may have an associated switch-flicking sound, which is activated by flicking the switch.
  • VR switching of the VR switch may cause the actuation of other events, either in the VR world or in actual physical devices and structures; for example, the flicking of the VR switch may cause an actual computer to be turned on or off.
  • virtual reality is not limited to simulations or representations of VR devices and information in VR worlds, but may also be extended to physical devices as well as, in hybrid implementations, to both physical and VR devices.
  • the human interface device includes hardware components consisting of a tracker assembly, a processing unit, and a display device.
  • the software/firmware components comprise a virtual environment manager (VEM) and a Multi-Dimensional Basic Input/Output subsystem (MD-BIOS), wherein the software and hardware components are utilized in combination to provide a virtual environment.
  • VWE virtual workstation environment
  • a physical computing system is simulated in a virtual environment including virtual hands and virtual input devices such as a keyboard, pointing device or other similar input devices.
  • VCS virtual gaming system
  • the present invention could be utilized to simulate a gaming environment wherein the user could virtually interact within the game.
  • the above examples are to be considered to be merely exemplary in that the present invention may be utilized in many other applications such as military use, flight simulation/training, corporate meetings, etc.
  • a computer system may be embodied as an Intel.RTM. central processor unit (CPU) based system a RISC based processor system, though it shall be understood that this should not be considered limiting in any manner in that a computer system for use with the present invention may be based on similar microprocessor devices.
  • CPU central processor unit
  • RISC RISC
  • the hardware components comprise a microprocessor based computer system, a display device, and a tracking system.
  • the present invention further includes software running stored in a computer readable medium, wherein the software is in association with the hardware components of the system.
  • the software of component of the present invention will be described in greater detail below in reference to the methods of the present invention.
  • the microprocessor based computer system 10 includes a central processing unit 20 , memory 30 , a communication bus 40 , a computer readable storage device 50 such as an optical storage device, magnetic storage device, flash memory storage device or similar computer readable storage mediums, and at least one communication port 60 .
  • the communication port 60 comprise any one of the following types of communication ports as well as a combination thereof: universal serial bus (USB), IEEE 1394 (firewire), serial port, parallel port, infrared port, 802.11b, 802.11a, 802.11g, Bluetooth.RTM. or similar communication ports and devices.
  • the tracker system 80 comprises a system electronic unit (SEU) 82 , at least one transmitter and at least one sensor each in communication with the SEU 82 .
  • SEU system electronic unit
  • the SEU 82 comprises communication ports that are in communication with at least one sensor 86 and at least one transmitter 84 .
  • the communication ports may comprise serial, parallel, universal serial bus, firewire.RTM. or other similar wired communication ports.
  • the SEU 82 additionally includes an analog section and a digital signal processing section controlled by a computer program stored within an erasable programmable memory device. The functionality of the SEU 82 and the interaction between the transmitter and the sensors will be described in greater detail in the methods section of the present invention.
  • the SEU 82 is in communication with the transmitter 84 and at least one sensor 86 .
  • the sensor 86 is configured to be coupled to the SEU 82 through the use of a cable communication device such as a serial port, parallel port, universal serial bus, firewire port, or similar wired communication ports.
  • the transmitter may be coupled to the SEU through similar communication devices such as those described above with regard to the sensors.
  • the transmitter 84 is configured to emit an electromagnetic signal, wherein the SEU 82 controls the transmission rate of the transmitter 84 .
  • Transmitters 84 that may be utilized with the current invention are those shown and described in U.S. Pat. No. 4,742,356, the entirety of which is herein incorporated by reference.
  • the transmitter 84 includes a plurality of radiating antennas for radiating electromagnetic energy. Each of the radiating antennas having independent components for defining a source reference coordinate frame.
  • the transmitted electromagnetic field will generally have a transmission range of about 0 to 30 feet, more preferably 0 to 15 feet and most preferred about 0 to 3 feet. The range of the transmitted magnetic field may be adjusted manually or automatically in accordance with the methods disclosed herein.
  • the transmitter 84 may be configured to include more than one transmitting device, wherein the two transmitting devices would be of different types.
  • the transmitter 84 may include a first magnetic transmitting device and a second transmitting device or third transmitting device.
  • the second or third transmitting devices may be configured to transmit acoustical, optical, or electromagnetic signals.
  • the system in accordance with the present invention may be configured to automatically choose between the two transmitters or the system may be manually configured. For example, if the user were to utilize the system in an environment having a large amount of stray magnetic fields that may interfere with the magnetic tracker, the system in accordance with the present invention may automatically switch to one of the other transmitting devices.
  • the transmitter 84 is configured to transmit at least one signal.
  • the transmitted signal may be electromagnetic, optical, acoustical, inertial etc. in a generally defined field of view.
  • the transmitting device may be configured to transmit a signal in a field of view having a spherical radius of between about 0 to 360 degrees, more preferably between about 0 and 270 degrees and most preferred between about 0 and 180 degrees. It shall be understood that although the present invention has been described as including only a single transmitting device shall not be considered limiting in any manner and that it is contemplated that additional transmitting devices may be utilized with the present invention to further expand the field of view of the transmitting device and or to increase accuracy and functionality.
  • FIG. 4 there is shown an exemplary embodiment of the hardware components of the present invention in use.
  • the computer system 10 display device 75 , and the tracker system 80 , wherein the tracker system 80 comprises the transmitter 84 , SEU 82 and sensors 86 .
  • the computer 10 is configured to be embodied in the form of a wearable computer system. Examples of preferred computer systems would be based on the Pocket PC.RTM. platform developed by the Microsoft.RTM. corporation. Although a Pocket PC platform is described as a preferred embodiment this shall not be considered limiting in any manner.
  • a user may be tethered to a conventional desktop computer system either wired or wirelessly or utilize other computer systems such as an Intel.RTM. or AMD.RTM. powered computing system, PalmPilot.RTM. or other similar computing devices.
  • the display device 75 may be configured to include the transmitter 84 of the tracker system 80 .
  • the transmitter 84 is configured to be retained on the display device 75 .
  • the display device 75 in a preferred embodiment is a liquid crystal display (LCD) device, wherein the LCD device is in communication with the computer system 10 through the use of a wired or wireless connection (not shown).
  • the display device 75 may comprise other types and styles of head mounted display devices, such as organic displays, thin film transistor (TFT) displays, light emitting diode (LED) displays.
  • TFT thin film transistor
  • LED light emitting diode
  • the display may incorporate a multi-layer device, wherein one layer is generally opaque and a second layer is a generally opaque dot-matrix layer, wherein the opaqueness of the first layer may be manually or automatically adjusted thereby allowing the heads-up display to be utilized in many different ambient light situations. While the present invention has been described as utilizing a single display device, it is contemplated that a second display device may be utilized for stereoscopic vision.
  • the second display device may be a conventional display device such as a cathode ray tube (CRT) device, a liquid crystal display (LCD) device, or a video projection device, wherein the transmitter 84 would be mounted onto the display device.
  • CTR cathode ray tube
  • LCD liquid crystal display
  • video projection device wherein the transmitter 84 would be mounted onto the display device.
  • a multi-layer liquid crystal display may be utilized in accordance with the present invention, wherein the multiple layers are capable of simulating three-dimensions.
  • a three dimensional workspace may be simulated.
  • Such a system could be utilized to simulate a “real” world experience in the virtual environment and would further provide haptic feedback.
  • the three-dimensional display device would be configured to interact with the software components of the present invention.
  • the present invention would utilize a display device having two LCD screens, wherein a left view model is generated and a right view model is generated by a graphic processing unit (GPU).
  • FIG. 5 An example of such a system can be seen in FIG. 5 , where there is shown an exemplary embodiment of the three dimensional system in accordance with the present invention.
  • a user's right and left eyes would focus on the right and left view model generated in the display device, wherein the user's brain can comprehend the information displayed on the display device and decide the distance and depth of the displayed object. Thus, the user would believe that they are seeing a true three dimensional display of the object displayed on the display device.
  • the display device may be a conventional display device such as a cathode ray tube (CRT) monitor or a liquid crystal display (LCD) monitor, wherein the tracker would be in association with the monitor frame.
  • the transmitter maybe positioned in the upper corner of the monitor frame, wherein the physical location of the transmitter will be utilized to establish an origin for the coordinate system of the present invention as will be described in greater detail. Further still, it is contemplated that the transmitter may be placed in other locations on the frame of the display device or alternatively in a location near the display device. In addition to conventional display devices as described above, other types of display devices may be utilized.
  • a video projector may be utilized to project an image on a surface, wherein the transmitter will be placed near one corner of the projected image.
  • the projected image may be directed onto a screen, wall or similar vertical surface, or alternatively, the projector may be mounted such that the image is projected onto a horizontal surface such as a conference table.
  • the display device may be a combination of devices, for example, a video camera and a CRT or LCD or projector, wherein the present system and methods may be utilized for video conferencing.
  • the display device may be embodied in the form of a physical display device such as a white board, chalkboard or a similar device, wherein the transmitter would be mounted to one corner of the board.
  • any of the display devices described above may be utilized in any combination.
  • a user may utilize a head mounted display device and a projector simultaneously in accordance with the methods described in the present application.
  • the present invention further includes at least one sensor device 86 disposed on the user.
  • the sensor device 86 is preferably disposed on the user's hand, and most preferable disposed on the user's fingertip. It is contemplated that at least one sensor 86 may be disposed on each of the user's fingertips. As shown and described in FIGS. 3 and 4 , the sensors are in communication with the SEU 80 .
  • the sensor(s) 86 are comprised of multiple coils encased in a protective housing, wherein the coils are configured to interact with the electromagnetic field generated by the transmitter 84 .
  • Each of the sensors are configured to generate at least one electrical signal, and more preferably between about three and six electrical signals in response to interaction with the signal generated by the tracking device.
  • the electrical signal(s) generated by the sensor 86 are passed through the cable connection to the SEU 82 , wherein the signals are amplified and converted into a digital signal.
  • the MD-BIOS further assigns a unique sensor id to each sensor.
  • the digital signals are embodied in the form of coordinate information of the sensor, such as, x, y, z, and yaw, pitch and roll information. This coordinate information is then passed from the SEU 82 to the computing system 10 .
  • the sensors 86 may be connected to a transducer 81 , wherein the transducer is configured to convert the analog signals to digital signals and wirelessly communicates the digital data to the SEU 82 .
  • the transducer 81 is configured to be hard wired to at least five sensors. In this configuration the transducer would assign a unique sensor identification (ID) to each sensor, wherein this unique sensor ID would be transmitted to the SEU 82 along with the coordinate data of each sensor.
  • ID unique sensor identification
  • transducer has been described as being wirelessly coupled to the SEU 82 it is contemplated that it may communicate with the SEU 82 through a wired communication port. It shall be understood that more than one transducer 81 may be utilized with the system in accordance with the present invention. For example, sensors may be disposed on each fingertip of a user's hands, wherein at least two transducers 81 will be utilized, one transducer 81 being disposed on each of the user's hands and in association with the sensors disposed on each of the user's fingertips respectively.
  • the SEU 82 has bee described as being an independent component separate from the computing system 10 it is contemplated that the SEU maybe integrally formed with the computing system.
  • the SEU may be configured to be a removable communications card in the form of a PCMCIA card, compact flash card, PCI card or other similar removable devices.
  • SEU may be integrated into the system board of the computing system.
  • the software components in accordance with the present invention comprise a virtual environment manager (VEM) 100 and a multi-dimensional basic input/output system (MD-BIOS) 110 , wherein the software components are configured to interact with the hardware components described above.
  • VEM virtual environment manager
  • MD-BIOS multi-dimensional basic input/output system
  • the software components of the present invention will be embodied in a computer readable media such as a cd-rom, dvd, hard drive, flash memory, programmable read only memory or any other type of computer readable media.
  • the VEM 100 receives coordinate data and a time stamp from MD-BIOS 110 , wherein the VEM 100 utilizes the coordinate data to simulate virtual device actions and display the virtual device actions on the display device. Additionally, MD-BIOS 110 and VEM 100 are in communication with an operating system. It shall be understood that the VEM 100 can be tailored for each application purpose. For example, if the system in accordance with the present invention is configured to virtually simulate a computer workstation, then the VEM 100 would be configured as such to generate a virtual workstation. Additionally, if a gaming system is to be replicated then the VEM 100 would be a virtual gaming manager, is a conferencing system is to be replicated than the VEM would be configured to be a Virtual Conferencing Manager. It shall be understood that the above examples should not be considered limiting in any manner, in that they have been provided for exemplary purposes only.
  • MD-BIOS 110 receives coordinate data from the SEU, wherein MD-BIOS adds time stamp information to the coordinate information.
  • FIG. 10 there is shown an exemplary embodiment of the memory structure of MD-BIOS, wherein data received from each sensor is interpreted and placed into appropriate memory location in the memory structure as shown.
  • MD-BIOS is capable of receiving data from a multiple number of sensors, wherein each sensor is assigned a sensor identification tag (id) by the MD-BIOS or transducer depending upon the system's configuration.
  • MD-BIOS not only can read the sensor id's and their properties, but can also check the sequence number to know the difference between each input to the application. Applications can also interact with MD-BIOS in one of two ways or in a combination of each. For example, it is contemplated that multiple programs running in a memory space of the personal computer may utilize different sets of sensors. For example, in association with the virtual workstation embodiment, a number of sensors associated with the user's fingertips may be utilized to control the virtual keyboard and mouse in a word processing program, while another program may be collecting data from other sensors associated with the present invention.
  • the present invention may be utilized in a laboratory setting, wherein in addition to manually entering data by utilizing the virtual keyboard, a scientist may wish to automatically collect data from an experiment or room conditions.
  • the scientist would dispose additional sensors to measure to desired properties. For example, one sensor would be utilized to measure room temperature, another for humidity, another to measure ph or another chemical property of an experiment, etc.
  • the first method of program interaction is referred to as synchronized interaction.
  • the applications need to register to MD-BIOS to listen to the set of sensor id's which each application has interest and leave a call-back function address with MD-BIOS. Therefore, whenever data is updated, MD-BIOS interrupts the application to acquire the data, then the application resumes processing the data. This provides synchronous operation of data processing and data acquisition.
  • MD-BIOS will filter out the application that requires a higher sampling frequency.
  • MD-BIOS will carry out its best resolution, if the requested frequency is outside the resolution of the tracker, then MD-BIOS will either return an error or will return a reduced frequency rate. In this case, some of the lower frequency applications may need to know the data acquisition gap and therefore utilize the sequence number of the sensor id' to determine the frequency.
  • MD-BIOS may additionally be utilized to create a true three dimensional virtual environment through the use of at least two display devices, wherein MD-BIOS generates a left and right view which are then displayed on a customized display device as previously described in the hardware section above.
  • MD-BIOS may be further configured to control the transmitter in combination with the system electronic unit, wherein the two systems could be utilized to adjust the transmission frequency of the transmitter or switch between the various transmitter types.
  • the transmitter is configured to emit an electromagnetic signal that the sensors interact with and produce an electrical signal that is converted into the coordinated received by MD-BIOS.
  • MD-BIOS compares the coordinate information against the previously received coordinate information for each sensor, if the deviation of each sensor is greater than a predetermined amount, for example, sensor movement of a quarter of an inch in one computing cycle at 120 Hz would be considered to be excessive.
  • MD-BIOS would automatically direct the transmitter to energize the alternative transmitting devices, wherein MD-BIOS would then sample the coordinate generated by all transmitting devices, if one set of coordinate information is outside the parameters as described above then that coordinate information is discarded and the other two sets of coordinate information are compared. If the two remaining coordinate information sets are similar, MD-BIOS chooses one of the transmitting systems and turns the remaining systems off. Therefore, MD-BIOS will automatically switch between the individual transmitting devices if interference is detected. Typically, MD-BIOS will automatically switch without the user's knowledge, though it is contemplated that the user may specify a desired system and adjust the default values to their preferences.
  • the Virtual Environment Manager is utilized to generate a virtual workstation environment.
  • the virtual workstation includes a virtual keyboard at least one virtual input device.
  • MD-BIOS receives coordinate data from the sensors adds a time stamp to the coordinate information and stores the coordinate data in a memory structure as shown in FIG. 10 .
  • the virtual environment manager includes a plurality of action thresholds to determine if sensor movement is to be interpreted to be an intended user input or is sensor movement can be attributed to inherent user movement such as natural heartbeat or muscle twitching. Initially, these threshold values are set at default levels, though these default levels may be adjusted by the user to suit the user's own preferences.
  • VEM utilizes for determining if sensor movement correlates to intended user input.
  • the coordinate data is utilized to determine whether the sensor movement is to be returned as a key press or other intended input or if the sensor movement is due to natural movement.
  • VEM generates a virtual keyboard having a known origin, wherein the coordinates generated for each of the sensors are compared to the origin coordinates to determine where the user's hands are in relation to the keys of the virtual keyboard.
  • a virtual input device such as a mouse is also generated wherein a set of origin coordinates are also established for the virtual input device, thereby allowing VEM to determine sensor location in relation to the virtual keyboard and virtual input device.
  • the key press threshold is a vector property wherein a value of less than negative one millimeter of a sensor's coordinates will be interpreted as a key press, wherein VEM will compare the sensor's coordinates to the origin coordinates of the virtual keyboard or mouse to determine the proper key press.
  • the key press will then be transmitted to the appropriate program, such as a word processing program, text-editing program, graphics program, etc.
  • a key release will be interpreted as a vector having a value of greater than 1 millimeter and having duration greater than one second.
  • a key hold will be determined by MD-BIOS as a vector having a value equal to one millimeter.
  • Movement of the virtual input device is determined by vector movement of the sensors, if the coordinate values of the sensors are moved less than one millimeter then VEM will not consider this to be a mouse movement, if the coordinate movement is greater than one millimeter this will be interpreted as a mouse move.
  • VEM utilizes the threshold values which have been pre-established for key presses on the virtual keyboard.
  • a virtual environment manager will be embodied in the form of a virtual workstation manager. That is the present invention will be utilized to replicate a personal computing system and the physical input devices in a virtual environment. As will be described in detail below, the virtual workstation manager will generate virtual input devices such as a keyboard, a mouse or other input devices.
  • the method according to the present invention comprises the steps of: initializing a computer system, including initializing the SEU, loading an operating system into a memory space, loading VEM and MD-BIOS into a memory space.
  • initializing a computer system including initializing the SEU, loading an operating system into a memory space, loading VEM and MD-BIOS into a memory space.
  • MD-BIOS begins to receive coordinate information from the SEU; VEM then begins to scan for sensor movement to determine if any sensor movement passes the threshold values.
  • a display device is initialized, wherein VEM (such as VWM) then displays virtual hands, virtual keyboard and at least one virtual input device on the display device.
  • VEM such as VWM
  • the methods according to the present invention may utilize hardware devices such as those described above or hardware devices similar to those shown and described above, wherein the present invention utilizes software or firmware to control the hardware in a manner to replicate a virtual environment.
  • the present invention comprises at least one software program or firmware code previously described herein wherein the software includes the VEM and MD-BIOS programs
  • a computing system is powered on, this includes powering on the SEU additionally if the SEU is embodied as an individual component separate from the computer system.
  • an operating system is loaded into a memory space.
  • MD-BIOS and the Virtual Environment Manager are loaded into a memory space within the computer system.
  • MD-BIOS After loading MD-BIOS and the virtual workstation manager into a memory space, MD-BIOS immediately begins receive coordinate information from the SEU at Box 215 .
  • the electrical signals generated by the sensors are converted to coordinate information and a time stamp is added by MD-BIOS and the data is stored in a memory location.
  • VEM compares the coordinates of each sensor to determine if motion of the sensor(s) has occurred and whether the sensor motion is intended to be a key press on the virtual keyboard or virtual mouse.
  • the transmission and scanning rate of the tracker is controlled by MD-BIOS. Wherein the scanning and transmitting frequency of the tracker is controlled in response to the physical location of the sensor(s) in relation to the tracker and the origin coordinates of the virtual devices. For example, as the sensors are moved closer to the virtual devices the scanning rate of the tracker will be increased, thereby increasing the accuracy and resolution of the system. When the sensors move away from the virtual devices, the scanning rate is reduced, thereby lowering the power consumption of the devices.
  • VEM In addition to controlling the scanning rate, VEM also may be configured to display a visual indicator of the sensor(s) position on the display device as well as provide auditory feedback to the user. For example, as the sensor(s) are moved to a location above the virtual keyboard, the sensor may change colors to indicate how close the sensor is to the keyboard, when the sensor moves toward the keyboard, VEM changes the color of the sensor to indicate a key press and provides an auditory response to the user to denote a key press.
  • the virtual keyboard may be configured to respond to absolute movement of each of the user's fingers (sensors).
  • the threshold values are set to zero or near zero, wherein any movement of the user's finger will correlate to a typing motion or mouse motion depending upon where the user's fingers are located in relation to the virtual devices.
  • the system prompts the user to enter the user's login and password.
  • the user's saved settings are loaded into a memory space.
  • a user's settings may control what is displayed on the virtual display device such as input devices (keyboard, mouse, tablet), the location of these input devices in the virtual setting and any other user definable preferences such as visual or audio feedback, tactile feedback and sensitivity of sensor movement, or other preferences.
  • user A may have saved settings from a previous use, wherein once logged into the system, VEM (such as VWM) displays a virtual keyboard, a virtual monitor and a virtual mouse, wherein the virtual workstation manager controls the origin coordinates of where each of these devices will be displayed in the virtual environment.
  • VEM such as VWM
  • the virtual workstation manager controls the origin coordinates of where each of these devices will be displayed in the virtual environment.
  • the system prompts the user whether or not they want to customize the default values. If the user chooses to customize the default values then Box 243 .
  • Box 243 the user's customized settings are saved under their login profile and then to Box 240 . If the user does not choose to customize the default settings then Box 240 .
  • the display device is initialized and at Box 250 the virtual devices are displayed in the virtual environment at the location dictated by either the user's settings or the loaded default values.
  • the virtual environment manager will be called wherein the user's preferred settings will be retrieved from a memory location and loaded into a memory space. If the user is new to the system, the Virtual Environment Manager will be continue to use system default values into memory. These default values may be changed by the user and saved under the user's personal profile before exiting the system, wherein these settings will be loaded when the user utilizes the system in the future.
  • Examples of user definable settings are the type and style of keyboard that will be displayed in the virtual environment.
  • the user may prefer to utilize a standard size keyboard having a standard QWERTY layout, or the user may prefer to utilize an ergonomic keyboard.
  • Additional settings would be the physical location of the keyboard in the virtual environment. Such that when the system is initialized the keyboard will be shown in the virtual environment wherein the coordinates of the keyboard are known.
  • additional settings may control the display of other virtual input devices such as a mouse, a virtual monitor or similar devices.
  • Each of the virtual devices displayed within the virtual environment will have origin coordinates known to the virtual environment manager. The user in the virtual environment may control the style, size and location on the virtual input devices.
  • the keyboard may include handles, wherein the user can grab the handles in the virtual environment and move, stretch, pull-apart the keyboard.
  • the other virtual input devices may also include handles to allow the devices to be re-positioned re-sized or stylized within the virtual environment.
  • the virtual environment manager can saves the user's settings, wherein the virtual environment may be loaded each time as left by the user in each previous use
  • the operating system and virtual environment manager and MD-BIOS finishes loading into memory with the user's settings.
  • the display device is initialize, Box 240 , wherein a graphical user interface (GUI) is displayed on the display device Box 250 .
  • GUI graphical user interface
  • the operating system is based on Microsoft Windows.RTM. then the standard GUI interface will be displayed in the display device.
  • virtual input devices will also be display on the display device. As described above, the placement, size, shape and orientation of the input devices as displayed within the virtual environment is dictated by either the users saved preferences or in the case of a new user the default settings of the Virtual Environment Manager.
  • a virtual set of hands may also be displayed if the user is wearing sensors on each of their fingertips.
  • the transmission rate of the signal is controlled by MD-BIOS and the virtual workstation manager in conjunction with the coordinate location of the sensors in relation to the tracker.
  • the transmission rate of the tracker may be controlled according to sensor position. For example, if it is determined that the sensor is disposed near the known location of an input device, then the tracker's scan rate will be increased to increase accuracy of the system, if the sensor is disposed at a given distance from any virtual input device than the transmission rate of the tracker will be reduced, thereby saving power and reducing processing cycles.
  • MD-BIOS will direct the transmitter to energize the alternative transmission sources, wherein the coordinates generated from these alternative transmission sources will be compared and the strongest or more accurate coordinate information will be utilized. Wherein MD-BIOS will direct the transmitter to turn off the alternative transmitting devices. In a preferred embodiment this process will occur automatically without any user interaction.
  • each of the sensors are constantly sampled by the SEU and coordinate data is sent to MD-BIOS where a timestamp is added and the information is placed into a memory location.
  • the virtual environment manager determines if the velocity component(s) and the vector component(s) of the detected sensor motion pass threshold values. Threshold values are utilized to filter out undesired input as well as increase system accuracy. For example there are natural vibrations associated with humans such as heartbeat, slight hand motion or other natural vibrations. If it is determined that the vector components are greater than the threshold values, then it is determined if the coordinates are in the vicinity of the known coordinates of the virtual input devices generated by the virtual environment manager. For example, if the coordinates received correlate to a keyboard position for a letter, the velocity component and the previous coordinates are compared to determine the intentions of the user.
  • the vector and velocity components will be utilized to determine if the user intended a mouse move, a mouse button depress/release, or a single or double click.
  • the movement of the sensors will be displayed on the display device in the form of a virtual set of hands or in the case of mouse motion, movement of a pointer within the virtual environment.
  • audio or tactile feedback maybe provided in the form of “clicking” sounds intended to replicate the physical sound of a standard keyboard.
  • the coordinate information generated from sensor motion is compared to the known coordinate information of the virtual input devices. For example, if a virtual keyboard is displayed according to the user's defined settings, coordinates of the keyboard are a known value that are stored within the shared memory manipulated through MD-BIOS API calls. Thus, when coordinates are calculated from sensor motion, the coordinates transmitted from the memory location of MD-BIOS are compared to the known coordinates of the keyboard. If the coordinates correspond to or are within a pre-defined range of an individual key disposed on the keyboard then the virtual environment manager will determine the user's intention as indicated in Box 330 . The virtual environment manager determines the user's intention by comparing the velocity component generated by sensor movement and comparing the velocity component to a known value.
  • the virtual workstation manager will interpret this to indicate a key press on the virtual keyboard. If the vector component is away from the plane of the keyboard it may also indicate that a key release was performed and the user's finger is moving away from the key. If the velocity component is zero or nearly zero this may be interpreted to indicate a key hold, such as a user holding down a shift key or the like. A vector of zero may also indicate a key repeating such as when a typed word includes two or more of the same characters. After the user's intention is determined then at Box 305 , the intended information is sent to the operating system for entry into the event queue or windows manager.
  • the key presses may be determined utilizing pure vector and velocity without consideration of the z-axis component. This would allow a user to utilize the present system while resting their hands on a surface, such as a tabletop or upon their person or the like.
  • the user's intended motion may be displayed on the virtual display device in the form of a virtual key-press on the virtual keyboard, motion of a virtual hand or pair of virtual hands, or an auditory or tactile response may be generated in response to the sensor's motion.
  • the system returns to Box 270 , wherein the system returns to scanning for sensor motion and the process is repeated until the system is powered off.
  • the sensor motion component has been described herein in reference to a key press on a keyboard it shall be understood that this should not be considered limiting in any manner.
  • the coordinates may correspond to any type of virtual input device, wherein each virtual input device has parameters which are utilized by the virtual environment manager to interpret the sensor motion. For example, if the coordinates received correlate to coordinated for a mouse, it will be determined if the coordinates define movement of the mouse, which will be displayed within the virtual environment. A mouse button press, mouse button release, a single button press or a double button press.
  • the virtual environment manager may be capable of distinguishing between different sets of sensors by each sensor(s) or transducer(s) unique ID. For example, if two users both utilizing a system in accordance with the present invention are near one another each tracker will only track those sensors to which the system is initialized. Thus, multiple systems can be utilized within close proximity to one another.
  • the functional flow diagram illustrates a graphic based system, wherein the virtual reality system in accordance with the present invention is configured to display three-dimensional virtual input devices in a virtual environment. It is further contemplated that the present invention may be utilized in a text-based system.
  • the functional flow diagram as shown in FIG. 12 would not include Box 200 wherein the operating system is loaded into a memory space.
  • the virtual keyboard generated by the virtual environment manager would be displayed one dimensionally and the virtual input device in a preferred embodiment would be a touch pad or a similar tracking device.
  • BIOS chip a portion of the software components of the present invention may be embodied in the form of a BIOS chip, wherein the BIOS chip would be installed on the computer system's mother board, such that when the computer system is powered on the software in accordance with the present invention would be loaded into a memory space from the BIOS.
  • FIGS. 13 and 14 there are shown exemplary embodiments of various virtual keyboards as displayed in the virtual environment.
  • the virtual keyboard 300 may be disposed over the top of the virtual mouse 310 thereby allowing a user to immediately toggle between the keyboard and the mouse without having to physically move the sensors a great amount.
  • This type of setup reduces motion associated with switching back and forth between the keyboard and the mouse, thereby potentially reducing repetitive stress injuries.
  • users of programs that require a user to switch back and forth between a pointing device and text input may be able to increase their productivity because less time is spent switching between the two devices. Referring now to FIG.
  • the virtual keyboard 300 and virtual mouse 310 wherein the virtual mouse is disposed over the virtual keyboard.
  • the virtual mouse can be transposed between the two positions shown either through specific hand motions wherein the sensors and VEM generate a specific signal to transpose the mouse from one position to another.
  • the mouse may be transposed utilizing a voice command, a hot key associated with the virtual keyboard or a hot spot located on within the virtual display.
  • a conventional computer mouse is shown being displayed in the virtual environment, it is contemplated that any type of input device may be displayed, for example the input device may be a touch pad, trackball, tablet and stylus or similar input devices.
  • the system of the present invention is configured as a virtual car control system (VCCS) 400 .
  • the VCCS comprises software and hardware components, wherein the software and hardware components interact to provide a virtual system that enables a user to control various systems within an automotive environment without physically removing their hands from the steering wheel and shifting their eyesight from the road.
  • the VCCS system 400 includes hardware and software elements to replicate and control mechanical and electrical controls within an automotive environment.
  • the VCCS includes a computer unit 410 , a tracker system 450 including a system electronic unit 420 , at least one sensor 425 and at least one transmitter 430 , a control button 460 and a heads up display device 470 or alternatively a see through HMD.
  • the VCCS further includes software components 480 stored on a computer readable medium disposed within the computer unit 410 .
  • the software components include an operating system 482 , a virtual environment manager program 484 and a multi-dimensional basic input/output subsystem (MD-BIOS) 486 program.
  • MD-BIOS multi-dimensional basic input/output subsystem
  • FIG. 16 there is shown an exemplary embodiment of the VCCS as seen from a user's perspective in accordance with the present invention.
  • an image showing three-dimensional buttons 500 would be projected onto the windshield 505 by a heads up display device 470 , when the buttons are displayed on the windshield coordinate information is associated with each of the buttons.
  • the coordinate information of each button locates each button in space in a plane spaced apart from the steering wheel, this can be seen by the three-dimensional buttons disposed on box 510 which are shown for exemplary purposes only and will not be actually seen by the user.
  • the origin coordinates for each button in a plane aligned with the physical steering wheel of the vehicle, the user may interact with the buttons without removing their hand's from the steering wheel.
  • the VCCS may further include at least one physical button.
  • the physical button(s) would be disposed on the steering wheel or steering column.
  • the physical buttons may be utilized to turn the heads up display on and off, menu return, enter, clear, dial, send or any similar command which may be utilized to control any aspect of the VCCS system.
  • At least one sensor would be disposed on a user's person, in a preferred embodiment at least one sensor would be disposed on each of the user's thumbs, but it is contemplated that additional sensors may be disposed on the user's additional fingers.
  • the sensor(s) interact with a signal generated from a transmitter, wherein the sensors produce a position signal due to interaction with the transmitted signal.
  • the position signal is converted into coordinate information by the SEU after being transmitted to the SEU.
  • the coordinate information is transmitted to the MD-BIOS software where time stamp data is added and the coordinate information is stored in a memory location.
  • a sensor identification tag may also be associated with the sensor, added by the SEU or added by MD-BIOS to the coordinate data.
  • the coordinate data is then utilized by the VEM to determine if motion of a sensor passes threshold values, and if so, what is the user's intent, the coordinate data is then compared to the origin coordinate information for each of the three-dimensional buttons virtually located in a plane adjacent to the steering wheel.
  • a virtual hand may also be displayed on the windshield to illustrate the motion of the sensor or a cursor or other visual marker may be displayed on the windshield so that the user may visually correlate their hand location on the steering wheel with the virtual three-dimensional buttons displayed on the windshield.
  • buttons would be displayed on the windshield as having three-dimensional characteristics. Wherein multiple layers of buttons may be utilized to control various systems. For example, as one button is pressed, additional buttons may be displayed on the windshield, wherein the additional buttons are displayed as being tiled over the previous layer of buttons. This may be better understood as shown in FIGS. 17-19 wherein there is shown an exemplary embodiment of the VCCS system in use in accordance with the present invention.
  • buttons as they would be displayed by the heads-up display device and projected onto an automobiles windshield.
  • the display device or the VCCS system may be controlled by a physical button wherein, after the button has been depressed the buttons would be displayed on the windshield.
  • the buttons displayed would have origin coordinates that would place the buttons in plane space adjacent to the user's left hand on the steering wheel wherein the first level of buttons displayed would be high-level system buttons.
  • one button will be displayed to access the climate control system, another for the GPS system, and another for an entertainment system and so on. It shall be understood that the list above should be considered exemplary and should not be considered limiting in any manner.
  • buttons would be displayed by the heads-up display device and projected onto the windshield as shown in FIG. 18 .
  • the additional buttons displayed would have origin coordinates that would place the buttons in plane space adjacent to the user's right hand on the steering wheel.
  • the additional buttons would provide the user with audio/video components as installed, such as AM/FM radio, cassette, compact disc, DVD audio and the like. As shown in FIG.
  • the user has selected the CD button, wherein the visual display of the CD button is changed, for example the button is made larger thereby appearing to be in a different plane than the other buttons and the color of the button may change to indicate that the button has been selected. Additionally, it is contemplated that an audible feedback may also be associated with button selection in addition to the visual feedback described above.
  • buttons will be displayed adjacent to the selected CD button wherein the additional buttons include controls associated with the CD player. For example, fast forward, skip, rewind, disc selection, mute, volume and menu return.
  • buttons may be displayed to control a navigation system such as a GPS system, climate control system, communication system such as a cellular phone, as well as automotive control systems such as traction control, system information and the like.
  • a navigation system such as a GPS system, climate control system, communication system such as a cellular phone, as well as automotive control systems such as traction control, system information and the like.
  • each of the top level buttons have been selected thereby causing the second menu to be displayed on the screen.
  • FIGS. 16-20 should be considered exemplary and not limiting in any manner in that the menus may be altered without deviating from the scope of the invention.
  • the VCCS system may be utilized in combination with mechanical switches or physical control means.
  • the entertainment system may be controlled with the UCC system by the driver and also may be controlled by the driver or a passenger with conventional means installed in the automobile.
  • the heads-up display may be controlled by a mechanical switch, a virtual switch or the display may be time controlled, wherein once activated, if no input is received from the user, the display will turn off, wherein the user can then re-activate the system.
  • the system may have additional restrictive controls such as a maximum speed control, wherein the system may not be accessed if the vehicle is traveling beyond a pre-set speed, or if the steering wheel has been rotated past a pre-set degree.
  • the VCCS system may utilize voice command software in addition to those hardware and software components described above.
  • the sensors are further capable of generating yaw, pitch and roll coordinate information in combination with the hardware and software components of the present invention.
  • the additional coordinate information may be utilized by applications not mentioned specifically herein. For example, if the virtual environment manager of the present invention was tailored to represent a virtual flight training simulator, the yaw, pitch and roll coordinates would be utilized to simulate movement of a virtual plane.
  • the present invention has been described with reference to a virtual workstation such as a personal computer or laptop computer it is contemplated that the system and methods in accordance with the present invention maybe utilized to replace other types of physical systems. Such as Virtual Game station or VR conference.
  • the present invention may be utilized to replace a gaming system such as an X-Box.RTM. or a Playstation.RTM..
  • the present invention may be utilized in combination with others, wherein the multiple systems may be utilized for communication between each of the users. Systems such as this may be utilized by military forces for covert communications, pilots, motorsport race teams and other similar careers that may require communication amongst more than one person.
  • Another contemplated use for the present invention is warehouse management and inventory control, wherein the user may roam freely around the warehouse entering inventory, fulfilling orders, or maintaining quality control.

Abstract

In accordance with the present invention there is provided devices and methods for controlling a microprocessor controlled device in a virtual environment through the use of at least one sensor disposed at least one user's finger and a scanner, said sensor and scanner being utilized to input position data, the method comprising the steps of: loading an operating system in a computing environment; displaying a virtual keyboard and a virtual pointing device on a display device; initializing coordinates defining individual keys of said keyboard; initializing coordinates defining a location of said pointing device in relation to the keyboard; monitoring the position of at least one sensor disposed on a user's finger; determining if sensor movement correlates to depression of a key on the keyboard and providing feedback to the user to indicate that a key was depressed on the keyboard; transmitting data correlating to the depression of the key; and returning to the monitoring step.

Description

    CROSS REFERENCE TO RELATED APPLICATION
  • The present application is a continuation of U.S. patent application Ser. No. 10/703,184 filed Nov. 5, 2003, and entitled “Virtual Workstation,” which claims the benefit under 35 U.S.C. § 119(e) to U.S. Provisional Patent Application No. 60/424,557 filed Nov. 6, 2002. Said applications are incorporated by reference into the present application in their entirety.
  • BACKGROUND OF THE INVENTION
  • 1. Field of the Invention
  • The present invention is generally related to methods and devices for producing a virtual computing environment. More specifically, the present invention is related to methods and devices that in combination mimic the functionality and behavior of a physical computer system in a virtual environment. In particular the methods and devices in accordance with the present invention includes a portable computing system wherein the system is configured to provide the user with a virtual hardware environment that may be utilized for computing, gaming or other uses.
  • 2. Description of the Relevant Art
  • With the advent of the modern computer input devices were invented to allow for the input of data, for example, early computer systems utilized punch cards to input data into the computer's memory. Although punch cards were effective at inputting data, a more simplified device was necessary, thus the modern keyboard was developed.
  • One of the most important factors contributing to the effective use of a computer is the interface between the computer and a person using it. Unquestionably the most popular computer interface device is the keyboard, which has a plurality of depressible keys each corresponding to a particular alphanumeric character, symbol, or computer function. While computer keyboards are widely accepted and quite suitable in many situations, keyboards are not always the most efficient, convenient, or easy to use devices.
  • A drawback of computer keyboards is that they include up to 110 individually marked keys mounted to a base with as many switches. All of these components must be produced and assembled, which accounts for considerable expense. Since keyboards are mechanical, are also more prone to failure than solid-state devices, additionally, due to the likelihood of failure, broken keyboards additionally present disposal problems. Further, the conventional keyboard cannot be quickly changed to a new keyboard layout, such as might be desired by those who have learned a keyboard layout other than the somewhat inefficient but traditional QWERTY layout.
  • Another drawback of computer keyboards is that they are built generally in one size for all users. As a result, users with relatively small or large hands must adapt to a keyboard size that is not optimal for their particular hand size. A person with smaller hands must stretch further to strike a key some distance from a home row of keys, whereas a person with larger hands will have a harder time accurately striking any desired key. Keyboard size that is optimized for a particular use may lead to decreased hand fatigue. However, keyboard manufacturers have determined an ergonomically acceptable compromise, which is a compromise nevertheless. Since keyboards are produced having only one size forces a user to type with his hands close together in an unnatural manner. It has been found that so called “split” keyboards, which are split into a separate keyboard for each hand, are more comfortable for the user and produce a slightly faster typing speed as a result. Additionally, as computers become more common in the workplace, a greater number of injuries have been reported due to utilizing a keyboard.
  • There have been attempts by various manufacturers to address the problems associated with mechanical keyboards. One such example is described in U.S. Pat. No. 5,581,484, wherein there is described a finger mounted computer input device. The finger-mounted device utilizes a series of pressure sensors to determine a users hand movement which then corresponds to a key on a keyboard. A problem with this type of system is that the user must still physically interact with a surface to generate a signal. Additionally, the sensors are usually disposed on a glove, wherein the user wears the glove to utilize the system. A problem associated with glove-based systems is that the material from which the glove has been fabricated has a fatigue life, and therefore will eventually wear out from prolonged usage. Additionally, a user may experience discomfort from using these types of gloves in that they may perspire inside the glove. The perspiration may further lead to degradation of the glove.
  • Another example of a virtual keyboard are produced by www.vkb.co.il, www.canasta.com, and www.virtualdevices.net. These types of keyboards utilize an infrared projection system, wherein a keyboard is projected onto a surface and a sensor detects the position of a finger on top of the projected keyboard image. A problem with they types of keyboards is that you can only utilize the system on a smooth clean non- transparent steady surface, additionally, if you rest your hands within the projected keyboard the sensor may interpret this motion as keystrokes, thereby resulting in errors. Further still, since the keyboard is projected onto the surface, the user may experience light interference from surrounding light sources.
  • Lastly, with the resurgence in tablet type computers having pressure sensitive screens, Microsoft.RTM. has released an on-screen keyboard in their latest version of Windows.RTM. that enables a user to utilize their fingers or a stylus to input data.
  • While each of these systems are a novel approach to overcoming the dependence on a physical keyboard, there are still shortcomings. Namely, each of the systems require the user to either be physically tethered to a computer, where pressure sensitive devices that must be depressed on a surface or find a smooth surface to set up a virtual keyboard. Additionally, these devices do not allow a user to customize the layout of the keyboard or adapt the keyboard to a users specific style of typing. In addition to being tethered to a computer, the use of physical input devices may cause injury to the user. For example, many claims are filed every year for repetitive stress injuries incurred from keyboard usage. Examples of common injuries are carpal tunnel syndrome, eye fatigue, neck and back strain, many of which are attributed to usage of a personal computer.
  • Attempts have been made to eliminate the use of a keyboard as an input device entirely. Many manufactures have attempted to produce voice recognition software systems, wherein a user could speak every command to a computer thereby eliminating the need for a physical or virtual keyboard. While this approach may be novel, presently voice recognition software has not advanced to the point of being reliable enough to replace a keyboard. In addition to requiring more hardware, a microphone, the voice recognition software is always running within a computer's operating system, thus requiring additional computing power. Also, voice recognition software must be custom tailored to each user's voice, inflections and/or accents, therefore once a system has been customized to an individual user other user's cannot readily utilize the system. Another shortcoming of voice recognition systems is that it is difficult to use voice recognition for editing, browsing the Internet, graphic design and similar input intensive programs. Additionally, constant talking may fatigue the user's voice, wherein the user's pitch and tone may change, thereby leading to additional input errors because the voice recognition software no longer recognizes the user's voice. Further still, voice recognition systems cannot be utilized in cubicle type work environments or similar “open” type environment where noise interference from other voices may confuse the voice recognition software.
  • Additional input devices may also be utilized in conjunction with keyboards. For example, pointing devices, such as “mouse” pointing devices and so called “track ball” devices are also popular computer interfaces. Generally, these types of devices provide velocity information, in both an X direction and an orthogonal Y direction, to the computer, as well as signals from one or more momentary contact push buttons. A pointing icon or other “tool” on a computer monitor responds to such velocity input by corresponding X and Y movement on the computer monitor. Graphics tablets are another type of “pointing” input device that provide the computer with X and Y positional information, as opposed to velocity information, which is used in much the same manner by the computer. Such devices are well suited for pointing to various software “push button” options on the screen, selecting portions of text or a group of software “objects,” freehand on-screen drawing, positioning a typing cursor location, and similar functions. However, such pointing devices are remarkably ill suited for text data input.
  • Other types of computer interfaces have been developed to overcome some of the above-mentioned drawbacks. For example, U.S. Pat. No. 5,212,372 to Quick et al. on May 18, 1993, teaches a glove device that has sensors for measuring the curvature of each finger at joints thereof. For entering numerical data, a person using this type of device curves his fingers to point to “zones,” or virtual keys, that each represents a particular number. While the input of alphabetical data is mentioned in the Quick disclosure, only numerical zones are illustrated and it remains unclear how such a device could possibly be used to enter the twenty-six additional characters of the alphabet, especially since the little finger is used solely for designating an “enter” key and is therefore not available for pointing to alphanumeric zones.
  • A variety of similar glove-based prior art devices exist, and in most cases each uses some type of joint flexing sensor to determine finger curvature. Many such devices are designed for communication with deaf or otherwise challenged individuals, and typically provide for computer interpretation of alphanumeric data formed by a single hand with standard sign language. It is a slow and fatiguing process for people, even those fluent in sign language, to use such devices to enter a large amount of data into a computer, such as might be required while typing a patent disclosure, for example. Further, while finger curvature is relatively easy to detect in a variety of sophisticated ways, such detection is only accomplished in one dimension. Lateral movement of the finger, for example from the “J” key to the “H” key of a standard QWERTY keyboard, cannot be detected by such joint flexure sensors as disclosed in the prior art. This drawback is also evident in many “virtual reality” data manipulation gloves, which also include a variety of motion sensors on similar gloves. As a result, such devices have limited use and are not well suited for prolonged data entry from a wide selection of character and command keys, such as those found on the standard computer keyboard. As previously described, these gloves are generally fragile and are not constructed for constant everyday usage. Additionally, the gloves are particularly sensitive to moisture such as sweat from the users hands or a wet environment, wherein moisture may cause sensor problems or lead to eventual failure of the glove.
  • Therefore there is a need for a device that eliminates the shortcomings of the presently available input devices, wherein the device may be utilized by a user in any physical configuration without requiring the user to remain physically limited by the device. Such a needed device would be adaptable to any individual, regardless of hand size or typing style. Further, such a needed device could be used equally well for both alphanumeric data entry, command entry, and position/velocity input. Such a needed device would be to a large extent software re-configurable, making use of the device immensely flexible and adaptable. The present invention fulfills these needs and provides further related advantages.
  • These and other objects, advantages, and features of the invention will become apparent to those persons skilled in the art upon reading the details of the methods and systems of the present invention, which are more fully described below.
  • SUMMARY OF THE INVENTION
  • The present invention provides systems and methods for interaction within a virtual environment. Wherein the methods of the present invention may be utilized to virtually interact with a computer system through the use of virtual input devices, wherein the methods and systems allow a user to input data into a computing system without physical limitations.
  • In accordance with the present invention there is provided a method for controlling a microprocessor based system in a virtual environment, the method comprising: loading a computer program into a memory space; transmitting at least one signal from at least one transmitter; displaying virtual input devices on a display device, wherein the virtual input devices initially have pre-determined coordinates; receiving data generated from movement of at least one sensor; calculating coordinates of the sensor movement; comparing calculated coordinates to the pre-determined coordinates of each virtual input device; calculating desired input from coordinates generated by the sensor movement; and displaying input on the display device and transmitting the input to the operating system.
  • In accordance with another embodiment of the present invention there is provided a method for controlling a microprocessor controlled device in a virtual environment, the method comprising: loading a computer program into a memory space; loading an operating system into a memory space; transmitting at least one signal from a transmitting device displaying a virtual keyboard and a virtual input device on a display device; initializing coordinates defining individual keys of the keyboard; initializing coordinates defining a location of the input device in relation to the keyboard; receiving data at least one sensor wherein the data received is converted into coordinate information and stored in a memory space; determining if coordinated derived from movement correlates to a key location of the virtual keyboard or movement of the virtual input device; and displaying sensor movement on the display device.
  • In accordance with the present invention there is provided a method of generating and controlling a virtual workstation, the method comprising: initializing an operating system and a controller system in a microprocessor based computer system; displaying virtual hands, a virtual keyboard, a virtual pointing device and a virtual workstation environment on a display device; monitoring movement of sensors disposed on a user's hands for movement; displaying movement of the virtual hands in response to movement of the sensors; and determining if movement of at least one sensor passes a user defined threshold.
  • In accordance with the present invention there is provided a method of generating a virtual gaming system, the method comprising: initializing an operating system and a controller system in a microprocessor based computer system; loading a game program into a memory space; displaying a virtual player on a display device; monitoring movement of sensors disposed on a user for movement; and displaying movement of the virtual player in response to movement of the sensors.
  • In accordance with the present invention there is provided a system for virtually controlling a microprocessor based system, the system comprising: a microprocessor based computer system having an operating system configured to be run thereon; a display device, at least one tracking device; at least one sensor, wherein said tracking device is configured to track movement of said sensor and determine coordinates of said sensor and time components of sensor movement within a pre-defined area; and a software component, wherein said software is stored in a computer readable medium, wherein said software is in communication with said tracker and said display device, wherein said software determines vector movement and acceleration of said sensor and displays said sensor movement on said display device.
  • In accordance with the present invention there is provided a system for implementing a virtual reality environment, the system comprising: a display device associated with a user, the display device being responsive to image data for generating and displaying an image simulating a physical computer system, including a physical keyboard, a physical input device, and physical representation of the user's hands, wherein each of the simulated components appear to the user to be in space independent of actual physical objects; an output device for transmitting a signal; an input device for generating data in response to interaction with the signal; a processor connected to the input and output device and the display device and operating a virtual environment manager program and a multi-dimensional basic input and output program for generating a virtual keyboard, a virtual input device, and virtual hands, the processor being responsive to data generated from the input device, for generating motion image data corresponding to the input device data; and wherein the display device is responsive to motion image data for generating a second image simulating physical motion of at least one virtual component.
  • In accordance with the present invention there is provided a system for implementing a virtual reality (VR) computing environment, the system comprising: VR display device including at least one display and worn by a user the one display viewable by the user, with the VR display, responsive to first image data, for generating and displaying a first VR image simulating a physical computer system including a virtual keyboard having a plurality of physical keys, a virtual mouse having at least one physical button, with the first VR image representing the VR keyboard and VR mouse, the VR keyboard and VR mouse having a first appearance corresponding to the first image data; an input and an output device for generating motion-representative data corresponding to motion of a user's body part; and a processor connected to the VR display device and operating a virtual environment manager (VEM) and multi-dimensional basic input/output subsystem(MD-BIOS) program, wherein the VEM and MD-BIOS provide the first image data to the VR display device, the processor being responsive to the motion-representative data generated from the input device, for generating motion image data corresponding to the motion; and wherein the VR display device is responsive to the motion image data for generating a second VR image simulating motion corresponding to the motion of the portion of the body of the user.
  • In accordance there is a need for smaller input/output interfaces as miniaturized portable computing devices become more common.
  • There is also a need for a system that recreates a full desktop computing experience without requiring the space needed for a full desktop computer or physically limiting a user to a physical location to utilize such a system.
  • There is an additional need for a system that can be utilized to minimize computer related injuries such as repetitive stress injury, carpal tunnel injuries and other such injuries that are related to physical computer use.
  • There is also a need for a system that is capable of displaying true three dimensional real world human-machine interactions.
  • It is the applicant's belief that the present invention addresses these needs with novel software and hardware solutions as described in detail below.
  • Other aspects, features and details of the present invention can be more completely understood by reference to the following detailed description of a preferred embodiment, taken in conjunction with the drawings and from the appended claims.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • To facilitate understanding, the same reference numerals have been used (where practical) to designate similar elements that are common to the Figures. Some such numbering has, however, been omitted for the sake of drawing clarity.
  • FIG. 1 is a block diagram illustrating exemplary mechanical devices that may be utilized with the methods in accordance with the present invention.
  • FIG. 2 is an exemplary embodiment of a computer system in accordance with the methods of the present invention.
  • FIG. 3 is an exemplary block diagram of the tracker system in accordance with the present invention.
  • FIG. 4 is an exemplary embodiment of the system in accordance with the present invention.
  • FIG. 5 is an exemplary embodiment illustrating an alternative embodiment of a display device in accordance with the present invention.
  • FIG. 6 an exemplary embodiment of an alternative display device in accordance with the present invention.
  • FIG. 7 an exemplary embodiment of a user's hand illustrating the wireless communication of the tracker system.
  • FIG. 8 is a functional flow diagram illustrating the method in accordance with the present invention.
  • FIG. 9 is a functional flow diagram illustrating the interaction between the software and hardware components of the present invention in accordance with the methods of the present invention.
  • FIG. 10 is a block diagram illustrating an exemplary embodiment of the memory structure of the multi-dimensional basic input/output system in accordance with the present invention.
  • FIG. 11 is a block diagram illustrating the threshold values and calculation methods of the virtual workstation manager in accordance with the present invention.
  • FIG. 12 is a functional flow diagram illustrating the method steps in accordance with the present invention.
  • FIG. 13 is an exemplary embodiment of a virtual workstation environment as seen from a user's perspective, wherein a virtual keyboard is shown disposed three dimensionally over a virtual mouse.
  • FIG. 14 is an exemplary embodiment of a virtual workstation environment as seen from a user's perspective, wherein the virtual mouse is shown disposed three dimensionally over the virtual keyboard.
  • FIG. 15 is a functional flow chart illustrating hardware components of a virtual car control center.
  • FIG. 16 is an exemplary embodiment of the present invention wherein the virtual environment manager has been configured as a virtual car control system.
  • FIG. 17 is an exemplary embodiment of the high-level virtual buttons in accordance with the virtual car control system.
  • FIG. 18 is an exemplary embodiment illustrating the high-level virtual buttons and the second level of virtual buttons in accordance with the virtual car control system.
  • FIG. 19 is an exemplary embodiment illustrating the high-level virtual buttons, the second level, and third level of virtual buttons in accordance with the virtual car control system.
  • FIG. 20 illustrates exemplary embodiments of additional second level virtual buttons after selection of the high-level virtual button.
  • DETAILED DESCRIPTION OF THE INVENTION
  • Before the present invention is described in such detail, it is to be understood that this invention is not limited to particular variations set forth herein as various changes or modifications may be made to the invention described and equivalents may be substituted without departing from the true spirit and scope of the invention. In addition, many modifications may be made to adapt a particular situation, material, composition of matter, process, process act(s) or step(s) to the objective(s), spirit or scope of the present invention. All such modifications are intended to be within the scope of the claims made herein.
  • Methods recited herein may be carried out in any order of the recited events that are logically possible, as well as the recited order of events. Furthermore, where a range of values is provided, it is understood that every intervening value, between the upper and lower limit of that range and any other stated or intervening value in that stated range is encompassed within the invention. Also, it is contemplated that any optional feature of the inventive variations described may be set forth and claimed independently, or in combination with any one or more of the features described herein.
  • All existing subject matter mentioned herein (e.g., publications, patents, patent applications and hardware) is incorporated by reference herein in its entirety except insofar as the subject matter may conflict with that of the present invention (in which case what is present herein shall prevail). The referenced items are provided solely for their disclosure prior to the filing date of the present application. Nothing herein is to be construed as an admission that the present invention is not entitled to antedate such material by virtue of prior invention.
  • Reference to a singular item, includes the possibility that there are plural of the same items present. More specifically, as used herein and in the appended claims, the singular forms “a,” “and,” “said” and “the” include plural referents unless the context clearly dictates otherwise. It is further noted that the claims may be drafted to exclude any optional element. As such, this statement is intended to serve as antecedent basis for use of such exclusive terminology as “solely,” “only” and the like in connection with the recitation of claim elements, or use of a “negative” limitation. Last, it is to be appreciated that unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
  • Referring to the detail drawings and the disclosure herein, the term “keyboard” is defined herein to include alphanumeric keyboards, subsets of alphanumeric keyboards, keypads including numerical keypads, telephone and DTMF keypads, security access input devices using buttons and labels, etc., and so it not limited to QWERTY alphanumeric keyboards. Accordingly, it is understood that the use of the term “keyboard” and the depiction in any of the figures of a keyboard such as a QWERTY alphanumeric keyboard typically used with personal computers and the like is only an example of a keyboard for use, interaction, and operation by a user for any application of keyboards for input and/or output devices. As defined herein, the term “keyboard” is more than a plurality of keys, since a keyboard includes a layout of the plurality of keys as well as keys, with the layout typically being predetermined. The keys may be associated with symbols such as alphabetical, numerical, mathematical, or other representations, and the keys may include associated pictorial or symbolic representations thereupon. Accordingly, a keyboard is not identical to a set of buttons but may be a plurality of buttons having a layout and a set of symbols associated with each key or button.
  • The term “virtual reality” and its abbreviation “VR” are herein defined to include, but not limited to, visual and/or other sensory applications implemented using software and/or hardware to simulate and/or provide representations of environments which may be different from the physical environment of the user. Such VR may provide visual and/or multimedia zones, worlds, and work areas in which the user and/or other software applications may change and interact representations of elements in the VR environment. For example, in a VR world, a graphical representation of a switch may be changed to represent the flicking or switching of the switch, which may have an associated switch-flicking sound, which is activated by flicking the switch. In addition, the VR switching of the VR switch may cause the actuation of other events, either in the VR world or in actual physical devices and structures; for example, the flicking of the VR switch may cause an actual computer to be turned on or off. Accordingly, the term “virtual reality” is not limited to simulations or representations of VR devices and information in VR worlds, but may also be extended to physical devices as well as, in hybrid implementations, to both physical and VR devices.
  • In accordance with the present invention, the detail description of the present invention will be divided into sections, wherein each section will be utilized to described components of the present invention. It shall be understood that the examples described herein should not be considered limiting in any manner and should be considered exemplary.
  • In accordance with the present invention there is provided devices, systems and methods for providing a human interface device that may be utilized to simulate a virtual environment. The human interface device includes hardware components consisting of a tracker assembly, a processing unit, and a display device. The software/firmware components comprise a virtual environment manager (VEM) and a Multi-Dimensional Basic Input/Output subsystem (MD-BIOS), wherein the software and hardware components are utilized in combination to provide a virtual environment. Examples of applications of the present invention include a virtual workstation environment (VWE) wherein a physical computing system is simulated in a virtual environment including virtual hands and virtual input devices such as a keyboard, pointing device or other similar input devices. Another contemplated application for the present invention is a virtual gaming system (VGS) wherein the present invention could be utilized to simulate a gaming environment wherein the user could virtually interact within the game. The above examples are to be considered to be merely exemplary in that the present invention may be utilized in many other applications such as military use, flight simulation/training, corporate meetings, etc.
  • The present invention will now be described in greater detail below with regard to the system's individual components.
  • Hardware
  • In accordance with the methods of the present invention there will be disclosed hardware devices that may be utilized in accordance with the methods of the present invention. It shall be understood that many of the hardware components are described in a general sense and that many other types/styles of similar hardware devices may be substituted for those described herein. For example, as described in the present invention a computer system may be embodied as an Intel.RTM. central processor unit (CPU) based system a RISC based processor system, though it shall be understood that this should not be considered limiting in any manner in that a computer system for use with the present invention may be based on similar microprocessor devices.
  • Referring now to FIG. 1, there is shown an exemplary embodiment of hardware components in accordance with the present invention. As shown in FIG. 1, the hardware components comprise a microprocessor based computer system, a display device, and a tracking system. In addition to the hardware shown, the present invention further includes software running stored in a computer readable medium, wherein the software is in association with the hardware components of the system. The software of component of the present invention will be described in greater detail below in reference to the methods of the present invention.
  • As shown in FIG. 2, the microprocessor based computer system 10 includes a central processing unit 20, memory 30, a communication bus 40, a computer readable storage device 50 such as an optical storage device, magnetic storage device, flash memory storage device or similar computer readable storage mediums, and at least one communication port 60. The communication port 60 comprise any one of the following types of communication ports as well as a combination thereof: universal serial bus (USB), IEEE 1394 (firewire), serial port, parallel port, infrared port, 802.11b, 802.11a, 802.11g, Bluetooth.RTM. or similar communication ports and devices.
  • Referring now to FIG. 3, there is shown an exemplary embodiment of the tracker system 80 in accordance with the present invention. The tracker system 80 comprises a system electronic unit (SEU) 82, at least one transmitter and at least one sensor each in communication with the SEU 82.
  • The SEU 82 comprises communication ports that are in communication with at least one sensor 86 and at least one transmitter 84. The communication ports may comprise serial, parallel, universal serial bus, firewire.RTM. or other similar wired communication ports. The SEU 82 additionally includes an analog section and a digital signal processing section controlled by a computer program stored within an erasable programmable memory device. The functionality of the SEU 82 and the interaction between the transmitter and the sensors will be described in greater detail in the methods section of the present invention.
  • As shown in FIG. 3, the SEU 82 is in communication with the transmitter 84 and at least one sensor 86. In one embodiment the sensor 86 is configured to be coupled to the SEU 82 through the use of a cable communication device such as a serial port, parallel port, universal serial bus, firewire port, or similar wired communication ports. The transmitter may be coupled to the SEU through similar communication devices such as those described above with regard to the sensors.
  • The transmitter 84 is configured to emit an electromagnetic signal, wherein the SEU 82 controls the transmission rate of the transmitter 84. Transmitters 84 that may be utilized with the current invention are those shown and described in U.S. Pat. No. 4,742,356, the entirety of which is herein incorporated by reference. The transmitter 84 includes a plurality of radiating antennas for radiating electromagnetic energy. Each of the radiating antennas having independent components for defining a source reference coordinate frame. The transmitted electromagnetic field will generally have a transmission range of about 0 to 30 feet, more preferably 0 to 15 feet and most preferred about 0 to 3 feet. The range of the transmitted magnetic field may be adjusted manually or automatically in accordance with the methods disclosed herein. It is further contemplated that the transmitter 84 may be configured to include more than one transmitting device, wherein the two transmitting devices would be of different types. For example, the transmitter 84 may include a first magnetic transmitting device and a second transmitting device or third transmitting device. The second or third transmitting devices may be configured to transmit acoustical, optical, or electromagnetic signals. As will be described in greater detail in the methods section, the system in accordance with the present invention may be configured to automatically choose between the two transmitters or the system may be manually configured. For example, if the user were to utilize the system in an environment having a large amount of stray magnetic fields that may interfere with the magnetic tracker, the system in accordance with the present invention may automatically switch to one of the other transmitting devices.
  • The transmitter 84 is configured to transmit at least one signal. The transmitted signal may be electromagnetic, optical, acoustical, inertial etc. in a generally defined field of view. For example, the transmitting device may be configured to transmit a signal in a field of view having a spherical radius of between about 0 to 360 degrees, more preferably between about 0 and 270 degrees and most preferred between about 0 and 180 degrees. It shall be understood that although the present invention has been described as including only a single transmitting device shall not be considered limiting in any manner and that it is contemplated that additional transmitting devices may be utilized with the present invention to further expand the field of view of the transmitting device and or to increase accuracy and functionality.
  • Referring now to FIG. 4, there is shown an exemplary embodiment of the hardware components of the present invention in use. As shown in FIG. 4 the computer system 10, display device 75, and the tracker system 80, wherein the tracker system 80 comprises the transmitter 84, SEU 82 and sensors 86. In a preferred embodiment the computer 10 is configured to be embodied in the form of a wearable computer system. Examples of preferred computer systems would be based on the Pocket PC.RTM. platform developed by the Microsoft.RTM. corporation. Although a Pocket PC platform is described as a preferred embodiment this shall not be considered limiting in any manner. It is contemplated that a user may be tethered to a conventional desktop computer system either wired or wirelessly or utilize other computer systems such as an Intel.RTM. or AMD.RTM. powered computing system, PalmPilot.RTM. or other similar computing devices.
  • Referring now to FIG. 4, there is shown one embodiment of an display device 75 in accordance with the present invention. As shown in FIG. 4, the display device 75 may be configured to include the transmitter 84 of the tracker system 80. In one embodiment the transmitter 84 is configured to be retained on the display device 75. The display device 75 in a preferred embodiment is a liquid crystal display (LCD) device, wherein the LCD device is in communication with the computer system 10 through the use of a wired or wireless connection (not shown). In alternative embodiments, the display device 75 may comprise other types and styles of head mounted display devices, such as organic displays, thin film transistor (TFT) displays, light emitting diode (LED) displays. Additionally, it is contemplated that the display may incorporate a multi-layer device, wherein one layer is generally opaque and a second layer is a generally opaque dot-matrix layer, wherein the opaqueness of the first layer may be manually or automatically adjusted thereby allowing the heads-up display to be utilized in many different ambient light situations. While the present invention has been described as utilizing a single display device, it is contemplated that a second display device may be utilized for stereoscopic vision. The second display device may be a conventional display device such as a cathode ray tube (CRT) device, a liquid crystal display (LCD) device, or a video projection device, wherein the transmitter 84 would be mounted onto the display device.
  • It is further contemplated that, a multi-layer liquid crystal display may be utilized in accordance with the present invention, wherein the multiple layers are capable of simulating three-dimensions. By utilizing two display devices a three dimensional workspace may be simulated. Such a system could be utilized to simulate a “real” world experience in the virtual environment and would further provide haptic feedback. As such, the three-dimensional display device would be configured to interact with the software components of the present invention. The present invention would utilize a display device having two LCD screens, wherein a left view model is generated and a right view model is generated by a graphic processing unit (GPU). An example of such a system can be seen in FIG. 5, where there is shown an exemplary embodiment of the three dimensional system in accordance with the present invention. As shown in FIG. 5, a user's right and left eyes would focus on the right and left view model generated in the display device, wherein the user's brain can comprehend the information displayed on the display device and decide the distance and depth of the displayed object. Thus, the user would believe that they are seeing a true three dimensional display of the object displayed on the display device.
  • Referring now to FIG. 6, there is shown an alternative display device in accordance with the present invention. As shown in FIG. 6, the display device may be a conventional display device such as a cathode ray tube (CRT) monitor or a liquid crystal display (LCD) monitor, wherein the tracker would be in association with the monitor frame. For example, the transmitter maybe positioned in the upper corner of the monitor frame, wherein the physical location of the transmitter will be utilized to establish an origin for the coordinate system of the present invention as will be described in greater detail. Further still, it is contemplated that the transmitter may be placed in other locations on the frame of the display device or alternatively in a location near the display device. In addition to conventional display devices as described above, other types of display devices may be utilized. For example, a video projector may be utilized to project an image on a surface, wherein the transmitter will be placed near one corner of the projected image. The projected image may be directed onto a screen, wall or similar vertical surface, or alternatively, the projector may be mounted such that the image is projected onto a horizontal surface such as a conference table. Further still, the display device may be a combination of devices, for example, a video camera and a CRT or LCD or projector, wherein the present system and methods may be utilized for video conferencing.
  • In another embodiment, it is contemplated that the display device may be embodied in the form of a physical display device such as a white board, chalkboard or a similar device, wherein the transmitter would be mounted to one corner of the board.
  • Still further, it is contemplated that any of the display devices described above may be utilized in any combination. For example, a user may utilize a head mounted display device and a projector simultaneously in accordance with the methods described in the present application.
  • As shown in FIGS. 1-6 the present invention further includes at least one sensor device 86 disposed on the user. The sensor device 86 is preferably disposed on the user's hand, and most preferable disposed on the user's fingertip. It is contemplated that at least one sensor 86 may be disposed on each of the user's fingertips. As shown and described in FIGS. 3 and 4, the sensors are in communication with the SEU 80.
  • The sensor(s) 86 are comprised of multiple coils encased in a protective housing, wherein the coils are configured to interact with the electromagnetic field generated by the transmitter 84. Each of the sensors are configured to generate at least one electrical signal, and more preferably between about three and six electrical signals in response to interaction with the signal generated by the tracking device. The electrical signal(s) generated by the sensor 86 are passed through the cable connection to the SEU 82, wherein the signals are amplified and converted into a digital signal. In addition to converting the sensor signals into digital form, the MD-BIOS further assigns a unique sensor id to each sensor. The digital signals are embodied in the form of coordinate information of the sensor, such as, x, y, z, and yaw, pitch and roll information. This coordinate information is then passed from the SEU 82 to the computing system 10.
  • Referring now to FIG. 7, there is shown an alternative embodiment of the tracker system 80 in accordance with the present invention. Wherein in the alternative embodiment it is contemplated that the sensors 86 may be connected to a transducer 81, wherein the transducer is configured to convert the analog signals to digital signals and wirelessly communicates the digital data to the SEU 82. In one embodiment in accordance with the present invention, the transducer 81 is configured to be hard wired to at least five sensors. In this configuration the transducer would assign a unique sensor identification (ID) to each sensor, wherein this unique sensor ID would be transmitted to the SEU 82 along with the coordinate data of each sensor. Although the transducer has been described as being wirelessly coupled to the SEU 82 it is contemplated that it may communicate with the SEU 82 through a wired communication port. It shall be understood that more than one transducer 81 may be utilized with the system in accordance with the present invention. For example, sensors may be disposed on each fingertip of a user's hands, wherein at least two transducers 81 will be utilized, one transducer 81 being disposed on each of the user's hands and in association with the sensors disposed on each of the user's fingertips respectively.
  • Although the SEU 82 has bee described as being an independent component separate from the computing system 10 it is contemplated that the SEU maybe integrally formed with the computing system. For example, the SEU may be configured to be a removable communications card in the form of a PCMCIA card, compact flash card, PCI card or other similar removable devices. Alternatively, SEU may be integrated into the system board of the computing system.
  • Although specific hardware has been described in conjunction with the present invention, it shall be understood that it is contemplated that variances in the described hardware may be undertaken without departing from the scope of the present invention.
  • Methods
  • In accordance with the present invention, methods of use of the present invention will be described in detail below, wherein it shall be understood that these components will be described in a general sense and should not be considered limiting in any manner. In accordance with the present invention, various mechanical components are utilized in conjunction with software and electronic components to define the system and methods in accordance with the present invention.
  • In accordance with the present invention there are provided software components configured to control the various hardware component of the present invention wherein the software and hardware components together form the system of the present invention. Referring now to FIG. 8 there is shown an exemplary functional flow chart illustrating the interaction between the various software components of the present invention. As shown in FIG. 8, the software components in accordance with the present invention comprise a virtual environment manager (VEM) 100 and a multi-dimensional basic input/output system (MD-BIOS) 110, wherein the software components are configured to interact with the hardware components described above. Furthermore, it is contemplated that the software components of the present invention will be embodied in a computer readable media such as a cd-rom, dvd, hard drive, flash memory, programmable read only memory or any other type of computer readable media.
  • As shown in FIG. 8, the VEM 100 receives coordinate data and a time stamp from MD-BIOS 110, wherein the VEM 100 utilizes the coordinate data to simulate virtual device actions and display the virtual device actions on the display device. Additionally, MD-BIOS 110 and VEM 100 are in communication with an operating system. It shall be understood that the VEM 100 can be tailored for each application purpose. For example, if the system in accordance with the present invention is configured to virtually simulate a computer workstation, then the VEM 100 would be configured as such to generate a virtual workstation. Additionally, if a gaming system is to be replicated then the VEM 100 would be a virtual gaming manager, is a conferencing system is to be replicated than the VEM would be configured to be a Virtual Conferencing Manager. It shall be understood that the above examples should not be considered limiting in any manner, in that they have been provided for exemplary purposes only.
  • Referring now to FIG. 9, there is shown an expanded embodiment of MD-BIOS 110 in accordance with the present invention. As shown in FIG. 9, MD-BIOS 110 receives coordinate data from the SEU, wherein MD-BIOS adds time stamp information to the coordinate information. Referring now to FIG. 10, there is shown an exemplary embodiment of the memory structure of MD-BIOS, wherein data received from each sensor is interpreted and placed into appropriate memory location in the memory structure as shown. MD-BIOS is capable of receiving data from a multiple number of sensors, wherein each sensor is assigned a sensor identification tag (id) by the MD-BIOS or transducer depending upon the system's configuration.
  • Applications that utilize MD-BIOS not only can read the sensor id's and their properties, but can also check the sequence number to know the difference between each input to the application. Applications can also interact with MD-BIOS in one of two ways or in a combination of each. For example, it is contemplated that multiple programs running in a memory space of the personal computer may utilize different sets of sensors. For example, in association with the virtual workstation embodiment, a number of sensors associated with the user's fingertips may be utilized to control the virtual keyboard and mouse in a word processing program, while another program may be collecting data from other sensors associated with the present invention. For example, the present invention may be utilized in a laboratory setting, wherein in addition to manually entering data by utilizing the virtual keyboard, a scientist may wish to automatically collect data from an experiment or room conditions. In this embodiment, the scientist would dispose additional sensors to measure to desired properties. For example, one sensor would be utilized to measure room temperature, another for humidity, another to measure ph or another chemical property of an experiment, etc.
  • The first method of program interaction is referred to as synchronized interaction. In the synchronized method, if one or more applications are interested in the same set of sensor id's the applications need to register to MD-BIOS to listen to the set of sensor id's which each application has interest and leave a call-back function address with MD-BIOS. Therefore, whenever data is updated, MD-BIOS interrupts the application to acquire the data, then the application resumes processing the data. This provides synchronous operation of data processing and data acquisition. In the asynchronized method, if one or more unequal frequency applications are interested in the same set of sensor id's, MD-BIOS will filter out the application that requires a higher sampling frequency. MD-BIOS will carry out its best resolution, if the requested frequency is outside the resolution of the tracker, then MD-BIOS will either return an error or will return a reduced frequency rate. In this case, some of the lower frequency applications may need to know the data acquisition gap and therefore utilize the sequence number of the sensor id' to determine the frequency.
  • Additionally as previously discussed and shown in FIG. 5, MD-BIOS may additionally be utilized to create a true three dimensional virtual environment through the use of at least two display devices, wherein MD-BIOS generates a left and right view which are then displayed on a customized display device as previously described in the hardware section above.
  • Further still, it is contemplated that MD-BIOS may be further configured to control the transmitter in combination with the system electronic unit, wherein the two systems could be utilized to adjust the transmission frequency of the transmitter or switch between the various transmitter types. As described herein in accordance with the present invention, the transmitter is configured to emit an electromagnetic signal that the sensors interact with and produce an electrical signal that is converted into the coordinated received by MD-BIOS. MD-BIOS compares the coordinate information against the previously received coordinate information for each sensor, if the deviation of each sensor is greater than a predetermined amount, for example, sensor movement of a quarter of an inch in one computing cycle at 120 Hz would be considered to be excessive. Therefore, MD-BIOS would automatically direct the transmitter to energize the alternative transmitting devices, wherein MD-BIOS would then sample the coordinate generated by all transmitting devices, if one set of coordinate information is outside the parameters as described above then that coordinate information is discarded and the other two sets of coordinate information are compared. If the two remaining coordinate information sets are similar, MD-BIOS chooses one of the transmitting systems and turns the remaining systems off. Therefore, MD-BIOS will automatically switch between the individual transmitting devices if interference is detected. Typically, MD-BIOS will automatically switch without the user's knowledge, though it is contemplated that the user may specify a desired system and adjust the default values to their preferences.
  • In accordance with the present invention the Virtual Environment Manager is utilized to generate a virtual workstation environment. The virtual workstation includes a virtual keyboard at least one virtual input device. As described above MD-BIOS receives coordinate data from the sensors adds a time stamp to the coordinate information and stores the coordinate data in a memory structure as shown in FIG. 10. The virtual environment manager includes a plurality of action thresholds to determine if sensor movement is to be interpreted to be an intended user input or is sensor movement can be attributed to inherent user movement such as natural heartbeat or muscle twitching. Initially, these threshold values are set at default levels, though these default levels may be adjusted by the user to suit the user's own preferences.
  • Referring now to FIG. 11 there is shown threshold values that the VEM utilizes for determining if sensor movement correlates to intended user input. After receiving coordinate data from MD-BIOS, the coordinate data is utilized to determine whether the sensor movement is to be returned as a key press or other intended input or if the sensor movement is due to natural movement. Initially, VEM generates a virtual keyboard having a known origin, wherein the coordinates generated for each of the sensors are compared to the origin coordinates to determine where the user's hands are in relation to the keys of the virtual keyboard. A virtual input device, such as a mouse is also generated wherein a set of origin coordinates are also established for the virtual input device, thereby allowing VEM to determine sensor location in relation to the virtual keyboard and virtual input device.
  • As shown in FIG. 11, the key press threshold is a vector property wherein a value of less than negative one millimeter of a sensor's coordinates will be interpreted as a key press, wherein VEM will compare the sensor's coordinates to the origin coordinates of the virtual keyboard or mouse to determine the proper key press. The key press will then be transmitted to the appropriate program, such as a word processing program, text-editing program, graphics program, etc. A key release will be interpreted as a vector having a value of greater than 1 millimeter and having duration greater than one second. A key hold will be determined by MD-BIOS as a vector having a value equal to one millimeter.
  • Movement of the virtual input device is determined by vector movement of the sensors, if the coordinate values of the sensors are moved less than one millimeter then VEM will not consider this to be a mouse movement, if the coordinate movement is greater than one millimeter this will be interpreted as a mouse move. To determine if the user intends to depress a button on the mouse, VEM utilizes the threshold values which have been pre-established for key presses on the virtual keyboard.
  • In accordance with the present invention a series of functional flow diagrams will be described herein, wherein the functional flow diagrams are utilized to illustrate the interaction between the various components of the present invention.
  • Methods
  • In accordance with the present invention there is provided methods for a virtual computing environment. Wherein a virtual environment manager will be embodied in the form of a virtual workstation manager. That is the present invention will be utilized to replicate a personal computing system and the physical input devices in a virtual environment. As will be described in detail below, the virtual workstation manager will generate virtual input devices such as a keyboard, a mouse or other input devices.
  • Referring now to FIG. 12, there is shown an exemplary embodiment of a method in accordance with the present invention, wherein the VEM and MD-BIOS are embodied in a computer readable medium. The method according to the present invention comprises the steps of: initializing a computer system, including initializing the SEU, loading an operating system into a memory space, loading VEM and MD-BIOS into a memory space. As soon as MD-BIOS is loaded into a memory space, MD-BIOS begins to receive coordinate information from the SEU; VEM then begins to scan for sensor movement to determine if any sensor movement passes the threshold values. A display device is initialized, wherein VEM (such as VWM) then displays virtual hands, virtual keyboard and at least one virtual input device on the display device. MD-BIOS each of these processes will be described in greater detail with reference to detailed functional flow diagrams.
  • The methods according to the present invention may utilize hardware devices such as those described above or hardware devices similar to those shown and described above, wherein the present invention utilizes software or firmware to control the hardware in a manner to replicate a virtual environment. In accordance with the present invention comprises at least one software program or firmware code previously described herein wherein the software includes the VEM and MD-BIOS programs
  • Referring now to FIG. 12 there is shown a functional flow diagram illustrating exemplary steps of the method in accordance with the present invention. At Box 190 a computing system is powered on, this includes powering on the SEU additionally if the SEU is embodied as an individual component separate from the computer system. As Box 200 an operating system is loaded into a memory space. At Box 210 MD-BIOS and the Virtual Environment Manager are loaded into a memory space within the computer system. After loading MD-BIOS and the virtual workstation manager into a memory space, MD-BIOS immediately begins receive coordinate information from the SEU at Box 215. As described in detail above the electrical signals generated by the sensors are converted to coordinate information and a time stamp is added by MD-BIOS and the data is stored in a memory location. VEM then compares the coordinates of each sensor to determine if motion of the sensor(s) has occurred and whether the sensor motion is intended to be a key press on the virtual keyboard or virtual mouse. The transmission and scanning rate of the tracker is controlled by MD-BIOS. Wherein the scanning and transmitting frequency of the tracker is controlled in response to the physical location of the sensor(s) in relation to the tracker and the origin coordinates of the virtual devices. For example, as the sensors are moved closer to the virtual devices the scanning rate of the tracker will be increased, thereby increasing the accuracy and resolution of the system. When the sensors move away from the virtual devices, the scanning rate is reduced, thereby lowering the power consumption of the devices. In addition to controlling the scanning rate, VEM also may be configured to display a visual indicator of the sensor(s) position on the display device as well as provide auditory feedback to the user. For example, as the sensor(s) are moved to a location above the virtual keyboard, the sensor may change colors to indicate how close the sensor is to the keyboard, when the sensor moves toward the keyboard, VEM changes the color of the sensor to indicate a key press and provides an auditory response to the user to denote a key press.
  • In an alternative embodiment the virtual keyboard may be configured to respond to absolute movement of each of the user's fingers (sensors). In this embodiment, the threshold values are set to zero or near zero, wherein any movement of the user's finger will correlate to a typing motion or mouse motion depending upon where the user's fingers are located in relation to the virtual devices.
  • At Box 230 the system prompts the user to enter the user's login and password. At Diamond 235 it is determined If the user already has a password and login then Box 237, where the user's saved settings are loaded into a memory space. A user's settings may control what is displayed on the virtual display device such as input devices (keyboard, mouse, tablet), the location of these input devices in the virtual setting and any other user definable preferences such as visual or audio feedback, tactile feedback and sensitivity of sensor movement, or other preferences. For example, user A may have saved settings from a previous use, wherein once logged into the system, VEM (such as VWM) displays a virtual keyboard, a virtual monitor and a virtual mouse, wherein the virtual workstation manager controls the origin coordinates of where each of these devices will be displayed in the virtual environment. If it is determined in Diamond 235 that a new user is logging in then at Box 239 default values for the virtual devices displayed on the display device are loaded into memory. At diamond 242, the system prompts the user whether or not they want to customize the default values. If the user chooses to customize the default values then Box 243. At Box 243 the user's customized settings are saved under their login profile and then to Box 240. If the user does not choose to customize the default settings then Box 240.
  • At Box 240 the display device is initialized and at Box 250 the virtual devices are displayed in the virtual environment at the location dictated by either the user's settings or the loaded default values.
  • If the user is a previous user, the virtual environment manager will be called wherein the user's preferred settings will be retrieved from a memory location and loaded into a memory space. If the user is new to the system, the Virtual Environment Manager will be continue to use system default values into memory. These default values may be changed by the user and saved under the user's personal profile before exiting the system, wherein these settings will be loaded when the user utilizes the system in the future.
  • Examples of user definable settings are the type and style of keyboard that will be displayed in the virtual environment. For example, the user may prefer to utilize a standard size keyboard having a standard QWERTY layout, or the user may prefer to utilize an ergonomic keyboard. Additional settings would be the physical location of the keyboard in the virtual environment. Such that when the system is initialized the keyboard will be shown in the virtual environment wherein the coordinates of the keyboard are known. In addition to displaying a keyboard as a user setting, additional settings may control the display of other virtual input devices such as a mouse, a virtual monitor or similar devices. Each of the virtual devices displayed within the virtual environment will have origin coordinates known to the virtual environment manager. The user in the virtual environment may control the style, size and location on the virtual input devices. For example, the keyboard may include handles, wherein the user can grab the handles in the virtual environment and move, stretch, pull-apart the keyboard. The other virtual input devices may also include handles to allow the devices to be re-positioned re-sized or stylized within the virtual environment. When the computer system is powered down, the virtual environment manager can saves the user's settings, wherein the virtual environment may be loaded each time as left by the user in each previous use
  • Once the user has logged into the computer system, the operating system and virtual environment manager and MD-BIOS finishes loading into memory with the user's settings. After loading the operating system and MD-BIOS into memory, the display device is initialize, Box 240, wherein a graphical user interface (GUI) is displayed on the display device Box 250. For example, if the operating system is based on Microsoft Windows.RTM. then the standard GUI interface will be displayed in the display device. In addition to displaying the operating system's GUI, virtual input devices will also be display on the display device. As described above, the placement, size, shape and orientation of the input devices as displayed within the virtual environment is dictated by either the users saved preferences or in the case of a new user the default settings of the Virtual Environment Manager. In addition to displaying virtual input devices on the display device, a virtual set of hands may also be displayed if the user is wearing sensors on each of their fingertips.
  • Referring now to Box 260, the transmission rate of the signal is controlled by MD-BIOS and the virtual workstation manager in conjunction with the coordinate location of the sensors in relation to the tracker. The transmission rate of the tracker may be controlled according to sensor position. For example, if it is determined that the sensor is disposed near the known location of an input device, then the tracker's scan rate will be increased to increase accuracy of the system, if the sensor is disposed at a given distance from any virtual input device than the transmission rate of the tracker will be reduced, thereby saving power and reducing processing cycles. Further still, as described above, if coordinates received from the SEU are outside the operating parameters of the system, MD-BIOS will direct the transmitter to energize the alternative transmission sources, wherein the coordinates generated from these alternative transmission sources will be compared and the strongest or more accurate coordinate information will be utilized. Wherein MD-BIOS will direct the transmitter to turn off the alternative transmitting devices. In a preferred embodiment this process will occur automatically without any user interaction.
  • Referring now to Box 270, each of the sensors are constantly sampled by the SEU and coordinate data is sent to MD-BIOS where a timestamp is added and the information is placed into a memory location.
  • At Diamond 280, the virtual environment manager and determines if the velocity component(s) and the vector component(s) of the detected sensor motion pass threshold values. Threshold values are utilized to filter out undesired input as well as increase system accuracy. For example there are natural vibrations associated with humans such as heartbeat, slight hand motion or other natural vibrations. If it is determined that the vector components are greater than the threshold values, then it is determined if the coordinates are in the vicinity of the known coordinates of the virtual input devices generated by the virtual environment manager. For example, if the coordinates received correlate to a keyboard position for a letter, the velocity component and the previous coordinates are compared to determine the intentions of the user. By comparing the previous coordinates and the present coordinates in addition to the velocity component it can be determine if the motion is intended to replicate a keystroke, key release, key hold or key repeating. If the coordinates correspond to the location of the mouse, then the vector and velocity components will be utilized to determine if the user intended a mouse move, a mouse button depress/release, or a single or double click.
  • The movement of the sensors will be displayed on the display device in the form of a virtual set of hands or in the case of mouse motion, movement of a pointer within the virtual environment. In addition to displaying motion within the virtual environment, it may also be desirable to provide feedback in the form of audio, tactile, or visual feedback of interaction between the user and the virtual input devices. For example, if a key is determined to be depressed on a keyboard, then the virtual keyboard will be displayed wherein the chosen key is illustrated as being depressed. In addition to the visual feedback, audio or tactile feedback maybe provided in the form of “clicking” sounds intended to replicate the physical sound of a standard keyboard.
  • At Box 300 and 305 the coordinate information generated from sensor motion is compared to the known coordinate information of the virtual input devices. For example, if a virtual keyboard is displayed according to the user's defined settings, coordinates of the keyboard are a known value that are stored within the shared memory manipulated through MD-BIOS API calls. Thus, when coordinates are calculated from sensor motion, the coordinates transmitted from the memory location of MD-BIOS are compared to the known coordinates of the keyboard. If the coordinates correspond to or are within a pre-defined range of an individual key disposed on the keyboard then the virtual environment manager will determine the user's intention as indicated in Box 330. The virtual environment manager determines the user's intention by comparing the velocity component generated by sensor movement and comparing the velocity component to a known value. For example, if the vector component is toward the plane of the keyboard, then the virtual workstation manager will interpret this to indicate a key press on the virtual keyboard. If the vector component is away from the plane of the keyboard it may also indicate that a key release was performed and the user's finger is moving away from the key. If the velocity component is zero or nearly zero this may be interpreted to indicate a key hold, such as a user holding down a shift key or the like. A vector of zero may also indicate a key repeating such as when a typed word includes two or more of the same characters. After the user's intention is determined then at Box 305, the intended information is sent to the operating system for entry into the event queue or windows manager. In accordance with the present invention, the key presses may be determined utilizing pure vector and velocity without consideration of the z-axis component. This would allow a user to utilize the present system while resting their hands on a surface, such as a tabletop or upon their person or the like.
  • At Box 300, the user's intended motion may be displayed on the virtual display device in the form of a virtual key-press on the virtual keyboard, motion of a virtual hand or pair of virtual hands, or an auditory or tactile response may be generated in response to the sensor's motion.
  • At Box, the system returns to Box 270, wherein the system returns to scanning for sensor motion and the process is repeated until the system is powered off.
  • It shall be understood that although the sensor motion component has been described herein in reference to a key press on a keyboard it shall be understood that this should not be considered limiting in any manner. In that the coordinates may correspond to any type of virtual input device, wherein each virtual input device has parameters which are utilized by the virtual environment manager to interpret the sensor motion. For example, if the coordinates received correlate to coordinated for a mouse, it will be determined if the coordinates define movement of the mouse, which will be displayed within the virtual environment. A mouse button press, mouse button release, a single button press or a double button press.
  • It is further contemplated that the virtual environment manager may be capable of distinguishing between different sets of sensors by each sensor(s) or transducer(s) unique ID. For example, if two users both utilizing a system in accordance with the present invention are near one another each tracker will only track those sensors to which the system is initialized. Thus, multiple systems can be utilized within close proximity to one another.
  • As shown in FIG. 12, the functional flow diagram illustrates a graphic based system, wherein the virtual reality system in accordance with the present invention is configured to display three-dimensional virtual input devices in a virtual environment. It is further contemplated that the present invention may be utilized in a text-based system. In a text based system the functional flow diagram as shown in FIG. 12 would not include Box 200 wherein the operating system is loaded into a memory space. In the text-based system the virtual keyboard generated by the virtual environment manager would be displayed one dimensionally and the virtual input device in a preferred embodiment would be a touch pad or a similar tracking device.
  • It is further contemplated that a portion of the software components of the present invention may be embodied in the form of a BIOS chip, wherein the BIOS chip would be installed on the computer system's mother board, such that when the computer system is powered on the software in accordance with the present invention would be loaded into a memory space from the BIOS.
  • Referring now to FIGS. 13 and 14 there are shown exemplary embodiments of various virtual keyboards as displayed in the virtual environment. As shown in FIG. 13, the virtual keyboard 300 may be disposed over the top of the virtual mouse 310 thereby allowing a user to immediately toggle between the keyboard and the mouse without having to physically move the sensors a great amount. This type of setup reduces motion associated with switching back and forth between the keyboard and the mouse, thereby potentially reducing repetitive stress injuries. Further still by providing such a layout, users of programs that require a user to switch back and forth between a pointing device and text input may be able to increase their productivity because less time is spent switching between the two devices. Referring now to FIG. 13, there is shown the virtual keyboard 300 and virtual mouse 310, wherein the virtual mouse is disposed over the virtual keyboard. As shown in FIGS. 13 and 14 the virtual mouse can be transposed between the two positions shown either through specific hand motions wherein the sensors and VEM generate a specific signal to transpose the mouse from one position to another. Alternatively, the mouse may be transposed utilizing a voice command, a hot key associated with the virtual keyboard or a hot spot located on within the virtual display. Additionally, although a conventional computer mouse is shown being displayed in the virtual environment, it is contemplated that any type of input device may be displayed, for example the input device may be a touch pad, trackball, tablet and stylus or similar input devices.
  • Referring now to FIGS. 15 and 16 there is shown yet another application of the system in accordance with the present invention, wherein the system of the present invention is configured as a virtual car control system (VCCS) 400. In the present embodiment, the VCCS comprises software and hardware components, wherein the software and hardware components interact to provide a virtual system that enables a user to control various systems within an automotive environment without physically removing their hands from the steering wheel and shifting their eyesight from the road.
  • The VCCS system 400 includes hardware and software elements to replicate and control mechanical and electrical controls within an automotive environment. The VCCS includes a computer unit 410, a tracker system 450 including a system electronic unit 420, at least one sensor 425 and at least one transmitter 430, a control button 460 and a heads up display device 470 or alternatively a see through HMD. The VCCS further includes software components 480 stored on a computer readable medium disposed within the computer unit 410. The software components include an operating system 482, a virtual environment manager program 484 and a multi-dimensional basic input/output subsystem (MD-BIOS) 486 program.
  • Referring now to FIG. 16 there is shown an exemplary embodiment of the VCCS as seen from a user's perspective in accordance with the present invention. As shown in FIG. 16, an image showing three-dimensional buttons 500 would be projected onto the windshield 505 by a heads up display device 470, when the buttons are displayed on the windshield coordinate information is associated with each of the buttons. The coordinate information of each button locates each button in space in a plane spaced apart from the steering wheel, this can be seen by the three-dimensional buttons disposed on box 510 which are shown for exemplary purposes only and will not be actually seen by the user. By placing the origin coordinates for each button in a plane aligned with the physical steering wheel of the vehicle, the user may interact with the buttons without removing their hand's from the steering wheel. In addition to the three-dimensional buttons displayed on the windshield, the VCCS may further include at least one physical button. In a preferred embodiment the physical button(s) would be disposed on the steering wheel or steering column. The physical buttons may be utilized to turn the heads up display on and off, menu return, enter, clear, dial, send or any similar command which may be utilized to control any aspect of the VCCS system.
  • In use, at least one sensor would be disposed on a user's person, in a preferred embodiment at least one sensor would be disposed on each of the user's thumbs, but it is contemplated that additional sensors may be disposed on the user's additional fingers. The sensor(s) interact with a signal generated from a transmitter, wherein the sensors produce a position signal due to interaction with the transmitted signal. The position signal is converted into coordinate information by the SEU after being transmitted to the SEU. The coordinate information is transmitted to the MD-BIOS software where time stamp data is added and the coordinate information is stored in a memory location. It is further contemplated that a sensor identification tag may also be associated with the sensor, added by the SEU or added by MD-BIOS to the coordinate data. The coordinate data is then utilized by the VEM to determine if motion of a sensor passes threshold values, and if so, what is the user's intent, the coordinate data is then compared to the origin coordinate information for each of the three-dimensional buttons virtually located in a plane adjacent to the steering wheel. A virtual hand may also be displayed on the windshield to illustrate the motion of the sensor or a cursor or other visual marker may be displayed on the windshield so that the user may visually correlate their hand location on the steering wheel with the virtual three-dimensional buttons displayed on the windshield.
  • As described above the VCCS system in accordance with the present invention provides a virtual interface allowing a user to control various components of an automobile. Each of the buttons would be displayed on the windshield as having three-dimensional characteristics. Wherein multiple layers of buttons may be utilized to control various systems. For example, as one button is pressed, additional buttons may be displayed on the windshield, wherein the additional buttons are displayed as being tiled over the previous layer of buttons. This may be better understood as shown in FIGS. 17-19 wherein there is shown an exemplary embodiment of the VCCS system in use in accordance with the present invention.
  • Referring now to FIG. 17 there is shown a series of three-dimensional buttons as they would be displayed by the heads-up display device and projected onto an automobiles windshield. As previously described, the display device or the VCCS system may be controlled by a physical button wherein, after the button has been depressed the buttons would be displayed on the windshield. The buttons displayed would have origin coordinates that would place the buttons in plane space adjacent to the user's left hand on the steering wheel wherein the first level of buttons displayed would be high-level system buttons. For example, one button will be displayed to access the climate control system, another for the GPS system, and another for an entertainment system and so on. It shall be understood that the list above should be considered exemplary and should not be considered limiting in any manner. If the user desires to access the entertainment system, they would move their thumb to a physical location which would correlate to the virtual entertainment system button, wherein motion of their thumb over the virtual entertainment button would be deemed to be a button press by the software components of the present invention. As a result, a second set of virtual buttons would be displayed by the heads-up display device and projected onto the windshield as shown in FIG. 18. The additional buttons displayed would have origin coordinates that would place the buttons in plane space adjacent to the user's right hand on the steering wheel. As shown, the additional buttons would provide the user with audio/video components as installed, such as AM/FM radio, cassette, compact disc, DVD audio and the like. As shown in FIG. 19, the user has selected the CD button, wherein the visual display of the CD button is changed, for example the button is made larger thereby appearing to be in a different plane than the other buttons and the color of the button may change to indicate that the button has been selected. Additionally, it is contemplated that an audible feedback may also be associated with button selection in addition to the visual feedback described above.
  • As shown in FIG. 19, after selecting the CD button, additional buttons will be displayed adjacent to the selected CD button wherein the additional buttons include controls associated with the CD player. For example, fast forward, skip, rewind, disc selection, mute, volume and menu return.
  • Referring now to FIG. 20 there is shown exemplary embodiments of additional buttons that may be displayed in accordance with the VCCS system of the present invention. As shown in FIG. 20, additional buttons may be displayed to control a navigation system such as a GPS system, Climate control system, communication system such as a cellular phone, as well as automotive control systems such as traction control, system information and the like. As shown in FIG. 20, each of the top level buttons have been selected thereby causing the second menu to be displayed on the screen. It shall be understood that the menus shown in FIGS. 16-20 should be considered exemplary and not limiting in any manner in that the menus may be altered without deviating from the scope of the invention.
  • Although the present application has been described in detail with reference to an entertainment system this shall not be considered limiting in any manner. Further still, it is contemplated that the VCCS system may be utilized in combination with mechanical switches or physical control means. For example, the entertainment system may be controlled with the UCC system by the driver and also may be controlled by the driver or a passenger with conventional means installed in the automobile. As described previously, the heads-up display may be controlled by a mechanical switch, a virtual switch or the display may be time controlled, wherein once activated, if no input is received from the user, the display will turn off, wherein the user can then re-activate the system. It is further contemplated that the system may have additional restrictive controls such as a maximum speed control, wherein the system may not be accessed if the vehicle is traveling beyond a pre-set speed, or if the steering wheel has been rotated past a pre-set degree. Additionally, the VCCS system may utilize voice command software in addition to those hardware and software components described above.
  • Although the coordinate information of the present invention has been described herein as being restricted to the x, y, and z-axis, the sensors are further capable of generating yaw, pitch and roll coordinate information in combination with the hardware and software components of the present invention. The additional coordinate information may be utilized by applications not mentioned specifically herein. For example, if the virtual environment manager of the present invention was tailored to represent a virtual flight training simulator, the yaw, pitch and roll coordinates would be utilized to simulate movement of a virtual plane.
  • Although the present invention has been described with reference to a virtual workstation such as a personal computer or laptop computer it is contemplated that the system and methods in accordance with the present invention maybe utilized to replace other types of physical systems. Such as Virtual Game station or VR conference. For example, the present invention may be utilized to replace a gaming system such as an X-Box.RTM. or a Playstation.RTM.. It is further contemplated that the present invention may be utilized in combination with others, wherein the multiple systems may be utilized for communication between each of the users. Systems such as this may be utilized by military forces for covert communications, pilots, motorsport race teams and other similar careers that may require communication amongst more than one person. Another contemplated use for the present invention is warehouse management and inventory control, wherein the user may roam freely around the warehouse entering inventory, fulfilling orders, or maintaining quality control. Although the present invention has been described with regards to specific examples these examples should not be considered limiting in any manner in that the methods of the present invention may be applied to a wide variety of technologies, many of which were not disclosed here.
  • The instant invention is shown and described herein in what is considered to be the most practical, and preferred embodiments. It is recognized, however, that departures may be made there from, which are within the scope of the invention, and that obvious modifications will occur to one skilled in the art upon reading this disclosure.

Claims (21)

1. (canceled)
2. A method for controlling a microprocessor-based system, the method comprising:
receiving data generated from movement that interacts with a substantially horizontal surface;
calculating coordinates of the movement relative to the surface;
using the coordinates to determine an input;
controlling the microprocessor-based system using the input; and
displaying at least one output on a display based on the controlling of the microprocessor.
3. The method of claim 2, wherein receiving data generated from movement that interacts with the surface comprises determining at least one movement that physically contacts the surface.
4. The method of claim 2, wherein receiving data generated from movement that interacts with the surface comprises only receiving data generated from movement that physically contacts the surface.
5. The method of claim 2, wherein receiving data generated from movement that interacts with the surface comprises simultaneously receiving a plurality of data generated from movements that interact with the surface.
6. The method of claim 2, wherein the surface comprises the display.
7. The method of claim 2, further comprising transmitting at least one signal from at least one transmitter, wherein receiving data generated from movement that interacts with the surface comprises receiving data generated from movement that interacts with the at least one signal.
8. A method for controlling a microprocessor-based system, the method comprising:
transmitting at least one signal from at least one transmitter;
simultaneously receiving a plurality of data generated from movements that interact with the at least one signal;
calculating coordinates of the movement relative to a display;
using the coordinates to determine an input;
controlling the microprocessor-based system using the input; and
displaying at least one output on the display based on the controlling of the microprocessor.
9. The method of claim 8, wherein transmitting the at least one signal comprises transmitting at least one of an optical signal, an electromagnetic signal, a magnetic signal and an acoustical signal.
10. The method of claim 8, wherein simultaneously receiving a plurality of data generated from movements that interact with the at least one signal comprises receiving a plurality of data generated from movements that interact with a substantially horizontal surface.
11. The method of claim 10, wherein the surface comprises the display.
12. The method of claim 10, wherein simultaneously receiving a plurality of data generated from movements that interact with the at least one signal comprises only receiving data generated from movements that physically contact the surface.
13. The method of claim 12, wherein the surface comprises the display.
14. The method of claim 10, wherein transmitting the at least one signal comprises transmitting within a range of about zero to three feet relative to the surface.
15. The method of claim 10, wherein transmitting the at least one signal comprises transmitting within a field of view of about zero to 180 degrees relative to the surface.
16. A microprocessor-based system, comprising:
an output device configured to transmit at least one signal;
an input device configured to simultaneously generate a plurality of data in response to movements that interact with the at least one signal;
a display device, at least one of the output device and the input device being associated with the display device such that the interaction with the at least one signal is relative to the display device; and
a processor in communication with at least the input device and the display device, the processor being configured to be controlled by the data generated by the input device and to cause the display device to display at least one output based on the control of the microprocessor.
17. The system of claim 16, wherein the input device is configured to simultaneously generate a plurality of data in response to movements that interact with the at least one signal and with a substantially horizontal surface.
18. The system of claim 17, wherein the surface comprises the display.
19. A microprocessor-based system, comprising:
an input device configured to generate data in response to movement that interacts with a substantially horizontal surface;
a display device associated with the input device;
a processor in communication with at least the input device and the display device, the processor being configured to be controlled by the data generated by the input device and to cause the display device to display at least one output based on the control of the microprocessor.
20. The system of claim 19, wherein the input device configured to simultaneously generate a plurality of data in response to movements that interact with the surface.
21. The system of claim 19, wherein the surface comprises the display.
US11/944,226 2002-11-06 2007-11-21 Virtual workstation Abandoned US20080150899A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US11/944,226 US20080150899A1 (en) 2002-11-06 2007-11-21 Virtual workstation

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US42455702P 2002-11-06 2002-11-06
US10/703,184 US7337410B2 (en) 2002-11-06 2003-11-05 Virtual workstation
US11/944,226 US20080150899A1 (en) 2002-11-06 2007-11-21 Virtual workstation

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US10/703,184 Continuation US7337410B2 (en) 2002-11-06 2003-11-05 Virtual workstation

Publications (1)

Publication Number Publication Date
US20080150899A1 true US20080150899A1 (en) 2008-06-26

Family

ID=32312833

Family Applications (2)

Application Number Title Priority Date Filing Date
US10/703,184 Expired - Fee Related US7337410B2 (en) 2002-11-06 2003-11-05 Virtual workstation
US11/944,226 Abandoned US20080150899A1 (en) 2002-11-06 2007-11-21 Virtual workstation

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US10/703,184 Expired - Fee Related US7337410B2 (en) 2002-11-06 2003-11-05 Virtual workstation

Country Status (3)

Country Link
US (2) US7337410B2 (en)
AU (1) AU2003291304A1 (en)
WO (1) WO2004044664A1 (en)

Cited By (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050212763A1 (en) * 2004-03-26 2005-09-29 Cannon Kabushiki Kaisha Information processing apparatus and method
US20050264527A1 (en) * 2002-11-06 2005-12-01 Lin Julius J Audio-visual three-dimensional input/output
US20060238439A1 (en) * 2005-04-22 2006-10-26 Microsoft Corporation State-based auxiliary display operation
US20090167508A1 (en) * 2007-12-31 2009-07-02 Apple Inc. Tactile feedback in an electronic device
US20100145518A1 (en) * 2008-12-10 2010-06-10 Samsung Electronics Co., Ltd. Robot and method thereof
US20100171694A1 (en) * 2009-01-06 2010-07-08 Chih-Hung Lu Electronic Apparatus with Virtual Data Input Device
US20100265183A1 (en) * 2009-04-20 2010-10-21 Microsoft Corporation State changes for an adaptive device
US20100265182A1 (en) * 2009-04-20 2010-10-21 Microsoft Corporation Context-based state change for an adaptive input device
US20110087493A1 (en) * 2008-06-16 2011-04-14 Stefan Sellschopp Communication System and Method for Representing Information in a Communication
US20110227947A1 (en) * 2010-03-16 2011-09-22 Microsoft Corporation Multi-Touch User Interface Interaction
US20120019373A1 (en) * 2007-10-12 2012-01-26 Immersion Corporation Method and Apparatus for Wearable Remote Interface Device
US20120025945A1 (en) * 2010-07-27 2012-02-02 Cyberglove Systems, Llc Motion capture data glove
US20120050154A1 (en) * 2010-08-31 2012-03-01 Adil Jagmag Method and system for providing 3d user interface in 3d televisions
US20120165099A1 (en) * 2010-12-22 2012-06-28 Nintendo Co., Ltd. Game system, game device, storage medium storing game program, and game process method
US20120169598A1 (en) * 2011-01-05 2012-07-05 Tovi Grossman Multi-Touch Integrated Desktop Environment
US20130257751A1 (en) * 2011-04-19 2013-10-03 Sony Computer Entertainment Inc. Detection of interaction with virtual object from finger color change
US20140117358A1 (en) * 2009-02-09 2014-05-01 Mitsubishi Electric Corporation Electro-optical device, shift register circuit, and semiconductor device
WO2014147686A1 (en) * 2013-03-21 2014-09-25 Sony Corporation Head-mounted device for user interactions in an amplified reality environment
WO2014106823A3 (en) * 2013-01-03 2014-12-11 Meta Company Extramissive spatial imaging digital eye glass for virtual or augmediated vision
US9612743B2 (en) 2011-01-05 2017-04-04 Autodesk, Inc. Multi-touch integrated desktop environment
US9751406B2 (en) 2014-04-03 2017-09-05 Audi Ag Motor vehicle and method for controlling a climate control system in a motor vehicle
US10782782B1 (en) * 2019-02-07 2020-09-22 Facebook Technologies, Llc Devices, systems, and methods for radar-based artificial reality tracking
US11010972B2 (en) 2015-12-11 2021-05-18 Google Llc Context sensitive user interface activation in an augmented and/or virtual reality environment
US11030459B2 (en) * 2019-06-27 2021-06-08 Intel Corporation Methods and apparatus for projecting augmented reality enhancements to real objects in response to user gestures detected in a real environment

Families Citing this family (59)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004127073A (en) * 2002-10-04 2004-04-22 Smk Corp Instruction input device
AU2003291304A1 (en) * 2002-11-06 2004-06-03 Julius Lin Virtual workstation
JP3728304B2 (en) * 2003-07-10 2005-12-21 キヤノン株式会社 Information processing method, information processing apparatus, program, and storage medium
US20050289453A1 (en) * 2004-06-21 2005-12-29 Tsakhi Segal Apparatys and method for off-line synchronized capturing and reviewing notes and presentations
JP4550059B2 (en) * 2004-07-29 2010-09-22 三菱電機株式会社 Wireless tag reading system, wireless tag reader, and wireless tag
JP4351599B2 (en) * 2004-09-03 2009-10-28 パナソニック株式会社 Input device
KR100663515B1 (en) * 2004-11-08 2007-01-02 삼성전자주식회사 A portable terminal apparatus and method for inputting data for the portable terminal apparatus
US20060119575A1 (en) * 2004-12-06 2006-06-08 Naturalpoint, Inc. Systems and methods for using a movable object to control a computer
US20060129935A1 (en) * 2004-12-15 2006-06-15 Honeywell International, Inc. Integrated information management system and method
US20070055949A1 (en) * 2005-01-29 2007-03-08 Nicholas Thomas Methods and apparatus for rfid interface control
US7930409B2 (en) * 2005-02-23 2011-04-19 Aol Inc. Configuring output on a communication device
DE102005017313A1 (en) * 2005-04-14 2006-10-19 Volkswagen Ag Method for displaying information in a means of transport and instrument cluster for a motor vehicle
WO2007030947A1 (en) * 2005-09-16 2007-03-22 Anthony Szturm Mapping motion sensors to standard input devices
JP4747792B2 (en) * 2005-11-10 2011-08-17 ソニー株式会社 Information processing apparatus and method for initializing setting items thereof
JPWO2007072675A1 (en) * 2005-12-20 2009-05-28 パナソニック株式会社 Content presentation apparatus and content presentation method
US9354715B2 (en) * 2006-04-12 2016-05-31 Sony Interactive Entertainment Inc. Dynamic arrangement of characters in an on-screen keyboard
AT504447B1 (en) * 2006-08-25 2008-11-15 Omv Refining & Marketing Gmbh METHOD FOR PRESENTING OPERATING STATES OF A CHEMICAL AND / OR PHYSICAL PLANT
US8418201B2 (en) * 2006-12-14 2013-04-09 Koninklijke Philips Electronics, N.V. System and method for reproducing and displaying information
DE08713883T1 (en) * 2007-01-22 2010-04-15 Bell Helicopter Textron, Inc., Fort Worth SYSTEM AND METHOD FOR CONTROLLING A VIRTUAL REALITY BY A REPRESENTATIVE IN VIRTUAL REALITY
US20080248871A1 (en) * 2007-03-15 2008-10-09 Szturm Anthony Interface device
US8358277B2 (en) 2008-03-18 2013-01-22 Microsoft Corporation Virtual keyboard based activation and dismissal
US20100103103A1 (en) * 2008-08-22 2010-04-29 Palanker Daniel V Method And Device for Input Of Information Using Visible Touch Sensors
US9207806B2 (en) * 2009-05-28 2015-12-08 Microsoft Technology Licensing, Llc Creating a virtual mouse input device
US9141284B2 (en) * 2009-05-28 2015-09-22 Microsoft Technology Licensing, Llc Virtual input devices created by touch input
AU2010310891A1 (en) * 2009-10-22 2012-04-19 Joshua Michael Young Human machine interface device
CN102062946A (en) * 2010-11-05 2011-05-18 鸿富锦精密工业(深圳)有限公司 Spectacles-type display device with communication function
ES2693060T3 (en) * 2011-03-17 2018-12-07 Ssi Schäfer Automation Gmbh Control and supervision of a warehouse installation and order preparation through movement and voice
CN102693066B (en) * 2011-03-25 2015-05-27 国基电子(上海)有限公司 Touch electronic device and virtual keyboard operation method thereof
JP5858641B2 (en) * 2011-05-10 2016-02-10 キヤノン株式会社 Information processing apparatus, system including information processing apparatus and external apparatus, system control method, and program
US8203502B1 (en) * 2011-05-25 2012-06-19 Google Inc. Wearable heads-up display with integrated finger-tracking input sensor
US8964008B2 (en) * 2011-06-17 2015-02-24 Microsoft Technology Licensing, Llc Volumetric video presentation
US9024843B2 (en) 2011-06-30 2015-05-05 Google Inc. Wearable computer with curved display and navigation tool
US20130002724A1 (en) * 2011-06-30 2013-01-03 Google Inc. Wearable computer with curved display and navigation tool
US9924907B2 (en) 2011-09-30 2018-03-27 Google Technology Holdings LLC Method and system for identifying location of a touched body part
KR101978740B1 (en) * 2012-02-15 2019-05-15 삼성전자주식회사 Tele-operation system and control method thereof
SE537730C2 (en) * 2012-05-14 2015-10-06 Scania Cv Ab Projected virtual vehicle entry system
US20140085177A1 (en) * 2012-09-21 2014-03-27 Nokia Corporation Method and apparatus for responding to input based upon relative finger position
KR20140055173A (en) 2012-10-30 2014-05-09 삼성전자주식회사 Input apparatus and input controlling method thereof
KR102219464B1 (en) * 2014-05-23 2021-02-25 삼성전자주식회사 Operating method and Electronic device for security
US10106172B2 (en) * 2014-08-18 2018-10-23 Ford Global Technologies, Llc Shared vehicle system
US10168833B2 (en) * 2014-09-03 2019-01-01 Hewlett-Packard Development Company, L.P. Presentation of a digital image of an object
US9733790B2 (en) * 2014-10-30 2017-08-15 Disney Enterprises, Inc. Haptic interface for population of a three-dimensional virtual environment
US9898865B2 (en) * 2015-06-22 2018-02-20 Microsoft Technology Licensing, Llc System and method for spawning drawing surfaces
US11125566B2 (en) * 2015-07-16 2021-09-21 Ford Global Technologies, Llc Method and apparatus for determining a vehicle ego-position
US10037085B2 (en) * 2015-12-21 2018-07-31 Intel Corporation Techniques for real object and hand representation in virtual reality content
CN105975089B (en) * 2016-06-17 2018-12-18 许艾云 A kind of virtual reality electromagnetism body-sensing Scene realization method and device
EP3260950B1 (en) * 2016-06-22 2019-11-06 Nokia Technologies Oy Mediated reality
US10225655B1 (en) 2016-07-29 2019-03-05 Relay Cars LLC Stereo user interface elements placed in 3D space for virtual reality applications in head mounted displays
CN106980362A (en) 2016-10-09 2017-07-25 阿里巴巴集团控股有限公司 Input method and device based on virtual reality scenario
US10824293B2 (en) * 2017-05-08 2020-11-03 International Business Machines Corporation Finger direction based holographic object interaction from a distance
US10866652B2 (en) 2017-11-13 2020-12-15 Samsung Electronics Co., Ltd. System and method for distributed device tracking
US11360558B2 (en) 2018-07-17 2022-06-14 Apple Inc. Computer systems with finger devices
US20200125235A1 (en) * 2018-10-23 2020-04-23 Microsoft Technology Licensing, Llc Adjustable Virtual User Input Devices To Accommodate User Physical Limitations
US10937244B2 (en) 2018-10-23 2021-03-02 Microsoft Technology Licensing, Llc Efficiency enhancements to construction of virtual reality environments
CN110942519B (en) * 2019-10-29 2023-04-28 华南师范大学 Computer assembly virtual experiment system and implementation method thereof
US11360587B1 (en) 2020-04-07 2022-06-14 Apple Inc. Deployment systems for computer system finger devices
CN111494927B (en) * 2020-04-20 2023-06-09 张洋 Maze with haptic feedback and method of generating haptic feedback for maze
CN112351324A (en) * 2020-10-27 2021-02-09 深圳Tcl新技术有限公司 Analog mouse control method, device, equipment and computer readable storage medium
CN116176430B (en) * 2023-05-04 2023-08-29 长城汽车股份有限公司 Virtual key display method and device, vehicle and storage medium

Citations (34)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4988981A (en) * 1987-03-17 1991-01-29 Vpl Research, Inc. Computer data entry and manipulation apparatus and method
US5581484A (en) * 1994-06-27 1996-12-03 Prince; Kevin R. Finger mounted computer input device
US5697001A (en) * 1996-04-24 1997-12-09 Eastman Kodak Company Camera for generating and recording object data with the recorded image
US5767842A (en) * 1992-02-07 1998-06-16 International Business Machines Corporation Method and device for optical input of commands or data
US6094625A (en) * 1997-07-03 2000-07-25 Trimble Navigation Limited Augmented vision for survey work and machine control
US6107997A (en) * 1996-06-27 2000-08-22 Ure; Michael J. Touch-sensitive keyboard/mouse and computing device using the same
US6201484B1 (en) * 1989-11-22 2001-03-13 Transforming Technologies, Llc Ergonomic customizeable user/computer interface device
US6215498B1 (en) * 1998-09-10 2001-04-10 Lionhearth Technologies, Inc. Virtual command post
US20020126026A1 (en) * 2001-03-09 2002-09-12 Samsung Electronics Co., Ltd. Information input system using bio feedback and method thereof
US6512838B1 (en) * 1999-09-22 2003-01-28 Canesta, Inc. Methods for enhancing performance and data acquired from three-dimensional image systems
US6600480B2 (en) * 1998-12-31 2003-07-29 Anthony James Francis Natoli Virtual reality keyboard system and method
US6611253B1 (en) * 2000-09-19 2003-08-26 Harel Cohen Virtual input environment
US6611258B1 (en) * 1996-01-11 2003-08-26 Canon Kabushiki Kaisha Information processing apparatus and its method
US6611252B1 (en) * 2000-05-17 2003-08-26 Dufaux Douglas P. Virtual data input device
US6614422B1 (en) * 1999-11-04 2003-09-02 Canesta, Inc. Method and apparatus for entering data using a virtual input device
US6650318B1 (en) * 2000-10-13 2003-11-18 Vkb Inc. Data input device
US6670894B2 (en) * 2001-02-05 2003-12-30 Carsten Mehring System and method for keyboard independent touch typing
US6690357B1 (en) * 1998-10-07 2004-02-10 Intel Corporation Input device using scanning sensors
US6711293B1 (en) * 1999-03-08 2004-03-23 The University Of British Columbia Method and apparatus for identifying scale invariant features in an image and use of same for locating an object in an image
US6750849B2 (en) * 2000-12-15 2004-06-15 Nokia Mobile Phones, Ltd. Method and arrangement for accomplishing a function in an electronic apparatus and an electronic apparatus
US20040178995A1 (en) * 2001-06-29 2004-09-16 Sterling Hans Rudolf Apparatus for sensing the position of a pointing object
US20040184640A1 (en) * 2003-03-17 2004-09-23 Samsung Electronics Co., Ltd. Spatial motion recognition system and method using a virtual handwriting plane
US20040258328A1 (en) * 2001-12-20 2004-12-23 Doron Adler Device, system and method for image based size analysis
US20050110771A1 (en) * 2003-11-21 2005-05-26 Hall Bernard J. Three dimensional position sensing apparatus and method for a display device
US6911972B2 (en) * 2001-04-04 2005-06-28 Matsushita Electric Industrial Co., Ltd. User interface device
US20050156914A1 (en) * 2002-06-08 2005-07-21 Lipman Robert M. Computer navigation
US6965374B2 (en) * 2001-07-16 2005-11-15 Samsung Electronics Co., Ltd. Information input method using wearable information input device
US20050264527A1 (en) * 2002-11-06 2005-12-01 Lin Julius J Audio-visual three-dimensional input/output
US6977643B2 (en) * 2002-01-10 2005-12-20 International Business Machines Corporation System and method implementing non-physical pointers for computer devices
US7042442B1 (en) * 2000-06-27 2006-05-09 International Business Machines Corporation Virtual invisible keyboard
US7162084B2 (en) * 2003-01-29 2007-01-09 Microsoft Corporation System and method for automatically detecting and extracting objects in digital image data
US7337410B2 (en) * 2002-11-06 2008-02-26 Julius Lin Virtual workstation
US20080130016A1 (en) * 2006-10-11 2008-06-05 Markus Steinbichler Method and an apparatus for the determination of the 3D coordinates of an object
US20090129628A1 (en) * 2004-11-30 2009-05-21 Iee International Electronics & Engineering S.A. Method for determining the position of an object from a digital image

Patent Citations (35)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4988981B1 (en) * 1987-03-17 1999-05-18 Vpl Newco Inc Computer data entry and manipulation apparatus and method
US4988981A (en) * 1987-03-17 1991-01-29 Vpl Research, Inc. Computer data entry and manipulation apparatus and method
US6201484B1 (en) * 1989-11-22 2001-03-13 Transforming Technologies, Llc Ergonomic customizeable user/computer interface device
US5767842A (en) * 1992-02-07 1998-06-16 International Business Machines Corporation Method and device for optical input of commands or data
US5581484A (en) * 1994-06-27 1996-12-03 Prince; Kevin R. Finger mounted computer input device
US6611258B1 (en) * 1996-01-11 2003-08-26 Canon Kabushiki Kaisha Information processing apparatus and its method
US5697001A (en) * 1996-04-24 1997-12-09 Eastman Kodak Company Camera for generating and recording object data with the recorded image
US6107997A (en) * 1996-06-27 2000-08-22 Ure; Michael J. Touch-sensitive keyboard/mouse and computing device using the same
US6094625A (en) * 1997-07-03 2000-07-25 Trimble Navigation Limited Augmented vision for survey work and machine control
US6215498B1 (en) * 1998-09-10 2001-04-10 Lionhearth Technologies, Inc. Virtual command post
US6690357B1 (en) * 1998-10-07 2004-02-10 Intel Corporation Input device using scanning sensors
US6600480B2 (en) * 1998-12-31 2003-07-29 Anthony James Francis Natoli Virtual reality keyboard system and method
US6711293B1 (en) * 1999-03-08 2004-03-23 The University Of British Columbia Method and apparatus for identifying scale invariant features in an image and use of same for locating an object in an image
US6512838B1 (en) * 1999-09-22 2003-01-28 Canesta, Inc. Methods for enhancing performance and data acquired from three-dimensional image systems
US6614422B1 (en) * 1999-11-04 2003-09-02 Canesta, Inc. Method and apparatus for entering data using a virtual input device
US6611252B1 (en) * 2000-05-17 2003-08-26 Dufaux Douglas P. Virtual data input device
US7042442B1 (en) * 2000-06-27 2006-05-09 International Business Machines Corporation Virtual invisible keyboard
US6611253B1 (en) * 2000-09-19 2003-08-26 Harel Cohen Virtual input environment
US6650318B1 (en) * 2000-10-13 2003-11-18 Vkb Inc. Data input device
US6750849B2 (en) * 2000-12-15 2004-06-15 Nokia Mobile Phones, Ltd. Method and arrangement for accomplishing a function in an electronic apparatus and an electronic apparatus
US6670894B2 (en) * 2001-02-05 2003-12-30 Carsten Mehring System and method for keyboard independent touch typing
US20020126026A1 (en) * 2001-03-09 2002-09-12 Samsung Electronics Co., Ltd. Information input system using bio feedback and method thereof
US6911972B2 (en) * 2001-04-04 2005-06-28 Matsushita Electric Industrial Co., Ltd. User interface device
US20040178995A1 (en) * 2001-06-29 2004-09-16 Sterling Hans Rudolf Apparatus for sensing the position of a pointing object
US6965374B2 (en) * 2001-07-16 2005-11-15 Samsung Electronics Co., Ltd. Information input method using wearable information input device
US20040258328A1 (en) * 2001-12-20 2004-12-23 Doron Adler Device, system and method for image based size analysis
US6977643B2 (en) * 2002-01-10 2005-12-20 International Business Machines Corporation System and method implementing non-physical pointers for computer devices
US20050156914A1 (en) * 2002-06-08 2005-07-21 Lipman Robert M. Computer navigation
US20050264527A1 (en) * 2002-11-06 2005-12-01 Lin Julius J Audio-visual three-dimensional input/output
US7337410B2 (en) * 2002-11-06 2008-02-26 Julius Lin Virtual workstation
US7162084B2 (en) * 2003-01-29 2007-01-09 Microsoft Corporation System and method for automatically detecting and extracting objects in digital image data
US20040184640A1 (en) * 2003-03-17 2004-09-23 Samsung Electronics Co., Ltd. Spatial motion recognition system and method using a virtual handwriting plane
US20050110771A1 (en) * 2003-11-21 2005-05-26 Hall Bernard J. Three dimensional position sensing apparatus and method for a display device
US20090129628A1 (en) * 2004-11-30 2009-05-21 Iee International Electronics & Engineering S.A. Method for determining the position of an object from a digital image
US20080130016A1 (en) * 2006-10-11 2008-06-05 Markus Steinbichler Method and an apparatus for the determination of the 3D coordinates of an object

Cited By (64)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050264527A1 (en) * 2002-11-06 2005-12-01 Lin Julius J Audio-visual three-dimensional input/output
US7774075B2 (en) 2002-11-06 2010-08-10 Lin Julius J Y Audio-visual three-dimensional input/output
US7453438B2 (en) * 2004-03-26 2008-11-18 Canon Kabushiki Kaisha Information processing apparatus and method
US20050212763A1 (en) * 2004-03-26 2005-09-29 Cannon Kabushiki Kaisha Information processing apparatus and method
US20060238439A1 (en) * 2005-04-22 2006-10-26 Microsoft Corporation State-based auxiliary display operation
US8959259B2 (en) 2005-04-22 2015-02-17 Microsoft Corporation State-based auxiliary display operation
US9383830B2 (en) 2005-04-22 2016-07-05 Microsoft Technology Licensing, Llc State-based auxiliary display operation
US10275201B2 (en) 2005-04-22 2019-04-30 Microsoft Technology Licensing, Llc State-based auxiliary display operation
US8692816B2 (en) * 2005-04-22 2014-04-08 Microsoft Corporation State-based auxiliary display operation
US9870187B2 (en) 2005-04-22 2018-01-16 Microsoft Technology Licensing, Llc State-based auxiliary display operation
US20120019373A1 (en) * 2007-10-12 2012-01-26 Immersion Corporation Method and Apparatus for Wearable Remote Interface Device
US8405612B2 (en) * 2007-10-12 2013-03-26 Immersion Corporation Method and apparatus for wearable remote interface device
US10123300B2 (en) 2007-12-31 2018-11-06 Apple Inc. Tactile feedback in an electronic device
US9520037B2 (en) 2007-12-31 2016-12-13 Apple Inc. Tactile feedback in an electronic device
US10420064B2 (en) 2007-12-31 2019-09-17 Apple, Inc. Tactile feedback in an electronic device
US9070262B2 (en) 2007-12-31 2015-06-30 Apple Inc. Tactile feedback in an electronic device
US8138896B2 (en) * 2007-12-31 2012-03-20 Apple Inc. Tactile feedback in an electronic device
US10616860B2 (en) 2007-12-31 2020-04-07 Apple, Inc. Wireless control of stored media presentation
US20090167509A1 (en) * 2007-12-31 2009-07-02 Apple Inc. Tactile feedback in an electronic device
US8373549B2 (en) * 2007-12-31 2013-02-12 Apple Inc. Tactile feedback in an electronic device
US8400283B2 (en) 2007-12-31 2013-03-19 Apple Inc. Tactile feedback in an electronic device
US20090167508A1 (en) * 2007-12-31 2009-07-02 Apple Inc. Tactile feedback in an electronic device
US8754759B2 (en) 2007-12-31 2014-06-17 Apple Inc. Tactile feedback in an electronic device
US9026449B2 (en) * 2008-06-16 2015-05-05 Audi Ag Communication system and method for representing information in a communication
US20110087493A1 (en) * 2008-06-16 2011-04-14 Stefan Sellschopp Communication System and Method for Representing Information in a Communication
US20100145518A1 (en) * 2008-12-10 2010-06-10 Samsung Electronics Co., Ltd. Robot and method thereof
US8571713B2 (en) * 2008-12-10 2013-10-29 Samsung Electronics Co., Ltd. Robot and method thereof
US20100171694A1 (en) * 2009-01-06 2010-07-08 Chih-Hung Lu Electronic Apparatus with Virtual Data Input Device
US9280942B2 (en) * 2009-02-09 2016-03-08 Mitsubishi Electric Corporation Electro-optical device, shift register circuit, and semiconductor device
US20140117358A1 (en) * 2009-02-09 2014-05-01 Mitsubishi Electric Corporation Electro-optical device, shift register circuit, and semiconductor device
US20100265183A1 (en) * 2009-04-20 2010-10-21 Microsoft Corporation State changes for an adaptive device
US20100265182A1 (en) * 2009-04-20 2010-10-21 Microsoft Corporation Context-based state change for an adaptive input device
US20110227947A1 (en) * 2010-03-16 2011-09-22 Microsoft Corporation Multi-Touch User Interface Interaction
US20120025945A1 (en) * 2010-07-27 2012-02-02 Cyberglove Systems, Llc Motion capture data glove
US20120050154A1 (en) * 2010-08-31 2012-03-01 Adil Jagmag Method and system for providing 3d user interface in 3d televisions
US8957910B2 (en) * 2010-12-22 2015-02-17 Nintendo Co., Ltd. Game system, game device, storage medium storing game program, and game process method
US9808717B2 (en) 2010-12-22 2017-11-07 Nintendo Co., Ltd. Game system, game device, storage medium storing game program, and game process method
US10300383B2 (en) 2010-12-22 2019-05-28 Nintendo Co., Ltd. Game system, game device, storage medium storing game program, and game process method
US20120165099A1 (en) * 2010-12-22 2012-06-28 Nintendo Co., Ltd. Game system, game device, storage medium storing game program, and game process method
US20120169598A1 (en) * 2011-01-05 2012-07-05 Tovi Grossman Multi-Touch Integrated Desktop Environment
US9612743B2 (en) 2011-01-05 2017-04-04 Autodesk, Inc. Multi-touch integrated desktop environment
US9600090B2 (en) * 2011-01-05 2017-03-21 Autodesk, Inc. Multi-touch integrated desktop environment
US9030425B2 (en) * 2011-04-19 2015-05-12 Sony Computer Entertainment Inc. Detection of interaction with virtual object from finger color change
US20130257751A1 (en) * 2011-04-19 2013-10-03 Sony Computer Entertainment Inc. Detection of interaction with virtual object from finger color change
US9720505B2 (en) 2013-01-03 2017-08-01 Meta Company Extramissive spatial imaging digital eye glass apparatuses, methods and systems for virtual or augmediated vision, manipulation, creation, or interaction with objects, materials, or other entities
US11073916B2 (en) 2013-01-03 2021-07-27 Meta View, Inc. Extramissive spatial imaging digital eye glass apparatuses, methods and systems for virtual or augmediated vision, manipulation, creation, or interaction with objects, materials, or other entities
US10540014B2 (en) 2013-01-03 2020-01-21 Meta View, Inc. Extramissive spatial imaging digital eye glass apparatuses, methods, and systems for virtual or augmediated vision, manipulation, creation, or interaction with objects, materials, or other entities
US11334171B2 (en) 2013-01-03 2022-05-17 Campfire 3D, Inc. Extramissive spatial imaging digital eye glass apparatuses, methods and systems for virtual or augmediated vision, manipulation, creation, or interaction with objects, materials, or other entities
US10168791B2 (en) 2013-01-03 2019-01-01 Meta Company Extramissive spatial imaging digital eye glass apparatuses, methods and systems for virtual or augmediated vision, manipulation, creation, or interaction with objects, materials, or other entities
WO2014106823A3 (en) * 2013-01-03 2014-12-11 Meta Company Extramissive spatial imaging digital eye glass for virtual or augmediated vision
US20230376121A1 (en) * 2013-01-03 2023-11-23 Campfire 3D, Inc. Extramissive spatial imaging digital eye glass apparatuses, methods and systems for virtual or augmediated vision, manipulation, creation, or interaction with objects, materials, or other entities
CN105190477A (en) * 2013-03-21 2015-12-23 索尼公司 Head-mounted device for user interactions in an amplified reality environment
US10466794B2 (en) * 2013-03-21 2019-11-05 Sony Corporation Gesture recognition areas and sub-areas for interaction with real and virtual objects within augmented reality
US20160034039A1 (en) * 2013-03-21 2016-02-04 Sony Corporation Information processing apparatus, operation control method and program
WO2014147686A1 (en) * 2013-03-21 2014-09-25 Sony Corporation Head-mounted device for user interactions in an amplified reality environment
US9751406B2 (en) 2014-04-03 2017-09-05 Audi Ag Motor vehicle and method for controlling a climate control system in a motor vehicle
US11010972B2 (en) 2015-12-11 2021-05-18 Google Llc Context sensitive user interface activation in an augmented and/or virtual reality environment
US10782782B1 (en) * 2019-02-07 2020-09-22 Facebook Technologies, Llc Devices, systems, and methods for radar-based artificial reality tracking
US11221404B1 (en) 2019-02-07 2022-01-11 Facebook Technologies, Llc Devices, systems, and methods for radar-based artificial reality tracking using zero-crossing detection
US11226406B1 (en) 2019-02-07 2022-01-18 Facebook Technologies, Llc Devices, systems, and methods for radar-based artificial reality tracking
US11474227B1 (en) 2019-02-07 2022-10-18 Meta Platforms Technologies, Llc Devices, systems, and methods for radar-based artificial reality tracking
US11650306B1 (en) 2019-02-07 2023-05-16 Meta Platforms Technologies, Llc Devices, systems, and methods for radar-based artificial reality tracking using polarized signaling
US11030459B2 (en) * 2019-06-27 2021-06-08 Intel Corporation Methods and apparatus for projecting augmented reality enhancements to real objects in response to user gestures detected in a real environment
US11682206B2 (en) 2019-06-27 2023-06-20 Intel Corporation Methods and apparatus for projecting augmented reality enhancements to real objects in response to user gestures detected in a real environment

Also Published As

Publication number Publication date
WO2004044664A9 (en) 2006-05-26
AU2003291304A8 (en) 2004-06-03
US7337410B2 (en) 2008-02-26
US20040128012A1 (en) 2004-07-01
AU2003291304A1 (en) 2004-06-03
WO2004044664A1 (en) 2004-05-27

Similar Documents

Publication Publication Date Title
US7337410B2 (en) Virtual workstation
US7774075B2 (en) Audio-visual three-dimensional input/output
JP4975634B2 (en) Method and device for controlling and entering data
US7161579B2 (en) Hand-held computer interactive device
US20150261310A1 (en) One-dimensional input system and method
US20100020036A1 (en) Portable electronic device and method of controlling same
Hinckley et al. Input/Output Devices and Interaction Techniques.
CN109074167B (en) Gadget for computing device multimedia management for blind or visually impaired people
JP2015519624A (en) Gesture recognition apparatus and method
Bergström et al. Human--Computer interaction on the skin
US11500452B2 (en) Displaying physical input devices as virtual objects
Gupta et al. Investigating remote tactile feedback for mid-air text-entry in virtual reality
US20050270274A1 (en) Rapid input device
EP1160651A1 (en) Wireless cursor control
Caporusso et al. Enabling touch-based communication in wearable devices for people with sensory and multisensory impairments
Dube et al. Shapeshifter: Gesture Typing in Virtual Reality with a Force-based Digital Thimble
US11409359B1 (en) Systems and methods for collective control of virtual objects
Golledge et al. Multimodal interfaces for representing and accessing geospatial information
KR20160111880A (en) A method for interlocking wearable 3d input devices with external devices
Habibi Detecting surface interactions via a wearable microphone to improve augmented reality text entry
Mulgund et al. Human/machine interface modalities for soldier systems technologies
Bowman et al. Pinch keyboard: Natural text input for immersive virtual environments
Liu Multimodal interaction: developing an interaction concept for a touchscreen incorporating tactile feedback
Zheng Enabling Expressive Keyboard Interaction with Finger, Hand, and Hand Posture Identification
Baber et al. Alternative interaction techniques

Legal Events

Date Code Title Description
AS Assignment

Owner name: INTELLIGENTEK CORP., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:LIN, JULIUS;REEL/FRAME:020492/0323

Effective date: 20071228

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION