WO1998036631A2 - System for sensing facial movements in virtual reality - Google Patents
System for sensing facial movements in virtual reality Download PDFInfo
- Publication number
- WO1998036631A2 WO1998036631A2 PCT/US1998/002552 US9802552W WO9836631A2 WO 1998036631 A2 WO1998036631 A2 WO 1998036631A2 US 9802552 W US9802552 W US 9802552W WO 9836631 A2 WO9836631 A2 WO 9836631A2
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- movement
- wearer
- data
- facial
- head
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/254—Analysis of motion involving subtraction of images
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/2224—Studio circuitry; Studio devices; Studio equipment related to virtual studio applications
Definitions
- VR systems are computer generated systems that provide the experience of a simulated environment that forms a three dimensional virtual world. These VR systems are used in many different applications including commercial flight simulators, entertainment systems, computer games and video arcade games, to name a few.
- a participant typically wears a head-mounted device that enables sensory receiving a virtual reality world generated by the computer.
- the system also includes an interaction means, such as a pointm ⁇ device or specially configured glove containing sensors and actuators, for interacting with objects in the virtual world.
- an interaction means such as a pointm ⁇ device or specially configured glove containing sensors and actuators, for interacting with objects in the virtual world.
- a full body suit also containing sensors and actuators, may be additionally provided so that the user can influence and has a realistic feel of objects m the virtual world.
- One means for interacting with animation in real time would be to provide a data input means that transmits appropriate data to a computer while the computer is simultaneously displaying and running the animation. The data received by the computer could then be integrated into the running animation for altering the animation in real time.
- U.S. Patent No. 5,507,716, to LeBerge et al. discloses equipment and methods used to induce lucid dreams in sleeping persons.
- the disclosed equipment and methods include a number of features prevalent in virtual reality systems.
- the system includes a headband having a face mask.
- the face mas contains two infrared emitter detector pairs, one detector pair for sensing eye movements and the other detector pair for sensing body movements.
- a state-test button and components that produce low intensity sensory stimuli, such as light and sound are also included in the disclosed face mask.
- a microprocessor is provided for momtorin ⁇ fluctuations in volta ⁇ es from the infrared detector pairs for the occurrence of a predetermined sequence of voltages. When the correct sequence occurs, the microprocessor triggers the stimuli producing components in the mask into activity, to cue the sleeper to become lucid.
- U.S. Patent No. 5,572,749 discloses a helmet mounting device for use in a virtual reality system that allows a helmet to be securely mounted on a user's head in a desired alignment, without irritating chin straps.
- the disclosed helmet includes a liner with a retention figured to be quickly mounted on, or removed from the head of the wearer, for applications such as theme parks.
- a disadvantage of the disclosed helmet is that it may still be substantially prone to movement about on the head of the wearer.
- a head mounted sensor input system which includes a head mount configured to track the movement of different facial regions of a wearer for generating data to manipulate a computer generated animation;
- the preferred embodiment of the present invention includes a head mount configured to inhibit relative movement between the head mount and head of a wearer.
- the preferred embodiment of the head mount is coupled to a computer through an umbilical data cable for processing data generated by the head mount in real time.
- Motion Data data generated by the facial movements of a person
- the Movement Data is continuously processed so that an object, such as a cartoon figure in a virtual reality program running on the computer, has facial expressions analogous to the facial expressions of the actor, as the actor is performing.
- a virtual reality computer generated character controlled by an actor wearing the head mount in the system of the preferred embodiment of the present invention is capable of conversm ⁇ with a human interviewee, in an interview situation for example, with the appearance that the human is actually interacting with the cartoon, as though the cartoon were another human.
- peripheral means may be provided for enhancing the facial expressions of the cartoon.
- the head mount of the preferred embodiment of the present invention is especially configured to inhibit relative movement between the head mount and head of the wearer for preventing the generation of incorrect data.
- the head mount includes a helmet portion having a rigid brow portion configured to engage the frontal region and forehead of the wearer's head to inhibit movement therebetween.
- a headband that extends about the occipital region of the head of the wearer and preferably is adjustable so that the head mount may be securely worn by wearers with various different head sizes.
- a data transmission frame secured to the helmet portion and coupled to the data cable is provided for coupling the head mount to the computer.
- the helmet portion and frame preferably comprise a suitable lightweight and rigid material so that the head mount is substantially light, weighing approximately one pound .
- Each of the cameras is attached to a positionable flexible arm for coupling the cameras to the umbilical data cable.
- the positionable arms enable precise positioning of the desired camera above the localized region of the wearer's face to be monitored, while maintaining the position of the camera
- Each of the cameras is preferably provided with a light emitting source adjacent to the lens thereof.
- the light sources of each of the cameras illuminate the adjacent region of the wearer's face to enhance the camera' s ability to sense movement in the region to generate the Movement Data indicative of facial expressions.
- a reflective material is adhered to the wearer' s face in each of the monitored regions.
- the reflective material reflects light emitted by the light source of the adjacent camera into the lens of the camera for generating data indicative of movement (the "Movement Data") in the region
- the camera monitoring the wearer's eye generates Movement Data indicating pupil movement and blinking.
- the data from each of the cameras is simultaneously transmitted to the computer in real time, for continuously changing the facial expressions and eye movement of the object in real time.
- a database is initially designed which contains the information necessary to support the animated image.
- the Movement Data generated by the preferred head mount then interacts with and manipulates the animation database to provide an altered image of facial expression.
- the Movement Data representative of the changing expressions is continuously transmitted to the computer and processed thereby for enabling the cartoon figure to have continuously changing facial expressions and eye movements that mimic the actor's.
- the actor can move freely about, even dance, without inaccurate data being generated by the cameras.
- the configuration of the head mount does not encumber the wearer, while the lightness thereof enables the head mount to be worn some time without fatiguing the neck and shoulders of the wearer.
- the head mount may be provided with a microphone.
- the micropality performance without the need to have the actor separately record a voice performance or to have another actor perform the voice portion of the virtual reality interaction.
- the adjustable mini video cameras are filtered to accept a narrow bandwidth of infra red light sources that illuminate 3m reflective materials that adhered to the face.
- the reflective spots are tracked in x and y real time through image processing and reported to a data collector PC.
- the head set is extremely light weight and can be worn simultaneously by the performer in the body suit.
- the system converts x, y motion data in real time to a set of morphing three dimensional geometric face sculptures (databases) that combine into a single real time computer generated rendered head of a character with an infinite number of variations to produce lip sync and facial expressions.
- FIG. 1 is a front elevational view of a preferred embodiment of a head mount for a system for manipulating a computer generated animation in real time of the present invention
- FIG. 3 is a side elevational view of the head mount of the present invention.
- FIGS. 1-3 of the drawings there is shown, generally at 10, a preferred embodiment of a system for manipulating computer generated animation in real time that includes a head mount 12 constructed according to the principles of the present invention.
- the invented head mount 12 is preferably configured to inhibit relative movement between the head mount 12 and head 14 of a wearer 16.
- the invented system 10 comprises the head mount 12 coupled to a computer 18 through an umbilical data cable 20 for processing data generated by the head mount 12 in real time.
- Movement Data generated by the facial movements of the wearer 16, such as when the wearer is an actor, are transmitted to the computer 18 for processing the data in real time.
- the Movement Data is continuously processed so that an object, such as a cartoon figure in a virtual reality program running on the computer 18, has facial expressions which correspond to and are analogous with the facial expressions of the actor 16, as the actor 16 is performing.
- a virtual reality cartoon controlled by the actor 16 wearing the head mount 12 in the invented system 10 is capable of conversing with a human interviewee, in an interview situation for example, with the appearance that the human is actually interacting with the cartoon, as though the cartoon were another human.
- peripheral means may be provided for enhancing the facial expressions of the cartoon.
- the head mount 12 is configured to inhibit relative movement between the head mount 12 and head 14 of the wearer 16 for preventing the generation of incorrect data.
- the head mount 12 includes a helmet portion 22 having a rigid brow portion 24 configured to engage a frontal region and forehead 26 (best seen in FIG. 2) of the wearer's head 14 to inhibit movement therebetween.
- the rigid brow 24 is curved in a base plane B and in a transverse plane T that extends substantially perpendicularly to the base plane B, such that the brow 24 forms a compound curve that mates with the frontal region and forehead 26 of the wearer 16.
- the brow 24 is preferably molded from a suitable known strong, rigid, and a lightweight polymeric material and is fabricated using methods known in the art.
- the helmet 22 further includes an adjustable headband 28 attached to the brow 24.
- the headband 28 is configured to extend- about an occipital region 30 of the wearer's head 14 and preferably is adjustable for adjusting the circumference of the helmet 22 so that the head mount 12 may be securely worn by wearers with various different head sizes.
- the headband 28 comprises flexible bands 32 that are attached to the brow 24 and are affixed to interlocking adjusting straps 34.
- a knob 36 is coupled to the interlocked straps 34 and rotated either clockwise or counterclockwise for adjusting the length of the headband 28 for changing the circumference of the helmet 22.
- a suitable foam padding 38 is affixed to an inner side 40 of the brow 24 and may be affixed to an inner side 42 of the bands 32.
- the padding 38 is provided to increase the coefficient of friction between the brow 24 and the wearer' s forehead 26 to prevent the brow 24 from sliding about on the forehead 26.
- the padding 38 also absorbs perspiration and provides comfort for the wearer 16.
- a data transmission frame 44 couples the head mount 12 to the computer 18.
- the data frame 44 comprises a hollow curvilinear tube that extends about the helmet 22 and is affixed to the brow 24.
- the frame 44 has ends 46 that terminate adjacent to the brow 24.
- the data frame 44 preferably comprises a lightweight, rigid metal alloy, such as aluminum for example.
- the frame 44 and helmet 22, along with each of the components comprising the head mount 12, to be thoroughly discussed hereinafter, comprise substantially lightweight materials so that the head mount 12 has a preferred weight range of approximately 0.5 to 1.5 pounds.
- the low weight of the head mount 12 enables the head mount to be worn for some time without fatiguing the neck and shoulders of the wearer 16, while the configuration thereof does not encumber the wearer.
- the preferred embodiment 10 of the present invention additionally includes a plurality of facial movement monitoring means 50.
- the monitoring means 50 continuously track movement in a plurality of localized regions of the wearer' s face 52 for continuously generating facial expression Movement Data representative of the changing facial expressions of the wearer 16.
- a monitoring means 50 is provided for monitoring a cheek region 54, a mouth and chin region 56, and an eyebrow and forehead region 58.
- a fourth monitoring means 50A monitors an eye region 60 for monitoring pupil movement and blinking of one of the wearer's eyes 62.
- the monitoring means 50 comprise infrared sensitive cameras, such as those manufactured by Chinon Kabushiki Kaisha, Suwa, Japan, for monitoring the localized regions 54, 56, 58 of the wearer' s face 52 for sensin ⁇ movements m each of these regions to generate data indicative of facial expressions
- the eye monitoring camera 50A continuously monitors the eye region 60 for generating data indicative eye movement and blinking.
- the cameras 50 are configured with known low-pass filter means (not shown) for filtering out undesired ambient light.
- Each of the cameras 50 are preferably provided with a plurality of light emitting sources 64 affixed adjacent to a lens 66 as close to the center of the lense without obstructing full viewing as possible
- the light emitting sources 64 comprise light emitting diodes 64 positioned about the lens 66 of each camera 50 for providing a focused light source to illuminate the adjacent facial region 54, 56, 58, 60 to be monitored.
- the light sources 64 illuminate the adjacent region of the wearer's face 52 to enhance the camera' s ability to detect movement in the monitored facial regions 54, 56, 58, 60 for generating representative facial expression and eye movement data.
- the eye monitoring camera 50A is provided with a plurality of LEDs 64, ranging from approximately 4 to approximately 8, for inundating the eye 62 with light
- the large number of LEDs 64 are provided with the eye monitoring camera 50A, since pupil movement and blinking are monitored by the lack of light reflected into the lens 66 thereof.
- each of the cameras 50 is attached to a positionable flexible arm 68 for coupling the cameras 50 to the computer 18 via the umbilical data cable 20.
- the positionable arms 68 enable precise positioning of the desired camera 50 above the localized region 54, 56, 58, 60 to be monitored.
- the arms 68 have ends 70 attached to connectors 72 affixed to the ends 46 of the frame 44, with the arm 68A supporting the camera 50 monitoring the eyebrow and forehead region 58 attached to the brow 24.
- the positionable arms 68 are hollow to provide an enclosed data line path from an output of cameras 50 through the arms 68 supporting the camera 50, through the hollow data frame 44, into the data cable 20, and to the computer 18, so that extraneous data lines do not interfere with the line of sight of the wearer 16 or with movements made thereby.
- the positionable arms 68 preferably comprise a series of articulated ball and socket type couplings for positioning each of the cameras 50 above the desired region to be monitored and for maintaining the desired position.
- the arms 68 are fabricated from a suitable rigid polymeric material, such as Delrint ⁇ .
- the series of hollow articulated ball and socket type couplings comprising the arms 68 are often used for a modular adjustable coolant hose, and are sold under the trademark "Loc-Line”, manufactured by Lockwood Products, Inc., Lake Oswego, Oregon; and sold under the trademark "Snap-Loc", manufactured by Cedar Berg Corporation, Minnesota.
- a plurality of epidermis position indicating means 74 are adhered to the epidermis 76 of the face 52 of a wearer 16 in each of the localized facial regions 54, 56, 58.
- Horizontal and vertical movement of the indicating means 74 indicates movement of the epidermis 76 to the adjacent camera 50 for indicating changing facial expressions.
- the indicating means 74 in one or more of the monitored regions 54, 56, 58 moves since the epidermis 76 moves due to the muscular contractions causing the facial expressions.
- the indicating means 74 moves, light emitted by the light sources 64 are reflected into the lens 66 of the adjacent camera 50 to indicate the changing horizontal and vertical positions of the indicating means 74 for continuously generating Movement Data representative of the changing facial expressions.
- each of the epidermis position indicating means 74 comprises a portion of reflective material, such as known reflective tape manufactured by Minnesota Mining and Manufacturing, Minnesota.
- One or more portions of the reflective tape 74 may be adhered to the wearer's face 52 in each of the monitored regions 54, 56, 58, depending upon such factors as the dimensions of the region to be monitored, expected or desired facial expressions to be made by the wearer 16, and desired portions of the region to be monitored, such as lips 77 for example.
- facial expression Movement Data is continuously generated by each camera 50 monitoring the facial expressions and eye movements of the wearer 16 and continuously transmitted to the computer 18 via the data cable 20.
- the Movement Data is then processed by the computer in real time, for continuously interacting with the database (not shown) representative of expressions and eye movements corresponding to the movements of the wearer 16.
- the object in the virtual reality program running on the computer 18 has facial expressions analogous to the facial expressions of the wearer as they are occurring.
- the data representative of the changing expressions enables the cartoon figure to have continuously changing facial expressions and eye movements that mimic the actor's. Since there is no relative movement between the head mount and the wearer's head, the actor can move freely about, even dance, without inaccurate data being generated by the cameras.
Abstract
Description
Claims
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP10536706A JP2001501346A (en) | 1997-02-06 | 1998-02-06 | System for sensing facial movements in virtual reality |
EP98906288A EP0938698A4 (en) | 1997-02-06 | 1998-02-06 | System for sensing facial movements in virtual reality |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US08/795,544 | 1997-02-06 | ||
US08/795,544 US6121953A (en) | 1997-02-06 | 1997-02-06 | Virtual reality system for sensing facial movements |
Publications (2)
Publication Number | Publication Date |
---|---|
WO1998036631A2 true WO1998036631A2 (en) | 1998-08-27 |
WO1998036631A3 WO1998036631A3 (en) | 1998-11-19 |
Family
ID=25165792
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US1998/002552 WO1998036631A2 (en) | 1997-02-06 | 1998-02-06 | System for sensing facial movements in virtual reality |
Country Status (4)
Country | Link |
---|---|
US (1) | US6121953A (en) |
EP (1) | EP0938698A4 (en) |
JP (1) | JP2001501346A (en) |
WO (1) | WO1998036631A2 (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6288704B1 (en) | 1999-06-08 | 2001-09-11 | Vega, Vista, Inc. | Motion detection and tracking system to control navigation and display of object viewers |
SG109411A1 (en) * | 1998-10-08 | 2005-03-30 | Sony Computer Entertainment Inc | Method of and system for adding information and recording medium |
US8801488B2 (en) | 2012-10-15 | 2014-08-12 | Disney Enterprises, Inc. | Chin strap sensor for triggering control of walk-around characters |
US10445863B2 (en) | 2014-08-04 | 2019-10-15 | Facebook Technologies, Llc | Method and system for reconstructing obstructed face portions for virtual reality environment |
Families Citing this family (93)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7076088B2 (en) * | 1999-09-03 | 2006-07-11 | Honeywell International Inc. | Near-infrared disguise detection |
JP2001145779A (en) * | 1999-11-22 | 2001-05-29 | Namco Ltd | Sign recognizing system, game system, and computer readable recording medium having game program recorded |
US7137711B1 (en) | 2000-03-21 | 2006-11-21 | Leonard Reiffel | Multi-user retro reflector data input |
WO2001084475A1 (en) | 2000-05-03 | 2001-11-08 | Leonard Reiffel | Dual mode data imaging product |
US6795068B1 (en) | 2000-07-21 | 2004-09-21 | Sony Computer Entertainment Inc. | Prop input device and method for mapping an object from a two-dimensional camera image to a three-dimensional space for controlling action in a game program |
AU2001283375A1 (en) * | 2000-08-18 | 2002-03-04 | Leonard Reiffel | Annotating imaged data product |
US7071914B1 (en) * | 2000-09-01 | 2006-07-04 | Sony Computer Entertainment Inc. | User input device and method for interaction with graphic images |
JP4294319B2 (en) * | 2000-12-15 | 2009-07-08 | ライフェル レナード | Coded data source tracking device with image display |
JP4103592B2 (en) * | 2000-12-15 | 2008-06-18 | ライフェル レナード | Multiple imaging devices, multiple data sources, and coded data source data input devices for multiple applications |
JP4081373B2 (en) * | 2000-12-15 | 2008-04-23 | ライフェル レナード | Coded data source converter by image |
US6774869B2 (en) * | 2000-12-22 | 2004-08-10 | Board Of Trustees Operating Michigan State University | Teleportal face-to-face system |
EP1390909A4 (en) * | 2001-04-19 | 2004-12-29 | Leonard Reiffel | Combined imaging coded data source data acquisition |
JP2003108984A (en) * | 2001-07-27 | 2003-04-11 | Toshiba Corp | Facial image authentication device, passage controller, facial image authentication method, and passage control method |
US20040135766A1 (en) * | 2001-08-15 | 2004-07-15 | Leonard Reiffel | Imaged toggled data input product |
US7027619B2 (en) * | 2001-09-13 | 2006-04-11 | Honeywell International Inc. | Near-infrared method and system for use in face detection |
FR2838904A1 (en) * | 2002-04-22 | 2003-10-24 | Delphine Software Internat | Facial motion capture imaging unit for video game generation has color video camera viewing subjects face through mirror with white light illumination |
US20040145588A1 (en) * | 2003-01-27 | 2004-07-29 | Scimed Life Systems, Inc. | System and method for reviewing an image in a video sequence using a localized animation window |
ITMI20030121A1 (en) * | 2003-01-27 | 2004-07-28 | Giuseppe Donato | MODULAR SURVEILLANCE SYSTEM FOR MONITORING OF CRITICAL ENVIRONMENTS. |
US7218320B2 (en) * | 2003-03-13 | 2007-05-15 | Sony Corporation | System and method for capturing facial and body motion |
US7573480B2 (en) * | 2003-05-01 | 2009-08-11 | Sony Corporation | System and method for capturing facial and body motion |
US7358972B2 (en) * | 2003-05-01 | 2008-04-15 | Sony Corporation | System and method for capturing facial and body motion |
US20060291797A1 (en) * | 2003-05-27 | 2006-12-28 | Leonard Reiffel | Multi-imager multi-source multi-use coded data source data input product |
US8593542B2 (en) | 2005-12-27 | 2013-11-26 | DigitalOptics Corporation Europe Limited | Foreground/background separation using reference images |
US8896725B2 (en) * | 2007-06-21 | 2014-11-25 | Fotonation Limited | Image capture device with contemporaneous reference image capture mechanism |
US7620218B2 (en) | 2006-08-11 | 2009-11-17 | Fotonation Ireland Limited | Real-time face tracking with reference images |
US8133115B2 (en) | 2003-10-22 | 2012-03-13 | Sony Computer Entertainment America Llc | System and method for recording and displaying a graphical path in a video game |
US7613494B2 (en) * | 2003-11-26 | 2009-11-03 | University Of Utah Research Foundation | Apparatus and methods for imaging using an anatomical positioning system |
US20050135659A1 (en) * | 2003-12-19 | 2005-06-23 | Smith John D. | Optical motion sensor |
US20060071933A1 (en) | 2004-10-06 | 2006-04-06 | Sony Computer Entertainment Inc. | Application binary interface for multi-pass shaders |
US8320641B2 (en) * | 2004-10-28 | 2012-11-27 | DigitalOptics Corporation Europe Limited | Method and apparatus for red-eye detection using preview or other reference images |
US20060187193A1 (en) * | 2005-02-22 | 2006-08-24 | Esquilin Eileen C | System and method for generating a virtual reality presentation |
US7508393B2 (en) * | 2005-06-07 | 2009-03-24 | Gordon Patricia L | Three dimensional animated figures |
US7636126B2 (en) | 2005-06-22 | 2009-12-22 | Sony Computer Entertainment Inc. | Delay matching in audio/video systems |
US7209577B2 (en) * | 2005-07-14 | 2007-04-24 | Logitech Europe S.A. | Facial feature-localized and global real-time video morphing |
US20070060830A1 (en) * | 2005-09-12 | 2007-03-15 | Le Tan Thi T | Method and system for detecting and classifying facial muscle movements |
US7880746B2 (en) | 2006-05-04 | 2011-02-01 | Sony Computer Entertainment Inc. | Bandwidth management through lighting control of a user environment via a display device |
US7965859B2 (en) | 2006-05-04 | 2011-06-21 | Sony Computer Entertainment Inc. | Lighting control of a user environment via a display device |
US7916897B2 (en) | 2006-08-11 | 2011-03-29 | Tessera Technologies Ireland Limited | Face tracking for controlling imaging parameters |
GB0703974D0 (en) * | 2007-03-01 | 2007-04-11 | Sony Comp Entertainment Europe | Entertainment device |
JP4823970B2 (en) * | 2007-05-24 | 2011-11-24 | オリンパスイメージング株式会社 | Image editing device |
KR100896065B1 (en) * | 2007-12-17 | 2009-05-07 | 한국전자통신연구원 | Method for producing 3d facial animation |
US8160311B1 (en) * | 2008-09-26 | 2012-04-17 | Philip Raymond Schaefer | System and method for detecting facial gestures for control of an electronic device |
US8289443B2 (en) * | 2008-09-29 | 2012-10-16 | Two Pic Mc Llc | Mounting and bracket for an actor-mounted motion capture camera system |
US9325972B2 (en) * | 2008-09-29 | 2016-04-26 | Two Pic Mc Llc | Actor-mounted motion capture camera |
US9390516B2 (en) * | 2008-09-29 | 2016-07-12 | Two Pic Mc Llc | Asynchronous streaming of data for validation |
US8493286B1 (en) | 2009-04-21 | 2013-07-23 | Mark T. Agrama | Facial movement measurement and stimulation apparatus and method |
US9372533B1 (en) * | 2009-04-21 | 2016-06-21 | Mark Agrama | Facial movement measurement and stimulation apparatus and method |
US10786736B2 (en) | 2010-05-11 | 2020-09-29 | Sony Interactive Entertainment LLC | Placement of user information in a game space |
US10474875B2 (en) * | 2010-06-07 | 2019-11-12 | Affectiva, Inc. | Image analysis using a semiconductor processor for facial evaluation |
US9342817B2 (en) | 2011-07-07 | 2016-05-17 | Sony Interactive Entertainment LLC | Auto-creating groups for sharing photos |
US8970692B2 (en) * | 2011-09-01 | 2015-03-03 | Industrial Technology Research Institute | Head mount personal computer and interactive system using the same |
US20130162536A1 (en) * | 2011-12-05 | 2013-06-27 | Vaibhav Narayan Khedkar | Systems and methods for enabling or accessing or viewing components, involving an input assembly and a screen |
CN103366782B (en) | 2012-04-06 | 2014-09-10 | 腾讯科技(深圳)有限公司 | Method and device automatically playing expression on virtual image |
US9265458B2 (en) | 2012-12-04 | 2016-02-23 | Sync-Think, Inc. | Application of smooth pursuit cognitive testing paradigms to clinical drug development |
US9380976B2 (en) | 2013-03-11 | 2016-07-05 | Sync-Think, Inc. | Optical neuroinformatics |
US9417653B2 (en) | 2013-05-29 | 2016-08-16 | Lawrence A. Quintal | Face-operated joystick control system |
US20160186965A1 (en) * | 2013-08-06 | 2016-06-30 | Jim Burns | Face illumination means |
EP3218074A4 (en) * | 2014-11-16 | 2017-11-29 | Guy Finfter | System and method for providing an alternate reality ride experience |
US11099008B2 (en) * | 2014-12-15 | 2021-08-24 | Sony Corporation | Capture device assembly, three-dimensional shape measurement device, and motion detection device |
KR101633057B1 (en) * | 2015-04-22 | 2016-06-23 | 재단법인 실감교류인체감응솔루션연구단 | Facial Motion Capture Method for Head-Mounted Display System |
WO2016183020A1 (en) * | 2015-05-11 | 2016-11-17 | Magic Leap, Inc. | Devices, methods and systems for biometric user recognition utilizing neural networks |
US10113913B2 (en) | 2015-10-03 | 2018-10-30 | Facense Ltd. | Systems for collecting thermal measurements of the face |
US10092232B2 (en) | 2015-06-14 | 2018-10-09 | Facense Ltd. | User state selection based on the shape of the exhale stream |
US10130308B2 (en) | 2015-06-14 | 2018-11-20 | Facense Ltd. | Calculating respiratory parameters from thermal measurements |
US10080861B2 (en) | 2015-06-14 | 2018-09-25 | Facense Ltd. | Breathing biofeedback eyeglasses |
US10523852B2 (en) | 2015-06-14 | 2019-12-31 | Facense Ltd. | Wearable inward-facing camera utilizing the Scheimpflug principle |
US10136856B2 (en) | 2016-06-27 | 2018-11-27 | Facense Ltd. | Wearable respiration measurements system |
US10130299B2 (en) | 2015-06-14 | 2018-11-20 | Facense Ltd. | Neurofeedback eyeglasses |
US10299717B2 (en) | 2015-06-14 | 2019-05-28 | Facense Ltd. | Detecting stress based on thermal measurements of the face |
US9968264B2 (en) | 2015-06-14 | 2018-05-15 | Facense Ltd. | Detecting physiological responses based on thermal asymmetry of the face |
US10045699B2 (en) | 2015-06-14 | 2018-08-14 | Facense Ltd. | Determining a state of a user based on thermal measurements of the forehead |
US10130261B2 (en) | 2015-06-14 | 2018-11-20 | Facense Ltd. | Detecting physiological responses while taking into account consumption of confounding substances |
US10159411B2 (en) | 2015-06-14 | 2018-12-25 | Facense Ltd. | Detecting irregular physiological responses during exposure to sensitive data |
US10045737B2 (en) | 2015-06-14 | 2018-08-14 | Facense Ltd. | Clip-on device with inward-facing cameras |
US10524696B2 (en) | 2015-06-14 | 2020-01-07 | Facense Ltd. | Virtual coaching based on respiration signals |
US10151636B2 (en) | 2015-06-14 | 2018-12-11 | Facense Ltd. | Eyeglasses having inward-facing and outward-facing thermal cameras |
US10076270B2 (en) | 2015-06-14 | 2018-09-18 | Facense Ltd. | Detecting physiological responses while accounting for touching the face |
US10524667B2 (en) | 2015-06-14 | 2020-01-07 | Facense Ltd. | Respiration-based estimation of an aerobic activity parameter |
US10216981B2 (en) | 2015-06-14 | 2019-02-26 | Facense Ltd. | Eyeglasses that measure facial skin color changes |
US10154810B2 (en) | 2015-06-14 | 2018-12-18 | Facense Ltd. | Security system that detects atypical behavior |
US10136852B2 (en) | 2015-06-14 | 2018-11-27 | Facense Ltd. | Detecting an allergic reaction from nasal temperatures |
US10076250B2 (en) | 2015-06-14 | 2018-09-18 | Facense Ltd. | Detecting physiological responses based on multispectral data from head-mounted cameras |
US10045726B2 (en) | 2015-06-14 | 2018-08-14 | Facense Ltd. | Selecting a stressor based on thermal measurements of the face |
US10064559B2 (en) | 2015-06-14 | 2018-09-04 | Facense Ltd. | Identification of the dominant nostril using thermal measurements |
DE102016110902A1 (en) | 2015-06-14 | 2016-12-15 | Facense Ltd. | Head-mounted devices for recording thermal readings |
US10085685B2 (en) | 2015-06-14 | 2018-10-02 | Facense Ltd. | Selecting triggers of an allergic reaction based on nasal temperatures |
WO2017048898A1 (en) * | 2015-09-18 | 2017-03-23 | Mazur Kai | Human-computer interface |
JP6742405B2 (en) * | 2015-09-29 | 2020-08-19 | バイナリーヴィーアール, インコーポレイテッドBinaryvr, Inc. | Head-mounted display with facial expression detection function |
CN105472224B (en) * | 2016-01-05 | 2017-03-15 | 北京度量科技有限公司 | A kind of passive optical motion capture equipment and its application |
CA3015658A1 (en) | 2016-03-11 | 2017-09-14 | Magic Leap, Inc. | Structure learning in convolutional neural networks |
US10701253B2 (en) | 2017-10-20 | 2020-06-30 | Lucasfilm Entertainment Company Ltd. | Camera systems for motion capture |
WO2020236993A1 (en) | 2019-05-21 | 2020-11-26 | Magic Leap, Inc. | Hand pose estimation |
WO2021257077A1 (en) * | 2020-06-18 | 2021-12-23 | Hewlett-Packard Development Company, L.P. | Emotive avatar animation with combined user pose data |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US3658054A (en) * | 1970-05-11 | 1972-04-25 | Gen Technical Services Inc | Adjustable helmet face mask |
US5089914A (en) * | 1989-10-02 | 1992-02-18 | Eev Limited | Thermal camera arrangement |
US5341181A (en) * | 1992-11-20 | 1994-08-23 | Godard Roger R | Systems and methods for capturing and presentng visual information |
US5553864A (en) * | 1992-05-22 | 1996-09-10 | Sitrick; David H. | User image integration into audiovisual presentation system and methodology |
US5572749A (en) * | 1994-06-28 | 1996-11-12 | The Walt Disney Company | Helmet mounting device and system |
US5635947A (en) * | 1993-08-16 | 1997-06-03 | Agency Of Industrial Science & Technology, Ministry Of International Trade & Industry | Eye movement tracking display |
US5704836A (en) * | 1995-03-23 | 1998-01-06 | Perception Systems, Inc. | Motion-based command generation technology |
Family Cites Families (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4141652A (en) * | 1977-11-25 | 1979-02-27 | Adaptive Optics Associates, Inc. | Sensor system for detecting wavefront distortion in a return beam of light |
US4461553A (en) * | 1982-06-18 | 1984-07-24 | The United States Of America As Represented By The Secretary Of The Navy | Dynamic head motion measuring system |
US4725138A (en) * | 1985-05-22 | 1988-02-16 | Adaptive Optics Associates Incorporated | Optical wavefront sensing system |
US5016282A (en) * | 1988-07-14 | 1991-05-14 | Atr Communication Systems Research Laboratories | Eye tracking image pickup apparatus for separating noise from feature portions |
US5588139A (en) * | 1990-06-07 | 1996-12-24 | Vpl Research, Inc. | Method and system for generating objects for a multi-person virtual world using data flow networks |
US5389790A (en) * | 1991-02-15 | 1995-02-14 | Sri International | Technique for detecting and identifying certain objects |
US5507716A (en) * | 1991-08-21 | 1996-04-16 | The Lucidity Institute, Inc. | Equipment and methods used to induce lucid dreams in sleeping persons |
JP3298072B2 (en) * | 1992-07-10 | 2002-07-02 | ソニー株式会社 | Video camera system |
US5712732A (en) * | 1993-03-03 | 1998-01-27 | Street; Graham Stewart Brandon | Autostereoscopic image display adjustable for observer location and distance |
US5689575A (en) * | 1993-11-22 | 1997-11-18 | Hitachi, Ltd. | Method and apparatus for processing images of facial expressions |
US5481622A (en) * | 1994-03-01 | 1996-01-02 | Rensselaer Polytechnic Institute | Eye tracking apparatus and method employing grayscale threshold values |
US5555090A (en) * | 1994-10-24 | 1996-09-10 | Adaptive Optics Associates | System for dimensioning objects |
-
1997
- 1997-02-06 US US08/795,544 patent/US6121953A/en not_active Expired - Fee Related
-
1998
- 1998-02-06 WO PCT/US1998/002552 patent/WO1998036631A2/en not_active Application Discontinuation
- 1998-02-06 EP EP98906288A patent/EP0938698A4/en not_active Withdrawn
- 1998-02-06 JP JP10536706A patent/JP2001501346A/en active Pending
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US3658054A (en) * | 1970-05-11 | 1972-04-25 | Gen Technical Services Inc | Adjustable helmet face mask |
US5089914A (en) * | 1989-10-02 | 1992-02-18 | Eev Limited | Thermal camera arrangement |
US5553864A (en) * | 1992-05-22 | 1996-09-10 | Sitrick; David H. | User image integration into audiovisual presentation system and methodology |
US5341181A (en) * | 1992-11-20 | 1994-08-23 | Godard Roger R | Systems and methods for capturing and presentng visual information |
US5635947A (en) * | 1993-08-16 | 1997-06-03 | Agency Of Industrial Science & Technology, Ministry Of International Trade & Industry | Eye movement tracking display |
US5572749A (en) * | 1994-06-28 | 1996-11-12 | The Walt Disney Company | Helmet mounting device and system |
US5704836A (en) * | 1995-03-23 | 1998-01-06 | Perception Systems, Inc. | Motion-based command generation technology |
Non-Patent Citations (1)
Title |
---|
See also references of EP0938698A2 * |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
SG109411A1 (en) * | 1998-10-08 | 2005-03-30 | Sony Computer Entertainment Inc | Method of and system for adding information and recording medium |
US6288704B1 (en) | 1999-06-08 | 2001-09-11 | Vega, Vista, Inc. | Motion detection and tracking system to control navigation and display of object viewers |
US8801488B2 (en) | 2012-10-15 | 2014-08-12 | Disney Enterprises, Inc. | Chin strap sensor for triggering control of walk-around characters |
US10445863B2 (en) | 2014-08-04 | 2019-10-15 | Facebook Technologies, Llc | Method and system for reconstructing obstructed face portions for virtual reality environment |
US10593025B2 (en) | 2014-08-04 | 2020-03-17 | Facebook Technologies, Llc | Method and system for reconstructing obstructed face portions for virtual reality environment |
US11238568B2 (en) | 2014-08-04 | 2022-02-01 | Facebook Technologies, Llc | Method and system for reconstructing obstructed face portions for virtual reality environment |
Also Published As
Publication number | Publication date |
---|---|
EP0938698A2 (en) | 1999-09-01 |
US6121953A (en) | 2000-09-19 |
EP0938698A4 (en) | 2001-09-12 |
WO1998036631A3 (en) | 1998-11-19 |
JP2001501346A (en) | 2001-01-30 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US6121953A (en) | Virtual reality system for sensing facial movements | |
US6774869B2 (en) | Teleportal face-to-face system | |
KR102391405B1 (en) | Enhanced optical and perceptual digital eyewear | |
CN105378632B (en) | The oriented user input of user focus control | |
US5321416A (en) | Head-mounted visual display apparatus | |
CN102591016B (en) | Optimized focal area for augmented reality displays | |
CN105452994B (en) | It is preferably watched while dummy object | |
CN102419631B (en) | Fusing virtual content into real content | |
US20100309097A1 (en) | Head mounted 3d display | |
US6400364B1 (en) | Image processing system | |
TWI250311B (en) | Image display device and simulation apparatus | |
KR20160105439A (en) | Systems and methods for gaze-based media selection and editing | |
US11074737B2 (en) | Information processing apparatus and method | |
TW201721228A (en) | Eye gaze responsive virtual reality headset | |
JP2019120953A (en) | Display device assembly | |
JP7177054B2 (en) | Head-mounted display with user head rotation guide | |
EP2470061A1 (en) | Wearable systems for audio, visual and gaze monitoring | |
US20190235246A1 (en) | Method and apparatus for showing emoji on display glasses | |
US20210278671A1 (en) | Head wearable device with adjustable image sensing modules and its system | |
US11045733B2 (en) | Virtual reality | |
KR100917100B1 (en) | Apparatus for displaying three-dimensional image and method for controlling location of display in the apparatus | |
JP5362357B2 (en) | Capture and process facial movement data | |
US20220094883A1 (en) | Augmented reality display device, and apparatus comprising same | |
JP7193539B2 (en) | Data processing | |
GB2534846A (en) | Head-mountable apparatus and systems |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AK | Designated states |
Kind code of ref document: A2 Designated state(s): JP |
|
AL | Designated countries for regional patents |
Kind code of ref document: A2 Designated state(s): AT BE CH DE DK ES FI FR GB GR IE IT LU MC NL PT SE |
|
ENP | Entry into the national phase |
Ref country code: JP Ref document number: 1998 536706 Kind code of ref document: A Format of ref document f/p: F |
|
WWE | Wipo information: entry into national phase |
Ref document number: 1998906288 Country of ref document: EP |
|
AK | Designated states |
Kind code of ref document: A3 Designated state(s): JP |
|
AL | Designated countries for regional patents |
Kind code of ref document: A3 Designated state(s): AT BE CH DE DK ES FI FR GB GR IE IT LU MC NL PT SE |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application | ||
WWP | Wipo information: published in national office |
Ref document number: 1998906288 Country of ref document: EP |
|
NENP | Non-entry into the national phase |
Ref country code: CA |
|
WWW | Wipo information: withdrawn in national office |
Ref document number: 1998906288 Country of ref document: EP |