US20150227211A1 - Gesture input system, method, and program - Google Patents
Gesture input system, method, and program Download PDFInfo
- Publication number
- US20150227211A1 US20150227211A1 US14/692,318 US201514692318A US2015227211A1 US 20150227211 A1 US20150227211 A1 US 20150227211A1 US 201514692318 A US201514692318 A US 201514692318A US 2015227211 A1 US2015227211 A1 US 2015227211A1
- Authority
- US
- United States
- Prior art keywords
- individual
- gesture
- gesture input
- input method
- motion information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
Definitions
- the subject matter of the present application relates to an interface for input to a computer.
- Multi-modal refers to multiple modes and a multi-modal interface refers to an interface using multiple media.
- a multi-modal interface refers to an interface combining voice information and video information.
- nonverbal interfaces using nonverbal information rather than using voice information are known (Non-Patent Document 1).
- Non-Patent Document 1 Takao Kurokawa “Non-verbal Interface Human Communication Engineering Series,” Ohmsha, 1994.
- an input interface using a gesture such as a human gesture or hand gesture
- a gesture such as a human gesture or hand gesture
- an input interface used when playing a game using motions on a whole-body in the field of amusement is known.
- use of gestures in an interface for input to a general computer has also been studied; however, the related-art gesture input has a room for improvement in terms of ease of use.
- an object of the present subject matter is to provide an input interface that uses human gestures (motions) and is natural for a human and operable intuitively.
- a gesture input system includes: motion information acquisition means for acquiring motion information based on a gesture; and input interface means for generating information for operating an object on a desktop of a computer on the basis of motion information acquired by the motion information acquisition means.
- the gesture input system preferably further includes template storage means for storing a template for recognizing a motion of a user.
- the input interface means preferably includes: template matching means for matching motion information acquired by the motion information acquisition means against a template stored in the template storage means; event output means for outputting an output event associated with a template matched by the template matching means; and operation information generation means for generating information for operating an object on a desktop on the basis of an output event outputted by the event output means.
- the object preferably includes a pie menu in which a menu item is disposed in a circular form, and a user preferably selects any menu item in the pie menu by a predetermined gesture.
- the operation information generation means preferably generates information for selecting a menu item in the pie menu in accordance with an angle at which a user twists a wrist thereof.
- the pie menu is preferably layered.
- a gesture input method includes: a motion information acquisition step of acquiring motion information based on a gesture; and a step of generating information for operating an object on a desktop on the basis of motion information acquired in the motion information acquisition step.
- a program according to the present application is characterized in that the program causes a computer to perform the steps of the gesture input method.
- the program according to the present application is distributed as a program product.
- the program is installed or loaded into a computer via various storage media such as an optical disk, e.g., a CD-ROM, a magnetic disk, and a semiconductor memory or by downloading the program via a communication network or the like.
- a “means” does not simply mean a physical means and a function of the means may be realized by software. Also, a function of one means may be realized by two or more physical means or functions of two or more means may be realized by one physical means.
- gesture input system By adopting the gesture input system according to the present application, it is made possible to provide an input interface that a human can operate naturally and intuitively using a gesture, which is one of information communication means that we usually use in a casual manner. Also, an input interface that does not require any special training and is easy to use for anyone is realized. Accordingly, such an input interface can become a base technology for next-generation input interfaces.
- FIG. 1 is a block diagram showing an overall configuration of a gesture input system 100 .
- FIGS. 2A and 2B include drawings showing the disposition of markers attached to the whole-body of a capture target.
- FIG. 3 is a drawing showing a coordinate system of capture data acquired by a motion capture system 12 .
- FIG. 4 is a diagram showing a structure of capture data acquired by the motion capture system 12 .
- FIG. 5 is a flowchart of a motion recognition process performed by an interface system 18 .
- FIGS. 6A-6D include diagrams showing examples of normalization.
- FIG. 7 is a diagram showing an example of a tree structure.
- FIGS. 8A-8D include drawings showing motions corresponding to a start template and an end template.
- FIG. 9 is a drawing showing parameters used when performing template-matching.
- FIG. 10 is a template corresponding to a jump motion.
- FIGS. 11A and 11B include drawings showing a cursor operation based on gesture input.
- FIGS. 12A-12C include drawings showing an operation method for changing the window size using both hands.
- FIGS. 13A-13C include drawings showing an operation method for changing the window size using one hand.
- FIGS. 14A and 14B include drawings showing an example of a pie menu.
- FIGS. 15A-15C include drawings showing an operation method for selecting a menu in a pie menu.
- FIG. 1 is a block diagram showing an overall configuration of a gesture input system 100 according to the subject matter of the present application.
- the gesture input system 100 is a system that functions as motion information acquisition means and input interface means, and includes multiple cameras 10 , a motion capture system 12 , a workstation 14 , a realtime engine 16 , an interface system 18 , and a client computer (not shown).
- the motion capture system 12 is a system that acquires three-dimensional information indicating a motion of a capture target from sensors or the like attached to the whole-body of a human who is the capture target.
- the motion capture system 12 photographs the capture target from different directions using special spheres (markers) for reflecting light and the multiple cameras 10 provided in predetermined positions and combines two-dimensional images (two-dimensional coordinate positions of the markers) obtained from the cameras to calculate data indicating the coordinates of the markers in a three-dimensional space to acquire motion information.
- the workstation 14 includes data processing software for shaping and converting capture data collected by the motion capture system 12 .
- This data processing software executes processes such as conversion (re-construction) of two-dimensional information into three-dimensional information and interpolation or deletion for correcting data missed when a marker falls into a blind spot of a cameral 10 . This makes it possible to acquire capture data in real time.
- the realtime engine 16 is a processing mechanism for generating the flow of data in real time.
- the interface system 18 is allowed to acquire capture data in real time through the mediation of the realtime engine 16 .
- the interface system (input interface means) 18 outputs a motion event to a client (not shown) on the basis of capture data acquired from the realtime engine 16 .
- the client performs a desktop operation or the like on the basis of the acquired motion event. The details will be described later.
- Vicon8i is used as the motion capture system 12 .
- the workstation 14 includes Vicon iQ. Tarsus is used as the realtime engine 16 .
- Dedicated or general-purpose computers including a processing unit such as a CPU, a memory such as a ROM or a RAM, an external storage unit for storing various types of information, an input/output interface, and a communication interface, and the like may be used as the motion capture system 12 , workstation 14 , realtime engine 16 , interface system 18 , and client computer.
- the systems and the like may be each made up of a single computer or multiple computers. Conversely, a single computer may include the functions of the multiple systems.
- FIG. 2 includes drawings showing the disposition of the markers attached to the whole-body of the capture target.
- the markers are disposed in 44 locations on the whole-body as shown in FIG. 2 ; however, the disposition and number of the markers are not limited to those shown in FIG. 2 .
- the real time nature at the time when capture data is used as an input interface is an important factor. Accordingly, it is preferable to omit unnecessary information to reduce the weight of the capture data while maintaining the quality of motion capture. For example, by using body data calculated by adding information about the skeleton of the human to information acquired by the markers attached to the whole-body of the capture target, the amount of information is made smaller than the amount of marker data and information about the motion of the human is more directly used.
- FIG. 3 is a drawing showing a coordinate system of capture data acquired by the motion capture system 12 .
- the capture data is acquired as data based on a global coordinate system using the center of a capture area as the origin point.
- FIG. 4 is a diagram showing a structure of capture data acquired by the motion capture system 12 .
- data 22 including a frame number 20 of the currently acquired data, a subject name that has been acquired by each marker and indicates the name of the measurement target, a label name indicating which marker's information the information is, and information indicating measured three-dimensional coordinates is acquired for each frame.
- FIG. 5 is a flowchart of a motion recognition process performed by the interface system 18 .
- the interface system 18 acquires data from the realtime engine 16 in real time (S 31 ), and then performs a preprocess of shaping the data so that the data is easily used for motion recognition (S 32 ). Subsequently, the interface system 18 matches the data against a template prepared in advance so as to recognize an intended motion (S 33 ). If the data is matched with the template, the interface system 18 outputs a motion event associated with the template (S 34 ).
- Step S 32 Preprocess
- FIG. 6 includes diagrams showing examples of normalization.
- FIG. 7 is a diagram showing an example of a tree structure.
- FIGS. 6(A) to 6(C) are pieces of body data indicating figures specific to individuals and
- FIG. 6(D) is a piece of body data indicating a normal figure provided as a base model in advance.
- the pieces of body data shown in FIGS. 6(A) to 6(C) are subjected to this normalization process so that the pieces of body data are corrected so as to come close to the normal figure shown in FIG. 6(D) .
- areas of a human body acquired from body data are defined in advance using a tree structure shown in FIG. 7 . Also, by defining the body height of a base model and the distance between each parent node and child node in the tree structure, the ratio between the upper-body and lower-body and the lengths of limbs are determined in advance.
- a basic pose of the capture target is acquired to obtain characteristics of the capture target. Then, the distances between each parent and child in the tree structure obtained from data about the body of the capture target are obtained on the basis of the defined body height of the base model so as to obtain the conversion ratio for each node. This process is performed when initialization is performed when capturing a motion of each capture target.
- a vector (unit vector) based on the positional relation between a parent and a child in the tree structure is obtained with respect to acquired data for each acquisition frame, and the positional relation is modified on the basis of the conversion ratio and the vector and is matched with the base model.
- a foot and toes (foot and toes shown in FIG. 7 ) it is preferable to consider only an influence on an X axis component and a Y axis component (except for a height component) with respect to a parent node (femur shown in FIG. 7 ) in order to prevent a swaying of the foot.
- Data acquired from the realtime engine 16 is data based on the global coordinate system using the center of the capture area as the origin point. For this reason, coordinate conversion is performed, that is, the coordinate system of this data is converted into a local coordinate system fixing the vision line direction (orientation of the body) and using the standing position as the origin point.
- this coordinate conversion is performed as follows. That is, first, an X axis component and a Y axis component of the acquired data except for a Z axis component (height component) thereof are converted into a coordinate system using a waist, which is the center of the capture target, as the origin point. Next, coordinate conversion is performed so that the vision line direction (orientation of the body) becomes the Y axis forward direction.
- the center of the capture target is used as the origin point and the vision line direction is fixed to the Y axis forward direction in the local coordinate system. Accordingly, it is possible to evaluate each acquired data uniformly without being influenced by the standing position or the orientation of the body even if the capture target makes a motion in any direction in any place.
- Step S 33 Template Matching
- the recognition result is generated as a motion event and transmitted to a client.
- whether the acquired data is matched with any of templates previously stored in a storage unit (template storage means) is determined (template matching means). If the acquired data is matched with any template, an event corresponding to the template is generated (event output means). Also, by preparing a start template and an end template, it is made possible to explicitly recognize the start and end of a motion to acquire an intended motion event.
- FIG. 8 is a drawing showing motions corresponding to the start template and end template.
- the lateral axes shown in FIG. 8 represent the time.
- the capture target makes a jump motion ( FIG. 8(A) ).
- an motion start event is generated (true in FIG. 8(D) ).
- no subsequent motion start event is generated until the acquired data is matched with the end template and thus a motion end event is generated.
- a motion end event is generated (JUMP_OFF: false in FIG. 8 (C))
- the state becomes a state in which a motion start event can be generated (false in FIG. 8(D) ).
- FIG. 9 is a drawing parameters used when performing template-matching.
- five elements that is, position, velocity, acceleration, distance, and direction are used as parameters when performing template-matching.
- the interface system 18 acquires these parameters for each frame. In this case, the interface system 18 acquires the parameters such as velocity acquired by performing a calculation extending over multiple frames. Thus, the interface system 18 is allowed to grasp the flow of a motion.
- FIG. 10 is a template corresponding to a jump motion as an example of a template.
- Each template includes a start template starting with “ ⁇ event name” and an end template starting with “-event name”.
- a label of body data, an evaluation parameter, and an evaluation value are sequentially described with respect to each of the start template and end template.
- the respective meanings of rows of the template corresponding to a jump motion are as follows.
- the vector of the head is equal to or more than the threshold thereof in the Z direction
- the vector of the waist is equal to or more than the threshold thereof in the Z direction
- the position of the right leg is equal to or more than the threshold thereof in the Z direction
- the position of the left leg is equal to or more than the threshold thereof in the Z direction
- the position of the left leg is equal to or less than the threshold in the Z direction
- an input interface process for generating information for performing a cursor operation or a window operation in real time using motion information acquired by the gesture input system 100
- operation information generation means for generating information for performing a cursor operation or a window operation in real time using motion information acquired by the gesture input system 100
- a window operation performed under a virtual desktop environment using a hand gesture will be described.
- two cursors are displayed on a screen (desktop), the cursors are associated with the left hand and right hand, respectively, in advance, and the cursors are operated on the basis of information indicating the coordinates of the right and left hands.
- Each cursor has a normal state and a selected state and the state of each cursor is shifted from the normal state to the selected state on the basis of a finger-pointing operation.
- FIG. 11 includes drawings showing a cursor operation based on gesture input. As shown in FIG. 11(A) , a cursor is moved with the palm of a hand directed downward. Then, by twisting the wrist when the cursor is placed in the selected state, the selection operation is performed ( FIG. 11(B) ).
- the selected operation may be performed by making a punch motion using one hand while operating the cursor on the basis of information indicating the coordinates of the other hand.
- the selected operation is preferably performed regardless of the state (normal state, selected state) of the cursor.
- FIG. 12 includes drawings showing an operation method for changing the window size using both hands.
- FIG. 12(B) by making a gesture in which the right and left hands are aligned, the window is shifted to a size change state.
- the window size is increased ( FIG. 12(A) ).
- the window size is reduced ( FIG. 12(C) ).
- the window size can be changed in three ways, that is, the lateral width thereof, the longitudinal width thereof, and both the longitudinal and lateral widths thereof can be changed.
- the lateral width is changed.
- the longitudinal width By moving both hands longitudinally, the longitudinal width is changed.
- both hands obliquely both the longitudinal and lateral widths are changed.
- FIG. 13 includes drawings showing an operation method for changing the window size using one hand.
- the window is shifted to the size change state by twisting the wrist, then, the coordinates of the hand in the size change state are used as the base point (FIG. 13 (B)), and then the widow size is increased by moving the hand toward the front of the base point ( FIG. 13(A) or the window size is reduced by moving the hand toward the rear of the base point ( FIG. 13(C) ).
- a specified window is maximized by making a punch motion, and the window is restored to its original size by making a punch motion in the maximized state.
- the window can be moved in the direction in which the hand is moved.
- a specified window can be closed.
- a gesture by using a gesture, not only information including the upward and downward directions and the left and right directions but three-dimensional information also including the forward and backward directions can be inputted. Accordingly, a movement toward the depth between objects can also be made. For example, among stacked windows, the back window is displayed in the front. In contrast, the front window can be hidden on the back.
- FIG. 14 includes drawings showing an example of a pie menu.
- a pie menu includes multiple pies (menu items) forming a circle and the pies are displayed in such a manner that the circle is divided from the center thereof. While the number of menu items in a pie menu is limited and thus selectable functions are limited, more functions can be included by providing pie menus in layers as shown in FIG. 14(B) .
- the names thereof are provided with predetermined marks, such as ⁇ , indicating that the menu items each have a menu in a lower layer.
- this pie menu is not displayed on the screen during normal operation, it is displayed in a popup form on the basis of a user's motion such as raising of the right leg.
- FIG. 15 includes drawings showing an operation method for selecting a menu in a pie menu.
- a user selects each pie using the angle of a wrist thereof.
- FIG. 15(B) when the palm of the hand is directed downward, the upper pie is selected.
- FIG. 15(A) by twisting the wrist clockwise, a pie is selected clockwise in accordance with the turned angle.
- FIG. 15(C) by twisting the wrist counterclockwise, a pie is selected counterclockwise in accordance with the turned angle.
- the user confirms the currently selected pie by making a punch motion and then performs a process corresponding to a menu item of the pie.
- a predetermined pie has been already selected immediately after a pie menu pops up, and by twisting the wrist clockwise, a pie may be sequentially selected clockwise, or by twisting the wrist counterclockwise, a pie may be sequentially selected counterclockwise.
- pie menus are layered and if a selected and then confirmed menu item has a menu in a lower layer, the pie menu in the lower layer is displayed. If the user returns from a child (lower layer) pie menu to a parent (upper layer) pie menu, the user returns to the parent pie menu by largely moving the position of the hand and then performing a selection operation. If the user makes the same motion when a root pie menu is displayed, the pie menu disappears.
- the recognition accuracy of the motion capture system 12 is reduced due to missing of information, or the like with respect to a body area such as the tip of a hand or a leg. For this reason, it is preferable that the client reads numerical information obtained by three-dimensionally vectorizing the inclination of the wrist so as to interpolate the missing data.
- an input interface that a human can operate naturally and intuitively can be provided by using a gesture, which is one of information communication means that we usually use in a casual manner, and thus using, as input information, not only two-dimensional information including the upward and downward directions and the left and right directions but also three-dimensional information additionally including the forward and backward directions (depth).
- a gesture which is one of information communication means that we usually use in a casual manner, and thus using, as input information, not only two-dimensional information including the upward and downward directions and the left and right directions but also three-dimensional information additionally including the forward and backward directions (depth).
- an input interface that does not require any special training and is easy to use for anyone is realized.
- this input interface is operable from a place remote from a panel.
- Motion information may be acquired using other methods. While motion information based on a hand gesture is acquired using the motion capture system in this embodiment, motion information based on a hand gesture may be acquired using, for example, a data glove or the like or using both the motion capture system and a data glove or the like.
- the present invention is not limited thereto.
- An input interface that captures movements on the whole-body can be mounted.
Abstract
Description
- The present application is a continuation of U.S. application Ser. No. 12/440,150, filed on Dec. 28, 2009, which is a U.S. national stage application under 35 USC §371 claiming the benefit of International Application No. PCT/JP2006/317730, filed on Sep. 7, 2006, the entire contents of which are incorporated herein by reference in their entirety.
- The subject matter of the present application relates to an interface for input to a computer.
- While various apparatuses such as a keyboard, a mouse, and a touch panel have been used as an interface for input to a computer, other various input interfaces have also been proposed so as to respond to voices of various users.
- Under these circumstances, multi-modal interfaces have been noted in recent years. “Multi-modal” refers to multiple modes and a multi-modal interface refers to an interface using multiple media. Typically, a multi-modal interface refers to an interface combining voice information and video information. Also, nonverbal interfaces using nonverbal information rather than using voice information are known (Non-Patent Document 1).
- [Non-Patent Document 1] Takao Kurokawa “Non-verbal Interface Human Communication Engineering Series,” Ohmsha, 1994.
- As for nonverbal interfaces, an input interface using a gesture such as a human gesture or hand gesture has been proposed. For example, an input interface used when playing a game using motions on a whole-body in the field of amusement is known. Also, use of gestures in an interface for input to a general computer has also been studied; however, the related-art gesture input has a room for improvement in terms of ease of use.
- The subject matter of the present application was made in view of the foregoing. Accordingly, an object of the present subject matter is to provide an input interface that uses human gestures (motions) and is natural for a human and operable intuitively.
- In order to solve the above-mentioned problem, a gesture input system according to the present application includes: motion information acquisition means for acquiring motion information based on a gesture; and input interface means for generating information for operating an object on a desktop of a computer on the basis of motion information acquired by the motion information acquisition means.
- The gesture input system preferably further includes template storage means for storing a template for recognizing a motion of a user. The input interface means preferably includes: template matching means for matching motion information acquired by the motion information acquisition means against a template stored in the template storage means; event output means for outputting an output event associated with a template matched by the template matching means; and operation information generation means for generating information for operating an object on a desktop on the basis of an output event outputted by the event output means.
- Also, the object preferably includes a pie menu in which a menu item is disposed in a circular form, and a user preferably selects any menu item in the pie menu by a predetermined gesture.
- Also, the operation information generation means preferably generates information for selecting a menu item in the pie menu in accordance with an angle at which a user twists a wrist thereof. Also, the pie menu is preferably layered.
- A gesture input method according to the present application includes: a motion information acquisition step of acquiring motion information based on a gesture; and a step of generating information for operating an object on a desktop on the basis of motion information acquired in the motion information acquisition step.
- A program according to the present application is characterized in that the program causes a computer to perform the steps of the gesture input method. The program according to the present application is distributed as a program product. The program is installed or loaded into a computer via various storage media such as an optical disk, e.g., a CD-ROM, a magnetic disk, and a semiconductor memory or by downloading the program via a communication network or the like.
- In this specification and the like, a “means” does not simply mean a physical means and a function of the means may be realized by software. Also, a function of one means may be realized by two or more physical means or functions of two or more means may be realized by one physical means.
- By adopting the gesture input system according to the present application, it is made possible to provide an input interface that a human can operate naturally and intuitively using a gesture, which is one of information communication means that we usually use in a casual manner. Also, an input interface that does not require any special training and is easy to use for anyone is realized. Accordingly, such an input interface can become a base technology for next-generation input interfaces.
- An embodiment of the present invention will be described in detail with reference to the accompanying drawings. Like elements are assigned like reference numerals and will not be repeatedly described.
-
FIG. 1 is a block diagram showing an overall configuration of agesture input system 100. -
FIGS. 2A and 2B include drawings showing the disposition of markers attached to the whole-body of a capture target. -
FIG. 3 is a drawing showing a coordinate system of capture data acquired by amotion capture system 12. -
FIG. 4 is a diagram showing a structure of capture data acquired by themotion capture system 12. -
FIG. 5 is a flowchart of a motion recognition process performed by aninterface system 18. -
FIGS. 6A-6D include diagrams showing examples of normalization. -
FIG. 7 is a diagram showing an example of a tree structure. -
FIGS. 8A-8D include drawings showing motions corresponding to a start template and an end template. -
FIG. 9 is a drawing showing parameters used when performing template-matching. -
FIG. 10 is a template corresponding to a jump motion. -
FIGS. 11A and 11B include drawings showing a cursor operation based on gesture input. -
FIGS. 12A-12C include drawings showing an operation method for changing the window size using both hands. -
FIGS. 13A-13C include drawings showing an operation method for changing the window size using one hand. -
FIGS. 14A and 14B include drawings showing an example of a pie menu. -
FIGS. 15A-15C include drawings showing an operation method for selecting a menu in a pie menu. -
-
- 10 camera
- 12 motion capture system
- 14 workstation
- 16 realtime engine
- 18 interface system (input interface means)
- 20 frame number
- 22 data (subject name, label name, and coordinate information)
- 100 gesture input system
-
FIG. 1 is a block diagram showing an overall configuration of agesture input system 100 according to the subject matter of the present application. As shown inFIG. 1 , thegesture input system 100 is a system that functions as motion information acquisition means and input interface means, and includesmultiple cameras 10, amotion capture system 12, aworkstation 14, arealtime engine 16, aninterface system 18, and a client computer (not shown). - The
motion capture system 12 is a system that acquires three-dimensional information indicating a motion of a capture target from sensors or the like attached to the whole-body of a human who is the capture target. In this embodiment, themotion capture system 12 photographs the capture target from different directions using special spheres (markers) for reflecting light and themultiple cameras 10 provided in predetermined positions and combines two-dimensional images (two-dimensional coordinate positions of the markers) obtained from the cameras to calculate data indicating the coordinates of the markers in a three-dimensional space to acquire motion information. - The
workstation 14 includes data processing software for shaping and converting capture data collected by themotion capture system 12. This data processing software executes processes such as conversion (re-construction) of two-dimensional information into three-dimensional information and interpolation or deletion for correcting data missed when a marker falls into a blind spot of a cameral 10. This makes it possible to acquire capture data in real time. - The
realtime engine 16 is a processing mechanism for generating the flow of data in real time. Theinterface system 18 is allowed to acquire capture data in real time through the mediation of therealtime engine 16. - The interface system (input interface means) 18 outputs a motion event to a client (not shown) on the basis of capture data acquired from the
realtime engine 16. The client performs a desktop operation or the like on the basis of the acquired motion event. The details will be described later. - In this embodiment, Vicon8i is used as the
motion capture system 12. Theworkstation 14 includes Vicon iQ. Tarsus is used as therealtime engine 16. - Dedicated or general-purpose computers including a processing unit such as a CPU, a memory such as a ROM or a RAM, an external storage unit for storing various types of information, an input/output interface, and a communication interface, and the like may be used as the
motion capture system 12,workstation 14,realtime engine 16,interface system 18, and client computer. Also, the systems and the like may be each made up of a single computer or multiple computers. Conversely, a single computer may include the functions of the multiple systems. -
FIG. 2 includes drawings showing the disposition of the markers attached to the whole-body of the capture target. In this embodiment, the markers are disposed in 44 locations on the whole-body as shown inFIG. 2 ; however, the disposition and number of the markers are not limited to those shown inFIG. 2 . - The real time nature at the time when capture data is used as an input interface is an important factor. Accordingly, it is preferable to omit unnecessary information to reduce the weight of the capture data while maintaining the quality of motion capture. For example, by using body data calculated by adding information about the skeleton of the human to information acquired by the markers attached to the whole-body of the capture target, the amount of information is made smaller than the amount of marker data and information about the motion of the human is more directly used.
-
FIG. 3 is a drawing showing a coordinate system of capture data acquired by themotion capture system 12. As shown inFIG. 3 , the capture data is acquired as data based on a global coordinate system using the center of a capture area as the origin point. -
FIG. 4 is a diagram showing a structure of capture data acquired by themotion capture system 12. As shown inFIG. 4 , as for capture data,data 22 including aframe number 20 of the currently acquired data, a subject name that has been acquired by each marker and indicates the name of the measurement target, a label name indicating which marker's information the information is, and information indicating measured three-dimensional coordinates is acquired for each frame. -
FIG. 5 is a flowchart of a motion recognition process performed by theinterface system 18. Theinterface system 18 acquires data from therealtime engine 16 in real time (S31), and then performs a preprocess of shaping the data so that the data is easily used for motion recognition (S32). Subsequently, theinterface system 18 matches the data against a template prepared in advance so as to recognize an intended motion (S33). If the data is matched with the template, theinterface system 18 outputs a motion event associated with the template (S34). Hereafter, the process steps will be described in detail. - (Step S32: Preprocess)
- It is difficult to directly use the capture data acquired from the
realtime engine 16 for motion recognition. For this reason, by performing a preprocess, that is, by performing normalization and coordinate conversion, the capture data is converted into data that is easy to use for motion recognition. - (Normalization)
- In order to directly use capture data acquired from the
realtime engine 16, templates created for individuals must be prepared, since the figures of humans are not uniform. For this reason, the capture data is corrected into a predetermined normal figure so that individual differences are accommodated. By doing so, whatever type of figure a human has, data about the human is handled as data about a human having an identical figure. This makes it possible to apply a single template to a motion. The template will be described later. -
FIG. 6 includes diagrams showing examples of normalization.FIG. 7 is a diagram showing an example of a tree structure.FIGS. 6(A) to 6(C) are pieces of body data indicating figures specific to individuals andFIG. 6(D) is a piece of body data indicating a normal figure provided as a base model in advance. The pieces of body data shown inFIGS. 6(A) to 6(C) are subjected to this normalization process so that the pieces of body data are corrected so as to come close to the normal figure shown inFIG. 6(D) . - Specific normalization steps are as follows.
- First, areas of a human body acquired from body data are defined in advance using a tree structure shown in
FIG. 7 . Also, by defining the body height of a base model and the distance between each parent node and child node in the tree structure, the ratio between the upper-body and lower-body and the lengths of limbs are determined in advance. - Next, a basic pose of the capture target is acquired to obtain characteristics of the capture target. Then, the distances between each parent and child in the tree structure obtained from data about the body of the capture target are obtained on the basis of the defined body height of the base model so as to obtain the conversion ratio for each node. This process is performed when initialization is performed when capturing a motion of each capture target.
- During the motion capture, a vector (unit vector) based on the positional relation between a parent and a child in the tree structure is obtained with respect to acquired data for each acquisition frame, and the positional relation is modified on the basis of the conversion ratio and the vector and is matched with the base model. As for a foot and toes, (foot and toes shown in
FIG. 7 ), it is preferable to consider only an influence on an X axis component and a Y axis component (except for a height component) with respect to a parent node (femur shown inFIG. 7 ) in order to prevent a swaying of the foot. - (Coordinate Conversion)
- Data acquired from the
realtime engine 16 is data based on the global coordinate system using the center of the capture area as the origin point. For this reason, coordinate conversion is performed, that is, the coordinate system of this data is converted into a local coordinate system fixing the vision line direction (orientation of the body) and using the standing position as the origin point. For example, this coordinate conversion is performed as follows. That is, first, an X axis component and a Y axis component of the acquired data except for a Z axis component (height component) thereof are converted into a coordinate system using a waist, which is the center of the capture target, as the origin point. Next, coordinate conversion is performed so that the vision line direction (orientation of the body) becomes the Y axis forward direction. - By performing the above-mentioned coordinate conversion, the center of the capture target is used as the origin point and the vision line direction is fixed to the Y axis forward direction in the local coordinate system. Accordingly, it is possible to evaluate each acquired data uniformly without being influenced by the standing position or the orientation of the body even if the capture target makes a motion in any direction in any place.
- Also, since motions involving travel, such as a walk and a run, may be evaluated more effectively using data based on the global coordinate system, it is desirable that data based on the global coordinate system be also available when recognizing a motion.
- (Step S33: Template Matching)
- In this embodiment, in order to make it easy to use a recognition result when constructing an interface on the basis of the recognition result, the recognition result is generated as a motion event and transmitted to a client. At that time, whether the acquired data is matched with any of templates previously stored in a storage unit (template storage means) is determined (template matching means). If the acquired data is matched with any template, an event corresponding to the template is generated (event output means). Also, by preparing a start template and an end template, it is made possible to explicitly recognize the start and end of a motion to acquire an intended motion event. That is, if the acquired data is matched with the start template in a state in which an event can be generated, a motion start event is generated and the state is shifted to a state in which an event cannot be generated (state transition). Also, if the acquired data is matched with the end template in a state in which an event cannot be generated, a motion end event is generated and the state is shifted to a state in which an event can be generated.
-
FIG. 8 is a drawing showing motions corresponding to the start template and end template. The lateral axes shown inFIG. 8 represent the time. Here, it is assumed that the capture target makes a jump motion (FIG. 8(A) ). In this case, if acquired data is matched with a template corresponding to the start of a jump motion (JUMP_ON: true in FIG. 8(B)), an motion start event is generated (true inFIG. 8(D) ). Thereafter, no subsequent motion start event is generated until the acquired data is matched with the end template and thus a motion end event is generated. If a motion end event is generated (JUMP_OFF: false in FIG. 8(C)), the state becomes a state in which a motion start event can be generated (false inFIG. 8(D) ). -
FIG. 9 is a drawing parameters used when performing template-matching. In this embodiment, five elements, that is, position, velocity, acceleration, distance, and direction are used as parameters when performing template-matching. Theinterface system 18 acquires these parameters for each frame. In this case, theinterface system 18 acquires the parameters such as velocity acquired by performing a calculation extending over multiple frames. Thus, theinterface system 18 is allowed to grasp the flow of a motion. -
FIG. 10 is a template corresponding to a jump motion as an example of a template. Each template includes a start template starting with “<event name” and an end template starting with “-event name”. A label of body data, an evaluation parameter, and an evaluation value are sequentially described with respect to each of the start template and end template. The respective meanings of rows of the template corresponding to a jump motion are as follows. - 1: A start template corresponding to a “JUMP” motion
- 2: The acceleration of the head is equal to or more than the threshold thereof
- 3: The vector of the head is equal to or more than the threshold thereof in the Z direction
- 4: The acceleration of the waist is equal to or more than the threshold thereof
- 5: The vector of the waist is equal to or more than the threshold thereof in the Z direction
- 6: The position of the right leg is equal to or more than the threshold thereof in the Z direction
- 7: The position of the left leg is equal to or more than the threshold thereof in the Z direction
- (whether the head and waist are raised with accelerations more than the thresholds thereof and both legs are floating is determined on the basis of the above-mentioned start template, and if all these conditions are met, the start of a jump is determined)
- 8: An end template of “JUMP”
- 9: The position of the right leg is equal to or less than the threshold in the Z direction
- 10: The position of the left leg is equal to or less than the threshold in the Z direction
- (whether both legs are landed on the ground is determined on the basis of the above-mentioned end template, and if all these conditions are met, the end of the jump is determined)
- Next, an input interface process (operation information generation means) for generating information for performing a cursor operation or a window operation in real time using motion information acquired by the
gesture input system 100 will be described. In this embodiment, a window operation performed under a virtual desktop environment using a hand gesture will be described. - In this embodiment, two cursors are displayed on a screen (desktop), the cursors are associated with the left hand and right hand, respectively, in advance, and the cursors are operated on the basis of information indicating the coordinates of the right and left hands. Each cursor has a normal state and a selected state and the state of each cursor is shifted from the normal state to the selected state on the basis of a finger-pointing operation. By providing each cursor with the normal state and selected state, an unintended event can be prevented from occurring during operation. Also, by making a wrist twisting motion when the cursor is placed in the selected state, an operation (hereafter referred to as a “selected operation”) equivalent to a click of a mouse is performed. Also, by moving a hand in the selected operation state, an operation equivalent to a drag is performed.
-
FIG. 11 includes drawings showing a cursor operation based on gesture input. As shown inFIG. 11(A) , a cursor is moved with the palm of a hand directed downward. Then, by twisting the wrist when the cursor is placed in the selected state, the selection operation is performed (FIG. 11(B) ). - As another embodiment, the selected operation may be performed by making a punch motion using one hand while operating the cursor on the basis of information indicating the coordinates of the other hand. In this case, the selected operation is preferably performed regardless of the state (normal state, selected state) of the cursor.
- Next, as an operation method that any existing desktop interface does not have, a method for operating each object (window) without performing a cursor operation will be described.
-
FIG. 12 includes drawings showing an operation method for changing the window size using both hands. As shown inFIG. 12(B) , by making a gesture in which the right and left hands are aligned, the window is shifted to a size change state. By increasing the width between the left hand and right hand in the size change state, the window size is increased (FIG. 12(A) ). By reducing the width between the left hand and right hand, the window size is reduced (FIG. 12(C) ). The window size can be changed in three ways, that is, the lateral width thereof, the longitudinal width thereof, and both the longitudinal and lateral widths thereof can be changed. By moving both hands laterally, the lateral width is changed. By moving both hands longitudinally, the longitudinal width is changed. By moving both hands obliquely, both the longitudinal and lateral widths are changed. -
FIG. 13 includes drawings showing an operation method for changing the window size using one hand. First, the window is shifted to the size change state by twisting the wrist, then, the coordinates of the hand in the size change state are used as the base point (FIG. 13(B)), and then the widow size is increased by moving the hand toward the front of the base point (FIG. 13(A) or the window size is reduced by moving the hand toward the rear of the base point (FIG. 13(C) ). - In addition to the changes in the window size, various operations can be realized using gesture input. For example, a specified window is maximized by making a punch motion, and the window is restored to its original size by making a punch motion in the maximized state. Also, by making a gesture for clearing away the window, the window can be moved in the direction in which the hand is moved. Also, by making a motion for waving a hand, a specified window can be closed.
- Also, by using a gesture, not only information including the upward and downward directions and the left and right directions but three-dimensional information also including the forward and backward directions can be inputted. Accordingly, a movement toward the depth between objects can also be made. For example, among stacked windows, the back window is displayed in the front. In contrast, the front window can be hidden on the back.
- Next, a process of selecting a pie menu using this gesture input will be described.
-
FIG. 14 includes drawings showing an example of a pie menu. As shown inFIG. 14(A) , a pie menu includes multiple pies (menu items) forming a circle and the pies are displayed in such a manner that the circle is divided from the center thereof. While the number of menu items in a pie menu is limited and thus selectable functions are limited, more functions can be included by providing pie menus in layers as shown inFIG. 14(B) . As for menu items that are each subdivided, the names thereof are provided with predetermined marks, such as ▴, indicating that the menu items each have a menu in a lower layer. - While this pie menu is not displayed on the screen during normal operation, it is displayed in a popup form on the basis of a user's motion such as raising of the right leg.
-
FIG. 15 includes drawings showing an operation method for selecting a menu in a pie menu. As shown inFIG. 15 , a user selects each pie using the angle of a wrist thereof. For example, as shown inFIG. 15(B) , when the palm of the hand is directed downward, the upper pie is selected. Also, as shown inFIG. 15(A) , by twisting the wrist clockwise, a pie is selected clockwise in accordance with the turned angle. Also, as shown inFIG. 15(C) , by twisting the wrist counterclockwise, a pie is selected counterclockwise in accordance with the turned angle. After the user selects the desired pie, the user confirms the currently selected pie by making a punch motion and then performs a process corresponding to a menu item of the pie. - As anther embodiment of menu selection in a pie menu, a predetermined pie has been already selected immediately after a pie menu pops up, and by twisting the wrist clockwise, a pie may be sequentially selected clockwise, or by twisting the wrist counterclockwise, a pie may be sequentially selected counterclockwise.
- If pie menus are layered and if a selected and then confirmed menu item has a menu in a lower layer, the pie menu in the lower layer is displayed. If the user returns from a child (lower layer) pie menu to a parent (upper layer) pie menu, the user returns to the parent pie menu by largely moving the position of the hand and then performing a selection operation. If the user makes the same motion when a root pie menu is displayed, the pie menu disappears.
- While a pie menu is operated by rotating the wrist, the recognition accuracy of the
motion capture system 12 is reduced due to missing of information, or the like with respect to a body area such as the tip of a hand or a leg. For this reason, it is preferable that the client reads numerical information obtained by three-dimensionally vectorizing the inclination of the wrist so as to interpolate the missing data. - As described above in detail, by adopting the gesture input system described herein, an input interface that a human can operate naturally and intuitively can be provided by using a gesture, which is one of information communication means that we usually use in a casual manner, and thus using, as input information, not only two-dimensional information including the upward and downward directions and the left and right directions but also three-dimensional information additionally including the forward and backward directions (depth). In this case, by adopting an operation method similar to a natural movement, an input interface that does not require any special training and is easy to use for anyone is realized. Also, unlike a touch panel, this input interface is operable from a place remote from a panel.
- The presently described subject matter is not limited to the above-mentioned embodiment and can be carried out in various ways without departing from the spirit and scope of the invention. For this reason, the above-mentioned embodiment is illustrative only in any respects and should not be construed in a limited manner. For example, the order of the above-mentioned process steps may be arbitrarily changed or any process steps may be performed simultaneously unless any contradiction occurs between processes.
- While information indicating a motion of a capture target is acquired using the optical motion capture system in this embodiment, the present invention is not limited thereto. Motion information may be acquired using other methods. While motion information based on a hand gesture is acquired using the motion capture system in this embodiment, motion information based on a hand gesture may be acquired using, for example, a data glove or the like or using both the motion capture system and a data glove or the like.
- While the input interface mainly using a hand gesture has been described in this embodiment, the present invention is not limited thereto. An input interface that captures movements on the whole-body can be mounted.
Claims (20)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/692,318 US20150227211A1 (en) | 2006-09-07 | 2015-04-21 | Gesture input system, method, and program |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/JP2006/317730 WO2008029467A1 (en) | 2006-09-07 | 2006-09-07 | Gesture input system, method and program |
US12/440,150 US9032336B2 (en) | 2006-09-07 | 2006-09-07 | Gesture input system, method and program |
US14/692,318 US20150227211A1 (en) | 2006-09-07 | 2015-04-21 | Gesture input system, method, and program |
Related Parent Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US12/440,150 Continuation US9032336B2 (en) | 2006-09-07 | 2006-09-07 | Gesture input system, method and program |
PCT/JP2006/317730 Continuation WO2008029467A1 (en) | 2006-09-07 | 2006-09-07 | Gesture input system, method and program |
Publications (1)
Publication Number | Publication Date |
---|---|
US20150227211A1 true US20150227211A1 (en) | 2015-08-13 |
Family
ID=39156912
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US12/440,150 Expired - Fee Related US9032336B2 (en) | 2006-09-07 | 2006-09-07 | Gesture input system, method and program |
US14/692,318 Abandoned US20150227211A1 (en) | 2006-09-07 | 2015-04-21 | Gesture input system, method, and program |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US12/440,150 Expired - Fee Related US9032336B2 (en) | 2006-09-07 | 2006-09-07 | Gesture input system, method and program |
Country Status (3)
Country | Link |
---|---|
US (2) | US9032336B2 (en) |
JP (1) | JP4489825B2 (en) |
WO (1) | WO2008029467A1 (en) |
Families Citing this family (36)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP4489825B2 (en) | 2006-09-07 | 2010-06-23 | 学校法人 大阪電気通信大学 | Gesture input system, method and program |
US20080098315A1 (en) * | 2006-10-18 | 2008-04-24 | Dao-Liang Chou | Executing an operation associated with a region proximate a graphic element on a surface |
KR20100101389A (en) * | 2009-03-09 | 2010-09-17 | 삼성전자주식회사 | Display apparatus for providing a user menu, and method for providing ui applied thereto |
EP2452258B1 (en) * | 2009-07-07 | 2019-01-23 | Elliptic Laboratories AS | Control using movements |
US20110054833A1 (en) * | 2009-09-02 | 2011-03-03 | Apple Inc. | Processing motion sensor data using accessible templates |
JP2011081480A (en) * | 2009-10-05 | 2011-04-21 | Seiko Epson Corp | Image input system |
JP2011081469A (en) * | 2009-10-05 | 2011-04-21 | Hitachi Consumer Electronics Co Ltd | Input device |
WO2011056657A2 (en) | 2009-10-27 | 2011-05-12 | Harmonix Music Systems, Inc. | Gesture-based user interface |
US9981193B2 (en) | 2009-10-27 | 2018-05-29 | Harmonix Music Systems, Inc. | Movement based recognition and evaluation |
JP5320332B2 (en) * | 2010-03-19 | 2013-10-23 | 株式会社コナミデジタルエンタテインメント | GAME DEVICE, GAME DEVICE CONTROL METHOD, AND PROGRAM |
CA2735325C (en) * | 2010-03-25 | 2015-01-20 | User Interface In Sweden Ab | System and method for gesture detection and feedback |
FR2959576A1 (en) * | 2010-05-03 | 2011-11-04 | Thomson Licensing | METHOD FOR DISPLAYING A SETTING MENU AND CORRESPONDING DEVICE |
EP2579955B1 (en) | 2010-06-11 | 2020-07-08 | Harmonix Music Systems, Inc. | Dance game and tutorial |
US8562403B2 (en) | 2010-06-11 | 2013-10-22 | Harmonix Music Systems, Inc. | Prompting a player of a dance game |
US9358456B1 (en) | 2010-06-11 | 2016-06-07 | Harmonix Music Systems, Inc. | Dance competition game |
US8416187B2 (en) | 2010-06-22 | 2013-04-09 | Microsoft Corporation | Item navigation using motion-capture data |
US20110317871A1 (en) * | 2010-06-29 | 2011-12-29 | Microsoft Corporation | Skeletal joint recognition and tracking system |
US20120120214A1 (en) * | 2010-11-16 | 2012-05-17 | Braun Gmbh | Product Demonstration |
US9870141B2 (en) * | 2010-11-19 | 2018-01-16 | Microsoft Technology Licensing, Llc | Gesture recognition |
JP2012190183A (en) * | 2011-03-09 | 2012-10-04 | Sony Corp | Image processing device, method, and program |
JP5658618B2 (en) * | 2011-05-16 | 2015-01-28 | パナソニックIpマネジメント株式会社 | Operation input device, program |
US9176608B1 (en) | 2011-06-27 | 2015-11-03 | Amazon Technologies, Inc. | Camera based sensor for motion detection |
US20130019175A1 (en) * | 2011-07-14 | 2013-01-17 | Microsoft Corporation | Submenus for context based menu system |
US9207852B1 (en) * | 2011-12-20 | 2015-12-08 | Amazon Technologies, Inc. | Input mechanisms for electronic devices |
WO2013103410A1 (en) * | 2012-01-05 | 2013-07-11 | California Institute Of Technology | Imaging surround systems for touch-free display control |
WO2014107434A1 (en) | 2013-01-02 | 2014-07-10 | California Institute Of Technology | Single-sensor system for extracting depth information from image blur |
US9442570B2 (en) | 2013-03-13 | 2016-09-13 | Google Technology Holdings LLC | Method and system for gesture recognition |
US10220303B1 (en) | 2013-03-15 | 2019-03-05 | Harmonix Music Systems, Inc. | Gesture-based music game |
JP6141108B2 (en) * | 2013-06-07 | 2017-06-07 | キヤノン株式会社 | Information processing apparatus and method |
US20150091841A1 (en) * | 2013-09-30 | 2015-04-02 | Kobo Incorporated | Multi-part gesture for operating an electronic personal display |
JP6268526B2 (en) | 2014-03-17 | 2018-01-31 | オムロン株式会社 | MULTIMEDIA DEVICE, MULTIMEDIA DEVICE CONTROL METHOD, AND MULTIMEDIA DEVICE CONTROL PROGRAM |
KR102265143B1 (en) * | 2014-05-16 | 2021-06-15 | 삼성전자주식회사 | Apparatus and method for processing input |
WO2016028629A1 (en) | 2014-08-16 | 2016-02-25 | Google Inc. | Identifying gestures using motion data |
US10660039B1 (en) | 2014-09-02 | 2020-05-19 | Google Llc | Adaptive output of indications of notification data |
US9804679B2 (en) | 2015-07-03 | 2017-10-31 | Google Inc. | Touchless user interface navigation using gestures |
US9703387B2 (en) | 2015-08-31 | 2017-07-11 | Konica Minolta Laboratory U.S.A., Inc. | System and method of real-time interactive operation of user interface |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5594469A (en) * | 1995-02-21 | 1997-01-14 | Mitsubishi Electric Information Technology Center America Inc. | Hand gesture machine control system |
US20060010400A1 (en) * | 2004-06-28 | 2006-01-12 | Microsoft Corporation | Recognizing gestures and using gestures for interacting with software applications |
Family Cites Families (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2602743Y2 (en) | 1991-10-04 | 2000-01-24 | 株式会社東芝 | Remote monitoring operation device |
JPH0728591A (en) * | 1993-05-13 | 1995-01-31 | Toshiba Corp | Space manipulation mouse system and space operation pattern input method |
JPH1031551A (en) * | 1996-07-15 | 1998-02-03 | Mitsubishi Electric Corp | Human interface system and high-speed moving body position detecting device using the same |
JPH11110125A (en) | 1997-10-02 | 1999-04-23 | Yuusuke Nonomura | Device and method for remote control |
JP4612928B2 (en) | 2000-01-18 | 2011-01-12 | マイクロストーン株式会社 | Body motion sensing device |
JP2001216069A (en) * | 2000-02-01 | 2001-08-10 | Toshiba Corp | Operation inputting device and direction detecting method |
US6554706B2 (en) * | 2000-05-31 | 2003-04-29 | Gerard Jounghyun Kim | Methods and apparatus of displaying and evaluating motion data in a motion game apparatus |
JP3631151B2 (en) | 2000-11-30 | 2005-03-23 | キヤノン株式会社 | Information processing apparatus, mixed reality presentation apparatus and method, and storage medium |
JP3847634B2 (en) | 2002-02-15 | 2006-11-22 | シャープ株式会社 | Virtual space simulation device |
US7161579B2 (en) * | 2002-07-18 | 2007-01-09 | Sony Computer Entertainment Inc. | Hand-held computer interactive device |
US7895536B2 (en) * | 2003-01-08 | 2011-02-22 | Autodesk, Inc. | Layer editor system for a pen-based computer |
JP2004356819A (en) | 2003-05-28 | 2004-12-16 | Sharp Corp | Remote control apparatus |
JP2006079533A (en) | 2004-09-13 | 2006-03-23 | Fuji Xerox Co Ltd | Information processor, information processing method, and computer program |
CN101308442B (en) * | 2004-10-12 | 2012-04-04 | 日本电信电话株式会社 | 3d pointing method and 3d pointing device |
CN101040242A (en) * | 2004-10-15 | 2007-09-19 | 皇家飞利浦电子股份有限公司 | System for 3D rendering applications using hands |
US7809214B2 (en) * | 2005-08-22 | 2010-10-05 | Samsung Electronics Co., Ltd. | Device and a method for identifying movement patterns |
JP2007109118A (en) * | 2005-10-17 | 2007-04-26 | Hitachi Ltd | Input instruction processing apparatus and input instruction processing program |
WO2007099762A1 (en) * | 2006-03-01 | 2007-09-07 | Nikon Corporation | Object-seeking computer program product, object-seeking device, and camera |
JP4489825B2 (en) | 2006-09-07 | 2010-06-23 | 学校法人 大阪電気通信大学 | Gesture input system, method and program |
-
2006
- 2006-09-07 JP JP2008533010A patent/JP4489825B2/en active Active
- 2006-09-07 US US12/440,150 patent/US9032336B2/en not_active Expired - Fee Related
- 2006-09-07 WO PCT/JP2006/317730 patent/WO2008029467A1/en active Application Filing
-
2015
- 2015-04-21 US US14/692,318 patent/US20150227211A1/en not_active Abandoned
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5594469A (en) * | 1995-02-21 | 1997-01-14 | Mitsubishi Electric Information Technology Center America Inc. | Hand gesture machine control system |
US20060010400A1 (en) * | 2004-06-28 | 2006-01-12 | Microsoft Corporation | Recognizing gestures and using gestures for interacting with software applications |
Also Published As
Publication number | Publication date |
---|---|
US9032336B2 (en) | 2015-05-12 |
JPWO2008029467A1 (en) | 2010-01-21 |
JP4489825B2 (en) | 2010-06-23 |
US20100138785A1 (en) | 2010-06-03 |
WO2008029467A1 (en) | 2008-03-13 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9032336B2 (en) | Gesture input system, method and program | |
US20210019215A1 (en) | System and Method for Error Detection and Correction in Virtual Reality and Augmented Reality Environments | |
JP5845830B2 (en) | Information processing apparatus, display control method, and program | |
CN103093453B (en) | Calculating can change posture and/or the shape of entity | |
US9075434B2 (en) | Translating user motion into multiple object responses | |
CN102184020B (en) | Gestures and gesture modifiers for manipulating a user-interface | |
US10019828B2 (en) | Image generating apparatus, image generating system, and non-transitory computer-readable storage medium | |
US11948376B2 (en) | Method, system, and device of generating a reduced-size volumetric dataset | |
JP4608326B2 (en) | Instruction motion recognition device and instruction motion recognition program | |
KR101519775B1 (en) | Method and apparatus for generating animation based on object motion | |
CN108564643B (en) | Performance capture system based on UE engine | |
CN102262438A (en) | Gestures and gesture recognition for manipulating a user-interface | |
CN103930944A (en) | Adaptive tracking system for spatial input devices | |
CN109308437B (en) | Motion recognition error correction method, electronic device, and storage medium | |
JP2010257081A (en) | Image procession method and image processing system | |
KR101654311B1 (en) | User motion perception method and apparatus | |
CN109395375A (en) | A kind of 3d gaming method of interface interacted based on augmented reality and movement | |
JP2003346150A (en) | Floor surface recognition device, floor surface recognition method, and robot device | |
Varga et al. | Survey and investigation of hand motion processing technologies for compliance with shape conceptualization | |
CN112215928B (en) | Motion capturing method based on visual image and digital animation production method | |
KR102477479B1 (en) | system and method for enabling content-based multi-party games by recognizing and classifying omnidirectional movements of multiple participants without errors using multiple sensors or devices | |
KR102359969B1 (en) | Character motion control apparatus by using motion sensors and animation data and method thereof | |
US11960660B2 (en) | Terminal device, virtual object manipulation method, and virtual object manipulation program | |
US20230367403A1 (en) | Terminal device, virtual object manipulation method, and virtual object manipulation program | |
Huang et al. | Interactive demonstration of pointing gestures for virtual trainers |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: OSAKA ELECTRO-COMMUNICATION UNIVERSITY, JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:UOI, HIROTAKA;KIMURA, KATSUTOSHI;SIGNING DATES FROM 20091201 TO 20091202;REEL/FRAME:035476/0951 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
AS | Assignment |
Owner name: CRESTLINE DIRECT FINANCE, L.P., TEXAS Free format text: SECURITY INTEREST;ASSIGNOR:EMPIRE TECHNOLOGY DEVELOPMENT LLC;REEL/FRAME:048373/0217 Effective date: 20181228 |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |
|
AS | Assignment |
Owner name: EMPIRE TECHNOLOGY DEVELOPMENT LLC, WASHINGTON Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CRESTLINE DIRECT FINANCE, L.P.;REEL/FRAME:049924/0794 Effective date: 20190501 |