WO2011119118A1 - A haptic system, a method of forming a haptic system and a method of controlling a haptic system - Google Patents

A haptic system, a method of forming a haptic system and a method of controlling a haptic system Download PDF

Info

Publication number
WO2011119118A1
WO2011119118A1 PCT/SG2011/000126 SG2011000126W WO2011119118A1 WO 2011119118 A1 WO2011119118 A1 WO 2011119118A1 SG 2011000126 W SG2011000126 W SG 2011000126W WO 2011119118 A1 WO2011119118 A1 WO 2011119118A1
Authority
WO
WIPO (PCT)
Prior art keywords
solenoids
haptic system
solenoid
virtual
haptic
Prior art date
Application number
PCT/SG2011/000126
Other languages
French (fr)
Inventor
Susanto Rahardja
Kin Mun Lye
Farzam Farbiz
Original Assignee
Agency For Science, Technology And Research
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Agency For Science, Technology And Research filed Critical Agency For Science, Technology And Research
Publication of WO2011119118A1 publication Critical patent/WO2011119118A1/en

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/016Input arrangements with force or tactile feedback as computer generated output to the user

Definitions

  • Various embodiments relate to a haptic system, a simulation system or a mixed reality environment system including the haptic system, a method of forming a haptic system and a method of controlling a haptic system.
  • Haptics is the science of applying touch (tactile) sensation and control, which for example may be applied for interaction with computer applications.
  • special input/output devices e.g. joysticks, data gloves, or other similar devices
  • users can receive feedback from computer applications in the form of felt sensations in the hand or other parts of the body.
  • haptic technology can be used to train people for tasks requiring hand-eye coordination, such as surgery and space ship manoeuvres. It can also be used for games. For example, in a mixed reality tennis game where the player or user can see the moving ball, by using the haptic device (e.g. in the form of a tennis racket), position and swing of the tennis racket, the user can feel the impact of the ball.
  • Haptic sensory information falls into two categories: tactile and kinesthetic (or force feedback) information.
  • the initial sense of contact is provided by the touch receptors in the skin, for example of a hand, which also provide information on the contact surface geometry, the surface texture of an object, and slippage.
  • kinesthetic information comes into play, providing details about the position and motion of the hand and arm, and the forces acting on them, to give a sense of total contact forces, surface compliance, and weight if the hand is supporting an object in some way.
  • tactile and kinesthetic sensing occurs simultaneously.
  • haptic feedback is based on electromagnetic motors, hydraulics, and pneumatic actuators.
  • Some low-end haptic devices are already common in the form of game controllers, in particular, in the form of joysticks and steering wheels.
  • Such features and/or haptic devices were optional components for game consoles but have increasingly become art of the game consoles to enhance the users' sense of reality during gameplay.
  • game controllers provide such haptic devices.
  • the Wii wireless remote controller also provides feedback, but uses a simpler vibration mechanism for haptic feedback compared to Sony's remote controller.
  • Another example of such game controller is the simulated automobile steering wheels that are programmed to provide a "feel" of the road. As the user makes a turn or accelerates, the steering wheel responds by resisting turns or slipping out of control.
  • vibration-based haptic feedback is using DC-motors or piezo-actuators based approach.
  • the main advantages of vibration-based haptic feedback are that it offers a low-cost and a tether-less approach.
  • the perceived haptic feedback is far from being realistic.
  • a haptic system for use in a simulation environment or a mixed reality environment.
  • the haptic system may include at least three solenoids configured to generate a torque feedback.
  • a simulation system or a mixed reality environment system is provided.
  • the simulation system or a mixed reality environment system may include a haptic system.
  • a method of forming a haptic system for use in a simulation environment or a mixed reality environment may include providing at least three solenoids configured to generate a torque feedback.
  • a method of controlling a haptic system for use in a simulation environment or a mixed reality environment may include actuating at least three solenoids configured to generate a torque feedback.
  • FIG. 1A shows a schematic block diagram of a haptic system for use in a simulation environment or a mixed reality environment, according to various embodiments.
  • FIG. IB shows a schematic block diagram of a haptic system for use in a simulation environment or a mixed reality environment, according to various embodiments.
  • FIG. 1C shows a schematic block diagram of a haptic system for use in a simulation environment or a mixed reality environment, according to various embodiments.
  • FIG. ID shows a schematic block diagram of a haptic system for use in a simulation environment or a mixed reality environment, according to various embodiments.
  • FIG. IE shows a schematic block diagram of a microcontroller in the haptic system of the embodiments of FIGS. IB and ID.
  • FIG. 2 shows a schematic block diagram of a simulation system or a mixed reality environment system, according to various embodiments.
  • FIG. 3 shows a method of forming a haptic system for use in a simulation environment or a mixed reality environment, according to various embodiments.
  • FIG. 4A shows a method of controlling a haptic system for use in a simulation environment or a mixed reality environment, according to various embodiments.
  • FIG. 4B shows a flow chart illustrating a method of actuating at least three solenoids according to the embodiment of FIG. 4A.
  • FIG. 5 shows a schematic illustration of a haptic system, according to various embodiments.
  • FIG. 6 shows a schematic illustration of a haptic system illustrating force feedback, according to various embodiments.
  • FIG. 7 shows a schematic block diagram of a haptic system, according to various embodiments.
  • FIG. 8 shows a schematic block diagram of a virtual sports simulation system including a haptic system of various embodiments.
  • Various embodiments provide a tether-less haptic approach and system that provide realistic haptic sensations and feedback force, for example reactive forces and impulsive forces, in a mixed reality environment, without or with reduced at least some of the associated disadvantages of conventional systems and approaches.
  • Various embodiments may provide a haptic system for mixed reality environment, for example mixed reality game simulations, where a user in the mixed reality environment interacts with a virtual object or objects using a physical device.
  • the haptic system may be implemented for mixed reality games, virtual sports simulations or human -computer interaction games.
  • the physical device may be a racket held by a user (human player), and the virtual object is a virtual ball.
  • the virtual sports simulation may include the game of badminton, where the virtual object is a virtual shuttlecock.
  • the virtual sports simulation is not limited to racquet sports simulations and the haptic system of various embodiments may be configured to be used with or on other sports equipment like a bat for sports simulations of cricket, baseball, etc.
  • the haptic system of various embodiments may also be adapted to be used with or on a sword or a striking weapon in mixed reality martial art games.
  • the haptic system may provide realistic feedback force that imparts a physical sensation and a reactive force sensation corresponding to the interaction with the virtual object or objects.
  • the haptic system incorporates a tether-less approach.
  • the haptic system employs a wireless approach for communication between a processing device (e.g. a computer) and the haptic system or a device for use in such a system, without any wire or cable connections that may limit the freedom of movement of the user(s).
  • Various embodiments may provide a haptic system with realistic haptic sensations or perceived haptic feedback for applications such as in robotics, simulators (e.g. aircraft simulators) and in medical applications (e.g. surgery). The haptic system may also be used for training purposes for tasks requiring hand-eye coordinations.
  • Various embodiments may provide a haptic system including at least one solenoid. In embodiments where a single solenoid is provided, for example attached to a fix position on a physical device, actuating the single solenoid may generate a vibration sensation and simulate an impulsive force on that position. In various embodiments, two solenoids may be provided, where actuating the two solenoids may simulate an impulsive force in a linear direction (i.e. in a line).
  • the haptic system is configured to simulate an impulsive force to cover a 2D surface of a device
  • at least three solenoids or solenoid sensors may be provided.
  • the surface area covered by the solenoids is that within the polygon connecting the solenoids.
  • actuating two or more solenoids may simulate the location or position of a virtual impact caused by a virtual object on the surface of the physical device.
  • a sensation of torque or energy generated from the virtual impact on the surface of the physical device may also be recreated or simulated.
  • Various embodiments may provide a haptic system configured to produce one or more impulsive forces and/or torque feedbacks on a surface area (e.g. a 2D surface area) of a device (e.g. a racket or sports equipment) covered by the solenoid sensors.
  • a surface area e.g. a 2D surface area
  • a device e.g. a racket or sports equipment
  • Various embodiments may provide a haptic system configured to simulate an impact or a collision caused by a virtual object incident on a surface (i.e. a surface impact) of a device, for example at an impact point.
  • various embodiments may provide a haptic system configured to provide force feedback in one axis, two axes, three axes or multiple axes or provide force feedback in one direction, two directions, three directions or multiple directions.
  • a haptic system including a plurality of solenoids arranged on a device.
  • the plurality of solenoids may be actuated or driven to generate a physical sensation and haptic feedback, such as impulsive force feedback and torque feedback, to a user using the device.
  • each solenoid may be actuated by a current, where the current is controlled by, for example a microcontroller circuit.
  • the plurality of solenoids may be arranged in any particular spatial pattern and may be actuated selectively in accordance with the haptic feedback to be generated.
  • solenoid includes an electromechanical solenoid.
  • the solenoid or solenoids may provide impulsive force feedback. Therefore, one or more solenoids may be used, for example on a device, to simulate and provide realistic impact sensations, for example a collision with a virtual object in a mixed reality environment such as a racquet game simulation.
  • components such as vibrators, motors and piezo -actuators generate a trembling sensation.
  • motors use rotary motions to generate vibrations which are not concentrated along a single axis.
  • a 'circuit' may be understood as any kind of a logic implementing entity, which may be special purpose circuitry or a processor executing software stored in a memory, firmware, or any combination thereof.
  • a 'circuit' may be a hard-wired logic circuit or a programmable logic circuit such as a programmable processor, e.g. a microprocessor (e.g. a Complex Instruction Set Computer (CISC) processor or a Reduced Instruction Set Computer (RISC) processor).
  • a 'circuit' may also be a processor executing software, e.g. any kind of computer program, e.g. a computer program using a virtual machine code such as e.g. Java. Any other kind of implementation of the respective functions which will be described in more detail below may also be understood as a 'circuit' in accordance with an alternative embodiment.
  • electrical communication may be achieved by, for example electrical interconnections (e.g. wire or bus).
  • FIG. 1A shows a schematic block diagram of a haptic system 10 for use in a simulation environment or a mixed reality environment, according to various embodiments.
  • the haptic system 10 includes at least two solenoids 12.
  • the at least two solenoids 12 may be configured for use on a device.
  • the at least two solenoids 12 may be arranged, for example on a circumference of the racquet head.
  • FIG. IB shows a schematic block diagram of a haptic system 20 for use in a simulation environment or a mixed reality environment, according to various embodiments.
  • the haptic system 20 includes at least two solenoids 22.
  • the at least two solenoids 22 may be configured for use on a device.
  • the at least two solenoids 22 may be arranged, for example on a circumference of the racquet head.
  • the haptic system 20 may further include at least two current drivers 24, wherein a respective current driver of the at least two current drivers 24 is in electrical communication with a respective solenoid of the at least two solenoids 22, and wherein the respective current driver is configured to actuate the respective solenoid.
  • the haptic system 20 may further include a microcontroller 26 in electrical communication with the at least two current drivers 24.
  • the haptic system 20 may further include a wireless receiver 28 in electrical communication with the microcontroller 26, wherein the wireless receiver 28 is configured to communicate with a processing device (e.g. a computer).
  • a processing device e.g. a computer
  • the haptic system 20 may further include at least one tracking sensor 30.
  • the haptic system 20 may further include a wireless transmitter 132 coupled to the at least one tracking sensor 30.
  • the haptic system 10, 20 may include at least three solenoids configured to generate a torque feedback.
  • FIG. 1C shows a schematic block diagram of a haptic system 100 for use in a simulation environment or a mixed reality environment, according to various embodiments.
  • the haptic system 100 includes at least three solenoids 102 configured to generate a torque feedback.
  • the at least three solenoids 102 may be configured for use on a device. In embodiments where the device is a sports racquet with a racquet head, the at least three solenoids 102 may be arranged, for example on a circumference of the racquet head.
  • FIG. ID shows a schematic block diagram of a haptic system 120 for use in a simulation environment or a mixed reality environment, according to various embodiments.
  • the haptic system 120 includes at least three solenoids 122 configured to generate a torque feedback.
  • the at least three solenoids 122 may be configured for use on a device.
  • the at least three solenoids 122 may be arranged, for example on a circumference of the racquet head.
  • the haptic system 120 may further include at least three current drivers 124, wherein a respective current driver of the at least three current drivers 124 is in electrical communication with a respective solenoid of the at least three solenoids 122, and wherein the respective current driver is configured to actuate the respective solenoid.
  • the haptic system 120 may further include a microcontroller 126 in electrical communication with the at least three current drivers 124.
  • the haptic system 120 may further include a wireless receiver 128 in electrical communication with the microcontroller 126, wherein the wireless receiver 128 is configured to communicate with a processing device (e.g. a computer).
  • a processing device e.g. a computer
  • the haptic system 120 may further include at least one tracking sensor 130.
  • the haptic system 120 may further include a wireless transmitter 132 coupled to the at least one tracking sensor 130.
  • the microcontroller 26, 126 may include a determination circuit 140 configured to determine a three-dimensional speed vector and a three- dimensional position of a virtual object; and an estimation circuit 142 configured to estimate a time of a virtual impact of the virtual object and an impact force of the virtual impact based on a position and an orientation of a device and the three-dimensional speed vector and a three-dimensional position of the virtual object.
  • the determination circuit 140 may further be configured to determine a relative impact force for a respective solenoid of the at least two solenoids 22 (FIG. IB) or the at least three solenoids 122 (FIG.
  • the microcontroller 26, 126 may further include a time delay circuit 144 configured to provide a time delay to the at least two solenoids 22 (FIG. IB) or the at least three solenoids 122 (FIG. ID).
  • the determination circuit 140, the estimation circuit 142 and the time delay circuit 144 may be in electrical communication with each other.
  • the at least three solenoids 102, 122 may be selectively actuated to generate a torque feedback.
  • the at least three solenoids 102, 122 may be arranged with an at least substantially uniform spacing or with a non-uniform spacing between adjacent solenoids.
  • the haptic system 10, 20, 100, 120 may comprise a memory which is for example used in the processing carried out by the haptic system 10, 20, 100, 120.
  • a memory used in the embodiments may be a volatile memory, for example a DRAM (Dynamic Random Access Memory) or a non-volatile memory, for example a PROM (Programmable Read Only Memory), an EPROM (Erasable PROM), EEPROM (Electrically Erasable PROM), or a flash memory, e.g., a floating gate memory, a charge trapping memory, an MRAM (Magnetoresistive Random Access Memory) or a PCRAM (Phase Change Random Access Memory).
  • DRAM Dynamic Random Access Memory
  • PROM Programmable Read Only Memory
  • EPROM Erasable PROM
  • EEPROM Electrical Erasable PROM
  • flash memory e.g., a floating gate memory, a charge trapping memory, an MRAM (Magnetoresistive Random Access Memory) or a PCRAM (P
  • FIG. 2 shows a schematic block diagram of a simulation system or a mixed reality environment system 200, according to various embodiments.
  • the simulation system or the mixed reality environment system 200 includes a haptic system 202.
  • the haptic system 202 may be one of the haptic system 10 of FIG. 1A, the haptic system 20 of FIG. IB, the haptic system 100 of FIG. 1C and the haptic system 120 of FIG. ID.
  • FIG. 3 shows a method 300 of forming a haptic system for use in a simulation environment or a mixed reality environment, according to various embodiments.
  • At 302 at least three solenoids configured to generate a torque feedback are provided.
  • the at least three solenoids may be arranged with an at least substantially uniform spacing between adjacent solenoids.
  • the at least three solenoids may be arranged on a device.
  • the at least three solenoids may be arranged, for example on a circumference of the racquet head.
  • At least three current drivers are provided, where a respective current driver of the at least three current drivers is in electrical communication with a respective solenoid of the at least three solenoids to actuate the respective solenoid.
  • a microcontroller is provided in electrical communication with the at least three current drivers.
  • a determination circuit configured to determine a three-dimensional speed vector and a three-dimensional position of a virtual object, and an estimation circuit configured to estimate a time of a virtual impact of the virtual object and an impact force of the virtual impact based on a position and an orientation of a device and the three-dimensional speed vector and a three-dimensional position of the virtual object, may be provided in the microcontroller.
  • a time delay circuit configured to provide a time delay to the at least three solenoids may be provided in the microcontroller.
  • a wireless receiver configured to communicate with a processing device (e.g. a computer) may be provided in electrical communication with the microcontroller.
  • a processing device e.g. a computer
  • At least one tracking sensor is provided.
  • a wireless transmitter may be coupled to the at least one tracking sensor.
  • FIG. 4A shows a method 400 of controlling a haptic system for use in a simulation environment or a mixed reality environment, according to various embodiments.
  • At 402 at least three solenoids configured to generate a torque feedback are actuated.
  • the at least three solenoids may be selectively actuated to generate the torque feedback.
  • the at least three solenoids may be arranged on a device.
  • the method 400 may further include communicating with a processing device (e.g. a computer).
  • FIG. 4B shows a flow chart 410 illustrating a method of actuating at least three solenoids according to the embodiment of FIG. 4A.
  • a position and an orientation of a device on which the at least three solenoids are arranged are tracked.
  • a three-dimensional speed vector and a three-dimensional position of a virtual object are determined.
  • a time of a virtual impact of the virtual object and an impact force of the virtual impact are estimated based on the position and the orientation of the device and the three-dimensional speed vector and a three-dimensional position of the virtual object.
  • a relative impact force for a respective solenoid of the at least three solenoids is determined based on a position of the respective solenoid of the at least three solenoids relative to a location of the virtual impact.
  • a current is provided to the respective solenoid of the at least three solenoids based on the relative impact force determined for the respective solenoid of the at least three solenoids.
  • the method of actuating at least three solenoids may further include providing a time delay to the at least three solenoids.
  • FIGS. 3, 4A and 4B may similarly be applicable to a haptic system including at least two solenoids. Accordingly, a reference to the terms “at least three solenoids” and “at least three current drivers” in the descriptions related to FIGS. 3, 4A and 4B, may be replaced by the terms “at least two solenoids” and “at least two current drivers”, respectively.
  • FIG. 5 shows a schematic illustration of a haptic system, according to various embodiments.
  • the haptic system includes a plurality of solenoids, for example as represented by 500a, 500b, 500c, for three solenoids, mounted and arranged on a physical device 502.
  • the physical device is a sport equipment such as a racket or racquet, such that the plurality of solenoids, e.g. 500a, 500b, 500c, are arranged and located on the racket 502 as shown in FIG. 5.
  • the plurality of solenoids e.g.
  • 500a, 500b, 500c may be arranged on or along the rim or frame 504 of a racket head 505 of the racket 502 (e.g. on a circumference or circumferential edge of an impact surface or area 506 of the racket head 505).
  • the plurality of solenoids e.g. 500a, 500b, 500c
  • the plurality of solenoids may be arranged with a non-uniform spacing between adjacent solenoids.
  • the plurality of solenoids are actuated or driven to generate a physical sensation and haptic feedback to a user holding or operating the racket 502 to simulate an impact or a collision occuring at an impact or collision point, for example as represented as 508, which the racket 502 experiences against a computer simulated virtual object, e.g. a ball.
  • the plurality of solenoids, e.g. 500a, 500b, 500c, arranged on the racket 502 produce impulsive force or torque feedback on the impact surface 506 of the racket 502.
  • the impact surface 506 of the racket head 505 of the racket 502 is the impact area for striking a virtual object such as a ball.
  • the impact point 508 causes a respective relative impact force as represented by the arrow 510a to the solenoid 500a, a respective relative impact force as represented by the arrow 510b to the solenoid 500b and a respective relative impact force as represented by the arrow 510c to the solenoid 500c.
  • Each of the relative impact forces 510a, 510b, 510c may be of a different force magnitude.
  • each solenoid of the plurality of solenoids may be actuated differently from another solenoid such that a vibration sensation felt from a solenoid to another solenoid is different, depending on the location of the impact point on the surface of the physical device (e.g. a racket) so as to provide the user of the physical device a sense of the location or position of the impact point caused by a virtual object on the surface of the physical device. Accordingly, the relative impact force to each solenoid may be different. In addition, therefore, a change in the location of the impact point also causes a change in the actuation of the same solenoid.
  • FIG. 6 shows a schematic illustration of a haptic system illustrating force feedback, according to various embodiments.
  • the haptic system includes two solenoids (a first solenoid Si 600a and a second solenoid S 2 600b), mounted or arranged along a frame 602, or a circumferential edge of the impact surface or area 604, or a circumference of a racket head 605 of a racket 606.
  • the first solenoid Si 600a and the second solenoid S 2 600b are arranged such that the pair of solenoids are on either side of a central axis, as represented by the dotted line 607, of the racket head 605 being parallel to the handle or grip 608 of the racket 606.
  • the first solenoid Si 600a and the second solenoid S 2 600b may also be arranged such that they are mirror images of each other about the central axis 607 as shown in FIG. 6.
  • the first solenoid Si 600a and the second solenoid S 2 600b may be arranged at least substantially centrally on the frame 602 of the racket head 605 as shown in FIG. 6.
  • first solenoid Si 600a and the second solenoid S 2 600b may be arranged on any locations along the frame 602 of the racket head 605 of the racket 606, for example on one side or either side of the central axis 607. Where the first solenoid Si 600a and the second solenoid S 2 600b are arranged on either side of the central axis 607, the first solenoid Si 600a and the second solenoid S 2 600b may or may not be arranged as mirror images about the central axis 607.
  • a torque or force feedback may be at a location or holding point 610 (e.g. where the user's hand grip is) at the handle 608 of the racket 606 when a virtual object (e.g. a ball) hits a part of the racket 606.
  • a virtual object e.g. a ball
  • the ball may hit any part of the racket 606, at any location on the impact area 604 of the racket 606, including for example at impact point represented as 612, and/or any part of the frame 602 of the racket 606.
  • the impact torque ⁇ at the grip position or holding point 610 may be given by the following equation:
  • the impact torque ⁇ is a vector.
  • f j is the distance vector of the distance n as represented by the arrow 616a, from the holding point 610 to the first solenoid Si 600a,
  • r 2 is the distance vector of the distance r 2 as represented by the arrow 616b, from the holding point 610 to the second solenoid Si 600b,
  • r is the distance vector of the distance r' i as represented by the arrow 618a, from the holding point 610, to the point 620, and
  • r 2 ' is the distance vector of the distance r' 2 as represented by the arrow 618b, from the holding point 610, to the point 622.
  • the impact force of each of the first solenoid Si 600a and the second solenoid S 2 600b is limited by its technical specification. Where the first solenoid Si 600a and the second solenoid S 2 600b have an at least substantially similar technical specifications, the impact force F s of the first solenoid S] 600a and the impact force F 3 ⁇ 4 of the second solenoid S 2 600b may not be larger than a certain maximum value
  • F Sm ax may be in a range of between about lg to about 20g (i.e. between about one time of gravity to about 20 times of gravity).
  • F s and F Sz are the force vectors of the impact force by the first solenoid Si 600a and the second solenoid S 2 600b, respectively, to generate the torque ⁇ ' .
  • the magnitude of the impact force F s of the first solenoid Si 600a and the magnitude of the impact force ⁇ Si of the second solenoid S 2 600b may not be larger than
  • F Si is greater than Fs ma x (i.e. F S] > Fsmax)
  • the maximum force of the impact force for the first solenoid Si 600a and of the impact force for the second solenoid S 2 600b, respectively, may be restricted to Fsmax, while the relation between F s and F Si may be maintained for generating the torque ⁇ ' , as shown in equation 4.
  • the maximum torque that may be generated at the holding point 610 may be determined from the following equation:
  • more than a pair of soleniods i.e. more than two solenoids
  • more than two solenoids may be used, for example two pairs, three pairs or any higher number of pairs of solenoids arranged on or along the frame 602 or a circumference of the racket head 605 of the racket 606.
  • any odd number or even number of solenoids may be provided, for example two solenoids, three solenoids, four solenoids, five solenoids, six solenoids, seven solenoids or any higher number of solenoids.
  • N pairs of solenoids may be arranged in a structure or configuration similar to that shown in FIG. 5, where the maximum generated torque are scaled up by a factor of N. Adding more solenoids to the system may provide challenges as this increases power consumption of the system as well as the total weight of the racket with the solenoids. Therefore, it should be appreciated that the number of solenoids incorporated depend on the desired torque and the total tolerable weight for the racket.
  • FIG. 7 shows a schematic block diagram of a haptic system 700, according to various embodiments.
  • the haptic system 700 may be used for mixed reality games or virtual sports simulations.
  • the haptic system 700 includes a plurality of solenoids, for example including a first solenoid 702a, a second solenoid 702b and further solenoids to an N-th solenoid 702c.
  • the first solenoid 702a, the second solenoid 702b and further solenoids to the N-th solenoid 702c may be arranged on a physical device similar to the embodiment shown in FIG. 5.
  • the haptic system 700 further includes a microcontroller 704 configured to control and deliver controlled current to each of the plurality of solenoids via a respective driver, such as a current driver, in order to actuate a respective solenoid of the plurality of solenoids and simulate a relative impact force on an impact surface of the physical device (e.g. a racket) from a virtual object (e.g. a virtual ball).
  • a driver 706a is in electrical communication with the first solenoid 702a
  • another driver 706b is in electrical communication with the second solenoid 702b
  • a further driver 706c is in electrical communication with the N-th solenoid 702c.
  • the drivers 706a, 706b, 706c are in turn in electrical communication with the microcontroller 704.
  • the microcontroller 704 is configured to control the operations of each of the first solenoid 702a, the second solenoid 702b and further solenoids to the N-th solenoid 702c based on signals wirelessly received from a computer 708, via a wireless transmitter 710 in communication with the computer 708 and a wireless receiver 712 in communication with the microcontroller 704.
  • the communication link between the wireless transmitter 710 and the wireless receiver 712 is represented by the dotted line 714.
  • the wireless transmitter 710 may be integral to the computer 708 or may be an external module.
  • the wireless receiver 712 may be integral to the haptic system 700 or may be an external module in communication with the haptic system 700.
  • the haptic system 700 may further include one or more wireless tracking sensors with a wireless transmitter, as represented by the block 716, configured to track a position and an orientation of a device incorporating the haptic system 700.
  • the microcontroller 704, the respective drivers, for example drivers 706a, 706b, 706c, the wireless receiver 712 and the one or more wireless tracking sensors with the wireless transmitter, as represented by the block 716, may be provided or integrated on a printed circuit board (PCB), which may be placed in a device incorporating the solenoids (e.g. the first solenoid 702a, the second solenoid 702b and further solenoids to the N-th solenoid 702c).
  • the PCB may be placed in the handle 608 of the racket 606.
  • the haptic system 700 tracks the racket and uses the location information of the racket to estimate the collision time and impact force based on the position and speed of the virtual ball.
  • the computer 708 sends an activation signal to the racket to activate all the solenoids (e.g. 702a, 702b, 702c) with a certain time delay such that the solenoids and the system are synchronized to the exact moment or time of the collision or impact.
  • the computer 708 calculates the relative impact force for each solenoid (e.g. 702a, 702b, 702c) individually based on the relative position of the solenoid with respect to the location or position of the collision (i.e. the impact point), similar to the descriptions and calculations as described earlier with respect to the embodiment of FIG. 6.
  • the computer 708 sends the relative impact force for each solenoid (e.g. 702a, 702b, 702c) wirelessly to the haptic system 700.
  • This information is received by the microcontroller 704 and is used to modify or vary the electric currents of each of the solenoids via the respective driver (e.g. 706a, 706b, 706c), which consequently affect the impact force of each solenoid in order to provide a realistic force feedback feeling or sensation to the user.
  • the force impact generated by each solenoid e.g. 702a, 702b, 702c
  • the force impact generated by each solenoid is at least substantially equal to the relative impact force determined from the impact force of the virtual impact for each solenoid (e.g. 702a, 702b, 702c).
  • the one or more wireless tracking sensors with a wireless transmitter may be configured to track the position and the orientation of the physical device. These tracking sensors 716 may be used to detect in real time, both the position and the orientation of the racket in a game environment, for example inside a room, in three dimensions. Such tracking information is subsequently sent to the computer 708 wirelessly. The computer 708 uses this information or data together with the three-dimensional speed vector and three-dimensional position of the virtual ball to estimate the moment of the collision or impact and its impact force. The estimated time and force are sent back to the racket wirelessly to activate the solenoids (e.g. 702a, 702b, 702c).
  • the solenoids e.g. 702a, 702b, 702c
  • the haptic system 700 may comprise a memory which is for example used in the processing carried out by the haptic system 700.
  • a memory used in the embodiments may be a volatile memory, for example a DRAM (Dynamic Random Access Memory) or a non-volatile memory, for example a PROM (Programmable Read Only Memory), an EPROM (Erasable PROM), EEPROM (Electrically Erasable PROM), or a flash memory, e.g., a floating gate memory, a charge trapping memory, an MRAM (Magnetoresistive Random Access Memory) or a PCRAM (Phase Change Random Access Memory).
  • the memory may be included in the microcontroller 704.
  • FIG. 8 shows a schematic block diagram of a virtual simulation system 800 (e.g. a tennis sports simulation system) including a haptic system of various embodiments, in the form of a haptic module 802.
  • the haptic module 802 may provide and/or receive haptic signals, as represented by the block 804, in correlation with the various modules or nodes within the virtual sports simulation system 800.
  • the haptic module 802 may be provided on a device, for example a racket, for use by a user or player 810.
  • the virtual simulation system 800 further includes a scene-graph rendering module 806 configured to provide graphics and a hybrid tracking module 808 configured to track the player 810 interacting with the virtual simulation system 800.
  • the hybrid tracking module 808 may be configured to track a location of the player 810, for example by sensing or tracking the movement of the player 810, such as tracking the movement of one or more parts or areas of the player 810 (e.g. the player's head and/or hand), and tracking the movement of one or more objects or items attached or held by the player 810.
  • the hybrid tracking module 808 may be further configured to compute or determine the trajectories of one or more parts or areas of the player 810 and/or the trajectories of one or more objects attached or held by the player 810, as a result of movement of the player 810.
  • the scene-graph rendering module 806 may include a root 812 (e.g. a processing device) which may control lighting 814, for example of the rendered scene, and include a navigation node 816 for providing different 3D models, for example to provide different adjustable views of the scene/simulation for the virtual simulation system 800 for interaction with the player 810.
  • the navigation node 816 may communicate with the hybrid tracking module 808, for example via a navigation controller 840 integrated with the hybrid tracking module 808.
  • the navigation node 816 may be in the form of a hardware controller.
  • the navigation node 816 may include a number of components configured to provide different independent models within the virtual simulation system 800.
  • the navigation node 816 may include a court model component configured to provide a 3D court model 818, and a net animation switch 820 configured to provide a number of 3D net models 822, for example of different shapes and colours.
  • the net animation switch 820 may be further configured to provide 3D net models 822 for the continuous movement of the net, for example when a ball hits the net, and/or 3D net models 822 for the display of gentle net movements, for example when there is a breeze.
  • the navigation node 816 may further include a ball animation node 824 configured to provide a 3D ball model 826.
  • the navigation node 816 may further include a virtual human animation node 828 configured to provide a number of 3D virtual human and racket models 830 for interacting with the player 810.
  • the virtual human animation node 828 may include data or inputs from an animation and artificial intelligence (AI) module 832 such that the virtual player may be skin-deformed and its 3D racket models may be manipulated for animation.
  • AI artificial intelligence
  • GPU enhanced animation 834 may also be provided to the virtual human animation node 828. It should be appreciated that other 3D models such as a ball serving machine or watching audience may be provided in order to extend the scene and simulation of the game environment.
  • the 3D ball model 826 interacts with other models such as the 3D court model 818, the 3D net models 822, and the 3D virtual human and racket models 830, through fast collision detection 836, which also provides accurate detection.
  • the fast collision detection 836 of the virtual ball and the racket of the player 810 may be achieved via the hybrid tracking module 808.
  • other models may be provided for the virtual simulation system 800 for different simulations which may include, but is not limited to, a military simulation, a driving simulation, and an architectural walkthrough.
  • the scene-graph rendering module 806 may communicate with the hybrid tracking module 808.
  • the scene-graph rendering module 806 and the hybrid tracking module 808 may provide and/or receive haptic signals 804, and therefore are also in communication with the haptic module 802.
  • the hybrid tracking module 808 may include an inertia tracking system 842 and a camera tracking system (e.g. an optical tracking system) 844.
  • the inertia tracking system 842 may include inertia tracking receivers or inertia trackers.
  • the camera tracking system 844 may include one or more infrared cameras. By using an infrared-based system, the camera tracking system 844 and therefore the hybrid tracking module 808, may be robust to lighting fluctuations.
  • the inertial trackers may provide three degrees of freedom (DOF) orientation tracking data.
  • DOF degrees of freedom
  • the inertia tracking system 842 includes an ultrasonic tracking system, such that the inertia tracking system 842 may be an ultrasonic -inertial tracking system. Therefore, the hybrid tracking module 808 may provide a robust hybrid tracking system including ultrasonic - inertial tracking and optical-based tracking, for example including ultrasonic-inertial tracking devices and infra-red (IR) cameras and markers, which may support fast moving requirements for example, for a tennis game simulation.
  • the ultrasonic-inertial tracking system may be the IS-900 system by Intersense.
  • the camera tracking system 844 may be a precision position tracking hybrid system (PPTH) employing infrared-based tracking. The PPTH system incorporates high-speed IR tracking technology (about 175 Hz) which may track fast moving objects such as, for example tennis rackets.
  • PPTH precision position tracking hybrid system
  • Infrared-based optical tracking technology such as Augmented Reality Tracker [A.R.T.] or precision position tracking hybrid system [PPTH] have been developed as a vision-based tracking method using multiple IR cameras. Although such technology may be sensitive to sunlight or incandescent light, it may be suitable for an indoor dark or florescent lighting environment. As most VR simulations are projection based and situated indoor, this technology has been widely used to provide the position tracking.
  • the hybrid tracking module 808 may provide ultrasonic-inertial tracking and optical-based tracking configured to track the player 810 and/or the object (e.g. a racket) held by the player 810 in order to provide three or more degrees of freedom position and orientation data, including instances where occlusion may occur. This may allow the tracking of fast or rapid movements of the player 810 and/or the tracking of the object or equipment held by the player.
  • optical tracking e.g. PPT-H
  • inertial tracking may be used to provide the three DOF orientation data
  • ultrasonic tracking may be used to complement optical tracking when occlusion occurs. Therefore, ultrasonic tracking may facilitate tracking even when occlusion occurs.
  • the hybrid tracking module 808 may further include a predictive filtering method or algorithm (e.g. a Kalman-filtering based sensor fusion method).
  • a predictive filtering method or algorithm e.g. a Kalman-filtering based sensor fusion method
  • the predictive filter used in the embodiment of FIG. 8 may be a Kalman filter.
  • the Kalman filter is a recursive stochastic approach to estimate the state of a dynamic system from a set of incomplete and noisy measurements.
  • the Kalman-filtering based sensor fusion method may be used to smooth the combined position and orientation data from the inertia tracking system 842 and the camera tracking system 844.
  • the Kalman- filtering based sensor fusion method may be further configured to smooth the combined position and orientation data with other data received from the other modules, such as the ball animation node 824 and the virtual human animation node 828 in the virtual simulation system 800. This may produce a more accurate result than individual ultrasonic tracking or optical tracking. Therefore, in various embodiments, the hybrid tracking module 808 may provide robust and smooth six DOF tracking in real time, even with high speed movements of the player 810 and/or the object held by the player 810.
  • a pre-filtering process may be performed to remove outlier (bad or noisy) data so that high fidelity data may be provided for the Kalman filtering process. Therefore, some measurements with obvious large errors may be pre- filtered, and subsequently left out of the Kalman filtering process. Removing these errors may help, for example in filtering out any discrepant estimations, for example by verifying that the distance between the current position and the previous position does not involve an excessive velocity for a moving person or a swinging racket.
  • the virtual simulation system 800 may provide an immersive, interactive and real-time simulation system with, for example a projection system with an L-shape 3D display.
  • the projection system may include two back projected HD Infitec projectors, in order to provide the player 810 with a 3D (stereoscopic) viewing with wide angles. Mirrors may also be provided to reduce the projection space constraint.
  • the scene-graph rendering module 806 provides the graphics and animation
  • the hybrid tracking module 808 tracks the head and racket movements of the player 810
  • the haptic module 802 generates force feedback and vibration for the racket held by the player 810.
  • the virtual simulation system 800 may be developed based on scenegraph architecture, and the virtual simulation system 800 may be easily scaled up to include more modules, depending on applications or when necessary.
  • the hybrid tracking module 808, including an ultrasonic- inertial tracking system (i.e. reference 842 of FIG. 8) with an optical tracking system (i.e. reference 844 of FIG. 8) may provide fast and accurate collision detection of the virtual ball and racket of the player 810.
  • the hybrid tracking module 808 may also constantly and accurately track the head and racket movements of the player 810.
  • the player 820 may be able to feel the virtual ball flying in the air towards the player 810. Together with the haptic feedback on the racket provided by the haptic module 802, when the virtual ball is detected to have collided with the racket through fast collision detection 836, the player 810 is able to feel the vibration and force feedback during game play. Therefore, the virtual simulation system 800 may be able to provide high-definition stereoscopic display, real-time hybrid tracking, animation and haptic (e.g. impulsive force or force feedback and/or torque feedback).
  • haptic e.g. impulsive force or force feedback and/or torque feedback

Abstract

According to embodiments of the present invention, a haptic system for use in a simulation environment or a mixed reality environment is provided. The haptic system includes at least three solenoids configured to generate a torque feedback.

Description

A HAPTIC SYSTEM, A METHOD OF FORMING A HAPTIC SYSTEM AND A METHOD OF CONTROLLING A HAPTIC SYSTEM
Cross-Reference To Related Applications
[0001] This application claims the benefit of priority of Singapore patent application No. 201002133-5, filed 26 March 2010, and Singapore patent application No. 201002211-9, filed 30 March 2010, the contents being hereby incorporated by reference in their entirety for all purposes.
Technical Field
[0002] Various embodiments relate to a haptic system, a simulation system or a mixed reality environment system including the haptic system, a method of forming a haptic system and a method of controlling a haptic system.
Background
[0003] Haptics is the science of applying touch (tactile) sensation and control, which for example may be applied for interaction with computer applications. By using special input/output devices (e.g. joysticks, data gloves, or other similar devices), users can receive feedback from computer applications in the form of felt sensations in the hand or other parts of the body. In combination with a visual display, haptic technology can be used to train people for tasks requiring hand-eye coordination, such as surgery and space ship manoeuvres. It can also be used for games. For example, in a mixed reality tennis game where the player or user can see the moving ball, by using the haptic device (e.g. in the form of a tennis racket), position and swing of the tennis racket, the user can feel the impact of the ball.
[0004] Haptic sensory information falls into two categories: tactile and kinesthetic (or force feedback) information. The initial sense of contact is provided by the touch receptors in the skin, for example of a hand, which also provide information on the contact surface geometry, the surface texture of an object, and slippage. When the hand applies more force, kinesthetic information comes into play, providing details about the position and motion of the hand and arm, and the forces acting on them, to give a sense of total contact forces, surface compliance, and weight if the hand is supporting an object in some way. In general, tactile and kinesthetic sensing occurs simultaneously.
[0005] Currently, the most common methods used for haptic feedback are based on electromagnetic motors, hydraulics, and pneumatic actuators. Some low-end haptic devices are already common in the form of game controllers, in particular, in the form of joysticks and steering wheels. Initially, such features and/or haptic devices were optional components for game consoles but have increasingly become art of the game consoles to enhance the users' sense of reality during gameplay. However, not many game controllers provide such haptic devices.
[0006] Some of the newer generation console controllers, joystick features and game controllers now have built-in haptic devices, for example such as that of Sony's Dualshock technology. The Wii wireless remote controller also provides feedback, but uses a simpler vibration mechanism for haptic feedback compared to Sony's remote controller. Another example of such game controller is the simulated automobile steering wheels that are programmed to provide a "feel" of the road. As the user makes a turn or accelerates, the steering wheel responds by resisting turns or slipping out of control.
[0007] The common approach for vibration-based haptic feedback is using DC-motors or piezo-actuators based approach. The main advantages of vibration-based haptic feedback are that it offers a low-cost and a tether-less approach. However, the perceived haptic feedback is far from being realistic.
[0008] One of the most popular methods in providing haptic force feedback is to use a series of cables attached to the user's body. Although such methods achieve realistic haptic feel, such tether solutions using cable systems restrict the user's movement and hence, they are not suitable for applications such as ball games (e.g. tennis). Summary
[0009] According to an embodiment, a haptic system for use in a simulation environment or a mixed reality environment is provided. The haptic system may include at least three solenoids configured to generate a torque feedback.
[0010] According to an embodiment, a simulation system or a mixed reality environment system is provided. The simulation system or a mixed reality environment system may include a haptic system.
[0011] According to an embodiment, a method of forming a haptic system for use in a simulation environment or a mixed reality environment is provided. The method may include providing at least three solenoids configured to generate a torque feedback.
[0012] According to an embodiment, a method of controlling a haptic system for use in a simulation environment or a mixed reality environment is provided. The method may include actuating at least three solenoids configured to generate a torque feedback.
Brief Description of the Drawings
[0013] In the drawings, like reference characters generally refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various embodiments of the invention are described with reference to the following drawings, in which:
[0014] FIG. 1A shows a schematic block diagram of a haptic system for use in a simulation environment or a mixed reality environment, according to various embodiments.
[0015] FIG. IB shows a schematic block diagram of a haptic system for use in a simulation environment or a mixed reality environment, according to various embodiments.
[0016] FIG. 1C shows a schematic block diagram of a haptic system for use in a simulation environment or a mixed reality environment, according to various embodiments. [0017] FIG. ID shows a schematic block diagram of a haptic system for use in a simulation environment or a mixed reality environment, according to various embodiments.
[0018] FIG. IE shows a schematic block diagram of a microcontroller in the haptic system of the embodiments of FIGS. IB and ID.
[0019] FIG. 2 shows a schematic block diagram of a simulation system or a mixed reality environment system, according to various embodiments.
[0020] FIG. 3 shows a method of forming a haptic system for use in a simulation environment or a mixed reality environment, according to various embodiments.
[0021] FIG. 4A shows a method of controlling a haptic system for use in a simulation environment or a mixed reality environment, according to various embodiments.
[0022] FIG. 4B shows a flow chart illustrating a method of actuating at least three solenoids according to the embodiment of FIG. 4A.
[0023] FIG. 5 shows a schematic illustration of a haptic system, according to various embodiments.
[0024] FIG. 6 shows a schematic illustration of a haptic system illustrating force feedback, according to various embodiments.
[0025] FIG. 7 shows a schematic block diagram of a haptic system, according to various embodiments.
[0026] FIG. 8 shows a schematic block diagram of a virtual sports simulation system including a haptic system of various embodiments.
Detailed Description [0027] The following detailed description refers to the accompanying drawings that show, by way of illustration, specific details and embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. Other embodiments may be utilized and structural, logical, and electrical changes may be made without departing from the scope of the invention. The various embodiments are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments.
[0028] Various embodiments provide a tether-less haptic approach and system that provide realistic haptic sensations and feedback force, for example reactive forces and impulsive forces, in a mixed reality environment, without or with reduced at least some of the associated disadvantages of conventional systems and approaches.
[0029] Various embodiments may provide a haptic system for mixed reality environment, for example mixed reality game simulations, where a user in the mixed reality environment interacts with a virtual object or objects using a physical device. The haptic system may be implemented for mixed reality games, virtual sports simulations or human -computer interaction games. For example, in a mixed reality game or virtual sports simulation, for example a tennis, squash or table tennis game simulation, the physical device may be a racket held by a user (human player), and the virtual object is a virtual ball. Furthermore, the virtual sports simulation may include the game of badminton, where the virtual object is a virtual shuttlecock. In addition, the virtual sports simulation is not limited to racquet sports simulations and the haptic system of various embodiments may be configured to be used with or on other sports equipment like a bat for sports simulations of cricket, baseball, etc. The haptic system of various embodiments may also be adapted to be used with or on a sword or a striking weapon in mixed reality martial art games. In various embodiments, the haptic system may provide realistic feedback force that imparts a physical sensation and a reactive force sensation corresponding to the interaction with the virtual object or objects.
[0030] In various embodiments, the haptic system incorporates a tether-less approach. In other words, the haptic system employs a wireless approach for communication between a processing device (e.g. a computer) and the haptic system or a device for use in such a system, without any wire or cable connections that may limit the freedom of movement of the user(s).
[0031] Various embodiments may provide a haptic system with realistic haptic sensations or perceived haptic feedback for applications such as in robotics, simulators (e.g. aircraft simulators) and in medical applications (e.g. surgery). The haptic system may also be used for training purposes for tasks requiring hand-eye coordinations. [0032] Various embodiments may provide a haptic system including at least one solenoid. In embodiments where a single solenoid is provided, for example attached to a fix position on a physical device, actuating the single solenoid may generate a vibration sensation and simulate an impulsive force on that position. In various embodiments, two solenoids may be provided, where actuating the two solenoids may simulate an impulsive force in a linear direction (i.e. in a line).
[0033] In embodiments where the haptic system is configured to simulate an impulsive force to cover a 2D surface of a device, at least three solenoids or solenoid sensors may be provided. The surface area covered by the solenoids is that within the polygon connecting the solenoids.
[0034] Therefore, actuating two or more solenoids, for example two solenoids, three solenoids, four solenoids, five solenoids or any higher number of solenoids, may simulate the location or position of a virtual impact caused by a virtual object on the surface of the physical device. A sensation of torque or energy generated from the virtual impact on the surface of the physical device may also be recreated or simulated.
[0035] Various embodiments may provide a haptic system configured to produce one or more impulsive forces and/or torque feedbacks on a surface area (e.g. a 2D surface area) of a device (e.g. a racket or sports equipment) covered by the solenoid sensors.
[0036] Various embodiments may provide a haptic system configured to simulate an impact or a collision caused by a virtual object incident on a surface (i.e. a surface impact) of a device, for example at an impact point. In addition, various embodiments may provide a haptic system configured to provide force feedback in one axis, two axes, three axes or multiple axes or provide force feedback in one direction, two directions, three directions or multiple directions.
[0037] In various embodiments, increasing the number of solenoids or sensors increases the surface area covered by the solenoids and the maximum force that may be produced by the haptic system. However, considerations should also be given to the increase in the total weight and energy consumption.
[0038] Various embodiments may provide a haptic system including a plurality of solenoids arranged on a device. The plurality of solenoids may be actuated or driven to generate a physical sensation and haptic feedback, such as impulsive force feedback and torque feedback, to a user using the device. In various embodiments, each solenoid may be actuated by a current, where the current is controlled by, for example a microcontroller circuit. In various embodiments, the plurality of solenoids may be arranged in any particular spatial pattern and may be actuated selectively in accordance with the haptic feedback to be generated.
[0039] In the context of various embodiments, the term "solenoid" includes an electromechanical solenoid. The solenoid or solenoids may provide impulsive force feedback. Therefore, one or more solenoids may be used, for example on a device, to simulate and provide realistic impact sensations, for example a collision with a virtual object in a mixed reality environment such as a racquet game simulation. In contrast, components such as vibrators, motors and piezo -actuators generate a trembling sensation. In addition, motors use rotary motions to generate vibrations which are not concentrated along a single axis.
[0040] In the context of various embodiments, a 'circuit' may be understood as any kind of a logic implementing entity, which may be special purpose circuitry or a processor executing software stored in a memory, firmware, or any combination thereof. Thus, in an embodiment, a 'circuit' may be a hard-wired logic circuit or a programmable logic circuit such as a programmable processor, e.g. a microprocessor (e.g. a Complex Instruction Set Computer (CISC) processor or a Reduced Instruction Set Computer (RISC) processor). A 'circuit' may also be a processor executing software, e.g. any kind of computer program, e.g. a computer program using a virtual machine code such as e.g. Java. Any other kind of implementation of the respective functions which will be described in more detail below may also be understood as a 'circuit' in accordance with an alternative embodiment.
[0041] In the context of various embodiments, "electrical communication" may be achieved by, for example electrical interconnections (e.g. wire or bus).
[0042] In the context of various embodiments, the terms "racket" and "racquet" may be used interchangeably.
[0043] In order that the invention may be readily understood and put into practical effect, particular embodiments will now be described by way of examples and not limitations, and with reference to the figures. [0044] FIG. 1A shows a schematic block diagram of a haptic system 10 for use in a simulation environment or a mixed reality environment, according to various embodiments. The haptic system 10 includes at least two solenoids 12.
[0045] The at least two solenoids 12 may be configured for use on a device. In embodiments where the device is a sports racquet with a racquet head, the at least two solenoids 12 may be arranged, for example on a circumference of the racquet head.
[0046] FIG. IB shows a schematic block diagram of a haptic system 20 for use in a simulation environment or a mixed reality environment, according to various embodiments. The haptic system 20 includes at least two solenoids 22.
[0047] The at least two solenoids 22 may be configured for use on a device. In embodiments where the device is a sports racquet with a racquet head, the at least two solenoids 22 may be arranged, for example on a circumference of the racquet head.
[0048] The haptic system 20 may further include at least two current drivers 24, wherein a respective current driver of the at least two current drivers 24 is in electrical communication with a respective solenoid of the at least two solenoids 22, and wherein the respective current driver is configured to actuate the respective solenoid.
[0049] The haptic system 20 may further include a microcontroller 26 in electrical communication with the at least two current drivers 24.
[0050] The haptic system 20 may further include a wireless receiver 28 in electrical communication with the microcontroller 26, wherein the wireless receiver 28 is configured to communicate with a processing device (e.g. a computer).
[0051] The haptic system 20 may further include at least one tracking sensor 30. The haptic system 20 may further include a wireless transmitter 132 coupled to the at least one tracking sensor 30.
[0052] In various embodiments, the haptic system 10, 20, may include at least three solenoids configured to generate a torque feedback.
[0053] FIG. 1C shows a schematic block diagram of a haptic system 100 for use in a simulation environment or a mixed reality environment, according to various embodiments. The haptic system 100 includes at least three solenoids 102 configured to generate a torque feedback. [0054] The at least three solenoids 102 may be configured for use on a device. In embodiments where the device is a sports racquet with a racquet head, the at least three solenoids 102 may be arranged, for example on a circumference of the racquet head.
[0055] FIG. ID shows a schematic block diagram of a haptic system 120 for use in a simulation environment or a mixed reality environment, according to various embodiments. The haptic system 120 includes at least three solenoids 122 configured to generate a torque feedback.
[0056] The at least three solenoids 122 may be configured for use on a device. In embodiments where the device is a sports racquet with a racquet head, the at least three solenoids 122 may be arranged, for example on a circumference of the racquet head.
[0057] The haptic system 120 may further include at least three current drivers 124, wherein a respective current driver of the at least three current drivers 124 is in electrical communication with a respective solenoid of the at least three solenoids 122, and wherein the respective current driver is configured to actuate the respective solenoid.
[0058] The haptic system 120 may further include a microcontroller 126 in electrical communication with the at least three current drivers 124.
[0059] The haptic system 120 may further include a wireless receiver 128 in electrical communication with the microcontroller 126, wherein the wireless receiver 128 is configured to communicate with a processing device (e.g. a computer).
[0060] The haptic system 120 may further include at least one tracking sensor 130. The haptic system 120 may further include a wireless transmitter 132 coupled to the at least one tracking sensor 130.
[0061] As shown in FIG. IE, the microcontroller 26, 126, may include a determination circuit 140 configured to determine a three-dimensional speed vector and a three- dimensional position of a virtual object; and an estimation circuit 142 configured to estimate a time of a virtual impact of the virtual object and an impact force of the virtual impact based on a position and an orientation of a device and the three-dimensional speed vector and a three-dimensional position of the virtual object. The determination circuit 140 may further be configured to determine a relative impact force for a respective solenoid of the at least two solenoids 22 (FIG. IB) or the at least three solenoids 122 (FIG. ID), respectively, based on a position of the respective solenoid of the at least two solenoids 22 (FIG. IB) or the at least three solenoids 122 (FIG. ID), relative to a location of the virtual impact. Based on the relative impact force, the respective current driver of the at least two current drivers 24 (FIG. IB) or the at least three current drivers 124 (FIG. ID) may be configured, respectively to provide a respective current to the respective solenoid of the at least two solenoids 22 (FIG. IB) or the at least three solenoids 122 (FIG. ID). The microcontroller 26, 126 may further include a time delay circuit 144 configured to provide a time delay to the at least two solenoids 22 (FIG. IB) or the at least three solenoids 122 (FIG. ID). In various embodiments, the determination circuit 140, the estimation circuit 142 and the time delay circuit 144 may be in electrical communication with each other.
[0062] In various embodiments, the at least three solenoids 102, 122, may be selectively actuated to generate a torque feedback. The at least three solenoids 102, 122, may be arranged with an at least substantially uniform spacing or with a non-uniform spacing between adjacent solenoids.
[0063] In various embodiments, the haptic system 10, 20, 100, 120, may comprise a memory which is for example used in the processing carried out by the haptic system 10, 20, 100, 120. A memory used in the embodiments may be a volatile memory, for example a DRAM (Dynamic Random Access Memory) or a non-volatile memory, for example a PROM (Programmable Read Only Memory), an EPROM (Erasable PROM), EEPROM (Electrically Erasable PROM), or a flash memory, e.g., a floating gate memory, a charge trapping memory, an MRAM (Magnetoresistive Random Access Memory) or a PCRAM (Phase Change Random Access Memory).
[0064] FIG. 2 shows a schematic block diagram of a simulation system or a mixed reality environment system 200, according to various embodiments. The simulation system or the mixed reality environment system 200 includes a haptic system 202. The haptic system 202 may be one of the haptic system 10 of FIG. 1A, the haptic system 20 of FIG. IB, the haptic system 100 of FIG. 1C and the haptic system 120 of FIG. ID.
[0065] FIG. 3 shows a method 300 of forming a haptic system for use in a simulation environment or a mixed reality environment, according to various embodiments.
[0066] At 302, at least three solenoids configured to generate a torque feedback are provided. [0067] The at least three solenoids may be arranged with an at least substantially uniform spacing between adjacent solenoids.
[0068] The at least three solenoids may be arranged on a device. In embodiments where the device is a sports racquet with a racquet head, the at least three solenoids may be arranged, for example on a circumference of the racquet head.
[0069] In various embodiments, at least three current drivers are provided, where a respective current driver of the at least three current drivers is in electrical communication with a respective solenoid of the at least three solenoids to actuate the respective solenoid.
[0070] In various embodiments, a microcontroller is provided in electrical communication with the at least three current drivers. In various embodiments, a determination circuit configured to determine a three-dimensional speed vector and a three-dimensional position of a virtual object, and an estimation circuit configured to estimate a time of a virtual impact of the virtual object and an impact force of the virtual impact based on a position and an orientation of a device and the three-dimensional speed vector and a three-dimensional position of the virtual object, may be provided in the microcontroller. In addition, a time delay circuit configured to provide a time delay to the at least three solenoids may be provided in the microcontroller.
[0071] In various embodiments, a wireless receiver configured to communicate with a processing device (e.g. a computer) may be provided in electrical communication with the microcontroller.
[0072] In various embodiments, at least one tracking sensor is provided. A wireless transmitter may be coupled to the at least one tracking sensor.
[0073] FIG. 4A shows a method 400 of controlling a haptic system for use in a simulation environment or a mixed reality environment, according to various embodiments.
[0074] At 402, at least three solenoids configured to generate a torque feedback are actuated. In various embodiments, the at least three solenoids may be selectively actuated to generate the torque feedback.
[0075] In various embodiments, the at least three solenoids may be arranged on a device. [0076] In various embodiments, the method 400 may further include communicating with a processing device (e.g. a computer).
[0077] FIG. 4B shows a flow chart 410 illustrating a method of actuating at least three solenoids according to the embodiment of FIG. 4A.
[0078] At 412, a position and an orientation of a device on which the at least three solenoids are arranged are tracked.
[0079] At 414, a three-dimensional speed vector and a three-dimensional position of a virtual object are determined.
[0080] At 416, a time of a virtual impact of the virtual object and an impact force of the virtual impact are estimated based on the position and the orientation of the device and the three-dimensional speed vector and a three-dimensional position of the virtual object.
[0081] At 418, a relative impact force for a respective solenoid of the at least three solenoids is determined based on a position of the respective solenoid of the at least three solenoids relative to a location of the virtual impact.
[0082] At 420, a current is provided to the respective solenoid of the at least three solenoids based on the relative impact force determined for the respective solenoid of the at least three solenoids.
[0083] In various embodiments, the method of actuating at least three solenoids may further include providing a time delay to the at least three solenoids.
[0084] It should be appreciated that the methods illustrated in the embodiments of FIGS. 3, 4A and 4B may similarly be applicable to a haptic system including at least two solenoids. Accordingly, a reference to the terms "at least three solenoids" and "at least three current drivers" in the descriptions related to FIGS. 3, 4A and 4B, may be replaced by the terms "at least two solenoids" and "at least two current drivers", respectively.
[0085] FIG. 5 shows a schematic illustration of a haptic system, according to various embodiments. The haptic system includes a plurality of solenoids, for example as represented by 500a, 500b, 500c, for three solenoids, mounted and arranged on a physical device 502. In various embodiments, the physical device is a sport equipment such as a racket or racquet, such that the plurality of solenoids, e.g. 500a, 500b, 500c, are arranged and located on the racket 502 as shown in FIG. 5. The plurality of solenoids, e.g. 500a, 500b, 500c, may be arranged on or along the rim or frame 504 of a racket head 505 of the racket 502 (e.g. on a circumference or circumferential edge of an impact surface or area 506 of the racket head 505). As shown in FIG. 5, the plurality of solenoids, e.g. 500a, 500b, 500c, are arranged with an at least substantially uniform spacing between adjacent solenoids. However, it should be appreciated that the plurality of solenoids, e.g. 500a, 500b, 500c, may be arranged with a non-uniform spacing between adjacent solenoids.
[0086] The plurality of solenoids, e.g. 500a, 500b, 500c, are actuated or driven to generate a physical sensation and haptic feedback to a user holding or operating the racket 502 to simulate an impact or a collision occuring at an impact or collision point, for example as represented as 508, which the racket 502 experiences against a computer simulated virtual object, e.g. a ball. The plurality of solenoids, e.g. 500a, 500b, 500c, arranged on the racket 502 produce impulsive force or torque feedback on the impact surface 506 of the racket 502. The impact surface 506 of the racket head 505 of the racket 502 is the impact area for striking a virtual object such as a ball.
[0087] Using the impact point 508 as an example, the impact point 508 causes a respective relative impact force as represented by the arrow 510a to the solenoid 500a, a respective relative impact force as represented by the arrow 510b to the solenoid 500b and a respective relative impact force as represented by the arrow 510c to the solenoid 500c. Each of the relative impact forces 510a, 510b, 510c, may be of a different force magnitude.
[0088] It should be appreciated that each solenoid of the plurality of solenoids may be actuated differently from another solenoid such that a vibration sensation felt from a solenoid to another solenoid is different, depending on the location of the impact point on the surface of the physical device (e.g. a racket) so as to provide the user of the physical device a sense of the location or position of the impact point caused by a virtual object on the surface of the physical device. Accordingly, the relative impact force to each solenoid may be different. In addition, therefore, a change in the location of the impact point also causes a change in the actuation of the same solenoid.
[0089] FIG. 6 shows a schematic illustration of a haptic system illustrating force feedback, according to various embodiments. For ease of understanding and clarity purposes, the haptic system includes two solenoids (a first solenoid Si 600a and a second solenoid S2 600b), mounted or arranged along a frame 602, or a circumferential edge of the impact surface or area 604, or a circumference of a racket head 605 of a racket 606. The first solenoid Si 600a and the second solenoid S2 600b are arranged such that the pair of solenoids are on either side of a central axis, as represented by the dotted line 607, of the racket head 605 being parallel to the handle or grip 608 of the racket 606. The first solenoid Si 600a and the second solenoid S2 600b may also be arranged such that they are mirror images of each other about the central axis 607 as shown in FIG. 6. The first solenoid Si 600a and the second solenoid S2 600b may be arranged at least substantially centrally on the frame 602 of the racket head 605 as shown in FIG. 6.
[0090] However, it should be appreciated that the first solenoid Si 600a and the second solenoid S2 600b may be arranged on any locations along the frame 602 of the racket head 605 of the racket 606, for example on one side or either side of the central axis 607. Where the first solenoid Si 600a and the second solenoid S2 600b are arranged on either side of the central axis 607, the first solenoid Si 600a and the second solenoid S2 600b may or may not be arranged as mirror images about the central axis 607.
[0091] In various embodiments, a torque or force feedback may be at a location or holding point 610 (e.g. where the user's hand grip is) at the handle 608 of the racket 606 when a virtual object (e.g. a ball) hits a part of the racket 606. It should be appreciated that the ball may hit any part of the racket 606, at any location on the impact area 604 of the racket 606, including for example at impact point represented as 612, and/or any part of the frame 602 of the racket 606.
[0092] In embodiments where the magnitude of the collision impact force of the ball and the racket 606 is F, and the distance from the holding point 610 to the point where the force acts (e.g. the impact point 612) is r as represented by the arrow 614, the impact torque τ at the grip position or holding point 610 may be given by the following equation:
τ = r x F (Equation 1), where F is the force vector, f is the displacement or distance vector (i.e. a vector from the point from which torque is measured, being the holding point 610, to the point where force is applied, being the impact point 612) and x denotes the cross product.
[0093] In the context of various embodiments, the impact torque τ is a vector.
[0094] Equation 1 may be expanded to the following equations: x = (?; + r2')x F (Equation 2), τ = (Equation 3),
Figure imgf000017_0001
where
r x r. r x r,
?,' = · - r, and ?,' = — 1 11 22, »
η x rJ
fj is the distance vector of the distance n as represented by the arrow 616a, from the holding point 610 to the first solenoid Si 600a,
r2 is the distance vector of the distance r2 as represented by the arrow 616b, from the holding point 610 to the second solenoid Si 600b,
r is the distance vector of the distance r' i as represented by the arrow 618a, from the holding point 610, to the point 620, and
r2' is the distance vector of the distance r'2 as represented by the arrow 618b, from the holding point 610, to the point 622.
[0095] In addition, the torque τ' generated by the first solenoid Si 600a and the second solenoid S2 600b may be given by the equation:
(Equation 4), where FS| is the force vector of the impact force on the first solenoid Si 600a and F¾ is the force vector of the impact force on the second solenoid S2 600b.
[0096] By comparing equation 3 and equation 4, in order to have τ' = T , the values of Fs and FSi may be determined by the following equations:
r x r,
(Equation 5),
(Equation 6).
Figure imgf000017_0002
[0097] In various embodiments, the impact force of each of the first solenoid Si 600a and the second solenoid S2 600b is limited by its technical specification. Where the first solenoid Si 600a and the second solenoid S2 600b have an at least substantially similar technical specifications, the impact force Fs of the first solenoid S] 600a and the impact force F¾ of the second solenoid S2 600b may not be larger than a certain maximum value
Fsmax according to the technical specification. In various embodiments, FSmax may be in a range of between about lg to about 20g (i.e. between about one time of gravity to about 20 times of gravity).
[0098] In various embodiments, by incorporating this limitation of Fsmax into equation 5 and equation 6, the required impact force, having a force vector F^ , for the first solenoid
51 600a and the required impact force, having a force vector Fg , for the second solenoid
52 600b, may be determined from the following equations: ¾ = ^¾ (Equation 7),
% = ¾==-¼, (Equation s), where FM = max( FS] , FSi , FSmax ), or in other words, F has a value equal to the maximum value of either the magnitude of the force vector Fs , the magnitude of the force vector FSi or Fsmax. [0099] As shown in equation 4, Fs and FSz are the force vectors of the impact force by the first solenoid Si 600a and the second solenoid S2 600b, respectively, to generate the torque τ' . As the magnitude of the impact force Fs of the first solenoid Si 600a and the magnitude of the impact force ¥Si of the second solenoid S2 600b may not be larger than
Fsmax, using equation 7 as an example and for Fs > FSi , when Fs is less than Fsmax (i.e. Fs < Fsmax), ¾ is equal to Fs (i.e. F^ = Fs ). Therefore, the force vector of the impact force by the first solenoid Si 600a is Fs . When FSi is greater than Fsmax (i.e. FS] > Fsmax),
¾ is equal to Fsmax (i.e. ¾ = Fsmax). Therefore, the force vector of the impact force by the first solenoid Si 600a is equivalent to Fsmax- [0100] Similarly, for FS5 > Fs , when F is less than Fsma (i-e. FSi < FSmax), ¾ is equal to Fs? (i.e. ¾ = FSi ). Therefore, the force vector of the impact force by the second solenoid S2 600b is F . When F& is greater than Fsmax (i.e. F& > Fsmax), ¾ is equal to
Fsmax (i-e- I¾ = Fsma ). Therefore, the force vector of the impact force by the second solenoid S2 600b is equivalent to Fsmax.
[0101] Therefore, according to equations 7 and 8, the maximum force of the impact force for the first solenoid Si 600a and of the impact force for the second solenoid S2 600b, respectively, may be restricted to Fsmax, while the relation between Fs and FSi may be maintained for generating the torque τ' , as shown in equation 4.
[0102] Therefore, based on the embodiment of FIG. 6 having the first solenoid Si 600a and the second solenoid S2 600b, the maximum torque that may be generated at the holding point 610 may be determined from the following equation:
*n™ = x FSmm + x Sraax = (/j + r2 ) x FSma (Equation 10)
[0103] In various embodiments, in order to increase the generated torque, more than a pair of soleniods (i.e. more than two solenoids) may be used, for example two pairs, three pairs or any higher number of pairs of solenoids arranged on or along the frame 602 or a circumference of the racket head 605 of the racket 606. However, it should be appreciated that any odd number or even number of solenoids may be provided, for example two solenoids, three solenoids, four solenoids, five solenoids, six solenoids, seven solenoids or any higher number of solenoids.
[0104] In various embodiments, N pairs of solenoids may be arranged in a structure or configuration similar to that shown in FIG. 5, where the maximum generated torque are scaled up by a factor of N. Adding more solenoids to the system may provide challenges as this increases power consumption of the system as well as the total weight of the racket with the solenoids. Therefore, it should be appreciated that the number of solenoids incorporated depend on the desired torque and the total tolerable weight for the racket.
[0105] FIG. 7 shows a schematic block diagram of a haptic system 700, according to various embodiments. The haptic system 700 may be used for mixed reality games or virtual sports simulations. The haptic system 700 includes a plurality of solenoids, for example including a first solenoid 702a, a second solenoid 702b and further solenoids to an N-th solenoid 702c. The first solenoid 702a, the second solenoid 702b and further solenoids to the N-th solenoid 702c may be arranged on a physical device similar to the embodiment shown in FIG. 5. The haptic system 700 further includes a microcontroller 704 configured to control and deliver controlled current to each of the plurality of solenoids via a respective driver, such as a current driver, in order to actuate a respective solenoid of the plurality of solenoids and simulate a relative impact force on an impact surface of the physical device (e.g. a racket) from a virtual object (e.g. a virtual ball). As shown in FIG. 7, a driver 706a is in electrical communication with the first solenoid 702a, another driver 706b is in electrical communication with the second solenoid 702b and a further driver 706c is in electrical communication with the N-th solenoid 702c. The drivers 706a, 706b, 706c, are in turn in electrical communication with the microcontroller 704.
[0106] In various embodiments, the microcontroller 704 is configured to control the operations of each of the first solenoid 702a, the second solenoid 702b and further solenoids to the N-th solenoid 702c based on signals wirelessly received from a computer 708, via a wireless transmitter 710 in communication with the computer 708 and a wireless receiver 712 in communication with the microcontroller 704. The communication link between the wireless transmitter 710 and the wireless receiver 712 is represented by the dotted line 714. In various embodiments, the wireless transmitter 710 may be integral to the computer 708 or may be an external module. In various embodiments, the wireless receiver 712 may be integral to the haptic system 700 or may be an external module in communication with the haptic system 700.
[0107] The haptic system 700 may further include one or more wireless tracking sensors with a wireless transmitter, as represented by the block 716, configured to track a position and an orientation of a device incorporating the haptic system 700.
[0108] In various embodiments, the microcontroller 704, the respective drivers, for example drivers 706a, 706b, 706c, the wireless receiver 712 and the one or more wireless tracking sensors with the wireless transmitter, as represented by the block 716, may be provided or integrated on a printed circuit board (PCB), which may be placed in a device incorporating the solenoids (e.g. the first solenoid 702a, the second solenoid 702b and further solenoids to the N-th solenoid 702c). Using FIG. 6 as an example, the PCB may be placed in the handle 608 of the racket 606.
[0109] In a mixed reality game or virtual sports simulation, for example a tennis, squash or table tennis game simulation, where the physical device is a racket held by a user (human player), and the ball is a virtual ball, the haptic system 700 tracks the racket and uses the location information of the racket to estimate the collision time and impact force based on the position and speed of the virtual ball. When a collision or impact is about to happen, the computer 708 sends an activation signal to the racket to activate all the solenoids (e.g. 702a, 702b, 702c) with a certain time delay such that the solenoids and the system are synchronized to the exact moment or time of the collision or impact. In addition to the activation signal, the computer 708 calculates the relative impact force for each solenoid (e.g. 702a, 702b, 702c) individually based on the relative position of the solenoid with respect to the location or position of the collision (i.e. the impact point), similar to the descriptions and calculations as described earlier with respect to the embodiment of FIG. 6.
[0110] Subsequently, the computer 708 sends the relative impact force for each solenoid (e.g. 702a, 702b, 702c) wirelessly to the haptic system 700. This information is received by the microcontroller 704 and is used to modify or vary the electric currents of each of the solenoids via the respective driver (e.g. 706a, 706b, 706c), which consequently affect the impact force of each solenoid in order to provide a realistic force feedback feeling or sensation to the user. In various embodiments, the force impact generated by each solenoid (e.g. 702a, 702b, 702c) is at least substantially equal to the relative impact force determined from the impact force of the virtual impact for each solenoid (e.g. 702a, 702b, 702c).
[0111] The one or more wireless tracking sensors with a wireless transmitter, as represented by the block 716, may be configured to track the position and the orientation of the physical device. These tracking sensors 716 may be used to detect in real time, both the position and the orientation of the racket in a game environment, for example inside a room, in three dimensions. Such tracking information is subsequently sent to the computer 708 wirelessly. The computer 708 uses this information or data together with the three-dimensional speed vector and three-dimensional position of the virtual ball to estimate the moment of the collision or impact and its impact force. The estimated time and force are sent back to the racket wirelessly to activate the solenoids (e.g. 702a, 702b, 702c).
[0112] The haptic system 700 may comprise a memory which is for example used in the processing carried out by the haptic system 700. A memory used in the embodiments may be a volatile memory, for example a DRAM (Dynamic Random Access Memory) or a non-volatile memory, for example a PROM (Programmable Read Only Memory), an EPROM (Erasable PROM), EEPROM (Electrically Erasable PROM), or a flash memory, e.g., a floating gate memory, a charge trapping memory, an MRAM (Magnetoresistive Random Access Memory) or a PCRAM (Phase Change Random Access Memory). In various embodiments, the memory may be included in the microcontroller 704.
[0113] FIG. 8 shows a schematic block diagram of a virtual simulation system 800 (e.g. a tennis sports simulation system) including a haptic system of various embodiments, in the form of a haptic module 802. The haptic module 802 may provide and/or receive haptic signals, as represented by the block 804, in correlation with the various modules or nodes within the virtual sports simulation system 800. The haptic module 802 may be provided on a device, for example a racket, for use by a user or player 810.
[0114] The virtual simulation system 800 further includes a scene-graph rendering module 806 configured to provide graphics and a hybrid tracking module 808 configured to track the player 810 interacting with the virtual simulation system 800. The hybrid tracking module 808 may be configured to track a location of the player 810, for example by sensing or tracking the movement of the player 810, such as tracking the movement of one or more parts or areas of the player 810 (e.g. the player's head and/or hand), and tracking the movement of one or more objects or items attached or held by the player 810. The hybrid tracking module 808 may be further configured to compute or determine the trajectories of one or more parts or areas of the player 810 and/or the trajectories of one or more objects attached or held by the player 810, as a result of movement of the player 810.
[0115] The scene-graph rendering module 806 may include a root 812 (e.g. a processing device) which may control lighting 814, for example of the rendered scene, and include a navigation node 816 for providing different 3D models, for example to provide different adjustable views of the scene/simulation for the virtual simulation system 800 for interaction with the player 810. The navigation node 816 may communicate with the hybrid tracking module 808, for example via a navigation controller 840 integrated with the hybrid tracking module 808. In various embodiments, the navigation node 816 may be in the form of a hardware controller.
[0116] In various embodiments, the navigation node 816 may include a number of components configured to provide different independent models within the virtual simulation system 800. The navigation node 816 may include a court model component configured to provide a 3D court model 818, and a net animation switch 820 configured to provide a number of 3D net models 822, for example of different shapes and colours. The net animation switch 820 may be further configured to provide 3D net models 822 for the continuous movement of the net, for example when a ball hits the net, and/or 3D net models 822 for the display of gentle net movements, for example when there is a breeze.
[0117] The navigation node 816 may further include a ball animation node 824 configured to provide a 3D ball model 826. The navigation node 816 may further include a virtual human animation node 828 configured to provide a number of 3D virtual human and racket models 830 for interacting with the player 810. The virtual human animation node 828 may include data or inputs from an animation and artificial intelligence (AI) module 832 such that the virtual player may be skin-deformed and its 3D racket models may be manipulated for animation. Furthermore, GPU enhanced animation 834 may also be provided to the virtual human animation node 828. It should be appreciated that other 3D models such as a ball serving machine or watching audience may be provided in order to extend the scene and simulation of the game environment.
[0118] During game play, the 3D ball model 826 interacts with other models such as the 3D court model 818, the 3D net models 822, and the 3D virtual human and racket models 830, through fast collision detection 836, which also provides accurate detection. The fast collision detection 836 of the virtual ball and the racket of the player 810 may be achieved via the hybrid tracking module 808. [0119] It should be appreciated that other models may be provided for the virtual simulation system 800 for different simulations which may include, but is not limited to, a military simulation, a driving simulation, and an architectural walkthrough.
[0120] In various embodiments, the scene-graph rendering module 806 may communicate with the hybrid tracking module 808. The scene-graph rendering module 806 and the hybrid tracking module 808 may provide and/or receive haptic signals 804, and therefore are also in communication with the haptic module 802.
[0121] As shown in FIG. 8, the hybrid tracking module 808 may include an inertia tracking system 842 and a camera tracking system (e.g. an optical tracking system) 844. The inertia tracking system 842 may include inertia tracking receivers or inertia trackers. The camera tracking system 844 may include one or more infrared cameras. By using an infrared-based system, the camera tracking system 844 and therefore the hybrid tracking module 808, may be robust to lighting fluctuations. The inertial trackers may provide three degrees of freedom (DOF) orientation tracking data. In various embodiments, the inertia tracking system 842 includes an ultrasonic tracking system, such that the inertia tracking system 842 may be an ultrasonic -inertial tracking system. Therefore, the hybrid tracking module 808 may provide a robust hybrid tracking system including ultrasonic - inertial tracking and optical-based tracking, for example including ultrasonic-inertial tracking devices and infra-red (IR) cameras and markers, which may support fast moving requirements for example, for a tennis game simulation. In various embodiments, the ultrasonic-inertial tracking system may be the IS-900 system by Intersense. In various embodiments, the camera tracking system 844 may be a precision position tracking hybrid system (PPTH) employing infrared-based tracking. The PPTH system incorporates high-speed IR tracking technology (about 175 Hz) which may track fast moving objects such as, for example tennis rackets.
[0122] Infrared-based optical tracking technology such as Augmented Reality Tracker [A.R.T.] or precision position tracking hybrid system [PPTH] have been developed as a vision-based tracking method using multiple IR cameras. Although such technology may be sensitive to sunlight or incandescent light, it may be suitable for an indoor dark or florescent lighting environment. As most VR simulations are projection based and situated indoor, this technology has been widely used to provide the position tracking. [0123] The hybrid tracking module 808 may provide ultrasonic-inertial tracking and optical-based tracking configured to track the player 810 and/or the object (e.g. a racket) held by the player 810 in order to provide three or more degrees of freedom position and orientation data, including instances where occlusion may occur. This may allow the tracking of fast or rapid movements of the player 810 and/or the tracking of the object or equipment held by the player.
[0124] In various embodiments, optical tracking (e.g. PPT-H) provides good quality three DOF position data, but may be occluded due to the inherent nature of vision-based tracking. Therefore, inertial tracking may be used to provide the three DOF orientation data, and ultrasonic tracking may be used to complement optical tracking when occlusion occurs. Therefore, ultrasonic tracking may facilitate tracking even when occlusion occurs.
[0125] The hybrid tracking module 808 may further include a predictive filtering method or algorithm (e.g. a Kalman-filtering based sensor fusion method). As an example and not limitations, the predictive filter used in the embodiment of FIG. 8 may be a Kalman filter. The Kalman filter is a recursive stochastic approach to estimate the state of a dynamic system from a set of incomplete and noisy measurements. The Kalman-filtering based sensor fusion method may be used to smooth the combined position and orientation data from the inertia tracking system 842 and the camera tracking system 844. The Kalman- filtering based sensor fusion method may be further configured to smooth the combined position and orientation data with other data received from the other modules, such as the ball animation node 824 and the virtual human animation node 828 in the virtual simulation system 800. This may produce a more accurate result than individual ultrasonic tracking or optical tracking. Therefore, in various embodiments, the hybrid tracking module 808 may provide robust and smooth six DOF tracking in real time, even with high speed movements of the player 810 and/or the object held by the player 810.
[0126] In various embodiments, a pre-filtering process may be performed to remove outlier (bad or noisy) data so that high fidelity data may be provided for the Kalman filtering process. Therefore, some measurements with obvious large errors may be pre- filtered, and subsequently left out of the Kalman filtering process. Removing these errors may help, for example in filtering out any discrepant estimations, for example by verifying that the distance between the current position and the previous position does not involve an excessive velocity for a moving person or a swinging racket.
[0127] In various embodiments, the virtual simulation system 800 may provide an immersive, interactive and real-time simulation system with, for example a projection system with an L-shape 3D display. The projection system may include two back projected HD Infitec projectors, in order to provide the player 810 with a 3D (stereoscopic) viewing with wide angles. Mirrors may also be provided to reduce the projection space constraint. In the virtual simulation system 800, the scene-graph rendering module 806 provides the graphics and animation, the hybrid tracking module 808 tracks the head and racket movements of the player 810, while the haptic module 802 generates force feedback and vibration for the racket held by the player 810. The virtual simulation system 800 may be developed based on scenegraph architecture, and the virtual simulation system 800 may be easily scaled up to include more modules, depending on applications or when necessary.
[0128] In various embodiments, the hybrid tracking module 808, including an ultrasonic- inertial tracking system (i.e. reference 842 of FIG. 8) with an optical tracking system (i.e. reference 844 of FIG. 8) may provide fast and accurate collision detection of the virtual ball and racket of the player 810. The hybrid tracking module 808 may also constantly and accurately track the head and racket movements of the player 810.
[0129] With the stereoscopic display, the player 820 may be able to feel the virtual ball flying in the air towards the player 810. Together with the haptic feedback on the racket provided by the haptic module 802, when the virtual ball is detected to have collided with the racket through fast collision detection 836, the player 810 is able to feel the vibration and force feedback during game play. Therefore, the virtual simulation system 800 may be able to provide high-definition stereoscopic display, real-time hybrid tracking, animation and haptic (e.g. impulsive force or force feedback and/or torque feedback).
[0130] While the invention has been particularly shown and described with reference to specific embodiments, it should be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. The scope of the invention is thus indicated by the appended claims and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced.

Claims

1. A haptic system for use in a simulation environment or a mixed reality environment, the haptic system comprising at least three solenoids configured to generate a torque feedback.
2. The haptic system as claimed in claim 1, wherein the at least three solenoids are selectively actuated to generate the torque feedback.
3. The haptic system as claimed in claim 1 or 2, wherein the at least three solenoids are arranged with an at least substantially uniform spacing between adjacent solenoids.
4. The haptic system as claimed in any one of claims 1 to 3, wherein the at least three solenoids are configured for use on a device, wherein the device comprises a sports racquet with a racquet head, and wherein the at least three solenoids are arranged on a circumference of the racquet head.
5. The haptic system as claimed in any one of claims 1 to 4, further comprising at least three current drivers, wherein a respective current driver of the at least three current drivers is in electrical communication with a respective solenoid of the at least three solenoids, and wherein the respective current driver is configured to actuate the respective solenoid.
6. The haptic system as claimed in claim 5, further comprising a microcontroller in electrical communication with the at least three current drivers.
7. The haptic system as claimed in claim 6, wherein the microcontroller comprises: a determination circuit configured to determine a three-dimensional speed vector and a three-dimensional position of a virtual object; and
an estimation circuit configured to estimate a time of a virtual impact of the virtual object and an impact force of the virtual impact based on a position and an orientation of a device and the three-dimensional speed vector and a three-dimensional position of the virtual object.
8. The haptic system as claimed in claim 7, wherein the determination circuit is further configured to determine a relative impact force for a respective solenoid of the at least three solenoids based on a position of the respective solenoid of the at least three solenoids relative to a location of the virtual impact.
9. The haptic system as claimed in claim 8, wherein the respective current driver of the at least three current drivers is configured to provide a respective current to the respective solenoid of the at least three solenoids based on the relative impact force.
10. The haptic system as claimed in any one of claims 6 to 9, wherein the microcontroller comprises a time delay circuit configured to provide a time delay to the at least three solenoids.
1 1. The haptic system as claimed in any one of claims 6 to 10, further comprising a wireless receiver in electrical communication with the microcontroller, wherein the wireless receiver is configured to communicate with a processing device.
12. The haptic system as claimed in any one of claims 1 to 11, further comprising at least one tracking sensor.
13. The haptic system as claimed in claim 12, further comprising a wireless transmitter coupled to the at least one tracking sensor.
14. The haptic system as claimed in any one of claims 1 to 13, wherein each of the at least three solenoids is an electromechanical solenoid.
15. A simulation system or a mixed reality environment system comprising a haptic system as claimed in any one of claims 1 to 14.
16. A method of forming a haptic system for use in a simulation environment or a mixed reality environment, the method comprising providing at least three solenoids configured to generate a torque feedback.
17. The method as claimed in claim 16, further comprising arranging the at least three solenoids with an at least substantially uniform spacing between adjacent solenoids.
18. The method as claimed in claim 16 or 17, further comprising arranging the at least three solenoids on a device.
19. The method as claimed in claim 18, wherein the device comprises a sports racquet with a racquet head; and arranging the at least three solenoids comprises arranging on a circumference of the racquet head.
20. The method as claimed in any one of claims 16 to 19, further comprising:
providing at least three current drivers, wherein a respective current driver of the at least three current drivers is in electrical communication with a respective solenoid of the at least three solenoids; and
actuating the respective solenoid with the respective current driver.
21. The method as claimed in claim 20, further comprising providing a microcontroller in electrical communication with the at least three current drivers.
22. The method as claimed in claim 21 , further comprising:
providing a determination circuit in the microcontroller, wherein the determination circuit is configured to determine a three-dimensional speed vector and a three-dimensional position of a virtual object; and
providing an estimation circuit in the microcontroller, wherein the estimation circuit is configured to estimate a time of a virtual impact of the virtual object and an impact force of the virtual impact based on a position and an orientation of a device and the three-dimensional speed vector and a three-dimensional position of the virtual object.
23. The method as claimed in claim 21 or 22, further comprising:
providing a time delay circuit in the microcontroller, wherein the time delay circuit is configured to provide a time delay to the at least three solenoids.
24. The method as claimed in any one of claims 21 to 23, further comprising providing a wireless receiver in electrical communication with the microcontroller, wherein the wireless receiver is configured to communicate with a processing device.
25. The method as claimed in any one of claims 16 to 24, further comprising providing at least one tracking sensor.
26. The method as claimed in claim 25, further comprising coupling a wireless transmitter to the at least one tracking sensor.
27. The method as claimed in any one of claims 16 to 26, wherein each of the at least three solenoids is an electromechanical solenoid.
28. A method of controlling a haptic system for use in a simulation environment or a mixed reality environment, the method comprising:
actuating at least three solenoids configured to generate a torque feedback.
29. The method as claimed in claim 28, further comprising selectively actuating the at least three solenoids to generate the torque feedback.
30. The method as claimed in claim 28 or 29, further comprising arranging the at least three solenoids on a device, and wherein actuating the at least three solenoids comprises tracking a position and an orientation of the device.
31. The method as claimed in claim 30, wherein actuating the at least three solenoids further comprises:
determining a three-dimensional speed vector and a three-dimensional position of a virtual object; and
estimating a time of a virtual impact of the virtual object and an impact force of the virtual impact based on the position and the orientation of the device and the three- dimensional speed vector and a three-dimensional position of the virtual object.
32. The method as claimed in claim 30 or 31, wherein actuating the at least three solenoids further comprises determining a relative impact force for a respective solenoid of the at least three solenoids based on a position of the respective solenoid of the at least three solenoids relative to a location of the virtual impact.
33. The method as claimed in claim 32, wherein actuating the at least three solenoids further comprises providing a current to the respective solenoid of the at least three solenoids based on the relative impact force determined for the respective solenoid of the at least three solenoids.
34. The method as claimed in any one of claims 28 to 33, wherein actuating the at least three solenoids comprises providing a time delay to the at least three solenoids.
35. The method as claimed in any one of claims 28 to 34, further comprising communicating with a processing device.
36. The method as claimed in any one of claims 28 to 35, wherein each of the at least three solenoids is an electromechanical solenoid.
PCT/SG2011/000126 2010-03-26 2011-03-28 A haptic system, a method of forming a haptic system and a method of controlling a haptic system WO2011119118A1 (en)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
SG201002133 2010-03-26
SG201002133-5 2010-03-26
SG201002211-9 2010-03-30
SG201002211 2010-03-30

Publications (1)

Publication Number Publication Date
WO2011119118A1 true WO2011119118A1 (en) 2011-09-29

Family

ID=44673471

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/SG2011/000126 WO2011119118A1 (en) 2010-03-26 2011-03-28 A haptic system, a method of forming a haptic system and a method of controlling a haptic system

Country Status (1)

Country Link
WO (1) WO2011119118A1 (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2012082072A1 (en) * 2010-12-13 2012-06-21 Agency For Science, Technology And Research A haptic actuator, a method of forming the haptic actuator and a method of controlling the haptic actuator, a haptic device and a method of forming the haptic device
CN105144248A (en) * 2013-04-16 2015-12-09 索尼公司 Information processing device and information processing method, display device and display method, and information processing system
CN106325513A (en) * 2016-08-24 2017-01-11 联想(北京)有限公司 Information processing method and information processing device
WO2018237256A1 (en) * 2017-06-22 2018-12-27 Centurion VR, LLC Virtual reality simulation

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6707443B2 (en) * 1998-06-23 2004-03-16 Immersion Corporation Haptic trackball device
US20050052415A1 (en) * 2000-09-28 2005-03-10 Braun Adam C. Directional tactile feedback for haptic feedback interface devices
US20060290662A1 (en) * 2005-06-27 2006-12-28 Coactive Drive Corporation Synchronized vibration device for haptic feedback
US7265750B2 (en) * 1998-06-23 2007-09-04 Immersion Corporation Haptic feedback stylus and other devices
US20100013613A1 (en) * 2008-07-08 2010-01-21 Jonathan Samuel Weston Haptic feedback projection system

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6707443B2 (en) * 1998-06-23 2004-03-16 Immersion Corporation Haptic trackball device
US7265750B2 (en) * 1998-06-23 2007-09-04 Immersion Corporation Haptic feedback stylus and other devices
US20050052415A1 (en) * 2000-09-28 2005-03-10 Braun Adam C. Directional tactile feedback for haptic feedback interface devices
US20060290662A1 (en) * 2005-06-27 2006-12-28 Coactive Drive Corporation Synchronized vibration device for haptic feedback
US20100013613A1 (en) * 2008-07-08 2010-01-21 Jonathan Samuel Weston Haptic feedback projection system

Cited By (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2012082072A1 (en) * 2010-12-13 2012-06-21 Agency For Science, Technology And Research A haptic actuator, a method of forming the haptic actuator and a method of controlling the haptic actuator, a haptic device and a method of forming the haptic device
CN105144248B (en) * 2013-04-16 2019-08-06 索尼公司 Information processing equipment and information processing method, display equipment and display methods and information processing system
CN105144248A (en) * 2013-04-16 2015-12-09 索尼公司 Information processing device and information processing method, display device and display method, and information processing system
EP2988275A4 (en) * 2013-04-16 2016-11-30 Sony Corp Information processing device and information processing method, display device and display method, and information processing system
US10269180B2 (en) 2013-04-16 2019-04-23 Sony Corporation Information processing apparatus and information processing method, display apparatus and display method, and information processing system
CN106325513A (en) * 2016-08-24 2017-01-11 联想(北京)有限公司 Information processing method and information processing device
CN106325513B (en) * 2016-08-24 2019-12-24 联想(北京)有限公司 Information processing method and information processing apparatus
WO2018237256A1 (en) * 2017-06-22 2018-12-27 Centurion VR, LLC Virtual reality simulation
US10279269B2 (en) 2017-06-22 2019-05-07 Centurion VR, LLC Accessory for virtual reality simulation
US10456690B2 (en) 2017-06-22 2019-10-29 Centurion Vr, Inc. Virtual reality simulation of a live-action sequence
US10265627B2 (en) 2017-06-22 2019-04-23 Centurion VR, LLC Virtual reality simulation of a live-action sequence
AU2018289561B2 (en) * 2017-06-22 2020-07-02 Centurion Vr, Inc. Virtual reality simulation
US10792573B2 (en) 2017-06-22 2020-10-06 Centurion Vr, Inc. Accessory for virtual reality simulation
US10792572B2 (en) 2017-06-22 2020-10-06 Centurion Vr, Inc. Virtual reality simulation of a live-action sequence
US10792571B2 (en) 2017-06-22 2020-10-06 Centurion Vr, Inc. Virtual reality simulation of a live-action sequence
US11052320B2 (en) 2017-06-22 2021-07-06 Centurion Vr, Inc. Virtual reality simulation of a live-action sequence
EP4011468A1 (en) * 2017-06-22 2022-06-15 Centurion VR, Inc. Virtual reality simulation
US11872473B2 (en) 2017-06-22 2024-01-16 Centurion Vr, Inc. Virtual reality simulation of a live-action sequence

Similar Documents

Publication Publication Date Title
CA2254854C (en) Method and apparatus for providing simulated physical interactions within computer generated environments
US10657696B2 (en) Virtual reality system using multiple force arrays for a solver
Bowman et al. Questioning naturalism in 3D user interfaces
US20170330362A1 (en) Augmented reality simulation continuum
JP5439367B2 (en) How to change game gearing
JP6005187B2 (en) Method for simulating specific movement by tactile feedback and apparatus for executing the method
JP5087101B2 (en) Program, information storage medium, and image generation system
EP2446341B1 (en) Virtual reality interface system
CN102004840B (en) Method and system for realizing virtual boxing based on computer
CN101515198A (en) Human-computer interaction method for grapping and throwing dummy object and system thereof
US20190212825A1 (en) Haptic feedback device, method and system
Tsai et al. Unity game engine: Interactive software design using digital glove for virtual reality baseball pitch training
KR100893669B1 (en) Virtual-Reality Based Haptic System
CN109240487B (en) Immersive system, control method, and non-transitory computer readable medium
WO2011119118A1 (en) A haptic system, a method of forming a haptic system and a method of controlling a haptic system
Song et al. An immersive VR system for sports education
JP2018126340A (en) Simulation system, program and controller
TW201016274A (en) Virtual sports system
Ishibashi et al. Spider hero: a VR application using pulling force feedback system
TW201944365A (en) A method to enhance first-person-view experience
Sargaana et al. Collaborative billiARds: Towards the ultimate gaming experience
WO2004053636A2 (en) Human-computer interfaces incorporating haptics and path-based interaction
Fogt Pairing a Robot to a Virtual Entity to Provide Haptic Feedback in Virtual Reality
Sargaana et al. billiARds: Augmented Reality System with Wearable Force-Feedback Device
Park et al. Track and Move or Move and Track?: Bi-Directional Interplay between Tracking and Ride Simulator

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 11759814

Country of ref document: EP

Kind code of ref document: A1

DPE1 Request for preliminary examination filed after expiration of 19th month from priority date (pct application filed from 20040101)
NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 11759814

Country of ref document: EP

Kind code of ref document: A1