US20060238549A1 - System and method for object tracking - Google Patents

System and method for object tracking Download PDF

Info

Publication number
US20060238549A1
US20060238549A1 US11/448,454 US44845406A US2006238549A1 US 20060238549 A1 US20060238549 A1 US 20060238549A1 US 44845406 A US44845406 A US 44845406A US 2006238549 A1 US2006238549 A1 US 2006238549A1
Authority
US
United States
Prior art keywords
tracking system
dimensional
edges
dimensional position
color
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US11/448,454
Inventor
Richard Marks
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Interactive Entertainment Inc
Original Assignee
Sony Computer Entertainment Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Computer Entertainment Inc filed Critical Sony Computer Entertainment Inc
Priority to US11/448,454 priority Critical patent/US20060238549A1/en
Assigned to SONY COMPUTER ENTERTAINMENT INC. reassignment SONY COMPUTER ENTERTAINMENT INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: SONY COMPUTER ENTERTAINMENT AMERICA INC.
Assigned to SONY COMPUTER ENTERTAINMENT AMERICA INC. reassignment SONY COMPUTER ENTERTAINMENT AMERICA INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: MARKS, RICHARD L.
Publication of US20060238549A1 publication Critical patent/US20060238549A1/en
Assigned to SONY INTERACTIVE ENTERTAINMENT INC. reassignment SONY INTERACTIVE ENTERTAINMENT INC. CHANGE OF NAME (SEE DOCUMENT FOR DETAILS). Assignors: SONY COMPUTER ENTERTAINMENT INC.
Abandoned legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0354Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of 2D relative movements between the device, or an operating part thereof, and a plane or surface, e.g. 2D mice, trackballs, pens or pucks
    • G06F3/03545Pens or stylus
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/0304Detection arrangements using opto-electronic means
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0346Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/60Analysis of geometric attributes
    • G06T7/62Analysis of geometric attributes of area, perimeter, diameter or volume
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/60Analysis of geometric attributes
    • G06T7/66Analysis of geometric attributes of image moments or centre of gravity
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/70Determining position or orientation of objects or cameras
    • G06T7/73Determining position or orientation of objects or cameras using feature-based methods
    • AHUMAN NECESSITIES
    • A63SPORTS; GAMES; AMUSEMENTS
    • A63FCARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F2300/00Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
    • A63F2300/10Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by input arrangements for converting player-generated signals into game device control signals
    • A63F2300/1012Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by input arrangements for converting player-generated signals into game device control signals involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
    • AHUMAN NECESSITIES
    • A63SPORTS; GAMES; AMUSEMENTS
    • A63FCARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F2300/00Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
    • A63F2300/10Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by input arrangements for converting player-generated signals into game device control signals
    • A63F2300/1087Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by input arrangements for converting player-generated signals into game device control signals comprising photodetecting means, e.g. a camera
    • A63F2300/1093Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by input arrangements for converting player-generated signals into game device control signals comprising photodetecting means, e.g. a camera using visible light

Definitions

  • the present invention relates to computer vision systems, and more particularly to a system in which an object is picked-up via an individual video camera, the camera image is analyzed to isolate the part of the image pertaining to the object, and the position and orientation of the object is mapped into a three-dimensional space.
  • a three-dimensional description of the object is stored in memory and used for controlling action in a game program, such as rendering of a corresponding virtual object in a scene of a video display.
  • Pixel positions of the ball resolved in a given digital image can be related to a specific three-dimensional position of the ball in play and, using triangulation from respective video images, a series of image frames are analyzed by a least-squares method, to fit the positions of the ball to trajectory equations describing unimpeded segments of motion of the ball.
  • Segen is concerned with determining the three-dimensional position of an object in motion from a plurality of two-dimensional video images captured at a point in time. Once the three-dimensional position of the “real” object is known, it is then possible to use this information to control a game program in any number of different ways which are generally known to game programmers.
  • the system of Segen relies on a plurality of video cameras for developing positional information about the object based on triangulation.
  • the detected object of Segen is a simple sphere which does not require information about the orientation (e.g. inclination) of the object in space.
  • the system of Segen is not capable of reconstructing position and orientation of an object, whether moving or at rest, from a two-dimensional video image using a single video camera.
  • the calculated three-dimensional information is used for fixing the position and orientation of the “virtual object” in a memory space of the game console, and then rendering of the image is performed by known projection processing to convert the three-dimensional information into a realistic perspective display.
  • an object tracking system comprising and input device configured to detect two-dimensional input pixel data from a prop device.
  • the system also comprises a multiprocessor unit configured to calculate three-dimensional position and orientation data associated with the proper device from the two-dimensional input pixel data.
  • the present invention also discloses an exemplary method for tracking an object.
  • pixel data is received from an input device.
  • Edges of an object are defined from the received pixel data and three-dimensional position and orientation data of the object are calculated, wherein the edges of the object are associated with the three-dimensional position and orientation data of the prop device.
  • a machine readable medium having embodied thereon a program being executable by a machine to perform a method for tracking an object is also disclosed herein. That tracking method, in accordance with the present exemplary embodiment, generally corresponds to the aforementioned tracking method.
  • FIG. 1 is a block diagram showing an example of a configuration of a main part of a video game console which is adapted to receive an input from a digital video camera.
  • FIG. 2 is an illustration showing movement of a hand held prop, as an auxiliary input device, in front of a digital camera for causing corresponding action on a video display of a game program.
  • FIG. 3 is a block diagram showing the functional blocks required for tracking and discrimination of the prop as it is manipulated by the user.
  • FIG. 4A illustrates a prop device according to one aspect of the present invention.
  • FIG. 4B illustrates a process for mapping two-dimensional pixel data of a cylinder corresponding to the prop device shown in FIG. 4A to a three-dimensional space.
  • FIG. 5A illustrates a prop device according to another aspect of the present invention.
  • FIG. 5B illustrates a process for mapping two-dimensional pixel data of a combined sphere and cylinder corresponding to the prop device shown in FIG. 5A to a three-dimensional space.
  • FIG. 6A illustrates a prop device according to still another aspect of the present invention.
  • FIG. 6B illustrates a process for mapping two dimensional pixel data of stripes provided on a cylinder corresponding to the prop device shown in FIG. 6A to a three-dimensional space on the basis of color transitions at the stripes.
  • FIG. 7 illustrates a prop device having a helical stripe thereon, and provides a description of principles of another aspect of the present invention whereby a rotational component of the prop can be determined.
  • FIGS. 8A and 8B are graphs for describing a two-dimensional chrominance color space, for illustrating principles by which color transitions associated with colored stripes provided on a manipulated object are selected to maximize their detectability.
  • FIG. 1 is a block diagram of a configuration of a main part of a video game console 60 adapted for use with a manipulated object (prop) serving as an alternative input device.
  • the game console 60 constitutes a component of an overall entertainment system 110 according to the present invention which, as shown in FIG. 1 is equipped by a multiprocessor unit MPU 112 for control of the overall system 110 , a main memory 114 which is used for various program operations and for storage of data, a vector calculation unit 116 for performing floating point vector calculations necessary for geometry processing, an image processor 120 for generating data based on controls from the MPU 112 , and for outputting video signals to a monitor 80 (for example a CRT), a graphics interface (GIF) 112 for carrying out mediation and the like over a transmission bus between the MPU 112 or vector calculation unit 116 and the image processor 120 , an input/output port 124 for facilitating reception and transmission of data to and from peripheral devices, an internal OSD functional ROM (OSDROM) 126 constituted by, for example, a flash memory, for performing control of a kernel or the like, and a real time clock 128 having calendar and timer functions.
  • OSDROM OSD functional ROM
  • the main memory 114 , vector calculation unit 116 , GIF 112 , OSDROM 126 , real time clock 128 , and input/output port 124 are connected to the MPU 112 over a data BUS 130 .
  • an image processing unit 138 which is a processor for expanding compressed moving images and texture images, thereby developing the image data.
  • the image processing unit 138 can serve functions for decoding and development of bit streams according to the MPEG2 standard format, macroblock decoding, performing inverse discrete cosine transformations, color space conversion, vector quantization and the like.
  • a sound system is constituted by a sound processing unit SPU 171 for generating musical or other sound effects on the basis of instructions form the MPU 112 , a sound buffer 173 into which waveform data may be recorded by the SPU 171 , and a speaker 175 for outputting the musical or other sound effects generated by the SPU 171 .
  • the speaker 175 may be incorporated as part of the display device 80 or may be provided as a separate audio line-out connection attached to an external speaker 175 .
  • a communications interface 140 is also provided, connected to the BUS 130 , which is an interface having functions of input/output of digital data, and for input of digital contents according to the present invention.
  • user input data may be transmitted to, and status data received from, a server terminal on a network.
  • An input device 132 also known as a controller
  • An optical disk device 136 for reproduction of the contents of an optical disk 70 , for example a CD-ROM or the like on which various programs and data (i.e. data concerning objects, texture data and the like), are connected to the input/output port 124 .
  • the present invention includes a digital video camera 190 which is connected to the input/output port 124 .
  • the input/output port 124 may be embodied by one or more input interfaces, including serial and USB interfaces, wherein the digital video camera 190 may advantageously make use of the USB input or any other conventional interface appropriate for use with the camera 190 .
  • the above-mentioned image processor 120 includes a rendering engine 170 , a main interface 172 , an image memory 174 and a display control device 176 (e.g. a programmable CRT controller, or the like).
  • the rendering engine 170 executes operations for rendering of predetermined image data in the image memory, through the memory interface 172 , and in correspondence with rendering commands which are supplied from the MPU 112 .
  • a first BUS 178 is connected between the memory interface 172 and the rendering engine 170
  • a second BUS is connected between the memory interface 172 and the image memory 174 .
  • First BUS 178 and second BUS 180 respectively, have a bit width of, for example 128 bits, and the rendering engine 170 is capable of executing high speed rendering processing with respect to the image memory.
  • the rendering engine 170 has the capability of rendering, in real time, image data of 320 ⁇ 240 pixels or 640 ⁇ 480 pixels, conforming to, for example, NTSC or PAL standards, and more specifically, at a rate greater than ten to several tens of times per interval of from 1/60 to 1/30 of a second.
  • the image memory 174 employs a unified memory structure in which, for example, a texture rendering region and a display rendering region, can be set in a uniform area.
  • the display controller 176 is structured so as to write the texture data which has been retrieved from the optical disk 70 through the optical disk device 136 , or texture data which has been created on the main memory 114 , to the texture rendering region of the image memory 174 , via the memory interface 172 , and then to read out, via the memory interface 172 , image data which has been rendered in the display rendering region of the image memory 174 , outputting the same to the monitor 80 whereby it is displayed on a screen thereof.
  • FIG. 2 an overall system configuration by which a user holding a prop object manipulates the object in front of a digital video camera, for causing an action to occur in a video game.
  • the prop may comprise a stick-like object which is made up of a handle 303 which is typically black in color, and a brightly colored cylinder (i.e. having a saturated color) 301 .
  • a user stands in front of the video camera 190 , which may comprise a USB webcam or a digital camcorder connected to an input/output port 124 of a game console 60 such as the “Playstation 2” manufactured by Sony Computer Entertainment Inc.
  • the features of the object relating to the cylinder are picked up by the camera 190 , and processing (tobe described later) is performed in order to isolate and discriminate a pixel group corresponding only the cylinder.
  • the three-dimensional description of the object in the memory 114 and a corresponding rendering of the object in the rendering area of image memory 174 , are continuously updated so that the position and orientation of the virtual object, or torch, on the monitor 80 changes as well.
  • the essential information which must be provided is a three-dimensional description of the object, which in the case of FIG. 2 is a cylinder.
  • the image which is picked up by the camera provides only two-dimensional pixel information about the object.
  • FIG. 3 is a block diagram showing the functional blocks used to track and discriminate a pixel group corresponding to the prop as it is being manipulated by the user. It shall be understood that the functions depicted by the blocks are implemented by software which is executed by the MPU 112 in the game console 60 . Moreover, not all of the functions indicted by the blocks in FIG. 3 are used for each embodiment. In particular, color transition localization is used only in the embodiment described in relation to FIGS. 6A and 6B , which shall be discussed below.
  • a color segmentation processing step S 201 is performed, whereby the color of each pixel is determined and the image is divided into various two-dimensional segments of different colors.
  • a color transition localization step S 203 is performed, whereby regions where segments of different colors adjoin are more specifically determined, thereby defining the locations of the image in which distinct color transitions occur.
  • a step for geometry processing S 205 is performed which, depending on the embodiment, comprises either an edge detection process or performing calculations for area statistics, to thereby define in algebraic or geometric terms the lines, curves and/or polygons corresponding to the edges of the object of interest.
  • the pixel area will comprise a generally rectangular shape corresponding to an orthogonal frontal view of the cylinder. From the algebraic or geometric description of the rectangle, it is possible to define the center, width, length and two-dimensional orientation of the pixel group corresponding only to the object.
  • step S 207 The three-dimensional position and orientation of the object are calculated in step S 207 , according to algorithms which are to be described in association with the subsequent descriptions of preferred embodiments of the present invention.
  • the data of three-dimensional position and orientation also undergoes a processing step S 209 for Kalman filtering to improve performance.
  • Such processing is performed to estimate where the object is going to be at a point in time, and to reject spurious measurements that could not be possible, and therefore are considered to lie outside the true data set.
  • Another reason for Kalman filtering is that the camera 190 produces images at 30 Hz, whereas the typical display runs at 60 Hz, so Kalman filtering fills the gaps in the data used for controlling action in the game program. Smoothing of discrete data via Kalman filtering is well known in the field of computer vision and hence will not be elaborated on further.
  • FIG. 4A a prop which is used according to the first embodiment shall be described, and in FIG. 4B a description is given which explains how three-dimensional information of the position and orientation of the prop of FIG. 4A is derived from a two-dimensional video image thereof.
  • the prop is a cylindrical body 301 having a single solid color attached to a handle 303 which is preferably black in color.
  • a position of a given point p typically the center of the object, in the X-Y plane and a depth quantity Z (i.e. the position of the point p on the Z axis) must be determined, together with angular information of the object in at least two different planes, for example, an inclination 0 of the object in the X-Y plane, and an inclination o of the object in the Y-Z plane.
  • the actual physical length and diameter of the cylinder 301 may be used for scaling, but are not essential for programming action in a game program since the virtual object shown on the display need not be of the same length and diameter, or even of the same shape, as the prop.
  • FIG. 4B shows a two-dimensional pixel image 305 of the object produced by the video camera 190 .
  • a frontal orthogonal view of the cylindrical object 301 is picked up in the video image which appears as a generally rectangular pixel group 307 , however, wherein the width of the pixel group can vary along the length l thereof as a result of the object being inclined in the phi ⁇ direction or as a result of the distance overall of the prop from the camera. It will be understood that the inclination in the phi ⁇ direction is not directly visible in the video image 305 .
  • Area statistics include the area, centroid, moment about the X-axis, moment about the Y-axis, principal moments, and the angle of principal moments, which typically are used for calculating moments of inertia of objects about a certain axis.
  • the lamina is of a shape having a geometric center, such as the rectangle in the case of FIG. 4B or a circle in the case of FIG. 5B (to be discussed later), the center of mass of such a lamina corresponds to the geometric center. More generally, if one knows the area statistics of the pixel region and, for example, that the two-dimensional shape is a rectangle, one can directly calculate its width, height and orientation. Similar calculations are possible with circular shapes to determine the center point and radius, for example. Representative calculations for cases of rectangles and circles can be found in standard college-level calculus or physics texts.
  • the X-Y position of the center point p can be derived directly from the image.
  • the theta ⁇ quantity is taken directly from the image simply by knowing any line l, determined in accordance with the geometry processing step S 205 described above, which runs along the longitudinal axis of the pixel group 307 corresponding to the cylinder 301 .
  • a longitudinal line l passing through the center point p is used for this purpose.
  • Determination of the phi ⁇ quantity requires some additional knowledge about the pixel width w in at least two different locations W 1 and W 2 wherein the ratio of the width quantities w 1 : w 2 provides a value which can be used for determining ⁇ . More specifically, if the cylinder 301 is inclined so that the top end thereof is closer to the camera 190 than the lower end of the cylinder, then, since the lower end of the cylinder is at a greater distance from the camera 190 , the pixel width quantity W 2 of the image will have a greater value than the pixel width quantity w 1 , and vice versa.
  • the ratio w 1 :w 2 is proportional to the inclination ⁇ of the cylinder 301 in the Y-Z plane, and therefore the phi quantity ⁇ can be determined from this ratio.
  • a plurality of equidistant measurements of pixel widths between ends of the pixel group 307 are taken, and averaging is performed to determine the ratio w 1 :w 2 .
  • Determination of the depth quantity Z can be done in different ways. However, it is important to recognize that the size and number of pixels making up the two-dimensional pixel group 307 are affected both by the inclination of the object in the ⁇ direction as well as by the actual distance of the physical object from the video camera 190 . More specifically, as the object inclines in the ⁇ direction, the apparent length of the object as seen by the video camera tends to shorten, so that the length l of the pixel group shortens as well. However, at the same time, as the object moves farther away from the camera along the Z-axis, the apparent size of the object overall, including its length l, also becomes smaller. Therefore, it is insufficient simply to look at the length l alone as an indicator of how far away from the camera the object is. Stated otherwise, the depth quantity Z must be determined as a function of both l and ⁇ .
  • a phi-weighted value of l which we may call l ⁇
  • the pixel length of l ⁇ in the image which changes as the object is moved closer or farther from the camera while assuming that ⁇ stays constant, then can be used to determine the depth quantity Z since l ⁇ , will be proportional to Z.
  • Another method for determining depth Z is to count the total number of pixels in the pixel group 307 corresponding to the object. As the object gets closer to or farther away from the camera, the number of pixels making up the pixel group 307 increases or decreases respectively, in proportion to the depth quantity Z. However, again, the number of pixels in the pixel group 307 is also affected by the inclination in the phi ⁇ direction, so the number of pixels N must first be weighted by phi ⁇ to result in a weighted quantity N ⁇ which is used for determining the depth quantity Z based on a proportional relationship between N ⁇ and Z.
  • Yet another advantageous method for determining the depth quantity Z is to use the average width w avg of the rectangle, which is calculated as the sum of a given number of width measurements of the rectangle divided by the number of width measurements taken. It should be clear that the average width of the pixel group is affected only by Z and not by the phi-inclination of the cylinder. It is also possible to determine phi ⁇ from the ratio of the total length of the pixel group to the average width (i.e. l: w avg ), and moreover, wherein the sign of the phi-inclination can be determined based on whether w 1 is greater or less than w 2 .
  • FIG. 5A a prop which is used according to another embodiment shall be described, and in FIG. 5B a description is given which explains how three-dimensional information of the position and orientation of the prop of FIG. 5A is derived from a two-dimensional video image thereof.
  • the prop according to the second embodiment similar to the first embodiment shown in FIG. 4A , comprises a cylindrical stick-shaped object, however in this case, a spherical object 309 of a different color is rigidly affixed to one end of the cylinder 301 .
  • a distal end of the cylinder may be provided which protrudes just slightly and is visible from an upper end of the sphere 309 .
  • the sphere 309 provides a simplified means for determining the depth quantity Z and the inclination of the object in the phi ⁇ direction, which does not require measurement of relative widths of the cylinder 301 , and which does not require any weighting of the length quantity by phi ⁇ in order to determine the depth quantity Z.
  • a pixel group 311 corresponding to the sphere 309 in the image appears as a two-dimensional circle.
  • a radius R and center point p s of the circle are determined according to area statistics calculations which have already been discussed above.
  • the total number of pixels making up the pixel group 311 of the circle can be counted for giving a pixel area of the circle. It will be appreciated that the circular pixel area will increase as the spherical object 309 comes closer to the camera 190 and vice versa, and therefore, since the total number of pixels in the pixel group 311 making up the circle is proportional to the depth quantity Z, the value for Z can be determined thereby.
  • the shape and size of the circular pixel group are not influenced as a result of the phi ⁇ angle of inclination. More specifically, even if the object overall is tilted in the phi direction, the sphere 309 and the pixel group 311 will retain their general shape and, unlike the length of the cylinder 301 , will not become foreshortened as a result of such tilting. Therefore, an advantage is obtained in that the total number of pixels of the pixel group making up the circle in the image can always be related proportionately to the depth quantity Z and, for determining Z, phi-weighting as in the previous embodiment is not required.
  • Determination of inclination of the object in the theta ⁇ direction is done directly from the image, just as in the previous embodiment, by determining the angle theta ⁇ between a center longitudinal line of the pixel group 307 corresponding to the cylinder 301 of the Y-axis.
  • Determining the angle of inclination in the phi ⁇ direction is handled somewhat differently than the previous embodiment. More specifically, such a quantity can be determined by knowledge of the depth quantity Z, determined as described above, and by the length l between the center point of the circle 311 and the center point of the pixel group 307 which corresponds to the cylinder 301 . For any known and fixed depth quantity Z, the length l (as viewed from the perspective of the camera) becomes shorter as the object is tilted in the phi ⁇ direction. Therefore, if the Z quantity is known, it is possible to determine, simply from the length l, the degree of inclination in the phi ⁇ direction, and it is not necessary to calculate a relative width quantity of ratio of widths, as in the embodiment shown by FIGS. 4A and 4B .
  • FIG. 6A illustrates a prop device according to still another aspect of the present invention.
  • the prop itself comprises a generally cylindrical body 301 .
  • three stripes S 1 , S 2 and S 3 having a different color than the cylinder itself are provided on the cylindrical body.
  • the stripes S 1 , S 2 and S 3 are each equal in width and are spaced equidistant from each other, at either end of the cylinder 301 and in the center thereof.
  • a pixel group making up the cylinder is extracted from the image to provide a two-dimensional line along which to look for color transitions.
  • positions are determined at which color transitions along any line l in the longitudinal direction of the cylinder 301 occur.
  • the chrominance values Cr and Cb which are output as part of the YCrCb signals from the video camera 190 are detected.
  • the angle ⁇ is taken directly as the angle between the longitudinal line of the cylinder and the Y axis, basically in the same manner as the preceding embodiments.
  • the ratio of the lengths l 1 :l 2 is used. For example, in the case (as shown) in which the cylinder is inclined in the ⁇ direction toward the camera 190 , with the upper end of the cylinder being closer to the camera than the lower end, the length l 1 will appear longer to the camera 190 (since it is closer) than the length 12 . It will also be appreciated that, although the apparent lengths l 1 and l 2 will also be affected by the overall distance Z of the object from the camera 190 , the ratio of these lengths l 1 :l 2 will not change and therefore this ratio provides a constant indication of the inclination of the cylinder 301 in the phi ⁇ direction.
  • is determined from the ratio of l 1 and l 2 , and once phi ⁇ is known, the total depth quantity Z can be determined from the sum of l 1 and l 2 .
  • Each of the tracking methods described above can be used to obtain five of the six degrees of freedom of the objects.
  • the only one missing is the rotation of the cylinder about its axis.
  • Information about the rotation of the cylinder would seem difficult to obtain because cylinders are symmetric in rotation about this axis.
  • the approach taken by the present invention to obtain this rotational component is to add a helical stripe S H that goes around the cylinder 301 exactly once. As the cylinder 301 is rotated, the height of the stripe S H will correspond to the rotation angle.
  • the cylinder 301 (or the cylinder-part of the prop in the case of FIGS. 5A and 5B ) includes the single helical strip S H thereon which goes around the object only once.
  • Information pertaining to the helical stripe is extracted, either from the entire pixel group 313 which makes up the helical stripe or by using the color transitions corresponding to the helical stripe S H , in order to determine, using the geometry processing discussed above, a helix H which best fits to the stripe S H .
  • a center line l of the pixel group corresponding to the cylinder is determined as described previously. Also the overall length l of the pixel group is determined.
  • the camera 190 only sees one side (or orthogonal projection) of the cylinder 301 at a time. Accordingly, the helix H determined from the extracted region of the camera image determines the degree of revolution of the cylinder 301 . More specifically, as shown, assuming no rotation (i.e. a rotational component of 0 degrees), a center line extending from one end to a point on the helix will have a first height h 1 , whereas if the object is rotated by 45 degrees, the height of the center line l between the lower end to the point where it intersects the helix H will have a shorter height h 2 . The condition shown by the far right-hand side of FIG.
  • additional information with respect to the object and orientation of the cylinder 301 in three-dimensional space can be provided. Such information can be used to control the rotation of a virtual object, for example, when displayed in a game program.
  • FIG. 8A shows a diagram of a color space defined by luminance and radial coordinates of hue and saturation.
  • Luminance is the brightness or intensity of the color
  • hue is the shift in the dominant wavelength of a spectral distribution
  • saturation is the concentration of a spectral distribution at one wavelength.
  • FIG. 8B shows a two-dimensional chrominance color space corresponding to the Cr and Cb chrominance output signals of the video camera.
  • video cameras output signals for controlling the color of each pixel making up a video image.
  • color can be defined using radial coordinates corresponding respectively to hue and saturation.
  • YCrCb color definition is another more useful standard for defining color, which is the most common representation of color used in the video world.
  • YCrCb represents each color by a single luma component (Y) and two components of chrominance Cr and Cb.
  • Y may be loosely related to brightness of luminance whereas Cr and Cb make up a quantities loosely related to hue.
  • Cr and Cb chrominance signals for each pixel are defined by Cartesian coordinates which also can be used to determine a location within the color wheel corresponding to a certain hue and saturation.
  • the color of the stripes S 1 , S 2 and S 3 and the color of the cylinder 301 are chosen in such a way as to maximize stripe detectability for the video camera.
  • Color-based tracking is notorious for its difficulties due to changes in lighting, which cause the apparent color to vary.
  • the color of blue as perceived by the camera, to vary to such a degree that accurate detection of the object is made difficult.
  • by looking for color transitions instead of absolute colors a more robust tracking solution can be attained. For example, in the embodiment of FIGS.
  • highly saturated colors of blue and orange are located at substantially diametrically opposed sides of the color wheel and are separated by a large distance D in the color space.
  • the actual distance D can be calculated as the hypotenuse of a triangle having sides defined by ⁇ Cr (i.e. the difference in the Cr chrominance signal values for the two colors of blue and orange) and ⁇ Cb (i.e. the difference in the Cb chrominance signal values for the same two colors), and hence the actual distance D is the square root of ( ⁇ Cr) 2 +( ⁇ Cb) 2 , as already discussed above in equation (4).
  • the method provides a general criteria whereby colors may be selected using their chrominance signals Cr and Cb in such a manner to maximize their separation in the color space.
  • a generally applicable method for the selection of colors, as well as for calculating distance between any two colors is performed in such a way that the distance between two colors is calculated as a distance projected onto a certain diameter-spoke of the color wheel.
  • a given diameter-spoke on the color wheel is selected having a certain angle of orientation ⁇ .
  • the angle of orientation of the selected diameter on the color wheel it is possible to select the color transitions one wants to detect. For example, if green is (1, 1) and magenta is ( ⁇ 1, ⁇ 1), the diameter of the spoke should be set at an orientation ⁇ of 45 degrees. Then the color separation distance is calculated simply by projecting the colors onto the 45 degree line.
  • the distance calculation shown by equation (5) can therefore also be used for setting the threshold D t based on a predetermined orientation defined by the angle ⁇ . For example, if the color transitions for the object were in fact green and magenta, the general distance calculation above can be used for threshold setting, while fixing the angle ⁇ of this equation at 45 degrees.
  • a “virtual object” that forms a moving image in a game display corresponding to how the “real” object is moved or positioned
  • the three-dimensional information can be used to control game programs in any number of different ways foreseeable to persons skilled in the art.
  • a “theremin” like musical effect can be achieved wherein changes in the position and orientation of the manipulated object could be used to influence volume, tone, pitch, rhythm and so forth of sounds produced by the sound processor.
  • Such a musical or rhythmic sound effect can be provided in combination with visual effects displayed on the screen of the game console, for enhancing the experience perceived by the game player.

Abstract

A system for tracking an object is disclosed. The exemplary tracking system comprises an input device configured to detect two-dimensional input pixel data from a prop device and a multiprocessor unit configured to calculate three-dimensional position and orientation data associated with the prop device from the two-dimensional input pixel data. An exemplary method for tracking an object is also disclosed. Through this exemplary method, pixel data is received from an input device and edges of an object are defined. Three-dimensional position and orientation data of the object are calculated, wherein the edges of the object are associated with the three-dimensional position and orientation data of the prop device.

Description

    CROSS-REFERENCE TO RELATED APPLICATIONS
  • The present application is a continuation and claims the priority benefit of U.S. patent application Ser. No. 10/928,778 entitled “System and Method for Object Tracking,” filed Aug. 26, 2004, which is a continuation and claims the priority benefit of U.S. patent application Ser. No. 09/621,578 entitled “Method for Mapping an Object from a Two-Dimensional Camera Image to a Three-Dimensional Space for Controlling Action in a Game Program,” filed Jul. 21, 2000, and now U.S. Pat. No. 6,795,068. The disclosure of this commonly owned application is incorporated herein by reference.
  • This application is related to U.S. patent application Ser. No. 10/927,918 entitled “Method for Color Transition Detection,” filed Aug. 26, 2004 and now U.S. patent number 7,______, which is a divisional and claims the priority benefit of U.S. patent application Ser. No. 09/621,578 entitled “Method for Mapping an Object from a Two-Dimensional Camera Image to a Three-Dimensional Space for Controlling Action in a Game Program,” filed Jul. 21, 2000, and now U.S. Pat. No. 6,795,068. The disclosure of this commonly owned application is incorporated herein by reference.
  • BACKGROUND OF THE INVENTION
  • 1. Field of the Invention
  • The present invention relates to computer vision systems, and more particularly to a system in which an object is picked-up via an individual video camera, the camera image is analyzed to isolate the part of the image pertaining to the object, and the position and orientation of the object is mapped into a three-dimensional space. A three-dimensional description of the object is stored in memory and used for controlling action in a game program, such as rendering of a corresponding virtual object in a scene of a video display.
  • 2. Background of the Invention
  • Tracking of moving objects using digital video cameras and processing the video images for producing various displays has been known in the art. One such application, for producing an animated video version of a sporting event, has been disclosed by Segen, U.S. Pat. No. 6,072,504, the disclosure of which is incorporated in the present specification by reference. According to this system, the position of a tennis ball during play is tracked using a plurality of video cameras, and a set of equations relating the three-dimensional points in the court to two-dimensional points (i.e. pixels) of digital images within the field of view of the cameras are employed. Pixel positions of the ball resolved in a given digital image can be related to a specific three-dimensional position of the ball in play and, using triangulation from respective video images, a series of image frames are analyzed by a least-squares method, to fit the positions of the ball to trajectory equations describing unimpeded segments of motion of the ball.
  • As described in some detail by Segen, once a three-dimensional description of position and motion of an object has been determined, various methods exist which are well known in the art for producing an animated representation thereof using a program which animates appropriate object movement in a video game environment.
  • Stated otherwise, Segen is concerned with determining the three-dimensional position of an object in motion from a plurality of two-dimensional video images captured at a point in time. Once the three-dimensional position of the “real” object is known, it is then possible to use this information to control a game program in any number of different ways which are generally known to game programmers.
  • However, the system of Segen relies on a plurality of video cameras for developing positional information about the object based on triangulation. Moreover, the detected object of Segen is a simple sphere which does not require information about the orientation (e.g. inclination) of the object in space. Thus, the system of Segen is not capable of reconstructing position and orientation of an object, whether moving or at rest, from a two-dimensional video image using a single video camera.
  • It is common for game programs to have virtual objects formed from a combination of three-dimensional geometric shapes, wherein during running of a game program, three-dimensional descriptions (positions and orientations) of the objects relative to each other are determined by control input parameters entered using an input device such as a joystick, game controller or other input device. The three-dimensional position and orientation of the virtual objects are then projected into a two-dimensional display (with background, lighting and shading, texture, and so forth) to create a three-dimensional perspective scene or rendition by means of the rendering processor functions of the game console.
  • As an example, there can be “virtual object” that forms a moving image in a game display corresponding to how one moves around the “real” object. To display the virtual object, the calculated three-dimensional information is used for fixing the position and orientation of the “virtual object” in a memory space of the game console, and then rendering of the image is performed by known projection processing to convert the three-dimensional information into a realistic perspective display.
  • However, in spite of the above knowledge and techniques, problems continue to hinder successful object tracking, and a particularly difficult problem is extracting precisely only those pixels of a video image which correspond unambiguously to an object of interest. For example, although movement of an object having one color against a solid background of another color, where the object and background colors vary distinctly from one another, can be accomplished with relative ease, tracking of objects, even if brightly colored, is not so easy in the case of multi-colored or non-static backgrounds. Changes in lighting also dramatically affect the apparent color of the object as seen by the video camera, and thus object tracking methods which rely on detecting a particular colored object are highly susceptible to error or require constant re-calibration as lighting conditions change. The typical home use environment for video game programs demands much greater flexibility and robustness than possible with conventional object tracking computer vision systems.
  • SUMMARY OF THE INVENTION
  • In one exemplary embodiment of the present invention, an object tracking system is provided. The exemplary tracking system comprises and input device configured to detect two-dimensional input pixel data from a prop device. The system also comprises a multiprocessor unit configured to calculate three-dimensional position and orientation data associated with the proper device from the two-dimensional input pixel data.
  • The present invention also discloses an exemplary method for tracking an object. Through this exemplary method, pixel data is received from an input device. Edges of an object are defined from the received pixel data and three-dimensional position and orientation data of the object are calculated, wherein the edges of the object are associated with the three-dimensional position and orientation data of the prop device.
  • A machine readable medium having embodied thereon a program being executable by a machine to perform a method for tracking an object is also disclosed herein. That tracking method, in accordance with the present exemplary embodiment, generally corresponds to the aforementioned tracking method.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • FIG. 1 is a block diagram showing an example of a configuration of a main part of a video game console which is adapted to receive an input from a digital video camera.
  • FIG. 2 is an illustration showing movement of a hand held prop, as an auxiliary input device, in front of a digital camera for causing corresponding action on a video display of a game program.
  • FIG. 3 is a block diagram showing the functional blocks required for tracking and discrimination of the prop as it is manipulated by the user.
  • FIG. 4A illustrates a prop device according to one aspect of the present invention.
  • FIG. 4B illustrates a process for mapping two-dimensional pixel data of a cylinder corresponding to the prop device shown in FIG. 4A to a three-dimensional space.
  • FIG. 5A illustrates a prop device according to another aspect of the present invention.
  • FIG. 5B illustrates a process for mapping two-dimensional pixel data of a combined sphere and cylinder corresponding to the prop device shown in FIG. 5A to a three-dimensional space.
  • FIG. 6A illustrates a prop device according to still another aspect of the present invention.
  • FIG. 6B illustrates a process for mapping two dimensional pixel data of stripes provided on a cylinder corresponding to the prop device shown in FIG. 6A to a three-dimensional space on the basis of color transitions at the stripes.
  • FIG. 7 illustrates a prop device having a helical stripe thereon, and provides a description of principles of another aspect of the present invention whereby a rotational component of the prop can be determined.
  • FIGS. 8A and 8B are graphs for describing a two-dimensional chrominance color space, for illustrating principles by which color transitions associated with colored stripes provided on a manipulated object are selected to maximize their detectability.
  • DETAILED DESCRIPTION
  • FIG. 1 is a block diagram of a configuration of a main part of a video game console 60 adapted for use with a manipulated object (prop) serving as an alternative input device.
  • The game console 60 constitutes a component of an overall entertainment system 110 according to the present invention which, as shown in FIG. 1 is equipped by a multiprocessor unit MPU 112 for control of the overall system 110, a main memory 114 which is used for various program operations and for storage of data, a vector calculation unit 116 for performing floating point vector calculations necessary for geometry processing, an image processor 120 for generating data based on controls from the MPU 112, and for outputting video signals to a monitor 80 (for example a CRT), a graphics interface (GIF) 112 for carrying out mediation and the like over a transmission bus between the MPU 112 or vector calculation unit 116 and the image processor 120, an input/output port 124 for facilitating reception and transmission of data to and from peripheral devices, an internal OSD functional ROM (OSDROM) 126 constituted by, for example, a flash memory, for performing control of a kernel or the like, and a real time clock 128 having calendar and timer functions.
  • The main memory 114, vector calculation unit 116, GIF 112, OSDROM 126, real time clock 128, and input/output port 124 are connected to the MPU 112 over a data BUS 130.
  • Further connected to the BUS 130 is an image processing unit 138 which is a processor for expanding compressed moving images and texture images, thereby developing the image data. For example, the image processing unit 138 can serve functions for decoding and development of bit streams according to the MPEG2 standard format, macroblock decoding, performing inverse discrete cosine transformations, color space conversion, vector quantization and the like.
  • A sound system is constituted by a sound processing unit SPU 171 for generating musical or other sound effects on the basis of instructions form the MPU 112, a sound buffer 173 into which waveform data may be recorded by the SPU 171, and a speaker 175 for outputting the musical or other sound effects generated by the SPU 171. It should be understood that the speaker 175 may be incorporated as part of the display device 80 or may be provided as a separate audio line-out connection attached to an external speaker 175.
  • A communications interface 140 is also provided, connected to the BUS 130, which is an interface having functions of input/output of digital data, and for input of digital contents according to the present invention. For example, through the communications interface 140, user input data may be transmitted to, and status data received from, a server terminal on a network. An input device 132 (also known as a controller) for input of data (e.g. key input data or coordinate data) with respect to the entertainment system 110, an optical disk device 136 for reproduction of the contents of an optical disk 70, for example a CD-ROM or the like on which various programs and data (i.e. data concerning objects, texture data and the like), are connected to the input/output port 124.
  • As a further extension or alternative to the input device, the present invention includes a digital video camera 190 which is connected to the input/output port 124. The input/output port 124 may be embodied by one or more input interfaces, including serial and USB interfaces, wherein the digital video camera 190 may advantageously make use of the USB input or any other conventional interface appropriate for use with the camera 190.
  • The above-mentioned image processor 120 includes a rendering engine 170, a main interface 172, an image memory 174 and a display control device 176 (e.g. a programmable CRT controller, or the like).
  • The rendering engine 170 executes operations for rendering of predetermined image data in the image memory, through the memory interface 172, and in correspondence with rendering commands which are supplied from the MPU 112.
  • A first BUS 178 is connected between the memory interface 172 and the rendering engine 170, and a second BUS is connected between the memory interface 172 and the image memory 174. First BUS 178 and second BUS 180, respectively, have a bit width of, for example 128 bits, and the rendering engine 170 is capable of executing high speed rendering processing with respect to the image memory.
  • The rendering engine 170 has the capability of rendering, in real time, image data of 320×240 pixels or 640×480 pixels, conforming to, for example, NTSC or PAL standards, and more specifically, at a rate greater than ten to several tens of times per interval of from 1/60 to 1/30 of a second.
  • The image memory 174 employs a unified memory structure in which, for example, a texture rendering region and a display rendering region, can be set in a uniform area.
  • The display controller 176 is structured so as to write the texture data which has been retrieved from the optical disk 70 through the optical disk device 136, or texture data which has been created on the main memory 114, to the texture rendering region of the image memory 174, via the memory interface 172, and then to read out, via the memory interface 172, image data which has been rendered in the display rendering region of the image memory 174, outputting the same to the monitor 80 whereby it is displayed on a screen thereof.
  • There shall now be described, with reference to FIG. 2, an overall system configuration by which a user holding a prop object manipulates the object in front of a digital video camera, for causing an action to occur in a video game.
  • As shown in FIG. 2, the prop may comprise a stick-like object which is made up of a handle 303 which is typically black in color, and a brightly colored cylinder (i.e. having a saturated color) 301. A user stands in front of the video camera 190, which may comprise a USB webcam or a digital camcorder connected to an input/output port 124 of a game console 60 such as the “Playstation 2” manufactured by Sony Computer Entertainment Inc. As the user moves the object in front of the camera 190, the features of the object relating to the cylinder are picked up by the camera 190, and processing (tobe described later) is performed in order to isolate and discriminate a pixel group corresponding only the cylinder. A three-dimensional description of the cylinder, including its position and orientation in three-dimensional space, is calculated, and this description is correspondingly stored in the main memory 114 of the game console 60. Then, using rendering techniques known in the art, the three-dimensional description of the object is used to cause action in a game program which is displayed on the display screen of the monitor 80. For example, a virtual object, shown as a torch for example, can be moved throughout the scene of the game, corresponding to the movements of the real object made by the user. As the user changes the position and orientation of the object by moving it, the three-dimensional description of the object in the memory 114, and a corresponding rendering of the object in the rendering area of image memory 174, are continuously updated so that the position and orientation of the virtual object, or torch, on the monitor 80 changes as well.
  • As noted above, the essential information which must be provided is a three-dimensional description of the object, which in the case of FIG. 2 is a cylinder. However, the image which is picked up by the camera provides only two-dimensional pixel information about the object. Moreover, it is necessary to discriminate the pixels which relate only the object itself before a three-dimensional description thereof can be calculated.
  • FIG. 3 is a block diagram showing the functional blocks used to track and discriminate a pixel group corresponding to the prop as it is being manipulated by the user. It shall be understood that the functions depicted by the blocks are implemented by software which is executed by the MPU 112 in the game console 60. Moreover, not all of the functions indicted by the blocks in FIG. 3 are used for each embodiment. In particular, color transition localization is used only in the embodiment described in relation to FIGS. 6A and 6B, which shall be discussed below.
  • Initially the pixel data input from the camera is supplied to the game console 60 through the input/output port interface 124, enabling the following processes to be performed thereon. First, as each pixel of the image is sampled, for example, on a raster basis, a color segmentation processing step S201 is performed, whereby the color of each pixel is determined and the image is divided into various two-dimensional segments of different colors. Next, for certain embodiments, a color transition localization step S203 is performed, whereby regions where segments of different colors adjoin are more specifically determined, thereby defining the locations of the image in which distinct color transitions occur. Then, a step for geometry processing S205 is performed which, depending on the embodiment, comprises either an edge detection process or performing calculations for area statistics, to thereby define in algebraic or geometric terms the lines, curves and/or polygons corresponding to the edges of the object of interest. For example, in the case of the cylinder shown in FIG. 2 the pixel area will comprise a generally rectangular shape corresponding to an orthogonal frontal view of the cylinder. From the algebraic or geometric description of the rectangle, it is possible to define the center, width, length and two-dimensional orientation of the pixel group corresponding only to the object.
  • The three-dimensional position and orientation of the object are calculated in step S207, according to algorithms which are to be described in association with the subsequent descriptions of preferred embodiments of the present invention.
  • Lastly, the data of three-dimensional position and orientation also undergoes a processing step S209 for Kalman filtering to improve performance. Such processing is performed to estimate where the object is going to be at a point in time, and to reject spurious measurements that could not be possible, and therefore are considered to lie outside the true data set. Another reason for Kalman filtering is that the camera 190 produces images at 30 Hz, whereas the typical display runs at 60 Hz, so Kalman filtering fills the gaps in the data used for controlling action in the game program. Smoothing of discrete data via Kalman filtering is well known in the field of computer vision and hence will not be elaborated on further.
  • In FIG. 4A, a prop which is used according to the first embodiment shall be described, and in FIG. 4B a description is given which explains how three-dimensional information of the position and orientation of the prop of FIG. 4A is derived from a two-dimensional video image thereof.
  • As shown in FIG. 4A, the prop is a cylindrical body 301 having a single solid color attached to a handle 303 which is preferably black in color. In order to fully define the position and orientation of the object in a three-dimensional space, a position of a given point p, typically the center of the object, in the X-Y plane and a depth quantity Z (i.e. the position of the point p on the Z axis) must be determined, together with angular information of the object in at least two different planes, for example, an inclination 0 of the object in the X-Y plane, and an inclination o of the object in the Y-Z plane. The actual physical length and diameter of the cylinder 301, together with knowledge of the focal length of the camera, may be used for scaling, but are not essential for programming action in a game program since the virtual object shown on the display need not be of the same length and diameter, or even of the same shape, as the prop.
  • Referring now to FIG. 4B, this figure shows a two-dimensional pixel image 305 of the object produced by the video camera 190. A frontal orthogonal view of the cylindrical object 301 is picked up in the video image which appears as a generally rectangular pixel group 307, however, wherein the width of the pixel group can vary along the length l thereof as a result of the object being inclined in the phi ø direction or as a result of the distance overall of the prop from the camera. It will be understood that the inclination in the phi ø direction is not directly visible in the video image 305.
  • To determine the length, center point, etc. of the pixel group 307 in accordance with the geometry processing step S205 discussed above, known area statistics calculations are used. Area statistics include the area, centroid, moment about the X-axis, moment about the Y-axis, principal moments, and the angle of principal moments, which typically are used for calculating moments of inertia of objects about a certain axis. For example, to determine the moments about the X and Y axes, respectively, if each pixel making up the pixel group is considered to correspond to a particle of a given uniform mass m in making up a thin homogeneous sheet or lamina, then the moments about x and y axes of a system of n such particles (or pixels) located in a coordinate plane are defined as follows: M x = k = 1 n m y k ( 1 ) M y = k = 1 n m x k ( 2 )
  • The center of mass of this system is located at the point (x, y) given by x = M y m , y = M x m ( 3 )
  • Further, assuming the lamina is of a shape having a geometric center, such as the rectangle in the case of FIG. 4B or a circle in the case of FIG. 5B (to be discussed later), the center of mass of such a lamina corresponds to the geometric center. More generally, if one knows the area statistics of the pixel region and, for example, that the two-dimensional shape is a rectangle, one can directly calculate its width, height and orientation. Similar calculations are possible with circular shapes to determine the center point and radius, for example. Representative calculations for cases of rectangles and circles can be found in standard college-level calculus or physics texts.
  • Because the image 305 is already taken to be in the X-Y plane, the X-Y position of the center point p can be derived directly from the image. Also, the theta θ quantity is taken directly from the image simply by knowing any line l, determined in accordance with the geometry processing step S205 described above, which runs along the longitudinal axis of the pixel group 307 corresponding to the cylinder 301. Typically, a longitudinal line l passing through the center point p is used for this purpose.
  • Determination of the phi ø quantity requires some additional knowledge about the pixel width w in at least two different locations W1 and W2 wherein the ratio of the width quantities w1: w2 provides a value which can be used for determining ø. More specifically, if the cylinder 301 is inclined so that the top end thereof is closer to the camera 190 than the lower end of the cylinder, then, since the lower end of the cylinder is at a greater distance from the camera 190, the pixel width quantity W2 of the image will have a greater value than the pixel width quantity w1, and vice versa. The ratio w1:w2 is proportional to the inclination ø of the cylinder 301 in the Y-Z plane, and therefore the phi quantity ø can be determined from this ratio. Typically, for better accuracy, a plurality of equidistant measurements of pixel widths between ends of the pixel group 307 are taken, and averaging is performed to determine the ratio w1:w2.
  • Determination of the depth quantity Z can be done in different ways. However, it is important to recognize that the size and number of pixels making up the two-dimensional pixel group 307 are affected both by the inclination of the object in the ø direction as well as by the actual distance of the physical object from the video camera 190. More specifically, as the object inclines in the ø direction, the apparent length of the object as seen by the video camera tends to shorten, so that the length l of the pixel group shortens as well. However, at the same time, as the object moves farther away from the camera along the Z-axis, the apparent size of the object overall, including its length l, also becomes smaller. Therefore, it is insufficient simply to look at the length l alone as an indicator of how far away from the camera the object is. Stated otherwise, the depth quantity Z must be determined as a function of both l and ø.
  • However, if the phi quantity ø has already been determined and is known, a phi-weighted value of l, which we may call lø, can be determined, and the pixel length of lø in the image, which changes as the object is moved closer or farther from the camera while assuming that ø stays constant, then can be used to determine the depth quantity Z since lø, will be proportional to Z.
  • Another method for determining depth Z is to count the total number of pixels in the pixel group 307 corresponding to the object. As the object gets closer to or farther away from the camera, the number of pixels making up the pixel group 307 increases or decreases respectively, in proportion to the depth quantity Z. However, again, the number of pixels in the pixel group 307 is also affected by the inclination in the phi ø direction, so the number of pixels N must first be weighted by phi ø to result in a weighted quantity Nø which is used for determining the depth quantity Z based on a proportional relationship between Nø and Z.
  • Yet another advantageous method for determining the depth quantity Z is to use the average width wavg of the rectangle, which is calculated as the sum of a given number of width measurements of the rectangle divided by the number of width measurements taken. It should be clear that the average width of the pixel group is affected only by Z and not by the phi-inclination of the cylinder. It is also possible to determine phi ø from the ratio of the total length of the pixel group to the average width (i.e. l: wavg), and moreover, wherein the sign of the phi-inclination can be determined based on whether w1 is greater or less than w2.
  • In FIG. 5A, a prop which is used according to another embodiment shall be described, and in FIG. 5B a description is given which explains how three-dimensional information of the position and orientation of the prop of FIG. 5A is derived from a two-dimensional video image thereof.
  • The prop according to the second embodiment, similar to the first embodiment shown in FIG. 4A, comprises a cylindrical stick-shaped object, however in this case, a spherical object 309 of a different color is rigidly affixed to one end of the cylinder 301. In addition, although not shown, a distal end of the cylinder may be provided which protrudes just slightly and is visible from an upper end of the sphere 309. As shall be explained below, the sphere 309 provides a simplified means for determining the depth quantity Z and the inclination of the object in the phi ø direction, which does not require measurement of relative widths of the cylinder 301, and which does not require any weighting of the length quantity by phi ø in order to determine the depth quantity Z.
  • As shown in FIG. 5B, a pixel group 311 corresponding to the sphere 309 in the image appears as a two-dimensional circle. According to this embodiment, a radius R and center point ps of the circle are determined according to area statistics calculations which have already been discussed above. In this case, further, the total number of pixels making up the pixel group 311 of the circle can be counted for giving a pixel area of the circle. It will be appreciated that the circular pixel area will increase as the spherical object 309 comes closer to the camera 190 and vice versa, and therefore, since the total number of pixels in the pixel group 311 making up the circle is proportional to the depth quantity Z, the value for Z can be determined thereby.
  • It should also be realized that, unlike the cylinder in the previous embodiment, the shape and size of the circular pixel group are not influenced as a result of the phi ø angle of inclination. More specifically, even if the object overall is tilted in the phi direction, the sphere 309 and the pixel group 311 will retain their general shape and, unlike the length of the cylinder 301, will not become foreshortened as a result of such tilting. Therefore, an advantage is obtained in that the total number of pixels of the pixel group making up the circle in the image can always be related proportionately to the depth quantity Z and, for determining Z, phi-weighting as in the previous embodiment is not required.
  • Determination of inclination of the object in the theta θ direction is done directly from the image, just as in the previous embodiment, by determining the angle theta θ between a center longitudinal line of the pixel group 307 corresponding to the cylinder 301 of the Y-axis.
  • Determining the angle of inclination in the phi ø direction is handled somewhat differently than the previous embodiment. More specifically, such a quantity can be determined by knowledge of the depth quantity Z, determined as described above, and by the length l between the center point of the circle 311 and the center point of the pixel group 307 which corresponds to the cylinder 301. For any known and fixed depth quantity Z, the length l (as viewed from the perspective of the camera) becomes shorter as the object is tilted in the phi ø direction. Therefore, if the Z quantity is known, it is possible to determine, simply from the length l, the degree of inclination in the phi ø direction, and it is not necessary to calculate a relative width quantity of ratio of widths, as in the embodiment shown by FIGS. 4A and 4B.
  • FIG. 6A illustrates a prop device according to still another aspect of the present invention.
  • As in the embodiment shown in FIG. 6A, the prop itself comprises a generally cylindrical body 301. In addition, three stripes S1, S2 and S3 having a different color than the cylinder itself are provided on the cylindrical body. Preferably, the stripes S1, S2 and S3 are each equal in width and are spaced equidistant from each other, at either end of the cylinder 301 and in the center thereof.
  • According to this embodiment, a pixel group making up the cylinder is extracted from the image to provide a two-dimensional line along which to look for color transitions. To determine the quantities Z, θ and ø, positions are determined at which color transitions along any line l in the longitudinal direction of the cylinder 301 occur.
  • More specifically, as shown in FIG. 6B, a group made up of only those pixels corresponding to a line l along the longitudinal direction of the cylinder body 301, as viewed by the camera, needs to be sampled in order to determine where along the line l distinct color transitions occur. In particular, for detecting such color transitions, the chrominance values Cr and Cb which are output as part of the YCrCb signals from the video camera 190 are detected. For reasons which shall be explained below in connection with the criteria for selecting the stripe colors, it is preferable to use a combined chrominance value D made up of a Pythagorean distance of the combined chrominance signals Cr and Cb for each color of the cylinder 301 and stripes S1, S2 and S3, respectively, thereby defining a separation in the two-dimensional chrominance color space used by the video camera 190, according to the following formula (1):
    D=√{square root over ((ΔCr)2+(ΔCb)2 )}  (4)
  • By selecting colors which maximize the value of D (to be explained in more detail later), it is possible to select a threshold Dt at which only color transitions above a certain separation where D>Dt are considered to correspond to the color transitions of the stripes S1, S2 and S3. Accordingly, the pixels along the line of the cylinder are filtered, using such a threshold, in order to find the large color transitions corresponding to the stripes S1, S2 and S3.
  • As shown in FIG. 6B, at positions along the line l where color transitions occur, for each stripe two spikes corresponding to positions where color transitions appear can be detected, and the center point between these spikes is taken to be the position of the stripes. Once the positions of the stripes are fixed, it is then a matter of course to determine the lengths l1 and l2 between the stripes, wherein the overall length of the cylinder is determined by the sum of l1 and l2.
  • It shall next be explained how knowledge of l1 and l2 provides sufficient information for determining the quantities of Z, θ and Φ, necessary for describing the position and orientation of the object in three dimensions.
  • First, since the line l defined by the pixels running along the length of the cylinder has already been determined, and since the camera is assumed to face normally to the X-Y plane, the angle θ is taken directly as the angle between the longitudinal line of the cylinder and the Y axis, basically in the same manner as the preceding embodiments.
  • For determining the angle of inclination in the phi ø direction, the ratio of the lengths l1:l2 is used. For example, in the case (as shown) in which the cylinder is inclined in the ø direction toward the camera 190, with the upper end of the cylinder being closer to the camera than the lower end, the length l1 will appear longer to the camera 190 (since it is closer) than the length 12. It will also be appreciated that, although the apparent lengths l1 and l2 will also be affected by the overall distance Z of the object from the camera 190, the ratio of these lengths l1:l2 will not change and therefore this ratio provides a constant indication of the inclination of the cylinder 301 in the phi ø direction.
  • For determining the depth quantity Z, a procedure similar to that of the first embodiment is employed, wherein a phi-weighted quantity 1Φ of the total length 1(1=l1+l2) is determined for giving Z. More specifically, the influence of the inclination angle ø on the total apparent length l of the object is first determined, and then the total length, properly weighted by the influence of ø, is proportional to the distance (or depth quantity) Z of the object from the camera 190.
  • Stated more simply, ø is determined from the ratio of l1 and l2, and once phi ø is known, the total depth quantity Z can be determined from the sum of l1 and l2.
  • There shall now be described, in connection with FIG. 7, a method for determining a rotational component of the prop. This method may be applied in conjunction with any of the embodiments which have discussed above, by further equipping the prop device with a helical stripe SH thereon.
  • Each of the tracking methods described above can be used to obtain five of the six degrees of freedom of the objects. The only one missing is the rotation of the cylinder about its axis. Information about the rotation of the cylinder would seem difficult to obtain because cylinders are symmetric in rotation about this axis. The approach taken by the present invention to obtain this rotational component is to add a helical stripe SH that goes around the cylinder 301 exactly once. As the cylinder 301 is rotated, the height of the stripe SH will correspond to the rotation angle.
  • More specifically, as shown in FIG. 7, the cylinder 301 (or the cylinder-part of the prop in the case of FIGS. 5A and 5B) includes the single helical strip SH thereon which goes around the object only once. Information pertaining to the helical stripe is extracted, either from the entire pixel group 313 which makes up the helical stripe or by using the color transitions corresponding to the helical stripe SH, in order to determine, using the geometry processing discussed above, a helix H which best fits to the stripe SH.
  • In addition to the helix H, a center line l of the pixel group corresponding to the cylinder is determined as described previously. Also the overall length l of the pixel group is determined.
  • For obtaining a degree of rotation of the cylinder, various heights h (only h1 and h2 are shown for simplicity) each of which define the distance between one end of the cylinder and the point p where the center line intersects the helix are determined.
  • As shown on the right-hand side of FIG. 7, the camera 190 only sees one side (or orthogonal projection) of the cylinder 301 at a time. Accordingly, the helix H determined from the extracted region of the camera image determines the degree of revolution of the cylinder 301. More specifically, as shown, assuming no rotation (i.e. a rotational component of 0 degrees), a center line extending from one end to a point on the helix will have a first height h1, whereas if the object is rotated by 45 degrees, the height of the center line l between the lower end to the point where it intersects the helix H will have a shorter height h2. The condition shown by the far right-hand side of FIG. 7, at a rotation of 90 degrees, represents to a unique case in which the center line will intersect the helix at two points. Hence, by calculating the heights of the center line l, a component of rotation of the cylinder 301 (or any other object affixed to the cylinder and rotated thereby) can be determined.
  • The specific quantity used for determining rotation is the ratio of the detected height between the lower end and the point on the helix to the total length I of the pixel group. This ratio gives a number from 0 to k (where k=hmax/1), which maps directly to a range of from 0 to 360 degrees. Thus, additional information with respect to the object and orientation of the cylinder 301 in three-dimensional space can be provided. Such information can be used to control the rotation of a virtual object, for example, when displayed in a game program.
  • Next, with respect to FIGS. 8A and 8B, a process for selection of colors for the stripes, according to the embodiments of FIGS. 6A and 6B shall now be described. More specifically, FIG. 8A shows a diagram of a color space defined by luminance and radial coordinates of hue and saturation. Luminance is the brightness or intensity of the color, hue is the shift in the dominant wavelength of a spectral distribution, and saturation is the concentration of a spectral distribution at one wavelength.
  • By contrast, FIG. 8B shows a two-dimensional chrominance color space corresponding to the Cr and Cb chrominance output signals of the video camera. It is well understood in the art that video cameras output signals for controlling the color of each pixel making up a video image. As shown by the color wheel diagram of FIG. 8A, color can be defined using radial coordinates corresponding respectively to hue and saturation. However, as it is needlessly complex for computerized image processing to use radial coordinates, another more useful standard for defining color is the YCrCb color definition, which is the most common representation of color used in the video world. YCrCb represents each color by a single luma component (Y) and two components of chrominance Cr and Cb. Y may be loosely related to brightness of luminance whereas Cr and Cb make up a quantities loosely related to hue. These components are defined more rigorously in ITU-R BT.601-4 (Studio encoding parameters of digital television for standard 4:3 and wide-screen 16:9 aspect ratios) published by the International Telecommunication Union. Thus, the Cr and Cb chrominance signals for each pixel are defined by Cartesian coordinates which also can be used to determine a location within the color wheel corresponding to a certain hue and saturation.
  • According to the present invention, the color of the stripes S1, S2 and S3 and the color of the cylinder 301 are chosen in such a way as to maximize stripe detectability for the video camera. Color-based tracking is notorious for its difficulties due to changes in lighting, which cause the apparent color to vary. As a result, if one is attempting to detect a certain color of blue corresponding to an object, for example, under certain lighting conditions it is possible for the color of blue, as perceived by the camera, to vary to such a degree that accurate detection of the object is made difficult. In the present invention, by looking for color transitions instead of absolute colors, a more robust tracking solution can be attained. For example, in the embodiment of FIGS. 6A and 6B, if the cylinder 301 is blue and the stripes S1, S2 and S3 are orange, if lighting conditions change, then the apparent colors will also change. However, the transition between these colors, as shown in FIG. 6B, will still be very evident.
  • As discussed above, video cameras capture data using the two-dimensional chrominance color space shown in FIG. 8B. By choosing colors for the object and stripes, respectively, which have a maximal separation D in this space, it is possible to significantly enhance the detectability of the color transitions.
  • More specifically, as shown in FIG. 8B, highly saturated colors of blue and orange are located at substantially diametrically opposed sides of the color wheel and are separated by a large distance D in the color space. The actual distance D can be calculated as the hypotenuse of a triangle having sides defined by ΔCr (i.e. the difference in the Cr chrominance signal values for the two colors of blue and orange) and ΔCb (i.e. the difference in the Cb chrominance signal values for the same two colors), and hence the actual distance D is the square root of (ΔCr)2+(ΔCb)2, as already discussed above in equation (4).
  • Although blue and orange have been described as an example, it will be appreciated that any other color pairs, for example green and magenta, which also possess a large separation in the chrominance color space may be used. In other words, the method provides a general criteria whereby colors may be selected using their chrominance signals Cr and Cb in such a manner to maximize their separation in the color space.
  • More specifically, a generally applicable method for the selection of colors, as well as for calculating distance between any two colors, is performed in such a way that the distance between two colors is calculated as a distance projected onto a certain diameter-spoke of the color wheel. First, a given diameter-spoke on the color wheel is selected having a certain angle of orientation θ. By choosing the angle of orientation of the selected diameter on the color wheel, it is possible to select the color transitions one wants to detect. For example, if green is (1, 1) and magenta is (−1, −1), the diameter of the spoke should be set at an orientation θ of 45 degrees. Then the color separation distance is calculated simply by projecting the colors onto the 45 degree line. In this manner, for the case of green and magenta, the computed distance is exactly the same as the Pythagorean distance D discussed above, however with a diameter-line orientation of 45 degrees, the distance between blue and orange is zero, because they both project to the origin. This tells us that, for a selected diameter line of 45 degrees, green and magenta are the optimal colors for detection, since they possess the maximum separation in the color space for this diameter.
  • Thus, for any given diameter angle of θ, which can be chosen from 0 to 180 degrees, the separation between two colors (Cr1, Cb1) and (Cr2, Cb2) may be calculated according to equation (5) as follows:
    D=[Cr 1·cos θ+Cb 1·sin θ]−[Cr 2·cos θ+Cb2·sin θ]  (5)
  • The distance calculation shown by equation (5) can therefore also be used for setting the threshold Dt based on a predetermined orientation defined by the angle θ. For example, if the color transitions for the object were in fact green and magenta, the general distance calculation above can be used for threshold setting, while fixing the angle θ of this equation at 45 degrees.
  • Herein have been described several methods for determining the position and orientation of a real object manipulated in front of a video camera, by mapping the two-dimensional image information of the object captured by the camera to a three-dimensional space, wherein a three dimensional description including position and orientation of the object may be used to control action in a game program.
  • Although one clear example of controlling a game program is to have a “virtual object” that forms a moving image in a game display corresponding to how the “real” object is moved or positioned, it will be appreciated that the three-dimensional information can be used to control game programs in any number of different ways foreseeable to persons skilled in the art. For example, a “theremin” like musical effect can be achieved wherein changes in the position and orientation of the manipulated object could be used to influence volume, tone, pitch, rhythm and so forth of sounds produced by the sound processor. Such a musical or rhythmic sound effect can be provided in combination with visual effects displayed on the screen of the game console, for enhancing the experience perceived by the game player.
  • It shall be understood that other modifications will be apparent and can be easily made by persons skilled in the art without departing from the scope and spirit of the present invention. Accordingly, the following claims shall not be limited by the descriptions or illustrations set forth herein, but shall be construed to cover with reasonable breadth all features which may be envisioned as equivalents by those skilled in the art.

Claims (22)

1. An object tracking system comprising:
an input device configured to detect two-dimensional input pixel data from a prop device; and
a multiprocessor unit configured to calculate three-dimensional position and orientation data associated with the prop device from the two-dimensional input pixel data.
2. The object tracking system of claim 1, wherein the multiprocessor unit comprises a memory configured to store the three-dimensional position and orientation data associated with the prop device.
3. The object tracking system of claim 1, wherein the multiprocessor unit comprises an image processor configured to execute operations for rendering the three-dimensional position and orientation data associated with the prop device.
4. The object tracking system of claim 1, further comprising a monitor for displaying action in a game program caused by rendering the three-dimensional position and orientation data associated with the prop device.
5. The object tracking system of claim 1, wherein the prop device comprises a saturated color.
6. The object tracking system of claim 1, wherein the input device is a camera.
7. The object tracking system of claim 2, wherein the memory further comprises a display rendering region.
8. The object tracking system of claim 2, wherein the memory further comprises a texture rendering region.
9. The object tracking system of claim 1, wherein the multiprocessor unit is further configured to:
determine the color of each pixel in the two-dimensional input pixel data; and
define edges of an object by dividing an image comprising the input pixel-data into two-dimensional segments of color, wherein the defined edges are associated with the three-dimensional position and orientation data of the prop device.
10. The object tracking system of claim 9, wherein the multiprocessor unit is further configured to localize color transitions whereby distinct color transitions are defined prior to defining the edges of the object in the image.
11. The object tracking system of claim 1 further comprising a filter, wherein the filter is configured to filter the three-dimensional position and orientation data.
12. The object tracking system of claim 11, wherein the filter comprises a Kalman filter.
13. The object tracking system of claim 9, wherein the multiprocessor unit is further configured to employ an edge detection process to define the edges of the object.
14. The object tracking system of claim 9, wherein the multiprocessor unit is further configured to employ area statistics calculations to define the edges of the object.
15. The object tracking system of claim 9, wherein the definition of an edge of the object is algebraic.
16. The object tracking system of claim 9, wherein the definition of an edge of the object is geometric.
17. A method for tracking an object, comprising:
receiving pixel data from an input device;
defining edges of an object from the received pixel data; and
calculating three-dimensional position and orientation data of the object, wherein the defined edges are associated with the three-dimensional position and orientation data of the object.
18. The method of claim 17, further comprising localizing color transitions whereby distinct color transitions are defined prior to defining the edges of the object.
19. The method of claim 17, further comprising the application of a Kalman filter to the three-dimensional position and orientation data.
20. The method of claim 17, wherein defining the edges of the object comprises employing an edge detection process.
21. The method of claim 17, wherein defining the edges of the object comprises employing area statistics calculations.
22. A machine readable medium having embodied thereon a program being executable by a machine to perform a method for tracking an object, the method comprising:
receiving pixel data from an input device;
defining edges of an object from the received pixel data; and
calculating three-dimensional position and orientation data of the object, wherein the defined edges are associated with the three-dimensional position and orientation data of the object.
US11/448,454 2000-07-21 2006-06-06 System and method for object tracking Abandoned US20060238549A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US11/448,454 US20060238549A1 (en) 2000-07-21 2006-06-06 System and method for object tracking

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US09/621,578 US6795068B1 (en) 2000-07-21 2000-07-21 Prop input device and method for mapping an object from a two-dimensional camera image to a three-dimensional space for controlling action in a game program
US10/928,778 US20050026689A1 (en) 2000-07-21 2004-08-26 System and method for object tracking
US11/448,454 US20060238549A1 (en) 2000-07-21 2006-06-06 System and method for object tracking

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US10/928,778 Continuation US20050026689A1 (en) 2000-07-21 2004-08-26 System and method for object tracking

Publications (1)

Publication Number Publication Date
US20060238549A1 true US20060238549A1 (en) 2006-10-26

Family

ID=24490736

Family Applications (4)

Application Number Title Priority Date Filing Date
US09/621,578 Expired - Lifetime US6795068B1 (en) 2000-07-21 2000-07-21 Prop input device and method for mapping an object from a two-dimensional camera image to a three-dimensional space for controlling action in a game program
US10/927,918 Expired - Lifetime US7113193B2 (en) 2000-07-21 2004-08-26 Method for color transition detection
US10/928,778 Abandoned US20050026689A1 (en) 2000-07-21 2004-08-26 System and method for object tracking
US11/448,454 Abandoned US20060238549A1 (en) 2000-07-21 2006-06-06 System and method for object tracking

Family Applications Before (3)

Application Number Title Priority Date Filing Date
US09/621,578 Expired - Lifetime US6795068B1 (en) 2000-07-21 2000-07-21 Prop input device and method for mapping an object from a two-dimensional camera image to a three-dimensional space for controlling action in a game program
US10/927,918 Expired - Lifetime US7113193B2 (en) 2000-07-21 2004-08-26 Method for color transition detection
US10/928,778 Abandoned US20050026689A1 (en) 2000-07-21 2004-08-26 System and method for object tracking

Country Status (3)

Country Link
US (4) US6795068B1 (en)
EP (1) EP1176559A3 (en)
JP (2) JP3901960B2 (en)

Cited By (25)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110119073A1 (en) * 2009-11-18 2011-05-19 Al Cure Technologies LLC Method and Apparatus for Verification of Medication Administration Adherence
US20110153360A1 (en) * 2009-12-23 2011-06-23 Al Cure Technologies LLC Method and Apparatus for Verification of Clinical Trial Adherence
US20110153361A1 (en) * 2009-12-23 2011-06-23 Al Cure Technologies LLC Method and Apparatus for Management of Clinical Trials
US20110231202A1 (en) * 2010-03-22 2011-09-22 Ai Cure Technologies Llc Method and apparatus for collection of protocol adherence data
US8605165B2 (en) 2010-10-06 2013-12-10 Ai Cure Technologies Llc Apparatus and method for assisting monitoring of medication adherence
US8884949B1 (en) 2011-06-06 2014-11-11 Thibault Lambert Method and system for real time rendering of objects from a low resolution depth camera
US9116553B2 (en) 2011-02-28 2015-08-25 AI Cure Technologies, Inc. Method and apparatus for confirmation of object positioning
US9256776B2 (en) 2009-11-18 2016-02-09 AI Cure Technologies, Inc. Method and apparatus for identification
US9293060B2 (en) 2010-05-06 2016-03-22 Ai Cure Technologies Llc Apparatus and method for recognition of patient activities when obtaining protocol adherence data
US9317916B1 (en) 2013-04-12 2016-04-19 Aic Innovations Group, Inc. Apparatus and method for recognition of medication administration indicator
US9342817B2 (en) 2011-07-07 2016-05-17 Sony Interactive Entertainment LLC Auto-creating groups for sharing photos
US9399111B1 (en) 2013-03-15 2016-07-26 Aic Innovations Group, Inc. Method and apparatus for emotional behavior therapy
US9436851B1 (en) 2013-05-07 2016-09-06 Aic Innovations Group, Inc. Geometric encrypted coded image
US9665767B2 (en) 2011-02-28 2017-05-30 Aic Innovations Group, Inc. Method and apparatus for pattern tracking
US9679113B2 (en) 2014-06-11 2017-06-13 Aic Innovations Group, Inc. Medication adherence monitoring system and method
US9754357B2 (en) 2012-03-23 2017-09-05 Panasonic Intellectual Property Corporation Of America Image processing device, stereoscoopic device, integrated circuit, and program for determining depth of object in real space generating histogram from image obtained by filming real space and performing smoothing of histogram
US9824297B1 (en) 2013-10-02 2017-11-21 Aic Innovations Group, Inc. Method and apparatus for medication identification
US9875666B2 (en) 2010-05-06 2018-01-23 Aic Innovations Group, Inc. Apparatus and method for recognition of patient activities
US9883786B2 (en) 2010-05-06 2018-02-06 Aic Innovations Group, Inc. Method and apparatus for recognition of inhaler actuation
US10116903B2 (en) 2010-05-06 2018-10-30 Aic Innovations Group, Inc. Apparatus and method for recognition of suspicious activities
US10558845B2 (en) 2011-08-21 2020-02-11 Aic Innovations Group, Inc. Apparatus and method for determination of medication location
US10762172B2 (en) 2010-10-05 2020-09-01 Ai Cure Technologies Llc Apparatus and method for object confirmation and tracking
US10786736B2 (en) 2010-05-11 2020-09-29 Sony Interactive Entertainment LLC Placement of user information in a game space
US10949983B2 (en) 2015-12-18 2021-03-16 Ricoh Company, Ltd. Image processing apparatus, image processing system, image processing method, and computer-readable recording medium
US11170484B2 (en) 2017-09-19 2021-11-09 Aic Innovations Group, Inc. Recognition of suspicious activities in medication administration

Families Citing this family (197)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7899511B2 (en) 2004-07-13 2011-03-01 Dexcom, Inc. Low oxygen in vivo analyte sensor
US9155496B2 (en) 1997-03-04 2015-10-13 Dexcom, Inc. Low oxygen in vivo analyte sensor
AUPQ363299A0 (en) * 1999-10-25 1999-11-18 Silverbrook Research Pty Ltd Paper based information inter face
US6795068B1 (en) * 2000-07-21 2004-09-21 Sony Computer Entertainment Inc. Prop input device and method for mapping an object from a two-dimensional camera image to a three-dimensional space for controlling action in a game program
DE60236866D1 (en) * 2001-05-24 2010-08-12 Tecey Software Dev Kg Llc OPTICAL BUS ARRANGEMENT FOR A COMPUTER SYSTEM
US8010174B2 (en) 2003-08-22 2011-08-30 Dexcom, Inc. Systems and methods for replacing signal artifacts in a glucose sensor data stream
US8364229B2 (en) * 2003-07-25 2013-01-29 Dexcom, Inc. Analyte sensors having a signal-to-noise ratio substantially unaffected by non-constant noise
US8260393B2 (en) 2003-07-25 2012-09-04 Dexcom, Inc. Systems and methods for replacing signal data artifacts in a glucose sensor data stream
US7613491B2 (en) 2002-05-22 2009-11-03 Dexcom, Inc. Silicone based membranes for use in implantable glucose sensors
US7161579B2 (en) 2002-07-18 2007-01-09 Sony Computer Entertainment Inc. Hand-held computer interactive device
US7646372B2 (en) 2003-09-15 2010-01-12 Sony Computer Entertainment Inc. Methods and systems for enabling direction detection when interfacing with a computer program
US7623115B2 (en) 2002-07-27 2009-11-24 Sony Computer Entertainment Inc. Method and apparatus for light input device
US7883415B2 (en) 2003-09-15 2011-02-08 Sony Computer Entertainment Inc. Method and apparatus for adjusting a view of a scene being displayed according to tracked head motion
US7102615B2 (en) * 2002-07-27 2006-09-05 Sony Computer Entertainment Inc. Man-machine interface using a deformable device
US8797260B2 (en) 2002-07-27 2014-08-05 Sony Computer Entertainment Inc. Inertially trackable hand-held controller
US8313380B2 (en) 2002-07-27 2012-11-20 Sony Computer Entertainment America Llc Scheme for translating movements of a hand-held controller into inputs for a system
US7760248B2 (en) 2002-07-27 2010-07-20 Sony Computer Entertainment Inc. Selective sound source listening in conjunction with computer interactive processing
US8686939B2 (en) 2002-07-27 2014-04-01 Sony Computer Entertainment Inc. System, method, and apparatus for three-dimensional input control
US7782297B2 (en) 2002-07-27 2010-08-24 Sony Computer Entertainment America Inc. Method and apparatus for use in determining an activity level of a user in relation to a system
US8570378B2 (en) 2002-07-27 2013-10-29 Sony Computer Entertainment Inc. Method and apparatus for tracking three-dimensional movements of an object using a depth sensing camera
US9393487B2 (en) 2002-07-27 2016-07-19 Sony Interactive Entertainment Inc. Method for mapping movements of a hand-held controller to game commands
US9474968B2 (en) 2002-07-27 2016-10-25 Sony Interactive Entertainment America Llc Method and system for applying gearing effects to visual tracking
US9682319B2 (en) 2002-07-31 2017-06-20 Sony Interactive Entertainment Inc. Combiner method for altering game gearing
JP2004199496A (en) * 2002-12-19 2004-07-15 Sony Corp Information processor and method, and program
US7283983B2 (en) * 2003-01-09 2007-10-16 Evolution Robotics, Inc. Computer and vision-based augmented interaction in the use of printed media
US9177387B2 (en) 2003-02-11 2015-11-03 Sony Computer Entertainment Inc. Method and apparatus for real time motion capture
US8072470B2 (en) 2003-05-29 2011-12-06 Sony Computer Entertainment Inc. System and method for providing a real-time three-dimensional interactive environment
RU2426590C2 (en) * 2003-07-18 2011-08-20 Бакстер Интернэшнл Инк. Method of production, use and composition of minor spherical particles produced in controlled phase separation
US7761130B2 (en) 2003-07-25 2010-07-20 Dexcom, Inc. Dual electrode system for a continuous analyte sensor
US7591801B2 (en) 2004-02-26 2009-09-22 Dexcom, Inc. Integrated delivery device for continuous glucose sensor
US20070208245A1 (en) * 2003-08-01 2007-09-06 Brauker James H Transcutaneous analyte sensor
US8845536B2 (en) * 2003-08-01 2014-09-30 Dexcom, Inc. Transcutaneous analyte sensor
US8160669B2 (en) * 2003-08-01 2012-04-17 Dexcom, Inc. Transcutaneous analyte sensor
US8060173B2 (en) 2003-08-01 2011-11-15 Dexcom, Inc. System and methods for processing analyte sensor data
US9135402B2 (en) 2007-12-17 2015-09-15 Dexcom, Inc. Systems and methods for processing sensor data
US20190357827A1 (en) 2003-08-01 2019-11-28 Dexcom, Inc. Analyte sensor
US20100168543A1 (en) 2003-08-01 2010-07-01 Dexcom, Inc. System and methods for processing analyte sensor data
US8886273B2 (en) 2003-08-01 2014-11-11 Dexcom, Inc. Analyte sensor
US7920906B2 (en) 2005-03-10 2011-04-05 Dexcom, Inc. System and methods for processing analyte sensor data for sensor calibration
US20140121989A1 (en) 2003-08-22 2014-05-01 Dexcom, Inc. Systems and methods for processing analyte sensor data
US7874917B2 (en) 2003-09-15 2011-01-25 Sony Computer Entertainment Inc. Methods and systems for enabling depth and direction detection when interfacing with a computer program
US8323106B2 (en) 2008-05-30 2012-12-04 Sony Computer Entertainment America Llc Determination of controller three-dimensional location using image analysis and ultrasonic communication
US8287373B2 (en) 2008-12-05 2012-10-16 Sony Computer Entertainment Inc. Control device for communicating visual information
US10279254B2 (en) 2005-10-26 2019-05-07 Sony Interactive Entertainment Inc. Controller having visually trackable object for interfacing with a gaming system
US9573056B2 (en) 2005-10-26 2017-02-21 Sony Interactive Entertainment Inc. Expandable control device via hardware attachment
US7285047B2 (en) * 2003-10-17 2007-10-23 Hewlett-Packard Development Company, L.P. Method and system for real-time rendering within a gaming environment
US8133115B2 (en) 2003-10-22 2012-03-13 Sony Computer Entertainment America Llc System and method for recording and displaying a graphical path in a video game
US9247900B2 (en) 2004-07-13 2016-02-02 Dexcom, Inc. Analyte sensor
WO2005051170A2 (en) * 2003-11-19 2005-06-09 Dexcom, Inc. Integrated receiver for continuous analyte sensor
US8364231B2 (en) 2006-10-04 2013-01-29 Dexcom, Inc. Analyte sensor
EP2239567B1 (en) * 2003-12-05 2015-09-02 DexCom, Inc. Calibration techniques for a continuous analyte sensor
US8287453B2 (en) 2003-12-05 2012-10-16 Dexcom, Inc. Analyte sensor
US11633133B2 (en) 2003-12-05 2023-04-25 Dexcom, Inc. Dual electrode system for a continuous analyte sensor
US8774886B2 (en) 2006-10-04 2014-07-08 Dexcom, Inc. Analyte sensor
US8423114B2 (en) 2006-10-04 2013-04-16 Dexcom, Inc. Dual electrode system for a continuous analyte sensor
US7663689B2 (en) 2004-01-16 2010-02-16 Sony Computer Entertainment Inc. Method and apparatus for optimizing capture device settings through depth information
CA2455359C (en) * 2004-01-16 2013-01-08 Geotango International Corp. System, computer program and method for 3d object measurement, modeling and mapping from single imagery
US8808228B2 (en) 2004-02-26 2014-08-19 Dexcom, Inc. Integrated medicament delivery device for use with continuous analyte sensor
US20050245799A1 (en) * 2004-05-03 2005-11-03 Dexcom, Inc. Implantable analyte sensor
US8277713B2 (en) * 2004-05-03 2012-10-02 Dexcom, Inc. Implantable analyte sensor
WO2005119356A2 (en) 2004-05-28 2005-12-15 Erik Jan Banning Interactive direct-pointing system and calibration method
US7857760B2 (en) 2004-07-13 2010-12-28 Dexcom, Inc. Analyte sensor
US7310544B2 (en) 2004-07-13 2007-12-18 Dexcom, Inc. Methods and systems for inserting a transcutaneous analyte sensor
US20060270922A1 (en) 2004-07-13 2006-11-30 Brauker James H Analyte sensor
US8547401B2 (en) 2004-08-19 2013-10-01 Sony Computer Entertainment Inc. Portable augmented reality device and method
US20060046851A1 (en) * 2004-08-24 2006-03-02 Hewlett-Packard Development Company, L.P. Remote gaming and projection
US20060071933A1 (en) 2004-10-06 2006-04-06 Sony Computer Entertainment Inc. Application binary interface for multi-pass shaders
US8842186B2 (en) 2004-10-25 2014-09-23 I-Interactive Llc Control system and method employing identification of a displayed image
US8760522B2 (en) 2005-10-21 2014-06-24 I-Interactive Llc Multi-directional remote control system and method
US8456534B2 (en) 2004-10-25 2013-06-04 I-Interactive Llc Multi-directional remote control system and method
US7796116B2 (en) 2005-01-12 2010-09-14 Thinkoptics, Inc. Electronic equipment for handheld vision based absolute pointing system
US20060262188A1 (en) * 2005-05-20 2006-11-23 Oded Elyada System and method for detecting changes in an environment
US7636126B2 (en) 2005-06-22 2009-12-22 Sony Computer Entertainment Inc. Delay matching in audio/video systems
US9285897B2 (en) * 2005-07-13 2016-03-15 Ultimate Pointer, L.L.C. Easily deployable interactive direct-pointing system and calibration method therefor
US7426029B2 (en) 2005-08-31 2008-09-16 Microsoft Corporation Color measurement using compact device
US7822270B2 (en) * 2005-08-31 2010-10-26 Microsoft Corporation Multimedia color management system
US7573620B2 (en) * 2005-09-01 2009-08-11 Microsoft Corporation Gamuts and gamut mapping
US8274714B2 (en) * 2005-11-30 2012-09-25 Microsoft Corporation Quantifiable color calibration
TWI286484B (en) * 2005-12-16 2007-09-11 Pixart Imaging Inc Device for tracking the motion of an object and object for reflecting infrared light
US9757061B2 (en) 2006-01-17 2017-09-12 Dexcom, Inc. Low oxygen in vivo analyte sensor
JP4530419B2 (en) 2006-03-09 2010-08-25 任天堂株式会社 Coordinate calculation apparatus and coordinate calculation program
US20070216711A1 (en) * 2006-03-14 2007-09-20 Microsoft Corporation Microsoft Patent Group Abstracting transform representations in a graphics API
EP2022039B1 (en) * 2006-05-04 2020-06-03 Sony Computer Entertainment America LLC Scheme for detecting and tracking user manipulation of a game controller body and for translating movements thereof into inputs and game commands
US7880746B2 (en) 2006-05-04 2011-02-01 Sony Computer Entertainment Inc. Bandwidth management through lighting control of a user environment via a display device
US7965859B2 (en) 2006-05-04 2011-06-21 Sony Computer Entertainment Inc. Lighting control of a user environment via a display device
US8601379B2 (en) * 2006-05-07 2013-12-03 Sony Computer Entertainment Inc. Methods for interactive communications with real time effects and avatar environment interaction
US7573489B2 (en) 2006-06-01 2009-08-11 Industrial Light & Magic Infilling for 2D to 3D image conversion
US7573475B2 (en) 2006-06-01 2009-08-11 Industrial Light & Magic 2D to 3D image conversion
US8013838B2 (en) 2006-06-30 2011-09-06 Microsoft Corporation Generating position information using a video camera
US8913003B2 (en) 2006-07-17 2014-12-16 Thinkoptics, Inc. Free-space multi-dimensional absolute pointer using a projection marker system
US7907117B2 (en) * 2006-08-08 2011-03-15 Microsoft Corporation Virtual controller for visual displays
US8013869B2 (en) * 2006-09-13 2011-09-06 Adobe Systems Incorporated Color selection interface
USRE48417E1 (en) 2006-09-28 2021-02-02 Sony Interactive Entertainment Inc. Object direction using video input combined with tilt angle information
US8310656B2 (en) 2006-09-28 2012-11-13 Sony Computer Entertainment America Llc Mapping movements of a hand-held controller to the two-dimensional image plane of a display screen
US8781151B2 (en) 2006-09-28 2014-07-15 Sony Computer Entertainment Inc. Object detection using video input combined with tilt angle information
US20080111789A1 (en) * 2006-11-09 2008-05-15 Intelligence Frontier Media Laboratory Ltd Control device with hybrid sensing system comprised of video-based pattern recognition and electronic signal transmission
JP2009134677A (en) * 2007-02-28 2009-06-18 Fuji Xerox Co Ltd Gesture interface system, wand for gesture input, application control method, camera calibration method, and control program
JP2008219788A (en) * 2007-03-07 2008-09-18 Toshiba Corp Stereoscopic image display device, and method and program therefor
US8154561B1 (en) 2007-03-22 2012-04-10 Adobe Systems Incorporated Dynamic display of a harmony rule list
US8094885B2 (en) * 2007-03-29 2012-01-10 Y.T. Ventures Ltd System and method for tracking an electronic device
US9176598B2 (en) 2007-05-08 2015-11-03 Thinkoptics, Inc. Free-space multi-dimensional absolute pointer with improved performance
US9070192B1 (en) * 2007-05-15 2015-06-30 Vision Interface Technologies, LLC Implementing rich color transition curve tracking for applications
US9536322B1 (en) 2007-05-15 2017-01-03 Peter Harmon Smith Implementation of multi-camera tracking applications using rich color transition curve target sequences
US20200037875A1 (en) 2007-05-18 2020-02-06 Dexcom, Inc. Analyte sensors having a signal-to-noise ratio substantially unaffected by non-constant noise
EP2152350A4 (en) 2007-06-08 2013-03-27 Dexcom Inc Integrated medicament delivery device for use with continuous analyte sensor
US8237656B2 (en) * 2007-07-06 2012-08-07 Microsoft Corporation Multi-axis motion-based remote control
GB2451461A (en) * 2007-07-28 2009-02-04 Naveen Chawla Camera based 3D user and wand tracking human-computer interaction system
US20090062002A1 (en) * 2007-08-30 2009-03-05 Bay Tek Games, Inc. Apparatus And Method of Detecting And Tracking Objects In Amusement Games
EP2227132B1 (en) 2007-10-09 2023-03-08 DexCom, Inc. Integrated insulin delivery system with continuous glucose sensor
US8417312B2 (en) 2007-10-25 2013-04-09 Dexcom, Inc. Systems and methods for processing sensor data
US8290559B2 (en) 2007-12-17 2012-10-16 Dexcom, Inc. Systems and methods for processing sensor data
US8542907B2 (en) 2007-12-17 2013-09-24 Sony Computer Entertainment America Llc Dynamic three-dimensional object mapping for user-defined control device
US8840470B2 (en) 2008-02-27 2014-09-23 Sony Computer Entertainment America Llc Methods for capturing depth data of a scene and applying computer actions
US8368753B2 (en) 2008-03-17 2013-02-05 Sony Computer Entertainment America Llc Controller with an integrated depth camera
CN102047201A (en) * 2008-05-26 2011-05-04 微软国际控股私有有限公司 Controlling virtual reality
US9352411B2 (en) 2008-05-28 2016-05-31 Illinois Tool Works Inc. Welding training system
EP2310798B1 (en) 2008-07-29 2016-10-26 Microsoft International Holdings B.V. Imaging system
WO2010014069A1 (en) * 2008-07-31 2010-02-04 Sti Medical Systems, Llc Single spot focus control
EP2347320B1 (en) * 2008-10-27 2021-03-31 Sony Interactive Entertainment Inc. Control device for communicating visual information
FR2939325B1 (en) 2008-12-04 2015-10-16 Parrot DRONES SYSTEM WITH RECONNAISSANCE BEACONS
US8961313B2 (en) 2009-05-29 2015-02-24 Sony Computer Entertainment America Llc Multi-positional three-dimensional controller
US8624962B2 (en) * 2009-02-02 2014-01-07 Ydreams—Informatica, S.A. Ydreams Systems and methods for simulating three-dimensional virtual interactions from two-dimensional camera images
US8527657B2 (en) 2009-03-20 2013-09-03 Sony Computer Entertainment America Llc Methods and systems for dynamically adjusting update rates in multi-player network gaming
WO2010108499A2 (en) * 2009-03-22 2010-09-30 Algreatly Cherif Atia 3d navigation method and system
US8342963B2 (en) 2009-04-10 2013-01-01 Sony Computer Entertainment America Inc. Methods and systems for enabling control of artificial intelligence game characters
US8142288B2 (en) 2009-05-08 2012-03-27 Sony Computer Entertainment America Llc Base station movement detection and compensation
US8393964B2 (en) 2009-05-08 2013-03-12 Sony Computer Entertainment America Llc Base station for position location
US20100295782A1 (en) 2009-05-21 2010-11-25 Yehuda Binder System and method for control based on face ore hand gesture detection
US8721444B2 (en) 2009-07-21 2014-05-13 Sony Corporation Game device for performing operation object control and non-operation object control
US8842096B2 (en) * 2010-01-08 2014-09-23 Crayola Llc Interactive projection system
WO2011085815A1 (en) * 2010-01-14 2011-07-21 Brainlab Ag Controlling a surgical navigation system
GB2478911B (en) 2010-03-22 2014-01-08 Timocco Ltd Object locating and tracking in video frames using smoothness check along specified line sections
KR101458939B1 (en) 2010-12-02 2014-11-07 엠파이어 테크놀로지 디벨롭먼트 엘엘씨 Augmented reality system
SG184582A1 (en) * 2011-03-07 2012-10-30 Creative Tech Ltd A method, system and electronic device for association based identification
WO2012142502A2 (en) 2011-04-15 2012-10-18 Dexcom Inc. Advanced analyte sensor calibration and error detection
US8845431B2 (en) 2011-05-31 2014-09-30 Microsoft Corporation Shape trace gesturing
US8657683B2 (en) 2011-05-31 2014-02-25 Microsoft Corporation Action selection gesturing
US8740702B2 (en) 2011-05-31 2014-06-03 Microsoft Corporation Action trigger gesturing
WO2012172548A1 (en) * 2011-06-14 2012-12-20 Youval Nehmadi Method for translating a movement and an orientation of a predefined object into a computer generated data
US9101994B2 (en) 2011-08-10 2015-08-11 Illinois Tool Works Inc. System and device for welding training
TWI478098B (en) * 2011-08-18 2015-03-21 Univ Nat Taiwan System and method of correcting a depth map for 3d image
KR101830966B1 (en) * 2011-09-21 2018-02-22 엘지전자 주식회사 Electronic device and contents generation method for electronic device
US8948498B1 (en) * 2012-10-17 2015-02-03 Google Inc. Systems and methods to transform a colored point cloud to a 3D textured mesh
US9583014B2 (en) 2012-11-09 2017-02-28 Illinois Tool Works Inc. System and device for welding training
US9317136B2 (en) * 2013-01-10 2016-04-19 UL See Inc. Image-based object tracking system and image-based object tracking method
US9583023B2 (en) 2013-03-15 2017-02-28 Illinois Tool Works Inc. Welding torch for a welding training system
US9666100B2 (en) 2013-03-15 2017-05-30 Illinois Tool Works Inc. Calibration devices for a welding training system
US9713852B2 (en) 2013-03-15 2017-07-25 Illinois Tool Works Inc. Welding training systems and devices
US9766709B2 (en) 2013-03-15 2017-09-19 Leap Motion, Inc. Dynamic user interactions for display control
US9728103B2 (en) 2013-03-15 2017-08-08 Illinois Tool Works Inc. Data storage and analysis for a welding training system
US9672757B2 (en) 2013-03-15 2017-06-06 Illinois Tool Works Inc. Multi-mode software and method for a welding training system
US9857876B2 (en) * 2013-07-22 2018-01-02 Leap Motion, Inc. Non-linear motion capture using Frenet-Serret frames
US10056010B2 (en) 2013-12-03 2018-08-21 Illinois Tool Works Inc. Systems and methods for a weld training system
US9454840B2 (en) * 2013-12-13 2016-09-27 Blake Caldwell System and method for interactive animations for enhanced and personalized video communications
US9757819B2 (en) 2014-01-07 2017-09-12 Illinois Tool Works Inc. Calibration tool and method for a welding system
US9589481B2 (en) 2014-01-07 2017-03-07 Illinois Tool Works Inc. Welding software for detection and control of devices and for analysis of data
US9724788B2 (en) 2014-01-07 2017-08-08 Illinois Tool Works Inc. Electrical assemblies for a welding system
US10105782B2 (en) 2014-01-07 2018-10-23 Illinois Tool Works Inc. Feedback from a welding torch of a welding system
US10170019B2 (en) 2014-01-07 2019-01-01 Illinois Tool Works Inc. Feedback from a welding torch of a welding system
US9751149B2 (en) 2014-01-07 2017-09-05 Illinois Tool Works Inc. Welding stand for a welding system
US9443311B2 (en) * 2014-06-12 2016-09-13 Topcon Positioning Systems, Inc. Method and system to identify a position of a measurement pole
US9937578B2 (en) * 2014-06-27 2018-04-10 Illinois Tool Works Inc. System and method for remote welding training
US9862049B2 (en) 2014-06-27 2018-01-09 Illinois Tool Works Inc. System and method of welding system operator identification
US10665128B2 (en) 2014-06-27 2020-05-26 Illinois Tool Works Inc. System and method of monitoring welding information
US10307853B2 (en) 2014-06-27 2019-06-04 Illinois Tool Works Inc. System and method for managing welding data
US9769494B2 (en) 2014-08-01 2017-09-19 Ati Technologies Ulc Adaptive search window positioning for video encoding
US9724787B2 (en) 2014-08-07 2017-08-08 Illinois Tool Works Inc. System and method of monitoring a welding environment
US11014183B2 (en) 2014-08-07 2021-05-25 Illinois Tool Works Inc. System and method of marking a welding workpiece
US9875665B2 (en) 2014-08-18 2018-01-23 Illinois Tool Works Inc. Weld training system and method
US10417934B2 (en) 2014-11-05 2019-09-17 Illinois Tool Works Inc. System and method of reviewing weld data
US10490098B2 (en) 2014-11-05 2019-11-26 Illinois Tool Works Inc. System and method of recording multi-run data
US10210773B2 (en) 2014-11-05 2019-02-19 Illinois Tool Works Inc. System and method for welding torch display
US10373304B2 (en) 2014-11-05 2019-08-06 Illinois Tool Works Inc. System and method of arranging welding device markers
US10204406B2 (en) 2014-11-05 2019-02-12 Illinois Tool Works Inc. System and method of controlling welding system camera exposure and marker illumination
US10402959B2 (en) 2014-11-05 2019-09-03 Illinois Tool Works Inc. System and method of active torch marker control
GB2533632B (en) * 2014-12-24 2018-01-03 Gen Electric Method and system for obtaining low dose tomosynthesis and material decomposition images
US10427239B2 (en) 2015-04-02 2019-10-01 Illinois Tool Works Inc. Systems and methods for tracking weld training arc parameters
US10438505B2 (en) 2015-08-12 2019-10-08 Illinois Tool Works Welding training system interface
US10373517B2 (en) 2015-08-12 2019-08-06 Illinois Tool Works Inc. Simulation stick welding electrode holder systems and methods
US10657839B2 (en) 2015-08-12 2020-05-19 Illinois Tool Works Inc. Stick welding electrode holders with real-time feedback features
US10593230B2 (en) 2015-08-12 2020-03-17 Illinois Tool Works Inc. Stick welding electrode holder systems and methods
CN106548127B (en) * 2015-09-18 2022-11-04 松下电器(美国)知识产权公司 Image recognition method
JP2017059207A (en) * 2015-09-18 2017-03-23 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカPanasonic Intellectual Property Corporation of America Image recognition method
US9898665B2 (en) 2015-10-29 2018-02-20 International Business Machines Corporation Computerized video file analysis tool and method
US10962363B2 (en) 2016-01-25 2021-03-30 Topcon Positioning Systems, Inc. Method and apparatus for single camera optical measurements
CA3077720A1 (en) 2017-10-24 2019-05-02 Dexcom, Inc. Pre-connected analyte sensors
US11331022B2 (en) 2017-10-24 2022-05-17 Dexcom, Inc. Pre-connected analyte sensors
CN107930125A (en) * 2017-11-27 2018-04-20 五华怪兽星球科技有限公司 A kind of interactive gaming system for having alarm function
GB2571953A (en) * 2018-03-13 2019-09-18 Massless Emea Ltd Single view tracking of cylindrical objects
CN111489711B (en) * 2019-01-28 2022-01-04 咸阳彩虹光电科技有限公司 Method and device for improving low color cast of visual angle by using algorithm and display panel
US20220120602A1 (en) * 2019-04-05 2022-04-21 Daedalus Technology Group, Inc. Calibration System and Method
US11288978B2 (en) 2019-07-22 2022-03-29 Illinois Tool Works Inc. Gas tungsten arc welding training systems
US11776423B2 (en) 2019-07-22 2023-10-03 Illinois Tool Works Inc. Connection boxes for gas tungsten arc welding training systems
GB2586059B (en) 2019-08-01 2023-06-07 Sony Interactive Entertainment Inc System and method for generating user inputs for a video game
CN111080719A (en) * 2019-12-26 2020-04-28 四川航天神坤科技有限公司 Video registration method and device
US11893808B2 (en) * 2020-11-30 2024-02-06 Mangolytics, Inc. Learning-based 3D property extraction
CN115129191B (en) * 2021-03-26 2023-08-15 北京新氧科技有限公司 Three-dimensional object pickup method, device, equipment and storage medium

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4991223A (en) * 1988-06-30 1991-02-05 American Innovision, Inc. Apparatus and method for recognizing image features using color elements
US5616078A (en) * 1993-12-28 1997-04-01 Konami Co., Ltd. Motion-controlled video entertainment system
US5836366A (en) * 1994-06-09 1998-11-17 Compagnie Generale Des Etablissements Michelin-Michelin & Cie Method of fitting an assembly formed of a tire and of a tread strip support
US5838366A (en) * 1995-11-03 1998-11-17 Snape; Tim Tracking apparatus for use in tracking an object
US5889505A (en) * 1996-04-04 1999-03-30 Yale University Vision-based six-degree-of-freedom computer input device
US6199093B1 (en) * 1995-07-21 2001-03-06 Nec Corporation Processor allocating method/apparatus in multiprocessor system, and medium for storing processor allocating program
US6323838B1 (en) * 1998-05-27 2001-11-27 Act Labs, Ltd. Photosensitive input peripheral device in a personal computer-based video gaming platform
US6720949B1 (en) * 1997-08-22 2004-04-13 Timothy R. Pryor Man machine interfaces and applications

Family Cites Families (103)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US561100A (en) * 1896-06-02 Andrew b
US3943277A (en) 1969-02-20 1976-03-09 The United States Of America As Represented By The Secretary Of The Navy Digital memory area correlation tracker
NL7509871A (en) * 1975-08-20 1977-02-22 Philips Nv COLOR TV CHROMA KEY SIGNAL GENERATOR.
US4133004A (en) 1977-11-02 1979-01-02 Hughes Aircraft Company Video correlation tracker
US4448200A (en) 1978-03-27 1984-05-15 University Of Southern California System and method for dynamic background subtraction
JPS5846783A (en) * 1981-09-12 1983-03-18 Sony Corp Chromakey device
JPS592040U (en) 1982-06-28 1984-01-07 日本電気ホームエレクトロニクス株式会社 joystick control device
FI68131C (en) * 1983-06-30 1985-07-10 Valtion Teknillinen REFERENCE FOR A WINDOW MACHINE WITH A GLASS LED WITH A LASER INDICATOR
IL69327A (en) 1983-07-26 1986-11-30 Elscint Ltd Automatic misregistration correction
US4675562A (en) 1983-08-01 1987-06-23 Fairchild Semiconductor Corporation Method and apparatus for dynamically controlling the timing of signals in automatic test systems
US4649504A (en) 1984-05-22 1987-03-10 Cae Electronics, Ltd. Optical position and orientation measurement techniques
US4672564A (en) * 1984-11-15 1987-06-09 Honeywell Inc. Method and apparatus for determining location and orientation of objects
JPS61131110A (en) 1984-11-30 1986-06-18 Fujitsu Ltd Input device
IL77610A (en) * 1986-01-15 1994-01-25 Technion Res & Dev Foundation Single camera three-dimensional head position sensing system
US4843568A (en) 1986-04-11 1989-06-27 Krueger Myron W Real time perception of and response to the actions of an unencumbered participant/user
JP2603947B2 (en) * 1986-09-26 1997-04-23 オリンパス光学工業株式会社 Apparatus for detecting corresponding areas between primary color images
US4864515A (en) 1987-03-30 1989-09-05 Honeywell Inc. Electronic sensing screen for measuring projectile parameters
US5162781A (en) * 1987-10-02 1992-11-10 Automated Decisions, Inc. Orientational mouse computer input system
US5363120A (en) 1987-10-14 1994-11-08 Wang Laboratories, Inc. Computer input device using orientation sensor
US4942538A (en) * 1988-01-05 1990-07-17 Spar Aerospace Limited Telerobotic tracker
US4933864A (en) 1988-10-04 1990-06-12 Transitions Research Corporation Mobile robot navigation employing ceiling light fixtures
US5045843B1 (en) * 1988-12-06 1996-07-16 Selectech Ltd Optical pointing device
US5034986A (en) 1989-03-01 1991-07-23 Siemens Aktiengesellschaft Method for detecting and tracking moving objects in a digital image sequence having a stationary background
WO1990016037A1 (en) 1989-06-20 1990-12-27 Fujitsu Limited Method for measuring position and posture of object
FR2652972B1 (en) * 1989-10-06 1996-11-29 Thomson Video Equip METHOD AND DEVICE FOR INTEGRATING SELF-ADAPTIVE COLOR VIDEO IMAGES.
GB9001468D0 (en) 1990-01-23 1990-03-21 Sarnoff David Res Center Computing multiple motions within an image region
US5668646A (en) * 1990-02-06 1997-09-16 Canon Kabushiki Kaisha Apparatus and method for decoding differently encoded multi-level and binary image data, the later corresponding to a color in the original image
ATE137377T1 (en) * 1990-02-06 1996-05-15 Canon Kk IMAGE PROCESSING DEVICE
US5128671A (en) * 1990-04-12 1992-07-07 Ltv Aerospace And Defense Company Control device having multiple degrees of freedom
US5208763A (en) 1990-09-14 1993-05-04 New York University Method and apparatus for determining position and orientation of mechanical objects
US5128794A (en) * 1990-12-31 1992-07-07 Honeywell Inc. Scanning laser helmet mounted sight
US5534917A (en) 1991-05-09 1996-07-09 Very Vivid, Inc. Video image based control system
US5548667A (en) 1991-05-24 1996-08-20 Sony Corporation Image processing system and method thereof in which three dimensional shape is reproduced from two dimensional image data
US5227985A (en) 1991-08-19 1993-07-13 University Of Maryland Computer vision system for position monitoring in three dimensions using non-coplanar light sources attached to a monitored object
US5212888A (en) 1991-09-16 1993-05-25 Calcomp Inc. Dual function sensor for a pen plotter
US5335557A (en) 1991-11-26 1994-08-09 Taizo Yasutake Touch sensitive input control device
US5631697A (en) * 1991-11-27 1997-05-20 Hitachi, Ltd. Video camera capable of automatic target tracking
US5230623A (en) 1991-12-10 1993-07-27 Radionics, Inc. Operating pointer with interactive computergraphics
US5680487A (en) 1991-12-23 1997-10-21 Texas Instruments Incorporated System and method for determining optical flow
US5577179A (en) * 1992-02-25 1996-11-19 Imageware Software, Inc. Image editing system
US5307137A (en) * 1992-03-16 1994-04-26 Mark F. Jones Terrain imaging apparatus and method
US5450504A (en) 1992-05-19 1995-09-12 Calia; James Method for finding a most likely matching of a target facial image in a data base of facial images
JP3391405B2 (en) 1992-05-29 2003-03-31 株式会社エフ・エフ・シー Object identification method in camera image
US5473736A (en) * 1992-06-08 1995-12-05 Chroma Graphics Method and apparatus for ordering and remapping colors in images of real two- and three-dimensional objects
JP3244798B2 (en) 1992-09-08 2002-01-07 株式会社東芝 Moving image processing device
US5982352A (en) 1992-09-18 1999-11-09 Pryor; Timothy R. Method for providing human input to a computer
US5469193A (en) 1992-10-05 1995-11-21 Prelude Technology Corp. Cordless pointing apparatus
JP3679426B2 (en) 1993-03-15 2005-08-03 マサチューセッツ・インスティチュート・オブ・テクノロジー A system that encodes image data into multiple layers, each representing a coherent region of motion, and motion parameters associated with the layers.
GB9308952D0 (en) * 1993-04-30 1993-06-16 Philips Electronics Uk Ltd Tracking objects in video sequences
US5297061A (en) * 1993-05-19 1994-03-22 University Of Maryland Three dimensional pointing device monitored by computer vision
EP0633546B1 (en) * 1993-07-02 2003-08-27 Siemens Corporate Research, Inc. Background recovery in monocular vision
US5598514A (en) * 1993-08-09 1997-01-28 C-Cube Microsystems Structure and method for a multistandard video encoder/decoder
JPH07160412A (en) 1993-12-10 1995-06-23 Nippon Telegr & Teleph Corp <Ntt> Pointed position detecting method
FR2714502A1 (en) 1993-12-29 1995-06-30 Philips Laboratoire Electroniq An image processing method and apparatus for constructing from a source image a target image with perspective change.
US5611000A (en) 1994-02-22 1997-03-11 Digital Equipment Corporation Spline-based image registration
US5649032A (en) 1994-11-14 1997-07-15 David Sarnoff Research Center, Inc. System for automatically aligning images to form a mosaic image
GB2295936B (en) 1994-12-05 1997-02-05 Microsoft Corp Progressive image transmission using discrete wavelet transforms
US5757360A (en) * 1995-05-03 1998-05-26 Mitsubishi Electric Information Technology Center America, Inc. Hand held computer control device
US5672820A (en) * 1995-05-16 1997-09-30 Boeing North American, Inc. Object location identification system for providing location data of an object being pointed at by a pointing device
US5913727A (en) 1995-06-02 1999-06-22 Ahdoot; Ned Interactive movement and contact simulation game
US5805745A (en) 1995-06-26 1998-09-08 Lucent Technologies Inc. Method for locating a subject's lips in a facial image
EP0852732A1 (en) 1995-09-21 1998-07-15 Omniplanar, Inc. Method and apparatus for determining position and orientation
US5970173A (en) 1995-10-05 1999-10-19 Microsoft Corporation Image compression and affine transformation for image motion compensation
US5818424A (en) * 1995-10-19 1998-10-06 International Business Machines Corporation Rod shaped device and data acquisition apparatus for determining the position and orientation of an object in space
US5825308A (en) 1996-11-26 1998-10-20 Immersion Human Interface Corporation Force feedback interface having isotonic and isometric functionality
FR2741769B1 (en) * 1995-11-23 1997-12-19 Thomson Broadcast Systems PROCESS FOR PROCESSING THE SIGNAL CONSTITUTED BY A SUBJECT EVOLVING BEFORE A COLORED BACKGROUND AND DEVICE IMPLEMENTING THIS METHOD
FR2741770B1 (en) * 1995-11-23 1998-01-02 Thomson Broadcast Systems METHOD FOR CALCULATING A CUTTING KEY OF A SUBJECT EVOLVING IN FRONT OF A COLORED BACKGROUND AND DEVICE IMPLEMENTING THIS METHOD
US5963209A (en) 1996-01-11 1999-10-05 Microsoft Corporation Encoding and progressive transmission of progressive meshes
US6049619A (en) 1996-02-12 2000-04-11 Sarnoff Corporation Method and apparatus for detecting moving objects in two- and three-dimensional scenes
US6009188A (en) 1996-02-16 1999-12-28 Microsoft Corporation Method and system for digital plenoptic imaging
US5764803A (en) * 1996-04-03 1998-06-09 Lucent Technologies Inc. Motion-adaptive modelling of scene content for very low bit rate model-assisted coding of video sequences
US5923318A (en) 1996-04-12 1999-07-13 Zhai; Shumin Finger manipulatable 6 degree-of-freedom input device
US5805170A (en) 1996-05-07 1998-09-08 Microsoft Corporation Systems and methods for wrapping a closed polygon around an object
US6356272B1 (en) * 1996-08-29 2002-03-12 Sanyo Electric Co., Ltd. Texture information giving method, object extracting method, three-dimensional model generating method and apparatus for the same
JPH1078768A (en) * 1996-09-04 1998-03-24 Alps Electric Co Ltd Gradation display control device
US6121953A (en) 1997-02-06 2000-09-19 Modern Cartoons, Ltd. Virtual reality system for sensing facial movements
WO1998037702A1 (en) * 1997-02-20 1998-08-27 Sony Corporation Video signal processing device and method, image synthesizing device, and editing device
US5864742A (en) * 1997-04-11 1999-01-26 Eastman Kodak Company Copy restrictive system using microdots to restrict copying of color-reversal documents
US5917937A (en) 1997-04-15 1999-06-29 Microsoft Corporation Method for performing stereo matching to recover depths, colors and opacities of surface elements
US6072504A (en) 1997-06-20 2000-06-06 Lucent Technologies Inc. Method and apparatus for tracking, storing, and synthesizing an animated version of object motion
US6049636A (en) 1997-06-27 2000-04-11 Microsoft Corporation Determining a rectangular box encompassing a digital picture within a digital image
US5990901A (en) 1997-06-27 1999-11-23 Microsoft Corporation Model based image editing and correction
US6009190A (en) 1997-08-01 1999-12-28 Microsoft Corporation Texture map construction method and apparatus for displaying panoramic image mosaics
US5986668A (en) 1997-08-01 1999-11-16 Microsoft Corporation Deghosting method and apparatus for construction of image mosaics
US5987164A (en) 1997-08-01 1999-11-16 Microsoft Corporation Block adjustment method and apparatus for construction of image mosaics
US6018349A (en) 1997-08-01 2000-01-25 Microsoft Corporation Patch-based alignment method and apparatus for construction of image mosaics
US6044181A (en) 1997-08-01 2000-03-28 Microsoft Corporation Focal length estimation method and apparatus for construction of panoramic mosaic images
AUPO894497A0 (en) * 1997-09-02 1997-09-25 Xenotech Research Pty Ltd Image processing method and apparatus
US6031934A (en) 1997-10-15 2000-02-29 Electric Planet, Inc. Computer vision system for subject characterization
US6072494A (en) 1997-10-15 2000-06-06 Electric Planet, Inc. Method and apparatus for real-time gesture recognition
US6101289A (en) 1997-10-15 2000-08-08 Electric Planet, Inc. Method and apparatus for unencumbered capture of an object
US5905894A (en) 1997-10-29 1999-05-18 Microsoft Corporation Meta-programming methods and apparatus
US6162123A (en) 1997-11-25 2000-12-19 Woolston; Thomas G. Interactive electronic sword game
US6172354B1 (en) 1998-01-28 2001-01-09 Microsoft Corporation Operator input device
US6512507B1 (en) * 1998-03-31 2003-01-28 Seiko Epson Corporation Pointing position detection device, presentation system, and method, and computer-readable medium
US6563499B1 (en) * 1998-07-20 2003-05-13 Geometrix, Inc. Method and apparatus for generating a 3D region from a surrounding imagery
JP3748172B2 (en) * 1998-12-09 2006-02-22 富士通株式会社 Image processing device
DE19917660A1 (en) * 1999-04-19 2000-11-02 Deutsch Zentr Luft & Raumfahrt Method and input device for controlling the position of an object to be graphically represented in a virtual reality
US6917692B1 (en) * 1999-05-25 2005-07-12 Thomson Licensing S.A. Kalman tracking of color objects
US6417836B1 (en) * 1999-08-02 2002-07-09 Lucent Technologies Inc. Computer input device having six degrees of freedom for controlling movement of a three-dimensional object
US6594388B1 (en) * 2000-05-25 2003-07-15 Eastman Kodak Company Color image reproduction of scenes with preferential color mapping and scene-dependent tone scaling
US6795068B1 (en) * 2000-07-21 2004-09-21 Sony Computer Entertainment Inc. Prop input device and method for mapping an object from a two-dimensional camera image to a three-dimensional space for controlling action in a game program
JP3581835B2 (en) * 2001-03-14 2004-10-27 株式会社イマジカ Color conversion method and apparatus in chroma key processing

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4991223A (en) * 1988-06-30 1991-02-05 American Innovision, Inc. Apparatus and method for recognizing image features using color elements
US5616078A (en) * 1993-12-28 1997-04-01 Konami Co., Ltd. Motion-controlled video entertainment system
US5836366A (en) * 1994-06-09 1998-11-17 Compagnie Generale Des Etablissements Michelin-Michelin & Cie Method of fitting an assembly formed of a tire and of a tread strip support
US6199093B1 (en) * 1995-07-21 2001-03-06 Nec Corporation Processor allocating method/apparatus in multiprocessor system, and medium for storing processor allocating program
US5838366A (en) * 1995-11-03 1998-11-17 Snape; Tim Tracking apparatus for use in tracking an object
US5889505A (en) * 1996-04-04 1999-03-30 Yale University Vision-based six-degree-of-freedom computer input device
US6720949B1 (en) * 1997-08-22 2004-04-13 Timothy R. Pryor Man machine interfaces and applications
US6323838B1 (en) * 1998-05-27 2001-11-27 Act Labs, Ltd. Photosensitive input peripheral device in a personal computer-based video gaming platform

Cited By (76)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10402982B2 (en) 2009-11-18 2019-09-03 Ai Cure Technologies Llc Verification of medication administration adherence
US11646115B2 (en) 2009-11-18 2023-05-09 Ai Cure Technologies Llc Method and apparatus for verification of medication administration adherence
US10380744B2 (en) 2009-11-18 2019-08-13 Ai Cure Technologies Llc Verification of medication administration adherence
US20110119073A1 (en) * 2009-11-18 2011-05-19 Al Cure Technologies LLC Method and Apparatus for Verification of Medication Administration Adherence
US11923083B2 (en) 2009-11-18 2024-03-05 Ai Cure Technologies Llc Method and apparatus for verification of medication administration adherence
US9256776B2 (en) 2009-11-18 2016-02-09 AI Cure Technologies, Inc. Method and apparatus for identification
US9652665B2 (en) 2009-11-18 2017-05-16 Aic Innovations Group, Inc. Identification and de-identification within a video sequence
US8781856B2 (en) 2009-11-18 2014-07-15 Ai Cure Technologies Llc Method and apparatus for verification of medication administration adherence
US10929983B2 (en) 2009-11-18 2021-02-23 Ai Cure Technologies Llc Method and apparatus for verification of medication administration adherence
US10297032B2 (en) 2009-11-18 2019-05-21 Ai Cure Technologies Llc Verification of medication administration adherence
US10388023B2 (en) 2009-11-18 2019-08-20 Ai Cure Technologies Llc Verification of medication administration adherence
US10297030B2 (en) 2009-11-18 2019-05-21 Ai Cure Technologies Llc Method and apparatus for verification of medication administration adherence
US10496796B2 (en) 2009-12-23 2019-12-03 Ai Cure Technologies Llc Monitoring medication adherence
US10296721B2 (en) 2009-12-23 2019-05-21 Ai Cure Technology LLC Verification of medication administration adherence
US20110153360A1 (en) * 2009-12-23 2011-06-23 Al Cure Technologies LLC Method and Apparatus for Verification of Clinical Trial Adherence
US10496795B2 (en) 2009-12-23 2019-12-03 Ai Cure Technologies Llc Monitoring medication adherence
US11222714B2 (en) 2009-12-23 2022-01-11 Ai Cure Technologies Llc Method and apparatus for verification of medication adherence
US9454645B2 (en) 2009-12-23 2016-09-27 Ai Cure Technologies Llc Apparatus and method for managing medication adherence
US10566085B2 (en) 2009-12-23 2020-02-18 Ai Cure Technologies Llc Method and apparatus for verification of medication adherence
US10303855B2 (en) 2009-12-23 2019-05-28 Ai Cure Technologies Llc Method and apparatus for verification of medication adherence
US8731961B2 (en) 2009-12-23 2014-05-20 Ai Cure Technologies Method and apparatus for verification of clinical trial adherence
US8666781B2 (en) 2009-12-23 2014-03-04 Ai Cure Technologies, LLC Method and apparatus for management of clinical trials
US10303856B2 (en) 2009-12-23 2019-05-28 Ai Cure Technologies Llc Verification of medication administration adherence
US20110153361A1 (en) * 2009-12-23 2011-06-23 Al Cure Technologies LLC Method and Apparatus for Management of Clinical Trials
US9183601B2 (en) 2010-03-22 2015-11-10 Ai Cure Technologies Llc Method and apparatus for collection of protocol adherence data
US10395009B2 (en) 2010-03-22 2019-08-27 Ai Cure Technologies Llc Apparatus and method for collection of protocol adherence data
US20110231202A1 (en) * 2010-03-22 2011-09-22 Ai Cure Technologies Llc Method and apparatus for collection of protocol adherence data
US11244283B2 (en) 2010-03-22 2022-02-08 Ai Cure Technologies Llc Apparatus and method for collection of protocol adherence data
US11094408B2 (en) 2010-05-06 2021-08-17 Aic Innovations Group, Inc. Apparatus and method for recognition of inhaler actuation
US10872695B2 (en) 2010-05-06 2020-12-22 Ai Cure Technologies Llc Apparatus and method for recognition of patient activities when obtaining protocol adherence data
US10116903B2 (en) 2010-05-06 2018-10-30 Aic Innovations Group, Inc. Apparatus and method for recognition of suspicious activities
US10646101B2 (en) 2010-05-06 2020-05-12 Aic Innovations Group, Inc. Apparatus and method for recognition of inhaler actuation
US10650697B2 (en) 2010-05-06 2020-05-12 Aic Innovations Group, Inc. Apparatus and method for recognition of patient activities
US9883786B2 (en) 2010-05-06 2018-02-06 Aic Innovations Group, Inc. Method and apparatus for recognition of inhaler actuation
US10262109B2 (en) 2010-05-06 2019-04-16 Ai Cure Technologies Llc Apparatus and method for recognition of patient activities when obtaining protocol adherence data
US11328818B2 (en) 2010-05-06 2022-05-10 Ai Cure Technologies Llc Apparatus and method for recognition of patient activities when obtaining protocol adherence data
US9293060B2 (en) 2010-05-06 2016-03-22 Ai Cure Technologies Llc Apparatus and method for recognition of patient activities when obtaining protocol adherence data
US11682488B2 (en) 2010-05-06 2023-06-20 Ai Cure Technologies Llc Apparatus and method for recognition of patient activities when obtaining protocol adherence data
US11862033B2 (en) 2010-05-06 2024-01-02 Aic Innovations Group, Inc. Apparatus and method for recognition of patient activities
US9875666B2 (en) 2010-05-06 2018-01-23 Aic Innovations Group, Inc. Apparatus and method for recognition of patient activities
US10786736B2 (en) 2010-05-11 2020-09-29 Sony Interactive Entertainment LLC Placement of user information in a game space
US11478706B2 (en) 2010-05-11 2022-10-25 Sony Interactive Entertainment LLC Placement of user information in a game space
US10762172B2 (en) 2010-10-05 2020-09-01 Ai Cure Technologies Llc Apparatus and method for object confirmation and tracking
US10506971B2 (en) 2010-10-06 2019-12-17 Ai Cure Technologies Llc Apparatus and method for monitoring medication adherence
US10149648B2 (en) 2010-10-06 2018-12-11 Ai Cure Technologies Llc Method and apparatus for monitoring medication adherence
US9844337B2 (en) 2010-10-06 2017-12-19 Ai Cure Technologies Llc Method and apparatus for monitoring medication adherence
US8605165B2 (en) 2010-10-06 2013-12-10 Ai Cure Technologies Llc Apparatus and method for assisting monitoring of medication adherence
US9486720B2 (en) 2010-10-06 2016-11-08 Ai Cure Technologies Llc Method and apparatus for monitoring medication adherence
US9116553B2 (en) 2011-02-28 2015-08-25 AI Cure Technologies, Inc. Method and apparatus for confirmation of object positioning
US9892316B2 (en) 2011-02-28 2018-02-13 Aic Innovations Group, Inc. Method and apparatus for pattern tracking
US10511778B2 (en) 2011-02-28 2019-12-17 Aic Innovations Group, Inc. Method and apparatus for push interaction
US9665767B2 (en) 2011-02-28 2017-05-30 Aic Innovations Group, Inc. Method and apparatus for pattern tracking
US9538147B2 (en) 2011-02-28 2017-01-03 Aic Innovations Group, Inc. Method and system for determining proper positioning of an object
US10257423B2 (en) 2011-02-28 2019-04-09 Aic Innovations Group, Inc. Method and system for determining proper positioning of an object
US8884949B1 (en) 2011-06-06 2014-11-11 Thibault Lambert Method and system for real time rendering of objects from a low resolution depth camera
US9342817B2 (en) 2011-07-07 2016-05-17 Sony Interactive Entertainment LLC Auto-creating groups for sharing photos
US10558845B2 (en) 2011-08-21 2020-02-11 Aic Innovations Group, Inc. Apparatus and method for determination of medication location
US11314964B2 (en) 2011-08-21 2022-04-26 Aic Innovations Group, Inc. Apparatus and method for determination of medication location
US10565431B2 (en) 2012-01-04 2020-02-18 Aic Innovations Group, Inc. Method and apparatus for identification
US10133914B2 (en) 2012-01-04 2018-11-20 Aic Innovations Group, Inc. Identification and de-identification within a video sequence
US11004554B2 (en) 2012-01-04 2021-05-11 Aic Innovations Group, Inc. Method and apparatus for identification
US9754357B2 (en) 2012-03-23 2017-09-05 Panasonic Intellectual Property Corporation Of America Image processing device, stereoscoopic device, integrated circuit, and program for determining depth of object in real space generating histogram from image obtained by filming real space and performing smoothing of histogram
US9399111B1 (en) 2013-03-15 2016-07-26 Aic Innovations Group, Inc. Method and apparatus for emotional behavior therapy
US11200965B2 (en) 2013-04-12 2021-12-14 Aic Innovations Group, Inc. Apparatus and method for recognition of medication administration indicator
US9317916B1 (en) 2013-04-12 2016-04-19 Aic Innovations Group, Inc. Apparatus and method for recognition of medication administration indicator
US10460438B1 (en) 2013-04-12 2019-10-29 Aic Innovations Group, Inc. Apparatus and method for recognition of medication administration indicator
US9436851B1 (en) 2013-05-07 2016-09-06 Aic Innovations Group, Inc. Geometric encrypted coded image
US10373016B2 (en) 2013-10-02 2019-08-06 Aic Innovations Group, Inc. Method and apparatus for medication identification
US9824297B1 (en) 2013-10-02 2017-11-21 Aic Innovations Group, Inc. Method and apparatus for medication identification
US10916339B2 (en) 2014-06-11 2021-02-09 Aic Innovations Group, Inc. Medication adherence monitoring system and method
US9679113B2 (en) 2014-06-11 2017-06-13 Aic Innovations Group, Inc. Medication adherence monitoring system and method
US11417422B2 (en) 2014-06-11 2022-08-16 Aic Innovations Group, Inc. Medication adherence monitoring system and method
US9977870B2 (en) 2014-06-11 2018-05-22 Aic Innovations Group, Inc. Medication adherence monitoring system and method
US10475533B2 (en) 2014-06-11 2019-11-12 Aic Innovations Group, Inc. Medication adherence monitoring system and method
US10949983B2 (en) 2015-12-18 2021-03-16 Ricoh Company, Ltd. Image processing apparatus, image processing system, image processing method, and computer-readable recording medium
US11170484B2 (en) 2017-09-19 2021-11-09 Aic Innovations Group, Inc. Recognition of suspicious activities in medication administration

Also Published As

Publication number Publication date
JP3901960B2 (en) 2007-04-04
US6795068B1 (en) 2004-09-21
EP1176559A2 (en) 2002-01-30
JP2002140705A (en) 2002-05-17
US7113193B2 (en) 2006-09-26
JP4381371B2 (en) 2009-12-09
JP2006178948A (en) 2006-07-06
US20050024379A1 (en) 2005-02-03
EP1176559A3 (en) 2007-11-14
US20050026689A1 (en) 2005-02-03

Similar Documents

Publication Publication Date Title
US7113193B2 (en) Method for color transition detection
US7928976B2 (en) Telestrator system
US6864886B1 (en) Enhancing video using a virtual surface
EP1552375B1 (en) Man-machine interface using a deformable device
US6529626B1 (en) 3D model conversion apparatus and method
US7646372B2 (en) Methods and systems for enabling direction detection when interfacing with a computer program
US7874917B2 (en) Methods and systems for enabling depth and direction detection when interfacing with a computer program
US6894686B2 (en) System and method for automatically editing captured images for inclusion into 3D video game play
US6999069B1 (en) Method and apparatus for synthesizing images
US6937255B2 (en) Imaging apparatus and method of the same
EP1033682B1 (en) Image processing apparatus and image processing method
US6914599B1 (en) Image processing apparatus
US5912670A (en) Method and apparatus for overlaying a bit map image on an environment map
JP2002233665A (en) Game system, game method, and variable recording medium
EP1271413A2 (en) Image processing method
JP2022029730A (en) Three-dimensional (3d) model generation apparatus, virtual viewpoint video generation apparatus, method, and program

Legal Events

Date Code Title Description
AS Assignment

Owner name: SONY COMPUTER ENTERTAINMENT AMERICA INC., CALIFORN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MARKS, RICHARD L.;REEL/FRAME:017959/0036

Effective date: 20021018

Owner name: SONY COMPUTER ENTERTAINMENT INC., JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SONY COMPUTER ENTERTAINMENT AMERICA INC.;REEL/FRAME:017984/0658

Effective date: 20030324

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION

AS Assignment

Owner name: SONY INTERACTIVE ENTERTAINMENT INC., JAPAN

Free format text: CHANGE OF NAME;ASSIGNOR:SONY COMPUTER ENTERTAINMENT INC.;REEL/FRAME:039239/0343

Effective date: 20160401