US20090285452A1 - Unified system and method for animal behavior characterization in home cages using video analysis - Google Patents
Unified system and method for animal behavior characterization in home cages using video analysis Download PDFInfo
- Publication number
- US20090285452A1 US20090285452A1 US12/354,727 US35472709A US2009285452A1 US 20090285452 A1 US20090285452 A1 US 20090285452A1 US 35472709 A US35472709 A US 35472709A US 2009285452 A1 US2009285452 A1 US 2009285452A1
- Authority
- US
- United States
- Prior art keywords
- animal
- posture
- behavior
- video
- images
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- A—HUMAN NECESSITIES
- A01—AGRICULTURE; FORESTRY; ANIMAL HUSBANDRY; HUNTING; TRAPPING; FISHING
- A01K—ANIMAL HUSBANDRY; CARE OF BIRDS, FISHES, INSECTS; FISHING; REARING OR BREEDING ANIMALS, NOT OTHERWISE PROVIDED FOR; NEW BREEDS OF ANIMALS
- A01K1/00—Housing animals; Equipment therefor
- A01K1/02—Pigsties; Dog-kennels; Rabbit-hutches or the like
- A01K1/03—Housing for domestic or laboratory animals
- A01K1/031—Cages for laboratory animals; Cages for measuring metabolism of animals
-
- A—HUMAN NECESSITIES
- A01—AGRICULTURE; FORESTRY; ANIMAL HUSBANDRY; HUNTING; TRAPPING; FISHING
- A01K—ANIMAL HUSBANDRY; CARE OF BIRDS, FISHES, INSECTS; FISHING; REARING OR BREEDING ANIMALS, NOT OTHERWISE PROVIDED FOR; NEW BREEDS OF ANIMALS
- A01K29/00—Other apparatus for animal husbandry
- A01K29/005—Monitoring or measuring activity, e.g. detecting heat or mating
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
- A61B5/1113—Local tracking of patients, e.g. in a hospital or private home
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
- A61B5/1116—Determining posture transitions
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
- A61B5/1118—Determining activity level
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
- A61B5/1126—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb using a particular sensing technique
- A61B5/1128—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb using a particular sensing technique using image analysis
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/40—Detecting, measuring or recording for evaluating the nervous system
- A61B5/4076—Diagnosing or monitoring particular conditions of the nervous system
- A61B5/4094—Diagnosing or monitoring seizure diseases, e.g. epilepsy
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/103—Static body considered as a whole, e.g. static pedestrian or occupant recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/23—Recognition of whole body movements, e.g. for sport training
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
- G16H40/60—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
- G16H40/67—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices for remote operation
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B2503/00—Evaluating a particular growth phase or type of persons or animals
- A61B2503/40—Animals
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B2503/00—Evaluating a particular growth phase or type of persons or animals
- A61B2503/42—Evaluating a particular growth phase or type of persons or animals for laboratory research
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
- A61B5/7267—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems involving training the classification device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
Definitions
- mice or rats Animals, for example mice or rats, are used extensively as human models in the research of drag development; genetic functions; toxicology research; understanding and treatment of diseases; and other research applications.
- their extensive genetic and neuroanatomical homologies give rise to a wide variety of behavioral processes that are widely conserved between species. Exploration of these shared brain functions will shed light on fundamental elements of human behavioral regulation. Therefore, many behavioral test experiments have been designed on animals like mice and rats to explore their behaviors. These experiments include, but not limited to, home cage behaviors, open field locomotion experiments, object recognition experiments, a variety of maze experiments, water maze experiments, and freezing experiments for conditioned fear.
- Animal's home cage activity patterns are important examination item on the general health list of animals, such as mice and rats. It provides many important indications of whether the animal's health status is normal or abnormal. Home cage behaviors are best observed by videotaping several 24-hour periods in the animal housing facility, and subsequent scoring of the videotape by two independent observers. However, this observation has rarely been done until our inventions came into play, due to the instability in long term human observation, the time consumed, and the huge costs associated with the observation.
- the invention relates generally to behavior analysis of animal objects. More particularly, one aspect of the invention is directed to monitoring and characterization of behaviors under specific behavioral paradigm experiments, including home cage behavior paradigms, locomotion or open field paradigm experiment, object recognition paradigm experiments, variety of maze paradigm experiments, water maze paradigm experiments, freezing paradigm experiments for conditioned fear, for an animal, for example, a mouse or a rat, using video analysis from a top view image or side view image, or the integration of both views.
- specific behavioral paradigm experiments including home cage behavior paradigms, locomotion or open field paradigm experiment, object recognition paradigm experiments, variety of maze paradigm experiments, water maze paradigm experiments, freezing paradigm experiments for conditioned fear, for an animal, for example, a mouse or a rat, using video analysis from a top view image or side view image, or the integration of both views.
- a revolutionary approach is invented to automatically measure animal's home cage activity patterns.
- This approach consists of defining a unique set of animal's, such as mice or rats, behavior category. This category includes behaviors like rearing, walking, grooming, eating, drinking, jumping, hanging, etc.
- Computer systems are designed and implemented that can produce digital video files of animal's behaviors in a home cage in real time or off-line mode.
- Software algorithms are developed to automatically understand and analyze the animal's behaviors in those video files. This analysis is based on the premise that the entire animal body, body parts, related color information, and their dynamic motion are taken advantage of in order to provide the measurement of complex behaviors and novel parameters.
- the present invention is directed to systems and methods for finding patterns of behaviors and/or activities of an animal using video.
- the invention includes a system with a video camera connected to a computer in which the computer is configured to automatically provide animal identification, animal motion tracking (for moving animal), animal shape, animal body parts, and posture classification, and behavior identification.
- the present invention is capable of automatically monitoring a video image to identify, track and classify the actions of various animals and their movements.
- the video image may be provided in real time from a camera and/or from a storage location.
- the invention is particularly useful for monitoring and classifying mice or rats behavior for testing drugs and genetic mutations, but may be used in a number of surveillance or other applications.
- the invention includes a system in which an analog/digital video camera and a video record/playback device (e.g., VCR) are coupled to a video digitization/compression unit.
- the video camera may provide a video image containing an animal to be identified.
- the video digitization/compression unit is coupled to a computer that is configured to automatically monitor the video image to identify, track and classify the actions of the animal and its movements over time within a sequence of video session image frames.
- the digitization/compression unit may convert analog video and audio into, for example, MPEG or other formats.
- the computer may be, for example, a personal computer, using either a Windows platform or a Unix platform, or a Macintosh computer and compatible platform.
- the computer is loaded and configured with custom software programs (or equipped with firmware) using, for example, MATLAB or C/C++ programming language, so as to analyze the digitized video for animal identification and segmentation, tracking, and/or behavior/activity characterization.
- This software may be stored in, for example, a program memory, which may include ROM, RAM, CD ROM and/or a hard drive, etc.
- the software (or firmware) includes a unique background subtraction method which is more simple, efficient, and accurate than those previously known.
- the system receives incoming video images from either the video camera in real time or pre-recorded from the video record/playback unit. If the video is in analog format, then the information is converted from analog to digital format and may be compressed by the video digitization/compression unit.
- the digital video images are then provided to the computer where various processes are undertaken to identify and segment a predetermined animal from the image.
- the animal is a mouse or rat in motion with some movement from frame to frame in the video, and is in the foreground of the video images.
- the digital images may be processed to identify and segregate a desired (predetermined) animal from the various frames of incoming video. This process may be achieved using, for example, background subtraction, mixture modeling, robust estimation, and/or other processes.
- the shape and location of the desired animal is then tracked from one frame or scene to another frame or scene of video images.
- the body parts of the animal such as head, mouth, tail, ear, abdomen, lower back, upper back, forelimbs, and hind limbs, are identified by novel approaches through body contour segmentation, contour segment classification, and relaxation labeling.
- the changes in the shapes, locations, body parts, and/or postures of the animal of interest may be identified, their features extracted, and classified into meaningful categories, for example, vertical positioned side view, horizontal positioned side view, vertical positioned front view, horizontal positioned front view, moving left to right, etc.
- the shape, location, body parts, and posture categories may be used to characterize the animal's activity into one of a number of pre-defined behaviors.
- some pre-defined normal behaviors may include sleeping, eating, drinking, walking, running, etc.
- pre-defined abnormal behavior may include spinning vertical, jumping in the same spot, etc.
- the pre-defined behaviors may be stored in a database in the data memory.
- the behavior may be characterized using, for example, approaches such as rule-based label analysis, token parsing procedure, and/or Hidden Markov Modeling (HMM).
- HMM Hidden Markov Modeling
- the system may be constructed to characterize the object behavior as new behavior and particular temporal rhythm.
- Each cage contains at least one animal or multiple animals.
- the multiple cameras may be taking video from different points of views such as one taking video images from the side of the cage, or one taking video images from the top of the cage.
- high throughput screening is achieved.
- video images taken from different points of views for example, one from the top view and another from the side view, are combined to identify animal's behaviors, integrated analysis is achieved.
- the system operates as follows. As a preliminary matter, normal postures and behaviors of the animals are defined and may be entered into a Normal Paradigm Parameters, Postures and Behaviors database. In analyzing, in a first instant, incoming video images are received. The system determines if the video images are in analog or digital format and input into a computer. If the video images are in analog format they are digitized and may be compressed, using, for example, an MPEG digitizer/compression unit. Otherwise, the digital video image may be input directly to the computer. Next, a background may be generated or updated from the digital video images and foreground objects detected. Next, the foreground animal features are extracted.
- body parts such as head, tail, ear, mouth, forelimbs, hind limbs, abdomen, and upper and lower back, are identified.
- Two different methods are pursuing from this point, depending on different behavior paradigms.
- the foreground animal shape is classified into various categories, for example, standing, sitting, etc.
- the foreground animal posture is compared to the various predefined postures stored in the database, and then identified as a particular posture or a new (unidentified) posture.
- various groups of postures and body parts are concatenated into a series to make up a foreground animal behavior compared against the sequence of postures, stored in for example a database in memory, that make up known normal or abnormal behaviors of the animal.
- the abnormal behaviors are then identified in terms of known abnormal behavior, new behavior and/or daily rhythm.
- behavioral processes and events are detected, and behavior parameters are calculated.
- behaviors 5 parameters give indications to animal health information related to learning and memory capability, anxiety, and relations to certain diseases.
- animal detection is performed through a unique method of background subtraction.
- the incoming digital video signal is split into individual images (frames) in real-time.
- the system determines if the background image derived from prior incoming video needs to be updated due to changes in the background image or a background image needs to be developed because there was no background image was previously developed. If the background image needs to be generated, then a number of frames of video image, for example 20, will be grouped into a sample of images. Then, the system creates a standard deviation map of the sample of images.
- the process removes a bounding box area in each frame or image where the variation within the group of images is above a predetermined threshold (i.e., where the object of interest or moving objects are located).
- the various images within the sample less the bounding box area are averaged.
- Final background is obtained by averaging 5-10 samples. This completes the background generation process.
- the background image does not remain constant for a great length of time due to various reasons.
- the background needs to be dynamically recalculated periodically as above or it can be recalculated by keeping track of the difference image and note any sudden changes.
- the newly dynamically generated background image is next subtracted from the current video image(s) to obtain foreground areas that may include the object of interest.
- regions of interest are obtained by identifying areas where the intensity difference generated from the subtraction is greater than a predetermined threshold, which constitute potential foreground object(s) being sought. Classification of these foreground regions of interest will be performed using the sizes of the ROIs, distances among these ROIs, threshold of intensity, and connectedness, to thereby identify the foreground objects.
- the foreground object identification/detection process may be refined by adaptively learning histograms of foreground ROIs and using edge detection to more accurately identify the desired object(s).
- the information identifying the desired foreground object is output. The process may then continue with the tracking and/or behavior characterization step(s).
- the systematically developed definition of mouse behavior that is detectable by the automated analysis according to the present invention makes precise and quantitative analysis of the entire mouse behavior repertoire possible for the first time.
- the various computer algorithms included in the invention for automating behavior analysis based on the behavior definitions ensure accurate and efficient identification of mouse behaviors.
- the digital video analysis techniques of the present invention improves analysis of behavior by leading to: (1) decreased variance due to non-disturbed observation of the animal; (2) increased experiment sensitivity due to the greater number of behaviors sampled over a much longer time span than ever before possible; and (3) the potential to be applied to all common normative behavior patterns, capability to assess subtle behavioral states, and detection of changes of behavior patterns in addition to individual behaviors.
- the invention may identify some abnormal behavior by using video image information (for example, stored in memory) of known abnormal animals to build a video profile for that behavior. For example, video image of vertical spinning while hanging from the cage top was stored to memory and used to automatically identify such activity in mice. Further, abnormalities may also result from an increase in any particular type of normal behavior. Detection of such new abnormal behaviors may be achieved by the present invention detecting, for example, segments of behavior that do not fit the standard profile.
- the standard profile may be developed for a particular strain of mouse whereas detection of abnormal amounts of a normal behavior can be detected by comparison to the statistical properties of the standard profile.
- the automated analysis of the present invention may be used to build profiles of the behaviors, their amount, duration, and daily cycle for each animal, for example each commonly used strain of mice.
- a plurality of such profiles may be stored in, for example, a database in a data memory of the computer.
- One or more of 5 these profiles may then be compared to a mouse in question and difference from the profile expressed quantitatively.
- the techniques developed with the present invention for automation of the categorization and quantification of all home-cage mouse behaviors throughout the daily cycle is a powerful tool for detecting phenotypic effects of gene manipulations in mice. As previously discussed, this technology is extendable to other behavior studies of animals and humans, as well as surveillance purposes. As will be described in detail below, the present invention provides automated systems and methods for automated accurate identification, tracking and behavior categorization of an object whose image is captured with video.
- Other variations of the present invention is directed particularly to automatically determining the behavioral characteristics of an animal in various behavioral experiment apparatus such as water maze, Y-maze, T-maze, zero maze, elevated plus maze, locomotion open field, field for object recognition study, and cued or conditioned fear.
- animal's body contour, center of mass, body parts including head, tail, forelimbs, hind limbs and etc. are accurately identified using the embodiments above. This allows excellent understanding of animal's behaviors within these specific experiment apparatus and procedures. Many novel and important parameters, which were beyond reach previously, are now successfully analyzed.
- These parameters include, but not limited to, traces of path of animal's center of mass, instant and average speed, instant and average of body turning angles, distance traveled, turning ratio, proximity score, heading error, stretch-and-attend, head-dipping, stay-across-arms, supported-rearing, sniffing (exploring) at particular objects, latency time to get to the goal (platform), time spent in specific arm/arena or specific zones within arm/arena, number of time entering and exiting arm/arena or specific zones within arm/arena, and etc.
- These parameters provide good indications for gene targeting, drug screening, toxicology research, learning and memory process study, anxiety study, understanding and treatment of diseases such as Parkinson's Diseases, Alzheimer Disease, ALS, and etc.
- FIG. 1 is a block diagram of one exemplary system configurable to find the position, shape, and behavioral characteristics of an object using automated video analysis, according to one embodiment of the present invention
- FIG. 2 is a block diagram of various functional portions of a computer system, such as the computer system shown in FIG. 1 , when configured to find the position, shape, and behavioral characteristics of an object using automated video analysis, according to one embodiment of the present invention
- FIG. 3 is a flow chart of a method of automatic video analysis for object identification and characterization, according to one embodiment of the present invention
- FIG. 4 is a flow chart of a method of automatic video analysis for object identification and characterization, according to another embodiment of the present invention.
- FIG. 5 is a flow chart of a method of automatic video analysis for object detection and identification, according to one variation of the present invention.
- FIG. 6 illustrates a sample video image frame with a mouse in a rearing up posture as determined using one variation of the present invention to monitor and characterize mouse behavior
- FIG. 7A is an image showing a mouse in its home cage
- FIG. 7B is a difference image between foreground and background for the image shown in FIG. 7A , according to one variation of the present invention as applied for monitoring and characterizing mouse behavior;
- FIG. 7C is the image shown in FIG. 7A after completing a threshold process for identifying the foreground image of the mouse which is shown as correctly identified, according to one variation of the present invention as applied for monitoring and characterizing mouse behavior;
- FIG. 7D is a computer generated image showing the outline of the foreground mouse shown in FIG. 7A after edge segmentation to demonstrate a contour-based approach to object location and outline identification, according to one variation of the present invention as applied for monitoring and characterizing mouse behavior;
- FIG. 8 is a chart illustrating one example of various mouse state transitions used in characterizing mouse behavior including: Horizontal Side View Posture (HS); Cuddled Up Posture (CU); Partially Reared Posture (PR); Rear Up Posture (RU); and Horizontal Front/Back View Posture (FB), along with an indication of duration of these states based on a sample, according to one variation of the present invention as applied for monitoring and characterizing mouse behavior;
- HS Horizontal Side View Posture
- CU Cuddled Up Posture
- PR Partially Reared Posture
- RU Rear Up Posture
- FB Horizontal Front/Back View Posture
- FIG. 9 shows the contour segmentation approach where the contour outline of the animal is split in smaller segments and each segment is classified as a body part
- FIG. 10 shows another embodiment in night light conditions, in which night conditions are simulated using dim red light.
- FIG. 11 shows another embodiment of the invention, a high-throughput system in which multiple cages can be analyzed at the same time.
- video analysis systems may now be configured to provide robust surveillance systems that can provide automated analysis and identification of various objects and characterization of their behavior.
- the present invention provides such systems and related methods.
- the present invention can automatically find the patterns of behaviors and/or activities of a predetermined object being monitored using video.
- the invention includes a system with a video camera connected to a computer in which the computer is configured to automatically provide object identification, object motion tracking (for moving objects), object shape and posture classification, and behavior identification.
- the system includes various video analysis algorithms.
- the computer processes analyze digitized video with the various algorithms so as to automatically monitor a video image to identify, track and classify the actions of one or more predetermined objects and its movements captured by the video image as it occurs from one video frame or scene to another.
- the system may characterize behavior by accessing a database of object information of known behavior of the predetermined object.
- the image to be analyzed may be provided in real time from one or more camera and/or from storage.
- the invention is configured to enable monitoring and classifying of animal behavior that result from testing drugs and genetic mutations on animals.
- the system may be similarly configured for use in any of a number of surveillance or other applications.
- the invention can be applied to various situations in which tracking moving objects is needed.
- One such situation is security surveillance in public areas like airports, military bases, or home security systems.
- the system may be useful in automatically identifying and notifying proper law enforcement officials if a crime is being committed and/or a particular behavior being monitored is identified.
- the system may be useful for monitoring of parking security or moving traffic at intersections so as to automatically identify and track vehicle activity.
- the system may be configured to automatically determine if a vehicle is speeding or has performed some other traffic violation.
- the system may be configured to automatically identify and characterize human behavior involving guns or human activity related to robberies or thefts.
- the invention may be capable of identifying and understanding subtle behaviors involving portions of body such as forelimb and can be applied to identify and understand human gesture recognition. This could help deaf individuals communicate.
- the invention may also be the basis for computer understanding of human gesture to enhance the present human-computer interface experience, where gestures will be used to interface with computers. The economic potential of applications in computer-human interface applications and in surveillance and monitoring applications is enormous.
- the invention includes a system in which an analog video camera 105 and a video storage/retrieval unit 110 may be coupled to each other and to a video digitization/compression unit 115 .
- the video camera 105 may provide a real time video image containing an object to be identified.
- the video storage/retrieval unit 110 may be, for example, a VCR, DVD, CD or hard disk unit.
- the video digitization/compression unit 115 is coupled to a computer 150 that is configured to automatically monitor a video image to identify, track and classify the actions (or state) of the object and its movements (or stillness) over time within a sequence of images.
- the digitization/compression unit 115 may convert analog video and audio into, for example, MPEG format, Real Player format, etc.
- the computer may be, for example, a personal computer, using either a Windows platform or a Unix platform, or a Macintosh computer and compatible platform.
- the computer may include a number of components such as (1) a data memory 151 , for example, a hard drive or other type of volatile or nonvolatile memory; (2) a program memory 152 , for example, RAM, ROM, EEPROM, etc.
- the computer may also include a video processor such as an MPEG encoder/decoder.
- a video processor such as an MPEG encoder/decoder.
- the computer 150 has been shown in FIG. 1 to include two memories (data memory 151 and program memory 152 ) and two processors (processor 153 and math co-processor 154 ), in one variation the computer may include only a single processor and single memory device or more then two processors and more than two memory devices. Further, the computer 150 may be equipped with user interface components such as a keyboard 155 , electronic mouse 156 , and display unit 157 .
- the system may be simplified by using all digital components such as a digital video camera and a digital video storage/retrieval unit 110 , which may be one integral unit.
- the video digitization/compression unit 115 may not be needed.
- the computer is loaded and configured with custom software program(s) (or equipped with firmware) using, for example, MATLAB or C/C++ programming language, so as to analyze the digitized video for object identification and segmentation, tracking, and/or behavior/activity characterization.
- This software may be stored in, for example, a program memory 152 or data memory that may include ROM, RAM, CD ROM and/or a hard drive, etc.
- the software includes a unique background subtraction method which is more simple, efficient, and accurate than those previously known which will be discussed in detail below.
- the algorithms may be implemented in software and may be understood as unique functional modules as shown in FIG. 2 and now described.
- the system is preloaded with standard object information before analyzing an incoming video including a predetermined object, for example, a mouse.
- a stream of digital video including a known object with known characteristics may be fed into the system to a standard object classifier module 220 .
- a user may then view the standard object on a screen and identify and classify various behaviors of the standard object, for example, standing, sitting, lying, normal, abnormal, etc.
- Data information representing such standard behavior may then be stored in the standard object behavior storage modules 225 , for example a database in data memory 151 .
- standard object behavior information data sets may be loaded directly into the standard object behavior storage module 225 from another system or source as long as the data is compatible with the present invention protocols and data structure.
- the system may be used to analyze and classify the behavior of one or more predetermined objects, for example, a mouse.
- digital video (either real-time and/or stored) of monitored objects to be identified and characterized is input to an object identification and segregation module 205 .
- This module identifies and segregates a predetermined type of object from the digital video image and inputs it to an object tracking module 210 .
- the object tracking module 210 facilitates tracking of the predetermined object from one frame or scene to another as feature information.
- This feature information is then extracted and input to the object shape and posture classifier 215 .
- This module classifies the various observed states of the predetermined object of interest into various shape and posture categories and sends it to the behavior identification module 230 .
- the behavior identification module 230 compares the object shape, motion, and posture information with shape, motion, and posture information for a standard object and classifies the behavior accordingly into the predefined categories exhibited by the standard object, including whether the behavior is normal, abnormal, new, etc. This information is output to the user as characterized behavior information on, for example, a display unit 157 .
- the system may receive incoming video images at step 305 , from the video camera 105 in real time, pre-recorded from the video storage/retrieval unit 110 , and/or a memory integral to the computer 150 . If the video is in analog format, then the information is converted from analog to digital format and may be compressed by the video digitization/compression unit 115 . The digital video images are then provided to the computer 150 for various computational intensive processing to identify and segment a predetermined object from the image.
- the object to be identified and whose activities are to be characterized is a moving object, for example a mouse, which has some movement from frame to frame or scene to scene in the video images and is generally in the foreground of the video images.
- the digital images may be processed to identify and segregate a desired (predetermined) object from the various frames of incoming video. This process may be achieved using, for example, background subtraction, mixture modeling, robust estimation, and/or other processes.
- various movements (or still shapes) of the desired object may then be tracked from one frame or scene to another frame or scene of video images.
- this tracking may be achieved by, for example, tracking the outline contour of the object from one frame or scene to another as it varies from shape to shape and/or location to location.
- the changes in the motion of the object such as the shapes, locations, and postures of the object of interest, may be identified and their features extracted and classified into meaningful categories. These categories may include, for example, vertical positioned side view, horizontal positioned side view, vertical positioned front view, horizontal positioned front view, moving left to right, etc.
- the states of the object may be used to characterize the objects activity into one of a number of pre-defined behaviors.
- some pre-defined normal behaviors may include sleeping, eating, drinking, walking, running, etc.
- predefined abnormal behavior may include spinning vertical, jumping in the same spot, etc.
- the pre-defined behaviors may be stored in a database in the data memory 151 .
- Types of behavior may also be characterized using, for example, approaches such as rule-based label analysis, token parsing procedure, and/or Hidden Markov Modeling (HMM).
- HMM Hidden Markov Modeling
- the HMM is particularly helpful in characterizing behavior that is determined with temporal relationships of the various motion of the object across a selection of frames. From these methods, the system may be capable of characterizing the object behavior as new behavior and particular temporal rhythm.
- step 415 video of the activities of a standard object and known behavior characteristics are input into the system.
- This information may be provided from a video storage/retrieval unit 110 in digitized video form into a standard object classified module 220 .
- This information may then be manually categorized at step 416 to define normal and abnormal activities or behaviors by a user viewing the video images on the display unit 157 and inputting their classifications. For example, experts in the field may sit together watching recorded scenes.
- an animal's e.g., a mouse
- behaviors may constitute the important posture and behavior database and are entered into a storage, for example a memory, of known activity of the standard object at step 420 .
- This information provides a point of reference for video analysis to characterize the behavior of non-standard objects whose behaviors/activities need to be characterized such as genetically altered or drug administered mice.
- normal postures and behaviors of the animals are defined and may be entered into a normal postures and behaviors database.
- the system may then be used to analyze incoming video images that may contain an object for which automated behavior characterization is desired.
- incoming video images are received.
- decision step 406 the system determines if the video images are in analog or digital format. If the video images are in analog format they are then digitized at step 407 .
- the video may be digitized and may be compressed, using, for example, a digitizer/compression unit 115 into a convenient digital video format such as MPEG, RealPlayer, etc. Otherwise, the digital video image may be input directly to the computer 150 . Now the object of interest is identified within the video images and segregated for analysis.
- a background may be generated or updated from the digital video images and foreground objects including a predetermined object for behavior characterization may be detected. For example, a mouse in a cage is detected in the foreground and segregated from the background. Then, at step 409 , features such as centroid, the principal orientation angle of the object, the area (number of pixels), the eccentricity (roundness), and the aspect ratio of the object, and/or shape in terms of contour, convex hull, or b-spline, of the foreground object of interest (e.g., a mouse) are extracted. Next, at step 410 , the foreground object shape and postures are classified into various categories, for example, standing, sitting, etc.
- the foreground object e.g., a mouse
- the foreground object posture may be compared to the various predefined postures in the set of known postures in the standard object storage of step 420 , which may be included in a database.
- the observed postures of the object contained in the analyzed video image may be classified and identified as a particular posture known for the standard object or a new previously unidentified posture.
- various groups of postures may be concatenated into a series to make up a foreground object behavior that is then compared against the sequence of postures, stored in for example a database in memory, that make up a known standard object behavior.
- This known standard behavior is, in a preferred embodiment, normal behavior for the type of animal being studied.
- the known activity of the standard object may be normal or abnormal behavior of the animal.
- the abnormal behaviors are then identified in terms of (1) known abnormal behavior; (2) new behavior likely to be abnormal; and/or (3) daily rhythm differences likely to be abnormal behavior.
- Known normal behavior may also be output as desired by the user. This information is automatically identified to the user for their review and disposition.
- the information output may include behavior information that is compatible with current statistical packages such as Systat and SPSS.
- object detection is performed through a unique method of background subtraction.
- incoming video is provided to the system for analysis. This video may be provided by digital equipment and input to the object identification and segregation module 205 of the computer 150 .
- the incoming digital video signal may be split into individual images (frames) in real-time. This step may be included if it is desired to carry out real-time analysis.
- decision step 506 the system determines if the background image needs to be developed because there was no background image developed previously or the background image has changed.
- a background image is generated by first grouping a number of frames or images into a sample of video images, for example 20 frames or images.
- the background may need to be updated periodically due to changes caused by, for example, lighting and displacement of moveable objects in the cage, such as the bedding.
- the system generates a standard deviation map of the group of images.
- an object(s) bounding box area is identified and removed from each frame or image to create a modified frame or image.
- the bounding box area is determined by sensing the area wherein the variation of a feature such as the standard deviation of intensity is above a predetermined threshold.
- an area in the digitized video image where the object of interest in motion is located is removed leaving only a partial image.
- the various modified images within the group, less the bounding box area are combined, for example averaged, to create a background image at step 511 .
- the background image does not remain constant for a great length of time due to various reasons. For example, the bedding in a mouse cage can shift due to the activity of the mouse. External factors such as change in illumination conditions also require background image recalculations. If the camera moves, then, background might need to be changed. Thus, the background typically needs to be recalculated periodically as described above or it can be recalculated by keeping track of the difference image and note any sudden changes such as an increase in the number of particular color (e.g., white) pixels in the difference image or the appearance of patches of the particular color (e.g., white) pixels in another area of the difference image. In any case, the newly generated background image may then be combined with any existing background image to create a new background image at step 511 .
- the newly generated background image may then be combined with any existing background image to create a new background image at step 511 .
- the newly generated background image is next, at step 512 , subtracted from the current video image(s) to obtain foreground areas that may include the object of interest. Further, if the background does not need to be updated as determined at decision step 506 , then the process may proceed to step 512 and the background image is subtracted from the current image, leaving the foreground objects.
- the object identification/detection process is performed.
- regions of interest ROI
- a predetermined threshold which constitute potential foreground object(s) being sought. Classification of these foreground regions of interest will be performed using the sizes of the ROIs, distances among these ROIs, threshold of intensity, and connectedness to identify the foreground objects.
- the foreground object identification/detection process may be refined by utilizing information about the actual distribution (histograms) of the intensity levels of the foreground object and using edge detection to more accurately identify the desired object(s).
- the system continuously maintains a distribution of the foreground object intensities as obtained.
- a lower threshold may be used to thereby permit a larger amount of noise to appear in the foreground image in the form of ROIs.
- a histogram is then updated with the pixels in the ROI.
- plotting a histogram of all the intensities of a particular color pixels over many images provides a bi-modal shape with the larger peak corresponding to the foreground object's intensity range and the smaller peak corresponding to the noise pixels in the ROI's images.
- step 516 having “learned” the intensity range of the foreground object, only those pixels in the foreground object that conform to this intensity range are selected, thereby identifying the foreground object more clearly even with background that is fairly similar.
- the foreground object of interest may be refined using edge information to more accurately identify the desired object.
- An edge detection mechanism such as Prewitt operator is applied to the original image. Adaptive thresholds for edge detections can be used.
- the actual boundary of the foreground object is assumed to be made up of one or more segments in the edge map, i.e., the actual contour of the foreground objects comprises edges in the edge map.
- the closed contour of the “detected” foreground object is broken into smaller segments, if necessary. Segments in the edge map that are closest to these contour segments according to a distance metric are found to be the desired contour.
- One exemplary distance metric is the sum of absolute normal distance to the edge map segment from each point in the closed contour of the “detected” foreground object.
- the previous embodiments are generally applicable to identifying, tracking, and characterizing the activities of a particular object of interest present in a video image, e.g., an animal, a human, a vehicle, etc.
- the invention is also particularly applicable to the study and analysis of animals used for testing new drugs and/or genetic mutations. As such, a number of variations of the invention related to determining changes in behavior of mice will be described in more detail below using examples of video images obtained.
- One variation of the present invention is designed particularly for the purpose of automatically determining the behavioral characteristics of a mouse.
- the need for sensitive detection of novel phenotypes of genetically manipulated or drug-administered mice demands automation of analyses. Behavioral phenotypes are often best detected when mice are unconstrained by experimenter manipulation. Thus, automation of analysis of behavior in a home cage would be a preferred means of detecting phenotypes resulting from gene manipulations or drug administrations. Automation of analysis as provided by the present invention will allow quantification of all behaviors and may provide analysis of the mouse's behavior as they vary across the daily cycle of activity.
- the automated system of the present invention may also detect behaviors that do not normally occur and present the investigator with video clips of such behavior without the investigator having to view an entire day or long period of mouse activity to manually identify the desired behavior.
- the systematically developed definition of mouse behavior that is detectable by the automated analysis of the present invention makes precise and quantitative analysis of the entire mouse behavior repertoire possible for the first time.
- the various computer algorithms included in the invention for automating behavior analysis based on the behavior definitions ensure accurate and efficient identification of mouse behaviors.
- the digital video analysis techniques of the present invention improves analysis of behavior by leading to: (1) decreased variance due to non-disturbed observation of the animal; (2) increased experiment sensitivity due to the greater number of behaviors sampled over a much longer time span than ever before possible; and (3) the potential to be applied to all common normative behavior patterns, capability to assess subtle behavioral states, and detection of changes of behavior patterns in addition to individual behaviors.
- Development activities have been complete to validate various scientific definition of mouse behaviors and to create novel digital video processing algorithms for mouse tracking and behavior recognition, which are embodied in software and hardware system according to the present invention.
- the first step in the analysis of home cage behavior is an automated initialization step that involves analysis of video images to identify the location and outline of the mouse, as indicated by step 310 .
- the location and outline of the mouse are tracked over time, as indicated by step 315 .
- Performing the initialization step periodically may be used to reset any propagation errors that appear during the tracking step.
- the mouse is tracked over time, its features including shape are extracted, and used for training and classifying the posture of the mouse from frame to frame, as indicated by step 320 .
- Posture labels are generated for each frame, which are analyzed over time to determine the actual behavior, as indicated by step 325 .
- the steps 305 , 310 , and 315 have been presented in U.S.
- the first step in analyzing a video of an animal and to analyze the behavior of the animal is to locate and extract the animal.
- a pre-generated background of the video clip in question is first obtained and it is used to determine the foreground objects by taking the intensity difference and applying a threshold procedure to remove noise.
- This step may involve threshold procedures on both the intensity and the size of region.
- An 8-connection labeling procedure may be performed to screen out disconnected small noisy regions and improve the region that corresponds to the mouse.
- all pixels in a frame will be assigned a label as foreground pixel or background pixel based on the threshold.
- the foreground pixels are further cleaned up by removing smaller components and leaving only the largest component as the foreground object.
- Those foreground pixels that border a background pixel form the contour for the object.
- the outline or contour of this foreground object is thus determined.
- the centroid (or center of mass) of the foreground object is calculated and is used for representing the location of the object (e.g., mouse).
- FIGS. 7A , 7 B, 7 C, and 7 D illustrate the results of the location and object outline identification for a mouse using the present invention.
- FIG. 7B illustrates a difference image between foreground and background for the image in FIG. 7A .
- FIG. 7C illustrates the image after thresholding showing the foreground mouse 705 object correctly identified.
- FIG. 7D illustrates the extracted contour of this object.
- the contour representation can be used as features of the foreground object, in addition to other features that include but not limited to: centroid, the principal orientation angle of the object, the area (number of pixels), the eccentricity (roundness), and the aspect ratio of object.
- Ideal tracking of foreground objects in the image domain involves a matching operation to be performed that identifies corresponding points from one frame to the next. This process may become computationally too consuming or expensive to perform in an efficient manner. Thus, one approach is to use approximations to the ideal case that can be accomplished in a short amount of time. For example, tracking the foreground object may be achieved by merely tracking the outline contour from one frame to the next in the feature space (i.e., identified foreground object image).
- tracking is performed in the feature space, which provides a close approximation to tracking in the image domain.
- the features include the centroid, principal orientation angle of the object, area (number of pixels), eccentricity (roundness), and the aspect ratio of object with lengths measured along the secondary and primary axes of the object.
- S be the set of pixels in the foreground object
- A denote the area in number of pixels
- C x , C y denote the centroid
- ⁇ denote the orientation angle
- E denote the eccentricity
- R denote the aspect ratio.
- m 0,2 ⁇ ( y ⁇ C y ) 2
- m 1,1 ⁇ ( x ⁇ C x )( y ⁇ C y )
- R is equal to the ratio of the length of the range of the points projected along an axis perpendicular to ⁇ , to the length of the range of the points projected along an axis parallel to ⁇ . This may also be defined as the aspect ratio (ratio of width to length) after rotating the foreground object by ⁇ .
- Tracking in the feature space involves following feature values from one frame to the next. For example, if the area steadily increases, it could mean that the mouse is coming out of a cuddled up position to a more elongated position, or that it could be moving from a front view to a side view, etc. If the position of the centroid of the mouse moves up, it means that the mouse may be rearing up on its hind legs. Similarly, if the angle of orientation changes from horizontal to vertical, it may be rearing up. These changes can be analyzed with combinations of features also.
- contour representation it is possible for a contour representation to be used to perform near-optimal tracking efficiently in the image domain (i.e., the complete image before background is subtracted).
- the foreground state of the mouse is classified into one of the given classes.
- This information may be stored in, for example, a database in, for example, a data memory.
- a Decision Tree classifier e.g., object shape and posture classifier 215
- the system of the present invention was exercised using these classifications. Performing a 10-fold cross-validation on the 6839 training samples, a combined accuracy of 94.6% was obtained indicating that the classifier was performing well. This is in the range of the highest levels of agreement between human observers.
- the present system provides good accuracy for mouse shape and posture recognition and classification.
- the contour of the animal object is split into smaller segments based on the curvature features. Segments are split at concave points along the contour. A segment comprising those contour pixels starting from a extreme concave point to the next extreme concave point and containing an extreme convex point is considered as a body segment. As shown in FIG. 9 , these body segments are classified into one of the following classes: Head, Forelimb, Abdomen, Hind Limb, Tail, Lower Back, Upper Back, and Ear.
- FIG. 6 A typical video frame of a mouse in its home cage is shown in FIG. 6 .
- a mouse In this video frame a mouse is shown in a rearing up posture. Many such frames make up the video of, for example, a 24 hour mouse behavior monitoring session.
- a small segment of successive frames of this video will correspond to one of the behaviors in the group of behaviors that have been modeled. The approach is to identify the correct segments and how to match those segments to the correct behavior. How each behavior is modeled is first described.
- Each behavior can be modeled as a sequence of postures of the mouse. If this particular pattern of postures is exhibited by the mouse, the corresponding behavior is detected.
- the following set of postures is being used: Horizontal Side View Posture, Vertical Posture, Cuddled Posture, Horizontal Front/Back View Posture, Partially Reared Posture, Stretched Posture, Hang Vertical Posture, Hang Cuddled Posture, Eating Posture and Drinking Posture.
- certain rules or conditions can be attached to the behavior description, which, only if satisfied will determine the corresponding behavior.
- the rules or conditions can be formulated using any of the available features or parameters including position and shape of specific body parts with or without respect to other objects, motion characteristics of the entire mouse body or individual body parts, etc. In the descriptions below, all such rules or conditions that augment the posture sequence requirement to derive the specific modeling of the behavior are stated. The behavior descriptions follow:
- Rear Up behavior is modeled as a sequence of postures starting from either of the cuddled, horizontal side-view, or horizontal front/back view postures to ending in a vertical or partially reared posture. This behavior is analogous to the standing up behavior.
- Come Down behavior is modeled as a sequence of postures starting from either vertical or partially reared posture to ending in one of cuddled, horizontal side view or horizontal front/back view postures. This behavior is analogous to the sitting down or laying down behavior.
- Eating behavior is modeled as a sequence of eating postures.
- An eating posture is an augmentation of one of the other postures by a condition that the mouth body part of the mouse is in contact with a food access area in the cage.
- Drinking behavior is modeled as a sequence of drinking postures.
- a drinking posture is an augmentation of one of the other postures by a condition that the mouth body part of the mouse is contact with a water spout in the cage.
- Digging behavior is determined by the aft movement of the bedding material in the cage by the animal with its fore and hind limbs. The displacement of the bedding is detected and the direction of movement of the bedding along with the orientation of the mouse is used to determine this behavior.
- Foraging behavior is determined by the movement of bedding material in the cage by the animal using the head and forelimbs. The displacement of the bedding is detected along with the position of the head and forelimbs and this is used to determine the foraging behavior.
- Jump behavior is modeled by a single up and down movement of the animal. Both the top of the animal and the bottom of the animal have to move monotonously up, and then, down, to determine this behavior.
- Repetitive jumping behavior is determined by several continuous up and down movements (individual jumps) of the animal.
- Sniffing behavior is determined by a random brisk movement of the mouth/nose tip of the head while the rest of the body remains stationary. The trace of the mouth tip is analyzed and the variance in its position is high relative to the bottom of the animal, a sniff is detected.
- Hang behavior is modeled as a sequence of postures starting from the vertical posture to ending in a hang vertical or hang cuddled posture.
- Land behavior is modeled as a sequence of postures starting from the hang vertical or hang cuddled posture to ending in a vertical posture.
- Sleep behavior is detected by analyzing the contour of the mouse body. If the amount of movement of this contour from one frame to the next is below a threshold value for a prolonged period of time, the mouse enters a sleep state.
- Twitch behavior is determined by the detection of a brief period of substantial movement and the resumption of sleep activity following this brief movement.
- Awaken behavior is determined by a prolonged substantial movement of the animal after sleep had set in.
- Grooming behavior is modeled as a brisk movement of limbs and head in a cyclical and periodic pattern. Variances of several shape and motion parameters, including the width and height, and area of the mouse, are calculated over time and if these variances exceed a threshold, for a prolonged period of time, groom is detected.
- Pause behavior is determined by a brief absence of movement of the animal. Similar criteria as those used for sleep detection is employed, except the duration of the behavior is much shorter, only lasting for several seconds.
- Urinate behavior is determined by the detection of the mouse tail being raised up and the mouse remaining stationary briefly while the tail is up.
- Turn behavior is modeled as a sequence of postures starting from horizontal side view or cuddled posture to ending in a horizontal front/back view posture, or vice versa. Accordingly, the turn behavior can further be classified as a Turn to Face Right, Turn to Face Left, Turn to Face Forward or Back behavior.
- Circling behavior is modeled as a succession of 3 or more turns.
- Walking or running behavior is determined by the continuous sideways movement of the centroid of the animal in one direction, to the left or right.
- the mouse needs to travel a certain minimal distance over a specified length of time for this behavior to be detected.
- Stretch behavior is modeled as a sequence of Stretched Postures.
- a Stretched posture is determined by the observation of the upper and lower back contours. If for a given frame, those body parts have a concave shape instead of a normal convex shape, and the overall shape of the animal is elongated, then a Stretched posture is detected for that frame. A sequence of these Stretched postures generates a Stretch behavior.
- This Stretch behavior can occur when the animal is horizontally elongated or vertically elongated. Horizontally elongated Stretching behavior occur after awaken behavior or when ducking under objects. Vertically elongated Stretch behavior occurs during sniffs or supported rearing behaviors.
- Chewing behavior is modeled as a movement of the mouth while the mouth is not in touch with a food container. Chews are detected only between two co-occurring Eat behaviors.
- Stationary behavior is detected when the animal remains in the same place and does not perform any of the other behaviors. It is often output as a default behavior when no other behavior can be detected. But, if the mouse moves and the movement pattern does not match any of the other behaviors, Unknown Behavior, not Stationary behavior, is selected.
- the behavior is deemed to be unknown.
- the approach is to determine those pre-defined behaviors as defined in the previous step. This process will be accomplished in real-time so that immediate results will be reported to investigators or stored in a database.
- One approach is to use a rule-based label analysis procedure (or a token parsing procedure) by which the sequence of labels is analyzed and to identify particular behaviors when its corresponding sequence of labels is derived from a video frame being analyzed. For example, if a long sequence (lasting for example several minutes) of the “Cuddled up position” label (Class 3) is observed, and if its centroid remains stationary, then, it may be concluded that the mouse is sleeping.
- the system can determine and identify that the mouse is drinking. It may also be useful for certain extra conditions to be tested such as, “some part (the mouth) of the mouse must touch the spout if drinking is to be identified” in addition to temporal characteristics of the behavior.
- HMMs Hidden Markov Models
- FIG. 8 various exemplary mouse state transitions tested in the present invention are illustrated.
- the five exemplary mouse state transitions include: (1) Horizontal Side View Posture (HS) 805 , (2) Horizontal Front/Back Posture (FB) 810 postures, (3) Cuddled Up Posture (CU) 815 , (4) Partially Reared Posture (PR) 820 , and (5) Reared Up Posture (RU) 825 .
- FIG. 8 shows the five posture states and the duration for which a mouse spent in each state in an exemplary sample video clip.
- the mouse usually passes through the partially reared posture (PR) 820 state to reach the reared up posture (RU) 825 state from the other three ground-level states.
- PR partially reared posture
- RU reared up posture
- the states are defined according to the posture classes mentioned previously.
- RUFP Rear Up From Partially Reared
- a sample sequence of postures can be 5, 5, 5, 2, 2, 2, where the numbers represent the posture class described earlier.
- another sample can be 5, 5, 2, 2, 2, 2, 2. More complicated behaviors will have more complicated patterns.
- certain augmentation rules needed to be applied, they were applied in a rule-based approach during the real-time analysis. For example, to detect grooming behavior, it is required that the variance of the width, height, and other measures be within a pre-set range while the animal has a certain sequence of postures. If both these conditions—the posture-based condition and the feature-based condition—the grooming behavior is detected.
- the mouse (or any object) may be analyzed in any location or environment.
- the invention in one variation may be used to automatically detect and characterize one or more particular behaviors.
- the system could be configured to automatically detect and characterize an animal freezing and/or touching or sniffing a particular object.
- the system could be configured to compare the object's behavior against a “norm” for a particular behavioral parameter. Other detailed activities such as skilled reaching and forelimb movements as well as social behavior among groups of animals can also be detected and characterized.
- the system of the present invention when a new video clip is analyzed, the system of the present invention first obtains the video image background and uses it to identify the foreground objects. Then, features are extracted from the foreground objects, which are in turn passed to the decision tree classifier for classification and labeling. This labeled sequence is passed to a behavior identification system module that identifies the final set of behaviors for the video clip.
- the image resolution of the system that has been obtained and the accuracy of identification of the behaviors attempted so far have been very good and resulted in an effective automated video image object recognition and behavior characterization system.
- the invention may identify some abnormal behavior by using video image information (for example, stored in memory) of known abnormal animals to build a video profile for that behavior. For example, video image of vertical spinning while hanging from the cage top was stored to memory and used to automatically identify such activity in mice. Further, abnormalities may also result from an increase in any particular type of normal behavior. Detection of such new abnormal behaviors may be achieved by the present invention detecting, for example, segments of behavior that do not fit the standard profile.
- the standard profile may be developed for a particular strain of mouse whereas detection of abnormal amounts of a normal behavior can be detected by comparison to the statistical properties of the standard profile.
- the automated analysis of the present invention may be used to build a profile of the behaviors, their amount, duration, and daily cycle for each animal, for example each commonly used strain of mice.
- a plurality of such profiles may be stored in, for example, a database in a data memory of the computer.
- the techniques developed with the present invention for automation of the categorization and quantification of all home-cage of mouse behaviors throughout the daily cycle is a powerful tool for detecting phenotypic effects of gene manipulations in mice.
- this technology is extendable to other behavior studies of animals and humans, as well as surveillance purposes.
- the present invention has proven to be a significant achievement in creating an automated system and methods for automated accurate identification, tracking and behavior categorization of an object whose image is captured in a video image.
- the analysis is performed under simulated night conditions with the use of red-light and regular visible range cameras, or with the use of no-light conditions and infra-red cameras.
- the systematically developed definitions of mouse behaviors that are detectable by the automated analysis according to the present invention makes precise and quantitative analysis of the entire mouse behavior repertoire possible for the first time.
- the various computer algorithms included in the invention for automating behavior analysis based on the behavior definitions ensure accurate and efficient identification of mouse behaviors.
- the digital video analysis techniques of the present invention improves analysis of behavior by leading to: (1) decreased variance due to non-disturbed observation of the animal; (2) increased experiment sensitivity due to the greater number of behaviors sampled over a much longer time span than ever before possible; and (3) the potential to be applied to all common normative behavior patterns, capability to assess subtle behavioral states, and detection of changes of behavior patterns in addition to individual behaviors.
- the present invention may also include audio analysis and/or multiple camera analysis.
- the video image analysis may be augmented with audio analysis since audio is typically included with most video systems today.
- audio may be an additional variable used to determine and classify a particular objects behavior.
- the analysis may be expanded to video image analysis of multiple objects, for example mice, and their social interaction with one another.
- the system may include multiple cameras providing one or more planes of view of an object to be analyzed.
- the camera may be located in remote locations and the video images sent via the Internet for analysis by a server at another site.
- the standard object behavior data and/or database may be housed in a remote location and the data files may be downloaded to a stand alone analysis system via the Internet, in accordance with the present invention.
Abstract
Systems and methods for finding the position and shape of an animal using video are disclosed. The invention includes a system with a video camera coupled to a computer in which the computer is configured to automatically provide animal segmentation and identification, animal motion tracking (for moving animals), animal posture classification, and behavior identification. In a preferred embodiment, the present invention may use background subtraction for animal identification and tracking, and a combination of decision tree classification and rule-based classification for posture and behavior identification. Thus, the present invention is capable of automatically monitoring a video image to identify, track and classify the actions of various animals and the animal's movements within the image. The image may be provided in real time or from storage.
Description
- This application is a continuation of U.S. patent application Ser. No. 10/698,044, now U.S. Pat. No. ______, which is a continuation-in-part of U.S. patent application Ser. No. 09/718,374, now U.S. Pat. No. 6,678,413. The subject matter of the related applications is hereby incorporated by reference in its entirety.
- Portions of the material in this specification arose as a result of Government support under grants MH58964, MH58964-02, and DA14889 between Clever Sys., Inc. and The National Institute of Mental Health, National Institute on Drug Abuse, National Institute of Health. The Government has certain rights in this invention.
- Animals, for example mice or rats, are used extensively as human models in the research of drag development; genetic functions; toxicology research; understanding and treatment of diseases; and other research applications. Despite the differing lifestyles of humans and animals, for example mice, their extensive genetic and neuroanatomical homologies give rise to a wide variety of behavioral processes that are widely conserved between species. Exploration of these shared brain functions will shed light on fundamental elements of human behavioral regulation. Therefore, many behavioral test experiments have been designed on animals like mice and rats to explore their behaviors. These experiments include, but not limited to, home cage behaviors, open field locomotion experiments, object recognition experiments, a variety of maze experiments, water maze experiments, and freezing experiments for conditioned fear.
- Animal's home cage activity patterns are important examination item on the general health list of animals, such as mice and rats. It provides many important indications of whether the animal's health status is normal or abnormal. Home cage behaviors are best observed by videotaping several 24-hour periods in the animal housing facility, and subsequent scoring of the videotape by two independent observers. However, this observation has rarely been done until our inventions came into play, due to the instability in long term human observation, the time consumed, and the huge costs associated with the observation.
- As discussed, all these apparatus and experiments use, in many cases, human observation of videotapes of the experiment sessions, resulting in inaccuracy, subjectivity, labor-intensive, and thus expensive experiments. Some automating software provides rudimentary and basic parameters, relying on tracking animal as a point in space, generating experiment results that are inaccurate and can not meet the demands for advanced features. Besides, each system software module works for only a specific experiment, resulting in potential discrepancy in the results across different systems due to differences in software algorithms used.
- All the observations of these behavioral experiments use video to record experiment processes and rely on human observations. This introduces the opportunity to utilize the latest technologies development in computer vision, image processing, and digital video processing to automate the processes and achieve better results, high throughput screening, and lower costs. Many of these experiments are conducted with observations performed from top view, that is, observation of the experiments from above the apparatus is used to obtain needed parameters. This also provides an opportunity to unify the approaches to observe and analyze these experiments' results.
- There are strong needs for automated systems and software that can automate the measurements of the experiments mentioned above, provide the measurements of meaningful complex behaviors and new revealing parameters that characterize animal behaviors to meet post-genomic era's demands, and obtain consistent results using novel approaches.
- The invention relates generally to behavior analysis of animal objects. More particularly, one aspect of the invention is directed to monitoring and characterization of behaviors under specific behavioral paradigm experiments, including home cage behavior paradigms, locomotion or open field paradigm experiment, object recognition paradigm experiments, variety of maze paradigm experiments, water maze paradigm experiments, freezing paradigm experiments for conditioned fear, for an animal, for example, a mouse or a rat, using video analysis from a top view image or side view image, or the integration of both views.
- A revolutionary approach is invented to automatically measure animal's home cage activity patterns. This approach consists of defining a unique set of animal's, such as mice or rats, behavior category. This category includes behaviors like rearing, walking, grooming, eating, drinking, jumping, hanging, etc. Computer systems are designed and implemented that can produce digital video files of animal's behaviors in a home cage in real time or off-line mode. Software algorithms are developed to automatically understand and analyze the animal's behaviors in those video files. This analysis is based on the premise that the entire animal body, body parts, related color information, and their dynamic motion are taken advantage of in order to provide the measurement of complex behaviors and novel parameters.
- In general, the present invention is directed to systems and methods for finding patterns of behaviors and/or activities of an animal using video. The invention includes a system with a video camera connected to a computer in which the computer is configured to automatically provide animal identification, animal motion tracking (for moving animal), animal shape, animal body parts, and posture classification, and behavior identification. Thus, the present invention is capable of automatically monitoring a video image to identify, track and classify the actions of various animals and their movements. The video image may be provided in real time from a camera and/or from a storage location. The invention is particularly useful for monitoring and classifying mice or rats behavior for testing drugs and genetic mutations, but may be used in a number of surveillance or other applications.
- In one embodiment the invention includes a system in which an analog/digital video camera and a video record/playback device (e.g., VCR) are coupled to a video digitization/compression unit. The video camera may provide a video image containing an animal to be identified. The video digitization/compression unit is coupled to a computer that is configured to automatically monitor the video image to identify, track and classify the actions of the animal and its movements over time within a sequence of video session image frames. The digitization/compression unit may convert analog video and audio into, for example, MPEG or other formats. The computer may be, for example, a personal computer, using either a Windows platform or a Unix platform, or a Macintosh computer and compatible platform. The computer is loaded and configured with custom software programs (or equipped with firmware) using, for example, MATLAB or C/C++ programming language, so as to analyze the digitized video for animal identification and segmentation, tracking, and/or behavior/activity characterization. This software may be stored in, for example, a program memory, which may include ROM, RAM, CD ROM and/or a hard drive, etc. In one variation of the invention the software (or firmware) includes a unique background subtraction method which is more simple, efficient, and accurate than those previously known.
- In operation, the system receives incoming video images from either the video camera in real time or pre-recorded from the video record/playback unit. If the video is in analog format, then the information is converted from analog to digital format and may be compressed by the video digitization/compression unit. The digital video images are then provided to the computer where various processes are undertaken to identify and segment a predetermined animal from the image. In a preferred embodiment the animal is a mouse or rat in motion with some movement from frame to frame in the video, and is in the foreground of the video images. In any case, the digital images may be processed to identify and segregate a desired (predetermined) animal from the various frames of incoming video. This process may be achieved using, for example, background subtraction, mixture modeling, robust estimation, and/or other processes.
- The shape and location of the desired animal is then tracked from one frame or scene to another frame or scene of video images. The body parts of the animal such as head, mouth, tail, ear, abdomen, lower back, upper back, forelimbs, and hind limbs, are identified by novel approaches through body contour segmentation, contour segment classification, and relaxation labeling. Next, the changes in the shapes, locations, body parts, and/or postures of the animal of interest may be identified, their features extracted, and classified into meaningful categories, for example, vertical positioned side view, horizontal positioned side view, vertical positioned front view, horizontal positioned front view, moving left to right, etc. Then, the shape, location, body parts, and posture categories may be used to characterize the animal's activity into one of a number of pre-defined behaviors. For example, if the animal is a mouse or rat, some pre-defined normal behaviors may include sleeping, eating, drinking, walking, running, etc., and pre-defined abnormal behavior may include spinning vertical, jumping in the same spot, etc. The pre-defined behaviors may be stored in a database in the data memory. The behavior may be characterized using, for example, approaches such as rule-based label analysis, token parsing procedure, and/or Hidden Markov Modeling (HMM). Further, the system may be constructed to characterize the object behavior as new behavior and particular temporal rhythm.
- In another embodiment of the invention, there are multiple cameras taking video images of experiment cages that contain animals. There is at least one cage, but as many as the computer computing power allows, say four (4) or sixteen (16) or even more, can be analyzed. Each cage contains at least one animal or multiple animals. The multiple cameras may be taking video from different points of views such as one taking video images from the side of the cage, or one taking video images from the top of the cage. When video images are taken of multiple cages and devices containing one or multiple animals, and are analyzed for identifying these animals' behaviors, high throughput screening is achieved. When video images taken from different points of views, for example, one from the top view and another from the side view, are combined to identify animal's behaviors, integrated analysis is achieved.
- In another preferred embodiment directed toward video analysis of animals such as mice or rats, the system operates as follows. As a preliminary matter, normal postures and behaviors of the animals are defined and may be entered into a Normal Paradigm Parameters, Postures and Behaviors database. In analyzing, in a first instant, incoming video images are received. The system determines if the video images are in analog or digital format and input into a computer. If the video images are in analog format they are digitized and may be compressed, using, for example, an MPEG digitizer/compression unit. Otherwise, the digital video image may be input directly to the computer. Next, a background may be generated or updated from the digital video images and foreground objects detected. Next, the foreground animal features are extracted. Also, body parts such as head, tail, ear, mouth, forelimbs, hind limbs, abdomen, and upper and lower back, are identified. Two different methods are pursuing from this point, depending on different behavior paradigms. In one method, the foreground animal shape is classified into various categories, for example, standing, sitting, etc. Next, the foreground animal posture is compared to the various predefined postures stored in the database, and then identified as a particular posture or a new (unidentified) posture. Then, various groups of postures and body parts are concatenated into a series to make up a foreground animal behavior compared against the sequence of postures, stored in for example a database in memory, that make up known normal or abnormal behaviors of the animal. The abnormal behaviors are then identified in terms of known abnormal behavior, new behavior and/or daily rhythm. In another method, behavioral processes and events are detected, and behavior parameters are calculated. These behaviors 5 parameters give indications to animal health information related to learning and memory capability, anxiety, and relations to certain diseases.
- In one variation of the invention, animal detection is performed through a unique method of background subtraction. First, the incoming digital video signal is split into individual images (frames) in real-time. Then, the system determines if the background image derived from prior incoming video needs to be updated due to changes in the background image or a background image needs to be developed because there was no background image was previously developed. If the background image needs to be generated, then a number of frames of video image, for example 20, will be grouped into a sample of images. Then, the system creates a standard deviation map of the sample of images. Next, the process removes a bounding box area in each frame or image where the variation within the group of images is above a predetermined threshold (i.e., where the object of interest or moving objects are located). Then, the various images within the sample less the bounding box area are averaged. Final background is obtained by averaging 5-10 samples. This completes the background generation process. However, often the background image does not remain constant for a great length of time due to various reasons. Thus, the background needs to be dynamically recalculated periodically as above or it can be recalculated by keeping track of the difference image and note any sudden changes. The newly dynamically generated background image is next subtracted from the current video image(s) to obtain foreground areas that may include the object of interest.
- Next, the object identification/detection process is performed. First, regions of interest (ROI) are obtained by identifying areas where the intensity difference generated from the subtraction is greater than a predetermined threshold, which constitute potential foreground object(s) being sought. Classification of these foreground regions of interest will be performed using the sizes of the ROIs, distances among these ROIs, threshold of intensity, and connectedness, to thereby identify the foreground objects. Next, the foreground object identification/detection process may be refined by adaptively learning histograms of foreground ROIs and using edge detection to more accurately identify the desired object(s). Finally, the information identifying the desired foreground object is output. The process may then continue with the tracking and/or behavior characterization step(s).
- Development activities have been completed to validate various scientific definitions of mouse behaviors and to create novel digital video processing algorithms for mouse tracking and behavior recognition, which are embodied in a software and hardware system according to the present invention. An automated method for analysis of mouse behavior from digitized 24 hours video has been achieved using the present invention and its digital video analysis method for object identification and segmentation, tracking, and classification. Several different methods and their algorithms, including Background Subtraction, Probabilistic approach with Expectation-Maximization, and Robust Estimation to find parameter values by best fitting a set of data measurements and results proved successful.
- The need for sensitive detection of novel phenotypes of genetically manipulated or drug-administered mice demands automation of analyses. Behavioral phenotypes are often best detected when mice are unconstrained by experimenter manipulation. Thus, automation of analysis of behavior in a known environment, for example a home cage, would be a powerful tool for detecting phenotypes resulting from gene manipulations or drug administrations. Automation of analysis would allow quantification of all behaviors as they vary across the daily cycle of activity. Because gene defects causing developmental disorders in humans usually result in changes in the daily rhythm of behavior, analysis of organized patterns of behavior across the day may also be effective in detecting phenotypes in transgenic and targeted mutant mice. The automated system may also be able to detect behaviors that do not normally occur and present the investigator with video clips of such behavior without the investigator having to view an entire day or long period of mouse activity to manually identify the desired behavior.
- The systematically developed definition of mouse behavior that is detectable by the automated analysis according to the present invention makes precise and quantitative analysis of the entire mouse behavior repertoire possible for the first time. The various computer algorithms included in the invention for automating behavior analysis based on the behavior definitions ensure accurate and efficient identification of mouse behaviors. In addition, the digital video analysis techniques of the present invention improves analysis of behavior by leading to: (1) decreased variance due to non-disturbed observation of the animal; (2) increased experiment sensitivity due to the greater number of behaviors sampled over a much longer time span than ever before possible; and (3) the potential to be applied to all common normative behavior patterns, capability to assess subtle behavioral states, and detection of changes of behavior patterns in addition to individual behaviors.
- The entire behavioral repertoire of individual mice in their home cage was categorized using successive iterations by manual videotape analysis. These manually defined behavior categories constituted the basis of automatic classification. Classification criteria (based on features extracted from the foreground object such as shape, position, movement) were derived and fitted into a decision tree (DT)classification algorithm. The decision tree could classify almost 7000 sample features into 8 different postures classes with accuracy over 94%. A set of HMMs have been built and used to classify the classified postures identified by the DT and yields an almost perfect mapping from input posture to output behaviors in mouse behavior sequences.
- The invention may identify some abnormal behavior by using video image information (for example, stored in memory) of known abnormal animals to build a video profile for that behavior. For example, video image of vertical spinning while hanging from the cage top was stored to memory and used to automatically identify such activity in mice. Further, abnormalities may also result from an increase in any particular type of normal behavior. Detection of such new abnormal behaviors may be achieved by the present invention detecting, for example, segments of behavior that do not fit the standard profile. The standard profile may be developed for a particular strain of mouse whereas detection of abnormal amounts of a normal behavior can be detected by comparison to the statistical properties of the standard profile.
- Thus, the automated analysis of the present invention may be used to build profiles of the behaviors, their amount, duration, and daily cycle for each animal, for example each commonly used strain of mice. A plurality of such profiles may be stored in, for example, a database in a data memory of the computer. One or more of 5 these profiles may then be compared to a mouse in question and difference from the profile expressed quantitatively.
- The techniques developed with the present invention for automation of the categorization and quantification of all home-cage mouse behaviors throughout the daily cycle is a powerful tool for detecting phenotypic effects of gene manipulations in mice. As previously discussed, this technology is extendable to other behavior studies of animals and humans, as well as surveillance purposes. As will be described in detail below, the present invention provides automated systems and methods for automated accurate identification, tracking and behavior categorization of an object whose image is captured with video.
- Other variations of the present invention is directed particularly to automatically determining the behavioral characteristics of an animal in various behavioral experiment apparatus such as water maze, Y-maze, T-maze, zero maze, elevated plus maze, locomotion open field, field for object recognition study, and cued or conditioned fear. In these experiment apparatuses, animal's body contour, center of mass, body parts including head, tail, forelimbs, hind limbs and etc. are accurately identified using the embodiments above. This allows excellent understanding of animal's behaviors within these specific experiment apparatus and procedures. Many novel and important parameters, which were beyond reach previously, are now successfully analyzed. These parameters include, but not limited to, traces of path of animal's center of mass, instant and average speed, instant and average of body turning angles, distance traveled, turning ratio, proximity score, heading error, stretch-and-attend, head-dipping, stay-across-arms, supported-rearing, sniffing (exploring) at particular objects, latency time to get to the goal (platform), time spent in specific arm/arena or specific zones within arm/arena, number of time entering and exiting arm/arena or specific zones within arm/arena, and etc. These parameters provide good indications for gene targeting, drug screening, toxicology research, learning and memory process study, anxiety study, understanding and treatment of diseases such as Parkinson's Diseases, Alzheimer Disease, ALS, and etc.
-
FIG. 1 is a block diagram of one exemplary system configurable to find the position, shape, and behavioral characteristics of an object using automated video analysis, according to one embodiment of the present invention; -
FIG. 2 is a block diagram of various functional portions of a computer system, such as the computer system shown inFIG. 1 , when configured to find the position, shape, and behavioral characteristics of an object using automated video analysis, according to one embodiment of the present invention; -
FIG. 3 is a flow chart of a method of automatic video analysis for object identification and characterization, according to one embodiment of the present invention; -
FIG. 4 is a flow chart of a method of automatic video analysis for object identification and characterization, according to another embodiment of the present invention; -
FIG. 5 is a flow chart of a method of automatic video analysis for object detection and identification, according to one variation of the present invention; -
FIG. 6 illustrates a sample video image frame with a mouse in a rearing up posture as determined using one variation of the present invention to monitor and characterize mouse behavior; -
FIG. 7A is an image showing a mouse in its home cage; -
FIG. 7B is a difference image between foreground and background for the image shown inFIG. 7A , according to one variation of the present invention as applied for monitoring and characterizing mouse behavior; -
FIG. 7C is the image shown inFIG. 7A after completing a threshold process for identifying the foreground image of the mouse which is shown as correctly identified, according to one variation of the present invention as applied for monitoring and characterizing mouse behavior; -
FIG. 7D is a computer generated image showing the outline of the foreground mouse shown inFIG. 7A after edge segmentation to demonstrate a contour-based approach to object location and outline identification, according to one variation of the present invention as applied for monitoring and characterizing mouse behavior; -
FIG. 8 is a chart illustrating one example of various mouse state transitions used in characterizing mouse behavior including: Horizontal Side View Posture (HS); Cuddled Up Posture (CU); Partially Reared Posture (PR); Rear Up Posture (RU); and Horizontal Front/Back View Posture (FB), along with an indication of duration of these states based on a sample, according to one variation of the present invention as applied for monitoring and characterizing mouse behavior; -
FIG. 9 shows the contour segmentation approach where the contour outline of the animal is split in smaller segments and each segment is classified as a body part; -
FIG. 10 shows another embodiment in night light conditions, in which night conditions are simulated using dim red light; and -
FIG. 11 shows another embodiment of the invention, a high-throughput system in which multiple cages can be analyzed at the same time. - The past few years have seen an increase in the integration of video camera and computer technologies. Today, the integration of the two technologies allows video images to be digitized, stored, and viewed on small inexpensive computers, for example, a personal computer. Further, the processing and storage capabilities of these small inexpensive computers has expanded rapidly and reduced the cost for performing data and computational intensive applications. Thus, video analysis systems may now be configured to provide robust surveillance systems that can provide automated analysis and identification of various objects and characterization of their behavior. The present invention provides such systems and related methods.
- In general, the present invention can automatically find the patterns of behaviors and/or activities of a predetermined object being monitored using video. The invention includes a system with a video camera connected to a computer in which the computer is configured to automatically provide object identification, object motion tracking (for moving objects), object shape and posture classification, and behavior identification. In a preferred embodiment the system includes various video analysis algorithms. The computer processes analyze digitized video with the various algorithms so as to automatically monitor a video image to identify, track and classify the actions of one or more predetermined objects and its movements captured by the video image as it occurs from one video frame or scene to another. The system may characterize behavior by accessing a database of object information of known behavior of the predetermined object. The image to be analyzed may be provided in real time from one or more camera and/or from storage.
- In various exemplary embodiments described in detail as follows, the invention is configured to enable monitoring and classifying of animal behavior that result from testing drugs and genetic mutations on animals. However, as indicated above, the system may be similarly configured for use in any of a number of surveillance or other applications. For example, the invention can be applied to various situations in which tracking moving objects is needed. One such situation is security surveillance in public areas like airports, military bases, or home security systems. The system may be useful in automatically identifying and notifying proper law enforcement officials if a crime is being committed and/or a particular behavior being monitored is identified. The system may be useful for monitoring of parking security or moving traffic at intersections so as to automatically identify and track vehicle activity. The system may be configured to automatically determine if a vehicle is speeding or has performed some other traffic violation. Further, the system may be configured to automatically identify and characterize human behavior involving guns or human activity related to robberies or thefts. Similarly, the invention may be capable of identifying and understanding subtle behaviors involving portions of body such as forelimb and can be applied to identify and understand human gesture recognition. This could help deaf individuals communicate. The invention may also be the basis for computer understanding of human gesture to enhance the present human-computer interface experience, where gestures will be used to interface with computers. The economic potential of applications in computer-human interface applications and in surveillance and monitoring applications is enormous.
- In one preferred embodiment illustrated in
FIG. 1 , the invention includes a system in which ananalog video camera 105 and a video storage/retrieval unit 110 may be coupled to each other and to a video digitization/compression unit 115. Thevideo camera 105 may provide a real time video image containing an object to be identified. The video storage/retrieval unit 110 may be, for example, a VCR, DVD, CD or hard disk unit. The video digitization/compression unit 115 is coupled to acomputer 150 that is configured to automatically monitor a video image to identify, track and classify the actions (or state) of the object and its movements (or stillness) over time within a sequence of images. The digitization/compression unit 115 may convert analog video and audio into, for example, MPEG format, Real Player format, etc. The computer may be, for example, a personal computer, using either a Windows platform or a Unix platform, or a Macintosh computer and compatible platform. In one variation the computer may include a number of components such as (1) adata memory 151, for example, a hard drive or other type of volatile or nonvolatile memory; (2) aprogram memory 152, for example, RAM, ROM, EEPROM, etc. that may be volatile or non-volatile memory; (3) aprocessor 153, for example, a microprocessor; and (4) a second processor to manage the computation intensive features of the system, for example, amath coprocessor 154. The computer may also include a video processor such as an MPEG encoder/decoder. Although thecomputer 150 has been shown inFIG. 1 to include two memories (data memory 151 and program memory 152) and two processors (processor 153 and math co-processor 154), in one variation the computer may include only a single processor and single memory device or more then two processors and more than two memory devices. Further, thecomputer 150 may be equipped with user interface components such as akeyboard 155,electronic mouse 156, anddisplay unit 157. - In one variation, the system may be simplified by using all digital components such as a digital video camera and a digital video storage/
retrieval unit 110, which may be one integral unit. In this case, the video digitization/compression unit 115 may not be needed. The computer is loaded and configured with custom software program(s) (or equipped with firmware) using, for example, MATLAB or C/C++ programming language, so as to analyze the digitized video for object identification and segmentation, tracking, and/or behavior/activity characterization. This software may be stored in, for example, aprogram memory 152 or data memory that may include ROM, RAM, CD ROM and/or a hard drive, etc. In one variation of the invention the software (or firmware) includes a unique background subtraction method which is more simple, efficient, and accurate than those previously known which will be discussed in detail below. In any case, the algorithms may be implemented in software and may be understood as unique functional modules as shown inFIG. 2 and now described. - Referring to
FIG. 2 , the system is preloaded with standard object information before analyzing an incoming video including a predetermined object, for example, a mouse. First, a stream of digital video including a known object with known characteristics may be fed into the system to a standardobject classifier module 220. A user may then view the standard object on a screen and identify and classify various behaviors of the standard object, for example, standing, sitting, lying, normal, abnormal, etc. Data information representing such standard behavior may then be stored in the standard objectbehavior storage modules 225, for example a database indata memory 151. Of course, standard object behavior information data sets may be loaded directly into the standard objectbehavior storage module 225 from another system or source as long as the data is compatible with the present invention protocols and data structure. In any case, once the standard object behavior data is entered into the standard objectbehavior storage module 225, the system may be used to analyze and classify the behavior of one or more predetermined objects, for example, a mouse. - In the automatic video analysis mode, digital video (either real-time and/or stored) of monitored objects to be identified and characterized is input to an object identification and
segregation module 205. This module identifies and segregates a predetermined type of object from the digital video image and inputs it to anobject tracking module 210. Theobject tracking module 210 facilitates tracking of the predetermined object from one frame or scene to another as feature information. This feature information is then extracted and input to the object shape andposture classifier 215. This module classifies the various observed states of the predetermined object of interest into various shape and posture categories and sends it to thebehavior identification module 230. Thebehavior identification module 230 compares the object shape, motion, and posture information with shape, motion, and posture information for a standard object and classifies the behavior accordingly into the predefined categories exhibited by the standard object, including whether the behavior is normal, abnormal, new, etc. This information is output to the user as characterized behavior information on, for example, adisplay unit 157. - Referring now to
FIG. 3 , a general method of operation for one embodiment of the invention will be described. In operation, in the video analysis mode the system may receive incoming video images atstep 305, from thevideo camera 105 in real time, pre-recorded from the video storage/retrieval unit 110, and/or a memory integral to thecomputer 150. If the video is in analog format, then the information is converted from analog to digital format and may be compressed by the video digitization/compression unit 115. The digital video images are then provided to thecomputer 150 for various computational intensive processing to identify and segment a predetermined object from the image. In a preferred embodiment, the object to be identified and whose activities are to be characterized is a moving object, for example a mouse, which has some movement from frame to frame or scene to scene in the video images and is generally in the foreground of the video images. In any case, atstep 310 the digital images may be processed to identify and segregate a desired (predetermined) object from the various frames of incoming video. This process may be achieved using, for example, background subtraction, mixture modeling, robust estimation, and/or other processes. - Next, at
step 315, various movements (or still shapes) of the desired object may then be tracked from one frame or scene to another frame or scene of video images. As will be discussed in more detail below, this tracking may be achieved by, for example, tracking the outline contour of the object from one frame or scene to another as it varies from shape to shape and/or location to location. Next, atstep 320, the changes in the motion of the object, such as the shapes, locations, and postures of the object of interest, may be identified and their features extracted and classified into meaningful categories. These categories may include, for example, vertical positioned side view, horizontal positioned side view, vertical positioned front view, horizontal positioned front view, moving left to right, etc. Then, atstep 325, the states of the object, for example the shape, location, and posture categories, may be used to characterize the objects activity into one of a number of pre-defined behaviors. For example, if the object is an animal, some pre-defined normal behaviors may include sleeping, eating, drinking, walking, running, etc., and predefined abnormal behavior may include spinning vertical, jumping in the same spot, etc. The pre-defined behaviors may be stored in a database in thedata memory 151. - Types of behavior may also be characterized using, for example, approaches such as rule-based label analysis, token parsing procedure, and/or Hidden Markov Modeling (HMM). The HMM is particularly helpful in characterizing behavior that is determined with temporal relationships of the various motion of the object across a selection of frames. From these methods, the system may be capable of characterizing the object behavior as new behavior and particular temporal rhythm.
- Referring now to
FIG. 4 a more detailed description of another preferred embodiment will be described. In this case the system is directed toward video analysis of animated objects such as animals. As a preliminary matter, atstep 415 video of the activities of a standard object and known behavior characteristics are input into the system. This information may be provided from a video storage/retrieval unit 110 in digitized video form into a standard object classifiedmodule 220. This information may then be manually categorized atstep 416 to define normal and abnormal activities or behaviors by a user viewing the video images on thedisplay unit 157 and inputting their classifications. For example, experts in the field may sit together watching recorded scenes. They may then define, for example, an animal's (e.g., a mouse) behavior(s), both qualitatively and quantitatively, with or without some help from systems like the Noldus Observer system. These cataloged behaviors may constitute the important posture and behavior database and are entered into a storage, for example a memory, of known activity of the standard object atstep 420. This information provides a point of reference for video analysis to characterize the behavior of non-standard objects whose behaviors/activities need to be characterized such as genetically altered or drug administered mice. For example, normal postures and behaviors of the animals are defined and may be entered into a normal postures and behaviors database. - Once information related to characterizing a standard object(s) is established, the system may then be used to analyze incoming video images that may contain an object for which automated behavior characterization is desired. First, at
step 405, incoming video images are received. Next, atdecision step 406, the system determines if the video images are in analog or digital format. If the video images are in analog format they are then digitized at step 407. The video may be digitized and may be compressed, using, for example, a digitizer/compression unit 115 into a convenient digital video format such as MPEG, RealPlayer, etc. Otherwise, the digital video image may be input directly to thecomputer 150. Now the object of interest is identified within the video images and segregated for analysis. As such, atstep 408, a background may be generated or updated from the digital video images and foreground objects including a predetermined object for behavior characterization may be detected. For example, a mouse in a cage is detected in the foreground and segregated from the background. Then, atstep 409, features such as centroid, the principal orientation angle of the object, the area (number of pixels), the eccentricity (roundness), and the aspect ratio of the object, and/or shape in terms of contour, convex hull, or b-spline, of the foreground object of interest (e.g., a mouse) are extracted. Next, atstep 410, the foreground object shape and postures are classified into various categories, for example, standing, sitting, etc. - Then, at
step 411, the foreground object (e.g., a mouse) posture may be compared to the various predefined postures in the set of known postures in the standard object storage ofstep 420, which may be included in a database. Atstep 412, the observed postures of the object contained in the analyzed video image may be classified and identified as a particular posture known for the standard object or a new previously unidentified posture. Next, atstep 413, various groups of postures may be concatenated into a series to make up a foreground object behavior that is then compared against the sequence of postures, stored in for example a database in memory, that make up a known standard object behavior. This known standard behavior is, in a preferred embodiment, normal behavior for the type of animal being studied. However, the known activity of the standard object may be normal or abnormal behavior of the animal. In either case, atstep 414, the abnormal behaviors are then identified in terms of (1) known abnormal behavior; (2) new behavior likely to be abnormal; and/or (3) daily rhythm differences likely to be abnormal behavior. Known normal behavior may also be output as desired by the user. This information is automatically identified to the user for their review and disposition. In one variation of the invention, the information output may include behavior information that is compatible with current statistical packages such as Systat and SPSS. - In one embodiment of the invention as illustrated in
FIG. 5 , object detection is performed through a unique method of background subtraction. First, atstep 405, incoming video is provided to the system for analysis. This video may be provided by digital equipment and input to the object identification andsegregation module 205 of thecomputer 150. Next, atstep 505, the incoming digital video signal may be split into individual images (frames) in real-time. This step may be included if it is desired to carry out real-time analysis. Then, atdecision step 506, the system determines if the background image needs to be developed because there was no background image developed previously or the background image has changed. If the background image needs to be generated or updated, then at step 507 a background image is generated by first grouping a number of frames or images into a sample of video images, for example 20 frames or images. The background may need to be updated periodically due to changes caused by, for example, lighting and displacement of moveable objects in the cage, such as the bedding. Then, atstep 508 the system generates a standard deviation map of the group of images. Next, atstep 509, an object(s) bounding box area is identified and removed from each frame or image to create a modified frame or image. The bounding box area is determined by sensing the area wherein the variation of a feature such as the standard deviation of intensity is above a predetermined threshold. Thus, an area in the digitized video image where the object of interest in motion is located is removed leaving only a partial image. Then, atstep 510, the various modified images within the group, less the bounding box area, are combined, for example averaged, to create a background image atstep 511. - Since varying pixels are not used in averaging, “holes” will be created in each image that is being used in the averaging process. Over time, not all frames will have these holes at the same location and hence, a complete background image is obtained
- after the averaging process. Final background is obtained by averaging 5-10 samples. This completes at least one iteration of the background generation process.
- The background image does not remain constant for a great length of time due to various reasons. For example, the bedding in a mouse cage can shift due to the activity of the mouse. External factors such as change in illumination conditions also require background image recalculations. If the camera moves, then, background might need to be changed. Thus, the background typically needs to be recalculated periodically as described above or it can be recalculated by keeping track of the difference image and note any sudden changes such as an increase in the number of particular color (e.g., white) pixels in the difference image or the appearance of patches of the particular color (e.g., white) pixels in another area of the difference image. In any case, the newly generated background image may then be combined with any existing background image to create a new background image at
step 511. - The newly generated background image is next, at
step 512, subtracted from the current video image(s) to obtain foreground areas that may include the object of interest. Further, if the background does not need to be updated as determined atdecision step 506, then the process may proceed to step 512 and the background image is subtracted from the current image, leaving the foreground objects. - Next, at steps 513-518, the object identification/detection process is performed. First, at
step 513, regions of interest (ROI) are obtained by identifying an area where the intensity difference is greater than a predetermined threshold, which constitute potential foreground object(s) being sought. Classification of these foreground regions of interest will be performed using the sizes of the ROIs, distances among these ROIs, threshold of intensity, and connectedness to identify the foreground objects. Next, the foreground object identification/detection process may be refined by utilizing information about the actual distribution (histograms) of the intensity levels of the foreground object and using edge detection to more accurately identify the desired object(s). - At
step 514, during both the background generation and background subtraction steps for object identification, the system continuously maintains a distribution of the foreground object intensities as obtained. A lower threshold may be used to thereby permit a larger amount of noise to appear in the foreground image in the form of ROIs. Thus, atstep 514, a histogram is then updated with the pixels in the ROI. Atstep 515, plotting a histogram of all the intensities of a particular color pixels over many images, provides a bi-modal shape with the larger peak corresponding to the foreground object's intensity range and the smaller peak corresponding to the noise pixels in the ROI's images. Now, atstep 516, having “learned” the intensity range of the foreground object, only those pixels in the foreground object that conform to this intensity range are selected, thereby identifying the foreground object more clearly even with background that is fairly similar. - In any case, next at
step 517 the foreground object of interest may be refined using edge information to more accurately identify the desired object. An edge detection mechanism such as Prewitt operator is applied to the original image. Adaptive thresholds for edge detections can be used. Once the edge map is obtained, the actual boundary of the foreground object is assumed to be made up of one or more segments in the edge map, i.e., the actual contour of the foreground objects comprises edges in the edge map. The closed contour of the “detected” foreground object is broken into smaller segments, if necessary. Segments in the edge map that are closest to these contour segments according to a distance metric are found to be the desired contour. One exemplary distance metric is the sum of absolute normal distance to the edge map segment from each point in the closed contour of the “detected” foreground object. Finally, atstep 518 the information identifying the desired foreground object is output. The process may then continue with tracking and/or behavior characterization steps. - The previous embodiments are generally applicable to identifying, tracking, and characterizing the activities of a particular object of interest present in a video image, e.g., an animal, a human, a vehicle, etc. However, the invention is also particularly applicable to the study and analysis of animals used for testing new drugs and/or genetic mutations. As such, a number of variations of the invention related to determining changes in behavior of mice will be described in more detail below using examples of video images obtained.
- One variation of the present invention is designed particularly for the purpose of automatically determining the behavioral characteristics of a mouse. The need for sensitive detection of novel phenotypes of genetically manipulated or drug-administered mice demands automation of analyses. Behavioral phenotypes are often best detected when mice are unconstrained by experimenter manipulation. Thus, automation of analysis of behavior in a home cage would be a preferred means of detecting phenotypes resulting from gene manipulations or drug administrations. Automation of analysis as provided by the present invention will allow quantification of all behaviors and may provide analysis of the mouse's behavior as they vary across the daily cycle of activity. Because gene defects causing developmental disorders in humans usually result in changes in the daily rhythm of behavior, analysis of organized patterns of behavior across the day may be effective in detecting phenotypes in transgenic and targeted mutant mice. The automated system of the present invention may also detect behaviors that do not normally occur and present the investigator with video clips of such behavior without the investigator having to view an entire day or long period of mouse activity to manually identify the desired behavior.
- The systematically developed definition of mouse behavior that is detectable by the automated analysis of the present invention makes precise and quantitative analysis of the entire mouse behavior repertoire possible for the first time. The various computer algorithms included in the invention for automating behavior analysis based on the behavior definitions ensure accurate and efficient identification of mouse behaviors. In addition, the digital video analysis techniques of the present invention improves analysis of behavior by leading to: (1) decreased variance due to non-disturbed observation of the animal; (2) increased experiment sensitivity due to the greater number of behaviors sampled over a much longer time span than ever before possible; and (3) the potential to be applied to all common normative behavior patterns, capability to assess subtle behavioral states, and detection of changes of behavior patterns in addition to individual behaviors. Development activities have been complete to validate various scientific definition of mouse behaviors and to create novel digital video processing algorithms for mouse tracking and behavior recognition, which are embodied in software and hardware system according to the present invention.
- Various lighting options for videotaping have been evaluated. Lighting at night as well as with night vision cameras was evaluated. It has been determined that good quality video was obtained with normal commercial video cameras using dim red light, a frequency that is not visible to rodents. Videos were taken in a standard laboratory environment using commercially
available cameras 105, for example a Sony analog camera, to ensure that the computer algorithms developed would be applicable to the quality of video available in the average laboratory. The commercially available cameras with white lighting gave good results during the daytime and dim red lighting gave good results at night time. - Referring again to
FIG. 3 , the first step in the analysis of home cage behavior is an automated initialization step that involves analysis of video images to identify the location and outline of the mouse, as indicated bystep 310. Second, the location and outline of the mouse are tracked over time, as indicated bystep 315. Performing the initialization step periodically may be used to reset any propagation errors that appear during the tracking step. As the mouse is tracked over time, its features including shape are extracted, and used for training and classifying the posture of the mouse from frame to frame, as indicated bystep 320. Posture labels are generated for each frame, which are analyzed over time to determine the actual behavior, as indicated bystep 325. Thesteps steps step 325. - Location and Outline Identification and Feature Extraction
- The first step in analyzing a video of an animal and to analyze the behavior of the animal is to locate and extract the animal. A pre-generated background of the video clip in question is first obtained and it is used to determine the foreground objects by taking the intensity difference and applying a threshold procedure to remove noise. This step may involve threshold procedures on both the intensity and the size of region. An 8-connection labeling procedure may be performed to screen out disconnected small noisy regions and improve the region that corresponds to the mouse. In the labeling process, all pixels in a frame will be assigned a label as foreground pixel or background pixel based on the threshold. The foreground pixels are further cleaned up by removing smaller components and leaving only the largest component as the foreground object. Those foreground pixels that border a background pixel form the contour for the object. The outline or contour of this foreground object is thus determined. The centroid (or center of mass) of the foreground object is calculated and is used for representing the location of the object (e.g., mouse).
-
FIGS. 7A , 7B, 7C, and 7D illustrate the results of the location and object outline identification for a mouse using the present invention.FIG. 7B illustrates a difference image between foreground and background for the image inFIG. 7A .FIG. 7C illustrates the image after thresholding showing theforeground mouse 705 object correctly identified.FIG. 7D illustrates the extracted contour of this object. - The contour representation can be used as features of the foreground object, in addition to other features that include but not limited to: centroid, the principal orientation angle of the object, the area (number of pixels), the eccentricity (roundness), and the aspect ratio of object.
- Mouse Tracking
- Ideal tracking of foreground objects in the image domain involves a matching operation to be performed that identifies corresponding points from one frame to the next. This process may become computationally too consuming or expensive to perform in an efficient manner. Thus, one approach is to use approximations to the ideal case that can be accomplished in a short amount of time. For example, tracking the foreground object may be achieved by merely tracking the outline contour from one frame to the next in the feature space (i.e., identified foreground object image).
- In one variation of the invention, tracking is performed in the feature space, which provides a close approximation to tracking in the image domain. The features include the centroid, principal orientation angle of the object, area (number of pixels), eccentricity (roundness), and the aspect ratio of object with lengths measured along the secondary and primary axes of the object. In this case, let S be the set of pixels in the foreground object, A denote the area in number of pixels, (Cx, Cy) denote the centroid, φ denote the orientation angle, E denote the eccentricity, and R denote the aspect ratio. Then,
-
- Let us define three intermediate terms, called second order moments,
-
m 2,0=Σ(x−C x)2 -
m 0,2=Σ(y−C y)2 -
m 1,1=Σ(x−C x)(y−C y) - Using the central moments, we define,
-
- R is equal to the ratio of the length of the range of the points projected along an axis perpendicular to φ, to the length of the range of the points projected along an axis parallel to φ. This may also be defined as the aspect ratio (ratio of width to length) after rotating the foreground object by φ.
- Tracking in the feature space involves following feature values from one frame to the next. For example, if the area steadily increases, it could mean that the mouse is coming out of a cuddled up position to a more elongated position, or that it could be moving from a front view to a side view, etc. If the position of the centroid of the mouse moves up, it means that the mouse may be rearing up on its hind legs. Similarly, if the angle of orientation changes from horizontal to vertical, it may be rearing up. These changes can be analyzed with combinations of features also.
- However, it is possible for a contour representation to be used to perform near-optimal tracking efficiently in the image domain (i.e., the complete image before background is subtracted).
- Mouse Posture Classification
- Once the features are obtained for the frames in the video sequence, the foreground state of the mouse is classified into one of the given classes. This involves building a classifier that can classify the shape using the available features. This information may be stored in, for example, a database in, for example, a data memory. In one variation of the invention a Decision Tree classifier (e.g., object shape and posture classifier 215) was implemented by training the classifier with 6839 samples of digitized video of a standard, in this case, normal mouse. Six attributes (or features) for each sample were identified. Ten posture classes for classification were identified as listed below.
-
- 1. Horizontal Side View Posture—Horizontally positioned, side view, either in normal state or elongated.
- 2. Vertical Posture—Vertically positioned in a reared state (e.g., See
FIG. 6 ). - 3. Cuddled Posture—Cuddled up position (like a ball).
- 4. Horizontal Front/Back View Posture—Horizontally positioned, but either front or back view, i.e., axis of mouse along the viewer's line of sight.
- 5. Partially Reared Posture—Partially reared, e.g., when drinking or eating, sitting on hind legs (e.g., See
FIG. 7A ). - 6. Stretched Posture—Stretched horizontally or vertically.
- 7. Hang Vertical Posture—Hanging vertically from the top of the cage or food bin.
- 8. Hang Cuddled Posture—Hanging cuddled up close to the top of the cage or on the food bin.
- 9. Eating Posture—In one of the earlier 8 posture with the added condition that the mouth is in touch with the food bin.
- 10. Drinking Posture—In one of the postures 1-8 with the added condition that the mouth is in touch with the water spout.
- The system of the present invention was exercised using these classifications. Performing a 10-fold cross-validation on the 6839 training samples, a combined accuracy of 94.6% was obtained indicating that the classifier was performing well. This is in the range of the highest levels of agreement between human observers. The present system provides good accuracy for mouse shape and posture recognition and classification.
- After the posture is classified, various body parts of the animal that can be obtained from that posture is detected. The contour of the animal object is split into smaller segments based on the curvature features. Segments are split at concave points along the contour. A segment comprising those contour pixels starting from a extreme concave point to the next extreme concave point and containing an extreme convex point is considered as a body segment. As shown in
FIG. 9 , these body segments are classified into one of the following classes: Head, Forelimb, Abdomen, Hind Limb, Tail, Lower Back, Upper Back, and Ear. - With the combination of the posture information and the body part information from a plurality of frames, behaviors are modeled and detected.
- Behavior Detection Methodology
- A typical video frame of a mouse in its home cage is shown in
FIG. 6 . In this video frame a mouse is shown in a rearing up posture. Many such frames make up the video of, for example, a 24 hour mouse behavior monitoring session. A small segment of successive frames of this video will correspond to one of the behaviors in the group of behaviors that have been modeled. The approach is to identify the correct segments and how to match those segments to the correct behavior. How each behavior is modeled is first described. - Each behavior can be modeled as a sequence of postures of the mouse. If this particular pattern of postures is exhibited by the mouse, the corresponding behavior is detected. The following set of postures is being used: Horizontal Side View Posture, Vertical Posture, Cuddled Posture, Horizontal Front/Back View Posture, Partially Reared Posture, Stretched Posture, Hang Vertical Posture, Hang Cuddled Posture, Eating Posture and Drinking Posture. Apart from modeling a behavior as a sequence of postures, certain rules or conditions can be attached to the behavior description, which, only if satisfied will determine the corresponding behavior. The rules or conditions can be formulated using any of the available features or parameters including position and shape of specific body parts with or without respect to other objects, motion characteristics of the entire mouse body or individual body parts, etc. In the descriptions below, all such rules or conditions that augment the posture sequence requirement to derive the specific modeling of the behavior are stated. The behavior descriptions follow:
- A. Rear Up to a Full or a Partially Reared Posture
- Rear Up behavior is modeled as a sequence of postures starting from either of the cuddled, horizontal side-view, or horizontal front/back view postures to ending in a vertical or partially reared posture. This behavior is analogous to the standing up behavior.
- B. Come Down Fully or to a Partially Reared Posture
- Come Down behavior is modeled as a sequence of postures starting from either vertical or partially reared posture to ending in one of cuddled, horizontal side view or horizontal front/back view postures. This behavior is analogous to the sitting down or laying down behavior.
- C. Eat
- Eating behavior is modeled as a sequence of eating postures. An eating posture is an augmentation of one of the other postures by a condition that the mouth body part of the mouse is in contact with a food access area in the cage.
- D. Drink
- Drinking behavior is modeled as a sequence of drinking postures. A drinking posture is an augmentation of one of the other postures by a condition that the mouth body part of the mouse is contact with a water spout in the cage.
- E. Dig
- Digging behavior is determined by the aft movement of the bedding material in the cage by the animal with its fore and hind limbs. The displacement of the bedding is detected and the direction of movement of the bedding along with the orientation of the mouse is used to determine this behavior.
- F. Forage
- Foraging behavior is determined by the movement of bedding material in the cage by the animal using the head and forelimbs. The displacement of the bedding is detected along with the position of the head and forelimbs and this is used to determine the foraging behavior.
- G. Jump
- Jump behavior is modeled by a single up and down movement of the animal. Both the top of the animal and the bottom of the animal have to move monotonously up, and then, down, to determine this behavior.
- H. Jump Repetitively
- Repetitive jumping behavior is determined by several continuous up and down movements (individual jumps) of the animal.
- I. Sniff
- Sniffing behavior is determined by a random brisk movement of the mouth/nose tip of the head while the rest of the body remains stationary. The trace of the mouth tip is analyzed and the variance in its position is high relative to the bottom of the animal, a sniff is detected.
- J. Hang
- Hang behavior is modeled as a sequence of postures starting from the vertical posture to ending in a hang vertical or hang cuddled posture.
- K. Land After Hanging
- Land behavior is modeled as a sequence of postures starting from the hang vertical or hang cuddled posture to ending in a vertical posture.
- L. Sleep
- Sleep behavior is detected by analyzing the contour of the mouse body. If the amount of movement of this contour from one frame to the next is below a threshold value for a prolonged period of time, the mouse enters a sleep state.
- M. Twitch During Sleep
- Twitch behavior is determined by the detection of a brief period of substantial movement and the resumption of sleep activity following this brief movement.
- N. Awaken from Sleep
- Awaken behavior is determined by a prolonged substantial movement of the animal after sleep had set in.
- O. Groom
- Grooming behavior is modeled as a brisk movement of limbs and head in a cyclical and periodic pattern. Variances of several shape and motion parameters, including the width and height, and area of the mouse, are calculated over time and if these variances exceed a threshold, for a prolonged period of time, groom is detected.
- P. Pause Briefly
- Pause behavior is determined by a brief absence of movement of the animal. Similar criteria as those used for sleep detection is employed, except the duration of the behavior is much shorter, only lasting for several seconds.
- Q. Urinate
- Urinate behavior is determined by the detection of the mouse tail being raised up and the mouse remaining stationary briefly while the tail is up.
- R. Turn
- Turn behavior is modeled as a sequence of postures starting from horizontal side view or cuddled posture to ending in a horizontal front/back view posture, or vice versa. Accordingly, the turn behavior can further be classified as a Turn to Face Right, Turn to Face Left, Turn to Face Forward or Back behavior.
- S. Circle
- Circling behavior is modeled as a succession of 3 or more turns.
- T. Walk
- Walking or running behavior is determined by the continuous sideways movement of the centroid of the animal in one direction, to the left or right. The mouse needs to travel a certain minimal distance over a specified length of time for this behavior to be detected.
- U. Stretch
- Stretch behavior is modeled as a sequence of Stretched Postures. A Stretched posture is determined by the observation of the upper and lower back contours. If for a given frame, those body parts have a concave shape instead of a normal convex shape, and the overall shape of the animal is elongated, then a Stretched posture is detected for that frame. A sequence of these Stretched postures generates a Stretch behavior. This Stretch behavior can occur when the animal is horizontally elongated or vertically elongated. Horizontally elongated Stretching behavior occur after awaken behavior or when ducking under objects. Vertically elongated Stretch behavior occurs during sniffs or supported rearing behaviors.
- V. Chew
- Chewing behavior is modeled as a movement of the mouth while the mouth is not in touch with a food container. Chews are detected only between two co-occurring Eat behaviors.
- W. Stationary
- Stationary behavior is detected when the animal remains in the same place and does not perform any of the other behaviors. It is often output as a default behavior when no other behavior can be detected. But, if the mouse moves and the movement pattern does not match any of the other behaviors, Unknown Behavior, not Stationary behavior, is selected.
- X. Unknown Behavior
- If the activity cannot be characterized by any of the behavior models, the behavior is deemed to be unknown.
- Behavior identification
- Using the posture labels assigned for the frames in the video clip, the approach is to determine those pre-defined behaviors as defined in the previous step. This process will be accomplished in real-time so that immediate results will be reported to investigators or stored in a database. One approach is to use a rule-based label analysis procedure (or a token parsing procedure) by which the sequence of labels is analyzed and to identify particular behaviors when its corresponding sequence of labels is derived from a video frame being analyzed. For example, if a long sequence (lasting for example several minutes) of the “Cuddled up position” label (Class 3) is observed, and if its centroid remains stationary, then, it may be concluded that the mouse is sleeping. If the location of the waterspout is identified, and if we observe a series of “partially reared” (Class 5) labels, and if the position of the centroid, and the mouse's angle of orientation fall within a small range that has been predetermined, the system can determine and identify that the mouse is drinking. It may also be useful for certain extra conditions to be tested such as, “some part (the mouth) of the mouse must touch the spout if drinking is to be identified” in addition to temporal characteristics of the behavior.
- While this approach is very straightforward, a better approach involves using a probabilistic model such as Hidden Markov Models (HMMs), where models may be built for each class of behavior with training samples. These models may then be used to identify behaviors based on the incoming sequence of labels. The HMM can provide significant added accuracy to temporal relationships for proper complex behavior characterization.
- Referring now to
FIG. 8 , various exemplary mouse state transitions tested in the present invention are illustrated. The five exemplary mouse state transitions include: (1) Horizontal Side View Posture (HS) 805, (2) Horizontal Front/Back Posture (FB) 810 postures, (3) Cuddled Up Posture (CU) 815, (4) Partially Reared Posture (PR) 820, and (5) Reared Up Posture (RU) 825. As illustrated,FIG. 8 shows the five posture states and the duration for which a mouse spent in each state in an exemplary sample video clip. One example of a pattern that is understandable and evident from the figure is that the mouse usually passes through the partially reared posture (PR) 820 state to reach the reared up posture (RU) 825 state from the other three ground-level states. The states are defined according to the posture classes mentioned previously. - Many important features can be derived from this representation, e.g., if the state changes are very frequent, it would imply that the mouse is very active. If the mouse remained in a single ground-level state such as “cuddled-up” (class 3) for an extended period of time, the system may conclude that the mouse is sleeping or resting. The sequence of transitions are also important, e.g., if the mouse rears (Class 2) from a ground-level state such as “Horizontally positioned” (Class 1), it should pass briefly through the partially reared state (Class 5). Techniques such as HMMs exploit these types of time-sequence-dependent information for performing classification.
- Each of the behaviors described in the previous section that can be modeled as a sequence of postures, was provided with a trained HMM representing that behavior only. Hence, there was a one-to-one correspondence between each HMM and a behavior that it represented. For example, an HMM corresponding to Rear Up From Partially Reared (RUFP) was created to represent the Rear Up behavior from a partially reared state fully to a reared up state. This was done during the training step.
- During HMM training, a posture sequence from real-video data was extracted
- that corresponded to various behaviors. Several samples for each behavior were collected. A separate HMM was generated for each of these behaviors that could be represented by a simple sequence of postures. For example, for a Rear Up From Partially Reared (RUFP) behavior, a sample sequence of postures can be 5, 5, 5, 2, 2, 2, where the numbers represent the posture class described earlier. Similarly, another sample can be 5, 5, 2, 2, 2, 2, 2. More complicated behaviors will have more complicated patterns.
- Once trained, these HMMs will match best with a sequence of labels that has
-
- a pattern similar to those used for training. For example, an input sequence of the
- form 5, 5, 5, 5, 5, 2, 2, 2 will match with the RUFP better than any other HMM. Hence, during analysis, the incoming sequence of labels is grouped and presented to all the HMMs and the winning HMM (or the best matching HMM) is selected as the corresponding behavior for that frame sequence. Continuing this process, all the behaviors that occur in succession are detected and output.
- One of the distinct advantages of using the HMM approach is that noise during analysis does not affect the match values much. So, the sequence 5, 5, 5, 7, 2, 2, 2, will still match with the RUFP HMM better than any other HMM.
- If certain augmentation rules needed to be applied, they were applied in a rule-based approach during the real-time analysis. For example, to detect grooming behavior, it is required that the variance of the width, height, and other measures be within a pre-set range while the animal has a certain sequence of postures. If both these conditions—the posture-based condition and the feature-based condition—the grooming behavior is detected.
- Although the above exemplary embodiment is directed to a mouse analyzed in a home cage, it is to be understood that the mouse (or any object) may be analyzed in any location or environment. Further, the invention in one variation may be used to automatically detect and characterize one or more particular behaviors. For example, the system could be configured to automatically detect and characterize an animal freezing and/or touching or sniffing a particular object. Also, the system could be configured to compare the object's behavior against a “norm” for a particular behavioral parameter. Other detailed activities such as skilled reaching and forelimb movements as well as social behavior among groups of animals can also be detected and characterized.
- In summary, when a new video clip is analyzed, the system of the present invention first obtains the video image background and uses it to identify the foreground objects. Then, features are extracted from the foreground objects, which are in turn passed to the decision tree classifier for classification and labeling. This labeled sequence is passed to a behavior identification system module that identifies the final set of behaviors for the video clip. The image resolution of the system that has been obtained and the accuracy of identification of the behaviors attempted so far have been very good and resulted in an effective automated video image object recognition and behavior characterization system.
- The invention may identify some abnormal behavior by using video image information (for example, stored in memory) of known abnormal animals to build a video profile for that behavior. For example, video image of vertical spinning while hanging from the cage top was stored to memory and used to automatically identify such activity in mice. Further, abnormalities may also result from an increase in any particular type of normal behavior. Detection of such new abnormal behaviors may be achieved by the present invention detecting, for example, segments of behavior that do not fit the standard profile. The standard profile may be developed for a particular strain of mouse whereas detection of abnormal amounts of a normal behavior can be detected by comparison to the statistical properties of the standard profile. Thus, the automated analysis of the present invention may be used to build a profile of the behaviors, their amount, duration, and daily cycle for each animal, for example each commonly used strain of mice. A plurality of such profiles may be stored in, for example, a database in a data memory of the computer. One or more of
- these profiles may then be compared to a mouse in question and difference from the profile expressed quantitatively.
- The techniques developed with the present invention for automation of the categorization and quantification of all home-cage of mouse behaviors throughout the daily cycle is a powerful tool for detecting phenotypic effects of gene manipulations in mice. As previously discussed, this technology is extendable to other behavior studies of animals and humans, as well as surveillance purposes. In any case, the present invention has proven to be a significant achievement in creating an automated system and methods for automated accurate identification, tracking and behavior categorization of an object whose image is captured in a video image.
- In another embodiment of the invention, the analysis is performed under simulated night conditions with the use of red-light and regular visible range cameras, or with the use of no-light conditions and infra-red cameras.
- In another embodiment of the invention, there are multiple cameras taking video images of experiment cages that contain animals. There is at least one cage, but as many as the computer computing power allows, say four (4) or sixteen (16) or even more, can be analyzed.
- The systematically developed definitions of mouse behaviors that are detectable by the automated analysis according to the present invention makes precise and quantitative analysis of the entire mouse behavior repertoire possible for the first time. The various computer algorithms included in the invention for automating behavior analysis based on the behavior definitions ensure accurate and efficient identification of mouse behaviors. In addition, the digital video analysis techniques of the present invention improves analysis of behavior by leading to: (1) decreased variance due to non-disturbed observation of the animal; (2) increased experiment sensitivity due to the greater number of behaviors sampled over a much longer time span than ever before possible; and (3) the potential to be applied to all common normative behavior patterns, capability to assess subtle behavioral states, and detection of changes of behavior patterns in addition to individual behaviors.
- Although particular embodiments of the present invention have been shown and described, it will be understood that it is not intended to limit the invention to the preferred or disclosed embodiments, and it will be obvious to those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the present invention. Thus, the invention is intended to cover alternatives, modifications, and equivalents, which may be included within the spirit and scope of the invention as defined by the claims.
- For example, the present invention may also include audio analysis and/or multiple camera analysis. The video image analysis may be augmented with audio analysis since audio is typically included with most video systems today. As such, audio may be an additional variable used to determine and classify a particular objects behavior. Further, in another variation, the analysis may be expanded to video image analysis of multiple objects, for example mice, and their social interaction with one another. In a still further variation, the system may include multiple cameras providing one or more planes of view of an object to be analyzed. In an even further variation, the camera may be located in remote locations and the video images sent via the Internet for analysis by a server at another site. In fact, the standard object behavior data and/or database may be housed in a remote location and the data files may be downloaded to a stand alone analysis system via the Internet, in accordance with the present invention. These additional features/functions add versatility to the present invention and may improve the behavior characterization capabilities of the present invention to thereby achieve object behavior categorization which is nearly perfect to that of a human observer for a broad spectrum of applications.
Claims (22)
1. A method for characterizing animal behavior, comprising:
segregating images of an animal from video images;
classifying a posture of the animal in each of a series of images of the animal;
identifying at least one body segment of the animal in each of the series of images of the animal; and
identifying behavior of the animal as one of a set of predetermined behaviors using the postures and the body segments of the animal over the series of images of the animal.
2. The method of claim 1 , wherein segregating images of an animal from video images includes subtracting a background image from a video image containing an image of an animal.
3. The method of claim 1 , wherein classifying a posture of the animal includes classifying a posture of the animal as one of a set of predetermined postures.
4. The method of claim 3 , wherein the set of predetermined postures includes horizontal side view posture, vertical posture, cuddled posture, horizontal front/back view posture, partially reared posture, stretched posture, hang vertical posture, hang cuddled posture, eating posture, and drinking posture.
5. The method of claim 1 , wherein the at least one body segment is a head.
6. The method of claim 1 , wherein the at least one body segment is a forelimb.
7. The method of claim 1 , wherein the at least one body segment is an abdomen.
8. The method of claim 1 , wherein the at least one body segment is a hind limb.
9. The method of claim 1 , wherein the at least one body segment is a tail.
10. The method of claim 1 , wherein the at least one body segment is a lower back.
11. The method of claim 1 , wherein the at least one body segment is an upper back.
12. The method of claim 1 , wherein the at least one body segment is an ear.
13. A method for characterizing animal behavior, comprising:
segregating images of an animal from video images;
identifying a posture of the animal in each of a series of images of the animal;
identifying each of a set of body segments of the animal in each of the series of images of the animal; and
characterizing behavior of the animal as one of a set of predetermined behaviors using the postures and the sets of body segments of the animal over the series of images of the animal.
14. The method of claim 13 , wherein segregating images of an animal from video images includes subtracting a background image from a video image containing an image of an animal.
15. The method of claim 13 , wherein classifying a posture of the animal includes classifying a posture of the animal as one of a set of predetermined postures.
16. The method of claim 15 , wherein the set of predetermined postures includes horizontal side view posture, vertical posture, cuddled posture, horizontal front/back view posture, partially reared posture, stretched posture, hang vertical posture, hang cuddled posture, eating posture, and drinking posture.
17. The method of claim 13 , wherein the set of body segments includes a head, a forelimb, an abdomen, a hind limb, a tail, a lower back, an upper back, and an ear.
18. A computer-readable medium including instructions for performing:
segregating images of an animal from video images;
identifying a posture of the animal in each of a series of images of the animal;
identifying each of a set of body segments of the animal in each of the series of images of the animal; and
characterizing behavior of the animal as one of a set of predetermined behaviors using the postures and the sets of body segments of the animal over the series of images of the animal.
19. The computer-readable medium of claim 18 , wherein segregating images of an animal from video images includes subtracting a background image from a video image containing an image of an animal.
20. The computer-readable medium of claim 18 , wherein classifying a posture of the animal includes classifying a posture of the animal as one of a set of predetermined postures.
21. The computer-readable medium of claim 20 , wherein the set of predetermined postures includes horizontal side view posture, vertical posture, cuddled posture, horizontal front/back view posture, partially reared posture, stretched posture, hang vertical posture, hang cuddled posture, eating posture, and drinking posture.
22. The computer-readable medium of claim 18 , wherein the set of body segments includes a head, a forelimb, an abdomen, a hind limb, a tail, a lower back, an upper back, and an ear.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US12/354,727 US20090285452A1 (en) | 2000-11-24 | 2009-01-15 | Unified system and method for animal behavior characterization in home cages using video analysis |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US09/718,374 US6678413B1 (en) | 2000-11-24 | 2000-11-24 | System and method for object identification and behavior characterization using video analysis |
US10/698,044 US7209588B2 (en) | 2000-11-24 | 2003-10-30 | Unified system and method for animal behavior characterization in home cages using video analysis |
US11/697,544 US20070175406A1 (en) | 2000-11-24 | 2007-04-06 | Unified system and method for animal behavior characterization in home cages using video analysis |
US12/354,727 US20090285452A1 (en) | 2000-11-24 | 2009-01-15 | Unified system and method for animal behavior characterization in home cages using video analysis |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/697,544 Continuation US20070175406A1 (en) | 2000-11-24 | 2007-04-06 | Unified system and method for animal behavior characterization in home cages using video analysis |
Publications (1)
Publication Number | Publication Date |
---|---|
US20090285452A1 true US20090285452A1 (en) | 2009-11-19 |
Family
ID=24885865
Family Applications (9)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/718,374 Expired - Lifetime US6678413B1 (en) | 2000-11-24 | 2000-11-24 | System and method for object identification and behavior characterization using video analysis |
US10/666,741 Expired - Lifetime US7068842B2 (en) | 2000-11-24 | 2003-09-19 | System and method for object identification and behavior characterization using video analysis |
US10/698,008 Abandoned US20040141635A1 (en) | 2000-11-24 | 2003-10-30 | Unified system and method for animal behavior characterization from top view using video analysis |
US10/698,044 Expired - Lifetime US7209588B2 (en) | 2000-11-24 | 2003-10-30 | Unified system and method for animal behavior characterization in home cages using video analysis |
US10/955,661 Active 2027-05-18 US8514236B2 (en) | 2000-11-24 | 2004-09-30 | System and method for animal gait characterization from bottom view using video analysis |
US11/697,544 Abandoned US20070175406A1 (en) | 2000-11-24 | 2007-04-06 | Unified system and method for animal behavior characterization in home cages using video analysis |
US12/354,727 Abandoned US20090285452A1 (en) | 2000-11-24 | 2009-01-15 | Unified system and method for animal behavior characterization in home cages using video analysis |
US12/365,149 Expired - Fee Related US7817824B2 (en) | 2000-11-24 | 2009-02-03 | Unified system and method for animal behavior characterization from top view using video analysis |
US12/881,154 Abandoned US20110007946A1 (en) | 2000-11-24 | 2010-09-13 | Unified system and method for animal behavior characterization with training capabilities |
Family Applications Before (6)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/718,374 Expired - Lifetime US6678413B1 (en) | 2000-11-24 | 2000-11-24 | System and method for object identification and behavior characterization using video analysis |
US10/666,741 Expired - Lifetime US7068842B2 (en) | 2000-11-24 | 2003-09-19 | System and method for object identification and behavior characterization using video analysis |
US10/698,008 Abandoned US20040141635A1 (en) | 2000-11-24 | 2003-10-30 | Unified system and method for animal behavior characterization from top view using video analysis |
US10/698,044 Expired - Lifetime US7209588B2 (en) | 2000-11-24 | 2003-10-30 | Unified system and method for animal behavior characterization in home cages using video analysis |
US10/955,661 Active 2027-05-18 US8514236B2 (en) | 2000-11-24 | 2004-09-30 | System and method for animal gait characterization from bottom view using video analysis |
US11/697,544 Abandoned US20070175406A1 (en) | 2000-11-24 | 2007-04-06 | Unified system and method for animal behavior characterization in home cages using video analysis |
Family Applications After (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US12/365,149 Expired - Fee Related US7817824B2 (en) | 2000-11-24 | 2009-02-03 | Unified system and method for animal behavior characterization from top view using video analysis |
US12/881,154 Abandoned US20110007946A1 (en) | 2000-11-24 | 2010-09-13 | Unified system and method for animal behavior characterization with training capabilities |
Country Status (5)
Country | Link |
---|---|
US (9) | US6678413B1 (en) |
EP (1) | EP1337962B9 (en) |
JP (1) | JP2004514975A (en) |
AU (1) | AU2002239272A1 (en) |
WO (1) | WO2002043352A2 (en) |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20100106743A1 (en) * | 2001-05-15 | 2010-04-29 | Psychogenics, Inc. | Method for Predicting Treatment Classes Using Behavior Informatics |
US20110007946A1 (en) * | 2000-11-24 | 2011-01-13 | Clever Sys, Inc. | Unified system and method for animal behavior characterization with training capabilities |
US20120140042A1 (en) * | 2007-01-12 | 2012-06-07 | International Business Machines Corporation | Warning a user about adverse behaviors of others within an environment based on a 3d captured image stream |
CN103325124A (en) * | 2012-03-21 | 2013-09-25 | 东北大学 | Target detecting and tracking system and method using background differencing method based on FPGA |
CN108846326A (en) * | 2018-05-23 | 2018-11-20 | 盐城工学院 | The recognition methods of pig posture, device and electronic equipment |
US11205088B2 (en) * | 2018-05-18 | 2021-12-21 | Hangzhou Hikvision Digital Technology Co., Ltd. | Method and apparatus for calculating a luminance value of a region of interest |
Families Citing this family (564)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7076102B2 (en) * | 2001-09-27 | 2006-07-11 | Koninklijke Philips Electronics N.V. | Video monitoring system employing hierarchical hidden markov model (HMM) event learning and classification |
US7484172B2 (en) * | 1997-05-23 | 2009-01-27 | Walker Digital, Llc | System and method for providing a customized index with hyper-footnotes |
GB2352076B (en) * | 1999-07-15 | 2003-12-17 | Mitsubishi Electric Inf Tech | Method and apparatus for representing and searching for an object in an image |
US20050146605A1 (en) * | 2000-10-24 | 2005-07-07 | Lipton Alan J. | Video surveillance system employing video primitives |
US7643655B2 (en) * | 2000-11-24 | 2010-01-05 | Clever Sys, Inc. | System and method for animal seizure detection and classification using video analysis |
GB2371936A (en) * | 2001-02-03 | 2002-08-07 | Hewlett Packard Co | Surveillance system for tracking a moving object |
US6778705B2 (en) * | 2001-02-27 | 2004-08-17 | Koninklijke Philips Electronics N.V. | Classification of objects through model ensembles |
JP3926572B2 (en) * | 2001-03-02 | 2007-06-06 | 株式会社日立製作所 | Image monitoring method, image monitoring apparatus, and storage medium |
US20090231436A1 (en) * | 2001-04-19 | 2009-09-17 | Faltesek Anthony E | Method and apparatus for tracking with identification |
US6909745B1 (en) | 2001-06-05 | 2005-06-21 | At&T Corp. | Content adaptive video encoder |
US6810086B1 (en) | 2001-06-05 | 2004-10-26 | At&T Corp. | System and method of filtering noise |
US6968006B1 (en) | 2001-06-05 | 2005-11-22 | At&T Corp. | Method of content adaptive video decoding |
US7773670B1 (en) | 2001-06-05 | 2010-08-10 | At+T Intellectual Property Ii, L.P. | Method of content adaptive video encoding |
US6970513B1 (en) * | 2001-06-05 | 2005-11-29 | At&T Corp. | System for content adaptive video decoding |
US7540011B2 (en) | 2001-06-11 | 2009-05-26 | Arrowsight, Inc. | Caching graphical interface for displaying video and ancillary data from a saved video |
JP4596221B2 (en) * | 2001-06-26 | 2010-12-08 | ソニー株式会社 | Image processing apparatus and method, recording medium, and program |
US20030004913A1 (en) * | 2001-07-02 | 2003-01-02 | Koninklijke Philips Electronics N.V. | Vision-based method and apparatus for detecting an event requiring assistance or documentation |
JP2003087771A (en) * | 2001-09-07 | 2003-03-20 | Oki Electric Ind Co Ltd | Monitoring system and monitoring method |
US20030058111A1 (en) * | 2001-09-27 | 2003-03-27 | Koninklijke Philips Electronics N.V. | Computer vision based elderly care monitoring system |
US7369680B2 (en) * | 2001-09-27 | 2008-05-06 | Koninklijke Phhilips Electronics N.V. | Method and apparatus for detecting an event based on patterns of behavior |
US7202791B2 (en) * | 2001-09-27 | 2007-04-10 | Koninklijke Philips N.V. | Method and apparatus for modeling behavior using a probability distrubution function |
US7110569B2 (en) * | 2001-09-27 | 2006-09-19 | Koninklijke Philips Electronics N.V. | Video based detection of fall-down and other events |
US7043075B2 (en) * | 2001-09-27 | 2006-05-09 | Koninklijke Philips Electronics N.V. | Computer vision system and method employing hierarchical object classification scheme |
US7432940B2 (en) * | 2001-10-12 | 2008-10-07 | Canon Kabushiki Kaisha | Interactive animation of sprites in a video production |
JP2003125401A (en) * | 2001-10-17 | 2003-04-25 | Mitsubishi Electric Corp | Video data reproducing method |
US6944421B2 (en) * | 2001-11-15 | 2005-09-13 | T.F.H. Publications, Inc. | Method and apparatus for providing training information regarding a pet |
US20030105880A1 (en) * | 2001-12-04 | 2003-06-05 | Koninklijke Philips Electronics N.V. | Distributed processing, storage, and transmision of multimedia information |
US20030115215A1 (en) * | 2001-12-18 | 2003-06-19 | Daniel Swarovski | Method and system for watching and tracking birds |
US7552030B2 (en) * | 2002-01-22 | 2009-06-23 | Honeywell International Inc. | System and method for learning patterns of behavior and operating a monitoring and response system based thereon |
EP1472869A4 (en) * | 2002-02-06 | 2008-07-30 | Nice Systems Ltd | System and method for video content analysis-based detection, surveillance and alarm management |
JP2004021495A (en) * | 2002-06-14 | 2004-01-22 | Mitsubishi Electric Corp | Monitoring system and monitoring method |
CA2492416A1 (en) * | 2002-07-15 | 2004-01-22 | Baylor College Of Medicine | Computer user interface facilitating acquiring and analyzing of biological specimen traits |
US7773112B2 (en) * | 2002-08-20 | 2010-08-10 | Tektronix, Inc. | Automatic measurement of video parameters |
US7200266B2 (en) * | 2002-08-27 | 2007-04-03 | Princeton University | Method and apparatus for automated video activity analysis |
JP2004096557A (en) * | 2002-09-02 | 2004-03-25 | Canon Inc | Image processor and image processing method |
DE60202374T2 (en) * | 2002-10-25 | 2005-12-08 | Evotec Technologies Gmbh | Method and apparatus for taking three-dimensional images of suspended micro-objects using high-resolution microscopy |
US7375731B2 (en) * | 2002-11-01 | 2008-05-20 | Mitsubishi Electric Research Laboratories, Inc. | Video mining using unsupervised clustering of video content |
ES2242484B1 (en) * | 2003-01-24 | 2007-01-01 | Pedro Monagas Asensio | ANIMAL ANALYZING DEVICE FOR MAMMALS. |
US20050134685A1 (en) * | 2003-12-22 | 2005-06-23 | Objectvideo, Inc. | Master-slave automated video-based surveillance system |
US7956889B2 (en) * | 2003-06-04 | 2011-06-07 | Model Software Corporation | Video surveillance system |
US7606417B2 (en) | 2004-08-16 | 2009-10-20 | Fotonation Vision Limited | Foreground/background segmentation in digital images with differential exposure calculations |
US7680342B2 (en) * | 2004-08-16 | 2010-03-16 | Fotonation Vision Limited | Indoor/outdoor classification in digital images |
US8238696B2 (en) * | 2003-08-21 | 2012-08-07 | Microsoft Corporation | Systems and methods for the implementation of a digital images schema for organizing units of information manageable by a hardware/software interface system |
US7106502B1 (en) * | 2003-08-21 | 2006-09-12 | The United States Of America As Represented By The Administrator Of National Aeronautics And Space Administration | Operation of a Cartesian robotic system in a compact microscope imaging system with intelligent controls |
US8166101B2 (en) | 2003-08-21 | 2012-04-24 | Microsoft Corporation | Systems and methods for the implementation of a synchronization schemas for units of information manageable by a hardware/software interface system |
US7590643B2 (en) * | 2003-08-21 | 2009-09-15 | Microsoft Corporation | Systems and methods for extensions and inheritance for units of information manageable by a hardware/software interface system |
US7835572B2 (en) * | 2003-09-30 | 2010-11-16 | Sharp Laboratories Of America, Inc. | Red eye reduction technique |
US7280673B2 (en) * | 2003-10-10 | 2007-10-09 | Intellivid Corporation | System and method for searching for changes in surveillance video |
US7333633B2 (en) * | 2003-10-31 | 2008-02-19 | Plexon, Inc. | Inter-frame video techniques for behavioral analysis of laboratory animals |
US20050104958A1 (en) * | 2003-11-13 | 2005-05-19 | Geoffrey Egnal | Active camera video-based surveillance systems and methods |
KR100601933B1 (en) * | 2003-11-18 | 2006-07-14 | 삼성전자주식회사 | Method and apparatus of human detection and privacy protection method and system employing the same |
US20050163345A1 (en) * | 2003-12-03 | 2005-07-28 | Safehouse International Limited | Analysing image data |
US7664292B2 (en) * | 2003-12-03 | 2010-02-16 | Safehouse International, Inc. | Monitoring an output from a camera |
US9311540B2 (en) | 2003-12-12 | 2016-04-12 | Careview Communications, Inc. | System and method for predicting patient falls |
US8675059B2 (en) | 2010-07-29 | 2014-03-18 | Careview Communications, Inc. | System and method for using a video monitoring system to prevent and manage decubitus ulcers in patients |
US7447333B1 (en) * | 2004-01-22 | 2008-11-04 | Siemens Corporate Research, Inc. | Video and audio monitoring for syndromic surveillance for infectious diseases |
JP4479267B2 (en) * | 2004-02-18 | 2010-06-09 | 株式会社日立製作所 | Surveillance camera video distribution system |
US7486815B2 (en) * | 2004-02-20 | 2009-02-03 | Microsoft Corporation | Method and apparatus for scene learning and three-dimensional tracking using stereo video cameras |
US7831094B2 (en) * | 2004-04-27 | 2010-11-09 | Honda Motor Co., Ltd. | Simultaneous localization and mapping using multiple view feature descriptors |
US7263472B2 (en) * | 2004-06-28 | 2007-08-28 | Mitsubishi Electric Research Laboratories, Inc. | Hidden markov model based object tracking and similarity metrics |
US20060007307A1 (en) * | 2004-07-12 | 2006-01-12 | Chao-Hung Chang | Partial image saving system and method |
US7562299B2 (en) * | 2004-08-13 | 2009-07-14 | Pelco, Inc. | Method and apparatus for searching recorded video |
JP4433948B2 (en) * | 2004-09-02 | 2010-03-17 | 株式会社セガ | Background image acquisition program, video game apparatus, background image acquisition method, and computer-readable recording medium recording the program |
US20080166015A1 (en) * | 2004-09-24 | 2008-07-10 | Object Video, Inc. | Method for finding paths in video |
US7469060B2 (en) * | 2004-11-12 | 2008-12-23 | Honeywell International Inc. | Infrared face detection and recognition system |
RU2323475C2 (en) * | 2004-11-12 | 2008-04-27 | Общество с ограниченной ответственностью "Центр Нейросетевых Технологий - Интеллектуальные Системы Безопасности" (ООО "ИСС") | Method (variants) and device (variants) for automated detection of intentional or incidental disruptions of technological procedure by operator |
US7602942B2 (en) * | 2004-11-12 | 2009-10-13 | Honeywell International Inc. | Infrared and visible fusion face recognition system |
US20060245500A1 (en) * | 2004-12-15 | 2006-11-02 | David Yonovitz | Tunable wavelet target extraction preprocessor system |
EP1834486A1 (en) * | 2004-12-24 | 2007-09-19 | Ultrawaves design holding B. V. | Intelligent distributed image processing |
US9325781B2 (en) | 2005-01-31 | 2016-04-26 | Invention Science Fund I, Llc | Audio sharing |
US7876357B2 (en) * | 2005-01-31 | 2011-01-25 | The Invention Science Fund I, Llc | Estimating shared image device operational capabilities or resources |
US8902320B2 (en) * | 2005-01-31 | 2014-12-02 | The Invention Science Fund I, Llc | Shared image device synchronization or designation |
US20060285150A1 (en) * | 2005-01-31 | 2006-12-21 | Searete Llc, A Limited Liability Corporation Of The State Of Delaware | Regional proximity for shared image device(s) |
US20060221197A1 (en) * | 2005-03-30 | 2006-10-05 | Jung Edward K | Image transformation estimator of an imaging device |
US20060170956A1 (en) | 2005-01-31 | 2006-08-03 | Jung Edward K | Shared image devices |
US9910341B2 (en) * | 2005-01-31 | 2018-03-06 | The Invention Science Fund I, Llc | Shared image device designation |
US9082456B2 (en) | 2005-01-31 | 2015-07-14 | The Invention Science Fund I Llc | Shared image device designation |
US20060187230A1 (en) * | 2005-01-31 | 2006-08-24 | Searete Llc | Peripheral shared image device sharing |
US20060173972A1 (en) * | 2005-01-31 | 2006-08-03 | Searete Llc, A Limited Liability Corporation Of The State Of Delaware | Audio sharing |
US7920169B2 (en) * | 2005-01-31 | 2011-04-05 | Invention Science Fund I, Llc | Proximity of shared image devices |
US8606383B2 (en) | 2005-01-31 | 2013-12-10 | The Invention Science Fund I, Llc | Audio sharing |
US9124729B2 (en) * | 2005-01-31 | 2015-09-01 | The Invention Science Fund I, Llc | Shared image device synchronization or designation |
US20070236505A1 (en) * | 2005-01-31 | 2007-10-11 | Searete Llc, A Limited Liability Corporation Of The State Of Delaware | Resampling of transformed shared image techniques |
US20060174203A1 (en) * | 2005-01-31 | 2006-08-03 | Searete Llc, A Limited Liability Corporation Of The State Of Delaware | Viewfinder for shared image device |
US20060187227A1 (en) * | 2005-01-31 | 2006-08-24 | Jung Edward K | Storage aspects for imaging device |
US20060171603A1 (en) * | 2005-01-31 | 2006-08-03 | Searete Llc, A Limited Liability Corporation Of The State Of Delaware | Resampling of transformed shared image techniques |
US20060187228A1 (en) * | 2005-01-31 | 2006-08-24 | Searete Llc, A Limited Liability Corporation Of The State Of Delaware | Sharing including peripheral shared image device |
US9489717B2 (en) | 2005-01-31 | 2016-11-08 | Invention Science Fund I, Llc | Shared image device |
US7903141B1 (en) * | 2005-02-15 | 2011-03-08 | Videomining Corporation | Method and system for event detection by multi-scale image invariant analysis |
US7710452B1 (en) | 2005-03-16 | 2010-05-04 | Eric Lindberg | Remote video monitoring of non-urban outdoor sites |
US7286056B2 (en) * | 2005-03-22 | 2007-10-23 | Lawrence Kates | System and method for pest detection |
US8139896B1 (en) * | 2005-03-28 | 2012-03-20 | Grandeye, Ltd. | Tracking moving objects accurately on a wide-angle video |
US7801328B2 (en) * | 2005-03-31 | 2010-09-21 | Honeywell International Inc. | Methods for defining, detecting, analyzing, indexing and retrieving events using video image processing |
US7760908B2 (en) * | 2005-03-31 | 2010-07-20 | Honeywell International Inc. | Event packaged video sequence |
US8704668B1 (en) * | 2005-04-20 | 2014-04-22 | Trevor Darrell | System for monitoring and alerting based on animal behavior in designated environments |
US20070098348A1 (en) * | 2005-10-31 | 2007-05-03 | Searete Llc, A Limited Liability Corporation Of The State Of Delaware | Degradation/preservation management of captured data |
US8681225B2 (en) * | 2005-06-02 | 2014-03-25 | Royce A. Levien | Storage access technique for captured data |
US8233042B2 (en) * | 2005-10-31 | 2012-07-31 | The Invention Science Fund I, Llc | Preservation and/or degradation of a video/audio data stream |
US9167195B2 (en) * | 2005-10-31 | 2015-10-20 | Invention Science Fund I, Llc | Preservation/degradation of video/audio aspects of a data stream |
US8253821B2 (en) * | 2005-10-31 | 2012-08-28 | The Invention Science Fund I, Llc | Degradation/preservation management of captured data |
US8072501B2 (en) * | 2005-10-31 | 2011-12-06 | The Invention Science Fund I, Llc | Preservation and/or degradation of a video/audio data stream |
US9942511B2 (en) | 2005-10-31 | 2018-04-10 | Invention Science Fund I, Llc | Preservation/degradation of video/audio aspects of a data stream |
US9621749B2 (en) * | 2005-06-02 | 2017-04-11 | Invention Science Fund I, Llc | Capturing selected image objects |
US9076208B2 (en) * | 2006-02-28 | 2015-07-07 | The Invention Science Fund I, Llc | Imagery processing |
US20070109411A1 (en) * | 2005-06-02 | 2007-05-17 | Searete Llc, A Limited Liability Corporation Of The State Of Delaware | Composite image selectivity |
US8964054B2 (en) * | 2006-08-18 | 2015-02-24 | The Invention Science Fund I, Llc | Capturing selected image objects |
US9967424B2 (en) * | 2005-06-02 | 2018-05-08 | Invention Science Fund I, Llc | Data storage usage protocol |
US9451200B2 (en) * | 2005-06-02 | 2016-09-20 | Invention Science Fund I, Llc | Storage access technique for captured data |
US7782365B2 (en) * | 2005-06-02 | 2010-08-24 | Searete Llc | Enhanced video/still image correlation |
US10003762B2 (en) | 2005-04-26 | 2018-06-19 | Invention Science Fund I, Llc | Shared image devices |
US20090144391A1 (en) * | 2007-11-30 | 2009-06-04 | Searete Llc, A Limited Liability Corporation Of The State Of Delaware | Audio sharing |
US9093121B2 (en) | 2006-02-28 | 2015-07-28 | The Invention Science Fund I, Llc | Data management of an audio data stream |
US7872675B2 (en) * | 2005-06-02 | 2011-01-18 | The Invention Science Fund I, Llc | Saved-image management |
US20070008326A1 (en) * | 2005-06-02 | 2007-01-11 | Searete Llc, A Limited Liability Corporation Of The State Of Delaware | Dual mode image capture technique |
US9819490B2 (en) * | 2005-05-04 | 2017-11-14 | Invention Science Fund I, Llc | Regional proximity for shared image device(s) |
US9191611B2 (en) * | 2005-06-02 | 2015-11-17 | Invention Science Fund I, Llc | Conditional alteration of a saved image |
US20070222865A1 (en) | 2006-03-15 | 2007-09-27 | Searete Llc, A Limited Liability Corporation Of The State Of Delaware | Enhanced video/still image correlation |
US9001215B2 (en) * | 2005-06-02 | 2015-04-07 | The Invention Science Fund I, Llc | Estimating shared image device operational capabilities or resources |
US20070139529A1 (en) * | 2005-06-02 | 2007-06-21 | Searete Llc, A Limited Liability Corporation Of The State Of Delaware | Dual mode image capture technique |
US20060260624A1 (en) * | 2005-05-17 | 2006-11-23 | Battelle Memorial Institute | Method, program, and system for automatic profiling of entities |
US20060274153A1 (en) * | 2005-06-02 | 2006-12-07 | Searete Llc, A Limited Liability Corporation Of The State Of Delaware | Third party storage of captured data |
US7720257B2 (en) * | 2005-06-16 | 2010-05-18 | Honeywell International Inc. | Object tracking system |
US20060291697A1 (en) * | 2005-06-21 | 2006-12-28 | Trw Automotive U.S. Llc | Method and apparatus for detecting the presence of an occupant within a vehicle |
US7944468B2 (en) * | 2005-07-05 | 2011-05-17 | Northrop Grumman Systems Corporation | Automated asymmetric threat detection using backward tracking and behavioral analysis |
CN101228555A (en) * | 2005-07-07 | 2008-07-23 | 独创目标实验室公司 | System for 3D monitoring and analysis of motion behavior of targets |
US20070031038A1 (en) * | 2005-08-03 | 2007-02-08 | Honeywell International Inc. | Boolean complement methods and systems for video image processing a region of interest |
US7545954B2 (en) * | 2005-08-22 | 2009-06-09 | General Electric Company | System for recognizing events |
US20070047834A1 (en) * | 2005-08-31 | 2007-03-01 | International Business Machines Corporation | Method and apparatus for visual background subtraction with one or more preprocessing modules |
US20070058717A1 (en) * | 2005-09-09 | 2007-03-15 | Objectvideo, Inc. | Enhanced processing for scanning video |
US20070071404A1 (en) * | 2005-09-29 | 2007-03-29 | Honeywell International Inc. | Controlled video event presentation |
US7382280B2 (en) * | 2005-10-17 | 2008-06-03 | Cleverdevices, Inc. | Parking violation recording system and method |
US7806604B2 (en) * | 2005-10-20 | 2010-10-05 | Honeywell International Inc. | Face detection and tracking in a wide field of view |
US20070120980A1 (en) * | 2005-10-31 | 2007-05-31 | Searete Llc, A Limited Liability Corporation Of The State Of Delaware | Preservation/degradation of video/audio aspects of a data stream |
US20070203595A1 (en) * | 2006-02-28 | 2007-08-30 | Searete Llc, A Limited Liability Corporation | Data management of an audio data stream |
US8228382B2 (en) * | 2005-11-05 | 2012-07-24 | Ram Pattikonda | System and method for counting people |
US20100310120A1 (en) * | 2005-11-05 | 2010-12-09 | Charlie Keith | Method and system for tracking moving objects in a scene |
US7558404B2 (en) * | 2005-11-28 | 2009-07-07 | Honeywell International Inc. | Detection of abnormal crowd behavior |
US7692696B2 (en) * | 2005-12-27 | 2010-04-06 | Fotonation Vision Limited | Digital image acquisition system with portrait mode |
US8265392B2 (en) * | 2006-02-07 | 2012-09-11 | Qualcomm Incorporated | Inter-mode region-of-interest video object segmentation |
US7986827B2 (en) * | 2006-02-07 | 2011-07-26 | Siemens Medical Solutions Usa, Inc. | System and method for multiple instance learning for computer aided detection |
US7822227B2 (en) * | 2006-02-07 | 2010-10-26 | International Business Machines Corporation | Method and system for tracking images |
US8150155B2 (en) | 2006-02-07 | 2012-04-03 | Qualcomm Incorporated | Multi-mode region-of-interest video object segmentation |
US8265349B2 (en) * | 2006-02-07 | 2012-09-11 | Qualcomm Incorporated | Intra-mode region-of-interest video object segmentation |
EP1987436B1 (en) * | 2006-02-14 | 2015-12-09 | FotoNation Limited | Image blurring |
IES20060559A2 (en) * | 2006-02-14 | 2006-11-01 | Fotonation Vision Ltd | Automatic detection and correction of non-red flash eye defects |
JP4607797B2 (en) * | 2006-03-06 | 2011-01-05 | 株式会社東芝 | Behavior discrimination device, method and program |
JP4970531B2 (en) * | 2006-03-28 | 2012-07-11 | ザ・ユニバーシティ・コート・オブ・ザ・ユニバーシティ・オブ・エディンバラ | A method for automatically characterizing the behavior of one or more objects. |
JP4589261B2 (en) * | 2006-03-31 | 2010-12-01 | パナソニック株式会社 | Surveillance camera device |
IES20060564A2 (en) * | 2006-05-03 | 2006-11-01 | Fotonation Vision Ltd | Improved foreground / background separation |
EP2023812B1 (en) | 2006-05-19 | 2016-01-27 | The Queen's Medical Center | Motion tracking system for real time adaptive imaging and spectroscopy |
CN101449293A (en) * | 2006-05-31 | 2009-06-03 | 汤姆森许可贸易公司 | Multi-track of video objects |
US7983448B1 (en) * | 2006-06-02 | 2011-07-19 | University Of Central Florida Research Foundation, Inc. | Self correcting tracking of moving objects in video |
US20080123959A1 (en) * | 2006-06-26 | 2008-05-29 | Ratner Edward R | Computer-implemented method for automated object recognition and classification in scenes using segment-based object extraction |
KR100716708B1 (en) * | 2006-07-11 | 2007-05-09 | 영남대학교 산학협력단 | Automatic velocity control running machine using pressure sensor array and fuzzy-logic |
US7930204B1 (en) | 2006-07-25 | 2011-04-19 | Videomining Corporation | Method and system for narrowcasting based on automatic analysis of customer behavior in a retail store |
US7974869B1 (en) | 2006-09-20 | 2011-07-05 | Videomining Corporation | Method and system for automatically measuring and forecasting the behavioral characterization of customers to help customize programming contents in a media network |
US20080112593A1 (en) * | 2006-11-03 | 2008-05-15 | Ratner Edward R | Automated method and apparatus for robust image object recognition and/or classification using multiple temporal views |
US7734623B2 (en) * | 2006-11-07 | 2010-06-08 | Cycorp, Inc. | Semantics-based method and apparatus for document analysis |
US8396250B2 (en) * | 2006-11-20 | 2013-03-12 | Adelaide Research & Innovation Pty Ltd | Network surveillance system |
US8165405B2 (en) * | 2006-12-18 | 2012-04-24 | Honda Motor Co., Ltd. | Leveraging temporal, contextual and ordering constraints for recognizing complex activities in video |
US8189926B2 (en) * | 2006-12-30 | 2012-05-29 | Videomining Corporation | Method and system for automatically analyzing categories in a physical space based on the visual characterization of people |
US8665333B1 (en) * | 2007-01-30 | 2014-03-04 | Videomining Corporation | Method and system for optimizing the observation and annotation of complex human behavior from video sources |
US20080184245A1 (en) * | 2007-01-30 | 2008-07-31 | March Networks Corporation | Method and system for task-based video analytics processing |
US9172918B2 (en) * | 2007-02-02 | 2015-10-27 | Honeywell International Inc. | Systems and methods for managing live video data |
WO2008098188A2 (en) * | 2007-02-08 | 2008-08-14 | Behavioral Recognition Systems, Inc. | Behavioral recognition system |
NZ553146A (en) * | 2007-02-09 | 2011-05-27 | Say Systems Ltd | Improvements relating to monitoring and displaying activities |
US7667596B2 (en) * | 2007-02-16 | 2010-02-23 | Panasonic Corporation | Method and system for scoring surveillance system footage |
US8456528B2 (en) * | 2007-03-20 | 2013-06-04 | International Business Machines Corporation | System and method for managing the interaction of object detection and tracking systems in video surveillance |
US7957565B1 (en) * | 2007-04-05 | 2011-06-07 | Videomining Corporation | Method and system for recognizing employees in a physical space based on automatic behavior analysis |
EP1988488A1 (en) * | 2007-05-03 | 2008-11-05 | Sony Deutschland Gmbh | Method for detecting moving objects in a blind spot region of a vehicle and blind spot detection device |
US20080310677A1 (en) * | 2007-06-18 | 2008-12-18 | Weismuller Thomas P | Object detection system and method incorporating background clutter removal |
US8103109B2 (en) * | 2007-06-19 | 2012-01-24 | Microsoft Corporation | Recognizing hand poses and/or object classes |
WO2009006605A2 (en) * | 2007-07-03 | 2009-01-08 | Pivotal Vision, Llc | Motion-validating remote monitoring system |
US8189905B2 (en) | 2007-07-11 | 2012-05-29 | Behavioral Recognition Systems, Inc. | Cognitive model for a machine-learning engine in a video analysis system |
US8005262B2 (en) * | 2007-07-16 | 2011-08-23 | Hugh Griffin | System and method for video object identification |
KR101375665B1 (en) * | 2007-08-08 | 2014-03-18 | 삼성전자주식회사 | Method and apparatus for estimating a background change, and method and apparatus for detecting a motion |
US20090062002A1 (en) * | 2007-08-30 | 2009-03-05 | Bay Tek Games, Inc. | Apparatus And Method of Detecting And Tracking Objects In Amusement Games |
US8009864B2 (en) | 2007-08-31 | 2011-08-30 | Accenture Global Services Limited | Determination of inventory conditions based on image processing |
US8189855B2 (en) | 2007-08-31 | 2012-05-29 | Accenture Global Services Limited | Planogram extraction based on image processing |
US8630924B2 (en) * | 2007-08-31 | 2014-01-14 | Accenture Global Services Limited | Detection of stock out conditions based on image processing |
US9135491B2 (en) | 2007-08-31 | 2015-09-15 | Accenture Global Services Limited | Digital point-of-sale analyzer |
US7949568B2 (en) * | 2007-08-31 | 2011-05-24 | Accenture Global Services Limited | Determination of product display parameters based on image processing |
EP2188778A1 (en) * | 2007-09-12 | 2010-05-26 | Digisensory Technologies Pty Ltd. | Smart network camera system-on-a-chip |
US8200011B2 (en) | 2007-09-27 | 2012-06-12 | Behavioral Recognition Systems, Inc. | Context processor for video analysis system |
US8300924B2 (en) * | 2007-09-27 | 2012-10-30 | Behavioral Recognition Systems, Inc. | Tracker component for behavioral recognition system |
US8175333B2 (en) * | 2007-09-27 | 2012-05-08 | Behavioral Recognition Systems, Inc. | Estimator identifier component for behavioral recognition system |
US8218811B2 (en) | 2007-09-28 | 2012-07-10 | Uti Limited Partnership | Method and system for video interaction based on motion swarms |
CN101420595B (en) * | 2007-10-23 | 2012-11-21 | 华为技术有限公司 | Method and equipment for describing and capturing video object |
JP5055092B2 (en) * | 2007-11-02 | 2012-10-24 | 株式会社日立国際電気 | Video processing apparatus and video processing method |
US9171454B2 (en) * | 2007-11-14 | 2015-10-27 | Microsoft Technology Licensing, Llc | Magic wand |
US20090137933A1 (en) * | 2007-11-28 | 2009-05-28 | Ishoe | Methods and systems for sensing equilibrium |
KR100936115B1 (en) * | 2007-12-20 | 2010-01-11 | 김세호 | Mouse Activity Measuring Instrument |
US8337404B2 (en) | 2010-10-01 | 2012-12-25 | Flint Hills Scientific, Llc | Detecting, quantifying, and/or classifying seizures using multimodal data |
US8382667B2 (en) | 2010-10-01 | 2013-02-26 | Flint Hills Scientific, Llc | Detecting, quantifying, and/or classifying seizures using multimodal data |
US8571643B2 (en) | 2010-09-16 | 2013-10-29 | Flint Hills Scientific, Llc | Detecting or validating a detection of a state change from a template of heart rate derivative shape or heart beat wave complex |
US8098888B1 (en) * | 2008-01-28 | 2012-01-17 | Videomining Corporation | Method and system for automatic analysis of the trip of people in a retail space using multiple cameras |
US20090195382A1 (en) * | 2008-01-31 | 2009-08-06 | Sensormatic Electronics Corporation | Video sensor and alarm system and method with object and event classification |
AU2008200926B2 (en) * | 2008-02-28 | 2011-09-29 | Canon Kabushiki Kaisha | On-camera summarisation of object relationships |
US8284249B2 (en) | 2008-03-25 | 2012-10-09 | International Business Machines Corporation | Real time processing of video frames for triggering an alert |
JP5213237B2 (en) * | 2008-04-17 | 2013-06-19 | パナソニック株式会社 | Imaging position determination method and imaging position determination apparatus |
US9579047B2 (en) | 2013-03-15 | 2017-02-28 | Careview Communications, Inc. | Systems and methods for dynamically identifying a patient support surface and patient monitoring |
US10645346B2 (en) | 2013-01-18 | 2020-05-05 | Careview Communications, Inc. | Patient video monitoring systems and methods having detection algorithm recovery from changes in illumination |
US9959471B2 (en) | 2008-05-06 | 2018-05-01 | Careview Communications, Inc. | Patient video monitoring systems and methods for thermal detection of liquids |
US9866797B2 (en) | 2012-09-28 | 2018-01-09 | Careview Communications, Inc. | System and method for monitoring a fall state of a patient while minimizing false alarms |
US9794523B2 (en) | 2011-12-19 | 2017-10-17 | Careview Communications, Inc. | Electronic patient sitter management system and method for implementing |
US8952894B2 (en) * | 2008-05-12 | 2015-02-10 | Microsoft Technology Licensing, Llc | Computer vision-based multi-touch sensing using infrared lasers |
US8009863B1 (en) | 2008-06-30 | 2011-08-30 | Videomining Corporation | Method and system for analyzing shopping behavior using multiple sensor tracking |
EP2230629A3 (en) | 2008-07-16 | 2012-11-21 | Verint Systems Inc. | A system and method for capturing, storing, analyzing and displaying data relating to the movements of objects |
US8396247B2 (en) * | 2008-07-31 | 2013-03-12 | Microsoft Corporation | Recognizing actions of animate objects in video |
US20100031202A1 (en) * | 2008-08-04 | 2010-02-04 | Microsoft Corporation | User-defined gesture set for surface computing |
US8847739B2 (en) | 2008-08-04 | 2014-09-30 | Microsoft Corporation | Fusing RFID and vision for surface object tracking |
US7710830B2 (en) * | 2008-09-02 | 2010-05-04 | Accuwalk Llc | Outing record device |
US8121968B2 (en) * | 2008-09-11 | 2012-02-21 | Behavioral Recognition Systems, Inc. | Long-term memory in a video analysis system |
US8126833B2 (en) * | 2008-09-11 | 2012-02-28 | Behavioral Recognition Systems, Inc. | Detecting anomalous events using a long-term memory in a video analysis system |
US9633275B2 (en) | 2008-09-11 | 2017-04-25 | Wesley Kenneth Cobb | Pixel-level based micro-feature extraction |
WO2010032247A2 (en) * | 2008-09-17 | 2010-03-25 | Ramot At Tel-Aviv University Ltd. | System and method for analyzing exploratory behavior |
JPWO2010035752A1 (en) * | 2008-09-24 | 2012-02-23 | 株式会社ニコン | Image generation apparatus, imaging apparatus, image reproduction apparatus, and image reproduction program |
US9141862B2 (en) * | 2008-09-26 | 2015-09-22 | Harris Corporation | Unattended surveillance device and associated methods |
US8634635B2 (en) * | 2008-10-30 | 2014-01-21 | Clever Sys, Inc. | System and method for stereo-view multiple animal behavior characterization |
US8694443B2 (en) | 2008-11-03 | 2014-04-08 | International Business Machines Corporation | System and method for automatically distinguishing between customers and in-store employees |
DE102008058020A1 (en) * | 2008-11-19 | 2010-05-20 | Zebris Medical Gmbh | Arrangement for training the gear |
US8471899B2 (en) | 2008-12-02 | 2013-06-25 | Careview Communications, Inc. | System and method for documenting patient procedures |
US9373055B2 (en) * | 2008-12-16 | 2016-06-21 | Behavioral Recognition Systems, Inc. | Hierarchical sudden illumination change detection using radiance consistency within a spatial neighborhood |
US8218877B2 (en) * | 2008-12-23 | 2012-07-10 | National Chiao Tung University | Tracking vehicle method by using image processing |
US20100157051A1 (en) * | 2008-12-23 | 2010-06-24 | International Business Machines Corporation | System and method for detecting and deterring rfid tag related fraud |
CA2748333A1 (en) * | 2008-12-24 | 2010-07-01 | Vehicle Monitoring Systems Pty Ltd | Method and system for detecting vehicle offences |
US20100169169A1 (en) * | 2008-12-31 | 2010-07-01 | International Business Machines Corporation | System and method for using transaction statistics to facilitate checkout variance investigation |
US20100182445A1 (en) * | 2009-01-22 | 2010-07-22 | Upi Semiconductor Corporation | Processing Device, Method, And Electronic System Utilizing The Same |
US8295546B2 (en) * | 2009-01-30 | 2012-10-23 | Microsoft Corporation | Pose tracking pipeline |
US8285046B2 (en) * | 2009-02-18 | 2012-10-09 | Behavioral Recognition Systems, Inc. | Adaptive update of background pixel thresholds using sudden illumination change detection |
US8878931B2 (en) | 2009-03-04 | 2014-11-04 | Honeywell International Inc. | Systems and methods for managing video data |
WO2010105105A2 (en) * | 2009-03-11 | 2010-09-16 | Crowley Davis Research, Inc. | Discrimination between multi-dimensional models using difference distributions |
US8416296B2 (en) * | 2009-04-14 | 2013-04-09 | Behavioral Recognition Systems, Inc. | Mapper component for multiple art networks in a video analysis system |
CN102438521A (en) * | 2009-04-24 | 2012-05-02 | 原子能和辅助替代能源委员会 | System and method for determining the posture of a person |
US9047742B2 (en) * | 2009-05-07 | 2015-06-02 | International Business Machines Corporation | Visual security for point of sale terminals |
US8608481B2 (en) * | 2009-05-13 | 2013-12-17 | Medtronic Navigation, Inc. | Method and apparatus for identifying an instrument location based on measuring a characteristic |
US9417700B2 (en) | 2009-05-21 | 2016-08-16 | Edge3 Technologies | Gesture recognition systems and related methods |
US20100299140A1 (en) * | 2009-05-22 | 2010-11-25 | Cycorp, Inc. | Identifying and routing of documents of potential interest to subscribers using interest determination rules |
US9740977B1 (en) * | 2009-05-29 | 2017-08-22 | Videomining Corporation | Method and system for recognizing the intentions of shoppers in retail aisles based on their trajectories |
US8320619B2 (en) | 2009-05-29 | 2012-11-27 | Microsoft Corporation | Systems and methods for tracking a model |
US8649594B1 (en) | 2009-06-04 | 2014-02-11 | Agilence, Inc. | Active and adaptive intelligent video surveillance system |
US20100315506A1 (en) * | 2009-06-10 | 2010-12-16 | Microsoft Corporation | Action detection in video through sub-volume mutual information maximization |
US8571259B2 (en) * | 2009-06-17 | 2013-10-29 | Robert Allan Margolis | System and method for automatic identification of wildlife |
US8462987B2 (en) * | 2009-06-23 | 2013-06-11 | Ut-Battelle, Llc | Detecting multiple moving objects in crowded environments with coherent motion regions |
US11004093B1 (en) | 2009-06-29 | 2021-05-11 | Videomining Corporation | Method and system for detecting shopping groups based on trajectory dynamics |
US20100332140A1 (en) * | 2009-06-30 | 2010-12-30 | Jonathan Livingston Joyce | Method of assessing the eating experience of a companion animal |
TWI386239B (en) * | 2009-07-24 | 2013-02-21 | Univ Far East | Animal experiment gait recording method |
JP5350928B2 (en) * | 2009-07-30 | 2013-11-27 | オリンパスイメージング株式会社 | Camera and camera control method |
US8493409B2 (en) * | 2009-08-18 | 2013-07-23 | Behavioral Recognition Systems, Inc. | Visualizing and updating sequences and segments in a video surveillance system |
US8340352B2 (en) * | 2009-08-18 | 2012-12-25 | Behavioral Recognition Systems, Inc. | Inter-trajectory anomaly detection using adaptive voting experts in a video surveillance system |
US8625884B2 (en) * | 2009-08-18 | 2014-01-07 | Behavioral Recognition Systems, Inc. | Visualizing and updating learned event maps in surveillance systems |
US8280153B2 (en) * | 2009-08-18 | 2012-10-02 | Behavioral Recognition Systems | Visualizing and updating learned trajectories in video surveillance systems |
US20110043689A1 (en) * | 2009-08-18 | 2011-02-24 | Wesley Kenneth Cobb | Field-of-view change detection |
US8379085B2 (en) * | 2009-08-18 | 2013-02-19 | Behavioral Recognition Systems, Inc. | Intra-trajectory anomaly detection using adaptive voting experts in a video surveillance system |
US9805271B2 (en) | 2009-08-18 | 2017-10-31 | Omni Ai, Inc. | Scene preset identification using quadtree decomposition analysis |
US8295591B2 (en) * | 2009-08-18 | 2012-10-23 | Behavioral Recognition Systems, Inc. | Adaptive voting experts for incremental segmentation of sequences with prediction in a video surveillance system |
US8358834B2 (en) * | 2009-08-18 | 2013-01-22 | Behavioral Recognition Systems | Background model for complex and dynamic scenes |
US8270733B2 (en) * | 2009-08-31 | 2012-09-18 | Behavioral Recognition Systems, Inc. | Identifying anomalous object types during classification |
US8285060B2 (en) * | 2009-08-31 | 2012-10-09 | Behavioral Recognition Systems, Inc. | Detecting anomalous trajectories in a video surveillance system |
US8797405B2 (en) * | 2009-08-31 | 2014-08-05 | Behavioral Recognition Systems, Inc. | Visualizing and updating classifications in a video surveillance system |
US8167430B2 (en) * | 2009-08-31 | 2012-05-01 | Behavioral Recognition Systems, Inc. | Unsupervised learning of temporal anomalies for a video surveillance system |
US8270732B2 (en) * | 2009-08-31 | 2012-09-18 | Behavioral Recognition Systems, Inc. | Clustering nodes in a self-organizing map using an adaptive resonance theory network |
US8786702B2 (en) | 2009-08-31 | 2014-07-22 | Behavioral Recognition Systems, Inc. | Visualizing and updating long-term memory percepts in a video surveillance system |
US8218819B2 (en) * | 2009-09-01 | 2012-07-10 | Behavioral Recognition Systems, Inc. | Foreground object detection in a video surveillance system |
US8218818B2 (en) * | 2009-09-01 | 2012-07-10 | Behavioral Recognition Systems, Inc. | Foreground object tracking |
US8170283B2 (en) * | 2009-09-17 | 2012-05-01 | Behavioral Recognition Systems Inc. | Video surveillance system configured to analyze complex behaviors using alternating layers of clustering and sequencing |
US8180105B2 (en) * | 2009-09-17 | 2012-05-15 | Behavioral Recognition Systems, Inc. | Classifier anomalies for observed behaviors in a video surveillance system |
FR2950989B1 (en) * | 2009-10-05 | 2011-10-28 | Alcatel Lucent | DEVICE FOR INTERACTING WITH AN INCREASED OBJECT. |
US8320621B2 (en) * | 2009-12-21 | 2012-11-27 | Microsoft Corporation | Depth projector system with integrated VCSEL array |
JP2011210139A (en) * | 2010-03-30 | 2011-10-20 | Sony Corp | Image processing apparatus and method, and program |
US20110246123A1 (en) * | 2010-03-30 | 2011-10-06 | Welch Allyn, Inc. | Personal status monitoring |
US8649871B2 (en) | 2010-04-29 | 2014-02-11 | Cyberonics, Inc. | Validity test adaptive constraint modification for cardiac data used for detection of state changes |
US8562536B2 (en) | 2010-04-29 | 2013-10-22 | Flint Hills Scientific, Llc | Algorithm for detecting a seizure from cardiac data |
US8831732B2 (en) | 2010-04-29 | 2014-09-09 | Cyberonics, Inc. | Method, apparatus and system for validating and quantifying cardiac beat data quality |
US8396252B2 (en) | 2010-05-20 | 2013-03-12 | Edge 3 Technologies | Systems and related methods for three dimensional gesture recognition in vehicles |
EP2395456A1 (en) * | 2010-06-12 | 2011-12-14 | Toyota Motor Europe NV/SA | Methods and systems for semantic label propagation |
US8670029B2 (en) | 2010-06-16 | 2014-03-11 | Microsoft Corporation | Depth camera illuminator with superluminescent light-emitting diode |
US8483481B2 (en) | 2010-07-27 | 2013-07-09 | International Business Machines Corporation | Foreground analysis based on tracking information |
US8641646B2 (en) | 2010-07-30 | 2014-02-04 | Cyberonics, Inc. | Seizure detection using coordinate data |
WO2012020856A1 (en) * | 2010-08-10 | 2012-02-16 | Lg Electronics Inc. | Region of interest based video synopsis |
US8582866B2 (en) | 2011-02-10 | 2013-11-12 | Edge 3 Technologies, Inc. | Method and apparatus for disparity computation in stereo images |
US8666144B2 (en) | 2010-09-02 | 2014-03-04 | Edge 3 Technologies, Inc. | Method and apparatus for determining disparity of texture |
US8655093B2 (en) | 2010-09-02 | 2014-02-18 | Edge 3 Technologies, Inc. | Method and apparatus for performing segmentation of an image |
WO2012030872A1 (en) | 2010-09-02 | 2012-03-08 | Edge3 Technologies Inc. | Method and apparatus for confusion learning |
WO2012040554A2 (en) | 2010-09-23 | 2012-03-29 | Stryker Corporation | Video monitoring system |
US8684921B2 (en) | 2010-10-01 | 2014-04-01 | Flint Hills Scientific Llc | Detecting, assessing and managing epilepsy using a multi-variate, metric-based classification analysis |
US20120094600A1 (en) | 2010-10-19 | 2012-04-19 | Welch Allyn, Inc. | Platform for patient monitoring |
KR20120052739A (en) * | 2010-11-16 | 2012-05-24 | 삼성전자주식회사 | Display driving device and method for compressing and decompressing image data in the same |
US9432639B2 (en) * | 2010-11-19 | 2016-08-30 | Honeywell International Inc. | Security video detection of personal distress and gesture commands |
JP5718632B2 (en) * | 2010-12-22 | 2015-05-13 | 綜合警備保障株式会社 | Part recognition device, part recognition method, and part recognition program |
AU2010257454B2 (en) * | 2010-12-24 | 2014-03-06 | Canon Kabushiki Kaisha | Summary view of video objects sharing common attributes |
GB2486913A (en) * | 2010-12-30 | 2012-07-04 | Delaval Holding Ab | Control and monitoring system for an animal installation |
US11080513B2 (en) * | 2011-01-12 | 2021-08-03 | Gary S. Shuster | Video and still image data alteration to enhance privacy |
US8970589B2 (en) | 2011-02-10 | 2015-03-03 | Edge 3 Technologies, Inc. | Near-touch interaction with a stereo camera grid structured tessellations |
US10025388B2 (en) * | 2011-02-10 | 2018-07-17 | Continental Automotive Systems, Inc. | Touchless human machine interface |
US9504390B2 (en) | 2011-03-04 | 2016-11-29 | Globalfoundries Inc. | Detecting, assessing and managing a risk of death in epilepsy |
US9498162B2 (en) | 2011-04-25 | 2016-11-22 | Cyberonics, Inc. | Identifying seizures using heart data from two or more windows |
US9402550B2 (en) | 2011-04-29 | 2016-08-02 | Cybertronics, Inc. | Dynamic heart rate threshold for neurological event detection |
DE102011075412A1 (en) * | 2011-05-06 | 2012-11-08 | Deutsche Telekom Ag | A method and system for detecting a static scene, determining raw events, and detecting free areas in an observation area |
TWI454150B (en) * | 2011-05-06 | 2014-09-21 | Altek Corp | Processing method for image file |
US8810640B2 (en) | 2011-05-16 | 2014-08-19 | Ut-Battelle, Llc | Intrinsic feature-based pose measurement for imaging motion compensation |
DE102011101939A1 (en) * | 2011-05-18 | 2012-11-22 | Biobserve GmbH | A method of creating a behavioral analysis of a rodent in an arena and method of creating an image of the rodent |
CN102221996A (en) * | 2011-05-20 | 2011-10-19 | 广州市久邦数码科技有限公司 | Implementation method for performing interaction between dynamic wallpaper and desktop component |
BR112013030404A2 (en) * | 2011-05-30 | 2016-12-13 | Koninkl Philips Nv | method for detecting body posture, equipment for detecting body postures on a bed, and using a method |
US9594430B2 (en) | 2011-06-01 | 2017-03-14 | Microsoft Technology Licensing, Llc | Three-dimensional foreground selection for vision system |
US8526734B2 (en) | 2011-06-01 | 2013-09-03 | Microsoft Corporation | Three-dimensional background removal for vision system |
CN102831442A (en) * | 2011-06-13 | 2012-12-19 | 索尼公司 | Abnormal behavior detection method and equipment and method and equipment for generating abnormal behavior detection equipment |
RU2455676C2 (en) * | 2011-07-04 | 2012-07-10 | Общество с ограниченной ответственностью "ТРИДИВИ" | Method of controlling device using gestures and 3d sensor for realising said method |
EP2739134A4 (en) * | 2011-08-03 | 2015-12-09 | Yeda Res & Dev | Systems and methods of monitoring social interactions in a group of organisms over a period of at least 24 hours in a semi-natural environment |
US9606209B2 (en) | 2011-08-26 | 2017-03-28 | Kineticor, Inc. | Methods, systems, and devices for intra-scan motion correction |
WO2013035902A1 (en) * | 2011-09-08 | 2013-03-14 | 엘지전자 주식회사 | Health care system based on video in remote health care solution and method for providing health care service |
TWI590193B (en) * | 2011-09-29 | 2017-07-01 | 國立清華大學 | Image method for classifying insects and insect classifying process |
US9124783B2 (en) | 2011-09-30 | 2015-09-01 | Camiolog, Inc. | Method and system for automated labeling at scale of motion-detected events in video surveillance |
US9549677B2 (en) | 2011-10-14 | 2017-01-24 | Flint Hills Scientific, L.L.C. | Seizure detection methods, apparatus, and systems using a wavelet transform maximum modulus algorithm |
US8442265B1 (en) * | 2011-10-19 | 2013-05-14 | Facebook Inc. | Image selection from captured video sequence based on social components |
US8437500B1 (en) * | 2011-10-19 | 2013-05-07 | Facebook Inc. | Preferred images from captured video sequence |
US9177208B2 (en) * | 2011-11-04 | 2015-11-03 | Google Inc. | Determining feature vectors for video volumes |
US9672609B1 (en) | 2011-11-11 | 2017-06-06 | Edge 3 Technologies, Inc. | Method and apparatus for improved depth-map estimation |
US20130273969A1 (en) * | 2011-12-01 | 2013-10-17 | Finding Rover, Inc. | Mobile app that generates a dog sound to capture data for a lost pet identifying system |
US9342735B2 (en) * | 2011-12-01 | 2016-05-17 | Finding Rover, Inc. | Facial recognition lost pet identifying system |
US9235895B2 (en) * | 2011-12-13 | 2016-01-12 | Hitachi, Ltd. | Method for estimating direction of person standing still |
US20150030252A1 (en) * | 2011-12-16 | 2015-01-29 | The Research Foundation For The State University Of New York | Methods of recognizing activity in video |
US8811938B2 (en) | 2011-12-16 | 2014-08-19 | Microsoft Corporation | Providing a user interface experience based on inferred vehicle state |
AU2012355375A1 (en) * | 2011-12-19 | 2014-07-10 | Birds In The Hand, Llc | Method and system for sharing object information |
KR20130076374A (en) * | 2011-12-28 | 2013-07-08 | 삼성전자주식회사 | Method for measuring quantity of exercise and display apparatus thereof |
US8948449B2 (en) * | 2012-02-06 | 2015-02-03 | GM Global Technology Operations LLC | Selecting visible regions in nighttime images for performing clear path detection |
US10076109B2 (en) | 2012-02-14 | 2018-09-18 | Noble Research Institute, Llc | Systems and methods for trapping animals |
IN2014DN08342A (en) | 2012-03-15 | 2015-05-08 | Behavioral Recognition Sys Inc | |
US20130279571A1 (en) * | 2012-04-18 | 2013-10-24 | Vixs Systems, Inc. | Video processing system with stream indexing data and methods for use therewith |
US10448839B2 (en) | 2012-04-23 | 2019-10-22 | Livanova Usa, Inc. | Methods, systems and apparatuses for detecting increased risk of sudden death |
US9681836B2 (en) | 2012-04-23 | 2017-06-20 | Cyberonics, Inc. | Methods, systems and apparatuses for detecting seizure and non-seizure states |
TWI484941B (en) * | 2012-05-10 | 2015-05-21 | Animal gait detection system and method | |
WO2013170129A1 (en) | 2012-05-10 | 2013-11-14 | President And Fellows Of Harvard College | A system and method for automatically discovering, characterizing, classifying and semi-automatically labeling animal behavior and quantitative phenotyping of behaviors in animals |
RU2531876C2 (en) * | 2012-05-15 | 2014-10-27 | Общество с ограниченной ответственностью "Синезис" | Indexing method of video data by means of card |
US9113143B2 (en) | 2012-06-29 | 2015-08-18 | Behavioral Recognition Systems, Inc. | Detecting and responding to an out-of-focus camera in a video analytics system |
US9911043B2 (en) | 2012-06-29 | 2018-03-06 | Omni Ai, Inc. | Anomalous object interaction detection and reporting |
US9723271B2 (en) | 2012-06-29 | 2017-08-01 | Omni Ai, Inc. | Anomalous stationary object detection and reporting |
EP2867860A4 (en) | 2012-06-29 | 2016-07-27 | Behavioral Recognition Sys Inc | Unsupervised learning of feature anomalies for a video surveillance system |
US9317908B2 (en) | 2012-06-29 | 2016-04-19 | Behavioral Recognition System, Inc. | Automatic gain control filter in a video analysis system |
US9111353B2 (en) | 2012-06-29 | 2015-08-18 | Behavioral Recognition Systems, Inc. | Adaptive illuminance filter in a video analysis system |
US9104918B2 (en) | 2012-08-20 | 2015-08-11 | Behavioral Recognition Systems, Inc. | Method and system for detecting sea-surface oil |
WO2014037890A2 (en) * | 2012-09-06 | 2014-03-13 | Cascube Ltd. | Position and behavioral tracking system and uses thereof |
US20140122488A1 (en) * | 2012-10-29 | 2014-05-01 | Elwha Llc | Food Supply Chain Automation Farm Testing System And Method |
US9704122B2 (en) | 2012-10-29 | 2017-07-11 | Elwha Llc | Food supply chain automation farm tracking system and method |
WO2014051262A1 (en) * | 2012-09-25 | 2014-04-03 | 에스케이텔레콤 주식회사 | Method for setting event rules and event monitoring apparatus using same |
US10380542B2 (en) * | 2012-10-01 | 2019-08-13 | Stephan Hammelbacher | Method and device for process tracking of operations or services relating to at least one object |
US10860683B2 (en) * | 2012-10-25 | 2020-12-08 | The Research Foundation For The State University Of New York | Pattern change discovery between high dimensional data sets |
WO2014075022A1 (en) | 2012-11-12 | 2014-05-15 | Behavioral Recognition Systems, Inc. | Image stabilization techniques for video surveillance systems |
DE102012111452B3 (en) * | 2012-11-27 | 2014-03-20 | Karlsruher Institut für Technologie | Optical arrangement for recording e.g. behaviors of non-human biological object in biology field, has planar matrix forming pattern, and optical filter passing wavelength of light beam, where wavelength lying within internal is weakened |
CN102970519A (en) * | 2012-11-29 | 2013-03-13 | 河海大学常州校区 | Non-rigid target behavior observation device and method based on visual perception network |
WO2014083433A2 (en) * | 2012-12-02 | 2014-06-05 | Agricam Ab | Systems and methods for predicting the outcome of a state of a subject |
US10220211B2 (en) | 2013-01-22 | 2019-03-05 | Livanova Usa, Inc. | Methods and systems to diagnose depression |
US9717461B2 (en) | 2013-01-24 | 2017-08-01 | Kineticor, Inc. | Systems, devices, and methods for tracking and compensating for patient motion during a medical imaging scan |
US9305365B2 (en) | 2013-01-24 | 2016-04-05 | Kineticor, Inc. | Systems, devices, and methods for tracking moving targets |
US10327708B2 (en) | 2013-01-24 | 2019-06-25 | Kineticor, Inc. | Systems, devices, and methods for tracking and compensating for patient motion during a medical imaging scan |
CN105392423B (en) | 2013-02-01 | 2018-08-17 | 凯内蒂科尔股份有限公司 | The motion tracking system of real-time adaptive motion compensation in biomedical imaging |
KR101993010B1 (en) * | 2013-02-28 | 2019-06-25 | 고려대학교 산학협력단 | Method and apparatus for analyzing video based on spatiotemporal patterns |
KR102361152B1 (en) | 2013-03-01 | 2022-02-09 | 클레버펫 엘엘씨 | Animal interaction device, system, and method |
EP2777490B1 (en) * | 2013-03-11 | 2021-12-08 | Biomedical International R + D GmbH | Non-invasive temperature and physical activity measurement of animals |
JP6057786B2 (en) * | 2013-03-13 | 2017-01-11 | ヤフー株式会社 | Time-series data analysis device, time-series data analysis method, and program |
US10721448B2 (en) | 2013-03-15 | 2020-07-21 | Edge 3 Technologies, Inc. | Method and apparatus for adaptive exposure bracketing, segmentation and scene organization |
KR102203884B1 (en) * | 2013-04-12 | 2021-01-15 | 삼성전자주식회사 | Imaging apparatus and controlling method thereof |
US10346680B2 (en) * | 2013-04-12 | 2019-07-09 | Samsung Electronics Co., Ltd. | Imaging apparatus and control method for determining a posture of an object |
WO2014210601A1 (en) * | 2013-06-28 | 2014-12-31 | Office Of Technology Transfer, National Institutes Of Health | Systems and methods of video monitoring for vivarium cages |
US20150022329A1 (en) * | 2013-07-16 | 2015-01-22 | Forget You Not, LLC | Assisted Animal Communication |
JP6220452B2 (en) * | 2013-07-16 | 2017-10-25 | ピンタレスト,インコーポレイテッド | Object-based context menu control |
WO2015021485A1 (en) | 2013-08-09 | 2015-02-12 | Behavioral Recognition Systems, Inc. | A cognitive neuro-linguistic behavior recognition system for multi-sensor data fusion |
US9117144B2 (en) | 2013-08-14 | 2015-08-25 | Qualcomm Incorporated | Performing vocabulary-based visual search using multi-resolution feature descriptors |
CN103488148B (en) * | 2013-09-24 | 2016-03-09 | 华北电力大学(保定) | A kind of animal behavior intelligent monitor system based on Internet of Things and computer vision |
US9355306B2 (en) * | 2013-09-27 | 2016-05-31 | Konica Minolta Laboratory U.S.A., Inc. | Method and system for recognition of abnormal behavior |
WO2015066460A2 (en) * | 2013-11-01 | 2015-05-07 | Children's Medical Center Corporation | Devices and methods for analyzing rodent behavior |
CN105745598B (en) * | 2013-11-27 | 2019-10-01 | 惠普发展公司,有限责任合伙企业 | Determine the shape of the expression of object |
US9230159B1 (en) * | 2013-12-09 | 2016-01-05 | Google Inc. | Action recognition and detection on videos |
US9495601B2 (en) | 2013-12-09 | 2016-11-15 | Mirsani, LLC | Detecting and reporting improper activity involving a vehicle |
US9396256B2 (en) * | 2013-12-13 | 2016-07-19 | International Business Machines Corporation | Pattern based audio searching method and system |
CN103676886A (en) * | 2013-12-17 | 2014-03-26 | 山东大学 | Standardized henhouse environment and breeding information monitoring and managing system |
JP6411373B2 (en) * | 2013-12-17 | 2018-10-24 | シャープ株式会社 | Recognition data transmission device, recognition data recording device, and recognition data recording method |
AU2013273778A1 (en) * | 2013-12-20 | 2015-07-09 | Canon Kabushiki Kaisha | Text line fragments for text line analysis |
US10986223B1 (en) | 2013-12-23 | 2021-04-20 | Massachusetts Mutual Life Insurance | Systems and methods for presenting content based on user behavior |
US10178222B1 (en) | 2016-03-22 | 2019-01-08 | Massachusetts Mutual Life Insurance Company | Systems and methods for presenting content based on user behavior |
SG10201501052XA (en) * | 2014-02-11 | 2015-09-29 | Agency Science Tech & Res | Method And System For Monitoring Activity Of An Animal |
DE102014203749A1 (en) * | 2014-02-28 | 2015-09-17 | Robert Bosch Gmbh | Method and device for monitoring at least one interior of a building and assistance system for at least one interior of a building |
CA3010117C (en) | 2014-03-10 | 2020-04-21 | Gojo Industries, Inc. | Hygiene tracking compliance |
CN106572810A (en) | 2014-03-24 | 2017-04-19 | 凯内蒂科尔股份有限公司 | Systems, methods, and devices for removing prospective motion correction from medical imaging scans |
US9237743B2 (en) | 2014-04-18 | 2016-01-19 | The Samuel Roberts Noble Foundation, Inc. | Systems and methods for trapping animals |
CN104079872A (en) * | 2014-05-16 | 2014-10-01 | 大连理工大学 | Video image processing and human-computer interaction method based on content |
US10277280B2 (en) | 2014-05-29 | 2019-04-30 | Sony Interactive Entertainment LLC | Configuration of data and power transfer in near field communications |
US9577463B2 (en) | 2014-05-29 | 2017-02-21 | Sony Corporation | Portable device to portable device wireless power transfer methods and systems |
US9843360B2 (en) | 2014-08-14 | 2017-12-12 | Sony Corporation | Method and system for use in configuring multiple near field antenna systems |
US10965159B2 (en) | 2014-05-29 | 2021-03-30 | Sony Corporation | Scalable antenna system |
TWI562636B (en) * | 2014-06-16 | 2016-12-11 | Altek Semiconductor Corp | Image capture apparatus and image compensating method thereof |
US10846536B2 (en) * | 2014-06-27 | 2020-11-24 | Nec Corporation | Abnormality detection device and abnormality detection method |
US9900748B2 (en) | 2014-07-16 | 2018-02-20 | Sony Corporation | Consumer electronics (CE) device and related method for providing stadium services |
US10127601B2 (en) | 2014-07-16 | 2018-11-13 | Sony Corporation | Mesh network applied to fixed establishment with movable items therein |
US9906897B2 (en) | 2014-07-16 | 2018-02-27 | Sony Corporation | Applying mesh network to pet carriers |
US9361802B2 (en) | 2014-07-16 | 2016-06-07 | Sony Corporation | Vehicle ad hoc network (VANET) |
US9426610B2 (en) | 2014-07-16 | 2016-08-23 | Sony Corporation | Applying mesh network to luggage |
US9516461B2 (en) | 2014-07-16 | 2016-12-06 | Sony Corporation | Mesh network applied to arena events |
EP3188660A4 (en) | 2014-07-23 | 2018-05-16 | Kineticor, Inc. | Systems, devices, and methods for tracking and compensating for patient motion during a medical imaging scan |
ES2874556T3 (en) * | 2014-08-29 | 2021-11-05 | Csb Sys Ag | Device and procedure for evaluating animal welfare in a slaughter animal |
US10117583B2 (en) | 2014-10-22 | 2018-11-06 | illumiSonics, Inc. | Photoacoustic remote sensing (PARS) |
US9449230B2 (en) | 2014-11-26 | 2016-09-20 | Zepp Labs, Inc. | Fast object tracking framework for sports video recognition |
CN104469299A (en) * | 2014-12-02 | 2015-03-25 | 柳州市瑞蚨电子科技有限公司 | Network camera shooting device |
CN104361724B (en) * | 2014-12-03 | 2017-01-18 | 京东方科技集团股份有限公司 | Device and method for detecting peeing of baby |
US10409910B2 (en) | 2014-12-12 | 2019-09-10 | Omni Ai, Inc. | Perceptual associative memory for a neuro-linguistic behavior recognition system |
US10409909B2 (en) | 2014-12-12 | 2019-09-10 | Omni Ai, Inc. | Lexical analyzer for a neuro-linguistic behavior recognition system |
US9305216B1 (en) | 2014-12-15 | 2016-04-05 | Amazon Technologies, Inc. | Context-based detection and classification of actions |
US9710712B2 (en) | 2015-01-16 | 2017-07-18 | Avigilon Fortress Corporation | System and method for detecting, tracking, and classifiying objects |
US10380486B2 (en) * | 2015-01-20 | 2019-08-13 | International Business Machines Corporation | Classifying entities by behavior |
WO2016168869A1 (en) * | 2015-04-16 | 2016-10-20 | California Institute Of Technology | Systems and methods for behavior detection using 3d tracking and machine learning |
US9619701B2 (en) | 2015-05-20 | 2017-04-11 | Xerox Corporation | Using motion tracking and image categorization for document indexing and validation |
NZ738150A (en) * | 2015-07-01 | 2019-06-28 | Viking Genetics Fmba | System and method for identification of individual animals based on images of the back |
US9943247B2 (en) | 2015-07-28 | 2018-04-17 | The University Of Hawai'i | Systems, devices, and methods for detecting false movements for motion correction during a medical imaging scan |
CA2999114A1 (en) | 2015-09-18 | 2017-03-23 | Children's Medical Center Corporation | Devices and methods for analyzing animal behavior |
EP3361935A4 (en) | 2015-10-14 | 2019-08-28 | President and Fellows of Harvard College | Automatically classifying animal behavior |
GB2544324A (en) * | 2015-11-13 | 2017-05-17 | Cathx Res Ltd | Method and system for processing image data |
US10716515B2 (en) | 2015-11-23 | 2020-07-21 | Kineticor, Inc. | Systems, devices, and methods for tracking and compensating for patient motion during a medical imaging scan |
KR101817583B1 (en) * | 2015-11-30 | 2018-01-12 | 한국생산기술연구원 | System and method for analyzing behavior pattern using depth image |
CN105574501B (en) * | 2015-12-15 | 2019-03-15 | 上海微桥电子科技有限公司 | A kind of stream of people's video detecting analysis system |
CN105512640B (en) * | 2015-12-30 | 2019-04-02 | 重庆邮电大学 | A kind of people flow rate statistical method based on video sequence |
US10327646B2 (en) | 2016-02-02 | 2019-06-25 | Illumisonics Inc. | Non-interferometric photoacoustic remote sensing (NI-PARS) |
US20170243354A1 (en) * | 2016-02-19 | 2017-08-24 | Xerox Corporation | Automatic frontal-view gait segmentation for abnormal gait quantification |
US11036219B2 (en) | 2016-02-22 | 2021-06-15 | Ketchup On, Inc. | Self-propelled device |
US9600717B1 (en) * | 2016-02-25 | 2017-03-21 | Zepp Labs, Inc. | Real-time single-view action recognition based on key pose analysis for sports videos |
CN105809711B (en) * | 2016-03-02 | 2019-03-12 | 华南农业大学 | A kind of pig movement big data extracting method and its system based on video frequency tracking |
US10750717B2 (en) * | 2016-03-04 | 2020-08-25 | Indiana University Research And Technology Corporation | Method and apparatus for spatial cognitive assessment of a lab animal |
CN105828030A (en) * | 2016-03-14 | 2016-08-03 | 珠海经济特区远宏科技有限公司 | Video investigation mobile terminal system |
EP3429471A4 (en) | 2016-03-18 | 2019-07-31 | President and Fellows of Harvard College | Automatically classifying animal behavior |
CN107221133B (en) * | 2016-03-22 | 2018-12-11 | 杭州海康威视数字技术股份有限公司 | A kind of area monitoring alarm system and alarm method |
US10306311B1 (en) | 2016-03-24 | 2019-05-28 | Massachusetts Mutual Life Insurance Company | Intelligent and context aware reading systems |
US10360254B1 (en) | 2016-03-24 | 2019-07-23 | Massachusetts Mutual Life Insurance Company | Intelligent and context aware reading systems |
CN105894536A (en) * | 2016-03-30 | 2016-08-24 | 中国农业大学 | Method and system for analyzing livestock behaviors on the basis of video tracking |
US9576205B1 (en) * | 2016-03-31 | 2017-02-21 | Pointgrab Ltd. | Method and system for determining location of an occupant |
CN108780576B (en) * | 2016-04-06 | 2022-02-25 | 赫尔实验室有限公司 | System and method for ghost removal in video segments using object bounding boxes |
US10241514B2 (en) | 2016-05-11 | 2019-03-26 | Brain Corporation | Systems and methods for initializing a robot to autonomously travel a trained route |
US10282849B2 (en) | 2016-06-17 | 2019-05-07 | Brain Corporation | Systems and methods for predictive/reconstructive visual object tracker |
US10016896B2 (en) * | 2016-06-30 | 2018-07-10 | Brain Corporation | Systems and methods for robotic behavior around moving bodies |
IL247101B (en) | 2016-08-03 | 2018-10-31 | Pointgrab Ltd | Method and system for detecting an occupant in an image |
CN106264569B (en) * | 2016-08-10 | 2020-03-06 | 深圳先进技术研究院 | Shared emotion nerve experiment system based on observational fear acquisition |
US10902249B2 (en) | 2016-10-31 | 2021-01-26 | Hewlett-Packard Development Company, L.P. | Video monitoring |
US10274325B2 (en) | 2016-11-01 | 2019-04-30 | Brain Corporation | Systems and methods for robotic mapping |
US10001780B2 (en) | 2016-11-02 | 2018-06-19 | Brain Corporation | Systems and methods for dynamic route planning in autonomous navigation |
CN106778537B (en) * | 2016-11-28 | 2021-02-02 | 中国科学院心理研究所 | Animal social network structure acquisition and analysis system and method based on image processing |
US10723018B2 (en) | 2016-11-28 | 2020-07-28 | Brain Corporation | Systems and methods for remote operating and/or monitoring of a robot |
US20180150697A1 (en) * | 2017-01-09 | 2018-05-31 | Seematics Systems Ltd | System and method for using subsequent behavior to facilitate learning of visual event detectors |
US10713792B1 (en) * | 2017-01-13 | 2020-07-14 | Amazon Technologies, Inc. | System and apparatus for image processing |
US10852730B2 (en) | 2017-02-08 | 2020-12-01 | Brain Corporation | Systems and methods for robotic mobile platforms |
US10310471B2 (en) * | 2017-02-28 | 2019-06-04 | Accenture Global Solutions Limited | Content recognition and communication system |
GB2560177A (en) | 2017-03-01 | 2018-09-05 | Thirdeye Labs Ltd | Training a computational neural network |
GB2560387B (en) | 2017-03-10 | 2022-03-09 | Standard Cognition Corp | Action identification using neural networks |
US10627338B2 (en) | 2017-03-23 | 2020-04-21 | Illumisonics Inc. | Camera-based photoacoustic remote sensing (C-PARS) |
JP6909960B2 (en) * | 2017-03-31 | 2021-07-28 | パナソニックIpマネジメント株式会社 | Detection device, detection method and detection program |
WO2018185718A1 (en) * | 2017-04-07 | 2018-10-11 | Smaluet Solutions Private Limited | A device and a method of learning a behavior of a pet in response to instructions provided to the pet |
US10034645B1 (en) * | 2017-04-13 | 2018-07-31 | The Board Of Trustees Of The Leland Stanford Junior University | Systems and methods for detecting complex networks in MRI image data |
US10373316B2 (en) * | 2017-04-20 | 2019-08-06 | Ford Global Technologies, Llc | Images background subtraction for dynamic lighting scenarios |
WO2018208319A1 (en) | 2017-05-12 | 2018-11-15 | Children's Medical Center Corporation | Devices and methods for analyzing animal behavior |
CN107292889B (en) * | 2017-06-14 | 2020-09-25 | 上海联影医疗科技有限公司 | Tumor segmentation method, system and readable medium |
US10157476B1 (en) * | 2017-06-15 | 2018-12-18 | Satori Worldwide, Llc | Self-learning spatial recognition system |
US10810414B2 (en) | 2017-07-06 | 2020-10-20 | Wisconsin Alumni Research Foundation | Movement monitoring system |
US11450148B2 (en) | 2017-07-06 | 2022-09-20 | Wisconsin Alumni Research Foundation | Movement monitoring system |
US10482613B2 (en) | 2017-07-06 | 2019-11-19 | Wisconsin Alumni Research Foundation | Movement monitoring system |
CA3071120A1 (en) * | 2017-07-31 | 2019-02-07 | Cubic Corporation | Automated scenario recognition and reporting using neural networks |
US10489654B1 (en) * | 2017-08-04 | 2019-11-26 | Amazon Technologies, Inc. | Video analysis method and system |
US10650545B2 (en) | 2017-08-07 | 2020-05-12 | Standard Cognition, Corp. | Systems and methods to check-in shoppers in a cashier-less store |
US10853965B2 (en) | 2017-08-07 | 2020-12-01 | Standard Cognition, Corp | Directional impression analysis using deep learning |
US10474988B2 (en) | 2017-08-07 | 2019-11-12 | Standard Cognition, Corp. | Predicting inventory events using foreground/background processing |
CA3072062A1 (en) * | 2017-08-07 | 2019-02-14 | Standard Cognition, Corp. | Predicting inventory events using foreground/background processing |
US11232687B2 (en) | 2017-08-07 | 2022-01-25 | Standard Cognition, Corp | Deep learning-based shopper statuses in a cashier-less store |
US11200692B2 (en) | 2017-08-07 | 2021-12-14 | Standard Cognition, Corp | Systems and methods to check-in shoppers in a cashier-less store |
US10445694B2 (en) | 2017-08-07 | 2019-10-15 | Standard Cognition, Corp. | Realtime inventory tracking using deep learning |
US11250376B2 (en) | 2017-08-07 | 2022-02-15 | Standard Cognition, Corp | Product correlation analysis using deep learning |
US10474991B2 (en) | 2017-08-07 | 2019-11-12 | Standard Cognition, Corp. | Deep learning-based store realograms |
US11023850B2 (en) | 2017-08-07 | 2021-06-01 | Standard Cognition, Corp. | Realtime inventory location management using deep learning |
CN109583452B (en) * | 2017-09-29 | 2021-02-19 | 大连恒锐科技股份有限公司 | Human identity identification method and system based on barefoot footprints |
CN107751011B (en) * | 2017-11-07 | 2020-12-04 | 山东天智信息科技有限公司 | Drinking water equipment based on drinking water adjunctie therapy |
US10410371B2 (en) | 2017-12-21 | 2019-09-10 | The Boeing Company | Cluttered background removal from imagery for object detection |
CN108133737B (en) * | 2017-12-26 | 2021-08-31 | 深圳先进技术研究院 | Rodent fear experiment video analysis method and device |
BR102018002876A2 (en) * | 2018-02-14 | 2019-09-10 | Guimaraes Hummig Ednilson | object locating platform |
CN108401177B (en) * | 2018-02-27 | 2021-04-27 | 上海哔哩哔哩科技有限公司 | Video playing method, server and video playing system |
CN112088530A (en) * | 2018-05-07 | 2020-12-15 | 苹果公司 | User interface for viewing live video feeds and recording videos |
US11132893B2 (en) * | 2018-05-11 | 2021-09-28 | Seagate Technology, Llc | Multi-sensor edge computing system |
CN108664942B (en) * | 2018-05-17 | 2021-10-22 | 西安理工大学 | Extraction method of mouse video multi-dimensional characteristic values and video classification method |
US11700836B2 (en) | 2018-05-21 | 2023-07-18 | Companion Labs, Inc. | System and method for characterizing and monitoring health of an animal based on gait and postural movements |
CN112469269B (en) * | 2018-05-21 | 2023-04-28 | 康帕宁实验室公司 | Method for autonomously training animals to respond to oral commands |
US11205508B2 (en) | 2018-05-23 | 2021-12-21 | Verb Surgical Inc. | Machine-learning-oriented surgical video analysis system |
US10905105B2 (en) * | 2018-06-19 | 2021-02-02 | Farm Jenny LLC | Farm asset tracking, monitoring, and alerts |
WO2020018469A1 (en) * | 2018-07-16 | 2020-01-23 | The Board Of Trustees Of The Leland Stanford Junior University | System and method for automatic evaluation of gait using single or multi-camera recordings |
US11048973B1 (en) | 2018-07-31 | 2021-06-29 | Objectvideo Labs, Llc | Action classification using aggregated background subtraction images |
US10810432B2 (en) * | 2018-08-02 | 2020-10-20 | Motorola Solutions, Inc. | Methods and systems for differentiating one or more objects in a video |
CN109272518B (en) * | 2018-08-17 | 2020-05-05 | 东南大学 | Morris water maze experiment image analysis system and method |
US10769799B2 (en) * | 2018-08-24 | 2020-09-08 | Ford Global Technologies, Llc | Foreground detection |
US10679743B2 (en) * | 2018-09-12 | 2020-06-09 | Verb Surgical Inc. | Method and system for automatically tracking and managing inventory of surgical tools in operating rooms |
WO2020072918A1 (en) * | 2018-10-05 | 2020-04-09 | The Trustees Of Princeton University | Automated system to measure multi-animal body part dynamics |
WO2020076225A1 (en) * | 2018-10-10 | 2020-04-16 | Delaval Holding Ab | Animal identification using vision techniques |
MX2021004399A (en) * | 2018-10-17 | 2021-07-06 | Group Ro Main Inc | Monitoring livestock. |
US11312594B2 (en) | 2018-11-09 | 2022-04-26 | Otis Elevator Company | Conveyance system video analytics |
US11093749B2 (en) | 2018-12-20 | 2021-08-17 | L'oreal | Analysis and feedback system for personal care routines |
CN109784208B (en) * | 2018-12-26 | 2023-04-18 | 武汉工程大学 | Image-based pet behavior detection method |
JP7297455B2 (en) * | 2019-01-31 | 2023-06-26 | キヤノン株式会社 | Image processing device, image processing method, and program |
CN111614703A (en) * | 2019-02-25 | 2020-09-01 | 南京爱体智能科技有限公司 | Method for combining Internet of things sensor with video analysis |
CN109831634A (en) * | 2019-02-28 | 2019-05-31 | 北京明略软件系统有限公司 | The density information of target object determines method and device |
US11103748B1 (en) | 2019-03-05 | 2021-08-31 | Physmodo, Inc. | System and method for human motion detection and tracking |
US11331006B2 (en) | 2019-03-05 | 2022-05-17 | Physmodo, Inc. | System and method for human motion detection and tracking |
CA3132432A1 (en) * | 2019-03-15 | 2020-09-24 | Parsin Haji Reza | Single source photoacoustic remote sensing (ss-pars) |
US11232575B2 (en) | 2019-04-18 | 2022-01-25 | Standard Cognition, Corp | Systems and methods for deep learning-based subject persistence |
KR102228350B1 (en) * | 2019-05-03 | 2021-03-16 | 주식회사 엘지유플러스 | Apparatus and method for monitering pet |
US11363071B2 (en) | 2019-05-31 | 2022-06-14 | Apple Inc. | User interfaces for managing a local network |
US10904029B2 (en) | 2019-05-31 | 2021-01-26 | Apple Inc. | User interfaces for managing controllable external devices |
US20200383299A1 (en) * | 2019-06-06 | 2020-12-10 | Edgar Josue Bermudez Contreras | Systems and methods of homecage monitoring |
CN110516535A (en) * | 2019-07-12 | 2019-11-29 | 杭州电子科技大学 | A kind of mouse liveness detection method and system and hygienic appraisal procedure based on deep learning |
CN110427865B (en) * | 2019-07-29 | 2023-08-25 | 三峡大学 | Human behavior video feature picture extraction and reconstruction method for high-voltage forbidden region |
CN110301364B (en) * | 2019-08-02 | 2021-09-24 | 中国人民解放军军事科学院军事医学研究院 | Experiment box for research on social behaviors of mice |
CN110456831B (en) * | 2019-08-16 | 2022-06-14 | 南开大学 | Mouse contact behavior tracking platform based on active vision |
CN110490161B (en) * | 2019-08-23 | 2022-01-07 | 安徽农业大学 | Captive animal behavior analysis method based on deep learning |
US11213015B2 (en) * | 2019-09-17 | 2022-01-04 | Eknauth Persaud | System and a method of lab animal observation |
US11321927B1 (en) * | 2019-09-23 | 2022-05-03 | Apple Inc. | Temporal segmentation |
AU2020104459A4 (en) * | 2019-10-14 | 2021-10-28 | TBIAS Pty Ltd | An automated behavioural monitoring unit |
TWI731442B (en) * | 2019-10-18 | 2021-06-21 | 宏碁股份有限公司 | Electronic apparatus and object information recognition method by using touch data thereof |
US11557151B2 (en) | 2019-10-24 | 2023-01-17 | Deere & Company | Object identification on a mobile work machine |
CN112764594B (en) * | 2019-11-01 | 2023-06-09 | 宏碁股份有限公司 | Electronic device and object information identification method using touch data thereof |
EP4046066A4 (en) * | 2019-11-07 | 2023-11-15 | Google LLC | Monitoring animal pose dynamics from monocular images |
US11587361B2 (en) | 2019-11-08 | 2023-02-21 | Wisconsin Alumni Research Foundation | Movement monitoring system |
US11109586B2 (en) * | 2019-11-13 | 2021-09-07 | Bird Control Group, Bv | System and methods for automated wildlife detection, monitoring and control |
CN110866559A (en) * | 2019-11-14 | 2020-03-06 | 上海中信信息发展股份有限公司 | Poultry behavior analysis method and device |
US11284824B2 (en) * | 2019-12-02 | 2022-03-29 | Everseen Limited | Method and system for determining a human social behavior classification |
WO2021123893A1 (en) | 2019-12-19 | 2021-06-24 | Illumisonics Inc. | Photoacoustic remote sensing (pars), and related methods of use |
US11755989B2 (en) | 2020-03-27 | 2023-09-12 | Toshiba Global Commerce Solutions Holdings Corporation | Preventing theft at retail stores |
US11482049B1 (en) | 2020-04-14 | 2022-10-25 | Bank Of America Corporation | Media verification system |
US20210315186A1 (en) * | 2020-04-14 | 2021-10-14 | The United States Of America, As Represented By Secretary Of Agriculture | Intelligent dual sensory species-specific recognition trigger system |
US11238634B2 (en) * | 2020-04-28 | 2022-02-01 | Adobe Inc. | Motion model refinement based on contact analysis and optimization |
US11513667B2 (en) | 2020-05-11 | 2022-11-29 | Apple Inc. | User interface for audio message |
US11786128B2 (en) | 2020-06-18 | 2023-10-17 | Illumisonics Inc. | PARS imaging methods |
US11122978B1 (en) | 2020-06-18 | 2021-09-21 | Illumisonics Inc. | PARS imaging methods |
US11589010B2 (en) | 2020-06-03 | 2023-02-21 | Apple Inc. | Camera and visitor user interfaces |
US11657614B2 (en) | 2020-06-03 | 2023-05-23 | Apple Inc. | Camera and visitor user interfaces |
US11918370B2 (en) * | 2020-06-10 | 2024-03-05 | The Board Of Trustees Of The Leland Stanford Junior University | Systems and methods for estimation of Parkinson's Disease gait impairment severity from videos using MDS-UPDRS |
US11361468B2 (en) | 2020-06-26 | 2022-06-14 | Standard Cognition, Corp. | Systems and methods for automated recalibration of sensors for autonomous checkout |
US11303853B2 (en) | 2020-06-26 | 2022-04-12 | Standard Cognition, Corp. | Systems and methods for automated design of camera placement and cameras arrangements for autonomous checkout |
CN111967321B (en) * | 2020-07-15 | 2024-04-05 | 菜鸟智能物流控股有限公司 | Video data processing method, device, electronic equipment and storage medium |
WO2022026886A1 (en) * | 2020-07-30 | 2022-02-03 | The Jackson Laboratory | Automated phenotyping of behavior |
WO2022035424A1 (en) * | 2020-08-11 | 2022-02-17 | Hitachi America, Ltd. | Situation recognition method and system for manufacturing collaborative robots |
CN116195261A (en) | 2020-09-05 | 2023-05-30 | 苹果公司 | User interface for managing audio of media items |
CN112189588B (en) * | 2020-10-10 | 2022-04-05 | 东北农业大学 | Cow image information collecting and processing method and system |
CN112215160B (en) * | 2020-10-13 | 2023-11-24 | 厦门大学 | Video three-dimensional human body posture estimation algorithm utilizing long-short period information fusion |
WO2022077113A1 (en) | 2020-10-14 | 2022-04-21 | One Cup Productions Ltd. | Animal visual identification, tracking, monitoring and assessment systems and methods thereof |
KR20230132483A (en) * | 2020-12-29 | 2023-09-15 | 더 잭슨 래보라토리 | Gait and posture analysis |
US11594032B1 (en) | 2021-02-17 | 2023-02-28 | Bank Of America Corporation | Media player and video verification system |
US11928187B1 (en) | 2021-02-17 | 2024-03-12 | Bank Of America Corporation | Media hosting system employing a secured video stream |
US11527106B1 (en) | 2021-02-17 | 2022-12-13 | Bank Of America Corporation | Automated video verification |
US11790694B1 (en) | 2021-02-17 | 2023-10-17 | Bank Of America Corporation | Video player for secured video stream |
US20220335446A1 (en) * | 2021-04-14 | 2022-10-20 | Sunshine Energy Technology Co., Ltd. | Real Food Honesty Display System |
CN113205032B (en) * | 2021-04-27 | 2022-11-01 | 安徽正华生物仪器设备有限公司 | Automatic analysis system and method for mouse suspension experiment based on deep learning |
CA3222789A1 (en) | 2021-05-27 | 2022-12-01 | Ai Thinktank Llc | 3d avatar generation and robotic limbs using biomechanical analysis |
US11615648B2 (en) | 2021-05-28 | 2023-03-28 | Sportsbox.ai Inc. | Practice drill-related features using quantitative, biomechanical-based analysis |
US11526548B1 (en) | 2021-06-24 | 2022-12-13 | Bank Of America Corporation | Image-based query language system for performing database operations on images and videos |
US11941051B1 (en) | 2021-06-24 | 2024-03-26 | Bank Of America Corporation | System for performing programmatic operations using an image-based query language |
US11784975B1 (en) * | 2021-07-06 | 2023-10-10 | Bank Of America Corporation | Image-based firewall system |
WO2023053229A1 (en) | 2021-09-28 | 2023-04-06 | 富士通株式会社 | Image processing program, image processing apparatus, and image processing method |
KR20230080219A (en) * | 2021-11-29 | 2023-06-07 | 주식회사 바딧 | Method, system and non-transitory computer-readable recording medium for supporting labeling to sensor data |
US20230206615A1 (en) * | 2021-12-29 | 2023-06-29 | Halliburton Energy Services, Inc. | Systems and methods to determine an activity associated with an object of interest |
CN114533040B (en) * | 2022-01-12 | 2024-04-09 | 北京京仪仪器仪表研究总院有限公司 | Method for monitoring specific activity of personnel in fixed space |
EP4276772A1 (en) * | 2022-05-12 | 2023-11-15 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Method, computer program and system for analysing one or more moving objects in a video |
CN114916452A (en) * | 2022-05-26 | 2022-08-19 | 浙江理工大学 | Device for testing influence of IVC environmental parameters on thermal comfort of SPF experimental animals |
CN115281604B (en) * | 2022-08-24 | 2023-11-21 | 深圳市具安科技有限公司 | Animal eye movement-based vertical axis rotation analysis method, device and medium |
Citations (55)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US3100473A (en) * | 1961-01-30 | 1963-08-13 | Mead Johnson & Co | Apparatus for measuring animal activity |
US3304911A (en) * | 1964-08-24 | 1967-02-21 | Shionogi & Co | Apparatus for automatically measuring the movement of an animal |
US3485213A (en) * | 1967-10-23 | 1969-12-23 | Edward J Scanlon | Animal exercising,conditioning and therapy and apparatus therefor |
US3803571A (en) * | 1971-10-21 | 1974-04-09 | Foerster Inst Dr Friedrich | Apparatus for monitoring test animal activity |
US3974798A (en) * | 1975-04-21 | 1976-08-17 | Meetze Jr Murray O | Method and apparatus for studying laboratory animal behavior |
US4337726A (en) * | 1980-07-07 | 1982-07-06 | Czekajewski Jan A | Animal activity monitor and behavior processor |
US4517593A (en) * | 1983-04-29 | 1985-05-14 | The United States Of America As Represented By The Secretary Of The Navy | Video multiplexer |
US4574734A (en) * | 1984-05-25 | 1986-03-11 | Omnitech Electronics, Inc. | Universal animal activity monitoring system |
US4600016A (en) * | 1985-08-26 | 1986-07-15 | Biomechanical Engineering Corporation | Method and apparatus for gait recording and analysis |
US4631676A (en) * | 1983-05-25 | 1986-12-23 | Hospital For Joint Diseases Or | Computerized video gait and motion analysis system and method |
US4888703A (en) * | 1986-09-09 | 1989-12-19 | Hitachi Engineering Co., Ltd. | Apparatus for monitoring the toxicant contamination of water by using aquatic animals |
US5299454A (en) * | 1992-12-10 | 1994-04-05 | K.K. Holding Ag | Continuous foot-strike measuring system and method |
US5428723A (en) * | 1992-09-09 | 1995-06-27 | International Business Machines Corporation | Method and apparatus for capturing the motion of an object in motion video |
US5546439A (en) * | 1995-11-02 | 1996-08-13 | General Electric Company | Systems, methods and apparatus for incrementally reconstructing overlapped images in a CT system implementing a helical scan |
US5581276A (en) * | 1992-09-08 | 1996-12-03 | Kabushiki Kaisha Toshiba | 3D human interface apparatus using motion recognition based on dynamic image processing |
US5596994A (en) * | 1993-08-30 | 1997-01-28 | Bro; William L. | Automated and interactive behavioral and medical guidance system |
US5708767A (en) * | 1995-02-03 | 1998-01-13 | The Trustees Of Princeton University | Method and apparatus for video browsing based on content and structure |
US5748775A (en) * | 1994-03-09 | 1998-05-05 | Nippon Telegraph And Telephone Corporation | Method and apparatus for moving object extraction based on background subtraction |
US5816256A (en) * | 1997-04-17 | 1998-10-06 | Bioanalytical Systems, Inc. | Movement--responsive system for conducting tests on freely-moving animals |
US5821945A (en) * | 1995-02-03 | 1998-10-13 | The Trustees Of Princeton University | Method and apparatus for video browsing based on content and structure |
US5870138A (en) * | 1995-03-31 | 1999-02-09 | Hitachi, Ltd. | Facial image processing |
US6010465A (en) * | 1991-10-10 | 2000-01-04 | Neurocom International, Inc. | Apparatus and method for characterizing gait |
US6061088A (en) * | 1998-01-20 | 2000-05-09 | Ncr Corporation | System and method for multi-resolution background adaptation |
US6072903A (en) * | 1997-01-07 | 2000-06-06 | Kabushiki Kaisha Toshiba | Image processing apparatus and image processing method |
US6072496A (en) * | 1998-06-08 | 2000-06-06 | Microsoft Corporation | Method and system for capturing and representing 3D geometry, color and shading of facial expressions and other animated objects |
US6081607A (en) * | 1996-07-25 | 2000-06-27 | Oki Electric Industry Co. | Animal body identifying device and body identifying system |
US6088468A (en) * | 1995-05-17 | 2000-07-11 | Hitachi Denshi Kabushiki Kaisha | Method and apparatus for sensing object located within visual field of imaging device |
US6144366A (en) * | 1996-10-18 | 2000-11-07 | Kabushiki Kaisha Toshiba | Method and apparatus for generating information input using reflected light image of target object |
US6212510B1 (en) * | 1998-01-30 | 2001-04-03 | Mitsubishi Electric Research Laboratories, Inc. | Method for minimizing entropy in hidden Markov models of physical signals |
US6231527B1 (en) * | 1995-09-29 | 2001-05-15 | Nicholas Sol | Method and apparatus for biomechanical correction of gait and posture |
US6242456B1 (en) * | 1998-03-09 | 2001-06-05 | Trustees Of Tufts College | Treatment of stereotypic, self-injurious and compulsive behaviors in man and animals using antagonists of NMDA receptors |
US6263088B1 (en) * | 1997-06-19 | 2001-07-17 | Ncr Corporation | System and method for tracking movement of objects in a scene |
US6295367B1 (en) * | 1997-06-19 | 2001-09-25 | Emtera Corporation | System and method for tracking movement of objects in a scene using correspondence graphs |
US6311644B1 (en) * | 2000-02-03 | 2001-11-06 | Carl S. Pugh | Apparatus and method for animal behavior tracking, predicting and signaling |
US6334187B1 (en) * | 1997-07-03 | 2001-12-25 | Matsushita Electric Industrial Co., Ltd. | Information embedding method, information extracting method, information embedding apparatus, information extracting apparatus, and recording media |
US6456737B1 (en) * | 1997-04-15 | 2002-09-24 | Interval Research Corporation | Data processing system and method |
US6468998B1 (en) * | 1998-11-02 | 2002-10-22 | Mitsubishi Pharma Corporation | Pyrrolidine compounds and medicinal utilization thereof |
US20030024482A1 (en) * | 2001-08-06 | 2003-02-06 | Vijay Gondhalekar | Programmable electronic maze for use in the assessment of animal behavior |
US6535131B1 (en) * | 1998-08-26 | 2003-03-18 | Avshalom Bar-Shalom | Device and method for automatic identification of sound patterns made by animals |
US20030100998A2 (en) * | 2001-05-15 | 2003-05-29 | Carnegie Mellon University (Pittsburgh, Pa) And Psychogenics, Inc. (Hawthorne, Ny) | Systems and methods for monitoring behavior informatics |
US6576237B1 (en) * | 1993-12-30 | 2003-06-10 | President And Fellows Of Harvard College | Vertebrate tissue pattern-inducing proteins, and uses related thereto |
US6601010B1 (en) * | 2000-06-05 | 2003-07-29 | The University Of Kansas | Force plate actometer |
US6630148B1 (en) * | 1993-12-30 | 2003-10-07 | Imperial Cancer Research Technology Ltd. | Compositions comprising hedgehog proteins |
US6630347B1 (en) * | 1996-03-15 | 2003-10-07 | The General Hospital Corporation | Endothelial NOS transgenic animals and methods of use |
US6650778B1 (en) * | 1999-01-22 | 2003-11-18 | Canon Kabushiki Kaisha | Image processing method and apparatus, and storage medium |
US6678413B1 (en) * | 2000-11-24 | 2004-01-13 | Yiqing Liang | System and method for object identification and behavior characterization using video analysis |
US20040009845A1 (en) * | 2002-07-15 | 2004-01-15 | Johnson Reginald A. | Balance and gait training board |
US20040020443A1 (en) * | 2000-05-16 | 2004-02-05 | Frauke Ohl | New screening tool for analyzing behavior of laboratory animals |
US6704502B2 (en) * | 1995-03-02 | 2004-03-09 | Canon Kabushiki Kaisha | Vibration correction apparatus and optical device |
US6715444B1 (en) * | 1999-11-11 | 2004-04-06 | Kowa Co., Ltd. | Method and device for measuring frequency of specific behavior of animal |
US6757444B2 (en) * | 1996-07-05 | 2004-06-29 | Canon Kabushiki Kaisha | Image extraction apparatus and method |
US6819796B2 (en) * | 2000-01-06 | 2004-11-16 | Sharp Kabushiki Kaisha | Method of and apparatus for segmenting a pixellated image |
US6899686B2 (en) * | 2001-09-17 | 2005-05-31 | The Curavita Corporation | Method and apparatus for monitoring locomotion kinematics in ambulating animals |
US6941239B2 (en) * | 1996-07-03 | 2005-09-06 | Hitachi, Ltd. | Method, apparatus and system for recognizing actions |
US7133537B1 (en) * | 1999-05-28 | 2006-11-07 | It Brokerage Services Pty Limited | Method and apparatus for tracking a moving object |
Family Cites Families (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPS58184810U (en) * | 1982-06-01 | 1983-12-08 | 鐘淵化学工業株式会社 | magnetic circuit device |
JPH0785080B2 (en) * | 1986-11-25 | 1995-09-13 | 株式会社日立製作所 | Fish condition monitor |
JPH07108285B2 (en) | 1988-10-14 | 1995-11-22 | 東洋産業株式会社 | Experimental animal behavior observer |
US5414644A (en) * | 1993-11-24 | 1995-05-09 | Ethnographics, Inc. | Repetitive event analysis system |
JPH0863603A (en) * | 1994-06-15 | 1996-03-08 | Olympus Optical Co Ltd | Image analyzer |
US5872865A (en) * | 1995-02-08 | 1999-02-16 | Apple Computer, Inc. | Method and system for automatic classification of video images |
JP3683929B2 (en) * | 1995-03-02 | 2005-08-17 | キヤノン株式会社 | Blur correction device and optical device |
GB9506324D0 (en) * | 1995-03-28 | 1995-05-17 | Vinten Group Plc | Improvements in or relating to linear force actuators |
JP3377659B2 (en) * | 1995-09-07 | 2003-02-17 | 株式会社日立国際電気 | Object detection device and object detection method |
US5969755A (en) * | 1996-02-05 | 1999-10-19 | Texas Instruments Incorporated | Motion based event detection system and method |
JP3540494B2 (en) * | 1996-03-15 | 2004-07-07 | 株式会社東芝 | Cooperative work adjusting device and cooperative work adjusting method |
JPH1152215A (en) * | 1997-07-29 | 1999-02-26 | Nikon Corp | Lens driving device |
JP3270005B2 (en) * | 1998-03-20 | 2002-04-02 | 勝義 川崎 | Automated method of observing behavior of experimental animals |
US6721454B1 (en) * | 1998-10-09 | 2004-04-13 | Sharp Laboratories Of America, Inc. | Method for automatic extraction of semantically significant events from video |
US7643655B2 (en) * | 2000-11-24 | 2010-01-05 | Clever Sys, Inc. | System and method for animal seizure detection and classification using video analysis |
-
2000
- 2000-11-24 US US09/718,374 patent/US6678413B1/en not_active Expired - Lifetime
-
2001
- 2001-11-19 AU AU2002239272A patent/AU2002239272A1/en not_active Abandoned
- 2001-11-19 JP JP2002544950A patent/JP2004514975A/en active Pending
- 2001-11-19 EP EP01987014A patent/EP1337962B9/en not_active Expired - Lifetime
- 2001-11-19 WO PCT/US2001/043282 patent/WO2002043352A2/en active Application Filing
-
2003
- 2003-09-19 US US10/666,741 patent/US7068842B2/en not_active Expired - Lifetime
- 2003-10-30 US US10/698,008 patent/US20040141635A1/en not_active Abandoned
- 2003-10-30 US US10/698,044 patent/US7209588B2/en not_active Expired - Lifetime
-
2004
- 2004-09-30 US US10/955,661 patent/US8514236B2/en active Active
-
2007
- 2007-04-06 US US11/697,544 patent/US20070175406A1/en not_active Abandoned
-
2009
- 2009-01-15 US US12/354,727 patent/US20090285452A1/en not_active Abandoned
- 2009-02-03 US US12/365,149 patent/US7817824B2/en not_active Expired - Fee Related
-
2010
- 2010-09-13 US US12/881,154 patent/US20110007946A1/en not_active Abandoned
Patent Citations (59)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US3100473A (en) * | 1961-01-30 | 1963-08-13 | Mead Johnson & Co | Apparatus for measuring animal activity |
US3304911A (en) * | 1964-08-24 | 1967-02-21 | Shionogi & Co | Apparatus for automatically measuring the movement of an animal |
US3485213A (en) * | 1967-10-23 | 1969-12-23 | Edward J Scanlon | Animal exercising,conditioning and therapy and apparatus therefor |
US3803571A (en) * | 1971-10-21 | 1974-04-09 | Foerster Inst Dr Friedrich | Apparatus for monitoring test animal activity |
US3974798A (en) * | 1975-04-21 | 1976-08-17 | Meetze Jr Murray O | Method and apparatus for studying laboratory animal behavior |
US4337726A (en) * | 1980-07-07 | 1982-07-06 | Czekajewski Jan A | Animal activity monitor and behavior processor |
US4517593A (en) * | 1983-04-29 | 1985-05-14 | The United States Of America As Represented By The Secretary Of The Navy | Video multiplexer |
US4631676A (en) * | 1983-05-25 | 1986-12-23 | Hospital For Joint Diseases Or | Computerized video gait and motion analysis system and method |
US4574734A (en) * | 1984-05-25 | 1986-03-11 | Omnitech Electronics, Inc. | Universal animal activity monitoring system |
US4600016A (en) * | 1985-08-26 | 1986-07-15 | Biomechanical Engineering Corporation | Method and apparatus for gait recording and analysis |
US4888703A (en) * | 1986-09-09 | 1989-12-19 | Hitachi Engineering Co., Ltd. | Apparatus for monitoring the toxicant contamination of water by using aquatic animals |
US6010465A (en) * | 1991-10-10 | 2000-01-04 | Neurocom International, Inc. | Apparatus and method for characterizing gait |
US5581276A (en) * | 1992-09-08 | 1996-12-03 | Kabushiki Kaisha Toshiba | 3D human interface apparatus using motion recognition based on dynamic image processing |
US5428723A (en) * | 1992-09-09 | 1995-06-27 | International Business Machines Corporation | Method and apparatus for capturing the motion of an object in motion video |
US5299454A (en) * | 1992-12-10 | 1994-04-05 | K.K. Holding Ag | Continuous foot-strike measuring system and method |
US5596994A (en) * | 1993-08-30 | 1997-01-28 | Bro; William L. | Automated and interactive behavioral and medical guidance system |
US6630148B1 (en) * | 1993-12-30 | 2003-10-07 | Imperial Cancer Research Technology Ltd. | Compositions comprising hedgehog proteins |
US6576237B1 (en) * | 1993-12-30 | 2003-06-10 | President And Fellows Of Harvard College | Vertebrate tissue pattern-inducing proteins, and uses related thereto |
US5748775A (en) * | 1994-03-09 | 1998-05-05 | Nippon Telegraph And Telephone Corporation | Method and apparatus for moving object extraction based on background subtraction |
US5708767A (en) * | 1995-02-03 | 1998-01-13 | The Trustees Of Princeton University | Method and apparatus for video browsing based on content and structure |
US5821945A (en) * | 1995-02-03 | 1998-10-13 | The Trustees Of Princeton University | Method and apparatus for video browsing based on content and structure |
US6704502B2 (en) * | 1995-03-02 | 2004-03-09 | Canon Kabushiki Kaisha | Vibration correction apparatus and optical device |
US5870138A (en) * | 1995-03-31 | 1999-02-09 | Hitachi, Ltd. | Facial image processing |
US6088468A (en) * | 1995-05-17 | 2000-07-11 | Hitachi Denshi Kabushiki Kaisha | Method and apparatus for sensing object located within visual field of imaging device |
US6231527B1 (en) * | 1995-09-29 | 2001-05-15 | Nicholas Sol | Method and apparatus for biomechanical correction of gait and posture |
US5546439A (en) * | 1995-11-02 | 1996-08-13 | General Electric Company | Systems, methods and apparatus for incrementally reconstructing overlapped images in a CT system implementing a helical scan |
US6630347B1 (en) * | 1996-03-15 | 2003-10-07 | The General Hospital Corporation | Endothelial NOS transgenic animals and methods of use |
US6941239B2 (en) * | 1996-07-03 | 2005-09-06 | Hitachi, Ltd. | Method, apparatus and system for recognizing actions |
US6757444B2 (en) * | 1996-07-05 | 2004-06-29 | Canon Kabushiki Kaisha | Image extraction apparatus and method |
US6081607A (en) * | 1996-07-25 | 2000-06-27 | Oki Electric Industry Co. | Animal body identifying device and body identifying system |
US6144366A (en) * | 1996-10-18 | 2000-11-07 | Kabushiki Kaisha Toshiba | Method and apparatus for generating information input using reflected light image of target object |
US6072903A (en) * | 1997-01-07 | 2000-06-06 | Kabushiki Kaisha Toshiba | Image processing apparatus and image processing method |
US6456737B1 (en) * | 1997-04-15 | 2002-09-24 | Interval Research Corporation | Data processing system and method |
US5816256A (en) * | 1997-04-17 | 1998-10-06 | Bioanalytical Systems, Inc. | Movement--responsive system for conducting tests on freely-moving animals |
US6263088B1 (en) * | 1997-06-19 | 2001-07-17 | Ncr Corporation | System and method for tracking movement of objects in a scene |
US6295367B1 (en) * | 1997-06-19 | 2001-09-25 | Emtera Corporation | System and method for tracking movement of objects in a scene using correspondence graphs |
US6334187B1 (en) * | 1997-07-03 | 2001-12-25 | Matsushita Electric Industrial Co., Ltd. | Information embedding method, information extracting method, information embedding apparatus, information extracting apparatus, and recording media |
US6061088A (en) * | 1998-01-20 | 2000-05-09 | Ncr Corporation | System and method for multi-resolution background adaptation |
US6212510B1 (en) * | 1998-01-30 | 2001-04-03 | Mitsubishi Electric Research Laboratories, Inc. | Method for minimizing entropy in hidden Markov models of physical signals |
US6242456B1 (en) * | 1998-03-09 | 2001-06-05 | Trustees Of Tufts College | Treatment of stereotypic, self-injurious and compulsive behaviors in man and animals using antagonists of NMDA receptors |
US6072496A (en) * | 1998-06-08 | 2000-06-06 | Microsoft Corporation | Method and system for capturing and representing 3D geometry, color and shading of facial expressions and other animated objects |
US6535131B1 (en) * | 1998-08-26 | 2003-03-18 | Avshalom Bar-Shalom | Device and method for automatic identification of sound patterns made by animals |
US6468998B1 (en) * | 1998-11-02 | 2002-10-22 | Mitsubishi Pharma Corporation | Pyrrolidine compounds and medicinal utilization thereof |
US6650778B1 (en) * | 1999-01-22 | 2003-11-18 | Canon Kabushiki Kaisha | Image processing method and apparatus, and storage medium |
US7133537B1 (en) * | 1999-05-28 | 2006-11-07 | It Brokerage Services Pty Limited | Method and apparatus for tracking a moving object |
US6715444B1 (en) * | 1999-11-11 | 2004-04-06 | Kowa Co., Ltd. | Method and device for measuring frequency of specific behavior of animal |
US6819796B2 (en) * | 2000-01-06 | 2004-11-16 | Sharp Kabushiki Kaisha | Method of and apparatus for segmenting a pixellated image |
US6311644B1 (en) * | 2000-02-03 | 2001-11-06 | Carl S. Pugh | Apparatus and method for animal behavior tracking, predicting and signaling |
US20040020443A1 (en) * | 2000-05-16 | 2004-02-05 | Frauke Ohl | New screening tool for analyzing behavior of laboratory animals |
US6601010B1 (en) * | 2000-06-05 | 2003-07-29 | The University Of Kansas | Force plate actometer |
US6678413B1 (en) * | 2000-11-24 | 2004-01-13 | Yiqing Liang | System and method for object identification and behavior characterization using video analysis |
US20040141635A1 (en) * | 2000-11-24 | 2004-07-22 | Yiqing Liang | Unified system and method for animal behavior characterization from top view using video analysis |
US20040141636A1 (en) * | 2000-11-24 | 2004-07-22 | Yiqing Liang | Unified system and method for animal behavior characterization in home cages using video analysis |
US20030100998A2 (en) * | 2001-05-15 | 2003-05-29 | Carnegie Mellon University (Pittsburgh, Pa) And Psychogenics, Inc. (Hawthorne, Ny) | Systems and methods for monitoring behavior informatics |
US7269516B2 (en) * | 2001-05-15 | 2007-09-11 | Psychogenics, Inc. | Systems and methods for monitoring behavior informatics |
US20030024482A1 (en) * | 2001-08-06 | 2003-02-06 | Vijay Gondhalekar | Programmable electronic maze for use in the assessment of animal behavior |
US6837184B2 (en) * | 2001-08-06 | 2005-01-04 | Psychogenics, Inc. | Programmable electronic maze for use in the assessment of animal behavior |
US6899686B2 (en) * | 2001-09-17 | 2005-05-31 | The Curavita Corporation | Method and apparatus for monitoring locomotion kinematics in ambulating animals |
US20040009845A1 (en) * | 2002-07-15 | 2004-01-15 | Johnson Reginald A. | Balance and gait training board |
Non-Patent Citations (1)
Title |
---|
Tillett et al: "Using model-based image processing to track animal movements", Computers and Electronics in Agriculture, 1997 * |
Cited By (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20110007946A1 (en) * | 2000-11-24 | 2011-01-13 | Clever Sys, Inc. | Unified system and method for animal behavior characterization with training capabilities |
US20100106743A1 (en) * | 2001-05-15 | 2010-04-29 | Psychogenics, Inc. | Method for Predicting Treatment Classes Using Behavior Informatics |
US7882135B2 (en) | 2001-05-15 | 2011-02-01 | Psychogenics, Inc. | Method for predicting treatment classes using behavior informatics |
US20120140042A1 (en) * | 2007-01-12 | 2012-06-07 | International Business Machines Corporation | Warning a user about adverse behaviors of others within an environment based on a 3d captured image stream |
US9208678B2 (en) * | 2007-01-12 | 2015-12-08 | International Business Machines Corporation | Predicting adverse behaviors of others within an environment based on a 3D captured image stream |
US9412011B2 (en) | 2007-01-12 | 2016-08-09 | International Business Machines Corporation | Warning a user about adverse behaviors of others within an environment based on a 3D captured image stream |
US10354127B2 (en) | 2007-01-12 | 2019-07-16 | Sinoeast Concept Limited | System, method, and computer program product for alerting a supervising user of adverse behavior of others within an environment by providing warning signals to alert the supervising user that a predicted behavior of a monitored user represents an adverse behavior |
CN103325124A (en) * | 2012-03-21 | 2013-09-25 | 东北大学 | Target detecting and tracking system and method using background differencing method based on FPGA |
US11205088B2 (en) * | 2018-05-18 | 2021-12-21 | Hangzhou Hikvision Digital Technology Co., Ltd. | Method and apparatus for calculating a luminance value of a region of interest |
CN108846326A (en) * | 2018-05-23 | 2018-11-20 | 盐城工学院 | The recognition methods of pig posture, device and electronic equipment |
Also Published As
Publication number | Publication date |
---|---|
US20040131254A1 (en) | 2004-07-08 |
US20040141635A1 (en) | 2004-07-22 |
JP2004514975A (en) | 2004-05-20 |
US20070229522A1 (en) | 2007-10-04 |
US6678413B1 (en) | 2004-01-13 |
US20090296992A1 (en) | 2009-12-03 |
WO2002043352A3 (en) | 2003-01-09 |
US20040141636A1 (en) | 2004-07-22 |
US7817824B2 (en) | 2010-10-19 |
US7209588B2 (en) | 2007-04-24 |
WO2002043352A2 (en) | 2002-05-30 |
EP1337962B1 (en) | 2012-09-26 |
EP1337962A2 (en) | 2003-08-27 |
US8514236B2 (en) | 2013-08-20 |
EP1337962A4 (en) | 2007-02-28 |
US20070175406A1 (en) | 2007-08-02 |
AU2002239272A1 (en) | 2002-06-03 |
EP1337962B9 (en) | 2013-02-13 |
US20110007946A1 (en) | 2011-01-13 |
US7068842B2 (en) | 2006-06-27 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7209588B2 (en) | Unified system and method for animal behavior characterization in home cages using video analysis | |
US9984183B2 (en) | Method for automatic behavioral phenotyping | |
US7643655B2 (en) | System and method for animal seizure detection and classification using video analysis | |
Yang et al. | Automatic recognition of sow nursing behaviour using deep learning-based segmentation and spatial and temporal features | |
US10121064B2 (en) | Systems and methods for behavior detection using 3D tracking and machine learning | |
US20060195199A1 (en) | Monitoring device | |
Wang et al. | An automated behavior analysis system for freely moving rodents using depth image | |
CN112580552A (en) | Method and device for analyzing behavior of rats | |
US8484143B2 (en) | System and method for analyzing exploratory behavior | |
CN114549371B (en) | Image analysis method and device | |
Twining et al. | Robust tracking and posture description for laboratory rodents using active shape models | |
Monteiro et al. | A depth-map approach for automatic mice behavior recognition | |
Sepúlveda et al. | Evaluation of background subtraction algorithms using MuHAVi, a multicamera human action video dataset | |
Tiwari et al. | Real-Time Fatigue Detection System using Computer Vision | |
Lai et al. | Mining motion patterns using color motion map clustering | |
Bulpitt et al. | Monitoring behavior of individuals in crowded scenes | |
Shimoide et al. | Automated behavioral phenotype detection and analysis using color-based motion tracking | |
Yu et al. | Automated home-cage behavioral phenotyping of mice | |
Szalai et al. | Tracking the saliency features in images based on human observation statistics | |
Huang | Tracking and analysis of Caenorhabditis elegans behavior using machine vision | |
Rao | DRIVER DROWSINESS MONITORING SYSTEM USING MACHINE LEARNING | |
Geng et al. | Detecting Attached Worm Eggs using Deformable Template Matching | |
ALEKHYA et al. | DRIVER DROWSINESS MONITORING SYSTEM USING VISUAL BEHAVIORS AND MACHINE LEARNING |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: CLEVER SYS, INC., VIRGINIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:WOLF, WAYNE;WILKS (LINDA S. CRNIC), STAN L.;REEL/FRAME:022116/0266;SIGNING DATES FROM 20080809 TO 20080928 Owner name: CLEVER SYS, INC., VIRGINIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:LIANG, YIQING;KOBLA, VIKRANT;BAI, XUESHENG;AND OTHERS;REEL/FRAME:022116/0258;SIGNING DATES FROM 20040405 TO 20040806 |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |