US20060120564A1 - Human identification apparatus and human searching/tracking apparatus - Google Patents
Human identification apparatus and human searching/tracking apparatus Download PDFInfo
- Publication number
- US20060120564A1 US20060120564A1 US11/342,651 US34265106A US2006120564A1 US 20060120564 A1 US20060120564 A1 US 20060120564A1 US 34265106 A US34265106 A US 34265106A US 2006120564 A1 US2006120564 A1 US 2006120564A1
- Authority
- US
- United States
- Prior art keywords
- information
- walking
- spatiotemporal
- image
- person
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/0304—Detection arrangements using opto-electronic means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
Definitions
- the present invention relates to a human identification apparatus which judges whether or not a human image included in one image sequence and a human image included in another image sequence represent the same person, and an apparatus which searches for or tracks a person.
- One of the conventional methods for judging a person for identification with the view to searching for or tracking a person represented by a human image in an image sequence is to correspond human image areas in the neighboring frames (for example pp. 3 and 6 and FIGS. 2 and 9, in the Japanese Laid-Open Application No. 2003-346159).
- FIGS. 1A through 1C are diagrams for describing the searching/tracking method described in the above-mentioned Application No. 2003-346159.
- FIGS. 1A and 1B shows temporally consecutive frame images which are obtained by shooting a person.
- the frame image A 10 shown in FIG. 1A is an image obtained by shooting a person A 11 who moves toward right.
- a rectangle A 12 is a circumscribed rectangle including head and body of the person which are detected, using motion vectors, as areas with small movement (i.e. steady human area) in the whole body.
- a circumscribed rectangle A 22 of the steady human area detected from a person A 21 is presented in a dashed line.
- the steady human areas of the circumscribed rectangles A 12 and A 22 obtained respectively from the frame images A 10 and A 20 are displayed at the same time in such a way that they overlap with each other.
- the person A 11 and the person A 21 are identified as the same person based on how the circumscribed rectangles A 12 and A 22 overlap as well as a continuity of respective motion vectors.
- the conventional method in which a person is searched in positions that are close between the frames, has a problem that, in the case where a detection of the person once fails in a certain frame, it becomes impossible to identify a human image in a frame prior to the failed frame with a human image in a frame following the failed frame, so as to track the person.
- An object of the present invention is to provide a human identification apparatus which can judge whether or not persons represented by human images respectively included in temporally-distant frames or frames shot with different cameras are the same person, as well as a human searching/tracking apparatus for searching for or tracking a person, with the use of such human identification apparatus.
- the human identification apparatus of the present invention judges whether or not persons represented by human images respectively included in different image sequences, are the same person, and includes: an image sequence receiving unit which receives inputs of a first image sequence and a second image sequence which is obtained at a time different from a time when the first image sequence is obtained or by an image sensor different from an image sensor used for obtaining the first image sequence; a walking sequence extracting unit which extracts, from the respective inputted first and second image sequences, first and second walking sequences, each being an image sequence indicating a walking state of a person; a gait information extracting unit which extracts first and second gait information, each being information which specifies a periodic motion in human gait, based on the respective extracted first and second walking sequences; a gait information verifying unit which verifies between the extracted first gait information and the extracted second gait information; and a judging unit which judges whether or not the persons represented by the human images respectively included in the first and second image sequences are the same person, based on
- the present invention it is possible to verify between human image sequences respectively obtained from temporally-distant frames or with different sensors, by using spatiotemporal period information, spatiotemporal phase information and spatiotemporal position information which can be obtained based on human gait.
- a human identification apparatus which can judge whether or not persons represented by the human images that are respectively obtained from the temporally-distant frames or the frames shot by different cameras are the same person, and a human searching/tracking apparatus which searches for or tracks a person using such human identification apparatus.
- FIG. 1A shows an example of a rectangle area detected by a conventional human searching/tracking apparatus
- FIG. 1B shows another example of the rectangle area detected by the conventional human searching/tracking apparatus
- FIG. 1C shows an example of a movement of the rectangle area detected by the conventional human searching/tracking apparatus
- FIG. 2 is a functional block diagram showing a structure of a human identification apparatus according to a first embodiment of the present invention
- FIG. 3 shows an example of an image sequence according to the first embodiment of the present invention
- FIG. 4A shows an example of an image sequence of a lower human body image
- FIG. 4B shows an example of a walking sequence according to the first embodiment of the present invention
- FIG. 4C shows an example of a form of the smallest pattern indicating a walking sequence
- FIG. 5 is a diagram showing spatiotemporal phase information and spatiotemporal position information, according to the first embodiment of the present invention.
- FIG. 6A shows an example of a walking trace according to the first embodiment of the present invention
- FIG. 6B shows an example of a state of walking estimated according to the first embodiment of the present invention
- FIG. 6C shows an example of walking states of different phases, according to the first embodiment of the present invention.
- FIG. 7A shows an example of a relationship between a walking trace and a change in stride length, according to the first embodiment
- FIG. 7B shows an example of the change in stride length, according to the first embodiment of the present invention.
- FIG. 7C shows another example of the change in stride length, according to the first embodiment of the present invention.
- FIG. 8A shows an example of a display of the walking sequence, according to the first embodiment of the present invention.
- FIG. 8B shows another example of a display of the walking sequence, according to the first embodiment of the present invention.
- FIG. 8C shows yet another example of a display of the walking sequence, according to the first embodiment of the present invention.
- FIG. 9 shows an example of a storage format obtained as a result of searching/tracking the walking sequence, according to the first embodiment of the present invention.
- FIG. 10 is a block diagram showing a structure of the human searching/tracking apparatus according to the first embodiment of the present invention.
- FIG. 11 shows an example of a screen for giving instructions on a search or a tracing of a person, according to the first embodiment
- FIG. 12 is a flowchart showing an example of a verification procedure according to the first embodiment of the present invention.
- FIG. 13A shows an example of an image sequence 1 according to the first embodiment of the present invention
- FIG. 13B shows an example of an image sequence 2 according to the first embodiment of the present invention
- FIG. 14 is a flowchart showing an example of a procedure for extracting a walking sequence, according to the first embodiment of the present invention.
- FIG. 15 is a flowchart showing an example of a procedure for extracting spatiotemporal period information, spatiotemporal phase information and spatiotemporal position information, according to the first embodiment of the present invention
- FIG. 16A shows an example of a detection of a specific walking state, according to the first embodiment of the present invention
- FIG. 16B shows an example of a template for detecting the specific walking state, according to the first embodiment of the present invention
- FIG. 16C shows an example of a process for detecting the specific walking state, according to the first embodiment of the present invention
- FIG. 17 shows an example of a display performed by a control unit according to the first embodiment of the present invention
- FIG. 18A shows an example of an image sequence 1 according to a second embodiment of the present invention.
- FIG. 18B shows an example of an image sequence 2 according to the second embodiment of the present invention.
- FIG. 19 shows a functional block showing a structure of the human identification apparatus according to the second embodiment of the present invention.
- the human identification apparatus judges whether or not persons represented by human images respectively included in different image sequences are the same person, and includes: an image sequence receiving unit which receives inputs of a first image sequence and a second image sequence obtained at a time different from a time when the first image sequence is obtained or by an image sensor different from an image sensor used for obtaining the first image sequence; a walking sequence extracting unit which extracts first and second walking sequences, each being an image sequence indicating a walking state of a person, from the respective inputted first and second image sequences; a gait information extracting unit which extracts first and second gait information, each being information which specifies a periodic walking movement of a person based on the respective extracted first and second walking sequences; a gait information verifying unit which verifies between the extracted first information and the extracted second gait information; and a judging unit operable to verify whether or not the persons represented by the human images respectively included in the first and second image sequences are the same person, based on a result of the verification performed by the gait
- the judgment for the identification of persons is made based on human gait, therefore, it is possible, compared to the conventional technique of judging that human images located in similar positions between frames represent a same person or identifying persons based on colors and image patterns, to judge for the identification of persons even in the case where a person to be searched for or tracked is hidden behind another person or an object, or a normal detection cannot be temporarily carried out in a frame due to change in lighting conditions.
- the present invention is conceived based on the facts that gait characteristics such as a walking cycle and a length of stride differ from person to person, and that an individual walks with the same gait characteristics. It is therefore possible to identify persons irrespective of one's temporal or spatial position, and also to judge whether human images respectively included in temporally-distant frames or in frames shot by different cameras represent the same person.
- the gait information verifying unit may verify the first and second gait information by comparing, based on first and second spatiotemporal phase information respectively included in the first and second gait information, times or positions at/in which the human images respectively included in the first and second image sequences take a predetermined walking posture.
- the gait information verifying unit has: a phase information estimating unit which estimates, for the human image included in the first image sequence, spatiotemporal phase information at a time or in a position different from a time or a position of the first image sequence; and a gait information verifying unit which verifies between the first gait information and the second gait information by comparing, based on the spatiotemporal phase information estimated by the phase information estimating unit and the second spatiotemporal phase information, times or positions in which the persons represented by the human images respectively included in the first and second image sequences take the predetermined walking posture.
- the gait information verifying unit may verify between the first and second gait information by comparing, based on the first and second spatiotemporal phase information respectively included in the first and second gait information, walking postures of the human images included in the respective first and second image sequences of a same time or position.
- the gait information verifying unit has: a phase information estimating unit which estimates, for the human image included in the first image sequence, spatiotemporal phase information indicating a time or a position different from a time or a position of the first image sequence; and a gait information verifying unit which verifies between the first gait information and the second gait information by comparing, based on the spatiotemporal phase information estimated by the phase information estimating unit and the second spatiotemporal phase information, walking postures of the human images respectively included in the first and second image sequences of a same time or position.
- a walking sequence may be, for example, an image presenting a cross-section in the case where each image sequence is cut based on time.
- it is an image obtainable by arranging fragmented image areas generated by cutting an image presenting the legs of the human images respectively included in the image sequences. In this way, it is possible to extract, temporally and spatially, gait characteristics of a person.
- the image sequence receiving unit may receive inputs of the first and second image sequences obtained by different image sensors which have shot the same place. Thus, it is possible to judge whether or not the persons, who are represented by the human images which are respectively included in the different image sequences that are obtained by shooting, from different angles, the same location having dead places, are the same person.
- the human identification apparatus may further include: a correction information holding unit which previously holds correction information indicating a correlation between a position within an image of the respective first and second image sequences, and a position in a shooting location; a correcting unit which makes, based on the correction information held in the correction information holding unit, a spatiotemporal correction for the extraction of the first and second gait information performed by the gait information extracting unit.
- a correction information holding unit which previously holds correction information indicating a correlation between a position within an image of the respective first and second image sequences, and a position in a shooting location
- a correcting unit which makes, based on the correction information held in the correction information holding unit, a spatiotemporal correction for the extraction of the first and second gait information performed by the gait information extracting unit.
- the correction information may be, for example, information which specifies grid lines that two-dimensionally section, at regular intervals, a surface on which a person walks in a shooting place.
- the human searching/tracking apparatus searches for or tracks a specific person in an image sequence obtained by shooting a person, and includes: the human identification apparatus; and a walking sequence storing unit operable to store, in an associated manner, the first and second walking sequences respectively corresponding to first and second gait information, in the case where the gait information verifying unit of the human identification apparatus verifies that the first gait information matches the second gait information.
- a walking sequence storing unit operable to store, in an associated manner, the first and second walking sequences respectively corresponding to first and second gait information, in the case where the gait information verifying unit of the human identification apparatus verifies that the first gait information matches the second gait information.
- the human searching/tracking apparatus searches for or tracks a specific person in an image sequence obtained by shooting a person, and includes: the human identification apparatus; and a display unit operable to display the first and second image sequences received by the image sequence receiving unit of the human identification apparatus, wherein said display unit is operable to display in a highlighted manner, the human images judged by the judging unit of the human identification apparatus to represent the same person, so that the identified human images are distinguished from the other human images, the identified human images being identified from among the human images included in the first and second image sequences.
- the identified persons can be promptly perceived by displaying the persons in a highlighted manner. This facilitates a search or tracking of a person.
- a human image judging unit which has: an image sequence receiving unit operable to receive inputs of a first image sequence, and a second image sequence which is obtained in one of the following manners: at a time different from a time when the first image sequence is obtained; and by an image sensor different from an image sensor used for obtaining the first image sequence; a walking sequence extracting unit operable to extract, from the respective received first and second image sequences, first and second walking sequences, each sequence being an image sequence indicating a walking state of a person; a gait information extracting unit operable to extract, from the respective extracted first and second walking sequences, first spatiotemporal period information and first spatiotemporal phase information as first gait information, and second spatiotemporal period information and second spatiotemporal phase information as second gait information, the spatiotemporal period information indicating a temporal or spatial walking period of a person, the
- Yet another aspect of the human searching/tracking apparatus of the present invention searches for or tracks a specific person in an image sequence obtained by shooting a person, and includes: a human identification apparatus which has: an image sequence receiving unit operable to receive inputs of a first image sequence, and a second image sequence which is obtained in one of the following manners: at a time different from a time when the first image sequence is obtained; and by an image sensor used for obtaining the first image sequence; a walking sequence extracting unit operable to extract, from the respective received first and second image sequences, first and second walking sequences, each sequence being an image sequence indicating a walking state of a person;
- a gait information extracting unit operable to extract, from the respective extracted first and second walking sequences, first spatiotemporal period information and first spatiotemporal phase information as first gait information, and second spatiotemporal period information and second spatiotemporal phase information as second gait information, the spatiotemporal period information indicating a temporal or spatial walking period of a person, the spatiotemporal phase information indicating a temporal or spatial phase information in periodic walking movement of a person, and the gait information specifying a periodic motion in human gait; a gait information verifying unit operable to verify between the extracted first gait information and the extracted second gait information; and a judging unit operable to judge whether or not the persons represented by the human images respectively included in the first and second image sequences are the same person; and a display unit operable to display the first and second image sequences received by the image sequence receiving unit of the human identification apparatus, wherein the display unit is operable to display, in a highlighted manner, the human images judged by the judging unit of the human identification apparatus
- Another aspect of the human identification apparatus of the present invention judges whether or not persons represented by human images respectively included in different image sequences are the same person, and includes: a walking sequence detecting unit operable to detect first and second walking sequences, each sequence being an image sequence indicating a walking state of first or second person respectively included in the image sequences; a walking posture transition estimating unit operable to estimate, based on the walking sequence of the first person, information indicating a transition of a walking posture in a periodic walking movement of the first person at a time or in a position different from a time or a position of the walking sequence of the first person; and a judging unit operable to verify whether or not the estimated information indicating the transition of the walking posture of the first person matches information indicating a transition of a walking posture of the second person, and to judge that the first person and the second person are the same person in the case of verifying that the information match with each other.
- a walking sequence detecting unit operable to detect first and second walking sequences, each sequence being an image sequence indicating a
- the judgment for the identification of persons is made based on human gait, so that it is possible to judge for the identification of persons even in the case where a normal detection cannot be temporarily carried out since a person to be searched for or tracked is hidden behind another person or an object, or due to change in lighting conditions. That is to say, the present invention is conceived based on the facts that gait characteristics such as a walking cycle and a length of stride differ from person to person, and that an individual keeps the same gait characteristics. It is therefore possible to identify persons irrespective of one's temporal or spatial position, and also to judge whether the persons represented by the human images respectively included in temporally-distant frames or in frames shot by different cameras are the same person.
- the present invention can be realized not only as such human identification apparatus and the human searching/tracking apparatus as described above, but also as a human image judging method, a human image searching/tracking method, or a program causing a computer to execute such methods, and even as a computer-readable storage medium on which the program is stored.
- FIG. 2 is a functional block diagram showing a structure of the human identification apparatus 10 according to the preset embodiment.
- the human identification apparatus 10 is an apparatus which judges whether or not persons represented by the human images respectively included in different sequences are the same person based on a continuity of walking sequence of a person.
- Such human identification apparatus 10 includes a walking posture detecting unit 200 , a walking state estimating unit 180 and a judging unit 190 .
- the walking posture detecting unit 200 is a processing unit which detects, in a moving picture, a walking sequence including a predetermined walking posture of a first person.
- the walking state estimating unit 180 is a processing unit which estimates, based on a walking sequence of the first person, a walking state (a transition state of posture in periodic walking movement) of the first person at a time or in a position different from a time or a position of the walking sequence of the first person.
- a walking state a transition state of posture in periodic walking movement
- the judging unit 190 is a processing unit which verifies whether or not a walking state of the first person matches a walking state of a second person, and judges that the first person and the second person are the same person in the case where the walking states match with each other.
- the walking posture detecting unit 200 includes an image sequence receiving unit 100 and a walking sequence extracting unit 110 .
- the walking state estimating unit 180 includes a spatiotemporal period information extracting unit 120 , a spatiotemporal phase information extracting unit 121 and a spatiotemporal position information extracting unit 122 .
- the judging unit 190 includes a spatiotemporal period information storing unit 130 , a spatiotemporal phase information storing unit 131 , a spatiotemporal position information storing unit 132 , a spatiotemporal period verifying unit 140 , a spatiotemporal phase verifying unit 141 , a spatiotemporal difference extracting unit 142 , a matching judging unit 150 and a control unit 160 .
- the walking posture detecting unit 200 is an example of the walking sequence detecting unit which detects a walking sequence being an image sequence indicating a walking state of a human image included in the image sequence.
- the walking state estimating unit 180 is an example of the walking posture transition estimating unit which estimates, based on the detected walking sequence, information indicating a transition of a walking posture in a periodic walking movement at a time or in a position different from a time or a position of the detected walking sequence.
- the judging unit 190 is an example of the judging unit which judges whether or not the information indicating respective transitions of two persons shot at different times or with different image sensors, and judges whether or not the two persons are identified as the same person.
- “information indicating a transition of walking posture” is information which includes the period information and phase information which will be mentioned later.
- the image sequence receiving unit 100 is an example of the image sequence receiving unit which receives inputs of first and second image sequences respectively obtained at a different time or with a different image sensor.
- the walking sequence extracting unit 110 is an example of the walking sequence extracting which extracts first and second walking sequences, each being an image sequence indicating a walking state of a person based on the respective first and second image sequences.
- the spatiotemporal phase information extracting unit 121 , the spatiotemporal position information extracting unit 122 and the spatiotemporal period information extracting unit 120 are examples of the gait information extracting unit which extracts first and second gait information, each being information which specifies a periodic walking movement of a person, based on the respective first and second walking sequences.
- the spatiotemporal phase verifying unit 141 , the spatiotemporal difference extracting unit 142 and the spatiotemporal period verifying unit 140 are examples of the gait information verifying unit which verifies between the extracted first gait information and the extracted second gait information.
- the matching judging unit 150 is an example of the judging unit which judges whether or not the human images respectively included in the first and second image sequences represent the same person, based on the result of the verification.
- the image sequence receiving unit 100 is a signal interface, or the like, which obtains an image sequence from a camera or an image storing apparatus.
- the “image sequence” is an array in which shot frame images are arranged along a temporal axis, as shown in FIG. 3 .
- the walking sequence extracting unit 110 is a processing unit which extracts a walking sequence from the image sequence obtained by the image sequence receiving unit 100 .
- the “walking sequence” is a sequence of a walking state obtained from a walking movement area in each frame image.
- FIG. 4B shows an example of such walking sequence.
- FIG. 4A shows a sequence of lower body areas, each being extracted from each frame, and the respective extracted areas are sequentially arranged in time order.
- FIG. 4B is a graph obtained by arranging the extracted areas along a temporal axis at a dashed line B 10 as shown in FIG. 4A .
- a black band in FIG. 4B shows a trace of moving toes (obtained by arranging respective images of each leg along the temporal axis). A method of calculating each walking sequence will be described later.
- the spatiotemporal period information extracting unit 120 is a processing unit which extracts period information being a spatiotemporal change in walking, based on the walking sequence extracted by the walking sequence extracting unit 110 .
- the “period information” is a result obtained by frequency number analysis made with respect to strides per predetermined period of time or predetermined distance and spatiotemporal positional change of a specific body part such as feet and hands, or a form of the smallest pattern to be repeated spatiotemporally and periodically.
- FIG. 4B shows an example of such spatiotemporal period information. In FIG.
- a black or white triangular mark denotes a spatiotemporal point at which the toes cross in temporal axis and a horizontal axis (as opposed to spatial axis) of the image.
- the spatiotemporal period information may be the number of strides per predetermined period of time or predetermined distance obtained based on spatial intervals between the neighboring black triangular marks as well as temporal intervals between the neighboring white triangular marks.
- a form per se of walking pattern surrounded by a dashed line linking the neighboring black triangular marks and a dashed line linking the neighboring white triangular marks ( FIG.
- a frequency characteristic (spectral intensity of a specific frequency band) obtained by frequency analysis made with respect to a temporal change in stride length as shown in a curved line B 11 in FIG. 4B , and a spatial change in stride length as shown in a curved line B 12 may be also used as the spatiotemporal period information.
- the spatiotemporal period information storing unit 130 is a memory for storing the spatiotemporal period information extracted by the spatiotemporal period information extracting unit 120 , together with the time and the position in the image which are detected.
- the spatiotemporal period verifying unit 140 is a processing unit which verifies between the spatiotemporal period information extracted by the spatiotemporal period information extracting unit 120 and the spatiotemporal period information held in the spatiotemporal period information storing unit 130 .
- the spatiotemporal phase information extracting unit 121 is a processing unit which extracts phase information indicating a spatiotemporal change in walking, based on the walking sequence extracted by the walking sequence extracting unit 110 .
- the “phase information” means a transition state (e.g. a position or time in which a predetermined walking posture is taken, or a walking posture in a specific position or at a specific time) during the periodic exercise of walking.
- the spatiotemporal phase information for example, is information indicating a spatiotemporal position in which a foot touched the ground (a predetermined walking posture) for the walking sequences of the same spatiotemporal period.
- the spatiotemporal phase information indicates also a difference in walking posture between two walking sequences compared at a same time or in a same position.
- FIG. 5 shows a walking trace of a foot position, as shown in FIG. 4B , but shows different walking traces A 010 and A 011 of different two persons who have the same spatiotemporal period but a different spatiotemporal phase. Comparing the walking traces A 010 and A 011 , a stride length and a walking period are the same, but a position or a time in/at which a foot touches the ground or a position or a time in/at which the legs are crossed are different.
- FIG. 5 shows a walking trace of a foot position, as shown in FIG. 4B , but shows different walking traces A 010 and A 011 of different two persons who have the same spatiotemporal period but a different spatiotemporal phase. Comparing the walking traces A 010 and A 011 , a stride length and a walking period are the same, but a position or a time in/at which a foot touches the ground or a position or a time in/at which the legs are crossed are different.
- phase information is a position or a time in/at which a predetermined walking posture is taken or a walking posture in a predetermined position or at a predetermined time.
- phase information is derived based on the same criterion.
- the spatiotemporal phase information extracting unit 121 estimates a walking posture at a time or in a position except for the time or the position at/in which a frame is shot, based on a position, a time or a periodicity of the predetermined posture (e.g. crossing of the legs) in the gait information 1802 of the first person (a dashed line 1803 in FIG. 6B ).
- the spatiotemporal phase information extracting unit 121 estimates a walking posture (a dashed line 1805 ) at a different time or in a different position based on a walking trace 1804 (in a dashed line) in the right of the obstacle 1800 .
- the spatiotemporal phase information extracting unit 121 then obtains a time or a position at/in which the predetermined walking posture is taken.
- crossing of the legs (a state in which a stride length becomes the smallest) is assumed to be the predetermined posture.
- FIG. 7A shows a temporal or positional change in stride length.
- the stride length can be derived, based on images, as a width between the tracings of the respective legs in a walking trace. As shown in FIG.
- the spatiotemporal phase information extracting unit 121 estimates a state (posture) 1903 a (in a dashed line) of the stride length at a time or in a position at/in which a person is not shot due to an obstacle 1900 a (a hatched area), based on stride length information 1902 a (a dashed line) of the first person.
- the spatiotemporal phase information extracting unit 121 obtains, as phase information, an area which is shot at an earlier time and in the left within the image.
- the time and the position are a time and a position at/in which the human image takes the predetermined posture.
- a time or position 1905 is obtained as phase information for an estimated walking state 1903 a of the first person, while a time or position 1906 is obtained for the walking state 1901 a of the second person.
- phase information 1910 is obtained for a walking state 1909 a (a dashed line) estimated for the walking state 1908 a (a dashed line), while phase information 1911 is obtained for a walking state 1907 a (a full line).
- the spatiotemporal phase information extracting unit 121 obtains a walking state or an estimated walking state at a predetermined time or in a predetermined position also in the case of using, as phase information, a posture at a predetermined time or in a predetermined position. In the case of FIG. 7B , the spatiotemporal phase information extracting unit 121 obtains a stride length (walking posture) at a predetermined time or in a predetermined position 1904 .
- the phase information for the first person shall be a value indicating an estimated walking state 1903 a (a dashed line) at the predetermined time or position 1904
- the phase information for the second person shall be a value indicating the walking state 1901 a (a full line) at the predetermined time or position 1904
- the value indicating an estimated walking state 1909 a at the predetermined time or position 1904 which is estimated based on the walking state 1908 a
- the value indicating the walking state 1907 a at the predetermined time or position 1904 are the phase information of the first person and the second person, respectively.
- an estimated posture is used only for the first person, however, an estimated posture may be obtained for the second person so that phase information is obtained for both the first and second persons based on the estimated postures.
- a location of an obstacle in FIGS. 6A through 6C and 7 A through 7 C and an area except for the area indicated in the diagram may be determined as fixed parts, so as to obtain phase information after deriving an estimated state in the fixed parts.
- the spatiotemporal phase information storing unit 131 is a memory for storing the spatiotemporal phase information extracted by the spatiotemporal phase information extracting unit 121 together with a time as well as a position in an image which are detected.
- the spatiotemporal phase verifying unit 141 is a processing unit which verifies between the spatiotemporal phase information extracted by the spatiotemporal phase information extracting unit 121 and the spatiotemporal phase information held in the spatiotemporal phase information storing unit 131 .
- the spatiotemporal position information extracting unit 122 refers to the walking sequence extracted by the walking sequence extracting unit 110 , extracts a spatiotemporal position at which the walking sequence is extracted, and generates spatiotemporal position information.
- the “spatiotemporal position information” means a time or a place where the walking sequence is detected. An example of such spatiotemporal position information is described in FIG. 5 .
- a cross-shape in dashed line presents a spatiotemporal position at which the legs are crossed for the first time with regard to respective two walking traces.
- the spatiotemporal position information is information indicating an absolute position of spatiotemporal walking.
- the spatiotemporal position information storing unit 132 is a memory for storing the spatiotemporal position information generated by the spatiotemporal position information extracting unit 122 .
- the spatiotemporal difference extracting unit 142 is a processing unit which obtains a difference between the spatiotemporal position information generated by the spatiotemporal position information extracting unit 122 and the spatiotemporal position information held in the spatiotemporal position information storing unit 132 .
- the matching judging unit 150 is a processing unit which judges whether or not different walking sequences match with each other, based on the results obtained by the spatiotemporal period verifying unit 140 , the spatiotemporal phase verifying unit 141 and the spatiotemporal difference extracting unit 142 . That is to say, the matching judging unit 150 judges whether or not the walking sequences are of the same person.
- An example of the method of judging on matching with regard to the spatiotemporal period information is as follows.
- the matching judging unit 150 judges that the two walking sequences match with each other, when the following equations are satisfied:
- the matching judging unit 150 judges the respective spatiotemporal phase information and spatiotemporal position information in the same way as in the case of the spatiotemporal period information, and in the case where the information match with each other for all the items or the specified number of items, the matching judging unit 150 judges that the two walking sequences match with each other. It should be noted that the judging unit is not limited to the one described above, and a method used for pattern recognition may be generally applied.
- the temporal phase information tp 1 and the spatial phase information (phase information of position) sp 1 which are obtained from the walking sequence of the first person in the moving picture are compared with the temporal phase information tp 2 and the spatial phase information (phase information of position) sp 2 which are obtained based on the walking sequence of the second person at a time or in a position different from the time and the position at/in which the phase information tp 1 and sp 1 are obtained.
- the matching judging unit 150 judges that the first person and the second person are the same person in the case where one or both of the following equations is/are satisfied:
- the matching judging unit 150 verifies that the spatiotemporal phase information of the walking state 1801 matches the spatiotemporal phase information of the walking state 1803 , and judges that the first person and the second person are the same person.
- FIG. 6B for example, by appropriately setting ⁇ t and ⁇ s, the matching judging unit 150 verifies that the spatiotemporal phase information of the walking state 1801 matches the spatiotemporal phase information of the walking state 1803 , and judges that the first person and the second person are the same person.
- FIG. 6B for example, by appropriately setting ⁇ t and ⁇ s, the matching judging unit 150 verifies that the spatiotemporal phase information of the walking state 1801 matches the spati
- the matching judging unit 150 verifies that the spatiotemporal phase information of the walking state 1801 does not match the spatiotemporal phase information of the walking state 1803 , and judges that the first person and the second person are not the same person.
- the matching judging unit 150 verifies that the spatiotemporal phase information of the walking state 1901 a matches the spatiotemporal phase information of the walking state 1909 a , and judges that the first person and the second person are the same person.
- the matching judging unit 150 verifies that the spatiotemporal phase information of the walking state 1907 a does not match the spatiotemporal phase information of the walking state 1909 a , and judges that the first person and the second person are different persons.
- the control unit 160 is a processing unit which performs display of an image sequence used for verification, as a control based on the result of the judgment made by the matching judging unit 150 .
- FIGS. 8A through 8C show examples of such display.
- FIG. 8A shows an example of magnified human image areas, one in the left and the other in the right, of the two image sequences in comparison, displayed together with shooting time and place of each image sequence.
- FIG. 8B shows an example of the result as same as shown in FIG. 8A , displayed together with a trace of human movement (arrows in the diagram).
- FIG. 8C shows an example of a display in which the area including a verified human image is highlighted with a rectangle in a dashed line, in the case where plural persons are included within an image.
- the control unit 160 may control not only display but also storage, into an external storage apparatus (e.g. position information storing apparatus), of matching information which indicates that different walking sequences are verified to match with each other and thus correlates the walking sequences. By thus storing the information indicating matching/mismatching of different walking sequences, such information can be utilized for searching for or tracking a human walking image.
- an external storage apparatus e.g. position information storing apparatus
- FIG. 9 shows an example of a storage format showing the result of a search or a tracking of a walking sequence by use of the process as described above.
- FIG. 9 presents three types of walking sequence information and information of five items for one walking sequence.
- the five items are sequence number, spatiotemporal period information, spatiotemporal phase information, spatiotemporal position information, and matched sequence number.
- a sequence number is an ID number assigned to respective walking sequences which are shot at different times with different cameras.
- the spatiotemporal period information presents, as (x,y), the number of strides x within a predetermined period and the number of strides y within a predetermined distance.
- the spatiotemporal phase information presents an amount of movement to a temporal or spatial position at which the legs are crossed for the first time since a temporal or spatial position to be used as a reference is passed with a ratio where a time and a movement amount required for a sequence of one stride are respectively determined as 1.0. For instance, in the case of sequence number 1, the legs are crossed at a time when a time required for 0.5 stride has passed from a reference time and after a distance required for 0.1 stride is walked from a reference position.
- the spatiotemporal position information indicates time and place at which a walking sequence is detected for the first time.
- the place is represented by pixel coordinator values in vertical and horizontal directions within an image.
- Matched sequence number indicates a sequence number of the walking sequence verified to match with a different walking sequence in a search or a tracking.
- the sequence number may be 0, and in the case where plural pairs of verified sequences are detected, plural sequence numbers may be described.
- sequence number may be associated with a reference destination (e.g. a file name and a storage address) of a shot image.
- a reference destination e.g. a file name and a storage address
- information about a spatial location in the spatiotemporal position information may be represented by coordinator system that is set for exclusive use, or a general coordinator such as latitude and longitude.
- FIG. 10 is a block diagram showing a structure of the human searching/tracking apparatus 20 onto which the human identification apparatus 10 is applied.
- the human searching/tracking apparatus 20 is a concrete example of a system or an apparatus which realizes a method of searching/tracking a person in an image sequence, according to the present embodiment.
- Such human searching/tracking apparatus 20 includes cameras 1010 and 1020 , a clock 1030 , a storing apparatus 1040 , a processing apparatus 1050 , a display apparatus 1060 , an input unit 1070 and a pointing device 1080 .
- the cameras 1010 and 1020 are examples of the image sequence receiving unit 100 , and shoot images of a person 1000 .
- the clock 1030 is a timer for obtaining a shooting time.
- the storing apparatus 1040 is a hard disk which stores the images shot with the cameras 1010 and 1020 , respective shooting time obtained by the clock 1030 and the results of a search or a tracking of a person.
- the processing apparatus 1050 is an apparatus which searches for or tracks a person in the image sequence obtained by the cameras 1010 and 1020 or the storing apparatus 1040 , and is equivalent of the human identification apparatus 10 shown in FIG. 2 .
- the display apparatus 1060 is a display which displays a result of the processing performed by the processing apparatus 1050 , while the input unit 1070 is a keyboard used for instructions on a search or tracking and the pointing device 1080 is a mouse used for the instructions on the search or tracking.
- FIG. 11 shows an example of pointing.
- a person 1091 to be searched/tracked is specified using a pointer 1090 , for searching for or tracking a walking sequence as same as that of the person 1091 .
- the communication path may be wired or wireless, and may include a private line and a public line
- FIGS. 13A and 13B respectively show examples of image sequences 1 and 2 .
- FIG. 13A shows an image sequence 1 in which a person 502 walks toward right on the left side of an obstacle 501 .
- FIG. 13B shows an image sequence 2 obtained by shooting the same place with the same camera as the image sequence 1 after ten seconds have passed since the image sequence 1 is shot.
- the image sequence 2 shows the obstacle 501 and persons 503 and 504 walking toward right.
- the image sequence receiving unit 100 receives an image sequence 1 (Step S 401 ). Then, a walking sequence (walking sequence 1 ) is extracted from the image sequence 1 (Step S 402 ). In the description, a sequence of lower body area as shown in FIG. 4A is used as a walking sequence.
- the walking sequence extracting unit 110 reads one frame image from the image sequence (Step S 601 ). A frame image is read from among the unread frame images in time order. The walking sequence extracting unit 110 then detects a human image area from the read-out frame image (Step S 602 ). For the detection of a human image area, method such as inter-frame difference method that is generally used for detecting a moving object and a difference background method for calculating a difference between the human image area and a previously-provided background image which does not include any human images.
- a template image of human may be prepared so that it is possible to detect a human image by extracting an area with high similarity to the template image, using similarity calculation (a template matching method such as a simple difference calculation and a normalization correlation calculation).
- the walking sequence extracting unit 110 then extracts, from the human image area, information indicating a walking state (Step S 603 ).
- Such information indicating a walking state may be information indicating a transition of time for a walking state such as trace information, as shown in FIG. 4B , indicating a tracing of toes.
- Steps S 602 and S 603 may be simultaneously performed as a sequence of process or a result of processing Step S 602 may be considered as a result of processing Step S 603 .
- processing may be executed so that an output of Step S 603 can be directly obtained.
- the walking state information may be directly obtained using the template matching method where the lower body image is used as a template.
- the walking sequence extracting unit 110 eventually judges whether or not the frame image that has just been read is the last frame (Step S 604 ), and ends the processing of extracting a walking sequence in the case where the image is the last frame, or returns to Step S 601 in the case where the last frame image still remains to be read.
- the spatiotemporal period information extracting unit 120 the spatiotemporal phase information extracting unit 121 and the spatiotemporal position information extracting unit 122 respectively extracts spatiotemporal period information, spatiotemporal phase information and spatiotemporal position information, based on the walking sequence 1 obtained in Step S 402 (Step S 403 ).
- the spatiotemporal period information extracting unit 120 and others detect a position of a specific walking state based on the walking sequence information (Step S 701 ).
- the description of the specific walking state will be described with reference to FIGS. 16A through 16C .
- FIG. 16A shows a result of the detection of a specific walking state with respect to the walking sequence shown in FIG. 4B .
- Two black wave-like bands, shown in FIG. 16A which cross with each other show a temporal trace of moving the toes.
- a horizontal axis presents a position of an image in a horizontal direction whereas a vertical line presents time.
- a specific walking state position is defined as a position at which the two bands cross the legs are crossed, namely, a position indicating a state in which the legs are crossed (a position indicated by respective dashed lines in the diagram).
- the crossing point can be detected by performing template matching or correlation operation using form patterns of the crossing section as shown in FIG. 16B which are provided beforehand.
- An example of the detecting process is shown in FIG. 16C .
- a degree of matching in form is calculated by shifting a position of a detection template 801 with respect to a walking trace 800 .
- the walking trace 800 is determined to be the specific walking state.
- the crossing points of dashed lines as shown in FIG. 16A , can be obtained.
- Step S 701 is performed until all the specific walking states are detected, and the process goes on to Step S 703 after that (Step S 702 ).
- the spatiotemporal period information extracting unit 120 then generates spatiotemporal period information by calculating an interval between the detected specific walking state positions (Step 703 ).
- the period information may be derived separately without obtaining such specific walking state positions, using the methods such as a Fourier transform, a wavelet transform and an auto-correlation method. In this case, these methods can be applied to a temporal change in a position of the wave-like bands shown in FIG. 16A or a spatiotemporal change in a gap between the two bands.
- a form of the smallest unit of a periodic walking trace, as shown in FIG. 16A may be used as a pattern. In such case, one wave form made up of two bands within the range sectioned by horizontal and vertical dashed lines may be used as a pattern.
- the spatiotemporal phase information extracting unit 121 derives spatiotemporal phase information (Step S 704 ).
- the spatiotemporal phase information represents an amount of spatiotemporal movement to a spatiotemporal position at which the legs are crossed (where a specific walking state appears) for the first time after a reference spatiotemporal position is passed.
- a time or a position at/in which a predetermined posture is taken a time or a position at/in which a stride length becomes the shortest is defined, in FIG. 7B , to be such spatiotemporal phase information.
- a value 1905 derived based on an estimated walking state 1903 a is defined as temporal phase information or spatial phase information
- a value 1906 is defined to be such phase information.
- a value 1904 (a value indicative of stride length) of an estimated walking state 1903 a is defined, for the walking 1902 a , as temporal phase information or spatial phase information, while the value 1904 is defined for the walking 1901 a as such phase information.
- a time between the upper left corner and a position at which the legs are crossed for the first time becomes the temporal phase information.
- a gap between the vertical line A 013 and the position, which is the nearest to the vertical line A 013 on the right side of the line A 013 and at which the legs are crossed becomes the spatial phase information.
- the way of representing such movement amount is not limited to the above examples, and the amount may be presented in a relative size based on a time required for a sequence of one step and its movement amount.
- the spatiotemporal phase information means phase information indicating an appearance of a specific walking state within an image, so that even in the case where the walking sequences have the same spatiotemporal period, different values may be obtained for the walking patterns whose spatiotemporal timings in a state of putting down a foot on the ground or crossing the legs are different.
- the walking sequences A 011 and A 012 have the same spatiotemporal period of walking, however, the spatiotemporal phase information as defined above respectively indicates a different value.
- the spatiotemporal position information extracting unit 122 derives spatiotemporal position information (Step S 705 ).
- the spatiotemporal position information is defined as a time and a position in image in which the first specific walking state is detected.
- the spatiotemporal position information is information indicating an absolute spatiotemporal position coordinate of the detected position of the specific walking state, and a position at which the second step is detected or a position at which the last step is detected may be used instead.
- the spatiotemporal period information, the spatiotemporal phase information and the spatiotemporal position information thus obtained in Step 403 are respectively stored into the spatiotemporal period information storing unit 130 , the spatiotemporal phase information storing unit 131 and the spatiotemporal position information storing unit 132 (Step S 404 ).
- the image sequence receiving unit 100 obtains an image sequence 2 for searching a person, as in Step S 401 (Step S 405 ).
- the walking sequence extracting unit 110 extracts a walking sequence 2 from the image sequence 2 (Step S 406 ).
- the image sequence receiving unit 100 judges whether or not another walking sequence can be found, as a result of Step S 406 (Step S 407 ). In the case where no sequence is found (No in Step S 407 ), the walking sequence extracting unit 110 ends the processing.
- the spatiotemporal period information extracting unit 120 , the spatiotemporal phase information extracting unit 121 and the spatiotemporal position information extracting unit 122 extract, as in the case of the walking sequence 1 , spatiotemporal period information, spatiotemporal phase information and spatiotemporal position information, from the walking sequence 2 (Step S 408 ).
- the spatiotemporal period verifying unit 140 , the spatiotemporal phase verifying unit 141 and the spatiotemporal difference extracting unit 142 respectively verify between the spatiotemporal period information, the spatiotemporal phase information and the spatiotemporal position information of the walking sequence 1 which are stored in Step S 404 and the spatiotemporal period information, the spatiotemporal phase information and the spatiotemporal position information extracted in Step 408 (Step S 409 ). All the three types of information may be used or either of the spatiotemporal period information and the spatiotemporal phase information may be used for the verification.
- a reference degree of matching may be pre-set for the respective three types of information, and in the case where all the three information satisfy the reference degree, it may be judged that the sequences match with each other.
- D x ⁇ x ; and
- D y ⁇ y , where each spatiotemporal period information is expressed as (x1, y1) and (x3, y3), it is judged that the sequences match with each other.
- ⁇ x and ⁇ y are predetermined thresholds.
- ⁇ t and ⁇ d are predetermined thresholds.
- each sum or product of the differences D x and D y in spatiotemporal period information, of the differences D w and D z in spatiotemporal phase information and of the differences D t and D xy in spatiotemporal position information may be compared with the predetermined reference degree of matching. For example, in the case where (D x +D y +D w +D z +D t +D xy ) ⁇ sum and (D x ⁇ D y ⁇ D w ⁇ D z ⁇ D t ⁇ D xy ) ⁇ mul are both satisfied, it is judged that the sequences match with each other.
- ⁇ mul is a predetermined threshold.
- the reference that is set for a difference in spatiotemporal period information and a difference in spatiotemporal phase information may be changed based on the difference in spatiotemporal position information.
- a difference in spatiotemporal position information gets larger since a value indicating spatiotemporal period and a value indicating spatiotemporal phase can vary as the spatiotemporal difference gets larger. Therefore, by reducing, as the difference in spatiotemporal position information increases, the reference references of matching for the difference in spatiotemporal period information and the difference in spatiotemporal phase information, it is possible to decrease the number of undetected sequences.
- the differences D t and D xy in spatiotemporal position information satisfies D t > ⁇ t or D xy > ⁇ xy
- the thresholds ⁇ x , ⁇ y , ⁇ w and ⁇ z are multiplied by ⁇ ( ⁇ is a constant indicating 0 or greater) for judging the verification.
- a value indicating the spatiotemporal period information does not easily change although a direction in which a person faces with respect to a camera changes, it is effective in that a search or tracking can be easily carried out between the temporally-distant image sequences.
- the matching judging unit 150 judges whether or not the walking sequence 1 matches the walking sequence 2 , based on the result of the verification performed in Step S 409 (Step S 410 ). In the case where the sequences do not match with each other (No in Step S 410 ), the matching judging unit 150 returns to Step S 406 and obtains a new walking sequence (Step S 406 ). In the case where the sequences match with each other (Yes in Step S 410 ), the control unit 160 displays the image sequence 1 and the image sequence 2 on the display apparatus 1060 , and also displays, in a highlighted manner, the human image areas of the respective matched walking sequences. In FIG. 17 , the image sequence 1 is displayed in an area 1130 and the image sequence 2 is displayed in an area 1140 , while the human image areas in the respective matched walking sequences are displayed, in a highlighted manner, using circumscribed rectangles 1110 and 1120 .
- the spatiotemporal period information i.e. walking pattern
- changes subjected to various elements such as personal habits (e.g. walking habit), individual circumstances (e.g. in a hurry, walking slowly), a type of footwear (e.g. a height of heel, and a movable range of ankle), belongings (e.g. whether something heavy is carried in the hand), clothes (e.g. a movable range of legs) and road surface conditions (e.g. slipperiness and slope). It is therefore difficult to identify an individual based on the spatiotemporal period information.
- personal habits e.g. walking habit
- individual circumstances e.g. in a hurry, walking slowly
- a type of footwear e.g. a height of heel, and a movable range of ankle
- belongings e.g. whether something heavy is carried in the hand
- clothes e.g. a movable range of legs
- road surface conditions e.g. slipperiness and slope
- the thresholds ⁇ x , ⁇ y , ⁇ w and ⁇ z are multiplied by ⁇ ( ⁇ is a constant less than 1.0) for judging the matching between sequences.
- the embodiment by detecting the spatiotemporal period information, the spatiotemporal phase information and the spatiotemporal position information obtained from a walking sequence, and judging whether the walking sequence matches a different walking sequence, based on the information, it is possible to search/track, with less influence by the size of a human image and a direction in which the human image faces in an image, a person presented by the human image included in an image sequence obtained at a different time.
- the matching of walking sequences is performed using both the spatiotemporal period information and the spatiotemporal phase information, however, it is possible to see if the walking sequences match with each other by using one of the above information, and thus it is possible to obtain the same effect as can be obtained with the present invention. With the combination of the both information, it is possible to perform more sophisticated verification and thereby to improve accuracy in a search or tracking.
- an image sequence to be received by the image sequence receiving unit 100 be a time length which includes a walking distance of at least one step or the number of frames, since the improvement of the verification accuracy can be expected when the number of steps is greater than one.
- time length it is also preferable that an image sequence last a half second or longer (i.e. 15 frames or more, as based on 30 frames per second), and that a specific walking state be detected two times or more.
- FIGS. 18A and 18B show examples of images.
- FIG. 18A shows the image sequence 1 while FIG. 18B shows the image sequence 2 .
- an obstacle 900 is displayed in the form of cuboid.
- a person 922 is hidden by the obstacle 900 and only a person 912 is presented.
- a person 921 and the person 922 are presented.
- the person 912 corresponds to the person 922 , having the same height and the same clothes, however, in the image sequence 2 , the persons 921 and 922 are walking closely to each other so that it is difficult to distinguish whether the person corresponding to the person 912 is the person 921 or the person 922 based on color, texture and motion vector.
- the ground is displayed in a lattice-like design using dashed lines for indicating a position coordinate to be used for verification between respective position information of the two image sequences.
- the lattice form in dashed lines can be obtained by an actual measurement of correspondence (spatiotemporal correction information) between a position within a camera screen and a position in a shooting location or by geometric calculation based on camera placement and optical specification.
- the lattice shown in FIG. 18A and the lattice shown in FIG. 18B present positions to be corresponded.
- Such coordinate conversion between corresponding points on different planes within the image can be represented in a homography H, therefore, the matrix H shall be calculated and held.
- the method of deriving such homography H the conventional method, as introduced in “Optimal homography computation with a reliability measure” in IEICE Transactions on Information and Systems written by K. Kanatani, N. Ohta and Y. Kanazawa, Vol. E83-D, No. 7, pp. 1369-1374 (July 2000), may be used.
- FIG. 19 is a function block diagram showing a structure of a human identification apparatus 15 according to the second embodiment.
- the human identification apparatus 15 includes a spatiotemporal correction unit 170 in addition to the components included in the human identification apparatus 10 according to the first embodiment.
- the spatiotemporal correction unit 170 is a processing unit which corrects spatiotemporal discordance between different image sequences using spatiotemporal correction information, when extracting a walking sequence or deriving spatiotemporal period information, spatiotemporal phase information and spatiotemporal position information.
- the spatiotemporal correction unit 170 is an example of a correction unit which performs spatiotemporal correction in the extraction of gait information.
- the human image judging unit 15 can process, in the same manner as described in the first embodiment, a video taken by a different camera, so as to identify, in the image sequence 2 , a human image corresponding to the human image included in the image sequence 1 .
- a location of a target object within an image differs due to the difference in placement of camera, therefore, grid lines, as spatiotemporal correction information, is previously held for each camera (or each image sequence) so that the grid lines can be used as correction information of spatial position coordinate for the extraction of a walking sequence. That is to say, the spatiotemporal correction unit 170 holds, as correction information, information which specifies grid lines that two-dimensionally sections at regular intervals, a surface on which a person walks in a shooting location.
- the spatiotemporal correction unit 170 can verify spatiotemporal period information, spatiotemporal phase information and spatiotemporal position information between the images taken by different cameras. For example, the spatiotemporal correction unit 170 spatiotemporally corrects the spatiotemporal period information, the spatiotemporal phase information and the spatiotemporal position information according to ratios of each side and dimension of a small area sectioned by the grid lines (i.e. multiply by a proportional factor).
- the spatiotemporal position at which a person is in a state of taking a specific walking posture for the first time after passing a specific grid line, may be used as the spatiotemporal phase information. With such structure, it is possible to identify a person (not shown) hidden behind the obstacle 900 with the person 921 , and also, the person 921 with the person 922 .
- a correspondence of the positions is used for extracting a walking sequence; however, an image in one image sequence may be converted before other processing so that a position of the image in one image sequence corresponds to a position of an image in the other image sequence.
- an image conversion by applying the homography H as described above to a pixel position on the planar surface (i.e. ground) on which a human image walks, it is possible to convert the pixel position into a position on a planar surface (i.e. ground) within the other corresponding image sequence. Therefore, an image can be converted by performing conversion for all the pixels in the same way.
- the spatiotemporal period information, the spatiotemporal phase information and the spatiotemporal position information after the correction using positional relationship it is possible to correspond different walking sequences without being affected by the placement of camera.
- a frame is taken out for every two frames from the latter image sequence so as to generate an image sequence at the frame rate of 15 frames per second, and then, detection of walking sequences and verification of the walking sequences between the image sequences are performed.
- spatiotemporal correction information of frame rate necessary for correction may be previously held in association with each image sequence, in a storing apparatus or the like.
- part or all of the processing described in the above embodiments may be executed by a device intended for such processing, or by a communication device such as a terminal and a base station, or a CPU incorporated in a computer, which executes a program.
- the present invention can be used as a human identification apparatus which judges whether or not human images included in different image sequences represent the same person, and as a human searching/tracking apparatus which searches for or tracks a person in image sequences, e.g., a monitor system that is set up on the street, or the like.
Abstract
Description
- This is a continuation application of PCT application No. PCT/JP05/013769 filed on Jul. 27, 2005, designating the United States of America.
- (1) Field of the Invention
- The present invention relates to a human identification apparatus which judges whether or not a human image included in one image sequence and a human image included in another image sequence represent the same person, and an apparatus which searches for or tracks a person.
- (2) Description of the Related Art
- In order to search for or track a specific person in an image sequence obtained by a surveillance camera, it is required to identify the specific person, namely, to judge whether or not a person that appears in one image or image sequence is as same as a person that appears in another image or image sequence.
- One of the conventional methods for judging a person for identification with the view to searching for or tracking a person represented by a human image in an image sequence is to correspond human image areas in the neighboring frames (for example pp. 3 and 6 and FIGS. 2 and 9, in the Japanese Laid-Open Application No. 2003-346159).
-
FIGS. 1A through 1C are diagrams for describing the searching/tracking method described in the above-mentioned Application No. 2003-346159.FIGS. 1A and 1B shows temporally consecutive frame images which are obtained by shooting a person. The frame image A10 shown inFIG. 1A is an image obtained by shooting a person A11 who moves toward right. A rectangle A12 is a circumscribed rectangle including head and body of the person which are detected, using motion vectors, as areas with small movement (i.e. steady human area) in the whole body. Similarly, in a frame image A20 shown inFIG. 1B , a circumscribed rectangle A22 of the steady human area detected from a person A21 is presented in a dashed line. In a frame image A30 shown inFIG. 1C , the steady human areas of the circumscribed rectangles A12 and A22 obtained respectively from the frame images A10 and A20 are displayed at the same time in such a way that they overlap with each other. According to the conventional method, the person A11 and the person A21 are identified as the same person based on how the circumscribed rectangles A12 and A22 overlap as well as a continuity of respective motion vectors. - However, the conventional method, in which a person is searched in positions that are close between the frames, has a problem that, in the case where a detection of the person once fails in a certain frame, it becomes impossible to identify a human image in a frame prior to the failed frame with a human image in a frame following the failed frame, so as to track the person.
- With the same method, it is conceivable to identify a human image in one frame with a human image in another frame, using colors and image patterns. Other problems, however, are that a positional change increases due to a movement of a person in the case of using temporally distant frames, and that changes of a direction in which the person stands with respect to a camera, a posing of the person and lighting conditions render it difficult to correspond the human image in one frame with the human image in another frame, even though the persons are the same person.
- Therefore, in the case where plural persons appear in an image, when a person to be searched for or tracked is hidden behind another person or an object, or when a normal detection cannot be temporarily carried out in a frame due to changes in the lighting conditions, a problem is that it is no longer possible to continue the search or tracking.
- The present invention is therefore conceived in order to solve the conventional problems as described above. An object of the present invention is to provide a human identification apparatus which can judge whether or not persons represented by human images respectively included in temporally-distant frames or frames shot with different cameras are the same person, as well as a human searching/tracking apparatus for searching for or tracking a person, with the use of such human identification apparatus.
- In order to achieve the above object, the human identification apparatus of the present invention judges whether or not persons represented by human images respectively included in different image sequences, are the same person, and includes: an image sequence receiving unit which receives inputs of a first image sequence and a second image sequence which is obtained at a time different from a time when the first image sequence is obtained or by an image sensor different from an image sensor used for obtaining the first image sequence; a walking sequence extracting unit which extracts, from the respective inputted first and second image sequences, first and second walking sequences, each being an image sequence indicating a walking state of a person; a gait information extracting unit which extracts first and second gait information, each being information which specifies a periodic motion in human gait, based on the respective extracted first and second walking sequences; a gait information verifying unit which verifies between the extracted first gait information and the extracted second gait information; and a judging unit which judges whether or not the persons represented by the human images respectively included in the first and second image sequences are the same person, based on a result of the verification performed by the gait information verifying unit. The gait information here may be, for instance, information indicating a temporal or spatial walking period of a person, or temporal or spatial phase information regarding periodic walking movement of a person, or temporal or spatial position information indicating periodic walking movement of a person.
- According to the present invention, it is possible to verify between human image sequences respectively obtained from temporally-distant frames or with different sensors, by using spatiotemporal period information, spatiotemporal phase information and spatiotemporal position information which can be obtained based on human gait.
- In other words, it is possible to provide a human identification apparatus which can judge whether or not persons represented by the human images that are respectively obtained from the temporally-distant frames or the frames shot by different cameras are the same person, and a human searching/tracking apparatus which searches for or tracks a person using such human identification apparatus.
- For further information about technical background to this application, the disclosure of Japanese Patent Application No. 2004-227083 filed on Aug. 3, 2004 including specification, drawings and claims is incorporated herein by reference in its entirety.
- These and other objects, advantages and features of the invention will become apparent from the following description thereof taken in conjunction with the accompanying drawings that illustrate a specific embodiment of the invention. In the Drawings:
-
FIG. 1A shows an example of a rectangle area detected by a conventional human searching/tracking apparatus; -
FIG. 1B shows another example of the rectangle area detected by the conventional human searching/tracking apparatus; -
FIG. 1C shows an example of a movement of the rectangle area detected by the conventional human searching/tracking apparatus; -
FIG. 2 is a functional block diagram showing a structure of a human identification apparatus according to a first embodiment of the present invention; -
FIG. 3 shows an example of an image sequence according to the first embodiment of the present invention; -
FIG. 4A shows an example of an image sequence of a lower human body image; -
FIG. 4B shows an example of a walking sequence according to the first embodiment of the present invention; -
FIG. 4C shows an example of a form of the smallest pattern indicating a walking sequence; -
FIG. 5 is a diagram showing spatiotemporal phase information and spatiotemporal position information, according to the first embodiment of the present invention; -
FIG. 6A shows an example of a walking trace according to the first embodiment of the present invention; -
FIG. 6B shows an example of a state of walking estimated according to the first embodiment of the present invention; -
FIG. 6C shows an example of walking states of different phases, according to the first embodiment of the present invention; -
FIG. 7A shows an example of a relationship between a walking trace and a change in stride length, according to the first embodiment; -
FIG. 7B shows an example of the change in stride length, according to the first embodiment of the present invention; -
FIG. 7C shows another example of the change in stride length, according to the first embodiment of the present invention; -
FIG. 8A shows an example of a display of the walking sequence, according to the first embodiment of the present invention; -
FIG. 8B shows another example of a display of the walking sequence, according to the first embodiment of the present invention; -
FIG. 8C shows yet another example of a display of the walking sequence, according to the first embodiment of the present invention; -
FIG. 9 shows an example of a storage format obtained as a result of searching/tracking the walking sequence, according to the first embodiment of the present invention; -
FIG. 10 is a block diagram showing a structure of the human searching/tracking apparatus according to the first embodiment of the present invention; -
FIG. 11 shows an example of a screen for giving instructions on a search or a tracing of a person, according to the first embodiment; -
FIG. 12 is a flowchart showing an example of a verification procedure according to the first embodiment of the present invention; -
FIG. 13A shows an example of animage sequence 1 according to the first embodiment of the present invention; -
FIG. 13B shows an example of animage sequence 2 according to the first embodiment of the present invention; -
FIG. 14 is a flowchart showing an example of a procedure for extracting a walking sequence, according to the first embodiment of the present invention; -
FIG. 15 is a flowchart showing an example of a procedure for extracting spatiotemporal period information, spatiotemporal phase information and spatiotemporal position information, according to the first embodiment of the present invention; -
FIG. 16A shows an example of a detection of a specific walking state, according to the first embodiment of the present invention; -
FIG. 16B shows an example of a template for detecting the specific walking state, according to the first embodiment of the present invention; -
FIG. 16C shows an example of a process for detecting the specific walking state, according to the first embodiment of the present invention; -
FIG. 17 shows an example of a display performed by a control unit according to the first embodiment of the present invention; -
FIG. 18A shows an example of animage sequence 1 according to a second embodiment of the present invention; -
FIG. 18B shows an example of animage sequence 2 according to the second embodiment of the present invention; and -
FIG. 19 shows a functional block showing a structure of the human identification apparatus according to the second embodiment of the present invention. - The human identification apparatus according to the present invention judges whether or not persons represented by human images respectively included in different image sequences are the same person, and includes: an image sequence receiving unit which receives inputs of a first image sequence and a second image sequence obtained at a time different from a time when the first image sequence is obtained or by an image sensor different from an image sensor used for obtaining the first image sequence; a walking sequence extracting unit which extracts first and second walking sequences, each being an image sequence indicating a walking state of a person, from the respective inputted first and second image sequences; a gait information extracting unit which extracts first and second gait information, each being information which specifies a periodic walking movement of a person based on the respective extracted first and second walking sequences; a gait information verifying unit which verifies between the extracted first information and the extracted second gait information; and a judging unit operable to verify whether or not the persons represented by the human images respectively included in the first and second image sequences are the same person, based on a result of the verification performed by the gait information verifying unit. The gait information here may be, for instance, information indicating temporal or spatial phase information regarding a periodic walking movement of a person or temporal or spatial position information indicating a periodic walking movement of a person.
- Thus, the judgment for the identification of persons is made based on human gait, therefore, it is possible, compared to the conventional technique of judging that human images located in similar positions between frames represent a same person or identifying persons based on colors and image patterns, to judge for the identification of persons even in the case where a person to be searched for or tracked is hidden behind another person or an object, or a normal detection cannot be temporarily carried out in a frame due to change in lighting conditions. That is to say that the present invention is conceived based on the facts that gait characteristics such as a walking cycle and a length of stride differ from person to person, and that an individual walks with the same gait characteristics. It is therefore possible to identify persons irrespective of one's temporal or spatial position, and also to judge whether human images respectively included in temporally-distant frames or in frames shot by different cameras represent the same person.
- The gait information verifying unit may verify the first and second gait information by comparing, based on first and second spatiotemporal phase information respectively included in the first and second gait information, times or positions at/in which the human images respectively included in the first and second image sequences take a predetermined walking posture. More precisely, the gait information verifying unit has: a phase information estimating unit which estimates, for the human image included in the first image sequence, spatiotemporal phase information at a time or in a position different from a time or a position of the first image sequence; and a gait information verifying unit which verifies between the first gait information and the second gait information by comparing, based on the spatiotemporal phase information estimated by the phase information estimating unit and the second spatiotemporal phase information, times or positions in which the persons represented by the human images respectively included in the first and second image sequences take the predetermined walking posture.
- Thus, accuracy of the judgment can be improved compared to the case of judging whether or not the human images represent the same person based only on spatiotemporal period information. For example, in the case where two persons having the same length of stride and walking speed are shot in the same image sequence, it is difficult to distinct one from the other since they have the same spatiotemporal period information. However, although they have the same spatiotemporal period information, if timings of criss-crossing the legs or positions differ between them, their spatiotemporal phase information differ as well. It is therefore possible to make a precise judgment using spatiotemporal phase information.
- The gait information verifying unit may verify between the first and second gait information by comparing, based on the first and second spatiotemporal phase information respectively included in the first and second gait information, walking postures of the human images included in the respective first and second image sequences of a same time or position. More concretely, the gait information verifying unit has: a phase information estimating unit which estimates, for the human image included in the first image sequence, spatiotemporal phase information indicating a time or a position different from a time or a position of the first image sequence; and a gait information verifying unit which verifies between the first gait information and the second gait information by comparing, based on the spatiotemporal phase information estimated by the phase information estimating unit and the second spatiotemporal phase information, walking postures of the human images respectively included in the first and second image sequences of a same time or position.
- A walking sequence may be, for example, an image presenting a cross-section in the case where each image sequence is cut based on time. To be specific, it is an image obtainable by arranging fragmented image areas generated by cutting an image presenting the legs of the human images respectively included in the image sequences. In this way, it is possible to extract, temporally and spatially, gait characteristics of a person.
- The image sequence receiving unit may receive inputs of the first and second image sequences obtained by different image sensors which have shot the same place. Thus, it is possible to judge whether or not the persons, who are represented by the human images which are respectively included in the different image sequences that are obtained by shooting, from different angles, the same location having dead places, are the same person.
- The human identification apparatus may further include: a correction information holding unit which previously holds correction information indicating a correlation between a position within an image of the respective first and second image sequences, and a position in a shooting location; a correcting unit which makes, based on the correction information held in the correction information holding unit, a spatiotemporal correction for the extraction of the first and second gait information performed by the gait information extracting unit. Thus, even in the case where the frames are obtained by different image sensors, a mismatch, which is detected between the images based on a difference in position in which the respective image sensors are set as well as a difference in shooting direction, is corrected, so that it is possible to judge for the identification of the persons represented by the human images respectively included in the frames obtained by different image sensors.
- The correction information may be, for example, information which specifies grid lines that two-dimensionally section, at regular intervals, a surface on which a person walks in a shooting place.
- In order to achieve the above object, the human searching/tracking apparatus according to the present invention searches for or tracks a specific person in an image sequence obtained by shooting a person, and includes: the human identification apparatus; and a walking sequence storing unit operable to store, in an associated manner, the first and second walking sequences respectively corresponding to first and second gait information, in the case where the gait information verifying unit of the human identification apparatus verifies that the first gait information matches the second gait information. Thus, each of the matched walking sequences is previously analyzed and held so that a person can be searched for or tracked while the image sequence is displayed, which speeds up the processing.
- In order to achieve the above object, the human searching/tracking apparatus according to the present invention searches for or tracks a specific person in an image sequence obtained by shooting a person, and includes: the human identification apparatus; and a display unit operable to display the first and second image sequences received by the image sequence receiving unit of the human identification apparatus, wherein said display unit is operable to display in a highlighted manner, the human images judged by the judging unit of the human identification apparatus to represent the same person, so that the identified human images are distinguished from the other human images, the identified human images being identified from among the human images included in the first and second image sequences. Thus, even when different images are simultaneously displayed, the identified persons can be promptly perceived by displaying the persons in a highlighted manner. This facilitates a search or tracking of a person.
- Another aspect of the human searching/tracking apparatus according to the present invention searches for or tracks a specific person in an image sequence obtained by shooting a person, and includes: a human image judging unit which has: an image sequence receiving unit operable to receive inputs of a first image sequence, and a second image sequence which is obtained in one of the following manners: at a time different from a time when the first image sequence is obtained; and by an image sensor different from an image sensor used for obtaining the first image sequence; a walking sequence extracting unit operable to extract, from the respective received first and second image sequences, first and second walking sequences, each sequence being an image sequence indicating a walking state of a person; a gait information extracting unit operable to extract, from the respective extracted first and second walking sequences, first spatiotemporal period information and first spatiotemporal phase information as first gait information, and second spatiotemporal period information and second spatiotemporal phase information as second gait information, the spatiotemporal period information indicating a temporal or spatial walking period of a person, the spatiotemporal phase information indicating a temporal or spatial phase information in periodic walking movement of a person, and the gait information indicating specifying a periodic motion in human gait; a gait information verifying unit operable to verify between the extracted first gait information and the extracted second gait information; and a judging unit operable to judge whether or not the respective persons represented by the human images included in the first and second image sequences are the same person, based on a result of the verification performed by the gait information verifying unit; and a walking sequence storing unit operable to store, in an associated manner, the first and second walking sequences respectively corresponding to the first and second gait information, in the case where the gait information verifying unit of the human identification apparatus verifies that the first gait information matches the second gait information.
- Yet another aspect of the human searching/tracking apparatus of the present invention searches for or tracks a specific person in an image sequence obtained by shooting a person, and includes: a human identification apparatus which has: an image sequence receiving unit operable to receive inputs of a first image sequence, and a second image sequence which is obtained in one of the following manners: at a time different from a time when the first image sequence is obtained; and by an image sensor used for obtaining the first image sequence; a walking sequence extracting unit operable to extract, from the respective received first and second image sequences, first and second walking sequences, each sequence being an image sequence indicating a walking state of a person;
- a gait information extracting unit operable to extract, from the respective extracted first and second walking sequences, first spatiotemporal period information and first spatiotemporal phase information as first gait information, and second spatiotemporal period information and second spatiotemporal phase information as second gait information, the spatiotemporal period information indicating a temporal or spatial walking period of a person, the spatiotemporal phase information indicating a temporal or spatial phase information in periodic walking movement of a person, and the gait information specifying a periodic motion in human gait; a gait information verifying unit operable to verify between the extracted first gait information and the extracted second gait information; and a judging unit operable to judge whether or not the persons represented by the human images respectively included in the first and second image sequences are the same person; and a display unit operable to display the first and second image sequences received by the image sequence receiving unit of the human identification apparatus, wherein the display unit is operable to display, in a highlighted manner, the human images judged by the judging unit of the human identification apparatus to represent the same person, so that the identified human images are distinguished from the other human images, the identified human images being identified from among the human images included in the first and second image sequences.
- Another aspect of the human identification apparatus of the present invention judges whether or not persons represented by human images respectively included in different image sequences are the same person, and includes: a walking sequence detecting unit operable to detect first and second walking sequences, each sequence being an image sequence indicating a walking state of first or second person respectively included in the image sequences; a walking posture transition estimating unit operable to estimate, based on the walking sequence of the first person, information indicating a transition of a walking posture in a periodic walking movement of the first person at a time or in a position different from a time or a position of the walking sequence of the first person; and a judging unit operable to verify whether or not the estimated information indicating the transition of the walking posture of the first person matches information indicating a transition of a walking posture of the second person, and to judge that the first person and the second person are the same person in the case of verifying that the information match with each other.
- Thus, the judgment for the identification of persons is made based on human gait, so that it is possible to judge for the identification of persons even in the case where a normal detection cannot be temporarily carried out since a person to be searched for or tracked is hidden behind another person or an object, or due to change in lighting conditions. That is to say, the present invention is conceived based on the facts that gait characteristics such as a walking cycle and a length of stride differ from person to person, and that an individual keeps the same gait characteristics. It is therefore possible to identify persons irrespective of one's temporal or spatial position, and also to judge whether the persons represented by the human images respectively included in temporally-distant frames or in frames shot by different cameras are the same person.
- Note that the present invention can be realized not only as such human identification apparatus and the human searching/tracking apparatus as described above, but also as a human image judging method, a human image searching/tracking method, or a program causing a computer to execute such methods, and even as a computer-readable storage medium on which the program is stored.
- The following describes the embodiments of the preset invention with reference to the diagrams.
-
FIG. 2 is a functional block diagram showing a structure of thehuman identification apparatus 10 according to the preset embodiment. Thehuman identification apparatus 10 is an apparatus which judges whether or not persons represented by the human images respectively included in different sequences are the same person based on a continuity of walking sequence of a person. Suchhuman identification apparatus 10 includes a walkingposture detecting unit 200, a walkingstate estimating unit 180 and ajudging unit 190. - The walking
posture detecting unit 200 is a processing unit which detects, in a moving picture, a walking sequence including a predetermined walking posture of a first person. - The walking
state estimating unit 180 is a processing unit which estimates, based on a walking sequence of the first person, a walking state (a transition state of posture in periodic walking movement) of the first person at a time or in a position different from a time or a position of the walking sequence of the first person. - The judging
unit 190 is a processing unit which verifies whether or not a walking state of the first person matches a walking state of a second person, and judges that the first person and the second person are the same person in the case where the walking states match with each other. - The walking
posture detecting unit 200 includes an imagesequence receiving unit 100 and a walkingsequence extracting unit 110. - The walking
state estimating unit 180 includes a spatiotemporal periodinformation extracting unit 120, a spatiotemporal phaseinformation extracting unit 121 and a spatiotemporal positioninformation extracting unit 122. - The judging
unit 190 includes a spatiotemporal periodinformation storing unit 130, a spatiotemporal phaseinformation storing unit 131, a spatiotemporal positioninformation storing unit 132, a spatiotemporalperiod verifying unit 140, a spatiotemporalphase verifying unit 141, a spatiotemporaldifference extracting unit 142, amatching judging unit 150 and acontrol unit 160. - Here, the walking
posture detecting unit 200 is an example of the walking sequence detecting unit which detects a walking sequence being an image sequence indicating a walking state of a human image included in the image sequence. The walkingstate estimating unit 180 is an example of the walking posture transition estimating unit which estimates, based on the detected walking sequence, information indicating a transition of a walking posture in a periodic walking movement at a time or in a position different from a time or a position of the detected walking sequence. The judgingunit 190 is an example of the judging unit which judges whether or not the information indicating respective transitions of two persons shot at different times or with different image sensors, and judges whether or not the two persons are identified as the same person. Note that “information indicating a transition of walking posture” is information which includes the period information and phase information which will be mentioned later. - The image
sequence receiving unit 100 is an example of the image sequence receiving unit which receives inputs of first and second image sequences respectively obtained at a different time or with a different image sensor. - The walking
sequence extracting unit 110 is an example of the walking sequence extracting which extracts first and second walking sequences, each being an image sequence indicating a walking state of a person based on the respective first and second image sequences. - The spatiotemporal phase
information extracting unit 121, the spatiotemporal positioninformation extracting unit 122 and the spatiotemporal periodinformation extracting unit 120 are examples of the gait information extracting unit which extracts first and second gait information, each being information which specifies a periodic walking movement of a person, based on the respective first and second walking sequences. - The spatiotemporal
phase verifying unit 141, the spatiotemporaldifference extracting unit 142 and the spatiotemporalperiod verifying unit 140 are examples of the gait information verifying unit which verifies between the extracted first gait information and the extracted second gait information. - The
matching judging unit 150 is an example of the judging unit which judges whether or not the human images respectively included in the first and second image sequences represent the same person, based on the result of the verification. - The image
sequence receiving unit 100 is a signal interface, or the like, which obtains an image sequence from a camera or an image storing apparatus. The “image sequence” is an array in which shot frame images are arranged along a temporal axis, as shown inFIG. 3 . - The walking
sequence extracting unit 110 is a processing unit which extracts a walking sequence from the image sequence obtained by the imagesequence receiving unit 100. The “walking sequence” is a sequence of a walking state obtained from a walking movement area in each frame image.FIG. 4B shows an example of such walking sequence.FIG. 4A shows a sequence of lower body areas, each being extracted from each frame, and the respective extracted areas are sequentially arranged in time order.FIG. 4B is a graph obtained by arranging the extracted areas along a temporal axis at a dashed line B10 as shown inFIG. 4A . A black band inFIG. 4B shows a trace of moving toes (obtained by arranging respective images of each leg along the temporal axis). A method of calculating each walking sequence will be described later. - The spatiotemporal period
information extracting unit 120 is a processing unit which extracts period information being a spatiotemporal change in walking, based on the walking sequence extracted by the walkingsequence extracting unit 110. The “period information” is a result obtained by frequency number analysis made with respect to strides per predetermined period of time or predetermined distance and spatiotemporal positional change of a specific body part such as feet and hands, or a form of the smallest pattern to be repeated spatiotemporally and periodically.FIG. 4B shows an example of such spatiotemporal period information. InFIG. 4B , a black or white triangular mark denotes a spatiotemporal point at which the toes cross in temporal axis and a horizontal axis (as opposed to spatial axis) of the image. The spatiotemporal period information may be the number of strides per predetermined period of time or predetermined distance obtained based on spatial intervals between the neighboring black triangular marks as well as temporal intervals between the neighboring white triangular marks. In addition, a form per se of walking pattern surrounded by a dashed line linking the neighboring black triangular marks and a dashed line linking the neighboring white triangular marks (FIG. 4C ), or a frequency characteristic (spectral intensity of a specific frequency band) obtained by frequency analysis made with respect to a temporal change in stride length as shown in a curved line B11 inFIG. 4B , and a spatial change in stride length as shown in a curved line B12 may be also used as the spatiotemporal period information. - The spatiotemporal period
information storing unit 130 is a memory for storing the spatiotemporal period information extracted by the spatiotemporal periodinformation extracting unit 120, together with the time and the position in the image which are detected. The spatiotemporalperiod verifying unit 140 is a processing unit which verifies between the spatiotemporal period information extracted by the spatiotemporal periodinformation extracting unit 120 and the spatiotemporal period information held in the spatiotemporal periodinformation storing unit 130. - The spatiotemporal phase
information extracting unit 121 is a processing unit which extracts phase information indicating a spatiotemporal change in walking, based on the walking sequence extracted by the walkingsequence extracting unit 110. The “phase information” means a transition state (e.g. a position or time in which a predetermined walking posture is taken, or a walking posture in a specific position or at a specific time) during the periodic exercise of walking. The spatiotemporal phase information, for example, is information indicating a spatiotemporal position in which a foot touched the ground (a predetermined walking posture) for the walking sequences of the same spatiotemporal period. The spatiotemporal phase information indicates also a difference in walking posture between two walking sequences compared at a same time or in a same position. An example of such spatiotemporal phase information will be described usingFIGS. 5, 6A through 6C and 7A through 7C.FIG. 5 shows a walking trace of a foot position, as shown inFIG. 4B , but shows different walking traces A010 and A011 of different two persons who have the same spatiotemporal period but a different spatiotemporal phase. Comparing the walking traces A010 and A011, a stride length and a walking period are the same, but a position or a time in/at which a foot touches the ground or a position or a time in/at which the legs are crossed are different.FIG. 6A shows two walking traces (a walking trace 1802 (in a dashed line) of a first person and a walking trace 1801 (in a full line) of a second person) which are detected as in discontinuity because of an obstacle 1800 (a hatched area). A case of detecting phase information based on the respective walking traces is considered here. Phase information, as is already described above, is a position or a time in/at which a predetermined walking posture is taken or a walking posture in a predetermined position or at a predetermined time. As for the walking traces of different times or positions, as in the case of thewalking track 1801 of the second person and thewalking track 1802 of the first person, phase information is derived based on the same criterion. In such case, the spatiotemporal phaseinformation extracting unit 121 estimates a walking posture at a time or in a position except for the time or the position at/in which a frame is shot, based on a position, a time or a periodicity of the predetermined posture (e.g. crossing of the legs) in thegait information 1802 of the first person (a dashedline 1803 inFIG. 6B ). Similarly, inFIG. 6C , the spatiotemporal phaseinformation extracting unit 121 estimates a walking posture (a dashed line 1805) at a different time or in a different position based on a walking trace 1804 (in a dashed line) in the right of theobstacle 1800. - The spatiotemporal phase
information extracting unit 121 then obtains a time or a position at/in which the predetermined walking posture is taken. Here, crossing of the legs (a state in which a stride length becomes the smallest) is assumed to be the predetermined posture.FIG. 7A shows a temporal or positional change in stride length. The stride length can be derived, based on images, as a width between the tracings of the respective legs in a walking trace. As shown inFIG. 7B , the spatiotemporal phaseinformation extracting unit 121 estimates a state (posture) 1903 a (in a dashed line) of the stride length at a time or in a position at/in which a person is not shot due to anobstacle 1900 a (a hatched area), based onstride length information 1902 a (a dashed line) of the first person. The spatiotemporal phaseinformation extracting unit 121 obtains, as phase information, an area which is shot at an earlier time and in the left within the image. The time and the position are a time and a position at/in which the human image takes the predetermined posture. InFIG. 7B , a time orposition 1905 is obtained as phase information for an estimated walkingstate 1903 a of the first person, while a time orposition 1906 is obtained for the walkingstate 1901 a of the second person. Similarly, inFIG. 7C , phase information 1910 is obtained for awalking state 1909 a (a dashed line) estimated for the walkingstate 1908 a (a dashed line), whilephase information 1911 is obtained for a walking state 1907 a (a full line). - The spatiotemporal phase
information extracting unit 121 obtains a walking state or an estimated walking state at a predetermined time or in a predetermined position also in the case of using, as phase information, a posture at a predetermined time or in a predetermined position. In the case ofFIG. 7B , the spatiotemporal phaseinformation extracting unit 121 obtains a stride length (walking posture) at a predetermined time or in apredetermined position 1904. The phase information for the first person shall be a value indicating an estimated walkingstate 1903 a (a dashed line) at the predetermined time orposition 1904, whereas the phase information for the second person shall be a value indicating the walkingstate 1901 a (a full line) at the predetermined time orposition 1904. Similarly, inFIG. 7C , the value indicating an estimated walkingstate 1909 a at the predetermined time orposition 1904, which is estimated based on the walkingstate 1908 a, and the value indicating the walking state 1907 a at the predetermined time orposition 1904 are the phase information of the first person and the second person, respectively. - In the above description, an estimated posture is used only for the first person, however, an estimated posture may be obtained for the second person so that phase information is obtained for both the first and second persons based on the estimated postures. A location of an obstacle in
FIGS. 6A through 6C and 7A through 7C and an area except for the area indicated in the diagram may be determined as fixed parts, so as to obtain phase information after deriving an estimated state in the fixed parts. - It should be noted that other state, such as a position at which a stride length becomes the largest and a state in which a change in stride length becomes the largest, may be used as the predetermined posture.
- The spatiotemporal phase
information storing unit 131 is a memory for storing the spatiotemporal phase information extracted by the spatiotemporal phaseinformation extracting unit 121 together with a time as well as a position in an image which are detected. The spatiotemporalphase verifying unit 141 is a processing unit which verifies between the spatiotemporal phase information extracted by the spatiotemporal phaseinformation extracting unit 121 and the spatiotemporal phase information held in the spatiotemporal phaseinformation storing unit 131. - The spatiotemporal position
information extracting unit 122 refers to the walking sequence extracted by the walkingsequence extracting unit 110, extracts a spatiotemporal position at which the walking sequence is extracted, and generates spatiotemporal position information. The “spatiotemporal position information” means a time or a place where the walking sequence is detected. An example of such spatiotemporal position information is described inFIG. 5 . InFIG. 5 , a cross-shape in dashed line presents a spatiotemporal position at which the legs are crossed for the first time with regard to respective two walking traces. Thus, the spatiotemporal position information is information indicating an absolute position of spatiotemporal walking. - The spatiotemporal position
information storing unit 132 is a memory for storing the spatiotemporal position information generated by the spatiotemporal positioninformation extracting unit 122. The spatiotemporaldifference extracting unit 142 is a processing unit which obtains a difference between the spatiotemporal position information generated by the spatiotemporal positioninformation extracting unit 122 and the spatiotemporal position information held in the spatiotemporal positioninformation storing unit 132. - The
matching judging unit 150 is a processing unit which judges whether or not different walking sequences match with each other, based on the results obtained by the spatiotemporalperiod verifying unit 140, the spatiotemporalphase verifying unit 141 and the spatiotemporaldifference extracting unit 142. That is to say, thematching judging unit 150 judges whether or not the walking sequences are of the same person. An example of the method of judging on matching with regard to the spatiotemporal period information is as follows. In the case where the number of strides x within a predetermined period and the number of strides y within a predetermined distance are used as the spatiotemporal period information, thematching judging unit 150 judges that the two walking sequences match with each other, when the following equations are satisfied: |Z1−Z2|<θ, where |Z1−Z2| denotes a difference between a vector Z1=(x1,y1) obtained for a pair of x and y based on one of the walking sequences to be judged for verification and a vector Z2=(x2,y2) obtained based on the other walking sequence, and θ denotes a predetermined threshold; and |x1−x2|<θx, |y1−y2|<θy (θx and θy are predetermined values). In other words, thematching judging unit 150 judges that the walking sequences are of the same person. - The
matching judging unit 150 judges the respective spatiotemporal phase information and spatiotemporal position information in the same way as in the case of the spatiotemporal period information, and in the case where the information match with each other for all the items or the specified number of items, thematching judging unit 150 judges that the two walking sequences match with each other. It should be noted that the judging unit is not limited to the one described above, and a method used for pattern recognition may be generally applied. - For example, in the case of using spatiotemporal phase information, the temporal phase information tp1 and the spatial phase information (phase information of position) sp1 which are obtained from the walking sequence of the first person in the moving picture are compared with the temporal phase information tp2 and the spatial phase information (phase information of position) sp2 which are obtained based on the walking sequence of the second person at a time or in a position different from the time and the position at/in which the phase information tp1 and sp1 are obtained. In such case, the
matching judging unit 150 judges that the first person and the second person are the same person in the case where one or both of the following equations is/are satisfied: |tp1−tp2|<θt (θt is a predetermined threshold); and |sp1−sp2|<θs (θs is a predetermined threshold). In the case ofFIG. 6B , for example, by appropriately setting θt and θs, thematching judging unit 150 verifies that the spatiotemporal phase information of the walkingstate 1801 matches the spatiotemporal phase information of the walkingstate 1803, and judges that the first person and the second person are the same person. In the case ofFIG. 6C , thematching judging unit 150 verifies that the spatiotemporal phase information of the walkingstate 1801 does not match the spatiotemporal phase information of the walkingstate 1803, and judges that the first person and the second person are not the same person. InFIG. 7B , thematching judging unit 150 verifies that the spatiotemporal phase information of the walkingstate 1901 a matches the spatiotemporal phase information of the walkingstate 1909 a, and judges that the first person and the second person are the same person. InFIG. 7C , thematching judging unit 150 verifies that the spatiotemporal phase information of the walking state 1907 a does not match the spatiotemporal phase information of the walkingstate 1909 a, and judges that the first person and the second person are different persons. - The
control unit 160 is a processing unit which performs display of an image sequence used for verification, as a control based on the result of the judgment made by thematching judging unit 150.FIGS. 8A through 8C show examples of such display.FIG. 8A shows an example of magnified human image areas, one in the left and the other in the right, of the two image sequences in comparison, displayed together with shooting time and place of each image sequence.FIG. 8B shows an example of the result as same as shown inFIG. 8A , displayed together with a trace of human movement (arrows in the diagram). Note that in these diagrams, shooting time and location are displayed in characters in accordance with an image sequence, however, a map may be displayed for the shooting location and a shooting position and a trace of the movement may be displayed in a superimposed manner.FIG. 8C shows an example of a display in which the area including a verified human image is highlighted with a rectangle in a dashed line, in the case where plural persons are included within an image. Note that thecontrol unit 160 may control not only display but also storage, into an external storage apparatus (e.g. position information storing apparatus), of matching information which indicates that different walking sequences are verified to match with each other and thus correlates the walking sequences. By thus storing the information indicating matching/mismatching of different walking sequences, such information can be utilized for searching for or tracking a human walking image. -
FIG. 9 shows an example of a storage format showing the result of a search or a tracking of a walking sequence by use of the process as described above.FIG. 9 presents three types of walking sequence information and information of five items for one walking sequence. The five items are sequence number, spatiotemporal period information, spatiotemporal phase information, spatiotemporal position information, and matched sequence number. A sequence number is an ID number assigned to respective walking sequences which are shot at different times with different cameras. The spatiotemporal period information presents, as (x,y), the number of strides x within a predetermined period and the number of strides y within a predetermined distance. The spatiotemporal phase information presents an amount of movement to a temporal or spatial position at which the legs are crossed for the first time since a temporal or spatial position to be used as a reference is passed with a ratio where a time and a movement amount required for a sequence of one stride are respectively determined as 1.0. For instance, in the case ofsequence number 1, the legs are crossed at a time when a time required for 0.5 stride has passed from a reference time and after a distance required for 0.1 stride is walked from a reference position. - The spatiotemporal position information indicates time and place at which a walking sequence is detected for the first time. The place is represented by pixel coordinator values in vertical and horizontal directions within an image.
- Matched sequence number indicates a sequence number of the walking sequence verified to match with a different walking sequence in a search or a tracking. In the case where no sequences are verified to match, the sequence number may be 0, and in the case where plural pairs of verified sequences are detected, plural sequence numbers may be described. By saving information indicating the result of the search or tracking, it is possible to omit the verification judgment by referring to the information, in the case of performing the same search or tracking.
- Note that the sequence number may be associated with a reference destination (e.g. a file name and a storage address) of a shot image. As shown in
FIG. 9 , information about a spatial location in the spatiotemporal position information may be represented by coordinator system that is set for exclusive use, or a general coordinator such as latitude and longitude. -
FIG. 10 is a block diagram showing a structure of the human searching/tracking apparatus 20 onto which thehuman identification apparatus 10 is applied. The human searching/tracking apparatus 20 is a concrete example of a system or an apparatus which realizes a method of searching/tracking a person in an image sequence, according to the present embodiment. Such human searching/tracking apparatus 20 includescameras clock 1030, astoring apparatus 1040, aprocessing apparatus 1050, adisplay apparatus 1060, aninput unit 1070 and apointing device 1080. - The
cameras sequence receiving unit 100, and shoot images of aperson 1000. Theclock 1030 is a timer for obtaining a shooting time. Thestoring apparatus 1040 is a hard disk which stores the images shot with thecameras clock 1030 and the results of a search or a tracking of a person. Theprocessing apparatus 1050 is an apparatus which searches for or tracks a person in the image sequence obtained by thecameras storing apparatus 1040, and is equivalent of thehuman identification apparatus 10 shown inFIG. 2 . Thedisplay apparatus 1060 is a display which displays a result of the processing performed by theprocessing apparatus 1050, while theinput unit 1070 is a keyboard used for instructions on a search or tracking and thepointing device 1080 is a mouse used for the instructions on the search or tracking.FIG. 11 shows an example of pointing. In the case of searching for or tracking a person, aperson 1091 to be searched/tracked is specified using apointer 1090, for searching for or tracking a walking sequence as same as that of theperson 1091. - Each of the components is connected via a communication path. The communication path may be wired or wireless, and may include a private line and a public line
- The following describes the operation of the
human identification apparatus 10 described above in the present embodiment, with reference to a flowchart shown inFIG. 12 . Note that, in this embodiment, thehuman identification apparatus 10 searches for or tracks a human image included in animage sequence 2 obtained at a different time but with the same camera as used for theimage sequence 1.FIGS. 13A and 13B respectively show examples ofimage sequences FIG. 13A shows animage sequence 1 in which aperson 502 walks toward right on the left side of anobstacle 501.FIG. 13B shows animage sequence 2 obtained by shooting the same place with the same camera as theimage sequence 1 after ten seconds have passed since theimage sequence 1 is shot. Theimage sequence 2 shows theobstacle 501 andpersons - Firstly, the image
sequence receiving unit 100 receives an image sequence 1 (Step S401). Then, a walking sequence (walking sequence 1) is extracted from the image sequence 1 (Step S402). In the description, a sequence of lower body area as shown inFIG. 4A is used as a walking sequence. - An example of the extraction of a walking sequence performed by the walking
sequence extracting unit 110 will be described with reference toFIG. 14 . Firstly, the walkingsequence extracting unit 110 reads one frame image from the image sequence (Step S601). A frame image is read from among the unread frame images in time order. The walkingsequence extracting unit 110 then detects a human image area from the read-out frame image (Step S602). For the detection of a human image area, method such as inter-frame difference method that is generally used for detecting a moving object and a difference background method for calculating a difference between the human image area and a previously-provided background image which does not include any human images. Even in the case of using still pictures, a template image of human may be prepared so that it is possible to detect a human image by extracting an area with high similarity to the template image, using similarity calculation (a template matching method such as a simple difference calculation and a normalization correlation calculation). The walkingsequence extracting unit 110 then extracts, from the human image area, information indicating a walking state (Step S603). Such information indicating a walking state may be information indicating a transition of time for a walking state such as trace information, as shown inFIG. 4B , indicating a tracing of toes. - Note that Steps S602 and S603 may be simultaneously performed as a sequence of process or a result of processing Step S602 may be considered as a result of processing Step S603. Instead of extracting explicitly a whole human image area as in Step S602, processing may be executed so that an output of Step S603 can be directly obtained. For example, for the lower body image area as described above, the walking state information may be directly obtained using the template matching method where the lower body image is used as a template. The walking
sequence extracting unit 110 eventually judges whether or not the frame image that has just been read is the last frame (Step S604), and ends the processing of extracting a walking sequence in the case where the image is the last frame, or returns to Step S601 in the case where the last frame image still remains to be read. - Next, the spatiotemporal period
information extracting unit 120, the spatiotemporal phaseinformation extracting unit 121 and the spatiotemporal positioninformation extracting unit 122 respectively extracts spatiotemporal period information, spatiotemporal phase information and spatiotemporal position information, based on thewalking sequence 1 obtained in Step S402 (Step S403). - An example of the extraction of the three types of information described above based on the walking sequence shown in
FIG. 4B will be described with reference toFIGS. 15 and 16 A through 16C. - Firstly, the spatiotemporal period
information extracting unit 120 and others detect a position of a specific walking state based on the walking sequence information (Step S701). The description of the specific walking state will be described with reference toFIGS. 16A through 16C .FIG. 16A shows a result of the detection of a specific walking state with respect to the walking sequence shown inFIG. 4B . Two black wave-like bands, shown inFIG. 16A , which cross with each other show a temporal trace of moving the toes. A horizontal axis presents a position of an image in a horizontal direction whereas a vertical line presents time. A specific walking state position is defined as a position at which the two bands cross the legs are crossed, namely, a position indicating a state in which the legs are crossed (a position indicated by respective dashed lines in the diagram). The crossing point can be detected by performing template matching or correlation operation using form patterns of the crossing section as shown inFIG. 16B which are provided beforehand. An example of the detecting process is shown inFIG. 16C . InFIG. 16C , a degree of matching in form is calculated by shifting a position of adetection template 801 with respect to awalking trace 800. When a value indicating a predetermined value or greater is obtained for the degree of matching in the verification, the walkingtrace 800 is determined to be the specific walking state. Thus, the crossing points of dashed lines, as shown inFIG. 16A , can be obtained. - Note that the specific walking state is not restricted to the state in which the legs are crossed, and a state in which the legs are widened the most may be set as the specific walking state. Such state, in the walking trace shown in
FIG. 16A , is a position at which a gap between the two bands is the widest (as indicated by respective dash-dot lines in the diagram). Step S701 is performed until all the specific walking states are detected, and the process goes on to Step S703 after that (Step S702). - The spatiotemporal period
information extracting unit 120 then generates spatiotemporal period information by calculating an interval between the detected specific walking state positions (Step 703). The period information may be derived separately without obtaining such specific walking state positions, using the methods such as a Fourier transform, a wavelet transform and an auto-correlation method. In this case, these methods can be applied to a temporal change in a position of the wave-like bands shown inFIG. 16A or a spatiotemporal change in a gap between the two bands. As for the spatiotemporal period information, a form of the smallest unit of a periodic walking trace, as shown inFIG. 16A , may be used as a pattern. In such case, one wave form made up of two bands within the range sectioned by horizontal and vertical dashed lines may be used as a pattern. - Then, the spatiotemporal phase
information extracting unit 121 derives spatiotemporal phase information (Step S704). The spatiotemporal phase information represents an amount of spatiotemporal movement to a spatiotemporal position at which the legs are crossed (where a specific walking state appears) for the first time after a reference spatiotemporal position is passed. In the case of using, as the spatiotemporal phase information, a time or a position at/in which a predetermined posture is taken, a time or a position at/in which a stride length becomes the shortest is defined, inFIG. 7B , to be such spatiotemporal phase information. Thus, for a walking 1902 a, avalue 1905 derived based on an estimated walkingstate 1903 a is defined as temporal phase information or spatial phase information, whereas for a walking 1901 a, avalue 1906 is defined to be such phase information. In the case of using a posture at a predetermined time or position, as the spatiotemporal phase information, a value 1904 (a value indicative of stride length) of an estimated walkingstate 1903 a is defined, for the walking 1902 a, as temporal phase information or spatial phase information, while thevalue 1904 is defined for the walking 1901 a as such phase information. - Looking at an example shown in
FIG. 5 , in the case of setting an upper left corner of the walking trace graph to be a reference temporal axis, a time between the upper left corner and a position at which the legs are crossed for the first time (a crossing point of the dashed lines) becomes the temporal phase information. Also, in the case of setting a vertical line A013 as a reference spatial position, a gap between the vertical line A013 and the position, which is the nearest to the vertical line A013 on the right side of the line A013 and at which the legs are crossed, becomes the spatial phase information. The way of representing such movement amount is not limited to the above examples, and the amount may be presented in a relative size based on a time required for a sequence of one step and its movement amount. - It should be noted that the spatiotemporal phase information means phase information indicating an appearance of a specific walking state within an image, so that even in the case where the walking sequences have the same spatiotemporal period, different values may be obtained for the walking patterns whose spatiotemporal timings in a state of putting down a foot on the ground or crossing the legs are different. In
FIG. 5 , the walking sequences A011 and A012 have the same spatiotemporal period of walking, however, the spatiotemporal phase information as defined above respectively indicates a different value. - Next, the spatiotemporal position
information extracting unit 122 derives spatiotemporal position information (Step S705). The spatiotemporal position information is defined as a time and a position in image in which the first specific walking state is detected. The spatiotemporal position information is information indicating an absolute spatiotemporal position coordinate of the detected position of the specific walking state, and a position at which the second step is detected or a position at which the last step is detected may be used instead. - By detecting a specific walking state, it is possible to obtain not only spatiotemporal period information of the walking but also its spatiotemporal phase information and spatiotemporal position information.
- The spatiotemporal period information, the spatiotemporal phase information and the spatiotemporal position information thus obtained in Step 403 are respectively stored into the spatiotemporal period
information storing unit 130, the spatiotemporal phaseinformation storing unit 131 and the spatiotemporal position information storing unit 132 (Step S404). - Then, the image
sequence receiving unit 100 obtains animage sequence 2 for searching a person, as in Step S401 (Step S405). The walkingsequence extracting unit 110 extracts awalking sequence 2 from the image sequence 2 (Step S406). The imagesequence receiving unit 100 then judges whether or not another walking sequence can be found, as a result of Step S406 (Step S407). In the case where no sequence is found (No in Step S407), the walkingsequence extracting unit 110 ends the processing. In the case otherwise (Yes in Step S407), the spatiotemporal periodinformation extracting unit 120, the spatiotemporal phaseinformation extracting unit 121 and the spatiotemporal positioninformation extracting unit 122 extract, as in the case of thewalking sequence 1, spatiotemporal period information, spatiotemporal phase information and spatiotemporal position information, from the walking sequence 2 (Step S408). - The spatiotemporal
period verifying unit 140, the spatiotemporalphase verifying unit 141 and the spatiotemporaldifference extracting unit 142 respectively verify between the spatiotemporal period information, the spatiotemporal phase information and the spatiotemporal position information of thewalking sequence 1 which are stored in Step S404 and the spatiotemporal period information, the spatiotemporal phase information and the spatiotemporal position information extracted in Step 408 (Step S409). All the three types of information may be used or either of the spatiotemporal period information and the spatiotemporal phase information may be used for the verification. - A reference degree of matching may be pre-set for the respective three types of information, and in the case where all the three information satisfy the reference degree, it may be judged that the sequences match with each other. For example, in comparing the sequences of different expressions as shown in
FIG. 9 , in the case where the following equations are both satisfied: |x1−x3|=Dx<θx; and |y1−y3|=Dy<θy, where each spatiotemporal period information is expressed as (x1, y1) and (x3, y3), it is judged that the sequences match with each other. Here, θx and θy are predetermined thresholds. Similarly, in the case where the following equations are satisfied: |w1−w3|=Dw<θw; and |z1−z3|=Dz<θz, where each spatiotemporal phase information is expressed as (w1, z1) and (w3, z3), it is judged that the sequences match with each other. Here again, θw and θz are predetermined thresholds. In the case where the following equations are satisfied: ∥t1−t3|=Dt<θt; and (xx1−xx3)*(xx1−xx3)+(yy1−yy3)*(yy1−yy3)=Dxy<θd, where each spatiotemporal position information is expressed as t1, (xx1, yy1) and t3, (xx3, yy3), it is judged that the sequences match with each other. Here, θt and θd are predetermined thresholds. - The three types of information described above may be associated with each other for judging the verification. For example, each sum or product of the differences Dx and Dy in spatiotemporal period information, of the differences Dw and Dz in spatiotemporal phase information and of the differences Dt and Dxy in spatiotemporal position information may be compared with the predetermined reference degree of matching. For example, in the case where (Dx+Dy+Dw+Dz+Dt+Dxy)<θsum and (Dx×Dy×Dw×Dz×Dt×Dxy)<θmul are both satisfied, it is judged that the sequences match with each other. Here, θmul is a predetermined threshold. The reference that is set for a difference in spatiotemporal period information and a difference in spatiotemporal phase information may be changed based on the difference in spatiotemporal position information. A difference in spatiotemporal position information gets larger since a value indicating spatiotemporal period and a value indicating spatiotemporal phase can vary as the spatiotemporal difference gets larger. Therefore, by reducing, as the difference in spatiotemporal position information increases, the reference references of matching for the difference in spatiotemporal period information and the difference in spatiotemporal phase information, it is possible to decrease the number of undetected sequences. For example, the differences Dt and Dxy in spatiotemporal position information satisfies Dt>θt or Dxy>θxy, the thresholds θx, θy, θw and θz are multiplied by α (α is a constant indicating 0 or greater) for judging the verification.
- In the case of using spatiotemporal period information, a value indicating the spatiotemporal period information does not easily change although a direction in which a person faces with respect to a camera changes, it is effective in that a search or tracking can be easily carried out between the temporally-distant image sequences.
- In the case of using spatiotemporal phase information, it is effective in that it is easy to distinguish the walking sequences of different persons, having the similar spatiotemporal period.
- Then, the
matching judging unit 150 judges whether or not thewalking sequence 1 matches thewalking sequence 2, based on the result of the verification performed in Step S409 (Step S410). In the case where the sequences do not match with each other (No in Step S410), thematching judging unit 150 returns to Step S406 and obtains a new walking sequence (Step S406). In the case where the sequences match with each other (Yes in Step S410), thecontrol unit 160 displays theimage sequence 1 and theimage sequence 2 on thedisplay apparatus 1060, and also displays, in a highlighted manner, the human image areas of the respective matched walking sequences. InFIG. 17 , theimage sequence 1 is displayed in anarea 1130 and theimage sequence 2 is displayed in anarea 1140, while the human image areas in the respective matched walking sequences are displayed, in a highlighted manner, using circumscribedrectangles - The spatiotemporal period information (i.e. walking pattern), in general, changes subjected to various elements such as personal habits (e.g. walking habit), individual circumstances (e.g. in a hurry, walking slowly), a type of footwear (e.g. a height of heel, and a movable range of ankle), belongings (e.g. whether something heavy is carried in the hand), clothes (e.g. a movable range of legs) and road surface conditions (e.g. slipperiness and slope). It is therefore difficult to identify an individual based on the spatiotemporal period information. By deriving a difference in spatiotemporal position between the two walking sequences to be verified, using the spatiotemporal position information, and allowing the reference degree of matching, which is used for judging whether or not the sequences match with each other, to vary depending on the difference in spatiotemporal position, it is possible to reflect, in the verification, the degree at which a place or a footwear, clothes and belongings may spatiotemporally vary, so as to prevent wrong verification. As in the examples shown in
FIGS. 13A and 13B , when a spatiotemporal difference is no more than ten seconds, a walking sequence with smaller differences in both spatiotemporal period information and spatiotemporal phase information may only be searched for or tracked. For example, when Dt is shorter than ten seconds, the thresholds θx, θy, θw and θz are multiplied by β (β is a constant less than 1.0) for judging the matching between sequences. With the method as described above, it is possible to appropriately select for aperson 502 in the image sequence shown inFIG. 13A , between the two persons in the image sequence shown inFIG. 13B , aperson 503 whose information about the walking sequence corresponds to the information about the walking sequence of theperson 502. - As has been described so far, according to the embodiment, by detecting the spatiotemporal period information, the spatiotemporal phase information and the spatiotemporal position information obtained from a walking sequence, and judging whether the walking sequence matches a different walking sequence, based on the information, it is possible to search/track, with less influence by the size of a human image and a direction in which the human image faces in an image, a person presented by the human image included in an image sequence obtained at a different time.
- It should be noted that, in the embodiment, the matching of walking sequences is performed using both the spatiotemporal period information and the spatiotemporal phase information, however, it is possible to see if the walking sequences match with each other by using one of the above information, and thus it is possible to obtain the same effect as can be obtained with the present invention. With the combination of the both information, it is possible to perform more sophisticated verification and thereby to improve accuracy in a search or tracking.
- Also, it is desirable that an image sequence to be received by the image
sequence receiving unit 100 be a time length which includes a walking distance of at least one step or the number of frames, since the improvement of the verification accuracy can be expected when the number of steps is greater than one. As such time length, it is also preferable that an image sequence last a half second or longer (i.e. 15 frames or more, as based on 30 frames per second), and that a specific walking state be detected two times or more. - It should be also noted that the first embodiment described the example of obtaining an image sequence directly from a camera, however, the same effect can be obtained by reading out an image sequence stored in the
storing apparatus 1040 or the like. - The following describes the second embodiment with reference to
FIGS. 18A, 18B and 19. - The structure of the human identification apparatus according to the second embodiment is basically the same as the one described in the first embodiment, however, the present apparatus searches for or tracks a person, using image sequences obtained by two different cameras, as an
image sequence 1 and animage sequence 2 respectively.FIGS. 18A and 18B show examples of images.FIG. 18A shows theimage sequence 1 whileFIG. 18B shows theimage sequence 2. In therespective image sequences obstacle 900 is displayed in the form of cuboid. In theimage sequence 1 shown inFIG. 18A , aperson 922 is hidden by theobstacle 900 and only aperson 912 is presented. In theimage sequence 2 shown inFIG. 18B , aperson 921 and theperson 922 are presented. Theperson 912 corresponds to theperson 922, having the same height and the same clothes, however, in theimage sequence 2, thepersons person 912 is theperson 921 or theperson 922 based on color, texture and motion vector. As shown inFIGS. 18A and 18B, the ground is displayed in a lattice-like design using dashed lines for indicating a position coordinate to be used for verification between respective position information of the two image sequences. The lattice form in dashed lines can be obtained by an actual measurement of correspondence (spatiotemporal correction information) between a position within a camera screen and a position in a shooting location or by geometric calculation based on camera placement and optical specification. The lattice shown inFIG. 18A and the lattice shown inFIG. 18B present positions to be corresponded. Such coordinate conversion between corresponding points on different planes within the image can be represented in a homography H, therefore, the matrix H shall be calculated and held. As for the method of deriving such homography H, the conventional method, as introduced in “Optimal homography computation with a reliability measure” in IEICE Transactions on Information and Systems written by K. Kanatani, N. Ohta and Y. Kanazawa, Vol. E83-D, No. 7, pp. 1369-1374 (July 2000), may be used. -
FIG. 19 is a function block diagram showing a structure of ahuman identification apparatus 15 according to the second embodiment. Thehuman identification apparatus 15 includes aspatiotemporal correction unit 170 in addition to the components included in thehuman identification apparatus 10 according to the first embodiment. Thespatiotemporal correction unit 170 is a processing unit which corrects spatiotemporal discordance between different image sequences using spatiotemporal correction information, when extracting a walking sequence or deriving spatiotemporal period information, spatiotemporal phase information and spatiotemporal position information. Thespatiotemporal correction unit 170 is an example of a correction unit which performs spatiotemporal correction in the extraction of gait information. - The human
image judging unit 15 can process, in the same manner as described in the first embodiment, a video taken by a different camera, so as to identify, in theimage sequence 2, a human image corresponding to the human image included in theimage sequence 1. A location of a target object within an image differs due to the difference in placement of camera, therefore, grid lines, as spatiotemporal correction information, is previously held for each camera (or each image sequence) so that the grid lines can be used as correction information of spatial position coordinate for the extraction of a walking sequence. That is to say, thespatiotemporal correction unit 170 holds, as correction information, information which specifies grid lines that two-dimensionally sections at regular intervals, a surface on which a person walks in a shooting location. - By performing correction using the spatiotemporal correction information, the
spatiotemporal correction unit 170 can verify spatiotemporal period information, spatiotemporal phase information and spatiotemporal position information between the images taken by different cameras. For example, thespatiotemporal correction unit 170 spatiotemporally corrects the spatiotemporal period information, the spatiotemporal phase information and the spatiotemporal position information according to ratios of each side and dimension of a small area sectioned by the grid lines (i.e. multiply by a proportional factor). The spatiotemporal position, at which a person is in a state of taking a specific walking posture for the first time after passing a specific grid line, may be used as the spatiotemporal phase information. With such structure, it is possible to identify a person (not shown) hidden behind theobstacle 900 with theperson 921, and also, theperson 921 with theperson 922. - In the above example, a correspondence of the positions is used for extracting a walking sequence; however, an image in one image sequence may be converted before other processing so that a position of the image in one image sequence corresponds to a position of an image in the other image sequence. For the image conversion, by applying the homography H as described above to a pixel position on the planar surface (i.e. ground) on which a human image walks, it is possible to convert the pixel position into a position on a planar surface (i.e. ground) within the other corresponding image sequence. Therefore, an image can be converted by performing conversion for all the pixels in the same way. By deriving the spatiotemporal period information, the spatiotemporal phase information and the spatiotemporal position information after the correction using positional relationship, it is possible to correspond different walking sequences without being affected by the placement of camera.
- Even in the case where frame rates are different between two image sequences, it is possible to search for or track a person over image sequences of different frame rates by one of the following operations performed by the spatiotemporal correction unit 170: performing, in advance, frame rate conversion to the frame rate of one image sequence so that the image sequences have the same frame rate; performing such frame rate conversion at the time of extracting a walking sequence; and correcting time information when deriving spatiotemporal period information, spatiotemporal phase information and spatiotemporal position information. For example, in the case where a frame rate of one image sequence is 15 frames per second while a frame rate of the other image sequence is 30 frames per second, a frame is taken out for every two frames from the latter image sequence so as to generate an image sequence at the frame rate of 15 frames per second, and then, detection of walking sequences and verification of the walking sequences between the image sequences are performed.
- In the case above, spatiotemporal correction information of frame rate necessary for correction may be previously held in association with each image sequence, in a storing apparatus or the like.
- Generally speaking, when a person is shot by different cameras, how the person looks like in an image greatly differs, in many cases, due to difference in a direction in which the person faces, lighting conditions and characteristics of a camera. It is therefore difficult to appropriately search for or track the person with the tracking method using colors, a continuity of motion vectors, and image patterns of human image area. However, by using the method according to the embodiment, it is possible to search for or track a person presented by the human images included in different image sequences, using the spatiotemporal period information and the spatiotemporal phase information obtained from a walking sequence of the person, even in the case where the human image to be judged is presented by a different color, or faces toward a different direction or has a different appearance compared to the person to be searched for or tracked. In particular, in the case where fields taken by different cameras include the same location, and the same person may be shot from different angles, it is possible to effectively correspond the different human sequences using spatiotemporal phase information.
- Note that, in the above example, a correspondence of positions between the images taken by different cameras is made beforehand, however, in the case of mainly using time period and time phase, it is possible, without such correspondence, to verify between the walking sequences so as to obtain the same effect as can be obtained with the present invention.
- It should be noted that part or all of the processing described in the above embodiments may be executed by a device intended for such processing, or by a communication device such as a terminal and a base station, or a CPU incorporated in a computer, which executes a program.
- Although only some exemplary embodiments of this invention have been described in detail above, those skilled in the art will readily appreciate that many modifications are possible in the exemplary embodiments without materially departing from the novel teachings and advantages of this invention. Accordingly, all such modifications are intended to be included within the scope of this invention.
- The present invention can be used as a human identification apparatus which judges whether or not human images included in different image sequences represent the same person, and as a human searching/tracking apparatus which searches for or tracks a person in image sequences, e.g., a monitor system that is set up on the street, or the like.
Claims (6)
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2004-227083 | 2004-08-03 | ||
JP2004227083 | 2004-08-03 | ||
PCT/JP2005/013769 WO2006013765A1 (en) | 2004-08-03 | 2005-07-27 | Person judging device and person search and tracking device |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2005/013769 Continuation WO2006013765A1 (en) | 2004-08-03 | 2005-07-27 | Person judging device and person search and tracking device |
Publications (2)
Publication Number | Publication Date |
---|---|
US20060120564A1 true US20060120564A1 (en) | 2006-06-08 |
US7397931B2 US7397931B2 (en) | 2008-07-08 |
Family
ID=35787054
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/342,651 Active 2026-09-11 US7397931B2 (en) | 2004-08-03 | 2006-01-31 | Human identification apparatus and human searching/tracking apparatus |
Country Status (4)
Country | Link |
---|---|
US (1) | US7397931B2 (en) |
JP (1) | JP3910629B2 (en) |
CN (5) | CN100474339C (en) |
WO (1) | WO2006013765A1 (en) |
Cited By (23)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070177765A1 (en) * | 2006-01-31 | 2007-08-02 | Canon Kabushiki Kaisha | Method for displaying an identified region together with an image, program executable in a computer apparatus, and imaging apparatus |
US20080024501A1 (en) * | 2006-07-04 | 2008-01-31 | Seiko Epson Corporation | Document editing support device, program, and storage medium |
US20080134021A1 (en) * | 2006-12-01 | 2008-06-05 | Seiko Epson Corporation | Document edit support device, document edit device, program and computer-readable storage medium |
US7397931B2 (en) * | 2004-08-03 | 2008-07-08 | Matsushita Electric Industrial Co., Ltd. | Human identification apparatus and human searching/tracking apparatus |
US20080260208A1 (en) * | 2007-04-20 | 2008-10-23 | Honda Motor Co., Ltd. | Automotive environment monitoring device, vehicle with the automotive environment monitoring device, and automotive environment monitoring program |
US20090046153A1 (en) * | 2007-08-13 | 2009-02-19 | Fuji Xerox Co., Ltd. | Hidden markov model for camera handoff |
US20090150359A1 (en) * | 2007-12-10 | 2009-06-11 | Canon Kabushiki Kaisha | Document processing apparatus and search method |
US20100188511A1 (en) * | 2009-01-23 | 2010-07-29 | Casio Computer Co., Ltd. | Imaging apparatus, subject tracking method and storage medium |
US20130314562A1 (en) * | 2012-05-24 | 2013-11-28 | Olympus Imaging Corp. | Object recognition apparatus, object recognition method and object recognition program |
US20140254876A1 (en) * | 2012-03-29 | 2014-09-11 | The Nielsen Company (Us), Llc | Methods and apparatus to count people in images |
US20150125073A1 (en) * | 2013-11-06 | 2015-05-07 | Samsung Electronics Co., Ltd. | Method and apparatus for processing image |
US9275285B2 (en) | 2012-03-29 | 2016-03-01 | The Nielsen Company (Us), Llc | Methods and apparatus to count people in images |
US9292736B2 (en) | 2012-03-29 | 2016-03-22 | The Nielsen Company (Us), Llc | Methods and apparatus to count people in images |
US20170018091A1 (en) * | 2015-07-16 | 2017-01-19 | Hanwha Techwin Co., Ltd. | Re-wandering alarm system and method |
US9594961B2 (en) | 2012-03-29 | 2017-03-14 | The Nielsen Company (Us), Llc | Methods and apparatus to count people in images |
US20170086075A1 (en) * | 2013-11-15 | 2017-03-23 | Alibaba Group Holding Limited | Identity authentication by using human biological characteristic |
US20190114787A1 (en) * | 2016-03-31 | 2019-04-18 | Nec Corporation | Image detection device, image detection method and storage medium storing program |
TWI697914B (en) * | 2018-11-29 | 2020-07-01 | 宏碁股份有限公司 | Monitoring device and method thereof |
US11080513B2 (en) * | 2011-01-12 | 2021-08-03 | Gary S. Shuster | Video and still image data alteration to enhance privacy |
US11461595B2 (en) * | 2016-12-16 | 2022-10-04 | Clarion Co., Ltd. | Image processing apparatus and external environment recognition apparatus |
US20230009480A1 (en) * | 2019-12-25 | 2023-01-12 | Nec Corporation | Estimation device, estimation system, estimation method, and program recording medium |
US11836217B2 (en) | 2018-12-27 | 2023-12-05 | Nec Corporation | Information processing apparatus, information processing method, and program |
US11847848B2 (en) | 2018-12-07 | 2023-12-19 | Nec Corporation | Information processing apparatus, information processing method, and program |
Families Citing this family (31)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP5032846B2 (en) * | 2004-08-31 | 2012-09-26 | パナソニック株式会社 | MONITORING DEVICE, MONITORING RECORDING DEVICE, AND METHOD THEREOF |
JP4257615B2 (en) * | 2006-07-14 | 2009-04-22 | ソニー株式会社 | Image processing apparatus and method, and program |
JP4263737B2 (en) * | 2006-11-09 | 2009-05-13 | トヨタ自動車株式会社 | Pedestrian detection device |
JP2009053815A (en) * | 2007-08-24 | 2009-03-12 | Nikon Corp | Subject tracking program and subject tracking device |
JP5211918B2 (en) * | 2008-07-30 | 2013-06-12 | 富士通株式会社 | Portable terminal device and authentication management method |
CN101350064B (en) * | 2008-08-29 | 2012-06-13 | 北京中星微电子有限公司 | Method and apparatus for estimating two-dimension human body guise |
CN101388114B (en) * | 2008-09-03 | 2011-11-23 | 北京中星微电子有限公司 | Method and system for estimating human body attitudes |
JP5029647B2 (en) * | 2009-04-08 | 2012-09-19 | 株式会社ニコン | Subject tracking device and camera |
JP5538781B2 (en) * | 2009-09-02 | 2014-07-02 | キヤノン株式会社 | Image search apparatus and image search method |
JP5960950B2 (en) * | 2011-03-29 | 2016-08-02 | セコム株式会社 | Image monitoring apparatus and program |
CN102999152B (en) * | 2011-09-09 | 2016-06-29 | 康佳集团股份有限公司 | A kind of gesture motion recognition methods and system |
WO2013103151A1 (en) * | 2012-01-04 | 2013-07-11 | 株式会社ニコン | Electronic device, method for generating information, and method for estimating position |
CN106104631B (en) * | 2014-03-11 | 2019-03-01 | 三菱电机株式会社 | Human detection device and human detection method |
JP6526953B2 (en) * | 2014-09-12 | 2019-06-05 | 株式会社日立国際電気 | Object identification method |
KR101732981B1 (en) * | 2015-10-29 | 2017-05-08 | 삼성에스디에스 주식회사 | System and method for analyzing personalized characteristics |
US9911198B2 (en) | 2015-12-17 | 2018-03-06 | Canon Kabushiki Kaisha | Method, system and apparatus for matching moving targets between camera views |
EP3494428A4 (en) | 2016-08-02 | 2020-04-08 | Atlas5D, Inc. | Systems and methods to identify persons and/or identify and quantify pain, fatigue, mood, and intent with protection of privacy |
JP6800820B2 (en) * | 2017-07-14 | 2020-12-16 | パナソニック株式会社 | People flow analysis method, people flow analyzer, and people flow analysis system |
JP7325745B2 (en) * | 2017-10-12 | 2023-08-15 | 株式会社コンピュータシステム研究所 | MONITORING DEVICE, MONITORING PROGRAM, STORAGE MEDIUM, AND MONITORING METHOD |
CN107730686A (en) * | 2017-11-01 | 2018-02-23 | 桐乡守敬应用技术研究院有限公司 | A kind of biological characteristic unlocking method |
JP7182778B2 (en) * | 2018-11-05 | 2022-12-05 | 公立大学法人大阪 | Gait analyzer |
WO2020115910A1 (en) | 2018-12-07 | 2020-06-11 | 日本電気株式会社 | Information processing system, information processing device, information processing method, and program |
EP3905188A4 (en) | 2018-12-27 | 2021-12-29 | NEC Corporation | Information processing device, information processing method, and program |
CN109859322B (en) * | 2019-01-22 | 2022-12-06 | 广西大学 | Spectral attitude migration method based on deformation graph |
JP7198196B2 (en) * | 2019-12-26 | 2022-12-28 | 株式会社日立ハイテク | Measuring device and measuring method |
US11315363B2 (en) * | 2020-01-22 | 2022-04-26 | Board Of Trustees Of Michigan State University | Systems and methods for gait recognition via disentangled representation learning |
WO2022030179A1 (en) * | 2020-08-05 | 2022-02-10 | 国立大学法人大阪大学 | Cyclic image restoring device and method, identification device and method, verification device and method, feature extraction device, training method, phase estimation device, and storage medium |
JP7296538B2 (en) | 2020-09-16 | 2023-06-23 | 株式会社シンギュラリティテック | Data preprocessing system, method, and program for gait recognition by artificial intelligence |
JP2022064719A (en) * | 2020-10-14 | 2022-04-26 | 富士通クライアントコンピューティング株式会社 | Information processor, information processing system, and information processing program |
WO2022201987A1 (en) * | 2021-03-23 | 2022-09-29 | 日本電気株式会社 | Image analysis device, image analysis system, image analysis method, and program |
JP7193104B1 (en) | 2021-11-25 | 2022-12-20 | 株式会社アジラ | Behavior identification system |
Citations (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5885229A (en) * | 1995-07-19 | 1999-03-23 | Nippon Telegraph & Telephone Corp. | Walking pattern processing method and system for embodying the same |
US20020167403A1 (en) * | 2001-03-15 | 2002-11-14 | Koninklijke Philips Electronics N.V. | Automatic system for monitoring persons entering and leaving changing room |
US6542621B1 (en) * | 1998-08-31 | 2003-04-01 | Texas Instruments Incorporated | Method of dealing with occlusion when tracking multiple objects and people in video sequences |
US20040252861A1 (en) * | 2003-02-14 | 2004-12-16 | Sony Corporation | Image processing apparatus and method, program, and recording medium |
US20050192516A1 (en) * | 2000-12-27 | 2005-09-01 | Sony Corporation | Gait detection system, gait detection apparatus, device, and gait detection method |
US20060093188A1 (en) * | 2002-02-22 | 2006-05-04 | Microsoft Corporation | Probabilistic exemplar-based pattern tracking |
US7113185B2 (en) * | 2002-11-14 | 2006-09-26 | Microsoft Corporation | System and method for automatically learning flexible sprites in video layers |
US7116323B2 (en) * | 1998-05-27 | 2006-10-03 | In-Three, Inc. | Method of hidden surface reconstruction for creating accurate three-dimensional images converted from two-dimensional images |
US7265777B2 (en) * | 2002-03-01 | 2007-09-04 | Huper Laboratories Co., Ltd. | Method for abstracting multiple moving objects |
US20070211925A1 (en) * | 2006-03-07 | 2007-09-13 | Kabushiki Kaisha Toshiba | Face authentication apparatus and face authentication method |
Family Cites Families (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2801362B2 (en) * | 1990-05-24 | 1998-09-21 | 日本電信電話株式会社 | Personal identification device |
US6205231B1 (en) * | 1995-05-10 | 2001-03-20 | Identive Corporation | Object identification in a moving video image |
US6263088B1 (en) * | 1997-06-19 | 2001-07-17 | Ncr Corporation | System and method for tracking movement of objects in a scene |
JP2000182060A (en) * | 1998-12-21 | 2000-06-30 | Nec Corp | Individual identification method/device |
US20030123703A1 (en) * | 2001-06-29 | 2003-07-03 | Honeywell International Inc. | Method for monitoring a moving object and system regarding same |
JP4187448B2 (en) * | 2002-03-07 | 2008-11-26 | 富士通マイクロエレクトロニクス株式会社 | Method and apparatus for tracking moving object in image |
JP2003346159A (en) * | 2002-05-28 | 2003-12-05 | Oki Electric Ind Co Ltd | Person tracking method and person tracking device |
CN100474339C (en) * | 2004-08-03 | 2009-04-01 | 松下电器产业株式会社 | Human identification apparatus |
-
2005
- 2005-07-27 CN CNB2005800008924A patent/CN100474339C/en active Active
- 2005-07-27 CN CN2008101459921A patent/CN101398891B/en active Active
- 2005-07-27 CN CN2008102100013A patent/CN101344923B/en active Active
- 2005-07-27 WO PCT/JP2005/013769 patent/WO2006013765A1/en active Application Filing
- 2005-07-27 CN CN2008101459917A patent/CN101398890B/en active Active
- 2005-07-27 CN CN2008102100009A patent/CN101398892B/en active Active
- 2005-07-27 JP JP2006531421A patent/JP3910629B2/en not_active Expired - Fee Related
-
2006
- 2006-01-31 US US11/342,651 patent/US7397931B2/en active Active
Patent Citations (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5885229A (en) * | 1995-07-19 | 1999-03-23 | Nippon Telegraph & Telephone Corp. | Walking pattern processing method and system for embodying the same |
US7116323B2 (en) * | 1998-05-27 | 2006-10-03 | In-Three, Inc. | Method of hidden surface reconstruction for creating accurate three-dimensional images converted from two-dimensional images |
US6542621B1 (en) * | 1998-08-31 | 2003-04-01 | Texas Instruments Incorporated | Method of dealing with occlusion when tracking multiple objects and people in video sequences |
US20050192516A1 (en) * | 2000-12-27 | 2005-09-01 | Sony Corporation | Gait detection system, gait detection apparatus, device, and gait detection method |
US6958045B2 (en) * | 2000-12-27 | 2005-10-25 | Sony Corporation | Gait detection system, gait detection apparatus, device, and gait detection method |
US20020167403A1 (en) * | 2001-03-15 | 2002-11-14 | Koninklijke Philips Electronics N.V. | Automatic system for monitoring persons entering and leaving changing room |
US20060093188A1 (en) * | 2002-02-22 | 2006-05-04 | Microsoft Corporation | Probabilistic exemplar-based pattern tracking |
US7265777B2 (en) * | 2002-03-01 | 2007-09-04 | Huper Laboratories Co., Ltd. | Method for abstracting multiple moving objects |
US7113185B2 (en) * | 2002-11-14 | 2006-09-26 | Microsoft Corporation | System and method for automatically learning flexible sprites in video layers |
US20070104383A1 (en) * | 2002-11-14 | 2007-05-10 | Microsoft Corporation | Stabilization of objects within a video sequence |
US20040252861A1 (en) * | 2003-02-14 | 2004-12-16 | Sony Corporation | Image processing apparatus and method, program, and recording medium |
US20070211925A1 (en) * | 2006-03-07 | 2007-09-13 | Kabushiki Kaisha Toshiba | Face authentication apparatus and face authentication method |
Cited By (44)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7397931B2 (en) * | 2004-08-03 | 2008-07-08 | Matsushita Electric Industrial Co., Ltd. | Human identification apparatus and human searching/tracking apparatus |
US7826639B2 (en) * | 2006-01-31 | 2010-11-02 | Canon Kabushiki Kaisha | Method for displaying an identified region together with an image, program executable in a computer apparatus, and imaging apparatus |
US20070177765A1 (en) * | 2006-01-31 | 2007-08-02 | Canon Kabushiki Kaisha | Method for displaying an identified region together with an image, program executable in a computer apparatus, and imaging apparatus |
US20080024501A1 (en) * | 2006-07-04 | 2008-01-31 | Seiko Epson Corporation | Document editing support device, program, and storage medium |
US8122347B2 (en) * | 2006-07-04 | 2012-02-21 | Seiko Epson Corporation | Document editing support device, program, and storage medium |
US20080134021A1 (en) * | 2006-12-01 | 2008-06-05 | Seiko Epson Corporation | Document edit support device, document edit device, program and computer-readable storage medium |
US7885430B2 (en) * | 2007-04-20 | 2011-02-08 | Honda Motor Co., Ltd. | Automotive environment monitoring device, vehicle with the automotive environment monitoring device, and automotive environment monitoring program |
US20080260208A1 (en) * | 2007-04-20 | 2008-10-23 | Honda Motor Co., Ltd. | Automotive environment monitoring device, vehicle with the automotive environment monitoring device, and automotive environment monitoring program |
US20090046153A1 (en) * | 2007-08-13 | 2009-02-19 | Fuji Xerox Co., Ltd. | Hidden markov model for camera handoff |
US8432449B2 (en) * | 2007-08-13 | 2013-04-30 | Fuji Xerox Co., Ltd. | Hidden markov model for camera handoff |
US20090150359A1 (en) * | 2007-12-10 | 2009-06-11 | Canon Kabushiki Kaisha | Document processing apparatus and search method |
US20100188511A1 (en) * | 2009-01-23 | 2010-07-29 | Casio Computer Co., Ltd. | Imaging apparatus, subject tracking method and storage medium |
US11600108B2 (en) * | 2011-01-12 | 2023-03-07 | Gary S. Shuster | Video and still image data alteration to enhance privacy |
US11080513B2 (en) * | 2011-01-12 | 2021-08-03 | Gary S. Shuster | Video and still image data alteration to enhance privacy |
US20210365670A1 (en) * | 2011-01-12 | 2021-11-25 | Gary S. Shuster | Video and still image data alteration to enhance privacy |
US9465999B2 (en) * | 2012-03-29 | 2016-10-11 | The Nielsen Company (Us), Llc | Methods and apparatus to count people in images |
US9275285B2 (en) | 2012-03-29 | 2016-03-01 | The Nielsen Company (Us), Llc | Methods and apparatus to count people in images |
US9292736B2 (en) | 2012-03-29 | 2016-03-22 | The Nielsen Company (Us), Llc | Methods and apparatus to count people in images |
US10242270B2 (en) | 2012-03-29 | 2019-03-26 | The Nielsen Company (Us), Llc | Methods and apparatus to count people in images |
US11527070B2 (en) | 2012-03-29 | 2022-12-13 | The Nielsen Company (Us), Llc | Methods and apparatus to count people in images |
US9594961B2 (en) | 2012-03-29 | 2017-03-14 | The Nielsen Company (Us), Llc | Methods and apparatus to count people in images |
US20140254876A1 (en) * | 2012-03-29 | 2014-09-11 | The Nielsen Company (Us), Llc | Methods and apparatus to count people in images |
US10810440B2 (en) | 2012-03-29 | 2020-10-20 | The Nielsen Company (Us), Llc | Methods and apparatus to count people in images |
US8976261B2 (en) * | 2012-05-24 | 2015-03-10 | Olympus Imaging Corp. | Object recognition apparatus, object recognition method and object recognition program |
US20130314562A1 (en) * | 2012-05-24 | 2013-11-28 | Olympus Imaging Corp. | Object recognition apparatus, object recognition method and object recognition program |
US9639758B2 (en) * | 2013-11-06 | 2017-05-02 | Samsung Electronics Co., Ltd. | Method and apparatus for processing image |
US20150125073A1 (en) * | 2013-11-06 | 2015-05-07 | Samsung Electronics Co., Ltd. | Method and apparatus for processing image |
US10902056B2 (en) * | 2013-11-06 | 2021-01-26 | Samsung Electronics Co., Ltd. | Method and apparatus for processing image |
US20170206227A1 (en) * | 2013-11-06 | 2017-07-20 | Samsung Electronics Co., Ltd. | Method and apparatus for processing image |
US20170086075A1 (en) * | 2013-11-15 | 2017-03-23 | Alibaba Group Holding Limited | Identity authentication by using human biological characteristic |
US9930533B2 (en) * | 2013-11-15 | 2018-03-27 | Alibaba Group Holding Limited | Identity authentication by using human biological characteristic |
US20170018091A1 (en) * | 2015-07-16 | 2017-01-19 | Hanwha Techwin Co., Ltd. | Re-wandering alarm system and method |
US9959460B2 (en) * | 2015-07-16 | 2018-05-01 | Hanwha Techwin Co., Ltd. | Re-wandering alarm system and method |
US10872423B2 (en) * | 2016-03-31 | 2020-12-22 | Nec Corporation | Image detection device, image detection method and storage medium storing program |
US11107223B2 (en) * | 2016-03-31 | 2021-08-31 | Nec Corporation | Image detection device, image detection method and storage medium storing program |
US11288816B2 (en) | 2016-03-31 | 2022-03-29 | Nec Corporation | Image detection device, image detection method and storage medium storing program |
US20190114787A1 (en) * | 2016-03-31 | 2019-04-18 | Nec Corporation | Image detection device, image detection method and storage medium storing program |
US11461595B2 (en) * | 2016-12-16 | 2022-10-04 | Clarion Co., Ltd. | Image processing apparatus and external environment recognition apparatus |
TWI697914B (en) * | 2018-11-29 | 2020-07-01 | 宏碁股份有限公司 | Monitoring device and method thereof |
US11847848B2 (en) | 2018-12-07 | 2023-12-19 | Nec Corporation | Information processing apparatus, information processing method, and program |
US11836217B2 (en) | 2018-12-27 | 2023-12-05 | Nec Corporation | Information processing apparatus, information processing method, and program |
US11928181B2 (en) | 2018-12-27 | 2024-03-12 | Nec Corporation | Information processing apparatus, information processing method, and program |
US11934483B2 (en) | 2018-12-27 | 2024-03-19 | Nec Corporation | Information processing apparatus, information processing method, and program |
US20230009480A1 (en) * | 2019-12-25 | 2023-01-12 | Nec Corporation | Estimation device, estimation system, estimation method, and program recording medium |
Also Published As
Publication number | Publication date |
---|---|
CN1842824A (en) | 2006-10-04 |
CN101398891B (en) | 2010-12-08 |
CN100474339C (en) | 2009-04-01 |
CN101398892A (en) | 2009-04-01 |
CN101398892B (en) | 2010-12-22 |
CN101398890B (en) | 2010-12-08 |
CN101344923B (en) | 2012-05-23 |
JPWO2006013765A1 (en) | 2008-05-01 |
JP3910629B2 (en) | 2007-04-25 |
US7397931B2 (en) | 2008-07-08 |
CN101398891A (en) | 2009-04-01 |
WO2006013765A1 (en) | 2006-02-09 |
CN101398890A (en) | 2009-04-01 |
CN101344923A (en) | 2009-01-14 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7397931B2 (en) | Human identification apparatus and human searching/tracking apparatus | |
JP5001260B2 (en) | Object tracking method and object tracking apparatus | |
Venkatesh et al. | Efficient object-based video inpainting | |
US8004528B2 (en) | Method, systems and computer product for deriving three-dimensional information progressively from a streaming video sequence | |
US9177229B2 (en) | Kalman filter approach to augment object tracking | |
JP4467838B2 (en) | Image recognition apparatus and image recognition method | |
GB2516173A (en) | Tracking assistance device, tracking assistance system and tracking assistance method | |
US20140301605A1 (en) | Posture estimation device and posture estimation method | |
US20130070105A1 (en) | Tracking device, tracking method, and computer program product | |
US20200374491A1 (en) | Forensic video exploitation and analysis tools | |
JP5634517B2 (en) | Video image monitoring | |
JP3836814B2 (en) | Image processing method, image processing apparatus, and image processing program | |
JP2018028784A (en) | Movable body group detection program, movable body group detection device, and movable body group detection method | |
KR101956275B1 (en) | Method and apparatus for detecting information of body skeleton and body region from image | |
US9256945B2 (en) | System for tracking a moving object, and a method and a non-transitory computer readable medium thereof | |
JP4728795B2 (en) | Person object determination apparatus and person object determination program | |
US7580546B2 (en) | Marker-free motion capture apparatus and method for correcting tracking error | |
JP5534432B2 (en) | Information terminal equipment | |
JP2006146413A (en) | Object tracking device | |
KR100994722B1 (en) | Method for tracking moving object on multiple cameras using probabilistic camera hand-off | |
WO2020013021A1 (en) | Detecting device, processing device, detecting method, and processing program | |
JP2006215655A (en) | Method, apparatus, program and program storage medium for detecting motion vector | |
JPH08167022A (en) | Image monitor device | |
JP2001012946A (en) | Dynamic image processor and processing method | |
JP2004118485A (en) | Image tracking device and method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: MATSUSHITA ELECTRIC INDUSTRIAL CO., LTD., JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:IMAGAWA, TARO;IWASAKI, MASAHIRO;REEL/FRAME:017404/0693 Effective date: 20051226 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
FEPP | Fee payment procedure |
Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
FPAY | Fee payment |
Year of fee payment: 4 |
|
AS | Assignment |
Owner name: PANASONIC INTELLECTUAL PROPERTY CORPORATION OF AMERICA, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:PANASONIC CORPORATION;REEL/FRAME:033033/0163 Effective date: 20140527 Owner name: PANASONIC INTELLECTUAL PROPERTY CORPORATION OF AME Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:PANASONIC CORPORATION;REEL/FRAME:033033/0163 Effective date: 20140527 |
|
FPAY | Fee payment |
Year of fee payment: 8 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 12TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1553); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 12 |