WO2012074352A1

WO2012074352A1 - System and method to detect loitering event in a region

Info

Publication number: WO2012074352A1
Application number: PCT/MY2011/000078
Authority: WO
Inventors: Mei Kuan Lim; Kim Meng Liang; Sze Ling Tang; Weng Kin Lai; Kadim Zulaikha; Samudin Norshuhada; Kiran Maleeha; A. Bahaa Ahmed
Original assignee: Mimos Bhd.
Priority date: 2010-11-29
Filing date: 2011-06-06
Publication date: 2012-06-07
Also published as: MY159285A

Abstract

The present invention relates to a system and method of surveillance to detect loitering event, more particularly to a system and method to detect loitering events during occlusion, non-occlusion or both in a region, wherein the system comprises a subtracting means (100), a tracking means (101), a timer means (102), a storage means (103), and a processor (104) electrically coupled to the storage means (105) for processing the methods of determining a loitering event. Since the system is able to detect loitering events during occlusion and non-occlusion, a method of implementing multi-timer approach is adapted into the system; additionally said adapted method is capable of detecting of re-appearing subjects.

Description

SYSTEM AND METHOD TO DETECT LOITERING EVENT IN A REGION

TECHNICAL FIELD The technical field of the invention relates to a system and method of surveillance to detect loitering event, more particularly to a system and method to detect loitering events during occlusion, non-occlusion or both in a region.

BACKGROUND OF INVENTION

Video surveillance has since progressed to capture images, digitize and analyze those images, and predicting response to events in those images based on the analysis. Amongst the common events in surveillance today include detection of pedestrian entering a shop, a vehicle turning against the traffic flow, or a person loitering about a secured area. Detection of loitering event is crucial, as the behavior of lingering around a specific area have been proven to be linked to harmful activities such as tailgating, drug dealing activity or vandalism. In a typical event, drug dealers will loiter around the bus stops while their customers come in on buses, purchase their material, and then leave on other buses. Thus, suspected drug-dealing activities can be detected by monitoring bus stops for individual loitering.

Several prior arts have disclosed systems and methods to detect loitering events. US 7088846 B2 disclosed a video surveillance system using rule-based reasoning and multiple-hypothesis scoring to detect predefined behaviors based on movement through zone patterns. Trajectory hypothesis spawning allows for trajectory splitting and/or merging and includes local pruning to manage hypothesis growth. This method requires robust tracker as the accuracy of detecting a loitering event is highly dependant on the tracking results. Also, the rule-based reasoning and multiple hypotheses require thorough understanding on the deployment environment as multiple zones are considered for the analysis. US 2009/002155 disclosed an introduction of electronic tracking device and a transmitter for a more accurate tracking information. Each of the transmitters can be configured accordingly to be associated with a particular subject or a group of subjects. The location information and trajectory pattern of the subject obtained from the transmitter is then analyzed to determine the behavior patterns or loitering event. This invention requires the tracking device and transmitter to be attached to a subject and is not suitable for most surveillance deployment in which monitoring is done non- intrusively. US 6985172 disclosed surveillance apparatus and methods effective for detecting incidents based upon improved image processing techniques applied to infrared and visible light spectrum images in a time sequence. This invention applies advanced algorithms to include temporal processing and model-based analysis to achieve machine perception and recognition of normal scene motions. Subsequent discrepancies between the scene and the model are then detected and compared to classification criteria for disposition. The model-based approach requires the scene to be monitored over a period of time to construct the model of the scene.

SUMMARY OF INVENTION

A system and a method of surveillance to detect loitering event in a region in accordance with the present invention are disclosed herein. The object of the present invention is to provide a system and method to detect loitering event in a region during occlusion and non-occlusion, wherein the system comprises a subtracting means for receiving and processing an input, a tracking means, a timer means, a storage means, and a processor electrically coupled to the storage means in a mutual relationship for processing and determining a loitering event. Preferably, the system to detect loitering event in a region comprising a tracking means for identifying and labeling subjects with an identity, and a timer means for receiving individual identity information of a plurality of subjects to measure time span of each subject in the region. Preferably, the timer means for receiving individual identity information of a plurality of subjects to measure time span of each subject in a region includes a first timer for measuring the time span of each subject during non-occlusion, and a second timer for measuring the time span of each subject during an occlusion.

Preferably, the system to detect loitering event in a region comprising a storage means for recording and retrieving individual identity information and timing information of each said subjects, and a processor electrically coupled to the storage means in a mutual relationship to provide time analysis, and loitering detection of the subjects for processing and determining a loitering event.

Preferably, the method to detect loitering event in a region comprises detecting at least a subject in the region from a series of captured frames, labeling each subject with an identity to establish correspondence between the subjects with the series of captured frames, determining time span for each subject, determining total time span for each subject, recording and storing the identity information and time span information of each subject, detecting a loitering event, and triggering an alarm when the loitering event is detected. Thus, the aim of the present invention is to solve common problems arising from the current systems and methods to detect loitering event, such as problems of unknown starting time or a first frame when a subject appears in a region due to occlusion or crowded scene, as well as the subject whom reappears in the region before the total time span of said subject exceeds the limit of the predefined loitering threshold, which will not be detected by a common system and method.

BRIEF DESCRIPTION OF DRAWINGS Figure 1: Describes a system of surveillance in accordance with the present invention.

Figure 2: Describes a process flow of a method of surveillance in accordance with the present invention. Figure 3: Describes a process flow of a timer analysis in accordance with the present invention.

Figure 4: Describes a loitering detection process flow in accordance with the present invention.

Figure 5: Describes an array of selected captured frames, of frame 3, frame 10, frame 1 1, frame 15, frame 21 and a result frame in accordance with the present invention.

DETAILED DESCRIPTION OF EMBODIMENTS

Described below are preferred embodiments of the present invention with reference to the accompanying drawings. Each of the following preferred embodiments describes an example

Referring to Figure 1, there is illustrated a system of surveillance to detect loitering event in a region, more particularly to detect a loitering event in a crowded scene comprising an input (100), a subtracting means (101) for receiving an input (100) and extracting a plurality of motion pixels generated from the input (100), a tracking means (102) for identifying a plurality of subjects from the motion pixels, a timer means (103) for receiving individual identity information of a plurality of subjects to measure time span of each subject in the region, a storage means (104) for recording and retrieving individual identity information and timing information of each said subjects, and a processor (105) electrically coupled to the storage means in a mutual relationship to provide time analysis of the subjects for processing and determining a loitering event.

Input (100) stated herein is the main source for analysis of a detection, wherein said input (100) is preferably a series of frames from a video scene of video cameras, closed circuit television, or any image producing media sources that are inputted into the subtracting means (101) for a preliminary analysis. The subtracting means (101) mainly extracts a plurality of motion pixel images from background scenes and processes outputs of binary map where binary zero pixels indicate the background, and binary non-zero pixels represent the motion pixels. A connecting component criterion, an application of widely developed algorithm, such as a heuristic approach to cluster identified data that represents structures of the motion pixels under uncertainty is then applied on the binary map to group motion pixels with similar properties as motion blobs. The key idea of grouping motion pixels into motion blobs is to emphasize blobs corresponding to subjects and eliminating noise as much as possible as smaller motion blobs which are usually due to noise will not be connected via this connecting component criterion, and can be removed by a way of setting a threshold. For example, any blobs with the size of less than 100 pixels will be removed from the system.

The tracking means (102) as part of the preferred embodiment in the present invention receives the motion blobs from the subtracting means (101) for identifying a plurality of subjects from the motion pixels, and labeling each subjects with an identity to establish correspondence between the subjects. Motion blobs that are fed into this tracking means (102) are processed of temporal image sequence to establish correspondence between the subjects across the series of frames and labeled with a consistent identity throughout their appearance in the image sequences. Preferably, an object classification procedure, where any existing algorithms that can be applied beforehand to discriminate between humans and non-humans, such as an example of using the height-width ration of the motion blob to discriminate between human and non-human so that only preferred subject is analyzed further by subsequent processes. This step is dependent on the aim or functionality of the system, where if the system is only interested in analyzing human subject, then a human classification method may be applied.

The labeled motion blobs are then fed into the timer means (103) where the computation of time span of subjects during occlusion is carried out in the timer means (103). The timer means (103) in the present invention herein preferably includes at least a first timer for measuring the time span of each subject during non- occlusion, and at least a second timer for measuring the time span of each subject during an occlusion. Firstly, the location of the subject is checked against a binary map representing a region of interest, where binary zero to indicate region of non- interest and binary one to indicate region of interest. Subsequent processes focuses only on subjects that are detected within the region of interest. Otherwise, the first timer information, and the second timer information of a first frame when subject exits the region of interest are stored in the storage means (104). This step is crucial to allow detection of subject that appears in the region of interest again at any instance, if the time span of the subject does not exceed a predefined loitering threshold.

The predefined loitering threshold is set upon configuration of the system, and prior to the deployment of this system because said predefined loitering threshold depends on the criticality of the environment the system is deployed and may vary from one image producing media source to another. For example, the predefined loitering threshold at a lobby may have a longer timestamp (i.e. 5 minutes) as compared to a corridor (i.e. 1 minute), due to the fact that a subject has higher tendency to spend more time at the lobby and thus should not be detected as a suspicious behavior, in this case loitering. However, a subject lingering about the corridor is not common and thus should be detected as loitering with a shorter timestamp.

For a subject that is detected within the region of interest, identity of the subject will be first examined against identity information stored in the storage means (104) to identify subject that re-appears in the region. The identity information may include unique identifier given based on the similarity between temporal features of the subject which includes but not limited to colour properties, texture information, size, and velocity. If the subject's identity information matches any of that in the storage means (104), then it is concluded that the particular subject had appeared in the region before. Thus, timer information of the subject in the storage means (104) is copied to the present subject, and the subject will now continue to carry its history of identity information, and the past timer information in the storage means (104) will be deleted.

For a subject that is detected within the region of interest and do not match any of the identity information in the storage means (104), the system proceeds computation based on concurrent timer information of the subject. Then, status of each subject is determined using a method of overlapping between motion blobs, where correspondence between frames is established via overlapping between motion blobs of the previous and current frame, and comparing similar temporal features to identify subjects to determine events of merging and splitting.

Some of the rules to discriminate between the events are explained as follows. If a blob from a current frame does not overlap with any of the blobs in a previous frame, it concludes that the current blob, is a new subject, and each of the new subjects detected will be stored into its respective region of interest database in the storage means (104) for further analysis. If one of the blobs from a previous frame overlaps with only one blob in a current frame, over an empirically defined threshold, it is concluded that the current blob is corresponding to the previous blob and thus is an existing subject.

If one of the blobs from a previous frame overlaps with more than one of the current blobs, it is concluded that splitting event occurs. When splitting event is detected, the present invention assumes both blobs as new subjects and they will be stored into their respective region of interest database in the storage means (104) for further analysis. This is to cater for new detection of subjects, which enters the region of interest in a group, or merely an occurrence of occlusion. Since, one previous blob is associated to more than one current blob, then splitting event is preferentially detected in the present invention and a first timer of the subject is activated. In other words, the first timer is only activated in situations where subject appears as single blob; when there is no occlusion.

If more than one of the blobs from a previous frame overlaps with one of the current blob, it is concluded that merging occurs, since more than one previous blob is associated to one current blob. In this situation, a second timer is activated and the first timer pauses as long as the subject reappears as a single blob. The second timer is activated when there is more than one subject in a single blob; when the tracking means (102) does not have sufficient information on the subject due to high occlusion. The first timer contributes to the total time span of each subject directly whereas the second timer contributes to the total time span according to a predefined sensitivity threshold. The predefined sensitivity threshold indicates the responsiveness of the system and can be set to a value as preferred upon configuration of said system. A low sensitivity of 5 seconds for example, will result in an alert of 5 seconds after a loitering event is detected. A sensitivity threshold of 0 seconds will result in an immediate alert upon detection of an event.

Finally, the processor (105) that is electrically coupled to the storage means (104) in a mutual relationship provides time analysis of the subjects for processing and determining a loitering event through obtaining the predefined sensitivity threshold from a user interface, depending on the deployment of the system framework. The threshold may be represented by a predetermined value to indicate the least sensitive, which detects reduced events, and by another predetermined value to indicate the most sensitive, which detects more events. Also, the predefined loitering threshold, which indicates detection for loitering, that differentiates between a normal and an abnormal time span of subjects in the region, is programmed where subsequently the total time span of each subject is computed. This is performed by summing up the value of the first timer and the value of the second timer, wherein a predefined sensitivity threshold is first multiplied with the second timer. Subsequently, the time span of each subject is compared against the predefined loitering threshold. Subject that appears in the region for a time span that exceeds the allowed predefined loitering threshold is deemed as loitering. Otherwise, the subject is deemed normal. Referring to Figure 2, there is illustrated a process flow of a method to detect an event in a region in accordance with the present invention. As simplified in the figure, the process flow initializes with primarily inputting a system of the present invention with frame sequences of a region of interest: the area of surveillance, and defining a sensitivity threshold and defining a loitering threshold that depends on the deployment of the system environment as mentioned in the earlier figure. Both thresholds are of time variable, therefore not affecting any detection for differences in the frame rate. Process flow continues with the preferred system to operate on background subtraction (51) of the available frame sequences to extract only motion pixels of interest. When this is done, motion tracking (52) will be communicated on the extracted motion pixels to identify and label subjects based on temporal image processing as explained previously such as colour properties, texture information, size, and velocity, with the intention that of forming motion blob for each subject. Next, the system will loop through each subject (53) to determine whether or not if any of the subjects are appearing in the region of interest for the very first instant, or if the subjects are reappearing in said region.

Following that, the well-identified motion blobs of subjects are now prepared for a time analysis (54), and ensuing that, the motion blobs are inspected for a loitering detection (55). As shown in the figure, next stage is decision making stage, where if a loitering is detected, the system triggers an alert and stores related information of the loitering subject into the storage means (104) for further investigations, or to remain unperturbed if no loitering events are detected. The alert, an alarm is preferably a highlight projected on the locality of the detected loitering event, and preferably accompanied by a sound alert. Referring now to Figure 3, there is illustrated a process flow of time analysis procedures in accordance with the method to detect loitering event in a region in the present invention. The process flow initiates a decision of whether or not a detected subject is in the region of interest. If the subject is detected to be in the region of interest, the system determines whether or not if the subject has been stored in a storage means (104) before, as to detect if the subject is of reappearance, so that the system retrieves the last timer information of the subject and activates time analysis for that particular subject, as well as deleting history of that subject from the storage means (104). For a fresh subject, the system proceeds to compute timer information of the subject as elaborated as follows.

Having detected and identified history of the subject and determined of said subject's timing information, now, the process flow continues with determining status of subjects using a method of overlapping between motion blobs in the previous and current frame in the technique of merging and splitting as explained in figure 1. As mentioned previously, the first timer will be activated upon detection of subject being a single motion blob, and the second timer will be activated upon detection of subjects in a group of motion blobs. On the contrary, if a subject is not detected to be in the region of interest, the system determines exit status of the subject, where the first timer information, and the second timer information of a first frame when said subject exits the region of interest are stored in the storage means (104).

Referring now to Figure 4, there is illustrated a loitering detection process flow in accordance with the method to detect a loitering event in a region in the present invention. As illustrated, loitering detection is dependent on the constant values of a predefined sensitivity threshold, and a predefined loitering threshold, which are then computed for the total time span of a subject in the region of interest. The total time span equals sum of value of the first timer with the product value of the second timer and a predefined sensitivity threshold. The total time span of each subject is compared against the predefined loitering threshold, and subjects that appear in the region for a total time span that exceeding allowed predefined loitering threshold are deemed as loitering. Otherwise, the subject is deemed not loitering. Referring now to Figure 5, the figure illustrates an array of selected captured frames, of frame 3 (71), frame 10 (72), frame 11 (73), frame 15 (74), frame 21 (75) and a result frame (76) to further explain a case study example on the operations of the preferred embodiment. The frame numbers are one-step incremental series starting from frame 1 , and the first timer and the second timer are represented as definite timer and ambiguous timer respectively in the shown frames. It is assumed that one frame equals to one second in this case study for ease of explanation. From frame 1 to frame 10 (72), a first timer will be activated, as a subject of interest is not occluded. At frame 11 (73), another subject occludes the subject, thus the second timer will be in activation mode until the subject appears as a single blob. In this case study, the sensitivity threshold is set at 80% and loitering threshold is set to be 15 seconds.

Hence, the total time span of the particular subject is the sum of its first timer and the product of second timer and sensitivity threshold, which is calculated to be 19 seconds - Π - in this case. Then, the total time span of the subject is compared against the loitering threshold, which is 15s. Since the total time span of the subject is greater than allowed, the subject is deemed as loitering, which concludes that loitering event has been detected.

The final stage is decision making, where a subject deemed as loitering, the system stores all information with regards to the subject in the storage means (104). This is important to allow post mortem analysis on the image sequences by an authorized personnel. Otherwise, if a subject is deemed normal, the system continues to process other subjects in the current frame and proceeds to analyze succeeding frames.

In as much as the present invention is subject to many variations, modifications and changes in detail, it is intended that all matter contained in the preceding description or shown in the accompanying drawings shall be interpreted as illustrative and not in a limiting sense within the scope of the present invention.

Claims

. A system to detect a loitering event in a region comprises:

a subtracting means (101) for receiving an input (100), and extracting a plurality of motion pixels generated from the input (100);

a tracking means (102) for identifying a plurality of subjects from the motion pixels, and labeling each subjects with an identity to establish correspondence between the subjects and said input (100);

a timer means (103) for receiving individual identity information of a plurality of subjects to measure time span of each subject in the region;

a storage means (104) for recording and retrieving individual identity information and timing information of each said subjects; and

a processor (105) electrically coupled to the storage means in a mutual relationship to provide time analysis of the subjects for processing and determining a loitering event;

characterized in that the timer means (103) includes at least a first timer for measuring the time span of each subject during non-occlusion, and at least a second timer for measuring the time span of each subject during an occlusion.

2. A system to detect a loitering event in a region in accordance to claim 1, wherein said subtracting means (101) preferably extracts motion pixels from the background scene.

A system to detect a loitering event in a region in accordance to claim 1, wherein said input (100) is preferably a series of frames from a video scene of video cameras, closed circuit television, or any image producing media sources or any combination thereof.

A system to detect a loitering event in a region in accordance to claim 1, wherein said subtracting means (101) preferably outputs a binary map.

5. A system to detect a loitering event in a region in accordance to claim 4, wherein the binary map preferably indicates binary zero pixels for the background.

A system to detect a loitering event in a region in accordance to claim 5, wherein the binary map preferably indicates binary non-zero pixels for the motion pixels.

A system to detect a loitering event in a region in accordance to claim 6, wherein said motion pixels with similar properties are preferably grouped together as motion blobs.

8. A system to detect a loitering event in a region in accordance to claim 7, wherein said motion blobs are preferably the subjects.

9. A system to detect a loitering event in a region in accordance to claim 1, wherein said tracking means (102) preferably performs an object classification method to discriminate objects in the region.

10. A system to detect a loitering event in a region in accordance to claim 1, wherein said tracking means (102) preferably identifies subjects based on temporal features of the subjects.

11. A system to detect a loitering event in a region in accordance to claim 10, wherein said temporal features are preferably but not limited to colour properties, texture, size, speed, velocity or any combination thereof.

12. A system to detect a loitering event in a region in accordance to claim 10, wherein said storage means (104) preferably stores the time span information of a first frame when the subject exits the region.

13. A method to detect a loitering event in a region characterized in that:

detecting at least a subject in the region from a series of captured frames;

labeling each subject with an identity to establish correspondence between the subjects with the series of captured frames;

determining time span for each subject, wherein the determination includes an application of at least a first timer during non-occlusion, and an application of at least a second timer during occlusion;

determining total time span for each subject, wherein the determination includes the first timer information, the second timer information, and a predefined sensitivity threshold;

recording and storing the identity information, and time span information of each subject in a storage means (104);

detecting a loitering event when the total time span of at least one subject is longer than a predefined loitering threshold; and

triggering an alarm when the loitering event is detected;

14. A method to detect a loitering event in a region in accordance to claim 13, wherein the series of captured frames are preferably from a video scene of video cameras, closed circuit television, or any image producing media sources or any combination thereof.

15. A method to detect a loitering event in a region in accordance to claim 13, wherein detecting and labeling at least a subject from a series of captured frames are preferably the results of a tracking means (102).

16. A method to detect a loitering event in a region in accordance to claim 15, wherein the tracking means (102) preferably performs an object classification method to discriminate objects in the region.

17. A method to detect a loitering event in a region in accordance to claim 16, wherein said tracking means (102) preferably identifies subjects based on temporal features of the subjects.

18. A method to detect a loitering event in a region in accordance to claim 17, wherein said temporal features are preferably but not limited to colour properties, texture, size, speed, velocity or any combination thereof.

19. A method to detect a loitering event in a region in accordance to claim 13, wherein the first timer for measuring timing information of said subject preferably activates during non-occlusion.

20. A method to detect a loitering event in a region in accordance to claim 13, wherein the second timer for measuring timing information of said subject preferably activates during occlusion.

21. A method to detect a loitering event in a region in accordance to claim 13, wherein the total time span for each subject is preferably summation of the first timer information with the product of second timer information and the predefined sensitivity threshold.

22. A method to detect a loitering event in a region in accordance to claim 13, wherein the predefined sensitivity threshold preferably represents responsiveness to detect the loitering event.

23. A method to detect a loitering event in a region in accordance to claim 13, wherein the stored identity information and time span information are preferably the recordings of the first frame where the subject exits the region.

24. A method to detect a loitering event in a region in accordance to claim 13, wherein the predefined loitering threshold distinguishes between a normal and an abnormal time span of the subject in said region.

25. A method to detect a loitering event in a region in accordance to claim 13, wherein the alarm is preferably a highlight projected on the locality of the detected loitering event.

26. A method to detect a loitering event in a region in accordance to claim 13, wherein the alarm is preferably a sound alert.