WO2008107713A1

WO2008107713A1 - Controlled high resolution sub-image capture with time domain multiplexed high speed full field of view reference video stream for image based biometric applications

Info

Publication number: WO2008107713A1
Application number: PCT/GB2008/050144
Authority: WO
Inventors: Thomas Heseltine; Justen Hyde
Original assignee: Aurora Computer Services Limited
Priority date: 2007-03-07
Filing date: 2008-03-03
Publication date: 2008-09-12
Also published as: GB0704325D0; GB2447246A; GB2447246B

Abstract

There is discloses a method and apparatus for image processing for object recognition applications. An image capture device is used to acquire a first, relatively low resolution image of a relatively wide field of view containing a plurality of objects. Apattern recognition algorithm is then applied to the first image so as to identify objects of interest, and at least one object of interest is selected from the first image and its coordinates within the image are determined. The determined coordinates are then used to control either the image capture device or an additional image capture device so as to obtain a sequence of second, relatively high resolution images of a relatively narrow field of view containing the at least one object of interest.

Description

CONTROLLED HIGH RESOLUTION SUB-IMAGE CAPTURE WITH TIME DOMAIN

MULTIPLEXED HIGH SPEED FULL FIELD OF VIEW REFERENCE VIDEO STREAM

FOR IMAGE BASED BIOMETRIC APPLICATIONS

The present invention relates to apparatus and methods for capturing high definition, high speed images from object specific targets in a range of biometric applications such as iris recognition and particularly, but not exclusively, for the capture of facial images for facial recognition systems.

BACKGROUND

The performance of software based facial recognition systems is determined by many factors. Enhancements in this technology now mean that many facial parameters are taken into consideration by the software. These include facial geometry, specific identifying marks such as moles, skin colouration and more specifically its variation across the facial area. However, one common factor is the requirement for high- resolution good quality electronic images to enable the software to analyse these features, a relatively simple task to achieve if the subject poses at a specific distance from a high definition camera under controlled lighting conditions. In such a defined posed situation, all of the key elements for good image capture can be controlled. Additionally, the image transfer time from the image capture device to the computer performing the facial recognition task is not critical. In uncontrolled environments the target may be moving and not looking directly at the camera.

BRIEF SUMMARY OF THE DISCLOSURE

According to a first aspect of the present invention, there is provided a method of image processing for object recognition applications, the method comprising the steps of: using an image capture device to acquire a first, relatively low resolution image of a relatively wide field of view containing a plurality of objects; applying a pattern recognition algorithm to the first image so as to identify objects of interest; selecting at least one object of interest from the first image and determining its coordinates within the image; and using the determined coordinates to control either the image capture device or an additional image capture device so as to obtain a sequence of second, relatively high resolution images of a relatively narrow field of view containing the at least one object of interest.

Te first image is advantageously time multiplexed with the sequence of second images.

A single image capture device may be used to acquire both the first and the second images. Alternatively, a first image capture device acquires the first image, and a second image capture device acquires the second images.

The image capture device may be controlled so as to perform digital zoom and optionally digital pan and/or tilt in order to select the relatively narrow field of view.

Alternatively, the image capture device may be controlled so as to perform optical zoom and optionally electromechanical pan and/or tilt in order to select the relatively narrow field of view.

Preferably, only the second images are used for object recognition processing.

According to a second aspect of the present invention, there is provided apparatus for image processing for object recognition applications, the apparatus comprising: an image capture device adapted to acquire a first, relatively low resolution image of a relatively wide field of view containing a plurality of objects; means for applying a pattern recognition algorithm to the first image so as to identify objects of interest; means for selecting at least one object of interest from the first image and determining its coordinates within the image; and means for using the determined coordinates to control either the image capture device or an additional image capture device so as to obtain a sequence of second, relatively high resolution images of a relatively narrow field of view containing the at least one object of interest.

The apparatus may comprise means for time multiplexing the first image with the sequence of second images.

The apparatus may comprise a single image capture device for acquiring both the first and the second images, or may comprise a first image capture device for acquiring the first image, and a second image capture device for acquiring the second images.

The image capture device may be controlled so as to perform digital zoom and optionally digital pan and/or tilt in order to select the relatively narrow field of view, or may be controlled so as to perform optical zoom and optionally electromechanical pan and/or tilt in order to select the relatively narrow field of view.

The present application describes a technique that acquires good quality images of the facial features of targets in both ideal and non-ideal locations where the target may be stationary or moving and fast image capture is required. By adapting both the imaged area and image resolution the method transmits the minimal data required for good facial recognition. Embodiments of the invention also provide the means to acquire good quality facial images in areas where significant numbers of people are present. Embodiments of the invention remove the requirement for moving parts, filtering, encoding or other image processing at the image source. As a result, embodiments of the invention provide excellent or optimal image quality of single or multiple facial areas within a viewing region whilst maintaining a high frame rate. It must also be stated that any image based biometric, regardless of the specifics of the biometric and the light spectrum used, places a similar requirement upon the image capture device.

It is true to say that in image based biometric recognition in general, high resolution, noise free images of the area used for analysis are highly beneficial, and that multiple instances of these biometric regions may be present in any image captured using the full viewing cone of the capture device. Iris recognition is an example where the eyes form the object specific target within the full facial image. Analysis is generally constrained by time in any practical application; hence high throughput of relevant image data to a biometric recognition engine is of great importance. For reasons of clarity and brevity, we will continue to address the issue of face recognition, though it must be borne in mind that any image-based biometric could be substituted for the face.

In areas with significant target numbers, the opportunity to locate a large number of faces arises provided each face occupies a small area of the available image. However this prevents image recognition due to poor image detail. Within densely populated places, a high-resolution camera image could be captured and processed with high- resolution facial detail. Transmission of such images is slow, and a key element of good facial recognition in such dynamic situations is the use of a sequence of closely time-related images. Face recognition systems make a comparison between a previously taken reference image and current images. Because of the comparison process, different pose positions present problems to the recognition system. To minimise or reduces this problem in dynamic situations, a sequence of images taken in quick succession allow the recognition system to select the best pose within the sequence. It is therefore important to maintain high frame rates during image capture.

Embodiments of the present invention use dynamic object specific digital pan and zoom, where a down sampled, low resolution, typically VGA high-speed representation of the full high-resolution image is time multiplexed with high resolution sub regions containing facial detail. The data reduction allows multiple sub regions to be transmitted within single frame periods without significant reduction in frame rates. Such images may be derived from 1.3 to 8Mpixel sensors and may rise as camera technology improves.

The VGA resolution image is first transmitted to the host facial recognition computer, facial areas are located by head and/or eye location algorithms running on the host. The coordinates of the target specific area is passed back to the camera. The camera operates digital pan and zoom to send a high-resolution subset of the full image area but transmits only the data containing the areas of the identified targets. Several highspeed transmissions of these areas can be made to optimise the facial position for the image recognition software. This approach provides target area location and high- resolution facial images at frame rates significantly higher than 25fps. This has the potential for capturing and processing crowded areas within the image area at any one time; with suitable lenses this may be achieved at a distance of tens of metres from the camera.

Many existing facial recognition systems employ standard closed circuit TV (CCTV) based on CCD technology cameras, to acquire the image for recognition. In security applications the pixel resolution of these cameras is often below the normal maximums of 752(H) x 582(V) for PAL and 768(H) x 494(V) for NTS. In non-posed applications, mechanical control of camera tilt, pan and zoom can be employed to optimise the image size but the response time of these mechanical systems is often too slow to optimise each image capture [Alessio Del Bue, Dorin Comaniciu, Visvanathan Ramesh, Carlo Regazzoni, "Smart Cameras With Real-Time Video Object Generation". IEEE International Conference Image Processing, (ICIP'02), Rochester, NY, VoI 3, 429 - 432, 2002].

Alternatively, a compromise may be made where the image of the facial area is a small subset of the total available image area allowing target acquisition over a larger area. In this situation the active area is large but the detail is insufficient for good facial recognition. In PAL applications where some level of posing is possible, the facial image size may be around 50% of the total available image (376 x 291 pixels) and in non-posed applications, facial image sizes of 10% of the total available image (75 x 58 pixels) may be the only ones available. With a resolution of only 58 pixels in the vertical and a head size of 280mm, the geometry of the face can only be measured to a resolution of 280mm/58 = 4.8mm. This results in an uncertainty on the size of features such as eye sockets of 4.8mm x 2 = 9.6mm. This poor geometric resolution limits the accuracy of the geometric component of the facial recognition system. If skin defects such as moles are considered these are generally smaller than normal features and the ability to identify these marks is significantly compromised.

An alternative is to use high resolution CMOS or CCD imaging technology that currently produces images with resolutions up to 8 megapixels (8Mp) and will continue to rise as imaging technology develops. A typical 5Mp device at present provides a resolution of 2592(H) x 1944(V). A like-for-like comparison with the aforementioned PAL system gives a 194 pixel resolution in the vertical for an image area of 10% of the total. This results in a geometric resolution of 280mm/194 = 1.4mm under the same conditions. The 8Mp versions reduce this further to only 1.1 mm. This is a significant improvement but requires a large increase in data transmission between the camera and the processing computer.

In the PAL system described, the total number of pixels is 752 x 582 = 438Kp, whilst the CMOS system has a total number of 2592 x 1944 = 5Mp, which is an 1 1 -fold increase in the data transmission requirement. In addition, the CMOS device produces a 10bit digital value for each pixel, unlike the PAL system, which transmits an analogue representation of the image. As transmission standards such as USB are 8 bit or 16 bit orientated, 5Mp x16 bit = 80 megabit (80Mb) is required to transfer a full frame with 10 bit colour depth. With standard transmission methods such as USB1 and full speed USB2 only 12Mb/s speeds are available; this determines a frame transfer speed of 6.6 seconds. In many applications this is much too slow. Other transmission formats, such as high speed USB2, transmit data at 480Mb/s but this only achieves a frame rate of 6fps. This is significantly slower than the 25fps associated with PAL systems. This problem becomes even more difficult if an 8Mp or higher resolution imaging device is employed.

Coding systems may be employed within the imaging device, for example JPEG or MPEG. This will increase the image transfer rate but requires additional hardware within the camera to provide the coding functions and requires decoding at the host computer. Compression techniques also reduce image quality if high levels of compression are employed; this limits the level of compression available and hence reduces the available frame rate.

Data reduction techniques described in US 6,829,391 'Adaptive resolution system and method for providing efficient low bit Rate transmission of image data for distributed applications, describe complex filtering of the data where upon data in the target area has high resolution whilst data in the surrounding area has a low level of detail'. This is based on the log polar mapping [Dorin Comaniciu, Fabio Berton, Visvanathan Ramesh, "Adaptive Resolution System for Distributed Surveillance"; Real Time Imaging, VoI 8, No 5, 427-437, 2002 and also Dorin Comaniciu, Visvanathan Ramesh, "Robust Detection and Tracking of Human Faces with an Active Camera"; IEEE lnt Workshop on Visual Surveillance, Dublin, Ireland, 1 1-18, 2000] technique in US 5,103,306 'Digital image compression employing a resolution gradient', a technique highly suited to images containing a single face where a single region of interest exists. However, it becomes much more complex when multiple faces, requiring multiple overlaid foveation patterns, exist within a single image. This increase in complexity significantly increases the computational load and raises the required bandwidth of the transmission if downgraded image quality is to be avoided. The alternative is to restrict the method to single target foveation, selecting only single facial images at any time regardless of the actual number of interest regions present in the image.

Embodiments of the present invention use multimode transmission with target specific area transmission to achieve high definition, high-speed transmission of facial images. The implementation requires minimal computation within the video data stream, requiring only the formatting of data to conform to the chosen output format. The embedded controller operating outside the main video data stream is subsequently able to operate at lower speeds and implements the main control functionality. High-speed head and eye location algorithms have been available for some time and are used in the early stage of facial recognition to locate and isolate the head from the rest of the image. The resulting image is then processed for recognition. Embodiments of the present invention use head location algorithms running on the host face recognition-processing computer to locate the position of a face on a low-resolution (typically 640 x 480 pixels) image. This image is derived from a high resolution CMOS or CCD imaging sensor, where the high-resolution image has been down sampled from for example 5Mp to 300Kp. The down sampled image is transmitted via a high-speed digital link to the processing host. If high-speed USB2 is employed, frame rates of around I OOfps can be transmitted at VGA resolution, the actual frame rate is determined by the speed of the image capture device and the number of sub frames transmitted per full frame. On receipt of the low-resolution image, the head is located and the coordinates of the head location are transmitted back to the imaging sensor. The coordinates either define the location of a default window size, at which point the entire default window area is transmitted, or alternatively two coordinates are used, determined by the face location algorithm, that define two diagonally opposed corners. In the case of the opposed corners, a variable window size results where the transmitted image area contains minimal data not relating to the facial region of interest. Image transmission rates in this system are therefore increased due to the high level of redundant data removal. The high-resolution subset of the image area is transmitted back to the host processor, allowing full facial recognition to be applied to the high- resolution image area of the facial region of interest. Once the facial area is located the coordinates of the facial area may be recalculated from the high-resolution current image. The updated coordinates may then be retransmitted to the imaging system, allowing tracking of the subject within the total available image area.

Due to the high data rate transmission available with transmission media such as USB2 and the high level of data reduction within embodiments of this invention, multiple images can be transmitted within each frame period. This allows time division multiplexing to intersperse full area images with high-resolution facial images within a single frame period. The following example is based on a 640 x 480 pixel reference image with three facial windows. Each facial window is determined from the previous reference image. This allows the system to track and identify the subject in real time with high frame rates. Full reference image 640x480x16bit=4.9Mbit

Facial window 1 300x200x16bit=1 Mbit

Facial window 2 600x400x16bit=3.8Mbit

Facial window 3 700x450x16bit=5Mbit

Total transmission required = 14.7Mbit

Data rate available on USB2; 480Mbit/s

Frame Rate = 480/14.7 = 32 frames/sec

These high frame rate high-resolution images allow multiple images of a subject to be acquired while in motion, providing the opportunity to capture multiple facial images with the optimal facial orientation.

In applications where the distance between the camera and the host processor is too large for low cost digital transmission to be implemented, the digital interface may be replaced or supplemented by PAL or NTSC analogue transmission. An auxiliary digital communications line is employed to dynamically pass the digital pan, zoom and other commands to the camera system. If the auxiliary communications is in the form of RS485 or RS422, long cable runs may be employed allowing remote mounting of camera systems. However, frame rates in this configuration reduce due to the limited transmission bandwidth.

Throughout the description and claims of this specification, the words "comprise" and "contain" and variations of the words, for example "comprising" and "comprises", means "including but not limited to", and is not intended to (and does not) exclude other moieties, additives, components, integers or steps.

Throughout the description and claims of this specification, the singular encompasses the plural unless the context otherwise requires. In particular, where the indefinite article is used, the specification is to be understood as contemplating plurality as well as singularity, unless the context requires otherwise.

Features, integers, characteristics, compounds, chemical moieties or groups described in conjunction with a particular aspect, embodiment or example of the invention are to be understood to be applicable to any other aspect, embodiment or example described herein unless incompatible therewith.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the present invention and to show how it may be carried into effect, reference shall now be made by way of example to the accompanying drawings, in which:

FIGURE 1 shows an image of an eye region of a subject taken with a low resolution image capture device;

FIGURE 2 shows an image of an eye region of a subject taken with a high resolution image capture device;

FIGURE 3 shows an image from a high resolution image capture device viewing an area where faces may appear;

FIGURE 4 shows a facial region selected from the image of Figure 3;

FIGURE 5 shows an implementation of an embodiment of the invention;

FIGURE 6 shows an implementation of another embodiment of the invention; and

FIGURE 7 shows an implementation of a further embodiment of the invention.

DETAILED DESCRIPTION

Figure 1 shows an image of the eye region of a subject taken with a CCD camera with a resolution of 752 x 582 pixels. The camera has been rotated by 90 degrees to improve vertical resolution and resulting image rotated back through 90 degrees. The subjects' facial area occupies approximately 25% of the full image area.

Figure 2 shows an image of the same eye region taken with a 3.1 Mp camera with the same total image area. Only the 640 x 480 pixels in the facial area were transmitted thus reducing the data transmission demand significantly compared to transmission of the full 3.1 Mp image. Dynamic object specific systems only transmit the required data, therefore resolution and picture quality is significantly enhanced for a given data transmission rate, which results in much higher accuracy within the facial recognition system.

Figure 3 shows the image from a 3.1 Mp camera viewing an area where faces may appear. The object specific head and eye location identifies the area of the image that holds a valid facial region and passes the coordinates to the camera. The camera then selects this region 1 and transmits the high definition version of this image area, this being shown in Figure 4.

This process requires the transmission of a 640 pixel x 480 pixel reference image and 224 x 241 pixel facial image. The resulting data is 5.8Mbit at 16bit colour depth. With 480Mbit/second on high speed USB2 the transmission time is around 12ms.

Figure 5 shows an implementation of the invention such that the high-resolution image is converted to a digital bitmap. A camera system 2 comprises a high resolution image sensor 3 linked to an analog-to-digital converter 4, which in turn is linked to a digital communications interface (DCI) 5. The camera system 2 components are controlled by a control microprocessor 6. Image data is sent to a host computer 7, which includes a DCI 8, a face recognition system 9 and a system 10 for locating head and eye regions in the image. The system 10 is configured to transmit head and eye coordinates back to the control microprocessor 6 of the camera system 2.

Sub sampling and windowing is performed by the image sensor 3 under the control of the control microprocessor 6, which in turn derives the windowing control data from the remote host computer 7. Image data is transmitted to the host 7 from the camera 2 via a suitable interface such as USB2 or FireWire. The digital communications may incorporate suitable encoding such as JPEG or MPEG. In this implementation the reference full frame is transmitted first. The host 7 determines the window coordinates, and a sequence of sub frames requested by the host 7. Each sub frame is derived from a new image captured by the sensing device 3 and is therefore not at exactly the same time frame as the reference image.

Figure 6 shows an implementation such that the operation is as in Figure 5 with the addition of a full frame memory 1 1. In this implementation the full high-resolution image is stored in the memory 1 1. Down sampling is then performed by reading subsets of the image data stored in the memory 1 1. Windowing is also performed by reading subsets of stored data. This implementation generates a reference image and captures subset images at the same instant in time.

Figure 7 shows an implementation such that the image is transferred to the host 7 via analog video encoders 12, 13 as standard analogue transmission media such as PAL or NTSC (rather than digitally). In this implementation, sub frame transmission speeds remain the same as full frame transmission speeds and must be interposed with the full frame reference image. Image transfer rates in this embodiment are lower than in the digital transmission implementations

Claims

CLAIMS:

1. A method of image processing for object recognition applications, the method comprising the steps of: using an image capture device to acquire a first, relatively low resolution image of a relatively wide field of view containing a plurality of objects; applying a pattern recognition algorithm to the first image so as to identify objects of interest; selecting at least one object of interest from the first image and determining its coordinates within the image; and using the determined coordinates to control either the image capture device or an additional image capture device so as to obtain a sequence of second, relatively high resolution images of a relatively narrow field of view containing the at least one object of interest.

2. A method according to claim 1 , wherein the first image is time multiplexed with the sequence of second images.

3. A method according to any preceding claim, wherein a single image capture device is used to acquire both the first and the second images.

4. A method according to claim 1 or 2, wherein a first image capture device acquires the first image, and a second image capture device acquires the second images.

5. A method according to any preceding claim, wherein the image capture device is controlled so as to perform digital zoom and optionally digital pan and/or tilt in order to select the relatively narrow field of view.

6. A method according to any one of claims 1 to 4, wherein the image capture device is controlled so as to perform optical zoom and optionally electromechanical pan and/or tilt in order to select the relatively narrow field of view.

7. A method according to any preceding claim, wherein only the second images are used for object recognition processing.

8. Apparatus for image processing for object recognition applications, the apparatus comprising: an image capture device adapted to acquire a first, relatively low resolution image of a relatively wide field of view containing a plurality of objects; means for applying a pattern recognition algorithm to the first image so as to identify objects of interest; means for selecting at least one object of interest from the first image and determining its coordinates within the image; and means for using the determined coordinates to control either the image capture device or an additional image capture device so as to obtain a sequence of second, relatively high resolution images of a relatively narrow field of view containing the at least one object of interest.

9. Apparatus as claimed in claim 8, comprising means for time multiplexing the first image with the sequence of second images.

10. Apparatus as claimed in claim 8 or 9, comprising a single image capture device for acquiring both the first and the second images.

1 1. Apparatus as claimed in claim 8 or 9, comprising a first image capture device for acquiring the first image, and a second image capture device for acquiring the second images.

12. Apparatus as claimed in any one of claims 8 to 1 1 , wherein the image capture device is controlled so as to perform digital zoom and optionally digital pan and/or tilt in order to select the relatively narrow field of view.

13. Apparatus as claimed in any one of claims 8 to 1 1 , wherein the image capture device is controlled so as to perform optical zoom and optionally electromechanical pan and/or tilt in order to select the relatively narrow field of view.

14. A method of image processing for object recognition applications substantially as hereinbefore described with reference to or as shown in the accompanying drawings.

15. Apparatus for image processing for object recognition applications substantially as hereinbefore described with reference to or as shown in the accompanying drawings.