WO2006072897A1

WO2006072897A1 - Method and device for detecting transparent regions

Info

Publication number: WO2006072897A1
Application number: PCT/IB2006/050007
Authority: WO
Inventors: Ahmet Ekin
Original assignee: Koninklijke Philips Electronics N.V.
Priority date: 2005-01-04
Filing date: 2006-01-02
Publication date: 2006-07-13

Abstract

The present invention concerns a method of detecting transparent regions in frames, comprising the steps of computing (4) a predetermined parameter for several parts of a frame, determining (6) a variation factor of said parameter for said parts of the frame, and determining (10) a likelihood of transparency for several regions of the frame by comparing the variation factors of the parts of the frame included in each region with predetermined values. This invention can be used for the determination of motion vectors and enhancement of transparent regions.

Description

METHOD AND DEVICE FOR DETECTING TRANSPARENT KILUILWNS

The present invention relates to a method of detection of transparent regions in frames, and to a corresponding detection device.

BACKGROUND OF THE INVENTION

Transparent overlays are used frequently in video and picture frames so that text and/or graphics can be superimposed on transparencies while simultaneously preserving the visibility of the original scenes. Numerous frame processing methods exist, as for example the motion event detection method for video indexing, described in the document EP 0 805 405. In this method, two metrics are used to detect an object that appears in one frame but does not in the other. It is based on a simple binary image AND operation. Detection of the objects is achieved by comparing changing areas in successive frames.

However, in the state of the art, there is no solution allowing to detect transparent overlays in picture frames or in video frames. This raises limitations in the processing of picture frames and video frames and also leads to incorrect application of motion estimation algorithms in video frames. Currently, in video frames, if the background, which is visible through transparency, is moving while the foreground comprising any text, graphics and other material superimposed on the transparent region is stationary, motion estimation algorithms usually produce inaccurate and inconsistent motion vectors which degrade the performance of motion compensated de-interlacing algorithms.

Therefore it is desirable to develop a new method to detect transparent regions in frames both in picture frames and video frames.

SUMMARY OF THE INVENTION

Accordingly, it is an object of the invention to provide a new method of detection of transparent regions in frames.

To this end, the invention relates to a method characterized in that it comprises the steps of :

- computing a predetermined parameter for several parts (B) of a frame;

- determining a variation factor of said parameter for said parts of the frame ; and - determining a likelihood of transparency for several regions (R) of the frame by comparing the variation factors of said parts of the frame included in each region with predetermined values. In an advantageous solution, computing a predetermined parameter is acmeveα on a pixel basis, one parameter value being computed for each pixel and said parts for which variation factors are determined being said pixels.

In another advantageous solution, computing a predetermined parameter is achieved on a block of pixels basis, one parameter value being computed for each block of pixels and said parts for which variation factors are determined being said block of pixels. With either of these two solutions, computing a predetermined parameter may include computing the variance of a parameter of the frame. Moreover, said predetermined parameter may be selected in the group including the intensity and the different colour channels Said parameter is preferably the energy, and, in this case, computing the parameter may include determining and normalizing the brightess. In this case also, the variation iactor may be a spatial one, determined by comparing the parameter of parts of the frame one to another.

Whatever these two solutions and their additional features, it may be advantageous that determining a likelihood of transparency includes:

- converting said regions of the frame into a binary image by assigning a specified binary value for each part of the regions for which the variation factor meets a predetermined requirement and the opposite binary value for each other part of the regions;

- evaluating a filling ratio of said specified binary value for each region of said frame; and

- comparing the filling ratio of each region to predetermined values to determine its likelihood of transparency, or, in a variant, that determining a likelihood of transparency includes:

- converting said regions of the frame into several binary images (/^₁ ) by assigning a specified binary value for each part of the regions for which the variation iactor meets one of predetermined requirements and the opposite binary value for each other part of the region;

- evaluating a requirement filling ratio of said specified binary value for each region of each binary image; - computing of a filling ratio for each region of the frame based upon said requirement filling ratios; and

- comparing the filling ratio of each region to predetermined values to determine its likelihood of transparency. Another object of the invention is to provide a device for detecting transparent regions in frames, characterized in that it comprises:

- a unit adapted for computing a predetermined parameter for several parts (B) of a frame; - a unit for determining a variation factor of said parameter for said parts (B) of the frame; and

- a unit for determining a likelihood of transparency for several regions (R) of the frame by comparing the variation iactors of the parts (B) of the frame included in each region (R) with predetermined values, said device being adapted to implement the method as described above.

These and other aspects of the invention will be described in greater detail hereinafter with reference to drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is a flow chart of the method of detecting transparent regions ; and Figures 2A to 2D are symbolical representations of frames as they appear during the application of the method described in figure 1.

DETAILED DESCRIPTION OF THE INVENTION

Referring to figure 1, a method of detecting transparent region in a frame is illustrated. In this embodiment the frame is a video frame that is part of a sequence of frames and thus has a previous frame and a following frame.

The method begins with a step 2 of determination of changing areas in the frame, which is achieved on a time basis by means of a comparison of the current frame at an instant t+1 and the same frame at the previous instant t, in order to detect the absolute differences between the current frame and its previous frame. Step 2 also includes global motion compensation achieved in a conventional manner. Accordingly the segments of the current frame and the previous frame that are compared one to another are corresponding segments after global motion compensation. Current frame and previous frame are respectively identified by references I₁₊₁ and I_t in figure 2A, while changing areas are identified by reference A. The method continues by computing a predetermined parameter, for example the variance of a parameter of the frame, in a step 4. In the described embodiment, the parameter used is the intensity of the frame, which is determined for blocks of pixels of the frame of predetermined width and height. These blocks are identified by a letter B in figure 2A. Step 4 is achieved only for the changing areas A of the frame for both the current frame I_t+1 and the previous frame I_t in order to deliver a variance value for each block of pixels of the changing areas and for both frames. The exact computation of the variance is achieved in a conventional way and the resulting variance for one block of pixels is indicated as σ_t+1 ² for the current frame and σ ² for the previous frame.

Thereafter the method continues by determining a variation factor of the variance of the intensity for several parts of the frame in a step 6. In the embodiment described, this variation factor is a temporal variation factor determined by comparing the variance of each relevant part of the frame, or block of pixels of the frame over the same block of pixels of the frame at a previous time. Most specifically the variation factor for the current frame is indicated as V₁₊₁ and is computed as a ratio between the current value of the variance σ_t+1 ² and the previous value of the variance σ₍ ².

Advantageously a small number indicated as "eps" is added to the variance value of the previous frame such that the variation factor V₁₊₁ is obtained by computing a ratio between two values of variance according to the following equation:

'^+U '^{y )} o_t ²(x,y) + eps

In this equation (x,y) identifies the coordinates of the processed block of pixels in the frame. Value "eps" is a small number that is inserted to overcome numerical instabilities such as division by zero. This variation factor reflects the transparency β of the block, where β indicates the contribution of the transparency and (1- β) indicates the contribution of the original parameter to the value of the block in the current frame. In case there is a transparent overlay on this block, with time its variation iactor V_t+1 will converge toward one. In the embodiment described, if the current frame I_t+1 has transparent regions and the previous frame I_t has not, the variation factor varies in an interval extending from 0 to 1.

The method continues by determining a likelihood of transparency for several regions of the frame by comparing the variation factors of the parts of the frame included in each region with predetermined values in a step 10. This step 10 includes a substep 12 of converting the current frame into binary images by assigning a specified binary value for each part of the frame for which the variation factor meets a predetermined requirement and the opposite binary value for other parts of the frame, where said variation factor does not meet it. The substep 12 of converting the frame into binary images is repeated over several predetermined requirements that are that the variations factors fall within predetermined intervals of values. Accordingly, a distinct binary image is generated for each predetermined requirement by comparing the variations factors with an interval of values corresponding to the respective requirement. More precisely the possible interval of 0 to 1 is divided into k intervals, for example 10 intervals with equal length and half overlapping parts. Of course other values could be used and intervals can be of various lengths and not overlapping at all or at least two overlapping one to another. Accordingly, k binary images are obtained as represented in figure 2B, by identifying the blocks of pixel of the frame for which the variation factor falls into the current segment and by assigning to these blocks of pixels the binary value of 1 and 0 to the other ones. Substep 12 is achieved only on changing areas A and is represented in figure 2B, each binary image being represented by the reference /^₁ . Substep 12 is followed by a substep 14 of processing of the binary images including binary morpho logical image processing. This operation of binary morphological image processing is achieved in a conventional way by first performing closing operations and then connect-component operations, to identify connected segments, and it allows determining of segments in each of the binary images, each segment comprising the different connected blocks of pixels having the binary value of 1. These segments are identified by the letter S in figure 2C. Substep 14 also includes the filtering of the binary images to eliminate noise by, for example, removing the small sized segment of the binary images.

The step 10 of determining a likelihood of transparency comprises then a substep 16 of automatic shape analysis of the segments by application of predetermined shapes over the different parts for which the variation factor has been determined. In the example, this is achieved by finding a minimum bounding rectangle for each surviving segment.

Substep 16 is followed by substep 18 for computing a filling ratio for each segment of the frame, as a percentage of the number of blocks of pixels having a binary value of 1 within the respective minimum bounding rectangle. Advantageously, substep 18 also includes removing minimum bounding rectangles that have a lower filling ratio than a predetermined threshold. Substep 18 is followed by substep 20 of verification of the existence of one or all of horizontal and/or vertical edges at the boundaries of each segment. Following the substep 20, the binary images appear as represented in figure 2C, each segment being replaced by a corresponding rectangle. In the example, a filling ratio is determined for each requirement tnat is tor eacn of the k intervals to which the variation iactor is compared. This filling ratio is determined for each segment of each binary image. The filling ratio for a region of the frame is based upon the different filling ratios which are computed. These regions are identified by the letter R on figure 2D. For example, the filling ratio for a region is equal to the maximum value of the different requirement filling ratios of the segments that are comprised within this region. In another embodiment, the filling ratio is equal to the average of the maximum value and its surrounding values. For example, if the maximum value is that of the fifth interval, the filling ratio is equal to the average of the requirement filling ratios of intervals four to six. Furthermore, in the example the method comprises the verification of the regions to determine if it contains any text and if such, it includes also increasing the likelihood of transparency. The method of the invention permits to determine the likelihood of transparency for different regions of a frame, this information being used as a criterion for further processing in another processing step designated 30 in figure 1. For example, the method of detecting transparent regions as described above is used in a method of determination of motion vectors in a video frame wherein confidence of motion vectors is based upon likelihood of transparency.

Further processing 30 can also include a modification of the transparency factor of the regions and an estimation of transparency colour. In this case, the transparency is computed in an accurate way. For each region for which the likelihood of transparency is sufficient, the following relation will apply:

/,₊₁ (*,;v) = β xr(*.;y) + (l - β)x/, (*. ;y)

In this equation, β is the transparency coefficient and T is the transparency intensity which is independent of the instant t, these two factors being unknown. This equation can be applied for several parameters and especially for the intensity and the different colour channels. By applying this equation to groups of at least two pixels the unknown values of β and T can be computed, for example by using a linear estimator such as least square or any convenient mathematical tool. To stronghold the estimation of the transparency factor and the transparency colour, computation is achieved on larger groups of pixels such as the blocks or the regions defined above. If the frame processed is extracted from a sequence, the above equation applies after global motion compensation. Accordingly, the references (x,y) do not identify an absolute location but the corresponding locations between two different frames. Another application achieved in step 30 includes enhancement ot me transparent regions such as removal of transparency, changing transparency coefficient and detection of overlay graphics, for example.

Many additional embodiments are possible. For example, the method of the invention is applied to the entire frames and not only to the changing areas, or the changing areas are determined upon a spatial basis by comparing parts of the frame one to another.

In another embodiment, the parameter used is the value of any colour channel like the value of a channel selected in the group consisting of green, red and blue channels, the computing also comprising the computing of the variance of the parameter. In another embodiment, the parameter used is the energy, in which case its computing advantageously includes determining and normalizing the brightness in an attempt to smooth the differences due to the different brightness of each colour. The energy of a pixel corresponds to the square value of its intensity and can be computed for each of the different colour channels as well as for the intensity. In such an embodiment, the variance of the energy is not computed as it would be zero on a non-textured surface, and the variation factor is computed directly with the parameter values, for example by a ratio between two different values. If the transparency is brighter than the background, the ratio will be comprised between 0 and 1 and the process will continue as described above. In the opposite case, the ratio is comprised between 1 and the infinite. Accordingly, the variation iactor is equal to the inverse of the ratio and the process continues.

Yet, in another embodiment, the computing of the parameter is achieved on a pixel basis, one parameter value and one variation factor being computed for each pixel.

In any embodiment, if the computed variance is zero, the block upon which it is calculated is increased in size to compute a non-zero variance. If a very large area is smooth, the likelihood of transparency drops to zero. Yet, in such cases, the use of another parameter like the energy may be adapted.

In another embodiment the variation factor is a spatial variation iactor determined by comparing a parameter of parts of the frame one to another. This is particularly adapted for frames that are not part of a sequence, like picture frames and also for the first frame of a sequence which has no previous frame, such as a new video shot.

In another embodiment, determination of the likelihood of transparency comprises only one step of conversion of the frame into a single binary image and one step of evaluation of a filling ratio for the regions, achieved upon one single predetermined requirement. In such an embodiment, the step 10 comprises the evaluation ot a tilling ratio ot a specified binary value achieved directly for each region of the frame. The determined filling ratios are then compared to predetermined values to determine the likelihood of transparency. Other embodiments are also possible using different requirement determined filling ratios and different mathematical relations between the filling ratios and the likelihood of transparency. For example, the likelihood of transparency is computed by applying probabilistic patterns and corresponding coefficients to the filling ratios.

The method of the invention can be achieved by various devices, such as computers and the like or dedicated devices, adapted to achieve the method as described above in any of the embodiments.

In one embodiment, such a device has :

- a unit adapted for computing a predetermined parameter for several parts of a frame; - a unit for determining a variation factor of said parameter for said parts of the frame; and

- a unit for determining a likelihood of transparency for several regions of the frame by comparing the variation iactors of the parts of the frame included in each region with predetermined values. The method of the invention can also be carried out by a computer program product for a processing unit comprising a set of instructions, which, when loaded into said processing unit, causes the processing unit to carry out the method as described above. There are numerous ways of implementing functions by means of items of hardware or software, or both. In this respect, the drawings are very diagrammatic, and represent only possible embodiments of the invention. Thus, although a drawing shows different functions as different blocks, this by no means excludes that a single item of hardware or software carries out several functions. Nor does it exclude that an assembly of items of hardware or software or both carry out a function.

The remarks made herein before demonstrate that the detailed description, with reference to the drawings, illustrates rather than limits the invention. There are numerous alternatives, which fall within the scope of the appended claims. Any reference sign in a claim should not be construed as limiting the claim. The words "comprising" or "comprise" do not exclude the presence of other elements or steps than those listed in a claim. The word "a" or "an" preceding an element or step does not exclude me presence ot a plurality of such elements or steps.

Claims

1. A method of detecting transparent regions in frames, characterized in that it comprises the steps of :

- computing a predetermined parameter for several parts (B) of a frame; - determining a variation factor of said parameter for said parts of the frame ; and

- determining a likelihood of transparency for several regions (R) of the frame by comparing the variation factors of said parts of the frame included in each region with predetermined values.

2. A method according to claim 1 , wherein computing a predetermined parameter is achieved on a pixel basis, one parameter value being computed for each pixel and said parts for which variation iactors are determined being said pixels.

3. A method according to claim 1, wherein computing a predetermined parameter is achieved on a block of pixels basis, one parameter value being computed for each block of pixels and said parts for which variation factors are determined being said block of pixels.

4. A method according to any of claims 1 to 3, wherein computing a predetermined parameter includes computing the variance of a parameter of the frame.

5. A method according to claim 4, wherein said predetermined parameter is selected in the group including the intensity and the different colour channels.

6. A method according to any of claims 1 to 3, wherein said parameter is the energy.

7. A method according to claim 6, wherein the parameter is the energy and wherein computing the parameter includes determining and normalising the brightness.

8. A method according to any of claims 1 to 6, wherein said variation iactor is a spatial variation factor determined by comparing the parameter of parts of the frame one to another.

9. A method according to any of claims 1 to 8, wherein said frame is part of a sequence of frames and said variation iactor is a temporal variation factor determined by comparing the parameter of each part of the frame (I₁₊₁ ) over the same part of the frame (l_t ) at a different time.

10. A method according to any of claims 1 to 9, wherein determining a likelihood of transparency includes: - converting said regions of the frame into a binary image by assigning a specified binary value for each part of the regions for which the variation factor meets a predetermined requirement and the opposite binary value for each other part of the regions;

- evaluating a filling ratio of said specified binary value for each region of said frame; and - comparing the filling ratio of each region to predetermined values to determine its likelihood of transparency.

11. A method according to any of claims 1 to 9, wherein determining a likelihood of transparency includes: - converting said regions of the frame into several binary images (/^₁ ) by assigning a specified binary value for each part of the regions for which the variation iactor meets one of predetermined requirements and the opposite binary value for each other part of the region;

- evaluating a requirement filling ratio of said specified binary value for each region of each binary image;

- computing of a filling ratio for each region of the frame based upon said requirement filling ratios; and

- comparing the filling ratio of each region to predetermined values to determine its likelihood of transparency.

12. A method according to claim 11 , wherein said predetermined requirements are that the variation factors fall within predetermined intervals of values.

13. A method according to any of claims 10 to 12, wherein said determination of likelihood of transparency comprises, for at least one binary image, its processing to determinate connected segments (S).

14. A method according to claim 13, wherein said processing of the binary image includes at least one processing comprised in the group consisting of binary morphological image processing comprising closing operations and connect component operations, and noise filtering.

15. A method according to any of claims 1 to 14, wherein said determination of likelihood of transparency comprises automatic shape analysis of the regions in the frame by application of predetermined shapes over the different parts of the frame for which the variation factor has been determined.

16. A method according to the claim 15, wherein said automatic shape analysis comprises finding minimum bounding rectangles including said different parts.

17. A method according to any of claims 1 to 16, wherein said determination of likelihood of transparency includes verifying the existence of one or all horizontal and/or vertical edges of said regions.

18. A method according to any of claims 1 to 17, wherein said determination ot likelihood of transparency includes verifying if a region has a text area and increasing its likelihood of transparency if such.

19. A method according to any of claims 1 to 18, characterized in that it includes the determination of changing areas (A) and in which further processing is achieved only on these changing areas.

20. Method of determination of motion vectors in a video frame, characterized in that it comprises a method of detecting transparent regions according to any of claims 1 to 19 and it further comprises determining confidence about motion vector based upon likelihood of transparency.

21. A device for detecting transparent regions in frames, characterized in that it comprises:

- a unit for determining a likelihood of transparency for several regions (R) of the frame by comparing the variation iactors of the parts (B) of the frame included in each region (R) with predetermined values.

22. A device according to claim 21, wherein the device is adapted to achieve the method claimed in anyone of claims 1 to 20.

23. A computer program product for a processing unit comprising a set of instructions, which, when loaded into said processing unit, causes the processing unit to carry out the steps of the method claimed in anyone of claims 1 to 20.