WO1996018976A1

WO1996018976A1 - Image processing

Info

Publication number: WO1996018976A1
Application number: PCT/GB1995/002942
Authority: WO
Inventors: Andrew Major
Original assignee: Snell & Wilcox Limited
Priority date: 1994-12-15
Filing date: 1995-12-15
Publication date: 1996-06-20
Also published as: GB9425328D0; EP0797813A1; AU4265696A

Abstract

In image processing, the coring procedure in a phase correlation process is improved by replacing a fixed noise threshold with a dynamically varying value, such as the RMS inter-field difference, over the block for which the Fourier transform is conducted. Ringing and overshooting in the correlation surface, is reduced by replacing a fixed window by a windowing function tailored to the actual spectral content of the signal.

Description

IMAGE PROCESSING

This invention relates to image processing and more particularly to the correlation of images, for example to identify movement. In one instance, the invention provides for the signal processing of video signals to provide a measure of correlation between two successive fields or other corresponding inputs. Correlation between successive video fields enables motion vectors to be identified in a video sequence.

Phase correlation is a known method by which motion vectors can be identified in a video signal. Reference is directed, for example, to:

PEARSON, HINES, GOLOSMAN, KUGLIN, 1977 "Video-rate image correlation processor." S.P.I. E. Vol. 119 Application of Digital Image

Processing (IOCC 1977);

GB-A-2 188 510

WO-A-92 05662

WO-A-9221201 WO-A-9305616

WO-A-9317520

WO-A-9317525

WO-A-9319430

WO-A-9401970 WO-A-9401830

Briefly stated, the phase correlation process takes blocks of luminance information at field intervals. A Fast Fourier Transfer (FFT) is performed to produce amplitude and phase signals. The phase signal is subtracted from a one-field-delayed signal to produce a phase difference signal. This is recombined with the amplitude signal in an inverse FFT to produce a correlation surface. By locating peaks in this correlation surface, candidate motion vectors can be identified for subsequent allocation on a pixel-by- pixel basis. It has previously been suggested that the amplitude signal be subjected to noise processing. A noise threshold is set, usually as a result of trial and error; components below the threshold are set to zero and components above the threshold are set to unity. Usually, the resulting, noise gated spectrum is then windowed. The form of the window is chosen to optimise the performance of the particular form of peak hunter chosen. One previous suggestion is to employ a Gaussian window with a peak hunter optimised to find quadratic peaks. The correlation surface undergoes logarithm processing before the peak hunter; the logarithm of a Gaussian produces a quadratic.

The prior approach is basically successful but has a number of limitations.

One limitation is that noise in the correlation surface hinders the accurate identification of peaks in the correlation surface, and thus reduces the likelihood of correct motion vectors being generated.

It is an object of one aspect of the present invention to provide an improved method of signal processing which is capable of producing more useful correlation surfaces.

Accordingly, the present invention consists, in one aspect, in a method of image processing to provide a measure of correlation between two images, comprising the steps of defining a set of corresponding samples in each image; performing transforms on said sample sets in each image to derive separate phase and amplitude signals; deriving a phase difference between said images; noise processing the amplitude signal by removing components beneath a noise threshold; and performing an inverse transform on the phase difference signal and noise-processed amplitude signal to provide a correlation signal, characterised in that the step of noise processing the amplitude signal comprises the steps of taking a measure of the amplitude difference between said images over said set and varying said noise threshold dynamically in response to variations in said measure. Preferably, the measure of the amplitude difference between said images over said set is an averaged value.

Suitably, the measure is a root mean square.

Advantageously, said set of samples comprises a corresponding block of pixels in successive video fields.

The present invention recognises that noise in the correlation surface can arise from large spectral differences. For example, when areas of the picture change in a manner which is not simply related to movement of an object between two fields, the associated phase difference information cannot help in identifying a motion vector. However, the associated amplitude signal - because it is large - is above any sensible noise threshold. Therefore, the "useless" phase difference information contributes fully to the correlation surface. This situation arises with regions of the picture that are revealed or obscured between the two fields of interest and also where there is motion which is exceptionally fast.

By dynamically varying the noise threshold in accordance with a measured amplitude difference, the method of the present invention is able to discriminate between - for example - a small but significant amplitude in two generally similar fields and a large but insignificant amplitude in fields which differ widely. In the former case, there is a reasonable prospect that the associated phase difference can be identified with movement; in the latter case, the differences between the two input fields are so great that no high precision measurement of motion is possible.

A second limitation of the prior art approach is related to the form of windowing. It is recognised that appropriate windowing in the frequency domain, which is equivalent to filtering in the time domain, is of considerable benefit in the subsequent flitering steps. Various types of window have been suggested and the particular benefit of a Gaussian window has already been mentioned. Hitherto, however, it has been assumed that the signal to be windowed is a full spectrum window that is to say a signal with significant contributions from all frequency bands in the bandwidth. The present invention, in a different aspect, recognizes that for significant numbers of real pictures, this assumption breaks down and breaks down in a manner which can now be seen to lead to material errors in motion estimation.

It is found, for example, that with picture material containing moving and out-of-focus objects, phase correlation according to the prior art tends to produce a correlation surface with spurious side peaks and undershoots which can completely mask a neighbouring "real" peak. With picture material containing a moving, one-dimensional object a similar ringing phenomenon is observed.

It is an object of the present invention, in a further aspect, to provide an improved method of signal processing which is capable of reducing significantly such impairments to the correlation surface.

Thus, in this further aspect, the present invention consists in a method of image processing to provide a measure of correlation between two images, comprising the steps of performing a transform to derive separate phase and amplitude signals; deriving a phase difference between said images; window processing the amplitude signal by attenuating components outside a defined frequency window; and performing an inverse transform on the phase difference signal and window-processed amplitude signal to provide a correlation signal, characterised in that the step of window processing the amplitude signal comprises the steps of determining the spectral content of the amplitude signal, generating a window function dynamically in accordance with said determination, and applying the window function to the amplitude signal.

Advantageously, the step of determining the spectral content of the amplitude signal comprises identifying those discrete frequencies at which significant amplitude information exists.

Suitably, a count is made of those discrete frequencies at which significant amplitude information exists.

The amplitude function will generally have a predetermined shape - such as a Gaussian - and will vary in dimension to fit the spectral content of the signal. Thus the width parameter, in one embodiment, is proportional to the count of those discrete frequencies at which significant amplitude information exists. Of course, other alternatives exist, such as selecting the width of the Gaussian to include a set proportion of said discrete frequencies.

With a full spectrum signal, a method according to the invention will produce the same result as the prior art. With a signal having an incomplete spectrum, such as the mentioned examples of out-of-focus or one- dimensional objects, however, the window in the present invention will reduce in frequency spread to ensure that sharp transitions are removed. In contrast, the prior art approach will reproduce a sharp frequency transition, if it falls towards the centre of the predetermined window.

The present invention will now be described by way of example with reference to the accompanying drawings, in which:-

Figure 1 is a block diagram illustrating a phase correlation approach according to the prior art; Figure 2 is a block diagram illustrating a modification according to the present invention;

Figures 3 and 4 are plots illustrating a simple moving object;

Figure 5 is a frequency spectrum for Figure 4.

Figure 6 illustrates a prior art approach to gating and windowing the spectrum of Figure 5;

Figure 7 is a plot similar to Figure 6 illustrating an approach according to the present invention;

Figure 8 illustrates a prior art correlation surface (depicted in one dimension) created from the amplitude terms shown in Figure 6; Figure 9 is a plot similar to Figure 8 illustrating a correlation surface achieved through use of the present invention;

Figures 10 to 16, correspond respectively with Figures 3 to 9, illustrating picture material having revealed detail;

Figures 17 to 23, correspond respectively with Figures 3 to 9, illustrating picture material having an out-of-focus moving object;

Figures 24 and 25 depict a moving one dimensional object in a two dimensional system; Figures 26 and 27 illustrate the spectrum of Figure 25, before and after coring, respectively;

Figure 28 depicts a prior art window function;

Figure 29 illustrates the prior art correlation surface resulting from use of the window depicted in Figure 28;

Figure 30 illustrates adaptive windowing according to the invention; and

Figure 31 illustrates the correlation surface resulting from use of the window depicted in Figure 30.

Referring to Figure 1, a known phase correlation process receives blocks of luminance picture information at field intervals, at input terminal (8). The blocks may be 64 by 64 pixels, overlapping such that every pixel of the picture is covered by four blocks. The sampled block has a Fast Fourier Transform performed on it in an FFT block (10), producing separate amplitude and phase signals, on lines (12) and (14), respectively.

The phase signal (14) passes to one input of a subtracter (16) directly, and to the other input of the subtracter through a one-field delay (18). The derived phase difference signal outputted from the subtracter (16) on line (20), being a measure of motion between the two fields, is taken to the phase input of an inverse FFT block (22).

The amplitude signal (12) from the FFT block (10) is taken through a noise gate (24). The principle behind noise gating is to normalise any significant component to full amplitude and to set to zero any components below the noise threshold. As a practical matter, the noise threshold is set empirically. The noise-gated amplitude signal is then passed through a frequency domain window (26). The purpose of the window is to filter the correlation surface, windowing in the frequency domain being the same as filtering in the time domain. The shape of the window is typically Gaussian and the size is such that the window decays to zero at the edge of the (assumed complete) spectrum. The window is intended to maximise the amount of available band width without introducing ringing on the correlation surface. Ringing is produced by having a sharp transition in the spectrum. The noise-gated and windowed amplitude terms (28) are then taken to the inverse FFT block (22).

The result of the inverse FFT is a correlation surface containing peaks, where the position of a peak represents a motion between the two fields. A peak hunter (30), which may for example be as described in WO-A- 94 01830, operates on the correlation surface to produce a menu of motion vectors for subsequent selection and allocation pixel-by-pixel.

Turning to Figure 2, there is illustrated a modification - according to the present invention - intended to replace the noise gate and window of

Figure 1.

The amplitude signal (12) identified from Figure 1 , is supplied directly and through one field delay (130) to an arrangement (132) serving to derive the RMS value of the inter-field spectral difference. This arrangement comprises a subtracter (134), a modulus block (136), a squaring block (138), a summing block (140) and a square root block (142). The summing and square root blocks are controlled through a block-enable signal provided at a terminal (144). In this way, the RMS inter-field spectral difference is available block-by-block. This difference signal forms the noise threshold for the otherwise conventional noise gate (24) identified from Figure 1.

Thus, the output from arrangement (132) is taken as the threshold signal to noise gate (24) which through a balancing block delay (146), also receives the amplitude signal.

It will be understood that the described arrangement (132), providing the RMS value of the difference between the two fields over the block, is but one example of numerous arrangements capable of providing a measure of the amplitude difference between two images over the pixel block (or other set of samples).

The noise-gated amplitude terms pass in turn to a spectral distribution measurement block (148) and an amplitude component generation block (150). As will be described in more detail, the amplitude component generation block (150) generates components by forming the product of the noise-gated amplitude terms with a window function derived from a measurement of spectral distribution in block (148). This may be contrasted with the prior art approach of applying a fixed window function.

The effects of the two improvements provided by this invention can now be understood in process terms.

With the frequency spectrum being expressed in discrete, digital form, it is helpful to regard the spectrum as consisting of "bins" which should, after the noise elimination and normalisation of the gate (24), either contain sensible phase information or be empty. The bins which contain sensible phase information are referred to as "active bins".

The first stage of the process, is to find the RMS level of the spectral difference between the two fields, measured over the total block area.

rms_dlfference -

This is the function performed block by block in arrangement (132).

The rms_difference value is then used to core the current input frame spectrum in gate (24):-

cored_spec = If (bin > rms_difference, then 1, else 0)

As previously noted, the present invention recognises that "noise" in the correlation surface, in the sense of information which is not related to movement between the two fields, can arise from large spectral differences. When areas of the picture change in a manner which is not simply related to movement of an object between two fields, the associated phase difference information cannot help in identifying a motion vector. However, the associated amplitude signal - because it is large - is above any sensible, fixed noise threshold applied according to the prior art. Therefore, the "useless" phase difference information contributes fully to the correlation surface. This situation arises with regions of the picture that are revealed or obscured between the two fields of interest and also where there is motion which is exceptionally fast.

By conducting the coring process with respect to an RMS measure of the difference between the two fields over the block, the method of the present invention is able to discriminate between - for example - a small but significant amplitude in two generally similar fields which is likely to be associated with an identifiable motion vector and a large but insignificant amplitude in fields which differ widely.

The total number of active bins is then counted to provide the "active bins factor" :- active_bins = Σ_{ωΛ→1J mjMr} bin ^.^^

With the known system, the amplitude component used for the inverse FFT is the noise-gated and windowed input spectrum. The present invention generates an amplitude component which is (in this case) a Gaussian that fits the available spectrum.

The size of the Gaussian used is related to the number of active bins. This can be achieved in a variety of ways and two examples are:-

Size - ^act^^ve~bins const

The size of the Gaussian in this example is purely a function of the total number of active bins.

Size - radius_where_75% bins_actlve const a

In this alternative, the spectrum is - effectively - searched from the

DC term outwards until 75% of the active bins have been found. The amplitude component is then found from:-

ampHtυde component - exp ^~0t* * Y²) size²

Turning now to the remaining figures, the advantages of the present invention will be discussed with reference to different types of picture information, contrasting in each case the results of using the prior art approach and the results achievable according to the present invention.

The example is taken in Figures 3 to 9 of a simple one dimensional, moving object. It will be seen later that in this highly stylized example, there is little to choose between the invention and the prior art, but the example is nonetheless useful for comparison with subsequent, more representative examples.

Figures 3 and 4 illustrate schematically a simple object moving from field 1 to field 2. The spectrum is shown in Figure 5; this can be from either field but is more conveniently from field 2 to avoid the use of a delay. The spectrum shows the amplitude value for each of the frequency bins utilised in the FFT.

In the prior art approach, a fixed noise threshold is employed, for example at the level A shown in Figure 5. There can be seen from Figure 6 the effect of prior art noise gating at this level A; amplitude terms beneath the threshold are zeroed and all others are set to unity. Figure 6 also shows the fixed Gaussian window function extending over the expected frequency range. It can be seen that as a result of the chosen noise level, a significant number of the frequency bins within the fixed window are empty. The resulting correlation surface (in one dimension) is shown in Figure 8. According to the present invention, the amplitude values are cored not a predetermined fixed threshold, but at the level for the block in question of the RMS field difference. In view of the close correspondence between field 1 and field 2 (or high inter-field spectral match), this value is low, shown by way of illustration at B in Figure 5. There is accordingly a high number of active bins; therefore a wide amplitude component is generated, corresponding with a broad window. Looking at the resulting correlation surfaces in Figure 8 for the prior art approach and in Figure 9 for the embodiment of this invention, it will be seen that the results of the adaptive and non-adaptive schemes are closely similar in terms of detectable peaks.

It can be observed that the surface of Figure 9 has reduced overshooting, but the overshooting of the prior art - in this stylized example - is probably not objectionable.

Turning now to Figures 10 and 11 , there is shown an example of a moving object with revealed detail. The spectrum is shown in Figure 12 and the prior art noise gating and windowing (utilising a fixed noise threshold shown at A) is depicted in Figure 13. It should be noted that a large number of the frequency bins within the window are not active.

In the same way as in the previous example, Figure 14 shows first the result of coring at the RMS level, depicted at B. This produces more active bins than the arrangement of Figure 14, but less active bins than in Figure 7 of the previous example; due to the revealed differences resulting in a low spectral match factor. Because there are fewer active bins, a narrower amplitude term is derived. Indeed, in comparison with Figure 13, the "window" is very much narrower. The practical effects of these distinctions are evident from comparison of the respective correlation surfaces in Figures 15 and 16, respectively.

The non-adaptive scheme of the prior art shown in Figure 15, produces an unsatisfactory result with large under shoots and very poor signal to noise ratio. The adaptive scheme produces in Figure 16 a slightly softer peak, reduced under shoots and improved signal to noise ratio. Compared to the previous example, the coring level or threshold is significantly higher; hence the correlation surface is cleaner but a bit softer. The example is now taken of picture material in which an incomplete spectrum results from out-of-focus images. Figures 17 and 18 depict the moving, out-of-focus object and the incomplete nature of the spectrum is clearly apparent from Figure 19. The inadequacies of the prior art approach are seen most clearly in Figure 20, where a very small fraction of the frequency bins within the window are active. In contrast, the approach of the present invention produces, as seen in Figure 21 , more active bins and a window which is tailored to fit the available spectral information. Turning to the correlation surfaces of Figures 22 and 23, totally unacceptable results are seen with the non-adaptive scheme of the prior art. The 100% undershoots which are produced at either side of the peak could totally remove other real motion peaks which fall at the same position. There are also two spurious peaks produced. The adaptive scheme of the present invention manages to produce one soft peak with virtually no ringing.

With this example, the spectral match factor is high, which allows nearly all of the available spectrum through the noise gate. But there are very few bins active, hence a narrow amplitude term is selected.

The final example that will be taken is of a one dimensional object in a two dimensional system. Figures 24 and 25 depict a moving bar. The resultant spectrum is shown in Figure 26 and after coring, in Figure 27. In contrast to the previous examples, the windowing operations of the prior art and the invention are depicted in separate plots, Figures 28 and 30, respectively. Figure 28 shows the conventional Gaussian; Figure 30 shows the result of tailoring the window to fit the amplitude spectrum of Figure 27.

The non-adaptive processing of Figures 28 and 29 is seen to result in a very smeary peak; reliable vectors are unlikely to be found. The amount of ringing is also a problem; it can seriously interfere with other true peaks. With the adaptive scheme of the present invention, seen from Figures 30 and 31 , a low number of active bins leads to the selection of a very small amplitude component. The resulting correlation has one soft circular peak, a much improved result.

Whilst the techniques of adapting a noise threshold to a measure of the amplitude difference between fields, and generating a window function dynamically in accordance with the spectral content of the signal, are particularly useful and synergistic in combination, there will be applications in which the techniques can be applied separately. It should be understood that in both its aspects, the invention has been described by way of examples only and a wide variety of modifications are possible without departing from the scope of the claimed invention.

Numerous techniques will exist, for example, for taking a measure of the amplitude difference between fields or other images to be correlated and varying a noise threshold in response to variations in this measure. Examples have been given of techniques for determining the spectral content of the amplitude signal and generating a window function in accordance with that determination. The approach of counting discrete frequencies at which cored amplitude information is present, has the benefit of simplicity and leads to straightforward techniques for selecting the width of an appropriate Gaussian window function. With amplitude terms then being "0" or "1", the step of applying the window function to the amplitude signal, occurs effectively simultaneously with generation of the window. Despite these advantages, other techniques may be useful in dtermining the spectral content, generating a dynamic window function and applying that function to the amplitude signal.

Claims

1. A method of image processing to provide a measure of correlation between two images, comprising the steps of defining a set of corresponding samples in each image; performing transforms on said sample sets in each image to derive separate phase and amplitude signals; deriving a phase difference between said images; noise processing the amplitude signal by removing components beneath a noise threshold; and performing an inverse transform on the phase difference signal and noise-processed amplitude signal to provide a correlation signal, characterised in that the step of noise processing the amplitude signal comprises the steps of taking a measure of the amplitude difference between said images over said set and varying said noise threshold dynamically in response to variations in said measure.

2. A method according to Claim 1 , wherein the measure of the amplitude difference between said images over said set is an averaged value.

3. A method according to Claim 2, wherein the measure is a root mean square.

4. A method according to any one of the preceding claims, wherein said set of samples comprises a corresponding block of pixels in successive video fields.

5. A method of image processing to provide a measure of correlation between two images, comprising the steps of performing a transform to derive separate phase and amplitude signals; deriving a phase difference between said images; window processing the amplitude signal by attenuating components outside a defined frequency window; and performing an inverse transform on the phase difference signal and window-processed amplitude signal to provide a correlation signal, characterised in that the step of window processing the amplitude signal comprises the steps of determining the spectral content of the amplitude signal, generating a window function dynamically in accordance with said determination, and applying the window function to the amplitude signal.

6. A method according to Claim 5, wherein the step of determining the spectral content of the amplitude signal comprises identifying those discrete frequencies at which significant amplitude information exists.

7. A method according to Claim 6, wherein a count is made of those discrete frequencies at which significant amplitude information exists.

8. A method according to any on of Claims 5 to 7, further comprising the step of noise processing the amplitude signal by taking a measure of the amplitude difference between said images and varying said noise threshold dynamically in response to variations in said measure.

9. A method according to Claim 8, wherein the measure of the amplitude difference between said images is an averaged value.

10. A method according to Claim 9, wherein the measure is a root mean square.