CA2088082C

CA2088082C - Dynamic bit allocation for three-dimensional subband video coding

Info

Publication number: CA2088082C
Application number: CA002088082A
Authority: CA
Inventors: John Hartung; Arnaud E. Jacquin; Thomas Andrew Michel; Christine Irene Podilchuk
Original assignee: American Telephone and Telegraph Co Inc
Current assignee: AT&T Corp
Priority date: 1992-02-07
Filing date: 1993-01-26
Publication date: 1999-01-19
Anticipated expiration: 2013-01-26
Also published as: DE69322531T2; CA2088082A1; EP0555016B1; JPH0646403A; JP2915238B2; DE69322531D1; EP0555016A2; US5309232A; EP0555016A3; US5481308A

Abstract

The present invention provides a method of coding a set of multi- level signals, such as video signals. The method comprises filtering the set of multi-level signals into one or more sets of multi-level sub-band signals; allocating a first quantity of bits for use in coding a first set of sub-band signals; coding the first set of multi-level sub-band signals using zero or more of the first quantity of allocated bits;
allocating a second quantity of bits for use in coding a second set of sub-band signals, the second quantity of bits comprising zero or more of the first quantity of bits not used in coding the first set of multi-level sub-band signals; and coding the second set of multi-level sub-band signals using zero or more of the second quantity of allocated bits. Three examples of the method are provided. A hardware implementation involving the use of digital signal processors operating in parallel is also provided.

Description

DYNAMIC BIT ALLOCATION FOR THREE-DIMENSIONAL
SUBBAND VIDEO CODING

Field of the Invention This invention relates to the efficient digital coding of multi-valued 5 signals, and more particularly, to the digital coding of video signals at low rates where the rate is fixed for each video frame.

Back~round of the Invention Good quality, low bit rate video coding is required for such appli-~tic)ns as tel~ol~re.~ cing over exi~ting and future netwol~s, as well as CD-ROM storage.
10 An effective low rate coder should remove the red~ln-l~ncies due to temporal and spatial correl~tion~ along with pcç~ ually irrelevant coml)ollents of an image sequence.
Subband digital coding techniques are well known in the art. See, e.g., N. S. Jayant and P. Noll, Digital Coding of Wa~erol~s: Principles and Applications 5 to Speech and Video (1984).
Subband coding techniques have been used for image coding in a three-cl;.~-en.~ion~l spatio t~ l subband Ll~~ .olk as desçribed in G. Karlsson and M. Vetterli, Three Dimensional Subband Coding of Video, Procee~lings ICASSP
(1988), 1100-1103. The technique described there employs mllltidim~n~ion~l 20 filt~ring to gv.lel~ spatio-~ l frequency bands or subbands using so called quadrature mirror filters. These latter filters are described, e.g., in J. D. Johnston, A
Filter Family Designed for Use in Quadrature Mirror Filter Bands, Procee(lings ICASSP (1980).

Summary of the Invention The present invention takes advantage of the three--lim~.n~ion~l subband Ll~ewc,lk in cletermining a dynamic bit ~lloc~tiQn that relies on pe.~;eplual criteria of the human visual system. This is done in terms of both the relative signific~nce of individual subbands, and the ~ignifir~nce of local spatial areas within such subbands.
Three illustrative embodi",P"I~i of the present invention are described below.
For example, in the first illustrative emb~im~nt of the present invention, an image sequence is sep~ted in dirrelellt spatio-temporal frequency bands. The temporal correlations are exploited by using conditional replenishm~nt on the subband data between frames. Unless the subband is discarded due to 7 ~ 7~

perceptually insignificant information as measured by the low signal energy content in the sub-band, conditional replenishment is applied to the data either on a pixel or block basis. The sub-band corresponding to the lowest spatio-temporal frequency components 5 needs to be encoded accurately due to the high signal energy present in the sub-band and its perceptual significance for video data. The lowest spatio-temporal frequency band is q~nti7e~1 using PCM with a uniform quantizer.
The perceptually significant upper sub-bands as measured by either overall signal energy are encoded either by Geometric Vector Qll~nti7~tion (described incommonly assigned C~n~ n Patent No. 2,039,416 which issued on September 3, 1996,entitled "Geometric Vector Qu~nti7~tionll~ and in commonly assigned C~n~ n Patent Application Serial No. 2,087,994, entitled "Geometric Vector Qu~nti7~tion", filed on January 25, 1993) or by conventionalPCM using a uniform quantizer.
The areas of the significant upper frequency sub-bands which are to be encoded 15 are chosen by using the adaptive bit allocation scheme of the present invention.
In accordance with one aspect of the present invention there is provided a method of coding an image signal sequence, the image signal sequence comprising a plurality of image signals, an image signal comprising a set of pixel signals, the method comprising the steps of: filtering the image signal sequence into a plurality of sets of 20 sub-band pixel signals; coding one or more pixel signals of a first set of sub-band pixel signals using one or more bits of a first quantity of bits, wherein each bit of the first quantity of bits is available for coding the first set of sub-band pixel signals; providing a second quantity of bits for use in coding a second set of sub-band pixel signals, the second quantity of bits comprising one or more bits of the first quantity of bits not used 25 in coding the first set of sub-band pixel signals; and coding one or more pixel signals of the second set of sub-band pixel signals using one or more bits of the second quantity of bits.
In accordance with another aspect of the present invention there is provided an apparatus for coding an image signal sequence, the image signal sequence comprising a 30 plurality of image signals, an image signal comprising a set of pixel signals, the apparatus comprising: means for filtering the image signal sequence into a plurality of sets of sub-band pixel signals; means for coding one or more pixel signals of a first set 'F ' ~; A

7~
- 2a-of sub-band pixel signals using one or more bits of a first quantity of bits, wherein each bit of the first quantity of bits is available for coding the first set of sub-band pixel signals; means for providing a second quantity of bits for use in coding a second 5 set of sub-band pixel signals, the second quantity of bits comprising one or more bits of the first quantity of bits not used in coding the first set of sub-band pixel signals;
and means for coding one or more pixel signals of the second set of sub-band pixel signals using one or more bits of the second quantity of bits.

Brief Description of the Drawing FIGs. 1 and 2 present a first illustrative embodiment of the present invention comprising digital image coder and decoder, respectively.
FIG. 3 shows a typical sub-band filter arrangement in accordance with one aspect of the present invention.
FIGs. 4 and 5 present a second illustrative embodiment of the present 15 invention comprising digital image coder and decoder, respectively.
FIG. 6 presents sub-bands arranged in order of perceived visibility of errors.
FIG. 7 presents a graph illustrating bit allocation as carried out by a bit allocation block.
FIG. 8 presents a third illustrative embodiment of the present invention 20 comprising an image coder.
FIG. 9 presents a graphical representation of the sub-bands including an indication of an illustrative high energy region and a mapping thereof among the sub-bands.

~ ., .~

FIG. 10 presents a flowchart of a procedure for determining quantizer parameters.

Detailed Description A. A First Illu~lr.~tive Eml)o.J:...~..I
FIGs. 1 and 2 present a first illustrative embodiment of the present invention.
FIG. 1 shows an image encoder which employs the adaptive bit allocation of the present invention together with conditional replenishm~nt and qu~nti7~tion based on PCM with a UllirUllll qll~nti7er or Geometric Vector 0 Q~l~nfi7~tion. Similarly, FIG. 2 shows an image decoder based on the same in~ e techniques. Each of the system cle.ll~nlg will now be descIibed individually.
FIG. 1 shows a l~,~l.,se.~ ;on of a graphical image, e.g., sequential frames of a video image. Since the subband filtering used in this illustrative image 15 coding application uses 2 taps for the te~ Jo~al filtPrin~, it proves con~,ellient to store two successive frames of the input signal in block 300. As a specific eY~mrl.o, each frame of the image cont~in~ 240x360 pixels which is known as the ~ommon Tntenn~diate Format (CIF). For present ~ oses, the image will be con~ .rcd to bea multi-level image having up to 256 possible intensity levels. Color images are20 amenable to coding using the present inventive techniques.
Referring to FIG. 1, snccessive frames of a video image are applied to the subband analysis filter block 300 (~lcstllt~d in greater detail in FIG 3). There, the spatio-tc~ ol~l frequency co~ ents of the image are generated using 2-tap Haarfilters for the temporal filtering and 10-tap one-~im~nsion~l q~ hl-e mirror25 filters (QMFs), of the type described in J. D. John~ton, A Filter Family Designedfor Use in Quadrature Mirror Filter Banks, F~ucee ling~ ICASSP (1980), for the spatial filtering In a typical embodiment of the present invention, the subband framework consists of 11 spatio--temporal frequency bands, as shown in FIG 3. Except for the particular choice of subband filters, this is the same basic structure as used in 30 G. Karlsson and M. Vetterli, Three-Dimensional Subband Coding of Video, Procee~ings ICASSP (1988). The terms HP and LP refer to high--pass filte.ring and low--pass filte~ing respectively while the subscripts t~h, and v refer to temporal, horizontal and vertical filtering respectively. The subbands whose signal energycontent is typically low are discarded without causing severe degradation in the , ~

reconstructed image sequence. The re~ ing subbands are encoded using conditional rep1~ni~hment to take advantage of the temporal correlation and either PCM with a wlirO~ q11~nti7~,r (such as that described by Jayant and Noll, Digital Coding of Waveforms: Principles and Applications to Speech and Video (1984)), or5 Geometric Vector Qu~nti7~tion, (such as that described in the patent application incorporated by reference).
The highest spatio-temporal frequency subbands are discarded due to their general per~eylual insignific~nce. For the particular scheme described here, the subbands labeled 9 through l l in FIG. 3 are discarded without c~ ing severe image lO quality distortion. In general, depending on the bit rate, quality sought, and subband rl~cw~lh, any numl)er of high frequency subbands may be discarded.
The ~ignifi~nt subbands are encoded using either a scalar qu~nti7~r or (~eo...el. ;c Vector Qu~nti7~tion shown as blocks 310i in FIG. l.
The dynamic bit allocation is lcyl~se ~le(i by block 320 in FIG l. The bit 15 allocation depends on the amount of con-lition~1 repleni~hment and the amount of motion data as det~ ~ by the signal energy in the a~yloyliate subband.
Con~lition~1 replenishment is applied to all of the si nific~nt subbands and is part of dynamic bit allocation block 320. Con-lition~1 rep1eni~hm~nt is ~Irol~ed on a pixel basis in the subbands using a scalar ~ n~ r~ and on a block 20 basis in the subbands using C~col,lcL-ic Vector Q~ tion The block si_e co~ ~nds to the block si_e used in the vector q11~nti7~tion scheme under consideration. Con-lition~1 repleni~hment on a pixel basis is de~-ribe~ by the following c~ ssion:

x(i j t) = ~ x(i,j,t-l) if Ix(i~j~t)--x(i~j~t--l)l<T~
Q{x(i,j,t) } otherwise 2s where x(i,j,t) is the original pixel value in the ith row, jth column and time t, and x(i,j ,t) is the q~1~nti7~1 pixel value in the ith row, jth column and time t. Q-l~lG~nls qu~nti7~tion of the term in the bracket while T~r is an empirically derived c-~n-lition~1 replenishmPnt scalar threshold. The choice for T~ de~e. ..~ .s howmuch of the subband data will be repeated from the previously encoded subbands.
30 As an example for some typical video sequences, we have found a good choice for T~ to be belwcen lO and 25. Performing conditional rep1eni~hment on a block basis is similar to the pixel--based approach except that the condition in Eq. (l) must hold for all the pixels in a block of data; otherwise the block must be qu~nti7~ and tr~n~mitted~

20880~2 s After conditional repleni~hmt nt is pelÇulllled on the subbands with signifi~nt information, the side information in(lic~ting which pixels are repeated from the previous frame and which pixels are qu~nti7~1 is sent to block 330, thee.ll,~y coder, to be encode~l The enl ~,~y coder may be any lossless coding s technique and for this example is the adaptive Lempel-Ziv algorithm. See, e.g., T. A.
Welch, A Techniq7,~e for High Performance Data Compression, IEEE CoLI~ulel (1988). The nuLub~ of bits that the entropy coder of block 330 in FIG. 1 needs to encode the conditional replPni~hm~.nt side--i--rulL,lalion is fed to the Dynamic Bit ~lloc~tiQn block 320 to update the nuLI~b~r of bits available to encode the subband 10 data. After ell~lû~y coding, the image signal is then multiplexed onto co.-....l..-ic~tion channel 345 for tr~n~mi~;on to a decoder (see discussion below).
The dynamic bit ~lloc~ti~n technique has two signific~nt parts. The first part is the ordering of the subbands based on their p~ Ual si~nifir~n~e. In thisregard, the lowest spatio t~ )Olal frequency band is treated as the most signific~nt 15 in that it CO--~il s the most signal energy and most of the origin~l image structure.
This band is very accurately encoded The next most ~ignifiç~nt subband is the motion subband (coll~ ollding to subband 8 in FIG. 3). This subband is given enough bits to encode the motion h~. ..~I;rn~ The ~ inil~g bits are used to encode the high spatial--low ltLUl)ol~l frequency subbands (coll~,~onding to 20 subbands 2--7 in FIG. 3). When the motion i lÇo. .~ on is high, more bits are given to the motion subband and fewer are left to encode the high spatial details. The high motion a. livhy should mask the loss in high spatial details. When the motion activity drops, more bits are left to encode the high spatial details which now become visible.
The second part of the dynamic bit allocation is to locate the ~ignific~nt areas of the image to encode. This is done across all the subbands by choosing the blocks with the highest average energy to encode.
Because the lowest spatio-temporal frequency subband (subband 1) includes much of the basic image structure and most of the signal energy, it is 30 im~ t to ~c-lr~tt~ly encode this subband. Thcl~irol~ the present embodiment _rst encodes the lûwest spatio t~ o-~l frequency band using PCM and a uniro -ll qu~nti7Pr. For a typical image scheme, the unifolm qu~nti7P,r consists of 6 -- 8 bits.
If the signal in the lowest rl~ucncy subband contains data which possesses a probability density function that is not uniÇollll, PCM is applied with a qll~nti7~r that 3s suits the probability density function of the signal. Applying a qu~nti7P,r suited to a signal's probability density function is a conventional technique and is described, -6- 7 ~

e.g., in the above-cited Jayant and Noll reference. The number of bits needed toencode the lowest spatio--temporal frequency band using PCM and a uniform qu~nti7~r iS fed into block 320 -- Dynamic Bit Allocation -- to update the number of bits available to encode the high spatio--temporal frequency bands.
s The next band that is encoded corresponds to the subband cont~ining high temporal frequency components and low spatial frequency components. For theexample presented in the framework illustrated in FIG. 3, this corresponds to subband 8. This frequency subband contains much of the motion inl~ ation of the video signal. The signal energy in this subband gives a good indication of the 10 amount of motion in the video sequence at any given time. Subband 8 is encoded by qll~nti7ing the data in the blocks whose local energy exceeds a predetermined - threshold value as given by:

N ~x(i,j,t)22Tm, (2) where the summation is performed over the block of data, N denotes the block size 5 and Tm is a predete.rmined scalar motion threshold value. For this example, a good value for Tm is 100. The blocks in the motion subband whose average energy exceeds the threshold Tm are encoded using Geometric Vector Q~n~i7~tion with either 2 or 3 levels (described in the above-referenced Canadian Patent No. 2,039,416). The amount of bits that are required to encode the motion data 20 accurately is fed back to block 320 -- Dynamic Bit allocation -- to update the number of bits left to encode the rem~ining subbands.
The subbands which are encoded last, correspond to the low temporal frequency components and high spatial frequency components. For the example framework illustrated in FIG. 3, this corresponds to the subbands labeled 2 through 2s 7. The bits which are left to encode subbands 2--7 are distributed to the blocks across all the l~mai~ g subbands with the largest local energy as defined in Eq. 2.
The blocks may be encoded using Geometric Vector Q~l~nti7~tion. Any of these remaining blocks, especially those corresponding to lower spatial frequencies (such as subbands 2--4 in FIG.3), may be encoded using the scalar q~l~nti7er described for 30 the lowest spatio--temporal frequency band. In this case, the pixels with the largest absolute value are encoded.
FIG. 2 shows a decoder counterpart to the coder in FIG. 1. Coded signals received from the channel are first demllltirlexed in unit 400. Side information is used by the deqll~nti7~r control unit 420 to deteImine which areas of ~,, the subbands have been repeated from the previously encoded subband as given by conditional replenishment, which areas have been ql!~nti7~, and which areas havebeen _eroed out. The deq~l~nti7er units labeled 410i reconstruct the data from the vector indices and m~nit~lde inform~tion as provided by the Geometric Vector 5 Qll~nti~r. The subband synthesis unit 430 ~e~rol lls the operations of the subband analysis unit 300 in reverse to reconstruct the images.
The above embodiment can be adapted to other frameworks besides the one illustrated in FIG. 3 allowing for a greater control over which subbands arepelc~lually most signific~nt and which areas of the image are pel.;el)lually most 10 signifi~nt B. A Second Illustrative Embodiment A second illustrative embodiment of the present invention codes video frame sequences at a lll~illlUm constrained bit rate. This embodiment provides coded video to be tr~nsmitteA or stored on a m~lillm with a specified constant or 5 time varying bit rate.

1. The Coder FIG. 4 presents an embodiment of the video coder according to the present invention. An input video image sequence is decomposed into temporal andspatial subbands by the subband analysis filter block 300. The subband structure is 20 the same as that shown in FIG. 4 of the above-referenced Can~ n Patent No. 2,039,416. The outputs of the subband filters are critically sampled, and bands 7, 9, 10, and 11 are set to ~ro. The decomposed video sequence, therefore,contains half the number of samples as the original input image sequence. The zeroed bands typically contain very low energy, therefore, ~roing these bands 25 introduces very little degradation in the reconstructed video sequence. Two-tap temporal filters are used to minimi7e delay, and tenth-order quadrature mirror filters are used for the spatial decomposition. Although the present embodiment is described only for the hlmin~nce component of a color video sequence,- the same advantages result when the present invention is applied to the chrominance 30 components of a color sequence.
The decomposition by filter block 300 produces subbands with reduced spatial re~nn~ncy. However, temporal reA-lntl~ncy within the lower temporal subbands and spatial correlations between the subbands remain. The technique of conditional replenishment is used to remove temporal reAIln(l~ncy from the coded -~088082 signal, thereby achieving higher decoded video quality at a specified coding rate.
Subband correlation may be taken advantage of by deqll~nti7~r blocks 610i and dequ~nsi7er control block 620 to further improve the quality of the decoded video.
During peak motion of objects in a video signal, the available coding S rate may be insllfficient to code all of the ~el~;ep~ually signifir~nt subband samples.
Th~erc,l~, this embodiment allocates coding bits within and among the subbands to ...;ni...;~ the ~lceived coding distortion. Q~l~ntir~tion errors are most noticeable in subband 1 because it co~ in~ most of the basic image structure, and in band 8 for moving areas of the video sequence.
Samples in bands 1, 2, 3, and 4 are subsampled by an ~l-lition~l factor of 2 beyond that of the other bands. Q~l~nti7~sion errors in these bands are more visible than those in the higher spatial bands because these s~mples are spatially interpolated by an a(~ sion~l factor during reconstruction, and IL~ ,role sampleerrors occupy a larger area in the decoded video frame. Band 4 usually has very low 15 energy, llle.~,rol~ it is pc.~plually less ~ignifir~nt than bands 5 and 6 for typical video se~luences. FIG. 6 shows the subbands arranged in order of she ~er~eiv~d visibility of errors.
This ordering is used to dele- ...;ne the priority of ~lloc~tin~ available bandwidth to each subband qU~nti7pr block 310i. Bandwidth or bit ~lloc~sion is 20 impl~ ed by dynamic bit ~ sion block 520. Because the bandwidth required to code dirr~.ent bands is correlated, a fixed pl~ollion of the ~ ining bandwidth may be allocated to the subbands at each stage of allocation. These ~.o~llions have been dete-...;n~l emririr~lly. FIG. 7 plesenls a graph illustrating bit ~lloc~tion as carried out by block 520. Each node of the graph corresponds to one of the 2s subbands, and each edge is labeled with the proportion of bits ~,n~ from the p,~lil~g subband which are ~lloc~ted to the lower subband node.
~~ The bits l~ h~ing after qn~t~ti7ing bands 1 and 8 are allocated equally to subbands 2, 3, 5, and 6. Although equal bandwidth is allocated to the lower priority bands, S and 6, fewer bits per sample are allocated to these bands since they 30 contain more samples. The advantage in ~ c~ting bits in this fashion is that the q~nti7P.rs for bands 1 and 8, and the ql-~nti7P.rs for bands 2, 3, 5, and 6 can be impl~ nled in parallel, yielding greater efficiency in operation.
Temporal re~lnntl~ncy in bands 1, 2, 3, 4, 5, and 6 is removed by selectively lGpea~ing subband samples which differ by less than a determined 35 threshold from the co..~,~ponding previously tr~nimitt~l value. This technique is referred to as conditional repleni~hm~nt and has been described above with .., reference tO the first illustrative embodiment. Conditional repleni~hment is applied in the subband analysis Ll~ewolk. An advantage is obtained by allocating bandwidth for repleni~hm~nt on a priority basis to those subbands which have thehighest visibility. This results in a ~.ce~Lually hl~ ùved decoded quality.
In order to determine which subband samples should be repeated, the m~gnihlde diLr~ ce between the subband samples being qu~nti7~ and the coll~,s~onding samples of a previously q~l~nti7Pd video frame is c~lc~ ted Each qu~nti7P,r block 510i includes a memory which contains the previously qu~nti7~
subband sa_ples. Since the samples are ~pl~,sellled with a linear eight-bit code, the o error m~nitnde is in the range (0, 256). A histogram of the oc-;ullcnce of the error values is effi(içntly generated by the q~l~nti7~r 310i using the error value, for each sample in the subband, as an address into one of 256 memory loc~tion~ which are ~ncf~.. l~d for each oc~;Ull~inCe of the coll~ollding error value. A coding error threshold is determin~d from the histogram by S.. ~ g the nnmher of occurrences 15 for each error value, starting with the largest error and proceeding towards the lowest error, until the bandwidth needed to quantize the nu~ ~ of s?mpl~-s in~ tyl by that sum equals the coding bandwidth available. The address of the location being ~ cesse~ when the s-~ ;on is ~ AI,~ iS equal to the coding error threshold.
The threshold is limited to be greater than an error which is not visible in the20 decod~ video s~uence. This .nil~in~....~ value has been empirically d~t~ ;ned to be three (3) for subbands 1, 2, and 3. Samples which have an error greater than the detefminfYl threshold are coded and tr~n~n-i~lr,d to the decoder as described below.
The l~ tion~ of samples which are below the threshold are effiriently coded using run-length coding, and entropy coding, e.g., Huffm~n coding, of the run-length~,2s impk..-~f.l-l~l by e~ up~ coder 530. Excess bits are available to be all(-c~ted to lower priority subbands when the error threshold detc. ..-;n~d by the histogram method falls below the ..~il-i.--...-- visibility threshold.
Sample values which are tr~n~mittP-l in bands 1, 2, 3, and 4 are coded using a scalar qu~nti7.o,r. Sample values which are ~ ~l in bands S and 6 are 30 coded using a scalar or vector quanti_er, and in particular the geometric vector q~l~nti7P,r described in the above-referenced and incorporated patent applit~tions The dele- . . .i n~tion to use the scalar or vector qu~nti7Pr is made by applying a threshold to the bandwidth allocated to subbands S and 6. An allocation below anempirically determin~d threshold in~ tes a high degree of motion in the video 35 sequence. It has been empirically dett rmin~A that ~rceived distortion is minimi7P~
when the vector qu~nti7~r is used for a bandwidth allocation falling below 60 percent of the maximum allocation. An advantage is achieved by this method because the distortion introduced by the vector qU~nti7P~r is m~k~d by motion in the video sequence. This allows a larger number of samples to be coded due to the higher çfficiçncy of this qU~n*7pr. When the bandwidth allocation increases above the S threshold, and thelefole the degree of m~king afforded by motion in the video sequence decreases, the scalar qU~nti7er is used to reduce the errors introduced by the vector q l~nti7P,r.

2. Geometric Vector Qll~nti7~tion This section desçribes an efficient imple, . .f ~ ;OI of the geometric lo vector qu~nti7~r used in subbands S, 6, and 8. This imple...~ ;on provides a means for consllaining the bandwidth necess~ry to code the given subband, and provides an effi(~ient technique to search for null vectors.
Null vectors correspond to repeated samples in bands 5 and 6, and refer to ~ro valued sa~nples in band 8. The geQn~l. ;r vector codebook described in 15 above-referenced C~n~ n Patent No. 2,039,416 requires approximately equal complexity to identify both null and non-null vectors. Because the majority of subband samples typically are coded as the null vector, a ~ignificant re~llction in vector search complexity may be achieved by an effici~-nt null vector search technique. Null vectors may be identifi~l by first determining the sample in each 20 vector having the greatest m~gnitllde~ for band 8, or the greatest m~gni1~ldedi~r~lc;nce, for bands S and 6, where conditional replenishment is used. A coding threshold is determined by the histogram method described above, using the maximum m~gnitlldes, for band 8, or maximum di~îel~nce m~gninlcles, for bands 5 and 6. Those vectors having a m~illlum m~gnit~l~le falling below the coding 25 threshold are identified as null vectors. The rem~ining vectors are coded using the efficient codebook search techniques described in the above referenced and incorporated patent applications. The location of null vectors is coded using run-length coding and entropy coding, e.g., Hllffm~n coding, of the run-lengths. Thecoding threshold in band 8 is limited to a miniml-m value of 6. Lf the coding value is 30 determined to be less than 6 by the histogram method, excess coding bandwidth is available to allocate to the lower priority bands.
The tr~n~mitted sample values are written into the corresponding locations in each subband sample memory in the quantizer blocks 510i. The channel multiplexer block 340 formats the information from the quantizer blocks SlOi so that 35 it can be tr~n~mitted or stored on the appropriate medium. This information includes . . .

the run-length encoded repeated sample locations and the trAn~mitte~l quAnti7~1 samples. Side i~ lion iS provided for bands 5 and 6 to specify whether scalar orvector qu~nti7~tion was used in the encoding process.

3. The Decoder s FIG. 5 presents an embodiment of the decoder according to the present invention. The channel de.m-lltiplexer block 400 opela~es in reciprocal fashion to multiplexer 340 in FIG. 4 to separate the coded subband inr(,. ..~Ation This informAtion is provided to the deqll~nti7P,r blocks 610i for each subband. For the case of vector quAnti7~tion in bands S, 6, and 8, the deq l~An~i7.o,r~ reconstruct the 10 vector samples as described in above-referenced C~n~ n Patent No. 2,039,416.
The ,~.,-A;nh~g samples in band 8 are set to _ero. The dequ~nti7Pr blocks 610i, COll~ SpOIl&g to subbands 1, 2, 3, 4, 5, and 6, include m~mories contAining the previously dequAnti7efl subband samples. Samples which are quAnti7e-1 and L[A Il!~l l li l l~i by the encoder replace the corresponding samples in these subband 15 m~mories, the re...Ail-il-g samples are repeated. The diLrel~nce between the trAn~mitted samples and previously stored samples, for subband 1, are calculated and stored for use as described below.
The selective mo~ific~tion of the subband samples in the decoder roves the peçceived quality of the reconstlucted video signal. This is 20 accomplished by deqll~nti7er blocks 610i and dequ~nti7er control block 620 asfollows. Objectionable errors in the decoded video sequence may occur when the coding rate is constrained such that there is in~nfficient bandwidth to ~ sllliL all of the subband samples with errors above a perceptible threshold. Under this condition, the decoder subbands using conditional repleni~hm~nt contain three types of 25 samples- (i) low error samples which are repeated, (ii) newly received samples, and (iii) large error samples which cause objectionable artifacts in the decoded video sequence. These large errors occur most frequently in the subbands with lower bandwidth allocation priority. The errors typically appear as edges which remainbehind moving objects.
Qll~nti7~ti~n errors in subband 1 remain small since it has the highest bandwidth allocation priority. This fact, along with the spatial correlation between the subbands, is used to identify samples in the upper subbands which have a large error. The technique com~es the band 1 difference signal, described above, to anempirically determined threshold. A useful threshold value is 12 for the range of intensity values in band 1 is (0, 256). If the dirrelellce is greater than the threshold, ~ =.

the corresponding spatial samples in subbands 2, 3, 4, 5, and 6 are tested further.
Each sample in subband 1 corresponds to four samples in subbands 5 and 6 since subband 1 is decimated by an addition~l factor of t~-vo in the analysis filters 300. If the co~ )onding upper subband samples have not been tr~n~mitted 5 by the encoder, they are ~j.. ed to have large error and are processed further to reduce the res~lltant distortion. A useful technique is to set these samples to zero in order to elimin~te trailing edges on moving areas. Other techniques, such as interpolating these samples ~l~n low error samples, may prove adv~nt~geo~l~ in smooth areas of the image. This technique provides an advantage by red~lcing 10 p~l.;Gived coding distortion without L~ ...iL~ g ~dditit~n~l side infollllaLion to the decoder.
The resulting subband samples are finally applied to the subband synthesis block 430, which reconstructs the sequence of video frames.
An illu..L~tive h~d~ hilt;clu.~ for a real-time impleme..~ n of 15 the present invention is pl-,sellled in the Appendix.

C. A Third nl~ll&li~ Embodiment The overall sLluclu,e of the encoder of this embodiment is pl~,senled in FIG. 8. A series of video frames are fed into a sub-band analysis filterbank 300which divides the frequency s~ecL- Ulll into sub-bands. The output of the sub-band 20 analysis is used by a dynamic bit alloc~tor 720. The dynamic bit allocator 720 is responsible for deci-ling how much of the total available bit rate should be allocated to each sub-band. The dynamic bit alloc~tor 720 also determines which spatial regions are more illl~~ to code within each sub-band. This illfollllaLion, alongwith the output of the sub-band analysis 300, is provided to the qu~nti7~,rs 710i.
25 Under certain ci~;ulll~nces, fee~lb~ from the el~ coder 330 may be needed to del~llii~ie the exact bit rate required to code some of the sub-bands. The reslllting bit stream is multiplexed 340 and llal~milled over a channel 345. The dynamic bit allocator 720 assumes that there is a fixed rate available as each frame enters the system, although that rate could change when the next frame enters.

A coding advantage can be achieved by dividing the sub-bands into regions and allocating more bits to regions that are deem.od to be pel~;e~lually more relevant. The definition of p~ce~lual relevance may vary for dirr~l~nl applications.
A convenient method for del~- ..-;ning perceptually relevant regions is to divide the high-pass lt;lllpOlal, low-pass spatial sub-band (sub-band 8 in FIG. 9) into blocks and to compute the energy in each block using, ek,l,t = ~ Xi,j,t i,j where x i,j,t ~ 3ellls the intensity value of a pixel at column i, row j, at time instant t. The co..~ rd energy values for each block, ek,l,t, are sorted in decreasing s order and c~ ified into two or more groups. Illustratively, two classes of blocks are ~.,.. ~1, high energy and low energy. In FIG. 8, the output from the sub-band analysis filterbank 300 is provided to the dynamic bit ~lloc~tor 720. The dynamic bit tor 720 is l~s~l-~iblP for dete ..~;ning the energy grouping.
Once the high and low energy blocks are de~t~ ined from sub-band 8, 0 this spatial info....~lion is used to q~.~n~i7~. an of the sub-bands to be coded (FIG. 9).
For example, if a block were c~ ified as a high energy block, then the co~ g block could be qu~ f~ using a finer qu~nti7f .r in each of the coded subbands. By plG~l villg higher quality in that block in all of the coded sub-bands, the lccolls~ ;t~,d image will have a higher SNR in the coll~,~,ponding block. This is 15 ideal for an applir~tion which requires more spatial detail in moving regions. Such an applir~tion might be, e.g., the coding of an image sequence of a person using the ~m~ri-~n Sign Language (used co.. only by deaf people). On the other hand, fewer bits could be ~110c~ted to these regions for applit~tion~ where spatial detail is not needed where motion occurs. There may be any number of high and low energy 20 blocks idel~tifi~d in sub-band 8.
When motion activity is low, sub-band images are not segregated into high and low motion regions. In this case, all the regions of the sub-band image are treated as being equally im~ t. The motion activity is determined by keeping a history of the e~ ~a of total energy in band 8. Illustratively, if the total energy of 2s sub-bana 8 for a current frame is within the bottom one-third of the dynamic range of the total energy history of the sub-band, motion activity is con~idered to be low.
After the high and low energy blocks have been dele. .,~ ed, an initial allotment of bits is dele~."i,-ed to code each sub-band. A typical initial allotment provides 50% of the total bits to sub-band 1, another 40% to sub-band 8, and the30 l~ ini~lg 10% is divided among sub-bands 2-7. The total number of bits is determined by the channel bandwidth and the frame rate.
The embodiment codes sub-band 1 first since sub-band 1 contains a signifiç~nt portion of the energy in the origin~l signal. Given the initial allotment of bits to code this sub-band, the number of pixels which can be accurately coded is .~..

computed by dividing the total number of bits allotted to the sub-band by the number of bits it takes to code each pixel. The number of bits needed to code each pixel is the sum of the bits needed to represent the address, sign, and m~nihl-lP of the lsl . .; l l~ pixel. The number of address and sign bits remain fixed widlin a sub-5 band, but the nulllber of bits needed fomcpl~senting pixel m~gnit~de is determinedusing an iLel~livt; procedure (see FIG. 10).
This procedure begins by computing a histogram of the absolute values of the dirrt;l~ ce between like-address pixels in the current and previous frames. The ... ~,~i.. - value is retained. The histogram is used to det~ e the threshold value 10 of a dead-zone qu~nti7P,r. A dead-zone qu~nti7~r ~ o,l~A~ lly elimin~tes coding pixel values which faU below a threshold. Sub-band dirr~nce values that are greater than the dead-zone threshold get coded, while those at or below the threshold are not coded but are repe~ted from the previous frame. The system uses an empiricaUy derived Illi~ threshold value as an initial e~ of the actual dead-zone 5 threshold. For a channel bit-rate of 384 kbs, illusLlaLivc .-~ ... threshold e,,~ s for bands 1-8 are: 3, 4, 4, 5, 5, 5, 5, and 5. The step size for the qu~nti7~r is then set to be twice the value of the dead-zone threshold. The Ill~illllllll amount of distortion that can be introduced by a single nncoded pixel (which falls at or below the threshold) is the threshold value. Since this level of distortion is ~snm~d to be 20 acceptable for pixels which are not coded, the same m~xim~lm distortion is aUowed on qll~nti7~1 pixels which are coded by setting the step size to twice the dead-zone threshold.
The .~-ini.~ threshold value is based on exl)f,. ;..-. ~.1~ which were ~IÇolll~ed to identify the amount of distortion that can be introduced into each sub-25 band before the noise becomes notice~ble. The reason a ...;.-i.--~ - threshold is introduced is to plcvcllt any pardcular sub-band from being overcoded.
Once the inidal dead-zone threshold, step size and m~xim~lm absolute dirrc,~nce value have been determin~, the number of qll~nti7~tion levels is tlete.,..i.~1 according to the following expression (max val- threshold) num levels = - .
- step_slze This value is then rounded up to the nearest power of 2 to create the ,ate~l number of levels for the bits available. Now the number of bits needed tocode a pixel can be evaluated by adding the number of bits needed for the address and sign to the number of bits needed for this ql~nti7er. Dividing the total number 35 of bits by the bits per pixel determines the number of pixels that can be coded within - 208~082 , ....

that sub-band region. A count of the number of pixels which fall above the dead-zone threshold is computed from the histogram. If this count is less than the allowed count, then the qu~n~-7P,r parameters are fixed. If, however, the count is larger than the aUowed count, then the dead-zone threshold is incremen~ed and the col~u~lions S are l~ealed as shown in FIG. 10. Increasing the dead-zone threshold forces thestep-size to increase since it is set to twice the dead-zone threshold. This, in turn, eventually reduces the nulllber of bits l~uiled to quantize the same data, albeit more coarsely. The dead-zone value is incl~ lled until the count of pixels to be coded falls below the null~er of coded pixels allowed. When this occurs, the q~l~nfi7f~r 10 pal~l~l~ are fixed and provided to the qu~nti7Pr 710i.
The ~u~ er p~,t~ are then used to actually code the sub-band.
There may be some bits leftover after coding sub-band 1 if the number of pixels to be coded is less than the number of pixels aUowed. This can happen in two ci~ ces. First, if the ~ J~II dead-zone threshold is used to prevent lS o~odhlg, then the actual count wiU usuaUy be less than the aUowed count.
Secon-1, since the histogram uses discrete values, the desired count may not match up exacdy with the allowed count. To p~ .nt the system from using more than its ~lloc~ted bit rate, the value of the dead-zone threshold is incl~ ~l until the actual count is less than or equal to dhe aUowed count. Under these two ci~ nces~ any 20 l~rlo~er bits are imm~ tely re-~lloc~ted to dhe next sub-band to be coded.
In this case, sub-band 8 is coded next. The exact same plocellule is foUowed for sub-band 8 as was used for sub-band 1, except that the ori~in~l frame is used rather than dhe diff~,nce be~ two frames. This method was chosen becau~ of the low correlation be~ successiv~ frames in sub-band 8. Once 25 again, any leftover bits are passed on to help code subsequent sub-bands.
Once sub-bands 1 and 8 are coded, each of the rem~ining low-pass tellll~olal sub-bands (sub-bands 2-7) are coded. Two ~lt~rn~tive methods for doing this are provided. The _rst method concern~ processing the sub-bands in a ~,~;~lually ~ignific~nt order: 2, 3, 5, 6, 4, 7. As with sub-band 8, any bits leftover 30 from previous coding are added to the prede-termined bit allocation for the current sub-band. Once again, the dead-zone q~l~nti7Pr on the dirr~ ce signal is used toupdate these sub-bands. As with sub-band 1, an empirically derived ~ini~
dead-_one threshold is imposed when coding these sub-bands. The second method uses an energy criterion which selects the highest values from the dirrelc;llce signal 35 from all of the sub-bands 2-7 rather than looking at each of bands 2-7 individually.

, ~.~

Of the high-pass temporal sub-bands, only sub-band 8 is coded.
Subbands 9, 10, and 11 are all ~t to zero and are not coded (as discussed above, this may be done by filterbank 300).
This embodiment employs the decoder discussed in section B.3, above, 5 and ~l~sel.ted in FIG. 5.

Claims

1. A method of coding an image signal sequence, the image signal sequence comprising a plurality of image signals, an image signal comprising a set of pixel signals, the method comprising the steps of:
filtering the image signal sequence into a plurality of sets of sub-band pixel signals;
coding one or more pixel signals of a first set of sub-band pixel signals using one or more bits of a first quantity of bits, wherein each bit of the first quantity of bits is available for coding the first set of sub-band pixel signals;
providing a second quantity of bits for use in coding a second set of sub-band pixel signals, the second quantity of bits comprising one or more bits of the first quantity of bits not used in coding the first set of sub-band pixel signals; andcoding one or more pixel signals of the second set of sub-band pixel signals using one or more bits of the second quantity of bits.

2. The method of claim 1 further comprising the step of generating a first signal reflecting that one or more sub-band pixel signals of said first set may be represented by one or more previously decoded sub-band pixel signals.

3. The method of claim 2 wherein the generating step comprises the step of comparing one or more sub-band pixel signals with one or more previous sub-band pixel signals to determine whether a difference between the compared signals satisfies an error criterion.

4. The method of claim 3 wherein the error criterion is based on the sub-band corresponding to the compared sub-band pixel signals.

5. The method of claim 1 wherein the sub-band corresponding to the first set of sub-band pixel signals has a greater perceptual significance than the sub-band corresponding to the second set of sub-band pixel signals.

6. The method of claim 5 wherein the sub-band corresponding to the first set of sub-band pixel signals reflects low spatial frequencies of the set of pixel signals.

7. The method of claim 1 wherein the step of coding one or more pixel signals of the first set of sub-band pixel signals comprises the step of determining which one or more sub-band pixel signals of said first set are more perceptuallysignificant relative to one or more other sub-band pixels of said first set.

8. An apparatus for coding an image signal sequence, the image signal sequence comprising a plurality of image signals, an image signal comprising a set of pixel signals, the apparatus comprising:
means for filtering the image signal sequence into a plurality of sets of sub-band pixel signals;
means for coding one or more pixel signals of a first set of sub-band pixel signals using one or more bits of a first quantity of bits, wherein each bit of the first quantity of bits is available for coding the first set of sub-band pixel signals;
means for providing a second quantity of bits for use in coding a second set of sub-band pixel signals, the second quantity of bits comprising one or more bits of the first quantity of bits not used in coding the first set of sub-band pixel signals; and means for coding one or more pixel signals of the second set of sub-band pixel signals using one or more bits of the second quantity of bits.

9. The apparatus of claim 8 further comprising means for generating a first signal reflecting that one or more sub-band pixel signals may be represented by one or more previously decoded sub-band pixel signals.

10. The apparatus of claim 9 wherein the means for generating comprises means for comparing one or more sub-band pixel signals with one or more previoussub-band pixel signals to determine whether a difference between the compared signals satisfies an error criterion.

11. The apparatus of claim 10 wherein the error criterion is based on the sub-band corresponding to the compared sub-band pixel signals.

12. The apparatus of claim 8 wherein the sub-band corresponding to the first set of sub-band pixel signals has a greater perceptual significance than the sub-band corresponding to the second set of sub-band pixel signals.

13. The apparatus of claim 12 wherein the sub-band corresponding to the first set of sub-band pixel signals reflects low spatial frequencies of the set of pixel signals.

14. The apparatus of claim 8 wherein the means for coding one or more pixel signals of the first set of sub-band pixel signals comprises means for determining which one or more sub-band pixel signals of said first set are more perceptuallysignificant relative to one or more other sub-band pixel signals of said first set.