WO1999037088A1 - Method and system for combining video sequences with spacio-temporal alignment - Google Patents

Method and system for combining video sequences with spacio-temporal alignment Download PDF

Info

Publication number
WO1999037088A1
WO1999037088A1 PCT/IB1999/000173 IB9900173W WO9937088A1 WO 1999037088 A1 WO1999037088 A1 WO 1999037088A1 IB 9900173 W IB9900173 W IB 9900173W WO 9937088 A1 WO9937088 A1 WO 9937088A1
Authority
WO
WIPO (PCT)
Prior art keywords
sequences
sequence
given
composite
video
Prior art date
Application number
PCT/IB1999/000173
Other languages
French (fr)
Inventor
Serge Ayer
Martin Vetterli
Original Assignee
Ecole Polytechnique Federale De Lausanne (Epfl)
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ecole Polytechnique Federale De Lausanne (Epfl) filed Critical Ecole Polytechnique Federale De Lausanne (Epfl)
Priority to AT99900616T priority Critical patent/ATE218780T1/en
Priority to BRPI9906887A priority patent/BRPI9906887B8/en
Priority to CA002318375A priority patent/CA2318375C/en
Priority to IL13727899A priority patent/IL137278A/en
Priority to AU19805/99A priority patent/AU1980599A/en
Priority to EP99900616A priority patent/EP1055324B1/en
Priority to JP2000540671A priority patent/JP4739520B2/en
Priority to DE69901681T priority patent/DE69901681T2/en
Publication of WO1999037088A1 publication Critical patent/WO1999037088A1/en
Priority to AU2003268578A priority patent/AU2003268578B2/en

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/2628—Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07C—TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
    • G07C1/00—Registering, indicating or recording the time of events or elapsed time, e.g. time-recorders for work people
    • G07C1/22—Registering, indicating or recording the time of events or elapsed time, e.g. time-recorders for work people in connection with sports or games
    • G07C1/24—Race time-recorders
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/265—Mixing

Definitions

  • the present invention relates to visual displays and, more specifically, to time-dependent visual displays.
  • a composite video sequence can be generated which includes visual elements from each of the given sequences, suitably synchronized and represented in a chosen focal plane. For example, given two video sequences with each showing a different contestant individually racing the same down-hill course, the composite sequence can include elements from each of the given sequences to show the contestants as if racing simultaneously.
  • a composite video sequence can be made also by similarly combining one or more video sequences with one or more different sequences such as audio sequences, for example.
  • Fig. 1 is a block diagram of a preferred embodiment of the invention.
  • Figs. 2A and 2B are schematics of different downhill skiers passing before a video camera.
  • Figs. 3 A and 3B are schematics of images recorded by the video camera, corresponding to Figs. 2A and 2B.
  • Fig. 4 is a schematic of Figs. 2 A and 2B combined.
  • Fig. 5 is a schematic of the desired video image, with the scenes of Fig. 3 A and 3B projected in a chosen focal plane.
  • Fig. 6 is a frame from a composite video sequence which was made with a prototype implementation of the invention.
  • the invention can be appreciated in analogy with 2- dimensional (2D) "mo hing", i.e. the smooth transformation, deformation or mapping of one image, II, into another, 12, in computerized graphics.
  • 2D 2- dimensional
  • mi hing i.e. the smooth transformation, deformation or mapping of one image, II, into another, 12, in computerized graphics.
  • Such morphing leads to a video sequence which shows the transformation of II into 12, e.g., of an image of an apple into an image of an orange, or of one human face into another.
  • the video sequence is 3 -dimensional, having two spatial and a temporal dimension. Parts of the sequence may be of special interest, such as intermediate images, e.g. the average of two faces, or composites, e.g. a face with the eyes from II and the smile from 12.
  • intermediate images e.g. the average of two faces
  • composites e.g. a face with the eyes from II and the smile from 12.
  • the invention is concerned with a more complicated task, namely the merging of two video sequences.
  • the morphing or mapping from one sequence to another leads to 4-dimensional data which cannot be displayed easily.
  • any intermediate combination, or any composite sequence leads to a new video sequence.
  • Video sequences obtained from two contestants having traversed a course separately can be time- synchronized by selecting the frames corresponding to the start of the race.
  • the sequences may be synchronized for coincident passage of the contestants at a critical point such as a slalom gate, for example.
  • the chosen focal plane may be the same as the focal plane of the one or the other of the given sequences, or it may be suitably constructed yet different from both.
  • the video sequences synchronized they can be further aligned spatially, e.g. to generate a composite sequence giving the impression of the contestants traversing the course simultaneously.
  • spatial alignment can be performed on a frame-by-frame basis.
  • the view in an output image can be extended to include background elements from several sequential images.
  • Forming a composite image involves representing component scenes in a chosen focal plane, typically requiring a considerable amount of computerized processing, e.g. as illustrated by Fig. 1 for the special case of two video input sequences.
  • Fig. 1 shows two image sequences ISl and IS2 being fed to a module 1 1 for synchronization into synchronized sequences 1ST and IS2'.
  • the sequences ISl and IS2 may have been obtained for two contestants in a down-hill racing competition, and they may be synchronized by the module 1 1 so that the first frame of each sequence corresponds to its contestant leaving the starting gate.
  • the synchronized sequences are fed to a module 12 for background- foreground extraction, as well as to a module 13 for camera coordinate transformation estimation.
  • the module 12 For each of the image sequences, the module 12 yields a weight-mask sequence (WMS), with each weight mask being an array having an entry for each pixel position and differentiating between the scene of interest and the background/foreground.
  • WMS weight-mask sequence
  • the generation of the weight mask sequence involves computerized searching of images for elements which, from frame to frame, move relative to the background.
  • the module 13 yields sequence parameters SPl and SP2 including camera angles of azimuth and elevation, and camera focal length and aperture among others. These parameters can be determined from each video sequence by computerized processing including interpolation and matching of images. Alternatively, a suitably equipped camera can furnish the sequence parameters directly, thus obviating the need for their estimation by computerized processing.
  • the weight-mask sequences WMS1 and WMS2 are fed to a module 13 for "alpha-layer" sequence computation.
  • the alpha layer is an array which specifies how much weight each pixel in each of the images should receive in the composite image.
  • the sequence parameters SPl and SP2 as well as the alpha layer are fed to a module 15 for projecting the aligned image sequences in a chosen focal plane, resulting in the desired composite image sequence. This is exemplified further by Figs. 2A, 2B, 3A, 3B, 4 and 5
  • Fig. 2 A shows a skier A about to pass a position marker 21, with the scene being recorded from a camera position 22 with a viewing angle ⁇ (A).
  • the position reached by A may be after an elapse of t(A) seconds from A's leaving the starting gate of a race event.
  • Fig. 2B shows another skier, B, in a similar position relative to the marker 21, and with the scene being recorded from a different camera position 23 and with a different, more narrow viewing angle ⁇ (B).
  • the position of skier B corresponds to an elapse of t(A) seconds from B leaving the starting gate.
  • skier B has traveled farther along the race course as compared with skier A.
  • Figs. 3A and 3B show the resulting respective images.
  • Fig. 4 shows a combination with Figs. 2A and 2B superposed at a common camera location.
  • Fig. 5 shows the resulting desired image projected in a chosen focal plane, affording immediate visualization of skiers A and B as having raced jointly for t(A) seconds from a common start.
  • Fig. 6 shows a frame from a composite image sequence generated by a prototype implementation of the technique, with the frame corresponding to a point of intermediate timing.
  • the value of 57.84 is the time, in seconds, that it took the slower skier to reach the point of intermediate timing, and the value of +0.04 (seconds) indicates by how much he is trailing the faster skier.
  • the prototype implementation of the technique was written in the "C" programming language, for execution on a SUN Workstation or a PC, for example.
  • Dedicated firmware or hardware can be used for enhanced processing efficiency, and especially for signal processing involving matching and interpolation.
  • background and foreground can be extracted using a suitable motion estimation method.
  • This method should be "robust", for background/foreground extraction where image sequences are acquired by a moving camera and where the acquired scene contains moving agents or objects.
  • Required also is temporal consistency, for the extraction of background/foreground to be stable over time.
  • temporal filtering can be used for enhanced temporal consistency.
  • background/foreground extraction Based on determinations of the speed with which the background moves due to camera motion, and the speed of the skier with respect to the camera, background/foreground extraction generates a weight layer which differentiates between those pixels which follow the camera and those which do not. The weight layer will then be used to generate an alpha layer for the final composite sequence.
  • Temporal alignment involves the selection of corresponding frames in the sequences, according to a chosen criterion. Typically, in sports racing competitions, this is the time code of each sequence delivered by the timing system, e.g. to select the frames corresponding to the start of the race. Other possible time criteria are the time corresponding to a designated spatial location such as a gate or jump entry, for example.
  • Spatial alignment is effected by choosing a reference coordinate system for each frame and by estimating the camera coordinate transformation between the reference system and the corresponding frame of each sequence. Such estimation may be unnecessary when camera data such as camera position, viewing direction and focal length are recorded along with the video sequence.
  • the reference coordinate system is chosen as one of the given sequences — the one to be used for the composite sequence.
  • spatial alignment may be on a single- frame or multiple-frame basis.
  • each video sequence is first analyzed over multiple frames for reconstruction of the scene, using a technique similar to the one for background/foreground extraction, for example. Once each scene has been separately reconstructed, e.g. to take in as much background as possible, the scenes can be spatially aligned as described above.
  • This technique allows free choice of the field of view of every frame in the scene, in contrast to the single-frame technique where the field of view has to be chosen as the one of the reference frame.
  • the field and/or angle of view of the composite image can be chosen such that all competitors are visible.
  • the techniques of the invention can be applied to other speed/distance sports such as car racing competitions and track and field, for example.
  • a composite video sequence made in accordance with the invention is apparent from Fig. 6, namely for determining differential time between two runners at any desired location of a race. This involves simple counting of the number of frames in the sequence between the two runners passing the location, and multiplying by the time interval between frames.
  • a composite sequence can be broadcast over existing facilities such as network, cable and satellite TV, and as video on the Internet, for example. Such sequences can be offered as on-demand services, e.g. on a channel separate from a strictly real-time main channel. Or, instead of by broadcasting over a separate channel, a composite video sequence can be included as a portion of a regular channel, displayed as a corner portion, for example.
  • composite video sequences can be used in sports training and coaching. And, aside from sports applications, there are potential industrial applications such as car crash analysis, for example. It is understood that composite sequences may be higher-dimensional, such as composite stereo video sequences.
  • one of the given sequences is an audio sequence to be synchronized with a video sequence.
  • the technique can be used to generate a voice-over or "lip-synch" sequence of actor A speaking or singing with the voice of B.
  • dynamic programming techniques can be used for synchronization.
  • the spatio-temporal realignment method can be applied in the biomedical field as well. For example, after orthopedic surgery, it is important to monitor the progress of a patient's recovery. This can be done by comparing specified movements of the patient over a period of time. In accordance with an aspect of the invention, such a comparison can be made very accurately, by synchronizing start and end of the movement, and aligning the limbs to be monitored in two or more video sequences.
  • Another application is in car crash analysis. The technique can be used for precisely comparing the deformation of different cars crashed in similar situations, to ascertain the extent of the difference. Further in car crash analysis, it is important to compare effects on crash dummies. Again, in two crashes with the same type of car, one can precisely compare how the dummies are affected depending on configuration, e.g. of safety belts.

Abstract

Given two video sequences, a composite video sequence can be generated which includes visual elements from each of the given sequences, suitably synchronized and represented in a chosen focal plane. For example, given two video sequences with each showing a different contestant individually racing the same down-hill course, the composite sequence can include elements from each of the given sequences to show the contestants as if racing simultaneously. A composite video sequence can be made also by similarly combining a video sequence with an audio sequence.

Description

METHOD AND SYSTEM FOR COMBINING VIDEO SEQUENCES WITH SPATIO-TEMPORAL ALIGNMENT
Technical Field
The present invention relates to visual displays and, more specifically, to time-dependent visual displays.
Background of the Invention
In video displays, e.g. in sports-related television programs, special visual effects can be used to enhance a viewer's appreciation of the action. For example, in the case of a team sport such as football, instant replay affords the viewer a second chance at "catching" critical moments of the game. Such moments can be replayed in slow motion, and superposed features such as hand-drawn circles, arrows and letters can be included for emphasis and annotation. These techniques can be used also with other types of sports such as racing competitions, for example.
With team sports, techniques of instant replay and the like are most appropriate, as scenes typically are busy and crowded. Similarly, e.g. in the 100- meter dash competition, the scene includes the contestants side-by-side, and slow- motion visualization at the finish line brings out the essence of the race. On the other hand, where starting times are staggered e.g. as necessitated for the sake of practicality and safety in the case of certain racing events such as downhill racing or ski jumping, the actual scene typically includes a single contestant.
Summary of the Invention
For enhanced visualization, by the sports fan as well as by the contestant and his coach, displays are desired in which the element of competition between contestants is manifested. This applies especially where contestants perform sole as in downhill skiing, for example, and can be applied also to group races in which qualification schemes are used to decide who will advance from quarter-final to half- final to final. We have recognized that, given two or more video sequences, a composite video sequence can be generated which includes visual elements from each of the given sequences, suitably synchronized and represented in a chosen focal plane. For example, given two video sequences with each showing a different contestant individually racing the same down-hill course, the composite sequence can include elements from each of the given sequences to show the contestants as if racing simultaneously.
A composite video sequence can be made also by similarly combining one or more video sequences with one or more different sequences such as audio sequences, for example.
Brief Description of the Drawing
Fig. 1 is a block diagram of a preferred embodiment of the invention.
Figs. 2A and 2B are schematics of different downhill skiers passing before a video camera.
Figs. 3 A and 3B are schematics of images recorded by the video camera, corresponding to Figs. 2A and 2B.
Fig. 4 is a schematic of Figs. 2 A and 2B combined.
Fig. 5 is a schematic of the desired video image, with the scenes of Fig. 3 A and 3B projected in a chosen focal plane.
Fig. 6 is a frame from a composite video sequence which was made with a prototype implementation of the invention.
Detailed Description
Conceptually, the invention can be appreciated in analogy with 2- dimensional (2D) "mo hing", i.e. the smooth transformation, deformation or mapping of one image, II, into another, 12, in computerized graphics. Such morphing leads to a video sequence which shows the transformation of II into 12, e.g., of an image of an apple into an image of an orange, or of one human face into another. The video sequence is 3 -dimensional, having two spatial and a temporal dimension. Parts of the sequence may be of special interest, such as intermediate images, e.g. the average of two faces, or composites, e.g. a face with the eyes from II and the smile from 12. Thus, morphing between images can be appreciated as a form of merging of features from the images.
The invention is concerned with a more complicated task, namely the merging of two video sequences. The morphing or mapping from one sequence to another leads to 4-dimensional data which cannot be displayed easily. However, any intermediate combination, or any composite sequence leads to a new video sequence.
Of particular interest is the generation of a new video sequence combining elements from two or more given sequences, with suitable spatio-temporal alignment or synchronization, and projection into a chosen focal plane. For example, in the case of a sports racing competition such as downhill skiing, video sequences obtained from two contestants having traversed a course separately can be time- synchronized by selecting the frames corresponding to the start of the race. Alternatively, the sequences may be synchronized for coincident passage of the contestants at a critical point such as a slalom gate, for example.
The chosen focal plane may be the same as the focal plane of the one or the other of the given sequences, or it may be suitably constructed yet different from both.
Of interest also is synchronization based on a distinctive event, e.g., in track and field, a high-jump contestant lifting off from the ground or touching down again. In this respect it is of further interest to synchronize two sequences so that both lift-off and touch-down coincide, requiring time scaling. The resulting composite sequence affords a comparison of trajectories.
With the video sequences synchronized, they can be further aligned spatially, e.g. to generate a composite sequence giving the impression of the contestants traversing the course simultaneously. In a simple approach, spatial alignment can be performed on a frame-by-frame basis. Alternatively, by taking a plurality of frames from a camera into consideration, the view in an output image can be extended to include background elements from several sequential images. Forming a composite image involves representing component scenes in a chosen focal plane, typically requiring a considerable amount of computerized processing, e.g. as illustrated by Fig. 1 for the special case of two video input sequences.
Fig. 1 shows two image sequences ISl and IS2 being fed to a module 1 1 for synchronization into synchronized sequences 1ST and IS2'. For example, the sequences ISl and IS2 may have been obtained for two contestants in a down-hill racing competition, and they may be synchronized by the module 1 1 so that the first frame of each sequence corresponds to its contestant leaving the starting gate.
The synchronized sequences are fed to a module 12 for background- foreground extraction, as well as to a module 13 for camera coordinate transformation estimation. For each of the image sequences, the module 12 yields a weight-mask sequence (WMS), with each weight mask being an array having an entry for each pixel position and differentiating between the scene of interest and the background/foreground. The generation of the weight mask sequence involves computerized searching of images for elements which, from frame to frame, move relative to the background. The module 13 yields sequence parameters SPl and SP2 including camera angles of azimuth and elevation, and camera focal length and aperture among others. These parameters can be determined from each video sequence by computerized processing including interpolation and matching of images. Alternatively, a suitably equipped camera can furnish the sequence parameters directly, thus obviating the need for their estimation by computerized processing.
The weight-mask sequences WMS1 and WMS2 are fed to a module 13 for "alpha-layer" sequence computation. The alpha layer is an array which specifies how much weight each pixel in each of the images should receive in the composite image. The sequence parameters SPl and SP2 as well as the alpha layer are fed to a module 15 for projecting the aligned image sequences in a chosen focal plane, resulting in the desired composite image sequence. This is exemplified further by Figs. 2A, 2B, 3A, 3B, 4 and 5
Fig. 2 A shows a skier A about to pass a position marker 21, with the scene being recorded from a camera position 22 with a viewing angle φ(A). The position reached by A may be after an elapse of t(A) seconds from A's leaving the starting gate of a race event.
Fig. 2B shows another skier, B, in a similar position relative to the marker 21, and with the scene being recorded from a different camera position 23 and with a different, more narrow viewing angle φ(B). For comparison with skier A, the position of skier B corresponds to an elapse of t(A) seconds from B leaving the starting gate. As illustrated, within t(A) seconds skier B has traveled farther along the race course as compared with skier A.
Figs. 3A and 3B show the resulting respective images.
Fig. 4 shows a combination with Figs. 2A and 2B superposed at a common camera location.
Fig. 5 shows the resulting desired image projected in a chosen focal plane, affording immediate visualization of skiers A and B as having raced jointly for t(A) seconds from a common start.
Fig. 6 shows a frame from a composite image sequence generated by a prototype implementation of the technique, with the frame corresponding to a point of intermediate timing. The value of 57.84 is the time, in seconds, that it took the slower skier to reach the point of intermediate timing, and the value of +0.04 (seconds) indicates by how much he is trailing the faster skier.
The prototype implementation of the technique was written in the "C" programming language, for execution on a SUN Workstation or a PC, for example. Dedicated firmware or hardware can be used for enhanced processing efficiency, and especially for signal processing involving matching and interpolation.
Individual aspects and variations of the technique are described below in further detail.
A. Background/Foreground Extraction
In each sequence, background and foreground can be extracted using a suitable motion estimation method. This method should be "robust", for background/foreground extraction where image sequences are acquired by a moving camera and where the acquired scene contains moving agents or objects. Required also is temporal consistency, for the extraction of background/foreground to be stable over time. Where both the camera and the agents are moving predictably, e.g. at constant speed or acceleration, temporal filtering can be used for enhanced temporal consistency.
Based on determinations of the speed with which the background moves due to camera motion, and the speed of the skier with respect to the camera, background/foreground extraction generates a weight layer which differentiates between those pixels which follow the camera and those which do not. The weight layer will then be used to generate an alpha layer for the final composite sequence.
B. Spatio-temporal Alignment of Sequences Temporal alignment involves the selection of corresponding frames in the sequences, according to a chosen criterion. Typically, in sports racing competitions, this is the time code of each sequence delivered by the timing system, e.g. to select the frames corresponding to the start of the race. Other possible time criteria are the time corresponding to a designated spatial location such as a gate or jump entry, for example.
Spatial alignment is effected by choosing a reference coordinate system for each frame and by estimating the camera coordinate transformation between the reference system and the corresponding frame of each sequence. Such estimation may be unnecessary when camera data such as camera position, viewing direction and focal length are recorded along with the video sequence. Typically, the reference coordinate system is chosen as one of the given sequences — the one to be used for the composite sequence. As described below, spatial alignment may be on a single- frame or multiple-frame basis.
B.l Spatial Alignment on a Single-frame Basis At each step of this technique, alignment uses one frame from each of the sequences. As each of the sequences includes moving agents/objects, the method for estimating the camera coordinate transformation needs to be robust. To this end, the masks generated in background/foreground extraction can be used. Also, as motivated for background/foreground extraction, temporal filtering can be used for enhancing the temporal consistency of the estimation process.
B.2 Spatial Alignment on a Multiple-frame Basis
In this technique, spatial alignment is applied to reconstructed images of the scene visualized in each sequence. Each video sequence is first analyzed over multiple frames for reconstruction of the scene, using a technique similar to the one for background/foreground extraction, for example. Once each scene has been separately reconstructed, e.g. to take in as much background as possible, the scenes can be spatially aligned as described above.
This technique allows free choice of the field of view of every frame in the scene, in contrast to the single-frame technique where the field of view has to be chosen as the one of the reference frame. Thus, in the multiple-frame technique, in case that all contestants are not visible in all the frames, the field and/or angle of view of the composite image can be chosen such that all competitors are visible.
C. Superimposing of Video Sequences After extraction of the background/foreground in each sequence and estimation of the camera coordinate transformation between each sequence and a reference system, the sequences can be projected into a chosen focal plane for simultaneous visualization on a single display. Alpha layers for each frame of each sequence are generated from the multiple background/foreground weight masks. Thus, the composite sequence is formed by transforming each sequence into the chosen focal plane and superimposing the different transformed images with the corresponding alpha weight.
D. Applications
Further to skiing competitions as exemplified, the techniques of the invention can be applied to other speed/distance sports such as car racing competitions and track and field, for example.
Further to visualizing, one application of a composite video sequence made in accordance with the invention is apparent from Fig. 6, namely for determining differential time between two runners at any desired location of a race. This involves simple counting of the number of frames in the sequence between the two runners passing the location, and multiplying by the time interval between frames. A composite sequence can be broadcast over existing facilities such as network, cable and satellite TV, and as video on the Internet, for example. Such sequences can be offered as on-demand services, e.g. on a channel separate from a strictly real-time main channel. Or, instead of by broadcasting over a separate channel, a composite video sequence can be included as a portion of a regular channel, displayed as a corner portion, for example.
In addition to their use in broadcasting, generated composite video sequences can be used in sports training and coaching. And, aside from sports applications, there are potential industrial applications such as car crash analysis, for example. It is understood that composite sequences may be higher-dimensional, such as composite stereo video sequences.
In yet another application, one of the given sequences is an audio sequence to be synchronized with a video sequence. Specifically, given a video sequence of an actor or singer, A, speaking a sentence or singing a song, and an audio sequence of another actor, B, doing the same, the technique can be used to generate a voice-over or "lip-synch" sequence of actor A speaking or singing with the voice of B. In this case, which requires more than mere scaling of time, dynamic programming techniques can be used for synchronization.
The spatio-temporal realignment method can be applied in the biomedical field as well. For example, after orthopedic surgery, it is important to monitor the progress of a patient's recovery. This can be done by comparing specified movements of the patient over a period of time. In accordance with an aspect of the invention, such a comparison can be made very accurately, by synchronizing start and end of the movement, and aligning the limbs to be monitored in two or more video sequences. Another application is in car crash analysis. The technique can be used for precisely comparing the deformation of different cars crashed in similar situations, to ascertain the extent of the difference. Further in car crash analysis, it is important to compare effects on crash dummies. Again, in two crashes with the same type of car, one can precisely compare how the dummies are affected depending on configuration, e.g. of safety belts.

Claims

1. A method for generating a composite video sequence from a plurality of given video sequences, comprising:
(a) synchronizing the given sequences; and (b) forming the composite sequence from the synchronized sequences as projected into a chosen focal plane.
2. The method according to claim 1, wherein synchronizing is with respect to a timed event in the given sequences.
3. The method according to claim 1, wherein synchronizing is with respect to a common spatial event in the given sequences.
4. The method according to claim 1 , wherein synchronizing is with respect to two events in each of the given sequences, with time scaling for equalizing time between the events.
5. The method according to claim 1, wherein the given video sequences have camera parameters including camera location and focal length, wherein the chosen focal plane corresponds to focal plane of the one of the given sequences, and wherein the composite sequence is as viewed from the camera location of the one of the given sequences.
6. The method according to claim 1, wherein forming the composite sequence is on a frame-by-frame basis.
7. The method according to claim 1 , wherein forming the composite sequence is based on several frames of at least one of the sequences, for an expanded field of view in the composite sequence as compared with the one of the sequences.
8. The method according to claim 1, wherein the given video sequences are from a sports event.
9. The method according to claim 8, wherein the sports event is a ski race.
10. The method according to claim 8, wherein the sports event is a car race.
11. A system for generating a composite video sequence from a plurality of given video sequences, comprising:
(a) means for synchronizing the given sequences and
(b) means for forming the composite sequence from the synchronized sequences as projected into a chosen focal plane.
12. The system according to claim 11, wherein the means for synchronizing comprises means for aligning the given sequences with respect to a timed event in the given sequences.
13. The system according to claim 11, wherein the means for synchronizing comprises means for aligning the given sequences with respect to a common spatial event in the given sequences.
14. The system according to claim 11, wherein the means for synchronizing comprises means for aligning the given sequences with respect to two events in each of the given sequences, and means for time scaling for equalizing time between the events.
15. The system according to claim 11, wherein the given video sequences have camera parameters including camera location and focal length, and wherein the means for forming the composite sequence comprises means for choosing the focal plane corresponding to focal plane of the one of the given sequences, and means for forming the composite sequence as viewed from the camera location of the one of the given sequences.
16. The system according to claim 11 , wherein the means for forming the composite sequence comprises means for processing the given sequences on a frame- by-frame basis.
17. The system according to claim 11 , wherein the means for forming the composite sequence comprises means for using several frames of at least one of the sequences, for an expanded field of view in the composite sequence as compared with the one of the sequences.
18. The system according to claim 11, wherein the given video sequences are from a sports event.
19. The system according to claim 18, wherein the sports event is a ski race.
20. The system according to claim 18, wherein the sports event is a car race.
21. A system for generating a composite video sequence from a plurality of given video sequences, comprising: (a) means for synchronizing the given sequences and
(b) a processor which is instructed for forming the composite sequence from the synchronized sequences as projected into a chosen focal plane.
22. A method for generating a composite image, comprising:
(a) synchronizing a plurality of video sequences and (b) forming the composite image from corresponding frames of the synchronized sequences as projected into a chosen focal plane.
23. A broadcast service comprising:
(a) synchronizing a plurality of video sequences; (b) forming a composite sequence from the synchronized sequences as projected into a chosen focal plane; and
(c) broadcasting the composite sequence.
24. A method for generating a composite video sequence from a given video sequence and a given audio sequence, comprising:
(a) synchronizing the given video sequence and the given audio sequence for synchrony between visual features of the video sequence and audio features of the audio sequence; and
(b) forming the composite sequence from the synchronized sequences, having a video portion corresponding to the given video sequence and an audio portion corresponding to the given audio sequence.
25. A system for generating a composite video sequence from a given video sequence and a given audio sequence, comprising:
(a) means for synchronizing the given video sequence and the given audio sequence for synchrony between visual features of the video sequence and audio features of the audio sequence; and
(b) means for forming the composite sequence from the synchronized sequences, having a video portion corresponding to the given video sequence and an audio portion corresponding to the given audio sequence.
26. A system for generating a composite video sequence from a given video sequence and a given audio sequence, comprising:
(a) means for synchronizing the given video sequence and the given audio sequence for synchrony between visual features of the video sequence and audio features of the audio sequence; and (b) a processor which is instructed for forming the composite sequence from the synchronized sequences, having a video portion corresponding to the given video sequence and an audio portion corresponding to the given audio sequence.
27. A method for determining differential time between two contestants at a specified location in a race, comprising: forming a composite video sequence from synchronized given video sequences of the contestants, projected into a chosen focal plane; and counting the number of frames between the contestants passing the location.
28. The method according to claim 1, wherein the given video sequences have biomedical significance.
29. The method according to claim 28, wherein biomedical significance comprises significance as to movement of a limb of a patient.
30. The method according to claim 1 , wherein the given video sequences comprise car crash test sequences.
31. The method according to claim 30, wherein the car crash test sequences comprise images of cars being tested.
32. The method according to claim 30, wherein the car crash test sequences comprise images of crash dummies in cars being tested.
PCT/IB1999/000173 1998-01-16 1999-01-15 Method and system for combining video sequences with spacio-temporal alignment WO1999037088A1 (en)

Priority Applications (9)

Application Number Priority Date Filing Date Title
AT99900616T ATE218780T1 (en) 1998-01-16 1999-01-15 METHOD AND DEVICE FOR COMBINING VIDEO SEQUENCES WITH TEMPO-SPATIAL ALIGNMENT
BRPI9906887A BRPI9906887B8 (en) 1998-01-16 1999-01-15 methods and systems of composite video sequence generation from plurality of given video sequences taken with single camera, method of determining differential time between two competitors at a specified race location and transmission system.
CA002318375A CA2318375C (en) 1998-01-16 1999-01-15 Method and system for combining video sequences with spatio-temporal alignment
IL13727899A IL137278A (en) 1998-01-16 1999-01-15 Method and system for combining video sequences with spatiotemporal alignment
AU19805/99A AU1980599A (en) 1998-01-16 1999-01-15 Method and system for combining video sequences with spacio-temporal alignment
EP99900616A EP1055324B1 (en) 1998-01-16 1999-01-15 Method and system for combining video sequences with spacio-temporal alignment
JP2000540671A JP4739520B2 (en) 1998-01-16 1999-01-15 Method and apparatus for synthesizing video sequence with spatio-temporal alignment
DE69901681T DE69901681T2 (en) 1998-01-16 1999-01-15 METHOD AND DEVICE FOR COMBINING VIDEO SEQUENCES WITH TIME-SPATIAL ORIENTATION
AU2003268578A AU2003268578B2 (en) 1998-01-16 2003-12-09 Method and System for Combining Video Sequences With Spatio-temporal Alignment

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US09/013,772 1998-01-16
US09/013,772 US6320624B1 (en) 1998-01-16 1998-01-16 Method and system for combining video sequences with spatio-temporal alignment

Publications (1)

Publication Number Publication Date
WO1999037088A1 true WO1999037088A1 (en) 1999-07-22

Family

ID=21761675

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB1999/000173 WO1999037088A1 (en) 1998-01-16 1999-01-15 Method and system for combining video sequences with spacio-temporal alignment

Country Status (12)

Country Link
US (1) US6320624B1 (en)
EP (1) EP1055324B1 (en)
JP (1) JP4739520B2 (en)
CN (1) CN1155229C (en)
AT (1) ATE218780T1 (en)
AU (1) AU1980599A (en)
BR (1) BRPI9906887B8 (en)
CA (1) CA2318375C (en)
DE (1) DE69901681T2 (en)
ES (1) ES2177220T3 (en)
IL (1) IL137278A (en)
WO (1) WO1999037088A1 (en)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1128667A2 (en) * 2000-02-26 2001-08-29 Orad Hi-Tec Systems Limited Television illustration system
EP1128668A2 (en) * 2000-02-26 2001-08-29 Orad Hi-Tec Systems Limited Methods and apparatus for enhancement of live events broadcasts by superimposing animation, based on real events
WO2002047031A2 (en) * 2000-12-05 2002-06-13 Yeda Research And Development Co. Ltd. Apparatus and method for alignment of spatial or temporal non-overlapping image sequences
EP1247255A1 (en) * 1999-11-24 2002-10-09 Dartfish SA Coordination and combination of video sequences with spatial and temporal normalization
EP1297487A2 (en) * 2000-05-11 2003-04-02 Yeda Research & Development Company, Ltd. Apparatus and method for spatio - temporal alignment of image sequences
EP1449357A1 (en) * 2001-06-19 2004-08-25 Ecole Polytechnique Fédérale de Lausanne (EPFL) Method and system for combining video with spatio-temporal alignment
US7042493B2 (en) 2000-04-07 2006-05-09 Paolo Prandoni Automated stroboscoping of video sequences
GB2432274A (en) * 2005-11-11 2007-05-16 Roke Manor Research Producing a combined image by determining the position of a moving object in a current image frame
WO2010109402A1 (en) * 2009-03-27 2010-09-30 Koninklijke Philips Electronics N.V. Synchronization of two image sequences of a periodically moving object

Families Citing this family (30)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE69732089T2 (en) * 1996-10-04 2005-12-08 Nippon Telegraph And Telephone Corp. DEVICE AND METHOD FOR THE TIMELY AND SPATIAL INTEGRATION AND MANAGEMENT OF A VARIETY OF VIDEOS AND STORAGE MEDIUM FOR STORING A PROGRAM THEREFOR
KR20030036160A (en) * 2000-05-22 2003-05-09 가부시키가이샤 소니 컴퓨터 엔터테인먼트 Information processing apparatus, graphic processing unit, graphic processing method, storage medium, and computer program
JP2001333348A (en) * 2000-05-24 2001-11-30 Minolta Co Ltd Transmitter, receiver, broadcast system and broadcast method
US8028314B1 (en) 2000-05-26 2011-09-27 Sharp Laboratories Of America, Inc. Audiovisual information management system
US8020183B2 (en) 2000-09-14 2011-09-13 Sharp Laboratories Of America, Inc. Audiovisual management system
US7904814B2 (en) 2001-04-19 2011-03-08 Sharp Laboratories Of America, Inc. System for presenting audio-video content
SG103289A1 (en) * 2001-05-25 2004-04-29 Meng Soon Cheo System for indexing textual and non-textual files
US7164369B2 (en) * 2001-06-19 2007-01-16 Sharp Laboratories Of America, Inc. System for improving storage efficiency of digital files
US7474698B2 (en) 2001-10-19 2009-01-06 Sharp Laboratories Of America, Inc. Identification of replay segments
US6667774B2 (en) * 2001-11-02 2003-12-23 Imatte, Inc. Method and apparatus for the automatic generation of subject to background transition area boundary lines and subject shadow retention
US8214741B2 (en) 2002-03-19 2012-07-03 Sharp Laboratories Of America, Inc. Synchronization of video and data
US7657907B2 (en) 2002-09-30 2010-02-02 Sharp Laboratories Of America, Inc. Automatic user profiling
SE0300286D0 (en) * 2003-02-05 2003-02-05 Axis Ab Method and apparatus for combining video signals into one comprehensive video signal
WO2005006773A1 (en) * 2003-06-20 2005-01-20 Ecole Polytechnique Federale De Lausanne (Epfl) Method and system for combining video sequences with spatio-temporal alignment
US7324166B1 (en) * 2003-11-14 2008-01-29 Contour Entertainment Inc Live actor integration in pre-recorded well known video
US8356317B2 (en) 2004-03-04 2013-01-15 Sharp Laboratories Of America, Inc. Presence based technology
US7594245B2 (en) 2004-03-04 2009-09-22 Sharp Laboratories Of America, Inc. Networked video devices
US8949899B2 (en) 2005-03-04 2015-02-03 Sharp Laboratories Of America, Inc. Collaborative recommendation system
US7457433B2 (en) * 2005-01-20 2008-11-25 International Business Machines Corporation System and method for analyzing video from non-static camera
US8689253B2 (en) 2006-03-03 2014-04-01 Sharp Laboratories Of America, Inc. Method and system for configuring media-playing sets
US8170278B2 (en) * 2008-08-06 2012-05-01 Sri International System and method for detecting and tracking an object of interest in spatio-temporal space
US8326087B2 (en) * 2008-11-25 2012-12-04 Xerox Corporation Synchronizing image sequences
US9025908B2 (en) * 2009-04-08 2015-05-05 The John Hopkins University System and method for aligning video sequences
US8941726B2 (en) * 2009-12-10 2015-01-27 Mitsubishi Electric Research Laboratories, Inc. Method and system for segmenting moving objects from images using foreground extraction
WO2014049989A1 (en) * 2012-09-28 2014-04-03 パナソニック株式会社 Image output device
WO2014155877A1 (en) 2013-03-26 2014-10-02 ソニー株式会社 Image processing device, image processing method, and program
US9648211B2 (en) 2015-05-14 2017-05-09 Xerox Corporation Automatic video synchronization via analysis in the spatiotemporal domain
EP3249651B1 (en) 2016-05-23 2018-08-29 Axis AB Generating a summary video sequence from a source video sequence
GB201615776D0 (en) 2016-09-16 2016-11-02 Rolls Royce Plc Spatiotemporal registation of image streams
CN109743625A (en) * 2018-12-20 2019-05-10 聚好看科技股份有限公司 A kind of method for processing video frequency and television set

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4843483A (en) * 1986-07-05 1989-06-27 Willy Bogner Method for the simultaneous depiction of at least two temporally sequential events on television, and equipment for implementing this method
US5367286A (en) * 1991-04-02 1994-11-22 Swiss Timing Ltd. System for instantaneously displaying the ranking of a competitor in a race with sequential starts
WO1995005873A1 (en) * 1993-08-24 1995-03-02 Easterbrook Norman J A system for instruction of a pupil
EP0669758A1 (en) * 1993-09-06 1995-08-30 Kabushiki Kaisha Oh-Yoh Keisoku Kenkyusho Time-varying image processor and display device

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4183056A (en) * 1977-05-23 1980-01-08 Kewp Electronic Systems, Inc. Apparatus and method for monitoring sports contests
US4157572A (en) 1977-09-12 1979-06-05 University Of Pittsburgh Superimposition of television images
DE2803653C3 (en) 1978-01-27 1986-05-28 Texas Instruments Deutschland Gmbh, 8050 Freising Alignment device
FR2576733B1 (en) 1985-01-28 1987-04-24 Telediffusion Fse IMAGE OVERPRINTING SYSTEM
JPH0666241B2 (en) 1985-10-14 1994-08-24 株式会社日立製作所 Position detection method
ATE56407T1 (en) 1986-07-07 1990-09-15 Edwin S Geary PROPELLER AND CONNECTOR.
US4974073A (en) * 1988-01-14 1990-11-27 Metavision Inc. Seamless video display
US5055926A (en) 1990-04-02 1991-10-08 The United States Of America As Represented By The United States Department Of Energy Video image position determination
GB9119964D0 (en) * 1991-09-18 1991-10-30 Sarnoff David Res Center Pattern-key video insertion
GB9217098D0 (en) 1992-08-12 1992-09-23 British Broadcasting Corp Derivation of studio camera position and motion from the camera image
EP0701758B1 (en) 1993-06-04 2000-11-02 Sarnoff Corporation System and method for electronic image stabilization
US5530483A (en) * 1994-10-11 1996-06-25 Pixel Instruments Corp. Delay detector apparatus and method for plural image sequences
CA2189109A1 (en) * 1995-10-30 1997-05-01 Kenji Mitsui Keying system and composite image producing method

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4843483A (en) * 1986-07-05 1989-06-27 Willy Bogner Method for the simultaneous depiction of at least two temporally sequential events on television, and equipment for implementing this method
US5367286A (en) * 1991-04-02 1994-11-22 Swiss Timing Ltd. System for instantaneously displaying the ranking of a competitor in a race with sequential starts
WO1995005873A1 (en) * 1993-08-24 1995-03-02 Easterbrook Norman J A system for instruction of a pupil
EP0669758A1 (en) * 1993-09-06 1995-08-30 Kabushiki Kaisha Oh-Yoh Keisoku Kenkyusho Time-varying image processor and display device

Cited By (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8675021B2 (en) * 1999-11-24 2014-03-18 Dartfish Sa Coordination and combination of video sequences with spatial and temporal normalization
EP1247255A1 (en) * 1999-11-24 2002-10-09 Dartfish SA Coordination and combination of video sequences with spatial and temporal normalization
EP1247255A4 (en) * 1999-11-24 2007-04-25 Dartfish Sa Coordination and combination of video sequences with spatial and temporal normalization
EP1128668A2 (en) * 2000-02-26 2001-08-29 Orad Hi-Tec Systems Limited Methods and apparatus for enhancement of live events broadcasts by superimposing animation, based on real events
EP1128667A2 (en) * 2000-02-26 2001-08-29 Orad Hi-Tec Systems Limited Television illustration system
EP1128668A3 (en) * 2000-02-26 2003-08-13 Orad Hi-Tec Systems Limited Methods and apparatus for enhancement of live events broadcasts by superimposing animation, based on real events
EP1128667A3 (en) * 2000-02-26 2004-02-11 Orad Hi-Tec Systems Limited Television illustration system
US7042493B2 (en) 2000-04-07 2006-05-09 Paolo Prandoni Automated stroboscoping of video sequences
JP2008282418A (en) * 2000-05-11 2008-11-20 Yeda Res & Dev Co Ltd System and method for spatio-temporal alignment of image sequences
EP1297487A4 (en) * 2000-05-11 2007-11-07 Yeda Res & Dev Apparatus and method for spatio - temporal alignment of image sequences
JP4625854B2 (en) * 2000-05-11 2011-02-02 イェダ リサーチ アンド デベロップメント カンパニー リミテッド Spatio-temporal normalized matching system and method for image sequences
EP1297487A2 (en) * 2000-05-11 2003-04-02 Yeda Research & Development Company, Ltd. Apparatus and method for spatio - temporal alignment of image sequences
US7428345B2 (en) 2000-05-11 2008-09-23 Yeda Research & Development Co. Ltd. Apparatus and method for spatio-temporal alignment of image sequences
WO2002047031A3 (en) * 2000-12-05 2002-12-19 Yeda Res & Dev Apparatus and method for alignment of spatial or temporal non-overlapping image sequences
US7596281B2 (en) 2000-12-05 2009-09-29 Yeda Research And Development Co. Ltd. Apparatus and method for alignment of spatial or temporal non-overlapping images sequences
WO2002047031A2 (en) * 2000-12-05 2002-06-13 Yeda Research And Development Co. Ltd. Apparatus and method for alignment of spatial or temporal non-overlapping image sequences
EP1449357A1 (en) * 2001-06-19 2004-08-25 Ecole Polytechnique Fédérale de Lausanne (EPFL) Method and system for combining video with spatio-temporal alignment
EP1449357A4 (en) * 2001-06-19 2006-10-04 Ecole Polytech Method and system for combining video with spatio-temporal alignment
GB2432274A (en) * 2005-11-11 2007-05-16 Roke Manor Research Producing a combined image by determining the position of a moving object in a current image frame
WO2010109402A1 (en) * 2009-03-27 2010-09-30 Koninklijke Philips Electronics N.V. Synchronization of two image sequences of a periodically moving object
CN102365654A (en) * 2009-03-27 2012-02-29 皇家飞利浦电子股份有限公司 Synchronization of two image sequences of a periodically moving object
US8781257B2 (en) 2009-03-27 2014-07-15 Koninklijke Philips N.V. Synchronization of two image sequences of a periodically moving object

Also Published As

Publication number Publication date
ES2177220T3 (en) 2002-12-01
IL137278A0 (en) 2001-07-24
JP2003523643A (en) 2003-08-05
IL137278A (en) 2005-06-19
ATE218780T1 (en) 2002-06-15
BR9906887A (en) 2000-11-14
JP4739520B2 (en) 2011-08-03
CA2318375C (en) 2008-08-05
US6320624B1 (en) 2001-11-20
BRPI9906887B1 (en) 2015-08-25
CN1288635A (en) 2001-03-21
DE69901681D1 (en) 2002-07-11
DE69901681T2 (en) 2003-01-30
EP1055324B1 (en) 2002-06-05
CA2318375A1 (en) 1999-07-22
CN1155229C (en) 2004-06-23
EP1055324A1 (en) 2000-11-29
AU1980599A (en) 1999-08-02
BRPI9906887B8 (en) 2017-03-07

Similar Documents

Publication Publication Date Title
CA2318375C (en) Method and system for combining video sequences with spatio-temporal alignment
US7843510B1 (en) Method and system for combining video sequences with spatio-temporal alignment
US8675021B2 (en) Coordination and combination of video sequences with spatial and temporal normalization
EP1287518B1 (en) Automated stroboscoping of video sequences
Yow et al. Analysis and presentation of soccer highlights from digital video
US6072504A (en) Method and apparatus for tracking, storing, and synthesizing an animated version of object motion
US20030030734A1 (en) System and method for transitioning between real images and virtual images
US20030030658A1 (en) System and method for mixed reality broadcast
JP4189900B2 (en) Event related information insertion method and apparatus
JP2002502160A5 (en)
CN113706720A (en) Image display method and device
WO2002104009A1 (en) Method and system for combining video with spatio-temporal alignment
RU2602792C2 (en) Motion vector based comparison of moving objects
AU2003268578B2 (en) Method and System for Combining Video Sequences With Spatio-temporal Alignment
CN114302234B (en) Quick packaging method for air skills
WO2005006773A1 (en) Method and system for combining video sequences with spatio-temporal alignment
JP2009519539A (en) Method and system for creating event data and making it serviceable
MXPA00007221A (en) Method and system for combining video sequences with spacio-temporal alignment
Muto et al. Life-size sequential photography in a mixed reality environment
Zhu et al. “Can You Move It?”: The Design and Evaluation of Moving VR Shots in Sport Broadcast
US9836863B2 (en) Method and system for visualization of athletes
Hirohashi et al. Producing a virtual object with realistic motion for a mixed reality space
Grimson et al. Immersive Sporting Events

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 99802201.2

Country of ref document: CN

AK Designated states

Kind code of ref document: A1

Designated state(s): AL AM AT AU AZ BA BB BG BR BY CA CH CN CU CZ DE DK EE ES FI GB GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MD MG MK MN MW MX NO NZ PL PT RO RU SD SE SG SI SK SL TJ TM TR TT UA UG UZ VN YU ZW

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): GH GM KE LS MW SD SZ UG ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE BF BJ CF CG CI CM GA GN GW ML MR NE SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
ENP Entry into the national phase

Ref document number: 2318375

Country of ref document: CA

Ref document number: 2318375

Country of ref document: CA

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 137278

Country of ref document: IL

WWE Wipo information: entry into national phase

Ref document number: 19805/99

Country of ref document: AU

NENP Non-entry into the national phase

Ref country code: KR

WWE Wipo information: entry into national phase

Ref document number: IN/PCT/2000/00095/DE

Country of ref document: IN

WWE Wipo information: entry into national phase

Ref document number: PA/a/2000/007221

Country of ref document: MX

WWE Wipo information: entry into national phase

Ref document number: 1999900616

Country of ref document: EP

REG Reference to national code

Ref country code: DE

Ref legal event code: 8642

WWP Wipo information: published in national office

Ref document number: 1999900616

Country of ref document: EP

WWG Wipo information: grant in national office

Ref document number: 1999900616

Country of ref document: EP