CA2202688C - Method and system for representing hierarchical time-based data structures and to extract information therefrom - Google Patents

Method and system for representing hierarchical time-based data structures and to extract information therefrom Download PDF

Info

Publication number
CA2202688C
CA2202688C CA002202688A CA2202688A CA2202688C CA 2202688 C CA2202688 C CA 2202688C CA 002202688 A CA002202688 A CA 002202688A CA 2202688 A CA2202688 A CA 2202688A CA 2202688 C CA2202688 C CA 2202688C
Authority
CA
Canada
Prior art keywords
layer
data elements
container
time
data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
CA002202688A
Other languages
French (fr)
Other versions
CA2202688A1 (en
Inventor
Raymond Hill
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Avid Technology Inc
Original Assignee
Avid Technology Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Avid Technology Inc filed Critical Avid Technology Inc
Priority to US08/847,536 priority Critical patent/US5930797A/en
Priority to CA002202688A priority patent/CA2202688C/en
Publication of CA2202688A1 publication Critical patent/CA2202688A1/en
Priority to US09/316,949 priority patent/US6023703A/en
Application granted granted Critical
Publication of CA2202688C publication Critical patent/CA2202688C/en
Anticipated expiration legal-status Critical
Expired - Fee Related legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G11INFORMATION STORAGE
    • G11BINFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
    • G11B27/00Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
    • G11B27/10Indexing; Addressing; Timing or synchronising; Measuring tape travel
    • G11B27/34Indicating arrangements 
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/40Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
    • GPHYSICS
    • G11INFORMATION STORAGE
    • G11BINFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
    • G11B27/00Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
    • G11B27/02Editing, e.g. varying the order of information signals recorded on, or reproduced from, record carriers
    • G11B27/031Electronic editing of digitised analogue information signals, e.g. audio or video signals
    • G11B27/034Electronic editing of digitised analogue information signals, e.g. audio or video signals on discs
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99931Database or file accessing
    • Y10S707/99933Query processing, i.e. searching
    • Y10S707/99934Query formulation, input preparation, or translation
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99941Database schema or data structure
    • Y10S707/99942Manipulating data structure, e.g. compression, compaction, compilation
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99941Database schema or data structure
    • Y10S707/99943Generating database or data structure, e.g. via user interface

Abstract

A method and system for representing hierarchical time-based data structures and to extract information therefrom comprises means to create a container, the container having one or more layers within which data elements are arranged along the container axis and to examine the container to produce virtual static graphs which represent active data elements and their interconnections within time durations represented within the container. The data elements in the container can change between active and inactive states with time and the connections between data elements can also change with time and the container represents this information.
A set of constraints is defined for the container such that interconnections between data elements only occur between data elements in different layers. Specifically, a data element can only receive input from one or more other data elements which are in lower levels and a data element can only provide output to one or more data elements which are in higher levels. Transitions in a lower layer, representing changes in the active status of a data element, are propagated to each higher layer, such that a higher layer can never have less transitions that a lower layer. Each layer includes a record of which data element outputs are available in the layer at all times.
When a request for information relating to the active data elements and their interconnections within a specified time period is received, the container is examined to produce a virtual static graph representing the minimal set of active data elements and their interconnections with the specified time.

Description

Method And System For Representing Hierarchical Time-Based Data Structures And To Extract Information Therefrom FIELD OF THE INVENTION
The present invention relates to a method and system of representing unidirectional data structures and extracting sets of data elements from such representations.
More specifically, the present invention relates to a method and system for extracting minimal sets of active data elements and their interconnections from hierarchical, unidirectional data structures such as time line-based data from nonlinear editors and the like.
BACKGROUND OF THE INVENTION
Hierarchical, unidirectional data structures are known and are commonly used to represent a variety of data. Such data structures can also be used to represent data elements with respect to a first axis, commonly a time axis, and wherein the data elements also have inter-relationships between themselves, such as child-parent hierarchies, etc.
One use for such data structures is to store time-based hierarchical data, such as the edits produced with a non-linear editor (NLE). In an NLE, data elements such as media sources, effects and filters are represented by objects which can be positioned and manipulated relative to a common time-based axis, or time line. Each data source can be represented by a clip object that has a duration which can be positioned relative to the time axis and is active (i.e.-generating an output) for all or a portion of that duration according to a time mask applied to the clip object by a user. The output of the source clip object can be a final output from the NLE or can be the input to another object in the NLE, such as a filter object or an effect object.
For example, a user may desire that a ten second portion of video information be output from a digital video stream with a total duration of forty seconds. The requested portion can be located from the tenth to the nineteenth second of the forty second video stream and it is desired to have that output at the thirty second point in the edit being produced, i.e. thirty seconds of some other output occurs and then ten seconds of this output. In such a case, the digital video stream can be represented within the NLE as a video source clip with a duration of forty seconds and is positioned by the user, with respect to a defined global time axis, such that the desired
2 duration of the video source clip (ten to nineteen seconds) commences at the thirty second point on the global time line, i.e. the beginning of the clip is located at the twenty second point of the global timeline. The user then places a time mask along the appropriate portion of the clip (i.e.
from thirty to thirty nine seconds relative to the global time line), activating the clip to output the requested duration of ten seconds, at the desired time. For the portions of the clip before and after the active duration, as indicated by the applied time mask, the clip is inactive and produces no output, although this can be subsequently changed by the user by adding additional time masks and/or changing the existing time mask.
In addition to being active or inactive, as mentioned above an object can be either the child or parent of another object. For example, the above mentioned video clip can be the child of a dissolve effect object and, in such a case, the output of the video clip is supplied to an input of the dissolve effect object for the specified duration and the dissolve effect processes the output from the video source clip and another video source clip to create an output comprising the desired dissolve effect between those clips. Also, an object can be the parent of another clip.
For example, the dissolve effect object is the parent of the video clip described above.
In addition to the time-based change between active and inactive states, the hierarchical relationships between data elements can also change with time.
For example, a thirty second video clip can be active and have no parent for a time t=0 to t=9 (i.e. - the output of the clip is output directly) and can be a child of a dissolve effect for a time t=10 to time t=19 (i.e.- the output of the clip is an input to a dissolve function) and can then be inactive for time t=20 onwards. In such a case, the parent-child relationships within the hierarchy of the NLE
change with time.
It is believed that, in the past, hierarchical unidirectional data structures such as NLE edits have been represented as graph structures which represent the hierarchical relationships between data elements. When it was desired to produce a portion of an edit, the graph structure would be walked, visiting each data element in the graph. Each visited data element would then be required to specify whether it was active or inactive within the desired time portion of the edit, which (if any) of its one or more output connections is active and which other data elements are supplying it with input or inputs.
3 As an NLE edit can be very complex, consisting of many source data clips, effects objects and filter objects, such a graph can be very deep and performing a walk of such a graph can be computationally expensive to implement. Further, such a structure requires that each data element maintain its own list of when it is active and/or its hierarchical relationships with other data elements and this can make constructing and/or updating such a graph structure to reflect changes in the NLE a complex and computationally expensive operation to perform.
It is desired to have a novel method and system of representing and using hierarchical, unidirectional data structures such as NLE edits and the like which is relatively efficient to implement and which allows the data structures to be created and/or updated in a flexible manner.
SUMMARY OF THE INVENTION
It is an object of the present invention to provide a novel method and system of representing and processing hierarchical, unidirectional data structures which obviates or mitigates at least one of the disadvantages of the prior art.
According to a first aspect of the present invention, there is provided a method of representing hierarchical time-based data elements wherein data elements can change between active and inactive states with time and wherein interconnections between said data elements can also change with time and extracting information from specified time portions thereof, comprising the steps of:
(i) determining the data elements to be represented;
(ii) creating a container to represent the states and hierarchy of said data elements;
(iii) placing each data element which receives no input from other data elements in a first, bottom layer of said container, and positioning each said data element with respect to a time line and indicating when each said data element is active and inactive;
(iv) creating a minimal set of one or more higher layers in which other data elements to be represented are placed, indicating when each said other data elements are active and inactive, said other data elements being placed such that they receive input only from data
4 elements in a lower layer and that, if they provide an output to other data elements, those data elements are located in a higher layer;
(v) for each layer in turn in said container, from the bottom layer to the top layer, examining the data elements in the layer to determine the transitions wherein each data element in the layer and in any adjacent lower layer changes between states;
(vi) creating for each layer a layer multiplexer to indicate each output from a data element which is available in each duration between determined transitions;
(vii) examining the top layer of said container to determine for each duration in said container to identify the outputs required for each duration and the data elements which provide said identified outputs;
(viii) for each data element which requires input from another data element, examining the layer multiplexers in lower layers to determine the other data elements which provide the required input; and (ix) in response to a request for active data elements and their interconnections within a specified portion of time represented within said container, outputting a representation of said active data elements a representation of their interconnections for each duration within said portion of time.
According to another aspect of the present invention, there is provided a system for representing hierarchical time-based data elements whose state can change between active and inactive states with time and wherein interconnections between said data elements can also change with time and for extracting information from specified time portions thereof, comprising:
a container having one or more layers, each layer being arranged in a hierarchy;
a database for each layer of said container, each said database including an indication the time of each state transition within a corresponding layer and an immediately adjacent lower layer in said hierarchy;
means to represent data elements within said layers of said container such that each data element which is to receive data from a second data element is represented in a higher layer than that of said second data element and each data element which is to provide an output to a second data element is in a lower layer than that of said second data element;
means to examine said container to determine those data elements which are in an active state for a requested time portion and their interconnections within said requested time portion; and means to output a representation of the determined data elements and their interconnections.
5 BRIEF DESCRIPTION OF THE DRAWINGS
Preferred embodiments of the present invention will now be described, by way of example only, with reference to the attached Figures, wherein:
Figure 1 shows an edit track in a nonlinear editing system;
Figure 2 shows a graph representation of the edit track of Figure 1;
Figures 3a through 3e show virtual static graphs each representing one of five durations within the edit track of Figure 1;
Figure 4 shows a media container in accordance with an embodiment of the present invention, the media container including a single source;
Figure 5 shows a schematic representation of the duration of the media container of Figure 4;
Figure 6 shows the media container of Figure 4 with an added second source;
Figure 7 shows schematic representations of the three durations of the media container of Figure 6;
Figure 8 shows the media container of Figure 6 wherein a time mask has been adjusted;
Figure 9 shows schematic representations of the two durations of the media container of Figure 8;
Figure 10 shows the media container of Figure 8 wherein an effect has been added;
Figure 11 shows schematic representations of the three durations of the media container of Figure 10;
Figure 12 shows the media container of Figure 10 with an added third source;
Figure 13 shows schematic representations of four durations of the media container of Figure 12;
Figure 14 shows the media container of Figure 12 wherein a second effect has been added;
6 Figure 15 shows schematic representations of the five durations of Figure 14;
and Figures 16a through 16e show virtual static graphs corresponding to each of the five durations of Figure 14.
DETAILED DESCRIPTION OF THE INVENTION
As is known, NLE edits can include one or more tracks. For example, an edit representing a television commercial can include a video track, a left audio track and a right audio track. A representation of an edit track created within a non-linear editing system in accordance with the present invention is shown generally at 20 in Figure 1.
Edit track 20 includes two data source clips, identified as S 1 and S2 in a first layer LO
and in a second layer L 1 includes two filters, identified as F 1 and F2 and a dissolve effect identified as E 1. In the case wherein edit track 20 is a video track, sources S 1 and S2 can be digital video data, in one of a variety of formats.
In Figure 1, source clip S 1 has a total duration of eight seconds and is positioned from time t0 to t8 relative to the global time line 24 of edit 20. A time mask 28 (indicated in the Figure by a black bar) has been applied to source clip S1 to make it active for four seconds, specifically from t0 to t4.
In the prior art of which the inventor is aware, the activity of a clip was represented solely by its presence or absence in the NLE edit. For example, if a clip was shown as being five seconds long, it was active for five seconds. To make it active for a longer time period required "stretching" the clip, for example to seven seconds in length, and similarly, reducing the time it was active required "clipping" the clip, for example to three seconds in length. In contrast, in the present invention a time mask is used to activate a portion or portions of a clip.
In Figure 1, source clip S2 has a total duration of six seconds and is positioned from time t2 to t8 of time line 24. A time mask 32 has been applied to source clip S2 to make it active from t2 to t7.
Filter obj ect F 1 acts on source clip S 1, as indicated by the "(S 1 )"
suffix in the
7 Figure, from t0 to tl and can, for example, be a fade-in which results in the output of source clip S 1 fading in from black to full visibility between t0 and tl in the output of edit track 20.
Dissolve effect object E1 acts on source clips S1 and S2, as indicated by the "(S1, S2)" suffix in the Figure, from t2 to t4 and represents a dissolve from the output of clip S
1 to the output of clip S2 over a two second duration. Finally, filter object F2 acts on source clip S2 from t6 to t7 and can, for example, be a fade-out effect which results in the output of source clip S2 fading from full visibility to black between t6 and t7 in the output of edit track 20.
Generally, in an NLE editor only a single output is produced at any one time for a given track. For example, a video output track and a right and a left audio output track can be created and output by an NLE edit, but two video outputs within a video track or two left audio outputs within a track will not be created and/or output at any one time. As will be apparent to those of skill in the art, the actual output on any track is determined by examining the layers of the edit, from top down. Thus, in Figure 1, it can be determined that filter object F1 generates the output from t0 to tl, source clip S1 generates the output from tl to t2, dissolve effect object E1 generates the output from t2 to t4, source clip S2 generates the output from t4 to t6 and filter object F2 generates the output from t6 to t7. As no object in track edit 20 is active after t7, the output ceases at that point.
When it is desired to output the result of an edit, or portion of an edit, the information representing the portion of the edit is output to the rendering system, which produces the desired output, according to the edit information. Specifically, the rendering system requires information to identify the active elements for the portion of the edit track, or tracks, to be rendered and their interconnections within the requested portion.
Figure 2 shows a prior art graph 50 which can be used to represent edit track 20.
In the graph each of the sources, effects and functions is represented as a data element, or node, and the data elements are arranged in a hierarchy for the entire duration of the edit track, i.e. -from t0 to t7, as shown. When it is desired to generate an output of edit track 20, the graph is walked, visiting each data element and querying the element to determine whether the data element creates an output (i.e. - is active), and at what time, and whether the data element receives input or inputs from other specified data elements and when. This information is then passed to the rendering system. As can be seen, these determinations are generally not easily made, nor is the graph easily updated to reflect changes in edit track 20 which may occur. Also, it is possible that redundant interconnections and/or data elements are presented to the rendering system, resulting in an inefficient use of the rendering system capacity.
In contrast, in the present invention one or more representations, referred to by the present inventor as virtual static graphs, are created as needed when a specified portion of an edit track is to be output. As used herein, the term virtual static graph is intended to comprise a graph, or graph-like representation, whose connections do not change with time and wherein the nodes in the graph are not duplicated, but instead the connections merely change.
Specifically, in the present invention an edit track is represented within a media container which is examined to determine all the durations between transitions in the track.
When it is desired to render a portion of an edit track, the media container is examined to create a representative virtual static graph or graphs which are output to the rendering hardware and/or software.
For example, edit track 20 of Figure 1 has five durations and Figures 3a, 3b, 3c, 3d and 3e show virtual static graphs which are created in accordance with the present invention to represent these durations. Figure 3a shows a virtual static graph representing the time span from t0 to tl, Figure 3b shows a virtual static graph representing the time span from tl to t2, Figure 3c shows a virtual static graph representing the time span from t2 to t4, Figure 3d shows a virtual static graph representing the time span from t4 to t6 and Figure 3e shows a virtual static graph representing the time span from t6 to t7. Each virtual static graph represents both the active data elements and their hierarchical relationships for each duration of edit track 20 and represents the duration with only those data elements which are active within the duration and with the minimum number of connections between data elements.
As will be apparent to those of skill in the art, virtual static graphs provide a simple, yet effect technique for representing the structure of portions of an NLE edit track to the rendering system. In the present invention, a minimum set of virtual static graphs is provided in response to all requests for an output portion of the NLE edit track. This set of minimal virtual static graphs is then processed, in any suitable manner, by the rendering system to produce the desired output. For example, if a user wishes to output edit track 20 from t2 to t6, the virtual static graphs shown in Figures 3c and 3d are produced for the rendering system. As will be apparent to those of skill in the art, in practice the rendering system will in fact be provided with an appropriate representation of the virtual static graphs, for example as one or more data structures in a suitable format, the data structures comprising the information represented by the virtual static graphs.
The method of constructing the set of minimal virtual static graphs for a requested duration of an NLE edit track is presently implemented as described below, with reference to Figures 4 to 16e. As presently implemented, the construction of virtual static graphs is performed on an as-needed basis, i.e. - when it is desired to produce an output from an NLE edit track.
Figure 4 shows a representation of a media container 100 employed within the present invention. Container 100 is constructed, as required, when one or more virtual static graphs are to be output to produce a requested duration of the edit.
Specifically, when one or more virtual static graphs are to be constructed for a requested portion, a container 100 is constructed and the virtual static graphs necessary to output the requested portion are produced therefrom. If a different portion is subsequently requested, without otherwise modifying the edit, the same container can be employed to extract another set of virtual static graphs which can be used to produce the different requested portion. However, if the edit has been modified, the container will be reconstructed, as described below. In any event, in a present embodiment of the invention, container 100 is not persisted when a edit is saved.
Container 100 comprises source objects, effects objects, filter objects and time masks arranged in layers and further includes a database object for each layer, referred to herein as a layer multiplexer or layer mux, which maintains a record of the objects in the layer of the container whose outputs are available at any given time and the transition times for objects in the layer and in any immediately adjacent lower layer. The conatiner also comprises, for each object in the container, a per object database which indicates, for each object, the connections to and from the object and the times they are active. As will be apparent to those of skill in the art, a transition occurs when the state of an object changes, for example from an active state to an inactive state, or vice versa.

A set of constraints is defined for the interconnection of objects in media container 100, the constraints requiring that objects be arranged in layers.
Specifically, under these constraints an object in a media container cannot receive input from another object in the same layer or from an object in a higher layer. Further, in the present embodiment of the invention an 5 object cannot provide an output to an object in the same layer or to an object in a lower layer.
Also, at the highest layer of a container 100 only a single output can be produced at any given time, although it is contemplated that this constraint can be waived, if desired, for particular applications of the present invention.
10 Finally, as mentioned above and discussed further below, transitions are propagated from lower layers to higher layers and thus, a higher layer can never have fewer transitions than a lower layer. In Figure 4, these constraints require only a single layer, L0, in media container 100.
In Figure 4, container 100 includes a single source clip S 1, which has a time mask 104 applied to it from time t10 to t40, and a layer multiplexer 108. Figure 5 shows a schematic view of media container 100. Layer multiplexer 108 includes indications of two transitions which define a duration from t10 to t40 wherein the output of S1 is available from t10 to t40.
Figure 6 shows the media container of Figure 4 wherein a second source, S2, has been added. Time mask 112 has been applied to the entire duration of source S2 and layer multiplexer 108 has been updated to include indications of transitions at t20 and t40 and shows that an output is available from S 1 from t10 to t20, from S 1 and S2 from t20 to t40 and from S2 from t40 to 50.
Figure 7 shows a schematic view of media container 100 in Figure 6, wherein the data elements which are not available to produce an output are shown with in broken line.
Specifically, from time t10 to t20 source S2 is not available to produce an output and is shown in broken line, as is source S 1 from time t40 to t50 when it is not available to produce an output.
In fact, as described above, in a presently preferred embodiment of the invention, the topmost layer of a media container can only have a single output at any given time and thus, in Figure 8 time mask 112 has been modified such that source S2 is only active from time t40 to t50 and layer multiplexer 108 has been updated accordingly. Figure 9 shows an updated schematic view of Figure 8.
In Figure 10, a dissolve effect E1 has been added to media container 100. As will require input from other items in media container 100, to comply with the above-mentioned connection constraints, E1 is placed in a second layer, L1. When layer L1 is added, a second layer multiplexer 120 is created and updated accordingly.
For example, assuming that, the track edit constructed by the user of the NLE
has E1 receiving input from sources S l and S2, from t20 to t40 and performing a dissolve between them. As indicated by layer multiplexer 108 in Figure 8, no output is available from source S2 from t20 to t40 and thus time mask 112 is updated, as shown in Figure 10, to be applied to the full duration of S2, from t 20 to t50. Further, a time mask 124 is applied to the full duration of E1 to activate it from t20 to t40. Layer multiplexer 120 is updated to indicate that an output is available from source S 1 for t10 to t20, an output is available from effect E1 from t20 to t40 and an output is available from source S2 from t40 to t50. Also, a per object database is created for E 1 which indicates the connection between E 1 and S 1 for t20 to t40 and between E 1 and S2 for the same time. The databases for S 1 and S2 are updated to indicate their respective connections to E 1 as well.
As will be apparent, and as mentioned above, layer L 1 has but a single output in each duration and thus the above-mentioned constraints are met. Of course, layer LO has two outputs from t20 to t40, but this is permitted as LO is not the topmost layer.
Figure 11 shows the corresponding schematic representations.
Figure 12 shows the media container of Figure 10 wherein another source S3 has been added. Source S3 has been added to layer LO and, in order to meet the single output constraint, a time mask 130 has only been applied for the duration t50 to t60.
Layer multiplexer 108 and layer multiplexer 120 have been updated accordingly. Figure 13 shows the corresponding schematic representation of the durations of media container 100. It should be noted that, as objects are added, removed or modified in container 100, creation and/or updating of media container 100 is performed in a bottom to top manner, with LO being processed first, L 1 next, etc.
Figure 14 shows media container 100 wherein a third layer, L2, has been added to receive a second effect E2 and a third layer multiplexer 134 has been created. E2 has been placed in L2 as E2 is active at the same time that E1 is active, specifically from t30 to t40, and thus E2 has been placed in L2 to meet the above-mentioned constraint that there only be a single output at the top layer of container 100. Further, it should be noted that layer multiplexes 120 shows two durations for E1, from t20 to t30 and from t30 to t40. This is due to the fact that, as mentioned above, transitions are propagated from lower layers to higher layers and, in fact a higher layer can never have fewer transitions than a lower layer. Thus, as there is a transition at t30 when S3 becomes active, this transition is propagated to layer multiplexes 120. Similarly, layer multiplexes 134 shows two durations for E2, from t30 to t40 and from t40 to t50, as E1 becomes inactive at t40 and this transition is propagated upwards to layer L2.
In fact, the transition wherein E 1 becomes inactive at t40 has been propagated up from LO
wherein S 1 becomes inactive at t40.
As defined by the user of the NLE, effect E2 is to be active from t30 to t50 as indicated by time mask 138 and is to be feed with source S2 and S3. As will be apparent from Figure 14, S2 is not available to feed E2 from layer L1 as effect El is occluding S2, as indicated by layer multiplexes 120 which only shows outputs from E1 and S3 in t30 to t40. Thus, effect E2 employs what the present inventor refers to as an indirect connection and connects to E1 from t30 to t40 as an indirect connection to source S2 and connects directly to source S2 for t40 to t50.
As will apparent to those of skill in the art, indirect connections allow for the layering of effects and/or filters onto sources, the output from one effect or filter serving as the input to another. With an indirect connection, a feeder will connect to the closest object which is connected, directly or otherwise, to the specified object. While not employed in a presently preferred embodiment of the invention, a connection can also be specified as a direct connection wherein a connection will only be made directly to the specified object.
Figure 15 shows the schematic representation of the five durations of media container 100 in Figure 14. In the schematic for t30 to t40, it is important to note that the direct connection between E2 and S2 is not active, S2 is shown in solid line only because it is active to E 1.
As will be apparent from the above, the process of creating a media container comprises the steps of:
(i) determining the data elements to be included in the container;
(ii) positioning each data element representing a data source in a first, bottom layer;
(iii) creating a minimal set of one or more higher layers in which other data elements are positioned, these other data elements being positioned such that they receive input only from data elements in a lower layer and that, if they provide an output to other data elements those data elements are located in a higher layer;
(iv) setting the current layer to be the lowest layer; and (v) for the current layer, creating a layer multiplexes containing an indication of each transition in the current layer and all transitions in the layer multiplexes of the adjacent lower layer, if any, each transition indicating the start and/or end of a duration and the layer multiplexes indicating the outputs available, if any, for each duration; and (vi) setting the current layer to be the next higher layer and repeating step (v) until a layer multiplexes has been created for all layers, the layer multiplexes for topmost layer including indications of all the transitions within the container.
The process of updating a media container comprises the steps o~
(i) ensuring that each data element is positioned within a layer such that any data elements which provide the data element under consideration with an input are in a lower layer and that any data element which the data element under consideration provides output to is in a higher later, moving data elements between layers and/or creating or removing layers as necessary; and (ii) for each layer in turn, commencing at the bottom layer and proceeding upwards to the top layer, updating the layer multiplexes to include an indication of each transition in the layer and in the layer multiplexes of the adjacent lower layer, if any, and the outputs, if any, available in each duration.

In fact, as will be apparent to those of skill in the art, the present invention permits incremental updating of existing media containers. Specifically, a modification to a layer, or layers only, necessitates updating of the lowest modified layer and the layers above. In a more complex case, improved incremental updating of media containers can be accomplished by considering, in the bottom most layer in which a modification occurred and layers above, the total duration within which the modifications have been made and those durations after it.
Virtual static graphs are formed from media containers by examining the container. Specifically, the examination of the container to form virtual static graphs comprises the following steps wherein the required outputs within a requested portion of an edit is determined and the layer multiplexers and the per object databases are examined to determine the required connections, the availability of the outputs and/or the data elements which provide inputs to the required outputs.
First, the desired output portion of media container is specified. Next, a determination is made of each duration of the media container within the specified portion. The transition times wherein each duration starts and ends within a media container are noted and, in media container 100 of Figure 14 and assuming a specified portion of from t10 to t60, these transition times are located at t10, t20, t30, t40, t50 and t60, defining five durations.
Next, at each transition time the required output data element is determined by examining the topmost layer multiplexer to determine the required output data element (if any) for each duration. In Figure 14, from layer multiplexer 134, the required data elements are: S 1 for t10 to t20; E1 for t20 to t30; E2 for t30 to t40; E2 for t40 to t50; and S3 for t50 to t60.
Next, if a duration's required output data element is a source element, a virtual static graph comprising the source element is produced. As S 1 and S3 are source data elements, the virtual static graphs for t10 to t20 and t50 to t60 are produced as shown in Figures 16a and 16e, respectively.
If the duration's required output data element is not a source data element, a determination is made from the per object database as to the connections required by the object within a duration. Next a determination is made as to whether the data element or elements which provide input to the required output data element are available. The determination commences by examining the layer multiplexes of the layer below and, if the required inputs are 5 not present in that layer multiplexes, proceeding to each lower layer multiplexes in turn until each input is identified. In the case of a data element permitting indirect connections, the data elements in each lower layer are examined, in turn, to determine if they receive the required input as an input. If this is the case, the intervening data element's output is used as the input to the required output data element.
For example, E1 requires input from S1 and S2 and, from layer multiplexes 120 of L 1 in Figure 14, a determination is made that S 1 and S2 are not available at layer L 1 for t20 to t30. From layer multiplexes 108, it is determined that source data elements S1 and S2 are available in layer L0. A virtual static graph produced for t20 to t30 is shown in Figure 16b.
For t30 to t40, the required output data element is E2 which is not a source data element and which receives input from S2 and S3. Accordingly, layer multiplexes 120 is examined to determine if S2 and S3 are available. From layer multiplexes 120, it is determined that for t30 to t40, E1 and S3 are available. If E2's inputs are defined as direct connection, no virtual static graph will be output for t30 to t40 as S2 is not directly available. However, in this example E2 has been defined as permitting indirect connections and thus an examination is made of E1. It is determined that one of El's inputs is S2 and thus El is supplied to E2, along with S3. The resulting virtual static graph produced for t30 to t40 is shown in Figure 16c.
For t40 to t50, the required output data element is still E2. However, from examining layer multiplexes 120 for this duration, it is apparent that S2 is directly available as an input to E2. The resulting virtual static graph produced for this duration is shown in Figure 16d.
The present invention allows for an effective and efficient representation and use of complex unidirectional data such as NLE edits and the like. When it is desired to render an edit represented by a portion of such data, the present invention produces the minimal set of data for the requested portion, allowing efficient processing of the data.
The above-described embodiments of the invention are intended to be examples of the present invention and alterations and modifications may be effected thereto, by those of skill in the art, without departing from the scope of the invention which is defined solely by the claims appended hereto.

Claims (5)

I claim:
1. A method of representing hierarchical time-based data elements wherein data elements can change between active and inactive states with time and wherein interconnections between said data elements can also change with time and extracting information from specified time portions thereof, comprising the steps of:
(i) determining the data elements to be represented;
(ii) creating a container to represent the states and hierarchy of said data elements;
(iii) placing each data element which receives no input from other data elements in a first, bottom layer of said container, and positioning each said data element with respect to a time line and indicating when each said data element is active and inactive;
(iv) creating a minimal set of one or more higher layers in which other data elements to be represented are placed, indicating when each said other data elements are active and inactive, said other data elements being placed such that they receive input only from data elements in a lower layer and that, if they provide an output to other data elements, those data elements are located in a higher layer;
(v) for each layer in turn in said container, from the bottom layer to the top layer, examining the data elements in the layer to determine the transitions wherein each data element in the layer and in any adjacent lower layer changes between states;
(vi) creating for each layer a layer multiplexer to indicate each output from a data element which is available in each duration between determined transitions;
(vii) examining the top layer of said container to determine for each duration in said container to identify the outputs required for each duration and the data elements which provide said identified outputs;
(viii) for each data element which requires input from another data element, examining the layer multiplexers in lower layers to determine the other data elements which provide the required input; and (ix) in response to a request for active data elements and their interconnections within a specified portion of time represented within said container, outputting a representation of said active data elements a representation of their interconnections for each duration within said portion of time.
2. The method of claim 1 wherein step (ix) is repeated for each request for a portion of time.
3. The method of claim 1 wherein, when the contents of said container are modified, only the lowest layer in which a modification has occurred and the layers above that layer are examined to update the positioning of data elements in these layers and/or to update the layer multiplexers.
4. The method of claim 3 wherein each examined layer is only examined from the earliest time position at which a modification has been effected.
5. A system for representing hierarchical time-based data elements whose state can change between active and inactive states with time and wherein interconnections between said data elements can also change with time and for extracting information from specified time portions thereof, comprising:
a container having one or more layers, each layer being arranged in a hierarchy;
a layer multiplexer for each layer of said container, each said layer multiplexer including an indication the time of each state transition within a corresponding layer and an immediately adjacent lower layer in said hierarchy;
means to represent data elements within said layers of said container such that each data element which is to receive data from a second data element is represented in a higher layer than that of said second data element and each data element which is to provide an output to a second data element is in a lower layer than that of said second data element;
means to examine said container to determine those data elements which are in an active state for a requested time portion and their interconnections within said requested time portion;
and means to output a representation of the determined data elements and their interconnections.
CA002202688A 1997-04-15 1997-04-15 Method and system for representing hierarchical time-based data structures and to extract information therefrom Expired - Fee Related CA2202688C (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
US08/847,536 US5930797A (en) 1997-04-15 1997-04-15 Method and system for representing hierarchical time-based data structures and to extract information therefrom
CA002202688A CA2202688C (en) 1997-04-15 1997-04-15 Method and system for representing hierarchical time-based data structures and to extract information therefrom
US09/316,949 US6023703A (en) 1997-04-15 1999-05-24 Method and system for representing hierarchical time-based data structures and to extract information therefrom

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US08/847,536 US5930797A (en) 1997-04-15 1997-04-15 Method and system for representing hierarchical time-based data structures and to extract information therefrom
CA002202688A CA2202688C (en) 1997-04-15 1997-04-15 Method and system for representing hierarchical time-based data structures and to extract information therefrom

Publications (2)

Publication Number Publication Date
CA2202688A1 CA2202688A1 (en) 1998-10-15
CA2202688C true CA2202688C (en) 2005-07-26

Family

ID=25679245

Family Applications (1)

Application Number Title Priority Date Filing Date
CA002202688A Expired - Fee Related CA2202688C (en) 1997-04-15 1997-04-15 Method and system for representing hierarchical time-based data structures and to extract information therefrom

Country Status (2)

Country Link
US (2) US5930797A (en)
CA (1) CA2202688C (en)

Families Citing this family (73)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6654933B1 (en) * 1999-09-21 2003-11-25 Kasenna, Inc. System and method for media stream indexing
EP0972272B1 (en) * 1997-04-03 2003-08-13 Intergraph Corporation Interactive timeline visualization
CA2202688C (en) * 1997-04-15 2005-07-26 Microsoft Corporation Method and system for representing hierarchical time-based data structures and to extract information therefrom
US6686918B1 (en) * 1997-08-01 2004-02-03 Avid Technology, Inc. Method and system for editing or modifying 3D animations in a non-linear editing environment
US6182080B1 (en) * 1997-09-12 2001-01-30 Netvoyage Corporation System, method and computer program product for storage of a plurality of documents within a single file
US6217331B1 (en) 1997-10-03 2001-04-17 Implant Innovations, Inc. Single-stage implant system
US6343291B1 (en) * 1999-02-26 2002-01-29 Hewlett-Packard Company Method and apparatus for using an information model to create a location tree in a hierarchy of information
US6314427B1 (en) * 1999-02-26 2001-11-06 Hewlett-Packard Company Method and apparatus for using an information model to organize an information repository into an extensible hierarchy of organizational information
US6345277B1 (en) * 1999-02-26 2002-02-05 Hewlett-Packard Company Method and apparatus for using an information model to organize an information repository into an extensible hierarchy of information
US6321229B1 (en) * 1999-02-26 2001-11-20 Hewlett-Packard Company Method and apparatus for using an information model to organize an information repository into a hierarchy of information
US6544294B1 (en) * 1999-05-27 2003-04-08 Write Brothers, Inc. Method and apparatus for creating, editing, and displaying works containing presentation metric components utilizing temporal relationships and structural tracks
US7057612B2 (en) * 2000-01-12 2006-06-06 Balfour Technologies Llc Method and system for a four-dimensional temporal visualization data browser
US7343617B1 (en) 2000-02-29 2008-03-11 Goldpocket Interactive, Inc. Method and apparatus for interaction with hyperlinks in a television broadcast
US7120924B1 (en) 2000-02-29 2006-10-10 Goldpocket Interactive, Inc. Method and apparatus for receiving a hyperlinked television broadcast
US7367042B1 (en) * 2000-02-29 2008-04-29 Goldpocket Interactive, Inc. Method and apparatus for hyperlinking in a television broadcast
US7257641B1 (en) * 2000-03-30 2007-08-14 Microsoft Corporation Multipoint processing unit
US7277956B2 (en) * 2000-07-28 2007-10-02 Kasenna, Inc. System and method for improved utilization of bandwidth in a computer system serving multiple users
US7310678B2 (en) * 2000-07-28 2007-12-18 Kasenna, Inc. System, server, and method for variable bit rate multimedia streaming
US20020078446A1 (en) * 2000-08-30 2002-06-20 Jon Dakss Method and apparatus for hyperlinking in a television broadcast
US6882891B2 (en) * 2000-12-06 2005-04-19 Microsoft Corporation Methods and systems for mixing digital audio signals
US7447754B2 (en) * 2000-12-06 2008-11-04 Microsoft Corporation Methods and systems for processing multi-media editing projects
US7114162B2 (en) 2000-12-06 2006-09-26 Microsoft Corporation System and methods for generating and managing filter strings in a filter graph
US7287226B2 (en) 2000-12-06 2007-10-23 Microsoft Corporation Methods and systems for effecting video transitions represented by bitmaps
US6834390B2 (en) * 2000-12-06 2004-12-21 Microsoft Corporation System and related interfaces supporting the processing of media content
US6954581B2 (en) * 2000-12-06 2005-10-11 Microsoft Corporation Methods and systems for managing multiple inputs and methods and systems for processing media content
US6774919B2 (en) * 2000-12-06 2004-08-10 Microsoft Corporation Interface and related methods for reducing source accesses in a development system
US7103677B2 (en) 2000-12-06 2006-09-05 Microsoft Corporation Methods and systems for efficiently processing compressed and uncompressed media content
US6959438B2 (en) 2000-12-06 2005-10-25 Microsoft Corporation Interface and related methods for dynamically generating a filter graph in a development system
US6912717B2 (en) 2000-12-06 2005-06-28 Microsoft Corporation Methods and systems for implementing dynamic properties on objects that support only static properties
US6768499B2 (en) * 2000-12-06 2004-07-27 Microsoft Corporation Methods and systems for processing media content
US7114161B2 (en) 2000-12-06 2006-09-26 Microsoft Corporation System and related methods for reducing memory requirements of a media processing system
US6983466B2 (en) * 2000-12-06 2006-01-03 Microsoft Corporation Multimedia project processing systems and multimedia project processing matrix systems
US6961943B2 (en) * 2000-12-06 2005-11-01 Microsoft Corporation Multimedia processing system parsing multimedia content from a single source to minimize instances of source files
US20040039934A1 (en) * 2000-12-19 2004-02-26 Land Michael Z. System and method for multimedia authoring and playback
AU2002231289A1 (en) * 2000-12-19 2002-07-01 Coolernet, Inc. System and method for multimedia authoring and playback
US7146374B1 (en) * 2001-02-21 2006-12-05 Oracle Corporation Automated multi-line labeling of a time axis
EP1374080A2 (en) * 2001-03-02 2004-01-02 Kasenna, Inc. Metadata enabled push-pull model for efficient low-latency video-content distribution over a network
US20030018978A1 (en) * 2001-03-02 2003-01-23 Singal Sanjay S. Transfer file format and system and method for distributing media content
US6961545B2 (en) * 2001-04-09 2005-11-01 Atheros Communications, Inc. Method and system for providing antenna diversity
US20020147739A1 (en) * 2001-04-10 2002-10-10 Netvoyage Corporation Methods and systems for tracking storage resources associated with a document distribution system
US7155450B2 (en) * 2001-05-10 2006-12-26 Automsoft R & D Limited Plant information management system
US7480864B2 (en) 2001-10-12 2009-01-20 Canon Kabushiki Kaisha Zoom editor
US7606819B2 (en) * 2001-10-15 2009-10-20 Maya-Systems Inc. Multi-dimensional locating system and method
US7680817B2 (en) * 2001-10-15 2010-03-16 Maya-Systems Inc. Multi-dimensional locating system and method
US8316306B2 (en) * 2001-10-15 2012-11-20 Maya-Systems Inc. Method and system for sequentially navigating axes of elements
US7937284B2 (en) * 2001-11-27 2011-05-03 Siebel Systems, Inc. Method and system for managing time-based organization hierarchies
US8600799B2 (en) * 2001-11-27 2013-12-03 Siebel Systems, Inc. Method and system for sales-credit assignment via time-based organization hierarchies
US8027869B2 (en) * 2001-11-27 2011-09-27 Siebel Systems, Inc. Method and system for monitoring achievement and attainment and calculating compensation via time-based organization hierarchies
US20080058106A1 (en) 2002-10-07 2008-03-06 Maya-Systems Inc. Multi-dimensional locating game system and method
US20040199650A1 (en) * 2002-11-14 2004-10-07 Howe John E. System and methods for accelerating data delivery
US7278111B2 (en) 2002-12-26 2007-10-02 Yahoo! Inc. Systems and methods for selecting a date or range of dates
US7496857B2 (en) * 2003-04-25 2009-02-24 Yahoo! Inc. Systems and methods for relating events to a date or date range selection
US20050262245A1 (en) * 2004-04-19 2005-11-24 Satish Menon Scalable cluster-based architecture for streaming media
US7821516B2 (en) * 2004-05-17 2010-10-26 Pixar Automatic pre-render pinning of change isolated assets methods and apparatus
US8296366B2 (en) * 2004-05-27 2012-10-23 Microsoft Corporation Efficient routing of real-time multimedia information
US20050267894A1 (en) * 2004-06-01 2005-12-01 Telestream, Inc. XML metabase for the organization and manipulation of digital media
US7793329B2 (en) * 2006-02-06 2010-09-07 Kasenna, Inc. Method and system for reducing switching delays between digital video feeds using multicast slotted transmission technique
JP4771831B2 (en) * 2006-03-02 2011-09-14 富士通株式会社 Graphic display program and graphic display method
US20080109557A1 (en) * 2006-11-02 2008-05-08 Vinay Joshi Method and system for reducing switching delays between digital video feeds using personalized unicast transmission techniques
US8826123B2 (en) * 2007-05-25 2014-09-02 9224-5489 Quebec Inc. Timescale for presenting information
CA2601154C (en) 2007-07-07 2016-09-13 Mathieu Audet Method and system for distinguising elements of information along a plurality of axes on a basis of a commonality
US8601392B2 (en) 2007-08-22 2013-12-03 9224-5489 Quebec Inc. Timeline for presenting information
CA2657835C (en) * 2008-03-07 2017-09-19 Mathieu Audet Documents discrimination system and method thereof
CA2677921C (en) 2008-09-12 2017-06-13 Mathieu Ma Audet Method of managing groups of arrays of documents
US8624898B1 (en) 2009-03-09 2014-01-07 Pixar Typed dependency graphs
US10387524B2 (en) * 2010-09-29 2019-08-20 Open Text Sa Ulc System and method for managing objects using an object map
US10095367B1 (en) * 2010-10-15 2018-10-09 Tivo Solutions Inc. Time-based metadata management system for digital media
US9122374B2 (en) 2011-02-01 2015-09-01 9224-5489 Quebec Inc. Expandable and collapsible arrays of documents
CA2790799C (en) 2011-09-25 2023-03-21 Mathieu Audet Method and apparatus of navigating information element axes
US9519693B2 (en) 2012-06-11 2016-12-13 9224-5489 Quebec Inc. Method and apparatus for displaying data element axes
US9646080B2 (en) 2012-06-12 2017-05-09 9224-5489 Quebec Inc. Multi-functions axis-based interface
US9728224B2 (en) * 2013-01-08 2017-08-08 Marcus van Bavel Video editing system user interface using virtual film equipment
US10671266B2 (en) 2017-06-05 2020-06-02 9224-5489 Quebec Inc. Method and apparatus of aligning information element axes

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3160017B2 (en) * 1991-08-28 2001-04-23 株式会社日立製作所 Network management display device
US5428730A (en) * 1992-12-15 1995-06-27 International Business Machines Corporation Multimedia system having software mechanism providing standardized interfaces and controls for the operation of multimedia devices
US5826102A (en) * 1994-12-22 1998-10-20 Bell Atlantic Network Services, Inc. Network arrangement for development delivery and presentation of multimedia applications using timelines to integrate multimedia objects and program objects
US5659793A (en) * 1994-12-22 1997-08-19 Bell Atlantic Video Services, Inc. Authoring tools for multimedia application development and network delivery
CA2202688C (en) * 1997-04-15 2005-07-26 Microsoft Corporation Method and system for representing hierarchical time-based data structures and to extract information therefrom

Also Published As

Publication number Publication date
CA2202688A1 (en) 1998-10-15
US6023703A (en) 2000-02-08
US5930797A (en) 1999-07-27

Similar Documents

Publication Publication Date Title
CA2202688C (en) Method and system for representing hierarchical time-based data structures and to extract information therefrom
US6243706B1 (en) System and method for managing the creation and production of computer generated works
JP2993528B2 (en) Text management and restoration method
EP0339901B1 (en) Improved version management tool
JPH02299064A (en) Deletion of marked point
US20040117730A1 (en) Non linear editing system and method of constructing an edit therein
EP0592914A2 (en) Multimedia complex form creation, display and editing method apparatus
KR20020025762A (en) Structure edit apparatus, video structuer edit apparatus, contents structure management method of object, contents structure display method, contents management method, and a computer-readable storage medium
EP0768624A3 (en) Roll-back during regeneration in a computer-aided design system
JP3028738B2 (en) Document common logical information editing device
JPH0863342A (en) Program management method and device therefor
US6664966B1 (en) Non linear editing system and method of constructing an edit therein
WO2001060060A1 (en) Control of sequence of video modifying operations
US20020199204A1 (en) Generation of a description in a markup language of a structure of a multimedia content
CN111760293A (en) Dynamic resource loading method and device for time axis tool and electronic equipment
JPH06274328A (en) Execution method for program consisting of plural processing modules
JPH0477960A (en) Control system for slip generation system
JPH01162982A (en) System for generating circuit library
JPH0778121A (en) Method for managing data in distributing processing system
JPH04111053A (en) Document processor
JPH09152965A (en) Method for rebuilding class library
JPH10177404A (en) System for generating production parts diagram
JPH11232271A (en) Document managing device and storage medium
JPH02206852A (en) Updating processing system for hierarchical structure data
JPS58223858A (en) Integrated backup restorage processing system

Legal Events

Date Code Title Description
EEER Examination request
MKLA Lapsed