WO2009154704A1

WO2009154704A1 - Methods and apparatus for splitting and combining scalable video coding transport streams

Info

Publication number: WO2009154704A1
Application number: PCT/US2009/003487
Authority: WO
Inventors: John Qiang Li; Xiuping Lu; Shemimon Manalikudy Anthru
Original assignee: Thomson Licensing
Priority date: 2008-06-17
Filing date: 2009-06-10
Publication date: 2009-12-23

Abstract

In the transport of Scalable Video Coding (SVC) video, base and enhancement layers are split into individual transport streams which can be used to provide differing levels of resolution for different users. A simple device can use only the base layer, whereas devices with greater resources can have the base layer stream and one or more of the enhancement layer streams combined to provide greater resolution. By thus splitting an SVC stream into different transport streams, differentiated quality of service and network traffic engineering is facilitated. A system incorporating SVC splitting and combining facilities as described can allow users with different capabilities to share the same encoding and transport infrastructure while receiving differentiated service and quality.

Description

METHODS AND APPARATUS FOR SPLITTING AND COMBINING SCALABLE VIDEO CODING TRANSPORT STREAMS

Related Patent Applications [0001] This application claims the benefit under 35 U.S.C. § 119(e) of United States Provisional Application No. 61/132,325, filed June 17, 2008, the entire contents and file wrapper of which are hereby incorporated by reference for all purposes into this application.

Field of Invention

[0002] The present invention generally relates to the communication of video data, particularly to the transport and reception of scalable video coding video data.

Background [0003] Scalable video coding (SVC) has many advantages over classical advanced video coding (AVC). An SVC stream usually consists of one base layer and one or more enhancement layers. The base layer can be independently decoded but an enhancement layer can only be decoded with the base layer and any other lower-level enhancement layers. [0004] In the transmission of SVC video using a multicast internet protocol (IP) network infrastructure, most video encoders will output an SVC stream (i.e., base and enhancement layers) over a single transport stream which is multicast to multiple end- users. This makes it difficult for a network operator to provide differentiated quality of service (QoS) for different SVC layers. For example, because the base layer is crucial to decoding SVC video, it should be provided with greater protection against transmission error than the enhancement layers. By transporting all layers over a single stream, the same level of protection has to be provided to all layers, regardless of their relative importance. Moreover, some end-user devices with limited resources, such as mobile devices, may only be able to decode and display lower-resolution video, in which case they would have no need for any enhancement layers received in an SVC stream. Nonetheless, such devices would still need to receive and parse through all of the layers in an SVC stream in order to identify the base layer packets, while discarding any enhancement layer packets. Such additional processing can tax the limited resources of such devices or require of such devices greater resources than would otherwise be needed.

Summary

[0005] In an exemplary embodiment of the invention, a transport entity splitter arranged between a Scalable Video Coding (SVC) video source and a distribution network converts a single SVC video transport stream over a single Real-time Transport Protocol (RTP) session into multiple transport streams over multiple RTP sessions. By assigning each layer to its own RTP session, the splitter allows different layers to be transported to different users with different capabilities. Additionally, differentiated quality of service (QoS) can be applied to different layers of SVC video streams.

[0006] In an exemplary embodiment of the invention, a transport entity combiner arranged between the network and an end-user device, or embodied within the end-user device, can convert multiple video transport streams over multiple RTP sessions to a single transport stream over a single RTP session. By thus combining various layers of an SVC stream, such a combiner allows end-user devices to work with flexible combinations of video transport streams according to a variety of factors, such as, for example, the devices' capabilities and/or the end-user's preferences. [0007] In view of the above, and as will be apparent from reading the detailed description, other embodiments and features are also possible and fall within the principles of the invention.

Brief Description of the Figures [0008] Some embodiments of apparatus and/or methods in accordance with embodiments of the present invention are now described, by way of example only, and with reference to the accompanying figures in which:

[0009] FIG. 1 illustrates an exemplary arrangement for splitting and combining an SVC stream for users with different resolution requirements. [0010] FIG. 2 illustrates an exemplary arrangement for splitting a single SVC stream into a base layer stream and a separate enhancement layer stream. [0011] FIG. 3 illustrates an exemplary arrangement for combining a base layer stream and an enhancement layer stream into a single SVC stream.

[0012] FIGs. 4 A and 4B illustrate scenarios in which enhancement and base layer streams to be combined experience packet loss. [0013] FIG. 5 is a block diagram of an exemplary embodiment of a one-to-two splitter. [0014] FIG. 6 is a block diagram of an exemplary embodiment of a two-to-one combiner. [0015] FIG. 7 is a block diagram of an exemplary embodiment of an SVC receiver device comprising a combiner.

Description of Embodiments

[0016] Other than the inventive concept, the elements shown in the figures are well known and will not be described in detail. For example, other than the inventive concept, familiarity with television broadcasting, receivers and video encoding is assumed and is not described in detail herein. For example, other than the inventive concept, familiarity with current and proposed recommendations for TV standards such as NTSC (National Television Systems Committee), PAL (Phase Alternation Lines), SECAM (SEquential Couleur Avec Memoire) and ATSC (Advanced Television Systems Committee) (ATSC), Chinese Digital Television System (GB) 20600-2006 and DVB-H is assumed. Likewise, other than the inventive concept, other transmission concepts such as eight-level vestigial sideband (8-VSB), Quadrature Amplitude Modulation (QAM), and receiver components such as a radio-frequency (RF) front-end (such as a low noise block, tuners, down converters, etc.), demodulators, correlators, leak integrators and squarers is assumed. Further, other than the inventive concept, familiarity with protocols such as Internet Protocol (IP), Real-time Transport Protocol (RTP), RTP Control Protocol (RTCP), User Datagram Protocol (UDP), is assumed and not described herein. Similarly, other than the inventive concept, formatting and encoding methods (such as Moving Picture Expert Group (MPEG)-2 Systems Standard (ISO/IEC 13818-1)) for generating transport bit streams are well-known and not described herein. It should also be noted that the inventive concept may be implemented using conventional programming techniques, which, as such, will not be described herein. Finally, like-numbers on the figures represent similar elements. [0017] FIG. 1 is a block diagram of an illustrative arrangement 100 in which an SVC encoder 1 10 generates an SVC video stream that can be decoded with high-definition (HD), standard-definition (SD), or common intermediate format (CIF) resolution. A splitter 120 splits the SVC stream into three separate streams 10-30, with stream 10 containing the base layer, stream 20 containing a first enhancement layer, and stream 30 containing a second enhancement layer of the SVC stream. A CIF decoder 125, as may be found in a mobile device, for example, can decode the base layer stream 10 with CIF resolution. For displaying the same picture with SD resolution, a combiner 130 combines the base layer stream 10 and enhancement layer stream 20 to generate the necessary stream for an SD decoder 135. For HD resolution, a combiner 140 combines the base layer stream 10 and the two enhancement layer streams 20 and 30 for provision to an HD decoder 145.

[0018] As can be appreciated, unless the original SVC stream is split, such as by the splitter 120, a device with only CIF capability, for example, such as a mobile device, would be required to handle the heavy traffic burden of all three SVC layers, even though it can only use the base layer packets and has to discard the enhancement layer packets. Moreover, unless the original SVC stream is split, any protection mechanism has to be applied to all three layers, which causes unnecessary overhead in the use of bandwidth. [0019] Exemplary splitting and combining arrangements will now be described with reference to FIGs. 2 and 3.

[0020] FIG. 2 shows a schematic representation of an exemplary one-to-two splitter 200. Its input is a single multicast SVC IP stream 210 which includes the base layer and one enhancement layer. The SVC IP stream 210 is associated with a first multicast IP address, for example 224.1.1.1. In the SVC stream 210, packets belonging to different layers are encapsulated in separate IP packets and are interleaved in sequence. In FIG. 2, Bl is the first packet of the base layer and El is the first packet of the enhancement layer, B2 is the second packet of the base layer and E2 is the second packet of the enhancement layer. [0021] In the exemplary embodiment described, the transport protocol is RTP, in which case each packet includes an RTP header with a sequence number and timestamp, among other information. To allow detection of packet loss, consecutive packets in the incoming SVC stream 210 should have consecutive RTP sequence numbers incremented by one, as is required by RTP. Illustrative sequence numbers are shown in FIG. 2 in the upper part of each packet. Moreover, the timestamps in the RTP headers of base and enhancement layer packets that refer to the same frame of video should be the same. [0022] Upon receiving the incoming SVC stream 210, the exemplary splitter 200 acts as a network proxy to perform the following functions.

[0023] First, the splitter 200 assigns a new multicast IP address to each layer. In the example illustrated in FIG. 2, the base layer is assigned address 224.1.1.2 and the enhancement layer is assigned address 224.1.1.3. In other words, the destination IP address in the IP headers of the base layer packets is changed to 224.1.1.2 and the destination IP address in the IP headers of the enhancement layer packets is changed to 224.1.1.3.

[0024] Second, the splitter 200 assigns a new RTP sequence number to each packet so that in the output streams 21 1, 212, the sequence numbers of consecutive packets are still consecutive and incremented by one.

[0025] An exemplary splitter in accordance with the invention can be implemented, for example, with software, hardware or a combination of both. The splitter may be a standalone entity or it may be incorporated, for example, into an encoder or a network router, among other possibilities. Furthermore, although a one-to-two splitter is described and shown, as can be appreciated, the principles of the present invention can also be applied to implement a one-to-N splitter, where N > 2.

[0026] FIG. 5 is a block diagram of an exemplary embodiment of a one-to-two splitter 500. An IP/UDP parser 510 decodes the IP/UDP packet headers of the incoming SVC IP stream and passes the packets to a RTP data handler 520. The RTP data handler 520 further examines each packet to determine its type; i.e., to which layer it belongs. If the packet is a base layer packet, the RTP data handler 520 passes the packet to RTP/IP/UDP encapsulation module 530. If the packet is an enhancement layer packet, the RTP data handler 520 passes the packet to RTP/IP/UDP encapsulation module 540. The encapsulation modules 530 and 540 generate a new base layer IP stream and a new enhancement layer IP stream, respectively, with fields in the RTP, IP and/or UDP headers modified as described above. [0027] FIG. 3 shows a schematic representation of a two-to-one combiner 300. Two IP streams, one stream 31 1 carrying an SVC base layer and the other stream 312 carrying an enhancement layer are provided as inputs to the combiner 300. If RTP is used, the combiner 300 relies on the RTP sequence numbers of the packets in each incoming stream to detect any packet losses. The combiner 300 uses the timestamps in the RTP headers to synchronize the incoming SVC base and enhancement layer streams 31 1, 312 and to combine them into a single RTP output stream 320, as shown in FIG. 3. The combined stream can be assigned a new IP address, and/or a new UDP port. [0028] In an exemplary method of operation, the combiner 300 allocates separate buffers for the incoming SVC base and enhancement layer streams 31 1, 312. After a certain buffering period, the combiner starts to combine the streams by sorting through all available packets in the incoming buffers based on the timestamp information in their RTP headers. Packets with the same timestamp are considered to be synchronized and placed into a single outgoing buffer. The packets in the outgoing buffer are ordered so that they can be decoded by an SVC decoder. The combiner 300 reconstructs the RTP header information for the outgoing packets. The timestamp values for the outgoing packets and the initialization and incrementing of the RTP sequence numbers for the outgoing packets are preferably compliant with existing standards. The packets are then sent out from the outgoing buffer as output stream 320. [0029] If a packet loss in either input stream 31 1, 312 is detected by the combiner 300, the output SVC stream 320 is generated with a gap in the RTP sequence numbers for each packet detected as lost. This allows downstream network elements (e.g., a router) to detect the packet loss based on the discontinuities in the RTP sequence numbers of the output stream 320. FIGs. 4A and 4B illustrate scenarios in which the incoming enhancement and base layer streams, respectively, are missing packets. Note that in both cases, the combiner generates an uninterrupted output stream 320 with gaps in the RTP sequence numbers indicative of the lost packets. Thus, for example, as shown in FIG. 4A, the output stream 320 is missing RTP sequence numbers 4 and 6 to indicate the loss of packets E2 and E3 from the input enhancement layer stream 312. Similarly, in FIG. 4B, the output stream 320 is missing RTP sequence numbers 3 and 5 to indicate the loss of packets B2 and B3 from the input base layer stream 31 1. [0030] In addition to packet losses, if any packets in the incoming base or enhancement layer streams 31 1, 312 are not available within a certain period (e.g., 10ms) because of network jitter, or the like, such packets are skipped and the rest of the packets are forwarded downstream. This time-bounded forwarding prevents an interruption in the combined output stream 320 because of a slower incoming stream.

[0031] An exemplary combiner in accordance with the invention can be implemented, for example, with software, hardware or a combination of both. Furthermore, although a tow- to-one combiner is described and shown, as can be appreciated, the principles of the present invention can also be applied to implement a N-to-one combiner, where N > 2. [0032] FIG. 6 is a block diagram of an exemplary embodiment of a two-to-one combiner 600. An IP/UDP parser 601 decodes the IP/UDP packet headers of the incoming base layer IP stream, whereas IPAJDP parser 602 decodes the IPAJDP packet headers of the incoming enhancement layer IP stream. A buffer 610 buffers the incoming base layer IP stream and a buffer 620 buffers the incoming enhancement layer IP stream. RTP parsers 630 and 640 decode the RTP header information of the base and enhancement layer packets, respectively.

[0033] After a certain buffering period, a combination module 650 starts to combine the buffered streams by sorting through all available packets in the incoming buffers based on the timestamp information in their RTP headers. Packets with the same timestamp are considered to be synchronized and placed into an outgoing buffer 660. The packets in the outgoing buffer 660 are ordered so that they can be decoded by an SVC decoder. The contents of the outgoing buffer 660 are provided to an encapsulation module 670 which generates a new SVC IP stream with modified fields in the RTP, IP, and/or UDP headers, as described above. The encapsulation module 670 may be omitted, for example, if the combined stream is provided directly to an SVC decoder, as described below.

[0034] An exemplary combiner in accordance with the invention can be implemented, for example, as a stand-alone network proxy or embedded within a receiver device, such as an SVC-capable set-top box (STB). FIG. 7 is a block diagram of an exemplary embodiment of a receiver device 700 comprising a combiner 710 and an SVC decoder 720. The combiner 710 can be implemented as described above, to combine an incoming base layer IP stream and an incoming enhancement layer IP stream into a combined SVC stream for provision to the SVC decoder 720. The SVC decoder 720, in turn, generates a decoded picture signal for provision to a display device (not shown). If implemented as part of an SVC receiver, as shown in FIG. 7, the reconstruction of the RTP header information for the combined outgoing SVC stream may be skipped and the combined SVC stream may be directly fed to the decoder 720.

[0035] In view of the above, the foregoing merely illustrates the principles of the invention and it will thus be appreciated that those skilled in the art will be able to devise numerous alternative arrangements which, although not explicitly described herein, embody the principles of the invention and are within its spirit and scope. For example, the inventive concept may be implemented in a stored-program-controlled processor, e.g., a digital signal processor, which executes associated software for carrying out a method in accordance with the principles of the invention. Further, the principles of the invention are applicable to different types of communications systems, e.g., satellite, Wireless- Fidelity (Wi-Fi), cellular, etc. Indeed, the inventive principals are applicable to stationary as well as mobile receivers. It is therefore to be understood that numerous modifications may be made to the illustrative embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention.

Claims

1. A method of splitting a scalable video coding (SVC) video stream into streams of layers of SVC video data, comprising: receiving an input SVC video stream containing base layer packets and enhancement layer packets, the input SVC video stream being associated with an input Real-time Transport Protocol (RTP) session; generating a first output stream containing the base layer packets, the first output stream being associated with a first output RTP session; and generating a second output stream containing the enhancement layer packets, the second output stream being associated with a second output RTP session.

2. The method of claim 1 , wherein: the input SVC video stream is associated with an input internet protocol (IP) address; the first output stream is associated with a first output IP address; and the second output stream is associated with a second output IP address.

3. The method of claim 1, comprising: assigning consecutive RTP sequence numbers to the base layer packets of the first output stream; and assigning consecutive RTP sequence numbers to the enhancement layer packets of the second output stream.

4. The method of claim 1, comprising: generating a further output stream containing packets of a further enhancement layer, the further output stream being associated with a further output RTP session.

5. A method of combining streams of layers of scalable video coding (SVC) video data into a SVC video stream, comprising: receiving a first input stream containing base layer packets, the first input stream being associated with a first input Real-time Transport Protocol (RTP) session; receiving a second input stream containing enhancement layer packets, the second input stream being associated with a second input RTP session; and generating an output SVC video stream containing the base layer packets and enhancement layer packets, the output SVC video stream being associated with an output RTP session.

6. The method of claim 5, wherein: the first input stream is associated with a first input internet protocol (IP) address; the second input stream is associated with a second input IP address; and the output SVC stream is associated with an output IP address.

7. The method of claim 5, comprising: checking the first input stream for packet loss; and checking the second input stream for packet loss.

8. The method of claim 5, wherein generating the output SVC video stream includes: input buffering the base layer packets of the first input stream; input buffering the enhancement layer packets of the second input stream; matching the buffered base layer packets and enhancement layer packets based on timestamps contained in each packet; and output buffering the matched base layer and enhancement layer packets, the matched base layer and enhancement layer packets being ordered so as to allow decoding.

9. The method of claim 5, comprising: assigning consecutive RTP sequence numbers to consecutive packets of the output SVC video stream.

10. The method of claim 5, comprising: reconstructing RTP headers for the packets of the output SVC video stream.

1 1. The method of claim 7, wherein if a packet loss is detected, assigning RTP sequence numbers to packets of the output SVC video stream with a gap indicating the packet loss.

12. The method of claim 7, wherein RTP sequence numbers of the packets of the first and second input streams are used to check for packet loss.

13. The method of claim 5, comprising: receiving a further input stream containing packets of a further enhancement layer, the further input stream being associated with a further input RTP session, wherein the output SVC video stream contains packets of the base layer, enhancement layer, and further enhancement layer.

14. Apparatus for splitting a scalable video coding (SVC) video stream into streams of layers of SVC video data, comprising: a parser, the parser decoding packet headers of an input SVC video stream containing base layer packets and enhancement layer packets, the input SVC video stream being associated with an input Real-time Transport Protocol (RTP) session; a RTP data handler, the RTP data handler determining whether each packet in the

SVC video stream is a base layer packet or an enhancement layer packet; a first encapsulation module, the first encapsulation module generating a first output stream containing the base layer packets, the first output stream being associated with a first output RTP session; and a second encapsulation module, the second encapsulation module generating a second output stream containing the enhancement layer packets, the second output stream being associated with a second output RTP session.

15. The apparatus of claim 14, wherein: the input SVC video stream is associated with an input internet protocol (IP) address; the first output stream is associated with a first output IP address; and the second output stream is associated with a second output IP address.

16. The apparatus of claim 14, wherein: the first encapsulation module assigns consecutive RTP sequence numbers to the base layer packets of the first output stream; and the second encapsulation module assigns consecutive RTP sequence numbers to the enhancement layer packets of the second output stream.

17. The apparatus of claim 14, comprising: a third encapsulation module, the third encapsulation module generating a further output stream containing packets of a further enhancement layer, the further output stream being associated with a further output RTP session.

18. Apparatus for combining streams of layers of scalable video coding (SVC) video data into a SVC video stream, comprising: a first input buffer, the first input buffer receiving a first stream containing base layer packets, the first input stream being associated with a first input Real-time Transport Protocol (RTP) session; a second input buffer, the second input buffer receiving a second input stream containing enhancement layer packets, the second input stream being associated with a second input RTP session; and a combination module, the combination module generating an output SVC video stream containing the base layer packets and enhancement layer packets, the output SVC video stream being associated with an output RTP session.

19. The apparatus of claim 18, wherein: the first input stream is associated with a first input internet protocol (IP) address; the second input stream is associated with a second input IP address; and the output SVC stream is associated with an output IP address.

20. The apparatus of claim 18, wherein the combination module matches the buffered base layer packets and enhancement layer packets based on RTP timestamps contained in each packet.

21. The apparatus of claim 20, comprising: an output buffer, the output buffer buffering the matched base layer and enhancement layer packets in an order allowing decoding.

22. The apparatus of claim 18, comprising: an encapsulation module, the encapsulation module assigning consecutive RTP sequence numbers to consecutive packets of the output SVC video stream.

23. The apparatus of claim 22, wherein the encapsulation module assigns RTP sequence numbers to packets of the output SVC video stream with a gap to indicate a packet loss if a packet loss is detected in at least one of the first and second input streams.

24. The apparatus of claim 18, comprising: a further input buffer, the further input buffer receiving a further input stream containing packets of a further enhancement layer, the further input stream being associated with a further input RTP session, wherein the combiner generates an output SVC video stream containing packets of the base layer, enhancement layer, and further enhancement layer.

25. Apparatus for receiving and decoding streams of layers of SVC video data, comprising the apparatus of claim 18 and an SVC decoder.