US8462629B2 - Cooperative operation of network transport and network quality of service modules - Google Patents
Cooperative operation of network transport and network quality of service modules Download PDFInfo
- Publication number
- US8462629B2 US8462629B2 US11/762,688 US76268807A US8462629B2 US 8462629 B2 US8462629 B2 US 8462629B2 US 76268807 A US76268807 A US 76268807A US 8462629 B2 US8462629 B2 US 8462629B2
- Authority
- US
- United States
- Prior art keywords
- packet
- transport module
- packets
- notification
- packet scheduler
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/12—Avoiding congestion; Recovering from congestion
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/18—End to end
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/19—Flow control; Congestion control at layers above the network layer
- H04L47/193—Flow control; Congestion control at layers above the network layer at the transport layer, e.g. TCP related
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/26—Flow control; Congestion control using explicit feedback to the source, e.g. choke packets
- H04L47/263—Rate modification at the source after receiving feedback
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/28—Flow control; Congestion control in relation to timing considerations
- H04L47/283—Flow control; Congestion control in relation to timing considerations in response to processing delays, e.g. caused by jitter or round trip time [RTT]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/30—Flow control; Congestion control in combination with information about buffer occupancy at either end or at transit nodes
Definitions
- the present invention relates to the area of improving the performance of transport protocols like the Transmission Control Protocol (TCP) in an Internet Protocol (IP) network.
- TCP Transmission Control Protocol
- IP Internet Protocol
- TCP When packets are dropped because of congestion, the transport protocol running in the end system affected by such dropped packets typically reacts by adjusting its sending rate downward.
- TCP follows the well-known additive-increase, multiplicative-decrease (AIMD) congestion control scheme. Roughly speaking, TCP increases its transmission window additively on each round-trip in which no packets are dropped, and decreases its window multiplicatively when packet loss is detected.
- AIMD additive-increase, multiplicative-decrease
- TCP's throughput in the best case is limited by the window size W divided by the roundtrip time RTT or (W/RTT).
- W/RTT roundtrip time
- TCP's throughput is also limited by the way in which the window opens and closes in the presence of packet loss.
- the throughput is proportional to 1/sqrt(p) where p is the rate of packet loss. Accordingly, the overall throughput of TCP is proportional to 1/(RTT*sqrt(p)). This means that increases in network latency and increases in packet loss rates each have a negative effect on TCP throughput, but also that the two problems in combination aggravate each other.
- TCP is widely used because TCP achieves a reliable ordered byte-stream service implemented by the end hosts over an underlying IP network that actually supports only best-effort (unreliable) delivery of datagrams with possible reordering en route.
- TCP is also “network friendly” to other TCP and non-TCP traffic that is sharing the same network link.
- a common test for a proposed transport protocol is whether it is similarly “TCP friendly” or “network friendly.”
- TCP friendly is similarly “TCP friendly” or “network friendly.”
- TCP's behavior is undesirable.
- ordinary TCP cannot distinguish between packet loss due to congestion and packet loss due to corruption.
- packet losses are caused by noise corrupting packets, it is undesirable for the sender to back off.
- a variety of systems have been proposed and/or implemented to improve TCP behavior in such circumstances.
- One worth noting involves the introduction of a snoop agent that both caches TCP packets (to avoid the need to resend them) and suppresses some TCP responses (to avoid shrinking the window inappropriately).
- TCP congestion-control mechanisms of TCP are powerful but far from perfect, and there are circumstances in which it is important to work in a better way than the ordinary behavior of TCP.
- Embodiments of the invention provide efficient and flexible networking quality of service as well as transport protocol design.
- a hybrid transport/network quality of service (HNTQ) scheme improves the performance of TCP over specific links or network paths that are subject to high latency, a high bandwidth-delay product, high packet loss, and/or bit errors.
- a callback mechanism is used between a packet scheduler and a transport module to control the transmission rate of packets across one or more connections or links.
- a method of communicating packets over a network is provided.
- a packet scheduler receives at least one packet from a transport module.
- the packet scheduler then services the received packet.
- the packet scheduler provides a notification to the transport module.
- the notification is provided to the transport module upon completion of the servicing of the received packet.
- the transport module receives data for at least one additional packet and buffers the data in a buffer. In response to receiving the notification from the packet scheduler, one or more additional packets are communicated from the transport module to the packet scheduler. In one aspect, a number of additional packets to communicate to the packet scheduler in response to the notification is included in the notification. In another aspect, the transport module determines whether to communicate the one or more additional packets for a particular connection based on that connection's flow control window and not based on a congestion window. In yet another aspect, the transport module determines a network connection associated with the received packet based on the notification. One or more additional packets corresponding to the associated network connection are selected to communicate to the packet scheduler.
- a buffer size of the transport module is changed based on the data in the buffer and the callbacks received from the packet scheduler.
- a size of the buffer is decreased when an amount of additional packets, which have been in a buffer of the transport module for longer than a round trip time, is larger than a threshold.
- a size of the buffer is increased when the buffer is full and there are no additional packets available for a particular network connection associated with a notification.
- the received packet includes an indicator adapted to facilitate a notification to the transport module from the packet scheduler upon servicing the received packet.
- the indicator identifies a procedure of the transport module. The procedure may be adapted to be invoked by the packet scheduler using a local procedure invocation. In another aspect, the indicator identifies a remote location of the transport module.
- the transport module and the packet scheduler are operated within a same device.
- the device is a network proxy.
- the transport module and the packet scheduler are operated within different devices connected via a network.
- a network system that communicates packets.
- the network system includes a transport module and a packet scheduler, which has an input for receiving a packet from the transport module.
- the packet scheduler includes logic for servicing the received packet and logic for providing a notification to the transport module in response to servicing the received packet.
- the transport module includes a first input for receiving the notification from the packet scheduler, a second input for receiving data for at least one additional packet, a buffer for buffering the data, and logic for communicating one or more additional packets to the packet scheduler in response to receiving the notification from the packet scheduler.
- a proxy is situated on a far end of a managed network path from the transport module and advertises a flow control window that is controlled via a configuration of said proxy or via automatic actions performed in said proxy. Logic may determine whether to communicate the one or more additional packets for a particular connection based on that connection's flow control window and not based on a congestion window.
- the transport module further includes logic for identifying a connection of the transport module as HTNQ-enabled and logic for including an indicator in a packet to be communicated to the packet scheduler through an HTNQ-enabled connection.
- the indicator identifies a packet as requiring a callback mechanism from the packet scheduler.
- the packet scheduler may further include logic for determining whether the received packet includes an indicator specifying an association with an HTNQ-enabled connection and logic for bypassing the step of providing the notification when the packet does not include the indicator.
- a number of additional packets to communicate to the packet scheduler in response to the notification is configurable by a network administrator or hard coded as a default value.
- the packet scheduler further includes a queue adapted to have N buffer sections, where N equals a total number of active connections of the transport module.
- a packet scheduler has an input for receiving a packet from a transport module, logic for servicing the received packet, and logic for providing a notification to the transport module in response to servicing the received packet.
- logic identifies an indicator in the received packet that is used to facilitate the providing of the notification to the transport module.
- the packet scheduler includes a queue and logic for dropping a non-HTNQ packet from the queue when a number of non HTNQ packets exceed a non-HTNQ queue limit.
- logic drops an HTNQ packet from the queue when a number of HTNQ packets exceed an HTNQ queue limit and provides an exception condition indicating a number of supportable connections has been exceeded when an HTNQ packet is dropped from the queue.
- a transport module has a first input for receiving, from a packet scheduler, a notification that is responsive to at least one packet sent from the transport module; a second input for receiving data for at least one additional packet; a buffer for buffering the data; and logic for communicating one or more additional packets to the packet scheduler in response to receiving the notification from the packet scheduler.
- the transport module has a plurality of connections. For each connection, there may be a bit stating whether the connection is active. In one aspect, logic sets a bit to indicate that a connection is active when data is received for that connection. In another aspect, logic sets a bit to indicate that a connection is idle when a notification is received from the packet scheduler and when no packets need to be transmitted for that connection. In another embodiment, the transport module includes a plurality of classes that are grouped in a hierarchical link-sharing organization, and wherein at least one group of the classes are HTNQ enabled and QoS controlled and another group of the classes are congestion-controlled.
- FIG. 1A shows a graph illustrating the transmission rate of a network using the AIMD method in TCP vs. time.
- FIG. 1B shows a system illustrating a communication model between two proxies of the network that may be improved by an embodiment of the present invention.
- FIG. 2 is a flow diagram illustrating a method for transmitting packets via a network connection according to an embodiment of the present invention.
- FIGS. 3A and 3B illustrate a method for sending packets to a queue from a transport protocol according to an embodiment of the present invention.
- FIG. 4 is a flow diagram illustrating a method for transitioning the status of a connection, e.g. from an idle state to an active state, according to an embodiment of the present invention.
- FIG. 5 shows a network system implementing a transport protocol connection according to an embodiment of the present invention.
- FIG. 6 illustrates a hierarchical link-sharing algorithm according to an embodiment of the present invention.
- FIG. 7 illustrates a method and system for performing network-based HTNQ according to an embodiment of the present invention.
- FIG. 8 illustrates a method for reducing overhead in a network-based HTNQ by processing multiple packets according to an embodiment of the present invention.
- FIG. 9A is a flow diagram illustrating a method for increasing the size of a socket buffer according to an embodiment of the present invention.
- FIG. 9B is a flow diagram illustrating a method for decreasing the size of a socket buffer according to an embodiment of the present invention.
- Embodiments of the invention provide efficient and flexible networking quality of service as well as transport protocol design.
- a hybrid transport/network quality of service (HNTQ) scheme improves the performance of TCP over specific links or network paths that are subject to high latency, a high bandwidth-delay product, high packet loss, and/or bit errors.
- a callback mechanism is used between a packet scheduler and a transport module to control the transmission rate of packets across one or more connections or links.
- routers and switches are devices that operate at the link and network layers and are distinct entities from transport end points that are implemented in separate systems called “hosts” in the TCP/IP nomenclature. There is no explicit communication between the hosts and the network. Hosts simply transmit packets and switches and routers attempt to forward said packets toward their destination. A host otherwise has no control over the router.
- Traffic classification entails assigning each packet to a “class”, which is typically specified by a network operator. For example, a class might be “voice traffic”, or “file server traffic”, or “Web traffic between the New York and Orlando offices”, etc. Typically, each class will be assigned to a particular queue. Note that more than one class may be assigned to the same queue, causing traffic from those multiple classes to be treated as a single aggregate. When different flows or collections of application sessions are aggregated in this fashion, the resulting scheme is often called Class of Service (CoS) resource management rather than QoS to emphasize the notion that network traffic is managed in a coarser grained fashion.
- CoS Class of Service
- Queue management entails how a queue is maintained as packets are inserted and removed from the queue and which packets are dropped when the queue becomes full (or begins to become full) in the event of congestion.
- a first-in, first-out (FIFO) queue with a drop-tail drop policy is a very simple exampling of a queue management scheme. More elaborate schemes like Random Early Detection (RED), Weighted Random Early Detection (WRED), Fair Queueing (FQ), Weighted Fair Queueing (WFQ), deficit round-robin (DRR), etc. have been developed.
- a networking device will manage multiple queues for each output port. Packets will be placed in the different queues according to policy that is controlled by traffic classification.
- a scheduling algorithm determines how and what queues are serviced each time a packet is ready to be transmitted over the output port.
- One of the simpler scheduling algorithms is a static-priority scheduler.
- each queue is assigned a priority and at each service time, the non-empty queue with the highest priority is chosen to be serviced.
- WFQ can be realized as a queue management scheme
- the WFQ algorithm can also be deployed as a scheduler. For example, a collection of FIFO queues might be serviced according to a WFQ schedule. Or, a collection of RED queues might be serviced according to a DRR schedule. Or, a collection of WFQ queues might be serviced according to a WFQ scheduler. This latter approach is sometimes called hierarchical packet fair queuing (H-PFQ).
- H-PFQ hierarchical packet fair queuing
- FIG. 1A shows a graph 100 illustrating the transmission rate 105 of a network using the AIMD method in TCP vs. time 110 ; and FIG. 1B shows a system 120 illustrating a communication model between two proxies of the network that may be improved by an embodiment of the present invention.
- a transport protocol 130 running on a host 125 which may be a proxy, increases the sending rate of a connection through port 135 .
- An output port in a data network is not limited in scope to a physical layer device like a T-1 interface card or a SONET interface. More generally, a port can be a transmission process that sends packets according to a bandwidth shaping rule, where the bandwidth may be fixed or may vary with time. For example, an output port may correspond to a virtual private network (VPN) tunnel where network traffic is groomed to a specified transmission rate over that tunnel.
- VPN virtual private network
- the port may corresponding to a rate-limited transmission of network traffic over a higher-capacity physical interface, e.g., a transmission process that sends packets at 1.5 Mb/s over a 1 Gb/s ethernet connection coupled to a router that is in turn coupled to a T1 connection.
- the network traffic is groomed to 1.5 Mb/s in a device that has a 1 Gb/s network interface so that those packets in turn can be transmitted smoothly over a slower speed link.
- link rate may refer interchangeably herein to either a physical interface rate or to rate of bandwidth shaping rule of a virtual port.
- TCP 130 When TCP 130 has data ready to send for a given connection (as allowed by the congestion and flow control windows), it immediately transmits packets 133 by passing them down the network stack to a network-layer device 145 , which processes those packets by a packet scheduler 140 . At minimum, the packets are queued on a device interface and transmitted over the interface's port 135 at the rate determined by the physical port, and in such cases, the device acts as a simple FIFO packet scheduler. In more complicated scenarios, one of a number of different scheduling algorithms may be configured in the system to manage a particular network interface.
- TCP regulates the transmission of packets 133 according to both its flow control window and its congestion window. If the next packet to send is within the range specified by both said windows, then TCP immediately transmits the packet. In effect, TCP can burst up to a window's worth of packets into the network and the queuing mechanisms within the network and within the local network stack absorb such bursts.
- a switch or router or even the local network stack of the host experiences congestion in the form of a queue that has grown to capacity, or more generally, in the form of a queue management algorithm that has detected incipient congestion.
- a packet scheduler 140 in a network-layer device 145 signals feedback to transport protocol 130 indicating that congestion has occurred. Such feedback is either implicit in that the congested packet scheduler 140 drops a packet and the end host 150 infers congestion by detecting the missing packet through coordination between the transport sender and receiver, or the feedback is explicit as in explicit congestion notification (ECN).
- ECN explicit congestion notification
- TCP contains logic to retransmit old packets that it believes have been lost in the network.
- TCP contains logic to retransmit old packets that it believes have been lost in the network.
- transport protocol 130 typically reacts by reducing the transmission rate of the corresponding connection, as in AIMD. Upon reducing the sending rate, the congestion is alleviated, and the queue dissipates. The sender begins increasing its rate again as in region 108 , and the process repeats.
- a transport protocol in a host implementing AIMD would increase its rate until the network becomes congested.
- the network bottleneck is packet scheduler 140 located within the host proxy 125 itself, e.g., because the proxy is additionally implementing QoS mechanisms to control the proxied traffic according to administrative policies.
- packet scheduler 140 would drop a packet or mark the packet's ECN bit.
- Transport protocol 130 would only detect the congestion after said marked packets or subsequent packets 155 are received at the end host 150 and acknowledgement packets 160 are then received a round-trip time later then indicating said congestion. Because of the modularity of the prior art architecture, the transport protocol's knowledge of the local congestion event is conveyed by a message exchange requiring a round trip, which round-trip is potentially traversing a wide-area network and thus delaying the notification by a significant amount of time.
- TCP used in system 120 is that its throughput tends to decrease as packet loss increases, which will also cause an increase in roundtrip time RTT.
- RTT roundtrip time
- increased RTT may produce longer queues and a higher likelihood of packet loss from queuing, which in turn further lowers throughput and potentially further increases queuing.
- the situation is slightly better.
- the local network stack detects incipient congestion (e.g., as a result of Linux's RED queuing algorithm) or if the local network stack decides to otherwise drop a packet, that information is returned up the protocol stack to the transport protocol.
- the local transport protocol can reduce its transmission rate in response to this locally generated congestion feedback, which is immediate. For example, in this case with respect to TCP, a TCP connection reduces its congestion window in response without having to wait a round-trip time to detect the condition.
- TCP adapts to the local congestion scenario in the same way it adapts to congestion in the network by essentially performing experimentation (by increasing its sending rate), observing the outcome of the experiments (by detecting packet loss), and adjusting the rate downward (by decreasing its congestion window).
- embodiments implement a better approach that tightly integrates the TCP sending process with the local packet scheduler. By timing the packet transmissions in the transport layer explicitly from the packet-scheduling layer, better performance can be achieved.
- a packet scheduler is generically extended with a mechanism to generate a notification when specially marked packets are serviced and a transport module is modified to mark packets as needing such treatment and is extended to act upon the notifications generated by the scheduler.
- Embodiments of the present invention advantageously do not maintain a separation between the transport protocol and a packet scheduler.
- Important examples of such a proxy are transaction accelerators as disclosed in both U.S. patent application Ser. No. 10/285,315, filed 30 Oct. 2002, entitled “Transaction Accelerator for Client-Server Communication Systems,” and in U.S. patent application Ser. No. 10/640,405, filed 12 Aug. 2003, entitled “Transparent Client-Server Transaction Accelerator.” It will be readily apparent to one skilled in the arts that there are also many other systems that use a substantially similar proxy architecture for a variety of purposes.
- An embodiment of the invention improves upon the prior art in both networking quality of service as well as transport protocol design with a hybrid transport/network quality of service (HTNQ) scheme. This embodiment also improves the performance of TCP over specific links or network paths that are subject to high latency, a high bandwidth-delay product, high packet loss, and/or bit errors through the use of HTNQ.
- HTNQ transport/network quality of service
- HTNQ utilizes a tight coupling between the network QoS mechanisms and the transport protocol.
- the transport protocol is modified such that it includes an HTNQ mode of operation called herein HTNQ mode.
- HTNQ mode is described with respect to a typical TCP implementation though it would be obvious to one skilled in the art how to adapt such mechanisms to other transport protocols and consequently this discussion is not intended to limit the scope of the present invention to the TCP protocol.
- the transport layer marks the packets of one or more connections with a unique identification as being admissible to HTNQ, and the network layer's packet scheduler provides feedback to the transport layer indicating the timing for which the transport layer is allowed to send subsequent packets.
- the modified packet scheduler and the modified transport protocol stack are co-located in the same device. In one embodiment, this scheme does not compute a window based on rate and round-trip information, assign a rate to each active TCP connection or otherwise try to implement additional flow control for each TCP connection, or explicitly track the set of active TCP connections and service them according to a constant bit rate.
- FIG. 2 is a flow diagram illustrating a method 200 for transmitting packets via a network connection according to an embodiment of the present invention.
- a transport layer module marks the packets of one or more connections with unique identification as being admissible to HTNQ.
- the TCP module inserts notification information in the buffer representing the packet. In an embodiment, this scheme of inserting and providing notification information could be realized by including a pointer to a function that represents the notification mechanism. In another embodiment, the notification information could include a bit in the packet buffer that indicates that the corresponding packet is an HTNQ packet.
- the transport module sends packets to network scheduler.
- the normal TCP mode bursts multiple packets into the scheduler, potentially generating a large queue, and then receives feedback tell the TCP to slow down.
- an embodiment of HTNQ effectively controls the rate of the transmission of packets on a given connection according to the local service that those packets would otherwise receive. Rather than build up a queue of packets inside the scheduler, HTNQ limits the number of packets per connection at any given time. This means that when the local packet scheduler is the bottleneck, TCP receives immediate and continuous feedback as to when and at what rate it can transmit packets. In one embodiment, the number of packets is limited to at most one packet per connection at any given time.
- the scheduler services a packet, which may contain the notification information inserted by the TCP module.
- the scheduler notifies the TCP module using the notification information in the packet buffer.
- the scheduler performs the notification by calling the notification function indirectly through the above-mentioned pointer.
- the packet scheduler in response to a bit being set, can call a well-known entry point in the TCP module that represents the callback for HTNQ packets. Packets from other protocols could also be marked in a similar manner, with the scheduler calling a different module-specific callback for such other protocols.
- the sending of the notification may be made upon the start, completion, or any other time during the servicing.
- the TCP module transmits the next packet to the scheduler, for example, as part of a generic callback mechanism that can work with any packet scheduler.
- a TCP connection obeys the principle that it transmits at most one packet at any given time, and no more packets until the connection has been notified via an HTNQ callback.
- the HTNQ mode allows for transmitting different numbers of packets in response to a HTNQ callback.
- HTNQ callback An instance of this process whereby the packet scheduler posts a notification callback is herein called an HTNQ callback.
- Packet buffers that include such notification information are called HTNQ packets and packets that do not include such information are called non-HTNQ packets.
- An embodiment of the invention allows HTNQ and non-HTNQ packets to coexist and ensures predictable behavior when these different types of packets are commingled in the same packet scheduler.
- the HTNQ notification model can be implemented in a reusable and modular fashion by performing such functions as a generic queuing discipline.
- queuing disciplines can be configured into different scheduling algorithms and mixed and matched in various fashions.
- an HTNQ queuing discipline might manage the transmit queue of a physical network interface.
- it might be the queuing discipline of a rate-controlled traffic shaper conforming to a token bucket.
- it could be configured as the queuing discipline of each queue that is managed by class-based weighted fair queuing.
- CBQ Floyd and Jacobson's class-based queuing
- HTB Linux's hierarchical token bucket
- HFSC Stoica's hierarchical fair service curve
- the HTNQ callback mechanism and queue limit techniques could be integrated into other queue management schemes like random early detection (RED), weighted fair queuing (WFQ), deficit round-robin (DRR), and so forth.
- RED random early detection
- WFQ weighted fair queuing
- DRR deficit round-robin
- the present invention is effective because the framework is indifferent as to how particular packets are chosen to be scheduled. No matter how complicated or simple the scheduling decision, once a packet is chosen to be serviced the callback mechanism causes TCP to generate the next packet in a way that works generally across any scheduling algorithm.
- FIGS. 3A and 3B illustrate a method 300 for sending packets to a queue from a transport protocol according to an embodiment of the present invention.
- a queuing discipline detects a packet 355 with HTNQ notification information.
- an HTNQ queuing discipline is a FIFO queue. For example, each time a packet is removed from the head of a queue 360 to be transmitted over the underlying physical device, the packet is checked for whether is contains HTNQ notification information.
- an embodiment of the TCP module marks each packet buffer from a connection in HTNQ mode with the notification information that enables the HTNQ callback.
- the queuing discipline sends a callback to the TCP module.
- the packet that is currently being serviced is passed to the TCP module.
- the HTNQ callback includes as an argument the packet that is being serviced.
- the notification information or HTNQ callback might include flow identification information like an address/port tuple (i.e., IP source and destination, and TCP source and destination port). This call back gives TCP the opportunity to send another packet if possible.
- the TCP module 370 determines which packet to send next.
- the TCP module can look up the data structure associated with the corresponding connection 380 and attempt to send the next packet with respect to that connection.
- the HTNQ callback includes a pointer to the actual connection data structure. However, care must be taken in the latter case to avoid race conditions where a data structure is deallocated while a packet happens to still be stored in a packet scheduler queue and the TCP module attempts to deference a pointer to memory that has been freed.
- the TCP transmits the next packet, which may subsequently be detected by the queuing discipline.
- This scheme is herein called callback FIFO, or CBFIFO.
- HTNQ with CBFIFO is self-clocking in the sense that when a TCP connection transmits a first packet, this causes a callback from CBFIFO packet scheduler to TCP, which causes TCP module to transmit a second packet, which in turn causes another callback from CBFIFO packet scheduler to TCP module, and so forth. Because the TCP module transmits these packets in response to callbacks from the packet scheduler, the TCP module transmits packets at precisely the rate of service dictated by the QoS administrative policy that governs the underlying packet traffic.
- the FIFO queue 360 within a CBFIFO module 390 is implemented such that it guarantees that it will never drop an HTNQ packet. Since the HTNQ convention is that the producer of such packets generates only one packet per callback, the FIFO queue 360 within CBFIFO module 390 is only large enough to hold at least one HTNQ packet per active connection. Thus, in this example, the queue 360 has only three buffer sections, one for each of the three active connections 380 .
- the FIFO queue 360 within the CBFIFO module 390 tracks the number of HTNQ packets as well as the number of non-HTNQ packets.
- the CBFIFO module 390 is configured with a queue limit. If the number of non-HTNQ packets exceeds this queue limit, then a non-HTNQ packet is dropped from the queue. This approach allows a single CBFIFO module to effectively process both HTNQ packets as well as non-HTNQ packets in a commingled fashion without letting the queue of non-HTNQ packets grow without bound since such traffic would not be adhering to the one-packet-per-callback HTNQ limit.
- the queue is also configured with a HTNQ limit.
- a packet is dropped.
- an HTNQ packet is dropped, the number of supportable connections has been exceeded and an exceptional condition should be raised, for example, by posting an alarm to a management system, by generating an SMTP trap, or by some other similar means.
- an exceptional condition should be raised, for example, by posting an alarm to a management system, by generating an SMTP trap, or by some other similar means.
- the status of a connection may be changed.
- FIG. 4 is a flow diagram illustrating a method 400 for transitioning the status of a connection, e.g. from an idle state to an active state, according to an embodiment of the present invention.
- An idle state is where there is no data to send and consequently no packets to send.
- An active state is where there is data that needs to be sent.
- an additional bit of state associated with each TCP connection is kept regarding whether or not there is a packet pending in the system and hence whether it is known that a callback is guaranteed to occur in the future.
- This bit of state, stored in the TCP connection data structure, is herein called the htnq_active bit.
- step 410 the connection is idle, and thus the htnq_active bit is clear.
- step 420 when the connection becomes active, the htnq_active bit is set. However, there is no pending packet for that connection in the respective CBFIFO queuing module.
- step 430 the TCP module simply sends a first packet to the network layer to launch the callback process.
- step 440 additional packets are then sent by the transport module in response to the callback process.
- the TCP module wants to send subsequent packets but the htnq_active bit is set, no such packets are generated but instead the data is buffered using the normal buffering mechanisms in a typical TCP implementation.
- step 450 an HTNQ callback is invoked but no packets need to be transmitted, thus the connection has become idle.
- the hntq_active bit is cleared to reflect that fact that no packet for that connection will be queued in the network layer.
- the net effect is that for each active TCP connection, there is exactly one packet belonging to that connection stored in the packet scheduler. That packet serves at least two purposes. First, it exists in the scheduler so that the scheduler can service it in accordance with the QoS policies that are being implemented with respect to the other traffic in the scheduler. And second, it serves as a sort of “on switch” for the connection, a reminder to the scheduler to invoke the HTNQ callback and generate the next HTNQ packet for that connection when the currently queued packet is serviced.
- the transport-level connections need not be explicitly tracked and managed by the network-level scheduler as each connection's existence is implied by existence of one of its HTNQ packets in the scheduler's buffers.
- the direct communication between the packet scheduler in the network layer and the TCP module is handled according to prescribed rules.
- the packet scheduler may invoke a procedure in the TCP module for a given TCP connection to perform the HTNQ callback when that connection state happens to be locked because other processing is concurrently operating within a critical section of code. Then, instead of performing the checks as to whether a new packet should be generate at that time, an embodiment of the present invention marks the connection as needing attention with respect to HTNQ.
- the embodiment can note that the HTNQ mechanism needs attention and can at that time determine whether another packet can be passed on to the network layer or whether the htnq_active bit should be cleared because there are no such packets that need be sent.
- the TCP module is extended with logic to interact properly with the underlying HTNQ queuing discipline.
- some policy external to the HTNQ system determines when and whether a given TCP connection is to be processed according to the HTNQ framework. For example, the policy might be that all TCP connections are processed this way; or, it might be that all TCP connections between the data center and any remote office are processed this way.
- such a policy could be integrated with a traffic classification scheme whereby the traffic class to which a connection is classified would include an attribute as to whether connections within said class would be processed in this mode, as further described below.
- a network path or link falls under the complete control of a single network administrative domain.
- an enterprise may have leased a dedicated circuit from a telecommunication provider, or purchased a VPN service with a guaranteed service-level agreement, or deployed a satellite network with a fixed amount of available bandwidth.
- such networks can present a challenging environment to the TCP congestion control algorithm, especially when buffers in network routers or switches are undersized or packet loss rates are otherwise high. Such conditions, when combined with high latency, causes performance problems with TCP congestion control.
- FIG. 5 shows a network system 500 implementing a transport protocol connection according to an embodiment of the present invention.
- all of the TCP connections traversing a given managed network link (path) 510 flow through an HTNQ-enabled proxy 520 .
- Said connections are terminated in the proxy and interact as described above with an HTNQ-aware packet scheduler 530 co-located in said device.
- congestion control 550 is disabled or non-existing.
- the TCP module determines whether to send a subsequent packet for a particular connection based only on that connection's flow control window and it otherwise ignores or does not implement a congestion window. This allows the TCP connection to run smoothly at the rate determined by the QoS policy whether or not there are significant packet losses in the network.
- a second proxy 580 is situated on the far end of the managed path so that the flow control window advertised by the receiving end of the TCP connection can be controlled by the network operator via configuration of said second proxy or via automatic actions performed in said second proxy.
- the flow control window is not influenced by the disposition of the client and server communicating the managed path and thus can be determined appropriately for the path in question such that the disposition of said flow control window never becomes the performance bottleneck.
- This approach herein is called QoS-controlled TCP because the QoS policies within the packet scheduler explicitly control the sending rates of the various TCP connections.
- QoS-controlled TCP with HTNQ is a powerful mechanism because network traffic can be controlled in a very precise and deliberate fashion using arbitrary QoS policies while attaining improved performance due to the explicit coupling of the TCP sending rates with said policies.
- such a mechanism allows for predictably managing a mixture of QoS-controlled TCP traffic with congestion-controlled TCP traffic, where the congestion-controlled traffic may not be traversing the same device as the QoS-controlled traffic and may only present itself as competing traffic in some bottleneck router along the managed path.
- the HTNQ-enabled CBFIFO queuing discipline is configured as the queuing discipline for the leaf classes of a hierarchical link-sharing algorithm like CBQ, HTB, or HFSC.
- the network operator would specify whether the class was to be HTNQ enabled and further whether TCP connections in said class are to be congestion-controlled or QoS-controlled.
- FIG. 6 illustrates a hierarchical link-sharing algorithm according to an embodiment of the present invention.
- a network operator needs to manage a link such that voice over IP (VoIP) traffic has high priority and two important applications (say a backup protocol and a file server protocol) are to be managed by HTNQ, while all other TCP traffic is not to be managed by HTNQ.
- VoIP voice over IP
- the operator creates “class 1” for VoIP traffic and assigns 10% of the link bandwidth to it.
- the operator creates “class 2” for the backup protocol and “class 3” for the file server protocol and assigns 35% of the link bandwidth to the former and 15% of the link bandwidth to the latter.
- the operator creates a sharing class with 50% of the link bandwidth that is the parent of class 2 and class 3.
- class 4 For all other traffic and assigns 40% of the link bandwidth to that class.
- Class 1, class 4, and the sharing class are all configured as children of the root class representing the entire link bandwidth.
- the operator configures class 2 and class 3 to be HTNQ-enabled and QoS-controlled, and leaves the remaining class configured as non-HTNQ and hence congestion-controlled.
- this scenario style of configuration has very powerful properties. For example, if there is only backup traffic present, it will be managed by HTNQ and receive the entire link bandwidth as the link sharing framework will allow the class 2 to utilize or “borrow” the unused bandwidth. On the other hand, if there is backup traffic in the presence of sustained load from other non-backup and non-file-server TCP traffic, then the other traffic will be guaranteed 40% of the bandwidth presuming such traffic is able to utilize that much bandwidth in light of the legacy TCP protocols; at the same time, the backup traffic will be guaranteed 50% of the traffic and will be groomed and optimized to by the HTNQ framework to fully utilize that bandwidth.
- HTNQ can be extended across a network such that the HTNQ-enabled transport protocol is implemented in a device that is separate from the HTNQ-enabled packet scheduler.
- the transport component might be implemented in an end host coincident to a client or server application running on that end host. Or it might be implemented in a proxy that is situated near such end host client and servers.
- the HTNQ-enabled packet scheduler might be implemented within a wide area router, or a link-layer multiplexer, or a link-layer switch or bridge, and so forth. This approach, which distributes HTNQ across at least two devices, is called herein network-based HTNQ.
- FIG. 7 illustrates a method and system for performing network-based HTNQ according to an embodiment of the present invention.
- the HTNQ callback is performed by having the scheduler 710 send a HTNQ callback message 730 over the network 740 to the HTNQ-enabled transport module 720 .
- Such a design can incur additional overhead compared to when the components are co-located in the same device. However, the overhead can be small compared to the advantages afforded by the design. For example, suppose a HTNQ-enabled proxy 750 interacts via network-based HTNQ with an HTNQ-enabled WAN router 760 that manages a moderate bandwidth WAN link.
- the inter-packet spacing times in this case might be on the order of milliseconds or tens of milliseconds.
- imposing a small, sub-millisecond LAN round-trip time on top of the much larger WAN packet time is a small overhead to incur.
- the overhead can be reduced by extending HTNQ to process multiple packets instead of a single packet with each HTNQ callback.
- FIG. 8 illustrates a method 800 for reducing overhead in a network-based HTNQ by processing multiple packets according to an embodiment of the present invention.
- packet scheduler locates an upstream transport module.
- the HTNQ transport module and the HTNQ scheduling module find each other or “rendezvous” over the network.
- additional signaling is provided for the packet scheduler to locate one or more upstream transport modules when these modules are not integrated in the same device as the packet scheduler.
- the notification information can act as an indicator to identify a remote location of the transport module.
- One approach for performing such rendezvous would be in band, where the notification information required for the HTNQ callback is stored in each and every packet, e.g., as a TCP option or an IP option.
- Such information might include the IP address and a UDP or TCP port over which the scheduler is to contact the transport module.
- the scheduler could send HTNQ callbacks as notification messages sent over a connectionless protocol like UDP.
- the scheduler could establish a control connection using TCP between itself and the transport module and send all such HTNQ callbacks over said TCP connection.
- TCP Transmission Control Protocol
- the scheduler could establish a control connection using TCP between itself and the transport module and send all such HTNQ callbacks over said TCP connection.
- a packet scheduler sends a callback to the transport module.
- the HTNQ callback specifies the number of packets that the HTNQ-enabled transport is allowed to send on each callback.
- the callback does not specify the number of packets to be sent.
- the limit on the number of packets provided in response to a HTNQ callback can be configured by a network administrator or built in to a particular implementation as a default value, which eliminates the need for the HTNQ callback to specify this limit.
- the HTNQ-enabled transport module sends from zero up to the specified number of packets in response to said callback.
- processing multiple packets is done for non-network-based HTNQ.
- the size of the HTNQ queue should be increased to handle multiple packets for each active connection, in accordance with the limit on the number of packets.
- the notification information may incorporate information regarding the group, such as the number of packets in the group.
- the TCP module automatically adjusts the size of the buffer that is used to hold application data while the performing the TCP protocol functions.
- buffers In order for TCP to perform well, such buffers need to be large enough to hold all of the data in transmitted on the network, i.e., large enough to “fill the pipe” between the sender and receiver, which is typically expressed as the bandwidth-delay product on the round-trip network path.
- bandwidth-delay product on the round-trip network path.
- such buffers are too large, they waste unnecessary memory and they cause increased end-to-end delay between the sending application and the receiving application. The latter effect can be problematic for an interactive application or for a proxy that is attempting to optimize and prioritize among different application payloads.
- these buffers are called “socket buffers” deriving from the terminology used by the original Berkeley Source Distribution (BSD) networking implementation of the TCP/IP stack and the socket programming interface defined for networked applications.
- BSD Berkeley Source Distribution
- socket buffer autosizing As such, prior art has explored the idea of automatically determining optimal socket buffer sizes based on observations of the congestion window. This approach is called herein socket buffer autosizing. However, in a QoS-controlled TCP connection there is no meaningful congestion window. Hence the prior art for socket buffer autosizing is not applicable.
- an embodiment of the present invention enables socket buffer autosizing in a different fashion.
- the size of the socket buffer is adjusted to be optimal with respect to the HTNQ-enabled scheduler instead of the TCP congestion window.
- the size of the buffer is adjusted in accordance with an embodiment of the HTNQ callback mechanism as follows.
- FIG. 9A is a flow diagram illustrating a method 900 for increasing the size of a socket buffer according to an embodiment of the present invention.
- a callback occurs for a specific connection.
- the socket buffer size is increased by some amount because the network is otherwise underutilized. This allows the application to place more data in the buffer and to further increase the network utilization. This solves the problem of growing the buffer when it is too small.
- FIG. 9B is a flow diagram illustrating a method 950 for decreasing the size of a socket buffer according to an embodiment of the present invention.
- the TCP module performs a check for whether the buffer is too large on each or other predetermine number of HTNQ callbacks.
- a round-trip time is calculated.
- the amount of data that hasn't been sent but has been present in the buffer for more than a round-trip time is determined. In other words, this check determines whether there is extra data in the buffer that cannot be sent in the near future because the transmission is network or scheduler limited.
- the amount of extra data is compared to a threshold.
- step 990 if this amount is larger than some threshold, then the buffer is decreased.
- the present invention can be implemented in the form of control logic in software or hardware or a combination of both.
- the control logic may be stored in an information storage medium as a plurality of instructions adapted to direct an information processing device to perform a set of steps disclosed in an embodiment of the present invention. Based on the disclosure and teachings provided herein, a person of ordinary skill in the art will appreciate other ways and/or methods to implement the present invention.
Abstract
Description
- [Stoica97] I. Stoica, H. Zhang, and T. S. E. Ng. “A Hierarchical Fair Service Curve Algorithm for Link-Sharing, Real-Time and Priority Services”. Proc ACM SIGCOMM '97.
- [Floyd95] S. Floyd and V. Jacobson. Link-sharing and Resource Management Models for Packet Networks. IEEE/ACM Transactions on Networking, Vol. 3, No. 4, August 1995.
- [Saltzer84] J. H. Saltzer, D. P. Reed, and D. D. Clark. End-to-end arguments in system design. ACM Transactions on
Computer Systems vol 2, no. 4, November 1984. - [Balakrishnan95] Hari Balakrishnan, Srinivasan Seshan, and Randy H. Katz. Improving Reliable Transport and Handoff Performance in Cellular Wireless Networks. ACM Wireless Networks,
vol 1, no. 4, December 1995.
Claims (43)
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US11/762,688 US8462629B2 (en) | 2006-06-14 | 2007-06-13 | Cooperative operation of network transport and network quality of service modules |
PCT/US2007/071254 WO2007147078A2 (en) | 2006-06-14 | 2007-06-14 | Cooperative operation of network transport and network quality of service modules |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US81386206P | 2006-06-14 | 2006-06-14 | |
US11/762,688 US8462629B2 (en) | 2006-06-14 | 2007-06-13 | Cooperative operation of network transport and network quality of service modules |
Publications (2)
Publication Number | Publication Date |
---|---|
US20070297414A1 US20070297414A1 (en) | 2007-12-27 |
US8462629B2 true US8462629B2 (en) | 2013-06-11 |
Family
ID=38832877
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/762,688 Active US8462629B2 (en) | 2006-06-14 | 2007-06-13 | Cooperative operation of network transport and network quality of service modules |
Country Status (2)
Country | Link |
---|---|
US (1) | US8462629B2 (en) |
WO (1) | WO2007147078A2 (en) |
Families Citing this family (21)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2007140482A2 (en) * | 2006-05-31 | 2007-12-06 | Riverbed Technology, Inc. | Service curve mapping |
US8462629B2 (en) | 2006-06-14 | 2013-06-11 | Riverbed Technology, Inc. | Cooperative operation of network transport and network quality of service modules |
US8102768B2 (en) * | 2006-10-18 | 2012-01-24 | D & S Consultants, Inc. | Method and system for traffic flow control in a communication network |
US8045456B1 (en) | 2006-11-27 | 2011-10-25 | Marvell International Ltd. | Hierarchical port-based rate limiting |
US8379515B1 (en) | 2007-02-01 | 2013-02-19 | F5 Networks, Inc. | TCP throughput control by imposing temporal delay |
US7782869B1 (en) * | 2007-11-29 | 2010-08-24 | Huawei Technologies Co., Ltd. | Network traffic control for virtual device interfaces |
US9578055B1 (en) | 2008-01-25 | 2017-02-21 | F5 Networks, Inc. | Thwarting drone-waged denial of service attacks on a network |
US7903562B2 (en) * | 2008-02-05 | 2011-03-08 | Lockheed Martin Corporation | Method and system for congestion control |
US8009560B2 (en) * | 2008-12-31 | 2011-08-30 | Microsoft Corporation | Detecting and managing congestion on a shared network link |
US8949444B1 (en) * | 2009-07-14 | 2015-02-03 | Juniper Networks, Inc. | Flow control scheme for parallel flows |
US8705552B1 (en) * | 2010-02-09 | 2014-04-22 | Marvell International Ltd. | Controlling latency variations in a packet node |
US20110282980A1 (en) * | 2010-05-11 | 2011-11-17 | Udaya Kumar | Dynamic protection of a resource during sudden surges in traffic |
US8340126B2 (en) | 2010-06-07 | 2012-12-25 | Lockheed Martin Corporation | Method and apparatus for congestion control |
CN101883007B (en) * | 2010-06-24 | 2015-01-28 | 中兴通讯股份有限公司 | Method and device for realizing service protection under ETREE networking of PTN (Packet Transport Network) network |
US8458327B1 (en) * | 2010-07-15 | 2013-06-04 | Google Inc. | System and method of reducing network latency |
JP5727633B2 (en) * | 2012-02-13 | 2015-06-03 | 日本電信電話株式会社 | Frame search processing apparatus and method |
US10397073B2 (en) * | 2013-03-15 | 2019-08-27 | Cisco Technology, Inc. | Supporting programmability for arbitrary events in a software defined networking environment |
US9749974B2 (en) * | 2013-01-16 | 2017-08-29 | Intel IP Corporation | Methods and arrangements for frame transmissions |
US20150236966A1 (en) * | 2014-02-18 | 2015-08-20 | Alcatel-Lucent Usa Inc. | Control of congestion window size of an information transmission connection |
US10965733B2 (en) * | 2016-08-28 | 2021-03-30 | Vmware, Inc. | Efficient, automated distributed-search methods and systems |
US11080564B2 (en) * | 2018-09-28 | 2021-08-03 | Microsoft Technology Licensing, Llc | Content classification tool with re-classification techniques |
Citations (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6021446A (en) * | 1997-07-11 | 2000-02-01 | Sun Microsystems, Inc. | Network device driver performing initial packet processing within high priority hardware interrupt service routine and then finishing processing within low priority software interrupt service routine |
US6233224B1 (en) * | 1997-09-25 | 2001-05-15 | Sony Computer Laboratory, Inc. | Communication method and data communications terminal, with data communication protocol for inter-layer flow control |
US6587469B1 (en) * | 1998-05-15 | 2003-07-01 | Nortel Networks Limited | Telecommunications system |
US6741575B1 (en) * | 1999-02-26 | 2004-05-25 | Hughes Electronics Corporation | Apparatus and method for efficient delivery of multicast data over personal access communications system (PACS) |
US20040215746A1 (en) | 2003-04-14 | 2004-10-28 | Nbt Technology, Inc. | Transparent client-server transaction accelerator |
US6973033B1 (en) | 1999-12-30 | 2005-12-06 | At&T Corp. | Method and apparatus for provisioning and monitoring internet protocol quality of service |
US20060015774A1 (en) * | 2004-07-19 | 2006-01-19 | Nguyen Huy T | System and method for transmitting data in storage controllers |
US20060039405A1 (en) * | 2004-08-18 | 2006-02-23 | Day Brian A | Systems and methods for frame ordering in wide port SAS connections |
US7010611B1 (en) | 1999-12-21 | 2006-03-07 | Converged Access, Inc. | Bandwidth management system with multiple processing engines |
US20060080671A1 (en) * | 2004-10-13 | 2006-04-13 | Day Brian A | Systems and methods for opportunistic frame queue management in SAS connections |
US7068660B2 (en) | 1999-06-18 | 2006-06-27 | Nokia Corporation | Method for measurement-based connection admission control (MBAC) in a packet data network |
US7075934B2 (en) | 2001-01-10 | 2006-07-11 | Lucent Technologies Inc. | Method and apparatus for hierarchical bandwidth distribution in a packet network |
US7076569B1 (en) * | 2002-10-18 | 2006-07-11 | Advanced Micro Devices, Inc. | Embedded channel adapter having transport layer configured for prioritizing selection of work descriptors based on respective virtual lane priorities |
US7120666B2 (en) | 2002-10-30 | 2006-10-10 | Riverbed Technology, Inc. | Transaction accelerator for client-server communication systems |
US7146425B2 (en) | 2000-12-22 | 2006-12-05 | Matsushita Electric Industrial Co., Ltd. | Measurement-based admission control utilizing effective envelopes and service curves |
US7209489B1 (en) * | 2002-01-23 | 2007-04-24 | Advanced Micro Devices, Inc. | Arrangement in a channel adapter for servicing work notifications based on link layer virtual lane processing |
US7280478B2 (en) * | 2001-11-27 | 2007-10-09 | Information And Communications University Educational Foundation | Control packet structure and method for generating a data burst in optical burst switching networks |
US20070237074A1 (en) * | 2006-04-06 | 2007-10-11 | Curry David S | Configuration of congestion thresholds for a network traffic management system |
US7292593B1 (en) * | 2002-03-28 | 2007-11-06 | Advanced Micro Devices, Inc. | Arrangement in a channel adapter for segregating transmit packet data in transmit buffers based on respective virtual lanes |
US20070297414A1 (en) | 2006-06-14 | 2007-12-27 | Riverbed Technology, Inc. | Cooperative Operation of Network Transport and Network Quality of Service Modules |
-
2007
- 2007-06-13 US US11/762,688 patent/US8462629B2/en active Active
- 2007-06-14 WO PCT/US2007/071254 patent/WO2007147078A2/en active Application Filing
Patent Citations (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6021446A (en) * | 1997-07-11 | 2000-02-01 | Sun Microsystems, Inc. | Network device driver performing initial packet processing within high priority hardware interrupt service routine and then finishing processing within low priority software interrupt service routine |
US6233224B1 (en) * | 1997-09-25 | 2001-05-15 | Sony Computer Laboratory, Inc. | Communication method and data communications terminal, with data communication protocol for inter-layer flow control |
US6587469B1 (en) * | 1998-05-15 | 2003-07-01 | Nortel Networks Limited | Telecommunications system |
US6741575B1 (en) * | 1999-02-26 | 2004-05-25 | Hughes Electronics Corporation | Apparatus and method for efficient delivery of multicast data over personal access communications system (PACS) |
US7068660B2 (en) | 1999-06-18 | 2006-06-27 | Nokia Corporation | Method for measurement-based connection admission control (MBAC) in a packet data network |
US7010611B1 (en) | 1999-12-21 | 2006-03-07 | Converged Access, Inc. | Bandwidth management system with multiple processing engines |
US6973033B1 (en) | 1999-12-30 | 2005-12-06 | At&T Corp. | Method and apparatus for provisioning and monitoring internet protocol quality of service |
US7146425B2 (en) | 2000-12-22 | 2006-12-05 | Matsushita Electric Industrial Co., Ltd. | Measurement-based admission control utilizing effective envelopes and service curves |
US7075934B2 (en) | 2001-01-10 | 2006-07-11 | Lucent Technologies Inc. | Method and apparatus for hierarchical bandwidth distribution in a packet network |
US7280478B2 (en) * | 2001-11-27 | 2007-10-09 | Information And Communications University Educational Foundation | Control packet structure and method for generating a data burst in optical burst switching networks |
US7209489B1 (en) * | 2002-01-23 | 2007-04-24 | Advanced Micro Devices, Inc. | Arrangement in a channel adapter for servicing work notifications based on link layer virtual lane processing |
US7292593B1 (en) * | 2002-03-28 | 2007-11-06 | Advanced Micro Devices, Inc. | Arrangement in a channel adapter for segregating transmit packet data in transmit buffers based on respective virtual lanes |
US7076569B1 (en) * | 2002-10-18 | 2006-07-11 | Advanced Micro Devices, Inc. | Embedded channel adapter having transport layer configured for prioritizing selection of work descriptors based on respective virtual lane priorities |
US7120666B2 (en) | 2002-10-30 | 2006-10-10 | Riverbed Technology, Inc. | Transaction accelerator for client-server communication systems |
US20040215746A1 (en) | 2003-04-14 | 2004-10-28 | Nbt Technology, Inc. | Transparent client-server transaction accelerator |
US20060015774A1 (en) * | 2004-07-19 | 2006-01-19 | Nguyen Huy T | System and method for transmitting data in storage controllers |
US20060039405A1 (en) * | 2004-08-18 | 2006-02-23 | Day Brian A | Systems and methods for frame ordering in wide port SAS connections |
US20060080671A1 (en) * | 2004-10-13 | 2006-04-13 | Day Brian A | Systems and methods for opportunistic frame queue management in SAS connections |
US20070237074A1 (en) * | 2006-04-06 | 2007-10-11 | Curry David S | Configuration of congestion thresholds for a network traffic management system |
US20070297414A1 (en) | 2006-06-14 | 2007-12-27 | Riverbed Technology, Inc. | Cooperative Operation of Network Transport and Network Quality of Service Modules |
Non-Patent Citations (14)
Title |
---|
Balakrishnan, S., et al., "Improving reliable transport and handoff performance in cellular wireless networks," Wireless Networks, vol. 1, No. 4, Dec. 1995, pp. 469-481. |
Balakrishnan, S., et al., "Improving reliable transport and handoff performance in cellular wireless networks," Wireless Networks, vol. 1, No. 4, Dec. 1995. pp. 469-481. |
Bennet and Zhang, "Hierarchical Packet Fair Queuing Algorithms", Proc. ACM SIGCOMM 1996. |
Cruz, R. L., "Service burstiness and dynamic burstiness measures: a framework", Journal of High Speed Networks, vol. 1, No. 2, 1992. |
Floyd, S. and V. Jacobson, Link-Sharing and Resource Management Models for Packet Networks, IEEE/ACM Transactions on Networking, vol. 3, No. 4, Aug. 1995. |
Floyd, S., et al., "Link-Sharing and Resource Management Models for Packet Networks," IEEE/ACM Transactions on Networking, vol. 3, No. 4, Aug. 1995, pp. 365-386. |
International Search Report and Written Opinion mailed Jul. 7, 2008 of PCT/US07/70158 filed May 31, 2007. |
International Search Report and Written Opinion mailed Sep. 22, 2008 of PCT/US07/71254 filed Jun. 14, 2007. |
Saltzer, J. H., et al., "End-to-End Arguments in System Design," ACM Transactions on Computer Systems, vol. 2, No. 4, Nov. 1984, pp. 277-288. |
Sariowan, H., R. Cruz, and G. Polyzos "Scheduling for Quality of Service Guarantees via Service Curves", Proc. ICCCN Sep. 1995. |
Stoica, I., et al., "A Hierarchal Fair Service Curve Algorithm for Link-Sharing, Real-Time and Priority Services," Proc ACM SIGCOMM '97, pp. 249-262, Cannes, France. |
Stoica, I., H. Zhang, and T. S. E. Ng in an article entitled "A Hierarchical Fair Service Curve Algorithm for Link-Sharing, Real-Time and Priority Service", Proc. ACM SIGCOMM, 1997, Cannes, France. |
U.S. Appl. No. 11/756,584, filed May 31, 2007, now App. Pub No. US2007-0297348 A1, Office Action mailed Jun. 12, 2008. |
U.S. Appl. No. 12/210,087, Lap Trac, filed Sep. 12, 2008. |
Also Published As
Publication number | Publication date |
---|---|
WO2007147078A3 (en) | 2009-01-15 |
WO2007147078A2 (en) | 2007-12-21 |
US20070297414A1 (en) | 2007-12-27 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8462629B2 (en) | Cooperative operation of network transport and network quality of service modules | |
US7801129B2 (en) | Method and apparatus for SIP message prioritization | |
US7697519B2 (en) | Packet processing | |
US7983170B2 (en) | In-band quality-of-service signaling to endpoints that enforce traffic policies at traffic sources using policy messages piggybacked onto DiffServ bits | |
KR100644445B1 (en) | Class-Based Rate Control Using a Multi-Threshold Leaky Bucket | |
AU2011279353B2 (en) | System, method and computer program for intelligent packet distribution | |
EP1017203B1 (en) | Monitoring of Internet differentiated services for transactional applications | |
US7916718B2 (en) | Flow and congestion control in switch architectures for multi-hop, memory efficient fabrics | |
US6839767B1 (en) | Admission control for aggregate data flows based on a threshold adjusted according to the frequency of traffic congestion notification | |
US20060187836A1 (en) | Communication device and method of prioritizing transference of time-critical data | |
US8547846B1 (en) | Method and apparatus providing precedence drop quality of service (PDQoS) with class-based latency differentiation | |
US7383349B2 (en) | Controlling the flow of packets within a network node utilizing random early detection | |
US8520520B2 (en) | System and method for per flow guaranteed throughput, multiple TCP flow bandwidth provisioning, and elimination of packet drops for transmission control protocol (TCP) and TCP-friendly protocols | |
US8203956B1 (en) | Method and apparatus providing a precedence drop quality of service (PDQoS) | |
US20120176903A1 (en) | Non-uniform per-packet priority marker for use with adaptive protocols | |
Astuti | Packet handling | |
Cisco | QC: Quality of Service Overview | |
Venkataraman et al. | A priority-layered approach to transport for high bandwidth-delay product networks | |
Semeria et al. | Supporting Differentiated Service Classes: Active Queue Memory Management | |
Fu et al. | Using parasite flows to camouflage flow traffic | |
Lee et al. | A Novel Scheme for Improving the Fairness of Queue Management in Internet Congestion Control | |
Krzyzanowski | Quality of Service | |
Akinyemi et al. | AN IMPROVED NETWORK CONGESTION AVOIDANCE MODEL | |
Smith | Responsive vs. Unresponsive Traffic: Active Queue Management for a Better-Than-Best-Effort Service | |
Kabari | Simulation of an Optimized Data Packet Transmission in a Congested Network |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: RIVERBED TECHNOLOGY, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:GUPTA, NITIN;WELCH, WILLIAM;MCCANNE, STEVEN;REEL/FRAME:019769/0965;SIGNING DATES FROM 20070821 TO 20070827 Owner name: RIVERBED TECHNOLOGY, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:GUPTA, NITIN;WELCH, WILLIAM;MCCANNE, STEVEN;SIGNING DATES FROM 20070821 TO 20070827;REEL/FRAME:019769/0965 |
|
AS | Assignment |
Owner name: MORGAN STANLEY & CO. LLC, MARYLAND Free format text: SECURITY AGREEMENT;ASSIGNORS:RIVERBED TECHNOLOGY, INC.;OPNET TECHNOLOGIES, INC.;REEL/FRAME:029646/0060 Effective date: 20121218 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
AS | Assignment |
Owner name: RIVERBED TECHNOLOGY, INC., CALIFORNIA Free format text: RELEASE OF PATENT SECURITY INTEREST;ASSIGNOR:MORGAN STANLEY & CO. LLC, AS COLLATERAL AGENT;REEL/FRAME:032113/0425 Effective date: 20131220 |
|
AS | Assignment |
Owner name: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT, NEW YORK Free format text: PATENT SECURITY AGREEMENT;ASSIGNOR:RIVERBED TECHNOLOGY, INC.;REEL/FRAME:032421/0162 Effective date: 20131220 Owner name: JPMORGAN CHASE BANK, N.A., AS ADMINISTRATIVE AGENT Free format text: PATENT SECURITY AGREEMENT;ASSIGNOR:RIVERBED TECHNOLOGY, INC.;REEL/FRAME:032421/0162 Effective date: 20131220 |
|
AS | Assignment |
Owner name: RIVERBED TECHNOLOGY, INC., CALIFORNIA Free format text: RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BARCLAYS BANK PLC;REEL/FRAME:035521/0069 Effective date: 20150424 |
|
AS | Assignment |
Owner name: MORGAN STANLEY SENIOR FUNDING, INC., AS COLLATERAL AGENT, NEW YORK Free format text: SECURITY INTEREST;ASSIGNOR:RIVERBED TECHNOLOGY, INC.;REEL/FRAME:035561/0363 Effective date: 20150424 Owner name: MORGAN STANLEY SENIOR FUNDING, INC., AS COLLATERAL Free format text: SECURITY INTEREST;ASSIGNOR:RIVERBED TECHNOLOGY, INC.;REEL/FRAME:035561/0363 Effective date: 20150424 |
|
AS | Assignment |
Owner name: RIVERBED TECHNOLOGY, INC., CALIFORNIA Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE CONVEYING PARTY NAME PREVIOUSLY RECORDED ON REEL 035521 FRAME 0069. ASSIGNOR(S) HEREBY CONFIRMS THE RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:JPMORGAN CHASE BANK, N.A.;REEL/FRAME:035807/0680 Effective date: 20150424 |
|
FPAY | Fee payment |
Year of fee payment: 4 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 8 |
|
AS | Assignment |
Owner name: ALTER DOMUS (US) LLC, AS COLLATERAL AGENT, ILLINOIS Free format text: PATENT SECURITY AGREEMENT;ASSIGNOR:RIVERBED TECHNOLOGY, INC.;REEL/FRAME:055514/0249 Effective date: 20201231 |
|
AS | Assignment |
Owner name: MACQUARIE CAPITAL FUNDING LLC, NEW YORK Free format text: SECURITY INTEREST;ASSIGNORS:RIVERBED HOLDINGS, INC.;RIVERBED TECHNOLOGY, INC.;ATERNITY LLC;REEL/FRAME:056397/0750 Effective date: 20210420 |
|
AS | Assignment |
Owner name: ATERNITY LLC, CALIFORNIA Free format text: RELEASE OF SECURITY INTEREST IN PATENTS RECORED AT REEL 056397, FRAME 0750;ASSIGNOR:MACQUARIE CAPITAL FUNDING LLC;REEL/FRAME:057983/0356 Effective date: 20211012 Owner name: RIVERBED TECHNOLOGY, INC., CALIFORNIA Free format text: RELEASE OF SECURITY INTEREST IN PATENTS RECORED AT REEL 056397, FRAME 0750;ASSIGNOR:MACQUARIE CAPITAL FUNDING LLC;REEL/FRAME:057983/0356 Effective date: 20211012 Owner name: RIVERBED HOLDINGS, INC., CALIFORNIA Free format text: RELEASE OF SECURITY INTEREST IN PATENTS RECORED AT REEL 056397, FRAME 0750;ASSIGNOR:MACQUARIE CAPITAL FUNDING LLC;REEL/FRAME:057983/0356 Effective date: 20211012 |
|
AS | Assignment |
Owner name: ALTER DOMUS (US) LLC, AS COLLATERAL AGENT, ILLINOIS Free format text: PATENT SECURITY AGREEMENT SUPPLEMENT - SECOND LIEN;ASSIGNORS:RIVERBED HOLDINGS, INC.;RIVERBED TECHNOLOGY, INC.;ATERNITY LLC;REEL/FRAME:057810/0559 Effective date: 20211013 Owner name: MORGAN STANLEY SENIOR FUNDING, INC., AS COLLATERAL AGENT, MARYLAND Free format text: PATENT SECURITY AGREEMENT SUPPLEMENT - FIRST LIEN;ASSIGNORS:RIVERBED HOLDINGS, INC.;RIVERBED TECHNOLOGY, INC.;ATERNITY LLC;REEL/FRAME:057810/0502 Effective date: 20211013 |
|
AS | Assignment |
Owner name: WILMINGTON TRUST, NATIONAL ASSOCIATION, MINNESOTA Free format text: PATENT SECURITY AGREEMENT;ASSIGNORS:RIVERBED TECHNOLOGY, INC.;ATERNITY LLC;REEL/FRAME:057943/0386 Effective date: 20211013 |
|
AS | Assignment |
Owner name: WILMINGTON TRUST, NATIONAL ASSOCIATION, AS U.S. COLLATERAL AGENT, MINNESOTA Free format text: SECURITY INTEREST;ASSIGNORS:RIVERBED TECHNOLOGY LLC (FORMERLY RIVERBED TECHNOLOGY, INC.);ATERNITY LLC;REEL/FRAME:058486/0216 Effective date: 20211207 |
|
AS | Assignment |
Owner name: ATERNITY LLC, MASSACHUSETTS Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:WILMINGTON TRUST, NATIONAL ASSOCIATION, AS U.S. COLLATERAL AGENT;REEL/FRAME:058593/0169 Effective date: 20211207 Owner name: RIVERBED TECHNOLOGY, INC., CALIFORNIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:WILMINGTON TRUST, NATIONAL ASSOCIATION, AS U.S. COLLATERAL AGENT;REEL/FRAME:058593/0169 Effective date: 20211207 Owner name: ATERNITY LLC, MASSACHUSETTS Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:ALTER DOMUS (US) LLC, AS COLLATERAL AGENT;REEL/FRAME:058593/0108 Effective date: 20211207 Owner name: RIVERBED TECHNOLOGY, INC., CALIFORNIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:ALTER DOMUS (US) LLC, AS COLLATERAL AGENT;REEL/FRAME:058593/0108 Effective date: 20211207 Owner name: ATERNITY LLC, MASSACHUSETTS Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC., AS COLLATERAL AGENT;REEL/FRAME:058593/0046 Effective date: 20211207 Owner name: RIVERBED TECHNOLOGY, INC., CALIFORNIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC., AS COLLATERAL AGENT;REEL/FRAME:058593/0046 Effective date: 20211207 |
|
AS | Assignment |
Owner name: RIVERBED TECHNOLOGY LLC, CALIFORNIA Free format text: CHANGE OF NAME;ASSIGNOR:RIVERBED TECHNOLOGY, INC.;REEL/FRAME:059232/0551 Effective date: 20211207 |
|
AS | Assignment |
Owner name: RIVERBED HOLDINGS, INC., CALIFORNIA Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:ALTER DOMUS (US) LLC, AS COLLATERAL AGENT;REEL/FRAME:064673/0739 Effective date: 20211207 Owner name: ATERNITY LLC, MASSACHUSETTS Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:ALTER DOMUS (US) LLC, AS COLLATERAL AGENT;REEL/FRAME:064673/0739 Effective date: 20211207 Owner name: RIVERBED TECHNOLOGY, INC., CALIFORNIA Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:ALTER DOMUS (US) LLC, AS COLLATERAL AGENT;REEL/FRAME:064673/0739 Effective date: 20211207 |