US20090164835A1 - Method and system for survival of data plane through a total control plane failure - Google Patents
Method and system for survival of data plane through a total control plane failure Download PDFInfo
- Publication number
- US20090164835A1 US20090164835A1 US12/002,873 US287307A US2009164835A1 US 20090164835 A1 US20090164835 A1 US 20090164835A1 US 287307 A US287307 A US 287307A US 2009164835 A1 US2009164835 A1 US 2009164835A1
- Authority
- US
- United States
- Prior art keywords
- originator
- route
- withdrawal
- connectivity failure
- failure occurred
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0654—Management of faults, events, alarms or notifications using network fault recovery
- H04L41/0668—Management of faults, events, alarms or notifications using network fault recovery by dynamic selection of recovery network elements, e.g. replacement by the most appropriate element after failure
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0805—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability
- H04L43/0811—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability by checking connectivity
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L45/00—Routing or path finding of packets in data switching networks
- H04L45/02—Topology update or discovery
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L45/00—Routing or path finding of packets in data switching networks
- H04L45/02—Topology update or discovery
- H04L45/033—Topology update or discovery by updating distance vector protocols
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L45/00—Routing or path finding of packets in data switching networks
- H04L45/02—Topology update or discovery
- H04L45/04—Interdomain routing, e.g. hierarchical routing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L45/00—Routing or path finding of packets in data switching networks
- H04L45/28—Routing or path finding of packets in data switching networks using route fault recovery
Definitions
- This application relates generally to communication networks, and more particularly, to a method and system for retaining learned routes in the event of a routing protocol connectivity failure between intermediate routers in a communication network.
- Computer network data communication involves the exchange of data between two or more entities interconnected by communication links and sub-networks.
- Routers interconnect the communication links and subnets to enable transmission of data between end nodes.
- Communication software executing on routers correlates and manages data communication with other routers.
- the routers typically communicate by exchanging discrete messages or packets of data according to predefined protocols, such as the Transmission Control Protocol/Internet Protocol (“TCP/IP”).
- TCP/IP Transmission Control Protocol/Internet Protocol
- a protocol consists of a set of rules defining how the routers interact with each other.
- network routing software executing on the intermediate routers allows expansion of communication to other nodes.
- An AS is a network or group of networks under a shared technical administration and with common routing policies.
- An Internet Service Provider (“ISP”) is an example of an AS that interconnects dispersed networks to provide Internet connectivity.
- IGPs Interior Gateway Protocols
- link-state protocols are intra-domain routing protocols that define the manner with which routing information and network-topology information are exchanged and processed within a particular AS.
- Examples of conventional link-state protocols include, but are not limited to, the Open Shortest Path First (“OSPF”) protocol and the Intermediate-System-to-Intermediate-System (“ISIS”) protocol.
- OSPF Open Shortest Path First
- ISIS Intermediate-System-to-Intermediate-System
- a plurality of interconnected AS domains may be configured to exchange messages in accordance with an inter-domain routing protocol, such as the Border Gateway Protocol (“BGP”).
- BGP Border Gateway Protocol
- BGP allows each AS to independently create its own routing policies to override distance metrics when appropriate. To address this flexibility, BGP advertises routes for carrying data to the address space indicated by the IP prefix of the announced route. When a BGP router advertises to a neighbor that it has a path for reaching a specific IP prefix, the neighbor has a high degree of confidence that the advertising BGP router will actively use the specific path to reach the target destination.
- the popularity of BGP is due, in part, to its ability to distribute reachability information selecting the best route to each destination according to policies specified for each AS.
- each routing domain can include at least one provider edge (“PE”) router that advertises routes to a PE router of another AS. Before transmitting such messages, however, the PE routers cooperate to establish a logical “peer” connection or session. These PE routers are also known as “originators” or “BGP speakers.” Two BGP speakers with an open BGP session for the purpose of exchanging routing information are referred to as “peers” or “neighbors.” BGP typically performs routing between AS domains by exchanging routing information among BGP speakers of each AS. BGP speakers also send update messages whenever a change in the topology occurs. For example, if a route is no longer accessible for any reason, a withdrawal of that route is propagated among the peers, which can delete a route from a router's control plane. BGP relies on pre-existing connectivity provided by IGP routes.
- PE provider edge
- Two BGP enabled PE routers not in the same AS can use external BGP (“eBGP”) to exchange routes.
- the routing information exchanged by eBGP neighbors typically includes the address of a PE router in another AS, which is also known as the “next hop” address.
- eBGP external BGP
- the speaker When a BGP speaker receives updates from multiple AS domains describing different paths to the same destination, the speaker typically chooses a single best path for reaching that destination. Once chosen, the speaker can use internal BGP (“iBGP”) to propagate that best path to its AS neighbors, including the “next hop” associated with that best path.
- Each route advertised by BGP must have a “next hop” address that is reachable through IGP in order for that route to be considered valid.
- iBGP speakers within an AS are typically required to connect in full mesh to ensure that all iBGP speakers receive route updates from other iBGP speakers. However, the full mesh requirement can become very burdensome in more complex topologies.
- RR route reflectors
- BGP In existing systems, the current implementation of BGP treats an originator and advertiser (“RR”) of a route equally in the event that there is a loss of BGP connectivity between routers. Such existing systems using the current implementation of BGP will withdraw all routes toward a particular “next hop” when there is a BGP connectivity failure between BGP speakers or an iBGP connectivity failure between an advertiser and one of its peers within an AS, regardless of whether the “next hop” is still accessible via IGP.
- RR originator and advertiser
- graceful restart which drops control plane connections to its routing peers for a short time, during which traffic forwarding continues, and restarts the control plane with a new instance of the routing tables.
- All peers must have compatible graceful restart extensions that are negotiated during start up. This compatibility requirement becomes problematic in more complex topologies.
- graceful restart unnecessarily drops the control plane connection when an intermediate advertiser or RR fails, but the route originator is still accessible via IGP.
- a system and method for retaining routes in a control plane learned by an inter-domain routing protocol in the event of a connectivity failure between routers Routers are classified as either route reflectors or originators. A determination is made whether the connectivity failure occurred between a route reflector and an originator, two originators, or two route reflectors. A determination is then made whether to propagate a withdrawal of learned routes based on whether the connectivity failure occurred between a route reflector and an originator, two originators, or two route reflectors. A withdrawal of learned routes is propagated to neighboring routers if the connectivity failure occurred between two originators, or between a route reflector and an originator that is inaccessible via an intra-domain routing protocol. No withdrawal of learned routes is propagated if the connectivity failure occurred between two route reflectors, or between a route reflector and an originator that is accessible via an intra-domain routing protocol.
- FIG. 1 is a high level diagram of a router that can be configured to be a border router, RR, or any other router configuration necessary for implementation of the method and system described herein;
- FIG. 2 is a schematic of an exemplary communication network or system having a route reflector topology
- FIG. 3 is flow chart illustrating a main sequence of steps for withdrawing all routes to an originator in the event of an originator failure
- FIG. 4 is a flow chart illustrating a main sequence of steps performed by an originator for withdrawing or retaining routes depending on whether the originator or an advertiser has failed;
- FIG. 5 is a flow chart illustrating a main sequence of steps performed by an advertiser for withdrawing or retaining routes depending on whether the originator or an advertiser has failed.
- a system and method for retaining learned routes in a data communication network despite a routing protocol (e.g., BGP) session failure are disclosed herein.
- Such networks include, but are not limited to, a network of routers running IGP protocols (e.g., OSPF, ISIS, iBGP, etc.), route advertising protocols (e.g., BGP, eBGP, iBGP, etc.), or a network supporting Virtual Private Network (“VPN”) services using BGP protocol.
- IGP protocols e.g., OSPF, ISIS, iBGP, etc.
- route advertising protocols e.g., BGP, eBGP, iBGP, etc.
- VPN Virtual Private Network
- the disclosed system and method can distinguish whether a route was learned from an originator (e.g., a provider edge (“PE”) router) or an advertiser (i.e., a route reflector (“RR”)).
- PE provider edge
- RR route reflector
- the method can be implemented in a router as part of a BGP program suite or installed separately.
- BGP which has become a de facto standard for facilitating communications between routers in different ASs—it is understood that this disclosure is not limited to BGP and can also be implemented with other routing protocols.
- FIG. 1 is a high level block diagram of an exemplary router 100 suitable for use in performing the functions described herein.
- Router 100 comprises a processor 110 (e.g., a central processing unit (“CPU”)), a memory 120 (e.g., random access memory (“RAM”) and/or read only memory (“ROM”)), a storage device 130 (e.g., tape drive, floppy drive, hard disk drive, compact disk drive, etc.), various input/output devices 140 (e.g., display, keyboard, mouse, speakers, buttons, etc.), and a network interface 150 .
- processor 110 e.g., a central processing unit (“CPU”)
- memory 120 e.g., random access memory (“RAM”) and/or read only memory (“ROM”)
- RAM random access memory
- ROM read only memory
- storage device 130 e.g., tape drive, floppy drive, hard disk drive, compact disk drive, etc.
- various input/output devices 140 e.g., display, keyboard, mouse, speakers, buttons,
- Router 100 runs a router operating system 160 and can be configured with BGP software 170 and IGP software 175 .
- Router 100 can also be configured as a Provider Edge (“PE”) router in a Multi Protocol Label Switching (“MPLS”) Virtual Private Network (“VPN”) running eBGP to peer with a customer edge (“CE”) router and iBGP to peer with a RR.
- PE Provider Edge
- MPLS Multi Protocol Label Switching
- VPN Virtual Private Network
- CE customer edge
- the processor 110 controls the overall operation of router 100 by executing computer program instructions from router operating system 160 and routing protocols 170 and 175 .
- the computer program instructions may be stored in storage device 130 or any other computer-readable medium, and loaded into memory 120 when the router operating system 160 boots.
- Router 100 is configured by software with a control plane process 180 that is responsible for building and maintaining the Internet Protocol (“IP”) routing table, which can define where an IP packet should be routed to based on a “next hop” IP address and the egress interface that the “next hop” is reachable from.
- IP Internet Protocol
- the control plane is the part of the router architecture that is concerned with drawing the network map or the information in a routing table that defines what to do with incoming packets.
- Control plane operations typically require an understanding of routing protocols and hence require some intelligence that is capable of supporting the complex algorithms and data structures associated with protocols such as IGP and BGP.
- the control plane operations required might vary between different routers depending on the routing protocol(s) configured. For example, a BGP withdrawal of a particular route can delete the “next hop” and egress interface for that route from the routing table.
- Router 100 is also configured by software with a data plane process 190 that is responsible for actually routing an IP packet based upon information learned by the control plane 180 .
- the data plane operations 190 are typically simple and fixed, since a packet is routed similarly regardless of the routing protocol used.
- the disclosed method can be a series of programming instructions executing in memory 120 .
- a user can interact or configure the router 100 via input/output interface 140 .
- Router 100 can have multiple network interfaces 150 for communicating with other routers or nodes in a network.
- FIG. 1 is a high level representation of some of the components of such a router for illustrative purposes.
- FIG. 2 is a schematic of an exemplary communication network or system 200 having a route reflector (“RR”) topology.
- the network 200 comprises a plurality of route reflectors RRw 1 , RRw 2 , RRe 1 , RRe 2 split into two geographical clusters, an East Cluster and a West Cluster.
- Each cluster services relevant provider edge (“PE”) routers in that geographic region.
- PE provider edge
- each PE router peers with both route reflectors in its geographical region. For example, route reflectors RRw 1 and RRw 2 service provider edge router PE in the West Cluster, and route reflectors RRe 1 and RRe 2 service provider edge router PE 1 in the East Cluster.
- FIG. 3 is flow chart illustrating a main sequence of steps performed by a router or similar device for withdrawing all routes to an originator or provider edge (“PE”) router in the event of an originator failure.
- PE provider edge
- the system announces a withdrawal of all routes to all AS neighbors via iBGP having a “next hop” attribute equal to or otherwise associated with the originator in step 303 and announces a withdrawal of the route to any other eBGP speaker in step 304 .
- originator PE of the West Cluster loses BGP connectivity to originator PE 1 of the East Cluster
- originator PE propagates a BGP withdrawal for all routes having a “next hop” attribute equal to or otherwise associated with originator PE 1 . This occurs regardless of whether originator PE can access originator PE 1 via IGP.
- FIG. 4 is a flow chart illustrating a main sequence of steps performed by an originator for withdrawing or retaining routes depending on whether the originator or an advertiser has failed.
- the system determines whether BGP connectivity to the originator was lost. If so, then the system propagates or announces a BGP withdrawal of all routes having a “next hop” attribute equal to or otherwise associated with that specific originator to all iBGP neighbors within the AS and any eBGP neighbors in step 402 .
- step 401 determines whether BGP connectivity was not lost to an originator. If it is determined in step 401 that BGP connectivity was not lost to an originator, then the system determines whether BGP connectivity to an advertiser or RR was lost in step 403 . If not, then the system retains all valid routes and abstains from propagating or announcing any withdrawal message of routes having a “next hop” attribute equal to or otherwise associated with the originator in step 404 .
- the system determines whether the originator or “next hop” is reachable via IGP in step 405 . If yes, then the system retains all valid routes and abstains from propagating or announcing any withdrawal message of routes having a “next hop” attribute equal to or otherwise associated with the originator in step 404 . If the originator or “next hop” is not reachable via IGP in step 405 , then, in step 402 , the system propagates or announces a BGP withdrawal of all routes having a “next hop” attribute equal to or otherwise associated with the originator to all iBGP neighbors within the AS and any eBGP neighbors.
- FIG. 5 is a flow chart illustrating a main sequence of steps performed by an advertiser or RR for withdrawing or retaining routes depending on whether an originator or an advertiser has failed.
- the system determines whether the advertiser or RR lost BGP connectivity to the originator. If BGP connectivity is lost to the originator, then the system determines whether the originator is still visible in the IGP domain in step 502 . If the originator is still visible in the IGP domain, then the system retains all valid routes and abstains from propagating or announcing a BGP withdrawal of all routes having a “next hop” attribute equal to or otherwise associated with the originator in step 503 .
- step 504 the system propagates or announces a BGP withdrawal of all routes having a “next hop” attribute equal to or otherwise associated with the originator to all iBGP neighbors within the AS and any eBGP neighbors
- step 505 the system determines whether the advertiser or RR lost BGP connectivity to another advertiser or RR in step 505 . If so, then the system proceeds to step 503 , retains all valid routes and abstains from propagating or announcing a BGP withdrawal of all routes having a “next hop” attribute equal to or otherwise associated with the originator. If it is determined in step 505 that the advertiser or RR has not lost BGP connection to another advertiser or RR, then no action is taken in step 506 .
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Environmental & Geological Engineering (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
Description
- This application relates generally to communication networks, and more particularly, to a method and system for retaining learned routes in the event of a routing protocol connectivity failure between intermediate routers in a communication network.
- Computer network data communication involves the exchange of data between two or more entities interconnected by communication links and sub-networks. Routers interconnect the communication links and subnets to enable transmission of data between end nodes. Communication software executing on routers correlates and manages data communication with other routers. The routers typically communicate by exchanging discrete messages or packets of data according to predefined protocols, such as the Transmission Control Protocol/Internet Protocol (“TCP/IP”). A protocol consists of a set of rules defining how the routers interact with each other. In addition, network routing software executing on the intermediate routers allows expansion of communication to other nodes. Collectively, these hardware and software components comprise a collection of communication networks.
- Since management of data communication networks can prove burdensome, smaller groups of one or more computer networks can be maintained as separate routing domains or autonomous systems (“AS”). An AS is a network or group of networks under a shared technical administration and with common routing policies. An Internet Service Provider (“ISP”) is an example of an AS that interconnects dispersed networks to provide Internet connectivity.
- Interior Gateway Protocols (“IGPs”), such as conventional link-state protocols, are intra-domain routing protocols that define the manner with which routing information and network-topology information are exchanged and processed within a particular AS. Examples of conventional link-state protocols include, but are not limited to, the Open Shortest Path First (“OSPF”) protocol and the Intermediate-System-to-Intermediate-System (“ISIS”) protocol.
- A plurality of interconnected AS domains may be configured to exchange messages in accordance with an inter-domain routing protocol, such as the Border Gateway Protocol (“BGP”). BGP allows each AS to independently create its own routing policies to override distance metrics when appropriate. To address this flexibility, BGP advertises routes for carrying data to the address space indicated by the IP prefix of the announced route. When a BGP router advertises to a neighbor that it has a path for reaching a specific IP prefix, the neighbor has a high degree of confidence that the advertising BGP router will actively use the specific path to reach the target destination. The popularity of BGP is due, in part, to its ability to distribute reachability information selecting the best route to each destination according to policies specified for each AS.
- To implement the BGP protocol, each routing domain can include at least one provider edge (“PE”) router that advertises routes to a PE router of another AS. Before transmitting such messages, however, the PE routers cooperate to establish a logical “peer” connection or session. These PE routers are also known as “originators” or “BGP speakers.” Two BGP speakers with an open BGP session for the purpose of exchanging routing information are referred to as “peers” or “neighbors.” BGP typically performs routing between AS domains by exchanging routing information among BGP speakers of each AS. BGP speakers also send update messages whenever a change in the topology occurs. For example, if a route is no longer accessible for any reason, a withdrawal of that route is propagated among the peers, which can delete a route from a router's control plane. BGP relies on pre-existing connectivity provided by IGP routes.
- Two BGP enabled PE routers not in the same AS can use external BGP (“eBGP”) to exchange routes. The routing information exchanged by eBGP neighbors typically includes the address of a PE router in another AS, which is also known as the “next hop” address. When a BGP speaker receives updates from multiple AS domains describing different paths to the same destination, the speaker typically chooses a single best path for reaching that destination. Once chosen, the speaker can use internal BGP (“iBGP”) to propagate that best path to its AS neighbors, including the “next hop” associated with that best path. Each route advertised by BGP must have a “next hop” address that is reachable through IGP in order for that route to be considered valid. iBGP speakers within an AS are typically required to connect in full mesh to ensure that all iBGP speakers receive route updates from other iBGP speakers. However, the full mesh requirement can become very burdensome in more complex topologies.
- Existing networks have alleviated this limitation by the use of advertisers or route reflectors (“RR”), which are special routers acting as a focal point for iBGP sessions. Multiple iBGP speakers can establish an iBGP peer with one RR, rather than establish an iBGP peering session with every other node in full mesh. The RR can take responsibility of re-advertising or “reflecting” learned routes from another BGP speaker within an AS.
- In existing systems, the current implementation of BGP treats an originator and advertiser (“RR”) of a route equally in the event that there is a loss of BGP connectivity between routers. Such existing systems using the current implementation of BGP will withdraw all routes toward a particular “next hop” when there is a BGP connectivity failure between BGP speakers or an iBGP connectivity failure between an advertiser and one of its peers within an AS, regardless of whether the “next hop” is still accessible via IGP.
- One approach to mitigating the effects of a BGP session failure is a BGP extension known as graceful restart, which drops control plane connections to its routing peers for a short time, during which traffic forwarding continues, and restarts the control plane with a new instance of the routing tables. For graceful restart to be successful, all peers must have compatible graceful restart extensions that are negotiated during start up. This compatibility requirement becomes problematic in more complex topologies. Furthermore, graceful restart unnecessarily drops the control plane connection when an intermediate advertiser or RR fails, but the route originator is still accessible via IGP.
- Thus, it is desirable to have a local solution requiring no extension compatibility between peers that can differentiate between the failure of an originator and an advertiser. Such a solution would retain routes learned from an intermediate advertiser or RR if the originator is still reachable via IGP, notwithstanding its BGP session termination.
- A system and method for retaining routes in a control plane learned by an inter-domain routing protocol in the event of a connectivity failure between routers. Routers are classified as either route reflectors or originators. A determination is made whether the connectivity failure occurred between a route reflector and an originator, two originators, or two route reflectors. A determination is then made whether to propagate a withdrawal of learned routes based on whether the connectivity failure occurred between a route reflector and an originator, two originators, or two route reflectors. A withdrawal of learned routes is propagated to neighboring routers if the connectivity failure occurred between two originators, or between a route reflector and an originator that is inaccessible via an intra-domain routing protocol. No withdrawal of learned routes is propagated if the connectivity failure occurred between two route reflectors, or between a route reflector and an originator that is accessible via an intra-domain routing protocol.
-
FIG. 1 is a high level diagram of a router that can be configured to be a border router, RR, or any other router configuration necessary for implementation of the method and system described herein; -
FIG. 2 is a schematic of an exemplary communication network or system having a route reflector topology; -
FIG. 3 is flow chart illustrating a main sequence of steps for withdrawing all routes to an originator in the event of an originator failure; -
FIG. 4 is a flow chart illustrating a main sequence of steps performed by an originator for withdrawing or retaining routes depending on whether the originator or an advertiser has failed; and -
FIG. 5 is a flow chart illustrating a main sequence of steps performed by an advertiser for withdrawing or retaining routes depending on whether the originator or an advertiser has failed. - A system and method for retaining learned routes in a data communication network despite a routing protocol (e.g., BGP) session failure are disclosed herein. Such networks include, but are not limited to, a network of routers running IGP protocols (e.g., OSPF, ISIS, iBGP, etc.), route advertising protocols (e.g., BGP, eBGP, iBGP, etc.), or a network supporting Virtual Private Network (“VPN”) services using BGP protocol. The disclosed system and method can distinguish whether a route was learned from an originator (e.g., a provider edge (“PE”) router) or an advertiser (i.e., a route reflector (“RR”)). The method can be implemented in a router as part of a BGP program suite or installed separately. Although the system and method are described in the context of BGP—which has become a de facto standard for facilitating communications between routers in different ASs—it is understood that this disclosure is not limited to BGP and can also be implemented with other routing protocols.
-
FIG. 1 is a high level block diagram of anexemplary router 100 suitable for use in performing the functions described herein.Router 100 comprises a processor 110 (e.g., a central processing unit (“CPU”)), a memory 120 (e.g., random access memory (“RAM”) and/or read only memory (“ROM”)), a storage device 130 (e.g., tape drive, floppy drive, hard disk drive, compact disk drive, etc.), various input/output devices 140 (e.g., display, keyboard, mouse, speakers, buttons, etc.), and anetwork interface 150. -
Router 100 runs arouter operating system 160 and can be configured withBGP software 170 andIGP software 175.Router 100 can also be configured as a Provider Edge (“PE”) router in a Multi Protocol Label Switching (“MPLS”) Virtual Private Network (“VPN”) running eBGP to peer with a customer edge (“CE”) router and iBGP to peer with a RR. - The
processor 110 controls the overall operation ofrouter 100 by executing computer program instructions fromrouter operating system 160 androuting protocols storage device 130 or any other computer-readable medium, and loaded intomemory 120 when therouter operating system 160 boots. -
Router 100 is configured by software with acontrol plane process 180 that is responsible for building and maintaining the Internet Protocol (“IP”) routing table, which can define where an IP packet should be routed to based on a “next hop” IP address and the egress interface that the “next hop” is reachable from. In other words, the control plane is the part of the router architecture that is concerned with drawing the network map or the information in a routing table that defines what to do with incoming packets. Control plane operations typically require an understanding of routing protocols and hence require some intelligence that is capable of supporting the complex algorithms and data structures associated with protocols such as IGP and BGP. The control plane operations required might vary between different routers depending on the routing protocol(s) configured. For example, a BGP withdrawal of a particular route can delete the “next hop” and egress interface for that route from the routing table. -
Router 100 is also configured by software with adata plane process 190 that is responsible for actually routing an IP packet based upon information learned by thecontrol plane 180. Thedata plane operations 190 are typically simple and fixed, since a packet is routed similarly regardless of the routing protocol used. The disclosed method can be a series of programming instructions executing inmemory 120. A user can interact or configure therouter 100 via input/output interface 140.Router 100 can havemultiple network interfaces 150 for communicating with other routers or nodes in a network. One skilled in the art will recognize that an implementation of an actual router may contain different components or configurations and thatFIG. 1 is a high level representation of some of the components of such a router for illustrative purposes. -
FIG. 2 is a schematic of an exemplary communication network orsystem 200 having a route reflector (“RR”) topology. In this exemplary embodiment, thenetwork 200 comprises a plurality of route reflectors RRw1, RRw2, RRe1, RRe2 split into two geographical clusters, an East Cluster and a West Cluster. Each cluster services relevant provider edge (“PE”) routers in that geographic region. For redundancy purposes, each PE router peers with both route reflectors in its geographical region. For example, route reflectors RRw1 and RRw2 service provider edge router PE in the West Cluster, and route reflectors RRe1 and RRe2 service provider edge router PE1 in the East Cluster. -
FIG. 3 is flow chart illustrating a main sequence of steps performed by a router or similar device for withdrawing all routes to an originator or provider edge (“PE”) router in the event of an originator failure. Instep 301, the system determines whether a router lost a BGP connection to an originator router. If there is no loss of connection to an originator router, then the system proceeds to step 302 and does not propagate a withdrawal of the route to the originator. - On the other hand, if there is a loss of BGP connectivity to an originator router, then the system announces a withdrawal of all routes to all AS neighbors via iBGP having a “next hop” attribute equal to or otherwise associated with the originator in
step 303 and announces a withdrawal of the route to any other eBGP speaker instep 304. For example, referring toFIG. 2 , if originator PE of the West Cluster loses BGP connectivity to originator PE1 of the East Cluster, then originator PE propagates a BGP withdrawal for all routes having a “next hop” attribute equal to or otherwise associated with originator PE1. This occurs regardless of whether originator PE can access originator PE1 via IGP. -
FIG. 4 is a flow chart illustrating a main sequence of steps performed by an originator for withdrawing or retaining routes depending on whether the originator or an advertiser has failed. Instep 401, the system determines whether BGP connectivity to the originator was lost. If so, then the system propagates or announces a BGP withdrawal of all routes having a “next hop” attribute equal to or otherwise associated with that specific originator to all iBGP neighbors within the AS and any eBGP neighbors instep 402. - On the other hand, if it is determined in
step 401 that BGP connectivity was not lost to an originator, then the system determines whether BGP connectivity to an advertiser or RR was lost instep 403. If not, then the system retains all valid routes and abstains from propagating or announcing any withdrawal message of routes having a “next hop” attribute equal to or otherwise associated with the originator instep 404. - If it is determined that the router lost BGP connectivity to an advertiser or RR in
Step 403, then the system determines whether the originator or “next hop” is reachable via IGP instep 405. If yes, then the system retains all valid routes and abstains from propagating or announcing any withdrawal message of routes having a “next hop” attribute equal to or otherwise associated with the originator instep 404. If the originator or “next hop” is not reachable via IGP instep 405, then, instep 402, the system propagates or announces a BGP withdrawal of all routes having a “next hop” attribute equal to or otherwise associated with the originator to all iBGP neighbors within the AS and any eBGP neighbors. -
FIG. 5 is a flow chart illustrating a main sequence of steps performed by an advertiser or RR for withdrawing or retaining routes depending on whether an originator or an advertiser has failed. Instep 501, the system determines whether the advertiser or RR lost BGP connectivity to the originator. If BGP connectivity is lost to the originator, then the system determines whether the originator is still visible in the IGP domain instep 502. If the originator is still visible in the IGP domain, then the system retains all valid routes and abstains from propagating or announcing a BGP withdrawal of all routes having a “next hop” attribute equal to or otherwise associated with the originator instep 503. If, on the other hand, the originator is no longer visible in the IGP domain, then, instep 504, the system propagates or announces a BGP withdrawal of all routes having a “next hop” attribute equal to or otherwise associated with the originator to all iBGP neighbors within the AS and any eBGP neighbors - Referring back to step 501 in
FIG. 5 , if the advertiser or RR did not lose connectivity to the originator, then, instep 505, the system determines whether the advertiser or RR lost BGP connectivity to another advertiser or RR instep 505. If so, then the system proceeds to step 503, retains all valid routes and abstains from propagating or announcing a BGP withdrawal of all routes having a “next hop” attribute equal to or otherwise associated with the originator. If it is determined instep 505 that the advertiser or RR has not lost BGP connection to another advertiser or RR, then no action is taken instep 506. - Having described and illustrated the principles of this application by reference to one or more preferred embodiments, it should be apparent that the preferred embodiment(s) may be modified in arrangement and detail without departing from the principles disclosed herein and that it is intended that the application be construed as including all such modifications and variations insofar as they come within the spirit and scope of the subject matter disclosed herein. For example, although described in the context of BGP, it is understood that this disclosure is not limited to BGP and can also be implemented with other routing protocols. In addition, the disclosed method and system could work in many different topologies other than the ones illustrated herein.
Claims (24)
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US12/002,873 US8396988B2 (en) | 2007-12-19 | 2007-12-19 | Method and system for survival of data plane through a total control plane failure |
US13/759,567 US8667174B2 (en) | 2007-12-19 | 2013-02-05 | Method and system for survival of data plane through a total control plane failure |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US12/002,873 US8396988B2 (en) | 2007-12-19 | 2007-12-19 | Method and system for survival of data plane through a total control plane failure |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/759,567 Continuation US8667174B2 (en) | 2007-12-19 | 2013-02-05 | Method and system for survival of data plane through a total control plane failure |
Publications (2)
Publication Number | Publication Date |
---|---|
US20090164835A1 true US20090164835A1 (en) | 2009-06-25 |
US8396988B2 US8396988B2 (en) | 2013-03-12 |
Family
ID=40790106
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US12/002,873 Expired - Fee Related US8396988B2 (en) | 2007-12-19 | 2007-12-19 | Method and system for survival of data plane through a total control plane failure |
US13/759,567 Expired - Fee Related US8667174B2 (en) | 2007-12-19 | 2013-02-05 | Method and system for survival of data plane through a total control plane failure |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/759,567 Expired - Fee Related US8667174B2 (en) | 2007-12-19 | 2013-02-05 | Method and system for survival of data plane through a total control plane failure |
Country Status (1)
Country | Link |
---|---|
US (2) | US8396988B2 (en) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120204222A1 (en) * | 2009-10-16 | 2012-08-09 | Nokia Siemens Networks Oy | Privacy policy management method for a user device |
EP2680515A1 (en) * | 2012-06-30 | 2014-01-01 | Juniper Networks, Inc. | Selective bgp graceful restart in redundant router deployments |
US20140082216A1 (en) * | 2012-09-14 | 2014-03-20 | Cisco Technology, Inc. | Performing offline bgp prefix origin and path validation at route reflectors |
US10659341B2 (en) * | 2018-06-28 | 2020-05-19 | Paypal, Inc. | System for dynamic election of route reflectors |
CN111835636A (en) * | 2019-06-02 | 2020-10-27 | 北京航空航天大学 | Network topology reconstruction method based on trunk path recursive expansion |
Families Citing this family (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN105024844B (en) * | 2014-04-30 | 2019-01-01 | 中国电信股份有限公司 | A kind of method calculating cross-domain routing, server and system |
CN104270307B (en) * | 2014-09-28 | 2018-07-24 | 新华三技术有限公司 | A kind of method for building up and equipment of bgp neighbor |
CN105703996A (en) * | 2014-11-25 | 2016-06-22 | 中兴通讯股份有限公司 | Media access control address convergence method and media access control address convergence device |
US10630585B2 (en) * | 2015-04-16 | 2020-04-21 | Arista Networks, Inc. | Method and system for withdrawing programmed routes in network devices |
Citations (31)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6339595B1 (en) * | 1997-12-23 | 2002-01-15 | Cisco Technology, Inc. | Peer-model support for virtual private networks with potentially overlapping addresses |
US20030012145A1 (en) * | 2001-07-13 | 2003-01-16 | Nigel Bragg | Routing for a communications network |
US6654882B1 (en) * | 2002-05-24 | 2003-11-25 | Rackspace, Ltd | Network security system protecting against disclosure of information to unauthorized agents |
US20040081154A1 (en) * | 2002-10-28 | 2004-04-29 | Cisco Technology, Inc. | Internal BGP downloader |
US6744739B2 (en) * | 2001-05-18 | 2004-06-01 | Micromuse Inc. | Method and system for determining network characteristics using routing protocols |
JP2004228885A (en) * | 2003-01-22 | 2004-08-12 | Ntt Docomo Inc | Mobile communication system, mobile terminal and multicast communication control method |
EP1463247A2 (en) * | 2003-03-28 | 2004-09-29 | NTT DoCoMo, Inc. | Mobile communication system, mobile terminal, and mobile communication method |
US6813644B1 (en) * | 1998-11-18 | 2004-11-02 | Nortel Networks Limited | Distribution of reachability information in data virtual private networks |
US6865611B1 (en) * | 2000-11-27 | 2005-03-08 | Nortel Networks Limited | Communications system, apparatus and method therefor |
US6963575B1 (en) * | 2000-06-07 | 2005-11-08 | Yipes Enterprise Services, Inc. | Enhanced data switching/routing for multi-regional IP over fiber network |
US20060083215A1 (en) * | 2004-10-19 | 2006-04-20 | James Uttaro | Method and apparatus for providing a scalable route reflector topology for networks |
US20060092940A1 (en) * | 2004-11-01 | 2006-05-04 | Ansari Furquan A | Softrouter protocol disaggregation |
US20060140136A1 (en) * | 2004-12-29 | 2006-06-29 | Clarence Filsfils | Automatic route tagging of BGP next-hop routes in IGP |
US20060182038A1 (en) * | 2005-02-15 | 2006-08-17 | Gargi Nalawade | Adaptive timing of update messages transmitted by routers employing the border gateway protocol |
US20060233181A1 (en) * | 2005-04-13 | 2006-10-19 | Robert Raszuk | Method and apparatus for accelerating border gateway protocol convergence |
US20060245374A1 (en) * | 2005-04-28 | 2006-11-02 | Keyur Patel | Method to scale hierarchical route reflectors using automated outbound route filtering-list mechanism |
US20060256724A1 (en) * | 2005-05-10 | 2006-11-16 | Luca Martini | Method of determining transit costs across autonomous systems |
WO2007003088A1 (en) * | 2005-07-06 | 2007-01-11 | Huawei Technologies Co., Ltd. | A method and system for updating a route |
US20070086461A1 (en) * | 2005-10-17 | 2007-04-19 | Ward David D | Method for recovery of a controlled failover of a border gateway protocol speaker |
US20070091796A1 (en) * | 2005-10-20 | 2007-04-26 | Clarence Filsfils | Method of implementing a backup path in an autonomous system |
US20070112975A1 (en) * | 2002-10-02 | 2007-05-17 | Christian Cassar | Redirecting network traffic through a multipoint tunnel overlay network using distinct network address spaces for the overlay and transport networks |
US20070162614A1 (en) * | 2006-01-12 | 2007-07-12 | Keyur Patel | Method and apparatus for achieving border gateway protocol convergence using alternate route information |
US20070183409A1 (en) * | 2006-02-07 | 2007-08-09 | Broadcom Corporation, A California Corporation | Facsimile servicing via peer-to-peer internet protocol telephony network |
US20070214280A1 (en) * | 2006-03-09 | 2007-09-13 | Patel Keyur P | Backup BGP paths for non-multipath BGP fast convergence |
US20080062986A1 (en) * | 2006-09-08 | 2008-03-13 | Cisco Technology, Inc. | Providing reachability information in a routing domain of an external destination address in a data communications network |
US20080089231A1 (en) * | 2006-10-13 | 2008-04-17 | Chandrashekhar Appanna | Suspending operation of Border Gateway Protocol |
US7688714B2 (en) * | 2004-04-28 | 2010-03-30 | Cisco Technology, Inc. | Network routing apparatus that performs soft graceful restart |
US7822027B2 (en) * | 2006-10-05 | 2010-10-26 | Cisco Technology, Inc. | Network routing to the socket |
US7860115B1 (en) * | 2003-12-18 | 2010-12-28 | Cisco Technology, Inc. | Withdrawing multiple advertised routes based on a single tag which may be of particular use in border gateway protocol |
US8179905B1 (en) * | 2006-09-27 | 2012-05-15 | At&T Intellectual Property Ii, L.P. | Method and apparatus for providing communication for virtual private networks |
US8238338B2 (en) * | 2007-09-14 | 2012-08-07 | Cisco Technology, Inc. | Interior gateway protocol summarization preserving internet protocol reachability information |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7307990B2 (en) * | 1999-01-19 | 2007-12-11 | Cisco Technology, Inc. | Shared communications network employing virtual-private-network identifiers |
EP1331772B1 (en) | 2002-01-24 | 2006-03-29 | Alcatel Canada Inc. | Method and apparatus for facilitating routing protocol redundancy in a network element |
US7236453B2 (en) | 2002-06-27 | 2007-06-26 | Jeremy Benjamin, Trustee | High available method for border gateway protocol version 4 |
JP4055955B2 (en) * | 2004-03-05 | 2008-03-05 | Kddi株式会社 | Route failure type identification method |
US8064467B2 (en) * | 2005-02-04 | 2011-11-22 | Level 3 Communications, Llc | Systems and methods for network routing in a multiple backbone network architecture |
US7865615B2 (en) * | 2006-05-08 | 2011-01-04 | Cisco Technology, Inc. | Maintaining IGP transparency of VPN routes when BGP is used as a PE-CE protocol |
US8160056B2 (en) * | 2006-09-08 | 2012-04-17 | At&T Intellectual Property Ii, Lp | Systems, devices, and methods for network routing |
-
2007
- 2007-12-19 US US12/002,873 patent/US8396988B2/en not_active Expired - Fee Related
-
2013
- 2013-02-05 US US13/759,567 patent/US8667174B2/en not_active Expired - Fee Related
Patent Citations (31)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6339595B1 (en) * | 1997-12-23 | 2002-01-15 | Cisco Technology, Inc. | Peer-model support for virtual private networks with potentially overlapping addresses |
US6813644B1 (en) * | 1998-11-18 | 2004-11-02 | Nortel Networks Limited | Distribution of reachability information in data virtual private networks |
US6963575B1 (en) * | 2000-06-07 | 2005-11-08 | Yipes Enterprise Services, Inc. | Enhanced data switching/routing for multi-regional IP over fiber network |
US6865611B1 (en) * | 2000-11-27 | 2005-03-08 | Nortel Networks Limited | Communications system, apparatus and method therefor |
US6744739B2 (en) * | 2001-05-18 | 2004-06-01 | Micromuse Inc. | Method and system for determining network characteristics using routing protocols |
US20030012145A1 (en) * | 2001-07-13 | 2003-01-16 | Nigel Bragg | Routing for a communications network |
US6654882B1 (en) * | 2002-05-24 | 2003-11-25 | Rackspace, Ltd | Network security system protecting against disclosure of information to unauthorized agents |
US20070112975A1 (en) * | 2002-10-02 | 2007-05-17 | Christian Cassar | Redirecting network traffic through a multipoint tunnel overlay network using distinct network address spaces for the overlay and transport networks |
US20040081154A1 (en) * | 2002-10-28 | 2004-04-29 | Cisco Technology, Inc. | Internal BGP downloader |
JP2004228885A (en) * | 2003-01-22 | 2004-08-12 | Ntt Docomo Inc | Mobile communication system, mobile terminal and multicast communication control method |
EP1463247A2 (en) * | 2003-03-28 | 2004-09-29 | NTT DoCoMo, Inc. | Mobile communication system, mobile terminal, and mobile communication method |
US7860115B1 (en) * | 2003-12-18 | 2010-12-28 | Cisco Technology, Inc. | Withdrawing multiple advertised routes based on a single tag which may be of particular use in border gateway protocol |
US7688714B2 (en) * | 2004-04-28 | 2010-03-30 | Cisco Technology, Inc. | Network routing apparatus that performs soft graceful restart |
US20060083215A1 (en) * | 2004-10-19 | 2006-04-20 | James Uttaro | Method and apparatus for providing a scalable route reflector topology for networks |
US20060092940A1 (en) * | 2004-11-01 | 2006-05-04 | Ansari Furquan A | Softrouter protocol disaggregation |
US20060140136A1 (en) * | 2004-12-29 | 2006-06-29 | Clarence Filsfils | Automatic route tagging of BGP next-hop routes in IGP |
US20060182038A1 (en) * | 2005-02-15 | 2006-08-17 | Gargi Nalawade | Adaptive timing of update messages transmitted by routers employing the border gateway protocol |
US20060233181A1 (en) * | 2005-04-13 | 2006-10-19 | Robert Raszuk | Method and apparatus for accelerating border gateway protocol convergence |
US20060245374A1 (en) * | 2005-04-28 | 2006-11-02 | Keyur Patel | Method to scale hierarchical route reflectors using automated outbound route filtering-list mechanism |
US20060256724A1 (en) * | 2005-05-10 | 2006-11-16 | Luca Martini | Method of determining transit costs across autonomous systems |
WO2007003088A1 (en) * | 2005-07-06 | 2007-01-11 | Huawei Technologies Co., Ltd. | A method and system for updating a route |
US20070086461A1 (en) * | 2005-10-17 | 2007-04-19 | Ward David D | Method for recovery of a controlled failover of a border gateway protocol speaker |
US20070091796A1 (en) * | 2005-10-20 | 2007-04-26 | Clarence Filsfils | Method of implementing a backup path in an autonomous system |
US20070162614A1 (en) * | 2006-01-12 | 2007-07-12 | Keyur Patel | Method and apparatus for achieving border gateway protocol convergence using alternate route information |
US20070183409A1 (en) * | 2006-02-07 | 2007-08-09 | Broadcom Corporation, A California Corporation | Facsimile servicing via peer-to-peer internet protocol telephony network |
US20070214280A1 (en) * | 2006-03-09 | 2007-09-13 | Patel Keyur P | Backup BGP paths for non-multipath BGP fast convergence |
US20080062986A1 (en) * | 2006-09-08 | 2008-03-13 | Cisco Technology, Inc. | Providing reachability information in a routing domain of an external destination address in a data communications network |
US8179905B1 (en) * | 2006-09-27 | 2012-05-15 | At&T Intellectual Property Ii, L.P. | Method and apparatus for providing communication for virtual private networks |
US7822027B2 (en) * | 2006-10-05 | 2010-10-26 | Cisco Technology, Inc. | Network routing to the socket |
US20080089231A1 (en) * | 2006-10-13 | 2008-04-17 | Chandrashekhar Appanna | Suspending operation of Border Gateway Protocol |
US8238338B2 (en) * | 2007-09-14 | 2012-08-07 | Cisco Technology, Inc. | Interior gateway protocol summarization preserving internet protocol reachability information |
Non-Patent Citations (2)
Title |
---|
Baykal et al., "An IP multicast handoff scheme with focus on IGMP sourced latency," High Speed Networks and Multimedia Communications 5th IEEE International Conference on , vol., no., pp. 361- 364, 2002. * |
Hardjono, et al., "Key establishment for IGMP authentication in IP multicast," Universal Multiservice Networks, 2000. ECUMN 2000. 1st European Conference on , vol., no., pp.247-252, 2000. * |
Cited By (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120204222A1 (en) * | 2009-10-16 | 2012-08-09 | Nokia Siemens Networks Oy | Privacy policy management method for a user device |
US9794268B2 (en) * | 2009-10-16 | 2017-10-17 | Nokia Solutions And Networks Oy | Privacy policy management method for a user device |
EP2680515A1 (en) * | 2012-06-30 | 2014-01-01 | Juniper Networks, Inc. | Selective bgp graceful restart in redundant router deployments |
US9178797B2 (en) | 2012-06-30 | 2015-11-03 | Juniper Networks, Inc. | Selective BGP graceful restart in redundant router deployments |
US20140082216A1 (en) * | 2012-09-14 | 2014-03-20 | Cisco Technology, Inc. | Performing offline bgp prefix origin and path validation at route reflectors |
US9338080B2 (en) * | 2012-09-14 | 2016-05-10 | Cisco Technology, Inc. | Performing offline BGP prefix origin and path validation at route reflectors |
US10659341B2 (en) * | 2018-06-28 | 2020-05-19 | Paypal, Inc. | System for dynamic election of route reflectors |
US11245613B2 (en) | 2018-06-28 | 2022-02-08 | Paypal, Inc. | System for dynamic election of route reflectors |
CN111835636A (en) * | 2019-06-02 | 2020-10-27 | 北京航空航天大学 | Network topology reconstruction method based on trunk path recursive expansion |
Also Published As
Publication number | Publication date |
---|---|
US8667174B2 (en) | 2014-03-04 |
US8396988B2 (en) | 2013-03-12 |
US20130151445A1 (en) | 2013-06-13 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10541905B2 (en) | Automatic optimal route reflector root address assignment to route reflector clients and fast failover in a network environment | |
US8667174B2 (en) | Method and system for survival of data plane through a total control plane failure | |
CN107409092B (en) | Optimal path selection method and system for optimal route reflection optimized border gateway protocol | |
US10057158B2 (en) | Control device discovery in networks having separate control and forwarding devices | |
US8824334B2 (en) | Dynamic shared risk node group (SRNG) membership discovery | |
US7978708B2 (en) | Automatic route tagging of BGP next-hop routes in IGP | |
US7436838B2 (en) | Automatic prioritization of BGP next-hop in IGP | |
US8537840B1 (en) | Angular distance calculation for BGP best path selection | |
EP3399703B1 (en) | Method for implementing load balancing, apparatus, and network system | |
US9647928B2 (en) | OSPF point-to-multipoint over broadcast or NBMA mode | |
RU2521092C2 (en) | Ldp and igp synchronisation for broadcast networks | |
US7702765B1 (en) | Techniques for automatically creating an iBGP mesh | |
CN112398740A (en) | Link state routing protocol adjacency state machine | |
Cisco | Configuring OSPF | |
CN114301824A (en) | Neighbor discovery for border gateway protocol in multi-access networks | |
CN111245716A (en) | Inter-domain routing method, device and system |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: AT&T LABS, INC.,TEXAS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:UTTARO, JAMES;REEL/FRAME:020646/0257 Effective date: 20080118 Owner name: AT&T LABS, INC., TEXAS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:UTTARO, JAMES;REEL/FRAME:020646/0257 Effective date: 20080118 |
|
FEPP | Fee payment procedure |
Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Free format text: PAYER NUMBER DE-ASSIGNED (ORIGINAL EVENT CODE: RMPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
REMI | Maintenance fee reminder mailed | ||
LAPS | Lapse for failure to pay maintenance fees | ||
STCH | Information on status: patent discontinuation |
Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362 |
|
FP | Lapsed due to failure to pay maintenance fee |
Effective date: 20170312 |