US20080314979A1

US20080314979A1 - Cable management systems

Info

Publication number: US20080314979A1
Application number: US12/015,934
Authority: US
Inventors: Bjorn Dag Johnsen; Ola Torudbakken; Hans Ludvig Opheim; Andreas Bechtolsheim
Original assignee: Sun Microsystems Inc
Current assignee: Sun Microsystems Inc
Priority date: 2007-06-22
Filing date: 2008-01-17
Publication date: 2008-12-25
Also published as: EP2034410A3; US20080316718A1; EP2018071B1; EP2018071A2; US8116332B2; EP2018071A3; US20080318465A1; US20080315985A1; US20080320117A1; US8516089B2; US20080317024A1; US7766692B2; US8015458B2; US7907624B2; EP2034410A2; EP2034410B1; US7850260B2; US20080315889A1; US20080317025A1

Abstract

In order to monitoring connectivity of cabling in a system comprising chassis components and cables interconnecting the chassis components, machine readable labels are applied to cable connectors and chassis component connectors, the machine readable labels on a cable connector and a chassis component to which the cable connector is connected are scanned, and the scanned connectivity information is recorded.

Description

RELATED APPLICATIONS

This application hereby claims priority under 35 U.S.C. §119 to U.S. Provisional Patent Application No. 60/945,778, filed on 22 Jun. 2007, entitled “COMMUNICATION SYSTEMS”, by inventor(s) Bjorn Johnsen et al. The present application hereby incorporates by reference the above-referenced provisional patent application.

BACKGROUND

The invention relates to the handling of cables in cable-based systems in communications systems, for example in cluster configurations and data-centers.
In large systems in particular, the handling of cables can present challenges both in terms of inventory handling and also in terms of system construction, re-configuration as well as trouble-shooting and diagnostics.
The present invention seeks to address such challenges.

SUMMARY

An aspect of the invention provides a method of monitoring connectivity of cabling in a system comprising chassis components and cables interconnecting the chassis components. The method comprises applying machine readable labels to cable connectors and chassis component connectors, scanning the machine readable labels on a cable connector and a chassis component to which the cable connector is connected and recording the scanned connectivity information.
Other aspects of the invention provide a machine readable medium comprising program instructions operable to control one or more processors to implement such a method and a hand held device operable to monitor connectivity of cabling by implementing such a method.
An aspect of the invention also provides a system comprising a plurality of chassis components having connectors for receiving cable connectors and a plurality of cables having cable connectors, the chassis component connectors and the cable connectors being provided with machine readable labels.
Although various aspects of the invention are set out in the accompanying independent and dependent claims, other aspects of the invention include any combination of features from the described embodiments and/or the accompanying dependent claims, possibly with the features of the independent claims, and not solely the combinations explicitly set out in the accompanying claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Specific embodiments are described by way of example only with reference to the accompanying Figures in which:

FIG. 1 is a schematic representation of the rear of an example switch chassis;

FIG. 2 is a schematic representation of the front of the example switch chassis;

FIG. 3 is a schematic representation of a midplane illustrating the logical connectivity through the midplane between cards at the rear and cards at the front orientated orthogonally with respect to each other;

FIG. 4A is a schematic diagram of an example management infrastructure;

FIG. 4B continues the schematic diagram of FIG. 4A;

FIGS. 5 to 11 are views of an example of a switch chassis;

FIG. 12 is a first isometric view of an example of a midplane;

FIG. 13 is a further isometric view of an example of a midplane;

FIG. 14 is an isometric view of an example of a line card;

FIG. 15 is an isometric view of an example of a fabric card;

FIG. 16 is schematic representations of part of a switch chassis;

FIG. 17 is a further schematic representation of part of a switch chassis;

FIG. 18 is a schematic representation of the connections of two cards orthogonally with respect to each other;

FIG. 19 is a schematic representation of an example of orthogonally arranged connectors;

FIG. 20 is a schematic side view of one of the connectors of FIG. 19;

FIG. 21 is a plan view of an example configuration of vias for the orthogonal connector pairing of FIG. 19;

FIG. 22 is a cross-section through of a via;

FIG. 23 is a schematic side view of example of an alternative to the connector of FIG. 20;

FIG. 24 is a schematic end view of an example cable connector;

FIG. 25 is a schematic side view of the example cable connector;

FIG. 26 represents a footprint of the cable connector;

FIGS. 27 and 28 illustrates example of signal routing for a cable connector;

FIG. 29 illustrates an example of a power supply for the cable connector;

FIG. 30 illustrates an example of cable status sense detection circuitry;

FIG. 31 illustrates an example of hot plug control circuitry;

FIG. 32 is a schematic representation of airflow though a switch chassis;

FIG. 33 illustrates an example cable connector and chassis connector configuration; and

FIG. 34 is a flow diagram illustrating an example method for monitoring connectivity of cabling in a system.

While the invention is susceptible to various modifications and alternative forms, specific embodiments are shown by way of example in the drawings and are herein described in detail. It should be understood, however, that drawings and detailed description thereto are not intended to limit the invention to the particular form disclosed, but on the contrary, the invention is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the present invention.

DETAILED DESCRIPTION

An example embodiment of a 3456-port InfiniBand 4×DDR switch in a custom rack chassis is described, with the switch architecture being based upon a 5-stage CLOS fabric. The rack chassis can form a switch enclosure.
The CLOS network, first described by Charles Clos in 1954, is a multi-stage fabric built from smaller individual switch elements that provides full-bisectional bandwidth for all end points, assuming effective dispersive routing.
Given that an external connection (copper or fiber) costs several times more per port than the silicon cost, the key to make large CLOS networks practical is to minimize the number of external cables required and to maximize the number of internal interconnections. This reduces the cost and increases the reliability. For example, a 5-stage fabric constructed with switching elements of size (n) ports supports (n*n/2*n/2) edge points, using (5*n/2*n/2) switch elements with a total of (3*n*n/2*n/2) connections. The ratio of total to external connections is 5:1, i.e. 80% of all connections can be kept internal. The switch elements (switch chips) in the described example can be implemented using a device with 24 4×DDR ports.
An example switch uses a connector that support 3 4× ports per connector, which can further to minimize a number of cables needed. This can provides a further 3:1 reduction in the number of cables. In a described example, only 1152 cables (⅓*n*n/2*n/2) are required.
In contrast if prior commercially available 288-port switches and 24-port switches were used to create a 3456-port fabric a total of 6912 cables (2*n*n/2*n/2) would be required.
The example switch can provide a single chassis that can implement a 5-stage CLOS fabric with 3456 4×DDR ports. High density external interfaces can be provided, including fiber, shielded copper, fiber and twisted pair copper. The amount of cabling can be reduced by 84.4% when compared to building a 3456-port fabric with commercially available 24-port and 288-port switches. In the present example, an orthogonal midplane design can be provided that is capable of DDR data rates.
An example switch can address a full range of HPC cluster computing from a few hundred to many thousand of nodes with a reliable and cost-effective solution that uses fewer chassis and cables than prior solutions.
FIGS. 1 and 2 are schematic diagrams of an example of a switch chassis as viewed from the rear (FIG. 1) and front (FIG. 2), respectively. This example comprises a custom rack chassis 10 that is 60″ high, 47″ wide, and 36″ deep, not including a cable management system. The present example provides a passive orthogonal midplane design (not shown in FIGS. 1 and 2) that provides a direct interface between Line Cards (LC) 12 and Fabric Cards (FC) 14. The line cards provide connections to external lines and the fabric card form switch fabric cards for providing switching functions.
In the present example, up to 18 fabric cards (FC0 to FC17) 12, FIG. 1 are provided. Each fabric card 12 plugs vertically into the midplane from the rear.
In the present example, up to 24 line cards (LC0 to LC23) 14, FIG. 2 can be provided. Each line card provides 144 4× ports (24 stacked 168-circuit cable connectors). Each line card plugs horizontally into the midplane from the front.
Up to 16 hot-pluggable power supply units (PS0-PS16) 16, FIG. 1 are each plugged into the chassis 10 from the rear. Each power supply unit 16 has an alternating current (AC) power supply inlet (not shown). The power supply units 16 plug into a power distribution board (PDB), which is not shown in FIGS. 1 and 2. Two busbars (not shown in FIGS. 1 and 2), one per group of 8 power supply units, distribute direct current (DC) supply to the line cards 12 and the fabric cards 14.
Two hot-pluggable Chassis Management Controllers (CMCs) 18, FIG. 2 plug into the power distribution board from the front. Each chassis management controller 18 comprises a mezzanine card.
The power distribution board is a passive power distribution board that supports up to 16 power supply units DC connectors and 2 chassis management controller slot connectors. The power distribution board connects to the midplane through ribbon cables that carry low-speed signals.
In the present example, up to 144 fan modules (Fan#0-Fan#143) 20 are provided, with 8 fan modules per fabric card 12 in the present instance. Cooling airflow in controlled to be from the front to the rear, using redundant fans on the fabric cards to pull the air from the line cards 14 through openings (not shown in FIGS. 1 and 2), in the midplane. The power supply units 16 have their own fans for cooling with the air exiting through the rear of the chassis. The power supply units 18 are also used to cool the chassis management controllers 18.
FIG. 3 is a schematic representation of a printed circuit board 30, which is configured as a midplane 30 in the switch chassis 10. The midplane 30 is configured in an orthogonal manner such that each fabric card 12 can connect to each of the line cards 14 without requiring any signal traces on the midplane 30. The orthogonal midplane design can provide excellent signal integrity in excess of 10 Gbps per differential pair.
The midplane 30 is represented schematically to show an array of midplane connector pairs 32 as black squares with ventilation openings shown as white rectangles. Each midplane connector pair 32 comprises a pair of connectors (to be explained in more detail later) with one connector on a first face of the midplane and a second connector on the other face of the midplane, the first and second connectors being electrically interconnected by way of pass-through vias (not shown in FIG. 3) formed in the midplane 30. As will be explained later, the first and second connectors of a midplane connector pair 32 are each multipath connectors. They are arranged orthogonally with respect to one another such that a first midplane connector of a midplane connector pair 32 is connectable to a fabric card 12 on a first side of the plane 30 in a first orientation and a second midplane connector of the midplane connector pair 32 is connectable to a line card on a second side of the plane 30 in a second orientation substantially orthogonally to the first orientation.
In an example described herein, each of the first connectors of the respective midplane connector pairs 32 of a column 31 of midplane connector pairs 32 can be connected to one fabric card 12. This can be repeated column by column for successive fabric cards 12. In an example described herein, each of the second connectors of the respective midplane connector pairs 32 of a row 33 of midplane connector pairs 32 can be connected to one line card 14. This can be repeated row by row for successive line cards 14. As a result, the midplane can be populated by vertically oriented fabric cards 12 on the first side of the midplane and horizontally orientated line cards 12 on the second side of the midplane 30.
In the present example the midplane 30 provides orthogonal connectivity between fabric cards 12 and the line cards 14 using orthogonal connector pairs. Each orthogonal connector pair provides 64 differential signal pairs, which is sufficient to carry the high-speed signals needed as well as a number of low-speed signals. The orthogonal connector pairs are not shown in FIG. 3, but are described later.
The midplane 30 is also configured to provide 3.3 VDC standby power distribution to all cards and to provide I2 C/System Management Bus connections for all fabric cards 12 and line cards 14.
Another function of the midplane 30 is to provide thermal openings for a front-to-rear airflow. The white holes in FIG. 3 (e.g., hole 34) form openings 34 in the midplane for airflow. In this example the midplane is approximately 50% open for airflow.
The fabric cards 12 each support 24 connectors and the line cards 14 each support 18 connectors.
FIG. 3 also illustrates an example of how the fabric cards 12, the midplane 20 and the line cards 14 interconnect. In this example there are 24 switch chips on a line card 14 and 8 chips on each of the 18 fabric cards 12.
As previously mentioned a 5-stage Clos fabric has a size n*n/2*n/2 in which n is the size of the switch element. The example switch element in FIG. 3 has n equal to 24 ports. Each line card 14 has 24 chips in 2 rows with 12 chips in each row. Each of 12 ports of each switch chip 35 in a first row 36 of the line card 14 is connected to 2 cable connectors 42, with 6 ports per cable connector. There are a total of 24 cable connectors per line card 14. Each cable connector can accommodate two physical independent cables that each carries 3 ports (links). Each cable connector 42 can accommodate 6 ports. The remaining 12 ports of each switch chip 35 in the first row 26 is connected to one chip 35 each in a second row 38 of chips 35.
There are 18 midplane connectors 32 per line card 14. Each midplane connector 32 provides one physical connection to one fabric card 14. Each midplane connector 32 can accommodate 8 4× links (there are 8 differential pairs per 4× link and a total of 64 differential pairs provided by the orthogonal connector)
12 ports of each of the switch chips 35 in the second row 38 of the line card 14 are connected to 2 line card connectors 40 that are used to connect the line card 14 to the midplane connectors 32 and thereby with the fabric cards 12 through the orthogonally oriented midplane connector pair. Of the 12 ports per switch chip 35, eight ports are connected to one line card connector 40, and the remaining four ports are connected to another line card connector 40 as represented by the numbers 8 and 4 adjacent the two left hand switch chips 35 in the second row 38. 2 switch chips are thereby connected to a group of 3 line card connectors 40 and hence to a group of three midplane connectors pairs 32.
The remaining 12 ports of each switch chip 35 in the second row 38 of the line card 14 are connected to each of the 12 switch chips 35 in the first row 36 of the line card 14.
At the fabric card 12 all links through an orthogonally oriented midplane connector pair 32 are connected to one line card 14. A single orthogonal connector 46 carries 8 links. These links are connected to one switch element 44 each at the fabric card 12.
Also shown in FIG. 3 are power connectors 37 on the midplane and power connectors 39 on the fabric cards 12.
There has been described a system with 24 line cards with 144 ports each, realized through 48 physical cable connectors that each carry 3 links. The switch fabric structure of each line card 14 is fully connected, so the line card 14 itself can be viewed upon as a fully non-blocking 144 port switch. In addition each line card 14 has 144 links that are connected to 18 fabric cards. The 18 fabric cards then connect all the line cards 14 together in a 5-stage non-blocking Clos topology.
FIGS. 4A and 4B are schematic diagrams of an example management infrastructure. This example provides redundant chassis management controllers 18. In addition each fabric card 12 and line card 14 supports an management controller. There are redundant management connections from each chassis management controller 18 to each of the fabric card and line card management controllers. In addition there are I2C connections to each of the power supply units 16. The management connections pass between the fabric cards 12, the line cards 14, the power supply units 16 and the chassis management cards 18 via the midplane and the power distribution board 22 in the present example.
FIGS. 5 to 11 provide various schematic views of an example of a switch chassis in accordance with the invention.
FIG. 5 is a front view of the switch chassis 10 showing cable management structures 50. FIG. 6 is a rear view of the switch chassis 10 showing the fabric cards 12, the power supply units 16 and cable management structures 50. FIG. 6 is a side view of the switch chassis 10 further showing the cable management structures 50. FIG. 8 is a side view of the switch chassis 10 further showing the cable management structures 50. FIG. 9 is an isometric view of the switch chassis 10 from the line card 14 (front) side further showing the cable management structures 50. FIG. 10 is an isometric view of the switch chassis 10 from the line card 14 (front) side showing four line cards 12 installed horizontally in the chassis 10 and part of the cable management structures 50. FIG. 11 is an isometric view of the switch chassis 10 from the fabric card 12 (rear) side showing four fabric cards 12 installed vertically in the chassis 10 and part of the cable management structures 50.
FIGS. 12 and 13 provide various schematic views of an example of a midplane 30 in accordance with the invention. FIG. 12 is an isometric view of the midplane 30 from the line card 14 (front) side and FIG. 13 is an isometric view of the midplane 30 from the fabric card 12 (rear) side. FIG. 12 shows the array formed from rows and columns of the second connectors 64 of the midplane connectors pairs 32 described with reference to FIG. 3. FIG. 13 shows the array formed from rows and columns of the first connectors 62 of the midplane connectors pairs 32 described with reference to FIG. 3.
FIG. 14 is an isometric view of an example of a line card 14. This shows the first and second rows 36 and 38 of switch chips 35, the line board connectors 40 and the cable connectors 42. As can be seen in FIG. 14, the cable connectors 42 are stacked double connectors such each cable connector can connect to two cables 52 and 54.
FIG. 15 is an isometric view of an example of a fabric card 12. This shows the fabric card connectors 46 and the switch elements 44.
FIG. 16 is a schematic representation of an example of two chassis management controllers 18 plugged into one side of a power distribution board 22 and 16 power supply units 16 plugged into the other side of the power distribution board 22. In the present example, the chassis management controllers 18 are plugged into the front side of the power distribution board 22 and the power supply units 16 are plugged into the rear side of the power distribution board 22 as mounted in the switch chassis. FIG. 17 illustrates bus bars 24 for a 3.3V standby supply.
In the present example the midplane 30 is a passive printed circuit board that has dimensions of 1066.8 mm (42″)×908.05 mm (35.75″)×7.1 mm (0.280″). The active area is 40″×34″. 864 8×8 midplane connectors (432 midplane connectors per side) are provided. There is a ribbon cable connection the power distribution board 22 and a 3.3V standby copper bar to the power distribution board 22.
In the present example a fabric card 12 comprises a printed circuit board with dimensions of 254 mm (10″)×1016 mm (40″)×4.5 mm (177″). It comprises 24 8×8 fabric card connectors 46, one power connector 39, 8 fan module connectors and 8 switch chips 44.
In the present example a line card 14 comprises a printed circuit board with dimensions of 317.5 mm (12.5″)×965.2 mm (38″)×4.5 mm (177″). It comprises 24 stacked cable 168- circuit connectors 42, 18 8×8 card connectors 40, 1 busbar connector and 24 switch chips 35.
In the present example a power distribution board 22 comprises a printed circuit board, 16 power supply DC connectors, 14 6×6 card connectors (7 connectors per chassis management card 18, ribbon cable connectors for low-speed connectivity to the midplane 30, and a 3.3V standby copper bar to the midplane 30.
In the present example a chassis management card 18 comprises 14 6×6 card connectors (7 connectors per chassis management card), two RJ45 connectors for Ethernet available on a chassis management card panel, two RJ45 connectors for serial available at the chassis management card panel, .three RJ45 for line card/fabric card debug console access at the chassis management card panel, three HEX rotary switches used to select between which line card/fabric card debug console is connected to the three RJ45s above, and a 220-pin connector for the mezzanine.
In the present example a mezzanine has dimensions: 92.0 mm×50.8 mm and comprises 4 mounting holes screw with either 5 mm or 8 mm standoff from the chassis management card board, a 220-pin connector for connectivity to chassis management board.
FIG. 18 is a schematic isometric view of an example of a midplane connector pair 32. As can be seen in FIG. 18, the connector comprises a first, fabric side, connector 62 and a second, line card side, connector 64. In this example, each of the connector 62 and 64 is substantially U-shaped and comprises an 8×8 array of contact pins.
It will be noted that the second connector 64 of the midplane connector pair 32 is rotated through substantially 90 degrees with respect to the first connector 62. The first connector 62 is configured to connect to a corresponding fabric card connector 46 of a fabric card 12. The second connector 62 is configured to connect to a corresponding fabric card connector 46 of a line card 14. Through the orientation of the second connector 64 of the midplane connector pair 32 substantially orthogonally to the orientation of the first connector 62, it can be seen that the line card 14 is mounted substantially orthogonally to the fabric card 12. In the present example the line card 14 is mounted substantially horizontally and the fabric card is mounted substantially vertically 12.
Each of the contact pins on the connector 62 is electrically connectable to a corresponding contact of the fabric card connector 46. Each of the contact pins on the connector 64 is electrically connectable to a corresponding contact of the line card connector 40. The connector pins of the respective connectors 62 and 64 are connected by means of pass-through vias in the midplane 30 as will now be described in more detail.
FIG. 19 illustrates an example of the configuration of a first midplane connector 62 and a second midplane connector 64 of a midplane connector pair 32 in more detail. In the example shown in FIG. 19 that second connector 64 (the line card side connector) comprises a substantially U-shaped frame 70 including a substantially planar base 71 and first and second substantially planar walls 72 and 74 that extend at substantially at 90 degrees from the base 71. The inside edges of the first and second substantially planar sides 72 and 74 are provided with ridges 76 and grooves 78 that provide guides for the line card connector 40.
As can be seen in FIG. 18, the line card connector 40 has a structure that comprises a plurality of contact planes 63 that are aligned side by side, such that it has a generally planar construction that extends up from the line card 14. Line card connector planes comprise printed circuit boards carrying traces leading to contacts. The traces and contacts can be provided on both sides of the printed circuit boards of the line card connector planes.
By comparing FIGS. 18 and 19, it can be seen that each contact plane 63 of the line card connector 40 can be entered into a respective one of the grooves 78 so that connectors of the line card connector 40 can then engage with contact pins 80 of the second connector 64. In the case of the line card side connector portion 64, the orientation of second connector 64 and the grooves 78 therein means that the line card 12 is supported in a substantially horizontal orientation. In the example shown in FIG. 19, an 8×8 array of connector pins 80 is provided.
The first midplane connector 62 (fabric card side connector) of the midplane connector pair 32 has substantially the same form as the second midplane connector 62 of the midplane connector pair 32, except that it is oriented at substantially 90 degrees to the second midplane connector 64. In this example the second midplane connector 62 comprises a substantially U-shaped support frame 75 including a substantially planar base and first and second substantially walls and that extend at substantially at 90 degrees from the base. The inside edges of the first and second substantially planar sides are provided with ridges and grooves that provide guides for the fabric card connector 46. The fabric card connector 46 has the same basic structure as that of the line card connector 40 in the present instance. Thus, in the same way as for the line card connector, each of a plurality of contact planes of the fabric card connector 46 can be entered into a respective one of the grooves so that connectors of the fabric card connector 46 can then engage with contact pins of the first connector 62. The orientation of the first connector 62 and the grooves therein means that the fabric card 12 is supported in a substantially vertical orientation.
In the example illustrated in FIG. 19, the orthogonal connector 60 provides an 8×8 array of connector pins 80 is provided that can support supports 64 differential pairs or 32 bi-directional serial channels (two wires per direction) in a footprint of 32.2×32.2 mm.
As mentioned above, the contact pins of the first and second midplane connectors 62 and 64 of a midplane connector pair 32 are connected by means of pass through vias in the midplane.
FIG. 20 illustrates a side view of an example of a midplane connector, for example the midplane connector 62 mounted on the midplane. In the example shown in FIG. 20 the midplane connector 64 comprises a substantially U-shaped frame 70 including a substantially planar base 71 and first and second substantially planar walls 72 and 74 that extend at substantially at 90 degrees from the base 71. The contact pins 80 are each connected to pairs of contact tails 81 that are arranged in sprung pairs that are arranged to be push fitted into pass through vias 83 in the midplane 30.
In use, the other midplane connector (e.g., the first midplane 62) of the midplane connector pair would be inserted into the pass through vias in the other side of the midplane 30 in the orthogonal orientation as discussed previously.
FIG. 21 is a schematic representation of an area of the midplane for receiving the midplane connectors 62 and 64 of the midplane connector pair 32. This shows the array of vias 83. FIG. 22 is a schematic cross-section though such a via 83 in the showing the conductive wall 85 of the via 83. The conductive wall 85 can be formed by metal plating the wall of the via, for example.
The examples of the midplane connectors described with reference to FIGS. 18 and 20 had a generally U-shape. However, other configurations for the midplane connectors are possible. For example FIG. 23 illustrates another example of a midplane connector pair 32′, where the first and second midplane connectors 62′ and 64′ are generally the same as the first and second midplane connectors 62 and 64 described with reference to FIG. 19 except that, in addition to the first and second walls 72 and 74, third and fourth walls 73 and 75 are provided. The additional walls provide a generally box-shaped configuration that can facilitate the insertion and support for the cards to be connected thereto.
It will be appreciated that in other examples the first and second midplane connectors could have different shapes and/or configurations appropriate for the connections for the cards to be connected thereto.
The array of midplane connector pairs 32 as described above provides outstanding performance in excess of 10 Gbps over a conventional FR4 midplane because the orthogonal connector arrangements allow signals to pass directly from the line card to the fabric card without requiring any signal traces on the midplane itself. The orthogonal arrangements of the cards that can result from the use of the array of orthogonally arranged connector pairs also avoids the problem of needing to route a large number of signals on the midplane to interconnect line and fabric cards, minimizing the number of layers required. This provides a major simplification compared to existing fabric switches. Thus, by providing an array of such orthogonal connectors, each of a set of horizontally arranged line cards 12 can be connected to each of a set of vertically aligned fabric cards without needing intermediate wiring.
FIGS. 24 and 25 provide an end view and a side view, respectively, of an example of a cable connector 42 as mentioned with reference to FIGS. 3 and 14. As shown in FIGS. 24 and 25, the cable connectors 24 and 25 include first and second cable connections 92 and 94 stacked within a single housing 90. This provides for a very compact design. Board contacts 96 are provided for connecting the connector to a line card 14. FIG. 26 is a plan view of the connector footprint for the board contact s 96 of the cable connector 42. The stacked arrangement facilitates the providing of line cards that are high density line cards supporting a 12× cable providing 24 line pairs with 3 4× links aggregated into a single cable. The cable connectors provide 12× cable connectors that are smaller than a conventional 4× connector, 3× denser than a standard InfiniBand 4× connector and electrically and mechanically superior. Using 12× cable (24 pairs) can be almost 50% more area efficient than three 4× cables and requires three times fewer cables to install and manage.
FIGS. 27 and 28 illustrate an example of the routing of signals from each of two 12× port sections 92 and 94 of a cable connector 42 to the equalizers and to a switch chip on a line card 14. FIG. 27 shown an example of routing from a first 12× port section. FIG. 28 shows an example of the routing from a second 12× port section. The transmit (Tx) lines are equalized, and can be connected directly from the switch chip to the cable connector. The can be routed on lower layers in order to minimize via stub effects.
FIG. 29 illustrates an example of a power supply for the cable connector and FIG. 30 illustrates an example of a cable status sense detection circuitry. The cable sense detection circuitry is operable to test from each end whether the other end is plugged or not, and, if plugged, to see if power from the power supply is on. Provisions are made such that “leaking” power from a powered to un-powered end is avoided. A valid status assumes that an active end is plugged. FIG. 31 is a schematic diagram of an example of a hot plug control circuit that enables hot plugging of cables. The switch chassis can thereby provide active cable support for providing active signal restoration at a cable connector. Active cable support can provides benefits of increased distances for copper cables as a result of active signal restoration at the connector, increased maximum cable distance by over 50%, using thinner and more flexible cables (e.g., reducing a cable diameter by up to 30%, which facilitates good cable management. A cable to connector interface can provide one, more or all of local and remote cable insertion detection, cable length indication, remote node power-on detection, remote power, a serial number and a management interface.
FIG. 32 is a schematic representation of the airflow through an example switch chassis. As illustrated by the arrows, the airflow is from the front to the rear, being drawn through by fans 20 in the fabric cards 12 and the power supplies 18.
The air inlet is via perforations at the line card 14 front panel. Fans 20 at the fabric cards 12 pull air across the line cards, though the openings 34 in the vertical midplane 30 and across the fabric cards 12.
Line card cooling is naturally redundant since the fabric cards are orientate orthogonally to the line cards. In other words, cooling air over each line card is as a result of the contribution of the effect of the fans of the fabric cards along the line card due to the respective orthogonal alignment. In the case that a fabric card fails or is removed, a portion of the cooling capacity is lost. However, as the cooling is naturally redundant the line cards will continue to operated and be cooled by the remaining fabric cards. Each fan is internally redundant and the fans on the fabric cards 12 can be individually hot swappable without removing the fabric card 12 itself. The fabric card 12 and line card 14 slots can be provided with blockers to inhibit reverse airflow when a card is removed. Empty line card 14 and fabric card 12 slots can be loaded with filler panels that prevent air bypass.
Each power supply has an internal fan that provides cooling for each power supply. Fans at the power supplies pull air through chassis perforations at the rear, across the chassis management cards 18, and through the power supply units 16. Chassis management card cooling is naturally redundant as multiple power supply units cool a single the chassis management card.
It will be appreciated that changes and modifications to the above described examples are possible. For example, although in the present example cooling if provided by drawing air from the front to the rear, in another example cooling could be from the rear to the front.
Also, although in the above described examples the fabric cards and the switch cards are described as being orthogonal to each other, they do not need to be exactly orthogonal to each other. Indeed, in an alternative example they could be angled with respect to each other but need not be exactly orthogonal to each other.
Also, in the above described examples the midplane connector pairs 32 are configured as first and second connectors 62 and 64, in another example they could be configured as a single connector that is assembled in the midplane. For example, through connectors could be provided that extend through the midplane vias. The through connectors could be manufactured to be integral with a first connector frame (e.g., a U-shaped frame or a box-shaped frame as in FIGS. 19 and 23, respectively) and the contacts inserted through the vias from a first side f the midplane 30. Then a second connector frame could be inserted over the connectors on the second side of the midplane 30 in a mutually orthogonal orientation to the first connector frame.
An example cable-based switch chassis can provide a very large switch having, for example, one or more of the following advantages, namely a 3456 ports non-blocking Clos (or Fat Tree) fabric, a 110 Terabit/sec bandwidth, major improvements in reliability, a 6:1 reduction in interconnect cables versus leaf and core switches, a new connector with superior mechanical design, major improvement in manageability, a single centralized switch with known topology that provides a 300:1 reduction in entities that need to be managed.
In a system such as that described herein, and generally in large cluster configurations and data-centers, the handling of such cables can present challenges both in terms of inventory handling and also in terms of system construction, re-configuration as well as trouble-shooting and diagnostics.
Example of these issues include the following issues A-D. A: How can each end of a single cable be identified reliably when the cable itself is part of a large cable-bundle (e.g., before and after the cable connectors have been attached to any system/chassis).
B: When a link associated with a local switch and/or adapter port is not operational, how can it be determined, for example, if there is a cable connected locally, if the cable is connected at the remote side, what is the state of the remote side (e.g., the remote switch and/or adapter), what is the identity of the remote side (e.g., to which connector and/or port in which chassis is the remote end(s) of the cable connected).
C: How can it be determined if there is there is a history of events and/or problems associated with any individual cable or a particular type of cable or a particular production batch.
D: How can it be determined if there are combinations of systems and/or cables (individuals, types or batches) that seem to have more problems than others.
Issue A mentioned above can to at least some extent be addressed by the use of human readable labels provided on all cable connectors. Issues B-D can to some extent be addressed by rigorous manual maintenance of a cable inventory database (e.g., based on the proper human readable labels). However, all these challenges can in general be classified as generic field replaceable unit (FRU) handling issues. By using cables where the connectors supports “FRUID” information that can be read electronically similar to conventional FRUs (e.g., “cards” etc., that plug into “slots” in a chassis and use I2C type access to ROM/PROM/EEPROM devices on the FRU itself), it is possible to keep track of both location, status and history of individual cables.
However, in an example of this aspect of the invention described herein, a difference from conventional FRU handling is that information is aggregated from more than a single chassis in order to keep track of the history of the complete “FRU”, that is the cable with two (or more) connectors.
The purely manual approach is potentially unreliable due to human error, and it is also very time consuming and not at all “user-friendly”.
The provision of an electrically readable FRUID approach would provide a solution for issues B, C and D (and the connected version of Issue A). However, it does not address the “un-connected” case of issue A. Also, it requires both sides (systems/chassis) to be operational in order to determine connectivity information. Also, implementing “FRUID” (or even only a unique serial number) as an electrically readable attribute of each cable connector is not always feasible from a cost, space or complexity perspective. In particular, in cases where a pure optical cable connector is used, then there may be no (practical) way to “communicate” with the cable connector itself.
An aspect of an invention described herein provides for the utilization of machine readable labels (e.g., small-footprint BAR-code labels) to emulate FRUID handling for cables in large distributed systems. The machine readable labels can be read by a device such as suitably configured mobile device (e.g. a PDA, etc.) with an appropriate scanner.
An example embodiment of this aspect of an invention described herein provides for providing a machine readable indicator label for the relevant FRUID on each cable connector.
In such an example embodiment, providing each cable connector with, for example, a BAR-code label that contains the relevant “FRUID” information (in particular a unique serial number) can address the human error aspects of keeping track of information that is only available as human readable text. As with other BAR-code usage, this BAR-code can very well be combined with human readable information that can be site specific, and/or can be a production feature of the cable (e.g., the serial number).
In order to fully address the short-comings of the electrically readable FRUID approach, it is desirable to be able reliably to associate a cable connector with which the chassis connector is attached. In principle, this can be done in various “semi-manual” ways,—e.g., to scan the BAR-code ID of a line-card FRU, and then scan the BAR-codes of the cables that are attached thereto. However, since not all connector positions may be occupied, it is in general difficult to keep track of which position a cable is connected to without manually entering this information as part of the scanning procedure.
Accordingly, in an example embodiment BAR codes can also be associated with each chassis cable-connector. This BAR code can identify a FRUID of a FRU (e.g., a line-card) with which it is associated, and also a corresponding connector position/number (e.g., a “port number”). In this way, it is possible to conduct scan-sequences that very reliably keep track of cable/position combinations. For example, the machine readable label of a cable connector can comprise a machine readable code that is independent of a chassis component connector with which it is associated. A label on a chassis component can include information to facilitate verification of a complete and/or correct order of scanning. A order of interleaving of cable and chassis component labels during a scan can determine the association of cables and chassis components and/or an empty chassis connector where the cable and chassis component labels are not read in a single atomic scan (i.e. an individual scan operation).
By making use of 2-dimensional BAR codes, it is possible to reduce the size of the labels to only a few millimeters in each dimension. Label locations can be provided on the cable connectors and be associated with the chassis connectors so that the labels on the cable connector as well as the related chassis connector can be read either as pairs (e.g., both labels are read in a single read operation), or in a defined sequence. In the case of “pair” readings, a scan can identify either empty connector positions on the switch FRU, or pairs of cable connector and switch FRU plus connector-position, whereby the information retrieved can be very reliable. In the case of a sequence, the information is potentially not quite as reliable. However, by the label locations can be arranged such that a single sequential scan will read the position labels in sequence and, if any cable is attached, then its label will be read between two “position readings”. As a result, the chance of undetected human errors are very small since any incorrect order of position labels can be arranged to cause the scan to be rejected (e.g., by the control software in a suitably configured mobile device (e.g., a PDA, etc.) that is used to control the scanning).
In both cases, chassis side BAR code labels can indicate a “first” and a “last” position so that a scan is always guaranteed to include all positions for the corresponding FRU (e.g., line-card), or to be rejected if not.
FIG. 33 is a schematic representation of a cable connector 306 at an end of a cable 310 for connection to a chassis component connector 314 of a chassis component 302. The cable connector 306 is provided with a two-dimensional BAR code 320 and the chassis component connector 314 is provided with a two-dimensional BAR code 322. The position of the BAR codes 320 and 322 are arranged such that when the cable connector 306 is engaged with the chassis component connector 314, the BAR codes are adjacent such that both BAR codes 320 and 322 can be read in a single action using a suitably configured mobile device 324 provided with a BAR code reader 326, as represented in FIG. 33.
As indicated above, an operator can use a BAR code reader associated with a hand-held mobile device that controls the scans (e.g., of a series of individual BAR code label readings), and is also able to communicate with external management software.
Such management software can both be associated with the specific chassis that is being scanned at the moment (e.g., the mobile device can be connected to the chassis via a network/console cable) or the mobile device can communicate with a central management server via a (secure) wireless network.
The mobile device can be configured to operate strictly as an un-intelligent proxy for the associated management software.
Alternatively it can be configured with application specific built-in logic to be able to operate independently of any external management software, and to record all scans, for example for later transfer to external management software. In this case, logic in the mobile device can receive and store information about how the system is to be configured, and the mobile device itself can be arranged to verify connectivity based on its accumulated knowledge about how the cables in the system is connected. In such an example, there need be no dependency on any of the involved system components needing to be operational, nor upon access to a management server.
If the chassis instances are operational at least at a basic power level, where a chassis management controller is able to communicate via a management network (e.g., communicate via the management network to both the mobile device as well as other relevant management components, for example other chassis management controllers and/or management servers, or via a dedicated link to the mobile device) and also able to control LED indicators, etc., associated with relevant connector positions, then it is also possible to provide guidance (in real time) to an operator when doing (re)cabling of the system or parts thereof.
It is to be noted that in a pure BAR-code based cable approach to identifying the presence and ID of cable instances is that they are only detected and recorded whenever a scan operation is performed. Hence, if cables are moved around without “re-scanning”, then the consistency of the “inventory database” would be lost, and recording of historical data would no longer be consistently associated with specific cable instances.
In addition to the use of standard procedures for always recording any cabling changes with appropriate scan operations, various mechanisms can be used to monitor that consistent information is maintained at all times.
It is to be noted that as long as a link is operational, the same cable has to be present. Accordingly, if the cable connectors support a “presence” status that can be monitored electrically from the chassis side, then it is possible to monitor that the same cable is connected locally as long as the chassis is operating in at least minimal power mode.
As long as a cable is always connected to at least one chassis that is able to monitor cable presence state, then the ID of the cable is inherently known whenever the remote end of the cable is connected again and the corresponding link becomes operational.
Whenever the system looses track of the ID of a cable, then this can generate an operator notification in order to request a re-scan of the “unknown” cable positions. Using the same reasoning as outlined above the system can then determine the point in time where the corresponding cable is known to have been inserted.
Whenever, the inventory database contents indicate that a cable is supposed to be present between two end-points and the link is not operational even if both end-points are operational, then the system can request the operator to investigate the situation.
It should also be noted that as long as BAR-code based labeling of cable connectors (or both cable and chassis connectors) does not impose significant increase in production cost, it is possible to use the BAR-codes as an enhancement to electrically readable FRU information for cables. In particular, the use of BAR-codes on the cables will help keeping track of cables that are not yet connected at both ends.
Another aspect to note is that the BAR-code labeling addresses both conventional cables with two connectors and single cable representing a single link, as well as multi-link cables that may have multiple individual cable segments and multiple connectors that represent either all the links or only a subset of the links.
The BAR-codes on each connector can identify exactly which links are connected so that the system level connectivity is always well defined without bringing any links up.
FIG. 34 is a flow diagram illustrating an example method 400 of monitoring connectivity of cabling in a system comprising chassis components and cables interconnecting the chassis components.
In step 402 machine readable labels applied to cable connectors and chassis component connectors are scanned on a cable connector and a chassis component to which the cable connector is connected.
In step 404, the scanned connectivity information is recorded in memory, either in memory in a hand held scanning device such as the mobile device 324 illustrated in FIG. 33 and/or in a management server.
In one example the machine readable label on the cable connector and the machine readable label on the chassis component connector are relatively located with respect to each when the cable connector is connected to the chassis component connector such that they can both be read in a single scanning operation.
In another example, the machine readable label on the cable connector and the machine readable label on the chassis component connector are relatively located with respect to each when the cable connector is connected to the chassis component connector such that they can be read in a predetermined sequence of scanning operations.
In step 406, the hand held device and/or the management server can be configured to verify an order of scanning of the machine readable labels by comparison to information stored in memory.
In step 408 the hand held device and/or the management server can be configured to reject the scanning operations in the event that the scanning is not performed in a predetermined order. In this case, an indication that the order of scanning is correct can be notified to the user of the hand held or other scanning device that an incorrect order of scanning had been followed. Alternatively, if scanning is performed in the correct order, then in step 410 the scanning operation is accepted. In one example embodiment, a label on a chassis component can include information to facilitate verification of a complete and/or correct order of scanning.
In an example embodiment, an order of interleaving of cable and chassis component labels during a scan, where cable and chassis component labels are not read in a single atomic scan, can be operable to determine at least one of an association of cables and chassis components and an empty chassis connector.
In step 412, if further cables and/or connectors are to be scanned, the process of steps 402-410 can be repeated.
In step 414, on completion of the scanning operations, the resulting recorded scanned information can be used to verify the connectivity of the system, and the recorded information can, for example, be used by a management server for system configuration and/or management.
As described herein, machine readable labels can be applied to cable and chassis connectors in a way that allows scanning operations to record the implied connectivity information without being subject to human errors in terms of multiple independent label readings. In an example embodiment BAR-code based information is used as an indirect way of populating the FRU inventory information associated with one or more chassis instances as well as FRUID information associated with the cable itself is another novel aspect.
Accordingly, the use of machine readable labels provides a simple and reliable way of keeping track of large number of interconnect cables and related connectivity without depending on the operational state of associated systems. It addresses the issues identified above without imposing any additional complexity for the implementation of the cable itself or associated connectors on either the cable or on the connected systems.
An example embodiment can facilitate the provision of a very large switch that can provide, for example one or more of the following advantages, namely a 3456 ports non-blocking Clos (or Fat Tree) fabric, a 110 Terabit/sec bandwidth, major improvements in reliability, a 6:1 reduction in interconnect cables versus leaf and core switches, a new connector with superior mechanical design, major improvement in manageability, a single centralized switch with known topology that provides a 300:1 reduction in entities that need to be managed.
Although the embodiments above have been described in considerable detail, numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated.

Claims

1. A method of monitoring connectivity of cabling in a system comprising chassis components and cables interconnecting the chassis components, the method comprising applying machine readable labels to cable connectors and chassis component connectors, scanning the machine readable labels on a cable connector and a chassis component to which the cable connector is connected, and recording the scanned connectivity information.

2. The method of claim 1, comprising scanning the machine readable labels using a hand held device.

3. The method of claim 1, wherein the machine readable label on the cable connector and the machine readable label on the chassis component connector are relatively located with respect to each when the cable connector is connected to the chassis component connector such that they can both be read in a single scanning operation.

4. The method of claim 1, wherein the machine readable label on the cable connector and the machine readable label on the chassis component connector are relatively located with respect to each when the cable connector is connected to the chassis component connector such that they can be read in a predetermined sequence of scanning operations.

5. The method of claim 3, comprising scanning the machine readable labels using a hand held device, wherein the hand held device is configured to verify an order of scanning of the machine readable labels and to reject the scanning operations in the event that the scanning is not performed in a predetermined order.

6. The method of claim 1, wherein the machine readable label of a cable connector comprises a machine readable code that is independent of a chassis component connector with which it is associated.

7. The method of claim 1, wherein a label on a chassis component includes information to facilitate verification of a complete and/or correct order of scanning.

8. The method of claim 1, wherein an order of interleaving of cable and chassis component labels during a scan, where cable and chassis component labels are not read in a single atomic scan, is operable to determine at least one of an association of cables and chassis components and an empty chassis connector.

9. The method of claim 1, wherein machine readable label comprises a BAR code.

10. The method of claim 9, wherein the BAR code is a two-dimensional BAR code.

11. A machine readable medium comprising program instructions operable to control one or more processors to implement a method of monitoring connectivity of cabling in a system comprising chassis components and cables interconnecting the chassis components, wherein machine readable labels are applied to cable connectors and chassis component connectors, the method comprising receiving data from scanning of the machine readable labels on a cable connector and a chassis component to which the cable connector is connected, and recording the scanned connectivity information.

12. A hand held device operable to monitor connectivity of cabling in a system comprising chassis components and cables interconnecting the chassis components, wherein machine readable labels are applied to cable connectors and chassis component connectors, the hand held device comprising means for scanning the machine readable labels on a cable connector and a chassis component to which the cable connector is connected, and means for recording the scanned connectivity information.

13. The hand held device of claim 12, comprising means for verifying an order of scanning of the machine readable labels and to reject the scanning operations in the event that the scanning is not performed in a predetermined order.

14. A system comprising a plurality of chassis components having connectors for receiving cable connectors and a plurality of cables having cable connectors, the chassis component connectors and the cable connectors being provided with machine readable labels.

15. The system of claim 14, wherein the machine readable label on a cable connector and the machine readable label on a chassis component connector are relatively located with respect to each when the cable connector is connected to the chassis component connector such that they can both be read in a single scanning operation by a hand held scanning device.

16. The system of claim 14, wherein the machine readable label on a cable connector and the machine readable label on a chassis component connector are relatively located with respect to each when the cable connector is connected to the chassis component connector such that they can be read in a predetermined sequence of scanning operations by a hand held scanning device.

17. The system of claim 14, comprising a hand held scanning device configured to verify an order of scanning of the machine readable labels and to reject the scanning operations in the event that the scanning is not performed in a predetermined order.

18. The system of claim 14, wherein the machine readable label of a cable connector comprises a machine readable code that is independent of a chassis component connector with which it is associated.

19. The system of claim 14, wherein a label on a chassis component includes information to facilitate verification of a complete and/or correct order of scanning.

20. The system of claim 14, comprising a hand held scanning device operable to determine from an order of interleaving of cable and chassis component labels during a scan the association of cables and chassis components.

21. The system of claim 14, wherein the machine readable labels comprise a BAR code.

22. The system of claim 21, wherein the BAR code is a two-dimensional BAR code.