Cisco UCS: What’s the maximum bandwidth per blade?

As I already mentioned in this article, Cisco UCS is a complex combination of components. With the release of the M3 type Cisco UCS blades the number of components have increased and so did the complexity. The result is that I receive calls from colleagues/customers on a regular basis, complaining about the amount of bandwidth per Cisco UCS blade.

I already described that, the maximum number of VIF’s depends on the number of uplinks to the chassis.

With the introduction of the M3 Cisco UCS blades we got a VIC1240 Modular Lan On Motherboard (mLOM) mezzanine adapter which gives 2 x 10Gbps to each Fabric Extender.

There’s also an additional mezzanine slot which can be used for a variety of additional hardware like a FusionIO- or LSI Nytro WarpDrive adapter. In this case we can also use this slot to expand the capacity of the VIC1240 by adding a port expander or a VIC1280. This mezzanine slot also provides 2 x 10Gbps to each Fabric Extender.

Now I learned that the amount of bandwidth for each fabric depends on the combination of the network interface(s) in the blade server and the type of Fabric Extender in the chassis.

How?

With the Cisco UCS blade enclosure you can choose from three Fabric Extenders:

• 2104XP (is replaced by the 2204XP, so I won’t go into that one.)
• 2204XP
• 2208XP

And you can choose three network interface combinations for your Cisco UCS M3 blade server, a:

• VIC1240
• VIC1240 & VIC1280
• VIC1240 & port expander

You can mix and match those components with different results. I have described the different combinations below.

2204XP combined with a VIC1240

The first combination is the simplest and I think most common configuration, a 2204XP Fabric Extender with the onboard VIC1240.

Because the host facing ports of the Fabric Extender alternately connect to the onboard VIC1240 and the mezzanine slot, there is only one connection per Fabric Extender per blade. Because each connection gives you 10Gbps, this combination gives you 10Gbps per fabric.

2204XP combined with a VIC1240 and a VIC1280

If you want more bandwidth but don’t want to change the Fabric Extender, you can add a VIC1280 to the mezzanine slot.

When you combine a VIC1240 with a VIC1280 UCS Manager sees them as two completely independent adapters. If you want to have VIC redundancy within a blade server, you should add a VIC1280 instead of a simple port expander.

By using an additional VIC1280, you can now use the host facing interfaces connecting to the mezzanine slot. You now have two active connections to each port group resulting in a 20Gbps per fabric.

2204XP combined with a VIC1240 and a port expander

An alternative, when you don’t need VIC redundancy, is to place a port expander which connects the mezzanine ports to the port groups on the onboard VIC1240.

When you combine the VIC1240 with a port expander UCS Manager sees the VIC1240 and the port expander as a single interface.

This has the same effect as with the VIC1280, a capacity of 20Gbps per fabric. But the the maximum you can achieve with a 2204XP Fabric Extender. If you want more bandwidth, you will have to upgrade to a 2208XP Fabric Extender.

2208XP combined with a VIC1240

The first combination delivers an equal amount of network bandwidth as the previous scenarios but this combination enables you to grow much further, must faster.

Because the 2208XP has double the amount of host facing ports, there are two connections to the VIC1240 per fabric. Because of this, this combination gives you 20Gbps.

2208XP combined with a VIC1240 and a VIC1280

When all that bandwidth if still not enough, you can add an additional VIC1280 to the mezzanine slot.

Again, if you want to have VIC redundancy within a blade server, you should add a VIC1280 instead of a simple port expander .

By using the additional VIC1280, you can now use the host facing interfaces connecting to the mezzanine slot. You now have four active connections to each port group resulting in a 40Gbps per fabric.

2208XP combined with a VIC1240 and a port expander

An alternative, when you don’t need VIC redundancy, is to place a port expander which connects the mezzanine ports to the port groups on the onboard VIC1240.

This has the same effect as with the VIC1280, a capacity of 40Gbps per fabric.

To wrap this up, different combinations of Fabric Extenders and network interfaces result in different amounts of bandwidth to the fabric.

The number of interfaces per Fabric Extender:

The number of 10Gbps interfaces per Fabric Extender per slot/blade server: