BlogIT » Switch Mishmash
Switch Mishmash
Last week, I stumbled upon an interesting problem. One of the projects I'm working on as an astronomical sensor grid, putting out about 2.6Gb/s of data per sensor station, coupled to a central super computing infrastructure. This system is still in the buildup/tender phase, so the network consists of networking equipment of various brands, to test out what the differences are and how well they work together. There are only 5 sensor stations at the moment, so the total amount of data is manageable. In the future the total data stream will grow to about 1Tb/s, to be processed real time.
In the past there were 2 network infrastructures, one a WAN (for connecting the sensors), one central and there was a firm boundary between them, in the form of a computing cluster that did data re-ordering. This cluster disappeared (because this task could also be done as a subtask of another cluster). The network came crashing down, due to some limitations posed by the computing infrastructure.
Nothing that couldn't be fixed, but this brought all the networking brands in the infrastructure together. The network code is now made up of Layer 2 and 3 switches from Nortel, Foundry and HP. This gave me a good feeling of the differences between the brands, on top of the experience I already have with testing several brands in a lab setup.
Note that I'm not a certified network anything, so if I'm wrong in someplace, or if there are features I missed please comment on this blog.
The challenge in this infrastructure that there are a relative small number of big monolithic data streams (600Mb/s) and total silent listeners, causing switches not to learn where the listeners are.
To make everything work the network relies heavily on VLAN and Multi-link trunking. Features every switch brand offers from their base model and up.
Performance differences
One of the first problems encountered is the bandwidth each switch offers. It is not trivial that on a 48 port GbE switch all ports can work at "full speed non-blocking", even for switches that do promise this performance. Typically in a switch, each cluster of ports is handled by one chip. And there is one chip handling the previous chip (a tree structure). There can be a significant limit om the amount of data that goes from chip to chip. So depending on which ports you use for what purpose you either get or don't get the performance you want.
Luckily most modern switches from the major switch brands don't have this limitation any more, but if you're into high performance networking, it is something to check.
Jumbo frames
In this infrastructure 9K jumbo Ethernet frames are used. Some brands put the jumbo frames at less than 9K. This not only happens at switch level but e.g. the max. frame size the network card in my computer can handle is 8K.
VLAN
The VLAN standard is well described. The implementation is not. Most implementation differences revolve around filtering and VLAN number 1 (the default VLAN). In all switch brands I've seen VLAN 1 is the default VLAN, the the port membership of VLAN 1 differs per brand. What I've seen:
- The port is a member of VLAN 1 until the port is added as an untagged port to another VLAN
- The port is a member of VLAN 1 until it is added to another VLAN (tagged or untagged)
- A port must specifically be removed from VLAN 1
- Unregistered (ID not registered at that switch) VLAN tags are or are not filtered
- Untagged packets are or are not filtered on a port that is supposed to handle only tagged packets (such as a trunk)
These implementation issues can drive you to madness creating loops where you don't expect them to be. Also it is important to know these implementation differences if only from the security point of view.
Some switches are more flexible when it comes to VLAN's then others. There are switches that allow you to set VLAN X for receiving data and VLAN Y for sending. While this kind of functionality is needed in some places, it makes you very vulnerable to mis-configurations.
Multi-link trunking
Most switches allow for two different types of multi-link trunking (MLT). A brand specific variant and LACP.
In the brand specific variants a set number of lines is allocated to the trunk and load is distributed over the lines. This is not done based on load, but based on an algorithm which is dependent on factors as source and destination MAC, IP address or IP port number.
In a normal network setting where there are many little data streams this works well. But if you have anly few streams (number of streams in the same order of magnitude are the number of lines in the trunk), this is not the case. Often too many streams are allocated to the same line from the trunk, while other lines are not used at all.
To effectively deploy MLT in such a case, you will need to know the algorithm and adapt your MAC addresses to it. The snag here is that the algorithm is almost alway classified information. The other snag here is that this kind of MLT only works between switches from the same brand.
LACP trunking is working a lot better in the environment I described. The number of lines participating the the trunk is dynamic. Lines are added to the trunk when the load on the current lines reaches the limit. This behaviour could lead to other performance issues (especially around the moment where extra lines are allocated to the trunk), but with long lasting continuous large data streams it is a perfect solution.
LACP trunks also work between switch brands, but a bit of caution is needed. At the switch level, each trunk has its own ID (key). For LACP trunks between switches, sometimes the ID doesn't need to match and sometimes it does. This could be a problem, especially if you use manual IDs for the trunk (each switch tries to auto allocate an ID, but also has a special range for manual IDs). e.g. the range for the manual configuration of IDs between HP and Foundry have no overlap. The trunk does not form. If the determination of the ID is let to the switches themselves, it works perfectly.
Controlling the Flow
It can be extremely hard to make your data go where it needs to go, or should I say not make it go where it doesn't need to be. If you got equipment on the network that is completely silent, a switch will never learn where the device is. Resulting in a broadcast of the data that needs to go to that device on all switch ports. That really is a problem if you have several 400Mb/s data streams. This brought down the network several times in the case I'm describing.
A possible solution is to enter addresses in the switch MAC table by hand; but this is not a standard feature of a switch. My guess is that only half the number of switches out there support this. Surprisingly some really cheap switches support this while some A-brands don't.
Another (partial) solution is defining uplink ports on the switch. If an address is not present in the MAC table the data only goes to the uplink ports. This reduces the problem untill the last switch the data goes through. This feature combined with a static MAC table is what you want to have if you have silent devices. It keeps the administration task manageable (only need to administer the MAC address once), while letting you have full control.
User interfaces
Most switches emulate Cisco in their configuration. This is a good thing, since it works very well, once you know it. The learning curve can be very steep for this kind of interface.
Some switches have menu based configuration and most also allow web-based configuration. While these are easy to use, I have never seen and menu based interface that allows you to configure all aspects of the switch.
Menu driven interfaces often try a more intelligent approach to configuring the switch. This works well when you're a rookie, but the result is, that if you configure one aspect of the switch, some other settings will be auto-changed as well.
One brand I know even has menu-ized the command prompt, while it works better than most web interfaces, I still prefer the good old command prompt and a "sh run" that only shows you the things did configure yourself.
Locked up
Almost each brand has some cool very handy features that no other brand has. The temptation to use these features can be quite big at times. There is no harm in using these features, but when you do, realize that a vendor lock-in might be created at that moment. The real hazard lies in the features of which you don't know that they are vendor specific (happened to me a couple of times). A lock-in could be created without knowing it.
Wrap-up
The bottom line to this story is that a switch is not a switch. Each brand has its strong and weak areas. These won't be encountered often in office based setups, but with the growing demand for performance and security these areas will be encountered more often.
This leaves a challenge for the network experts, determining what brand (or mix of brands) is most optimal for which case can be, I'm happy to say, not easy.
