mysite blog

Does TCP/IP have a future in the data centre?

Mon, 29 Sep 2008 00:00:00 +0200

Network Convergence is a hot topic nowadays. This is driven mostly by storage networking. The first products which can deliver Fibre Channel over Ethernet (FCoE) are starting to appear even now. There are a couple of drivers for developing Ethernet for storage networks:

Ethernet development is ahead of Fibre Channel and Infiniband in terms of speed
Common network layer simplifies infrastructure management
Reduction in the number of network interfaces and thus initial hardware cost

Ethernet is a best effort protocol by nature and as such not suitable for storage networking, which requires guaranteed delivery of the data. Furthermore, native Fibre Channel and Infiniband have lower latencies that common Ethernet. To bring all these worlds together, new developments are taking place on the IEEE802.1 standard, which describes Ethernet. These developments fall under the common denominator Converged Enhanced Ethernet (CEE) also known as Lossless Ethernet. Though the first FCoE products are starting to appear, the new standard is not finished yet and the new equipment is pre-standard. CEE is expected to be ready for large scale deployment in 2010.

Average data centre networks use Ethernet. Even for inter-data centre connects Ethernet is used more and more. On top of that the most used protocols are TCP/IP and UDP/IP. If Ethernet is made lossless, one could wonder what the advantage of TCP over UDP really is. A list of functional specifications of the IPv6, UDP and TCP protocols:

IPv6:

Can make use of heterogeneous network
Addressing
1. Address scopes (link local/site local/global)
2. Routing; the ability of packets to traverse from one network to another. The use and usage of routing within the data centre is limited. Routes are static and different sub networks for security, but only in combination with VLANs to provide network separation.
3. Address auto configuration; IPv6 interfaces can configure themselves automatically, starting by querying the nearest router using their link-local address, which is normally derived from the interface's Ethernet MAC address.
4. Multicast; The possibility to address a select group of hosts at once
IPsec; IP layer encryption
Prioritization; Used for flow control. Is now well described yet.
Jumbograms; IPv6 datagrams can be much bigger than IPv4 datagrams. Sizes up to 4Gb are allowable.

So how does this compare to Ethernet:

We are talking network convergence here. The network is uniform by definition (OK)
Ethernet only has the MAC address (OK)
1. No scopes exist. Addressing each individual system is a major task. Since the MAC address, for our purpose, is just a random number, some mechanism to manage the MAC addresses is needed. An equivalent to DNS at the very least. (Partially)
2. As mentioned earlier, routing within the data centre is limited. Network bridges can be created. 802.1aq, Shortest Path Bridging, will optimize the network paths (Partially)
3. Each interface comes with is MAC address preconfigured (OK)
4. Ethernet has multi casting capabilities as well (OK)
802.1X, Network Access Control and 802.1AR, Secure Device Identifiers can provide part of the functionality. As far as I know no current standard for encrypting the Ethernet payload exists (Partially)
Prioritization is handled in 801.1p, Class Based Flow Control (CBFC) which uses the 3 bit Priority Code Point from the VLAN header (OK)
Ethernet can use Jumbo Frames which size up to 9K. Since even the Ipv6 Jumbograms must use Ethernet, the best efficiency from a network point of view will be Jumbo Frames, without any IP at all (OK)

Now on to UDP. UDP provides checksumming and application multiplexing. Checksumming is also provided in the Ethernet header. Application multiplexing has no Ethernet equivalent. Application multiplexing (the usage of UDP ports) is a core requirement on networking.

The benefits of using TCP over UDP:

Ordered data transfer
Retransmission and discarding of duplicate packets
Flow control
Congestion control

These benefits provide nothing new over what CEE already offers:

Ethernet relies on in order reception of packets. Each packet will take the same route (OK)
not needed, since in principle Ethernet should not lose any packets (OK)
Flow control is handled by 802.1p and az. Per priority pause frames can be send, selectively throttling data flows. This mechanism is probably more efficient than TCP's bandwidth scaling algorithm. (OK)
is handled by 802.1au, Congestion Notification. Congestion points en route (bridges) notify end points with rate limiting capacity. (OK)

So there it is, for local network traffic, using TCP will be overkill and a reduction of network efficiency. Looking at the trend in Ethernet development, more and more functionality from the higher OSI layers is pulled into the Ethernet protocol. The basic reason why we have IP at all is the fact that the Internet is a heterogeneous environment. With Ethernet becoming more and more the by default standard, the need for higher protocols evaporates. Though Ethernet will never cover the full functionality of the IP stack, though I could envision a protocol that is backwards compatible with Ethernet that can, but that will probably not happing within the next 20 years. Network management would be so much simpler, if there were one standard protocol (not a protocol stack) that would give guaranteed data delivery to one or more systems within a given time.

Back to Square One

Tue, 22 Jul 2008 00:00:00 +0200

In 1992 when I started my study in Astronomy, most students at the physics department were using X-terminals. Workstations were still fairly expensive then and had more power than any one person needed (on average). The X-terminal was a fairly dumb device. It had an X server for graphical representation and tftp client for booting. They worked very well in combination with UNIX servers.

Around 1996, the Linux era began. PC's were even cheaper than terminals, with computing power that was beginning to rival the workstation. Lots of terminals were replaced with Linux PC's. This brought the administrators more work, but as the demand for computing power kept growing, this was good solution as the total cost of the infrastructure as a whole was hardly growing. An additional advantage was, that the PC's could be installed with Windows (which was/is not a multi-user OS) and Linux, giving the user a choice of what to use.

So you might say that Windows and Linux collaborated to push the terminal out.

Now that almost every employee has swapped his terminal (graphical or not) for a PC, we see that the PC at every desktop model has some disadvantages. In today’s environment it becomes harder and harder to keep the desktop environment under control. Viruses are rampant, software lifecycles become shorter and the pressure to keep IT cost under control higher.

A solution for this is to revert to the terminal of old. The main advantage of the terminal is that there are no moving parts. Terminals hardly ever break and if one does, make sure there is a spare in the broom closet. Every person who can unplug a cable can install a new terminal.

Working from a central environment has several advantages, among which:

Backups are a breeze, since all desktops are near tape drives
Migration becomes easy as well, no need to go out to the desktop to install programs. Full platform migration, for which a reinstall on the desktop would be needed, becomes much easier as well, rollback scenarios included.
Patch management, patch when needed, not when the desktop happens to be on
Reduce in power usage. One server that replaces ten desktops or more uses a lot less power. The terminal devices themselves typically use less power than a single hard drive
Terminal devices even exist in laptop form. Add a UMTS card to it and an employee could work everywhere, without taking sensitive company data outside the corporate infrastructure. Terminals in combination with VPN technology could also be used from home using a normal Internet connection.
Terminals are cheaper than PC's and last much longer. So hardware expenditure should be lower as well.

The reason behind the terminal's revival? Desktop Virtualization! Most office workers are still working with the Windows OS, which still isn't a true multi-user operating system. On top of that, many users are loath to give up the privacy of their own desktop. This last argument is more psychological than real, since the administrator already has free access to their desktop.

New products in this area allow you to "remote" the display of the virtual system to a dumb device. This method also works around the typical difficulties with Terminal Server. This is important, because not all applications, even from Microsoft, are working well when started from a terminal. With the desktop virtualization products programs don't even have to know that they are running from a terminal, so those problems are solved.

However, desktop virtualization is not a must for using remote desktops/thin clients. Before any such project is started, an inventory should be created on which users could just work directly of a Windows server and which users would need their own virtual desktop. Chances are that most users will be able to share a server, reducing the maintenance effort for lots of desktop images and the license cost for the virtualization product.

Migrating to Linux or any other UNIX for that matter will fully eliminate the need for any virtual desktops, since multi-user is one of the core strengths of UNIX. If you do need Windows only applications, Codeweavers Crossover Office might be a good suggestion. I myself have been using it to run Outlook and Visio on Linux for years now. (Note that it works on Apple too!)

Using remote/virtual desktops has a few downsides though. At the core of these downsides is the network. During operation, the terminal will send keystrokes and mouse movements to the server and the server sends back the changes in graphical representations on the screen. For office work, this amounts to something between 50kb/s to 100kb/s. For an active user this goes up to around 1Mb/s. From this perspective, a remote office with about 10 employees could still be connected through an ADSL/SDSL connection.

Trouble begins with the new and flashy desktop effects present in Vista and also many new Unices. Enabling these will have a major impact on the network. Since these effects are not strictly necessary, they could be disabled. Playing a movie on a terminal will have the same effect though, so the web broadcast of the New Year’s speech of the CEO will cripple the network. Solutions in this area are coming though, in the form of MPEG decryption at the terminal level instead of the server.

Another problem is more day to day and has a larger impact; printing. Many printers in the corporate environment use PostScript as the printing language. A document containing a few images (the company logo in the template!) translates to a printjob several, of not tens of, megabytes in size. Now we return to the remote office with the 10 employees. A printjob in this environment might take minutes, up to hours.

About the same problem rises when an employee connects his/hers USB stick to the terminal to take a document home.

A whole constellation of options exist to resolve these problems. A few examples:

Change the way people work. By distributing e-paper based devices, the amount of print work could be reduced. The environment could also benefit from this.
Some specialized network devices exist today, than can compress network traffic on the fly and do some other smart network tricks to optimize and usage of bandwidth
Change printer protocols. There must be some printers out there that can work natively with PDF files instead of PostScript. PDF's use compression, so the size of the printjob could be reduced massively.

In my opinion the terminal is back and here to stay. This new generation of the terminal is maturing rapidly, though some details still need to be ironed out. The remote desktop should not be confused with the virtual desktop though. The virtual desktop is solution for users who do require the privacy of their own desktop and applications that won't work well in the multi-user environment. From that perspective, it is a very useful add-on for the remote desktop, but for most users, it won't be needed.

Virtualisation Tips III: Technicalities

Fri, 06 Jun 2008 00:00:00 +0200

In the previous 2 blogs, the soft side of virtualisation was touched. This part is about the real stuff (no offence to non-techies meant). Though this part is technical, there are some tips to save money below, so non-techies may want to read this as well.

Tip III.1: Bring on the load Most appliances in the data centre have optimal efficiency at about 80% load. This is true of server power supplies, CPU's themselves, but also the air-conditioning. So in a data centre that is not full, either because it is new, or because all services have migrated to virtual servers and the rest of the hardware is turned off, turn off some air-conditioning units. 18-20 degrees Celsius is cool enough. One unit with 80% load uses a lot less energy than two running at 40%.

From the cost perspective, this is what virtualisation is all about. Making more efficient use of hardware. It also means, that is is not wise to push beyond the 80% barrier, as energy consumption will that to rise exponentially.

Tip III.2: Start small For the deployment of new services, start as small as possible with hardware. Virtual machines can grow, dependent on the Hypervisor, even without downtime. Create an agreement for fast delivery of new physical servers with a hardware vendor and scale up when you need it.

Normally hardware for a new service was created with the 3 or 5 year depreciation cycle in mind. As long as the infrastructure is set up in a scalable way, you only need to scale to survive the delivery period of new hardware, which is typically one month.

Tip III.3: Think big Big boxes (servers) bring flexibility. With the current technology, virtual systems can only scale to the limit of the physical server. Furthermore, to make optimum use of virtualisation, at least more than a couple of virtual machines should be running on the same hardware. So if you use virtual machine with more than 2 virtual CPU's, you'll want big machines.

The biggest x86_64 compatible servers currently on the market have 32 CPU cores and can allocate 512GB of memory. These boxes are ideal. Also, because they have lots of usable PCI-e slots. You'll need these for you I/O interfaces.

Many vendors try to make you believe that blade servers are the optimal choice for virtualisation. Not so. Blades typically have only few CPU cores and have problems if more than 2 network interfaces are needed.

Blades to to be seen as a competitive technology to virtualisation. (It has its merits, but this us a blog about virtualisation)

In terms of network, the ideal network for bix boxes is 10GbE. The technology is coming down in cost rapidly. With 10GbE, you'll probably not need more than 2 interfaces per server. You'll need at least 2 for availability purposes. More than 2 interfaces will actually be useless, as a typical server will even have problems fully utilizing one 10GbE card.

Tip III.4: 64 not 32 Virtualisation should only be used in combination with 64 bit operating systems. 32 bit systems have a limit of 4GB of memory, which is not enough. Theoretically it is possible to run 32bit virtual systems on a 64bit Hypervisor, but is is best to keep everything 64bit.

Tip III.5: Use the dynamics Modern Hypervisors ans operating systems may allow you to dynamically manage resources. Extra CPU's and storage can be added at runtime. This allows you to shift resources around. More resources to batch oriented applications at night and more to on-line during the daytime, if you're not a 24x7 web shop of course.

Though resources can be added and removed on the fly, it is not a good idea to do this with internal memory allocations. Adding more memory will not be a problem, but taking it away might be, since all unices try to use all memory in the box, either for applications, or for cache.

Dynamic allocations can also be used for fail over purposes. Most Java based application servers can be clustered. Typical deployments contain at least 2 application servers and a load balancer. For many applications this is overkill. Virtualisation in combination with central storage can offer a solution using live migration of running virtual systems to other physical machines. This can be done without interruption. However, if the original physical machine suddenly dies, live migration is not possible and the virtual system will have to be booted on another piece of hardware. This will cause a service interruption of a few minutes.

If this violates your requirements, you could use a cluster of 2 virtual machines. One to handle all requests and one as standby. The standby virtual system can be run with very little resources (no load anyway) and will thus have little impact on the other virtual machines running on that same physical box. If the physical box with the primary virtual system dies, the load balancer will transfer the load to the second virtual system. This one can then scale up in resources. This one is not my idea, but was given to me by a consultant from RedHat. He assured me that this was working well in real world situations.

Virtualisation Tips II: Service Management

Mon, 05 May 2008 00:00:00 +0200

At the introduction of virtualisation in any organisation its impact on service management is often overlooked. Virtualisation is implemented as just another technology. To gain maximum profit from virtualisation changes in the way services are managed are needed. Something, that became clear already in "Virtualisation Tips I". Changes in processes and procedures might even be necessary to remain legally compliant. If cost is the main driver behind virtualisation, the systems management approach is absolutely critical. Not only in processes and procedures, but also in tooling.

Tip II.1: Review the service chain Hosting is a component in the service chain. If virtualisation encompasses on OS platform and one set of service levels, it is best to leave it that way. However, when several different sets of service levels and multiple OS are involved, not to mention the virtual teams from "Virtualisation Tips I" is is probably best to split this component from the service chain in 2, hosting and virtual hosting.

This helps in a couple of ways:

It gives mote insight into operational cost and comparison before and after are more clear;
Clearer picture on how the final service levels are composed;
Helps in separating the virtual from the physical, the physical part might even be outsourced.

Tip II.2: See virtual and physical as separate The virtual and physical worlds are interconnected. Full separation is not possible. However, by keeping them separated as much as possible, flexibility increases. If the virtual infrastructure is not dependent on the physical infrastructure at all, than all virtual infrastructures layouts are possible. The complete infrastructure could be remodeled at the push of a button. This might be a good feature at the introduction of a radically new application, company reorganization or a change in business model. It reduces the financial risk in any IT project involved.

If physical and virtual are separated, it also creates more degrees of freedom in outsourcing deals. One might decide to keep management of the virtual infrastructure and outsource the physical (cloud computing by Amazon/Yahoo/Google/etc. provides a lot of bang for the buck), involve two outsourcings partners, or keep even rent out the surplus physical infrastructure to another company.

Tip II.3: Licenses! Many license models use a per CPU model. But how does that compare to virtual CPUs? Licenses could be bould to a hardware serial number, but those are configurable for virtual machines; thus allowing a second, unpaid, deployment of the same application, without an alarm bell going off. Does the vendor even support deployment of its application on a virtual platform?

The biggest risk here is the deployment of licenses not paid for, in the form of new virtual machines or by adding resources to existing virtual machines. When this happens and more important, if it is discovered, it could result in damages from fines and corporate image.

On the other side, some vendors, you can gain an extra cost advantage. Take RedHat's support contracts for instance. If you deploy Linux on VMware, you pay VMware license and support contract and one support contract for each Linux deployment. If you host Linux on Linux (both RedHat's of course), you only have to pay for one support contract. This could lead to a cost advantage of several thousands of euros per machine per year.

Tip II.4: CMDB The key to keep II.3 is the Configuration Management Database (CMDB). This database should be adapted to virtualisation. A properly configured database will keep track on which system is running where with which service levels and contracts. The CMDB will report will report any problems with regard to licensing/contracts upon entering or changing information.

Another advantage becomes clear when disaster recovery is needed. If a system breaks, where should be virtual systems be reallocated to? It doesn't do to reallocate a virtual system to a physical system with incompatible service levels. A close relation should exists between disaster recovery tools/procedures and the CMDB

Tip II.5: Tooling is essential, not the Hypervisor Systems/Service management should be leading when choosing a virtualisation product. True, one Hypervisor gives better performance than the other, but the bult of the cost for any service is not hardware, but rather the management of that hardware. Windows oriented management departments tend to be more comfortable with VMware or XenSources own Xen distribution and Linux/Unix oriented departments like Xen (any) or even things like AIX partitions more. This has a huge influence on the learning curve of the department and the actual functionality they can realize with the product chosen.

An important part of this is the provisioning of new virtual machines. Provisioning of new virtual machines within an hour should be possible. If this was already the case for physical machines, the same tooling could no doubt do the same for virtual ones. If not, new tooling should be introduced (the simplest approach is template based a.k.a. creating one machine and copy it around).

Virtualisation tips I: The Human Factor

Mon, 14 Apr 2008 00:00:00 +0200

The human factor is often overlooked when implementing new technologies in the data center. Most organizations do think about training their personnel and most actually do spend budget on that, but that is it.

If companies truly want to reap the full benefits of virtualisation some other things concerning the human factor are important as well. The main reason why is because virtualisation introduces a fundamental change in the way we look at infrastructures.

Tip I.1: Boundaries between existing specialized organizational units/departments fade. This tip involves large IT departments only. Thos usually have specialized Windows, Unix and network teams.

In a virtual infrastructure, it shouldn't matter where the virtual system is actually running. This means that the persons managing the hypervisor must work together with the persons managing the virtual machines. This means that whether you are a Windows, Unix or even BSD guy, you'll have to communicate with your peers of all specializations. Not just to change a setting, but to actually dive deep into the lower bowels of the operating system to solve e.g. performance problems.

How about the network? In an enterprise infrastructure, most systems are assigned to one VLAN or the other. If a virtual machine should be ably yo run at any place in the infrastructure, this means that the actual physical server needs to be a member of multiple, if not all, VLANs. All of a sudden not the network engineers decide which packet needs to go where, but the persons managing the hypervisor. This can only happen when departments trust each other.

Some persons will welcome these changes, but there is also a breed of technologist that absolutely abhors to have anything to do with the neighboring technologies (as proof I point to the various flame wars on usenet and public fora). One way to solve this problem is to create a virtual team/organization consisting of persons from all current units/departments and make them responsible for the infrastructure core; the hypervisors and VLAN definitions. Because this team is multi-disciplinary other units/departments will have less trouble trusting and are more willing to work with this new virtual group.

Tip I.2: Human error People make mistakes, it is a fact of life. Most mistakes do not have a very big impact but some do lead to service interruptions. Virtualisation is also about consolidation. More services/applications will be running on one piece of hardware amd one operating system (the hypervisor). This means that the probability that a human mistake has far reaching consequences increases. Something has top be done to balance this increased risk. There are a few options, among which:

Follow tip I.1 and put only the persons with the best track records in the virtual team
Change operating procedure. In many cases, a lot can be gained by looking at the operating procedures carefully. Classify procedures into high, medium and low risk. Then think about ways to lower the risk of at least the high risk procedures.
Consider carefully which application runs where, or which virtual machine runs where. If a service is made up of 3 applications, running in 3 different virtual systems, it is probably best they all run together on the same piece of hardware. This limits the number of dependencies (only one hypervisor involved) and thus reduces the chance of a service interrupting mistake. Of course there are other things to consider as well. From the security point of view you might not want to run all three applications on the same system. A balance will have to be foun.
Cluster. Build clusters of physical machines automatic fail-over capability. This will not safeguard against all problems and mistakes, but will reduce the impact of them. An added benefit is now that all applications are high available.

Next week Part II of the tips, appearing somewhere next week, will be about service management.

Switch Mishmash

Fri, 29 Feb 2008 00:00:00 +0100

Last week, I stumbled upon an interesting problem. One of the projects I'm working on as an astronomical sensor grid, putting out about 2.6Gb/s of data per sensor station, coupled to a central super computing infrastructure. This system is still in the buildup/tender phase, so the network consists of networking equipment of various brands, to test out what the differences are and how well they work together. There are only 5 sensor stations at the moment, so the total amount of data is manageable. In the future the total data stream will grow to about 1Tb/s, to be processed real time.

In the past there were 2 network infrastructures, one a WAN (for connecting the sensors), one central and there was a firm boundary between them, in the form of a computing cluster that did data re-ordering. This cluster disappeared (because this task could also be done as a subtask of another cluster). The network came crashing down, due to some limitations posed by the computing infrastructure.

Nothing that couldn't be fixed, but this brought all the networking brands in the infrastructure together. The network code is now made up of Layer 2 and 3 switches from Nortel, Foundry and HP. This gave me a good feeling of the differences between the brands, on top of the experience I already have with testing several brands in a lab setup.

Note that I'm not a certified network anything, so if I'm wrong in someplace, or if there are features I missed please comment on this blog.

The challenge in this infrastructure that there are a relative small number of big monolithic data streams (600Mb/s) and total silent listeners, causing switches not to learn where the listeners are.

To make everything work the network relies heavily on VLAN and Multi-link trunking. Features every switch brand offers from their base model and up.

Performance differences One of the first problems encountered is the bandwidth each switch offers. It is not trivial that on a 48 port GbE switch all ports can work at "full speed non-blocking", even for switches that do promise this performance. Typically in a switch, each cluster of ports is handled by one chip. And there is one chip handling the previous chip (a tree structure). There can be a significant limit om the amount of data that goes from chip to chip. So depending on which ports you use for what purpose you either get or don't get the performance you want.

Luckily most modern switches from the major switch brands don't have this limitation any more, but if you're into high performance networking, it is something to check.

Jumbo frames In this infrastructure 9K jumbo Ethernet frames are used. Some brands put the jumbo frames at less than 9K. This not only happens at switch level but e.g. the max. frame size the network card in my computer can handle is 8K.

VLAN The VLAN standard is well described. The implementation is not. Most implementation differences revolve around filtering and VLAN number 1 (the default VLAN). In all switch brands I've seen VLAN 1 is the default VLAN, the the port membership of VLAN 1 differs per brand. What I've seen:

The port is a member of VLAN 1 until the port is added as an untagged port to another VLAN
The port is a member of VLAN 1 until it is added to another VLAN (tagged or untagged)
A port must specifically be removed from VLAN 1
Unregistered (ID not registered at that switch) VLAN tags are or are not filtered
Untagged packets are or are not filtered on a port that is supposed to handle only tagged packets (such as a trunk)

These implementation issues can drive you to madness creating loops where you don't expect them to be. Also it is important to know these implementation differences if only from the security point of view.

Some switches are more flexible when it comes to VLAN's then others. There are switches that allow you to set VLAN X for receiving data and VLAN Y for sending. While this kind of functionality is needed in some places, it makes you very vulnerable to mis-configurations.

Multi-link trunking Most switches allow for two different types of multi-link trunking (MLT). A brand specific variant and LACP.

In the brand specific variants a set number of lines is allocated to the trunk and load is distributed over the lines. This is not done based on load, but based on an algorithm which is dependent on factors as source and destination MAC, IP address or IP port number.

In a normal network setting where there are many little data streams this works well. But if you have anly few streams (number of streams in the same order of magnitude are the number of lines in the trunk), this is not the case. Often too many streams are allocated to the same line from the trunk, while other lines are not used at all.

To effectively deploy MLT in such a case, you will need to know the algorithm and adapt your MAC addresses to it. The snag here is that the algorithm is almost alway classified information. The other snag here is that this kind of MLT only works between switches from the same brand.

LACP trunking is working a lot better in the environment I described. The number of lines participating the the trunk is dynamic. Lines are added to the trunk when the load on the current lines reaches the limit. This behaviour could lead to other performance issues (especially around the moment where extra lines are allocated to the trunk), but with long lasting continuous large data streams it is a perfect solution.

LACP trunks also work between switch brands, but a bit of caution is needed. At the switch level, each trunk has its own ID (key). For LACP trunks between switches, sometimes the ID doesn't need to match and sometimes it does. This could be a problem, especially if you use manual IDs for the trunk (each switch tries to auto allocate an ID, but also has a special range for manual IDs). e.g. the range for the manual configuration of IDs between HP and Foundry have no overlap. The trunk does not form. If the determination of the ID is let to the switches themselves, it works perfectly.

Controlling the Flow It can be extremely hard to make your data go where it needs to go, or should I say not make it go where it doesn't need to be. If you got equipment on the network that is completely silent, a switch will never learn where the device is. Resulting in a broadcast of the data that needs to go to that device on all switch ports. That really is a problem if you have several 400Mb/s data streams. This brought down the network several times in the case I'm describing.

A possible solution is to enter addresses in the switch MAC table by hand; but this is not a standard feature of a switch. My guess is that only half the number of switches out there support this. Surprisingly some really cheap switches support this while some A-brands don't.

Another (partial) solution is defining uplink ports on the switch. If an address is not present in the MAC table the data only goes to the uplink ports. This reduces the problem untill the last switch the data goes through. This feature combined with a static MAC table is what you want to have if you have silent devices. It keeps the administration task manageable (only need to administer the MAC address once), while letting you have full control.

User interfaces Most switches emulate Cisco in their configuration. This is a good thing, since it works very well, once you know it. The learning curve can be very steep for this kind of interface.

Some switches have menu based configuration and most also allow web-based configuration. While these are easy to use, I have never seen and menu based interface that allows you to configure all aspects of the switch.

Menu driven interfaces often try a more intelligent approach to configuring the switch. This works well when you're a rookie, but the result is, that if you configure one aspect of the switch, some other settings will be auto-changed as well.

One brand I know even has menu-ized the command prompt, while it works better than most web interfaces, I still prefer the good old command prompt and a "sh run" that only shows you the things did configure yourself.

Locked up Almost each brand has some cool very handy features that no other brand has. The temptation to use these features can be quite big at times. There is no harm in using these features, but when you do, realize that a vendor lock-in might be created at that moment. The real hazard lies in the features of which you don't know that they are vendor specific (happened to me a couple of times). A lock-in could be created without knowing it.

Wrap-up The bottom line to this story is that a switch is not a switch. Each brand has its strong and weak areas. These won't be encountered often in office based setups, but with the growing demand for performance and security these areas will be encountered more often.

This leaves a challenge for the network experts, determining what brand (or mix of brands) is most optimal for which case can be, I'm happy to say, not easy.

Using OpenDNS with your own DNS

Thu, 21 Feb 2008 00:00:00 +0100

Yesterday I read an article about OpenDNS http://www.opendns.com. OpenDNS adds the possibility of domain name based filtering. You can configure your own white and blacklist, but they also have a list of categories you can choose from. Most interesting category I found was phising, but I image that if you got small children at home you'd also want to block porn and such. Domain name spelling correction is an added bonus, plus they promise almost 100% service availability, superior performance, domain name spell checking and statistics gathering (can be turned off).

The service is free, but you have to register and put in your source IP address, because the service is configured via the OpenDNS website. Teh only thing you need to change on your PC is the DNS address to use. I suppose most people have this on auto-configure (via DHCP), so this add a (small) piece of configuration. A downside is that this free service must get its funding from someplace, so ads are shown when pages are blocked or not found, but I can live with this.

Myself, I run a couple of services on a home server, among which this webserver and my mailserver. Because of this I have an internal DNS server which forwards all non gloudemans.info queries to the DNS of my ISP. All internal clients also use my local DNS, or they wouldn't be able to find my own webserver. Now I could just change my forwarders to point at the OpenDNS DNS servers, but this could have a couple of issues:

Some mail is relayed via a server from my ISP. The DNS name of this server is only in the ISP's DNS
It adds another party that can see who I'm sending mail to. I'm willing to trust my ISP, but to trust some free service off the Internet is another matter.
The network path to OpenDNS DNS servers is longer, though the OpenDNS servers might provide a better uptime, I doubt the actual uptime measured as I see it (including the network between me and OpenDNS) is better. I can live with this for browsing, but I'd like better availability for the services I have running.
e.g. the mailserver doesn't need the functionality OpenDNS offers. Bugs in thew OpenDNS software, like spell checking domain names might even have an adverse effect. This is a nonexistent issue at a moment, but it was in the past and illustrates my point.

Now if the performance of OpenDNS is really better, I could think about using it for my mailserver as well (but then again, what's the importance if a mail goes out within 2 or one second). So I decided to do a quick test with this script. It looks up a couple of domains and displays the time it took in seconds. I ran the script twice for my ISP's DNS and for OpenDNS and got these results. As you can see, the performance of both DNS servers is very similar. The ISP DNS servers have a few spikes in them, but on the whole there isn't much difference. So there is no business case to use it on mail, but I still would like it for my browser.

Luckily the DNS server software used for most Linuxes (Bind) has a featured called views. WIth views you can control the actions of the name server based on the client. First, there needs to be a clear distinction between server and client network addresses. You can enter each client address into the DNS configuration by hand, but I subnetted my local network from a class C (netmask 24) to a netmask 26, giving me 4 distict IP ranges. I still use the 24 netmask on all systems, but now I have the ability to make a distinction. 1-63 is all server related, 64 to 127 is wired client, 128-191 is wireless client and 192 to 255 is DHCP range.

My named.conf looked like (not a complete working config)

options
{       
        // Put files that named is allowed to write in the data/ directory:
        directory "/var/named"; // the default
        dump-file               "data/cache_dump.db";
        statistics-file         "data/named_stats.txt";
        memstatistics-file      "data/named_mem_stats.txt";
        auth-nxdomain yes;
        listen-on { 127.0.0.1; 10.0.0.2; };
        forwarders {
              //ISP DNS one
              192.192.192.192;
              //ISP DNS two
              192.192.129.193;
        };       
        allow-transfer {
                none;
        };
};
zone "0.0.10.in-addr.arpa." IN {
        type master;
        file "10.0.0.db";
};
zone "gloudemans.info." IN {
        type master;
        file "gloudemans.info.db";
};

Using my newly defined subnets and using views, I changed this into (non working example):

acl serverips {
        //Dont forget the localhost
        127.0.0.1;
        10.0.0.0/26;
};
acl clientips {
        // Wired stations
        10.0.0.64/26;
        // Wireless stations
        10.0.0.128/26;
        // DHCP range
        10.0.0.192/26;
};
options
{       
        // Put files that named is allowed to write in the data/ directory:
        directory "/var/named"; // the default
        dump-file               "data/cache_dump.db";
        statistics-file         "data/named_stats.txt";
        memstatistics-file      "data/named_mem_stats.txt";
        auth-nxdomain yes;
        listen-on { 127.0.0.1; 10.0.0.2; };
        allow-transfer {
                none;
        };
};
view servers {
        match-clients { serverips; };
        forwarders {
              //ISP DNS one
              192.192.192.192;
              //ISP DNS two
              192.192.129.193;
        };       
        zone "0.0.10.in-addr.arpa." IN {
                type master;
                file "10.0.0.db";
        };
        zone "gloudemans.info." IN {
                type master;
                file "gloudemans.info.db";
        };
};
view clients {
        match-clients { clientips; };
        forwarders {
              //OpenDNS one
              208.67.222.222;
              //OpewnDNS two
              208.67.220.220;
        };       
        zone "0.0.10.in-addr.arpa." IN {
                type master;
                file "10.0.0.db";
        };
        zone "gloudemans.info." IN {
                type master;
                file "gloudemans.info.db";
        };
};

Now all server related stuff, like mail and local webserver statistics gathering etc. is using the ISP DNS servers, while all local web browsers resolve non local domains via OpenDNS and this profit from all its features. Note that this setup is not secure. If someone doesn't want to use the OpenDNS on purpose. A change of the client IP number is enough. This setup will work to keep your little children away from sites they should not see, it won't work for the teenager (unless you only use 127.0.0.1 for the server IP), but at that age a proper conversation is probably better idea than just blocking content anyway.

The Triangle of Amplification

Thu, 14 Feb 2008 00:00:00 +0100

SOA, SOX Compliance, Virtualisation, Security, Identity Management, Outsourcing; all trends of the current time. Organizations are struggling to adapt and incorporate. Consultancy bureaus thrive on this (I belong in this category). Now is the time to step back and look on how all of these trends interact and what the demands are on he organization from a holistic point of view.

There are three key concepts in technical infrastructure trends:

Fragmentation
- Services are built from increasingly small parts. This fits the Service Oriented Architecture (SOA) model, but at the same time complicates infrastructure management. Especially if those parts are the responsibility of different entities
- Grid; having a large pool of resources that is dynamically allocated to the most needy task sounds and is great. But how to ever guarantee service levels on such an infrastructure; what is the pool of resources is spread over multiple separate entities?
- ...
Security
- Identity management simplifies the maintenance of authorization and authentication records and makes life easier for all users involved, but also makes any security incident more widespread
- More and more laws are posing boundary conditions on services and infrastructures
- ...
Virtualisation
- Potentially brings down hardware and license cost
- But eradicates the traditional boundaries in the infrastructure. Separation of responsibilities as implemented in most organizations is gone (see also http://www.gloudemans.info/virtualisation-overview)
- ...

Other trends are more or less related to one or more of these concepts.

These three concepts enhance/amplify each other in every way. They enable each other, but also amplify each others problems. Examples:

Because of the fragmentation of services, each component needs authentication and security services
Virtualisation allows for many smaller service components to run on one piece of hardware, thus enabling fragmentation
Virtualisation poses a new security challenge in the area of infrastructure and service management
Chinese wall policies might encumber the full usage of virtualisation and grid technologies
A successful identity management implementation enables implementation of SOA

Outsourcing commonly plays at least on one of these concepts. For example: Outsourcing means new trust relations (playing on security), fragmentation (more than one outsourcings partner is common) and virtualisation (giving the outsourcing partner(s) a piece of infrastructure to test on).

Triggering one area will trigger the other two as well, directly of in the future.

Ground rules for support organizations on how to deal with modern infrastructures:

Awareness; be aware of the service chains and how they interact, be aware of security implications if a component changes. This is a business responsibility, but in todays highly complex infrastructures mistakes are easy to make and can have a devastating influence on the business
Multi-Domain knowledge. The era of the system and network manager are past. The new IT professional is skilled in several areas and is a team player by definition. No one can have complete overview, so IT professionals must increasingly rely on the knowledge of their peers
The support organization must have as little interfaces as possible. Depending on the user to identify the problem and calling the right service desk of the right service component is not possible. The user will be more heavily involved in identifying the real problem, but relaying him from service desk to service desk is counter productive.
No boundaries. Boundaries between entities in the same organization cannot exist if a complicated/fragmented service is to be managed efficiently. Different roles still must exists (something to do with security), but unhampered interaction between these roles is mandatory.
The governance model for the support organization must be excellent. Not only for the complex services under its own control, but for interfacing to the organizations which have other parts of the same service under control as well.

End to all Desktop problems?

Tue, 12 Feb 2008 00:00:00 +0100

Some consultants would you believe that virtualisation technology will solve all your desktop problems.... Yeah, right!

The assumption is based on the fact that many people believe that desktop stability and the data loss that goes with it, is the biggest problem. It is true that one misbehaving application can bring down your complete desktop computer. If every application is run in its own virtual machine (VM), which contains only the absolute essentials for the application to work. This would:

stop applications from influencing each other. A problem in one application will not affect the other one
prevent the famous windows DLL dependency hell (or shared object hell if you're a *nix person)
keep the Operating System (OS) to a bare minimum. Since all logic to access the hardware is in the VM. The operating system is just a Hypervisor and maybe some user interface elements.
drastically reduce the chance of security exploits and bugs in the OS, since there is less code in the OS
a driver problem only kills the application using that driver

Now, in my opinion, this makes no sense at all. While I do believe, it is a good idea to run e.g. a web browser in a VM, for most desktop applications is isn't. These are the reasons that spring up in my mind:

No interaction between applications means no data exchange between them. The concept of "copy and paste" and "drag en drop" needs a complete overhaul
All the applications together will make up most of the complete OS from the "Old" environment, the only thing changed is the shift in responsibility
Application developers don't want to reinvent the wheel from scratch for each application. There will be standard libraries is each application. What happens if a security flaw is discovered in that? Not one hole to patch, but many!
Each application still needs access to the data, so an exploit in your application still puts your valuable data at risk

Virtualisation can help creating a more secure desktop, but it'll have to be used on todays desktop technology.

The best way to a more secure and stable desktop environment must come from the OS itself. Proper role and memory separation and a good Quality Assurance program for driver development. While companies like Microsoft have an excellent QA program, it is not mandatory. The Open Source model actually helps driver development. The developer can see all the kernel code, copy, reuse and discuss with the rest of the world.

A stable and secure desktop is possible now, it is just a case of picking the right technology and actively work on putting legacy applications of of business, or in a VM. Linux anyone? (or BSD for that matter)

Away with ITIL

Mon, 04 Feb 2008 00:00:00 +0100

ITIL has become the de facto standard for system management in most large organizations and many small ones. ITIL has revolutionized system management . One of the strongest points of ITIL is the dictionary that comes with it. Now we can all understand each-other when talking about system and service management.

But has the quality of IT services really improved because of ITIL? I think not as much as it could have. In ITIL implementations, organization structures became a mirror of the ITIL processes. Organizations which previously had one monolithic systems management department, now have several departments or groups. One responsible for first line support, one for second line, one for service management etc. ITIL has inadvertently created new boundaries within the organizations implementing it. This is not a problem by definition, but all to often boundaries lead to "kingdoms".

Knowledge and experience becomes locked up between these boundaries and employees become fed up with their jobs (if you are a real technician, delivering first line support can be a true horror).

The solution? Quite simple; use ITIL as a representation to the outside world, but remove the boundaries internally. Create two groups, one the business side of systems/service management and one for the technical side and fine a balance of experienced staff and rookies within these groups. For large organizations create teams for each application/service domain within the groups. Make each team responsible for meeting the service levels for the services running in their domain. If possible, do not appoint a team lead, but let the teams find out for themselves and move members between teams on a regular (yearly?) basis, to promote standardization.

Each team must contain all the skills needed to manage the application/service domain, not only first to third level, but also from hardware to application, from tactical to strategical etc. In these multi-skilled teams of diverse experience, the learning curve for the inexperienced will be very steep, while the experienced may find a new challenge in coaching the inexperienced and enjoy the increased freedom. Competition between the teams (based on service levels met) must be encouraged. The only boundary condition; the outside interface is ITIL and it must be the same for all teams. (Knowledge about ITIL within the teams is essential)

I think, this will really improve the quality of services delivered. For instance the number of escalations, technical or management will go down drastically. If the teams do a proper job, the person to escalate to is already involved! At the same time, customers and team members will be more content and thus less likely to switch vendor or employer.

Certification; killing innovation?

Sat, 02 Feb 2008 00:00:00 +0100

Many employees require their personnel to be certified in certain areas. This could be area specific certificates, like security, or vendor specific certificates, like MCE or CCNA. The reason behind it seems very obvious; the certificate is proof that one has mastered the necessary knowledge. It reduces the risk for the employer or customer of having an inadequate staff.

But what is the value of certification really? In real life, I have seen certified technicians that I wouldn't even let near my home network, let alone the a business critical infrastructure. Actually, this is a well known phenomenon and is due to various reasons. The most obvious are:

Most important one is the quality of the course leading to certification. I think the earlier Microsoft certifications are a great example of this. People learned how things were done in the GUI, but they forgot the part of telling people how the underlying standards worked and why they were what they were.
The fact that many certificates don't need to be renewed. Especially if the certificate is gained by performing well on an exam, chances are that most of the knowledge needed is in short term memory.
Even if certificates do need to be renewed, employers and customers hardly ever check if they are still valid. I'm sure that there are a lot of people out there who write that they are certified on their resumes, while they actually are not (any more)
There's no substitute for experience. But experience noted on a resume might be misleading. Every project has its leaders and followers; their resumes often looks the same.

Most of the certificates that are held in high regard, have a way to cope with these previous points and they deliver highly trained professionals, who really do know what they are doing. I think the real problem with certification is that people learn to think along certain lines. This is something that struck me, while I was at a security conference. Everyone I met, approached a given problem from the same angle and came to the same, predictable, solution. Talking about security, this is a bit scaring, because if reactions are predictable, this gives the hacker (who if most often not certified and/or trained and thus highly unpredictable) an opportunity to go around them.

To find the optimal solution for a given problem, the problem must be approached from multiple angles. In my experience certification (and cources) interfere with this. Certified processionals, will deliver an excellent solution for a given problem, but it may not be the best one possible.

My remark is not only valid for the field of security, but business as well. If you want to know what your competitor is going to do next, dig up his resume and look for his courses and certificates.

Innovation happens when problems are approached from unusual directions, also called "out of the box thinking". Courses and certifications put you firmly in the box, making it harder to escape. So next time you, or your boss, wants you to get a certificate, think hard on the reason why. Buying some books and starting a study group with some colleagues might have the same effect, without putting yourself in the box.

As for how to get the next job, with a seemingly poor resume? Just put your thoughts on certification in the letter, if you do that right, you will spark some interest, get invited for an interview and dazzle them with your innovative insights.

High Speed Firewalling in Linux

Thu, 31 Jan 2008 00:00:00 +0100

In this age of virtualization, cluster and grid applications, the load on the server network interface tends to increase. Multi-homed servers were rare not more than 5 years ago, nowadays servers come with 4 NICs pre-installed. This, of-course, increases complexity and network cost. Today I made a proposal for a couple of servers with more than 10 Gigabit interfaces. With the current state of technology, we are approaching the break even point for 10 Gigabit network interfaces. This year will see the breakthrough of 10Gigabit over copper in the datacentre.

The reduction of the number of interfaces will at least simplify the physical infrastructure, but I think management cost will decrease as well.

But what performance can we expect from 10GB/s interfaces. Not too long ago, servers had trouble even saturating a single 1Gigabit interface. I did some tests myself, using a Sun X4100 server running Centos 5 (Linux) and some quick kernel tuning. I was not disappointed with the results. I managed to get a throughput of 8.5 Gigabit per second, using TCP/IP over Ethernet (using Jumbo frames). These kinds of throughput are only possible with a NIC that supports TCP/IP offloading, like a Myricom card. I did this test also with an older 10Gigibit Intel NIC and I got stuck at 3.5 Gigabit per second. In all cases one CPU core was fully used to process and transmit/receive.

For normal business applications, or high speed Internet connections the Firewalling is a must. So what is the performance penalty?

It is surprisingly small. Network latency anly increases by 2 microseconds (1-2% increase) and bandwidth suffers 10-25Mbit per second; that's less than 0.5%
I am still analyzing the test results for one of the customers I work for, but I'd say that the (Linux) server is ready for the next step in networking.

The image below shows a graph of network frame spacing against bandwidth. Each line represents an IPtables ruleset compared against the baseline performance (without rulesets)