blog

blog http://www.gloudemans.info/blogit/ Migrate Paravirtualized Xen to KVM under RHEL http://www.gloudemans.info/migrate-paravirtualized-xen-to-kvm-under-rhel/ Update July 11, 2009: Re-registering VMs at RHN uses an extra entitlement with RHEL5.4Beta Update July 15, 2009: Swap usage, clock and disk cache of the virtual machine Update July 16, 2009: Replace virsh create with virsh define &amp;amp; start to create a managed domain and not a transient one Update September 2, 2009: Re-registering with RHN works Update September 2, 2009: RHEL5.4 has been released. Added a note about services on the physical host Update September 6, 2009: Updating TimeKeeping and Hugepages Update April 1, 2010: Hugepages configuration for RHEL 5.5 Update July 3, 2010: Make Hugepages mountpoint persistent RedHat Enterprise Linux version 5.4 is out. It heralds the arrival of KVM as RedHat's official hypervisor. RedHat will be supporting Xen for the rest of the RHEL5 life cycle, so for the moment, there is no need to migrate to KVM.However migrating to KVM has some advantages. For one KVM looks simpler from the outside, another is that it works with a normal kernel, meaning that all drivers that work on a normal kernel work as well. This not only encompasses display drivers, but CPU scaling (dynamically adapting the speed of the CPU) as well. This is not only very "green" but makes a difference is your or the companies wallet as well.RedHat put a lot of work into making Xen easier to manage in RHEL5.0-5.3. As a result Xen uses a single disk image from which it can boot. The format of this image is the same as for KVM. One would suspect that migrating from one Hypervisor to another would be easy and it is. This blog will describe a step-by-step scenario on how to do it.The starting situation is a RHEL5.3 Physical host with RHEL5.3 paravirtualized guests. The guests have two networking interfaces, one bridged to the physical network interface, and one bridged to a dummy network interface for an internal host network. Note that the minimum requirement to run with virtio is RHEL5.3.Note: I had some trouble with selinux in the rhel 5.4 beta. It is related to the attributes on /var/lib/libvirt. I do not use this directory to store the images, bit I use raw LVM volumes. To get my system running again, I just disabled selinux.Configure the virtio drivers Open /etc/modprobe.conf in the editor. In our case /etc/modprobe.conf contains the following lines: <div class="codesnippet">alias eth0 xennet alias eth1 xennet alias scsi_hostadapter xenblkchange it toalias eth0 virtio_net alias eth1 virtio_net alias scsi_hostadapter virtio_blk</div>Now add the virtio drivers to the kernel boot image (modify this lane to mirror the latest kernel version)<div class="codesnippet">mkinitrd -f --with=virtio_blk --with=virtio_pci --builtin=xenblk initrd-2.6.18-128.1.16.el5.img 2.6.18-128.1.16.el5 The --builtin is necessary only when currently running under a xen kernel in paravirtualized mode</div>Internal clock The internal clock of KVM is less stable than the clock under Xen. Heavy loads have been know to cause clock drift. There are two workarounds: <ul><li> Boot with divider=10 notsc (see earlier) and start ntpd at boot (chkconfig --level 2345 ntpd on; configure ntp first) </li><li> Use the -no-kvm-pit-reinjection option with qemu-kvm. One of the improvements added to the final version is that libvirt seems to add this option by default now, so everything should work out of the box. You still need to start ntp though. </li></ul> Also see (<a href="https://bugzilla.redhat.com/show_bug.cgi?id=507834">https://bugzilla.redhat.com/show_bug.cgi?id=507834</a>)Now shut down the virtual system (shutdown -h now)Updating the host The physical host needs some updating as well. First, before you start, make sure all virtual systems are stopped (xm list) and that you are logged on as root. If RHEL5.4 is already released, yum will update the system automatically to this version. If now, the system needs to be subscribed to the RHEL5.4 beta channel. You can do this at <a href="https:///rhn.redhat.com">RedHat network</a>, if your system is subscribed to rhn. Also make sure the system has access to the Virtual Platform channel beta. Aside from the updates, some new packages need to be installed as well and all virtualization services must be disabled at boot time until we are ready with the configuration work. <div class="codesnippet">yum clean all #for safety yum update yum install kernel kvm kvm-tools kmod-kvm kvm-qemu-img bridge-utils chkconfig --level 2345 xend off chkconfig --level 2345 xendomains off chkconfig --level 2345 rhn-virtualization-host off</div>Edit /boot/grub/menu.lst and set the default boot kernel to the newest non-xen kernel (see example grub config)Network configuration By default only a network that is connected via NAT to the outside world is created. There are three options, leave it as is, but check that the IP range does not conflict with anything on the local network, change the IP range, or convert it to a host only network. I left the network, but adapted the IP range and created a new network for host-only networking. Be sure to change the uuid of the network. The format of the uuid should not change. Change any hex number [0-9|a-f] in the uuid string. <div class="codesnippet">/etc/libvirt/qemu/networks/default.xml &lt;network&gt;&lt;name&gt;default&lt;/name&gt;&lt;uuid&gt;cc06c2a2-0766-45ee-baaa-896e04c7a3be&lt;/uuid&gt;&lt;forward mode="nat"/&gt;&lt;bridge name="virbr0" stp="on" forwarddelay="0"/&gt;&lt;ip address="a.b.c.d" netmask="255.255.255.0"&gt;&lt;dhcp&gt;&lt;range start="a.b.c.e" end="a.b.c.f"/&gt;&lt;/dhcp&gt;&lt;/ip&gt;&lt;/network&gt; /etc/libvirt/qemu/networks/hostonly.xml &lt;network&gt;&lt;name&gt;hostonly&lt;/name&gt;&lt;uuid&gt;04255669-803e-d8f6-352a-086fa45ae09d&lt;/uuid&gt;&lt;bridge name="virbr1" stp="on" forwarddelay="0"/&gt;&lt;ip address="a.b.g.h" netmask="255.255.255.0"&gt;&lt;dhcp&gt;&lt;range start="a.b.g.i" end="a.b.g.j"/&gt;&lt;/dhcp&gt;&lt;/ip&gt;&lt;/network&gt;</div> The host-only network should be started at boot, so ln -s /etc/libvirt/qemu/networks/hostonly.xml /etc/libvirt/qemu/networks/autostart. Note that his network will replace the network coupled to the dummy0 interface, so dummy0 should not start up after a reboot. To do this, move /etc/sysconfig/network-scripts/ifcfg-dummy0 to a safe location, or edit it and change the ONBOOT option from "yes" to "no".Note: If you run any services on the physical host, which are bound to the network interface of the host only network, you need to watch the boot order. Most services are started before libvirtd. The Virtual bridges only exist after libvirtd has been started. Any services started before libvirtd will not be able to bind to the virbrX interface. Named (bind) for instance binds to the interfaces. If you use the host only network to access a nameserver on the physical hosts, you need to restart named after boot (of the physical host), or the guests cannot access the nameserver.The bridged network is a bit more complex. Use the configuration file of eth0 as a basis. cp /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-br0. Remove the lines crossed out below and change/add the bold statements. <div class="codesnippet">&amp;#13; /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 <del>BOOTPROTO=static</del> <del>BROADCAST=a.b.c.255</del> HWADDR=ab:cd:ef:gh:ij:kl <del>IPADDR=a.b.c.d</del> <del>NETMASK=255.255.255.0</del> <del>NETWORK=a.b.c.0</del> BRIDGE=br0 ONBOOT=yes /etc/sysconfig/network-scripts/ifcfg-br0 DEVICE=br0 BOOTPROTO=static BROADCAST=a.b.c.255 <del>HWADDR=ab:cd:ef:gh:ij:kl</del> IPADDR=a.b.c.d NETMASK=255.255.255.0 NETWORK=a.b.c.0 ONBOOT=yes TYPE=Bridge</div> Now br0 can be used as a bridge interface. All traffic over the bridge interface is subject to filtering by IPtables. I think this is a great feature and allows you to centralize firewalling on each host. Even better is that the firewall rules are now susceptible to change if the virtual machine is ever compromised. However, Xen worked in a different fashion. Our Xen based images will have their own firewall rules. To skip the firewall rules for the physical host do: <div class="codesnippet">echo net.bridge.bridge-nf-call-ip6tables = 0 &amp;gt;&amp;gt; /etc/sysctl.conf echo net.bridge.bridge-nf-call-iptables = 0 &amp;gt;&amp;gt; /etc/sysctl.conf echo net.bridge.bridge-nf-call-arptables = 0 &amp;gt;&amp;gt; /etc/sysctl.conf</div>Swap usage and caching If your physical machine is only running Virtual Machines and the memory is not oversubscribed (all VM's together use not more than 80-90%) of total memory, you might want to limit swapfile usage. Since the kernel sees the VMs as a process, rules for processes apply as well. One of those rules means that pages that are not referenced for a while are paged out to swap. The purpose is to free up memory to use for other processes or cache. This speeds up things that are being used. For a VM this is unwanted behavior. On a dedicated host nothing else does run and I don't want my VMs being cached, since that is already happening inside the VM. Double caching gives inconsistent performance behavior, let alone the effects when the host crashes.There are two ways to put a stop to paging and swapping. The first is not to create a swapfile at all. The second one is to set the kernel dwappiness parameter to a low value. I've set it to 0. <div class="codesnippet">echo vm.swappiness = 0 &amp;gt;&amp;gt; /etc/sysctl.conf</div> See the virtual machine config file on how to turn off disk caching for virtual machines.Converting the virtual machine configuration file There are two ways of converting to KVM. The easiest one is to use virt-manager and create a new virtual machine with exactly the same details as the old one, but point it to a different virtual disk (smallest possible) to prevent overwriting any existing data. Then stop the machine (no need to really install anything) and change the configuration file in /etc/libvirt/qemu by hand to point at the right disk image. This method requires you to reboot first. Else the configuration tools wont see the networks we just created.The other method is to convert the virtual machine definition by hand. Below is an old Xen definition file (/etc/xen/test1: <div class="codesnippet">name = "test1" uuid = "4a07fde8-f244-2a6d-9603-85ff2179a9bb" maxmem = 512 memory = 512 vcpus = 2 bootloader = "/usr/bin/pygrub" on_poweroff = "destroy" on_reboot = "restart" on_crash = "restart" vfb = [ "type=vnc,vncunused=1,keymap=en-us" ] disk = [ "tap:aio:/var/lib/xen/images/test1.img,xvda,w" ] vif = [ "mac=00:16:3e:1a:d0:96,bridge=xenbr0", "mac=00:16:3e:1a:d0:97,bridge=xenbr1" ]</div> This information can be converted into a KVM configuration file (/etc/libvirt/qemu/test1.xml. Take care to use the same MAC addresses for the network interfaces or else they won't be recognized when the virtual machine is booted. Also watch the serial and console arguments to not point to the same serial port for multiple VMs. You could use virsh list and virsh dumpxml as a starting point. However you must do this before starting with this howto. <div class="codesnippet">&lt;domain type="kvm"&gt;&lt;name&gt;test1&lt;/name&gt;&lt;uuid&gt;48156322-4e0c-b658-b80a-1bf3b608b49d&lt;/uuid&gt;&lt;memory&gt;524288&lt;/memory&gt;&lt;currentmemory&gt;524288&lt;/currentmemory&gt;&lt;vcpu&gt;2&lt;/vcpu&gt;&lt;os&gt;&lt;type arch="x86_64" machine="pc"&gt;hvm&lt;/type&gt;&lt;boot dev="hd"/&gt;&lt;/os&gt;&lt;features&gt;&lt;acpi/&gt;&lt;apic/&gt;&lt;pae/&gt;&lt;/features&gt;&lt;clock offset="utc"/&gt;&lt;on_poweroff&gt;destroy&lt;/on_poweroff&gt;&lt;on_reboot&gt;restart&lt;/on_reboot&gt;&lt;on_crash&gt;restart&lt;/on_crash&gt;&lt;devices&gt;&lt;emulator&gt;/usr/libexec/qemu-kvm&lt;/emulator&gt;&lt;disk type="file" device="disk"&gt;&lt;driver name="qemu" cache="none"/&gt;&lt;source file="/var/lib/xen/images/test1.img"/&gt;&lt;target dev="vda" bus="virtio"/&gt;&lt;/disk&gt;&lt;interface type="bridge"&gt;&lt;mac address="00:16:3e:1a:d0:96"/&gt;&lt;source bridge="br0"/&gt;&lt;model type="virtio"/&gt;&lt;/interface&gt;&lt;interface type="network"&gt;&lt;mac address="00:16:3e:1a:d0:97"/&gt;&lt;source network="hostonly"/&gt;&lt;model type="virtio"/&gt;&lt;/interface&gt;&lt;serial type="pty"&gt;&lt;source path="/dev/pts/2"/&gt;&lt;target port="0"/&gt;&lt;/serial&gt;&lt;console type="pty"&gt;&lt;source path="/dev/pts/2"/&gt;&lt;target port="0"/&gt;&lt;/console&gt;&lt;input type="mouse" bus="ps2"/&gt;&lt;graphics type="vnc" port="-1" autoport="yes" keymap="en-us"/&gt;&lt;/devices&gt;&lt;/domain&gt;</div> If you are using a partition as a virtual disk the Xen configuration disk = [ "phy:/dev/vgvm/lvmyvolume,xvda,w" ] translates to: <div class="codesnippet">&lt;disk device="disk" type="block"&gt;&lt;driver cache="none"/&gt;&lt;source dev="/dev/vgvm/lvmyvolume"/&gt;&lt;target dev="vda" bus="virtio"/&gt;&lt;/disk&gt;</div> If you want to bind the virtual cpu to a physical one use the following vcpu syntax: <div class="codesnippet">&lt;vcpu cpuset="cpu1,cpu2,cpu3"&gt;virtual cpus&lt;/vcpu&gt; for example &lt;vcpu cpuset="0,1"&gt;4&lt;/vcpu&gt;</div> Also see <a href="http://libvirt.org/formatdomain.html">http://libvirt.org/formatdomain.html</a> If you want to verify that the xml file is correct, use the virt-xml-validate command. now reboot the hostStarting the virtual machines You can now start the virtual machines by using the virsh command. Open a console directly after starting the domain to monitor boot progress. You also might want to start the machine after booting. <div class="codesnippet">virsh define /etc/libvirt/qemu/[mymachine.xml] virsh list virsh start [mymachines ID] virsh console [mymachines ID] virsh autostart [mymachines ID]</div>Improving Performance with Hugepages Note: There could be some unwanted interaction with SELinux here. If you run into problems, either don't use Hugepages or turn SELinux offKVM uses 4kB memory pages by default, just like any other process. One of the main differences between a normal average process and a kvm virtual machine process is the amount of memory allocated to it. Virtual machines normally use hundreds or even gigabytes of memory. This means a lot of overhead when the CPU switches between virtual machines since large memory tables need to be updated each time.&amp;#13;RHEL 5.4 and Hugepages Linux also has Hugepages, special memory pages that are 1,2 or 4MB in size, shortening the list of memory pages dramatically and improving performance up to 10%. Sadly, support for Hugepages hasn't been implemented into libvirt. There is work on it in Fedora 12, but I don't expect to see those developments in RHEL5. There is a way however. First lets start by reserving the Hugepages. The file /proc/meminfo should contain the Hugepage size of the system somewhere in the last lines.Now calculate the amount of Hugepages needed for the virtual machines and add at least 6 pages extra for each virtual machines. If you do not reserve enough pages, your virtual machine won't start. KVM uses some additional pages when starting up the VM, so if you don't add those 6 pages, the last VM will not start. Add the total of Hugepages to your kernel configuration by doing: <div class="codesnippet">echo vm.nr_hugepages = XXXX &amp;gt;&amp;gt; /etc/sysctl.conf</div> Make the Hugepages accessible to KVM <div class="codesnippet">mkdir /hugepages echo hugetlbfs /hugepages hugetlbfs defaults 0 0</div> Now the Hugepages are set-up (they become accessible after a system reboot). Lets rig libvirt so the Hugepages are actually used after a system reboot. To do this we need to move the qemu-kvm binary and replace it with a script of our own. The binary is located in /usr/libexec. Execute mv /usr/libexec/qemu-kvm /usr/libexec/qemu-kvm2. Now create the script /usr/libexec/qemu-kvm with the following contents: <div class="codesnippet">#!/bin/bash exec /usr/libexec/qemu-kvm2 -mem-path /hugepages "$@"</div> Now reboot the system and start your virtual machines like normal. Note: Be careful when updating the libvirt package. An update will overwrite our script, so you need to reapply the change after each libvirt update.RHEL 5.5 and Hugepages RHEL 5.5 has native support for Hugepages. First make sure that the libhugetlbfs package is installed. Then execute the huge_page_setup_helper command and answer the questions. <div class="codesnippet">[root@aurora ~]# rpm -qa | grep huge libhugetlbfs-1.3-7.el5 libhugetlbfs-1.3-7.el5 [root@aurora ~]# huge_page_setup_helper.py Current configuration: * Total System Memory......: 7909 MB * Shared Mem Max Mapping...: 7100 MB * System Huge Page Size....: 2 MB * Number of Huge Pages.....: 3550 * Total size of Huge Pages.: 7100 MB * Remaining System Memory..: 809 MB * Huge Page User Group.....: root (0)How much memory would you like to allocate for huge pages? (input in MB, unless postfixed with GB):</div> Now add the Hugepages mountpoint to /etc/fstab <div class="codesnippet">mkdir /dev/hugepages echo hugetlbfs /dev/hugepages hugetlbfs defaults 0 0</div> On next reboot there will be a problem, as /dev is governed by udev. That means that the hugepages mountpoint disappears automatically on reboot. To fix this a patch must be applied to /sbin/start_udev. To make sure thuis patch stays in place, even after rolling out a udev patch, a script has been created to see if the patch has been applied and if not, applies it.Download the patch <a href="http://www.gloudemans.info/publish/udev.patch">here</a> and place it in /usr/local/bin Download the init script <a href="http://www.gloudemans.info/publish/libvirt_hugepages">here</a> and place it in /etc/init.dThen do: <div class="codesnippet">chkconfig --add libvirt_hugepages chkconfig libvirt_hugepages on</div> As a last step, add the following to the virtual machile XML config files in /etc/libvirt/qemu (on the same level as &lt;memory&gt;) <div class="codesnippet">&lt;memorybacking&gt;&lt;hugepages/&gt;&lt;/memorybacking&gt;</div>Now reboot the system and the virtual machines should be started using Hugepages memory. You can verify this by looking at the qemu-kvm command in the process list. It should contain a -mem-path parameter now. If the Hugepages mountpoint is added after the system has rebooted, restart libvirtd, or else libvirt won't see the Hugepages.&lt;/memory&gt; Thu, 09 Jul 2009 20:08:00 +0200 http://www.gloudemans.info/migrate-paravirtualized-xen-to-kvm-under-rhel/ Canned Meat http://www.gloudemans.info/canned-meat/ Preventing Spam e-mails is so easy and yet, because a seemingly substantial amount server admins refuse to read up on the standards, so very hard. For instance, when you try to implement all the measures at <a href="http://calum.org/posts/postfix-antispam">http://calum.org/posts/postfix-antispam</a>, you'll find that a substantial amount of legitimate mail is refused by your mailserver. When I tried here in the Netherlands, almost 25% of all mail was rejected. I am talking about a seemingly amount of admins, because often it is not the admin who is to blame, but the ISP.For instance, a mailserver should have a forward and a reverse DNS lookup defined, as required per RFC1912. Next, a mailserver should HELO/EHLO (mailserver handshake) with a FQDN that is defined in DNSImplementing these rules alone will mean that many legitimate mails will not be accepted by your mailserver. All legitimate servers have the forward DNS lookup defined, but the reverse lookup if often missing or not matching the server name. This is almost always true for consumer type ADSL lines and also for many corporate SDSL ones. This is a flaw on the side of ISPs.Problems with the second rule are also common. Many mailservers have a different name on the internal network than on the Internet. This is not a problem, because with most mailservers you can configure the name the server uses in the handshake. In practice, many admins forget this.If these criteria cannot be used to fight Spam and Malware what measures are left? Plenty. Many of these are easy to implement. A how to implement in Postfix in included with each point.Use blacklists. There are many sources for good blacklists out there. A well known one is Spamhouse (<a href="http://www.spamhouse.org">www.spamhouse.org</a>) another one is njabl (<a href="http://www.njabl.org">www.njabl.org</a>). These list work using DNS lookups and are realtime. These lists can be applied in two ways; tag the incoming message as Spam when the sending server is on the blacklist or reject the mail altogether. I am using the last option now for a couple of years and I am quite happy with it. It keeps the load on the server down and hardly ever creates false positives.<div class="codesnippet">In /etc/postfix/main.cf smtpd_client_restrictions = permit_mynetworks,permit_sasl_authenticated,check_client_access hash:/etc/postfix/client_restrictions,reject_rbl_client zen.spamhaus.org,reject_rbl_client bl.spamcop.net,reject_rbl_client dnsbl.njabl.org,reject_rbl_client dnsbl.sorbs.net,reject_rhsbl_client rhsbl.sorbs.netMake sure to keep this order or you might inadvertently reject mail from the internal network or from users logging on to send their mail. Many blocks of IP numbers given uit by ISPs are on one blacklist ore more out of principle (for ISPs that do not support mailservers. The client_restrictions file contains a whitelist. You can also set a customized reply to the sending server to help instances who are sending legitimate mail: default_rbl_reply = $rbl_code Service unavailable; $rbl_class [$rbl_what] blocked using $rbl_domain${rbl_reason?; $rbl_reason}. Also see {some url with further explanations}</div>Use SPF (or DKIM). Sender Policy Framework (<a href="http://www.openspf.org/">www.openspf.org</a>) can be a good measure to not only prevent Spam from going in, but it also prevents others from using your domain name for Spamming purposes. DNS already contains a record for each server dat receives mail on behalf of your domain. An SPF record defines which servers are authorized to send mail on behalf of your domain. <div class="codesnippet">In /etc/postfix/main.cf smtpd_recipient_restrictions = reject_non_fqdn_recipient, permit_sasl_authenticated, permit_mynetworks, reject_unlisted_recipient, reject_unauth_destination, reject_invalid_hostname, check_policy_service unix:private/policyd-spf Again, the order of statements is important. The reject_unlisted_recipient is the implementation of the 4th point. The following should be added to /etc/postfix/master.cf policyd-spf unix - n n - 0 spawn user=nobody argv=/usr/bin/python /usr/bin/policyd-spf Before restarting postfix the SPF software should also be installed. Get policyd-spf at <a href="http://www.openspf.org/Software">http://www.openspf.org/Software</a> and the SPF library at <a href="http://sourceforge.net/project/showfiles.php?group_id=139894&amp;package_id=191419">http://sourceforge.net/project/showfiles.php?group_id=139894&amp;package_id=191419</a> . If you are running RHEL5 or Centos5, get the src.rpm and rebuild the installable package with "rpmbuild --rebuild {source-package}"</div>Use Greylisting (<a href="http://www.greylisting.org/">www.greylisting.org</a>). This measure is not without controversy. Greylisting looks at the sender and receiver addresses as well as the sending server. If this combination is new, the server will return a try again later message. A legitimate mailserver will keep retrying until the mail is delivered. The time after which the mail is accepted is configurable. Spam servers will give up, or will be identified by a blacklist when trying again. The next time a mail cones in with that same recipient, sender and server the mail is accepted immediately. Greylisting can cause problems. Some legitimate servers will not retry and mail is getting delayed, so users should be aware of this mechanism. <div class="codesnippet">For RHEL compatible systems, get the postgrey daemon at <a href="http://dag.wieers.com/rpm/packages/postgrey/">http://dag.wieers.com/rpm/packages/postgrey/</a>. Make sure it is started before postfix. Next in /etc/postfix/main.cf smtpd_recipient_restrictions = reject_non_fqdn_recipient,permit_sasl_authenticated, permit_mynetworks, reject_unlisted_recipient, reject_unauth_destination, reject_invalid_hostname, check_policy_service unix:private/policyd-spf,check_policy_service unix:postgrey/socket</div>Only accept mail that can be delivered. This sounds trivial, but many servers will accept mail to any username on a given domain. If the mail cannot be delivered to an end-user the server will send an undeliverable message back. This adds to the load of the mailserver and, more importantly, Spammers are using this mechanism to get Spam into your mailbox. Send a Spam message with the intended recipient as sender to a non-existing user on a legitimate server. That server will send a non-delivery report, with the original message to the intended recipient. This is called backscatter Spam. Implementing this point is harder than it looks. It is easy enough for an all in one mailserver, but when a mail gateway, backup mail gateway or a Spam/virus scanning appliance is used, this is often not implemented if at all possible. Also, do be careful when implementing. All my users are in a LDAP directory. The MTA can talk to the LDAP server directly, but what happens when the LDAP server is down? Right, all mail gets rejected. That is why I use a script to generate a recipient list a couple of times per day.<div class="codesnippet">Implementation has already been shown. Below is the script I use to populate the recipient map from LDAP #!/bin/sh# Generate the clear file with addresses ldapsearch -l 60 -x mail zimbramailalias | grep "@" | cut -f 2 -d ' ' | sort -u &gt; /tmp/my_local_recipient_table.txtlines=`cat /tmp/my_local_recipient_table.txt | wc -l`if [ $lines -lt 15 ] then echo Local recipient table generation failed. Exiting exit fi# Move over to postfix configuration if [ -f /etc/postfix//my_local_recipient_table.txt ] then rm /etc/postfix/my_local_recipient_table.txt fifor locrcpt in `cat /tmp/my_local_recipient_table.txt` do username=`echo $locrcpt | cut -f 1 -d '@'` echo $locrcpt $username &gt;&gt; /etc/postfix/my_local_recipient_table.txt done/usr/sbin/postmap /etc/postfix/my_local_recipient_table.txt Implement the list as follows in /etc/postfix/main.cf: local_recipient_maps = hash:/etc/postfix/my_local_recipient_table.txt unknown_local_recipient_reject_code = 550</div>Use the right tool for the trade and read up. Not an experienced admin? No problem, but then you should stay away from an Exchange server, because you can configure it without knowing what you are doing, but stay away from Sendmail as well because it is too complex. Rather start with a simpler MTA like Postfix or Courier.RHEL/CentOS use sendmail by default. Executing <div class="codesnippet">yum install postfix system-switch-mail -or- up2date -i postfix system-switch-mail system-switch-mail</div> You can change the default from sendmail to postfix. Suse systems use postfix by defaultThese measures all sound very fine, but if you are in the circumstance that your ISP requires you to send mail via their gateway, you are out of luck. Number 1 and 3 won't work, since mail is coming from the gateway always and 2 cannot work for inbound mail. It could work for outbound mail, but when I asked my previous provider (KPN) for a list of gateway addresses they told me it was secret. I switched providers immediately.Now lets look to my setup, or at least, how it was until a couple of weeks ago. My mail solution comprises of 2 servers. One is a Zimbra installation (open source edition) and one acts as filtering MTA. I chose this setup, because Zimbra is rather intolerant of other software on the server and configurations done outside of the Zimbra GUI or command line interface is bound to get lost on upgrades. Zimbra includes Spam and virus scanning and is doing a good job, but I wanted more.<img src="/assets/blog/MTAMailScanner.png" />That's why I added a separate MTA, in which all of the above rules are implemented. On top of the measures in Postfix I also added <a href="http://www.mailscanner.info/">MailScanner</a> to the solution because I wanted: <ol style="list-style-type:decimal;"><li>have a second virusscanner </li><li> prevent problems because of HTML mails which redirect users to malicious sites </li><li> blacklist some file types (can be done in Zimbra as well) </li><li> MailScanner can be used to convert incoming Word documents to text. Ideal if you want to have a quick scan of the included document. </li></ol>This setup worked really well for most of the time, but I had some problems with it as well:<ol style="list-style-type:decimal;"><li> Mail that is forwarded from other mail accounts, or fetched using fetchmail bypasses all checks in Postfix. There is still a double check on virusses, but enabling SpamAssassin (<a href="http://www.spamassassin.org/">http://www.spamassassin.org</a>) is pointless, since Zimbra is using SpamAssassin as well. Performance of SpamAssassin is still impressive though. Only 5% of all Spam in forwarded mail was let through, which translates to 4 to 5 Spam mails in my personal box per day. </li><li> MailScanner will automatically try to update virus and phishing definition libraries, but does not detect of the update was successful. Because of this the MTA went without virus definition updates for a month. </li><li> MailScanner has a few conflicts with other packages on the system. This causes problems when those other packages need to be patched. (The commercial version of MailScanner doesn't have this problem) </li><li> MailScanner is relatively slow. For instance, if you want MailScanner to scan for viruses, you can use it with almost every scanner out there. Sadly, only the command line versions of those scanners are free for non-commercial use, except ClamAV (<a href="http://www.clamav.com/">www.clamav.com</a>). But since Zimbra is already using ClamAV, it is pointless to use it with MailScanner as well. The result is, that for each incoming mail the virus scanner needs to be started. This causes quite a load. </li></ol>In December I got an invitation from the People at <a href="http://www.bitdefender.com/">BitDefender</a> to partake in the beta of the next version of the Unix desktop virus scanner product. Out of curiosity I downloaded the beta. Before I knew it I was ranked as the top tester and god rewarded with a BitDefender product of choice for 35 users with a one year license. After sending a mail back with the message that I was not representing a company and that a one-year license was not of too much use, they kindly granted me a three year license for the same.Last september I became a dad. This is a wonderful experience, but tends to diminish the time you have available to look after your server. From a commercial product I expect an emphasis on Total Cost of Ownership/Return On Investment (choose your poison ;-) ). In my mind this translates not only to a good hit rate on viruses and spam, but also to low effort for administration. This together with the 4 points mentioned earlier lead me to replace MailScanner with <a href="http://www.bitdefender.com/PRODUCT-32-en--BitDefender-Security-for-Mail-Servers.html">BitDefender for Unix Mailservers</a>. I had to sacrifice some functionality like the conversion of word documents but I would gain:<ol style="list-style-type:decimal;"><li> A second sophisticated Spam scanner </li><li> Guaranteed updates of the virus scanner plus alerts </li><li> A slick interface (GUI and command line) to configure BitDefender </li><li> A better maintained server with less effort </li></ol>BitDefender works a bit different from MailScanner. Where MailScanner picked up queue files, BitDefender works as a SMTP proxy. Since the earlier mentioned anti-spam measures are not implemented in BitDefender (except real-time blacklists, but only as tag as Spam only), I need a Postfix in front of BitDefender and one after to deliver the mail. The BitDefender installation process configures all this automatically. Starting from a properly configured Postfix, the installation an configuration only took a few minutes.<img src="/assets/blog/MTABitDefender.png" />The setup is not optimal in terms of performance, but compared to the old MailScanner setup it is better. Performance can be enhanced by following the instructions in the Postfix manual (<a href="http://www.postfix.org/FILTER_README.html">http://www.postfix.org/FILTER_README.html</a>). <div class="codesnippet">The BitDefender install makes the following changes; to /etc/postfix/main.cf it adds content_filter = smtp:127.0.0.1:10025 to /etc/postfix/master.cf it adds 127.0.0.1:10026 inet n - n - 10 smtpd -o content_filter for better performance change this to: 127.0.0.1:10026 inet n - n - 10 smtpd -o content_filter= -o receive_override_options=no_unknown_recipient_checks, no_header_body_checks, no_milters, no_address_mappings -o smtpd_helo_restrictions= -o smtpd_client_restrictions= -o smtpd_sender_restrictions= -o smtpd_recipient_restrictions=permit_mynetworks,reject -o mynetworks=127.0.0.0/8 -o smtpd_authorized_xforward_hosts=127.0.0.0/8</div>The setup has been working flawlessly ever since. One patch came out after installation. I was alerted automatically and patch installation cost me 2 minutes. This is much less compared to the 30 minutes I had to spend at least for each MailScanner update.As for the Spam volume that is getting through? I'm now down to one Spam mail per day. I estimate this is less than 1% of the total Spam volume. I had to set BitDefender to a medium aggressive setting. And no false positives so far. The fact that I have nothing more to report on BitDefender shows it strength; it just works. Sat, 07 Feb 2009 15:01:00 +0100 http://www.gloudemans.info/canned-meat/ Get Thin and make the World a better place http://www.gloudemans.info/get-thin-and-make-the-world-a-better-place/ I'm evaluating Suns <a href="http://www.sun.com/sunray">SunRay</a> technology for several customers at the moment. The SunRay is a very dumb graphical terminal requiring only between 3 and 4 watts of power. The desktop environment is running on a central computer. This can either be a virtual machine per user (VDI) or a multi-user operating system like Linux. (<a href="http://www.gloudemans.info/back-to-square-one/">suggested reading before this blog</a>)Sun delivered a terminal to me for demonstration purposes at the client location. After a successful demo I didn't have to return the unit.I thought it was cool to have in the home, but didn't really have a purpose for it. I have a server, more like a desktop with some extra memory added, which is on 24x7 and is located in a spot where I don't hear the noise. The server however was not sized with remote desktops in mind. The specs were dimensioned for a mail and web server.After some thinking I realized that I could use the living room TV also as a monitor. The SunRay has a DVI connector and DVI to HDMI cables are readily available. Since it would be connected to the TV, I would also need a wireless keyboard with integrated trackball, with sufficient range.These are hard to come by, especially if you want one that works with a SunRay. The software is more aimed at a Sun specific keyboard than supporting the variety type of keyboard available at the local electronics store (most of these things are aimed solely at Windows users, even a modern Linux variant already has problems with most of them). Eventually I found <a href="http://www.trust.com/products/product_detail.aspx?item=14909">one</a>, which is not too expensive. The only downside is that it does not have a third mouse button.In Linux/Unix the 3rd button is used for pasting copied text, I use is quite often. For the rest this keyboard worked out of the box.The next problem was the network connection.I still had some ports available on the switch near to the server, but the problem was getting the cable from the SunRay to the switch. I was thinking about using the duct also used for cable TV. Not the most pretty solution since that would mean a network connector box on the wall. The amplifier is just below the TV an to it is connected a <a href="http://www.slimdevices.com/">Squeezebox</a> by Logitech. This streaming media player is using the wireless network, but can also be connected to a wired network. What's more, the Ethernet port can be bridged to the wireless network. I could just plug the SunRay into the Squeezebox!The last step was creating a virtual machine to run the SunRay and desktop software in. In the beginning, performance was horribly slow. But after adding some memory to the server, performance was fine.The SunRay only gives you near-desktop performance, so no advanced window switching effects etc., but for reading mail and browsing this is just fine.Initially I had some problems with the Sun software (SRSS 4.1). After a couple of days I couldn't login anymore and sound didn't work. The first problem was easily solved. The SRSS software creates some directory and files in the /tmp filesystem. RedHat Linux has a cron job cleaning up /tmp daily. After a few days some of the files from SRSS were removed, causing the problem. This can be solved by adding "-x /tmp/SUNWut" to /etc/cron.daily/tmpwatchSound was harder to resolve. Sound is realized using a kernel module, as is remote storage (you can connect external storage to a SunRay station). The first problem is that the kernel modules need to be recompiled each time the kernel is patched. I've created a script to do this. The second problem is that the sound module is an OSS module. All modern Linuxes use ALSA. This is why the sound daemon crashes on startup. To resolve this, the sound daemon has to be recompiled without ALSA support. You can download the resulting RPM for 64bit RHEL5 <a href="/assets/Uploads/esound-0.2.36-30.x8664.rpm">here</a> and the adapted source RPM <a href="/assets/Uploads/esound-0.2.36-30.src.rpm">here</a>.The last problem to get sound working is in /tmp again. A /tmp/.esd directory is needed. This needs to be created at boot time. It also needs to be excluded in the tmpwatch script.Now sound works. For some reason the gnome sound configuration dialog does not work on my desktop, but you can enable desktop sounds with gconf-editor as well (/desktop/gnome/sound).At that point I got my second SunRay station. Not really knowing what to do with it, I connected it to the second input of my desktop monitor, thinking it would be easy if I wanted to quickly check my mail. The SunRay is almost instant on. To my suprise haven't been using my desktop computer at all since that time. I'm one of those persons who spends some time tweaking the spiffy desktop effects each time I install my PC, but for some reason the instant on (and off) capability, combined with the absolute silence of the SunRay has won me over. Additional advantage is that I am using the same desktop when I'm using the SunRay in the living room.So lets get back to the title. The most power hungry system is the house is the desktop system. It probably uses 150W when not doing much. Before the SunRays that system was on at least 3 hours per day year round. With the SunRay I'm only using the desktop for 30 minutes per day (gaming and processing digital images). 150W for 3 hours per day is about 165kWh per year. 30 minutes is 28kWh. So I'm saving 137kWh per year. This equals to 34 euros, but more importantly to 62kg (450grams per kWh) of CO2. In most companies where desktops are using 100W year round, this could be as much as 200 euros per desktop or about 390kg of CO2. This comparison is not really true, because my server only uses a bit more power when using the SunRay, but a company has to run extra servers. But no more that half of those savings need to be invested in servers.Another thing to mention is that I only use Solar, Wind and Water power to power the computer systems, so I'm almost producing no CO2, but I am saving 35 euros per year. Mon, 26 Jan 2009 13:30:00 +0100 http://www.gloudemans.info/get-thin-and-make-the-world-a-better-place/ Nvidia RAID Users; Fedora 10 might eat your data http://www.gloudemans.info/nvidia-riad-users-fedora-10-might-eat-your-data/ We interrupt this boring strings of blogs for some real news.Out of habit and curiosity I always install the latest Fedora release on my desktop as soon as it is proven that the Nvidia display driver works (Hey, if you work for large cooperations, Quake 3 or OpenArena is a necessity ;-) ). This is exactly what I did when Fedora 10 came out. It installed OK and then ran fine ... until I decided to change something in the grub config.I made a typo, so the system didn't boot any more. No worries, the rescue CD is here. As soon as I repaired the error I got some disk warnings on the new spiffy Fedora start screen. Initially I ignored these. In the session that followed I got bus errors from Java. My wine apps wouldn't even start any more, so I rebooted and pressed &lt;esc&gt; during the boot so I could see the start-up messages scrolling by. I had forgone this initial step during first boot; and with disastrous results. During boot the system is complaining about a duplicate physical volume (PV) ID and it tells me it is using /dev/sdb and not /dev/sda. Hang on; I have the raid controller on my Asus A8N-E (Nforce4 chipset) configured. Shouldn't it be using /dev/dm-[something] of even something from /dev/mapper? Turns out the raid set is not initialized and because it is one of those fake-raids, the individual disks are reported to the system as well.So here is what happened; the raid set was never initiated after installation (during installation it went fine) and only the second disk from the raid set was used. This will work fine, as long as this disk doesn't run into problems. The rescue CD however initializes the raid controller properly. When that is the case, the first disk from the set is leading. So when I repaired grub using the rescue disks, I actually corrupted the disk Fedora 10 was using.This bus is already reported to the Fedora bugzilla: <a href="https://bugzilla.redhat.com/show_bug.cgi?id=474697">https://bugzilla.redhat.com/show_bug.cgi?id=474697</a>.The scary part of this is what will happen if this bug is fixed. If no special attention is called to it, after the installation of the patch for this problem all Nvidia raid users will, after reboot, be confronted with the system as it was right after initial install, with no more access to the data.The only way out then is to boot from the rescue disk. Do not mount the filesystems on disk (this is what caused my problem), but do initialize the network card. If you are using lvm you can then discover and activate the volume group without initializing the raid array ("lvm vgscan" and "lvm vgchange -a y [volume group name]"). If you are not using lvm, you can skip that step. You can then mount the volume or partition to see if you still have access to your data, but I suspect you will have to do a disk check first (use fsck -f -a; because there will be a lot of errors). After I repaired the disk, most of my stuff was under a subdir in the lost+found of that disk. I didn't lose anything (except my pride) and luckily it happened only days after installation, so I still had a recent backup.What should you do when you have a Nvidia raid controller and haven't noticed yet? <ul><li> Hit &lt;esc&gt; during boot and see if you are hit by this problem. If you see something about a duplicate PV you have a problem </li><li> Continue the boot and backup your data to another system of DVD </li><li> Reboot you system and disable the raid controller from the BIOS setup </li><li> Reboot and re-install Fedora using software RAID. And no, you will not loose any performance, the hardware RAID was fake anyway. You were already using your main CPU to mirror the data. </li></ul> Sun, 07 Dec 2008 08:45:00 +0100 http://www.gloudemans.info/nvidia-riad-users-fedora-10-might-eat-your-data/ Does TCP/IP have a future in the data centre? http://www.gloudemans.info/does-tcp-ip-have-a-future-in-the-data-centre/ Network Convergence is a hot topic nowadays. This is driven mostly by storage networking. The first products which can deliver Fibre Channel over Ethernet (FCoE) are starting to appear even now. There are a couple of drivers for developing Ethernet for storage networks: <ul><li> Ethernet development is ahead of Fibre Channel and Infiniband in terms of speed </li><li> Common network layer simplifies infrastructure management </li><li> Reduction in the number of network interfaces and thus initial hardware cost </li></ul>Ethernet is a best effort protocol by nature and as such not suitable for storage networking, which requires guaranteed delivery of the data. Furthermore, native <a href="http://en.wikipedia.org/wiki/Fibre_channel">Fibre Channel</a> and <a href="http://en.wikipedia.org/wiki/Infiniband">Infiniband</a> have lower latencies that common Ethernet. To bring all these worlds together, new developments are taking place on the IEEE802.1 standard, which describes Ethernet. These developments fall under the common denominator Converged Enhanced Ethernet (CEE) also known as Lossless Ethernet. Though the first FCoE products are starting to appear, the new standard is not finished yet and the new equipment is pre-standard. CEE is expected to be ready for large scale deployment in 2010.Average data centre networks use Ethernet. Even for inter-data centre connects Ethernet is used more and more. On top of that the most used protocols are TCP/IP and UDP/IP. If Ethernet is made lossless, one could wonder what the advantage of TCP over UDP really is. A list of functional specifications of the IPv6, UDP and TCP protocols:IPv6: <ol style="list-style-type:decimal;"><li> Can make use of heterogeneous network </li><li> Addressing <ol style="list-style-type:lower-alpha;"><li> Address scopes (link local/site local/global) </li><li> Routing; the ability of packets to traverse from one network to another. The use and usage of routing within the data centre is limited. Routes are static and different sub networks for security, but only in combination with VLANs to provide network separation. </li><li> Address auto configuration; IPv6 interfaces can configure themselves automatically, starting by querying the nearest router using their link-local address, which is normally derived from the interface's Ethernet MAC address. </li><li> Multicast; The possibility to address a select group of hosts at once </li></ol> </li><li> IPsec; IP layer encryption </li><li> Prioritization; Used for flow control. Is now well described yet. </li><li> Jumbograms; IPv6 datagrams can be much bigger than IPv4 datagrams. Sizes up to 4Gb are allowable. </li></ol>So how does this compare to Ethernet: <ol style="list-style-type:decimal;"><li> We are talking network convergence here. The network is uniform by definition (OK) </li><li> Ethernet only has the MAC address (OK) <ol style="list-style-type:lower-alpha;"><li> No scopes exist. Addressing each individual system is a major task. Since the MAC address, for our purpose, is just a random number, some mechanism to manage the MAC addresses is needed. An equivalent to DNS at the very least. (Partially) </li><li> As mentioned earlier, routing within the data centre is limited. Network bridges can be created. <a href="http://www.ieee802.org/1/files/public/docs2006/aq-nfinn-shortest-path-2-0106.pdf">802.1aq</a>, Shortest Path Bridging, will optimize the network paths (Partially) </li><li> Each interface comes with is MAC address preconfigured (OK) </li><li> Ethernet has multi casting capabilities as well (OK) </li></ol> </li><li> 802.1X, Network Access Control and <a href="http://www.ieee802.org/1/files/public/docs2006/ar-borza-draft-0_3prime.pdf">802.1AR</a>, Secure Device Identifiers can provide part of the functionality. As far as I know no current standard for encrypting the Ethernet payload exists (Partially) </li><li> Prioritization is handled in <a href="http://en.wikipedia.org/wiki/IEEE_802.1p">801.1p</a>, Class Based Flow Control (CBFC) which uses the 3 bit Priority Code Point from the VLAN header (OK) </li><li> Ethernet can use Jumbo Frames which size up to 9K. Since even the Ipv6 Jumbograms must use Ethernet, the best efficiency from a network point of view will be Jumbo Frames, without any IP at all (OK) </li></ol>Now on to UDP. UDP provides checksumming and application multiplexing. Checksumming is also provided in the Ethernet header. Application multiplexing has no Ethernet equivalent. Application multiplexing (the usage of UDP ports) is a core requirement on networking.The benefits of using TCP over UDP: <ol style="list-style-type:decimal;"><li> Ordered data transfer </li><li> Retransmission and discarding of duplicate packets </li><li> Flow control </li><li> Congestion control </li></ol>These benefits provide nothing new over what CEE already offers: <ol style="list-style-type:decimal;"><li> Ethernet relies on in order reception of packets. Each packet will take the same route (OK) </li><li> not needed, since in principle Ethernet should not lose any packets (OK) </li><li> Flow control is handled by 802.1p and az. Per priority pause frames can be send, selectively throttling data flows. This mechanism is probably more efficient than TCP's bandwidth scaling algorithm. (OK) </li><li> is handled by 802.1au, Congestion Notification. Congestion points en route (bridges) notify end points with rate limiting capacity. (OK) </li></ol>So there it is, for local network traffic, using TCP will be overkill and a reduction of network efficiency. Looking at the trend in Ethernet development, more and more functionality from the higher OSI layers is pulled into the Ethernet protocol. The basic reason why we have IP at all is the fact that the Internet is a heterogeneous environment. With Ethernet becoming more and more the by default standard, the need for higher protocols evaporates. Though Ethernet will never cover the full functionality of the IP stack, though I could envision a protocol that is backwards compatible with Ethernet that can, but that will probably not happing within the next 20 years. Network management would be so much simpler, if there were one standard protocol (not a protocol stack) that would give guaranteed data delivery to one or more systems within a given time. Mon, 29 Sep 2008 08:45:00 +0200 http://www.gloudemans.info/does-tcp-ip-have-a-future-in-the-data-centre/ Back to Square One http://www.gloudemans.info/back-to-square-one/ In 1992 when I started my study in Astronomy, most students at the physics department were using X-terminals. Workstations were still fairly expensive then and had more power than any one person needed (on average). The X-terminal was a fairly dumb device. It had an X server for graphical representation and tftp client for booting. They worked very well in combination with UNIX servers.Around 1996, the Linux era began. PC's were even cheaper than terminals, with computing power that was beginning to rival the workstation. Lots of terminals were replaced with Linux PC's. This brought the administrators more work, but as the demand for computing power kept growing, this was good solution as the total cost of the infrastructure as a whole was hardly growing. An additional advantage was, that the PC's could be installed with Windows (which was/is not a multi-user OS) and Linux, giving the user a choice of what to use. So you might say that Windows and Linux collaborated to push the terminal out.Now that almost every employee has swapped his terminal (graphical or not) for a PC, we see that the PC at every desktop model has some disadvantages. In today’s environment it becomes harder and harder to keep the desktop environment under control. Viruses are rampant, software lifecycles become shorter and the pressure to keep IT cost under control higher.A solution for this is to revert to the terminal of old. The main advantage of the terminal is that there are no moving parts. Terminals hardly ever break and if one does, make sure there is a spare in the broom closet. Every person who can unplug a cable can install a new terminal.Working from a central environment has several advantages, among which: <ul><li> Backups are a breeze, since all desktops are near tape drives </li><li> Migration becomes easy as well, no need to go out to the desktop to install programs. Full platform migration, for which a reinstall on the desktop would be needed, becomes much easier as well, rollback scenarios included. </li><li> Patch management, patch when needed, not when the desktop happens to be on </li><li> Reduce in power usage. One server that replaces ten desktops or more uses a lot less power. The terminal devices themselves typically use less power than a single hard drive </li><li> Terminal devices even exist in laptop form. Add a UMTS card to it and an employee could work everywhere, without taking sensitive company data outside the corporate infrastructure. Terminals in combination with VPN technology could also be used from home using a normal Internet connection. </li><li> Terminals are cheaper than PC's and last much longer. So hardware expenditure should be lower as well. </li></ul>The reason behind the terminal's revival? Desktop Virtualization! Most office workers are still working with the Windows OS, which still isn't a true multi-user operating system. On top of that, many users are loath to give up the privacy of their own desktop. This last argument is more psychological than real, since the administrator already has free access to their desktop.New products in this area allow you to "remote" the display of the virtual system to a dumb device. This method also works around the typical difficulties with Terminal Server. This is important, because not all applications, even from Microsoft, are working well when started from a terminal. With the desktop virtualization products programs don't even have to know that they are running from a terminal, so those problems are solved.However, desktop virtualization is not a must for using remote desktops/thin clients. Before any such project is started, an inventory should be created on which users could just work directly of a Windows server and which users would need their own virtual desktop. Chances are that most users will be able to share a server, reducing the maintenance effort for lots of desktop images and the license cost for the virtualization product. Migrating to Linux or any other UNIX for that matter will fully eliminate the need for any virtual desktops, since multi-user is one of the core strengths of UNIX. If you do need Windows only applications, <a href="http://www.codeweavers.com/">Codeweavers Crossover Office</a> might be a good suggestion. I myself have been using it to run Outlook and Visio on Linux for years now. (Note that it works on Apple too!)Using remote/virtual desktops has a few downsides though. At the core of these downsides is the network. During operation, the terminal will send keystrokes and mouse movements to the server and the server sends back the changes in graphical representations on the screen. For office work, this amounts to something between 50kb/s to 100kb/s. For an active user this goes up to around 1Mb/s. From this perspective, a remote office with about 10 employees could still be connected through an ADSL/SDSL connection.Trouble begins with the new and flashy desktop effects present in Vista and also many new Unices. Enabling these will have a major impact on the network. Since these effects are not strictly necessary, they could be disabled. Playing a movie on a terminal will have the same effect though, so the web broadcast of the New Year’s speech of the CEO will cripple the network. Solutions in this area are coming though, in the form of MPEG decryption at the terminal level instead of the server.Another problem is more day to day and has a larger impact; printing. Many printers in the corporate environment use PostScript as the printing language. A document containing a few images (the company logo in the template!) translates to a printjob several, of not tens of, megabytes in size. Now we return to the remote office with the 10 employees. A printjob in this environment might take minutes, up to hours.About the same problem rises when an employee connects his/hers USB stick to the terminal to take a document home.A whole constellation of options exist to resolve these problems. A few examples: <ul><li> Change the way people work. By distributing e-paper based devices, the amount of print work could be reduced. The environment could also benefit from this. </li><li> Some specialized network devices exist today, than can compress network traffic on the fly and do some other smart network tricks to optimize and usage of bandwidth </li><li> Change printer protocols. There must be some printers out there that can work natively with PDF files instead of PostScript. PDF's use compression, so the size of the printjob could be reduced massively. </li></ul>In my opinion the terminal is back and here to stay. This new generation of the terminal is maturing rapidly, though some details still need to be ironed out. The remote desktop should not be confused with the virtual desktop though. The virtual desktop is solution for users who do require the privacy of their own desktop and applications that won't work well in the multi-user environment. From that perspective, it is a very useful add-on for the remote desktop, but for most users, it won't be needed. Tue, 22 Jul 2008 08:44:00 +0200 http://www.gloudemans.info/back-to-square-one/ Virtualisation Tips III: Technicalities http://www.gloudemans.info/virtualisation-tips-iii-technicalities/ In the previous 2 blogs, the soft side of virtualisation was touched. This part is about the real stuff (no offence to non-techies meant). Though this part is technical, there are some tips to save money below, so non-techies may want to read this as well.Tip III.1: Bring on the load Most appliances in the data centre have optimal efficiency at about 80% load. This is true of server power supplies, CPU's themselves, but also the air-conditioning. So in a data centre that is not full, either because it is new, or because all services have migrated to virtual servers and the rest of the hardware is turned off, turn off some air-conditioning units. 18-20 degrees Celsius is cool enough. One unit with 80% load uses a lot less energy than two running at 40%.From the cost perspective, this is what virtualisation is all about. Making more efficient use of hardware. It also means, that is is not wise to push beyond the 80% barrier, as energy consumption will that to rise exponentially.Tip III.2: Start small For the deployment of new services, start as small as possible with hardware. Virtual machines can grow, dependent on the Hypervisor, even without downtime. Create an agreement for fast delivery of new physical servers with a hardware vendor and scale up when you need it.Normally hardware for a new service was created with the 3 or 5 year depreciation cycle in mind. As long as the infrastructure is set up in a scalable way, you only need to scale to survive the delivery period of new hardware, which is typically one month.Tip III.3: Think big Big boxes (servers) bring flexibility. With the current technology, virtual systems can only scale to the limit of the physical server. Furthermore, to make optimum use of virtualisation, at least more than a couple of virtual machines should be running on the same hardware. So if you use virtual machine with more than 2 virtual CPU's, you'll want big machines.The biggest x86_64 compatible servers currently on the market have 32 CPU cores and can allocate 512GB of memory. These boxes are ideal. Also, because they have lots of usable PCI-e slots. You'll need these for you I/O interfaces.Many vendors try to make you believe that blade servers are the optimal choice for virtualisation. Not so. Blades typically have only few CPU cores and have problems if more than 2 network interfaces are needed.Blades to to be seen as a competitive technology to virtualisation. (It has its merits, but this us a blog about virtualisation)In terms of network, the ideal network for bix boxes is 10GbE. The technology is coming down in cost rapidly. With 10GbE, you'll probably not need more than 2 interfaces per server. You'll need at least 2 for availability purposes. More than 2 interfaces will actually be useless, as a typical server will even have problems fully utilizing one 10GbE card.Tip III.4: 64 not 32 Virtualisation should only be used in combination with 64 bit operating systems. 32 bit systems have a limit of 4GB of memory, which is not enough. Theoretically it is possible to run 32bit virtual systems on a 64bit Hypervisor, but is is best to keep everything 64bit.Tip III.5: Use the dynamics Modern Hypervisors ans operating systems may allow you to dynamically manage resources. Extra CPU's and storage can be added at runtime. This allows you to shift resources around. More resources to batch oriented applications at night and more to on-line during the daytime, if you're not a 24x7 web shop of course.Though resources can be added and removed on the fly, it is not a good idea to do this with internal memory allocations. Adding more memory will not be a problem, but taking it away might be, since all unices try to use all memory in the box, either for applications, or for cache.Dynamic allocations can also be used for fail over purposes. Most Java based application servers can be clustered. Typical deployments contain at least 2 application servers and a load balancer. For many applications this is overkill. Virtualisation in combination with central storage can offer a solution using live migration of running virtual systems to other physical machines. This can be done without interruption. However, if the original physical machine suddenly dies, live migration is not possible and the virtual system will have to be booted on another piece of hardware. This will cause a service interruption of a few minutes.If this violates your requirements, you could use a cluster of 2 virtual machines. One to handle all requests and one as standby. The standby virtual system can be run with very little resources (no load anyway) and will thus have little impact on the other virtual machines running on that same physical box. If the physical box with the primary virtual system dies, the load balancer will transfer the load to the second virtual system. This one can then scale up in resources. This one is not my idea, but was given to me by a consultant from RedHat. He assured me that this was working well in real world situations. Fri, 06 Jun 2008 08:44:00 +0200 http://www.gloudemans.info/virtualisation-tips-iii-technicalities/ Virtualisation Tips II: Service Management http://www.gloudemans.info/virtualisation-tips-ii-service-management/ At the introduction of virtualisation in any organisation its impact on service management is often overlooked. Virtualisation is implemented as just another technology. To gain maximum profit from virtualisation changes in the way services are managed are needed. Something, that became clear already in "<a href="http://www.gloudemans.info/virtualisation-tips-i-the-human-factor/">Virtualisation Tips I</a>". Changes in processes and procedures might even be necessary to remain legally compliant. If cost is the main driver behind virtualisation, the systems management approach is absolutely critical. Not only in processes and procedures, but also in tooling. Tip II.1: Review the service chain Hosting is a component in the service chain. If virtualisation encompasses on OS platform and one set of service levels, it is best to leave it that way. However, when several different sets of service levels and multiple OS are involved, not to mention the virtual teams from "<a href="http://www.gloudemans.info/virtualisation-tips-i-the-human-factor/">Virtualisation Tips I</a>" is is probably best to split this component from the service chain in 2, hosting and virtual hosting.This helps in a couple of ways: <ul><li> It gives mote insight into operational cost and comparison before and after are more clear; </li><li> Clearer picture on how the final service levels are composed; </li><li> Helps in separating the virtual from the physical, the physical part might even be outsourced. </li></ul>Tip II.2: See virtual and physical as separate The virtual and physical worlds are interconnected. Full separation is not possible. However, by keeping them separated as much as possible, flexibility increases. If the virtual infrastructure is not dependent on the physical infrastructure at all, than all virtual infrastructures layouts are possible. The complete infrastructure could be remodeled at the push of a button. This might be a good feature at the introduction of a radically new application, company reorganization or a change in business model. It reduces the financial risk in any IT project involved.If physical and virtual are separated, it also creates more degrees of freedom in outsourcing deals. One might decide to keep management of the virtual infrastructure and outsource the physical (cloud computing by <a href="http://www.amazon.com/EC2-AWS-Service-Pricing/b/ref=sc_fe_l_2/105-3428288-0580454?ie=UTF8&amp;node=201590011&amp;no=3440661&amp;me=A36L942TSJ2AJA">Amazon</a>/<a href="http://yhoo.client.shareholder.com/press/releasedetail.cfm?ReleaseID=301187">Yahoo</a>/<a href="http://code.google.com/appengine/">Google</a>/etc. provides a lot of bang for the buck), involve two outsourcings partners, or keep even rent out the surplus physical infrastructure to another company.Tip II.3: Licenses! Many license models use a per CPU model. But how does that compare to virtual CPUs? Licenses could be bould to a hardware serial number, but those are configurable for virtual machines; thus allowing a second, unpaid, deployment of the same application, without an alarm bell going off. Does the vendor even support deployment of its application on a virtual platform?The biggest risk here is the deployment of licenses not paid for, in the form of new virtual machines or by adding resources to existing virtual machines. When this happens and more important, if it is discovered, it could result in damages from fines and corporate image.On the other side, some vendors, you can gain an extra cost advantage. Take RedHat's support contracts for instance. If you deploy Linux on VMware, you pay VMware license and support contract and one support contract for each Linux deployment. If you host Linux on Linux (both RedHat's of course), you only have to pay for one support contract. This could lead to a cost advantage of several thousands of euros per machine per year.Tip II.4: CMDB The key to keep II.3 is the Configuration Management Database (CMDB). This database should be adapted to virtualisation. A properly configured database will keep track on which system is running where with which service levels and contracts. The CMDB will report will report any problems with regard to licensing/contracts upon entering or changing information.Another advantage becomes clear when disaster recovery is needed. If a system breaks, where should be virtual systems be reallocated to? It doesn't do to reallocate a virtual system to a physical system with incompatible service levels. A close relation should exists between disaster recovery tools/procedures and the CMDBTip II.5: Tooling is essential, not the Hypervisor Systems/Service management should be leading when choosing a virtualisation product. True, one Hypervisor gives better performance than the other, but the bult of the cost for any service is not hardware, but rather the management of that hardware. Windows oriented management departments tend to be more comfortable with VMware or XenSources own Xen distribution and Linux/Unix oriented departments like Xen (any) or even things like AIX partitions more. This has a huge influence on the learning curve of the department and the actual functionality they can realize with the product chosen.An important part of this is the provisioning of new virtual machines. Provisioning of new virtual machines within an hour should be possible. If this was already the case for physical machines, the same tooling could no doubt do the same for virtual ones. If not, new tooling should be introduced (the simplest approach is template based a.k.a. creating one machine and copy it around). Mon, 05 May 2008 08:44:00 +0200 http://www.gloudemans.info/virtualisation-tips-ii-service-management/ Virtualisation tips I: The Human Factor http://www.gloudemans.info/virtualisation-tips-i-the-human-factor/ The human factor is often overlooked when implementing new technologies in the data center. Most organizations do think about training their personnel and most actually do spend budget on that, but that is it.If companies truly want to reap the full benefits of virtualisation some other things concerning the human factor are important as well. The main reason why is because virtualisation introduces a fundamental change in the way we look at infrastructures.Tip I.1: Boundaries between existing specialized organizational units/departments fade. This tip involves large IT departments only. Thos usually have specialized Windows, Unix and network teams.In a virtual infrastructure, it shouldn't matter where the virtual system is actually running. This means that the persons managing the hypervisor must work together with the persons managing the virtual machines. This means that whether you are a Windows, Unix or even BSD guy, you'll have to communicate with your peers of all specializations. Not just to change a setting, but to actually dive deep into the lower bowels of the operating system to solve e.g. performance problems.How about the network? In an enterprise infrastructure, most systems are assigned to one VLAN or the other. If a virtual machine should be ably yo run at any place in the infrastructure, this means that the actual physical server needs to be a member of multiple, if not all, VLANs. All of a sudden not the network engineers decide which packet needs to go where, but the persons managing the hypervisor. This can only happen when departments trust each other. Some persons will welcome these changes, but there is also a breed of technologist that absolutely abhors to have anything to do with the neighboring technologies (as proof I point to the various flame wars on usenet and public fora). One way to solve this problem is to create a virtual team/organization consisting of persons from all current units/departments and make them responsible for the infrastructure core; the hypervisors and VLAN definitions. Because this team is multi-disciplinary other units/departments will have less trouble trusting and are more willing to work with this new virtual group.Tip I.2: Human error People make mistakes, it is a fact of life. Most mistakes do not have a very big impact but some do lead to service interruptions. Virtualisation is also about consolidation. More services/applications will be running on one piece of hardware amd one operating system (the hypervisor). This means that the probability that a human mistake has far reaching consequences increases. Something has top be done to balance this increased risk. There are a few options, among which: <ul><li> Follow tip I.1 and put only the persons with the best track records in the virtual team </li><li> Change operating procedure. In many cases, a lot can be gained by looking at the operating procedures carefully. Classify procedures into high, medium and low risk. Then think about ways to lower the risk of at least the high risk procedures. </li><li> Consider carefully which application runs where, or which virtual machine runs where. If a service is made up of 3 applications, running in 3 different virtual systems, it is probably best they all run together on the same piece of hardware. This limits the number of dependencies (only one hypervisor involved) and thus reduces the chance of a service interrupting mistake. Of course there are other things to consider as well. From the security point of view you might not want to run all three applications on the same system. A balance will have to be foun. </li><li> Cluster. Build clusters of physical machines automatic fail-over capability. This will not safeguard against all problems and mistakes, but will reduce the impact of them. An added benefit is now that all applications are high available. </li></ul>Next week Part II of the tips, appearing somewhere next week, will be about service management. Mon, 14 Apr 2008 08:44:00 +0200 http://www.gloudemans.info/virtualisation-tips-i-the-human-factor/ Switch Mishmash http://www.gloudemans.info/switch-mishmash/ Last week, I stumbled upon an interesting problem. One of the projects I'm working on as an astronomical sensor grid, putting out about 2.6Gb/s of data per sensor station, coupled to a central super computing infrastructure. This system is still in the buildup/tender phase, so the network consists of networking equipment of various brands, to test out what the differences are and how well they work together. There are only 5 sensor stations at the moment, so the total amount of data is manageable. In the future the total data stream will grow to about 1Tb/s, to be processed real time.In the past there were 2 network infrastructures, one a WAN (for connecting the sensors), one central and there was a firm boundary between them, in the form of a computing cluster that did data re-ordering. This cluster disappeared (because this task could also be done as a subtask of another cluster). The network came crashing down, due to some limitations posed by the computing infrastructure.Nothing that couldn't be fixed, but this brought all the networking brands in the infrastructure together. The network code is now made up of Layer 2 and 3 switches from Nortel, Foundry and HP. This gave me a good feeling of the differences between the brands, on top of the experience I already have with testing several brands in a lab setup.Note that I'm not a certified network anything, so if I'm wrong in someplace, or if there are features I missed please comment on this blog.The challenge in this infrastructure that there are a relative small number of big monolithic data streams (600Mb/s) and total silent listeners, causing switches not to learn where the listeners are.To make everything work the network relies heavily on VLAN and Multi-link trunking. Features every switch brand offers from their base model and up.Performance differences One of the first problems encountered is the bandwidth each switch offers. It is not trivial that on a 48 port GbE switch all ports can work at "full speed non-blocking", even for switches that do promise this performance. Typically in a switch, each cluster of ports is handled by one chip. And there is one chip handling the previous chip (a tree structure). There can be a significant limit om the amount of data that goes from chip to chip. So depending on which ports you use for what purpose you either get or don't get the performance you want.Luckily most modern switches from the major switch brands don't have this limitation any more, but if you're into high performance networking, it is something to check.Jumbo frames In this infrastructure 9K jumbo Ethernet frames are used. Some brands put the jumbo frames at less than 9K. This not only happens at switch level but e.g. the max. frame size the network card in my computer can handle is 8K.VLAN The VLAN standard is well described. The implementation is not. Most implementation differences revolve around filtering and VLAN number 1 (the default VLAN). In all switch brands I've seen VLAN 1 is the default VLAN, the the port membership of VLAN 1 differs per brand. What I've seen: <ul><li> The port is a member of VLAN 1 until the port is added as an untagged port to another VLAN </li><li> The port is a member of VLAN 1 until it is added to another VLAN (tagged or untagged) </li><li> A port must specifically be removed from VLAN 1 </li><li> Unregistered (ID not registered at that switch) VLAN tags are or are not filtered </li><li> Untagged packets are or are not filtered on a port that is supposed to handle only tagged packets (such as a trunk) </li></ul>These implementation issues can drive you to madness creating loops where you don't expect them to be. Also it is important to know these implementation differences if only from the security point of view.Some switches are more flexible when it comes to VLAN's then others. There are switches that allow you to set VLAN X for receiving data and VLAN Y for sending. While this kind of functionality is needed in some places, it makes you very vulnerable to mis-configurations.Multi-link trunking Most switches allow for two different types of multi-link trunking (MLT). A brand specific variant and LACP.In the brand specific variants a set number of lines is allocated to the trunk and load is distributed over the lines. This is not done based on load, but based on an algorithm which is dependent on factors as source and destination MAC, IP address or IP port number. In a normal network setting where there are many little data streams this works well. But if you have anly few streams (number of streams in the same order of magnitude are the number of lines in the trunk), this is not the case. Often too many streams are allocated to the same line from the trunk, while other lines are not used at all.To effectively deploy MLT in such a case, you will need to know the algorithm and adapt your MAC addresses to it. The snag here is that the algorithm is almost alway classified information. The other snag here is that this kind of MLT only works between switches from the same brand.LACP trunking is working a lot better in the environment I described. The number of lines participating the the trunk is dynamic. Lines are added to the trunk when the load on the current lines reaches the limit. This behaviour could lead to other performance issues (especially around the moment where extra lines are allocated to the trunk), but with long lasting continuous large data streams it is a perfect solution.LACP trunks also work between switch brands, but a bit of caution is needed. At the switch level, each trunk has its own ID (key). For LACP trunks between switches, sometimes the ID doesn't need to match and sometimes it does. This could be a problem, especially if you use manual IDs for the trunk (each switch tries to auto allocate an ID, but also has a special range for manual IDs). e.g. the range for the manual configuration of IDs between HP and Foundry have no overlap. The trunk does not form. If the determination of the ID is let to the switches themselves, it works perfectly.Controlling the Flow It can be extremely hard to make your data go where it needs to go, or should I say not make it go where it doesn't need to be. If you got equipment on the network that is completely silent, a switch will never learn where the device is. Resulting in a broadcast of the data that needs to go to that device on all switch ports. That really is a problem if you have several 400Mb/s data streams. This brought down the network several times in the case I'm describing.A possible solution is to enter addresses in the switch MAC table by hand; but this is not a standard feature of a switch. My guess is that only half the number of switches out there support this. Surprisingly some really cheap switches support this while some A-brands don't.Another (partial) solution is defining uplink ports on the switch. If an address is not present in the MAC table the data only goes to the uplink ports. This reduces the problem untill the last switch the data goes through. This feature combined with a static MAC table is what you want to have if you have silent devices. It keeps the administration task manageable (only need to administer the MAC address once), while letting you have full control.User interfaces Most switches emulate Cisco in their configuration. This is a good thing, since it works very well, once you know it. The learning curve can be very steep for this kind of interface.Some switches have menu based configuration and most also allow web-based configuration. While these are easy to use, I have never seen and menu based interface that allows you to configure all aspects of the switch. Menu driven interfaces often try a more intelligent approach to configuring the switch. This works well when you're a rookie, but the result is, that if you configure one aspect of the switch, some other settings will be auto-changed as well.One brand I know even has menu-ized the command prompt, while it works better than most web interfaces, I still prefer the good old command prompt and a "sh run" that only shows you the things did configure yourself. Locked up Almost each brand has some cool very handy features that no other brand has. The temptation to use these features can be quite big at times. There is no harm in using these features, but when you do, realize that a vendor lock-in might be created at that moment. The real hazard lies in the features of which you don't know that they are vendor specific (happened to me a couple of times). A lock-in could be created without knowing it.Wrap-up The bottom line to this story is that a switch is not a switch. Each brand has its strong and weak areas. These won't be encountered often in office based setups, but with the growing demand for performance and security these areas will be encountered more often.This leaves a challenge for the network experts, determining what brand (or mix of brands) is most optimal for which case can be, I'm happy to say, not easy. Fri, 29 Feb 2008 08:44:00 +0100 http://www.gloudemans.info/switch-mishmash/ Using OpenDNS with your own DNS http://www.gloudemans.info/using-opendns-with-your-own-dns/ Yesterday I read an article about OpenDNS <a href="http://www.opendns.com/">http://www.opendns.com</a>. OpenDNS adds the possibility of domain name based filtering. You can configure your own white and blacklist, but they also have a list of categories you can choose from. Most interesting category I found was phising, but I image that if you got small children at home you'd also want to block porn and such. Domain name spelling correction is an added bonus, plus they promise almost 100% service availability, superior performance, domain name spell checking and statistics gathering (can be turned off).The service is free, but you have to register and put in your source IP address, because the service is configured via the OpenDNS website. Teh only thing you need to change on your PC is the DNS address to use. I suppose most people have this on auto-configure (via DHCP), so this add a (small) piece of configuration. A downside is that this free service must get its funding from someplace, so ads are shown when pages are blocked or not found, but I can live with this.Myself, I run a couple of services on a home server, among which this webserver and my mailserver. Because of this I have an internal DNS server which forwards all non gloudemans.info queries to the DNS of my ISP. All internal clients also use my local DNS, or they wouldn't be able to find my own webserver. Now I could just change my forwarders to point at the OpenDNS DNS servers, but this could have a couple of issues: <ul><li> Some mail is relayed via a server from my ISP. The DNS name of this server is only in the ISP's DNS </li><li> It adds another party that can see who I'm sending mail to. I'm willing to trust my ISP, but to trust some free service off the Internet is another matter. </li><li> The network path to OpenDNS DNS servers is longer, though the OpenDNS servers might provide a better uptime, I doubt the actual uptime measured as I see it (including the network between me and OpenDNS) is better. I can live with this for browsing, but I'd like better availability for the services I have running. </li><li> e.g. the mailserver doesn't need the functionality OpenDNS offers. Bugs in thew OpenDNS software, like spell checking domain names might even have an adverse effect. This is a nonexistent issue at a moment, but it was in the past and illustrates my point. </li></ul>Now if the performance of OpenDNS is really better, I could think about using it for my mailserver as well (but then again, what's the importance if a mail goes out within 2 or one second). So I decided to do a quick test with <a href="assets/blog/dnstest.sh">this</a> script. It looks up a couple of domains and displays the time it took in seconds. I ran the script twice for my ISP's DNS and for OpenDNS and got <a href="assets/blog/DNS-comparison.pdf">these</a> results. As you can see, the performance of both DNS servers is very similar. The ISP DNS servers have a few spikes in them, but on the whole there isn't much difference. So there is no business case to use it on mail, but I still would like it for my browser.Luckily the DNS server software used for most Linuxes (<a href="http://www.isc.org/index.pl?/sw/bind/">Bind</a>) has a featured called views. WIth views you can control the actions of the name server based on the client. First, there needs to be a clear distinction between server and client network addresses. You can enter each client address into the DNS configuration by hand, but I subnetted my local network from a class C (netmask 24) to a netmask 26, giving me 4 distict IP ranges. I still use the 24 netmask on all systems, but now I have the ability to make a distinction. 1-63 is all server related, 64 to 127 is wired client, 128-191 is wireless client and 192 to 255 is DHCP range.My named.conf looked like (not a complete working config) <div class="codesnippet">options { // Put files that named is allowed to write in the data/ directory: directory "/var/named"; // the default dump-file "data/cache_dump.db"; statistics-file "data/named_stats.txt"; memstatistics-file "data/named_mem_stats.txt"; auth-nxdomain yes; listen-on { 127.0.0.1; 10.0.0.2; }; forwarders { //ISP DNS one 192.192.192.192; //ISP DNS two 192.192.129.193; }; allow-transfer { none; }; }; zone "0.0.10.in-addr.arpa." IN { type master; file "10.0.0.db"; }; zone "gloudemans.info." IN { type master; file "gloudemans.info.db"; };</div>Using my newly defined subnets and using views, I changed this into (non working example): <div class="codesnippet">acl serverips { //Dont forget the localhost 127.0.0.1; 10.0.0.0/26; }; acl clientips { // Wired stations 10.0.0.64/26; // Wireless stations 10.0.0.128/26; // DHCP range 10.0.0.192/26; }; options { // Put files that named is allowed to write in the data/ directory: directory "/var/named"; // the default dump-file "data/cache_dump.db"; statistics-file "data/named_stats.txt"; memstatistics-file "data/named_mem_stats.txt"; auth-nxdomain yes; listen-on { 127.0.0.1; 10.0.0.2; }; allow-transfer { none; }; }; view servers { match-clients { serverips; }; forwarders { //ISP DNS one 192.192.192.192; //ISP DNS two 192.192.129.193; }; zone "0.0.10.in-addr.arpa." IN { type master; file "10.0.0.db"; }; zone "gloudemans.info." IN { type master; file "gloudemans.info.db"; }; }; view clients { match-clients { clientips; }; forwarders { //OpenDNS one 208.67.222.222; //OpewnDNS two 208.67.220.220; }; zone "0.0.10.in-addr.arpa." IN { type master; file "10.0.0.db"; }; zone "gloudemans.info." IN { type master; file "gloudemans.info.db"; }; };</div>Now all server related stuff, like mail and local webserver statistics gathering etc. is using the ISP DNS servers, while all local web browsers resolve non local domains via OpenDNS and this profit from all its features. Note that this setup is not secure. If someone doesn't want to use the OpenDNS on purpose. A change of the client IP number is enough. This setup will work to keep your little children away from sites they should not see, it won't work for the teenager (unless you only use 127.0.0.1 for the server IP), but at that age a proper conversation is probably better idea than just blocking content anyway. Thu, 21 Feb 2008 08:44:00 +0100 http://www.gloudemans.info/using-opendns-with-your-own-dns/ The Triangle of Amplification http://www.gloudemans.info/the-triangle-of-amplification/ SOA, SOX Compliance, Virtualisation, Security, Identity Management, Outsourcing; all trends of the current time. Organizations are struggling to adapt and incorporate. Consultancy bureaus thrive on this (I belong in this category). Now is the time to step back and look on how all of these trends interact and what the demands are on he organization from a holistic point of view.<div style="text-align:center"><img src="assets/blog/triangleofamplification.png" /></div>There are three key concepts in technical infrastructure trends: <ul><li> Fragmentation <ul><li> Services are built from increasingly small parts. This fits the Service Oriented Architecture (SOA) model, but at the same time complicates infrastructure management. Especially if those parts are the responsibility of different entities </li><li> Grid; having a large pool of resources that is dynamically allocated to the most needy task sounds and is great. But how to ever guarantee service levels on such an infrastructure; what is the pool of resources is spread over multiple separate entities? </li><li> ...</li></ul> </li><li> Security <ul><li> Identity management simplifies the maintenance of authorization and authentication records and makes life easier for all users involved, but also makes any security incident more widespread </li><li> More and more laws are posing boundary conditions on services and infrastructures </li><li> ...</li></ul> </li><li> Virtualisation <ul><li> Potentially brings down hardware and license cost </li><li> But eradicates the traditional boundaries in the infrastructure. Separation of responsibilities as implemented in most organizations is gone (see also <a href="http://www.gloudemans.info/virtualisation-overview">http://www.gloudemans.info/virtualisation-overview</a>) </li><li> ... </li></ul></li></ul>Other trends are more or less related to one or more of these concepts.These three concepts enhance/amplify each other in every way. They enable each other, but also amplify each others problems. Examples: <ul><li> Because of the fragmentation of services, each component needs authentication and security services </li><li> Virtualisation allows for many smaller service components to run on one piece of hardware, thus enabling fragmentation </li><li> Virtualisation poses a new security challenge in the area of infrastructure and service management </li><li> Chinese wall policies might encumber the full usage of virtualisation and grid technologies </li><li> A successful identity management implementation enables implementation of SOA </li></ul>Outsourcing commonly plays at least on one of these concepts. For example: Outsourcing means new trust relations (playing on security), fragmentation (more than one outsourcings partner is common) and virtualisation (giving the outsourcing partner(s) a piece of infrastructure to test on).Triggering one area will trigger the other two as well, directly of in the future.Ground rules for support organizations on how to deal with modern infrastructures: <ul><li> Awareness; be aware of the service chains and how they interact, be aware of security implications if a component changes. This is a business responsibility, but in todays highly complex infrastructures mistakes are easy to make and can have a devastating influence on the business </li><li> Multi-Domain knowledge. The era of the system and network manager are past. The new IT professional is skilled in several areas and is a team player by definition. No one can have complete overview, so IT professionals must increasingly rely on the knowledge of their peers </li><li> The support organization must have as little interfaces as possible. Depending on the user to identify the problem and calling the right service desk of the right service component is not possible. The user will be more heavily involved in identifying the real problem, but relaying him from service desk to service desk is counter productive. </li><li> No boundaries. Boundaries between entities in the same organization cannot exist if a complicated/fragmented service is to be managed efficiently. Different roles still must exists (something to do with security), but unhampered interaction between these roles is mandatory. </li><li> The governance model for the support organization must be excellent. Not only for the complex services under its own control, but for interfacing to the organizations which have other parts of the same service under control as well.</li></ul> Thu, 14 Feb 2008 08:44:00 +0100 http://www.gloudemans.info/the-triangle-of-amplification/ End to all Desktop problems? http://www.gloudemans.info/end-to-all-desktop-problems/ Some consultants would you believe that virtualisation technology will solve all your desktop problems.... Yeah, right!The assumption is based on the fact that many people believe that desktop stability and the data loss that goes with it, is the biggest problem. It is true that one misbehaving application can bring down your complete desktop computer. If every application is run in its own virtual machine (VM), which contains only the absolute essentials for the application to work. This would: <ul><li> stop applications from influencing each other. A problem in one application will not affect the other one </li><li> prevent the famous windows DLL dependency hell (or shared object hell if you're a *nix person) </li><li> keep the Operating System (OS) to a bare minimum. Since all logic to access the hardware is in the VM. The operating system is just a Hypervisor and maybe some user interface elements. </li><li> drastically reduce the chance of security exploits and bugs in the OS, since there is less code in the OS </li><li> a driver problem only kills the application using that driver </li></ul>Now, in my opinion, this makes no sense at all. While I do believe, it is a good idea to run e.g. a web browser in a VM, for most desktop applications is isn't. These are the reasons that spring up in my mind: <ul><li> No interaction between applications means no data exchange between them. The concept of "copy and paste" and "drag en drop" needs a complete overhaul </li><li> All the applications together will make up most of the complete OS from the "Old" environment, the only thing changed is the shift in responsibility </li><li> Application developers don't want to reinvent the wheel from scratch for each application. There will be standard libraries is each application. What happens if a security flaw is discovered in that? Not one hole to patch, but many! </li><li> Each application still needs access to the data, so an exploit in your application still puts your valuable data at risk </li></ul>Virtualisation can help creating a more secure desktop, but it'll have to be used on todays desktop technology. The best way to a more secure and stable desktop environment must come from the OS itself. Proper role and memory separation and a good Quality Assurance program for driver development. While companies like Microsoft have an excellent QA program, it is not mandatory. The Open Source model actually helps driver development. The developer can see all the kernel code, copy, reuse and discuss with the rest of the world.A stable and secure desktop is possible now, it is just a case of picking the right technology and actively work on putting legacy applications of of business, or in a VM. Linux anyone? (or BSD for that matter) Tue, 12 Feb 2008 08:44:00 +0100 http://www.gloudemans.info/end-to-all-desktop-problems/ Away with ITIL http://www.gloudemans.info/away-with-itil/ ITIL has become the de facto standard for system management in most large organizations and many small ones. ITIL has revolutionized system management . One of the strongest points of ITIL is the dictionary that comes with it. Now we can all understand each-other when talking about system and service management.But has the quality of IT services really improved because of ITIL? I think not as much as it could have. In ITIL implementations, organization structures became a mirror of the ITIL processes. Organizations which previously had one monolithic systems management department, now have several departments or groups. One responsible for first line support, one for second line, one for service management etc. ITIL has inadvertently created new boundaries within the organizations implementing it. This is not a problem by definition, but all to often boundaries lead to "kingdoms".Knowledge and experience becomes locked up between these boundaries and employees become fed up with their jobs (if you are a real technician, delivering first line support can be a true horror).The solution? Quite simple; use ITIL as a representation to the outside world, but remove the boundaries internally. Create two groups, one the business side of systems/service management and one for the technical side and fine a balance of experienced staff and rookies within these groups. For large organizations create teams for each application/service domain within the groups. Make each team responsible for meeting the service levels for the services running in their domain. If possible, do not appoint a team lead, but let the teams find out for themselves and move members between teams on a regular (yearly?) basis, to promote standardization.Each team must contain all the skills needed to manage the application/service domain, not only first to third level, but also from hardware to application, from tactical to strategical etc. In these multi-skilled teams of diverse experience, the learning curve for the inexperienced will be very steep, while the experienced may find a new challenge in coaching the inexperienced and enjoy the increased freedom. Competition between the teams (based on service levels met) must be encouraged. The only boundary condition; the outside interface is ITIL and it must be the same for all teams. (Knowledge about ITIL within the teams is essential)I think, this will really improve the quality of services delivered. For instance the number of escalations, technical or management will go down drastically. If the teams do a proper job, the person to escalate to is already involved! At the same time, customers and team members will be more content and thus less likely to switch vendor or employer. Mon, 04 Feb 2008 08:44:00 +0100 http://www.gloudemans.info/away-with-itil/ Certification; killing innovation? http://www.gloudemans.info/certification-killing-innovation/ Many employees require their personnel to be certified in certain areas. This could be area specific certificates, like security, or vendor specific certificates, like MCE or CCNA. The reason behind it seems very obvious; the certificate is proof that one has mastered the necessary knowledge. It reduces the risk for the employer or customer of having an inadequate staff.But what is the value of certification really? In real life, I have seen certified technicians that I wouldn't even let near my home network, let alone the a business critical infrastructure. Actually, this is a well known phenomenon and is due to various reasons. The most obvious are: <ul><li> Most important one is the quality of the course leading to certification. I think the earlier Microsoft certifications are a great example of this. People learned how things were done in the GUI, but they forgot the part of telling people how the underlying standards worked and why they were what they were. </li><li> The fact that many certificates don't need to be renewed. Especially if the certificate is gained by performing well on an exam, chances are that most of the knowledge needed is in short term memory. </li><li> Even if certificates do need to be renewed, employers and customers hardly ever check if they are still valid. I'm sure that there are a lot of people out there who write that they are certified on their resumes, while they actually are not (any more) </li><li> There's no substitute for experience. But experience noted on a resume might be misleading. Every project has its leaders and followers; their resumes often looks the same. </li></ul>Most of the certificates that are held in high regard, have a way to cope with these previous points and they deliver highly trained professionals, who really do know what they are doing. I think the real problem with certification is that people learn to think along certain lines. This is something that struck me, while I was at a security conference. Everyone I met, approached a given problem from the same angle and came to the same, predictable, solution. Talking about security, this is a bit scaring, because if reactions are predictable, this gives the hacker (who if most often not certified and/or trained and thus highly unpredictable) an opportunity to go around them.To find the optimal solution for a given problem, the problem must be approached from multiple angles. In my experience certification (and cources) interfere with this. Certified processionals, will deliver an excellent solution for a given problem, but it may not be the best one possible.My remark is not only valid for the field of security, but business as well. If you want to know what your competitor is going to do next, dig up his resume and look for his courses and certificates.Innovation happens when problems are approached from unusual directions, also called "out of the box thinking". Courses and certifications put you firmly in the box, making it harder to escape. So next time you, or your boss, wants you to get a certificate, think hard on the reason why. Buying some books and starting a study group with some colleagues might have the same effect, without putting yourself in the box. As for how to get the next job, with a seemingly poor resume? Just put your thoughts on certification in the letter, if you do that right, you will spark some interest, get invited for an interview and dazzle them with your innovative insights. Sat, 02 Feb 2008 08:44:00 +0100 http://www.gloudemans.info/certification-killing-innovation/ High Speed Firewalling in Linux http://www.gloudemans.info/high-speed-firewalling-in-linux/ In this age of virtualization, cluster and grid applications, the load on the server network interface tends to increase. Multi-homed servers were rare not more than 5 years ago, nowadays servers come with 4 NICs pre-installed. This, of-course, increases complexity and network cost. Today I made a proposal for a couple of servers with more than 10 Gigabit interfaces. With the current state of technology, we are approaching the break even point for 10 Gigabit network interfaces. This year will see the breakthrough of 10Gigabit over copper in the datacentre.The reduction of the number of interfaces will at least simplify the physical infrastructure, but I think management cost will decrease as well.But what performance can we expect from 10GB/s interfaces. Not too long ago, servers had trouble even saturating a single 1Gigabit interface. I did some tests myself, using a Sun X4100 server running Centos 5 (Linux) and some quick kernel tuning. I was not disappointed with the results. I managed to get a throughput of 8.5 Gigabit per second, using TCP/IP over Ethernet (using Jumbo frames). These kinds of throughput are only possible with a NIC that supports TCP/IP offloading, like a Myricom card. I did this test also with an older 10Gigibit Intel NIC and I got stuck at 3.5 Gigabit per second. In all cases one CPU core was fully used to process and transmit/receive.For normal business applications, or high speed Internet connections the Firewalling is a must. So what is the performance penalty? It is surprisingly small. Network latency anly increases by 2 microseconds (1-2% increase) and bandwidth suffers 10-25Mbit per second; that's less than 0.5% I am still analyzing the test results for one of the customers I work for, but I'd say that the (Linux) server is ready for the next step in networking. The image below shows a graph of network frame spacing against bandwidth. Each line represents an IPtables ruleset compared against the baseline performance (without rulesets) <img src="assets/blog/10Gthroughput.jpg" /> Thu, 31 Jan 2008 08:44:00 +0100 http://www.gloudemans.info/high-speed-firewalling-in-linux/