Archive for the ‘ Everything Else ’ Category

Rethinking my blog

I’m re-thinking my blog.  Please excuse the mess.


A note on patching XenServer hosts in a resource pool

Don’t do it during your initial build-out. 

Actually, its not so terrible, but its time consuming.  I have three hosts I’m patching that are all in a resource pool.  To ensure the safety of the vm’s in the pool Xen goes through and patches a host, reboots, and then moves onto the next one.  Had the machines been removed from the pool, they could have been patched simultaneously.  Its actually a cool safety feature for production, but seeing as I don’t have any VM’s in the pool yet I wish I had realized what was going on sooner and split the pool out first, or patched them before joining the pool. 

Ok, lesson learned. Fully patch the hosts, then join them to the pool.



I was talking to a guy who specialized in Citrix XenDesktop and he told me that the real trick to the patching is not to allow XenCenter to auto-reboot the Hosts after applying patches.  I assumed XenCenter knew what it was doing, but apparently it’s just inefficient.  Instead, when applying each patch, specify that you want to manually reboot after the patch is installed.  After it is applied you then skip the reboot and apply the next patch, and then the next, until all are applied and you wrap it up with a single reboot. 

I’m paranoid though, so I recommend leaving it out of the pool either way.

Change the DNS servers by command line on XenServer

To change the DNS servers after the original setup you have to modify the server from the command line.  To do so you first need to know the uuid of the management interface.  In my case I know the management interface is eth0.  I can get the uuid of eth0 by issuing the following command:

xe pif-list device=eth0

Which will return something like the following:

    uuid ( RO)                  : c8d514cc-dfee-1877-c288-2a5d84cc3275
device ( RO): eth0
currently-attached ( RO): true
VLAN ( RO): -1
network-uuid ( RO): c5837a4a-ceed-8521-8ce3-4a4e81cd66fe

Then I issue the following command to update the resolv.conf file with the DNS server list.

xe pif-reconfigure-ip uuid=<uuid_from_above> mode=static IP=<eth0_ip_address> netmask=<eth0_subnetmask> gateway=<eth0_dfg> DNS=dnsserver1,dnsserver2

No reboot required to make this change. Just ping a hostname using a short netbios name to test.

Change the name of a Citrix XenServer and add a DNS suffix list

Today I decided that I wanted to change the Hostnames of my XenServers.  After a little more thought it occurred to me that I should also add a DNS suffix lookup order to them to address some name resolution issues.  Here I show how to complete each task. 

First objective: change the hostname from the command line

The command I’m going to use to do this requires the UUID, so the first thing I need to do is figure out what it is.  I can do this using the following command:


the results of which will look something like this:

     uuid ( RO)                : 6123425d-46gg-553c-bsbg-ada4d03f1edc 
          name-label ( RW): SERVER1 
name-description ( RW): Default install of XenServer

The uuid is in bold text above. Now I can change the name of the server like this:

xe host-set-hostname-live host-name=CIPT0655 host-uuid=6123425d-46gg-553c-bsbg-ada4d03f1edc host-name=SERVER2

And now to check and see if the hostname is applied:


And I see that the name has now been changed. Objective one completed, but note that this is NOT the same as changing the display name in XenCenter.  You can do that through the XenCenter gui, or you can issue the following command:

xe host-param-set name-label=<new_hostname> uuid=<uuid_as_noted_above>

Second objective: add the DNS suffix list

The steps for this are defined in this Citrix article

First we will need to get the uuid of the management adapter.  I happen to know that this is eth0.  To get the uuid I run the following:

xe pif-list device=eth0

Which will return something like the following:

    uuid ( RO)                  : c8d514cc-dfee-1877-c288-2a5d84cc3275
                   device ( RO): eth0
currently-attached ( RO): true
                     VLAN ( RO): -1
        network-uuid ( RO): c5837a4a-ceed-8521-8ce3-4a4e81cd66fe

So now I ffed the uuid into the following command and then reboot the server:

xe pif-param-set uuid=<pif uuid> other-config:domain=sweeneyops.lab

Note that I could add multiple domains to the list by way of a comma separated list, which would change the other-config parameter to look something like this:

… other-config:domain=sweeneyops.lab,dom1.sweeneyops.lab

After my reboot, I simply run the following to check the results:

cat /etc/resolv.conf

Manually setting the time in Citrix XenServer

We hit another issue today with two of our blades in one of our XenServer pools.  Each blade thought it had been up for well over a year, which was interesting since we had only installed them last week.  It was even more unusual because we had configured it to use our NTP server to pull its time.  Using the XenServer Console I attempted to update the time, but to no avail, so I moved to the command line and used the Linux DATE command to manually set the time. 

date –s “15 NOV 2011 15:00:00”

Going back to the XenServer Clinet I saw that the uptime had not been corrected, so I then instructed the server to reboot. 

shutdown –r now

Missing management NIC in Citrix XenServer

After a lengthy proof of concept we finally started our production deployment of Citrix XenDektop.  I’m a VMWare fanboy, but I really like anything that has to do with virtualization, so I was excited to get started on it.  To top it off, we’re deploying it in a Cisco UCS environment, which is pretty cool stuff. 

I started by deploying the Citrix XenServers themselves, along with the web and licensing servers, and built out my clusters.  In Citrix they are referred to as resource pools, but they’re really clusters.  Shortly after completing that, however, we realized that we had to upgrade a lot of the firmware on the UCS chassis and blades. 

I assumed there would be some risk, but the results surprised me, for after completing the firmware updates I found that the Citrix resource pools/clusters were inaccessible.  Going in through the UCS KVM manager I found that my management NIC’s had disappeared!  After a little research in the forums, and some trial and error, I found that I could recover the pool using the following method.  I thought I would record the process for posterity’s sake.

First, I needed to recover the master server in each pool.  To do this I logged in through the command line on the first pool’s master and tried to perform an emergency transition to master.  I found that I couldn’t, however, because HA was enabled. To disable HA on the Host I executed the following command:

xe host-emergency-ha-disable –force

Note the “-force” switch.  Also note that the command itself warns that this is a dangerous thing to do.  It doesn’t actually say why it’s dangerous though, and it did allow me to move on so I’m taking the warning with a grain of salt.  I tried the emergency transition to master again:

xe pool-emergency-transition-to-master

This time it worked.  I was able to see the management NIC again and view the host in XenCenter console.  On to the second host (there were three in all).  This time I started by disabling HA.

xe host-emergency-ha-disable –force

That worked as expected, so I moved on to the master transition:

xe pool-emergency-transition-to-master

That worked as well, but when checking the management NIC I found that it now had an old IP I had accidentally used earlier.  Weird. I wonder where this got cached?  Oh well. Move on.  I tried to change IP, but found that I couldn’t because external authentication was enabled.  This was because I had enabled AD authentication earlier.  Another thing to disable.  Fortunately, there is a command for that as well:

xe pool-disable-external-auth

This took a while to run and reported errors because it couldn’t disable external authentication on other systems it knew to be in the same cluster, but it actually did disable it on the host in question.  Now I could change the IP on the second host.  I still couldn’t join pool so I had to perform the emergency master transition again:

xe pool-emergency-transition-to-master

Now the second host thinks it’s the master so it won’t join the cluster… err… pool.  To fix this, I have to tell it to point to another master instead.  I find a command to do this, but it doesn’t work because couldn’t resolve hostname.  After some testing it seems to be network related, so I tried to restart management agent (xapi):


I gave it 30 minutes to do its thing, then tried to reset the master again.

xe pool-emergency-reset-master master-address x.x.x.x

Woot!  That worked!  The second host appeared in XenCenter as a part of the cluster… umm… pool.  I repeated the same thing on the third host in the pool, and then the entire process on the second pool and I was back up and running.

Filesystem Hierarchy Standard (for *NIX OS)

So I finally found a good file system reference for the Linux OS that helps me understand the function of the standard directories.  The entire reference can be found here:

Be prepared. It’s a bit of a dry read. The key table in this for me was the following:


Directory Description
bin Essential command binaries
boot Static files of the boot loader
dev Device files
etc Host-specific system configuration
lib Essential shared libraries and kernel modules
media Mount point for removable media.
mnt Mount point for mounting a filesystem temporarily
opt Add-on applications and software packages
sbin Essential system binaries
srv Data for services provided by this system
tmp Temporary files
usr Secondary hierarchy (user home folders)
var Variable data

Each of these directories are discussed in greater detail in the source document.  In future entries I’ll discuss them one by one as I learn more about each.