Vmware on Evan Hoffman

"Call HostServiceSystem.Start for object XX on vCenter Server YY failed" when starting snmpd on ESXi 5.1

Mon, 05 Aug 2013 16:59:35 +0000

SSH into the host and run these:

~ # esxcli system snmp set --communities 
~ # esxcli system snmp set --enable true

After that, make sure you have the port (161/udp) open in the firewall config and it should start up ok. (The root problem is the file /etc/vmware/snmp.xml - those commands fix it).

Use PowerShell to disconnect the CDROMs from all VMs in vCenter

Sat, 20 Apr 2013 14:59:36 +0000

Recently I was moving all VMs from one NFS datastore to another so we could destroy the old volume. Storage vMotion took care of this for the most part, but even after moving the files, vCenter still showed that the VMs were using the old datastore. It turned out this was due to the VMs having mounted ISOs on that datastore. The solution was to eject/unmount the CDrom, but I didn’t want to do “Edit Settings…” and manually remove the CDrom for 200+ VMs.

VMWare Tools Compatibility Matrix

Wed, 20 Feb 2013 10:15:22 +0000

Question: If I install the latest version of VMWare Tools on a guest, what kind of compatibility do I get when I vMotion to an older version of ESX? In other words, is VMware Tools backwards compatible with older versions of ESX/ESXi?

Answer: Yes, it’s backwards compatible. From this tool:

Rescan SATA bus (aka hot-adding a SATA disk on a Linux guest in VMware without rebooting)

Mon, 24 Oct 2011 15:20:51 +0000

Linux supports hot-adding disks but whenever I add a new vdisk in VMware the new disk doesn’t show up unless I reboot, which defeats the purpose of hot-add. This command forces a rescan of the bus:

echo "- - -" > /sys/class/scsi_host/host0/scan

dmesg shows the new disk has been found:

Vendor: VMware Model: Virtual disk Rev: 1.0 
 Type: Direct-Access ANSI SCSI revision: 02
 target0:0:2: Beginning Domain Validation
 target0:0:2: Domain Validation skipping write tests
 target0:0:2: Ending Domain Validation
 target0:0:2: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)
SCSI device sdd: 1048576000 512-byte hdwr sectors (536871 MB)
sdd: Write Protect is off
sdd: Mode Sense: 03 00 00 00
sdd: cache data unavailable
sdd: assuming drive cache: write through
SCSI device sdd: 1048576000 512-byte hdwr sectors (536871 MB)
sdd: Write Protect is off
sdd: Mode Sense: 03 00 00 00
sdd: cache data unavailable
sdd: assuming drive cache: write through
 sdd: unknown partition table
sd 0:0:2:0: Attached scsi disk sdd
sd 0:0:2:0: Attached scsi generic sg3 type 0

Now, why there’s no “rescan_sata” command is something I can’t fathom, but that’s Linux for you.

VMWare 5's new licensing model.

Sun, 17 Jul 2011 10:20:56 +0000

After reading up on the new VMware licensing&pricing model I understand the uproar. Limiting vRAM is a reasonable constraint, but 32GB per socket for Enterprise? 48 GB for “Enterprise Plus”? If you have a dual CPU server with 144 GB (easily configurable last year), with 4.1 you’d only need 2 enterprise licenses to use all 144 GB, since in 4.1 an “Enterprise” license covered 1 CPU (up to 6 cores) and up to 256 GB memory on the host.

VMware Workstation 7 - virtual ethernet fails to start after changing vmnet8 subnet

Wed, 23 Feb 2011 10:29:22 +0000

After my recent wipe of my laptop, I reinstalled VMware Workstation and my Win XP VM was working fine. The one wrinkle I faced was that the subnet for the vmnet8 (NAT) vnic had changed from 192.168.250.0/24 to 173.16.132.0/24. The host machine had been 192.168.250.1 so rather than reconfiguring everything on the guest to point to a new IP for the host I figured it would be easier to change the subnet for vmnet8. I went into the Virtual Network Editor and just changed the subnet. Seemed to work correctly, but after doing a release/renew in Win XP I couldn’t get an IP.

Upgrading ESX 4.0 to ESX 4.1 with Update Manager

Thu, 21 Oct 2010 23:57:54 +0000

This turned out to be so ridiculously easy I probably shouldn’t even mention it, but only because I remembered that Update Manager exists. I thought I was going to have to burn an ISO, go to the datacenter, pop the disc in, etc. But with Update Manager this is all handled remotely. Just download the files from vmware.com and upload them to your vCenter server through the client. This YouTube video by VMware walks you through the procedure:

vCenter Data Migration tool doesn't restore SQL Express DB - Migrating vCenter 4.0 to 4.1 (part 2)

Tue, 19 Oct 2010 21:44:01 +0000

I’m into my second day on this project. It appears that the backup and restore procedures run to completion but on the target machine the database is empty. This is pretty easy to check; I just went in with SQL Management Studio, looked in VIM_VCDB -> Tables -> dbo.VPX_DATASTORE and did “Open Table.” On the source machine I saw all the datastores in there (as expected). On the target machine, however, the table was empty. I found KB 1024380 describing a failure of the migration tool with a symptom, “Data Migration Tool logs (restore.log) does not indicate any errors, but the database is blank.” The KB says:

vCenter Data Migration tool doesn't restore SQL Express DB - Migrating vCenter 4.0 to 4.1 (part 2)

Tue, 19 Oct 2010 21:44:01 +0000

Migrating vCenter 4.0 to 4.1 (part 1)

Mon, 18 Oct 2010 22:28:12 +0000

When vSphere 4.1 was released earlier this year, I was mildly excited. Our ESX servers are still running 4.0.0 and I had been planning to upgrade to 4.0u1 for a while. When 4.1 came out it seemed to have vindicated my procrastination: now I could just upgrade to 4.1 and take advantage of whatever new features there were (I read something about memory compression… whatever). I assumed the upgrade procedure for vCenter itself would be pretty simple: uninstall 4.0 and install 4.1. Well, it turns out that’s not the case! I sat down one day to create my game plan for the upgrade and right away I learned this was going to be a lot more work than I’d guessed. From VMware’s KB:

The Barracuda Spam Firewall VMware Appliance (Vx) finally exists!

Mon, 04 Oct 2010 18:24:12 +0000

When I started at my current company, spam was handled with a separate server running SpamAssassin and a few other services. This sort of got the job done but required babysitting. I wasn’t part of the Sysadmin team at that point but I know they had to restart SpamAssassin relatively frequently, manually clear out the email queue when people noticed they weren’t receiving email, etc.

After a few months, someone wised up and purchased a Barracuda Spam Firewall. I assumed this was basically a pretty GUI wrapper around the same tools we had already been using, but it was certainly worth the money. In addition to the basic filtering of spam it filtered for viruses, performed recipient verification against AD via LDAP, had nice graphing and reporting.

The VMware datastore LUN 2TB (well, 1.99999 TB) size limit

Fri, 24 Sep 2010 16:00:15 +0000

I started migrating our physical machines to VMs using VMware a few years ago and the first problem I ran into is still the most annoying one: the size limit for LUNs is, per VMware’s docs, (2TB - 512B). That’s 512 bytes shy of 2TB, so basically 1.99999 TB, or 2047.999 GB. So when I create a new LUN for a datastore in the SAN the max size is 2047 GB. Now, as the VMware KB article states, this is a limitation of SCSI, not VMware per se, but that doesn’t make it any less annoying. When I first setup ESX, I created a 5 TB LUN for the datastore. It showed up in vCenter as 1 TB. After some Googling I learned of the 2 TB limit – the usable space is basically usable space = (size of lun) % 2TB, where % is the modulo operator – and found something suggesting using extents to expand the datastore across luns. I did that, but I later learned that there seems to be a consensus that extents should be avoided.

The Joy of Migrating from Exchange 2003 to 2010

Tue, 20 Jul 2010 13:57:13 +0000

I’ve been working on migrating from Exchange 2003 to Exchange 2010 for several weeks. Actually, at this point it feels like several months. Now that I think about it, I guess that’s because it’s actually been several months.

Back in January or February, I got fed up with the Exchange setup I inherited: our Exchange 2003 server was running on a server in the basement of our office, on non-UPS power, with a power company that likes to pull shenanigans (like 3-4 hour outages every few months). In addition, the physical machine itself has some weird bug where it would hang at the POST screen complaining about some USB device, even though there are no USB devices plugged in, and USB is disabled in the BIOS. Meanwhile, in the datacenter, I had recently finished migrating most of our ancient physical servers to virtual machines on beautiful new hardware. It didn’t take long to see the solution that seemed to be obvious: move Exchange to the datacenter, in a VM.

vCenter: Error parsing the server "(server IP)" "clients.xml" file

Wed, 23 Jun 2010 12:18:59 +0000

I got the above error today after running Windows Update on my XP VM a few days ago. A quick search showed that the error is caused by a Microsoft update to the .NET framework. To resolve it, remove update KB980773 (Add/Remove programs, make sure “Show Updates” is checked; KB980773 is under “Microsoft .NET Framework 2.0 Service Pack 2”). I removed it and was able to log in without problems.

vCenter: Error parsing the server "(server IP)" "clients.xml" file

Wed, 23 Jun 2010 12:18:59 +0000

Windows XP guests hang on shutdown in VMware Workstation on Linux

Fri, 23 Apr 2010 16:45:30 +0000

I had this problem with FC11, where I couldn’t properly shut down or suspend the Windows XP VM I run (mostly for Outlook). When I’d shutdown inside the guest, it would mostly do what you’d expect, but then the screen would go blue, then black, and then the only way I could get the thing to exit was to kill the pid of vmware-vmx (or a ‘killall vmware-vmx’). I solved it somehow by removing some RPM, but when I went to FC12 the other day it came back, and I couldn’t remember how I’d fixed it initially. I Googled for nearly a full day before I found the original blog post that told me which RPMs to remove - for some reason I thought it was fprintd, but it was something else completely. Here’s the original post, and when I just read it I was going to comment on it to say thanks for posting it, but apparently I did that already. Anyway, I removed pcsc-lite again and everything appears to be good now.

Moving an Exchange 2003 server to another location with minimal risk and disruption?

Wed, 14 Apr 2010 01:38:51 +0000

So our Exchange server is located in our office building. This made sense at the time because that’s where the users are. Over time though, this has proved problematic for a few reasons. Primarily, our office is certainly not a datacenter and doesn’t offer the amenities of one - clean, reliable power, and redundant cooling. In an average year we lose power probably 10-15 times, often for an hour or more. The rest of our production environment is hosted in a top-tier datacenter, so after a while I started to wonder why our Exchange server wasn’t there, and making plans to move it there. Oh, and did I mention I’m not an Exchange admin in any sense of the term? I just inherited the Exchange server about 2 months ago.

More fun with ovftool

Sun, 20 Sep 2009 14:28:11 +0000

I tried restoring my .ova into vCenter and it failed. It said something like “Device ‘virtual disk’ uses a controller that is not supported. this is a general limitation of the virtual hardware version.” This kind of pissed me off, especially when I couldn’t really find anything in Google about what the error meant, but it turned out to be pretty easy to fix - I just clicked “upgrade virtual hardware” in VMWare Server to upgrade the hardware version from 4 to 7. I then created another .ova, uploaded it to the datacenter and successfully deployed it in vCenter in its own resource pool. Yay!

Exporting VMWare Server VMs to OVA/OVF with ovftool

Fri, 11 Sep 2009 18:56:25 +0000

Well, I thought this was going to be a huge pain in the ass, but it turns out not to be a really big deal. I found this post which includes instructions and a link to download VMWare’s “ovftool”. There’s a win32 version and a 32-bit and 64-bit linux version:

-rwxr--r-- 1 evan evan 19326184 Sep 11 17:16 VMware-ovftool-installer-1.0.0-166674-lin.i386.sh*
-rwxr--r-- 1 evan evan 17212108 Sep 11 17:16 VMware-ovftool-installer-1.0.0-166674-lin.x86_64.sh*
-rwxr-xr-x 1 evan evan 43833393 Sep 11 17:33 jdk-1_5_0_13-linux-amd64.bin*

I installed JDK 1.5.0_r13 because that’s what I had laying around. The 64-bit installer for ovftool ran without a hitch. I haven’t yet tried importing the .ova to my vCenter cluster (the export was from our Vmware Server (free) environment) but the conversion itself seemed fine. The one problem I encountered was that it was complaining that the cdrom was connected. The VM in question had an ISO configured as its cdrom drive, but it wasn’t connected, yet the ovftool still complained about it. I shutdown the VM, deleted the device completely, and then the conversion went fine.

VMWare rules

Sun, 05 Jul 2009 12:28:22 +0000

I love VMWare. I’ve been using it for less than a year but it’s easily one of my favorite products. I upgraded my ESX 3.5 cluster to 4.0 a little over a week ago and it was a pretty simple procedure - the vCenter server pushes out the new ISO to the ESX server, installs it, and reboots it. I haven’t really played with most of the 4.0 features yet, but the new performance overview graphs are awesome.

VMWare is pretty cool.

Fri, 24 Apr 2009 18:53:01 +0000

Over the past few months I’ve gotten to love VMWare. We had purchased a SAN around May 2008 for one project, and all the SAN vendors kept asking us if we did anything with virtualization. The first few times we kind of furrowed our brows and said no, but eventually I started wondering if this was something I should be looking into.

I’d used some VMWare products before - the Workstation and the free VMWare Server product - and didn’t have great experiences. We tried using VMs for development environments but the performance was poor and there was a serious problem with the system clock in the VMs losing 2-3 seconds per real-time second; the clock skew was so bad that ntp couldn’t even set the clock, it diverged too quickly. The performance problems were largely our fault, I realize now: we tried running 3 or 4 VMs on a host with only 2 GB of memory (each VM wasn’t allocated that much), and the machine only had a single dual-core Opteron. Plus, the host machine was also serving as a dev Apache server for our main web app. This setup started out okay but it degraded over the course of about a year to the point that the VMs were basically unusable. We used VMs for a few other minor projects but after this initial project it was always something that made me crinkle my nose.