Windows on Evan Hoffman

Exchange (OWA) CAS crashes with 503 error - again

Sun, 18 Sep 2011 12:22:46 +0000

This just started happening again, with these errors appearing in the event viewer:

Log Name: System
Source: Microsoft-Windows-WAS
Date: 9/18/2011 11:16:33 AM
Event ID: 5011
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: exch2010fe1
Description:
A process serving application pool 'MSExchangeOWAAppPool' suffered a
fatal communication error with the Windows Process Activation Service.
The process id was '3760'. The data field contains the error number.

Log Name: System Source: Microsoft-Windows-WAS Date: 9/17/2011 6:47:07 AM Event ID: 5009 Task Category: None Level: Warning Keywords: Classic User: N/A Computer: exch2010fe1 Description: A process serving application pool ‘MSExchangeOWAAppPool’ terminated unexpectedly. The process id was ‘3108’. The process exit code was ‘0x800703e9’.

Renaming a single-label domain to a FQDN

Thu, 07 Apr 2011 17:56:54 +0000

Long ago – eons, perhaps – before I had anything to do with the Windows environment here, someone created the AD domain in my company as a single-label domain (e.g. instead of “example.com” our domain is just “example”). Over the years this has led to lots of “fun” on the part of Windows admins who’ve worked here as the implications of this choice became more apparent.

Since I inherited this system about a year ago, I haven’t really bumped up against any problems stemming from the single-label domain issue… until now. I recently attempted to add a new Windows 2008r2 file server to our DFS replication group/namespace. This totally failed for some mysterious reason. Well, I shouldn’t say “totally” failed, as I was able to add it to the DFS replication group, but unable to add it to the DFS namespace. In my attempt to debug the namespace issue, I deleted the namespace and attempted to recreate it, but just kept getting this error: The namespace cannot be queried. The specified domain either does not exist or could not be contacted.. I couldn’t do anything with the namespace - even clicking on it in the DFS Management console brought up an error. After some searching I found that this was likely due to having a single-label domain. I wasn’t sure why the error was happening even on Windows 2003 machines though, maybe joining a 2008r2 box to the domain made some schema changes? I tried a few suggestions like editing the hosts file but nothing seemed to resolve this.

Autodiscover mysteriously stopped working (Exchange 2010)

Tue, 01 Mar 2011 12:30:30 +0000

I had Autodiscover working for months but recently it just stopped. I’m not sure why, but it may be related to removing the last Exchange 2003 servers from service recently. Maybe some setting got wiped from AD when I uninstalled Exchange 2003 (as per the procedure Microsoft gives). Basically what was happening was that the email address field was being autopopulated by the user’s UPN rather than their email address. Since we have a single label domain, the UPN isn’t a valid email address, and autodiscovery fails.

Wasted time with Exchange 2010, SquirrelMail, and IMAP-SSL

Tue, 30 Nov 2010 17:32:11 +0000

I’m setting up SquirrelMail to point to my Exchange 2010 server via IMAP (don’t ask) and couldn’t get SM to talk to Exchange on port 993 (imaps). Even though the servers on the same subnet, any time passwords are being sent over the network I like to opt for SSL. I found a couple of sites suggesting that the problem was that there was no SSL certificate installed, but I knew for a fact there was a valid certificate because I could get to https://webmail.example.com/ for OWA.

Using Zabbix for SNMP monitoring disk usage percent for Windows hosts

Fri, 05 Nov 2010 09:44:22 +0000

A few years ago we moved from Nagios to Zabbix for our server monitoring needs. I wasn’t a big fan of Nagios, finding it a pain to manage with its myriad configuration files. It’s probably gotten better since I last toyed with it but since we moved to Zabbix I haven’t had much reason to look at Nagios again.

I also try to use SNMP monitoring for everything. SNMP is widely supported - all sorts of hardware has SNMP support, and with the net-snmp package you can pretty easily create your own SNMP-monitorable stuff on Linux. Since almost all of our stuff runs on Linux this has worked out pretty well, but our Exchange server is probably going to be running on Windows for the foreseeable future. Windows has SNMP support, it’s just not on by default. However, even when it’s enabled it doesn’t have the simple “dskPercent” monitoring I’ve come to know and love with net-snmp on Linux, which simply tells you how full a given disk is as a percent. This makes it easy to set alerts when a disk reaches 80% full.

Exchange 2010 Post-Upgrade weirdness: can't edit Mail Non-Universal Group or Security Group

Thu, 21 Oct 2010 22:29:38 +0000

Now that everyone’s been moved to Exchange 2010 we’ve started using the 2010 Exchange Managment Console/Shell exclusively which has revealed some weirdness. First, we created a new group in AD using an old script (which used LDAP) and created a Mail-enabled Global Security group. We put people in the group, and everything seemed to be working fine until it was discovered that users in the group couldn’t see the group in the Global Address List. Users not in the group had no problem seeing the group. Additionally, users in the group couldn’t see users added directly in 2010. This only appeared to affect the GAL; the users were able to send/receive email fine with the full SMTP addresses.

Exchange 2010 Post-Upgrade weirdness: can't edit Mail Non-Universal Group or Security Group

Thu, 21 Oct 2010 22:29:38 +0000

Finally, all users moved from Exchange 2003 to Exchange 2010.

Tue, 12 Oct 2010 15:29:44 +0000

I’ve been working on migrating our Exchange environment from 2003 to 2010 for several months. My first post about this is from April 14th, when I was just trying to virtualize our existing Exchange 2003 system. Once that was complete, I started playing around with Exchange 2010 around June or July, and had most of the users moved over to 2010 by the end of August. The last holdouts were Blackberry users. I couldn’t move their mailboxes because our BES was hosted on our original Exchange 2003 server.

One reason I hate iTunes.

Thu, 16 Sep 2010 08:04:43 +0000

I’ve always hated iTunes. It’s a huge pile of bloatware and it’s slow as poo. It’s like 100 mb or more for an mp3 player. I remember winamp playing mp3s when it was a 500k download. Anyway.

I keep all my music on a Linux machine running samba. This way it’s available to every machine in the house. When I had Winamp on all my machines this was wonderful. But now that I’m forced into iTunes (thanks to having an iPhone), it turns out to be a major pain. In iTunes I unchecked the box for “let iTunes keep my libary organized” to prevent it from copying the entire library to each computer’s local disk. Initially adding my library of ~4000 tracks to iTunes takes over an hour (100 mbit wire) - it would take about 5 minutes in Winamp, even reading the ID3 tags for each track as it was added (rather than lazily as the song was played).

Converting Exchange 2003 conference rooms to Exchange 2010

Wed, 01 Sep 2010 13:25:27 +0000

I’m wrapping up moving mailboxes to Exchange 2010. The last ones to be moved (except for BlackBerry users… thanks BES) are the conference rooms. So the first step was to move them using the Local Move tool, which was pretty simple. But I don’t want them in 2010 as user mailboxes if they can be designated as “rooms,” which they can. So here’s how I’m doing it:

Identify the mailboxes to be moved

ldapsearch example on Active Directory

Thu, 26 Aug 2010 10:40:18 +0000

Just putting this here for safekeeping since I couldn’t remember the exact syntax.

[evan@ehoffman 10:35:50 ~]$ ldapsearch -x -LLL -D "ldapuser@example.com" -w password -b "OU=Users,DC=example,DC=com" -s sub -H ldaps://activedirectory.example.com "(sn=hoffman)" cn mail displayName samaccountname
dn: CN=Evan Hoffman,OU=Tech,OU=Users,DC=example,DC=com
cn: Evan Hoffman
displayName: Evan D. Hoffman
sAMAccountName: ehoffman
mail: Evan.Hoffman@example.com

Explanation: Connect to activedirectory.example.com using ldaps (SSL) with simple authentication, binding as ldapuser@example.com with password password; search for (sn=hoffman) within the OU=Users,DC=example,DC=com search base (branch), and search the subtree. Return the cn, displayName, and samaccountname fields.

ldapsearch on Active Directory

Thu, 26 Aug 2010 10:40:18 +0000

Just putting this here for safekeeping since I couldn’t remember the exact syntax.

[evan@ehoffman 10:35:50 ~]$ ldapsearch -x -LLL -D "ldapuser@example.com" -w password -b "OU=Users,DC=example,DC=com" -s sub -H ldaps://activedirectory.example.com "(sn=hoffman)" cn mail displayName samaccountname
dn: CN=Evan Hoffman,OU=Tech,OU=Users,DC=example,DC=com
cn: Evan Hoffman
displayName: Evan D. Hoffman
sAMAccountName: ehoffman
mail: Evan.Hoffman@example.com

Exchange 2010 - Out-of-office response (OOF) won't turn off?

Wed, 11 Aug 2010 12:44:34 +0000

Two users reported the same problem this week: they turned on their out-of-office reply while they were out, then came back and turned it off. Except even after they turned it off, the autoreply was still being sent out. I had them log in to OWA and make sure it was off (maybe some weird bug with Outlook not registering the change in the server), which it was in both cases. I Googled hard and fast and couldn’t find anyone with this same problem.

Exchange 2010 - Out-of-office response (OOF) won't turn off?

Wed, 11 Aug 2010 12:44:34 +0000

Changing Active Directory Password in Browser through OWA 2010

Thu, 05 Aug 2010 11:29:41 +0000

A few months ago I was on a quest to figure out how to change my Active Directory password via a browser (for Linux/Mac users). I finally figured it out, but since I’ve been working on this Exchange 2010 migration I noticed one of the features of OWA (Outlook Web App) in Exchange 2010 is that you can change the AD password right in the browser from within the app:

Outlook 2007 & Exchange 2010 Autodiscover SSL certificate error annoyance

Tue, 27 Jul 2010 11:51:54 +0000

One of the more annoying side effects of migrating my mailbox to Exchange 2010 has been the nagging of Outlook 2007’s Autodiscovery feature. Now, every time I start Outlook I get hit with a certificate error for autodiscover.domain.com. Now, autodiscover.domain.com is a CNAME to mail.domain.com, which is the OWA URL for the CAS. The SSL certificate is valid - but it’s valid for mail.domain.com. I could buy a SSL certificate from GoDaddy for $12.99 (an insanely great price, btw) for “autodiscover” but that would also require using another IP address on the CAS (since you can can only bind one SSL certificate to an IP:port pair), and that seems like a waste of an IP address.

The Joy of Migrating from Exchange 2003 to 2010

Tue, 20 Jul 2010 13:57:13 +0000

I’ve been working on migrating from Exchange 2003 to Exchange 2010 for several weeks. Actually, at this point it feels like several months. Now that I think about it, I guess that’s because it’s actually been several months.

Back in January or February, I got fed up with the Exchange setup I inherited: our Exchange 2003 server was running on a server in the basement of our office, on non-UPS power, with a power company that likes to pull shenanigans (like 3-4 hour outages every few months). In addition, the physical machine itself has some weird bug where it would hang at the POST screen complaining about some USB device, even though there are no USB devices plugged in, and USB is disabled in the BIOS. Meanwhile, in the datacenter, I had recently finished migrating most of our ancient physical servers to virtual machines on beautiful new hardware. It didn’t take long to see the solution that seemed to be obvious: move Exchange to the datacenter, in a VM.

vCenter: Error parsing the server "(server IP)" "clients.xml" file

Wed, 23 Jun 2010 12:18:59 +0000

I got the above error today after running Windows Update on my XP VM a few days ago. A quick search showed that the error is caused by a Microsoft update to the .NET framework. To resolve it, remove update KB980773 (Add/Remove programs, make sure “Show Updates” is checked; KB980773 is under “Microsoft .NET Framework 2.0 Service Pack 2”). I removed it and was able to log in without problems.

vCenter: Error parsing the server "(server IP)" "clients.xml" file

Wed, 23 Jun 2010 12:18:59 +0000

Hygiene Management?

Thu, 03 Jun 2010 15:55:39 +0000

I’m installing Exchange 2010 and in the docs it shows a bunch of groups that get created in the AD Schema during the domain prep part. After running prep, I looked to see if the groups were there, and sure enough they were (yay). What caught my eye was that one of the groups is called Hygiene Management. I thought maybe this was an Easter Egg from MS, but apparently it’s just the name of the group of people who can manage the Exchange antivirus/antispam features. Still funny though.

Moving an Exchange 2003 server to another location with minimal risk and disruption?

Wed, 14 Apr 2010 01:38:51 +0000

So our Exchange server is located in our office building. This made sense at the time because that’s where the users are. Over time though, this has proved problematic for a few reasons. Primarily, our office is certainly not a datacenter and doesn’t offer the amenities of one - clean, reliable power, and redundant cooling. In an average year we lose power probably 10-15 times, often for an hour or more. The rest of our production environment is hosted in a top-tier datacenter, so after a while I started to wonder why our Exchange server wasn’t there, and making plans to move it there. Oh, and did I mention I’m not an Exchange admin in any sense of the term? I just inherited the Exchange server about 2 months ago.

Victory! Change Active Directory Password via LDAP through browser

Wed, 13 Jan 2010 13:51:43 +0000

I had to give up on PHP and go to Perl, but it turned out not to be so bad. Users can now change their Active Directory passwords via a self-service web page that doesn’t require admin credentials. The Perl code is below. Authentication to the script is done via .htaccess LDAP authentication, so the REMOTE_USER env variable is assumed to contain the user’s username (sAMAccountName) by the time this script is called. There is a simple check for $ENV{HTTPS} to ensure the script is called via SSL, and AD requires password changes to be done via ldaps, so the whole thing should be encrypted end to end.