VMWare vSphere Upgrade at AGLT2

In September 2012 VMware release a signficant upgrade to vSphere: 5.1. There are a number of nice features that we want to benefit from at AGLT2 and with the MSU installation just starting we would like to base our work on 5.1. You can see details about what 5.1 provides at http://www.vmware.com/files/pdf/products/vsphere/vmware-what-is-new-vsphere51.pdf

Some useful URLs:

vSphere 5.1 upgrade best practices
Comparing the behavior of vCenter Single Sign On with earlier versions of vCenter Server
Methods of upgrading to vSphere 5.1
Installing vCenter Single Sign On in a multisite deployment
VMware 5.1 Documents
vSphere 5.1 Storage Appliance Release Notes

Procedures

We need to download the required software and then

1. Upgrade the UMVMGT_VM vCenter from 5.0 to 5.1
2. Verify new web interface functionality
3. Setup the VMware Update Manager to allow it to upgrade our ESXi 5.0 systems to 5.1
4. Do a "rolling" upgrade of the ESXi servers: UMVM01.AGLT2.ORG, UMVM02.AGLT2.ORG and UMVM03.AGLT2.ORG
5. Upgrade VMs to new hardware and vmware-tools installs
6. Verify and update plugins and added components (Dell plugin for VMware and MDxxxx storage mostly)

Software Download

We downloaded most of the "Enterprise Plus" items from the VMware Download site and stored them on UMVMGT_VM in the C:\VMware directory. We use "Virtual Clonedrive" on this system to "Mount" the ISO on drive E:.

Upgrade the UMVMGT_VM vCenter System

To prepare for upgrades I did two things:

1. Made a "clone" of the VM called UMVMGT_VM-prior-to-51
2. Created a "Snapshot" prior to changes

Examining the requirements showed that a "simple" upgrade (one which uses a single host to run the SSO, inventory service and vCenter) would require 100+GB of disk and 10G of memory. The existing VM didn't have sufficient memory so I shut it down and increased it to 10G. While it was done I updated the "hardware" to v8 and setup the option to "hot-add" memory or CPU.

When I started the VM was being hosted on the MD3000i storage. I had two issues with the system "hanging" for now reason. To give the system more I/O capacity I moved it to an MD3600i LUN.

My first try at the upgrade failed in setting up the SSO. You have to do a bit of work with the Database config. I had problems getting it to work regarding the needed DB setup. The "pre-install" check I ran showed no problems but getting the system to work with an existing DB takes more steps. I ended up running the provided scripts (with some required customization edits) to create the "RSA" database on our MSSQL 2008 R2 server and setup the users prior to running the upgrade. However the install still failed because it couldn't connect to the DB?!

The problem was that the existing install of MSSQL 2008 R2 was using "dynamic" TCP ports while the install scripts expect to find MSSQL running at port 1433. I had to run the MSSQL configuration utility, reconfigure for static ports and reboot the system. Then the MSSQL service was showing up at port 1433.

Install SSO on vSphere 5.1

First prompt is to setup the password of the 'admin@System-Domain' account. I used our "stock" UM 'root' pw.

Next step is the DB choice: I picked "Use an existing DB"

The setup screen for the SSO DB looked like this:

Inventory Service

The next part of the install is the Inventory Service. I choose to not overwrite the Inventory Database.

Next I hit this problem:

I found this in the ESXi 5.0 docs on http://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vsphere.upgrade.doc_50%2FGUID-093777CF-BB5A-4D23-A41D-5B791789E33C.html

Grant the following permissions to the vCenter user in the vCenter database:

GRANT ALTER ON SCHEMA :: <schema> to <user>;
GRANT REFERENCES ON SCHEMA :: <schema> to <user>;
GRANT INSERT ON SCHEMA :: <schema>  to <user>;
GRANT CREATE TABLE to <user>;
GRANT CREATE VIEW to <user>;
GRANT CREATE Procedure to <user>;

Grant the following permissions to the user in the MSDB database:

GRANT SELECT on msdb.dbo.syscategories to <user>;
GRANT SELECT on msdb.dbo.sysjobsteps to <user>;
GRANT SELECT ON msdb.dbo.sysjobs to <user>;
GRANT EXECUTE ON msdb.dbo.sp_add_job TO <user>;
GRANT EXECUTE ON msdb.dbo.sp_delete_job TO <user>;
GRANT EXECUTE ON msdb.dbo.sp_add_jobstep TO <user>;
GRANT EXECUTE ON msdb.dbo.sp_update_job TO <user>;
GRANT EXECUTE ON msdb.dbo.sp_add_category TO <user>;
GRANT EXECUTE ON msdb.dbo.sp_add_jobserver TO <user>;
GRANT EXECUTE ON msdb.dbo.sp_add_jobschedule TO <user>;

Here is the SQL I created to fix the problem:

use VIM_VCDB;
GRANT ALTER ON SCHEMA :: dbo to virtual_center;
GRANT REFERENCES ON SCHEMA :: dbo to virtual_center;
GRANT INSERT ON SCHEMA :: dbo  to virtual_center;
GRANT CREATE TABLE to virtual_center;
GRANT CREATE VIEW to virtual_center;
GRANT CREATE Procedure to virtual_center;
use msdb;
CREATE USER virtual_center for LOGIN virtual_center
GRANT SELECT on msdb.dbo.syscategories to virtual_center;
GRANT SELECT on msdb.dbo.sysjobsteps to virtual_center;
GRANT SELECT ON msdb.dbo.sysjobs to virtual_center;
GRANT EXECUTE ON msdb.dbo.sp_add_job TO virtual_center;
GRANT EXECUTE ON msdb.dbo.sp_delete_job TO virtual_center;
GRANT EXECUTE ON msdb.dbo.sp_add_jobstep TO virtual_center;
GRANT EXECUTE ON msdb.dbo.sp_update_job TO virtual_center;
GRANT EXECUTE ON msdb.dbo.sp_add_category TO virtual_center;
GRANT EXECUTE ON msdb.dbo.sp_add_jobserver TO virtual_center;
GRANT EXECUTE ON msdb.dbo.sp_add_jobschedule TO virtual_center;

The fix in Studio looks like this:Here is what it looks like in the

The next issue (after fixing permissions) was another permissions error:

I mistakenly thought it was referring to the stored procedures from the previous install so I used the MS Studio to backup the VIM_VCDB database and then deleted all the non-system stored procedures:

The real problem was the SQL Agent Jobs shown here:

They were not owned by the virtual_center account. I changed the owner of all the jobs referring to VCDB to virtual_center. I am assuming the install will replace/recreate the stored procedures above that I removed.

So trying to continue I then get this:

Dang! This is frustrating! So I think I have to cancel the install, restart the vCenter services and login so I can change this setting. I hope this doesn't mess something up.

So I exited the install. It mentioned having to install "individual components" when if I wanted to install (not "Simple Install"). I started the vCenter Server and corresponding Webserver, connected the client and made the "SSL Settings" change. No ESXi hosts showed up. I exited vCenter client, stopped the vCenter Server and tried to restart the "Simple Install" but it wouldn't let me. It turned out I just need to (re)install the Inventory service and then do the vCenter itself:

The "install" process had me remove the Inventory service when I started it and then I just reinstalled it. Went very easily this time.

Next is the vCenter Server and I selected "Updating the database" option (I copied the SSL certificates to 'SSL-Backup' as before running it). This part is slow (DB update takes ~15 minutes).

No problems finishing the vCenter update nor the vSphere Client nor the vSphere Web Client.

I hit a problem with the VMware Update Manager though. After a long delay it popped up an error box saying it couldn't connect. I tried again and got the same issue. At this point I rebooted the system. Once it came back the upgrade of VUM worked fine with the minor exception of a DB issue:

This problem was remedied by giving the virtual_center MSSQL login "db_owner" on the MSDB schema.

I also installed all the VMware Support Tools listed above with the exception of the "Authentication Proxy" which required IIS to be installed.

Using the new vCenter Clients

Once things were installed I tried out both the VMware vSphere Client and the web client.

If you connect with the older VMware vSphere Client it will prompt you to download and upgrade. This worked without a problem. One note: none of the new 5.1 features are supported in the vSphere (non-Web) Client. The plugins from Dell seemed to work fine in the new 5.1 client.

You can also open a web page to http://umvmgt.physics.lsa.umich.edu:

If you click on the web client it will open https://umvmgt.physics.lsa.umich.edu:9443/vsphere-client/
At this point you can login with either the SSO "admin" (admin@System-Domain) or the UMVMGT "Administrator" accounts. They will show you different views since, for example, the SSO "admin" isn't authorized on the ESXi 5.0 hosts we have right now.

I did have a problem connecting via both the client and the web client:
This was just due to the "vCenter Server" service being down on UMVMGT. Not sure why it stopped but a simple "Start" on the service fixed things.

-- ShawnMcKee - 17 Sep 2012