VMware - My experience of upgrading from 4.1 to 5.0 update 2

**************************************************
Update:

We found an interesting article prior to updating the Hosts to ESXi 5.0 in which a NFS disconnect issue is highlighted - so we haven't upgraded to 5.0 as yet. Also impacts 5.1.

Impacted products: vSphere 5.0, NetApp also EMC.
***************************************************
 
My experiences of upgrading from 4.1 to 5.0.......

The test upgrade of a database (existing VC DB restored to another server) worked fine. Just the normal glitches of not being given the correct rights to upgrade the DB the first time of asking - dbo right on the DB and MSDB (the one that gets missed).

The test of the new feature of Web Client Server went well - the test of VPN connection using a user account proved access/connectivity to the nominated VM.

The testing ticked all the boxes. Excellent!

Production upgrade didn't go too well - it failed with a SQL script error every time on the first two days.
A call was logged with VMware to get assistance as no amount of Googling helped.

The VMware Engineer asked for a number of SQL DB checks to be done and set the Logs to unlimited growth. On the third day all steps were followed and the upgrade still failed. After the second restore and prior to the roll back to vSphere 4.1 one of the DBAs checked to see what differences there were between the Production DB and the Test DB - the only difference was the Compatibility level in the Database Properties (Test set to SQL Server 2008 (100) and Production set to SQL Server 2005 (90)). Production had its setting changed to SQL Server 2008 (100) and the upgrade worked!!

Once the upgrade completed VUM was upgraded followed by the Web Client Server install - no problems were encountered with either of these.

Now for the interesting bits.....

• Some SSL issues mainly around the Web Client Server.
• My VPN access isn't least privilege and therefore port 9443 wasn't open for general users - Firewall and VPN change required.
• Web Client users getting  "Empty Inventory" when they connect which happens to be a known error - fix can be found here. I used the Domain Users group and gave them Read rights at the top level with no propagation.
• Web Client users unable to see VMs they have console access to. The previous fix had been applied approximately 1 hour prior to the user trying Web Client. My tests, adding the console rights the user had to my user account, also failed to see the any VMs. Approximately another hour later the user checks and can see most of the VMs now. Checking my user account I can see ALL of the VMs that the user has access to. Permissions propagation issue????
• During my testing to see why the user couldn't access the VMs I applied Read Only right to the Domain Users group at a Host and one of the impacted VMs - which looking back wasn't a good idea. Why? The least privilege rule applied which meant my Domain Admin account went from having full Virtual Center Admin rights to Read Only. This meant I couldn't do anything - at the time we thought it was something to do with the upgrade. This then had a knock on affect to our nightly backup as this account also belonged to the Domain Users group - the NetApp backup failed just because ONE machine out of 300 didn't have the right permissions. What happened to error checking??  Logged a call with VMware and before the lady rang us the epiphany hit (least privilege trump card). With VMware support we went through logging into the VI Client as the Administrator account (in our case the local admin account on the VC server) and since it had nothing to do with Domain authentication and had the highest level permissions I could remove the Domain Users Read Only from the Host and VM. Checking Web Client access and everything was working.
• Annotations permissions have changed - Support staff can't edit the notes. The next day, with no changes applied, they could edit the notes....

Lessons learnt:

• Have a VPN account which is mimics least privileged access.