Solaris - finding and restarting services

The other day I was looking to restart a service on a Solaris 10 server as a management console was reporting that is couldn't connect to the service.

(In this example I'm looking for all Sophos services)

# svcs -a | grep sophos

online         Feb_18   svc:/com/sophos/sav/sav-protect:default
online         11:09:43 svc:/com/sophos/sav/sav-update:default
online         11:51:47 svc:/com/sophos/sav/sav-rms:default

So I've found the services: How do I restart one?

# svcadm restart svc:/com/sophos/sav/sav-rms:default

Or using its short unique name:

# svcadm restart sav-rms

online         Feb_18   svc:/com/sophos/sav/sav-protect:default
online         11:09:43 svc:/com/sophos/sav/sav-update:default
online         11:51:47 svc:/com/sophos/sav/sav-rms:default

The service should now be back online but what happens when the service doesn't comeback as online?
You have to enable the service if it goes into maintenance (not sure why this happened as I had run the same command on other servers without the service going into maintenance).

online         Feb_18   svc:/com/sophos/sav/sav-protect:default
online         11:09:43 svc:/com/sophos/sav/sav-update:default
maintenance11:51:47 svc:/com/sophos/sav/sav-rms:default

# svcadm enable sav-rms

online         Feb_18   svc:/com/sophos/sav/sav-protect:default
online         11:09:43 svc:/com/sophos/sav/sav-update:default
online         11:51:47 svc:/com/sophos/sav/sav-rms:default

VMware - My experience of upgrading from 4.1 to 5.0 update 2

**************************************************
Update:

We found an interesting article prior to updating the Hosts to ESXi 5.0 in which a NFS disconnect issue is highlighted - so we haven't upgraded to 5.0 as yet. Also impacts 5.1.

Impacted products: vSphere 5.0, NetApp also EMC.
***************************************************
 
My experiences of upgrading from 4.1 to 5.0.......

The test upgrade of a database (existing VC DB restored to another server) worked fine. Just the normal glitches of not being given the correct rights to upgrade the DB the first time of asking - dbo right on the DB and MSDB (the one that gets missed).

The test of the new feature of Web Client Server went well - the test of VPN connection using a user account proved access/connectivity to the nominated VM.

The testing ticked all the boxes. Excellent!

Production upgrade didn't go too well - it failed with a SQL script error every time on the first two days.
A call was logged with VMware to get assistance as no amount of Googling helped.

The VMware Engineer asked for a number of SQL DB checks to be done and set the Logs to unlimited growth. On the third day all steps were followed and the upgrade still failed. After the second restore and prior to the roll back to vSphere 4.1 one of the DBAs checked to see what differences there were between the Production DB and the Test DB - the only difference was the Compatibility level in the Database Properties (Test set to SQL Server 2008 (100) and Production set to SQL Server 2005 (90)). Production had its setting changed to SQL Server 2008 (100) and the upgrade worked!!

Once the upgrade completed VUM was upgraded followed by the Web Client Server install - no problems were encountered with either of these.

Now for the interesting bits.....

• Some SSL issues mainly around the Web Client Server.
• My VPN access isn't least privilege and therefore port 9443 wasn't open for general users - Firewall and VPN change required.
• Web Client users getting  "Empty Inventory" when they connect which happens to be a known error - fix can be found here. I used the Domain Users group and gave them Read rights at the top level with no propagation.
• Web Client users unable to see VMs they have console access to. The previous fix had been applied approximately 1 hour prior to the user trying Web Client. My tests, adding the console rights the user had to my user account, also failed to see the any VMs. Approximately another hour later the user checks and can see most of the VMs now. Checking my user account I can see ALL of the VMs that the user has access to. Permissions propagation issue????
• During my testing to see why the user couldn't access the VMs I applied Read Only right to the Domain Users group at a Host and one of the impacted VMs - which looking back wasn't a good idea. Why? The least privilege rule applied which meant my Domain Admin account went from having full Virtual Center Admin rights to Read Only. This meant I couldn't do anything - at the time we thought it was something to do with the upgrade. This then had a knock on affect to our nightly backup as this account also belonged to the Domain Users group - the NetApp backup failed just because ONE machine out of 300 didn't have the right permissions. What happened to error checking??  Logged a call with VMware and before the lady rang us the epiphany hit (least privilege trump card). With VMware support we went through logging into the VI Client as the Administrator account (in our case the local admin account on the VC server) and since it had nothing to do with Domain authentication and had the highest level permissions I could remove the Domain Users Read Only from the Host and VM. Checking Web Client access and everything was working.
• Annotations permissions have changed - Support staff can't edit the notes. The next day, with no changes applied, they could edit the notes....

Lessons learnt:

• Have a VPN account which is mimics least privileged access.

VMware - SSL

Having recently upgraded from vSphere 4.1 to vSphere 5.0 Update 2 we are currently suffering from some SSL issues - all appeared to be resolved apart from one (which is the new Web Client service).

Being lucky enough to be in a team large enough where we have some specialists the SSL certificate errors were resolved quickly.

Goggling around a bit there seems to be a number of people suffering from SSL issues and I found some interesting articles written by Michael Webster which are listed below:

http://longwhiteclouds.com/2012/02/24/updating-ca-ssl-certificates-in-vsphere-5/

http://longwhiteclouds.com/2012/02/07/the-trouble-with-ca-ssl-certificates-and-vcenter-5/

http://longwhiteclouds.com/2012/09/15/vcert-manager-changing-vmware-ssl-certs-made-easy/

Solaris - recover a root password in a local zone

Someone managed to reset the root password of a local zone incorrectly which resulted in a machine we couldn't log in as root......

To fix it log in to the global zone as root.

Edit the shadow file of the offending local zone (e.g. Local zone is called LZ01)

# vi /zones/LZ01/root/etc/shadow

Edit the root entry in the shadow file like so:

root::15435::::::

The save the entry (esc key, colon, wq!)

Log in to the console of the local zone from the global zone:

# zlogin -C LZ01

Log in as root (which now just logs you in without prompting for a password).

Reset the root password.

# passwd root

Follow the prompts.......

vSphere - Monitoring vSMP VMs (VMA & resxtop)

A server was built in the VI and I received the healthcheck email the next morning stating there was a 4 vCPU VM - which is unusual as the max vCPU we normally have is 2! The supplier was quite insistent that the server needed to have 4 CPUs.

The warnings VMware give with regards vSMP VMs is (as quoted from the VMworld 2011 VSP3866 session):

"vSMP VMs may not always use those vCPUs. Test your applications and verify that the threads for that application are being split among the processors equitably. An idle vCPU incurs a scheduling penalty.

Pay attention to the %CSTP counter on vSMP VMs. The more you see this, the more your processing is unbalanced. (The ESX 4.x relaxed co-scheduler is needing to catch all vCPUs up to a vCPU that is much further advanced.)".

In short - if there are too many vCPUs on a VM you could get the opposite affect to what you want.

How to check:

1. Log into the vMA using putty (or similar).
2. Set a target - in this case the Virtual Center server.

# vifptarget -s <vc-server>

3. Run resxtop.

# resxtop --server <esxi-host>

4. Login using the ESXi account when prompted (Host needs to be out of Lockdown Mode - Hosts are connected using AdAuth) .

5. Look for any increase from 0.00 in %CSTP.

6. Exit resxtop CTRL-C.

You can then decide what to do - add/remove vCPUs.

Description of %CSTP (extract from DOC-11812)
The percentage of time the world spent in ready, co-deschedule state. This co-deschedule state is only meaningful for SMP VMs. Roughly speaking, ESX CPU scheduler deliberately puts a VCPU in this state, if this VCPU advances much farther than other VCPUs. VCPU with high %CSTP is "stopped" from executing so that another VCPU in the same virtual machine could be run to "catch-up".

References:
VSP3866 VMworld 2011 sessions (USA and EMEA)

VMware communities interpreting esxtop 4.1 stats
http://communities.vmware.com/docs/DOC-11812

VMware communities interpreting esxtop stats
http://communities.vmware.com/docs/DOC-9279

Solaris - Third party management tools

So a New Year is upon us and I'd thought I'd better document what tools I use for managing the Solaris environment - if anyone read my ramblings I would be interested to see what you use.

Free Tools:

SSH connectivity - putty
SFTP connections to move files around - coreftp
GUI connectivity - Xming

I've used putty for a number of years on and off and it does the job admirably. I do like the fact it can be setup to record input and output which has helped me out a couple of times.

For file transfer I've started using coreftp as I read somewhere that the FileZilla client stored the SFTP password in clear text..... I hadn't read anything bad about coreftp.

We did have some older Exceed licenses for GUI connectivity and when I got given a Windows 7 build they weren't valid for a supported install - so I went looking for a freebie and stumbled across Xming which connects great but it doesn't like to be logged out so I just click on the X in the top right hand corner to kill the window. Nothing flags in the logs as being a problem afterwards but I'd rather not exit it that way and I will investigate a little more when I have time

HP - Service Pack for ProLiant

After the HP NC522SFP saga one of the nuggets of information I gratefully received was the HP Service Pack for ProLiant. Which was great! Apart from where can it be found on the HP website!!!!!!

I decided to hunt the page down to enable me to find it quickly when I needed to (also regularly check if there are new releases or not).

Home page: http://h18004.www1.hp.com/products/servers/management/spp/index.html

Download page: http://h18004.www1.hp.com/products/servers/service_packs/en/index.html

Extract from the site on benefits of using SPP:

Leverages the Power of HP Smart Update Manager (HP SUM).

Broad portfolio support for HP ProLiant servers, BladeSystem enclosures, and HP CloudSystem

Radically simplified updates with a single step installation process containing both firmware and systems software with drivers packaged together

• Consolidation -- single solution for all supported ProLiant servers
• Interdependency testing of drivers and firmware
• Offered from the web to provide convenient access
• PXE bootable ISO images to reduce customer:
• Qualification cycles
• Resource usage
• Maintenance windows
• Downtime

Integral Part of the HP Server Experience

• Essential management tool designed to simplify IT management in a dynamic and demanding 7x24x365 environment
• Increases Ease of Server Management
  • More efficient enclosure updates because of enhancements to HP SUM
  • Consolidated sets of tested firmware and HP System Software (drivers, agents, utilities)
  • Enhances IT staff productivity while reducing downtime
  • Provides Reliable Configuration by:
  • Assisting with server deployment, integrating server support software and firmware, and minimizing configuration and update issues
  • Ensuring consistency and reliability and maximizing system uptime
  • Simplified delivery of ProLiant software and firmware.