vSphere - Monitoring vSMP VMs (VMA & resxtop)

A server was built in the VI and I received the healthcheck email the next morning stating there was a 4 vCPU VM - which is unusual as the max vCPU we normally have is 2! The supplier was quite insistent that the server needed to have 4 CPUs.

The warnings VMware give with regards vSMP VMs is (as quoted from the VMworld 2011 VSP3866 session):

"vSMP VMs may not always use those vCPUs. Test your applications and verify that the threads for that application are being split among the processors equitably. An idle vCPU incurs a scheduling penalty.

Pay attention to the %CSTP counter on vSMP VMs. The more you see this, the more your processing is unbalanced. (The ESX 4.x relaxed co-scheduler is needing to catch all vCPUs up to a vCPU that is much further advanced.)".

In short - if there are too many vCPUs on a VM you could get the opposite affect to what you want.

How to check:

1. Log into the vMA using putty (or similar).
2. Set a target - in this case the Virtual Center server.

# vifptarget -s <vc-server>

3. Run resxtop.

# resxtop --server <esxi-host>

4. Login using the ESXi account when prompted (Host needs to be out of Lockdown Mode - Hosts are connected using AdAuth) .

5. Look for any increase from 0.00 in %CSTP.

6. Exit resxtop CTRL-C.

You can then decide what to do - add/remove vCPUs.

Description of %CSTP (extract from DOC-11812)
The percentage of time the world spent in ready, co-deschedule state. This co-deschedule state is only meaningful for SMP VMs. Roughly speaking, ESX CPU scheduler deliberately puts a VCPU in this state, if this VCPU advances much farther than other VCPUs. VCPU with high %CSTP is "stopped" from executing so that another VCPU in the same virtual machine could be run to "catch-up".

References:
VSP3866 VMworld 2011 sessions (USA and EMEA)

VMware communities interpreting esxtop 4.1 stats
http://communities.vmware.com/docs/DOC-11812

VMware communities interpreting esxtop stats
http://communities.vmware.com/docs/DOC-9279

Solaris - Third party management tools

So a New Year is upon us and I'd thought I'd better document what tools I use for managing the Solaris environment - if anyone read my ramblings I would be interested to see what you use.

Free Tools:

SSH connectivity - putty
SFTP connections to move files around - coreftp
GUI connectivity - Xming

I've used putty for a number of years on and off and it does the job admirably. I do like the fact it can be setup to record input and output which has helped me out a couple of times.

For file transfer I've started using coreftp as I read somewhere that the FileZilla client stored the SFTP password in clear text..... I hadn't read anything bad about coreftp.

We did have some older Exceed licenses for GUI connectivity and when I got given a Windows 7 build they weren't valid for a supported install - so I went looking for a freebie and stumbled across Xming which connects great but it doesn't like to be logged out so I just click on the X in the top right hand corner to kill the window. Nothing flags in the logs as being a problem afterwards but I'd rather not exit it that way and I will investigate a little more when I have time

HP - Service Pack for ProLiant

After the HP NC522SFP saga one of the nuggets of information I gratefully received was the HP Service Pack for ProLiant. Which was great! Apart from where can it be found on the HP website!!!!!!

I decided to hunt the page down to enable me to find it quickly when I needed to (also regularly check if there are new releases or not).

Home page: http://h18004.www1.hp.com/products/servers/management/spp/index.html

Download page: http://h18004.www1.hp.com/products/servers/service_packs/en/index.html

Extract from the site on benefits of using SPP:

Leverages the Power of HP Smart Update Manager (HP SUM).

Broad portfolio support for HP ProLiant servers, BladeSystem enclosures, and HP CloudSystem

Radically simplified updates with a single step installation process containing both firmware and systems software with drivers packaged together

• Consolidation -- single solution for all supported ProLiant servers
• Interdependency testing of drivers and firmware
• Offered from the web to provide convenient access
• PXE bootable ISO images to reduce customer:
• Qualification cycles
• Resource usage
• Maintenance windows
• Downtime

Integral Part of the HP Server Experience

• Essential management tool designed to simplify IT management in a dynamic and demanding 7x24x365 environment
• Increases Ease of Server Management
  • More efficient enclosure updates because of enhancements to HP SUM
  • Consolidated sets of tested firmware and HP System Software (drivers, agents, utilities)
  • Enhances IT staff productivity while reducing downtime
  • Provides Reliable Configuration by:
  • Assisting with server deployment, integrating server support software and firmware, and minimizing configuration and update issues
  • Ensuring consistency and reliability and maximizing system uptime
  • Simplified delivery of ProLiant software and firmware.

Solaris - Patching Zones with NFS mount points

Further to previous posts (and test I've done) I came across issues where I've patched a couple of servers running Sparse Zones which have NFS mount points. The issues are:

1. Mount points disappear and have to be recreated upon reboot. Symptoms: no mount points and directories disappeared but the entries in vfstab are still there.

2. The worst of the two is where upon reboot (post BE activation) the Zones failed to start and when manually started the 3 errors appeared:
libpool(3LIB) error: System error
dedicated-cpu setting cannot be instantiated
call to zoneadmd failed

After logging a call on MOS they support engineer has linked the problem to a bug:
ID# 15774198 lucreate(1M)and lumake(1M)cannot copy NFS mountpoint non-global zone.

As of today (3rd January2013) I have had no further updates.....

Solaris - Find a process using a specific port

One of the DBAs found/knocked up a useful script which searches for what process is using the entered port number.

Background:

From a NETSTAT the DBAs found that a unknown process was hogging a port number that another application they were installing wanted to use.

Netstat syntax:

# netstat -a | grep <port-number>

Process finding script:

#!/bin/bash

# Get the process which listens on port

# $1 is the port we are looking for

if [ $# -lt 1 ]

then

echo "Please provide a port number parameter for this script"

echo "e.g. $0 22"

exit

fi

echo "Greping for your port, please be patient (CTRL+Z breaks) ... "

for i in `ls /proc`

do

pfiles $i | grep AF_INET | grep $1

if [ $? -eq 0 ]

then

echo Is owned by pid $i

fi

done

Solaris - ZFS disk failure reporting

Our previous servers had Solaris 9 running and when we replaced the hardware we also moved to Solaris 10.

On the old servers they had a SVM script, running regularly via crontab, which monitored disk status. Any failure messages would be emailed to a shared monitored mailbox.

A bit of digging around and it didn't make sense to run the same script on the new servers as we had also switched to ZFS. So we could monitor the status of the ZFS (the disks are in pairs and therefore mirrored) using the zpool status command.

Steps:

1. Create script.
2. Configure mail relay.
3. Schedule script frequency in crontab.

1. Create script

# vi /usr/local/zfscheck


#!/usr/bin/ksh
zpool status -x | grep 'all pools are healthy'
if [ $? -ne 0 ]; then
    date > /var/tmp/zfscheck.log
    echo >> /var/tmp/zfscheck.log
    hostname >> /var/tmp/zfscheck.log
    echo >> /var/tmp/zfscheck.log
    zpool status -xv >> /var/tmp/zfscheck.log
    cat /var/tmp/zfscheck.log | mail -s "Disk failure in server : `hostname`" name@mailaddress
fi


(save and exit)

# chmod +x zfscheck

2. Configure mail relay

Edit the sendmail.cf file with your mail relay information by editing the line:
# "Smart" relay host (may be null)
DS

# vi /etc/mail/sendmail.cf

# "Smart" relay host (may be null)
DSmailrelay.yourdomain.com

(On our Exchange Front Ends we edited the the SMTP node with the server IP address to enable relay.By default Exchange is set to deny relaying).

Restart the sendmail service:
# svcadm restart sendmail

3. Schedule the task to run every 30 minutes

To set an editor for crontab:
# bash
# export EDITOR=vi
# crontab -e

NOTE: If you are in the default shell (bourne) then you have to use the following to be able to edit using vi:
# EDITOR=vi
# export EDITOR

Edit crontab with the following setting then exit and save:

# ZFS pool check
0,30 * * * * /usr/local/zfscheck

Check that the entry has taken:
# crontab -l

I checked that the whole thing worked by setting it all up on a test server and then pulled a drive out. After a few minutes I got an email!



 

Solaris - Searching for growing files

Still being quite new to Solaris and *NIX variants we hit a minor issue when one of the servers reported its disk space being nearly full and thought "How do we find out what's filling the space up?".

Application team wanted to know what files were involved, was it system or application? - so after a bit of hunting around and testing we came up with:

# find / -type f -mount -size +100000000c -mtime -1

(look for files on the local disk that are greater than 100Mb in size that have grown since yesterday)

You'll get an output giving the file path and name of the file which you can then go and check how big the actual file is:

# cd /location
# ls -hal

In this case it was a log (not in /var/adm) which was logging a failed service.

Stopped the service - deleted the log (after have a quick peek to see what was up), checked and fixed the issue then restarted the service.

We found the files causing the issues and took remedial action and disk usage went down from 97% to 78%, the lowest it had been since the server was installed.

On the same theme to find directories that are large then:

# cd /location
# du -dks * | sort -n

From the above output you can see which directories are the largest and go a hunting..... (cd into the one of the offending large directories and repeat the command).