Wednesday, January 30, 2019

vSphere Syslog Slam

Maybe you are like me and you have a new syslog destination (SIEM is grand).  Maybe you want to ensure that all of your vSphere hosts are configured the same.

Automation being king, this will do it.  The general process for each host:
  1. Set the syslog destination 
  2. Insure a firewall exception 
  3. Restart the syslog service.
In vSphere 5.1 the Set-VMHostSysLogServer cmdlet was added to PowerCLI.  While there are other ways to accomplish the same task, I tend to prefer using any method that is more obvious when coding.  Set-VMHostSysLogServer it is.

Cheers!

Wednesday, December 26, 2018

VMware Authorization Service High CPU Usage

When running VMware Workstation on Microsoft Windows, the VMware Authorization Service (vmware-authd.exe) may place a high CPU load on the system.  Upgrading from Workstation 14 to the latest Workstation 15 had no impact, so it's not something that VMware recently fixed.  I remember the issue from many years ago, but not how to resolve the situation.

I'm documenting here so that I don't forget the solution. It's my remind-future-self post.

While I found plenty of Google hits with a similar complaint, I hadn't found any solutions that worked for me.

The vCommunity is great, so a quick tweet was met with a quick reply from buddy Bob Plankers:

Bob's reply sent me to an article on michlG's blog.   The proposed solution did not fit my particular circumstances, but it got me on the right track: looking at Windows performance counters.  

When running PerfMon (I suppose that I should do so more often) I received an error:



My next stop was a Technet article which did the trick:

  1. lodctr /r
  2. lodctr /q to find anything disabled.
  3. lodctr /e:"performance counter" to enable






Tuesday, December 18, 2018

Windows Registry: Loading and Unloading All User Hives

Today my team was discussing the need to search all user registry hives when scanning for malware.

It's not too bad to do this manually if there are a two user profiles on a system, but it gets considerably more tedious on common-use PCs with more user profiles.  And I'm lazy.

The process might also prove handy for other registry-related tasks, where perhaps can't use Group Policy to be lazy.

Queue Load-UserHives.ps1. This quick-and-dirty PowerShell will search the system for available user profiles and load each key.  Later, you can reverse this with the -Unload switch.  Rather than externally tracking what was loaded prior, a simple naming prefix is used for the unload process.



Thursday, January 19, 2017

LUNs with Multiple Exports

We have a vSphere 5.5 environment with your standard multiple paths to shared storage.  In our case, 2 paths to each of 2 nodes in a 7000-series HPE 3PAR array.

While investigating some stability issues we noticed something interesting:


That's one device and the LUN IDs should be the same.  In this configuration, vSphere sees the same device multiple times.

What's more, the vSphere GUI is doing us a "favor" and condensing the output.  Using esxcli we can confirm that there are, in fact, more than 4 paths:

Whoa Nelly!  I've condensed the output some, but that's 12 paths instead of 4.  Each LUN ID represents one set of paths.

The 3PAR GUI confirms:


This particular virtual volume was exported directly to the host twice and once indirectly via a Host Set.  Yeah, don't do that!

Determining the Scope of the Issue

I found a thread on VMware Communities from Markus Kraus, where he saw the same issue, although ultimately from a different cause.  Markus has a blog post too, with a great script and vCheck plugin!

Using Markus' script, I was able to determine all devices with more than one LUN ID.  Now we know how prevalent the issue is and which LUNs are involved on each particular vSphere host.

Unwinding

While we like 3PAR Host Sets, our implementation was incomplete.  Moving toward completion has the potential of causing more problems because exports are also tied to the Host Sets.  The biggest sin here was mixing the use of direct host exports and exports in Host Sets.

With the right attack plan we can work on one vSphere host at a time without any outage:

  1. Place a host into Maintenance Mode.
     
  2. Remove the host from any 3PAR Host Set.
     
  3. Remove host-based duplicate virtual volume exports one at a time.
     
  4. Rescan HBAs.
     
  5. Exit Maintenance Mode.
We are not going to use the 3PAR GUI because it's not granular enough.  Removing a host from a host set would also remove the exports and risk impact on other hosts.  ssh into the 3PAR to access their rather nice CLI.
 
Our Command Set
 

List all Host Sets and their members:
showhostset
Show Virtual Volumes exported to a host:
showvlun -sortcol 1 -host <host> 
Remove the host from the Host Set:
removehostset <setname> <host>
Unexport from a host:
removevlun <vvname> <lunid> <host>
What I found to work best is to first remove the host from any Host Set followed by a rescan of the vSphere host HBA.  When that is completed, unexport any duplicate exports by removing the highest LUN ID(s).  Then rescan HBAs again.

Keep in mind that VVNames and Host Set Names are case sensitive.


Example Time



  1. removehostset My-Host-Set host-9
     
  2. Rescan HBA
     
  3. removevlun NL-THIN-3 34 host-9
     
  4. removevlun NL-THIN-3 33 host-9
     
  5. Rescan HBA
     
  6. showvlun -sortcol 1 -host host-9
     
  7. Verify with esxcli storage core path list --device <device>


Monday, March 14, 2016

Microsoft SQL Server on vSphere Best Practices Guide

Some of us have been running Microsoft SQL Server on top of vSphere for years. Maybe it was only for your pre-VCSA vCenter or maybe you have dozens of instances--but that doesn't stop an official Best Practices Guide from being good reading.

VMware has just updated, re-titled, and published  "Architecting Microsoft SQL Server on VMware vSphere -- Best Practices Guide."

There looks to be some pretty good information within, so get comfortable and grab the PDF here.


Wednesday, December 9, 2015

vSphere Host Logging Levels

If you are configuring vSphere hosts for syslog collection you may be overwhelmed by the amount of data thrown at your collector. By default hostd and vpxa are configured for a logging level of “verbose.”

To view the configuration for all of your vSphere hosts, break out PowerCLI.  I’m forever forgetting how to retrieve information from certain cmdlets and include the host name.  Here it is for me to find again:

 

Setting hostd and vpxa logging levels on all hosts to “info” is pretty easy:

Friday, October 30, 2015

Note to Self: Learn Get-View

Queue Deflation

I’m rather red-faced right now.  I chimed in with a reply in the VMTN VMware vSphere PowerCLI forum and promptly got schooled.

Which is great; these things should happen.

Where’s the Template?

In this thread, VMTN Communities member denjoh44 asked how one would obtain a list of templates along with the name of the vSphere hosts on which they reside.

I haven’t had the need to do this myself, but I wanted to give it a shot. I fired up PowerShell ISE and plugged away:

And it works, which is nice.

I See Your 12 Lines and…

It wasn’t long after that I revisited the thread only to find a much more elegant solution posted by Christophe Calvet:

My personality dictates that I should feel silly about my reply.  Christophe nailed it with a one-liner.  Instead, I’m going to focus on my inexperience with Get-View and do some reading.  I'll endeavor to always keep calculated expressions in mind as well.

Time to Learn

Brian Graf has a 3-part blog series on Get-View which I’ve seen before and marked for reading in more detail “later.”  If you are interested in learning more about Get-View, start here:
While you are at it, Christophe has a blog worth checking out at TheCrazyConsultant.com