Send Active vSphere Alarms to Microsoft Teams

One of the ways that we have been driving adoption of Microsoft Teams at work is by automatically pushing useful content into channels via its webhook support.

I've pushed alarms from our general monitoring system into Teams and it removes the limitation of short text messages while providing more details on an issue before resorting to using the corporate VPN.  I simply arm myself with the Microsoft Teams app on my phone.

I thought about what else might benefit from the ability to see details while on the go and came up with active vSphere alarms.  I wanted to be able to get details on active alarms, the time they started, their status (severity, if you ask me) and the objects and clusters involved.

I started playing around with PowerShell and soon had rudimentary alarm data going into a Microsoft Teams channel.  But what if I am or can be connected to work? 

I added a button to open the vSphere Client and then an alarm-specific button that opens the vSphere Client summary page for the object: 

Example vSphere Alarms Alert

Setting it all up is pretty straightforward: create an incoming webhook connector in Microsoft Teams, save the URL, and schedule the PowerShell to run at a frequency of your choosing.

Adding a Webhook

Open Microsoft Teams, choose or create a new Team, choose or create a new channel within that Team, and then select the three dots to the right of the channel name to create a connector.  The screenshots below will help lead you along, but it's pretty simple.  Be sure to save the link that is generated for your connector.  You can always go back and manage the configured connectors on your channel if you need to get the URL again.

Scheduling

Save the PowerShell at the bottom of this article and save it on a server somewhere so that you can set it up as a Scheduled Task.  Invocation is easy, and you will need to provide your vCenter FQDN and Microsoft Teams webhook URL as part of the program/scripts arguments:

-ExecutionPolicy Bypass -file Send-vSphereAlarms.ps1 -vcenter vcenter-fqdn -TeamsUri uri

I like to configure the task to run every 10 minutes, but pick what works best for you and your environment.  

I'm certain that a lot can be changed in the script. Play around and enjoy!

Send-vSphereAlarms.ps1

vSphere Syslog Slam

Maybe you are like me and you have a new syslog destination (SIEM is grand).  Maybe you want to ensure that all of your vSphere hosts are configured the same.

Automation being king, this will do it.  The general process for each host:

1. Set the syslog destination 
2. Insure a firewall exception 
3. Restart the syslog service.

In vSphere 5.1 the Set-VMHostSysLogServer cmdlet was added to PowerCLI.  While there are other ways to accomplish the same task, I tend to prefer using any method that is more obvious when coding.  Set-VMHostSysLogServer it is.

Cheers!

VMware Authorization Service High CPU Usage

When running VMware Workstation on Microsoft Windows, the VMware Authorization Service (vmware-authd.exe) may place a high CPU load on the system.  Upgrading from Workstation 14 to the latest Workstation 15 had no impact, so it's not something that VMware recently fixed.  I remember the issue from many years ago, but not how to resolve the situation.

I'm documenting here so that I don't forget the solution. It's my remind-future-self post.

While I found plenty of Google hits with a similar complaint, I hadn't found any solutions that worked for me.

The vCommunity is great, so a quick tweet was met with a quick reply from buddy Bob Plankers:

https://t.co/q62OpGpke1

— Bob Plankers (@plankers) November 30, 2018

Bob's reply sent me to an article on michlG's blog.   The proposed solution did not fit my particular circumstances, but it got me on the right track: looking at Windows performance counters.  

When running PerfMon (I suppose that I should do so more often) I received an error:

My next stop was a Technet article which did the trick:

1. lodctr /r
2. lodctr /q to find anything disabled.
3. lodctr /e:"performance counter" to enable

Windows Registry: Loading and Unloading All User Hives

Today my team was discussing the need to search all user registry hives when scanning for malware.

It's not too bad to do this manually if there are a two user profiles on a system, but it gets considerably more tedious on common-use PCs with more user profiles.  And I'm lazy.

The process might also prove handy for other registry-related tasks, where perhaps can't use Group Policy to be lazy.

Queue Load-UserHives.ps1. This quick-and-dirty PowerShell will search the system for available user profiles and load each key.  Later, you can reverse this with the -Unload switch.  Rather than externally tracking what was loaded prior, a simple naming prefix is used for the unload process.

LUNs with Multiple Exports

We have a vSphere 5.5 environment with your standard multiple paths to shared storage.  In our case, 2 paths to each of 2 nodes in a 7000-series HPE 3PAR array.

While investigating some stability issues we noticed something interesting:

That's one device and the LUN IDs should be the same.  In this configuration, vSphere sees the same device multiple times.

What's more, the vSphere GUI is doing us a "favor" and condensing the output.  Using esxcli we can confirm that there are, in fact, more than 4 paths:

Whoa Nelly!  I've condensed the output some, but that's 12 paths instead of 4.  Each LUN ID represents one set of paths.

The 3PAR GUI confirms:

This particular virtual volume was exported directly to the host twice and once indirectly via a Host Set.  Yeah, don't do that!

Determining the Scope of the Issue

I found a thread on VMware Communities from Markus Kraus, where he saw the same issue, although ultimately from a different cause.  Markus has a blog post too, with a great script and vCheck plugin!

Using Markus' script, I was able to determine all devices with more than one LUN ID.  Now we know how prevalent the issue is and which LUNs are involved on each particular vSphere host.

Unwinding

While we like 3PAR Host Sets, our implementation was incomplete.  Moving toward completion has the potential of causing more problems because exports are also tied to the Host Sets.  The biggest sin here was mixing the use of direct host exports and exports in Host Sets.

With the right attack plan we can work on one vSphere host at a time without any outage:

1. Place a host into Maintenance Mode.
    
2. Remove the host from any 3PAR Host Set.
    
3. Remove host-based duplicate virtual volume exports one at a time.
    
4. Rescan HBAs.
    
5. Exit Maintenance Mode.

We are not going to use the 3PAR GUI because it's not granular enough.  Removing a host from a host set would also remove the exports and risk impact on other hosts.  ssh into the 3PAR to access their rather nice CLI.
 

Our Command Set
 

List all Host Sets and their members:

showhostset

Show Virtual Volumes exported to a host:

showvlun -sortcol 1 -host <host> 

Remove the host from the Host Set:

removehostset <setname> <host>

Unexport from a host:

removevlun <vvname> <lunid> <host>

What I found to work best is to first remove the host from any Host Set followed by a rescan of the vSphere host HBA.  When that is completed, unexport any duplicate exports by removing the highest LUN ID(s).  Then rescan HBAs again.

Keep in mind that VVNames and Host Set Names are case sensitive.

Example Time

1. removehostset My-Host-Set host-9
    
2. Rescan HBA
    
3. removevlun NL-THIN-3 34 host-9
    
4. removevlun NL-THIN-3 33 host-9
    
5. Rescan HBA
    
6. showvlun -sortcol 1 -host host-9
    
7. Verify with esxcli storage core path list --device <device>

Microsoft SQL Server on vSphere Best Practices Guide

Some of us have been running Microsoft SQL Server on top of vSphere for years. Maybe it was only for your pre-VCSA vCenter or maybe you have dozens of instances--but that doesn't stop an official Best Practices Guide from being good reading.

VMware has just updated, re-titled, and published  "Architecting Microsoft SQL Server on VMware vSphere -- Best Practices Guide."

There looks to be some pretty good information within, so get comfortable and grab the PDF here.

vSphere Host Logging Levels

If you are configuring vSphere hosts for syslog collection you may be overwhelmed by the amount of data thrown at your collector. By default hostd and vpxa are configured for a logging level of “verbose.”

To view the configuration for all of your vSphere hosts, break out PowerCLI.  I’m forever forgetting how to retrieve information from certain cmdlets and include the host name.  Here it is for me to find again:

 

Setting hostd and vpxa logging levels on all hosts to “info” is pretty easy: