Send Active vSphere Alarms to Microsoft Teams

One of the ways that we have been driving adoption of Microsoft Teams at work is by automatically pushing useful content into channels via its webhook support.

I've pushed alarms from our general monitoring system into Teams and it removes the limitation of short text messages while providing more details on an issue before resorting to using the corporate VPN.  I simply arm myself with the Microsoft Teams app on my phone.

I thought about what else might benefit from the ability to see details while on the go and came up with active vSphere alarms.  I wanted to be able to get details on active alarms, the time they started, their status (severity, if you ask me) and the objects and clusters involved.

I started playing around with PowerShell and soon had rudimentary alarm data going into a Microsoft Teams channel.  But what if I am or can be connected to work? 

I added a button to open the vSphere Client and then an alarm-specific button that opens the vSphere Client summary page for the object: 

Example vSphere Alarms Alert

Setting it all up is pretty straightforward: create an incoming webhook connector in Microsoft Teams, save the URL, and schedule the PowerShell to run at a frequency of your choosing.

Adding a Webhook

Open Microsoft Teams, choose or create a new Team, choose or create a new channel within that Team, and then select the three dots to the right of the channel name to create a connector.  The screenshots below will help lead you along, but it's pretty simple.  Be sure to save the link that is generated for your connector.  You can always go back and manage the configured connectors on your channel if you need to get the URL again.

Scheduling

Save the PowerShell at the bottom of this article and save it on a server somewhere so that you can set it up as a Scheduled Task.  Invocation is easy, and you will need to provide your vCenter FQDN and Microsoft Teams webhook URL as part of the program/scripts arguments:

-ExecutionPolicy Bypass -file Send-vSphereAlarms.ps1 -vcenter vcenter-fqdn -TeamsUri uri

I like to configure the task to run every 10 minutes, but pick what works best for you and your environment.  

I'm certain that a lot can be changed in the script. Play around and enjoy!

Send-vSphereAlarms.ps1

vSphere Syslog Slam

Maybe you are like me and you have a new syslog destination (SIEM is grand).  Maybe you want to ensure that all of your vSphere hosts are configured the same.

Automation being king, this will do it.  The general process for each host:

1. Set the syslog destination 
2. Insure a firewall exception 
3. Restart the syslog service.

In vSphere 5.1 the Set-VMHostSysLogServer cmdlet was added to PowerCLI.  While there are other ways to accomplish the same task, I tend to prefer using any method that is more obvious when coding.  Set-VMHostSysLogServer it is.

Cheers!

VMware Authorization Service High CPU Usage

When running VMware Workstation on Microsoft Windows, the VMware Authorization Service (vmware-authd.exe) may place a high CPU load on the system.  Upgrading from Workstation 14 to the latest Workstation 15 had no impact, so it's not something that VMware recently fixed.  I remember the issue from many years ago, but not how to resolve the situation.

I'm documenting here so that I don't forget the solution. It's my remind-future-self post.

While I found plenty of Google hits with a similar complaint, I hadn't found any solutions that worked for me.

The vCommunity is great, so a quick tweet was met with a quick reply from buddy Bob Plankers:

https://t.co/q62OpGpke1

— Bob Plankers (@plankers) November 30, 2018

Bob's reply sent me to an article on michlG's blog.   The proposed solution did not fit my particular circumstances, but it got me on the right track: looking at Windows performance counters.  

When running PerfMon (I suppose that I should do so more often) I received an error:

My next stop was a Technet article which did the trick:

1. lodctr /r
2. lodctr /q to find anything disabled.
3. lodctr /e:"performance counter" to enable

LUNs with Multiple Exports

We have a vSphere 5.5 environment with your standard multiple paths to shared storage.  In our case, 2 paths to each of 2 nodes in a 7000-series HPE 3PAR array.

While investigating some stability issues we noticed something interesting:

That's one device and the LUN IDs should be the same.  In this configuration, vSphere sees the same device multiple times.

What's more, the vSphere GUI is doing us a "favor" and condensing the output.  Using esxcli we can confirm that there are, in fact, more than 4 paths:

Whoa Nelly!  I've condensed the output some, but that's 12 paths instead of 4.  Each LUN ID represents one set of paths.

The 3PAR GUI confirms:

This particular virtual volume was exported directly to the host twice and once indirectly via a Host Set.  Yeah, don't do that!

Determining the Scope of the Issue

I found a thread on VMware Communities from Markus Kraus, where he saw the same issue, although ultimately from a different cause.  Markus has a blog post too, with a great script and vCheck plugin!

Using Markus' script, I was able to determine all devices with more than one LUN ID.  Now we know how prevalent the issue is and which LUNs are involved on each particular vSphere host.

Unwinding

While we like 3PAR Host Sets, our implementation was incomplete.  Moving toward completion has the potential of causing more problems because exports are also tied to the Host Sets.  The biggest sin here was mixing the use of direct host exports and exports in Host Sets.

With the right attack plan we can work on one vSphere host at a time without any outage:

1. Place a host into Maintenance Mode.
    
2. Remove the host from any 3PAR Host Set.
    
3. Remove host-based duplicate virtual volume exports one at a time.
    
4. Rescan HBAs.
    
5. Exit Maintenance Mode.

We are not going to use the 3PAR GUI because it's not granular enough.  Removing a host from a host set would also remove the exports and risk impact on other hosts.  ssh into the 3PAR to access their rather nice CLI.
 

Our Command Set
 

List all Host Sets and their members:

showhostset

Show Virtual Volumes exported to a host:

showvlun -sortcol 1 -host <host> 

Remove the host from the Host Set:

removehostset <setname> <host>

Unexport from a host:

removevlun <vvname> <lunid> <host>

What I found to work best is to first remove the host from any Host Set followed by a rescan of the vSphere host HBA.  When that is completed, unexport any duplicate exports by removing the highest LUN ID(s).  Then rescan HBAs again.

Keep in mind that VVNames and Host Set Names are case sensitive.

Example Time

1. removehostset My-Host-Set host-9
    
2. Rescan HBA
    
3. removevlun NL-THIN-3 34 host-9
    
4. removevlun NL-THIN-3 33 host-9
    
5. Rescan HBA
    
6. showvlun -sortcol 1 -host host-9
    
7. Verify with esxcli storage core path list --device <device>

vSphere Host Logging Levels

If you are configuring vSphere hosts for syslog collection you may be overwhelmed by the amount of data thrown at your collector. By default hostd and vpxa are configured for a logging level of “verbose.”

To view the configuration for all of your vSphere hosts, break out PowerCLI.  I’m forever forgetting how to retrieve information from certain cmdlets and include the host name.  Here it is for me to find again:

 

Setting hostd and vpxa logging levels on all hosts to “info” is pretty easy:

Note to Self: Learn Get-View

Queue Deflation

I’m rather red-faced right now.  I chimed in with a reply in the VMTN VMware vSphere PowerCLI forum and promptly got schooled.

Which is great; these things should happen.

Where’s the Template?

In this thread, VMTN Communities member denjoh44 asked how one would obtain a list of templates along with the name of the vSphere hosts on which they reside.

I haven’t had the need to do this myself, but I wanted to give it a shot. I fired up PowerShell ISE and plugged away:

And it works, which is nice.

I See Your 12 Lines and…

It wasn’t long after that I revisited the thread only to find a much more elegant solution posted by Christophe Calvet:

My personality dictates that I should feel silly about my reply.  Christophe nailed it with a one-liner.  Instead, I’m going to focus on my inexperience with Get-View and do some reading.  I'll endeavor to always keep calculated expressions in mind as well.

Time to Learn

Brian Graf has a 3-part blog series on Get-View which I’ve seen before and marked for reading in more detail “later.”  If you are interested in learning more about Get-View, start here:

• Get-View Part 1: Introduction
• Get-View Part 2: Views and Extension Data
• Get-View Part 3: Performance Impact

While you are at it, Christophe has a blog worth checking out at TheCrazyConsultant.com

Tech Field Day Recap – Virtualization Field Day 2

I’ll have more on an individual session or two in the coming days, but I wanted to take the time to provide a brief recap of Virtualization Field Day 2.

It was great to see the familiar faces of Edward Haletky, Mike Laverick, Roger Lund,  David Owen, and Rick Schlander while meeting Rodney Haywood, Bill Hill, Dwayne Lessner, Scott Lowe, Robert Novak, Brandon Riley, and Chris Wahl for the first time. It’s solid group of people on both personal and professional levels.

Day zero allowed time for all of the delegates to arrive and closed with a welcome dinner at Zeytoun. We had great food; our hometown gift exchange; a shrine and webcam chat with Stephen who was not able to make it to TFD for the first time, but for the right reason.

For sponsors on day one we had Symantec, who's competing strategy appeared to PowerPoint overload, Zerto with their solid continuous data protection DNA, and Xangati with their monitoring solutions and perfect bacon execution.

Day one ended with a private tour and mystery theatre at Winchester Mystery House.

On tap for day two was flashy Pure Storage, Pivot3 with their integrated storage and compute solution and W. Curtis Preston with TruthinIT for a “battle of the bloggers.”  To be honest, I surprised myself by how much I enjoyed that last session.

There was a lot of good content, but the ones that really appealed to me were Pure Storage, Zerto, Pivot3, and Xangati so I’ll summarize each of those briefly for now.

Pure Storage

Fast, cool, exciting technology in an environment that matches. Wowed by their space from when we walked in the door to their tech in the presentation and to the lab. And the Psycho Donuts seemed to go over well for many.  RBaaS (Red Bull as a Service) was saw a trial run as well.

Zerto

Gil was back for another Tech Field Day and this time with CTO and Co-founder Oded Kedem to talk about the upcoming 2.0 release of their flagship BC/DR product. Zerto visited us at Tech Field Day 8 in Boston and won this year’s vmworld Best of Show.  Each time I’ve seen their tech I have wished that I worked in the segment to which it is marketed.

Pivot3

They came with an appliance, out of the box, configured it during the presentation in 40 minutes, and then used it in a demo. How great is that?  Pivot3’s tech DNA comes from the likes of Adaptec and Compaq and is branching out from the surveillance industry. Having founder and CTO Bill Galloway present showed their commitment to Tech Field Day.

Xangati

Bacon, ice cream, and Star Wars references abound is not a bad way to start with this crowd, and Xangati appears to have one of the best monitoring products for your virtual infrastructure. They’ve been concentrating on VDI lately and their focus was on the VDI Dashboard.

 

We brought it all to a close at the end of day two with great food and conversation at Antonella’s.  Unfortunately Rodney Haywood could not join us; he needed to start his long journey home.

 

Disclaimer
My travel, accommodations, and meals were paid for by the Tech Field Day sponsors. Often there is swag involved as well. Like me, all TFD delegates are independents. We tweet, write, and say whatever we wish at all times, including during the sessions.

Follow the other delegates!

Capturing CPU Trends with PowerCLI

Inspired by the creeping CPU that we see in Linux guests and helped greatly by @BoerLowie at his blog, I’ve come up with a little PowerCLI to capture CPU trends of the top consumers per cluster.
This is my first cut and will likely see changes over time, like any script should. HTML output and emailed results are the most likely candidates.
The script should be fairly self-explanatory. For each cluster, traverse all VMs and get their OverallCpuUsage (the number that you see in the vSphere Client when selecting a cluster and then the Virtual Machines tab).  Take the top X consumers based on that number and get their average CPU usage performance statistic for N days back in time and compare it to today’s.
The output looks something like this:



So here you go:

#

#  Produce guest CPU trending from a time period back versus a shorter 

#  more immediate time frame.  e.g. 30 days ago versus past 2 days.

#

param(

    [string] $vCenter

)

$DaysOld = -30        # compare to full day stats this many days back

$DaysRecent = -1    # get stats for this many recent days.

$GetTop = 10        # look at top x CPU consumers

Add-PSSnapin VMware.VimAutomation.Core -ErrorAction SilentlyContinue

#if ($vCenter -eq "") {

#    $vCenter = Read-Host "VI Server: "

#}

#if ($DefaultVIServers.Count) {

#    Disconnect-VIServer -Server * -Force -Confirm:$false

#}

#Connect-VIServer $vCenter

$AllClusters = Get-Cluster

Foreach ($Cluster in $AllClusters) {

    Write-Host "`n$($Cluster.Name)"

    $VMs = Get-Cluster $Cluster | Get-VM | `

        Where-Object { $_.PowerState -eq "PoweredOn" }

    $NumVMs = $VMs.Count

    # Get the Overall CPU Usage for each VM in the cluster.  Then cap that 

    # list at the top $GetTop highest for Overall CPU Usage

    $vm_list = @()

    $Count = 0

    Foreach ($vm in $VMs)

    {

        $Count += 1

        Write-Progress -Activity "Getting VM views" -Status "Progress:" `

            -PercentComplete ($Count / $NumVMs * 100)

        # the vSphere .Net view object has the OverallCpuUsage 

        # (VirtualMachineQuickStats)

        # http://www.vmware.com/support/developer/vc-sdk/visdk400pubs/ReferenceGuide/vim.vm.Summary.QuickStats.html

        $view = Get-View $vm

        $objOutput = "" | Select-Object VMName, CpuMhz

        $objOutput.VMName = $view.Name

        $objOutput.CpuMhz = $view.Summary.QuickStats.OverallCpuUsage

        $vm_list += $objOutput

    }

    # Reduce to our Top X

    $vm_list = $vm_list | sort-object CpuMhz -Descending | select -First $GetTop 

    #

    # For each of those VMs, get the statistics for past and current CPU usage

    $NumVMs = $vm_list.Count

    $Out_List = @()

    $Count = 0

    Foreach ($vm in $vm_list)

    {

        $Count += 1

        Write-Progress -Activity "Compiling CPU stats" -Status "Progress:" `

            -PercentComplete ($Count / $NumVMs * 100)

           [Double] $ldblPerfAged = (Get-Stat -Entity $vm.VMName -Stat cpu.usage.average `

            -Start $((Get-Date).AddDays($DaysOld)) `

            -Finish $((Get-Date).AddDays($DaysOld + 1)) -ErrorAction Continue | `

            Measure-Object -Average Value).Average

        If ($ldblPerfAged -gt 0) {

               [Double] $lblPerfNow = (Get-Stat -Entity $vm.VMName -Stat cpu.usage.average `

                -Start $((Get-Date).AddDays($DaysRecent)) `

                -ErrorAction Continue | Measure-Object -Average Value).Average

            [Int] $lintTrend = (($lblPerfNow - $ldblPerfAged) / $ldblPerfAged) * 100

            $objOutput = "" | Select-Object VMName, CpuMhz, PerfAged, PerfNow, Trend

            $objOutput.VMName = $vm.VMName

            $objOutput.CpuMhz = $vm.CpuMhz

            $objOutput.PerfAged = "{0:f2}%" -f $ldblPerfAged

            $objOutput.PerfNow = "{0:f2}%" -f $lblPerfNow

            $objOutput.Trend = "{0}%" -f $lintTrend

            $out_list += $objOutput

        }

    }

    # Spit 'er out

    Write-Host "Top CPU Consumers Trending, $($DaysOld) days vs today`n"

    $out_list | Format-Table -Property VMName, `

        @{Expression={$_.CpuMhz};Name='CPU Mhz';align='right'}, `

        @{Expression={$_.PerfAged};Name='CPU Aged';align='right'}, `

        @{Expression={$_.PerfNow};Name='CPU Now';align='right'}, `

        @{Expression={$_.Trend};Name='Trend';align='right'}

}

Linux Guest CPU Creep

We run a lot of tiny VMs on vSphere 4 in a rather unique environment.  The densities are high and the kernel OS is officially unsupported Fedora Core 8 (2.6.26 kernel). This causes us to be more tolerant of aberrations.

The biggest aberration of note has been CPU creep.  The tiny guests will run along just fine using 30 - 40 MHz of CPU and then start a slow upward trend.  It will creep slowly over the course of a week.  No useful perspective can be gained from within the guest using traditional means.  More interesting, performing a guest-initiated reboot will reveal a slow crawl all the way through the BIOS at boot and no CPU dip beyond the new baseline.  They are stuck, and a reset from the vSphere client resolves the issue.

This has been acceptable so far.  The guests are stateless, only a few are impacted at any one time, and no one guest is critical by itself.  We automated the remediation, became accustomed, and moved on.  The issue has stuck to one functional cluster and persisted across minor vSphere 4 upgrades.

Becoming accustomed caused us to miss another occurrence.

The software architects have been busy troubleshooting the core application running in a separate vSphere cluster on Ubuntu Server 8.04 LTS (2.6.24 kernel).  CPU has been creeping slowly up for the past couple of months with a marked recent acceleration. We’ve been attributing it to increased load as we grow.  The software was optimized and the CPU remained steady and on its upward path.

The solution:

Stop all running processes, verify a higher than expected CPU load, and reset the VM.  We’re down substantially.

In a small shop with few resources and too many projects, it’s time to implement trending alerts.

Have you experienced this behavior before?