Send Active vSphere Alarms to Microsoft Teams

One of the ways that we have been driving adoption of Microsoft Teams at work is by automatically pushing useful content into channels via its webhook support.

I've pushed alarms from our general monitoring system into Teams and it removes the limitation of short text messages while providing more details on an issue before resorting to using the corporate VPN.  I simply arm myself with the Microsoft Teams app on my phone.

I thought about what else might benefit from the ability to see details while on the go and came up with active vSphere alarms.  I wanted to be able to get details on active alarms, the time they started, their status (severity, if you ask me) and the objects and clusters involved.

I started playing around with PowerShell and soon had rudimentary alarm data going into a Microsoft Teams channel.  But what if I am or can be connected to work? 

I added a button to open the vSphere Client and then an alarm-specific button that opens the vSphere Client summary page for the object: 

Example vSphere Alarms Alert

Setting it all up is pretty straightforward: create an incoming webhook connector in Microsoft Teams, save the URL, and schedule the PowerShell to run at a frequency of your choosing.

Adding a Webhook

Open Microsoft Teams, choose or create a new Team, choose or create a new channel within that Team, and then select the three dots to the right of the channel name to create a connector.  The screenshots below will help lead you along, but it's pretty simple.  Be sure to save the link that is generated for your connector.  You can always go back and manage the configured connectors on your channel if you need to get the URL again.

Scheduling

Save the PowerShell at the bottom of this article and save it on a server somewhere so that you can set it up as a Scheduled Task.  Invocation is easy, and you will need to provide your vCenter FQDN and Microsoft Teams webhook URL as part of the program/scripts arguments:

-ExecutionPolicy Bypass -file Send-vSphereAlarms.ps1 -vcenter vcenter-fqdn -TeamsUri uri

I like to configure the task to run every 10 minutes, but pick what works best for you and your environment.  

I'm certain that a lot can be changed in the script. Play around and enjoy!

Send-vSphereAlarms.ps1

vSphere Syslog Slam

Maybe you are like me and you have a new syslog destination (SIEM is grand).  Maybe you want to ensure that all of your vSphere hosts are configured the same.

Automation being king, this will do it.  The general process for each host:

1. Set the syslog destination 
2. Insure a firewall exception 
3. Restart the syslog service.

In vSphere 5.1 the Set-VMHostSysLogServer cmdlet was added to PowerCLI.  While there are other ways to accomplish the same task, I tend to prefer using any method that is more obvious when coding.  Set-VMHostSysLogServer it is.

Cheers!

VMware Authorization Service High CPU Usage

When running VMware Workstation on Microsoft Windows, the VMware Authorization Service (vmware-authd.exe) may place a high CPU load on the system.  Upgrading from Workstation 14 to the latest Workstation 15 had no impact, so it's not something that VMware recently fixed.  I remember the issue from many years ago, but not how to resolve the situation.

I'm documenting here so that I don't forget the solution. It's my remind-future-self post.

While I found plenty of Google hits with a similar complaint, I hadn't found any solutions that worked for me.

The vCommunity is great, so a quick tweet was met with a quick reply from buddy Bob Plankers:

https://t.co/q62OpGpke1

— Bob Plankers (@plankers) November 30, 2018

Bob's reply sent me to an article on michlG's blog.   The proposed solution did not fit my particular circumstances, but it got me on the right track: looking at Windows performance counters.  

When running PerfMon (I suppose that I should do so more often) I received an error:

My next stop was a Technet article which did the trick:

1. lodctr /r
2. lodctr /q to find anything disabled.
3. lodctr /e:"performance counter" to enable

Windows Registry: Loading and Unloading All User Hives

Today my team was discussing the need to search all user registry hives when scanning for malware.

It's not too bad to do this manually if there are a two user profiles on a system, but it gets considerably more tedious on common-use PCs with more user profiles.  And I'm lazy.

The process might also prove handy for other registry-related tasks, where perhaps can't use Group Policy to be lazy.

Queue Load-UserHives.ps1. This quick-and-dirty PowerShell will search the system for available user profiles and load each key.  Later, you can reverse this with the -Unload switch.  Rather than externally tracking what was loaded prior, a simple naming prefix is used for the unload process.

LUNs with Multiple Exports

We have a vSphere 5.5 environment with your standard multiple paths to shared storage.  In our case, 2 paths to each of 2 nodes in a 7000-series HPE 3PAR array.

While investigating some stability issues we noticed something interesting:

That's one device and the LUN IDs should be the same.  In this configuration, vSphere sees the same device multiple times.

What's more, the vSphere GUI is doing us a "favor" and condensing the output.  Using esxcli we can confirm that there are, in fact, more than 4 paths:

Whoa Nelly!  I've condensed the output some, but that's 12 paths instead of 4.  Each LUN ID represents one set of paths.

The 3PAR GUI confirms:

This particular virtual volume was exported directly to the host twice and once indirectly via a Host Set.  Yeah, don't do that!

Determining the Scope of the Issue

I found a thread on VMware Communities from Markus Kraus, where he saw the same issue, although ultimately from a different cause.  Markus has a blog post too, with a great script and vCheck plugin!

Using Markus' script, I was able to determine all devices with more than one LUN ID.  Now we know how prevalent the issue is and which LUNs are involved on each particular vSphere host.

Unwinding

While we like 3PAR Host Sets, our implementation was incomplete.  Moving toward completion has the potential of causing more problems because exports are also tied to the Host Sets.  The biggest sin here was mixing the use of direct host exports and exports in Host Sets.

With the right attack plan we can work on one vSphere host at a time without any outage:

1. Place a host into Maintenance Mode.
    
2. Remove the host from any 3PAR Host Set.
    
3. Remove host-based duplicate virtual volume exports one at a time.
    
4. Rescan HBAs.
    
5. Exit Maintenance Mode.

We are not going to use the 3PAR GUI because it's not granular enough.  Removing a host from a host set would also remove the exports and risk impact on other hosts.  ssh into the 3PAR to access their rather nice CLI.
 

Our Command Set
 

List all Host Sets and their members:

showhostset

Show Virtual Volumes exported to a host:

showvlun -sortcol 1 -host <host> 

Remove the host from the Host Set:

removehostset <setname> <host>

Unexport from a host:

removevlun <vvname> <lunid> <host>

What I found to work best is to first remove the host from any Host Set followed by a rescan of the vSphere host HBA.  When that is completed, unexport any duplicate exports by removing the highest LUN ID(s).  Then rescan HBAs again.

Keep in mind that VVNames and Host Set Names are case sensitive.

Example Time

1. removehostset My-Host-Set host-9
    
2. Rescan HBA
    
3. removevlun NL-THIN-3 34 host-9
    
4. removevlun NL-THIN-3 33 host-9
    
5. Rescan HBA
    
6. showvlun -sortcol 1 -host host-9
    
7. Verify with esxcli storage core path list --device <device>

Microsoft SQL Server on vSphere Best Practices Guide

Some of us have been running Microsoft SQL Server on top of vSphere for years. Maybe it was only for your pre-VCSA vCenter or maybe you have dozens of instances--but that doesn't stop an official Best Practices Guide from being good reading.

VMware has just updated, re-titled, and published  "Architecting Microsoft SQL Server on VMware vSphere -- Best Practices Guide."

There looks to be some pretty good information within, so get comfortable and grab the PDF here.

vSphere Host Logging Levels

If you are configuring vSphere hosts for syslog collection you may be overwhelmed by the amount of data thrown at your collector. By default hostd and vpxa are configured for a logging level of “verbose.”

To view the configuration for all of your vSphere hosts, break out PowerCLI.  I’m forever forgetting how to retrieve information from certain cmdlets and include the host name.  Here it is for me to find again:

 

Setting hostd and vpxa logging levels on all hosts to “info” is pretty easy:

Note to Self: Learn Get-View

Queue Deflation

I’m rather red-faced right now.  I chimed in with a reply in the VMTN VMware vSphere PowerCLI forum and promptly got schooled.

Which is great; these things should happen.

Where’s the Template?

In this thread, VMTN Communities member denjoh44 asked how one would obtain a list of templates along with the name of the vSphere hosts on which they reside.

I haven’t had the need to do this myself, but I wanted to give it a shot. I fired up PowerShell ISE and plugged away:

And it works, which is nice.

I See Your 12 Lines and…

It wasn’t long after that I revisited the thread only to find a much more elegant solution posted by Christophe Calvet:

My personality dictates that I should feel silly about my reply.  Christophe nailed it with a one-liner.  Instead, I’m going to focus on my inexperience with Get-View and do some reading.  I'll endeavor to always keep calculated expressions in mind as well.

Time to Learn

Brian Graf has a 3-part blog series on Get-View which I’ve seen before and marked for reading in more detail “later.”  If you are interested in learning more about Get-View, start here:

• Get-View Part 1: Introduction
• Get-View Part 2: Views and Extension Data
• Get-View Part 3: Performance Impact

While you are at it, Christophe has a blog worth checking out at TheCrazyConsultant.com

vSphere 5.5 Update 3 Patch Available

The Best Intentions

It was obvious and natural; the Right Thing to do.  In preparation for a deployment across all vSphere hosts we took the step of insuring the latest updates. And since we are running vSphere 5.5, this meant Update 3.

There was even some chest-thumping among our small team of generalists as they completed the task within short order.  And why not?

That was Short-Lived

Within 2 weeks we were crestfallen. Update 3 introduced a service-impacting defect involving seldom-used (sic) snapshots.  KB 2133118 was released to announce "unexpected signal: 11."  Consolidate or delete a snapshot and you may suffer a VM outage.

To be honest, I find the KB lacking in details--there are no specifics to indicate contributing factors or degree of impact across customers.

A New Hope?

While we haven't been impacted (whew!), I'm happy to see that today brings the release of patch ESXi550-201510401-BG to address this issue.

Fire up Update Manager and start patching!  Let us know how it goes, because I'm not sure that we wish to be first on this one.


2

Snazzy-Up Your PowerShell with a GUI

I’d like to say that I work with virtualization every day, and in a way I do.  Virtually every server in my employer’s environment is virtualized (see what I did there?).

That we are a small team in an SME environment dictates that we are I.T. generalists.  That’s not a curse—that’s variety and opportunity alleviating the mundane.  It’s also an opportunity to automate the mundane. 

For the purposes of user provisioning I wanted to use PowerShell for identity management on a shoestring budget.   As with many things, creating is harder than destroying; this provisioning needs a GUI.  Pick an OU, pick a user/template, and copy with new details.  While that’s how I got to the point of driving a GUI with PowerShell, it’s difficult to effectively convey a company-specific workflow.  So I’ll demonstrate with something virtualization-focused.

With a little bit of Googling I found an older option or two along with .NET-based Windows Presentation Foundation (WPF).  .NET this plus .NET that sounded appealing with me, so I dug in further. 

There are solid posts to be found that give you all that you need.  I’m working against some of my musings, so here I share my journey toward the same end result.

At a higher level the process is relatively simple:

• Generate a GUI form in Extensible Application Markup Language (XAML).
• Load the XAML in PowerShell with some minor (automated) transformations.
• Instantiate the WPF form.
• Create variables for working with our GUI elements.
• Define PowerShell event handlers for each GUI element, incorporating our workflow within.

Generating Your GUI’s XAML

To generate the GUI we need something that will generate XAML that WPF can handle.  Anything from Visual Studio 2010 on up will do here.  I used Visual Studio 2015 Community since it is full featured with an investment of $0.

Open Visual Studio and create a new project.  Choose WPF Application under Visual C# Templates and give it a name.  Any name will do—we only want the XAML which this project will generate.

What you will notice is a blank window in the main center pane, labeled MainWindow.xaml.  The pane below contains the XAML in which we are interested. 

I find it best to turn on the toolbox from the View menu and pin it to the side bar this yields an environment tailored for creating window forms.

Let’s use the toolbox to drag and position a few controls:

• A ComboBox selector for Cluster
• A DataGrid to display VMs within a selected Cluster.
• A Button for launching a selected VM’s console.
• A Label or three for clarity.

While we can add these controls in any order, Visual Studio will set the form’s tab order based on the order of definition.  That’s not a particularly big deal because of a neat Visual Studio feature:  the GUI form is tied to the XAML and the XAML is tied to the GUI form. We can adjust the tab order by re-ordering the text inside the XAML pane copy and paste!

With each control use the properties pane and give it a unique name, which we will  use later to bind to PowerShell variables.  I like to start all label control names with “label_” for easier exclusion from this binding process.

Loading the XAML

All we need from Visual Studio is the XAML, which we will copy and paste into a PowerShell here-string.  The string isn’t perfect for PowerShell processing yet, so we’ll also rework it to take care of these issues and cast as XML:

Instantiating the GUI Form

Next we need to load the PresentationFramework assembly and process our form so that it can be presented:

Creating GUI Variables

We need to be able to work with the controls on form that comprise our GUI, and that means variables that let us get/set values and work with other properties and methods afforded to us by WPF.   

We can do this rather handily in PowerShell by invoking Set-Variable for nodes within the XML that comprises our form.  The iteration of those nodes will return Label controls as well, which we’ll ignore based on giving them a prefix of “Label_”, which was mentioned earlier:

Defining Event Handlers

WPF will send asynchronous signals (events) to our script when the GUI is acted upon.  Examples include when a selection is made, a button is clicked, and when the window is instantiated or destroyed. 

Defining event handler functions in our PowerShell lets us dictate our workflow:

• When the window is created, connect to vCenter, get a list of clusters, and populate the cluster ComboBox.
• When a selection is made in the cluster ComboBox, get a list of cluster VMs and populate the details DataGrid.
• When a VM selection is made in the DataGrid, enable the console Button.
• When the console Button is clicked, launch a vSphere Remote Console window.
• When the window is closed, disconnect from vCenter

Anything is possible, of course.  Here’s our event handlers:

Final Bits

There’s not much left!  Define the vCenter server against which we will operate and display the form:

Reveal

Run this little puppy and off we go.  Pick a cluster from the drop-down to see a list of VMs and click the column headers to if you don’t like it ordered by VM name.  Choose a VM and the console button becomes active so long as the VM state is “PoweredOn.”

What else?

• My error checking and validations are non-existent--Add some.
• Add another button to perform another task.
• Change what VM properties appear in the DataGrid.
• Do something else and have fun with it!

Full Code

You can grab the full code from here.

Proper Attribution is Proper

• Stephen Owen’s XAML to GUI
• The Scripting Guys’ I've Got a PowerShell Secret: Adding a GUI to Scripts

Musings of an Infrequent Blogger

Writing is hard.
That's not the case for everyone, of course, but it holds true for me.  If you know me on a personal or professional level you've heard me say this before.  The odd thing is, I enjoy writing.  And because of that I tend to focus on this topic.

I have reasons not to write.

It's a silly idea/topic/thought

This is a powerful one for me and stops me the most often. It's also the quickest way out.  I question myself by nature, so there's no surprise here.  I'm still not certain that I'll hit publish on this post!

Suck it up and write anyway.

It's never quite right

One of the things that I enjoy most in the writings of others is the multitude of ways to say the same thing. The simplicity of some, the eloquence of others.  Let's not even talk about grammar, misspellings, etc.  Whether I'm writing here or within the corporate walls, I can tell you that I've reworked virtually every paragraph multiple times.

Practice makes perfect?  No, but it certainly improves.

Others have already written about this

Why bother covering a topic that so many others already have?  Fear of getting lost among similar technical articles and not managing to stand out.

Not to be glib, but meh--one more can't hurt.

Time

How do others find the time?  I'm up early and put in long days. There is often weekend work. Around it all I must find time for family. There needs to be wind-down time and sleep in there somewhere.

Others do it. There are ways.  Give up a little time that's normally spent by myself anyway.

---

Why have I shared this?  Certainly it is therapeutic to a degree and I can refer to it in the future to push me forward.
But that's not my hope.

I want to reach someone else and push them forward. I'm an avid reader of blogs, sometimes churning through dozens a day. I make it part of my daily routine. I thoroughly enjoy reading what others have to say.

If you are like me, I'd like you to write--or write more.  

Because I'd like to read what you have to say.

Honored

This year, after years of deliberation and encouragement from friends, I finally submitted a vExpert application.

I am honored and humbled to have been awarded vExpert for 2015 by VMware.  To be fair, I'm also rather ecstatic!

See the full list of stellar individuals at the official vExpert 2015 Announcement.

vSphere Claim Rules with PowerCLI

We’ve been implementing HP 3PAR 7000 series into our environment. During this task we’ve tryied to be good admins by following the HP 3PAR StoreServ Storage and VMware vSphere 5 best practices.

Within this excellent resource there is a section for automating round robin policy for all 3PAR LUNs with a custom SATP rule.  That’s great, but it involves an esxcli command that must be run on each host:


For a small number of hosts that’s not too terrible--until you realize that you must either reboot each host or modify the policy for each LUN currently presented.  That’s not something that I’m likely to do.

With my aid of my technology buddy Google, I discovered several tidbits online:

• Philip Sellers has a blog that works through much of this in PowerCLI.
• Cormac Hogan has a blog that re-iterates the above, along with the commands required to unload each already claimed device (LUN), reload the claim rules, and rescan the HBAs.

Most of what I needed was out there already, of course. But I wanted to reload the claim rules in order to test the custom rule and without setting the policy individually on each LUN. I also didn’t want to impact every iSCSI LUN—just 3PAR—and there’s no need to reclaim a LUN that is already configured for round robin.

Here’s how I did it for each host:


Of course, I wrapped all of this around more PowerCLI that obtained a list of hosts in each cluster and managed the connections for esxcli.

If you receive any errors during the unclaim, make sure that your HA configuration is not using those LUNs as heartbeat datastores.  I experienced that problem which was easy to remedy in our environment.

Office 365 Staged Migrations with PowerShell

I have a love-hate relationship with PowerShell.  I want to love it and often do--except when my knowledge is lacking or it acts mysteriously (to me).
To aid my memory and perhaps help others, I've decided to post snippets as I continue my journey toward improving my PowerShell skills (I hear that shaking Luc Dekens' and Alan Renouf's hand at the same time increases one's PS Mojo).
With that out of the way, onward and upward.

---

We are in the midst of migrating from Exchange Server 2003 to Office 365 as a Staged Migration.  While not a particularly arduous task, it does involve pulling mailbox statistics and email addresses and using that information for the planning and creation of migration batches. It's tedious and error-prone.
Given a set of Exchange Servers, Domain Controllers, and a Universal Security Group, Get-MailboxSizes.ps1 generates a CSV containing mailbox names, server, size, number of items, office location, department, active mailbox flag, and string suitable for directly pasting into a Staged Migration CSV file.


The ActiveMbx column is TRUE if the user is not enabled or a member of a Universal Security Group used for tracking progress and automating other things.

The odd column header at the last column should be directly pasted into your Staged Migration CSV file as row 1.  For successive entries, copy that column value in filtered rows as you plan the migration.   Please note that this column assumes Active Directory Synchronization with Office 365.

We've been known to paste entries from the last column into Notepad for a quick Search-Replace of ",,FALSE" with ";".  That yields a list that you can paste into the To field of a notification email or add members box of the Universal Security Group.

I’d like to thank Iain Brighton who lent a hand when I was stuck with Custom Objects. Yea Twitter!

Traditional DR…and its Imminent Demise?

 

My primary focus at VMworld 2012 was Disaster Recovery, which caused me to think a fair amount about the future of DR in general—it’s necessity, utility, and longevity.  Have we really escaped “traditional” DR?  Will the methods employed today exist as we know them in 10 years, or just be another integral part of the infrastructure?

Each session invariably started off comparing the “traditional” disaster recovery of yesterday against the virtualization-enabled DR of today, where the old machinations are replaced with flipping a software switch.

With the exception of American National Bank’s and Varrow’s Active/Active datacenter (INF-BCO1883), I can’t help but see this as still being traditional DR—only with today’s tools.

Let’s take a look at some of the main points in “traditional” DR versus today’s:

• Before virtualization, restoring to the same hardware used in production was a challenge. If it could not be met, time was wasted.  Virtualization gives us a common hardware set, eliminating those hardware compatibility woes. 
  
  While a valid point, what if you could always purchase bland hardware, generic x86 servers at Wal-Mart as a commodity, like that which virtualization presents to the OS?
• Tapes could not always be restored and took too much time. Virtualization gets us to replication technologies that avoid tape.
  
  Excluding array-based replication, disk-to-disk-to-tape solutions with replication were pitched as disaster recovery aids 10 years ago, specifically to get around the problems of tape.
• New systems/applications required new servers.
  
  You got me there. Virtualization wins, and I’m happy with that.

Same process, new tools—albeit faster, better, stronger tools. It’s still traditional DR to me.

Now enter Cloud-based DR.  With DR in the cloud, there is no need to lease or require that which remains idle. In essence, keep an off-site copy of your data—a Good Thing anyway—and pay for what you need when you need.  Disaster Recovery has now moved fully from capex to opex.  It’s cloud being used for what cloud is intended. 

But not at the application level.

In an application-centric world, everything behaves like the modern applications to which we’ve become accustomed.  We are blissfully unaware, yet fully appreciative of Facebook, Google, Twitter, and the like spanning multiple datacenters.  We aren’t exposed to datacenter failures that they may encounter and we shouldn’t be. Nor should your customers. 

Your line-of-business systems need to be heading this way, today, for it is the key to availability across datacenters and devices (EUC was a big push at VMworld this year).  They shouldn’t care any more about what datacenter they occupy than how many instances are deployed.

The pieces are there. We’re seeing the increased popularity of orchestration with the likes of Chef and Puppet (in no particular order). Infrastructure manipulation via APIs such as Amazon provides into their Elastic Load Balancer.  Data—big or otherwise—replication, and sharding is becoming commonplace.

The hold-outs are back office systems that won’t get where we need them to be soon enough, yet demonstrate significant movement in this direction when you consider Office 365 and the like.

Once achieved, is expanding from private to public cloud based on increased load any different than contracting from one to the other based on availability?

Private, public, or hybrid the cloud is an extension of your datacenter. It’s the elasticity of your workloads at web-scale, that need not be within one datacenter.  If well orchestrated, you have “simple” contractions of your cloud based on not only load, but availability.

I see the agile system encompassing multiple datacenters at any point in time, expanding and contracting as load and availability changes. This, will be the new DR—no DR.  Just a well-designed modern system.

What are your thoughts?

Takeaway: Developers need to be aware of infrastructure; this could be interesting.

Tech Field Day Recap – Virtualization Field Day 2

I’ll have more on an individual session or two in the coming days, but I wanted to take the time to provide a brief recap of Virtualization Field Day 2.

It was great to see the familiar faces of Edward Haletky, Mike Laverick, Roger Lund,  David Owen, and Rick Schlander while meeting Rodney Haywood, Bill Hill, Dwayne Lessner, Scott Lowe, Robert Novak, Brandon Riley, and Chris Wahl for the first time. It’s solid group of people on both personal and professional levels.

Day zero allowed time for all of the delegates to arrive and closed with a welcome dinner at Zeytoun. We had great food; our hometown gift exchange; a shrine and webcam chat with Stephen who was not able to make it to TFD for the first time, but for the right reason.

For sponsors on day one we had Symantec, who's competing strategy appeared to PowerPoint overload, Zerto with their solid continuous data protection DNA, and Xangati with their monitoring solutions and perfect bacon execution.

Day one ended with a private tour and mystery theatre at Winchester Mystery House.

On tap for day two was flashy Pure Storage, Pivot3 with their integrated storage and compute solution and W. Curtis Preston with TruthinIT for a “battle of the bloggers.”  To be honest, I surprised myself by how much I enjoyed that last session.

There was a lot of good content, but the ones that really appealed to me were Pure Storage, Zerto, Pivot3, and Xangati so I’ll summarize each of those briefly for now.

Pure Storage

Fast, cool, exciting technology in an environment that matches. Wowed by their space from when we walked in the door to their tech in the presentation and to the lab. And the Psycho Donuts seemed to go over well for many.  RBaaS (Red Bull as a Service) was saw a trial run as well.

Zerto

Gil was back for another Tech Field Day and this time with CTO and Co-founder Oded Kedem to talk about the upcoming 2.0 release of their flagship BC/DR product. Zerto visited us at Tech Field Day 8 in Boston and won this year’s vmworld Best of Show.  Each time I’ve seen their tech I have wished that I worked in the segment to which it is marketed.

Pivot3

They came with an appliance, out of the box, configured it during the presentation in 40 minutes, and then used it in a demo. How great is that?  Pivot3’s tech DNA comes from the likes of Adaptec and Compaq and is branching out from the surveillance industry. Having founder and CTO Bill Galloway present showed their commitment to Tech Field Day.

Xangati

Bacon, ice cream, and Star Wars references abound is not a bad way to start with this crowd, and Xangati appears to have one of the best monitoring products for your virtual infrastructure. They’ve been concentrating on VDI lately and their focus was on the VDI Dashboard.

 

We brought it all to a close at the end of day two with great food and conversation at Antonella’s.  Unfortunately Rodney Haywood could not join us; he needed to start his long journey home.

 

Disclaimer
My travel, accommodations, and meals were paid for by the Tech Field Day sponsors. Often there is swag involved as well. Like me, all TFD delegates are independents. We tweet, write, and say whatever we wish at all times, including during the sessions.

Follow the other delegates!

Virtualization Field Day Two

I am honored to have been invited back for another virtualization-themed Tech Field Day. For those unfamiliar, Tech Field Day is a great opportunity for vendors to get together on a technical level with independents that represent their target market.  It's a great short feedback loop that generates much excitement on both sides.


This second virtualization-themed event marks a big step for GestaltIT and Tech Field Day as Matt Simmons will be the primary "show runner", paving the way for expanded Tech Field Day events.  As a side note, this frees Stephen Foskett to be with his son as he is presented the award for winning the Ohio Civil Rights MLK Essay Contest. Go Grant!


This time around Symantec and Zerto return and are joined by Xangati, PureStorage, and Pivot3 and a blogger event with truthinIT.


Join us, February 23 and 24th and be sure to follow us on Twitter with hashtag #VFD2.


I look forward to seeing many familiar faces among the delegates and the opportunity to meet some new ones.  

Capturing CPU Trends with PowerCLI

Inspired by the creeping CPU that we see in Linux guests and helped greatly by @BoerLowie at his blog, I’ve come up with a little PowerCLI to capture CPU trends of the top consumers per cluster.
This is my first cut and will likely see changes over time, like any script should. HTML output and emailed results are the most likely candidates.
The script should be fairly self-explanatory. For each cluster, traverse all VMs and get their OverallCpuUsage (the number that you see in the vSphere Client when selecting a cluster and then the Virtual Machines tab).  Take the top X consumers based on that number and get their average CPU usage performance statistic for N days back in time and compare it to today’s.
The output looks something like this:



So here you go:

#

#  Produce guest CPU trending from a time period back versus a shorter 

#  more immediate time frame.  e.g. 30 days ago versus past 2 days.

#

param(

    [string] $vCenter

)

$DaysOld = -30        # compare to full day stats this many days back

$DaysRecent = -1    # get stats for this many recent days.

$GetTop = 10        # look at top x CPU consumers

Add-PSSnapin VMware.VimAutomation.Core -ErrorAction SilentlyContinue

#if ($vCenter -eq "") {

#    $vCenter = Read-Host "VI Server: "

#}

#if ($DefaultVIServers.Count) {

#    Disconnect-VIServer -Server * -Force -Confirm:$false

#}

#Connect-VIServer $vCenter

$AllClusters = Get-Cluster

Foreach ($Cluster in $AllClusters) {

    Write-Host "`n$($Cluster.Name)"

    $VMs = Get-Cluster $Cluster | Get-VM | `

        Where-Object { $_.PowerState -eq "PoweredOn" }

    $NumVMs = $VMs.Count

    # Get the Overall CPU Usage for each VM in the cluster.  Then cap that 

    # list at the top $GetTop highest for Overall CPU Usage

    $vm_list = @()

    $Count = 0

    Foreach ($vm in $VMs)

    {

        $Count += 1

        Write-Progress -Activity "Getting VM views" -Status "Progress:" `

            -PercentComplete ($Count / $NumVMs * 100)

        # the vSphere .Net view object has the OverallCpuUsage 

        # (VirtualMachineQuickStats)

        # http://www.vmware.com/support/developer/vc-sdk/visdk400pubs/ReferenceGuide/vim.vm.Summary.QuickStats.html

        $view = Get-View $vm

        $objOutput = "" | Select-Object VMName, CpuMhz

        $objOutput.VMName = $view.Name

        $objOutput.CpuMhz = $view.Summary.QuickStats.OverallCpuUsage

        $vm_list += $objOutput

    }

    # Reduce to our Top X

    $vm_list = $vm_list | sort-object CpuMhz -Descending | select -First $GetTop 

    #

    # For each of those VMs, get the statistics for past and current CPU usage

    $NumVMs = $vm_list.Count

    $Out_List = @()

    $Count = 0

    Foreach ($vm in $vm_list)

    {

        $Count += 1

        Write-Progress -Activity "Compiling CPU stats" -Status "Progress:" `

            -PercentComplete ($Count / $NumVMs * 100)

           [Double] $ldblPerfAged = (Get-Stat -Entity $vm.VMName -Stat cpu.usage.average `

            -Start $((Get-Date).AddDays($DaysOld)) `

            -Finish $((Get-Date).AddDays($DaysOld + 1)) -ErrorAction Continue | `

            Measure-Object -Average Value).Average

        If ($ldblPerfAged -gt 0) {

               [Double] $lblPerfNow = (Get-Stat -Entity $vm.VMName -Stat cpu.usage.average `

                -Start $((Get-Date).AddDays($DaysRecent)) `

                -ErrorAction Continue | Measure-Object -Average Value).Average

            [Int] $lintTrend = (($lblPerfNow - $ldblPerfAged) / $ldblPerfAged) * 100

            $objOutput = "" | Select-Object VMName, CpuMhz, PerfAged, PerfNow, Trend

            $objOutput.VMName = $vm.VMName

            $objOutput.CpuMhz = $vm.CpuMhz

            $objOutput.PerfAged = "{0:f2}%" -f $ldblPerfAged

            $objOutput.PerfNow = "{0:f2}%" -f $lblPerfNow

            $objOutput.Trend = "{0}%" -f $lintTrend

            $out_list += $objOutput

        }

    }

    # Spit 'er out

    Write-Host "Top CPU Consumers Trending, $($DaysOld) days vs today`n"

    $out_list | Format-Table -Property VMName, `

        @{Expression={$_.CpuMhz};Name='CPU Mhz';align='right'}, `

        @{Expression={$_.PerfAged};Name='CPU Aged';align='right'}, `

        @{Expression={$_.PerfNow};Name='CPU Now';align='right'}, `

        @{Expression={$_.Trend};Name='Trend';align='right'}

}