Thursday, January 19, 2017

LUNs with Multiple Exports

We have a vSphere 5.5 environment with your standard multiple paths to shared storage.  In our case, 2 paths to each of 2 nodes in a 7000-series HPE 3PAR array.

While investigating some stability issues we noticed something interesting:

That's one device and the LUN IDs should be the same.  In this configuration, vSphere sees the same device multiple times.

What's more, the vSphere GUI is doing us a "favor" and condensing the output.  Using esxcli we can confirm that there are, in fact, more than 4 paths:

Whoa Nelly!  I've condensed the output some, but that's 12 paths instead of 4.  Each LUN ID represents one set of paths.

The 3PAR GUI confirms:

This particular virtual volume was exported directly to the host twice and once indirectly via a Host Set.  Yeah, don't do that!

Determining the Scope of the Issue

I found a thread on VMware Communities from Markus Kraus, where he saw the same issue, although ultimately from a different cause.  Markus has a blog post too, with a great script and vCheck plugin!

Using Markus' script, I was able to determine all devices with more than one LUN ID.  Now we know how prevalent the issue is and which LUNs are involved on each particular vSphere host.


While we like 3PAR Host Sets, our implementation was incomplete.  Moving toward completion has the potential of causing more problems because exports are also tied to the Host Sets.  The biggest sin here was mixing the use of direct host exports and exports in Host Sets.

With the right attack plan we can work on one vSphere host at a time without any outage:

  1. Place a host into Maintenance Mode.
  2. Remove the host from any 3PAR Host Set.
  3. Remove host-based duplicate virtual volume exports one at a time.
  4. Rescan HBAs.
  5. Exit Maintenance Mode.
We are not going to use the 3PAR GUI because it's not granular enough.  Removing a host from a host set would also remove the exports and risk impact on other hosts.  ssh into the 3PAR to access their rather nice CLI.
Our Command Set

List all Host Sets and their members:
Show Virtual Volumes exported to a host:
showvlun -sortcol 1 -host <host> 
Remove the host from the Host Set:
removehostset <setname> <host>
Unexport from a host:
removevlun <vvname> <lunid> <host>
What I found to work best is to first remove the host from any Host Set followed by a rescan of the vSphere host HBA.  When that is completed, unexport any duplicate exports by removing the highest LUN ID(s).  Then rescan HBAs again.

Keep in mind that VVNames and Host Set Names are case sensitive.

Example Time

  1. removehostset My-Host-Set host-9
  2. Rescan HBA
  3. removevlun NL-THIN-3 34 host-9
  4. removevlun NL-THIN-3 33 host-9
  5. Rescan HBA
  6. showvlun -sortcol 1 -host host-9
  7. Verify with esxcli storage core path list --device <device>

Monday, March 14, 2016

Microsoft SQL Server on vSphere Best Practices Guide

Some of us have been running Microsoft SQL Server on top of vSphere for years. Maybe it was only for your pre-VCSA vCenter or maybe you have dozens of instances--but that doesn't stop an official Best Practices Guide from being good reading.

VMware has just updated, re-titled, and published  "Architecting Microsoft SQL Server on VMware vSphere -- Best Practices Guide."

There looks to be some pretty good information within, so get comfortable and grab the PDF here.

Wednesday, December 9, 2015

vSphere Host Logging Levels

If you are configuring vSphere hosts for syslog collection you may be overwhelmed by the amount of data thrown at your collector. By default hostd and vpxa are configured for a logging level of “verbose.”

To view the configuration for all of your vSphere hosts, break out PowerCLI.  I’m forever forgetting how to retrieve information from certain cmdlets and include the host name.  Here it is for me to find again:


Setting hostd and vpxa logging levels on all hosts to “info” is pretty easy:

Friday, October 30, 2015

Note to Self: Learn Get-View

Queue Deflation

I’m rather red-faced right now.  I chimed in with a reply in the VMTN VMware vSphere PowerCLI forum and promptly got schooled.

Which is great; these things should happen.

Where’s the Template?

In this thread, VMTN Communities member denjoh44 asked how one would obtain a list of templates along with the name of the vSphere hosts on which they reside.

I haven’t had the need to do this myself, but I wanted to give it a shot. I fired up PowerShell ISE and plugged away:

And it works, which is nice.

I See Your 12 Lines and…

It wasn’t long after that I revisited the thread only to find a much more elegant solution posted by Christophe Calvet:

My personality dictates that I should feel silly about my reply.  Christophe nailed it with a one-liner.  Instead, I’m going to focus on my inexperience with Get-View and do some reading.  I'll endeavor to always keep calculated expressions in mind as well.

Time to Learn

Brian Graf has a 3-part blog series on Get-View which I’ve seen before and marked for reading in more detail “later.”  If you are interested in learning more about Get-View, start here:
While you are at it, Christophe has a blog worth checking out at

Tuesday, October 6, 2015

vSphere 5.5 Update 3 Patch Available

The Best Intentions

It was obvious and natural; the Right Thing to do.  In preparation for a deployment across all vSphere hosts we took the step of insuring the latest updates. And since we are running vSphere 5.5, this meant Update 3.

There was even some chest-thumping among our small team of generalists as they completed the task within short order.  And why not?

That was Short-Lived

Within 2 weeks we were crestfallen. Update 3 introduced a service-impacting defect involving seldom-used (sic) snapshots.  KB 2133118 was released to announce "unexpected signal: 11."  Consolidate or delete a snapshot and you may suffer a VM outage.

To be honest, I find the KB lacking in details--there are no specifics to indicate contributing factors or degree of impact across customers.

A New Hope?

While we haven't been impacted (whew!), I'm happy to see that today brings the release of patch ESXi550-201510401-BG to address this issue.

Fire up Update Manager and start patching!  Let us know how it goes, because I'm not sure that we wish to be first on this one.


Thursday, September 24, 2015

Snazzy-Up Your PowerShell with a GUI

I’d like to say that I work with virtualization every day, and in a way I do.  Virtually every server in my employer’s environment is virtualized (see what I did there?).

That we are a small team in an SME environment dictates that we are I.T. generalists.  That’s not a curse—that’s variety and opportunity alleviating the mundane.  It’s also an opportunity to automate the mundane. 

For the purposes of user provisioning I wanted to use PowerShell for identity management on a shoestring budget.   As with many things, creating is harder than destroying; this provisioning needs a GUI.  Pick an OU, pick a user/template, and copy with new details.  While that’s how I got to the point of driving a GUI with PowerShell, it’s difficult to effectively convey a company-specific workflow.  So I’ll demonstrate with something virtualization-focused.

With a little bit of Googling I found an older option or two along with .NET-based Windows Presentation Foundation (WPF).  .NET this plus .NET that sounded appealing with me, so I dug in further. 

There are solid posts to be found that give you all that you need.  I’m working against some of my musings, so here I share my journey toward the same end result.

At a higher level the process is relatively simple:
  • Generate a GUI form in Extensible Application Markup Language (XAML).
  • Load the XAML in PowerShell with some minor (automated) transformations.
  • Instantiate the WPF form.
  • Create variables for working with our GUI elements.
  • Define PowerShell event handlers for each GUI element, incorporating our workflow within.

Generating Your GUI’s XAML

To generate the GUI we need something that will generate XAML that WPF can handle.  Anything from Visual Studio 2010 on up will do here.  I used Visual Studio 2015 Community since it is full featured with an investment of $0.

Open Visual Studio and create a new project.  Choose WPF Application under Visual C# Templates and give it a name.  Any name will do—we only want the XAML which this project will generate.

2015-09-23 20_26_39-Start Page - Microsoft Visual Studio

What you will notice is a blank window in the main center pane, labeled MainWindow.xaml.  The pane below contains the XAML in which we are interested. 

LetUsWPF - Main Window

I find it best to turn on the toolbox from the View menu and pin it to the side bar this yields an environment tailored for creating window forms.

Let’s use the toolbox to drag and position a few controls:
  • A ComboBox selector for Cluster
  • A DataGrid to display VMs within a selected Cluster.
  • A Button for launching a selected VM’s console.
  • A Label or three for clarity.
While we can add these controls in any order, Visual Studio will set the form’s tab order based on the order of definition.  That’s not a particularly big deal because of a neat Visual Studio feature:  the GUI form is tied to the XAML and the XAML is tied to the GUI form. We can adjust the tab order by re-ordering the text inside the XAML pane copy and paste!

With each control use the properties pane and give it a unique name, which we will  use later to bind to PowerShell variables.  I like to start all label control names with “label_” for easier exclusion from this binding process.

LetUsWPF - Drawn Form

Loading the XAML

All we need from Visual Studio is the XAML, which we will copy and paste into a PowerShell here-string.  The string isn’t perfect for PowerShell processing yet, so we’ll also rework it to take care of these issues and cast as XML:

Instantiating the GUI Form

Next we need to load the PresentationFramework assembly and process our form so that it can be presented:

Creating GUI Variables

We need to be able to work with the controls on form that comprise our GUI, and that means variables that let us get/set values and work with other properties and methods afforded to us by WPF.   

We can do this rather handily in PowerShell by invoking Set-Variable for nodes within the XML that comprises our form.  The iteration of those nodes will return Label controls as well, which we’ll ignore based on giving them a prefix of “Label_”, which was mentioned earlier:

Defining Event Handlers

WPF will send asynchronous signals (events) to our script when the GUI is acted upon.  Examples include when a selection is made, a button is clicked, and when the window is instantiated or destroyed. 

Defining event handler functions in our PowerShell lets us dictate our workflow:
  • When the window is created, connect to vCenter, get a list of clusters, and populate the cluster ComboBox.
  • When a selection is made in the cluster ComboBox, get a list of cluster VMs and populate the details DataGrid.
  • When a VM selection is made in the DataGrid, enable the console Button.
  • When the console Button is clicked, launch a vSphere Remote Console window.
  • When the window is closed, disconnect from vCenter
Anything is possible, of course.  Here’s our event handlers:

Final Bits

There’s not much left!  Define the vCenter server against which we will operate and display the form:


Run this little puppy and off we go.  Pick a cluster from the drop-down to see a list of VMs and click the column headers to if you don’t like it ordered by VM name.  Choose a VM and the console button becomes active so long as the VM state is “PoweredOn.”

LetUsWPF - Pick Cluster

LetUsWPF - Pick VM

What else?

  • My error checking and validations are non-existent--Add some.
  • Add another button to perform another task.
  • Change what VM properties appear in the DataGrid.
  • Do something else and have fun with it!

Full Code

You can grab the full code from here.

Proper Attribution is Proper

Thursday, September 17, 2015

Musings of an Infrequent Blogger

Writing is hard.
That's not the case for everyone, of course, but it holds true for me.  If you know me on a personal or professional level you've heard me say this before.  The odd thing is, I enjoy writing.  And because of that I tend to focus on this topic.

I have reasons not to write.

It's a silly idea/topic/thought

This is a powerful one for me and stops me the most often. It's also the quickest way out.  I question myself by nature, so there's no surprise here.  I'm still not certain that I'll hit publish on this post!

Suck it up and write anyway.

It's never quite right

One of the things that I enjoy most in the writings of others is the multitude of ways to say the same thing. The simplicity of some, the eloquence of others.  Let's not even talk about grammar, misspellings, etc.  Whether I'm writing here or within the corporate walls, I can tell you that I've reworked virtually every paragraph multiple times.

Practice makes perfect?  No, but it certainly improves.

Others have already written about this

Why bother covering a topic that so many others already have?  Fear of getting lost among similar technical articles and not managing to stand out.

Not to be glib, but meh--one more can't hurt.


How do others find the time?  I'm up early and put in long days. There is often weekend work. Around it all I must find time for family. There needs to be wind-down time and sleep in there somewhere.

Others do it. There are ways.  Give up a little time that's normally spent by myself anyway.

Why have I shared this?  Certainly it is therapeutic to a degree and I can refer to it in the future to push me forward.
But that's not my hope.

I want to reach someone else and push them forward. I'm an avid reader of blogs, sometimes churning through dozens a day. I make it part of my daily routine. I thoroughly enjoy reading what others have to say.

If you are like me, I'd like you to write--or write more.  

Because I'd like to read what you have to say.