LUNs with Multiple Exports

We have a vSphere 5.5 environment with your standard multiple paths to shared storage.  In our case, 2 paths to each of 2 nodes in a 7000-series HPE 3PAR array.

While investigating some stability issues we noticed something interesting:

That's one device and the LUN IDs should be the same.  In this configuration, vSphere sees the same device multiple times.

What's more, the vSphere GUI is doing us a "favor" and condensing the output.  Using esxcli we can confirm that there are, in fact, more than 4 paths:

Whoa Nelly!  I've condensed the output some, but that's 12 paths instead of 4.  Each LUN ID represents one set of paths.

The 3PAR GUI confirms:

This particular virtual volume was exported directly to the host twice and once indirectly via a Host Set.  Yeah, don't do that!

Determining the Scope of the Issue

I found a thread on VMware Communities from Markus Kraus, where he saw the same issue, although ultimately from a different cause.  Markus has a blog post too, with a great script and vCheck plugin!

Using Markus' script, I was able to determine all devices with more than one LUN ID.  Now we know how prevalent the issue is and which LUNs are involved on each particular vSphere host.

Unwinding

While we like 3PAR Host Sets, our implementation was incomplete.  Moving toward completion has the potential of causing more problems because exports are also tied to the Host Sets.  The biggest sin here was mixing the use of direct host exports and exports in Host Sets.

With the right attack plan we can work on one vSphere host at a time without any outage:

1. Place a host into Maintenance Mode.
    
2. Remove the host from any 3PAR Host Set.
    
3. Remove host-based duplicate virtual volume exports one at a time.
    
4. Rescan HBAs.
    
5. Exit Maintenance Mode.

We are not going to use the 3PAR GUI because it's not granular enough.  Removing a host from a host set would also remove the exports and risk impact on other hosts.  ssh into the 3PAR to access their rather nice CLI.
 

Our Command Set
 

List all Host Sets and their members:

showhostset

Show Virtual Volumes exported to a host:

showvlun -sortcol 1 -host <host> 

Remove the host from the Host Set:

removehostset <setname> <host>

Unexport from a host:

removevlun <vvname> <lunid> <host>

What I found to work best is to first remove the host from any Host Set followed by a rescan of the vSphere host HBA.  When that is completed, unexport any duplicate exports by removing the highest LUN ID(s).  Then rescan HBAs again.

Keep in mind that VVNames and Host Set Names are case sensitive.

Example Time

1. removehostset My-Host-Set host-9
    
2. Rescan HBA
    
3. removevlun NL-THIN-3 34 host-9
    
4. removevlun NL-THIN-3 33 host-9
    
5. Rescan HBA
    
6. showvlun -sortcol 1 -host host-9
    
7. Verify with esxcli storage core path list --device <device>

vSphere Claim Rules with PowerCLI

We’ve been implementing HP 3PAR 7000 series into our environment. During this task we’ve tryied to be good admins by following the HP 3PAR StoreServ Storage and VMware vSphere 5 best practices.

Within this excellent resource there is a section for automating round robin policy for all 3PAR LUNs with a custom SATP rule.  That’s great, but it involves an esxcli command that must be run on each host:


For a small number of hosts that’s not too terrible--until you realize that you must either reboot each host or modify the policy for each LUN currently presented.  That’s not something that I’m likely to do.

With my aid of my technology buddy Google, I discovered several tidbits online:

• Philip Sellers has a blog that works through much of this in PowerCLI.
• Cormac Hogan has a blog that re-iterates the above, along with the commands required to unload each already claimed device (LUN), reload the claim rules, and rescan the HBAs.

Most of what I needed was out there already, of course. But I wanted to reload the claim rules in order to test the custom rule and without setting the policy individually on each LUN. I also didn’t want to impact every iSCSI LUN—just 3PAR—and there’s no need to reclaim a LUN that is already configured for round robin.

Here’s how I did it for each host:


Of course, I wrapped all of this around more PowerCLI that obtained a list of hosts in each cluster and managed the connections for esxcli.

If you receive any errors during the unclaim, make sure that your HA configuration is not using those LUNs as heartbeat datastores.  I experienced that problem which was easy to remedy in our environment.