LUNs with Multiple Exports

We have a vSphere 5.5 environment with your standard multiple paths to shared storage.  In our case, 2 paths to each of 2 nodes in a 7000-series HPE 3PAR array.

While investigating some stability issues we noticed something interesting:

That's one device and the LUN IDs should be the same.  In this configuration, vSphere sees the same device multiple times.

What's more, the vSphere GUI is doing us a "favor" and condensing the output.  Using esxcli we can confirm that there are, in fact, more than 4 paths:

Whoa Nelly!  I've condensed the output some, but that's 12 paths instead of 4.  Each LUN ID represents one set of paths.

The 3PAR GUI confirms:

This particular virtual volume was exported directly to the host twice and once indirectly via a Host Set.  Yeah, don't do that!

Determining the Scope of the Issue

I found a thread on VMware Communities from Markus Kraus, where he saw the same issue, although ultimately from a different cause.  Markus has a blog post too, with a great script and vCheck plugin!

Using Markus' script, I was able to determine all devices with more than one LUN ID.  Now we know how prevalent the issue is and which LUNs are involved on each particular vSphere host.

Unwinding

While we like 3PAR Host Sets, our implementation was incomplete.  Moving toward completion has the potential of causing more problems because exports are also tied to the Host Sets.  The biggest sin here was mixing the use of direct host exports and exports in Host Sets.

With the right attack plan we can work on one vSphere host at a time without any outage:

1. Place a host into Maintenance Mode.
    
2. Remove the host from any 3PAR Host Set.
    
3. Remove host-based duplicate virtual volume exports one at a time.
    
4. Rescan HBAs.
    
5. Exit Maintenance Mode.

We are not going to use the 3PAR GUI because it's not granular enough.  Removing a host from a host set would also remove the exports and risk impact on other hosts.  ssh into the 3PAR to access their rather nice CLI.
 

Our Command Set
 

List all Host Sets and their members:

showhostset

Show Virtual Volumes exported to a host:

showvlun -sortcol 1 -host <host> 

Remove the host from the Host Set:

removehostset <setname> <host>

Unexport from a host:

removevlun <vvname> <lunid> <host>

What I found to work best is to first remove the host from any Host Set followed by a rescan of the vSphere host HBA.  When that is completed, unexport any duplicate exports by removing the highest LUN ID(s).  Then rescan HBAs again.

Keep in mind that VVNames and Host Set Names are case sensitive.

Example Time

1. removehostset My-Host-Set host-9
    
2. Rescan HBA
    
3. removevlun NL-THIN-3 34 host-9
    
4. removevlun NL-THIN-3 33 host-9
    
5. Rescan HBA
    
6. showvlun -sortcol 1 -host host-9
    
7. Verify with esxcli storage core path list --device <device>