Vmworld is right on the corner. There ain’t much new exciting information surfacing during this waiting period. I think I can use this time to gather my energy to have a little bit dive (not very deep though) before Vmworld hit on the ground.
I’m going to talk about PSA (Pluggable Storage Architecture) as part of requirement of VCAP study list. As usual, I try to make my post as simple as I can and also very welcome to any comments.
I remember when I started to read esxcli command and concept of PSA, I was simply overwhelmed by so many different parameters and options. And just like any one, I was lost. But after I started to abstract the detail command but try to understand what these commands tell me, everything is getting clear.
Please be aware: esxcli should be only used on esx(i) ssh session. vicfg-* should be used via vMA.
PSA Concept
To manage storage multipathing, ESXi uses a special VMkernel layer, the Pluggable Storage Architecture(PSA). The PSA is an open, modular framework that coordinates the simultaneous operation of multiplemultipathing plug-ins (MPPs).The VMkernel multipathing plug-in that ESXi provides by default is the VMware Native Multipathing Plug-In (NMP). The NMP is an extensible module that manages sub plug-ins. There are two types of NMP sub plugins,Storage Array Type Plug-Ins (SATPs), and Path Selection Plug-Ins (PSPs). SATPs and PSPs can be built-inand provided by VMware, or can be provided by a third party.
Let’s be reasonable. If a VM send a SCSI command to access data on the SAN, VMkernel needs to know how to access and which path it should choose. That’s where whole PSA kicks in. The PSA is a framework. It contains different modules and their sub-modules.
Note: PSA has 3 layers. Mulitpathing Layers->SATP layers->PSP layers
From the above picture, we can see PSA module need to choose between VMWARE NMP(Vmware own multipathing module) and MPP (thirdparty multipathing plug-ins, like EMC Powerpath) first. If you choose VMWARE NMP, please be aware you are not necessary using exclusive vmware product from this point. NMP still able to load third party sub-modules (SATP, PSP).
NMP&MPP layers
It basically decide which SATP to choose. This layer will see what kind of physical hardware (SAN) you have. EMC? NETAPP? DELL? It will load appropriate SATP, PSP to do other jobs. Please be aware you have bunch of SATP and PSP to choose. You can let NMP to decide or manually assign (or claim) new rules.
Please be aware MASK_PATH(MASK LUN in esx3.5) are consider as a NMP level plug-in.
SATP layer
Storage Array Type Plug-Ins (SATPs) run in conjunction with the VMware NMP and are responsible for arrayspecific
operations.
ESXi offers a SATP for every type of array that VMware supports. It also provides default SATPs that support
non-specific active-active and ALUA storage arrays, and the local SATP for direct-attached devices. Each SATP
accommodates special characteristics of a certain class of storage arrays and can perform the array-specific
operations required to detect path state and to activate an inactive path. As a result, the NMP module itself
can work with multiple storage arrays without having to be aware of the storage device specifics.
After the NMP determines which SATP to use for a specific storage device and associates the SATP with the
physical paths for that storage device, the SATP implements the tasks that include the following:
- Monitors the health of each physical path.
- Reports changes in the state of each physical path.
- Performs array-specific actions necessary for storage fail-over. For example, for active-passive devices, it can activate passive paths.
Please be aware SATP can be thirdparty one. But I don’t have thirdparty loaded in this picture.
PSP layers
Path Selection Plug-Ins (PSPs) run with the VMware NMP and are responsible for choosing a physical path for I/O requests.
The VMware NMP assigns a default PSP for each logical device based on the SATP associated with the physical paths for that device. You can override the default PSP.
These path policy are reflected to path policy in vCenter.
I’m not going to discuss each PSP here. If you have more questions, please refer to vmware docs.
PSA command line
There are few command lines related to PSA.
esxcli, vicfg-mpath, vicfg-mpath35
so what exact does command do? What kind of information I will pull out or change?
vicfg-mpath (vicfg-mpath35 is for esx 3.5)
This command is to list all available path and all detailed information about your device as well.
It also has ability to disable a path and active a path.
esxcli is much powerful command comparing with vicfg-mpath.
You need to be aware that esxcli is much more just to adjust PSA structure. It also control network, swiscsi, vaai, and vms behavior in vmKernel leve.
Let’s take a brief look what esxcli can do.
As you can see from the picture, from PSA wise, we only focus on corestorage and nmp.
Please consider esxcli is a command which you can use to interactive information of vmKernel pool. Let’s take look what kind of information you can access.
Configuration information (or Claim Rules)
Run command from esxi server ssh connection:
esxcli corestorage claimrule list
On the left side, we have rule class. There are 3 types of rule class(MP,FILTER,VAAI). It would appear if you use my last command. In this example, it didn’t because Hardware acceleration are not enabled and neither VAAI.
Rule runs from small to big number (or lower number as Vmware prefer). Be aware from Rule 0 to 101 are Vmware reserved rule. Between 102 to 60000 user can create their own rules. After 60000, Vmware claims those rules again.
When you build a rule, you need to build a pair rule(runtime and file). The file parameter in the class column indicates rule is defined. The runtime paramter indicates that rule has been loaded into your system.
Plugin also means module. In this example, we have NMP(Vmware module), MASK_PATH(Vmware LUN Mask) and MPP_1(thirdparty module for NewVend).
MASK Lun method has been changed. In ESX 3.5, we use vCenter to Mask LUN you don’t want hosts to see. In ESX 4, we have to use command and create MASK_PATH rule to dedicate LUNs.
Matches column is actually conditions for rules to apply. You can clearly see what kind of conditions rule 0-4 will apply and so forth. The last rule is like last rule of ISA firwall. Every other conditions which were not defined by previous rules will fall in this rule.
Device information
Have you ever wondering where you can see new UUID, vml, and other information for LUNs or devices connecting to your host? You can read my post at here or you got vicfg-mpath -l to do the job. You also can use esxcli nmp device list as well. But esxcli is starting from PSA wise.
SATP & PSP options
If you want to see what kind of SATP&PSP you have and you can choose, you can use
esxcli nmp satp list
esxcli nmp psp list
All right. I believe it should be easy to understand now. vmKernel has lots of information about PSA. You can use esxcli and vicfg-mpath to get information and modify as you want. I have to say, this is an understanding doc not reference doc. If you do want to add new path, MASK LUNs, or use different rules, you still need to check out all docs before you actually execute any commands.
Please do leave any comments if you want.
Reference:
ESXi configuration Guide
iSCSI SAN Configration Guide
GeekSilver's Blog 