Today we'll learn What is VMware VMFS Locking Mechanism. It is another post within our Tips category, so if you know your way, just skip. New people coming to learn VMware technology usually need this information to understand how VMFS locking works within the shared storage environment. And it is especially for those people that we're creating this post.
We will also talk about VMFS metadata which is basically a maping file assuring a global view of where all files (such as VMDKs, directories, RDM descriptor file….) are located. Metadata is updated each time you perform datastore or virtual machine management operations (Power On or OFF a VM, deploying VM from a template, vMotion migration, expanding VMFS datastore etc).
VMware VMFS locking mechanism is covered within VMware documentation, but we completed the knowledge from other sources too, including VMware forums.
So this post is partly put together from all those resources.
You'll find the links at the bottom of the post if you want to study further. Let's start with a short quote which explains the concept.
Quote:
In a shared storage environment, when multiple hosts access the same VMFS datastore, specific locking mechanisms are used. These locking mechanisms prevent multiple hosts from concurrently writing to the metadata and ensure that no data corruption occurs.
What is VMware VMFS Locking Mechanism?
There are two types of locking mechanisms:
ATS-Only Mechanism – For storage devices that support T10 standard-based VAAI specifications, VMFS provides ATS locking, also called “hardware-assisted locking“. The ATS algorithm supports discrete locking per disk sector. All newly formatted VMFS5 (and higher) datastores use the ATS-only mechanism if the underlying storage supports it, and never use SCSI reservations.
When you create a multi-extent datastore where ATS is used, vCenter Server filters out non-ATS devices. This filtering allows you to use only those devices that support the ATS primitive. In certain cases, you might need to turn off the ATS-only setting for a VMFS5 datastore.
ATS+SCSI Mechanism – A VMFS datastore that supports the ATS+SCSI mechanism is configured to use ATS and attempts to use it when possible. If ATS fails, the VMFS datastore reverts to SCSI reservations. In contrast with the ATS locking, the SCSI reservations lock an entire storage device while an operation that requires metadata protection is performed. After the operation completes, VMFS releases the reservation and other operations can continue.
Datastores that use the ATS+SCSI mechanism include VMFS5 datastores that were upgraded from VMFS3. In addition, new VMFS5 datastores on storage devices that do not support ATS use the ATS+SCSI mechanism.
How to find out in my VMware vSphere environment?
Connect via SSH (use Putty for example) to your host and enter the following command:
esxcli storage vmfs lockmode list
You'll see an output like this: (click to enlarge)
As you can see in my environment, I have some VMFS-6 and also VMFS-5 formatted datastores.
The documentation states that it is possible to upgrade the locking mechanism to ATS Only if your hardware supports it. You can find out about it within the third column from the right, a column called “ATS compatible“. In my case, this is not possible.
If not you can use this command for the upgrade of the locking mechanism: Source: VMware Docs
esxcli storage vmfs lockmode set -a|–ats -l|–volume-label= VMFS label -u|–volume-uuid= VMFS UUID.
You then need to unmount and remount the datastore o all hosts which are accessing this datastore. An additional way to complete the upgrade would be to put the datastore into a maintenance mode and then exit the maintenance mode.
Check if the output by using the first command again, whether the output changed to ATS-only mode.
You may have other outputs. for example, If the Locking Mode displays any other status, (ATS UPGRADE PENDING), you can check which host has not yet processed the upgrade by running this command:
esxcli storage vmfs host list
This is about it about the VMware VMFS locking mechanism.
Lab screenshot showing a shared Drobo datastore….
VMware VMFS Locking mechanism and “On-disk locking”. How that works?
The “on-disk locking” does prevent powering on a virtual machine (VM) on several hosts at once. Pretty serious, right… The fact that host where the VM is powered on, it locks the VM files, and in this case, the VM can't be powered on another host within the cluster.
Quote:
On-disk locking in VMFS ensures that a virtual machine is not powered on by multiple ESXi Servers at the same time. With VMware HA enabled, if a server fails the on-disk lock for each virtual machine is released, allowing the virtual machine to be restarted on other physical servers.
Screenshot from VMware…
What Is a Distributed Lock Manager?
Yes, this is another interesting concept which is part of the on-disk locking. DLM (Distributed Lock Manager), provides distributed software applications with a means to synchronize their accesses to shared resources. The key here is the distributed aspect of the locking method.
DLM runs on each host which is part of the cluster, and the lock management is distributed across all hosts in the cluster. Each ESXi host stores its virtual machine files in a specific subdirectory on the VMFS file system. When a VM is running, VMFS has a lock on those files so that other ESXi hosts cannot update them. VMFS ensures that the VM cannot be opened by more than one ESXi Server in the cluster.
Wrap Up:
As you can see VMware locking mechanisms are pretty solid allowing to control the writes to shared storage. It's pretty obvious that it has to be managed somehow. Now we know that it is thanks to the ATS or the ATS+SCSI locking mechanisms.
For the VM on-disk locking it works a bit different way, but still, pretty cool knowing that clusters with shared storage can handle that. And VMs can be restarted on another host within the cluster in case there is a host failure, without that the VM stays locked by the failed host.
More from ESX Virtualization:
- What is VMware Snapshot?
- What is VMware vMotion?
- What is VMware Enhanced vMotion Compatibility (EVC)
- What is VMware Storage DRS (SDRS)?
- VMware vSphere Standard vs Enterprise Plus
Stay tuned through RSS, and social media channels (Twitter, FB, YouTube)
Sources: VMware Docs , then Google “vSphere Storage Guide” …, (PDF), VMware KB article Investigating virtual machine file locks on ESXi.
