LUN and HBA Queue Depth on ESX(i) Hosts
This is a guest post by Andy Grant.
Jason Boche has posted an interesting article on the VMkernel parameter Disk.SchedNumReqOutstanding. I confess that I have never thought much about the setting other than following VMware’s recommendation when changing HBA queue depth.
Changing the Queue Depth for QLogic and Emulex HBAs
“The recommendation is to set both Disk.SchedNumReqOutstanding and the adapter queue depth to the same value.”
Both HBA and LUN (Disk.SchedNumReqOutstanding) default to 32. Changing one means changing both and this brings up some interesting questions that I think I will have to lab to find the answer.
- SIOC dynamically changes LUN queue depth (DQLEN) depending on load, how does this effect Disk.SchedNumReqOutstanding?
- The adaptive queue depth algorithm dynamically changes DQLEN, how does this effect Disk.SchedNumReqOutstanding?
- Many consider VAAI to be the god-send for increasing VM per datastore ratios with hardware accelerated fine-grained block-level SCSI locking, so will the array per-LUN queue depth be the next bottlenext? See the Scalable Storage Performance page 5 for more on this.
I think I have my lab-work cut out for me next week.
Update: Re-framed questions per https://www.vmware.com/files/pdf/techpaper/vsp_41_perf_SIOC.pdf. SIOC does not throttle HBA queue depth (AQLEN), it throttles LUN queue depth (DQLEN).
“SIOC, if enabled, manages the distribution of queue slots from the array-level I/O queue among the hosts by throttling the device queue“
