How to configure
Starting with vSphere 6.5 it is possible to configure IO limits using Storage Policies Based Management (SPBM). Alternatively you can edit VM settings and configure the limit directly at the VMDKs.
To configure using SPBM:
- Go to Policies and Profiles in WebClient
- Click VM Storage Policies
- Click Create VM Storage Policy
- Under 2a Common rules check Use common rules in VM Storage policy
- Click Add component, select Storage I/O Control, select Custom
- At this point you can enter IOPs limit
- Continue rule creation to the end of the wizard
- Select VMs you want to apply this policy, right-click your selection and select VM Policies and Edit VM Storage Policies. Select you policy and press OK. After this all VMDKs get this policy applied. If you want to exclude VMDKs from policy, edit VM settings and set storage policy for VMDK back to Datastore default.
To configure limits by editing VM settings:
- Right-click VM and click Edit settings
- Expand VMDK to limit and enter Limit – IOPs
- Click OK.
Both way can be done online, changes gets applied immediately.
How does it work
All the work is done by a component called mClock, running on ESXi host. To execute IO limits, mClock leverages kernel IO queuing. What is kernel IO queuing? This is something you don’t want to see during normal operations. It happens, when IOs triggert by VMs on a datastore exceed its device queue. When device queue is full, kernel IO queue jumps in to hold these IOs and put them into device queue when possible. For VM IOs this behavior is fully transparent.
Because of this you get a downside of IO limits: higher latency!
During normal IO operations, kernel IO queues should not be used. Here is a screenshot of an IOmeter test in esxtop.
Short column description:
- DQLEN: device (LUN/volume) queue lenght
- ACTV: current count of IOs in device queue
- QUED: IOs in kernel queue
- CMDS/s: IOs per second
- DAVG/cmd: storage device latency
- KAVG/cmd: kernel latency
- GADG/cmd: guest (VM) latency
- QAVG/cmd: kernel queue latency
As you can see, no kernel queue is used, therefore device latency is the same as guest latency.
An example when kernel queue is used is when a VM uses a larger IO queue in virtual disk controller than DQLEN and uses this queue excessively . Here an example of such situation. Disk controller in VM has a configured length or size of 256. I used IOmeter to run a test using 100 outstanding IOs:
You can see, QUED, KAVG/cmd and QAVG/cmd are > 0. You can also see ACTV + QUED ~ 100 which is the used queue length in VM. When you have no hardware problems in your SAN (cable, SFP), QAVG/cmd should be the same as KAVG/cmd.
- IO limit for VMDK is set to 2000
- IO-size in IOmeter is set to 32kb
- Outstanding IOs: 16.
You can see:
- ACTV = 0 –> device queue is not used
- QUED = outstanding IOs –> kernel IO queuing
- CMDS/s ~ IO limit set
- KAVG/cmd > 0 –> kernel IO queuing
- GAVG/cmd = DAVG/cmd + KAVG/cmd
Conclusio: kernel queue is used to handle the amount of IOs put to device queue to slowdown IO processing. Because IOs have to wait in kernel queue, latency is increased.
About IO Size
You define IO limit using an absolute value. But mClock does not count every IO as one IO. The size of an IO matters! mClock “normalizes” IOs to a multiple of 32KB. In a mathematical notation:
means to round up to next integer (). So an IO-size in range of 1-32KB count as 1, 33-64KB count as 2, …
To show this behavior, I used the same test as before. Only differnce: 33KB instead of 32KB IO size:
IOs cut in half.
Next 65KB IO size:
…only one third.
- Despite you configure a Storage IO Control (SIOC) policy when using SPBM, SIOC must not be enabled on datastore.
- When storage policy is applied at a VM that already uses manually set IO limit, lower limit wins and is executed.
- When using SPBM, IO Limits are not shown when edit VM settings. There you can only see applied policy name.
- When storage policy gets detached, disk IO Limits are reset to Unlimited.