Skip navigation

Tag Archives: qos

One of biggest changes for vSphere 4.1 is introduction of Network I/O control and Storage I/O.

This post will give you an introduction and understanding of what Network I/O control (NetIOC) is. This is a new technology and we still need to wait and see more real case in the future. But for now, Let’s see what Network IO control is.

Why do we need to have Network IO Control (NetIOC)?

1. 1Gbit network is not enough

As you may know, we have more and more demanding on the network traffic. FT traffice, iSCSI Traffic(Don’t you team up?) and NFS Traffic, vMotion Traffic etc. Although you can team up multiple physical nics together, but from a single VM perspective, it can only allow to use one physical nic at one time no matter what kind of teaming method you are using. Plus, network team has already started to talk about 100Gbit network and it’s about time to push 10Gbit network into public.

2. Blade server demands

All new blade server has 10Gbit ports switch in the blade. The architecture of Blade server has changed from each blade has it’s own ports to central ethernet Module. It saves a lot of resource and traffic can be easily Qos and scaled.

Prerequisites for Network IO control

1. You need Enterprise Plus license

The reason for that is NetIOC is only available for vDS. For vSS, you can only control outbound traffic.

2. You need vSphere 4.1 and ESX 4.1

With vSphere 4.0, you do can control traffic by port group. But you can’t preconfigure traffic by type (or you can call it by class). This is fundamental architecture change. We will talk about it later.  ESX 4.1 is also required otherwise you won’t see the new tab in the vCenter.

How does Network IO Control (NetIOC) work?

If you recall vSphere 4.0, we also have ingress and egress traffic control for vDS.(for vSS, we only have outbound control) Traffic shaping is controlled by Average Bandwidth, pea bandwidth and bust size. You have manually divide dvUplinks by functions. Like this dvUplink is for FT, this is for vMotion, this is for iSCSI etc. Everything is done by manual.

With new vSphere 4.1, we are not only able to control traffic by port group, we are also control traffic by class.

The NetIOC concept revolves around resource pools that are similar in many ways to the ones already existing for CPU and Memory.
NetIOC classifies traffic into six predefined resource pools as follows:
• vMotion
• FT logging
• Management
• Virtual machine traffic

If you open vCenter, you will see the new tab of dvSwitch  in your ESX i 4.1 server.

This means all traffic go through this vDS will be under Qos by these rules. Remember, it only works for this vDS.

Now, let’s see the architecture picture first and then, we talk about how this thing work.

As you can see, there are 3 layers in NETIOC. Teaming Policy, shaper and Scheduler. As what my previous post mentioned, vDS is actually a combination of special hidden vSS and policy profiles downloaded from vCenter.

Teaming policy (New policy, LBT)

There is a new method of teaming called LBT(Load base teaming). It basically detect how busy those physical nics are, then it will move the flows to different cards. LBT will only move a flow when the mean send or receive utilization on an uplink exceeds 75 percent of capacity over a 30-second period. LBT will not move flows more often than every 30 seconds.

Best practice 4: We recommend that you use LBT as your vDS teaming policy while using NetIOC in order to maximize the networking capacity utilization.
NOTE: As LBT moves flows among uplinks it may occasionally cause reordering of packets at the receiver.

I haven’t done any tests on how much extra CPU cycles are required to run LTB, but we will keep eyes on it.


There are two attributes( Shares and Limit) you can control over traffic via Resource Allocation. Resource Allocation is controlling base on vDS and only apply to this vDS. It applies on vDS level not on port group or dvUplink level. Shaper is where limits apply. It limits traffic by the class of traffic.  Be noticed at this 4.1, each vDS has it’s own resource pool and resource pool are not shared between vDS.

A user can specify an absolute shaping limit for a given resource-pool flow using a bandwidth capacity limiter. As opposed to shares that are enforced at the dvUplink level, limits are enforced on the overall vDS set of dvUplinks, which means that a flow of a given resource pool will never exceed a given limit for a vDS out of a given vSphere host.


Shares apply to dvUplink Level and each share rates will be calculated base on traffic of each dvUplink. It controls share value of traffic going through this particular dvUplink and make sure share percentage is correct.

the network flow scheduler is the entity responsible for enforcing shares and therefore is in charge of the overall arbitration under overcommitment. Each resource-pool flow has its own dedicated software queue inside the scheduler so that packets from a given resource pool won’t be dropped due to high utilization by other flows.

NetIOC Best Practices

NetIOC is a very powerful feature that will make your vSphere deployment even more suitable for your I/O-consolidated datacenter. However, follow these best practices to optimize the usage of this feature:

Best practice 1: When using bandwidth allocation, use “shares” instead of “limits,” as the former has greater flexibility for unused capacity redistribution. Partitioning the available network bandwidth among different types of network traffic flows using limits has shortcomings. For instance, allocating 2Gbps bandwidth by using a limit for the virtual machine resource pool provides a maximum of 2Gbps bandwidth for all the virtual machine traffic even if the team is not saturated. In other words, limits impose hard limits on the amount of the bandwidth usage by a traffic flow even when there is network bandwidth available.

Best practice 2: If you are concerned about physical switch and/or physical network capacity, consider imposing limits on a given resource pool. For instance, you might want to put a limit on vMotion traffic flow to help in situations where multiple vMotion traffic flows initiated on different ESX hosts at the same time could possibly oversubscribe the physical network. By limiting the vMotion traffic bandwidth usage at the ESX host level, we can prevent the possibility of jeopardizing performance for other flows going through the same points of contention.

Best practice 3: Fault tolerance is a latency-sensitive traffic flow, so it is recommended to always set the corresponding resource-pool shares to a reasonably high relative value in the case of custom shares. However, in the case where you are using the predefined default shares value for VMware FT, leaving it set to high is recommended.

Best practice 4: We recommend that you use LBT as your vDS teaming policy while using NetIOC in order to maximize the networking capacity utilization.

NOTE: As LBT moves flows among uplinks it may occasionally cause reordering of packets at the receiver.

Best practice 5: Use the DV Port Group and Traffic Shaper features offered by the vDS to maximum effect when configuring the vDS. Configure each of the traffic flow types with a dedicated DV Port Group. Use DV Port Groups as a means to apply configuration policies to different traffic flow types, and more important, to provide additional Rx bandwidth controls through the use of Traffic Shaper. For instance, you might want to enable Traffic Shaper for the egress traffic on the DV Port Group used for vMotion. This can help in situations when multiple vMotions initiated on different vSphere hosts converge to the same destination vSphere server.

Let me know if you have more questions.


Click to access VMW_Netioc_BestPractices.pdf