Skip navigation

Tag Archives: Network Distributed Switch

One of biggest changes for vSphere 4.1 is introduction of Network I/O control and Storage I/O.

This post will give you an introduction and understanding of what Network I/O control (NetIOC) is. This is a new technology and we still need to wait and see more real case in the future. But for now, Let’s see what Network IO control is.

Why do we need to have Network IO Control (NetIOC)?

1. 1Gbit network is not enough

As you may know, we have more and more demanding on the network traffic. FT traffice, iSCSI Traffic(Don’t you team up?) and NFS Traffic, vMotion Traffic etc. Although you can team up multiple physical nics together, but from a single VM perspective, it can only allow to use one physical nic at one time no matter what kind of teaming method you are using. Plus, network team has already started to talk about 100Gbit network and it’s about time to push 10Gbit network into public.

2. Blade server demands

All new blade server has 10Gbit ports switch in the blade. The architecture of Blade server has changed from each blade has it’s own ports to central ethernet Module. It saves a lot of resource and traffic can be easily Qos and scaled.

Prerequisites for Network IO control

1. You need Enterprise Plus license

The reason for that is NetIOC is only available for vDS. For vSS, you can only control outbound traffic.

2. You need vSphere 4.1 and ESX 4.1

With vSphere 4.0, you do can control traffic by port group. But you can’t preconfigure traffic by type (or you can call it by class). This is fundamental architecture change. We will talk about it later.  ESX 4.1 is also required otherwise you won’t see the new tab in the vCenter.

How does Network IO Control (NetIOC) work?

If you recall vSphere 4.0, we also have ingress and egress traffic control for vDS.(for vSS, we only have outbound control) Traffic shaping is controlled by Average Bandwidth, pea bandwidth and bust size. You have manually divide dvUplinks by functions. Like this dvUplink is for FT, this is for vMotion, this is for iSCSI etc. Everything is done by manual.

With new vSphere 4.1, we are not only able to control traffic by port group, we are also control traffic by class.

The NetIOC concept revolves around resource pools that are similar in many ways to the ones already existing for CPU and Memory.
NetIOC classifies traffic into six predefined resource pools as follows:
• vMotion
• FT logging
• Management
• Virtual machine traffic

If you open vCenter, you will see the new tab of dvSwitch  in your ESX i 4.1 server.

This means all traffic go through this vDS will be under Qos by these rules. Remember, it only works for this vDS.

Now, let’s see the architecture picture first and then, we talk about how this thing work.

As you can see, there are 3 layers in NETIOC. Teaming Policy, shaper and Scheduler. As what my previous post mentioned, vDS is actually a combination of special hidden vSS and policy profiles downloaded from vCenter.

Teaming policy (New policy, LBT)

There is a new method of teaming called LBT(Load base teaming). It basically detect how busy those physical nics are, then it will move the flows to different cards. LBT will only move a flow when the mean send or receive utilization on an uplink exceeds 75 percent of capacity over a 30-second period. LBT will not move flows more often than every 30 seconds.

Best practice 4: We recommend that you use LBT as your vDS teaming policy while using NetIOC in order to maximize the networking capacity utilization.
NOTE: As LBT moves flows among uplinks it may occasionally cause reordering of packets at the receiver.

I haven’t done any tests on how much extra CPU cycles are required to run LTB, but we will keep eyes on it.


There are two attributes( Shares and Limit) you can control over traffic via Resource Allocation. Resource Allocation is controlling base on vDS and only apply to this vDS. It applies on vDS level not on port group or dvUplink level. Shaper is where limits apply. It limits traffic by the class of traffic.  Be noticed at this 4.1, each vDS has it’s own resource pool and resource pool are not shared between vDS.

A user can specify an absolute shaping limit for a given resource-pool flow using a bandwidth capacity limiter. As opposed to shares that are enforced at the dvUplink level, limits are enforced on the overall vDS set of dvUplinks, which means that a flow of a given resource pool will never exceed a given limit for a vDS out of a given vSphere host.


Shares apply to dvUplink Level and each share rates will be calculated base on traffic of each dvUplink. It controls share value of traffic going through this particular dvUplink and make sure share percentage is correct.

the network flow scheduler is the entity responsible for enforcing shares and therefore is in charge of the overall arbitration under overcommitment. Each resource-pool flow has its own dedicated software queue inside the scheduler so that packets from a given resource pool won’t be dropped due to high utilization by other flows.

NetIOC Best Practices

NetIOC is a very powerful feature that will make your vSphere deployment even more suitable for your I/O-consolidated datacenter. However, follow these best practices to optimize the usage of this feature:

Best practice 1: When using bandwidth allocation, use “shares” instead of “limits,” as the former has greater flexibility for unused capacity redistribution. Partitioning the available network bandwidth among different types of network traffic flows using limits has shortcomings. For instance, allocating 2Gbps bandwidth by using a limit for the virtual machine resource pool provides a maximum of 2Gbps bandwidth for all the virtual machine traffic even if the team is not saturated. In other words, limits impose hard limits on the amount of the bandwidth usage by a traffic flow even when there is network bandwidth available.

Best practice 2: If you are concerned about physical switch and/or physical network capacity, consider imposing limits on a given resource pool. For instance, you might want to put a limit on vMotion traffic flow to help in situations where multiple vMotion traffic flows initiated on different ESX hosts at the same time could possibly oversubscribe the physical network. By limiting the vMotion traffic bandwidth usage at the ESX host level, we can prevent the possibility of jeopardizing performance for other flows going through the same points of contention.

Best practice 3: Fault tolerance is a latency-sensitive traffic flow, so it is recommended to always set the corresponding resource-pool shares to a reasonably high relative value in the case of custom shares. However, in the case where you are using the predefined default shares value for VMware FT, leaving it set to high is recommended.

Best practice 4: We recommend that you use LBT as your vDS teaming policy while using NetIOC in order to maximize the networking capacity utilization.

NOTE: As LBT moves flows among uplinks it may occasionally cause reordering of packets at the receiver.

Best practice 5: Use the DV Port Group and Traffic Shaper features offered by the vDS to maximum effect when configuring the vDS. Configure each of the traffic flow types with a dedicated DV Port Group. Use DV Port Groups as a means to apply configuration policies to different traffic flow types, and more important, to provide additional Rx bandwidth controls through the use of Traffic Shaper. For instance, you might want to enable Traffic Shaper for the egress traffic on the DV Port Group used for vMotion. This can help in situations when multiple vMotions initiated on different vSphere hosts converge to the same destination vSphere server.

Let me know if you have more questions.


Click to access VMW_Netioc_BestPractices.pdf

I just came back from Vmware Seminar 2010. There are lots of information I would like to share with you. You can clearly see  where Vmware is heading for it’s own future.

Future of Vmware: Cloud

You may heard some news about VmForce (Vmware combine with Salesforce to make cloud level ERP system), Vmware acquired SpringSource and public cloud, private clound. I was quite confused before I attended this seminar since I could hear everyone is talking about cloud but no real clouds system for private enterprise to merge or any real cases about cloud. This puzzle is resolved by this Seminar.

VMware wants to get rid of Microsoft

That’s it. That’s root cause why Vmware did all sorts of weird activities in past year. This is what Vmware has planned:

Make all companies virtualization 50% up (√) (85% world companies are using VMware tech)

Make all companies 100% virtualized

  • Vmware SRM helps DR (Expensive plan which requires 100% virtualization & DR budget, only about 5% companies are doing that)
  • vSphere helps servers platform (facing challenges from MS and Citrix)
  • VMware View helps Desktop (unsuccessful and beaten by Citrix  XenDesktop)
  • vThinapps (very few companies have actually used in the production, this is prestep for stripping apps from OS level in the future)

Using ESXi to replace ESX (has confirmed from all VMWARE people, it will happen in 2011. Vmware can finally get rid of head ache Red Hat for SC and have hardware appliance alike ESXi as house bricks)

Using VMSafe products (like Trend Macro, Agentless anti-virus. It will be available very soon).

Private Cloud era (VMware believes applications don’t need to run on Microsoft OS. They can let applications run directly on VMKernel. It’s obviously that Microsoft won’t agree with this idea. That’s why VMware bought SpringSource(Java application company) and try to make application platformless. I believe Microsoft will soon to push out their own cloud system and also use MS version thinApps and stream APPs to fight with Vmware)

Hybrid cloud (It’s also called Redwood project. This is next generation VM OS. If Private cloud is ready, meaning all apps can run on VMkernel without MS OS. Any apps can be seamlessly transferred between private cloud to hybrid cloud and even public cloud).

Public cloud (At that day, every system will run on a standard industry module  and can accept and transfer all applications)

In terms of convincing CIOs and IT Managers to purchase equipment to do VM DR and 100% virtualization, Vmware put lots of efforts on CapacityIQ, ChargeBack, how to shift attention from CAPEX to OPEX during the seminar. It only broadcasts one message. Come and virtualize everything!

Good plans, but there are concerns:

I agreed Vmware has drawn a beautiful picture of future I.T. But whether they can actually pull off this show is really a question mark. With all those components, like servers, Desktops, Networks, any parts of failure may cause huge disaster for Vmware. Vmware View is still not promising from angle of optimizing qos via WAN. PCoIP completely when it competes with Citrix ICA/HDX. Vmware users have to go back and use MS RDP protocol to connect Virtual desktop which gives Microsoft a chance to regain the market. Even in the latest demonstration, Vmware view still hasn’t fixed issue. From my personal understanding, Java applications has tons of issue. Slowness, stability problem. Novell has to dropped off Java console from Netware few years ago doesn’t mean anything to Vmware?

This is part 2 of vDS (vNetwork Distributed Switch), My Understanding.

How does vDS work?

What will your instructor tell you? “Please don’t consider vDS is a switch connecting to Hosts. vDS is just a template”  Well, that’s what you always heard from all your instructors. but template of what? The answer is vDS is template of HIDDEN vSwitch sitting on your local host. vDS(the template) is managed by vCenter(high level operation) and your local Host(low level operation). Let’s see a diagram.

From this diagram, you can see there are two hosts. Each host has hidden switch which received template (vDS) from vCenter. The local template will be updated every 5 minutes like what I mentioned in Part 1.

Now, let’s open this hidden switch and see what’s happening in there.

As you can see, the hidden switch has forwarding engine and teaming engine which will be configured and controlled by setting in vCenter. There are two IO filters (not just one) is to be used in VMSafe. So what VMSafe does is let third party software (for example, the Trend Micro)  build a VM appliance and be certified by VMWARE to prove it won’t do any damage. That special VM will use special API to monitor traffice (like firewall) or check virus. Meaning, if you want to use VMSafe product, you have to use vDS, meaning you have buy Enterprise Plus license! I guess that’s why VMSafe product is not popular.

ok. Back to vDS. Let’s make a small conclusion. vDS is also a vSS. But it’s hidden in the Host. This hidden vSS is using template made by vCenter and Local Host so you can control traffic and share switch data between hosts.

Few things you need to know about vDS

vDS is capable to do everything vSS can do because it’s basically a super (hidden) vSS. Once you assign a vmnic OR VMkernal, SC or VM to vDS, you won’t be able to use them in vSS. It’s same thing as vSS.

I won’t say there are not much point to use vDS but if you do want to use vDS, you would either use Cisco Nexus to replace vDS or you want to use VMSafe product. Or you have Enterprise Plus license and want to use host profiles.

vDS timeout issue

vDS is not as convenient as vSS which only connects to single Host. sometimes, it’s not easy to remove vDS switches or even switch physical nic to different vDS switches. If vDS believes a port is busy, vCenter won’t allow you to delete vDS or remove a host from it. By the default, vCenter automatically forces all “busy” ports on all distributed switches to time out every 24 hours.

You can make change on vpxd.cfg to make it as 10 minutes.

vpxd.conf is located at c:\documents and settings\all users\application data\VMware\VMware VirtualCenter\vpxd.cfg

In vpxd.cfg, add the line <vpxd><dvs><portReserveTimeout>10</portReserveTimeout></dvs></vpxd> and save the file.

Restart vCenter. The default timeout is now set to ten minutes.

After the port reservation has timed out, remove the vNetwork Distributed Switch or dvPort group.

Reset the default timout by removing the line you previous added to vpxd.cfg.

Restart vCenter.

Best Practise for vDS daily operation

If you run into problem with vDS, always start checking from vCenter->Networking level. Because it has general view to tell you all vDS details such as IP on each port group, PVLAN info and which VM or kernal ports it connects. Then, you should drill down to single host ->Configuration->Networking to add or remove objects. If you do have issue, try to remove all objects to another vDS and then, make your change.

For the rest of details, like PVLAN, blocking of individual ports, you can check this file to continue your journey of vDS.

Click to access vsphere-vnetwork-ds-migration-configuration-wp.pdf


Click to access vsphere-vnetwork-ds-migration-configuration-wp.pdf

vSphere has introduced many new features. One of new feature is vDS (vNetwork Distributed Switch) which always confuses me and lots of people. I’m trying to explain it as simple and easy, also deep to you as I can. If I make any mistakes, please feel free to leave comment. Thank you.

So What is vDS? What’s difference between vSS and vDS from configure file structure?

vDS is a new Virtual Switch introduced by Vmware. The old vSS is more like local Host property. All switch data saved in the local Host. Other Host is not aware what kind of vSS other Host has.Not only vCenter can’t do anything about it, it causes trouble when you do vMotion.  vDS is saved in both vCenter and Host. One copy in the vCenter, vDS is saved in the SQL database. In the local host, vDS has another local database cache copy sits at /etc/vmware/dvsdata.db. This local cache will be updated by vCenter every 5 minutes.

You can use following command to help you to get copy to read the local host database.

cd /usr/lib/vmware/bin

./net-dvs > /tmp/dvs.txt

then, you can read dvs.txt

Also, after you configure vDS on your local host, your esx.conf (/etc/vmware) has record shows brief configuration information of vDS.

Those 3 configuration combines vDS structure. This also makes vDS can work even after Host disconnected from vCenter.

What’s difference between vSS and vDS on control level?

With vSS, everything should be controlled on local host. Basically, you go to Local Host->Configuration->Networking. Then, you start everything from there. But vDS is different. vDS divide control into 2 different level. I call them high level and low level.

High Level: High level is to create/remove, management teaming, distribution port group etc. This level sits at vCenter->Inventory->Networking.

Low Level: This level is to connect your vm, vmkernel, and your local physical cards to vDS. Please be aware that your vm, vmkernel, etc are connecting to distribute port group. Unlike local vSS (you have create same vswitch, same vswitch port group on all hosts), vDS is pushed from vCenter to all Hosts. As long as you are connecting to same vDS, you will have same distribute port group.

With local physical nic card, they need to connect to dvUPlink side. You can choose any number of local nics to connect or even no nic at all. But what you can’t do is to setup teaming (only work for 2 nics from same host), traffic shaping, VLAN because you need to setup on high level.

To be continued. ……..