Skip navigation

Tag Archives: Vmware


As you guys may notice, I have spent some hours on vSphere vShield product recently. I have came cross a design flaw issue I would like to discuss with you.

First all, let me briefly describe my test environment.

I have two physical HP boxes and a EMC SAN as my test box. In this case, I have built a vCenter as VM sitting on one of ESX host. Therefore, I can even make snapshot if I want to. However, this has been generate some issues for vShield product.

Symptoms:

In terms of testing installing and configuring vShield product. I normally install vShield on one host and move some test VMs to new host to see how VMs respond. Then, I will vMotion vCenter VM to new host and install vShield on the second host since some of vShield components requires reboot host. I have done that couple of times. Eventually, it happened.

shissue-03

I initialled vMotion from a host which has zone, firewall, vApp to a host which doesn’t have those settings. vCenter got frozen.

I was waiting for couple of minutes but I was still not able to connect to vCenter. Not even pingable.

so I jump on new host with directly vClient and I found vCenter is up running in the new host. But it’s not pingable. Other VMs sitting in the same vSwitch are not having issues at all. I vMotioned vCenter before I install vShield without any issues. Why I can’t connect to vCenter VM this time?

Cause:

The reason is simple. It’s caused by vShield Zone and other components. Let’s take a look to see what happens when I vMotion a normal VM to a host installed with vShield.

shissue-01

 

The normal procedure should be:

  1. Query
  2. Migrate a new VM into new host.

 

However, as you can see from the picture, it actually reconfigured the VM afterwards.

Notice:

And  if you monitor vMotion ping status, the ping drop during vMotion from 1 time out become 10 times out depends on how you configure vShield.

shissue-02

 

so what exactly this reconfiguration step do?

The answer is that virtual machine vmx file has been reconfigured with vShield information. The more important thing is this step is done by vCenter!!

With a host installed with vShield products(like Zone), any VMs vMotion into that host will automatically configured with vZone. If vZone information is not configured, the VM will not able to communicate with other VM even if VMs in the same vSwitch because it’s caused at vNic leve.

Just imagine what happened if you try to vMotion a vCenter? No one is going to modify vCenter VM since it’s temporary disconnect from network!!

Solution:

I think this is a design flaw since use VM as vCenter is an option provided by VMware.

What I did was to use putty to connect to ESX host and manually modify vmx file of vCenter VM.

This is what old vmx looks like. This host has all vShield parts.

shissue-05

We need to remove filter0.name and param1 and add vEndpoint to match whatever new host got. The result is following.

shissue-04

After modification, the vCenter is able to start and connect to network.

Conclusion:

vShield is still a new product. VMware needs to resolve issues when vCenter in VM mode and let host , instead of vCenter, to reconfigure vmx files everytime a new VM vmotion into host or register a new VM.

Plus, the reconfiguration takes too long to finish. For important time sensitive machine, 10 time out may not be acceptable.


In my previous post, I described about vShield Endpoint. In this post, I will talk about the only real product which is actually using and design with this concept. Trend Micro Deep Security 7.5.

Before I started to roll out details, I would like to thank Trend Micro Australia’s help to give me support when I stuck. Thanks guys.

trenddp_08

What can Trend Micro Deep Security 7.5 do?

First time I saw this product is on the Vmware seminar. When Trend Micro representative standing on the stage and demonstrate how Deep Security can use only 20% of resource to scan in the virtualization environment.  That was mind blowing because imaging VDI and VMs are calling for schedule scan at same time. How much pressure it will cost to ESX Host? This product is only working with vSphere 4.1. It’s using vShield Endpoint and must use vShield point to do it’s job.   Well, at least, that’s what Trend Micro claimed. So is this true? Please continue to read.

Note: DS 7.5 is actually merely designed for VM environment. It means it’s not a complete solution at this stage. If you want to protect your physical boxes or workstation, you better still use OfficeScan product.

Deep Security provides comprehensive protection, including:

  • Anti-Malware (detect&clean virus)
  • Intrusion Detection and Prevention (IDS/IPS) and Firewall (malicious attack pattern protection)
  • Web Application Protection (malicious attack pattern protection)
  • Application Control (malicious attack pattern protection)
  • Integrity Monitoring (Registry & file modification trace)
  • Log Inspection (inspect logs and event on vm)

The interesting about DS 7.5 and vShield Endpoint is that none of this product can provide complete solution for end users. Each of them play a certain roles in the system. So the result is actually combination of both software.

Let’s take a look with clear table.

trenddp_09

Note:

My suggestion for installing is to install both vShield Endpoint Agent and DS Agent on your VMs. That’s the only way you can protect your VMs.

Components of Deep Security 7.5

Deep Security consists of the following set of components that work together to provide protection:

Deep Security Manager, the centralized management component which administrators use to configure security policy and deploy protection to enforcement components: Deep Security Virtual Appliance and Deep Security Agent. (You need to install it on one of windows server)

Deep Security Virtual Appliance is a security virtual machine built for VMware vSphere environments, that provides Anti-Malware, IDS/IPS, Firewall , Web Application Protection and Application Control protection. (It will be pushed from DS manager to each ESX)

Deep Security Agent is a security agent deployed directly on a computer which can provide IDS/IPS, Firewall, Web Application Protection, Application Control, Integrity Monitoring and Log Inspection protection. (It need to be installed on the protected VMs)

As matter of fact, you need to download following files from Trend Micro website. Don’t forget to download filter-driver which will be pushed from DS Manager to each ESX host.

trenddp_10

Architecture of Deep Security 7.5

Let’s take a look.

trenddp_02

There should be only have one DS manager unless you want to have redundancy.

ESX Host must be installed with vShield Endpoint.

Each ESX has it’s own Virtual appliance.

Each VM should have both vShield Endpoint and DS Agent installed.

How does Deep Security 7.5 work?

trenddp_16

For malware and virus check:

DS is using vShield Endpoint to monitor protected VM memory. The vSheild Endpoint Agent (or AKA vShield Endpoint thin driver) will open a special channel to allow DS virtual appliance to scan it’s memory via special vSwitch which is running on ESX kernel driver layer.

Since VMware needs to make sure the isolation of VMs traffic and memory, hard disk and no other application should breach this protection, vShield Endpoint is a back door opened by VMware to let third party to scan VM content legally and logically.

For registry keys and logs and other components of VM, we have to relay on DS Agent because vShield Endpoint can allow do so much. That’s why the solution must combine both vShield Endpint and DS agent.

Install Deep Security 7.5

I did encounter some interesting errors during the installation.

But let’s sort out the steps of installation first.

  1. Install Endpoint on your VMware ESXs.
  2. hostInstall DS manager on one of your windows box.
  3. Push Virtual Appliance, filter driver to each ESX host. It will add a appliance into vShield protected vSwitch. Filter driver will be loaded in the ESX kernel.
  4. Install DS agent, vShield Point Agent on VMs you want to protect.

Install Endpoint on your VMware ESXs.

Please click here to see how to do it.

Install DS manager on one of your windows box

Those are easy step. I believe any admin can do his job well.

Let’s me skip some easy parts.

trenddp_11

skip,skip

trenddp_12

Once you finish installation of DS Manager. You need to configure the DS Manager.

trenddp_13 trenddp_14

trenddp_15

This is really tricky part. What are those IP for?

The answer is those IP must not be occupied and it must be in the same subnet as rest of your vShield components are.

Check out this diagram and find out your own vShield  subnet.

On your ESX host(which has Endpoint installed already), you should find this.

trenddp_17

so what’s your vSheild Subnet?

The rest is easy part. skip,skip

trenddp_18

trenddp_19

Basic Configure DS Manager

By now, you have already connect to vCenter and vShield Manager. You suppose to see something like that.

trenddp_20

Notice nothing is actually managed and ready. That’s because you need to “Prepare ESX”.

Notice:

Before you “Prepare ESX”, you need to make sure vShield Endpoint has already installed and you have already download all DS components.

trenddp_21

trenddp_22

If you didn’t setup your vShield subnet correct, you will run into this error.

trenddp_23

In my case, I just need to right click vCenter->Properties-> Network Configuration

trenddp_24

please be aware you need to put your ESX into maintenance mode and restart it in terms of pushing DS virtual appliance and filter driver.

trenddp_25

You need to import your downloaded files into DS Manager. If you didn’t import before, you will have chance to import again or download.

trenddp_26

As usually, I skip some steps.

trenddp_27

trenddp_28

Here is another tricky. Because my ESX has different default IP as DS default. so once the DS Manager deploy the virtual appliance to ESX, the appliance only has default DHCP IP which is wrong in my case also the virtual network is also wrong. I encounter this problem.

trenddp_29

All what you need to do is to jump on ESX and virtual appliance console to change IP of that appliance. The default username and password is dsva.

trenddp_30

trenddp_31

Once you changed the IP, reboot this VM. Go back to DS Manager and double click dsva object to activate it.

trenddp_32

Make sure the security profile is loaded. That’s very important!!

trenddp_33

System will automatically offer you some VMs to protect. You can choose “no” at this stage. Why? because you haven’t installed vShield Endpoint agent and DS agent on your VMs yet.

trenddp_34

By now, the installation steps have finished here.

In my next post, I will talk about how to configure Trend Micro Deep Security 7.5 and performance result comparing with OfficeScan and virus testing.

Let me show you a picture what a DS manager look like when a VM is fully protected to finish this post.

trenddp_36

Reference:

Trend Micro Deep security installation guide

Trend Micro Deep security User guide


This is going to be a long post regarding vShield Endpoint and Trend Micro Deep Security 7.5. In this post, I will go through What is Endpoint, DP 7.5. How to install and basic configuration. How system work and performance comparison between two Trend products. Deep Security and OfficeScan.

Like what I said, this is going to be a long post. Let’s turn to Page one. 😉

In my past posts, I have describe what vShield is and different modules of vShield. You can find my previous post from here.

What is vShield Endpoint?

Let’s take a look what vShield is.

Strengthen security for virtual machines and their hosts while improving performance by orders of magnitude for endpoint protection, with VMware vShield Endpoint, part of the VMware vShield family. Offload antivirus and anti-malware processing to dedicated security-hardened virtual machines delivered by VMware partners. Leverage existing investments and manage antivirus and anti-malware policies for virtualized environments with the same management interfaces as physical environments.

  • Streamline and accelerate antivirus and anti-malware deployment
  • Improve virtual machine performance and eliminate antivirus and anti-malware bottlenecks
  • Reduce risk by eliminating agents susceptible to attack and enforce remediation more easily
  • Satisfy audit requirements with detailed logging of antivirus and anti-malware activities

This is what you can read from vmware.com. But what vShield Endpoint real does is a set of common interface or opening window to let third Party Anti-virus virtual appliance to scan/query memory of ESX host. If  you do remember what Vmware said about memory of each individual VM is secured separated for each VM. Well, vShield Endpoint is a back door to allow certain VM (like virtual appliance) to access all VMs memory at same time. As we all know, all information has to go through memory. Regardless it is opening ports or data saved on the virtual harddisk. However, it ain’t entire solution. As matter of fact, it can only do part of solutions. It can open window to AV appliance to scan memory, use firewall rule to deny unwanted access but it doesn’t understand registry key and logic structure of your servers.

How does vShield Endpoint work?

trenddp_03

The endpoint doesn’t have it’s own VM in the system unlike vApp and Edge. Well, in fact it does require a virtual appliance but it’s provided by third party.

Endpoint will install a special module in your ESX.

trenddp_01

This module will read data from protected VM and handled it to third party appliance to check virus/malware. This third party will sit in a secured vSwitch which will only be accessed by special module in ESX host. From protected VM angle, CPU usage is very low and memory utilization is low as well. The resource consumption has been transferred and reduced to AV appliance. But it doesn’t mean Hard disk are not used. We will discuss it in performance section.

What you need to do is to enable Endpoint on your host. Install Endpoint driver (or thin agent) on VMs you want to protect. Then, install third party appliance and everything will be fine.

How to install vShield Endpoint?

This procedure is similar as vEdge and vApp.

trenddp_04

trenddp_05

trenddp_06

Once you have install everything including Endpoint, and thirdparty of Antivirus. You will see something like this.

trenddp_07

Well, for more details, please wait for second post. I will review Trend Micro Deep Security 7.5 and how to install, configure.


First of all, I would like to apologize for updating my blog late since I was called away last week and not able to do too much.

I’m going to talk about vShield Edge and vApp. First of all, let’s review why we need vShield Edge. The last post can be found here.

What is vEdge?

vShield Edge is deployed as a virtual appliance to provide firewall,VPN, Web(HTTP only) load balancer, NAT, and DHCP services. Eliminate the need for VLANs by creating a barrier between the virtual machines protected by vShield Edge and the external network for port group isolation. Satisfy your network security within virtualized environments:

  • Consolidate edge security hardware: Provision edge security services, including firewall and VPN, using existing vSphere resources, eliminating the need for hardware-based solutions.
  • Ensure performance and availability of web services: Efficiently manage inbound web traffic across virtual machine clusters with web load balancing capabilities.
  • Accelerate IT compliance: Get increased visibility and control over security at the network edge, with the logging and auditing controls you need to demonstrate compliance with internal policies and external regulatory requirements.

Why do we need vEdge?

VMware is trying to design cloud system which can be used by ISP to host multiple Enterprise clouds on one datacenter.

vshield-edge01

VMware needs a cheap and efficient way to manage internal network to make sure the data between different clouds can be isolated from different network level but also be connected with well control. vEdge is used to allow you to isolate different cloud with NAT, load balance, DHCP and VPN.

Here is a good example for NAT using. There are two Test environment coexists in the same network because NAT function vEdge provides.

vshield-edge02

With vEdge, you can separate your Network tenancy into different connections without security breach or other threat.

vshield-edge03

Install vEdge

Installing vEdge is required to install license first. It’s the same location as you will do for others.

vshield-edge04

The next step is to choose which vSwitch (vSS or vDS) you want to deploy vEdge. Not like Zone which can be installed on vNic level, vEdge can be only setup on PortGroup.

vshield-edge05

All what you need to do is to choose a portgroup and click Edge menu on the right hand and provide information for vEdge VM and click to install.

vshield-edge06

Since vShield zone is base on Network crossing host, only one VM will be created and deployed by vShield Manager.  vSheild-Edge-DvPorgGroup can be migrated to other Host without any issues.

vshield-edge07

There is option when you install vEdge on Portgroup. It’s called Port Group Isolation.

You can prepare and install a port group isolation on vDS. It is an option for vEdge and it only works for vDS based vShield Edge. The port group Isolation creates a barrier between the protected VM and external network. Only NAT nuels or VLAN tags are configured.

At same time, a new vShield-PGI-dvSwitch will be created to handle traffic control. Each port group isolation will create a new VM.

Configuring vEdge

Everyone configures it differently. Please check out screen shots.

vshield-edge08

Firewall

vshield-edge09

NAT

vshield-edge10

DHCP

vshield-edge11

VPN

vshield-edge12

Load Balancer

Load Balancer is only for HTTP protocol at this stage. It’s designed for front web servers.

vshield-edge13

Few things to be aware:

  • At this day, vEdge can handdle 40,000 concurrent sessions.
  • You can make rules in the different layer, but new rules don’t apply to established sessions unless you manually apply it.
  • You can always create security groups as logical unit to manage your rules.
  • There is no package capture functions in vShield.
  • vEdge license can be included in Vmware View premium version.
  • vZone license can be included in vSphere Advanced.
  • vApp license can be included in vCloud director.

We will talk about vApp in next post.


This is second part of vShield. We will spend some time on vShield zone about Installation and configuration, of course, understanding as well.

Installation of vSheild Manager

Like what I have mentioned in the last post, vShield control module is vShield manager. And vShield Zone is it’s back bone which provides platform all other applications can run on it.

1.Download and Install

You can download evaluation version of vShield from Vmware as ova format. It’s a 500MB ova file and use vClient to deploy this ova into your vmware environment. You don’t need to worry about this vManager too much as it can be freely vMotion to any hosts in your cluster.

vshield-21

Once you imported the ova, you can file it up and use username “admin” and password as “default” to log in.

vshield-22

Type enable into cmd window and run setup

2. Configure IP and gateway.

 

vshield-23

You should be able to ping vManager.

3. Connect vManager with Internet Browser

vshield-24

vshield-25

4. Restart vClient and log in

After giving information to vManager, you should be able to see a new tab on vClient.

vshield-26

By now, vSheild Manager has been installed. But vZone or any other real vShield components haven’t been installed on any hosts. What you have done is merely a frame.

You can choose to configure all other aspects if you want.

vshield-27

Install vShield Zone

The next step is to install vShield Zone. vShield Zone is vShield App basic version. It shares same theory as vApp.

When you deploy vShield Zone from vManager, vManager will ask you to provide which host you want to install and a new set of IP for vShield Zone VM.

Each host will be bond with a new Linux VM and that VM will be fixed on that host and can’t be vMotion to other host since this VM will talk directly to a special module running in that host as same method of vSwitch.

In other word, that new VM will in charge all filtering jobs specific targeting on one host.

Notice: if you are running cluster, vShield Zone will only protect VMs running on host which as vZone installed. For example, you have host A, B. VM C,D. VM C running on host A and VM D running on host B. If you install vZone on Host B. Only VM D will be effected by vZone setting. If  you vMotion VM C from host A to B, then, VM C will be effected too.

vshield-38

However, if you are running a cluster (Host A, B), then by installing vZone on Host B won’t protect any VM until you install vZone on all Hosts in the cluster.

1. go to vShield tab and select a host to install

vshield-29

2. Provide a vZone VM IP set and Install

vshield-30

 

3.  System will deploy a new VM on that host

vshield-31

Apart from deploying a new VM, there are other couple of things this installing script has done.

  • Install a new module in the host.
  • Modify vmx belong to that host
  • Create a new vSwitch for firewall

 

Install a new module in the host

vshield-32

Modify vmx belong to that host

vshield-33

Create a new vSwitch for firewall

vshield-34

vshield-37

 

Let’s see a diagram and understand how it works at logic level.

vshield-28

All network traffic can be considered with a special detour before they reach to VM.

In the host level, we can use VMSafe diagram to understand since they share similar structure. It’s similar as VMsafe Net but it use it’s own filter (vShiled-dvfilter).

vshield-35

 

 

Management of vZone

vZone management is very similar as ISA. It has divided into multiple levels.

Hierarchy of Zones Firewall Rules
Each vShield Zones instance enforces Zones Firewall rules in top-to-bottom ordering. A vShield Zones instance checks each traffic session against the top rule in the Zones Firewall table before moving down thesubsequent rules in the table. The first rule in the table that matches the traffic parameters is enforced.
Zones Firewall rules are enforced in the following hierarchy:
1. Data Center High Precedence Rules
2. Cluster Level Rules
3. Data Center Low Precedence Rules (seen as Rules below this level have lower precedence than cluster level rules when a datacenter resource is selected)
4. Secure Port Group Rules
5. Default Rules
vshield-36

 

Few things you need to know:

1. Make sure vManager, vZone VM are all pingable to each other.

2. If you are using cluster, make sure all hosts are installed vZone.

3. If you try to uninstall vZone, a restart of host is involved!!

4. No restart involved when you install vZone on host.

5. vZone VM can’t be montioned.

6. How much overhead will be consumed by vShield in prod is unknown.

7. How much impact on network traffic by vShield is unknown.

Reference:

vShield Administration Guide


Here we go. This is another big chunk of Vmware technology. I should start this article long time ago, but I am always got carried away. Therefore, I have decided to discuss this topic in couple of posts(it’s big, isn’t it). Due to not too much information around, I will do my best to explain what I have learned and understand. If I made mistakes, please let me know. Thanks

Why do we need vShield?

Before we start to explain how and what, we need to understand why VMWARE makes this product. It’s all about vCloud. Vmware ambition is focusing holding multiple company infrastructures into a virtual Datacenter. In other word, a vDC needs to hold up different companies private clouds and hybrid clouds. Hence, Vmware need a product to isolate vClouds and acting as either internal firewall(isolation) and gateway between datacenter and Enterprise private cloud. Plus with a neutral anti-virus system which will scan the VMs without causing any performance and confidential information leaking issues. Hence that’s why vShield is a must have software with vCloud Director.

Family members of VMware vShield

VMvShield has different parts for different reasons. Let’s take a look.

vshield-01

At first glance, vEdge, vZone,vApp, vEndpoint and even manager are look so similar. That’s where  you start to get headache. The strategy of Vmware is clear. VMware give you different appliance (ova file) and you install them into your vmware platform and running them as just normal Linux VMs. Each linux VM will start to install components into ESX hosts and change vm configuration file in terms of let module running on host work or effect.

VMware vShield Manager:

Why do I introduce this part first? Because this part is back bone of whole vShield products. It installed a new tab into  your vSphere Client and allow you to manage entire vShield family. It’s base on linux and support SSH, WEB console, vSphere Client and REST API, most of importantly, it generate other components of vShield to install. If you got on VMware website, you can download this ova file.

This Open Virtualization Archive (OVA) file includes vShield Manager, vShield Edge, vShield App and vShield Endpoint. vShield App, Endpoint, and Edge components are managed by vShield Manager. The minimum requirement for vShield products are vSphere 4.0 U1 (Essentials Plus and above), vCenter 4.0 and vSphere Client 4.1. Only vShield Endpoint requires vSphere 4.1.

Please be aware: vSphere Manager VM is vMotionable.

VMware vShield Zones:

vshield-05

This is basic firewall product and vShield App is upgrade version of Zones. The vSheild Manage will generate a customized ova file (according to your answers on the wizard) and install it on host you want vZones on. Each It is loaded to each ESX/ESXi host as part of kernal module and it create its own vSwitch to filter the traffic. Please be aware each Host will have it’s own Linux vm running as VMware Zone VM and it can’t be vMotioned! You may have to manually power off if you want to enter maintenance mode.

vshield-03

Notice: As you see from the picture, each host will have their own vZone VM.

vshield-02

Error I got when I tried to vMotion vZone VM.

 

Well, it’s is firewall after all. It does have same infrastructure as vShield App but it can’t App work due to license issue.  According to Vmware site,

Get basic firewalling of traffic between virtual machines with vShield Zones, allowing for connections to be filtered and grouped based on the 5-tuple – source IP address, destination IP address, source port, destination port, protocol. Depending on how services are virtualized, this may be sufficient for security policies that do not require much granularity.

so what it didn’t do? Let’s check out vShield Apps

Vmware vShield App:

vshield-04

Here it is. The advanced version of firewall for internal protection purpose. It’s not only do what vZone does, it can understand traffic at application level.

Because vShield is working on logic concept to group VMs. Therefore, you can group VMs by function, department or organizational need instead of just IP or VLAN which is the part vShield try to avoid to use. In the traditional infrastructure, Internal firewall can’t only use VLAN to isolate VMs in the cluster. Now, you have much more options and power.

 

VMware vShield Edge:

vshield-06

This is purely design for vDC to holding different private clouds in their platform. If we consider vSheild app is for internal Firewall, then vEdge is for external firewall.

Get essential security capabilities including port group isolation, network security gateway services and web load balancing for performance and availability. vShield Edge is deployed as a virtual appliance to provide firewall,VPN, Web load balancer, NAT, and DHCP services. Eliminate the need for VLANs by creating a barrier between the virtual machines protected by vShield Edge and the external network for port group isolation.

VMvSheild Endpoint

vshield-07

This is Vmware cloud base anti-virus solution. It’s designed for Cloud base and VDI base. There are lots of details and pictures I would like to show you. But let’s just take a brief concept first. What it can do.

 

Offload key antivirus and anti-malware functions to a hardened, tamperproof security virtual machine, eliminating agent footprint. The robust and secure hypervisor introspection capabilities in vSphere prevent compromise of the antivirus and anti-malware service. vShield Endpoint plugs directly into vSphere and consists of hardened security virtual machine (delivered by VMware partners), a driver for virtual machines to offload file events, and the VMware Endpoint security (EPSEC) loadable kernel module (LKM) to link the first two components at the hypervisor layer.

 

Like what I mentioned from beginning, it’s big topic. In the next post, I will break down vShield into small piece. Let’s see how it goes.

 

 

Reference:

 

What is REST API?

http://searchsoa.techtarget.com/sDefinition/0,,sid26_gci823682,00.html


This is always interesting topic about using 1 core in VM most likely get better performance comparing with using 4 cores, not mention 8 cores. However, there are cases you want to use 8 cores vCPUs. I have recently experienced this real case and I would like to share it to you.

Why do we need to have multiple cores in VM?

Well, first of all, let me introduce our environment to you. We are using Dynamics AX 2009 and recently are conducting MRP model Test. MRP model requires to run batch jobs which could take up to 7 hours to finish on single core VM. The database of Dynamics AX 2009 is on our SQL box but , with batch job, most of them are CPU work and it runs on a VM.

As what I mentioned above, with single core (Dynamics AX 2009 MRP natively only run one thread even on multiple cores machine), the time of finishing batch job is unacceptable in real world. Therefore, Microsoft develops “helper” to assist. Each helper suppose to represent a core. It means, if I run batch job on 4 core VM, I need to setup 3 helpers (plus original one thread to make it 4).

Microsoft is not recommending to run batch jobs on VM (because their hyper-v sucks? 😉 ) but I’m pretty happy to put it into test. Before you continue to read on, I have to remind you MRP helpers are very new to this world. It is far from perfect….. yes, far far from………

My test Environment:

SQL: SQL 2005 with latest patch running on physical box

VM: Windows 2003 Standard 32bit

ESX Host software: ESXi, 4.1.0 260247 with Evaluation license

There is only one VM running on that ESX Host.

ESX Host hardware: HP Proliant DL380 G5, 2 Quad Core X5460, 16GB mem.

Storage: SAN, EMC CX3

Tools involved: Performance Monitor on  Windows, ESXtop, vMA 4.1, FASTSCP,Excel,ESXplot

Number of core: 4, 8 (each Test involves different number of cores)

Single Core Test

This is a Test running without any helpers and distributions. It means batch server is running single thread on 4 cores VM. Distribution is number of job list. In theory, number of distribution should equal number of cores.

First Test, bench mark test

Test Num distribution Helpers Job Name Running time
1 0 0 FP20 260 min

 

Batch VM Performance (the performance monitor is setup as 8 cores, but VM only has 4 cores)

4core_01

From this picture, you can see only one core has been used. It’s about 38% utlization.

Line graphic of VM CPU

4core_03

HOST status

4core_02

This is result I got from esxtop. This is total CPU loader status. Since we are using VM, so single virtual core job is distributed to 8 physical cores. It runs about 13% of physical CPU resources. This is utilization of pCPU which include pCPU over head.

Test 2 with 3 helpers and 4 distributions

Test Num distribution Helpers Job Name Running time
1 0 0 FP20 260 min
2 4 3 FP20 207min

 

Notice we are using much less time in this test!! The new test is only using 79% time of single thread.

4core_04

This is 4 cores VM. Notice the blue core utilization is very low. It’s possible that windows reserve one core for it’s OS. All cores were utilized very low!

However, as what I said, the helpers are very new for MRP. So it’s very poor coded. Let’s see what each vCPU has done during the time.

vCPU0

4core_05

Notice there are time vCPU0 was very low utilized..

vCPU1

4core_06

vCPU2

4core_07

My best guess is this is reserved core for OS.

4core_08

Poor coding….

Let’s check out HOST CPU

4core_09

Notice that physical CPU usage is actually higher than single thread.

Test 6 with 8 cores VM and 12 helpers, 6 distribution

I did some other tests with 8 cores. I setup vm with 8 cores and lots of helpers and distributions. As you can see, the running time is shorted again. But as what I said, due to poor coding, it’s not always effective as I expected.

Test Num distribution Helpers Job Name Running time
5 5 7 FP20 180min
6 6 12 FP20 168min

 

4core_10

None of cores are running more than 40%. Still, it’s coding issue.

vCPU0

4core_11

vCPU1

4core_12

The problem of this poor coding is it doesn’t use all cores in all the time. There are lots of time only few cores are used.

vCPU2,4,6

4core_13

There 3 cores running in this shape. It’s pretty pity resources are wasted.

vCPU3,7

4core_14

 

This is not bad usage.

ESX Host CPU

4core_15

As you can see, the maximum usage has reached to 40%. but for the rest period, the usage dropped due to few cores were used.

Let’s see what a single physical CPU doing on ESX host

pCPU07

4core_16

There are lots of up and down and spikes due to distribution by ESX layer.

 

Conclusion of this Test:

1. Dynamics AX batch server can run on VM. As matter of fact, it works pretty good with current MRP helpers patch. You can load up with other VMs to utilize more CPU resources.

2. 8 cores does help a lot in this case. Since all cores were only used less than 40%! Thank God we are using virtualization layer and all virtual cpu jobs are distributed to physical CPUs.

Leave ur comments. 😉


I do understand that there are quite few articles around talking about how to enable multiple cores. But I did encounter few issues when I tried to do it by myself. Therefore, I write this down just as reference.

Why do we need to have Multiple cores in your VM?

The reason is simple. Microsoft Windows only support limited number of processors on their OS.

 

Win OS Version Max Processor
Win2k3/2k8 Standard 4 CPUs
Win2k3/2k8 Enterprise 8 CPUs
Windows 2003 DataCen 32bit 32 CPUs
Windows 2003 DataCen 64bit 64 CPUs
Windows 2008 DataCen 64bit 64 CPUs
Win 2k8 R2 64 physical CPU or 256 logical CPU

 

Let’s say, if you are running a Dynamics AX AOS server, it is required to run multiple cores to do your batch jobs. With AOS server, you can setup number of helpers to work with distributed batch jobs. The maximum number of helpers (number of supporting cores) is 21. If I run AOS on Windows 2003 Standard, I need to enable multicores so a Win2k3 standard can run 8 cores to help me improve calculating.

How do we enable multicore on VM?

You need to do that in vm 7 hardware which means it should be vSphere 4.x version. (I was told it could work on ESX 3.5, but I haven’t tried yet). Most important and difficult condition is you need a Enterprise Plus license to let VM to support 8 vCPUs.

You can work around by installing a new ESXi host(not your vCenter since CPU license is related to Host license)

Let’s say if  you want to have your vm to have 4 vCPUs and 2 cores on each vCPU. Here is a trick. You need to give this VM 8 vCPUs(4vCPU x 2 cores= 8vCPUs) instead of 4 vCPUs.

mulcor-01

This will give VM 8 vCPU all together. Now you need to setup 2 cores for each vCPU. In terms of doing that, we need to add a special Configuration Parameters. Turn off you VM->Edit VM Setting

mulcor-02

Manually add this new row into your sheet.

mulcor-03

 

Then, you turn on VM.

How to verify I’m using multicores?

Well, the basic way is to open Task Manager and Device Manager.

mulcor-04

Please be notice that the above picture is from another example. It should show up with 8 cores but it only shows 4 cores because that’s another machine. I will replace picture later.

There is another way to see number of cores if you running windows 2008 above.

Get into your machine->cmd->wmic->cpu

mulcor-05

Move your scroll bar to middle. Here is just example. Again, it’s not related to my test machine. And you need to remember this method only works on Win2k8. For W2k3, you need other third party tools.

mulcor-06

 

Possible problems you may run into

Like what I said before, you need to have Enterprise Plus license or Evaluation license on host. Otherwise, you will be pleasure to see this error.

mulcor-07

 

What difference between multicores and multiCPUs?

The answer is, from performance wise, nothing. It’s just different way to call it. VM still pass on the CPU cycle to host and process it. The best practise is if you want to have 8 vCPU, you can either have 4 vCPU x 2 cores or 2 vCPU x 4 cores.

It’s total you choose and your call.

 

Reference:

Vmware KB Article: 1010184


Ok. This is second part of EMC Celerra Unified, CLARiiON storage with Vmware – Deduplication and Compression Part 1. In my last post, I explained how EMC block compression works and where it will applies. In this post, I’m going to talk about another important part of EMC Celerra Unified storage technology. Celerra file level Deduplication& compression. AKA, Celerra Data deduplication.

Considering Celerra Unified storage is combination of Celerra (NAS) and CALRiiON (block storage, FC). The essential Celerra is actually focusing on NAS parts. It is not only has compression function(but it’s file level), but also has file level deduplication. Let’s talk about how it works.

The sequence of Celerra Data Deduplication is to look for cold data, compress first, then deduplicate later.

Please be aware although NAS is basically a file system. but Celerra is not exactly operating on file leve.  Let’s see how it works.

Celerra Compression

If you check out latest version of Celerra docs, you may notice Celerra compression is not longer a file compression. It’s actually just labeled a “compression”.

Plus, EMC used to claim that do not use deduplication on NFS for Vmware, but recentely, EMC just released the Celerra Vmware Plug-in which allow  you to compress single VM. First of all, let’s talk about “Initial compression and deduplication”

Initial and scheduled compression & deduplication

In the Celerra, the compression and deduplication  is done together. Celerra periodically scan file system and compress “cold” (not recently active files) and hash them into metadatabase. If there are same hash exist in the database, this file is deduplicated.

So the process is:

Compress->hash->compare meta database->Copy to hidden space of this LUN (or not)->Update meta database.

Reading Compressed file

Compress Read is not much difference from EMC CLARiiON read. Celerra directly load compressed data into memory and extract data in the memory. Nothing is writing back to disk. Therefore, in some cases, reading compression data is even faster than uncompression data but with some CPU cycles cost.

Writing to a compressed & deduplicated file

Writing a compressed file is a long procedure because it involves writing to disk. A write to or a modification of a deduplicated file cause a copy of the requested portion of the file data to be reduplicated for this particular instance while preserving the deduplicated data for the remaining references to this file.

The entire file is not decompressed and reduplicated on the disk until the sum of the number of individual changed blocks and the number of blocks in the corresponding deduplicated file are larger than the logical file size.

Compression with Vmware environment

Celerra Vmware Plug-in just be released not long ago. It can only work on NAS file system. It has “Thin provisioning”, “Fast/Full Clone”, “Compression/Decompression” features. Just like CLARiiON system, you can use this plug-in to off load those operations from host to SAN. One of operation is compression. It allows you to compress a VM regardless it’s think or thin disk. VMs that are selected for storage optimization are compressed by using the same compression algorithm that is used in Celerra Data Deduplication. The algorithm comresses the VMDK file (virtual disk) and leaves intact other files that make up the VMs. When a compressed VMDK is read, only the data blocks containing the actual data are decompressed. Likewise, when data is written to the VMDK, instead of compressing it “on the fly”, data is written to a set-aside buffer, which then gets compressed as a background task. In most situations, the amount of data read or written is a small percentage of total amount of data stored.  VM performance will typically be impacted maginally (less than 10%) when the corrsponding VMDK file is compressed.

I still think there are plenty of things EMC can do. Celerra plug – in is just a beginning. I will keep eye on it and post more later.


First of all, I need to point out that I ain’t work for EMC nor VMWARE. I won’t do anything like Chad in Virtual Geek to say, “Yes, it’s true. EMC really IS #1 for VMware.” What I’m going to to talk about is purely from a customer point of view. A tech who doesn’t favorite either EMC nor Netapps. I just want to draw a picture of EMC storage and Vmware related technologies in front of you so we can put everything on the table and discuss it.

As usual, any comments and discussions are very welcome here.

EMC Storage – Celerra, CLARiiON and Celerra Unifed Storage

Well, back in years ago, There are three production lines in the EMC storage. Celerra, CLARiiON and Symmetrix. After years of Virtualization development, the line between Celerra and CLARiiON is getting really blurry. Celerra used to dedicate to NAS file system. It provides NAS only. With CLARiiON(like CX3 series), it mainly focus on Block level storage like FC or iSCSI. Now, with new product line Celerra Unified storage coming out, I don’t really think anyone would buy old system any more. Because new Unified Storage provides both Celerra & CLARiiON in one box. EMC call them block enabled Celerra system (NS-120,480,960,etc). However, as you may know the technology using in NAS are quite different from technology using in the block storage. If you are as confused as me, please read rest of article and hope it can help you clear your mind. This article is focusing on Celerra Unified storage only.

EMC Deduplication and Compression

As everyone knows, one of key elements of storage is disk capacity. How to utilize disk space and tier down unused data and files and compress them to small space becomes the major reason when we select Storage. Even after years and years research, EMC still insists deduplication should only happens on file level instead of block level. so what does that mean to customers who bought Celerra Unified Storage? It means you can only use block compression if you use block storage (like FC, iSCSI, this is as part of CLARiiON technology) and you can only use file level Compression and file level deduplication when you use NAS (as part of Celerra, if you use NFS for VM or NFS,CIFS for file systems). In other word, how  you divide your LUNs and what kind of block or file system you use will dramatically impact your system. Let’s break down those technology and see what they are.

EMC compression

As what I mentioned above, depends on what kind of system (NAS or block) you use, Celerra will use different ways to compress the data. Let’s talk about block level compression first.

Block level compression

As what this name indicates, this compression should only work on FC or iSCSI LUNs. The block size compression works on is 64KB. Each block is processed independtly. The typical result of compression is as much as 2x while it use modest CPU resource.

Note:

In default, CX4-480 can have 8 concurrent threads on compression. When all threads running at same time, the consumption of CPU will be compression rate(speed) as Low (15% CPU), medium (30~50% CPU) and high(60%~80% CPU).

How block compression works

1.Initial compression- This occurs when you enable compression. It compress entire LUN and can’t be paused in the middle. But it can be disabled during the procedure. No damage will be done.

2. Compression of new data – When new data is written, it is written uncompressed and compressed asynchronously. It keeps doing that until you disable compression. In default, when 10% of user capacity of LUN or 10GB new data are written, and total amount new data is larger than 1GB, compress starts. It does use some of SP cache memory for swapping. When you compress a LUN, that LUN will automatically migrate to a thin LUN in different pool if LUN is a normal RAID lun. If it’s a think LUN, it will reminds in the same pool.

3. Decompression when compression is disabled- if the original LUN is a thin LUN, it will reminds thin LUN. If the original LUN is a thick LUN or RAID group LUN, it will write zeros to unallocated capacity till full while it reminds a thin LUN. System will pause at 90% and stop at 98% if the LUN has filled up too much.

Limits of block compression

  • The following cannot be compressed:
  • Private LUNs (including write intent logs, clone private LUNs, reserved LUNs, meta LUNs, and component LUNs)
  • Snapshot LUNs
  • Celerra iSCSI or file system LUNs ( Personally, I don’t think that’s right. I’m confirming with EMC now)
  • A LUN is already being migrated and expanding or shrinking.
  • A mirrored LUN replicating to a storage system running pre-29 FLARE code.

Interactive of compression with other functionalities

Basically, a compressed LUN is transparent to other operations like replication or migration. But by saying that, it’s better not migrating or copying while compress at same time. It’s always easy for SAN to enable compress after migration.

How to setup compression?

All what you need to do is to connect Celerra Unified storage control station with your Internet Browser. You will have Unisphere Manager running directly from SAN or you can install Unified Manager on a windows server and connect to your box. Compression function is a licensed feature and you should have it directly from console. Unlike Celerra NAS part, there is no VMWARE plug-in availabe for compression so you need to use Unisphere to do the job.

There is no Vmware plug-in?

It is very interesting that Celerra Unified NAS part got a vmware plug-in while CLARiiON reminds nothing. I reckon vSphere may use VAAI API to offload clone from host to SAN but why it doesn’t work for Celerra NAS part? If anyone can answer this question, it will be appreciated.

To be continued……

Reference: