Skip navigation

Tag Archives: Architecture

As usually, I would thank you for continuing browsing my blog although I haven’t uploaded for couple of months. I was caught by my personal errands till, today, one of my friends said, “Silver, why don’t you update your blog? Even just write some nonsense into it”.

Well, personally, I don’t write any useless information in this tech blog. But I do need to update. So here it is. Hope you can enjoy it.

I will show you how to configure VMware Orchestrator. This software is coming with vSphere but it is installed silently and you need to manually configure it. Reason to use VMware Orchestrator will be 2.

A. You have very large and complex Vmware environment and you would like to dig deep and become guru.

B. You need to prepare for VCAP-DCA exam.

Regardless which reason you may have, this post will give you a hand and knock the door for you.


Configure VMware Orchestrator

The first thing you need to do is to check out Service “VMware vCenter Orchestrator Configuration” is running. In default, it is manual for start up.


Once you started the Orchestrator Configuration, you can just run “Configuration”



You should see this page coming from IE.



The default username and password is vmware/vmware.

You should see main interface like this.




There is nothing you need to configure in the General class for now.



so Let’s jump on “Network”. the network configuration is for Orchestrator. So You need to put IP and DNS and keep settings. No drama on that.

Notice “SSL Certificate” page here, but we don’t configure it for now. You can choose to use CA certificate or your own certificate. In this case, we will generate Orchestrator own certificate first, then we can configure it. Please see Chapter “Server Certificate” below.


The purpose of LDAP is to let you use AD account to log in to Orchestrator client.

You need fill those blank with your DC servers, and LDAP path.


For the root and other group path information, you don’t need to run some scripts to get it. All what you need to do is to run AD Users and computers.

Right click the object (for example, the root of your AD) and click –> “Properties” and go to Attribute Editor and find distinguishedName as follow.



Same thing for the rest of page.






It’s pretty straight forward for configuring database. I’m using SQL database and



Once you install the database, jump on SQL server and verify it.



Server Certificate:

You should generate your own Server certificate here. For some reason, the certificate generate by my Domain CA doesn’t work well here. so I would suggest you do it by yourself.


Once you generate certificate, you need to export it to a file protected with password.


The next step supposes to import certificate back to “Network –>SSL configuration”. If you don’t that, you won’t get “License” right.



This is where you gain license from vCenter and also license for plug-ins.


If you don’t import SSL certificate here, you won’t get right result. Because we need to use secure channel.


You need to import your license which you export above.

This will also setup Network configuration->SSL part as well.


please be noticed:  You may need to restart vCenter to let license work!!

Start up option:

You must make sure the status is “Running”. I was stuck at “Unknown” status for a very long time even after I restart vCenter and Orchestrator services and server. The only way to resolve it is to click those “Restart” buttons in this page. Trust me, they are here for reasons.


The rest parts are very easy to configure. I just paste picture here as guide.












This is for connection to your hosts.


vCenter Server:

This is where you configure your vCenter.



once you finished the configuration, you can get into Orchestrator now via it’s own client. Run it under Vmware you shall see this interface.




There are some tricks to setup Orchestrator. But the difficult part is actually to use it since there are lack of good examples and documents.

I would suggest VMworld Orchestrator Lab manual is a very good start. If you do want to know me to give you some examples, please leave your messages.


So this is last part of this series. Hopefully, I don’t need to write another post.

From previous post, I discussed about how to install and configure Trend Deep Security 7.5 on vSheild. This post will talk little bit more about configuration and performance review.

In my last post, I have installed vShield Zone on host, Install DS Manager one of my VMs which is also vCenter, and push DS Virtual Appliance on to one of hosts.

Then, I changed the IP and network configuration on the DS VA and activate it with Deep Security Virtual Appliance.

Please be aware that Security Policy is playing an important role in the DS. You need to make sure all protected VMs having correct Security Policy.

Once you finished the VA, we can go back to DS manager and take a quick look.

I would like to list some common issues you may encounter.


If anti-Malware status is not Capable, it means vEndpoint is not installed on this ESX host.


If Anti-Malware is on, but the color is blue. It means you haven’t assigned correct policy on this VM. In default, there is no policy at all. Just right click the VM and follow the instruction.



You better actually create your own policy before you apply. Some default policy(like windows 2k3) doesn’t have all protection on and doesn’t allow certain protocol (e.g: RDP). The best way is to make copy of old policy and customize a new one for yourself.

The next step is to prepare your VMs. All what you need to do is to install vShield Driver agent and DS Agent. Once you finish installation, you must reactivate your vm from DS Manager to let DS Manager to check VM status.


If you have installed both agents and apply right policy, reactivate your vm from DS Manager. You should see something like this in the DS Manager.


It should have all greens and Agent should running. Your VM should be protected at each level from crossing both Appliance(working with vEndpoint) and Agent.

One more thing when you try to install DS Agent, you need to copy the installation on local disk of VM and install. Otherwise, you will encounter this error.


Virus download test

I have a protected VM which has all features turned on. Let’s see how it react when I tried to download a virus sample file from Internet.


It actually worked!

Does Deep Security actually reduce resource consumption?

Here is the big question. The reason we spent so much time to deploy this product is the rumour that it can save the resource comparing with traditional AV solution. Let’s take a look.

I installed OfficeScan on one of test machines. I monitored the resource which has been consumed from CPU, Memory,DISK,Network for both test VM and Host as base line. I will scan a vm with officescan once. And also scan it with DS.

Protected VM CPU

Protected VM CPU with OfficeScan


CPU: 50% of one core. It lasts 10 mins.

Protected VM CPU with DS



only 22% on CPU comparing with 50% on Office Scan.

Note: I ran twice on this test.

Protected VM DISK

Protected VM disk with OfficeScan


Disk: 5000KBps for 10 mins.

Protected VM disk with DS:ds-15

It’s very interesting to see the first run disk but nothing on second. The reason is the first run has already load disk data into memory and it doesn’t require to load again at second time. It proves DS is load to memory and scan only memory theory. The DS scan finished in 4.5 mins.

Protected VM Memory

Protected VM with OfficeScan


Memory: Consumed memory is 1.25GB, and active memory is 4GB.

Protected VM with DS


50% of active memory in 4.5 mins. I ran twice.

Protected VM Network

Protected VM with OfficeScan


Network: OfficeScan tried to contact OfficeScan server at beginning. Then, it went quiet.

Protected VM Network Activity with DS:


There is almost nothing on network. It means DS is using ESX module to scan memory directly. It doesn’t go through normal network channel. Because it is using similar theory as vSwitch, I call it a protected vSwitch channel.

From what I can see via Protected VM angle, the resource has been consumed almost 50% less and use only half time to finish scan.

Because using DA actually involves to use Deep Security Virtual Appliance to scan. We need to take look about DS VA.



The truth behind scene is DS VA is actually scanning the data instead of protected VM. That’s why you see low utilization on VM because all what it did was to load data into memory and call vShield Endpoint driver to let DS VA to scan.

DS VA Disk:


Almost nothing on disk VA disk activity.

DS VA Memory:


It consume 1.5GB memory on VA. It’s understandable.

DS VA Network:


This is very interesting. According to this chart, the network activity on DS VA is very high during scanning. It means vShield Endpoint will open port for all VMs sitting on that protected vSwitch instead of just DS VA.


This is the vSwitch vShield Endpoint use. It’s just normal vSwith and you can add adapters if you want. It does bring my concern whether this could be potential security breach.

Here is moment of truth. Will DS actually save resource from ESX perspective?

Following is the data from Physical ESX Host:

ESX CPU utilization

ESX CPU with OfficeScan


4% of total CPUs on ESX box.  I have nothing else was running on that host.

ESX Host CPU Performance on DS


It does finish scan in half time but it actually use 6% of CPUs. Be aware this is not including overhead of ESX host CPU. It’s 2% of higher than OfficeScan.

ESX Disk with OfficeScan


Disk activity on ESX host.

ESX Disk activity with DS


It’s same disk activity but with half loading time.

There ain’t much point to check memory since everything is happening in the memory. Just one module to scan another chunk of memory in the host. That’s all.


Let’s sum up with what we have learned from those data. Please be aware I’m only test single machine scan.

Resource consumption:

ESX Host

OfficeScan DS 7.5
CPU Util 4% 6%
CPU Used time 10 mins 4.5 mins
DISK Util 200CMD/s 200CMD/s
DISK Used time 10 mins 4.5 mins
Memory Same Same
Network 0 0 Nothing on pNIC

It does seem like Host CPU is consumed more resource than officeScan.

but It seems that DS VA doesn’t support multiple threads scanning at same time. If that’s the case, a host can hold about 30 VMs max. So DS Manager will schedule to scan all machines in different time.

This is the end of this Session of this year!

I wish everyone has a wonderful Christmas and Happy New Year!!



In my previous post, I described about vShield Endpoint. In this post, I will talk about the only real product which is actually using and design with this concept. Trend Micro Deep Security 7.5.

Before I started to roll out details, I would like to thank Trend Micro Australia’s help to give me support when I stuck. Thanks guys.


What can Trend Micro Deep Security 7.5 do?

First time I saw this product is on the Vmware seminar. When Trend Micro representative standing on the stage and demonstrate how Deep Security can use only 20% of resource to scan in the virtualization environment.  That was mind blowing because imaging VDI and VMs are calling for schedule scan at same time. How much pressure it will cost to ESX Host? This product is only working with vSphere 4.1. It’s using vShield Endpoint and must use vShield point to do it’s job.   Well, at least, that’s what Trend Micro claimed. So is this true? Please continue to read.

Note: DS 7.5 is actually merely designed for VM environment. It means it’s not a complete solution at this stage. If you want to protect your physical boxes or workstation, you better still use OfficeScan product.

Deep Security provides comprehensive protection, including:

  • Anti-Malware (detect&clean virus)
  • Intrusion Detection and Prevention (IDS/IPS) and Firewall (malicious attack pattern protection)
  • Web Application Protection (malicious attack pattern protection)
  • Application Control (malicious attack pattern protection)
  • Integrity Monitoring (Registry & file modification trace)
  • Log Inspection (inspect logs and event on vm)

The interesting about DS 7.5 and vShield Endpoint is that none of this product can provide complete solution for end users. Each of them play a certain roles in the system. So the result is actually combination of both software.

Let’s take a look with clear table.



My suggestion for installing is to install both vShield Endpoint Agent and DS Agent on your VMs. That’s the only way you can protect your VMs.

Components of Deep Security 7.5

Deep Security consists of the following set of components that work together to provide protection:

Deep Security Manager, the centralized management component which administrators use to configure security policy and deploy protection to enforcement components: Deep Security Virtual Appliance and Deep Security Agent. (You need to install it on one of windows server)

Deep Security Virtual Appliance is a security virtual machine built for VMware vSphere environments, that provides Anti-Malware, IDS/IPS, Firewall , Web Application Protection and Application Control protection. (It will be pushed from DS manager to each ESX)

Deep Security Agent is a security agent deployed directly on a computer which can provide IDS/IPS, Firewall, Web Application Protection, Application Control, Integrity Monitoring and Log Inspection protection. (It need to be installed on the protected VMs)

As matter of fact, you need to download following files from Trend Micro website. Don’t forget to download filter-driver which will be pushed from DS Manager to each ESX host.


Architecture of Deep Security 7.5

Let’s take a look.


There should be only have one DS manager unless you want to have redundancy.

ESX Host must be installed with vShield Endpoint.

Each ESX has it’s own Virtual appliance.

Each VM should have both vShield Endpoint and DS Agent installed.

How does Deep Security 7.5 work?


For malware and virus check:

DS is using vShield Endpoint to monitor protected VM memory. The vSheild Endpoint Agent (or AKA vShield Endpoint thin driver) will open a special channel to allow DS virtual appliance to scan it’s memory via special vSwitch which is running on ESX kernel driver layer.

Since VMware needs to make sure the isolation of VMs traffic and memory, hard disk and no other application should breach this protection, vShield Endpoint is a back door opened by VMware to let third party to scan VM content legally and logically.

For registry keys and logs and other components of VM, we have to relay on DS Agent because vShield Endpoint can allow do so much. That’s why the solution must combine both vShield Endpint and DS agent.

Install Deep Security 7.5

I did encounter some interesting errors during the installation.

But let’s sort out the steps of installation first.

  1. Install Endpoint on your VMware ESXs.
  2. hostInstall DS manager on one of your windows box.
  3. Push Virtual Appliance, filter driver to each ESX host. It will add a appliance into vShield protected vSwitch. Filter driver will be loaded in the ESX kernel.
  4. Install DS agent, vShield Point Agent on VMs you want to protect.

Install Endpoint on your VMware ESXs.

Please click here to see how to do it.

Install DS manager on one of your windows box

Those are easy step. I believe any admin can do his job well.

Let’s me skip some easy parts.




Once you finish installation of DS Manager. You need to configure the DS Manager.

trenddp_13 trenddp_14


This is really tricky part. What are those IP for?

The answer is those IP must not be occupied and it must be in the same subnet as rest of your vShield components are.

Check out this diagram and find out your own vShield  subnet.

On your ESX host(which has Endpoint installed already), you should find this.


so what’s your vSheild Subnet?

The rest is easy part. skip,skip



Basic Configure DS Manager

By now, you have already connect to vCenter and vShield Manager. You suppose to see something like that.


Notice nothing is actually managed and ready. That’s because you need to “Prepare ESX”.


Before you “Prepare ESX”, you need to make sure vShield Endpoint has already installed and you have already download all DS components.



If you didn’t setup your vShield subnet correct, you will run into this error.


In my case, I just need to right click vCenter->Properties-> Network Configuration


please be aware you need to put your ESX into maintenance mode and restart it in terms of pushing DS virtual appliance and filter driver.


You need to import your downloaded files into DS Manager. If you didn’t import before, you will have chance to import again or download.


As usually, I skip some steps.



Here is another tricky. Because my ESX has different default IP as DS default. so once the DS Manager deploy the virtual appliance to ESX, the appliance only has default DHCP IP which is wrong in my case also the virtual network is also wrong. I encounter this problem.


All what you need to do is to jump on ESX and virtual appliance console to change IP of that appliance. The default username and password is dsva.



Once you changed the IP, reboot this VM. Go back to DS Manager and double click dsva object to activate it.


Make sure the security profile is loaded. That’s very important!!


System will automatically offer you some VMs to protect. You can choose “no” at this stage. Why? because you haven’t installed vShield Endpoint agent and DS agent on your VMs yet.


By now, the installation steps have finished here.

In my next post, I will talk about how to configure Trend Micro Deep Security 7.5 and performance result comparing with OfficeScan and virus testing.

Let me show you a picture what a DS manager look like when a VM is fully protected to finish this post.



Trend Micro Deep security installation guide

Trend Micro Deep security User guide

This is second part of vShield. We will spend some time on vShield zone about Installation and configuration, of course, understanding as well.

Installation of vSheild Manager

Like what I have mentioned in the last post, vShield control module is vShield manager. And vShield Zone is it’s back bone which provides platform all other applications can run on it.

1.Download and Install

You can download evaluation version of vShield from Vmware as ova format. It’s a 500MB ova file and use vClient to deploy this ova into your vmware environment. You don’t need to worry about this vManager too much as it can be freely vMotion to any hosts in your cluster.


Once you imported the ova, you can file it up and use username “admin” and password as “default” to log in.


Type enable into cmd window and run setup

2. Configure IP and gateway.



You should be able to ping vManager.

3. Connect vManager with Internet Browser



4. Restart vClient and log in

After giving information to vManager, you should be able to see a new tab on vClient.


By now, vSheild Manager has been installed. But vZone or any other real vShield components haven’t been installed on any hosts. What you have done is merely a frame.

You can choose to configure all other aspects if you want.


Install vShield Zone

The next step is to install vShield Zone. vShield Zone is vShield App basic version. It shares same theory as vApp.

When you deploy vShield Zone from vManager, vManager will ask you to provide which host you want to install and a new set of IP for vShield Zone VM.

Each host will be bond with a new Linux VM and that VM will be fixed on that host and can’t be vMotion to other host since this VM will talk directly to a special module running in that host as same method of vSwitch.

In other word, that new VM will in charge all filtering jobs specific targeting on one host.

Notice: if you are running cluster, vShield Zone will only protect VMs running on host which as vZone installed. For example, you have host A, B. VM C,D. VM C running on host A and VM D running on host B. If you install vZone on Host B. Only VM D will be effected by vZone setting. If  you vMotion VM C from host A to B, then, VM C will be effected too.


However, if you are running a cluster (Host A, B), then by installing vZone on Host B won’t protect any VM until you install vZone on all Hosts in the cluster.

1. go to vShield tab and select a host to install


2. Provide a vZone VM IP set and Install



3.  System will deploy a new VM on that host


Apart from deploying a new VM, there are other couple of things this installing script has done.

  • Install a new module in the host.
  • Modify vmx belong to that host
  • Create a new vSwitch for firewall


Install a new module in the host


Modify vmx belong to that host


Create a new vSwitch for firewall




Let’s see a diagram and understand how it works at logic level.


All network traffic can be considered with a special detour before they reach to VM.

In the host level, we can use VMSafe diagram to understand since they share similar structure. It’s similar as VMsafe Net but it use it’s own filter (vShiled-dvfilter).




Management of vZone

vZone management is very similar as ISA. It has divided into multiple levels.

Hierarchy of Zones Firewall Rules
Each vShield Zones instance enforces Zones Firewall rules in top-to-bottom ordering. A vShield Zones instance checks each traffic session against the top rule in the Zones Firewall table before moving down thesubsequent rules in the table. The first rule in the table that matches the traffic parameters is enforced.
Zones Firewall rules are enforced in the following hierarchy:
1. Data Center High Precedence Rules
2. Cluster Level Rules
3. Data Center Low Precedence Rules (seen as Rules below this level have lower precedence than cluster level rules when a datacenter resource is selected)
4. Secure Port Group Rules
5. Default Rules


Few things you need to know:

1. Make sure vManager, vZone VM are all pingable to each other.

2. If you are using cluster, make sure all hosts are installed vZone.

3. If you try to uninstall vZone, a restart of host is involved!!

4. No restart involved when you install vZone on host.

5. vZone VM can’t be montioned.

6. How much overhead will be consumed by vShield in prod is unknown.

7. How much impact on network traffic by vShield is unknown.


vShield Administration Guide

Vmworld is right on the corner. There ain’t much new exciting information surfacing during this waiting period. I think I can use this time to gather my energy to have a little bit dive (not very deep though) before Vmworld hit on the ground.

I’m going to talk about PSA (Pluggable Storage Architecture) as part of requirement of VCAP study list. As usual, I try to make my post as simple as I can and also very welcome to any comments.

I remember when I started to read esxcli command and concept of PSA, I was simply overwhelmed by so many different parameters and options. And just like any one, I was lost. But after I started to abstract the detail command but try to understand what these commands tell me, everything is getting clear.

Please be aware: esxcli should be only used on esx(i) ssh session. vicfg-* should be used via vMA.

PSA Concept

To manage storage multipathing, ESXi uses a special VMkernel layer, the Pluggable Storage Architecture(PSA). The PSA is an open, modular framework that coordinates the simultaneous operation of multiplemultipathing plug-ins (MPPs).The VMkernel multipathing plug-in that ESXi provides by default is the VMware Native Multipathing Plug-In (NMP). The NMP is an extensible module that manages sub plug-ins. There are two types of NMP sub plugins,Storage Array Type Plug-Ins (SATPs), and Path Selection Plug-Ins (PSPs). SATPs and PSPs can be built-inand provided by VMware, or can be provided by a third party.

Let’s be reasonable. If a VM send a SCSI command to access data on the SAN, VMkernel needs to know how to access and which path it should choose. That’s where whole PSA kicks in. The PSA is a framework. It contains different modules and their sub-modules.

Note: PSA has 3 layers. Mulitpathing Layers->SATP layers->PSP layers

From the above picture, we can see PSA module need to choose between VMWARE NMP(Vmware own multipathing module) and MPP (thirdparty multipathing plug-ins, like EMC Powerpath) first. If  you choose VMWARE NMP, please be aware you are not necessary using exclusive vmware product from this point. NMP still able to load third party sub-modules (SATP, PSP).

NMP&MPP layers

It basically decide which SATP to choose. This layer will see what kind of physical hardware (SAN) you have. EMC? NETAPP? DELL? It will load appropriate SATP, PSP to do other jobs. Please be aware you have bunch of SATP and PSP to choose. You can let NMP to decide or manually assign (or claim) new rules.

Please be aware MASK_PATH(MASK LUN in esx3.5) are consider as a NMP level plug-in.

SATP layer

Storage Array Type Plug-Ins (SATPs) run in conjunction with the VMware NMP and are responsible for arrayspecific


ESXi offers a SATP for every type of array that VMware supports. It also provides default SATPs that support

non-specific active-active and ALUA storage arrays, and the local SATP for direct-attached devices. Each SATP

accommodates special characteristics of a certain class of storage arrays and can perform the array-specific

operations required to detect path state and to activate an inactive path. As a result, the NMP module itself

can work with multiple storage arrays without having to be aware of the storage device specifics.

After the NMP determines which SATP to use for a specific storage device and associates the SATP with the

physical paths for that storage device, the SATP implements the tasks that include the following:

  • Monitors the health of each physical path.
  • Reports changes in the state of each physical path.
  • Performs array-specific actions necessary for storage fail-over. For example, for active-passive devices, it can activate passive paths.

Please be aware SATP can be thirdparty one. But I don’t have thirdparty loaded in this picture.

PSP layers

Path Selection Plug-Ins (PSPs) run with the VMware NMP and are responsible for choosing a physical path for I/O requests.

The VMware NMP assigns a default PSP for each logical device based on the SATP associated with the physical paths for that device. You can override the default PSP.

These path policy are reflected to path policy in vCenter.

I’m not going to discuss each PSP here. If you have more questions, please refer to vmware docs.

PSA command line

There are few command lines related to PSA.

esxcli, vicfg-mpath, vicfg-mpath35

so what exact does command do? What kind of information I will pull out or change?

vicfg-mpath (vicfg-mpath35 is for esx 3.5)

This command is to list all available path and all detailed information about your device as well.

It also has ability to disable a path and active a path.

esxcli is much powerful command comparing with vicfg-mpath.

You need to be aware that esxcli is much more just to adjust PSA structure. It also control network, swiscsi, vaai, and vms behavior in vmKernel leve.

Let’s take a brief look what esxcli can do.

As you can see from the picture, from PSA wise, we only focus on corestorage and nmp.

Please consider esxcli is a command which you can use to interactive information of vmKernel pool. Let’s take look what kind of information you can access.

Configuration information (or Claim Rules)

Run command from esxi server ssh connection:

esxcli corestorage claimrule list

On the left side, we have rule class. There are 3 types of rule class(MP,FILTER,VAAI). It would appear if you use my last command. In this example, it didn’t because Hardware acceleration are not enabled and neither VAAI.

Rule runs from small to big number (or lower number as Vmware prefer). Be aware from Rule 0 to 101 are Vmware reserved rule. Between 102 to 60000 user can create their own rules. After 60000, Vmware claims those rules again.

When you build a rule, you need to build a pair rule(runtime and file). The file parameter in the class column indicates rule is defined. The runtime paramter indicates that rule has been loaded into your system.

Plugin also means  module. In this example, we have NMP(Vmware module), MASK_PATH(Vmware LUN Mask) and MPP_1(thirdparty module for NewVend).

MASK Lun method has been changed. In ESX 3.5, we use vCenter to Mask LUN you don’t want hosts to see. In ESX 4, we have to use command and create MASK_PATH rule to dedicate LUNs.

Matches column is actually conditions for rules to apply. You can clearly see what kind of conditions rule 0-4 will apply and so forth. The last rule is like last rule of ISA firwall. Every other conditions which were not defined by previous rules will fall in this rule.

Device information

Have you ever wondering where you can see new UUID, vml, and other information for LUNs or devices connecting to your host? You can read my post at here or you got vicfg-mpath -l to do the job. You also can use esxcli nmp device list as well. But esxcli is starting from PSA wise.

SATP & PSP options

If you want to see what kind of SATP&PSP you have and you can choose, you can use

esxcli nmp satp list

esxcli nmp psp list

All right. I believe it should be easy to understand now. vmKernel has lots of information about PSA. You can use esxcli and vicfg-mpath to get information and modify as you want. I have to say, this is an understanding doc not reference doc. If you do want to add new path, MASK LUNs, or use different rules, you still need to check out all docs before you actually execute any commands.

Please do leave any comments if you want.


ESXi configuration Guide

iSCSI SAN Configration Guide

I have chance to get involved into a EMC pre-sale meeting today. During the meeting, the EMC pre-sale Engineer introduced F.AS.T v1 and V2 to us. I did know what FAST it was before, but this presentation really opened my eyes and also Engineer was able to answer few of my questions abour Netapps Deduplication vs EMC Compression. I will bring details into this post. However, because there ain’t much available data in the Internet, I have to draw an ugly diagram to help me expressing my idea. I may make mistakes, please feel free to point out.

What is EMC FAST v1?

As you can see from full name of F.A.S.T, It’s about tiering your storage automatically. As you may know the transitional SAN storage contains FC disk and SATA disk. FC is fast and expensive and SATA is slow for random w/r and cheaper. As SAN administrator in the company, your job would be give right LUNs to appropriate servers to fit SLA requirement.

With F.A.S.T, it basically did following things:

1. Add EFD(Enterprise Flash Disk) layer.

As we all know, SSD (solid storage disk) is 100 times faster than FC. It has SLC(single layer cell) and MLC(multiple layer cell) two types. All SSD are short life product. So how does EMC manage to overcome these issues?

These EFD are made of  SLC SSD not MLC, meaning it’s faster than MLC SSD. As you may have heard, SSD is easy to be broken. The reason for easy damaged SSD(same as your usb flash disk) is rewrite same location repeatedly. For the normal system, you write first block of flash disk, and wipe out and write again. So the first block of flash disk is used too many times and easy to damaged. EMC EFD won’t use same spot twice until it has finished all other available spots in the SSD.

Each EFD has 3 components. Cache area (fastest area), normal storage area and hotspot area. All data will write to fast cache first and then, write to normal storage. The spot of normal storage will be discarded after few times reusing and it will start to use spot in the hotspot area to avoid potential bad spot. Same thing apply to cache area, if one of spot is damaged it will start to use spot in the normal area. According to EMC, the EFD has 5 years warranty.

2. It added a virtual LUN layer

Virtual lun can isolated Host and actually storage details. Host doesn’t need to know which physical LUNs (FC,EFD,SATA) it’s operating. With virtual LUN technology, the FAST true mean can work under SAN layer.

3. Auto moving LUNs between tiers

This is what FAST for. F.A.S.T can automatically (or manually) move your LUNs to different tier. Busy and high demanding LUNs will move to fastest tier (EFD) or FC. The low priority LUNs can be shift to SATA to safe fast speed tier for SLA requirement.

What is FAST v2?

We have briefly introduced FAST v1 system as above. After EMC push this technology to it’s customers, they discovered most of customers actually bought lots of FC disks instead of SATA disk. Because FAST v1 is operating on LUN level. Everytime it moves, it has to move whole LUN which is slow and inefficient. so FAST v2 comes to alive.

FAST v2 made some big changes.

1. Let’s making pool

Well, basically, you need to create pool first. A pool is combination of different tiers resource. For example, you can make a pool which has 3xEFD, 4x FC, 5xSATA with all RAID 5. Then, you can create LUNs on this pool. The LUN will be built cross all tiers instead of sitting on one.

2. Let’s move 1GB data segment.

From FAST v1, we move whole LUN which takes long long time and also may not be effective as well. With this version of FAST, we move data with 1GB data segment as smallest operation unit. Meaning if one LUN got hit very hard, the system will use fast cache hold the data and started to move that most busy segment from SATA to EFD. Then, it will move other segments later on according to utilization of LUNs.

EMC compression vs Netapps Deduplication

I have an interesting conversation with EMC Engineer. EMC has preach block level compression to all systems instead of deduplication like NetApps did. This compression and decompression can be done on the fly. It will add about 5% performance overhead which you may not notice. However, it gives you almost 50% compress ratio comparing with deduplication ratio which is only 30% most of time. For the SP utilization, the compression will cost 5% utilization and dedup will cost around 20% CPU.

EMC is very cautious about CPU utilization on Storage. They reckon the normal utilization should be around 25% of single CPU. If one of your SP failed, then, your load will be 50% on remain CPU. They don’t want to use deduplication cost too much cpu resource at this time. At least, not with current CPU horsepower. According to them, the CPU will be much powerful in 2 years which will not only allow to do deduplication, compression, it will also allow you to directly run VMs (like  WAN accelerate appliances) on it. In short, EMC is quite conservative company but it does provide awesome technology especially for long run.

Please leave your comments if you want.


I just came back from Vmware Seminar 2010. There are lots of information I would like to share with you. You can clearly see  where Vmware is heading for it’s own future.

Future of Vmware: Cloud

You may heard some news about VmForce (Vmware combine with Salesforce to make cloud level ERP system), Vmware acquired SpringSource and public cloud, private clound. I was quite confused before I attended this seminar since I could hear everyone is talking about cloud but no real clouds system for private enterprise to merge or any real cases about cloud. This puzzle is resolved by this Seminar.

VMware wants to get rid of Microsoft

That’s it. That’s root cause why Vmware did all sorts of weird activities in past year. This is what Vmware has planned:

Make all companies virtualization 50% up (√) (85% world companies are using VMware tech)

Make all companies 100% virtualized

  • Vmware SRM helps DR (Expensive plan which requires 100% virtualization & DR budget, only about 5% companies are doing that)
  • vSphere helps servers platform (facing challenges from MS and Citrix)
  • VMware View helps Desktop (unsuccessful and beaten by Citrix  XenDesktop)
  • vThinapps (very few companies have actually used in the production, this is prestep for stripping apps from OS level in the future)

Using ESXi to replace ESX (has confirmed from all VMWARE people, it will happen in 2011. Vmware can finally get rid of head ache Red Hat for SC and have hardware appliance alike ESXi as house bricks)

Using VMSafe products (like Trend Macro, Agentless anti-virus. It will be available very soon).

Private Cloud era (VMware believes applications don’t need to run on Microsoft OS. They can let applications run directly on VMKernel. It’s obviously that Microsoft won’t agree with this idea. That’s why VMware bought SpringSource(Java application company) and try to make application platformless. I believe Microsoft will soon to push out their own cloud system and also use MS version thinApps and stream APPs to fight with Vmware)

Hybrid cloud (It’s also called Redwood project. This is next generation VM OS. If Private cloud is ready, meaning all apps can run on VMkernel without MS OS. Any apps can be seamlessly transferred between private cloud to hybrid cloud and even public cloud).

Public cloud (At that day, every system will run on a standard industry module  and can accept and transfer all applications)

In terms of convincing CIOs and IT Managers to purchase equipment to do VM DR and 100% virtualization, Vmware put lots of efforts on CapacityIQ, ChargeBack, how to shift attention from CAPEX to OPEX during the seminar. It only broadcasts one message. Come and virtualize everything!

Good plans, but there are concerns:

I agreed Vmware has drawn a beautiful picture of future I.T. But whether they can actually pull off this show is really a question mark. With all those components, like servers, Desktops, Networks, any parts of failure may cause huge disaster for Vmware. Vmware View is still not promising from angle of optimizing qos via WAN. PCoIP completely when it competes with Citrix ICA/HDX. Vmware users have to go back and use MS RDP protocol to connect Virtual desktop which gives Microsoft a chance to regain the market. Even in the latest demonstration, Vmware view still hasn’t fixed issue. From my personal understanding, Java applications has tons of issue. Slowness, stability problem. Novell has to dropped off Java console from Netware few years ago doesn’t mean anything to Vmware?

This is part 2 of whole series of Using ESXi to replace ESX. ESXi comes long way and still not taking major market for production. But I personally believer RCLI, VCenter+ESXi will be the future. Vmware has already developed hidden page to guide everyone to upgrade from ESX to ESXi. Yes, please read on and I will explain it later.

So this is sort of deep dive to ESXi system. However, there ain’t much information about ESXi4 so I have to add lots of own opinions. If I made mistake, please feel free to point out.

Ok. Let’s take look what the difference between ESX and ESXi at Architecture level.

This is ESX4 architecture. Essentially, everything is running on Service Console (Red Hat). VMKernel itself is an operating system just like all other OS. But it relies on SC to do all communication works and runs different agent. VMware agents (vpxa, hostd, etc) runs on SC and always run into stability issue after people install all other Hardware monitoring agents on SC. According to my experience, we always have some HA agents issue, SC stop responding to ping and heartbeats. Then, the issue resolved by itself after few minutes. All command lines are running on SC and then, SC forwards to VMKernel and wait for reply. This is long way to go and consumes lots of extra resources and bring tons of headache to VMWARE.

The above picture is ESXi diagram. I have to say, ESXi is not only a free product, but also a brand new design from architecture level. It has following advantages comparing with ESX.

Vmware agetns ported to run directly on VMKernel.

Let me bring up another diagram so  you can take a close look.

As you can see, vpxa, hostd and other important processors have migrated from SC to VMKernal. They are running on User world API stack and waiting for the communication from RCLI, vPowershell, and VC.

Authorized 3rd party modules can also run in Vmkernel. These provide specific functionality

  • Hardware monitoring
  • Hardware drivers

There are quite big changes in the Hardware monitoring world. VMware in default weak SNMP protocol (no SNMP trap set for ESXi) and focus on CIM broker.

The Common Information Model (CIM) is an open standard that defines how computing resources can be represented and managed. It enables a framework for agentless, standards-based monitoring of hardware resources for ESXi.

Basically, instead of using SNMP trap and query SNMP to your HOST, you should enable WBEM to do all the jobs. If  you want to deploy your ESXi system, you should not download ESXi directly from vmware site, but instead, you should go to your server company to download their special version of ESXi. For example, HP provides HP WBEM(Web Base Enterprise Management) embedded ESXi for free downloading. ESXi allows third-party to pre-install CIM Plug-ins and ESXi and plug-in can be upgraded separately. All what you need to do is to download HP SIM Manager and start querying. (In default, WBEM queries every 2 minutes)

With this design, ESXi can use agentless framework to let hardware monitoring system get full details of Host and also secured and prevent unexpected error caused by HW Agent (like HP SIM Agents).

The “dual-image” approach lets you revert to prior image if desired

This is very interesting design special for ESXi.

The ESXi system has two independent banks of memory, each of which stores a full system image, as a fail-safe for applying updates. When you upgrade the system, the new version is loaded into the inactive bank of memory, and the system is set to use the updated bank when it reboots. If any problem is detected during the boot process, the system automatically boots from the previously used bank of memory. You can also intervene manually at boot time to choose which image to use for that boot, so you can back out of an update if necessary.
At any given time, there are typically two versions of VI Client and two versions of VMware Tools in the store partition, corresponding to the hypervisor versions in the two boot banks. The specific version to use is determined by which boot bank is currently active.
As what the pdf says, ESXi alwasy keep another version of configuration file and other components. If boot fails, it can switches over like “Last good configuration” function in MS.
If you runs command fdisk -l in the ESXi, you will get following picture.

As you can see, the first part is Extended partition, also called Store partition. It’s about 917MB in ESXi4 instead of 750MB in ESXi 3. It stores Auxiliary files like VI client, VMWare tools, runtime storage etc.

The second partition is 4GB as what VMWARE called Scratch partition. Next one is VMFS partition. Partition FAT16<32M is bootloader partition. It remains as 4MB to choose which boot bank should be loaded. Boot bank (255MB in ESXi) contains core hypervisor code. Diagonistic Partition (112MB in ESXi4) is for core dump purpose. And the last one is Hypervisor 3 Locker. Once you start Locker mode, no remote access will be accepted.

As you can see, ESXi4 has 3 different boot options. Primary, backup and locker mode. It provides failover and security as well. I will review this part again in my next part (part 3).

No other arbitrary code is allowed on the system

Essentially, ESXi should be consider as an appliance with firmware. Yes, there are still quite few things you can mock around, like open SSH, setup SNMP TRAP, Backup configuration settings without host profile function, but comparing ESX 4, it’s much simple, easy, fast, efficient and safe.

Much less updates means much better stable system

Let’s see this diagram from Vmware, then it will explain by itself.

Finally, at last but not least.

VMWARE has developed web page to help user to Upgrade from ESX to ESXi. But you can’t find link from it’s parents page which is vSphere page.

I provide the link so you can see it by yourself.

To be continued….


One of my friends just questioned about ESXi and think ESX is better environment to execute his precious codes. I think Service Console has nothing to do with implement or execute code because this job is done by vSphere API. ESXi has vSphere API just like ESX and has no issue to execute any codes running on ESX.

For better understanding, I’m showing you this picture to prove my point.