Skip navigation

Category Archives: EMC

I have thinking recently about what and where I would be in 3 yrs regarding my career path. If I go through the certificates I got, I have a strong feeling that I don’t have any cloud certificates. so here is question.

I have VCP certificates, do I need anything else?

First of all, you need to be aware that we are talking about cloud Certificates here. It’s not virtualization certificates. Virtualization is ground brick of Cloud, but it can’t represent the concept and IT as Service. We do need something in general concept and help us to convert business mode from normal traditional EA software license to become user consumption, department consumption mode as what I called now, IT as service.

The reason for that is we need to fully understand mode and details of each department and each software usage for business so in one day, we can break down some pieces and shift them to Public cloud.

Now, back to my own topic, Certificates.


So far, I have only found one set of Cloud certificates which is EMC CIS.

The path of getting all certificates are following:


Become an IT professional who demonstrates cross-domain expertise and focus on designing cloud-based IT service solutions that drive business transformations for the enterprise and service provider organizations. This course is for those assessing, architecting, and designing IT-as-a-Service solutions as part of the transformation and optimization of virtual data centers into cloud-based IT-as-a-Service environments. Prepare for your Expert-level Cloud Architect Certification.
Exam and Practice Test
Expert E20-918 (To be announced)
Specialist E20-018
Associate E20-002




To be honest, the cost of taking those training are huge. You are basically looking at $3000 just for video training and $5000 for Lab training with limited region. As parent company of Vmware, EMC believes it earns it’s place to issue it’s certificates as first in industry. But will  you really want to get EMC certificates?



Cisco has been really pushing on Virtualization and working extreme well. The flag product is UCS which earns respect and become default Blade system any company would want to have. Cisco certificates are not new to I.T so here is something new from Cisco in Cloud side.


This is new released by Cisco and that’s what Cisco picture itself in Cloud business. Since there is no doubt on network part, with help of Vmware, I’m very sure Cisco will become a true leader of Cloud certification.

However, there is one specific cloud certificate from Cisco yet, so UCS certificates should be the one you can get.

Interesting enough, not only you need to pass Cisco Exam, but also must own Vmware certificates. Hence, we know how strong the relation between Cisco and Vmware.


Here we go. There is no reason I don’t mention Vmware here. But the tricky thing is even after so many effort from Vmware for Virtualization and Cloud, there is no Vmware Cloud certificate!

My guess is Vmware is still working on how to make Cloud Director really working as it should be. All other components are made but not mature yet. We should expect to see some sort of Vmware Cloud certificates in next 2 or 3 years.



I never be a fan of Citrix. In my mind, it’s complicated, not user friendly, consuming too much resource, overhead for administration and too expensive and too many on licenses. The only reason we are still hitting on Citrix is the XenDesktop which is great on low bandwidth. Apart from that, I don’t see any attractions.

No cloud certificates on Citrix but I believe it will kick in pretty soon.


Microsoft is keeping it’s own way on Cloud definition. It seems it doesn’t like to share whatever technology it’s using. Hyper-v 3 is finally taking vDS into it but still lack of hardware vendor’s support. Windows Azure is slowly slowly moving forward with few companies doing DEV and test on it. Office 365 could be a good one but it’s charging too much and limited on customization. Leave your product into black box and you can’t manage and don’t know how it works is a scary strategy to take.


Well, as usual, drop a line to me and see what I have missed.






Ok. This is second part of EMC Celerra Unified, CLARiiON storage with Vmware – Deduplication and Compression Part 1. In my last post, I explained how EMC block compression works and where it will applies. In this post, I’m going to talk about another important part of EMC Celerra Unified storage technology. Celerra file level Deduplication& compression. AKA, Celerra Data deduplication.

Considering Celerra Unified storage is combination of Celerra (NAS) and CALRiiON (block storage, FC). The essential Celerra is actually focusing on NAS parts. It is not only has compression function(but it’s file level), but also has file level deduplication. Let’s talk about how it works.

The sequence of Celerra Data Deduplication is to look for cold data, compress first, then deduplicate later.

Please be aware although NAS is basically a file system. but Celerra is not exactly operating on file leve.  Let’s see how it works.

Celerra Compression

If you check out latest version of Celerra docs, you may notice Celerra compression is not longer a file compression. It’s actually just labeled a “compression”.

Plus, EMC used to claim that do not use deduplication on NFS for Vmware, but recentely, EMC just released the Celerra Vmware Plug-in which allow  you to compress single VM. First of all, let’s talk about “Initial compression and deduplication”

Initial and scheduled compression & deduplication

In the Celerra, the compression and deduplication  is done together. Celerra periodically scan file system and compress “cold” (not recently active files) and hash them into metadatabase. If there are same hash exist in the database, this file is deduplicated.

So the process is:

Compress->hash->compare meta database->Copy to hidden space of this LUN (or not)->Update meta database.

Reading Compressed file

Compress Read is not much difference from EMC CLARiiON read. Celerra directly load compressed data into memory and extract data in the memory. Nothing is writing back to disk. Therefore, in some cases, reading compression data is even faster than uncompression data but with some CPU cycles cost.

Writing to a compressed & deduplicated file

Writing a compressed file is a long procedure because it involves writing to disk. A write to or a modification of a deduplicated file cause a copy of the requested portion of the file data to be reduplicated for this particular instance while preserving the deduplicated data for the remaining references to this file.

The entire file is not decompressed and reduplicated on the disk until the sum of the number of individual changed blocks and the number of blocks in the corresponding deduplicated file are larger than the logical file size.

Compression with Vmware environment

Celerra Vmware Plug-in just be released not long ago. It can only work on NAS file system. It has “Thin provisioning”, “Fast/Full Clone”, “Compression/Decompression” features. Just like CLARiiON system, you can use this plug-in to off load those operations from host to SAN. One of operation is compression. It allows you to compress a VM regardless it’s think or thin disk. VMs that are selected for storage optimization are compressed by using the same compression algorithm that is used in Celerra Data Deduplication. The algorithm comresses the VMDK file (virtual disk) and leaves intact other files that make up the VMs. When a compressed VMDK is read, only the data blocks containing the actual data are decompressed. Likewise, when data is written to the VMDK, instead of compressing it “on the fly”, data is written to a set-aside buffer, which then gets compressed as a background task. In most situations, the amount of data read or written is a small percentage of total amount of data stored.  VM performance will typically be impacted maginally (less than 10%) when the corrsponding VMDK file is compressed.

I still think there are plenty of things EMC can do. Celerra plug – in is just a beginning. I will keep eye on it and post more later.

First of all, I need to point out that I ain’t work for EMC nor VMWARE. I won’t do anything like Chad in Virtual Geek to say, “Yes, it’s true. EMC really IS #1 for VMware.” What I’m going to to talk about is purely from a customer point of view. A tech who doesn’t favorite either EMC nor Netapps. I just want to draw a picture of EMC storage and Vmware related technologies in front of you so we can put everything on the table and discuss it.

As usual, any comments and discussions are very welcome here.

EMC Storage – Celerra, CLARiiON and Celerra Unifed Storage

Well, back in years ago, There are three production lines in the EMC storage. Celerra, CLARiiON and Symmetrix. After years of Virtualization development, the line between Celerra and CLARiiON is getting really blurry. Celerra used to dedicate to NAS file system. It provides NAS only. With CLARiiON(like CX3 series), it mainly focus on Block level storage like FC or iSCSI. Now, with new product line Celerra Unified storage coming out, I don’t really think anyone would buy old system any more. Because new Unified Storage provides both Celerra & CLARiiON in one box. EMC call them block enabled Celerra system (NS-120,480,960,etc). However, as you may know the technology using in NAS are quite different from technology using in the block storage. If you are as confused as me, please read rest of article and hope it can help you clear your mind. This article is focusing on Celerra Unified storage only.

EMC Deduplication and Compression

As everyone knows, one of key elements of storage is disk capacity. How to utilize disk space and tier down unused data and files and compress them to small space becomes the major reason when we select Storage. Even after years and years research, EMC still insists deduplication should only happens on file level instead of block level. so what does that mean to customers who bought Celerra Unified Storage? It means you can only use block compression if you use block storage (like FC, iSCSI, this is as part of CLARiiON technology) and you can only use file level Compression and file level deduplication when you use NAS (as part of Celerra, if you use NFS for VM or NFS,CIFS for file systems). In other word, how  you divide your LUNs and what kind of block or file system you use will dramatically impact your system. Let’s break down those technology and see what they are.

EMC compression

As what I mentioned above, depends on what kind of system (NAS or block) you use, Celerra will use different ways to compress the data. Let’s talk about block level compression first.

Block level compression

As what this name indicates, this compression should only work on FC or iSCSI LUNs. The block size compression works on is 64KB. Each block is processed independtly. The typical result of compression is as much as 2x while it use modest CPU resource.


In default, CX4-480 can have 8 concurrent threads on compression. When all threads running at same time, the consumption of CPU will be compression rate(speed) as Low (15% CPU), medium (30~50% CPU) and high(60%~80% CPU).

How block compression works

1.Initial compression- This occurs when you enable compression. It compress entire LUN and can’t be paused in the middle. But it can be disabled during the procedure. No damage will be done.

2. Compression of new data – When new data is written, it is written uncompressed and compressed asynchronously. It keeps doing that until you disable compression. In default, when 10% of user capacity of LUN or 10GB new data are written, and total amount new data is larger than 1GB, compress starts. It does use some of SP cache memory for swapping. When you compress a LUN, that LUN will automatically migrate to a thin LUN in different pool if LUN is a normal RAID lun. If it’s a think LUN, it will reminds in the same pool.

3. Decompression when compression is disabled- if the original LUN is a thin LUN, it will reminds thin LUN. If the original LUN is a thick LUN or RAID group LUN, it will write zeros to unallocated capacity till full while it reminds a thin LUN. System will pause at 90% and stop at 98% if the LUN has filled up too much.

Limits of block compression

  • The following cannot be compressed:
  • Private LUNs (including write intent logs, clone private LUNs, reserved LUNs, meta LUNs, and component LUNs)
  • Snapshot LUNs
  • Celerra iSCSI or file system LUNs ( Personally, I don’t think that’s right. I’m confirming with EMC now)
  • A LUN is already being migrated and expanding or shrinking.
  • A mirrored LUN replicating to a storage system running pre-29 FLARE code.

Interactive of compression with other functionalities

Basically, a compressed LUN is transparent to other operations like replication or migration. But by saying that, it’s better not migrating or copying while compress at same time. It’s always easy for SAN to enable compress after migration.

How to setup compression?

All what you need to do is to connect Celerra Unified storage control station with your Internet Browser. You will have Unisphere Manager running directly from SAN or you can install Unified Manager on a windows server and connect to your box. Compression function is a licensed feature and you should have it directly from console. Unlike Celerra NAS part, there is no VMWARE plug-in availabe for compression so you need to use Unisphere to do the job.

There is no Vmware plug-in?

It is very interesting that Celerra Unified NAS part got a vmware plug-in while CLARiiON reminds nothing. I reckon vSphere may use VAAI API to offload clone from host to SAN but why it doesn’t work for Celerra NAS part? If anyone can answer this question, it will be appreciated.

To be continued……


This is a post regarding VAAI (vStorage APIs for Array Integration). I have read some posts from Chad and personally, I don’t think I can do any better job than him to explain what VAAI is. However, I do can explain it from a user angle instead of manufacture perspective.

All right, let’s hit on the road.

What is vStorage API for Array Integration(VAAI)?

Essentially, VAAI is technology that Vmware use to offload certain disk activity/jobs to SAN to save Host I/O resource and improve performance and it’s enabled by default. From user point of view, this is still nice to have feature since the tasks you can offload are following.

•Hardware-Accelerated Locking = 10-100x better metadata scaling

–Replace LUN locking with extent-based locks for better granularity

–Reduce number of “lock” operations required by using one efficient SCSI command to perform pre-lock, lock, and post-lock operations

–Increases locking efficiency by an order of magnitude

•Use Cases –Bigger clusters with more VMs –View, Lab Manager, Project Redwood –More & Faster VM Snapshotting

•Hardware-Accelerated Zero = 2-10x lower number of IO operations

–Eliminate redundant and repetitive host-based write commands with optimized internal array commands

Use Cases-Reduced IO when writing to new blocks in the VMDK for any VM,Time to create VMs (particularly FT-enabled VMs

•Hardware-Accelerated Copy = 2-10x better data movement

–Leverage native array Copy capability to move blocks

Use Cases -Storage VMotion -VM Creation from Template

What kind of benefit you can get from using VAAI?

I believe the Vmware View will gain some performance boost when it try to provision more desktops. SAN can take load off from HOST. So it should be faster for View host to create more snapshots.

The other one is Anti-virus. For example, the new Trend Macro is using VMSafe API to scan and respond. Let’s see what Trend said about it.

How Core Protection for Virtual Machines product works is we take a snapshot of the Virtual Machine via the VMsafe API and scan the snapshot when performing the full system scheduled & manual scans, if malware has been detected we then send an instruction to the real-time agent for the mitigation process to initiate. The real-time scanning agent on the other hand performs On-Access scanning on the virtual machines which subsequently initiates the mitigation process if malware is identified.

With lots of creating snapshots during the scan, VAAI is definitely help a lot to Trend or CA anti-virus software.

so are you wondering how much performance it can gain?

Let’s see what Chad said about it.

Just one example (of many)… using the Full Copy API:

  1. We reduced the time for many VMware storage-related tasks by 25% or more (in some cases up to 10x)
  2. We reduced the amount of CPU load on the ESX host and the array by 50% or more (in some cases up to 10x)(by reducing the impact during the operation and reducing the duration) for many tasks.
  3. We reduced the traffic on the network by 99% for those tasks.  Yes you read that right, 99%. During these activities (storage vmotion is what we used in the example), so much storage network traffic can be so heavy (in the example 240MBps worth of traffic), it impacts all the other VMs using the storage network and storage arrays.

And like Intel VT with vSphere 4, these improvements just “appear” for customers using vSphere 4.1 and storage arrays that support these new hardware acceleration offloads.

Where do these integrations occur?

There is nothing better than Chad’s blog, I just quote here.

To understand the dialog that goes with this, listen to the webcast (link below).   As a point of note – while all vendors are working hard to integrate with VMware (which is good – highlights the importance of VMware in customer environments), to date, as far as I know, EMC is the only single vendor to have products available that integrate witheach of the areas in green.   BTW – “co-op” means it’s an area where it’s not an integration API per se, but that there is a lot of cooperative development.

What are VAAI requirements?

Well, In general speaking, everything you purchased more than 2 years old won’t support the VAAI unless you update FLARE or firmware. As you can see from above picture, it requires “Vendor-specific VAAI SCSI command” loaded in the SAN. For users who are still using CX3 (32bit), it’s time to move on and end your current lease of SAN to get new one.

Still, quote from  Chad:

What do you need to benefit from this hardware acceleration of storage functions?

  • Well, of course, you need vSphere 4.1.  VAAI is supported in Enterprise and Enterprise Plus editions.
  • If you’re an EMC Unified (an EMC Celerra purchased in the last year – NS-120, NS-480, NS-960) or EMC CLARiiON CX4 customer, you need to:
    • Be running FLARE 30 (which also adds Unisphere, Block Compression, FAST VP aka sub-LUN automated tiering, FAST Cache and more).   You can read more about all the FLARE 30 goodness here and here if you’re interested in more detail about what’s new in that release.   FLARE 30 is going to be GA any day now…
    • Also, ESX hosts need to be configured to use ALUA (failover mode 4).   If you’re using a modern EMC Unified or CLARiiON CX4 array, using ALUA (with the round robin PSP or PowerPath/VE) with vSphere 4.0 or vSphere 4.1 is a best practice (for iSCSI, FC and FCoE).   We will be automating configuration of this shortly in the always free EMC Virtual Storage Integrator vCenter plugin, but for now it’s pretty easy to setup manually.
  • If you’re an EMC VMAX customer – it will be a bit longer – but not much, but VAAI support is in the next major Enginuity update scheduled for Q4 2010.
  • It is supported on all block protocols (FC, iSCSI, FCoE)

What are the catch?

  • When does a VAAI offload NOT work (and the datamover falls back to the legacy software codepath) if all of the above are true?
    • The source and destination VMFS volumes have different block sizes (a colleague, Itzik Reich, already ran into this one at a customer, here – not quite a bug, but it does make it clear – “consistent block sizes” is a “good hygiene” move)
    • The source file type is RDM and the destination file type is non-RDM (regular file)
    • The source VMDK type is eagerzeroedthick and the destination VMDK type is thin
    • The source or destination VMDK is any sort of sparse or hosted format
    • The logical address and/or transfer length in the requested operation are not aligned to the minimum alignment required by the storage device (all datastores created with the vSphere Client are aligned automatically)
    • The VMFS has multiple LUNs/extents and they are all on different arrays

Last but not Least:

First of all, there are other voices from Netapps. From this link, you can clearly see How Netapps work with VAAI technology. If we think about from Microsoft and Citrix angle, what are they going to do to offload snapshot and vm copy from host to storage? Will VAAI become a public protocol and APIs to be loaded by all Virtual platform?

And what happened to that missing feature of VAAI, Thin Provisioning Stun?


I have chance to get involved into a EMC pre-sale meeting today. During the meeting, the EMC pre-sale Engineer introduced F.AS.T v1 and V2 to us. I did know what FAST it was before, but this presentation really opened my eyes and also Engineer was able to answer few of my questions abour Netapps Deduplication vs EMC Compression. I will bring details into this post. However, because there ain’t much available data in the Internet, I have to draw an ugly diagram to help me expressing my idea. I may make mistakes, please feel free to point out.

What is EMC FAST v1?

As you can see from full name of F.A.S.T, It’s about tiering your storage automatically. As you may know the transitional SAN storage contains FC disk and SATA disk. FC is fast and expensive and SATA is slow for random w/r and cheaper. As SAN administrator in the company, your job would be give right LUNs to appropriate servers to fit SLA requirement.

With F.A.S.T, it basically did following things:

1. Add EFD(Enterprise Flash Disk) layer.

As we all know, SSD (solid storage disk) is 100 times faster than FC. It has SLC(single layer cell) and MLC(multiple layer cell) two types. All SSD are short life product. So how does EMC manage to overcome these issues?

These EFD are made of  SLC SSD not MLC, meaning it’s faster than MLC SSD. As you may have heard, SSD is easy to be broken. The reason for easy damaged SSD(same as your usb flash disk) is rewrite same location repeatedly. For the normal system, you write first block of flash disk, and wipe out and write again. So the first block of flash disk is used too many times and easy to damaged. EMC EFD won’t use same spot twice until it has finished all other available spots in the SSD.

Each EFD has 3 components. Cache area (fastest area), normal storage area and hotspot area. All data will write to fast cache first and then, write to normal storage. The spot of normal storage will be discarded after few times reusing and it will start to use spot in the hotspot area to avoid potential bad spot. Same thing apply to cache area, if one of spot is damaged it will start to use spot in the normal area. According to EMC, the EFD has 5 years warranty.

2. It added a virtual LUN layer

Virtual lun can isolated Host and actually storage details. Host doesn’t need to know which physical LUNs (FC,EFD,SATA) it’s operating. With virtual LUN technology, the FAST true mean can work under SAN layer.

3. Auto moving LUNs between tiers

This is what FAST for. F.A.S.T can automatically (or manually) move your LUNs to different tier. Busy and high demanding LUNs will move to fastest tier (EFD) or FC. The low priority LUNs can be shift to SATA to safe fast speed tier for SLA requirement.

What is FAST v2?

We have briefly introduced FAST v1 system as above. After EMC push this technology to it’s customers, they discovered most of customers actually bought lots of FC disks instead of SATA disk. Because FAST v1 is operating on LUN level. Everytime it moves, it has to move whole LUN which is slow and inefficient. so FAST v2 comes to alive.

FAST v2 made some big changes.

1. Let’s making pool

Well, basically, you need to create pool first. A pool is combination of different tiers resource. For example, you can make a pool which has 3xEFD, 4x FC, 5xSATA with all RAID 5. Then, you can create LUNs on this pool. The LUN will be built cross all tiers instead of sitting on one.

2. Let’s move 1GB data segment.

From FAST v1, we move whole LUN which takes long long time and also may not be effective as well. With this version of FAST, we move data with 1GB data segment as smallest operation unit. Meaning if one LUN got hit very hard, the system will use fast cache hold the data and started to move that most busy segment from SATA to EFD. Then, it will move other segments later on according to utilization of LUNs.

EMC compression vs Netapps Deduplication

I have an interesting conversation with EMC Engineer. EMC has preach block level compression to all systems instead of deduplication like NetApps did. This compression and decompression can be done on the fly. It will add about 5% performance overhead which you may not notice. However, it gives you almost 50% compress ratio comparing with deduplication ratio which is only 30% most of time. For the SP utilization, the compression will cost 5% utilization and dedup will cost around 20% CPU.

EMC is very cautious about CPU utilization on Storage. They reckon the normal utilization should be around 25% of single CPU. If one of your SP failed, then, your load will be 50% on remain CPU. They don’t want to use deduplication cost too much cpu resource at this time. At least, not with current CPU horsepower. According to them, the CPU will be much powerful in 2 years which will not only allow to do deduplication, compression, it will also allow you to directly run VMs (like  WAN accelerate appliances) on it. In short, EMC is quite conservative company but it does provide awesome technology especially for long run.

Please leave your comments if you want.