Skip navigation

Tag Archives: esxi


With new release of ESXi4.1, we all need to update vSphere components which includes HP CIM agent.

It’s very easy step if you have prepared.

Presteps:

1. We need to download hp CIM agent for ESXi 4.1 from here

2. We need vMA 4.1 or vSphere CLI available. Please refer to my last post about vMA 4.1.

3. We need to have http server ready. A simple MS windows server can do the job. You need to make sure IIS is installed.

4. Put your host into maintenance mode

Steps:

1. Downloading the 4MBs zip file from HP website. Here

2. Setup IIS so you can download zip from your internal site.

vihostupdate supports both local zip file and http zip file. But we don’t want to upload to little poor vMA (only 5GB size). so you want to build a IIS virtual directory just let vMA able to download zip file from other local server.

After you download the zip file, you put it into one of folders where IIS is installed (in my case, it’s E:\Install\vmdownload).

Then, you need to open IIS Manager and right click Default Web site and choose “New” for Virtual Directory.

Go through the wizard as default (read only will be fine and build a virtual direcotry. (In my case, it’s vmdownload).

You can just try to download that file from any IE and to test it.

Make sure you type full address and include filename as well because you don’t have browse rights to that folder.

3. Use vMA to update hp cim to your host

You need to log in vMA and target your host first.

Then, you need to check out and see what has been installed.

vihostupdate.pl –server <server> –query

If you can’t find the bundle file, you can use these command to install

vihostupdate.pl –server <server> –install –bundle <local_path>/rollup.zip –bulletin

After installation, you are required to reboot your hosts.

After reboot, you can run command to verity the installation.

Reference:

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=15351&prodSeriesId=1121516&swItem=MTX-b98795300e7947d88f19ab56d6&prodNameId=3288134&swEnvOID=4091&swLang=8&taskId=135&mode=4&idx=1


If anyone can recall, I wrote a post about vMA 4.0 before. With new vSphere 4.1 released, vMA has released a new version 4.1 to work with new vSphere 4.1.

During the installation and configuring vMA 4.1, I have encounter multiple errors. I would like to thank William Lam’s help from the forum. If you want to read more about vMA 4.1 scripting, please follow William’s blog in the reference.

What’s New about vMA 4.1?

Apart from vMA is using new OS (CentOS) and it’s using vSphere CLI 4.1, SDK for Perl 4.1 and upgrade version of VMware Tools, the new version if vMA brought us a different way of authentication.  AD Authentication. Also there are some new commands to replace the old one. I’m going to elaborate as follow.

Download vMA 4.1

Downloading vMA 4.1 is pretty easy. Anyone can go to here to download OVF file and related documents. vMA 4.1 is able to load on both vCenter 4.0 and vCenter 4.1. You can get pretty good idea about how to install from vma_guide. However, there are some mistakes in the docs I would like to point out later.

Configuration vMA 4.1

When you first time run vMA, it will give  you a wizard to let you configure vMA. If you miss the chance, you can run

sudo system-config-network-tui

to reactive the wizard.

Join vMA, and ESX(i) into Active Directory

Concept

First of all, let’s talk about the concept behind this topic. Why do we need to join vMA and ESX(i) into AD?

The reason we join the ESX(i) into AD is to easy our management and try to use less username and passwords to control ESX(i). As you all know, vCenter is in the AD already. In default, Domain admin has rights to log on vCenter and manage it. However, ESX(i) use local user database and you have to use root every time in terms of logging and execute command.

I believe the second reason for ESX(i) to join the domain is to help domain users for vCLI access. Let’s image you can log on vMA(or use vSphere CLI and your script files) with your own domain accounts and execute commands against the vCenter and Hosts directly. No need to remember another set of username and passwords anymore. Everything will be integrated with same service account or domain user account.

Join ESXi to Active Domain

Connect to your vCenter which has ESXi 4.1 as host.

If you type your domain in the filed then click “Join domain” button, you must use “username” instead of “domain\username”.

I followed the smooth blog to configure it, I got following error. So you must not user domain\username format.

After you join the ESX(i) 4.1 to AD, you can connect ESX directly with vSphere Client and go to permission and add your domain account into local user database. For the rest, you can follow with smooth blog in reference.

Join vMA 4.1 into Active Directory

This is also pretty straight forward operation.

You log on vMA 4.1 with vi-admin account (vi-user hasn’t enabled yet, you have to do it manually). then, you type

sudo domainjoin-cli join your_domain your_domain_admin_user

then, you type password as what vma_guide indicated. But you may see following warning after you join the domain.

Those pam module are part of CentOS module and they are designed to not only join vMA to Windows AD, so does Linux AD. So it’s normal for you to see those warning.

You can use sudo domainjoin-cli query to verify as what I did.

Connect to vCenter and ESX(i) Hosts

There are two different ways you can authenticate your vMA to vCenter and Hosts.

Active Directory Authentication

Like what I have mentioned above, the concept for this one is to let your admin to log on with vMA with their own domain account and able to run commands against vCenter and Host without typing multiple times username and password. Comparing with fastpass authentication, vMA doesn’t store username and password into local vMA box. More secure in certain way. You don’t need to have extra passwords to memorize.

PreSteps:

Your vMA must joined the domain.

Your vCenter must joined the domain.

If you want to directly operate on Host without using “–vihost”, your ESX needs to join domain.

DNS host file must be preconfigured so vMA will know what your vCenter/host IPs are.

customize server list

Modify DNS host files

Well, the reason we setup DNS hosts file is we want to just type server name or host name to make it work. No one wants to type 10.163.x.x all day.

The solution is using hosts file just like what we did on lmhosts for windows.

Steps:

Open console (or connect vMA with ssh tool , like putty) of vMA.

Login as vi-admin

The host file is located at /etc

You must use “sudo chmod a+w hosts” to make hosts file writeable.

Use “sudo vi hosts” to add your vcenter and host IP

Save and quit vi

One thing I must point out is all server name must be FQDN and no exception!

customize server list

vMA needs to know how many servers you may connect to (although it can only operate on one server a time). vMA needs to know which servers you are going to use AD authentication and which servers you are going to use fastpass authentication. That’s why you need to build a server list.

You must log on with vi-admin to build server list.

To view current server list.

vifp listserver -l

You must use “-l” parameter in terms of to see authentication method.

If server you want is not in the list, make sure DNS host file has configured and you can use following command to add.

vifp addserver yourhost –authpolicy adauth (this is for AD authentication)

or

vifp addserver yourhost (this is for fastpass authentication)

If you try to add vCenter, you must use domain admin account because vi-admin doesn’t exist in vCenter unless you manually added in. For Host, you need to type root password and vMA will automatically add vi-admin users into Host.

Notice: There is a big trick here. If system prompt and ask you username and password, you can type “domain\username”. But if you want to use domain\username in the command line, you have to use “domain\\username”.

Now, you are ready to connect your server.

Steps:

1. Log in vMA with your domain admin account (normal domain account will work too!! But they don’t have rights to operate on vCenter).

2. target your server (vCenter or Host).

You must target one object to send command with. If you don’t do that, you will get error message like

“Error connecting to server at ‘https://localhost/sdk/webService&#8217;: Connection refused”

3. Send command to object

If you target to a vCenter and your command is a HOST base command, you must “–vihost your_host_name” to tell vCenter which Host you want. Also, the name must be FQDN!.

Notice: I was told from Vmware Support, if you use “–vihost” , then you will be asked to type username and password again!

If you target to a Host, you can just use command and it should work.

Here is the tricky thing. It should work and you shouldn’t type any credentials anymore. But some of users like me do get asked to type username and password again! Maybe it’s a bug of vMA 4.1. I’m investigating this matter with Vmware as I’m typing.

——————————————————————————————————————–

New Updates about this issue.
I just got call from Vmware Support and they admited this is a bug in the vMA 4.1. They will

fix this issue in the next release.

——————————————————————————————————————–

Fastpass authentication

This is old authentication method as previous version. Basically, the vMA stored your credentials in the local and you don’t need to type multiple times when you operate on Hosts and vCenter. The reason for that is vMA actually create vi-admin accounts into Hosts.

PreSteps:

DNS host file must be preconfigured so vMA will know what your vCenter/host IPs are.

customize server list

Please check above post to look for details about how to do it.

This is reference for fastpass authentication.

Steps:

1. Log in vMA with vi-admin.

2. target your server (vCenter or Host).

You must target one object to send command with. If you don’t do that, you will get error message like

“Error connecting to server at ‘https://localhost/sdk/webService&#8217;: Connection refused”

3. Send command to object

If you target to a vCenter and your command is a HOST base command, you must “–vihost your_host_name” to tell vCenter which Host you want. Also, the name must be FQDN!.

Reference:

http://communities.vmware.com/community/vmtn/vsphere/automationtools/vima

http://www.virtuallyghetto.com/2010/07/vma-41-active-directory-intergration.html

http://www.smoothblog.co.uk/2010/07/15/esxi-4-1-active-directory-integration/

http://www.virtuallyghetto.com/2010/07/vma-41-authentication-policy-fpauth-vs.html


This is fix up for PVSCSI issue in ESX 4.0 U2. Previous link https://geeksilver.wordpress.com/2010/06/21/upgrade-esx-3-5-to-vsphere-upgrade-your-vms-to-get-performance-jump-part-1/

Basically, there are 2 errors you may encounter during adding pvscsi device  to windows 2k3 and 2k8.

1. PXE issue

After you added new disks, you encounter PXE issue. Even if disk you installed is the secondary disk, you will still encounter this issue.

Reason:

Vmware bug

New Updates:

Just spoke to Vmware support and they are able to reproduce this issue in their lab.

Solution:

Vmware vCenter 4 U2 can’t take too many options in adding and changing hardware at same time. so do one step a time.

E.g: add disk. then, click ok. Get into setting, change type of scsi, ok. etc.

New fix:

This issue is caused by boot SCSI card sequence has changed after you delete and add new SCSI controller.

All what you need to do is to make sure the SCSI (0:0) is in the first bootable position like what  you can see in the diagram.

2. Blue screen of windows

After changing boot disk SCSI controller type, you can see windows started, then, you encounter blue screen. The system keep restarts.

Cause: Windows doesn’t have your SCSI driver.

Solution: You must have all SCSI drivers available in device manager before you load all type of disks. If you build machine with PVSCSI, you won’t have LSI SCSI driver. So you need to add secondary disk of LSI to let vmtools to install driver.

-Silver


Before you start:

I have fixed PVSCSI few issues in my another post. Please check out if you do encounter problems.

https://geeksilver.wordpress.com/2010/06/21/upgrade-esx-3-5-to-vsphere-fixing-part-1-5/

I really hope you guys like my last series ESXi era: Using ESXi to replace ESX. I will start a new series which is upgrading ESX 3.5 to vSphere. I believe lots of companies are facing with this issue while ESX 3.5 support reaches it’s own lifetime and vSphere become more matured. This series is not only talking about how to upgrade, it also discuss how to optimize your performance and new technology used in vSphere.

Since this is a large projects if you have big environment, I would like to break them down and bring them to you with not exactly project wise order. I will make a list of ordered posts at the very end of this series.

All right, let’s hit the ground and roll out!

One of issues for upgrading ESX3.5 to vSphere is to upgrade your VMs. Your VMs are using Vmware hardware version 4 which doesn’t give you too much performance boosts. I’m going to guide you through how to upgrade a VM (I’m using Windows 2008 R2) to new version and optimize more. There are lots of detail information will be blended in this process.

Test Environment

In my test environment, I have a vCenter 4 U2(build 258672) and 1 ESX 3.5 U5 (build 226117) and ESXi 4.0.1 U2(Build 261974, You have to redownloaded this version of ESX 4 update 2 since latest update has issues with View).

I added those 2 hosts into vCenter at same time which will be exactly similar situation you will encounter in the future. So I created a VM(Windows 2008 R2) from ESX3.5 and ready to be moved to vSphere. This VM has 3 hard disks. 1 OS and 2 data disk. As you can see from this picture.

be noticed that the nic card the vm current has is E1000. Also the SCSI controller is LSI Logic Parallel.

As you can see from above picture, the OS is win2k8r2 and it’s running on hardware version 4.

That’s the vmtools version running on that VM.

Step 1: vMotion and Upgrade VMtools

First step is to vMotion to your ESXi server. I’m not going to solve the issues you may encounter on the vMotion, but I can recommend you to shutdown VM if you have to. That will help you a lot in many cases. You are going to have outage for the following procedures anyway. the next one is to upgrade VMtools. You must upgrade VMtools before you upgrade VM hardware. Otherwise, you may encounter missing and hidden drives issue I will mention later. So you  can either upgrade manually in VM console or just right click VM and choose guest->Upgrade vmtools.

Be noticed Upgrading VMtools will cause restart your VMs. Make sure you have turned off your monitoring services.

After upgrade VMtools, you should able to see new VMtools version.

Step 2: Upgrade VM hardware

VM hardware upgrade option will only appear when the VM is off. Hence, you need to power off your VM and right click Upgrade Virtual Hardware.

Notice: Once  you upgrade your VM HW to v7. You can’t vMotion VM back to ESX 3.5 server.

This is result of upgrading VM hardware. The Hardware version is 7. Nic is still using same nic which is E1000 in this case. Be noticed there is a new VMCI device added into your hardware list. VMCI in default is disabled. so What is VMCI?

The Virtual Machine Communication Interface (VMCI) is an infrastructure that provides fast and efficient communication between a virtual machine and the host operating system and between two or more virtual machines on the same host. The VMCI SDK facilitates development of applications that use the VMCI infrastructure.

First of all, VMCI is fast. Instead of going through 1Gb/s virtual nic to communicate  other VMs, it directly use memory to exchange data with other VMs on the same host. It can reach 24 times as fast as 1Gbps network connection in some case depends on your physical memory structure.

then, why not uses it? VMCI basically requires not only OS to support it, it also requires application to use VMCI stack instead of traditional TCP/IP stack. At this day, not many applications can do that. That’s why it’s disabled in default.

Step 3: Optimize your VMs with new technology

All right, let’s move on. so Hardware upgrade finished here? No. If you stopped here, you will lost a huge chunk of performance boost. There are at least 2 technologies you can apply to your VM so you can gain performance up and utilization of CPU down.

First technology, PVSCSI.

They were able to achieve 350,000 I/O operations per second on a single vSphere host (ESX 4.0) and with just 3 virtual machines. Their testing utilized the EMC Enterprise Flash Drives, which have an incredibly high throughput. They talked about how the VMware Paravirtual SCSI (PVSCSI) adapter was able to achieve 12% more throughput with 18% less CPU cost compared to the LSI virtual adapter.


There are also downsides with PVSCSI. First of all, it does  support bootable disk officially after U1.I have tested building new windows 2003 and windows 2008 with PVSCSI as boot disk successfully. All what you need to do is to connect floppy disk image when you install windows. Then, you need to hook up with PVSCSI floppy image which you can get them into 2 ways. Please follow this link to get more information.  From my tests, it failed when I tried to use it on existing boot disk on windows 2003 server and windows 2008. It also doesn’t support FT as well. But It’s worthy to use it.

The other technology is new VMnic card.

The paravirtualized network adapters in the VMXNET family implement an idealized network interface that
passes network traffic between the virtual machine and the physical network interface cards with minimal
overhead. Drivers for VMXNET-family adapters are available for most guest operating systems supported by
ESX.
The VMXNET family contains VMXNET, Enhanced VMXNET (available since ESX 3.5), and VMXNET
Generation 3 (VMXNET3; newly-added in ESX 4.0).

The paravirtualized network adapters in the VMXNET family implement an idealized network interface thatpasses network traffic between the virtual machine and the physical network interface cards with minimaloverhead. Drivers for VMXNET-family adapters are available for most guest operating systems supported byESX.The VMXNET family contains VMXNET, Enhanced VMXNET (available since ESX 3.5), and VMXNETGeneration 3 (VMXNET3; newly-added in ESX 4.0).

There are some tricky way to implement those hardwares. Let me demonstrate you as follow.

First of all, you need to know what’s your current IP configuration of E1000 card. Then, you can deleted the E1000 card when VM is power off. Now, switch to one of non-boot disk and notice that disk is connecting to SCSI(0:1) which means it connects the our first SCSIcontroller.

Now, you need to remove (disconnect) your 2 non-boot disks.

Make sure you are using “Remove from virtual Machin”. You don’t want to delete vmdk file from disk.

then, you press ok to apply these operation.

Your VM settings should like this picture. Notice VMCI device is not enabled.

We have already removed all old hardware now we need to reinstall them. Click Add to add new Ethernet Adapter.

Select VMXNET 3 card. Finish the wizard.

Now, let’s add (connect) removed hard disks. Be aware you will gain second SCSI controller during the procedure as well.

.

You have to choose SCSI (1:0). It will start install additional SCSI controller. At this stage,  you don’t have choice to choose type of SCSI card, but you can do it later.

.

Be aware new SCSI controller is there. Also new hard disk. Now, you can modify the type of that new SCSI controller. With vSphere, you can let multiple VMs to access same VMDK as well.

It’s time to add another removed disk.

That’s what it look like after finishing configuration.

After finishing configuration, it’s still few steps from making it work. You added and removed hardware for that VM, but VMWARE didn’t actually removed the hardware from OS level. What it did was to hide removed devices instead of actually removing it. so you have to do it by yourself.

Now, let’s start VM and restart it with it’s request since we added new hardware.

Just be aware the IP address has lost since we are using new nic. We have to get into system to remove old hidden nic card so we can add ip to new card. As what I mentioned above, the old card is still hidden and with IP configuration. Since Windows doesn’t allow same  IP to apply to nics in the same box, so you have to removed the old card.

After you finish restart, you log into vm via console and open command window and do following step.

There is also another trick. You must start device manager not computer manager which contains device manager. If you use computer manager, you won’t see hidden devices.

so once you opened device manager, you should see following picture.

You still to enable show hidden device. But if you don’t set environment variable and open device manager, you won’t able to see those hidden devices.

Next step is to remove those hidden devices.

then, you can add IP and finial picture should look like this one.

All right. This is end of part 1 of Upgrading ESX 3.5 to vSphere. I explained how to upgrade VMs and also use the new technology in vSphere. I hope you will like it and please feel free to leave comments.

Reference:

http://www.vmware.com/resources/techresources/10041

http://pubs.vmware.com/vmci-sdk/VMCI_intro.html

http://blog.petecheslock.com/2009/06/03/how-to-add-vmware-paravirtual-scsi-pvscsi-adapters/

http://derek858.blogspot.com/2010/06/extracting-vmware-esxi-pvscsi-boot.html


The fifth part of Using ESXi to replace ESX. After Staring point, Architecture, Installation, Configuration, this is the last part of ESXi era. There are still much more I can write about after this post, like daily check script, monitoring VMs on the fly, but most of rest of stuff are similar as what you would do to ESX4. Plus, I think it’s time for me to finish this series so I can start to work on something else later on.

I hope no matter who read this post can point out what I have missed or mistakes I made. I would much appreciate it.

Every IT administrators know how important Log files are. Without log files, we will loss our senses and don’t know what’s going on in the black box. ESX4 has quite complex logs which I will reveal briefly in this post. ESXi logs are easier than ESX but lack of documentation always troubles me. I will try my best to explain what I know about ESXi logs.

How many logs are there and where you can find them?

One thing I dislike about Vmware is they intend to put logs everywhere instead of have one central locations to store all logs (log server can do that, I will talk about it later). There are few tools we can use to collect logs but let’s check out what those logs are.

Logs for vCenter Server:

vpxd log: vpxd log is vpx daemon (or service) logs. It’s archived when it reaches 5MB or vpxd.exe service is restarted. In default, maximum 10 vpxd log files are maintained. The eight oldest log files are compressed.

vpxd-profiler log: this log file is to gather performance-related information.

location: C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs

All other windows installed components logs can be found at similar path as showing above.

Logs for ESX & ESXi Host:

main host log files:

/var/log/messages                              -Operating a system log (ESX/ESXi, 4 weeks)

/var/log/vmware/hostd.log           -Host agent log(ESX/ESXi, max 10 files)

/var/log/vmware/vpx/vpxa.log  -vCenter Agent log (ESX/ESXi, max 10 files)

/var/log/vmkernel                             -VMkernel log (ESX only, 36months rotate)

/var/log/vmkwarning                       -VMkernel warnings (ESX only, 4 weeks)

How to change Log level and log file locations?

Changing log level is important. Like what I have mentioned before, Vmware support can’t find much information from normal log level which is verbose. In terms of getting all details you want, you may want to setup your log level to “trivia”.

With ESXi, you have to go into different folder and modify different configuration file to switch the level and change your log file location. You don’t want to your log files stuff up your in-memory file system. Like all settings you want to change, you need to backup the original file first.

messages log

Messages log can be changed from vCenter console as what you can see from my last post.

or you can modify /etc/syslog.conf to make change.

hostd log

Configuration file: /etc/vmware/hostd/config.xml

Original file looks like this:

After modification:

vpxa log files

vpxa is agent which talks to vCenter vpxd. If your HOST is showing disconnected from vCenter, this is a must check place.

Configuration file: /etc/opt/vmware/vpxa/vpxa.cfg

vpxa.cfg has verbose keyword but it lacks of location for where log files need to be stored. We can just manually add directory into it.

You have to modify log level and manually add directory in.

After  you have done all those things, restart hostd daemon.

services.sh restart hostd

then, you can check out your local storage and see any files have created by system.

Forward your logs to central log server

We can forward logs to a central log server. This central log server can be vMA sitting on Linux, but I wouldn’t suggest since the size of vMA ovf is only 5GB altogether. Imaging if you have 20 HOSTs and each host (regardless ESX or ESXi) is using trivia log level. How much 5GB disk can hold up?

Plus, do you want log files sitting on vm or a physical box when issue happens?

Therefore, my recommendation is using dedicated syslog application and installed on your vCenter. vCenter should be a physical box and collect all logs. Plus, you should have logs sitting on your local host VMFS as well just in case vCenter is not available.

Setup syslog server on your vCenter

So you can either download free log server software like Kiwi log server from solarwind.

http://www.solarwinds.com/products/freetools/kiwi_syslog_server/

The only problem of using free version of kiwi log is you can’t rotate file and have to keep watching it in case it comes oversize and stuff you disk. Also, log files are sent via UDP(port  514) in realtime, it doesn’t actually sort different logs into different files. Hence, you will get one big file instead.

Setup ESXi Server to forward logs to syslog server

It’s very easy for esxi to setup for forwarding Logs to central syslog server. Just put server FQDN to syslog.remote.hostname.

Collecting and reviewing logs

There are different ways to collect logs and export logs.

You can use vm-support -w <path> to sets working directory used for output files. This is critical for ESXi since ESXi doesn’t have much space in in-memory file system.

You can use vCenter->File->Export System Logs, or view logs from vclient

You can use DCUI to view logs

oh, Well. I guess this is the end of my series of Using ESXi to replace ESX. One of my friends just told me that his VMWARE rep told him ESX will be rolled out of market and replaced by esxi in 2011. I’m glad I have made this series to help myself and everyone about ESXi. Please leave your message and comments. I will reply them once I got time. Thank you for reading.

-Silver


I just came back from Vmware Seminar 2010. There are lots of information I would like to share with you. You can clearly see  where Vmware is heading for it’s own future.

Future of Vmware: Cloud

You may heard some news about VmForce (Vmware combine with Salesforce to make cloud level ERP system), Vmware acquired SpringSource and public cloud, private clound. I was quite confused before I attended this seminar since I could hear everyone is talking about cloud but no real clouds system for private enterprise to merge or any real cases about cloud. This puzzle is resolved by this Seminar.

VMware wants to get rid of Microsoft

That’s it. That’s root cause why Vmware did all sorts of weird activities in past year. This is what Vmware has planned:

Make all companies virtualization 50% up (√) (85% world companies are using VMware tech)

Make all companies 100% virtualized

  • Vmware SRM helps DR (Expensive plan which requires 100% virtualization & DR budget, only about 5% companies are doing that)
  • vSphere helps servers platform (facing challenges from MS and Citrix)
  • VMware View helps Desktop (unsuccessful and beaten by Citrix  XenDesktop)
  • vThinapps (very few companies have actually used in the production, this is prestep for stripping apps from OS level in the future)

Using ESXi to replace ESX (has confirmed from all VMWARE people, it will happen in 2011. Vmware can finally get rid of head ache Red Hat for SC and have hardware appliance alike ESXi as house bricks)

Using VMSafe products (like Trend Macro, Agentless anti-virus. It will be available very soon).

Private Cloud era (VMware believes applications don’t need to run on Microsoft OS. They can let applications run directly on VMKernel. It’s obviously that Microsoft won’t agree with this idea. That’s why VMware bought SpringSource(Java application company) and try to make application platformless. I believe Microsoft will soon to push out their own cloud system and also use MS version thinApps and stream APPs to fight with Vmware)

Hybrid cloud (It’s also called Redwood project. This is next generation VM OS. If Private cloud is ready, meaning all apps can run on VMkernel without MS OS. Any apps can be seamlessly transferred between private cloud to hybrid cloud and even public cloud).

Public cloud (At that day, every system will run on a standard industry module  and can accept and transfer all applications)

In terms of convincing CIOs and IT Managers to purchase equipment to do VM DR and 100% virtualization, Vmware put lots of efforts on CapacityIQ, ChargeBack, how to shift attention from CAPEX to OPEX during the seminar. It only broadcasts one message. Come and virtualize everything!

Good plans, but there are concerns:

I agreed Vmware has drawn a beautiful picture of future I.T. But whether they can actually pull off this show is really a question mark. With all those components, like servers, Desktops, Networks, any parts of failure may cause huge disaster for Vmware. Vmware View is still not promising from angle of optimizing qos via WAN. PCoIP completely when it competes with Citrix ICA/HDX. Vmware users have to go back and use MS RDP protocol to connect Virtual desktop which gives Microsoft a chance to regain the market. Even in the latest demonstration, Vmware view still hasn’t fixed issue. From my personal understanding, Java applications has tons of issue. Slowness, stability problem. Novell has to dropped off Java console from Netware few years ago doesn’t mean anything to Vmware?


Here we are. The forth part of Using ESXi to replace ESX. Somehow I start to have feeling I’m not just updating my blog but also I’m writing a book? This ESXi series is getting longer and longer. But there are still much more to talk about if we want to use ESXi to replace ESX in the production.

This post is about how to configure ESXi host. I’m not going to discuss about same stuff you can do on ESX. What I’m discussing here is how to initialize and setup your ESXi so you won’t have further troubles in the future. As what I always said, I may make mistakes, please feel free to leave comments.  Thanks

At the end of last post, I was showing everyone this picture.

This is default result after you finish installation on your host. This interface called DCUI.

Direct Console User Interface (DCUI) — the low-level configuration and management interface, accessible through the console of the server, used primarily for initial basic configuration.

Please consider DCUI as lower level which means it’s more like backdoor of ESXi which allows you by pass any other security mechanism. You can use DCUI to setup root password and manage your ESX box. There are couple of things I would like to point out within DCUI.

Default root password is empty. Using DCUI to setup one ASAP.

The only way to setup ESXi initially is to use DCUI since ESXi box doesn’t have IP, root password is empty and SSH or other protocol is disabled in default. All what you need to do is to assign nic to management network.

Careful plan with your management network

Unlike ESX installation which will give you choice to choose which nic will be vmnic0, you don’t have choose in ESXi. Meaning, you better make sure all ESXi HOSTs are using same Hardward otherwise, you may have some issues on choose right nic and plug right nics to right switch ports.

Do not try to setup secondary gateway in you vCenter it will bring your Service Console offline.ESXi doesn’t have Service Console meaning you can’t setup 2 different gateway for your vmotion network and management network (in ESXi, we call it Management Network) or iSCSI. Considering Management Network as normal network which can have multiple network cards but only one gateway is allowed. The solution for this is to use esxcfg-route to setup multiple routing in your ESXi just like normal Windows.

esxcfg-route –h to get all command parameters

esxcfg-route -l to see the current list

esxcfg-route -a 192.168.100.0/24 192.168.100.1 to add new network routing
By saying that, your secondary network gateway in default is local subnet. So you don’t need to setup this one unless you want to customize it.

Use DCUI to clean up wrong assigned NICs if you made mistake.

The other function of DCUI is to recover nics from other vSwitch if you accidently lost connection to Management network by assigning wrong NICs.

Configuring SSH

By default, SSH is not enabled. You can enable ssh by following my instructions. Please be aware that enable SSH is not safe from security wise. ESXi Update doesn’t update SSH parts since Vmware believes it’s not official to access your ESXi box. Best practise is enable your ssh only when you want it. Disable it after you finish your operation.

1. Go to the ESXi console and press alt+F1
2. Type: unsupported
3. Enter the root password(No prompt, typing is blindly)
4. At the prompt type “vi /etc/inetd.conf”
5. Look for the line that starts with “#ssh” (you can search with pressing “/”)
6. Remove the “#” (press the “x” if the cursor is on the character)
7. Save “/etc/inetd.conf” by typing “:wq!”
8. Restart the management service “/sbin/services.sh restart”
10 – Kill inetd : kill `ps | grep inetd | cut -f2 -d” “‘
11 – Start inetd: inetd


inetd is a daemon which is monitoring all network activity and it will starts appropriate daemon to interactive with request. Using KILL command to terminate inetd will make inetd reread inetd.conf and restart daemon.

Enable TechSupportMode

Yes, that’s actually enabled by default, believe or not. If you press Alt+F1 at Console, and type “unsupported”, you will get chance to type in password so you can login and run some commands. There are too much discussion about this on the Internet. so I don’t need to explain too much.

You need make sure that check box is selected to get this function.

If you read my previous post, you would know ESXi automatically create 4GB scratch partition. This partition is to use to store vmware upgrade file or other temp files and even log files. You need to make sure the space of that partition is big enough. Double check the path in this column. If it’s necessary, feel free to move local storage.

Syslog local path and remote syslog server

The funny thing about log is it is setup as in normal verbose by default. But verbose log won’t help you at all when your server crash. You pick up phone to ring Vmware support and they ask you to export your log and tell you that your log isn’t detail enough to give whatever they want. So you have to come back and setup to different level and wait for disaster happening again.

Also, you can move your log files to another location and setup remote log server at same time. I will discuss it in future post. Feel free to move logs location to local storage in VMFS.

Backup and Restore ESXi configuration

Warning: Please be cautious when you backup and restore your data. You have to make sure no VMs (on/off doesn’t matter) not sitting on this ESXi HOST. Otherwise, there will be orphaned VMS from Local Host database meaning that some inconsistent vms will become orphan from VC.

Solution for this one is to remove orphan VM from VC and reimport it from storage.

(This picture shows how to backup. But you have to connect to right host first).

With this new version of ESXi, you can backup your ESXi data into a single file from vMA(please refer to my previous post). Then,you can schedule to backup daily. The content of backup is not txt format. It is not only including network configuration, special settings on Advanced Settings, it also includes all registered VMs information as well. Therefore, when you restore your ESXi configuration, you will encounter orphan VMs.

(This picture shows how to restore and you have to type “yes” to continue, reboot after that)

You have to face with it when  you restore your configuration. Clean up duplicate orphan VMs. That’s all.

To be continued……..


This is third part of Using ESXi to replace ESX. I really hope I can myself clear and anyone who visit my sites would like it and enjoy the time when you are here. Please do leave comments and footprints. Thanks.

After discussing about staring point and architecture, it’s time for us to install and configure ESXi. There are some tips and tricks I would like to share with you in this chapter.

Install ESXi 4

First of all, you need to download vendor version of ESXi (please refer to PART 2) and you should load it with your ILO or something similar. You can ask your SAN team to block HBA or move that HOST out of Storage group but in this case, I didn’t do that since it’s merely lab test server.

The installation is pretty straight forward. You don’t have many choice to go. Basically, it’s either install or not. Once  you get into installation, the only thing you would be offer to make choice is where you want to put your files on.

As you can see, the first one is my local disk. Disk 1 is KVM adapter. Disk 2 is my Test SAN Lun. So go for local disk is what I want.

ESXi Partitions

I did brief ESXi partitions in last article. I will give you more details in this one. By default, the ESXi builds 3 partitions out of your local disk.

They are:

  • Swap Partition also called vFat Scratch partition (Used for vm-support to store temp space, upgrade, 4GB)
  • Diagnostic Partition (for CPU, memory core dump, 112MB).
  • bootloader Partition (4MB)
  • Primary boot bank (Core hypervisor(32MB), VMKernel, Server manufacturer customizatons, 255MB)
  • Backup boot bank (started with empty, will be filled later as backup of boot bank, 255MB)
  • Store partition (Auxiliary files, VI Client, Vmware tools, runtime storage, 917MB)
  • VMFS Partition (rest of DISK space)

Plus, visorfs (325.5MB) runs in-memory file system which holds /var/logs, /tmp,/etc/vmware, etc.

I mentioned this picture in my last post, but this is better version. From this picture, you can clearly see which partitions are and what’s they are for.  fdisk -l shows disk information. It’s all physical.  Please be aware the size of partition has been increased since ESXI4.

df -h is to display file system command. Be aware the first one (visorfs) is in memory. This is only display partition list which has been mounted from file system. It’s not all current not existed partitions.

This pictures indicates what has been mounted under /vmfs/volumes. Notice there are 6 mounted but only 5 of them got link files. The scratch doesn’t have link file but you can access from /scratch.

Well, this is ESXi. So you don’t get any other options to choose for the rest of installation. Let’s just quick go through it.

blah, blah, blah

blah,blah,blah

After reboot, you suppose to see this picture from your ILO or KVM. Then, ESXi installation is finished.

This interface is not just some quick menu of ESXi, it’s DCUI (Direct Console User Interface). I will leave it for next post.

Hope you can enjoy my post and not falling sleep. :p

To be continued…..


This is part 2 of whole series of Using ESXi to replace ESX. ESXi comes long way and still not taking major market for production. But I personally believer RCLI, VCenter+ESXi will be the future. Vmware has already developed hidden page to guide everyone to upgrade from ESX to ESXi. Yes, please read on and I will explain it later.

So this is sort of deep dive to ESXi system. However, there ain’t much information about ESXi4 so I have to add lots of own opinions. If I made mistake, please feel free to point out.

Ok. Let’s take look what the difference between ESX and ESXi at Architecture level.

This is ESX4 architecture. Essentially, everything is running on Service Console (Red Hat). VMKernel itself is an operating system just like all other OS. But it relies on SC to do all communication works and runs different agent. VMware agents (vpxa, hostd, etc) runs on SC and always run into stability issue after people install all other Hardware monitoring agents on SC. According to my experience, we always have some HA agents issue, SC stop responding to ping and heartbeats. Then, the issue resolved by itself after few minutes. All command lines are running on SC and then, SC forwards to VMKernel and wait for reply. This is long way to go and consumes lots of extra resources and bring tons of headache to VMWARE.

The above picture is ESXi diagram. I have to say, ESXi is not only a free product, but also a brand new design from architecture level. It has following advantages comparing with ESX.

Vmware agetns ported to run directly on VMKernel.

Let me bring up another diagram so  you can take a close look.

As you can see, vpxa, hostd and other important processors have migrated from SC to VMKernal. They are running on User world API stack and waiting for the communication from RCLI, vPowershell, and VC.

Authorized 3rd party modules can also run in Vmkernel. These provide specific functionality

  • Hardware monitoring
  • Hardware drivers

There are quite big changes in the Hardware monitoring world. VMware in default weak SNMP protocol (no SNMP trap set for ESXi) and focus on CIM broker.

The Common Information Model (CIM) is an open standard that defines how computing resources can be represented and managed. It enables a framework for agentless, standards-based monitoring of hardware resources for ESXi.

Basically, instead of using SNMP trap and query SNMP to your HOST, you should enable WBEM to do all the jobs. If  you want to deploy your ESXi system, you should not download ESXi directly from vmware site, but instead, you should go to your server company to download their special version of ESXi. For example, HP provides HP WBEM(Web Base Enterprise Management) embedded ESXi for free downloading. ESXi allows third-party to pre-install CIM Plug-ins and ESXi and plug-in can be upgraded separately. All what you need to do is to download HP SIM Manager and start querying. (In default, WBEM queries every 2 minutes)

With this design, ESXi can use agentless framework to let hardware monitoring system get full details of Host and also secured and prevent unexpected error caused by HW Agent (like HP SIM Agents).

The “dual-image” approach lets you revert to prior image if desired

This is very interesting design special for ESXi.

The ESXi system has two independent banks of memory, each of which stores a full system image, as a fail-safe for applying updates. When you upgrade the system, the new version is loaded into the inactive bank of memory, and the system is set to use the updated bank when it reboots. If any problem is detected during the boot process, the system automatically boots from the previously used bank of memory. You can also intervene manually at boot time to choose which image to use for that boot, so you can back out of an update if necessary.
At any given time, there are typically two versions of VI Client and two versions of VMware Tools in the store partition, corresponding to the hypervisor versions in the two boot banks. The specific version to use is determined by which boot bank is currently active.
As what the pdf says, ESXi alwasy keep another version of configuration file and other components. If boot fails, it can switches over like “Last good configuration” function in MS.
If you runs command fdisk -l in the ESXi, you will get following picture.

As you can see, the first part is Extended partition, also called Store partition. It’s about 917MB in ESXi4 instead of 750MB in ESXi 3. It stores Auxiliary files like VI client, VMWare tools, runtime storage etc.

The second partition is 4GB as what VMWARE called Scratch partition. Next one is VMFS partition. Partition FAT16<32M is bootloader partition. It remains as 4MB to choose which boot bank should be loaded. Boot bank (255MB in ESXi) contains core hypervisor code. Diagonistic Partition (112MB in ESXi4) is for core dump purpose. And the last one is Hypervisor 3 Locker. Once you start Locker mode, no remote access will be accepted.

As you can see, ESXi4 has 3 different boot options. Primary, backup and locker mode. It provides failover and security as well. I will review this part again in my next part (part 3).

No other arbitrary code is allowed on the system

Essentially, ESXi should be consider as an appliance with firmware. Yes, there are still quite few things you can mock around, like open SSH, setup SNMP TRAP, Backup configuration settings without host profile function, but comparing ESX 4, it’s much simple, easy, fast, efficient and safe.

Much less updates means much better stable system

Let’s see this diagram from Vmware, then it will explain by itself.

Finally, at last but not least.

VMWARE has developed web page to help user to Upgrade from ESX to ESXi. But you can’t find link from it’s parents page which is vSphere page.

I provide the link so you can see it by yourself.

http://www.vmware.com/products/vsphere/esxi-upgrade/architecture.html

To be continued….

Add-on:

One of my friends just questioned about ESXi and think ESX is better environment to execute his precious codes. I think Service Console has nothing to do with implement or execute code because this job is done by vSphere API. ESXi has vSphere API just like ESX and has no issue to execute any codes running on ESX.

For better understanding, I’m showing you this picture to prove my point.


Reference:

http://www.vmware.com/files/pdf/vmware_esxi_architecture_wp.pdf

http://www.vmware.com/products/vsphere/esxi-upgrade/architecture.html

http://docs.hp.com/en/5991-6518/ch01s06.html


First of all, let me express my personal feeling to people actually visit my site day by day. I have been away for one week to get MCITP:EA certificate and haven’t updated my blog for one week. But I’m just surprised by how many people actually keen to come back my blog.

Now, I’m off MS for a while and finally can dive into VMware world again and hopefully, I can get practical and deep as well. With no further duo, please allow me to bring up this new series topic, Using ESXi to replace ESX.

Why would you to use ESXi to replace ESX in the production?

I’m pretty sure that’s the first question you would ask. That’s also first question my manager asked as well. In the old time, ESXi is a free product and not even get supported by Vmware. And what’s wrong with ESX?

Well, if we double check what vmware said recently.

§First choice is ESXi Embedded, whenever possible.

ESXi is the future.

VMware has announced its intention to eventually replace ESX with ESXi.

§Design the management infrastructure to support VMware® ESXi, even if deploying VMware ESX™.

Limit the use of service console–based command-line management and monitoring agents

Develop ESXi management techniques now because ESX will eventually be phased out.

This message is from a vSphere Design workshop and blogged in vFrank.

As far as I understand, Vmware had enough headache to fixing RedHat version of Service Console. More than 50% percentage of Updates are delivered for SC.Thousands Support call are related to this poor Management console and Vmware believes ESXi can work better, lighter, and easier. According to my personal experiences, I’m very troubled by ESX console ping dropping. I have 2 or 3 software monitoring SC via ping, SNMP and there are always issues.

Concerns about ESXi

It’s free product, do I get support?

Yes, you do. I just have Vmware License team shoot an official Email to me to prove ESXi is supported as long as you have full production support. You will get exactly same support as esxi.

Does ESXi support vMotion, HA, FT, DRS, blah,blah?

Yes, they are. ESXi support all those functions with no issues.  I would like to provide you a link which is comparison of ESX and ESXi. http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1015000

So what kind of features does ESXi support?

I can show you a picture with most functions of ESX and ESXi supported. If you want to get full list, please follow this link. http://www.vmware.com/products/esxi/facts.html This page shows there are no functions ESX supports while ESXi doesn’t.

To be continued…….

Reference:

http://www.vfrank.org/2010/04/13/esx-will-be-phased-out-esxi-is-the-future/

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1015000

http://www.vmware.com/products/esxi/facts.html