The fifth part of Using ESXi to replace ESX. After Staring point, Architecture, Installation, Configuration, this is the last part of ESXi era. There are still much more I can write about after this post, like daily check script, monitoring VMs on the fly, but most of rest of stuff are similar as what you would do to ESX4. Plus, I think it’s time for me to finish this series so I can start to work on something else later on.
I hope no matter who read this post can point out what I have missed or mistakes I made. I would much appreciate it.
Every IT administrators know how important Log files are. Without log files, we will loss our senses and don’t know what’s going on in the black box. ESX4 has quite complex logs which I will reveal briefly in this post. ESXi logs are easier than ESX but lack of documentation always troubles me. I will try my best to explain what I know about ESXi logs.
How many logs are there and where you can find them?
One thing I dislike about Vmware is they intend to put logs everywhere instead of have one central locations to store all logs (log server can do that, I will talk about it later). There are few tools we can use to collect logs but let’s check out what those logs are.
Logs for vCenter Server:
vpxd log: vpxd log is vpx daemon (or service) logs. It’s archived when it reaches 5MB or vpxd.exe service is restarted. In default, maximum 10 vpxd log files are maintained. The eight oldest log files are compressed.
vpxd-profiler log: this log file is to gather performance-related information.
location: C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs
All other windows installed components logs can be found at similar path as showing above.
Logs for ESX & ESXi Host:
main host log files:
/var/log/messages -Operating a system log (ESX/ESXi, 4 weeks)
/var/log/vmware/hostd.log -Host agent log(ESX/ESXi, max 10 files)
/var/log/vmware/vpx/vpxa.log -vCenter Agent log (ESX/ESXi, max 10 files)
/var/log/vmkernel -VMkernel log (ESX only, 36months rotate)
/var/log/vmkwarning -VMkernel warnings (ESX only, 4 weeks)
How to change Log level and log file locations?
Changing log level is important. Like what I have mentioned before, Vmware support can’t find much information from normal log level which is verbose. In terms of getting all details you want, you may want to setup your log level to “trivia”.
With ESXi, you have to go into different folder and modify different configuration file to switch the level and change your log file location. You don’t want to your log files stuff up your in-memory file system. Like all settings you want to change, you need to backup the original file first.
messages log
Messages log can be changed from vCenter console as what you can see from my last post.
or you can modify /etc/syslog.conf to make change.
hostd log
Configuration file: /etc/vmware/hostd/config.xml
Original file looks like this:
vpxa log files
vpxa is agent which talks to vCenter vpxd. If your HOST is showing disconnected from vCenter, this is a must check place.
Configuration file: /etc/opt/vmware/vpxa/vpxa.cfg
vpxa.cfg has verbose keyword but it lacks of location for where log files need to be stored. We can just manually add directory into it.
You have to modify log level and manually add directory in.
After you have done all those things, restart hostd daemon.
services.sh restart hostd
then, you can check out your local storage and see any files have created by system.
Forward your logs to central log server
We can forward logs to a central log server. This central log server can be vMA sitting on Linux, but I wouldn’t suggest since the size of vMA ovf is only 5GB altogether. Imaging if you have 20 HOSTs and each host (regardless ESX or ESXi) is using trivia log level. How much 5GB disk can hold up?
Plus, do you want log files sitting on vm or a physical box when issue happens?
Therefore, my recommendation is using dedicated syslog application and installed on your vCenter. vCenter should be a physical box and collect all logs. Plus, you should have logs sitting on your local host VMFS as well just in case vCenter is not available.
Setup syslog server on your vCenter
So you can either download free log server software like Kiwi log server from solarwind.
http://www.solarwinds.com/products/freetools/kiwi_syslog_server/
The only problem of using free version of kiwi log is you can’t rotate file and have to keep watching it in case it comes oversize and stuff you disk. Also, log files are sent via UDP(port 514) in realtime, it doesn’t actually sort different logs into different files. Hence, you will get one big file instead.
Setup ESXi Server to forward logs to syslog server
It’s very easy for esxi to setup for forwarding Logs to central syslog server. Just put server FQDN to syslog.remote.hostname.
Collecting and reviewing logs
There are different ways to collect logs and export logs.
You can use vm-support -w <path> to sets working directory used for output files. This is critical for ESXi since ESXi doesn’t have much space in in-memory file system.
You can use vCenter->File->Export System Logs, or view logs from vclient
You can use DCUI to view logs
oh, Well. I guess this is the end of my series of Using ESXi to replace ESX. One of my friends just told me that his VMWARE rep told him ESX will be rolled out of market and replaced by esxi in 2011. I’m glad I have made this series to help myself and everyone about ESXi. Please leave your message and comments. I will reply them once I got time. Thank you for reading.
-Silver
GeekSilver's Blog 