VM search

grep -i vmname hostd* | less
/vmname
grep -H -r vmname /vmfs/volume/*/*.vmx  would give the entires of vm which
are resgestered but not named on backened.
vmware-cmd -l|grep -i vmname
vm-support -l| grep -i vmname
vm-support -x |grep -i vmname
ps auxww | grep -i vmname

VM Resource Management

Respurce management in VM:

Memoery  Reservation:
Dedicated amount of pyscial memeory  for a VM all the time.It is the min amount of RAM required for VM to power on. If the reservation is not set vkernal uses Vmkernal Swap rather than the physical memeory which would degrade the performance of VM but the VM is powered on.
Once the memeory is reservered it can not be reclamied or used by other vm's or hypervisor.Only transperent page sharing can be used with memory reservation.
Memory resrvation incsrease SWAP file size decreases.Number of VCPU and Configured memoery determine the overhead.A Host has 4 GB of moeoery and you allocate 1 GB reservation to 4 VM's then you can not overcommit meoery since you have reserved all the memeory.
-------------------------------------------------------------------------------------------------------------
Memoery Limit:
The Key problem with the use of memory limits is thta they are enforced without aby guest OS awareness.
If a VM is configured  4GB of RAM guest OS thinks it has 4 GB of RAM and works accordinly .Then if you limit the memory to 2 GB the VMkernal forces the the Limit os 2GB out of 4GB and guest OS thinks it is using all 4GB and thus causing  VM disk swapping and degrades the performce

Bottamline :Vm with less memoery would perform better than VM is more moery and Memory limit.
Memeory Limit is useful in cases when you have limited resources and you want to put a host into maintence mode in such case to limit the reources of VM.

-------------------------------------------------------------------------------------------------------------

Memoery Share :
Unlike memory limit and reservation .Memoery share would only come into effect when the ESXi is memoery constraint.VM with more share vlaue would get the preference in such cases.
Some people orgue that memeory share can be achived by memeory reservation yes they are correct but you are commiting the moemeory and thus limiting the number of VM's per host.
------------------------------------------------------------------------------------------------
Cpu reservation :
CPU reservation is smiliar to memoery reservation but differ in one accept that memeory reserved can never be reclaimed ,page to disk and shared memmory but CPU reservation can  use idle cpu cycles to other VM's and will be recliamed when the original VM needs it.
------------------------------------------------------------------------------------------------
Expandble Reservation in Resource pool :
Allows resource pools to barrow resources from parent host or parent resource pools only to satisfy VM reservations not to stisfy excess resource reservation.

NETIOC - > Net I/O --> DV switch -4.1 -->outgoing
stroage IOC -> datastore should be managed by single Vcenter -- > Latency --- >per disk.

Important concepts

Tranparent memeory page sharing is enabled by default and cannot be trun off (T/F) ?
False : enabled by default but you can trunoff using advanced setting on host.
service console is constantly sheduled on diffrent HECs(Hardware Execution Context HEC) (T/F) ?
false : service console does not participate in vKernal in constant sheduling and resheduling on HEC's
and remians on the same CPU at all the times.

------------------------------------------------------------------------------------------
vCPU needs to be sheduled on HEC .HEC is a thread that is sheduled on a physical processor.
vKernal shedules and resdules vCPU on diffrent HEC every 20 milliseconds with the exception of service consle
which is always sheduled on first HEC or physical CPU 0 and never chnaged.
-----------------------------------------------------------------------------------------
Ballon driver max can be set to 0-max75% default is 65% in advanced config
-----------------------------------------------------------------------------------------
Virtual machine swap file is created when it is powered on and deleted when it is powered off.
-----------------------------------------------------------------------------------------
Hyperthreading :The technology that allows for sheduling of mutilple threads on the same CPU is known as
hyperthreading
------------------------------------------------------------------------------------------
DRS cluster yellow overcommited resources
DRS cluster RED insufficent resources

DRS five start:
Most conservative :*****
ModeratelyConservative :****
Default:***
Modertely Aggressive:**
Aggresive :*
-----------------------------------------------------------------------------------------
vMotion Requirments :-
Vmotion does not work across CPU vendors
Vmotion does not work with cpu with different cpu setping or familes
Hyperthreading and cores are not relevant to vMoition
vMotion does not work cpu with diffrent multimedia instrations
NX/XD
vMotion :
When vmotion is intiated ,memoery bitmap is created to track the chnages and the process of copying the
pysical RAM from host to another begins .Quiese(cut-over) the VM and copy the contents of bit-map to destination.
vMotion and Svmotion can not be done performed at once on the VM and can be done consecutively.
-------------------------------------------------------------------------------------------------------
How can you reduce the amount of balloning the occur iside the VM ? reboot or poweroff
VM backup type ? file or image level backups.
File level backup : agent should be instaled in guest OS.
Back up VM files.VCB gives option for both files and image level backups.
VCB spports file backsup on windows-based systems and image level backups for all guest operating systems.
VCB and VDR are agerntless backup solutions
----------------------------------------------------------------------------------------
Outgoing port for HA heart beats and state synchronization
TCP 2050-2250
UDP 2050-2250
incomming port for HA and state sync
TCP 8042-8045
udp 8042-8045
----------------------------------------------------------------------------------------
HA concept:
Memeber of cluster send the heart beat every 1sec  in 4.0 and 10 sec in 10sec in 4.1.
When the other host does not respond for 15 sec the cluster intiates the VM restarts on other boxes.
DNS resolutions should work in all the hosts in cluster.
host isolation :split brain occurs when the host stoped reciving the heart beat in a HA.

Kill VM

Vmware-cmd ./guestname getstate
vmware-cmd ./guestname stop
ps -auxww | grep 7885
Kill -9 7885

vSphere 5 HA features

In vSphere 5.0, the vSphere HA feature has been completely rewritten by VMware from the ground up. This was done to increase the scalability, reliability, and usability of vSphere HA

Changes in VMware HA :
Scalability :-
One of the biggest changes with vSphere HA is that the concept of primary and secondary nodes has been
completely removed. The new model incorporates a master-slave relationship between the nodes in a cluster,
where one node is elected to be a master and the rest are slaves. The master node coordinates all availability
actions with the other nodes and is responsible for communicating that state to the VMware vCenter Server.

Realibility :-
One enhancement was the elimination of dependencies on any external component by vSphere HA. Specifically,vSphere HA no longer has any type of dependency upon DNS resolution by each host in the cluster. Eliminating this reduces the odds that an outage of an external component will have an effect on the operation of vSphere HA.

Another enhancement is the ability to enable communication between the nodes within a cluster through the
storage subsystem. Now vSphere HA will utilize multiple paths of communication, through the network and
storage. Not only does this allow for a greater level of redundancy, but it also enables better identification of the health of a node and the virtual machines running on it.

Virtual Machine Scalability and Added Device Support :-
Up to 32 virtual CPUs (vCPUs) and up to 1TB of RAM
Enhanced graphics capabilities, including 3D graphics support that enables a richer desktop experience
Broader device coverage, including support for 3.0 USB devices, smart card readers and EFI BIOS
New user interface support for configuring multicore vCPUs
Support for new guest operating systems including Mac OS X Server v10.6 (“Snow Leopard”)


Compatibility with Older Versions of VMware Tools and Virtual Hardware


vSphere 5 HA replaced AAM with FDM (Fault Domain Manager)
If you are doing host maintance or network uncheck host monitoring in the HA
Master-Slave /etc/opt/vmware/fdm

Network Partition: is the term used to describe the suitation in which one or more slave hosts cannot comminicate with the Master even though they still in network connected .HA uses heart beat datastores to detect wheather the partitioned hosts are still live and whether actions needs to be taken to protect VM's on those hosts.


Network Isolation :
Is the situation in which one or more slave hosts have lost all managment network connectivity.Isolated hosts can neither communicate with HA master or commnicated with other ESXi hosts.

Network Tunning

Network Management :-
VM Directpath :- Bypasses the Virtualization layer and lets the Virtual device talk directly to a PCI physical Device.but this is lose features like vmotion,snapshots,memoery over-commitment,FT,VMsafe .(Avanced Configuration)
-----------------------------------------
Jumbo Frames :-Less processing and more throughput .normal ethenet payload is up to 1500 bytes payload with jumbo frame we can send payloads up to  9000 bytes . To send jumbo frames all devices  end to end should be enabled  and VMXnet (enhanced ) or VMXNET3 should be used for VM's.
-------------------------------------------
TCP Offload :-
TCP offload Engine(TOE) NIC feature is not supported by vSphere
TCP Segmentation Offload (TSO) supported by vSphere on some limited NIC -
" a feature of some NICs that offloads the packetization of data from the CPU to the NIC "

-------------------------------------------
NICO :- Network I/O Control (NIOC) from 4.1 require Vdistributed switch configured on DVSwitch
 Similar to SI/OC with NIOC you can use shares to prorotize virtual network bandwidth .
--------------------------------------------------------------------------------
VMCI (VIrtual Machine Communication Interface (VMCI))
Super fast coomunication option between host and VM or between VM and VM .ONly works with programs designed to work with socket API like a web application and backned database.Does not work with TCP/IP.

VMCI should be enabled on VM properties and must be installed.
------------------------------------------------------------------------------
Network Metric's
net.usage.average
net.droppedRx.summation
net.droppedTx.summation
net.recived.average
net.trnasmitted.average
net.usage.minimum
net.usage.maximum.
--------------------------------------------------------------
esxtop --n for networking
       -- f for options
------------------------------------------------------
How to find if network is the problem
Primarly look at the usage in KBps and see if exceeding the bandwidth of the netwrokinterface
Drop packets
No traffice sent /recived
Errors on physical switch port.
-----------------------------------------------------

Memory Tunning

Memory Metrics:-
Granted :- is total amount of memory configured to all the Virtual Machines in  total.
Active :- is the amount of memory pages the VM touched in recent times.Amount of guests "physical" memory actively used.
Consumed :- is the amount of memory the VM thinks it is going to use.Consumed memory is shared memory and memory that might be reserved but not actually used.Consumed is something like application requires like SQL +Vcenter service may need 3.5 GB but it might not use all 3.5Gb all the time.

----------------------------
Average Memoery Active
mem.active.average on host or VM
Memoery estimated to be used based on recently touched memeory pages -this is the smallest of the active,consumed and granted.
----------------------------
Average memory consumed
mem.consumed.average
Amount of memory consumed by one or all virtual machines is Memory granted - Memmory Saved by sharing
---------------------------------
Average Memeory Swapped in or out
-mem.swapin.average or mem.swapout.average
Amount of memmory swapped in or out of disk.
--------------------------------
Average memoery Swapped
Total amount of memory swapped.
--------------------------------
Average Memory reclaimed by Ballooning
-Mem.vmmemctl.average
-Memory reclamied by using ballooning
----------------------------------
ESXTOP :
h : To add or remove feilds
f : to added fields
----------------------------------
vKernel capacityview free tool and App view

Storage Performance

Capacity  (GB or TB)
Performance(MBytes persec)
Throughput (I/O  persec)
Latency (Milleseconds)

7200 RPM SATA :80 i/o ps
10K RPM SATA/SAS/FC :1200 i/o ps
15k RPM SAS/Fiberchannel :180 i/o ps
SSD (Solid State Device) MLC :1000-2000 i/o ps
EFD (Entp Flash Drive) SLC Single cell Tech :6000-30,000 i/ops

VMFS filesystem in SAN,iSCSI,FC ;NFS filesystem in NAS.

Blocked Based Storage:-SAN Fiber Channel (1- 20 GB speed) or iSCSI (100MB to 10 GB speed)

File Based Storage:-NFS (100 to 10 GB speed)

VMFS is used on local SCSI, FC, or iSCSI SAN (but not NFS)

SIOC(Storage I/O Control) is only implemented from 4.1
This will allows us to allocate share on per VM basis so that VM can access Storage I/O according
to the share allocated to them.
When Storage I/O Control is applied to Datastore cobfigured to multiple ESX hosts accessing a same stroage array Storage I/O Controller make sure that VM get it allocated i/o share.

Stroage Performance Factors:-

1)Number of VM on LUN
2)SAN/NAS configuration-Caching,RAID level, and number of spindles on a LUN
3)VMFS or RDMS RDMS offer no better performce than VMFS in thick mode
4) VMDK thin or thick
5)LUN queue depth(howbusy the LUN on a storage Array
6)Storage Protcol being used.
7) Storage I/O control
------------------------------------------------
Storage Performance :-

1)Average Disk Read Rate and Write Rate in KBps/disk (disk.read.average and disk.write.average)

2)Disk.devicelatency.average,disk.kernallatency.average,disk.queue.latency.average,disk.totallatency.average

per disk in millisecs

3)Device latency shouldn't exceed 25 ms,
Kernel and  Queue latency shouldn't exceed 2 ms,
toal latency or command latency shouldn't exceed 27 ms

4)disk.commandsAborted.summation & disk.busRests.summation
Per disk number of commands aborted with Windows VMs,an aborted command will occur if the I/O command wasn't fulfilled in 60 seconds,by default
Any number > 0 indicates major storage I/O bottlenecks or physical storage issues

Esxtop:Davg(not exceed 25),Kavg,Gavg=dvag+Kavg(guest)total(not exceed 27),Qavg
u disk devices
F to change field
 
WWN :- 50:00:fn:jh:kj:th:kl First four 50:00 tell you the vendor name Qlogic,Emulex,CNA
CNA is only for FCOE.

VSAN is similar to VLAN .They Provide isloation between multiple logical SAN's exisitng
on  a common physical Platform.


Zoning : To ensure that a LUN that is required to be visible to multiple hosts in a cluster.

Port based Zoning :

WWN base zoning :

Each VMFS volume has a metadata file which contains information  about File locks.

SATP : SATP module handles path failover for a given  storage array and determine the failover

type for LUN.

PSP :Handles the actual path used for every I/O .

How do we know VMFS datastore is overloaded ?

Q-length  resxtop :U  stroage QUED colum

Profile Driven Storage : Works with VASA (Vspshere Stroage API for storage awarness )
prinicple behind profile driven storage simples allow vsphere admins to build
VM storage profiles that describe the specific storage attribute that a VM requires.

Vspshere 5 has a software ISCSI intiator.

Once the LUN is presented you have to go and create a VMFS datastore.


Expanding VMFS Datastore :

1)You can dynamically expand the VMFS datastore if you have space avalible in  LUN.

2)You can add extent if you do not have space in the LUN and then increase the VMFS volume

3)Each VMFS has a UNIQUE UUID

4)Choose keep exsisting signature if you want to mount the datastore copy without writing

a new signature only possible if the orginal is deleted or unmounted becoz of similar UUID


5)Choose Assign a new signature if you want to write a new signature onto the VMFS datastore

keep in  mind this operation is irrversiable .If you have resignaured the datstore containing VM's

you will likely need to reregister those VM's in the Vcenter Server since path is changed.


No-root Squash option will give full explicit read/write acess.

Memory Management in VM

Memory Over Commitment:-
Transperent Page sharing
Memoery Ballooning
Hypervisor Swapping
Memory Comperssion
----------------------------------------------------
On a physical server MMU is part of Mother board which allocates memoery to OS.
For Virtual machines just like the how Physical have dedicated Hardware MMU to manage memory on the OS.MMU is virtulized for VM using either software using Shadow page tables or Hardware Intel EPT(extended page tables) or AMD RVI (Rapid Virtulized Index).

When a VM starts it has no physical RAM allocated to it and RAM is allocated as needed and recliamed when not in use using Transperent Page sharing,Memoery Ballooning,Hypervisor Swapping and Memory Comperssion 4.1.

Hypervisor  use Pmap  for each VM.Pmap manages what the VM's has to the host Physical Memoery.
Shaow pages tables in Hypervisor handles what Guest VM and Host pysical and SPtables are used by the MMU

http://software.intel.com/en-us/articles/best-practices-for-paravirtualization-enhancements-from-intel-virtualization-technology-ept-and-vt-d/

Hardware Page Table Virtualization provides a hardware assist to memory virtualization, which includes the partitioning and allocation of physical memory among VMs. Memory virtualization causes VMs to see a contiguous address space, which is not actually contiguous within the underlying physical memory. The guest OS stores the mapping between virtual and physical memory addresses in page tables.

Because the guest OSs do not have native direct access to physical system memory, the VMM must perform another level of memory virtualization in order to accommodate multiple VMs simultaneously. That is, mapping must be performed within the VMM between the physical memory and the page tables in the guest OSs. In order to accelerate this additional layer of memory virtualization, both Intel and AMD have announced technologies to provide a hardware assist. Intel's is called Extended Page Tables (EPT), and AMD's is called Nested Page Tables (NPT). These two technologies are very similar at a conceptual level.

Transperent Page sharing:-
Vsphere reclaims memeory by removing redundant memoery pages with identical Content.This is similar to
Dedupe performed on Stroage volumes.Say if you have three VM with 2008 any identiacal meomory among the three is hashed so that memory can be reclaimed through transparent page sharing.

Memoery Ballooning:-
VMEMCTL is loaded into guest OS using VM's tools this puts presure on the Guest OS when hypervisor runs low on memoery .Guest OS invokes its memoery cleanup and reclimation process freeing up pages for the ballon so the physical memoery pages assigned to ballon are free for other VM's.

Hypervisor Swapping.:- more latency becoz host need to swap memeory to disk and free up the Page and memoery.

Memoery Comperssion:
Host memmory Pages swapped out are stored in a compressed format in the host memoery for faster access.

Vcenter Linked mode Installation

Vcenter Linked Mode Installation :-
Make sure Frewall is trun off.
Install the vcenter application with generic account domain admin rights.
Make sure Generic account can login as service and login as batch.
SQL Native Driver installed from Microsoft SQL based on the version.

Creating 64-bit DSN :-
1)Program files -ODBC driver for 64-bit
2)system DSN -> ADD
3)Put the name of vCenter,Descrition and SQL server name
4)Make sure the account you login has access to the Database and MSDB

Creating 32-bit DSN for Update manager which is 32 Bit always:-
1) C:\Windows\SysWOW64\odbccad32.exe
2)system DSN -> ADD
3)Put the name of Updatemanager,Descrition and SQL server name
4)Make sure the account you login has access to the Database and MSDB

Create a new SQL Server database user with database operator (DBO) rights

1  Create a vCenter Server database user with the db_datawriter and db_datareader permissions.
2  Open a query analyzer window with a user having DBO rights on the vCenter Server and MSDB   databases.
3  Locate the dbschema scripts in the vCenter Server installation package /<installation directory>/vpx/dbschema directory.
4  Run the scripts in sequence on the database.
The DBO user must own the objects created by these scripts. Open the scripts one at a time in the Query Analyzer window and press F5 to execute each script in the order shown here.
VCDB_mssql.SQLpurge_stat1_proc_mssql.sqlpurge_stat2_proc_mssql.sqlpurge_stat3_proc_mssql.sqlpurge_usage_stats_proc_mssql.sqlstats_rollup1_proc_mssql.sqlstats_rollup2_proc_mssql.sqlstats_rollup3_proc_mssql.sqlcleanup_events_mssql.sqldelete_stats_proc_mssql.sqlupsert_last_event_proc_mssql.sqlload_usage_stats_proc_mssql.sqlload_stats_proc_mssql.sql5  For all supported editions of Microsoft SQL Server (except Microsoft SQL Server 2005 Express), ensure that the SQL Server Agent service is running by using these additional scripts to set up scheduled jobs on the database.
job_schedule1_mssql.sqljob_schedule2_mssql.sqljob_schedule3_mssql.sqljob_cleanup_events_mssql.sql

Remove intallation on Flash cache memory Partition.

1. Run ESXi installer and hit "ALT F1" to get into shell prompt after installer has completely loaded.

2. Run fdisk -l to verify path to 4GB device.

3. Run fdisk <path to 4GB device>.

4. Delete partitions off of 4GB device with the "d" command and selected each partition one at a time.

5. Execute fdisk commands via the "w" write command.

6. Verified that the 4GB disk was cleared of the ESXi install via the fdisk -l command.

7. Rebooted the host to make sure ESXi loaded ok.

Script to look VM's booted on host during a HA event

Query against the hostd logs in /var/log/vmware of the vm-support bundles:

grep "2011-06-20" hostd*|grep "No bootable device"|grep -v ha-eventmgr|awk '{print $3}'|awk -F "/" '{print $5}'

ESXi Configuration Backups

Config Backup commands would back up all the Switch,Nic,VLans and all the configurations on the hosts.

Hosts requires a reboot.

vicfg-cfgbackup -s(save)   /tmp/backup
vicfg-cfgbackup -l(load)   /tmp/backup

Storage Identifier & Logs

Stroage Identifier are unique identifer  for each of the stroage devices connected to ESX/ESXi.
NAA:Network Address Authority
T10:any SCSI device
iSCSI IQN :iSCSI qualified name
MPX:Vmware generated,dev that doesn't respond

Stroage Logs:
PSA modules loaded at boot is found in
ESX- /var/log/boot-logs/sysboot.log
ESXi-/var/log/sysboot.log

Critical Stroage events:
ESX :/var/log/vmkernel
ESXi:/var/log/messages

Storage Commands

@esxcfg-nas   :Configure NAS storage
@esxcfg-mpath :Configure and view mutiple storage paths
@esxcfg-rescan:Rescan Lans
@esxcfg-scsidevs:To view all scsi devices that esxi host can see
@esxcfg-swiscsi:Software iscsi device
@esxcfg-volumes:
@vmkiscsi-tool:
@esxcli:Analyzing PSA & Mutlipathing,esxcli gives you the namespaces to dig deep into
Namespaces esxcli:
corestroage :to configure LUN masking or storage path masking.
Network:Network commands
nmp:Configure and view multipathing
swiscsi:configure software ISCSI

@esxcli nmp satp list
@esxcli nmp psp list
@esxcli nmp path list (all path of stroage devices)
@esxcli nmp device list(to view avalible devices)
@vmkload_mod:loads and unloads modules for VMkernel
@vicfg-module or esxcfg-module :to view/load/unload the modules
Note: when a module is enabled it does not take affect until it next reboot.
@vmware-cmd (gives all the operations we can perform on VM)
------------------------------------------------------------
esxtop & vscsiStats
@esxtop :Gives stroage latency and throughput.Latency is only provided for iSCSI and FC
@vscsiStats:gives reports on stroage counters from Virtual SCSI device in kernel,per VMDK for all.vscsiStats is only on ESX classic and can be installed on ESXi also
@esxtop d device
@esxtop v per VM stats

Storage terms

VMFS Datastores can be stored onlocal stroage ,FC SAN or iSCSI SAN.You can store VM's on NFS NAS but that doesnt use VMFS.

PSA :Pluggable Storage Architecture is a special layer of VMkernal making maltipathing flexible  and allows
3rd party mutipathing plugins (MPP).PSA takes effect when VMkernal sendsa SCSI command to access data on block device.

MPP:Multi pathing Plugins ( Like EMC Power path)

NMP:Native multipathing plugin is the VMware NMP(Native mutipathing plugin).NMP manages subplugins SATP and PSP

SATP:Storage Array Type Plugins knows how to comminicate with stroage arrays

PSP:Path selection Plugins takes the path to get to the stroage arrays

ALUA:

Notes :
MPP and NMP can  coexist and can be used on a LUN or per array basis.
MPP or NMP job is to discover physical Stroage Devices and determine the "Claim rules" to export the logical device.Claimrule are found in /etc/vmware/esx.conf.Path failover is delegated to SATP.Path Load Balancing is delegated to PSP.

esxtop for 2 hours

#vm-support -s -i 300 -d 7200 will collect logs for 2 hours in the VM support bundle

for i in `seq 1`;do esxtop -a -b -d 5 -n 360 > /vmfs/volumes/host1.csv.$i;done

Private Vlans

Private Vlans:

Evolution of PVLANS :
Vlans=Virtual Lans
1) Layer 2 broadcast Domain segmented into Logical networks.
2) Each Vlan has it own Ip Subnet and layer three device like router or a layer three Switch  is needed to route traffice between Vlans.
VLan ID: 0 to 4095 and if these are not enough for we go for private Vlans.

Private Vlans :
Pvlans segment a vlan into more vlans.PVlans allow us to isolate traffic (at layer2) between machines in the Vlans
Reqirments :DVSWITCH (Enterprise plus)
PVLANS Types:Primary or promiscus,isolated & comminity

Change Passwd in Single user mode

Single user mode :
Step1 :#reboot
Step2# Press key to enter grub
Step3# e-  to edit
Step4# select the second line to edit
Step5#Go to the end of line and add "single"
Step 6#select b to boot
Step 7# passwd <username>

vMA Logging the ESX/ESXi servers

vilogger to consolidate ESX/ESXi  logs on diffrent servers.

Adding an ESX/ESXi servers to vMA logging :-

vilogger enable --server <esx servername> --numrotation 20 --maxfilesize 10(MB) --collectionperiod 10

./vilogger --help

cd /var/log/vmware

Important Things ESXi:
ESXi servers don't save log files when the power is lost or when servers are rebooted

vMA Session Files

Session Files : Alternative way to Fast Pass

#cd  /usr/share/doc/vmware-vcli/samples/session

#./save_session.pl --help

Adding a ESX/ESXi server to session files :-
#./save_session.pl --server <esx servername> --username XXXX --password XXXXX --savesessionfile /home/vi-admin/esx147

#vicfg-nics -l --sessionfile /home/vi-admin/esx147

vMA Fastpass

vi-Fastpass :
Managing mutiple esx and esxi servers
vifptarget --help

Adding an ESX/ESXi server:
vifp addserver <esx servername> (To add the servers)
credientials

Listing the ESX/ESXi servers added:-
vifp listservers (to look at servers added to Vi-Fastpass) we can also add Vcenter

Setting an ESX/ESXi as target:-
vifptarget -s <esxi servername> (To set the target)
vicfg-vSwitch -l

Clear an ESX/ESXi targets:-
vifptarget  -c (to clear the target)

vMA(vSphere Management Assitant)

Vsphere Managment Assitant

64-bit Red Hat  Enterprise Linux,Vmware tools,vCLI
Esx should be 64 bit,ESX/ESXi 3.5 U2 or 4.0 later
Requirments :Single CPU only,512 MB,5GB Virtual Disk
Download vMA as OVF template from vmware and deploy

vMA Commands:
root#cd  /usr/bin
root#ls *vicfg*

Runing an vMA Command
vicfg-nics --list --server <esxi servername>
credetails

Configuring SNMP for ESX host

Step1 :root@etc]# cd snmp
Step 2:root@snmp]# ls
Step 3:root@snmp]# vi snmpd.conf
Step 4:root@snmp]# chkconfig snmpd on
Step 5:root@snmp]# esxcfg-firewall -e snmpd
Step 6:root@snmp]# service snmpd start
Step 7 :root@snmp]# snmpwalk -v1 -c public localhost .1.3.6.1.4.1.6876 |grep 6876.1

Intalling HP SIM Manually

Installing HP SIM agent on ESX box:

Step1 :tar -xzvf  hpmgmt-8.1.1-vmware3x.tgz
Step 2:rpm -ivh package name
Step 3:rpm -qa | grep hp
Step 4:root# service pegasus start
Step5:root# service pegasus status
Step 6: hpasm  activate or hpasm reconfigure
Step 7:service hpasm start
Step 8: Serivce snmpd start
Step 9: Firewall Config
esxcfg-firewall --openport 2301,tcp,in,hp-sim
esxcfg-firewall --openport 280,tcp,out,sim-cert
esxcfg-firewall --openport 2381,tcp,in,hpim
Step 10:service firewall restart
Step 11:chkconfig snmpd on
Step 12:esxcfg-firewall -e snmpd
Step 13:service snmpd start

opening and closing Firewall ports

Opening the Firewall ports :
Syntax: esxcfg-firewall --openPort <port_number>,tcp|udp,in|out,<port_name>

Examples :
 root]# esxcfg-firewall --openport 2301,tcp,in,hp-sim

 root]# esxcfg-firewall --openport 514,udp,out,syslog

 root]# esxcfg-firewall --openport 280,tcp,out,sim-cert

 root]# esxcfg-firewall --openport 2381,tcp,in,hpim

Closing the Firewall ports:
Syntax: esxcfg-firewall --closePort <port_number>,tcp|udp,in|out,<port_name>

Example :
 root]#esxcfg-firewall --closePort 6380,tcp,in


service firewall restart
esxcfg-firewall-e| d SERVICE

Syslog Config

Syslog Config

 # vi syslog.conf
# service syslog restart
# esxcfg-firewall -q|grep syslog
        syslog              : port 514 udp.out

KB Article :1005030

DRS

DRS Requirments:
You should have Enterprise or Enterprise plus
Requirments :
Vmotion requirments should be met.
-VM's have access to same networks
-Compatible CPUs on source and destination hosts
-Hosts have Vmotion enabled Vkernal interface
-No attached CD/DVD drives or VMDirectPath.

Packet/Protocol Analyser

Packet/Protocol Analyser

Physical:
Port-mirror,SPAN (Switch Port Analyzer),RSPAN
Basically you connect your Laptop to a Port or Group ports with same Vlan in promiscous mode.
promiscous mode accetps all traffic.

Linux Packet capture :TCPDUMP and Wireshark tcp anaylse the traffic.
Step1:Configure promiscuous mode to a new Portgroup.
Step2:Your packet capture VM where Wireshark is installed should be on the same portgroup where promiscous mode is conifgured.

TCPDUMP:
-i =specify the interface
-v =Verbose
-W =Write packets to a file
-r =Read packets from a file
tcpdump -i vswif0 host <ipaddress> to watch DNS traffic,arp traffic becoz of DNS lookup

Check for NX flag

VMotion fails : Incompatible CPU

Check for NX flag in the ESX host and  disable it in BIOS.

To check if NX flag is set on ESX host

root# cat  /proc/cpuinfo

VMware KB Article :1993

Uploading Diagnostic information to Vmware

FTP the bundles to Vmware

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1008525&sliceId=1&docTypeID=DT_KB_1_1&dialogID=175002379&stateId=0%200%20175000597

Exporting VCenter Logs
Exporting ESX and ESXi logs
Creating VMware Support Log File Bundles

Collecting Diagnostic information for Vmware products

Below links gives you information about collecting Diagnostic information of Various products in Vmware.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1008524

Vcenter and ESX and ESXi logfiles

Vcenter Log files :
-------------------
Windows 2008 C:\programdata\vmware\vmware Virtualcenter\Logs
Windows 2003 C:\Documents and settings\All Users\Application Data\Vmware\VirtualCenter\logs
Size,location,name and rotation can be manipulated with vpxd.cfg (5MB)file.

Vmware KB Article :
1021804

ESX log Files:-
---------------
/var/log/vmkernel  ---Vkernel messages
/var/log/vmkwarning
/var/log/vmksummary
/var/log/vmware/hostd.log--ESX service
/var/log/messages ---Service Console
/var/log/vmware/vpx/vpxa.log---vSphere Client Agent
/var/log/vmware/aam/vmware_<hstname>-xxx.log---HA
/var/log/vmkiscsid.og---iscsi
/var/log/boot-logs/sysboot.log ---boot log

Vmware KB Article:
1021800

ESXi log files :-
-----------------
/var/log/vmware/hostd.log--ESXi service
/var/log/messages ---Combinaton of Vkernel messages ,Hostd,Vmkwarning
/var/log/vmware/vpx/vpxa.log---vSphere Client Agent
/var/log/vmware/aam/vmware_<hstname>-xxx.log---HA
/var/log/boot-logs/sysboot.log ---boot log
ESXi logs vanish when reboot or poweroff.
Vmware KB Article:
1021801

Configuring Log rotation for the log files :
--------------------------------------------
/etc/logrotate.conf
hostd.log
/etc/vmware/hostd/config.xml
vpxa.log
/etc/opt/vmware/vpxa/vpxa.cfg


Acessing the logs:
-------------------
Vsphre clinet
SSH
DCUI with ESXi service Console

Troubleshooting Vmware HA

Common VMHA problems :

1)Misconfigured admission control
2)Lack of management network redendancy

VMHA agents monitor other hosts in the cluster with heartbeats sent on the management network
ESX it is vswif and ESXi it is vmk.After 15 secs if no heart is recived host is failed vms are failed over.

3)Misunderstood host roles (primary/secondary)
4)HA agent not loaded

VMHA Admission Control polices:
1)Amonut of host failures
2)Percentage reserved
3)Designated failover host (Reserving the Host for failure)

Troubleshooting NFS

Adding an NFS volume from ESX:
esxcfg-nas -a <datastore name> -o <nfs servername or ip> -s <mount point>
esxcfg-nas -l   list NFS volumes

Adding an NFS volume from VCLI:
vicfg-nas.pl --server <esx hostname> --add nfsshare --nasserver <ip/dns name> --share <mount point>

NFS logs :
ESX --->\var\log\vmkernel
ESXi--->\var\log\messages
cat \var\log\vmkernel | grep nfs

Troubleshooting steps for NFS :
1) Refresh the storage
2)Check VMkernel is configured correclty
3)Verify firewall ports UDP/TCP 111 & 2049 are open
4)Verify NFS client is open in Security profile/firewall
5) Test connectivity -vmkping from esx to nas server
6) Verify share & permissions on NAS server
7)Check the messages or vmkernel log file

References :
Vmware KB 1003967

Troubleshooting Vcenter issues

Vcenter Service VPXD  is the main services  that should run for the Vcenter to function properly.

Location of config files and logs in Win2008:
C:\programdata\Vmware\VmwareVirtualCenter\vpxd.cfg
C:\programdata\Vmware\VmwareVirtualCenter\logs

1) Make sure VPXD services in Vcenter is started.
2) Look for the credentials used for VPXD are  correct
3)Verify the Hostname or IP of the Database changed
4)Vcenter Database name changed
5)Username and Password changed for database
6)DSN changed (Data Source Name)
7)Verify type of DSN (32 or 64 bit)
8)Check the port 80,443,906 on Vcenter if they are used by some other services
9)Out of space on SQL servers or DB

Referrences :
Refer Vmware KB articles 1003979 & 1003928 & 1003926

Notes :-
netstat -an | more to check all listening ports
TCPVIEW from MS sysinternals to view all the ports with names.

ESX & ESXi Performance

Absolute best tool for ESX/ESXi Performance .Best way to see the performance of 
VM on a Host is using .This gives you real-time resource utilization statistics

esxtop for ESX
resxtop for ESXi  using VMA

Q- Quit
C- Switch to CPU Utlization Screen
1)look for CPU Ready value this is the amount of time it waits for Disk  or Network.
2)The %CSTP value represents The %CSTP value represents the amount of time a virtual machine with multiple virtual CPUs is waiting to be scheduled on multiple cores on the physical host. The higher the value, the longer it waits and the worse its performance.  Lowering the number of vCPUs reduces the scheduling wait time. the amount of time a virtual machine with multiple virtual CPUs is waiting to be scheduled on multiple cores on the physical host. The higher the value, the longer it waits and the worse its performance.  Lowering the number of vCPUs reduces the scheduling wait time.

M- Switch to memory Utilization Screen
D- Switch to Stroage (disk) adapter resource Utilization Screen
U- Switch to Stroage (disk) device resource Utilization Screen
V-Switch to Stroage (disk) VM resource Utilization Screen
N-Switch to Network resource Utilization Screen
I-Switch to interrupt Utilization Screen
P-Switch to Power Utilization Screen

These options apply both for ESX and ESXi
http://communities.vmware.com/docs/DOC-9279

My

My Favorite Song till Now

Reinstall HA to a ESX 3.5 host

1. Disable HA in your Cluster

2. Ensure the Startup/Shutdown is set to Disabled. To check this, go to the Configuration tab of your ESX host and check the Virtual Machine Startup/Shutdown settings. Ensure its set to Disabled.

3. Disconnect the ESX host in VirtualCenter. Right-click the host and click Disconnect At the service console, issue the commands:

      service mgmt-vmware stop

      service vmware-vpxa stop

      service vmware-vmkauthd stop

      service xinetd restart

     

4. Query the vpxa package

      rpm -qa | grep vpxa

5. At the service console, run the "rpm -e" command on the rpm file that displayed in the previous command.

      rpm -e <rpm_name>

6. This uninstalls the agent from the ESX host. Reconnecting the host to VirtualCenter re-installs the agent.

7. Remove the AAM Agents:

      rpm -qa | grep -i aam | awk '{print $1}' | xargs rpm -ef $1

8. Restart the services:

      service mgmt-vmware start && service vmware-vmkauthd start

9. Execute the vmware-cmd -l command to let you know when hostd is up and running so you can reattach it to VirtualCenter.

10. Log in to the VirtualCenter Server using VMware Infrastructure (VI) Client.

   Select the ESX host it in the inventory list.

11. Right-click on the ESX host and click Connect. This step initiates a re-install of the agent.

12. Reconnect the ESX host in VirtualCenter. Right-click the host and click Connect

13. Re-enable HA

Find Old Snapshot in Vcenter

param ( $Age = 30 )
$vm = Get-VM
$snapshots = Get-Snapshot -VM $vm
Write-Host -ForegroundColor Red "Old snapshots found:"
foreach ( $snap in $snapshots ) {
 if ( $snap.Created -lt (Get-Date).AddDays( -$Age ) ) {
  Write-Host "Name: " $snap.Name "  Size: " $snap.SizeMB "  Created: " $snap.Created
 }
}

Destroy Virtual Machine from Vcenter

$c = Get-content C:\VM.txt

foreach($a in $c)
{

get-vm $a | Remove-VM –DeleteFromDisk

}

Network Virtulization

Cisco Nexus 1000v (Virtulized Network Switch)

Application Virtulization

Server Virtulization

Check for Virtual Technology Enabled

root]# esxcfg-info|grep "HV Support"

         |----HV Support............................................3

0 - VT/AMD-V indicates that support is not available for this hardware.
1 - VT/AMD-V indicates that VT or AMD-V might be available but it is not supported for this hardware.
2 - VT/AMD-V indicates that VT or AMD-V is available but is currently not enabled in the BIOS.
3 - VT/AMD-V indicates that VT or AMD-V is enabled in the BIOS and can be used.

Script to add a New NFS Datastore to all hosts in cluster

New-datastore –nfs –vmhost (get-cluster “$clustername” | get-vmhost ) –name $datastorename –path $/path/nfs –nfshost $nfsserverip

Script to Add a New Portgroup to a Cluster or a Host

Get-Cluster -Name $clustername | Get-VMhost | Get-VirtualSwitch -Name $Vswitchname|New-virtualportgroup $portgroupname -Vlanid $vlanIDname