ESX Virtualization

VMware ESXi, vSphere, VMware Backup, Hyper-V... how-to, videos....

VCP6.5-DCV Objective 7.3 – Troubleshoot vSphere Upgrades and Migrations

By | Last Updated:

Shares

Today's objective is VCP6.5-DCV Objective 7.3 – Troubleshoot vSphere Upgrades and Migrations. Its kind of a “pain” topic where you must remember many different log paths for different products, remember some error codes, but that's the way it is. Remeber and succeed or die hard trying. Work hard and win. Motivate yourself, go forward and learn.

Check our VCP6.5-DCV Study Guide page which is starting really to take shape and fills up with topics from the Exam Preparation Guide (previously called Exam Blueprint). I highly recommend getting the full vSphere 6.5 documentation set and the latest exam preparation guide (PDF) as well when preparing for the exam. I'm not linking directly in purpose as those PDFs can change URL, but you can find those easily through Google search.

You have a choice to study towards the VCP6-DCV – Exam Number: 2V0-621, ( it has 28 Objectives) or going for the VCP6.5-DCV (Exam Code: 2V0-622) which is few chapters longer (it has 32 Objectives). Both exams are valid for two years, then you have to renew. You can also go further and pass VCAP exam, then VCDX.

Note: VMware released new “bridge” upgrade path. Any VCP who is two versions behind the most current available version in the same solution track can upgrade to the latest by only taking one exam. Previously, you would first have to earn the VCP6-DCV (and passing the delta exam) and then upgrade to the VCP6.5-DCV (again, through the delta exam.) Now you can go straight from VCP5 to VCP6.5.

Exam Price: $250 USD, there are 70 Questions (single and multiple answers), passing score 300, and you have 105 min to complete the test.

Check our VCP6.5-DCV Study Guide Page. 

You can download your free copy via this link – Download Free VCP6.5-DCV Study Guide at Nakivo.

VCP6.5-DCV Study guide PDF

Troubleshoot vSphere Upgrades and Migrations

Collect upgrade diagnostic information

You can collect upgrade diagnostic information through logs being made during the installation/upgrade process. For VMware vCenter server appliance (VCSA) access the appliance shell. You can do so by pressing the Alt + F1 if you have access directly to the appliance.

If you want to connect remotely, you can use SSH and start a remote session. Once connected, run the pi shell command to access the Bash shell. In the Bash shell run the vc-support.sh script to generate a support bundle.

It generates a support bundle as .tgz in /var/tmp.

Export the generated support bundle to the [email protected]:/tmp folder.

scp /var/tmp/vc-etco-vm-vlan11-dhcp-63-151.eng.vmware.com-2014-02-28–21.11.tgz [email protected]:/tmp

Determine which firstboot script failed.

cat /var/log/firstboot/firstbootStatus.json

Collect Installation Logs by Using the Installation Wizard – You can use the Setup Interrupted page of the installation wizard to browse to the generated .zip file of the vCenter Server for Windows installation log files.

If the installation fails, the Setup Interrupted page appears with the log collection checkboxes selected by default.

Leave the check boxes selected and click Finish.

The installation files are collected in a .zip file on your desktop, for example, VMware-VCS-logs-time-of-installation-attempt.zip, where time-of-installation-attempt displays the year, month, date, hour, minutes, and seconds of the installation attempt. Get the log files from the .zip file on your desktop.

Manually get the logs – the installation directories

%PROGRAMDATA%\VMware\vCenterServer\logs directory, usually C:\ProgramData\VMware\vCenterServer\logs

%TEMP% directory, usually C:\Users\username\AppData\Local\Temp

The files in the %TEMP% directory include vc-install.txt, vminst.log, pkgmgr.log, pkgmgr-comp-msi.log, and vim-vcs-msi.log.

Open the installation log files in a text editor for examination.

Collect Database Upgrade Logs – You can retrieve the database upgrade logs after you complete the vCenter Server upgrade process.

Navigate to the database upgrade log locations. Open the database upgrade logs in a text editor for examination.

Database Upgrade Locations:

  • For pre-upgrade checks, review the %TEMP%\..\vcsUpgrade\vcdb_req.out file. The vcdb_req.err file tracks any errors that were identified during the pre-upgrade phase.
  • For export details, review the %TEMP%\..\vcsUpgrade\vcdb_export.out file. The vcdb_export.err file contains errors that were identified during the export phase of the upgrade.
  • For import details, review the ProgramData\Vmware\CIS\logs\vmware\vpx\vcdb_import.out file. The vcdb_import.err file contains errors that were identified during the import phase of the upgrade process.
  • For in-place upgrade log details, review the ProgramData\Vmware\CIS\logs\vmware\vpx\vcdb_inplace.out file.
    The vcdb_inplace.err file contains in-place upgrade errors.

Collect Logs to Troubleshoot ESXi Hosts – Enter the “vm-support” command in the ESXi Shell or through SSH.

Collect logs for ESXi with VM support command

Navigate to the /var/tmp/ directory.

Retrieve the log files from the .tgz file.

Recognize common upgrade and migration issues with vCenter Server and vCenter Server Appliance

The installation and upgrade precheck script runs tests to identify problems on the host machine that can cause an installation, upgrade, or migration to fail.

For interactive installations, upgrades, and migrations, the errors or warnings are displayed on the final panel of the installer, where you are asked to confirm or cancel the installation or upgrade. For scripted installations, upgrades, or migrations, the errors or warnings are written to the installation log file.

vSphere Update Manager provides custom messages for these errors or warnings. To see the original errors and warnings returned by the precheck script during an Update Manager host upgrade scan, you should review the Update Manager log file vmware-vum-server-log4cpp.log.

Error and Warning Codes That Are Returned by the Installation and Upgrade Precheck Script

64BIT_LONGMODESTATUS – The host processor must be 64-bit.

COS_NETWORKING – Warning. An IPv4 address was found on an enabled service console virtual NIC that has no corresponding address in the same subnet in the vmkernel. A separate warning appears for each such occurrence.

CPU_CORES – The host must have at least two cores.

DISTRIBUTED_VIRTUAL_SWITCH – If the Cisco Virtual Ethernet Module (VEM) software is found on the host, the test checks that the upgrade also contains the VEM software. The test also determines whether the upgrade supports the same version of the Cisco Virtual Supervisor Module (VSM) as the existing version on the host. If the software is missing or is compatible with a different version of the VSM, the test returns a warning. The result indicates which version of the VEM software was expected on the upgrade ISO and which versions, if any, were found. You can use ESXi Image Builder CLI to create a custom installation ISO that includes the appropriate version of the VEM software.

HARDWARE_VIRTUALIZATION – Warning. If the host processor doesn’t have hardware virtualization or if hardware virtualization is not turned on in the host BIOS, host performance suffers. Enable hardware virtualization in the host machine boot options. See your hardware vendor’s documentation.

MD5_ROOT_PASSWORD – This test checks that the root password is encoded in MD5 format. If a password is not encoded in MD5 format, it might be significant only to eight characters. In this case, any characters after the first eight are no longer authenticated after the upgrade, which can create a security issue.

MEMORY_SIZE – The host requires the specified amount of memory to upgrade.

PACKAGE_COMPLIANCE – vSphere Update Manager only. This test checks the existing software on the host against the software contained on the upgrade ISO to determine whether the host has been successfully upgraded. If any of the packages are missing or are an older version than the package on the upgrade ISO, the test returns an error and indicates which software was found on the host and which software was found on the upgrade ISO.

PARTITION_LAYOUT – You can upgrade or migrate software only if at most one VMFS partition on the disk is being upgraded and the VMFS partition must start after sector 1843200.

POWERPATH – This test checks for installation of EMC PowerPath software, consisting of a CIM module and a kernel module. If either of these components is found on the host, the test checks that matching components,such as CIM, vmkernel and module, also exist in the upgrade. If they do not exist, the test returns a warning that indicates which PowerPath components were expected on the upgrade ISO and which, if any, were found.

PRECHECK_INITIALIZE – This test checks that the precheck script can be run.

SANE_ESX_CONF – The /etc/vmware/esx.conf file must exist on the host.

SPACE_AVAIL_ISO – vSphere Update Manager only. The host disk must have enough free space to store the contents of the installer CD or DVD.

SPACE_AVAIL_CONFIG – vSphere Update Manager only. The host disk must have enough free space to store the legacy configuration between reboots.

SUPPORTED_ESX_VERSION – You can upgrade or migrate to ESXi 6.5 only from version 5.5 or 6.0 ESXi hosts.

TBOOT_REQUIRED – This message applies only to vSphere Update Manager upgrades. The upgrade fails with this error when the host system is running in trusted boot mode (tboot), but the ESXi upgrade ISO does not contain any tboot VIBs. This test prevents an upgrade that can make the host less secure.

UNSUPPORTED_DEVICES – Warning. This test checks for unsupported devices. Some PCI devices are not supported in ESXi 6.5.

UPDATE_PENDING – This test checks the host for VIB installations that require a reboot. This test fails if one or more such VIBs is installed, but the host has not yet been rebooted. In these conditions, the precheck script is unable to reliably determine which packages are currently installed on the host, so it might not be safe to rely on the rest of the precheck tests to determine whether an upgrade is safe.

If you encounter this error, restart the host and retry the upgrade.

Microsoft SQL Database Set to Unsupported Compatibility Mode Causes vCenter Server Installation or Upgrade to Fail – vCenter Server installation with a Microsoft SQL database fails when the database is set to compatibility mode with an unsupported version.

The following error message appears: The DB User entered does not have the required permissions needed to install and configure vCenter Server with the selected DB. Please correct the following error(s): %s

The database version must be supported for vCenter Server. For SQL, even if the database is a supported version, if it is set to run in compatibility mode with an unsupported version, this error occurs. For example, if SQL 2008 is set to run in SQL 2000 compatibility mode, this error occurs.

Make sure the vCenter Server database is a supported version and is not set to compatibility mode with an unsupported version.

Create/Locate VMware log bundles

Web Client and log in to the vCenter Server system > Global Inventory Lists, select vCenter Servers > Click the vCenter Server that contains the ESX/ESXi hosts from which you want to export logs >  Click the Monitor tab > System Logs > Click Export System Logs > Select the ESX/ESXi hosts > Select the Include vCenter Server and vSphere Web Client logs (optional).

Select the system logs that are to be exported > Select Gather performance data to include performance data information in the log files > Next > Click Generate Log Bundle. The Download Log Bundles dialog appears when the Generating Diagnostic Bundle task completes. Click Download Log Bundle to save it to your local computer.

Determine alternative methods to upgrade ESXi hosts in event of failure

Upgrade Hosts Interactively – To upgrade ESXi 5.5 hosts or ESXi 6.0 hosts to ESXi 6.5, you can boot the ESXi installer from a CD, DVD, or USB flash drive.

Installing or Upgrading Hosts by Using a Script – You can quickly deploy ESXi hosts by using scripted, unattended installations or upgrades. Scripted installations or upgrades provide an efficient way to deploy multiple hosts.

PXE Booting the ESXi Installer – You can use the preboot execution environment (PXE) to boot a host. Starting with vSphere 6.0, you can PXE boot the ESXi installer from a network interface on hosts with legacy BIOS or using UEFI.

Upgrading Hosts by Using esxcli Commands – By using vSphere CLI, you can upgrade a ESXi 5.5 host or ESXi 6.0 host to version 6.5 and update or patch ESXi 5.5, ESXi 6.0, and ESXi 6.5 hosts.

Here are some posts (step-by-steps) which was a lab work:

Configure vCenter Server logging options

vSphere Web Client > Resources, select vCenter Servers > Click the vCenter Server to update the level of logging > Settings tab > General > Edit > Logging Settings > Select the level of logging from the Logging Options dropdown > Click OK when finished

The available options are:

  • None (Disable Logging)
  • Turns off logging
  • Error (Errors Only) – Displays only error log entries
  • Warning (Errors and Warnings) – Displays warning and error log entries
  • Info (Normal Logging – Default) – Displays information, error, and warning log entries
  • Verbose (Verbose) – Displays information, error, warning, and verbose log entries
  • Trivia (Extended Verbose) – Displays information, error, warning, verbose, and trivia log entries

Another objective finished today. VCP6.5-DCV Objective 7.3 – Troubleshoot vSphere Upgrades and Migrations. Make sure to bookmark the VCP6.5-DCV Study page.

Again, we higly recommend to get the latest PDF documentation set when stugdying for the exam.

More from ESX Virtualization

Stay tuned through RSS, and social media channels (Twitter, FB, YouTube)

Shares
Vote !

About Vladan SEGET

This website is maintained by Vladan SEGET. Vladan is as an Independent consultant, professional blogger, vExpert x16, Veeam Vanguard x9, VCAP-DCA/DCD, ESX Virtualization site has started as a simple bookmarking site, but quickly found a large following of readers and subscribers.

Connect on: Facebook. Feel free to network via Twitter @vladan.

x