Recovering from a Cisco 9500 Firmware ISSU Upgrade

Today, I had a activity to execute an IOS XE upgrade on a Cisco 9500 switch using from the current version to a newer version which is running in SSO mode.

I started by performing pre-checks on the verify that the switch is in the correct state and working as expected.

The pre-checks included:

  • Check the switch current IOS XE version

    C9500# show versions | in IOS XE
  • Check the switches connection

    C9500# show switch
  • Check the Flash free space

    C9500# dir flash: | in free
  • Check the state of switch ISSU

    C9500# show issu state detail
  • Check the install state

    C9500# show install summary

  • Check the switch is in SSO mode

    C9500# show redundancy
  • Check the switch that auto-boot is enabled

    C9500# show boot system
  • Check the exiting boot file on flash

    C9500# dir flash: | in bin


    Once the pre-checks has been completed. It is time to clean up the previous installation files.
    Use the following commands to perform this activity:
  • Copy the old image and backup current configuration to TFTP or USB drive

    C9500# copy bootflash:<old_image.bin> usbflash
    C9500# copy running_config usbflash

    OR

    C9500# copy bootflash:<old_image.bin> tftp:
    C9500# copy running_config tftp:
  • Remove any old image installation files from the flash (note that if you copy new image before this command, it will be included in the list of files to be removed)

    C9500# install remove inactive
  • Copy new image to be switch flash

    C9500# copy usbflash:<new_image.bin> bootflash:
    C9500# copy tftp:new_image.cfg bootflash:
  • Verify new image hash with Cisco website

    C9500# verify /md5 flash:<new_image.bin>
  • Start the ISSU to upgrade IOS XE

    C9500# install add file flash:<new_image.bin> activate issu commit

My colleague had recommended and experiencing switch upgrades first hand, it is a good practice to monitor the switch’s IOS XE upgrades from the console port while the switch is reloading.
You have visibility of any errors that shows on the console.
It is worth having the console session logged during the activity for reference that may be used for troubleshooting and support.

Recovering from the standby switch that booted into ROMMON mode with the error failed to Boot URL not found or incorrect.

  • Executed these commands on the standby switch in SSO mode

    ROMMON 1> dir flash:
    ROMMON 1> boot flash:<image-name.bin>
    OR
    ROMMON 1> boot flash:package.conf00-

The standby switch booted into the previous state.

  • Aborted the ISSU state on the active switch before standby switch loaded

    C9500# install abort issu

  • If ISSU failed to abort, these are other useful commands that was not necessary in my case. Enable service install that is required to clear install state

    C9500# configure terminal
    C9500(config)# service internal
    C9500(config)# end

    C9500# clear install state

I hope these commands were useful for managing switch upgrades.

reference:

https://www.cisco.com/c/en/us/support/docs/interfaces-modules/catalyst-6000-backplane-clock-modules/223261-troubleshoot-issu-upgrade-failures-on.html

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9500/software/release/17-18/configuration_guide/ha/b_1718_ha_9500_cg/configuring_issu_on_stackwise_virtual.html

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9500-series-switches/214406-in-service-software-upgrade-issu-on-ca.html#toc-hId–458892867

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9500-series-switches/222279-upgrading-catalyst-9500-switches.html

Recovering from a BitLocker System Lockout after a BIOS Update

We had a user who was not abiding by the rules to store all the company data on the network drives to ensure it is protected and backed up by our systems.

The user’s laptop got a BIOS upgrade and unfortunately the laptop was rebooting requesting Bitlocker password. In our environment, Bitlocker passwords are registered in Active Directory upon activation but for some reason unknown to us, it was not there for this particular laptop.

Our colleague at the site, research a way to overcome this challenge and found the following solution:

  1. Download an Application that will allow you to build a bootable USB drive in DOS 32-bit. E.g. RUFUS
  2. Execute your selected application to build bootable USB drive (You might have to first format USB drive at 32-bit).
  3. Download all BIOS updates for the computer model from the vendor support site. (If you remember the previously installed BIOS version, you can download that version from the site)
  4. Boot up the computer with the bootable USB drive and run each BIOS from the command prompt (At this point, you are trying to find the BIOS version before the update was done, if you already know the version just install it).
  5. After updating the BIOS, reboot the computer to verify if Windows boots up successfully.
  6. If unsuccessful, go back to Step 4 and execute the next BIOS version file.
  1.  

    NOTE: Execute your BIOS from the latest to the oldest.