Recovering from a Cisco 9500 Firmware ISSU Upgrade

Today, I had a activity to execute an IOS XE upgrade on a Cisco 9500 switch using from the current version to a newer version which is running in SSO mode.

I started by performing pre-checks on the verify that the switch is in the correct state and working as expected.

The pre-checks included:

  • Check the switch current IOS XE version

    C9500# show versions | in IOS XE
  • Check the switches connection

    C9500# show switch
  • Check the Flash free space

    C9500# dir flash: | in free
  • Check the state of switch ISSU

    C9500# show issu state detail
  • Check the install state

    C9500# show install summary

  • Check the switch is in SSO mode

    C9500# show redundancy
  • Check the switch that auto-boot is enabled

    C9500# show boot system
  • Check the exiting boot file on flash

    C9500# dir flash: | in bin


    Once the pre-checks has been completed. It is time to clean up the previous installation files.
    Use the following commands to perform this activity:
  • Copy the old image and backup current configuration to TFTP or USB drive

    C9500# copy bootflash:<old_image.bin> usbflash
    C9500# copy running_config usbflash

    OR

    C9500# copy bootflash:<old_image.bin> tftp:
    C9500# copy running_config tftp:
  • Remove any old image installation files from the flash (note that if you copy new image before this command, it will be included in the list of files to be removed)

    C9500# install remove inactive
  • Copy new image to be switch flash

    C9500# copy usbflash:<new_image.bin> bootflash:
    C9500# copy tftp:new_image.cfg bootflash:
  • Verify new image hash with Cisco website

    C9500# verify /md5 flash:<new_image.bin>
  • Start the ISSU to upgrade IOS XE

    C9500# install add file flash:<new_image.bin> activate issu commit

My colleague had recommended and experiencing switch upgrades first hand, it is a good practice to monitor the switch’s IOS XE upgrades from the console port while the switch is reloading.
You have visibility of any errors that shows on the console.
It is worth having the console session logged during the activity for reference that may be used for troubleshooting and support.

Recovering from the standby switch that booted into ROMMON mode with the error failed to Boot URL not found or incorrect.

  • Executed these commands on the standby switch in SSO mode

    ROMMON 1> dir flash:
    ROMMON 1> boot flash:<image-name.bin>
    OR
    ROMMON 1> boot flash:package.conf00-

The standby switch booted into the previous state.

  • Aborted the ISSU state on the active switch before standby switch loaded

    C9500# install abort issu

  • If ISSU failed to abort, these are other useful commands that was not necessary in my case. Enable service install that is required to clear install state

    C9500# configure terminal
    C9500(config)# service internal
    C9500(config)# end

    C9500# clear install state

I hope these commands were useful for managing switch upgrades.

reference:

https://www.cisco.com/c/en/us/support/docs/interfaces-modules/catalyst-6000-backplane-clock-modules/223261-troubleshoot-issu-upgrade-failures-on.html

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9500/software/release/17-18/configuration_guide/ha/b_1718_ha_9500_cg/configuring_issu_on_stackwise_virtual.html

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9500-series-switches/214406-in-service-software-upgrade-issu-on-ca.html#toc-hId–458892867

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9500-series-switches/222279-upgrading-catalyst-9500-switches.html

Leave a comment