Tag: ios

Recovering from a Cisco 9500 Firmware ISSU Upgrade

by Samuel Mitchell, posted in Cisco, Routing & Switching, Uncategorized

Today, I had a activity to execute an IOS XE upgrade on a Cisco 9500 switch using from the current version to a newer version which is running in SSO mode.

I started by performing pre-checks on the verify that the switch is in the correct state and working as expected.

The pre-checks included:

  • Check the switch current IOS XE version

    C9500# show versions | in IOS XE
  • Check the switches connection

    C9500# show switch
  • Check the Flash free space

    C9500# dir flash: | in free
  • Check the state of switch ISSU

    C9500# show issu state detail
  • Check the install state

    C9500# show install summary
  • Check the switch is in SSO mode

    C9500# show redundancy
  • Check the switch that auto-boot is enabled

    C9500# show boot system
  • Check the exiting boot file on flash

    C9500# dir flash: | in bin


    Once the pre-checks has been completed. It is time to clean up the previous installation files.
    Use the following commands to perform this activity:
  • Copy the old image and backup current configuration to TFTP or USB drive

    C9500# copy bootflash:<old_image.bin> usbflash
    C9500# copy running_config usbflash

    OR

    C9500# copy bootflash:<old_image.bin> tftp:
    C9500# copy running_config tftp:
  • Remove any old image installation files from the flash (note that if you copy new image before this command, it will be included in the list of files to be removed)

    C9500# install remove inactive
  • Copy new image to be switch flash

    C9500# copy usbflash:<new_image.bin> bootflash:
    C9500# copy tftp:new_image.cfg bootflash:
  • Verify new image hash with Cisco website

    C9500# verify /md5 flash:<new_image.bin>
  • Start the ISSU to upgrade IOS XE

    C9500# install add file flash:<new_image.bin> activate issu commit

My colleague had recommended and experiencing switch upgrades first hand, it is a good practice to monitor the switch’s IOS XE upgrades from the console port while the switch is reloading.
You have visibility of any errors that shows on the console.
It is worth having the console session logged during the activity for reference that may be used for troubleshooting and support.

Recovering from the standby switch that booted into ROMMON mode with the error failed to Boot URL not found or incorrect.

  • Executed these commands on the standby switch in SSO mode

    ROMMON 1> dir flash:
    ROMMON 1> boot flash:<image-name.bin>
    OR
    ROMMON 1> boot flash:package.conf00-

The standby switch booted into the previous state.

  • Aborted the ISSU state on the active switch before standby switch loaded

    C9500# install abort issu

  • If ISSU failed to abort, these are other useful commands that was not necessary in my case. Enable service install that is required to clear install state

    C9500# configure terminal
    C9500(config)# service internal
    C9500(config)# end

    C9500# clear install state

I hope these commands were useful for managing switch upgrades.

reference:

https://www.cisco.com/c/en/us/support/docs/interfaces-modules/catalyst-6000-backplane-clock-modules/223261-troubleshoot-issu-upgrade-failures-on.html

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9500/software/release/17-18/configuration_guide/ha/b_1718_ha_9500_cg/configuring_issu_on_stackwise_virtual.html

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9500-series-switches/214406-in-service-software-upgrade-issu-on-ca.html#toc-hId–458892867

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9500-series-switches/222279-upgrading-catalyst-9500-switches.html

Cisco 2960 Error: %ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error: Power Controller reports power supply VEE under voltage and lockout

by Samuel Mitchell, posted in Cisco, Routing & Switching, VoIP

I got a report that all our VoIP phone were down in a particular area that were connected to a Cisco 2960 switch. Upon investigation,

Initially, I ran the following show commands to check the status of the ports and power:

show power inline

The switch output:

showpowerinline

It was indicating that there was no device drawing power from the switch which was strange because there were VoIP phones connected to this switch.

With further checks, I came across these errors in the event logs:

poe_error

This error was telling me either the switch power module failed or some device caused a surge in power that causes the port power module to overload.

My resolution: Restart the switch which restored the switch to normal operations.

Being curious about the error, I researched it and came across this Cisco report:

https://quickview.cloudapps.cisco.com/quickview/bug/CSCsw18530

It was stating that this error is due to a bug in a particular version of IOS for 2960 switches.

The solution is:

Workaround:

Longer cable (>50ft) seems to fix the issue in most cases.

Solution

This issue is fixed in 12,2(55)SE3 , 12.2(58)SE1 and later releases. The
following interface configuration commands have to be applied to resolve the
issue,

Switch(config)# int x/y
Switch(config-if)# power inline port 2x-mode
Switch(config-if)# shut
Switch(config-if)# no shut

The fix is present in the following platforms :

3750-E,3560-E, 3750-X, 3560-X, 3560-C, 2960S, 2960,2960C,2960X

The other platforms do not support 2X power mode and the workaround would be to
use a longer cable.