A couple of nights ago, we upgraded the firmware of our Catalyst 6509. It was not supposed to be so complicated. Making backups and copying images back and forth would take a little bit of time. Rebooting a Catalyst 6509 will also take several minutes. But we wouldn’t anticipate taking more than half hour ordinarily.
We know Murphy’s Law. Anything that can go wrong, will. Murphy’s Law came more true than we expected.
The root cause of the problem we encountered was a corrupted firmware image. What we had on disk was not what we transfered to the Catalyst 6509’s flash memory. Can it get more wrong? Yes, eventually it turned out that what we had on disk to begin with was actually also corrupted. So, of course, the Catalyst 6509 switch could not boot up from corrupted firmware.
If that was not enough, our upgrade exercise got waylaid by a few other weird hardware glitches. First, the switch indicated a “minor fault” and a “temperature alarm”. The fan tray had a red light. We realized the fans were not spinning at all. One of the line cards had an orange status light also indicating a “minor fault”. While we were puzzling over these problems, just suddenly out of the blue, both power supply units showed a “DC output failure” light, and the switch was dead. Turn off, turn on, fortunately the switch started up fine, and the fans were spinning again. Hmm, but wait, the standby supervisor module failed to come online.
It was altogether a rather strange upgrade exercise.