STACKING CISCO SWITCHES

Each switch in the stack is referred to a member of the stack. Members work together as a unified system, appearing administratively as a single switch. A switch stack of 3650/3850s (note you can’t mix these in a stack) can have up to nine stacking-capable switches connected through their stackwise ports.

The alternative way of interconnecting the switches would be to daisychain the switches using the ports on the front of the switch. Cosmetics aside using stackwise is a better option for a number of reasons:

  • Doesn’t use up existing switchports
  • Stackwise maximises bandwidth between switches in stack (160G)
  • All switches in the stack appear as a single switch on the cli
  • Hot pluggable/swappable

What’s the most efficient way to configure these switches into a stack?

We’ll cover three areas.

  1. Configuring a stack of new switches
  2. Adding or replacing a switch in an existing stack
  3. Upgrading IOS

CONFIGURE A NEW STACK


First physically connect the switches using stackwise cables. Here’s Cisco’s recommendation:

347682

Note: The switch to be added to the stack must be off. Otherwise, the hole stack will reload.

1.- Add the new switch to the stack by connecting the stack cables to the switch (every cable has a cisco logo on the connector, it must be in the upright position) as shown in the picture below (1):

log.jpg

Each switch has two stack ports to which stackwise cables are connected. Viewed as above from the rear the stack port on the left is Port1 and the one on the right Port2. This is true of all switches in the stack.

These two stack ports on each switch are connected to their neighbor, defined as physically located above and below. The top and bottom switches in the stack are neighbors for the purpose of this connectivity.

It doesn’t matter which ports we use to arrive at this connectivity (stack port 1 or 2), but without what is effectively a ring topology between all switches in the stack we won’t have redundancy should one of the stackwise connections fail. We then run the risk of the stack being “split” (into two stacks due to lack of connectivity between switches) should only one link fail, resulting in connectivity issues for attached end devices as their shared network exists in two different places. Another reason to stack the switches in a ring is the stackwise cables are similar in function to a backplane in a chassis switch. Therefore if a member only has connectivity to one neighbor we’ve halved it’s bandwidth (see note 1 below).

Onto our lab. We have a stack of 3 x 3650s connected as per the above.

Switch#show switch stack-ports
Switch# Port1 Port2 
----------------------------
1 OK OK 
2 OK OK 
3 OK OK 

Switch#

OK. We’ve verified connectivity on each of the stack ports.

https://supportforums.cisco.com/document/12575126/how-form-catalyst-3850-data-stack-and-power-stack

CONCEPT: SWITCH STACK ROLES

Active – aka master. Operates the stack. Holds run and startup configs for stack. Only one active switch in stack. Standby – the “backup active” if there’s a problem. Only one standby switch in stack. Member – a switch in the stack that isn’t an active or standby switch. All other switches in the stack have this role. Just as with OSPF DR election, the stack master is non-preemptive. This means that to engineer the desired stack roles, the order in which switches are powered on can affect which switch in the stack becomes the active aka the master switch (although if we did use this method it’s a pretty poor way of doing it – see note 2 below). In view of the above output, in our lab the switches are physically stacked as follows

stack mac

In our lab the switches were taken out of sealed boxes, connected with stackwise cables in the recommended way and powered on simultaneously.

Switch#show switch 
Switch/Stack Mac Address : 74a0.2f45.2380 - Local Mac Address
Mac persistency wait time: Indefinite
                                          H/W   Current
Switch#  Role    Mac Address   Priority Version  State 
------------------------------------------------------------
*1      Active   74a0.2f45.2380    1      V01    Ready 
 2      Standby  74a0.2f58.7180    1      V01    Ready 
 3      Member   a0ec.f936.4d00    1      V01    Ready 

Switch#

Why have the switch roles resulted as per the above show switch output?

The answer is that should all switch priorities be equal, the switch with the LOWEST mac address wins and takes the active role. The next lowest mac in the stack takes the standby role, that is it will take over active should the current active fail (in which instance a re-election is avoided and there is no interruption to service). Other switches in the stack assume the member role up to the maximum total stack size of 9 switches.

The switch stack will operate fine in this configuration with one bugbear…

CONCEPT: SWITCH # (MEMBER NUMBER)

Note the first column in the show switch output above. Out of the box a switch will have a default member number of 1 (see note 1 below). Observe that the member numbers have been derived from the switch role, with active role taking a member number of  1. It’s the member number that determines the interface numbering for a switch in the stack.

A member number of 1 means the interfaces on that individual switch will be Gi1/0/1-48. However in our lab this switch is physically the second physical switch in the stack! Similarly Gi2/0/1-48 are on the standby switch, physically the first switch in the stack. To make administration of the switches less confusing, lets’s reconfigure the member number on the stack to align with how the switches are physically stacked.

Switch#switch 1 renumber 2
WARNING: Changing the switch number may result in a configuration change for that 
switch. The interface configuration associated with the old switch number will 
remain as a provisioned configuration. Do you want to continue?[y/n]y
Switch#switch 2 renumber 1
WARNING: Changing the switch number may result in a configuration change for that 
switch. The interface configuration associated with the old switch number will 
remain as a provisioned configuration. Do you want to continue?[y/n]y
Switch#show switch
Switch/Stack Mac Address : 74a0.2f45.2380 - Local Mac Address
Mac persistency wait time: Indefinite
                                           H/W   Current
Switch#   Role   Mac Address    Priority Version  State 
------------------------------------------------------------
*1       Active  74a0.2f45.2380    1      V01     Ready 
 2       Standby 74a0.2f58.7180    1      V01     Ready 
 3       Member  a0ec.f936.4d00    1      V01     Ready

Switch#wr
Building configuration...
Compressed configuration from 6558 bytes to 2242 bytes[OK]

Switch#show switch
Switch/Stack Mac Address : 74a0.2f45.2380 - Local Mac Address
Mac persistency wait time: Indefinite
                                           H/W   Current
Switch#   Role   Mac Address    Priority Version  State 
------------------------------------------------------------
*1       Active  74a0.2f45.2380    1      V01     Ready 
 2       Standby 74a0.2f58.7180    1      V01     Ready 
 3       Member  a0ec.f936.4d00    1      V01     Ready
Switch#reload
Reload command is being issued on Active unit, this will reload the whole stack
Proceed with reload? [confirm]y
<snip>
Press RETURN to get started!


Switch>en
Switch#sh switch
Switch/Stack Mac Address : 74a0.2f45.2380 - Local Mac Address
Mac persistency wait time: Indefinite
                                            H/W   Current
Switch#   Role   Mac Address     Priority Version  State 
------------------------------------------------------------
 1      Standby  74a0.2f58.7180     1       V01    Ready 
*2      Active   74a0.2f45.2380     1       V01    Ready 
 3      Member   a0ec.f936.4d00     1       V01    Ready 

Switch#

Note that whilst our original member numbers were derived from switch role, reconfiguring member numbers has no influence on election of active/standby. In this case the Active and Standby roles are still elected using mac address, all priority values being equal.

With our interface numbering fixed by changing the member number, aligning the logical switch numbering with the physical, it’s worth noting that technically it doesn’t really matter which switch is active and standby. Convention is to configure these roles as physical switch 1 and switch 2 in the stack respectively.

CONCEPT: SWITCH PRIORITY

Each switch in the stack has a priority value between 1 and 15, default 1 out of the box. Rather than using the mac address tie breaker during election of the active and standby switches, we can manually engineer which switch we want to assume these switch roles by changing the priority value. Higher is better. Let’s change the priority of the physical switch 1 at the top of our stack to 15 to ensure it’s takes on the active role.

Switch#switch 1 priority 15
WARNING: Changing the switchpriority may result in a configuration change for that
switch. Do you want to continue?[y/n]y
Switch#sh switch
Switch/Stack Mac Address : 74a0.2f45.2380 - Local Mac Address
Mac persistency wait time: Indefinite
                                            H/W   Current
Switch#   Role   Mac Address     Priority Version  State 
------------------------------------------------------------
 1      Standby  74a0.2f58.7180     15      V01    Ready 
*2      Active   74a0.2f45.2380     1       V01    Ready 
 3      Member   a0ec.f936.4d00     1       V01    Ready 

Switch#

Note that whilst the priority value has changed the role has not, election being non-preemptive. A reboot will cause a reelection using the revised priority values. For brevity, lets also change the priority values of physical switch 2 and 3, then reload the whole stack.

Switch#switch 2 priority 14
WARNING: Changing the switchpriority may result in a configuration change for that
switch. Do you want to continue?[y/n]y
Switch#switch 3 priority 13
WARNING: Changing the switchpriority may result in a configuration change for that
switch. Do you want to continue?[y/n]y
Switch#sh switch
Switch/Stack Mac Address : 74a0.2f45.2380 - Local Mac Address
Mac persistency wait time: Indefinite
                                            H/W   Current
Switch#   Role   Mac Address     Priority Version  State 
------------------------------------------------------------
 1      Standby  74a0.2f58.7180     15      V01    Ready 
*2      Active   74a0.2f45.2380     14      V01    Ready 
 3      Member   a0ec.f936.4d00     13      V01    Ready 

Switch#reload
<snip>
Press RETURN to get started.

Switch>sh switch
Switch/Stack Mac Address : 74a0.2f45.2380 - Local Mac Address
Mac persistency wait time: Indefinite
                                            H/W   Current
Switch#   Role   Mac Address     Priority Version  State 
------------------------------------------------------------
*1      Active   74a0.2f58.7180     15      V01    Ready 
 2      Standby  74a0.2f45.2380     14      V01    Ready 
 3      Member   a0ec.f936.4d00     13      V01    Ready
Switch>

Note that by setting the priority value of physical – physical switch 1 to priority 15, physical switch 2 to priority 14 etc. we also influence the logical switch member number to align with the physical switches. This will result in our interface numbering being correct. So, if we want a quick and easy way to setup a switch stack, priority is the single value to change (See the quickstart guide to provisioning a stack using only priority here)

Better understanding switch roles and member numbers however will assist with any troubleshooting when provisioning or making changes to existing switch stacks.

Quickstart: Provisioning a new stack

Step 1 Rack em and stack em. Physically install all switches to be stacked in a powered off state. Connect all stackwise cables. Did i mention don’t power on the switches?

Step 2 Power on physical switch 1 (top of stack). Insert a usb stick with your preferred IOS on into the usb slot on the front of the switch.

Switch#copy usbflash0:cat3k_caa-universalk9.SPA.03.07.03.E.152-3.E3.bin flash:
Destination filename [cat3k_caa-universalk9.SPA.03.07.03.E.152-3.E3.bin]? 
Copy in progress...
322991728 bytes copied in 52.370 secs (6167495 bytes/sec)

Switch#verify /md5 flash:cat3k_caa-universalk9.SPA.03.07.03.E.152-3.E3.bin
...Done!
verify /md5 (flash:cat3k_caa-universalk9.SPA.03.07.03.E.152-3.E3.bin) 
= 71d48b44bb5ec13d4b4d47d8c3dc9dd7

You can find the md5 of the file on Cisco.com or grab it from the file using md5 checker software e.g WinMd5 (other md5 checkers are available).

Step 3  Enable auto-upgrade of IOS. This will be used by other switch members joining the stack. This is off by default.

Switch(config)#software auto-upgrade enable

Step 4  Change the switch priority of physical switch 1 to priority 15, save. Now initiate the software upgrade which will also reload the switch.

Switch#sh switch
WARNING: Changing the switchpriority may result in a configuration change for that
switch. Do you want to continue?[y/n]y
Switch#sh switch
Switch/Stack Mac Address : 74a0.2f45.2380 - Local Mac Address
Mac persistency wait time: Indefinite
                                   H/W    Current
Switch#   Role    Mac Address    Priority Version State 
------------------------------------------------------------
*1       Active   74a0.2f45.2380     1      V01   Ready 

Switch#switch 1 priority 15
WARNING: Changing the switchpriority may result in a configuration change for that switch. Do you want to continue?[y/n]y
Switch#sh switchWARNING: Changing the switchpriority may result in a configuration
change for that switch. Do you want to continue?[y/n]y
Switch#sh switch
Switch/Stack Mac Address : 74a0.2f45.2380 - Local Mac Address
Mac persistency wait time: Indefinite
                                   H/W    Current
Switch#   Role    Mac Address    Priority Version State 
------------------------------------------------------------
*1       Active   74a0.2f45.2380     15     V01   Ready 

Switch#
Switch#wr 
Switch#software install file 
flash:cat3k_caa-universalk9.SPA.03.07.03.E.152-3.E3.bin 
Preparing install operation ... 
[1]: Starting install operation 
[1]: Expanding bundle flash:cat3k_caa-universalk9.SPA.03.07.03.E.152-3.E3.bin 
[1]: Copying package files 
[1]: Package files copied 
[1]: Finished expanding bundle flash:cat3k_caa-universalk9.SPA.03.07.03.E.152-3.E3.bin 
[1]: Verifying and copying expanded package files to flash: 
[1]: Verified and copied expanded package files to flash: 
[1]: Starting compatibility checks 
[1]: Finished compatibility checks 
[1]: Starting application pre-installation processing 
[1]: Finished application pre-installation processing 
[1]: Old files list: Removed cat3k_caa-base.SPA.03.06.00E.pkg Removed cat3k_caa-drivers.SPA.03.06.00E.pkg Removed cat3k_caa-infra.SPA.03.06.00E.pkg Removed cat3k_caa-iosd-universalk9.SPA.152-2.E.pkg Removed cat3k_caa-platform.SPA.03.06.00E.pkg Removed cat3k_caa-wcm.SPA.10.2.102.0.pkg 
[1]: New files list: Added cat3k_caa-base.SPA.03.07.03E.pkg Added cat3k_caa-drivers.SPA.03.07.03E.pkg Added cat3k_caa-infra.SPA.03.07.03E.pkg Added cat3k_caa-iosd-universalk9.SPA.152-3.E3.pkg Added cat3k_caa-platform.SPA.03.07.03E.pkg Added cat3k_caa-wcm.SPA.10.3.130.0.pkg 
[1]: Creating pending provisioning file 
[1]: Finished installing software. New software will load on reboot. 
[1]: Committing provisioning file 
[1]: Do you want to proceed with reload? [yes/no]: yes 
[1]: Reloading Switch#

Step 5  Power on physical switch 2 (next one down). This will be the standby switch for the stack. The switch will reload twice, the second time due to auto-upgrade of the IOS before it joins the stack.

Switch#sh switch
Switch/Stack Mac Address : 74a0.2f45.2380 - Local Mac Address
Mac persistency wait time: Indefinite
                                   H/W    Current
Switch#   Role    Mac Address    Priority Version State 
------------------------------------------------------------
*1       Active   74a0.2f45.2380     15     V01   Ready 
 2       Standby  74a0.2f58.7180     1      V01   Ready

Now change the second physical switch priority to 14

Switch#switch 2 priority 14 
WARNING: Changing the switchpriority may result in a configuration change for that
switch. Do you want to continue?[y/n]y
Switch#sh switch
Switch/Stack Mac Address : 74a0.2f45.2380 - Local Mac Address
Mac persistency wait time: Indefinite
                                   H/W    Current
Switch#   Role    Mac Address    Priority Version State 
------------------------------------------------------------
*1       Active   74a0.2f45.2380     15     V01   Ready 
 2       Standby  74a0.2f58.7180     14     V01   Ready

Step 6  Continue working down the stack powering on each additional member switch, one at a time. There’s no need to wait for each switch to fully boot before powering on the next as a switch joining the stack will take the lowest available member number. To ensure this order remains after reboot, continue to decrement the switch priority i.e. physical switch 3 priority 13, switch 4 priority 12 etc as you onboard each switch. Once you have configured all the member switches, save and reload the entire stack. Verify.

CHANGING AN EXISTING STACK

SWAPPING OR ADDING A SWITCH IN AN EXISTING STACK

Pitfall 1: Adding a new switch that’s already powered on to the stack (i.e before you connect the stack cables to it) will cause the entire stack to reload and potentially re-elect roles (see note 3). For uninterrupted service ensure the switch being added or removed is powered down/off before connecting or disconnecting stack cables to/from it. This also minimises the risk of accidentally splitting the stack as you disconnect stackwise cables, depending on how they are connected up.

If swapping a switch, provided that the replacement is the exact same model it will automatically take up the relevant config and operate as per the one removed.

ADDING MULTIPLE SWITCHES TO AN EXISTING STACK

Just as with provisioning a new stack, power on each switch you’re adding to an existing stack top down in the order they are physically stacked. A switch is automatically assigned the next lowest available stack member number when it enters the stack. You can then fix its place in the stack topology by changing the priority value. Use the

Switch#reload slot [switch member number]

command to reload each newly provisioned switch individually to verify the stack member numbers are stable on reboot.

REMOVING STACK CONFIGURATION FROM A SWITCH

Have you ever powered on a switch that has been taken out of production only to find that it has old stack configuration on and the interface numbers start with gi3/0/1? Annoying. Have you ever carried on working with it in this state regardless because you couldn’t remember the commands to reset the stack configuration? Don’t be that guy.

There’s two ways to tackle this.

Option 1: Go nuclear. Factory reset the switch

Note: This is an option listed in the Cisco documentation however in my testing this didn’t reset the stack priority to 1 i.e the pre-factory reset value was retained.

Step 1 No need to cycle power. Erase the startup-config if it exists (see note 4 below). Whilst the switch is powered press and hold the Mode button. The switch LEDs begin blinking after about 3 seconds.

Step 2 Continue holding down the Mode button. The LEDs stop blinking after 7 more seconds, and then the switch restarts.

Option 2: Reconfigure the existing stack configuration

There’s no way to turn off or reset the stack feature on 3560/3850s. Even a standalone switch runs in a stack, albeit on its own. The following command is present in a switch that has just been factory defaulted and powered up standalone.

switch 1 provision ws-c3650-48pd

Verification

Switch#show switch
Switch/Stack Mac Address : 74a0.2fc4.4600 - Local Mac Address
Mac persistency wait time: Indefinite
                                             H/W  Current
Switch#  Role      Mac Address    Priority Version State 
------------------------------------------------------------
*1      Active     74a0.2fc4.4600    1       V01   Ready

Switch#

Here’s a switch that’s been removed from a stack of two.

Switch#show run | in provision
switch 1 provision ws-c3650-48pd 
switch 2 provision ws-c3650-48pd
Switch#sh switch
Switch/Stack Mac Address : 74a0.2f59.dd80 - Local Mac Address
Mac persistency wait time: Indefinite
 H/W Current
Switch# Role Mac Address Priority Version State 
------------------------------------------------------------
*1 Active 74a0.2f59.dd80 14 V01 Ready 
 2 Member 0000.0000.0000 0 0 Provisioned

Provisioned means there was a switch there but it’s not there any more. We can Unprovision this redundant switch and upon the next reload it won’t be there any more.

Switch(config)#no switch 2 provision ws-c3650-48pd
Switch(config)#end
Switch#sh switch
*Jun 22 23:29:14.255: %SYS-5-CONFIG_I: Configured from console by consolewitch
Switch/Stack Mac Address : 74a0.2f59.dd80 - Local Mac Address
Mac persistency wait time: Indefinite
 H/W Current
Switch# Role Mac Address Priority Version State 
------------------------------------------------------------
*1 Active 74a0.2f59.dd80 14 V01 Ready 
 2 Member 0000.0000.0000 0 0 Unprovisioned

Switch#

removes the old stack config.

Switch#switch 1 priority 1

restores the default priority. Neither of our changes will take effect until after a reload

Switch#reload
<snip>
Press RETURN to get started.
Switch#sh switch
*Jun 22 23:29:14.255: %SYS-5-CONFIG_I: Configured from console by consolewitch
Switch/Stack Mac Address : 74a0.2f59.dd80 - Local Mac Address
Mac persistency wait time: Indefinite
 H/W Current
Switch# Role Mac Address Priority Version State 
------------------------------------------------------------
*1 Active 74a0.2f59.dd80 14 V01 Ready

SUMMARY

For simple stack setup change the switch priority value.

Be aware of all stack configuration options (i.e not just priority) that can affect the stack topology.

Take note of the order of operation when making changes to switch stacks to minimise disruption.

NOTES

1. The mode button on the front of the switch is not only useful for checking the stack priority. Look at the port leds on the last two ports on the switch (10 gig or sfp ports). If both lit, this visual indicator shows the switch is operating at full bandwidth. We can also verify this on the cli

Switch#show switch stack-ring speed

Stack Ring Speed : 160G
Stack Ring Configuration: Down
Stack Ring Protocol : StackWise

2. Power up the switch you wish to be active first on it’s own. It’s a one horse race. Any switches powered up 2 mins after this will not participate in the election. Note this approach wouldn’t guarantee a stable stack topology on reload. A final note on non-preemptive behaviour. If the active switch is removed from the stack the standby switch becomes active. Any switch subsequently re-inserted into the stack with a higher priority (even if the original active switch is subsequently re-inserted) will not cause the active switch to release it’s role or an election to be triggered.

3. Result of connecting two powered standalone switches together using a single stackwise. Despite having different priorities, each calls a stack merge and both reload.

Switch#
*Jun 22 21:13:59.869: %STACKMGR-1-STACK_LINK_CHANGE: 1 stack-mgr: Stack port 2 on switch 1 is up 
*Jun 22 21:14:00.252: %IOSXE-3-PLATFORM: 1 process stack-mgr: : -Traceback=1#87f75377a54ddceb043862d594ac8de7 :54DA2000+9B564 :54DA2000+1F854 :54DA2000+67B54 :54DA2000+5E5D4 :54DA2000+60914 ngwcutils:2B263000+BE84 ngwcutils:2B263000+DA3C pthread:2AD53000+5DC8 
*Jun 22 21:14:00.262: %IOSXE-3-PLATFORM: 1 process stack-mgr: : -Traceback=1#87f75377a54ddceb043862d594ac8de7 :54DA2000+1F854 :54DA2000+67B54 :show 
<Wed Jun 22 21:14:00 2016> Message from sysmgr: Reason Code:[4] Reset Reason:Reset/Reload requested by [stack-manager]. [stack merge]

Unmounting ng3k filesystems...
Unmounted /dev/sda3...
Warning! - some ng3k filesystems may not have unmounted cleanly...
Please stand by while rebooting the system...
Restarting system.

Booting...

4. Output of pressing the mode button before and after erasing the startup-config:

Switch#
*Jun 22 20:37:36.950: %EXPRESS_SETUP-6-MODE_BUTTON_RESET_IGNORED: mode button pressed for more than 10Sec and startup config is present hence not reloading
Switch#erase startup-config 
Erasing the nvram filesystem will remove all configuration files! Continue? [confirm]
[OK]
Erase of nvram: complete
Switch#
*Jun 22 20:38:06.795: %SYS-7-NV_BLOCK_INIT: Initialized the geometry of nvram
Switch#
consoleless_setup_process: Entering Setup Mode
*Jun 22 20:38:15.625: %EXPRESS_SETUP-6-MODE_ENTERED: 
*Jun 22 20:38:40.953: %EXPRESS_SETUP-5-CONFIG_IS_RESET: The configuration is reset and the system will now reboot
*Jun 22 20:38:46.209: %SYS-5-RELOAD: Reload requested by NGWC led process. Reload Reason: Reload due to Express Setup.