Whether you’re looking at the Cisco Nexus 5000 line for the Unified Fabric feature, the 1.92Tbps of throughput, or simply as an upgrade from the traditional Catalyst 6509 chassis, you will definitely want to take advantage of its Virtual Port-Channel (vPC) capability.
In this example I am going to focus on a very simple scenario: Implementation of a dual Cisco Nexus 5596UP Aggregation Layer in an infrastructure with a non-VSS Dual Core and Catalyst 3560E Access Layer using dual 10Gbps uplinks.
Design Overview
Equipment:
- Core Layer: Dual Cisco Catalyst 6509 Switches (non-VSS mode)
- Aggregation Layer : Dual Cisco 5596UP Switches
- Access Layer: Cisco Catalyst 3560E
Diagram:
Configuration
1. License Check
If your Nexus 5000 isn’t “New In Box” (NIB), then the first thing you will want to do after powering up is to verify that the software licenses match what is installed:
NEXUS1# show license usage Feature Ins Lic Status Expiry Date Comments Count ------------------------------------------------------------------- FCOE_NPV_PKG No - Unused - FM_SERVER_PKG No - Unused - ENTERPRISE_PKG No - Unused - FC_FEATURES_PKG No - Unused - VMFEX_FEATURE_PKG No - Unused - ENHANCED_LAYER2_PKG No - Unused - LAN_BASE_SERVICES_PKG No - Unused - LAN_ENTERPRISE_SERVICES_PKG No - Unused - -------------------------------------------------------------------
Pay particular attention to the LAN_BASE_SERVICES_PKG. If you have it installed without an accompanying Layer 3 Daughter Card any VLAN Interfaces you create in the Default VRF will remain in “Non-Routable VDC Mode” with a “down” status. You have to clear the license in order to resolve the problem:
nexus# show license brief license_FOX1773GE6A_12.lic nexus# clear license license_FOX1773GE6A_12.lic
2. Enable Features
In order to create Virtual Port-channels you will need to first enable the “feature”. This is also the time to take a look at the different available features and enable any others you need for your design. For this case I am using the following features:
feature tacacs+ feature interface-vlan feature lacp feature vpc
If you’re new to the Nexus platform and not in a particularly large environment you may want to disable Cisco Fabric Services (CFS). CFS allows the automatic synchronization of certain applications on CFS-capable switches such as TACACS+ and User Roles. If you’re going to disable it, do so before configuring your vPC otherwise you will get the following error:
NEXUS1(config)# no cfs distribute This will prevent CFS from distributing the configuration to other switches. Are you sure? (y/n) [n] y Operation failed: Please disable feature vPC before disabling cfs distribution. To disable feature vPC use no feature vPC
3. Create vPC Domain
The first step in configuring vPC is to create a domain with an id number (must match on both switches) and assign a non-preemptive role priority to each switch where lowest number = highest priority.
A new feature was released in NX-OS version 5.2(1) called vPC auto-recovery. Cisco strongly recommends that you enable this feature to help prevent dual active scenarios, so we’ll enable it. More information can be found here: https://supportforums.cisco.com/docs/DOC-24939
Nexus1 (Primary Switch) configuration:
vpc domain 999 role priority 100 auto-recovery
Nexus2 configuration:
vpc domain 999 role priority 110 auto-recovery
4. Create vPC Peer Link
Next we will create a Port-Channel to be used as the vPC Peer Link. The purpose of the Peer Link is to synchronize BPDU’s, HSRP, and MAC address tables while also forwarding unknown unicast, multicast, and broadcast traffic.
In most cases you’ll never see more than 40Mbps max on the Peer Link, but you still need to dedicate at least two of your interfaces to this Port-Channel for redundancy. Be sure to include the “spanning-tree port type network” and “vpc peer-link” commands to your the configuration.
In this example we are creating Port-Channel 4096 as the Peer Link and assigning one interface on module 1 and one interface on the 16 port flexible GEM card in slot 4 to the channel:
Nexus1 (Primary Switch) configuration:
interface port-channel4096 description vPC PEER LINK switchport mode trunk spanning-tree port type network vpc peer-link ! interface Ethernet1/48 description NEXUS2: E1/48 [Po4096] switchport mode trunk spanning-tree port type network channel-group 4096 mode active ! interface Ethernet4/16 description NEXUS2: E4/16 [Po4096] switchport mode trunk spanning-tree port type network channel-group 4096 mode active !
Nexus2 configuration:
interface port-channel4096 description vPC PEER LINK switchport mode trunk spanning-tree port type network vpc peer-link ! interface Ethernet1/48 description NEXUS1: E1/48 [Po4096] switchport mode trunk spanning-tree port type network channel-group 4096 mode active ! interface Ethernet4/16 description NEXUS1: E4/16 [Po4096] switchport mode trunk spanning-tree port type network channel-group 4096 mode active !
5. Create vPC Peer-Keepalive
The Keepalive link is used to resolve dual-active failures and although not documented, Cisco’s recommended configuration is to directly connect the two management interfaces together with an Ethernet cable to form the Peer-Keepalive.
Although this clearly seems like the wrong way to do it, especially considering that you lose the benefits of the management VRF and interface, Cisco TAC will tell you that customers using virtual interfaces in the default VRF for the Keepalive endpoints have experienced strange behavior even when correctly configured (i.e. even when on a VLAN not carried on the vPC Peer Link).
I’m not a fan of this suggestion, but I chose to follow it and recommend you do too.
In this example we will be connecting an Ethernet cable between our management interfaces for use as the Peer-Keepalive interface and configuring them as hosts on the 1.1.1.0/30 network.
Nexus1 (Primary Switch) configuration:
interface mgmt0 ip address 1.1.1.1/30 ! vpc domain 999 role priority 100 system-priority 1024 peer-keepalive destination 1.1.1.2 auto-recovery
Nexus2 configuration:
interface mgmt0 ip address 1.1.1.2/30 ! vpc domain 999 role priority 110 system-priority 1024 peer-keepalive destination 1.1.1.1 auto-recovery
Verification:
Nexus1# show vpc peer-keepalive vPC keep-alive status : peer is alive --Peer is alive for : (3503360) seconds, (556) msec --Send status : Success --Last send at : 2013.01.15 21:29:29 13 ms --Sent on interface : mgmt0 --Receive status : Success --Last receive at : 2013.01.15 21:29:29 12 ms --Received on interface : mgmt0 --Last update from peer : (0) seconds, (760) msec vPC Keep-alive parameters --Destination : 1.1.1.2 --Keepalive interval : 1000 msec --Keepalive timeout : 5 seconds --Keepalive hold timeout : 3 seconds --Keepalive vrf : management --Keepalive udp port : 3200 --Keepalive tos : 192
5. Create vPC Port-Channels and Ports
The only difference between creating a vPC interface and a regular Port-Channel interface is the vpc command. For ease of administration purposes I suggest making your vPC and Port-Channel ID’s match and also taking the time to properly configure the description field for all interfaces. It makes life easier, especially if you are using Cacti to monitor your switches as discussed in my previous blog post: Dress Up Your Cacti Graphs.
In this example we will be using vpc 101-102 for connectivity to the core switches and vpc 1-44 for connectivity to the access switches.
Note that our design does not include any Port-Channels directly between the two non-VSS Core Switches. We will leverage the Nexus vPC for connectivity.
Nexus1 (Primary Switch) configuration:
interface port-channel1 description SW-R01-01A: [vpc1] switchport mode trunk vpc 1 interface port-channel2 description SW-R01-02A: [vpc2] switchport mode trunk vpc 2 interface port-channel3 description SW-R01-03A: [vpc3] switchport mode trunk vpc 3 interface port-channel4 description SW-R01-04A: [vpc4] switchport mode trunk vpc 4 interface port-channel5 description SW-R01-05A: [vpc5] switchport mode trunk vpc 5 . . . interface port-channel101 description CR1: [vpc101] switchport mode trunk spanning-tree port type network vpc 101 interface port-channel102 description CR2: [vpc102] switchport mode trunk vpc 102 interface Ethernet1/1 description SW-R01-01A: Te0/1 [Po1] switchport mode trunk channel-group 1 mode active interface Ethernet1/2 description SW-R01-02A: Te0/1 [Po2] switchport mode trunk channel-group 2 mode active interface Ethernet1/3 description SW-R01-03A: Te0/1 [Po3] switchport mode trunk channel-group 3 mode active interface Ethernet1/4 description SW-R01-04A: Te0/1 [Po4] switchport mode trunk channel-group 4 mode active . . . interface Ethernet1/46 description CORE1: Te7/1 [Po101] switchport mode trunk spanning-tree port type network channel-group 101 mode active interface Ethernet1/47 description CORE2: Te7/1 [Po102] switchport mode trunk spanning-tree port type network channel-group 102 mode active interface Ethernet1/48 description NEXUS2: E1/48 [Po4096] switchport mode trunk spanning-tree port type network channel-group 4096 mode active
Nexus2 configuration:
interface port-channel1 description SW-R01-01A: [vpc1] switchport mode trunk vpc 1 interface port-channel2 description SW-R01-02A: [vpc2] switchport mode trunk vpc 2 interface port-channel3 description SW-R01-03A: [vpc3] switchport mode trunk vpc 3 interface port-channel4 description SW-R01-04A: [vpc4] switchport mode trunk vpc 4 interface port-channel5 description SW-R01-05A: [vpc5] switchport mode trunk vpc 5 . . . interface port-channel101 description CR1: [vpc101] switchport mode trunk vpc 101 interface port-channel102 description CR2: [vpc102] switchport mode trunk vpc 102 interface Ethernet1/1 description SW-R01-01A: Te0/2 [Po1] switchport mode trunk channel-group 1 mode active interface Ethernet1/2 description SW-R01-02A: Te0/2 [Po2] switchport mode trunk channel-group 2 mode active interface Ethernet1/3 description SW-R01-03A: Te0/2 [Po3] switchport mode trunk channel-group 3 mode active interface Ethernet1/4 description SW-R01-04A: Te0/2 [Po4] switchport mode trunk channel-group 4 mode active . . . interface Ethernet1/46 description CORE1: Te7/2 [Po101] switchport mode trunk spanning-tree port type network channel-group 101 mode active interface Ethernet1/47 description CORE2: Te7/2 [Po102] switchport mode trunk spanning-tree port type network channel-group 102 mode active
6. Network Management Configuration
Since we burned our management VRF and interface on the vPC Peer-Keepalive, we’ll need to create a Virtural Interface for management tasks before adding configuration for things such as tacacs+, snmp, and syslog. This isn’t dramatically different than on a Cisco IOS device so rather than step you through it I’ll just post my abbreviated running configuration for use as reference.
Nexus1 (Primary Switch) full configuration:
version 5.2(1)N1(2a) logging level feature-mgr 0 hostname NEXUS1 feature tacacs+ feature interface-vlan feature lacp feature vpc feature lldp ip domain-lookup ip domain-name domain.com ip domain-list domain-int ip name-server 10.0.0.10 tacacs-server key 7 "XXXXXXX" ip tacacs source-interface Vlan10 tacacs-server host 10.0.0.32 tacacs-server host 10.0.0.33 aaa group server tacacs+ TACACS+ server 10.0.0.32 server 10.0.0.33 source-interface Vlan10 logging message interface type ethernet description logging event link-status default service unsupported-transceiver snmp-server source-interface trap Vlan10 snmp-server source-interface inform Vlan10 snmp-server user admin network-admin auth md5 ... snmp-server host 10.0.0.18 traps version 2c ... snmp-server enable traps aaa server-state-change snmp-server enable traps upgrade UpgradeOpNotifyOnCompletion snmp-server enable traps upgrade UpgradeJobStatusNotify snmp-server enable traps feature-control FeatureOpStatusChange snmp-server enable traps sysmgr cseFailSwCoreNotifyExtended snmp-server enable traps config ccmCLIRunningConfigChanged snmp-server enable traps snmp authentication snmp-server enable traps link cisco-xcvr-mon-status-chg snmp-server community ... group network-operator ntp server 10.0.0.101 use-vrf default ntp source-interface Vlan10 ntp logging aaa authentication login default group TACACS+ local aaa authentication login ascii-authentication tacacs-server directed-request vrf context management vlan 1 vlan 5 vlan 10 vlan 13 vlan 14 vlan 15 no spanning-tree bridge assurance vpc domain 999 role priority 100 system-priority 1024 peer-keepalive destination 1.1.1.2 auto-recovery port-profile default max-ports 512 interface Vlan1 interface Vlan10 no shutdown management ip address 10.0.0.104/16 interface port-channel1 description SW-R01-01A: [vpc1] switchport mode trunk vpc 1 interface port-channel2 description SW-R01-02A: [vpc2] switchport mode trunk vpc 2 interface port-channel3 description SW-R01-03A: [vpc3] switchport mode trunk vpc 3 interface port-channel4 description SW-R01-04A: [vpc4] switchport mode trunk vpc 4 interface port-channel5 description SW-R01-05A: [vpc5] switchport mode trunk vpc 5 . . . interface port-channel101 description CR1: [vpc101] switchport mode trunk vpc 101 interface port-channel102 description CR2: [vpc102] switchport mode trunk vpc 102 interface port-channel4096 description vPC PEER LINK switchport mode trunk spanning-tree port type network vpc peer-link interface Ethernet1/1 description SW-R01-01A: Te0/1 [Po1] switchport mode trunk channel-group 1 mode active interface Ethernet1/2 description SW-R01-02A: Te0/1 [Po2] switchport mode trunk channel-group 2 mode active interface Ethernet1/3 description SW-R01-03A: Te0/1 [Po3] switchport mode trunk channel-group 3 mode active interface Ethernet1/4 description SW-R01-04A: Te0/1 [Po4] switchport mode trunk channel-group 4 mode active . . . interface Ethernet1/46 description CORE1: Te7/1 [Po101] switchport mode trunk spanning-tree port type network channel-group 101 mode active interface Ethernet1/47 description CORE2: Te7/1 [Po102] switchport mode trunk spanning-tree port type network channel-group 102 mode active interface Ethernet1/48 description NEXUS2: E1/48 [Po4096] switchport mode trunk spanning-tree port type network channel-group 4096 mode active interface mgmt0 ip address 1.1.1.1/30 line console line vty boot kickstart bootflash:/n5000-uk9-kickstart.5.2.1.N1.2a.bin boot system bootflash:/n5000-uk9.5.2.1.N1.2a.bin ip route 0.0.0.0/0 10.0.0.1 logging logfile LOGS 6 logging server 10.0.0.18 7 use-vrf default
Nexus2 full configuration:
version 5.2(1)N1(2a) logging level feature-mgr 0 hostname NEXUS2 feature tacacs+ feature interface-vlan feature lacp feature vpc feature lldp ip domain-lookup ip domain-name domain.com ip domain-list domain-int ip name-server 10.0.0.10 tacacs-server key 7 "XXXXXXX" ip tacacs source-interface Vlan10 tacacs-server host 10.0.0.32 tacacs-server host 10.0.0.33 aaa group server tacacs+ TACACS+ server 10.0.0.32 server 10.0.0.33 source-interface Vlan10 logging message interface type ethernet description logging event link-status default service unsupported-transceiver snmp-server source-interface trap Vlan10 snmp-server source-interface inform Vlan10 snmp-server user admin network-admin auth md5 ... snmp-server host 10.0.0.18 traps version 2c ... snmp-server enable traps aaa server-state-change snmp-server enable traps upgrade UpgradeOpNotifyOnCompletion snmp-server enable traps upgrade UpgradeJobStatusNotify snmp-server enable traps feature-control FeatureOpStatusChange snmp-server enable traps sysmgr cseFailSwCoreNotifyExtended snmp-server enable traps config ccmCLIRunningConfigChanged snmp-server enable traps snmp authentication snmp-server enable traps link cisco-xcvr-mon-status-chg snmp-server community ... group network-operator ntp server 10.0.0.101 use-vrf default ntp source-interface Vlan10 ntp logging aaa authentication login default group TACACS+ local aaa authentication login ascii-authentication tacacs-server directed-request vrf context management vlan 1 vlan 5 vlan 10 vlan 13 vlan 14 vlan 15 no spanning-tree bridge assurance vpc domain 999 role priority 110 system-priority 1024 peer-keepalive destination 1.1.1.2 auto-recovery port-profile default max-ports 512 interface Vlan1 interface Vlan10 no shutdown management ip address 10.0.0.105/16 interface port-channel1 description SW-R01-01A: [vpc1] switchport mode trunk vpc 1 interface port-channel2 description SW-R01-02A: [vpc2] switchport mode trunk vpc 2 interface port-channel3 description SW-R01-03A: [vpc3] switchport mode trunk vpc 3 interface port-channel4 description SW-R01-04A: [vpc4] switchport mode trunk vpc 4 interface port-channel5 description SW-R01-05A: [vpc5] switchport mode trunk vpc 5 . . . interface port-channel101 description CR1: [vpc101] switchport mode trunk vpc 101 interface port-channel102 description CR2: [vpc102] switchport mode trunk vpc 102 interface port-channel4096 description vPC PEER LINK switchport mode trunk spanning-tree port type network vpc peer-link interface Ethernet1/1 description SW-R01-01A: Te0/2 [Po1] switchport mode trunk channel-group 1 mode active interface Ethernet1/2 description SW-R01-02A: Te0/2 [Po2] switchport mode trunk channel-group 2 mode active interface Ethernet1/3 description SW-R01-03A: Te0/2 [Po3] switchport mode trunk channel-group 3 mode active interface Ethernet1/4 description SW-R01-04A: Te0/2 [Po4] switchport mode trunk channel-group 4 mode active . . . interface Ethernet1/46 description CORE1: Te7/2 [Po101] switchport mode trunk spanning-tree port type network channel-group 101 mode active interface Ethernet1/47 description CORE2: Te7/2 [Po102] switchport mode trunk spanning-tree port type network channel-group 102 mode active interface Ethernet1/48 description NEXUS1: E1/48 [Po4096] switchport mode trunk spanning-tree port type network channel-group 4096 mode active interface mgmt0 ip address 1.1.1.2/30 line console line vty boot kickstart bootflash:/n5000-uk9-kickstart.5.2.1.N1.2a.bin boot system bootflash:/n5000-uk9.5.2.1.N1.2a.bin ip route 0.0.0.0/0 10.0.0.1 logging logfile LOGS 6 logging server 10.0.0.18 7 use-vrf default
7. Additional Information
A few other useful tips:
- The hidden command service unsupported-transceiver allows you to use things such as Laserwire optics.
- Disable spanning-tree bridge assurance globally if you are peering with Catalyst 6509’s and using the spanning-tree port type network configuration on your Port-Channels.
Great Post Paul. Exactly the sort of thing I was after.
Thanks Paul. Fabulous article. Saved our a$$ on an installation of disparate 5020 switches and incompatibility with VPC peering. Even though Cisco text failed us, you saved the day. Would like to send you a Starbucks giftcard for the help. From one CCNP and even CCIP+Metro, thanks brother.
You rock!
Andy
Very happy to hear that! Thanks for taking the time to post some feedback 🙂
Hi, good article, question on the cats its purely a single port channel on each catalyst switch, as not really doing Vpc
Cheers,
Scott
thanks, Paul. for your explanation on the mgmt interface and how to connect them together.
i was confused about that one and have search all over for answers until i found your post.
Thanks dude! Saved me alot of work, especially the part about best practice using mgmt as peer-addresses. It’s posts like these who help so many people! 🙂
Very nice blog, we have 2 locations and at each location we have 2 x 5548 en 2200. Between location we have 2 x dark fiber of 10 gb. Can you explain how to link the two sites together?
Nice blog mate! Few questions
The default route on the Nexus is pointing to 10.0.0.1 would that be one of the cat6500 address’s ?
In regard to further design btw Core and Distribution layer since you have a vPC towards the 6500 switches would you create a layer 3 port-channel and run L3 routing towards the 5500?
Also another design question btw Access layer and Distribution layer, would you run HSRP/VRRP across multiple SVI’s on the 5500?
Would be great to know more on this design aspect.