Cisco Nexus 5000 Series vPC Design and Configuration

Whether you’re looking at the Cisco Nexus 5000 line for the Unified Fabric feature, the 1.92Tbps of throughput, or simply as an upgrade from the traditional Catalyst 6509 chassis, you will definitely want to take advantage of its Virtual Port-Channel (vPC) capability.

In this example I am going to focus on a very simple scenario: Implementation of a dual Cisco Nexus 5596UP Aggregation Layer in an infrastructure with a non-VSS Dual Core and Catalyst 3560E Access Layer using dual 10Gbps uplinks.

Design Overview

Equipment:

  • Core Layer: Dual Cisco Catalyst 6509 Switches (non-VSS mode)
  • Aggregation Layer : Dual Cisco 5596UP Switches
  • Access Layer: Cisco Catalyst 3560E

Diagram:

Nexus Configuration: Example #1

Configuration

1. License Check

If your Nexus 5000 isn’t “New In Box” (NIB), then the first thing you will want to do after powering up is to verify that the software licenses match what is installed:

 NEXUS1# show license usage
 Feature                      Ins  Lic   Status Expiry Date Comments
 Count
 -------------------------------------------------------------------
 FCOE_NPV_PKG                  No    -   Unused             -
 FM_SERVER_PKG                 No    -   Unused             -
 ENTERPRISE_PKG                No    -   Unused             -
 FC_FEATURES_PKG               No    -   Unused             -
 VMFEX_FEATURE_PKG             No    -   Unused             -
 ENHANCED_LAYER2_PKG           No    -   Unused             -
 LAN_BASE_SERVICES_PKG         No    -   Unused             -
 LAN_ENTERPRISE_SERVICES_PKG   No    -   Unused             -
 -------------------------------------------------------------------

Pay particular attention to the LAN_BASE_SERVICES_PKG. If you have it installed without an accompanying Layer 3 Daughter Card any VLAN Interfaces you create in the Default VRF will remain in “Non-Routable VDC Mode” with a “down” status. You have to clear the license in order to resolve the problem:

nexus# show license brief
license_FOX1773GE6A_12.lic
nexus# clear license license_FOX1773GE6A_12.lic

2. Enable Features

In order to create Virtual Port-channels you will need to first enable the “feature”. This is also the time to take a look at the different available features and enable any others you need for your design. For this case I am using the following features:

feature tacacs+
feature interface-vlan
feature lacp
feature vpc 

If you’re new to the Nexus platform and not in a particularly large environment you may want to disable Cisco Fabric Services (CFS). CFS allows the automatic synchronization of certain applications on CFS-capable switches such as TACACS+ and User Roles. If you’re going to disable it, do so before configuring your vPC otherwise you will get the following error:

NEXUS1(config)# no cfs distribute
This will prevent CFS from distributing the configuration to other switches.
Are you sure? (y/n)  [n] y
Operation failed: Please disable feature vPC before disabling cfs distribution.
To disable feature vPC use no feature vPC 

3. Create vPC Domain

The first step in configuring vPC is to create a domain with an id number (must match on both switches) and assign a non-preemptive role priority to each switch where lowest number = highest priority.

A new feature was released in NX-OS version 5.2(1) called vPC auto-recovery. Cisco strongly recommends that you enable this feature to help prevent dual active scenarios, so we’ll enable it. More information can be found here: https://supportforums.cisco.com/docs/DOC-24939

Nexus1 (Primary Switch) configuration:

vpc domain 999
role priority 100
auto-recovery
 

Nexus2 configuration:

vpc domain 999
role priority 110
auto-recovery 

4. Create vPC Peer Link

Next we will create a Port-Channel to be used as the vPC Peer Link. The purpose of the Peer Link is to synchronize BPDU’s, HSRP, and MAC address tables while also forwarding unknown unicast, multicast, and broadcast traffic.

In most cases you’ll never see more than 40Mbps max on the Peer Link, but you still need to dedicate at least two of your interfaces to this Port-Channel for redundancy. Be sure to include the “spanning-tree port type network” and “vpc peer-link” commands to your the configuration.

In this example we are creating Port-Channel 4096 as the Peer Link and assigning one interface on module 1 and one interface on the 16 port flexible GEM card in slot 4 to the channel:

Nexus1 (Primary Switch) configuration:

interface port-channel4096
description vPC PEER LINK
switchport mode trunk
spanning-tree port type network
vpc peer-link
!
interface Ethernet1/48
description NEXUS2: E1/48 [Po4096]
switchport mode trunk
spanning-tree port type network
channel-group 4096 mode active
!
interface Ethernet4/16
description NEXUS2: E4/16 [Po4096]
switchport mode trunk
spanning-tree port type network
channel-group 4096 mode active
! 

Nexus2 configuration:

interface port-channel4096
description vPC PEER LINK
switchport mode trunk
spanning-tree port type network
vpc peer-link
!
interface Ethernet1/48
description NEXUS1: E1/48 [Po4096]
switchport mode trunk
spanning-tree port type network
channel-group 4096 mode active
!
interface Ethernet4/16
description NEXUS1: E4/16 [Po4096]
switchport mode trunk
spanning-tree port type network
channel-group 4096 mode active
! 

5. Create vPC Peer-Keepalive

The Keepalive link is used to resolve dual-active failures and although not documented, Cisco’s recommended configuration is to directly connect the two management interfaces together with an Ethernet cable to form the Peer-Keepalive.

Although this clearly seems like the wrong way to do it, especially considering that you lose the benefits of the management VRF and interface, Cisco TAC will tell you that customers using virtual interfaces in the default VRF for the Keepalive endpoints have experienced strange behavior even when correctly configured (i.e. even when on a VLAN not carried on the vPC Peer Link).

I’m not a fan of this suggestion, but I chose to follow it and recommend you do too.

In this example we will be connecting an Ethernet cable between our management interfaces for use as the Peer-Keepalive interface and configuring them as hosts on the 1.1.1.0/30 network.

Nexus1 (Primary Switch) configuration:

interface mgmt0
ip address 1.1.1.1/30
!
vpc domain 999
role priority 100
system-priority 1024
peer-keepalive destination 1.1.1.2
auto-recovery

Nexus2 configuration:

interface mgmt0
ip address 1.1.1.2/30
!
vpc domain 999
role priority 110
system-priority 1024
peer-keepalive destination 1.1.1.1
auto-recovery

Verification:

Nexus1# show vpc peer-keepalive

vPC keep-alive status           : peer is alive
--Peer is alive for             : (3503360) seconds, (556) msec
--Send status                   : Success
--Last send at                  : 2013.01.15 21:29:29 13 ms
--Sent on interface             : mgmt0
--Receive status                : Success
--Last receive at               : 2013.01.15 21:29:29 12 ms
--Received on interface         : mgmt0
--Last update from peer         : (0) seconds, (760) msec

vPC Keep-alive parameters
--Destination                   : 1.1.1.2
--Keepalive interval            : 1000 msec
--Keepalive timeout             : 5 seconds
--Keepalive hold timeout        : 3 seconds
--Keepalive vrf                 : management
--Keepalive udp port            : 3200
--Keepalive tos                 : 192 

5. Create vPC Port-Channels and Ports

The only difference between creating a vPC interface and a regular Port-Channel interface is the vpc command. For ease of administration purposes I suggest making your vPC and Port-Channel ID’s match and also taking the time to properly configure the description field for all interfaces. It makes life easier, especially if you are using Cacti to monitor your switches as discussed in my previous blog post: Dress Up Your Cacti Graphs.

In this example we will be using vpc 101-102 for connectivity to the core switches and vpc 1-44 for connectivity to the access switches.

Note that our design does not include any Port-Channels directly between the two non-VSS Core Switches. We will leverage the Nexus vPC for connectivity.

Nexus1 (Primary Switch) configuration:

interface port-channel1
description SW-R01-01A: [vpc1]
switchport mode trunk
vpc 1

interface port-channel2
description SW-R01-02A: [vpc2]
switchport mode trunk
vpc 2

interface port-channel3
description SW-R01-03A: [vpc3]
switchport mode trunk
vpc 3

interface port-channel4
description SW-R01-04A: [vpc4]
switchport mode trunk
vpc 4

interface port-channel5
description SW-R01-05A: [vpc5]
switchport mode trunk
vpc 5
.
.
.
interface port-channel101
description CR1: [vpc101]
switchport mode trunk
  spanning-tree port type network
  vpc 101

interface port-channel102
description CR2: [vpc102]
switchport mode trunk
  vpc 102

interface Ethernet1/1
description SW-R01-01A: Te0/1 [Po1]
switchport mode trunk
channel-group 1 mode active

interface Ethernet1/2
description SW-R01-02A: Te0/1 [Po2]
switchport mode trunk
channel-group 2 mode active

interface Ethernet1/3
description SW-R01-03A: Te0/1 [Po3]
switchport mode trunk
channel-group 3 mode active

interface Ethernet1/4
description SW-R01-04A: Te0/1 [Po4]
switchport mode trunk
channel-group 4 mode active
.
.
.
interface Ethernet1/46
description CORE1: Te7/1 [Po101]
switchport mode trunk
spanning-tree port type network
channel-group 101 mode active

interface Ethernet1/47
description CORE2: Te7/1 [Po102]
switchport mode trunk
spanning-tree port type network
channel-group 102 mode active

interface Ethernet1/48
description NEXUS2: E1/48 [Po4096]
switchport mode trunk
spanning-tree port type network
channel-group 4096 mode active

Nexus2 configuration:

interface port-channel1
description SW-R01-01A:  [vpc1]
switchport mode trunk
vpc 1

interface port-channel2
description SW-R01-02A:  [vpc2]
switchport mode trunk
vpc 2

interface port-channel3
description SW-R01-03A: [vpc3]
switchport mode trunk
vpc 3

interface port-channel4
description SW-R01-04A: [vpc4]
switchport mode trunk
vpc 4

interface port-channel5
description SW-R01-05A: [vpc5]
switchport mode trunk
vpc 5
.
.
.
interface port-channel101
description CR1: [vpc101]
switchport mode trunk
vpc 101

interface port-channel102
description CR2: [vpc102]
switchport mode trunk
vpc 102

interface Ethernet1/1
description SW-R01-01A: Te0/2 [Po1]
switchport mode trunk
channel-group 1 mode active

interface Ethernet1/2
description SW-R01-02A: Te0/2 [Po2]
switchport mode trunk
channel-group 2 mode active

interface Ethernet1/3
description SW-R01-03A: Te0/2 [Po3]
switchport mode trunk
channel-group 3 mode active

interface Ethernet1/4
description SW-R01-04A: Te0/2 [Po4]
switchport mode trunk
channel-group 4 mode active
.
.
.
interface Ethernet1/46
description CORE1: Te7/2 [Po101]
switchport mode trunk
spanning-tree port type network
channel-group 101 mode active

interface Ethernet1/47
description CORE2: Te7/2 [Po102]
switchport mode trunk
spanning-tree port type network
channel-group 102 mode active

6. Network Management Configuration

Since we burned our management VRF and interface on the vPC Peer-Keepalive, we’ll need to create a Virtural Interface for management tasks before adding configuration for things such as tacacs+, snmp, and syslog. This isn’t dramatically different than on a Cisco IOS device so rather than step you through it I’ll just post my abbreviated running configuration for use as reference.

Nexus1 (Primary Switch) full configuration:

version 5.2(1)N1(2a)
logging level feature-mgr 0
hostname NEXUS1

feature tacacs+
feature interface-vlan
feature lacp
feature vpc
feature lldp

ip domain-lookup
ip domain-name domain.com
ip domain-list domain-int
ip name-server 10.0.0.10
tacacs-server key 7 "XXXXXXX"
ip tacacs source-interface Vlan10
tacacs-server host 10.0.0.32
tacacs-server host 10.0.0.33
aaa group server tacacs+ TACACS+
server 10.0.0.32
server 10.0.0.33
source-interface Vlan10
logging message interface type ethernet description
logging event link-status default
service unsupported-transceiver
snmp-server source-interface trap Vlan10
snmp-server source-interface inform Vlan10
snmp-server user admin network-admin auth md5 ...
snmp-server host 10.0.0.18 traps version 2c ...
snmp-server enable traps aaa server-state-change
snmp-server enable traps upgrade UpgradeOpNotifyOnCompletion
snmp-server enable traps upgrade UpgradeJobStatusNotify
snmp-server enable traps feature-control FeatureOpStatusChange
snmp-server enable traps sysmgr cseFailSwCoreNotifyExtended
snmp-server enable traps config ccmCLIRunningConfigChanged
snmp-server enable traps snmp authentication
snmp-server enable traps link cisco-xcvr-mon-status-chg
snmp-server community ... group network-operator
ntp server 10.0.0.101 use-vrf default
ntp source-interface  Vlan10
ntp logging
aaa authentication login default group TACACS+ local
aaa authentication login ascii-authentication
tacacs-server directed-request
vrf context management
vlan 1
vlan 5
vlan 10
vlan 13
vlan 14
vlan 15
no spanning-tree bridge assurance
vpc domain 999
role priority 100
system-priority 1024
peer-keepalive destination 1.1.1.2
auto-recovery
port-profile default max-ports 512

interface Vlan1

interface Vlan10
no shutdown
management
ip address 10.0.0.104/16

interface port-channel1
description SW-R01-01A: [vpc1]
switchport mode trunk
vpc 1

interface port-channel2
description SW-R01-02A: [vpc2]
switchport mode trunk
vpc 2

interface port-channel3
description SW-R01-03A: [vpc3]
switchport mode trunk
vpc 3

interface port-channel4
description SW-R01-04A: [vpc4]
switchport mode trunk
vpc 4

interface port-channel5
description SW-R01-05A: [vpc5]
switchport mode trunk
vpc 5
.
.
.
interface port-channel101
description CR1: [vpc101]
switchport mode trunk
vpc 101

interface port-channel102
description CR2: [vpc102]
switchport mode trunk
vpc 102

interface port-channel4096
description vPC PEER LINK
switchport mode trunk
spanning-tree port type network
vpc peer-link

interface Ethernet1/1
description SW-R01-01A: Te0/1 [Po1]
switchport mode trunk
channel-group 1 mode active

interface Ethernet1/2
description SW-R01-02A: Te0/1 [Po2]
switchport mode trunk
channel-group 2 mode active

interface Ethernet1/3
description SW-R01-03A: Te0/1 [Po3]
switchport mode trunk
channel-group 3 mode active

interface Ethernet1/4
description SW-R01-04A: Te0/1 [Po4]
switchport mode trunk
channel-group 4 mode active
.
.
.
interface Ethernet1/46
description CORE1: Te7/1 [Po101]
switchport mode trunk
spanning-tree port type network
channel-group 101 mode active

interface Ethernet1/47
description CORE2: Te7/1 [Po102]
switchport mode trunk
spanning-tree port type network
channel-group 102 mode active

interface Ethernet1/48
description NEXUS2: E1/48 [Po4096]
switchport mode trunk
spanning-tree port type network
channel-group 4096 mode active

interface mgmt0
ip address 1.1.1.1/30
line console
line vty
boot kickstart bootflash:/n5000-uk9-kickstart.5.2.1.N1.2a.bin
boot system bootflash:/n5000-uk9.5.2.1.N1.2a.bin
ip route 0.0.0.0/0 10.0.0.1
logging logfile LOGS 6
logging server 10.0.0.18 7 use-vrf default
 

Nexus2 full configuration:

version 5.2(1)N1(2a)
logging level feature-mgr 0
hostname NEXUS2

feature tacacs+
feature interface-vlan
feature lacp
feature vpc
feature lldp

ip domain-lookup
ip domain-name domain.com
ip domain-list domain-int
ip name-server 10.0.0.10
tacacs-server key 7 "XXXXXXX"
ip tacacs source-interface Vlan10
tacacs-server host 10.0.0.32
tacacs-server host 10.0.0.33
aaa group server tacacs+ TACACS+
server 10.0.0.32
server 10.0.0.33
source-interface Vlan10
logging message interface type ethernet description
logging event link-status default
service unsupported-transceiver
snmp-server source-interface trap Vlan10
snmp-server source-interface inform Vlan10
snmp-server user admin network-admin auth md5 ...
snmp-server host 10.0.0.18 traps version 2c ...
snmp-server enable traps aaa server-state-change
snmp-server enable traps upgrade UpgradeOpNotifyOnCompletion
snmp-server enable traps upgrade UpgradeJobStatusNotify
snmp-server enable traps feature-control FeatureOpStatusChange
snmp-server enable traps sysmgr cseFailSwCoreNotifyExtended
snmp-server enable traps config ccmCLIRunningConfigChanged
snmp-server enable traps snmp authentication
snmp-server enable traps link cisco-xcvr-mon-status-chg
snmp-server community ... group network-operator
ntp server 10.0.0.101 use-vrf default
ntp source-interface  Vlan10
ntp logging
aaa authentication login default group TACACS+ local
aaa authentication login ascii-authentication
tacacs-server directed-request
vrf context management
vlan 1
vlan 5
vlan 10
vlan 13
vlan 14
vlan 15
no spanning-tree bridge assurance
vpc domain 999
role priority 110
system-priority 1024
peer-keepalive destination 1.1.1.2
auto-recovery
port-profile default max-ports 512

interface Vlan1

interface Vlan10
no shutdown
management
ip address 10.0.0.105/16

interface port-channel1
description SW-R01-01A:  [vpc1]
switchport mode trunk
vpc 1

interface port-channel2
description SW-R01-02A:  [vpc2]
switchport mode trunk
vpc 2

interface port-channel3
description SW-R01-03A: [vpc3]
switchport mode trunk
vpc 3

interface port-channel4
description SW-R01-04A: [vpc4]
switchport mode trunk
vpc 4

interface port-channel5
description SW-R01-05A: [vpc5]
switchport mode trunk
vpc 5
.
.
.
interface port-channel101
description CR1: [vpc101]
switchport mode trunk
vpc 101

interface port-channel102
description CR2: [vpc102]
switchport mode trunk
vpc 102

interface port-channel4096
description vPC PEER LINK
switchport mode trunk
spanning-tree port type network
vpc peer-link

interface Ethernet1/1
description SW-R01-01A: Te0/2 [Po1]
switchport mode trunk
channel-group 1 mode active

interface Ethernet1/2
description SW-R01-02A: Te0/2 [Po2]
switchport mode trunk
channel-group 2 mode active

interface Ethernet1/3
description SW-R01-03A: Te0/2 [Po3]
switchport mode trunk
channel-group 3 mode active

interface Ethernet1/4
description SW-R01-04A: Te0/2 [Po4]
switchport mode trunk
channel-group 4 mode active
.
.
.
interface Ethernet1/46
description CORE1: Te7/2 [Po101]
switchport mode trunk
spanning-tree port type network
channel-group 101 mode active

interface Ethernet1/47
description CORE2: Te7/2 [Po102]
switchport mode trunk
spanning-tree port type network
channel-group 102 mode active

interface Ethernet1/48
description NEXUS1: E1/48 [Po4096]
switchport mode trunk
spanning-tree port type network
channel-group 4096 mode active

interface mgmt0
ip address 1.1.1.2/30
line console
line vty
boot kickstart bootflash:/n5000-uk9-kickstart.5.2.1.N1.2a.bin
boot system bootflash:/n5000-uk9.5.2.1.N1.2a.bin
ip route 0.0.0.0/0 10.0.0.1
logging logfile LOGS 6
logging server 10.0.0.18 7 use-vrf default

 

7. Additional Information

A few other useful tips:

  1. The hidden command service unsupported-transceiver allows you to use things such as Laserwire optics.
  2. Disable spanning-tree bridge assurance globally if you are peering with Catalyst 6509’s and using the spanning-tree port type network configuration on your Port-Channels.

8 comments

  1. Thanks Paul. Fabulous article. Saved our a$$ on an installation of disparate 5020 switches and incompatibility with VPC peering. Even though Cisco text failed us, you saved the day. Would like to send you a Starbucks giftcard for the help. From one CCNP and even CCIP+Metro, thanks brother.

    You rock!
    Andy

  2. Hi, good article, question on the cats its purely a single port channel on each catalyst switch, as not really doing Vpc

    Cheers,

    Scott

  3. thanks, Paul. for your explanation on the mgmt interface and how to connect them together.
    i was confused about that one and have search all over for answers until i found your post.

  4. Thanks dude! Saved me alot of work, especially the part about best practice using mgmt as peer-addresses. It’s posts like these who help so many people! πŸ™‚

  5. Very nice blog, we have 2 locations and at each location we have 2 x 5548 en 2200. Between location we have 2 x dark fiber of 10 gb. Can you explain how to link the two sites together?

  6. Nice blog mate! Few questions

    The default route on the Nexus is pointing to 10.0.0.1 would that be one of the cat6500 address’s ?

    In regard to further design btw Core and Distribution layer since you have a vPC towards the 6500 switches would you create a layer 3 port-channel and run L3 routing towards the 5500?

    Also another design question btw Access layer and Distribution layer, would you run HSRP/VRRP across multiple SVI’s on the 5500?

    Would be great to know more on this design aspect.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s