Troubleshooting Nexus 5500s and 1 GE SFPs

The newer versions of NX-OS support 1G or 10G on the N5K, but the N5K is not auto-speed sensing. A mini case study follows.

One of my customers just put some some GLC-SX-MM optics and a fiber patch between a 3750 and a N5K. He happened to look at status on Network Assistant on the 3750, and this error appeared:

Description: Gi1/1/1: This port has been disabled because Non Compliant  Gigabit Interface Converter (GBIC) connector detected.

Recommendation: Replace connector with cisco compliant Gigabit Interface Converter (GBIC) connector. Refer switch technical documentation to determine cisco compliant connector. Enable the port again.

This seemed odd, since the GLC-SX-MM is supported on both devices. I tried shut/no shut on both ends of the port, but that did not help. I then wondered – does he have a bad SFP? I poked around a bit.

The CLI show log info on the 3750 was also unhappy:

*Apr  16 01:48:41.190: %PHY-4-SFP_NOT_SUPPORTED: The SFP in Gi1/1/1 is not supported
*Apr  16 01:48:41.190: %PM-4-ERR_DISABLE: gbic-invalid error detected on Gi1/1/1, putting Gi1/1/1 in err-disable state

Strange, the GLC-SX-MM is supported on the 3750. The interface capabilities looked fine:

3750_sw1#sh int gig 1/1/1 capa
GigabitEthernet1/1/1
  Model:                 WS-C3750X-48P
  Type:                  1000BaseSX SFP
  Speed:                 1000
  Duplex:                full
  Trunk encap. type:     802.1Q,ISL
  Trunk mode:            on,off,desirable,nonegotiate
  Channel:               yes
  Broadcast suppression: percentage(0-100)
  Flowcontrol:           rx-(off,on,desired),tx-(none)
  Fast Start:            yes
  QoS scheduling:        rx-(not configurable on per port basis),
                         tx-(4q3t) (3t: Two configurable values and one fixed.)
  CoS rewrite:           yes
  ToS rewrite:           yes
  UDLD:                  yes
  Inline power:          no
  SPAN:                  source/destination
  PortSecure:            yes

  Dot1x:                 yes
3750_sw1#

The status on the 3750 was not connected:

3750_sw1#sh int gig 1/1/1 status

Port      Name               Status       Vlan       Duplex  Speed Type
Gi1/1/1                      notconnect   1            auto   auto 1000BaseSX SFP
3750_sw1#

I then went to look at the N5K. It claimed it has an invalid SFP of an unkown type:

N5K1# sh int e1/5 status

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex  Speed   Type
--------------------------------------------------------------------------------
Eth1/5        --                 sfpInvali 1         full    10G     SFP-1000BAS
N5K1#
N5K1# sh int e1/5 capa
Ethernet1/5
  Model:                 N5K-C5548UP-SUP
  Type (SFP capable):    10Gbase-(unknown)
  Speed:                 1000,10000
  Duplex:                full
  Trunk encap. type:     802.1Q
  Channel:               yes
  Broadcast suppression: no
  Flowcontrol:           rx-(off/on),tx-(off/on)
  Rate mode:             none
  QOS scheduling:        rx-(6q1t),tx-(1p6q0t)
  CoS rewrite:           no
  ToS rewrite:           no
  SPAN:                  yes
  UDLD:                  yes
  Link Debounce:         yes
  Link Debounce Time:    yes
  MDIX:                  no
  Pvlan Trunk capable:   yes
  TDR capable:           no
  FabricPath capable:    yes
  Port mode:             Switched
  FEX Fabric:            yes

N5K1#

The N5K also claimed there was no transceiver:

N5K1#sh log | inc 1/5
...
2012 Apr 6 18:10:26 N5K1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet1/5 is down (None)
...
2012 Apr 6 19:06:10 N5K1 %ETHPORT-5-IF_HARDWARE: Interface Ethernet1/5, hardware type changed to No-Transceiver

I scratched my head a bit, and then came up with the underlying issue – when installing 1GE optics in the N5K, you need to manually set the speed. If you look back, you will see the previous ‘sh int e1/5 status‘ on the N5K shows a speed of 10G. So after the appropriate “speed 1000” command to the e1/5 interface on the N5K, the link came up on both ends.

N5K1# sh int e1/5 stat

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex  Speed   Type
--------------------------------------------------------------------------------
Eth1/5        --                 connected 1         full    1000     SFP-1000BAS
N5K1#

Summary: The newer versions of NX-OS support 1G or 10G on the N5K, but the N5K is not auto-speed sensing. If you run into something similar, even error messages on the remote device, check the port speed on the N5K.

— cwr