Fibre Channel card initialization

I had a new SUN M4000 server with two identical QLOGIC FC cards installed. The first FC interface came up without any problem and I kind of forgot about the second one.

Well, in order to move the server into production I needed to multipath the two cards together using the stmsboot -e command, but the second card wasn't responding. I checked the physical connections between the FC switch and the server, but that wasn't it. I tried to poke the system using the cfgadm command -- no luck...


# cfgadm -c configure c2
# cfgadm -al
Ap_Id Type Receptacle Occupant Condition
[...]
c1 fc-fabric connected configured unknown
c2 fc connected unconfigured unknown

# fcinfo hba-port
HBA Port WWN: xxxxxxxx294
OS Device Name: /dev/cfg/c1
Manufacturer: QLogic Corp.
Model: 375-3355-02
Firmware Version: 4.04.01
FCode/BIOS Version: BIOS: 1.24; fcode: 1.24; EFI: 1.8;
Serial Number: 0402G00-0905674110
Driver Name: qlc
Driver Version: 20080617-2.29
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: 4Gb
Node WWN: xxxxxxxx294
HBA Port WWN: xxxxxxxx594
OS Device Name: /dev/cfg/c2
Manufacturer: QLogic Corp.
Model: 375-3355-02
Firmware Version: 4.04.01
FCode/BIOS Version: BIOS: 1.24; fcode: 1.24; EFI: 1.8;
Serial Number: xxx-xxx
Driver Name: qlc
Driver Version: 20080617-2.29
Type: unknown
State: offline
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: not established
Node WWN: xxxxxxxx594

# luxadm -e port
/devices/pci@3,700000/SUNW,qlc@0/fp@0,0:devctl CONNECTED
/devices/pci@2,600000/SUNW,qlc@0/fp@0,0:devctl NOT CONNECTED

No commands would bring the card to life so the FC switch wouldn't see anything attached to the port.

After a lot of troubleshooting I took it to basics and halted the system and reset the bus:

{1} ok
{1} ok reset-all
{1} ok probe-scsi-all

now the second card was responding, so I rebooted the server:

{1} ok boot -rv.

and now it's fine...

# cfgadm -al
[...]
c1 fc-fabric connected configured unknown
[bunch of disks]
c2 fc-fabric connected configured unknown
[bunch of disks]

Solaris 10 routes

If you have a Solaris 10 server with several NIC's and a different subnet plugged into each NIC, you might set up zones to use a specific NIC. In order for the non-global zone to route correctly, you will need to set up routing on the global zone.

I've found that putting in a startup script in /etc/rc3.d/S99add-route doesn't always work.

By the way, I use these commands in the script:

#!/bin/sh
/usr/sbin/ifconfig nxge0 plumb
/usr/sbin/ifconfig nxge1 plumb
/usr/sbin/route add default 10.10.10.1 -ifp nxge0
/usr/sbin/route add default 10.10.20.1 -ifp nxge1

It seems that SUN has added a "-p" flag to the route command that now makes the statement persistent across reboots, rendering the script moot:

# /usr/sbin/route -p add default 10.10.10.1 -ifp nxge0
add net default: gateway 10.10.10.1: entry exists
add persistent net default: gateway 10.10.10.1

# /usr/sbin/route -p add default 10.10.20.1 -ifp nxge1
add net default: gateway 10.10.20.1: entry exists
add persistent net default: gateway 10.10.20.1

The entries are written to the file: /etc/inet/static_routes

# File generated by route(1M) - do not edit.
default 10.10.10.1 -ifp nxge0
default 10.10.20.1 -ifp nxge1