TestDevicesNX3031_HP

NX3031 is a network card that we use in testing, e.g. in TestNetwork10gNet1.

Dual-socket (module), card is NetXen model NX3031 (brand is HP, NC522SFP, maybe NC522), both sockets have module by Intel (model: 🚧...?) rated at 10 Gbit/sec (entire card is rated at 20 Gbit/s)

Summary: the card works somewhat-well, from Debian 8, Debian 9; Driver install not required (default system drivers) - but seems installing the non-free firmware makes it work better (debian 8 repo non-free, package firmware-netxen)

• so far, card achieves max 13.5 Gbit/s in SUM, that is if you send through both sockets at once, the speed spreads across them and is NOT reaching twice as one module, stays as 1 module - see experiment below
• small problem, on CompWolf tested in TestNetwork10gNet1 on IP .114 (so module PHYAD: 0) was turning itself off during sending, two times (was de-configuring IP address) - but after a while it fixed itself (bad default kernel/driver params for timer?) - perf: interrupt took too long (3190 > 3187), lowering kernel.perf_event_max_sample_rate to 62500 . No such problem on CompDragon
• It is picky (or kernel/BIOS fault) and sometimes it works only in certain PCIe port (e.g. on CompHarry-v1 only in the top PCIe port).

See also TestNetwork10gNet1 for more details - and example use.

References/materials:

• https://wiki.pdl.cmu.edu/pub/OpenCloud/CloudManuals/NetXen_Users_Guide_rev40_01-07-09.pdf

01:00.0 Ethernet controller: NetXen Incorporated NX3031 Multifunction 1/10-Gigabit Server Adapter (rev 42)
        Subsystem: Hewlett-Packard Company NC522SFP Dual Port 10GbE Server Adapter
01:00.1 Ethernet controller: NetXen Incorporated NX3031 Multifunction 1/10-Gigabit Server Adapter (rev 42)
        Subsystem: Hewlett-Packard Company NC522SFP Dual Port 10GbE Server Adapter

Parts owned (by our community)

Part "nx3031-1"

Since 2017-10
Device S/N 59-69-46-61-6e-48-73-75 (in lspci) , also "CR...1R?" (on the device label?)

• port1 MAC 78:e3:b5:f5:23:40
• port2 MAC 78:e3:b5:f5:23:44

ipartid=p17xND3K9

Part "nx3031-2"

Since 2017-10
Device S/N ... (in lspci) , also "..." (on the device label?)

• port1 MAC
• port2 MAC

ipartid=p17xYWNTA

Part "nx3031-3"

Since 2017-12

Part "nx3031-4"

Since 2017-12
Device S/N 59-69-46-61-6e-48-73-75 (in lspci).

• port1 MAC e8:39:35:02:1d:20
• port2 MAC e8:39:35:02:1d:24

Last worked: CompDragon 2017-12-09 22:30 UTC (in 1st PCI-e slot, after reboots).

Installing

To get it to work:

Error waiting for rom done, netxen_nic probe failed with error -5

Dec  9 22:45:56 dragon kernel: [    1.344655] QLogic/NetXen Network Driver v4.0.82
Dec  9 22:45:56 dragon kernel: [    1.344893] netxen_nic 0000:01:00.0: 2MB memory map
Dec  9 22:45:56 dragon kernel: [    1.345303] Error waiting for rom done
Dec  9 22:45:56 dragon kernel: [    1.345449] netxen_nic: probe of 0000:01:00.0 failed with error -5
Dec  9 22:45:56 dragon kernel: [    1.345583] netxen_nic 0000:01:00.1: 2MB memory map
Dec  9 22:45:56 dragon kernel: [    1.345928] Error waiting for rom done
Dec  9 22:45:56 dragon kernel: [    1.346031] netxen_nic: probe of 0000:01:00.1 failed with error -5

This usually means the device will NOT work on THIS BOOTUP.

Solution: soft-rebooted (with command reboot from Linux, without disconnecting power nor shutting down) got it to work after first failing as above (perhaps after few reboots).

Correctly it looks like:

Dec  9 23:27:02 dragon kernel: [    1.340748] QLogic/NetXen Network Driver v4.0.82
Dec  9 23:27:02 dragon kernel: [    1.340939] netxen_nic 0000:01:00.0: 2MB memory map

Dec  9 23:27:02 dragon kernel: [    1.343634] netxen_nic 0000:01:00.0: firmware: direct-loading firmware phanfw.bin

Dec  9 23:27:02 dragon kernel: [    1.546808] netxen_nic 0000:01:00.0: Gen2 strapping detected
Dec  9 23:27:02 dragon kernel: [    1.546830] netxen_nic 0000:01:00.0: using 64-bit dma mask

Dec  9 23:27:02 dragon kernel: [    1.907160] netxen_nic: Dual XGb SFP+ LP Board S/N ZQ23BK2658  Chip rev 0x42
Dec  9 23:27:02 dragon kernel: [    1.907164] netxen_nic 0000:01:00.0: Driver v4.0.82, firmware v4.0.593 [legacy]
Dec  9 23:27:02 dragon kernel: [    1.907275] netxen_nic 0000:01:00.0: using msi-x interrupts
Dec  9 23:27:02 dragon kernel: [    1.907277] netxen_nic 0000:01:00.0: ULA adapter
Dec  9 23:27:02 dragon kernel: [    1.907416] netxen_nic 0000:01:00.0: eth2: XGbE port initialized
Dec  9 23:27:02 dragon kernel: [    1.907479] netxen_nic 0000:01:00.1: 2MB memory map
Dec  9 23:27:02 dragon kernel: [    1.907600] netxen_nic 0000:01:00.1: using 64-bit dma mask


Dec  9 23:27:02 dragon kernel: [    1.943194] netxen_nic 0000:01:00.1: Driver v4.0.82, firmware v4.0.593 [legacy]
Dec  9 23:27:02 dragon kernel: [    1.943264] netxen_nic 0000:01:00.1: using msi-x interrupts
Dec  9 23:27:02 dragon kernel: [    1.943407] netxen_nic 0000:01:00.1: eth3: XGbE port initialized

log

Obtained 2017.10.13 (first devices), more on 2017.12.

lspci (in Dragon)

01:00.0 Ethernet controller: NetXen Incorporated NX3031 Multifunction 1/10-Gigabit Server Adapter (rev 42)
        Subsystem: Hewlett-Packard Company NC522SFP Dual Port 10GbE Server Adapter
        Flags: bus master, fast devsel, latency 0, IRQ 24
        Memory at f6200000 (64-bit, non-prefetchable) [size=2M]
        Memory at f4000000 (64-bit, non-prefetchable) [size=32M]
        Expansion ROM at f6400000 [disabled] [size=64K]
        Capabilities: [40] MSI-X: Enable+ Count=64 Masked-
        Capabilities: [80] Power Management version 3
        Capabilities: [a0] MSI: Enable- Count=1/32 Maskable- 64bit+
        Capabilities: [c0] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number 59-69-46-61-6e-48-73-75
        Kernel driver in use: netxen_nic

01:00.1 Ethernet controller: NetXen Incorporated NX3031 Multifunction 1/10-Gigabit Server Adapter (rev 42)
        Subsystem: Hewlett-Packard Company NC522SFP Dual Port 10GbE Server Adapter
        Flags: bus master, fast devsel, latency 0, IRQ 24
        Memory at f6000000 (64-bit, non-prefetchable) [size=2M]
        Memory at f2000000 (64-bit, non-prefetchable) [size=32M]
        Capabilities: [40] MSI-X: Enable+ Count=64 Masked-
        Capabilities: [80] Power Management version 3
        Capabilities: [a0] MSI: Enable- Count=1/32 Maskable- 64bit+
        Capabilities: [c0] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number 59-69-46-61-6e-48-73-75
        Kernel driver in use: netxen_nic

ethtool (general)

root@dragon:~# ethtool eth3
Settings for eth3:
        Supported ports: [ TP FIBRE ]
        Supported link modes:   10000baseT/Full 
        Supported pause frame use: No
        Supports auto-negotiation: No
        Advertised link modes:  10000baseT/Full 
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Speed: 10000Mb/s
        Duplex: Full
        Port: FIBRE
        PHYAD: 1
        Transceiver: external
        Auto-negotiation: off
        Supports Wake-on: g
        Wake-on: g
        Link detected: yes
root@dragon:~# ethtool eth2
Settings for eth2:
        Supported ports: [ TP FIBRE ]
        Supported link modes:   10000baseT/Full 
        Supported pause frame use: No
        Supports auto-negotiation: No
        Advertised link modes:  10000baseT/Full 
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Speed: 10000Mb/s
        Duplex: Full
        Port: FIBRE
        PHYAD: 0
        Transceiver: external
        Auto-negotiation: off
        Supports Wake-on: g
        Wake-on: g
        Link detected: yes

firmware, versions

Debian 8 default (free?) driver results in:

root@harry:~# ethtool -i eth2
driver: netxen_nic
version: 4.0.82
firmware-version: 4.0.593
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

installing there firmware-netxen from non-free repository, seems to give same effect, same firmware 4.0.593 and version 4.0.82

ethtool -c

# ethtool -c eth3
Coalesce parameters for eth3:
Adaptive RX: off  TX: off
stats-block-usecs: 0
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0

rx-usecs: 3
rx-frames: 256
rx-usecs-irq: 0
rx-frames-irq: 0

tx-usecs: 4
tx-frames: 64
tx-usecs-irq: 0
tx-frames-irq: 0

rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0

rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0

Following settings seem to reduce interrupts count (from ~95,000 to ~40,000) while doing test like sending 2cards * 10Gbit (settings on CompDragon, sending from CompHarry).

for NIC in eth2 eth3 ; do ethtool -C $NIC  rx-usecs 60 rx-frames 1024 ; done

misc

cooling

needs external cooling - in CompDragon

[  673.891309] eth2: Device temperature 93 degrees C exceeds operating range. Immediate action needed.
[  673.947368] eth3: Device temperature 93 degrees C exceeds operating range. Immediate action needed.
[  874.475180] eth2: Device temperature 100 degrees C exceeds maximum allowed. Hardware has been shut down.
[  874.543239] eth3: Device temperature 100 degrees C exceeds maximum allowed. Hardware has been shut down.

experiment

testing max speed when sending via both modules at once

UDP sending: http://p.l/p/aqpx-netexpWolf10gbitBothModule.ogv

TCP sending: http://p.l/p/aqoy-netexpWolf10gbitBothModule-tcp.ogv

Property drivers:

RedHat linux server 7

there is no property drivers for kernel > 2.6.32 (for RedHat 6). The only option seen to be is using netxen_nic drivers