$FreeBSD: head/sys/dev/vxge/README 221167 2011-04-28 14:33:15Z gnn $ ''_Readme for FreeBSD X3100 Series 10GbE PCIe I/O Virtualized Server Adapter Drivers_''' === Introduction === FreeBSD Driver for X3100 10GbE Server/Storage adapters * Drivers support all X3100 10GbE adapters with FreeBSD version 7.x, 8.x and 9.x * Supports both i386 and amd64 architectures * Features: Jumbo frames (up to 9600), LRO (Large Receive Offload), TSO (TCP segmentation offload), RTH (Receive Traffic Hash). Also, Extended Message Signaled Interrupts (MSI-X). ''Features'' a. Jumbo frames: X3110 and X3120 supports MTU up to 9600 bytes, modifiable using ifconfig command. b. LRO (Large Receive Offload): LRO can be enabled/disabled before loading driver. Set lro_enable in vxge.conf to 1 before loading driver. c. TSO (TCP Segmentation Offload) TSO can be enabled/disabled before loading driver. Set tso_enable in vxge.conf to 1 before loading driver. d. RTH (Receive Traffic Hash) Receive side steering for better scaling. Set rth_enable in vxge.conf to 1 before loading driver. e. MSI-X Can be enabled on platforms which support it, resulting in noticeable performance improvement. f. Multi-VPaths Up to 17 hardware based transmit and receive data channels, with multiple steering options. ''X3100 & Driver configuration: vxge.conf'' The vxge.conf contains following attributes. ''msix_enable'' Enable MSI (Message Signaled Interrupts) feature in driver. 0 - INTA 1 - MSI-X Default: 1 ''rth_enable'' Enables Receive side steering for better scaling (RTH - Receive Traffic Hash) Range: 0 - 1 Default: 1 ''lro_enable'' Enables LRO (Large Receive Offload) feature in driver. Range: 0 - 1 Default: 1 ''tso_enable'' Enables TSO (TCP Segmentaton Offload) feature in driver. Range: 0 - 1 Default: 1 ''no_of_vpath'' Specifies maximum VPATH(s) configured for each device function. Valid range: 1-17 Default: Optimized by driver ''func_mode'' Change PCI function mode 0 - SF1_VP17 (1 function with 17 VPATHs) 1 - MF8_VP2 (8 functions with 2 VPATHs per function) 8 - MF2_VP8 (2 functions, 8 Paths/Function) 9 - MF4_VP4 (4 Functions, 4 Paths/Function) 11 - MF8P_VP2 (8 functions with 2 VPATHS per function required for DirectIO) Default: -1 ''port_mode'' Change the default dual port mode 2 - Active Passive 3 - Single Port 4 - Dual Port ''l2_switch'' Turn on/off the inter function traffic through l2 switch 0 - Disallow inter function traffic 1 - Allow inter function traffic Default: -1 ''bandwidth_0 - bandwidth_7'' Desired max receive/transmit bandwidth,in Mbps for function 0 to function 7 Minimum value is 100 Mbps, for 1 Gbps specify a value of 1024. ''priority_0 - priority_7'' Desired receive/transmit priority for function 0 to function 7 ''intr_coalesce'' Adaptive interrupt coalescing 0 - Disable 1 - Enable ''Low Latency'' 0 - Disable 1 - Enable === Installation Instructions === ''Identifying the Adapter'' The X3100 adapter is identified by the board ID number on the adapter. Look for a label that has a barcode and a number, for example, SXT0425072. The factory-burned MAC address (hardware address) shows up on the board above the serial number, (similar to 000CFC000449 -- 00:0C:FC:00:04:49). ''Kernel Driver Source Package'' This package contains kernel_update.sh script which is to be used to copy driver sources to kernel path. It creates vxge folder with source code in /usr/src/sys/dev and Makefile in /usr/src/sys/modules. Loadable parameters can be changed by putting below lines in /boot/device.hints and set values as desired. hint.vxge.0.msix_enable="1" hint.vxge.0.rth_enable="1" hint.vxge.0.lro_enable="1" hint.vxge.0.tso_enable="1" hint.vxge.0.tx_steering="1" hint.vxge.0.no_of_vpath="-1" hint.vxge.0.func_mode="-1" hint.vxge.0.port_mode="-1" hint.vxge.0.fw_upgrade="1" hint.vxge.0.bandwidth_0="-1" hint.vxge.0.bandwidth_1="-1" hint.vxge.0.bandwidth_2="-1" hint.vxge.0.bandwidth_3="-1" hint.vxge.0.bandwidth_4="-1" hint.vxge.0.bandwidth_5="-1" hint.vxge.0.bandwidth_6="-1" hint.vxge.0.bandwidth_7="-1" hint.vxge.0.priority_0="-1" hint.vxge.0.priority_1="-1" hint.vxge.0.priority_2="-1" hint.vxge.0.priority_3="-1" hint.vxge.0.priority_4="-1" hint.vxge.0.priority_5="-1" hint.vxge.0.priority_6="-1" hint.vxge.0.priority_7="-1" hint.vxge.0.intr_coalesce="0" hint.vxge.0.low_latency="0" e.g., set hint.vxge.0.msix_enable to 0 to load driver in INTA mode. Then reboot the system to add loadable parameters to kenv. ''Standalone Driver Source Package'' vxge (FreeBSD package) This directory contains FreeBSD driver sources for X3100 device(s), Makefile, and X3100 Hardware Abstraction headers and sources (vxgehal and include folders) vxgehal This directory contains the X3100 HAL sources. The driver uses HAL to perform operations on the X3100 hardware. include The include subdirectory contains HAL header files. Source code for vxge-manage tool (used to get statistics, pciconfig and register dump) are included in the freebsd directory. ''Building the driver'' The driver is distributed in the source form. Driver and installation utility executables need to be built for the target platform. In the directory containing Makefile for building Exar driver for FreeBSD, #make clean #make Please unload previously installed Exar drivers before proceeding with following steps. #make uninstall ''Loading the driver'' Use "kldload" to load driver module vxge.ko. #kldload ./vxge.ko Run "kldstat" and find an entry for vxge kernel module to ensure driver installation was successful. #kldstat | grep vxge 3 1 0xc22cc000 26000 vxge.ko ''Enabling interface and assigning IP address'' #ifconfig up will be similar to vxge0, vxge1 etc. and can be found by executing "ifconfig -a". Neterion adapters typically have MAC addresses starting with "00:0C:FC" or "00:11:25". Example: #ifconfig vxge0 10.2.2.40 up Enables vxge0 interface and assigns to it the IP address 10.2.2.40. vxge0: flags=8843 metric 0 mtu 1500 options=53b ether 00:0c:fc:00:da:47 inet6 fe80::20c:fcff:fe00:da47%vxge0 prefixlen 64 scopeid 0x3 inet 10.2.2.40 netmask 0xff000000 broadcast 17.255.255.255 media: Ethernet autoselect (10Gbase-SR ) status: active ''Disabling the interface'' #ifconfig down Example: #ifconfig vxge0 down ''Unloading the Driver'' #kldunload vxge.ko === Performance Suggestions === Sysctl Tuning Parameters #sysctl net.inet.tcp.sendspace=786432 #sysctl net.inet.tcp.recvspace=786432 #sysctl net.inet.tcp.recvbuf_max=16777216 #sysctl net.inet.tcp.sendbuf_max=16777216 #sysctl net.inet.tcp.blackhole=1 #sysctl net.inet.tcp.rfc1323=1 #sysctl net.inet.tcp.path_mtu_discovery=1 #sysctl net.inet.tcp.inflight.enable=0 #sysctl net.inet.ip.maxfragsperpacket=2147483647 #sysctl kern.ipc.maxsockbuf=8388608 #sysctl kern.ipc.nmbclusters=2147483647 #sysctl kern.ipc.nmbjumbop=262144 #sysctl kern.ipc.maxsockets=81920 #sysctl hw.intr_storm_threshold=9000 ''Usage & Troubleshooting'' For general information and support, please visit Neterion support website at http://www.neterion.com/support/support.html Make sure that the operating system identifies the X3100 adapter. Note that Neterion vendor ID is 0x17D5 and X3110 and X3120 adapters can be fixed to both PCIe slots. The rest of this section details troubleshooting tips and information. Some of them are general and some are more specific. For online Troubleshooting tips and faqs, please visit http://trac.neterion.com/cgi-bin/trac.cgi/wiki/TitleIndex?anonymous ''Loading the driver and initializing the device'' The first time FreeBSD identifies the device it stores the corresponding device/vendor IDs in an enumerated tree of PCI devices. Note that Neterion vendor id is 0x17d5. After cold reboot FreeBSD finds the device and tries to load the corresponding driver. If it fails, try to switch the card and/or cable. And, in parallel, send us the "tail" of the "/var/log/messages". We also need traces, register dump and statistics(Use vxge-manage tool). ''Collect information'' If you're reporting a problem to Neterion, please describe: host(s) and adapter(s), switch (if used), software version. ''ARP'' If this is a basic connectivity issue (e.g., cannot connect, cannot ping), make sure first that ARP works Do you see ARPs coming through a switch (in case switch is used)? Do you see frame drops at the switch when pinging? Do you see frame counts increasing via statistics? ''Have you tried'' a. A previous driver release; b. A different adapter in the same PCI slot; c. A different PCI slot; d. Back-to-back setup so that the switch is excluded from the equation. e. To replace the cables? f. To use a different PCI slot? ''Start clean'' a. Have you tried to reboot the switch? Cold-reboot the host? b. Make sure that the latest released driver gets loaded after the host reboot, and that the 10GE interface shows up via ifconfig. ''LEDs'' Do you see LED going green after everything is connected and drivers loaded? How/when does the color changes? ''ifconfig'' Run "ifconfig -a" at the command prompt and check whether the output Looks as expected. Include the output in your problem report. Note for instance that "all-foxes" i.e, FF:FF:FF:FF:FF:FF MAC address could explain a general connectivity issue, if that's what you see. ''Log'' Please attach the generated log, with traces enabled. Note that the driver's logging facility is configurable at compile-time. Errors and traces can be compiled out on a per-component basis. The components are: HAL fifos and rings, device, etc., see VXGE_COMPONENT_HAL_??? in the Makefile. Here's how you enable all except data path traces: CFLAGS_VXGE = -DVXGE_DEBUG_MODULE_MASK=0xffffffbf \ -DVXGE_DEBUG_ERR_MASK=0xffffffbf Recompile with traces and include the log in the report. === Utilities === ''Statistics'' To print hardware and software statistics for interface instance 0 (i.e., vxge0), run: #vxge-manage vxgeX stats common #vxge-manage vxgeX stats mrpcim #vxge-manage vxgeX stats driver #vxge-manage vxgeX pciconfig #vxge-manage vxgeX hwinfo #vxge-manage vxgeX bw_pri_get #vxge-manage vxgeX bw_pri_get vf_id #vxge-manage vxgeX port_mode_get The vxge-manage tool generates log file in the working directory. Once done, ping a few times, and collect the statistics again (Ping both from this and the remote machines). Many counters could be of interest. For example, "rx_vld_frms" counts all valid incoming Ethernet frames seen by the adapter. Information could be derived from the fact that (for instance) counter stay constant during ping, if that is what happening. For detailed description of the X3100 counters, please refer to the "X3100 User Guide". Please include the statistics into your problem report. ''X3100 registers'' Use vxge-manage to dump all X3100 BAR0 space registers. Include this register dump into your problem report. #vxge-manage vxgeX regs ''PCI configuration space'' Use vxge-manage to retrieve PCI vendor, device, etc. Include the PCI configutation space in your problem report. #vxge-manage vxgeX pciconfig ''Hardware Info'' To retrieve hardware info of device, e.g, serial / part number and function mode etc. use vxge-manage #vxge-manage vxgeX hwinfo ''Bandwidth and Priority'' Use vxge-manage to display Bandwidth and Priority information. #vxge-manage vxgeX bw_pri_get or #vxge-manage vxgeX bw_pri_get vf_id vxge-manage can also be used to set bandwidth and priority for individual VF. #vxge-manage vxgeX bw_pri_set vf_id bandwidth or #vxge-manage vxgeX bw_pri_set vf_id bandwidth priority Example: #vxge-manage vxge0 bw_pri_set 0 1500 or #vxge-manage vxge0 bw_pri_set 0 1500 1 ''Port mode'' Use vxge-manage to display Port mode setting #vxge-manage vxgeX port_mode_get vxge-manage can also be used to set Port mode. #vxge-manage vxgeX port_mode_set port_mode_value Example: #vxge-manage vxge0 port_mode_set 2 === Known Issues === === Available Downloads === For latest available drivers or further support please contact your network adapter provider or neterionsupport@exar.com. =================================================================== Exar Corp., Proprietary COPYRIGHT (c) 2002-2011 Exar corp., ALL RIGHTS RESERVED