This README is for OpenSM and the InfiniBand diagnostic utilities in this directory (management). The master source repository is git://git.openfabrics.org/~sashak/management.git and can be cloned by: git clone git://git.openfabrics.org/~sashak/management.git Packages -------- libibcommon - common stuff libibumad - interface to ib_umad module (user_mad) library libibmad - generic MAD handling library opensm - OpenSM infiniband-diags - various diagnostic tools Building -------- To make this unpack tarballs and in directories libibcommon, libibumad, libibmad, opensm, infiniband-diags (in that order) run: ./configure && make && make install (If you are building the cloned repository run also ./autogen.sh first) Typically the autogen and configure steps only need be done the first time unless configure.in or Makefile.am changes in the directories. Libraries are installed by default at /usr/local/lib and binaries at /usr/local/sbin. Running ------- After compiling and installing, you can run opensm by invoking /usr/local/sbin/opensm opensm must be run as root. Run 'opensm --help' to see the options. Note also that you must have udev mount /dev/infiniband or do it manually. See .../src/linux-kernel/docs/user_mad.txt. Also, ib_umad module must be loaded. opensm will run on the first existing port on the first IB device (HCA). You can override that by using "-g ". Verify that the first port is active. This assumes the port is plugged into another IB device. In case of problems, run the opensm with -V and send the log file (/var/log/opensm.log). IMPORTANT: Don't forget to modprobe ib_umad and make sure udev is configured before using any of the userspace programs. OpenSM Limitations: 1. Retry mechanism in SM is primitive and needs enhancing to deal with ports which are active but don't respond to SM MADs. 2. Async events are not yet supported (by OpenSM). The only one supported is local LID change (and this is handled in the mthca driver). Future versions of OpenSM may need to act on more local events. Tuning OpenSM for Large Clusters -------------------------------- Currently OpenSM is compiled with debug and no optimization. This should be changed to at least -O2 (and perhaps -O4) but I would start with -O2. This results in a 2x speedup for some code paths. OpenSM supports a pipelining mode for SMPs. The default is 4 outstanding SMPs. -maxsmps <#> indicates the number of outstanding SMPs allowed and should speed up the initialization. Useful values of this are 16 and 32. Beyond this, there may be some issue with a link which is causing timeout and retries to kick in. The OpenSM log should have some messages in there indicating this. Other utilities (infiniband diagnostics) --------------------------------------- ibstat - show host adapters status ibstatus - similar to ibstat but implemented as a script ibnetdiscover - scan topology ibaddr - shows the lid range and default GID of the target (default is the local port) ibroute - display unicast and multicast forwarding tables of switches ibtracert - display unicast or multicast route from source to destination ibping - ping/pong between IB nodes (currently using vendor MADs) ibsysstat - obtain basic information for node (hostname, cpus, memory, utilization) which may be remote sminfo - query the SMInfo attribute on a node smpdump - simple solicited SMP query tool. Output is hex dump (unless requested otherwise, e.g. using -s) smpquery - formatted SMP query tool perfquery - dump (and optionally clear) the performance (including error) counters of the destination port ibcheckport - perform some basic tests on the specified port ibchecknode - perform some basic tests on the specified node ibcheckerrs - check if the error counters of the port/node have passed some predefined thresholds ibchecknet - perform port/node/errors check on the subnet. ibnetdiscover output can be used as in input topology ibswitches - scan the net or use existing net topology file and list all switches ibhosts - scan the net or use existing net topology file and list all hosts ibnodes - scan the net or use existing net topology file and list all nodes ibportstate - get the logical and physical port state of an IB port or disable or enable the port (only on a switch) ibcheckwidth - perform port width check on the subnet. Used to find ports with 1x link width. ibcheckportwidth - perform 1x port width check on specified port ibcheckstate - perform port state (and physical port state) check on the subnet. Used to find ports not in LinkUp physical port state and not Active port state ibcheckportstate - perform port state (and physical port state) check on specified port ibcheckerrors - perform error check on subnet. Used to find ports with error counters (PMA PortCounters) beyond the indicated thresholds ibclearerrors - clear all error counters on subnet ibclearcounters - clear all port counters on subnet ibdiscover.pl - takes output of ibnetdiscover and a map file and produces a topology file (local node GUID and port connected to remote node GUID and port) saquery - issue some SA queries Note that the above list is not up to date and the infiniband-diags subdirectory should be checked for the latest tools.