I need to optimize the PCI Express performance for my QDR/FDR IB HCA?
The MaxReadRequest feature is used to ensure a fairer bandwidth allocation of PCI Express. It specifies the largest read request any PCI Express device can generate. To reduce the way the bus is hogged by large reads one reduces the MaxReadRequest size.
The MaxReadRequest size is set too low and will affect performance. It will provide excellent bus sharing at the cost of bus data transfer rates. Although not a critical issue, it may be worth considering setting the MaxReadRequest size to 256 or 512 to increase throughput on the PCI Express bus.
Detecting the card:
[root@node003 ~]# lspci | grep Mellanox03:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3] Instead of the vendor name the hex vendor identifier can be specified:
[root@node003 ~]# lspci -n | grep 15b303:00.0 0280: 15b3:1003[root@node003 ~]#
The following command will reveal the PCI Express link speed:
setpci -d "15b3:" 72
If the output is neither 81 nor 82, then the card is NOT installed in an x8 PCI Express slot.
The following command will reveal the value of MaxReadReq:
[root@node003 ~]# setpci -d "15b3:" 68.w2000
To modify this value for all the nodes it is advised to do that via an "initialize script". Doing at this early stage ensures that this setting will be used also during the node provisioning if the nodes are set to be provisioned over IB.
[root@demo ~]# cmsh[demo]% category use mlnx-mic[demo->category[mlnx-mic]]% set initializescript[demo->category[mlnx-mic]]% commit
"set initialzescript" will open an editor session e.g. vim.
There you can add the setpci command:
/sbin/setpci -d "15b3:" 68.w