Transport Notes

Table of Contents

• Introduction
  • Overriding transport parameters
• UDP Transport (Sockets)
  • Transport parameters
  • Flow control parameters
  • Resize socket buffers
  • Latency Optimization
  • Linux specific notes
  • Windows specific notes
  • Mac OS X specific notes
• USB Transport (LibUSB)
  • Transport parameters
  • Setup Udev for USB (Linux)
  • Install USB driver (Windows)
• PCIe Transport (NI-RIO)
  • Transport parameters

Introduction

A transport is the layer between the packet interface and a device IO interface. The advanced user can pass optional parameters into the underlying transport layer through the device address. These optional parameters control how the transport object allocates memory, resizes kernel buffers, spawns threads, etc. When not specified, the transport layer will use values for these parameters that are known to perform well on a variety of systems. The transport parameters are defined below for the various transports in the UHD software.

Overriding transport parameters

On MPMD-based and X300 devices, applications may wish to further modify certain transport parameters beyond the default values chosen by the UHD software or those supplied via the device address. For the following transport parameters, if a value is specified in the device arguments, that value is used to configure the transport, overriding the default value chosen by UHD. Additionally, if a value is specified in the stream arguments, that value takes priority over the value specified in the device arguments for the transport associated with the stream.

Note: These default values apply to UDP transports.

• num_send_frames and num_recv_frames
  • Default value: 32
  • Note: Value is only applied to TX and RX links, not control links
• send_frame_size
  • Default value (X3x0): 1472 (if link rate is GigE) or 8000 (if link rate is 10GigE)
  • Default value (MPMD): 1472 (if link rate is GigE); 8000 (if link rate is 10GigE or 100GigE); TX MTU (other link rates)
  • Note: Value will be capped at the link's send MTU
• recv_frame_size
  • Default value (X3x0 and MPMD): 1472 (if link rate is GigE); 8000 (if link rate is 10GigE or 100GigE); RX MTU (other link rates)
  • Note: Value will be capped at the link's receive MTU
• send_buff_size
  • Default value (X3x0 and MPMD): 20 ms of data at the link rate (2.5 MB for 1GigE, 25 MB for 10GigE, 250 MB for 100GigE)
• recv_buff_size
  • Default value (X3x0 and MPMD): 20 ms of data at the link rate (X3x0: OR 64 1472-byte packets, whichever is larger)

Note: Be aware that values may be further limited due to platform- specific restrictions. See the platform-specific notes below for more details.

UDP Transport (Sockets)

The UDP transport is implemented with user-space sockets. This means standard Berkeley sockets API using send()/recv().

Transport parameters

The following parameters can be used to alter the transport's default behavior. These options can be passed to a USRP device as arguments at initialization time (see Device Configuration through address string). For MPMD-based and X3x0 devices, some of these parameters may also be passed via stream arguments (see Streaming Arguments (Stream Args)), in which case the values passed as device arguments are overridden. See Overriding transport parameters for more details on default values and the priority of device argument and stream argument overrides.

• recv_frame_size: The size of a single receive buffer in bytes
• num_recv_frames: The number of receive buffers to allocate
• send_frame_size: The size of a single send buffer in bytes
• num_send_frames: The number of send buffers to allocate
• recv_buff_fullness: The targeted fullness factor of the the buffer (typically around 90%)
• ups_per_sec: USRP2 only. Flow control ACKs per second on TX.
• ups_per_fifo: USRP2 only. Flow control ACKs per total buffer size (in packets) on TX.

Notes:

• num_recv_frames does not affect performance.
• num_send_frames does not affect performance.
• recv_frame_size and send_frame_size can be used to increase or decrease the maximum number of samples per packet. The frame sizes default to an MTU of 1472 bytes per IP/UDP packet and may be increased if permitted by your network hardware.

Flow control parameters

The host-based flow control expects periodic update packets from the device. These update packets inform the host of the last packet consumed by the device, which allows the host to determine throttling conditions for the transmission of packets. The following mechanisms affect the transmission of periodic update packets:

• ups_per_fifo: The number of update packets for each FIFO's worth of bytes sent into the device
• ups_per_sec: The number of update packets per second (defaults to 20 updates per second)

Resize socket buffers

It may be useful to increase the size of the socket buffers to move the burden of buffering samples into the kernel or to buffer incoming samples faster than they can be processed. However, if your application cannot process samples fast enough, no amount of buffering can save you. The following parameters can be used to alter socket's buffer sizes:

• recv_buff_size: The desired size of the receive buffer in bytes
• send_buff_size: The desired size of the send buffer in bytes

Note: Large send buffers tend to decrease transmit performance.

Latency Optimization

Latency is a measurement of the time it takes a sample to travel between the host and device. Most computer hardware and software is bandwidth optimized, which may negatively affect latency. If your application has strict latency requirements, please consider the following notes:

Note1: The time taken by the device to populate a packet is proportional to the sample rate. Therefore, to improve receive latency, configure the transport for a smaller frame size.

Note2: For overall latency improvements, look for "Interrupt Coalescing" settings for your OS and Ethernet chipset. It seems the Intel Ethernet chipsets offer fine-grained control in Linux. Also, consult:

• http://www.ibm.com/support/knowledgecenter/SSQPD3_2.6.0/com.ibm.wllm.doc/batchingnic.html

Linux specific notes

On Linux, the maximum buffer sizes are capped by the sysctl values net.core.rmem_max and net.core.wmem_max. To change the maximum values, run the following commands: :

sudo sysctl -w net.core.rmem_max=<new value>
sudo sysctl -w net.core.wmem_max=<new value>

Set the values permanently by editing /etc/sysctl.conf.

It is also possible to tune the network interface controller (NIC) by using ethtool. Increasing the number of descriptors for TX or RX can dramatically boost performance on some hosts.

To change the number of TX/RX descriptors, run the following command:

sudo ethtool -G <interface> tx <N> rx <N>

One can query the maximums and current settings with the following command:

ethtool -g <interface>

Tx pause frame support must be enabled when using 100GigE with X410 to ensure proper streaming performance.

To enable Tx pause frames run the following command on the device in a Linux prompt.

ethtool -A <interface> tx on

If you are using both SFP ports, you will need to run the above command separately on both of them.

One can query the the current enable status of TX pause frames by running the following command on the X410 device in a Linux prompt.

ethtool -a <interface>

The above changes will be volatile and will need to be re-applied every time the device boots or the SFP ports are reconfigured, e.g. after a fpga image load.

Automatic enable of TX pause frames will be possible in the future when systemd v2.4.6 or higher is used on the device. This will be done by using the TxFlowControl parameter in the .network files(s) used for configuring the SFP port(s) being used. E.g. When using sfp0 port, add a line TxFlowControl=1 as shown below to the /data/network/sfp0.network file.

[Link]
TxFlowControl=1

Windows specific notes

UDP send fast-path: It is important to change the default UDP behavior such that 1500 byte packets still travel through the fast path of the sockets stack. This can be adjusted with the FastSendDatagramThreshold registry key:

• FastSendDatagramThreshold registry key documented here:
  • http://www.microsoft.com/windows/windowsmedia/howto/articles/optimize_web.aspx#appendix_e
• Double click and run <install-path>/share/uhd/FastSendDatagramThreshold.reg
• A system reboot is recommended after the registry key change.

Power profile: The Windows power profile can seriously impact instantaneous bandwidth. Application can take time to ramp-up to full performance capability. It is recommended that users set the power profile to "high performance".

Mac OS X specific notes

OS X restricts the value of the send_buff_size and recv_buff_size transport parameters to a maximum value of 1 MiB (1048576 bytes).

USB Transport (LibUSB)

The USB transport is implemented with LibUSB. LibUSB provides an asynchronous API for USB bulk transfers.

Transport parameters

The following parameters can be used to alter the transport's default behavior:

• recv_frame_size: The size of a single receive transfers in bytes
• num_recv_frames: The number of simultaneous receive transfers
• send_frame_size: The size of a single send transfers in bytes
• num_send_frames: The number of simultaneous send transfers

Setup Udev for USB (Linux)

On Linux, Udev handles USB plug and unplug events. The following commands install a Udev rule so that non-root users may access the device:

cd <install-path>/lib/uhd/utils
sudo cp uhd-usrp.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules
sudo udevadm trigger

Install USB driver (Windows)

A driver package must be installed to use a USB-based product with UHD software:

• Download the driver from the UHD wiki page here.
• Unzip the file into a known location. We will refer to this as the <directory>.
• Open the device manager and plug in the USRP device. You will see an unrecognized USB device in the device manager.
• Right click on the unrecognized USB device and select update/install driver software (may vary for your OS).
• In the driver installation wizard, select "browse for driver", browse to the <directory>, and select the .inf file.
• Continue through the installation wizard until the driver is installed.

PCIe Transport (NI-RIO)

The NI-RIO-based PCIe transport is only used with the X300/X310. It uses a separate driver stack (NI-RIO) which must be installed separately (see also NI RIO Kernel Modules for X-Series PCIe Connectivity).

More information on how to set it up can be found here: PCI Express (Desktop).

The X3x0 PCIe transport has 6 separate bidirectional DMA channels, and UHD will use two of those for command, control, and asynchronous messages. That means a total of four DMA channels can be used for streaming (either 4xRX, for TwinRX operations, or 2xRX + 2xTX for full-duplex operation).

Transport parameters

The following parameters can be used to alter the transport's default behavior:

• recv_frame_size: The size of a single receive transfers in bytes
• num_recv_frames: The number of simultaneous receive transfers
• recv_buff_size: The socket buffer size. Must be a multiple of pages
• send_frame_size: The size of a single send transfers in bytes
• num_send_frames: The number of simultaneous send transfers
• send_buff_size: The socket buffer size. Must be a multiple of pages