4.2. Running OMPIF applications

Multi-node multi-fpga applications developed using the OMPIF cluster API need special setup that is not needed in regular OmpSs@FPGA applications.

4.2.1. Install cluster scripts

This includes loading bitstreams into the allocated FPGAs and setting up routing tables.

This tasks are automated for the meep machine by a series of scripts:

https://gitlab.pm.bsc.es/ompss-at-fpga/rtl/meep-ompss-fpga-cluster.git

This path is referred as $MEEP_SCRIPTS throughout the document.

4.2.2. Application execution

This section covers cluster environment setup and application execution. It is assumed that vivado and xtasks_server are available in the path. The xtasks_server binary is in the xtasks installation path ${XTASKS_INSTALL}/bin.

If application is to be run using multiple FPGAs in the same node, Start xtasks servers can be skipped.

Creating cluster description file

A json file describing the cluster needs to be created for the cluster to be automatically configured. It contains, for each FPGA, its index inside the node, the node index and the bitstream:

[
    { "fpga": FPGA_INDEX, "node": NODE_INDEX, "bitstream": BITSTREAM_PATH }
]

The following example configures a cluster using 4 FPGAs in 2 different nodes (2 FPGAs each node):

[
    { "node": 1, "fpga": 1, "bitstream" : "bitstream.bit" },
    { "node": 1, "fpga": 2, "bitstream" : "bitstream.bit" },
    { "node": 2, "fpga": 1, "bitstream" : "bitstream.bit" },
    { "node": 2, "fpga": 2, "bitstream" : "bitstream.bit" }
]

Information regarding FPGAs, can be found in /etc/motd in each of the FPGA nodes.

Configuring the FPGA cluster

Once the cluster file is created, the cluster can be configured using the create_cluster.py script from the meep-ompss-fpga-cluster repo:

python3 $MEEP_SCRIPTS/scripts/create_cluster.py cluster.json

If there are FPGAs on remote nodes, the script will automatically launch servers and connect to them. For each node, a log file ${NODENAME}_fpga_mng.log is created in the home directory. You can change the path with the --log_prefix flag. Also, for each node the script creates the xtasks_devs_$(hostname).sh file, with the XTASKS_PCI_DEV and XDMA_QDMA_DEV environment variables. By default it is created in the current working directory, but it can be changed with the --xtasks_cluster_prefix flag. The use of this file is explained in Start xtasks servers. The script also creates the xtasks.cluster file needed by your application in the current working directory. This file must be in the same directory where you launch the application, or also you can set the path in the XTASKS_CLUSTER_FILE environment variable. If not, the application will assume you are executing in single-node mode, and will not connect to the remote servers.

Start xtasks servers

A remote server that listens for FPGA tasks needs to be started in each of the remote nodes:

python3 $MEEP_SCRIPTS/scripts/launch_servers.py cluster.json --xtasks

Log files are created in the home directory named ${NODENAME}_xtasks.log. You can change the path with the --log_prefix flag. The script launches the server in the current working directory, so you must have the xtasks.cluster file in the same directory. For the moment, it can’t be set with an environment variable. Also, by default all xtasks_devs_$(hostname).sh files must be in the current directory as well. However, you can set another path with the --script_prefix flag. Then, the cluster application can be run as usual.

Debugging

There are many debug registers that can be read with QDMA, including the number of received messages, number of corrupted messages, number of send/receive tasks, etc. More details in POM AXI-Lite interface memory map.