6.2. Xaloc cluster installation

The OmpSs-2@FPGA releases are automatically installed in the Xaloc cluster. They are available through a module file for each target architecture. This document describes how to load and use the modules to compile an example application. Once the modules are loaded, the workflow in the Xaloc cluster should be the same as in the Docker images.

6.2.1. General remarks

  • The OmpSs@FPGA toolchain is installed in a version folder under the /opt/bsc/ directory.
  • Third-party libraries required to run some programs are installed in the corresponding folder under the /opt/lib/ directory.
  • The rest of the software (Xilinx toolchain, slurm, modules, etc.) is installed under the /tools/ directory.
  • During the updates, the installation will not be available for the users’ usage.
  • Usually, the installation takes about 30 minutes.
  • After the installation, an informative email will be sent.

6.2.3. Logging into xaloc

Xaloc is accessible from HCA ssh.hca.bsc.es Alternatively, it can be accessed through the 4810 port in HCA and ssh connection will be redirected to the actual host:

ssh -p 4810 ssh.hca.bsc.es

Also, this can be automated by adding a xaloc host into ssh config:

Host xaloc
    HostName ssh.hca.bsc.es
    Port 8410

6.2.4. Module structure

The ompss-2 modules are:

  • ompss-2/x86_64/*[release version]*

This will automatically load the default Vivado version, although an arbitrary version can be loaded before ompss:

module load vivado/2023.2 ompss-2/x86_fpga/git

To list all available modules in the system run:

module avail

6.2.5. Build applications

To generate an application binary and bitstream, you could refer to Compile OmpSs-2@FPGA programs as the steps are general enough.

Note that the appropriate modules need to be loaded. See Module structure.

6.2.6. Running applications

6.2.6.1. Get access to an installed fpga

Xaloc cluster uses SLURM in order to manage access to computation resources. Therefore, to be able to use the resources of an FPGA, an allocation in one of the partitions has to be made.

There is 1 partition in the cluster:

  • fpga: a Versal VCK5000 board

The easiest way to allocate an FPGA is to run bash through srun with the --gres option:

srun --gres=fpga:BOARD:N --pty bash

Where BOARD is the FPGA to allocate, in this case versal, and N the number of FPGAs to allocate, that is 1.

For instance, the command:

srun --gres=fpga:versal:1 --pty bash

Will allocate the FPGA and run an interactive bash with the required tools and file permissions already set by slurm. To get information about the active slurm jobs, run:

squeue

The output should look similar to this:

JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
 1312 fpga          bash afilguer  R      17:14      1 xaloc

6.2.6.2. Loading bistreams

The FPGA bitstream needs to be loaded before the application can run. The load_bitstream utility is provided in order to simplify the FPGA configuration.

load_bitstream bitstream.bit [index]

The utility receives a second optional parameter to indicate which of the allocated FPGAs to program, the default behavior is to program all the allocated FPGAs with the bitstream.

To know which FPGAs indices have been allocated, run load_bitstream with the help (-h) option. The output should be similar to this:

Usage load_bitstream bitstream.bit [index]
Available devices:
index:  jtag           pcie          usb
0:      XXXXXXXXXXXXX  0000:02:00.0  002:002

6.2.6.3. Set up qdma queues

Note

This step is performed by load_bitstream script, which creates a single bidirectional memory mapped queue. This is only needed if other configuration is needed.

For DMA transfers to be performed between system main memory and the FPGA memory, qdma queues has to be set up by the user prior to any execution.

In this case dmactl tool is used. For instance: In order to create and start a memory mapped qdma queue with index 1 run:

dmactl qdma02000 q add idx 1 mode mm dir bi
dmactl qdma02000 q start idx 1 mode mm dir bi

OmpSs runtime system expects an mm queue at index 1, which can be created with the commands listed above.

In the same fashion, these queues can also be removed:

dmactl qdma02000 q stop idx 1 mode mm dir bi
dmactl qdma02000 q del idx 1 mode mm dir bi

For more information, see

dmactl --help

6.2.6.4. Get current bitstream info

In order to get information about the bitstream currently loaded into the FPGA, the tool read_bitinfo is installed in the system.

read_bitinfo

Note that an active slurm reservation is needed in order to query the FPGA.

This call should return something similar to the sample output for a matrix multiplication application:

Reading bitinfo of FPGA 0000:b3:00.0
Bitstream info version: 11
Number of acc:  8
AIT version:    7.1.0
Wrapper version 13
Board base frequency (Hz)   156250000
Interleaving stride 32768
Features:
[ ] Instrumentation
[ ] Hardware counter
[x] Performance interconnect
[ ] Simplified interconnection
[ ] POM AXI-Lite
[x] POM task creation
[x] POM dependencies
[ ] POM lock
[x] POM spawn queues
[ ] Power monitor (CMS) enabled
[ ] Thermal monitor (sysmon) enabled
Cmd In addr 0x2000 len 128
Cmd Out addr 0x4000 len 128
Spawn In addr 0x6000 len 1024
Spawn Out addr 0x8000 len 1024
Managed rstn addr 0xA000
Hardware counter addr 0x0
POM AXI-Lite addr 0x0
Power monitor (CMS) addr 0x0
Thermal monitor (sysmon) addr 0x0

xtasks accelerator config:
type        count   freq(KHz)   description
5839957875  1   300000      matmulFPGA
7602000973  7   300000      matmulBlock

ait command line:
ait --name=matmul --board=alveo_u200 -c=300 --memory_interleaving_stride=32K --simplify_interconnection --interconnect_opt=performance --interconnect_regslice=all --floorplanning_constr=all --slr_slices=all --placement_file=u200_placement_7x256.json --wrapper_version 13

Hardware runtime VLNV:
bsc:ompss:picosompssmanager:7.3

bitinfo note:
''

6.2.6.5. Remote debugging

Although it is possible to interact with Vivado’s Hardware Manager through ssh-based X forwarding, Vivado’s GUI might not be very responsive over remote connections. To avoid this limitation, one might connect a local Hardware Manager instance to targets hosted on Quar, completely avoiding X forwarding, as follows.

  1. On Xaloc, when allocating an FPGA with slurm, a Vivado HW server is automatically launched for each FPGA:
    • FPGA 0 uses port 3120
  2. On the local machine, assuming that Xaloc’s HW Server runs on port 3121, let all connections to port 3121 be forwarded to Xaloc by doing ssh -L 3121:xaloc:3121 [USER]@ssh.hca.bsc.es -p 8410 .
  3. Finally, from the local machine, connect to Xaloc’s hardware server:
    • Open Vivado’s Hardware Manager.
    • Launch the “Open target” wizard.
    • Establish a connection to the local HW server, which will be just a bridge to the remote instance.