6.3. xaloc user guide

xaloc takes its name from the Catalan form of Sirocco, which is the name of the Mediterranean wind that comes from the southeast.

The OmpSs-2@FPGA releases are automatically installed in the server. They are available through a module file for each target architecture. This document describes how to load and use the modules to compile an example application. Once the modules are loaded, the workflow in the server should be the same as in the Docker images.

6.3.1. General remarks

  • The OmpSs-2@FPGA toolchain is installed in a version folder under the /opt/bsc/ directory.

  • Third-party libraries required to run some programs are installed in the corresponding folder under the /opt/lib/ directory.

  • The rest of the software (Xilinx toolchain, slurm, modules, etc.) is installed under the /tools/ directory.

6.3.2. Node specifications

6.3.3. Logging into the system

xaloc is accessible from HCA ssh.hca.bsc.es Alternatively, it can be accessed through the 8410 port in HCA and ssh connection will be redirected to the actual host:

ssh -p 8410 ssh.hca.bsc.es

Also, this can be automated by adding a xaloc host into ssh config:

Host xaloc
    HostName ssh.hca.bsc.es
    Port 8410

6.3.4. Module structure

The ompss-2 modules are:

  • ompss-2/x86_64/*[release version]*

This will automatically load the default Vivado version, although an arbitrary version can be loaded before ompss-2:

module load vivado/2023.2 ompss-2/x86_64/git

To list all available modules in the system run:

module avail

6.3.5. Build applications

To generate an application binary and bitstream, you could refer to Compile OmpSs-2@FPGA programs as the steps are general enough.

Note that the appropriate modules need to be loaded. See Module structure.

6.3.6. Running applications

Warning

Although the Versal board is installed and can be allocated via Slurm there is no toolchain support yet.

Get access to an installed fpga

The server uses Slurm in order to manage access to computation resources. Therefore, to be able to use the resources of an FPGA, an allocation in one of the partitions has to be made.

You can check the number and name of partitions and nodes by running:

sinfo -Nel

There is 1 partition in the node:

  • fpga: versal

In order to make an allocation of computing resources, you must run salloc with the --gres option.

For instance:

salloc -p fpga --gres=fpga:BOARD:N

Where BOARD is the FPGA to allocate and N the number of FPGAs to allocate.

This command will allocate the number of specified FPGAs with the required tools and file permissions already set by slurm and prevent other users from using those resources.

Once inside an allocation you can run a script or an interactive job with a subset of the allocated resources with srun:

For an interactive job, run:

srun --gres=fpga:BOARD:N --pty bash

To execute a script, run:

srun --gres=fpga:BOARD:N script.sh

Note

You can also allocate and run a job in a single command with srun. There is no need to pre-allocate resources with salloc.

Warning

Just running an salloc will not set the OmpSs-2@FPGA environment variables. In order to do so, you must run your job through srun.

Alternatively you can also run your jobs asynchronously through an sbatch command, passing a slurm job script as argument:

sbatch --gres=fpga:BOARD:N job_script.sh

Being an example job_script.sh:

#!/bin/bash
#
#SBATCH --job-name=ompss-2_fpga_test
#SBATCH --output=out.txt
#SBATCH --time=05:00
#SBATCH --gres=fpga:BOARD:N
#SBATCH -p fpga

module load ompss-2/x86_64/git

cd test
make binary

srun --gres=fpga:BOARD:N exec_test.sh

To get information about the active slurm jobs, run:

squeue

The output should look similar to this:

JOBID   PARTITION   NAME    USER        ST  TIME    NODES   NODELIST(REASON)
1312    fpga        bash    afilguer    R   17:14   1       quar

To know which FPGAs have been allocated, you can run the report_slurm_node tool. The output should be similar to this:

LOCAL_ID  PCI_DEV       USB_DEV  QDMA_DEV  HWSERVER_PORT  GLOBAL_ID
0         0000:02:00.0  002:002  02000     13330          0