6.2. crdbmaster user guide

crdbmaster takes its name from CRDB, a compute node developed within the EuroEXA project.

The OmpSs-2@FPGA releases are automatically installed in the server. They are available through a module file for each target architecture. This document describes how to load and use the modules to compile an example application. Once the modules are loaded, the workflow in the server should be the same as in the Docker images.

6.2.1. General remarks

  • The OmpSs-2@FPGA toolchain is installed in a version folder under the /opt/bsc/ directory.

  • Third-party libraries required to run some programs are installed in the corresponding folder under the /opt/lib/ directory.

  • The rest of the software (Xilinx toolchain, slurm, modules, etc.) is installed under the /tools/ directory.

6.2.2. Node specifications

6.2.3. Logging into the system

crdbmaster login node is accessible via ssh at crdbmaster.bsc.es.

ssh crdbmaster.bsc.es

6.2.4. Module structure

The ompss-2 modules are:

  • ompss-2/arm64/*[release version]*

This will automatically load the default Vivado version, although an arbitrary version can be loaded before ompss-2:

module load vivado/2023.2 ompss-2/arm64/git

To list all available modules in the system run:

module avail

6.2.5. Build applications

To generate an application binary and bitstream, you could refer to Compile OmpSs-2@FPGA programs as the steps are general enough.

Note that the appropriate modules need to be loaded. See Module structure.

6.2.6. Running applications

Get access to an installed fpga

The server uses Slurm in order to manage access to computation resources. Therefore, to be able to use the resources of an FPGA, an allocation in one of the partitions has to be made.

You can check the number and name of partitions and nodes by running:

sinfo -Nel

There are 2 partitions in the node:

  • arm64: kv260, zcu102

  • arm32: zedboard, zynq702

In order to make an allocation of computing resources, you must run srun.

For instance:

srun -p arm32 --pty bash

Or allocate a specific board with:

srun -p arm64 --nodelist=zcu102 --pty bash

These commands will allocate an FPGA and run an interactive bash inside the FPGA node with the required tools and file permissions already set by slurm.

To get information about the active slurm jobs, run:

squeue

The output should look similar to this:

JOBID   PARTITION   NAME    USER        ST  TIME    NODES   NODELIST(REASON)
1312    arm32       bash    afilguer    R   17:14   1       zynq702

Loading bistreams

The FPGA bitstream needs to be loaded before the application can run. Xilinx provides the fpgautil utility in order to simplify bitstream loading.

fpgautil -b bitstream.bin

Get current bitstream info

In order to get information about the bitstream currently loaded into the FPGA, the tool read_bitinfo is installed in the system.

read_bitinfo

Note that an active slurm reservation is needed in order to query the FPGA.

This call should return something similar to the sample output for a matrix multiplication application:

Bitinfo version: 16
Bitstream user-id: 0xDD3ABA12
AIT version: 8.2.0
Wrapper version: 13
Number of accelerators: 3
Board base frequency: 125.00 MHz
Dedicated FPGA memory: not available
Memory interleaving: not enabled

Features:
[ ] Instrumentation
[ ] Hardware counter
Interconnect optimization
    [ ] Area
    [x] Performance
Picos OmpSs Manager
    [ ] AXI-Lite
    [x] Task creation
    [x] Dependencies
    [ ] Lock
    [x] Spawn queues
[ ] Power monitor (CMS)
[ ] Thermal monitor (sysmon)
[ ] OMPIF
[ ] IMP

Address map:
Managed rstn - address 0x80001000
CmdIn - address 0x80002000 length 256
CmdOut - address 0x80004000 length 256
SpawnIn - address 0x80006000 length 1024
SpawnOut - address 0x80008000 length 1024
Hardware counter - not enabled
POM AXI-Lite - not enabled
Power monitor (CMS) - not enabled
Thermal monitor (sysmon) - not enabled

xtasks accelerator config:
type        count   freq(KHz)   description
5839957875  1   100000      matmulFPGA
7602000973  2   100000      matmulBlock

ait command line:
ait --name=matmul --board=zynq702 -c=100 --interconnect_regslices --wrapper_version 13

Hardware runtime VLNV:
bsc:ompss:picos_ompss_manager:7.5