6.4. crdbmaster cluster installation

The OmpSs@FPGA releases are automatically installed in the crdbmaster cluster. They are available through a module file for each target architecture. This document describes how to load and use the modules to compile an example application. Once the modules are loaded, the workflow in the crdbmaster cluster should be the same as in the Docker images.

6.4.1. General remarks

  • All software is installed in a version folder under the /opt/bsc directory.

  • During the updates, the installation will not be available for the users’ usage.

  • Usually, the installation just takes 20 minutes.

  • After the installation, an informative email will be sent.

6.4.2. System overview

Current setup consists of an x86 login node and a CRDB directly connected to it. Serial lines and jtag are connected to the login node, allowing node management as well as debug and programming.

6.4.2.1. CRDB

CRDB is a FPGA development board developed within the euroexa project.

It has two discrete devices, a Zynq Ultrascale XCZU9EG and a Virtex Ultrascale+ XCVU9P. Both devices are directly connected.

CRDB itself has 16GB system memory and 16x3 GB FPGA memory.

6.4.3. Logging into the system

Crdbmaster login node is accessible via ssh at crbmaster.bsc.es

6.4.4. Module structure

The ompss modules are:

  • ompss/arm64_fpga/[release version]

It requires having some vivado loaded:

module load vivado ompss/arm64_fpga/git

To list all available modules in the system run:

module avail

6.4.5. Build applications

To generate an application binary and bitstream, you could refer to Compile OmpSs@FPGA programs as the steps are general enough.

Note that the appropriate modules need to be loaded. See Module structure.

6.4.6. Running applications

6.4.6.1. Get access to an installed fpga

crdbmaster cluster uses SLURM in order to manage access to computation resources. Therefore, to be able to use the resources of an FPGA, an allocation in one of the partitions has to be made.

There is 1 partition in the cluster: * fpga: EuroEXA CRDB board

In order to make an allocation, you must run salloc:

salloc -p [partition]

For instance:

salloc -p fpga

Then get the node that has been allocated for you:

squeue

The output should look similar to this:

JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
 1312 fpga          bash afilguer  R      17:14      1 crdb

Then, you can log into your node:

ssh crdb

6.4.6.2. OmpSs Initialization

Before the application can be executed, the ZU9 environment has to be set up in order to run ompss applications, and the VU9 design must be loaded.

The following command loads the needed kernel modules and resets device to device communications. Note that the ZU9 design is already loaded upon Linux boot.

init_ompss-[ompss-fpga-release]-vu9-programming [vu9-bitstream.bit]

For instance, to initialize environment for 3.2.0 release, run:

init_ompss-3.2.0-vu9-programming mybitstream.bit

6.4.6.3. Loading bistreams

Since system has two FPGA devices, Devices can be independently. This is useful when reloading the bitstream after ompss initialization.

VU9 can be loaded using:

load_vu9 myNewBitstream.bit

Also, Zynq bitstream can be reloaded on runtime. However this is not needed during normal operation

load_zu9 myZynqBitstream.bit

6.4.6.4. CRDB cold reboot

6.4.6.4.1. Attach to serial ports

Board serial ports and debug port are only available to the user that has the active running job on the board. Also, attached minicom instances will be killed upon job end. Devices are /dev/ttyUSB0 for the management controller and /dev/ttyUSB2 for the processing system serial line.

minicom -D /dev/ttyUSB0     # management
minicom -D /dev/ttyUSB2     # serial line

6.4.6.4.2. Serial port settings

Minicom serial port settings in order to connect to the CRDBs Serial ports

A -    Serial Device      : /dev/ttyUSB0
B - Lockfile Location     : /var/lock
C -   Callin Program      :
D -  Callout Program      :
E -    Bps/Par/Bits       : 115200 8N1
F - Hardware Flow Control : No
G - Software Flow Control : No

In order to use these settings, this has to be saved into minicom’s config file.

Set ~/.minirc.dfl contents as follows:

pu port /dev/ttyUSB0
pu baudrate 115200
pu rtscts No
pu xonxoff No

6.4.6.4.3. Hard reboot

Once connected to the management serial line, you should see a menu like this: (You may need to press enter for it to refresh)

MainMenu
0) Interrupt config mcu
1) I2C Scan menu
2) I2C CMD menu (1 byte register)
3) I2C IO expander 1 menu
4) I2C IO expander 2 menu
5) I2C CMD menu (2 byte register)
6) Execute startup sequence eMMC
7) Execute startup sequence SD

Then select 3

i2cIOExpander1Menu
Polling: DISABLED
Pin: 00,    output enabled: 1,                   description: EN_5V
Pin: 01,    output enabled: 1,                   description: EN_3V3
Pin: 02,    output enabled: 1,                   description: EN_12V
Pin: 03,    output enabled: 0,                   description: nSYS_RESET
Pin: 04,    output enabled: 0,                   description: nCB_RESET
Pin: 05, interrupt enabled: 0, actual value: 0, description: -
Pin: 06, interrupt enabled: 0, actual value: 1, description: nTHRM
Pin: 07, interrupt enabled: 0, actual value: 1, description: nSMB_ALERT
Pin: 10, interrupt enabled: 0, actual value: 0, description: PERST
Pin: 11, interrupt enabled: 0, actual value: 1, description: PEWAKE
Pin: 12, interrupt enabled: 0, actual value: 1, description: nTHRM_TRIP
Pin: 13, interrupt enabled: 0, actual value: 1, description: PWR_OK
Pin: 14,    output enabled: 1,                   description: nPWRBTN
Pin: 15, interrupt enabled: 0, actual value: 0, description: nTYPE(0)
Pin: 16, interrupt enabled: 0, actual value: 1, description: nTYPE(1)
Pin: 17, interrupt enabled: 0, actual value: 0, description: nTYPE(2)
Enter [pin] to toggle the interrupt enabled/output state
P to toggle polling enable status
C to change input/output configuration
I to initialize
R to refresh

Then enter 02 twice to switch off and on the 12V rail

6.4.6.5. FPGA jtag debug

A vivado 2020.1 hw_server is running on the controller node so users can remotely access jtag interface.

This is done by specifying a remote target in vivado hardware manager: crdbmaster.bsc.es:3121

Users also can run a local vivado instance by running vivado in the master node and forwarding X graphics. However, this is not recommended as waveform visualization is graphics intensive and latency is not negligible.