6.4. crdbmaster cluster installation¶
The OmpSs@FPGA releases are automatically installed in the crdbmaster cluster. They are available through a module file for each target architecture. This document describes how to load and use the modules to compile an example application. Once the modules are loaded, the workflow in the crdbmaster cluster should be the same as in the Docker images.
6.4.1. General remarks¶
- All software is installed in a version folder under the
/opt/bsc
directory. - During the updates, the installation will not be available for the users’ usage.
- Usually, the installation just takes 20 minutes.
- After the installation, an informative email will be sent.
6.4.2. System overview¶
Current setup consists of an x86 login node and a CRDB directly connected to it. Serial lines and jtag are connected to the login node, allowing node management as well as debug and programming.
6.4.2.1. CRDB¶
CRDB is a FPGA development board developed within the euroexa project.
It has two discrete devices, a Zynq Ultrascale XCZU9EG and a Virtex Ultrascale+ XCVU9P. Both devices are directly connected.
CRDB itself has 16GB system memory and 16x3 GB FPGA memory.
6.4.3. Logging into the system¶
Crdbmaster login node is accessible via ssh at crbmaster.bsc.es
6.4.4. Module structure¶
The ompss modules are:
- ompss/arm64_fpga/[release version]
It requires having some vivado loaded:
module load vivado ompss/arm64_fpga/git
To list all available modules in the system run:
module avail
6.4.5. Build applications¶
To generate an application binary and bitstream, you could refer to compile-ompssatfpga-programs as the steps are general enough.
Note that the appropriate modules need to be loaded. See Module structure.
6.4.6. Running applications¶
6.4.6.1. Get access to an installed fpga¶
crdbmaster cluster uses SLURM in order to manage access to computation resources. Therefore, to be able to use the resources of an FPGA, an allocation in one of the partitions has to be made.
There is 1 partition in the cluster: * fpga
: EuroEXA CRDB board
In order to make an allocation, you must run salloc
:
salloc -p [partition]
For instance:
salloc -p fpga
Then get the node that has been allocated for you:
squeue
The output should look similar to this:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1312 fpga bash afilguer R 17:14 1 crdb
Then, you can log into your node:
ssh crdb
6.4.6.2. OmpSs Initialization¶
Before the application can be executed, the ZU9 environment has to be set up in order to run ompss applications, and the VU9 design must be loaded.
The following command loads the needed kernel modules and resets device to device communications. Note that the ZU9 design is already loaded upon Linux boot.
init_ompss-[ompss-fpga-release]-vu9-programming [vu9-bitstream.bit]
For instance, to initialize environment for 3.2.0 release, run:
init_ompss-3.2.0-vu9-programming mybitstream.bit
6.4.6.3. Loading bistreams¶
Since system has two FPGA devices, Devices can be independently. This is useful when reloading the bitstream after ompss initialization.
VU9 can be loaded using:
load_vu9 myNewBitstream.bit
Also, Zynq bitstream can be reloaded on runtime. However this is not needed during normal operation
load_zu9 myZynqBitstream.bit
6.4.6.4. CRDB cold reboot¶
6.4.6.4.1. Attach to serial ports¶
Board serial ports and debug port are only available
to the user that has the active running job on the board.
Also, attached minicom instances will be killed upon job end.
Devices are /dev/ttyUSB0
for the management controller and /dev/ttyUSB2
for the processing system serial line.
minicom -D /dev/ttyUSB0 # management
minicom -D /dev/ttyUSB2 # serial line
6.4.6.4.2. Serial port settings¶
Minicom serial port settings in order to connect to the CRDBs Serial ports
A - Serial Device : /dev/ttyUSB0
B - Lockfile Location : /var/lock
C - Callin Program :
D - Callout Program :
E - Bps/Par/Bits : 115200 8N1
F - Hardware Flow Control : No
G - Software Flow Control : No
In order to use these settings, this has to be saved into minicom’s config file.
Set ~/.minirc.dfl contents as follows:
pu port /dev/ttyUSB0
pu baudrate 115200
pu rtscts No
pu xonxoff No
6.4.6.4.3. Hard reboot¶
Once connected to the management serial line, you should see a menu like this: (You may need to press enter for it to refresh)
MainMenu
0) Interrupt config mcu
1) I2C Scan menu
2) I2C CMD menu (1 byte register)
3) I2C IO expander 1 menu
4) I2C IO expander 2 menu
5) I2C CMD menu (2 byte register)
6) Execute startup sequence eMMC
7) Execute startup sequence SD
Then select 3
i2cIOExpander1Menu
Polling: DISABLED
Pin: 00, output enabled: 1, description: EN_5V
Pin: 01, output enabled: 1, description: EN_3V3
Pin: 02, output enabled: 1, description: EN_12V
Pin: 03, output enabled: 0, description: nSYS_RESET
Pin: 04, output enabled: 0, description: nCB_RESET
Pin: 05, interrupt enabled: 0, actual value: 0, description: -
Pin: 06, interrupt enabled: 0, actual value: 1, description: nTHRM
Pin: 07, interrupt enabled: 0, actual value: 1, description: nSMB_ALERT
Pin: 10, interrupt enabled: 0, actual value: 0, description: PERST
Pin: 11, interrupt enabled: 0, actual value: 1, description: PEWAKE
Pin: 12, interrupt enabled: 0, actual value: 1, description: nTHRM_TRIP
Pin: 13, interrupt enabled: 0, actual value: 1, description: PWR_OK
Pin: 14, output enabled: 1, description: nPWRBTN
Pin: 15, interrupt enabled: 0, actual value: 0, description: nTYPE(0)
Pin: 16, interrupt enabled: 0, actual value: 1, description: nTYPE(1)
Pin: 17, interrupt enabled: 0, actual value: 0, description: nTYPE(2)
Enter [pin] to toggle the interrupt enabled/output state
P to toggle polling enable status
C to change input/output configuration
I to initialize
R to refresh
Then enter 02
twice to switch off and on the 12V rail
6.4.6.5. FPGA jtag debug¶
A vivado 2020.1 hw_server is running on the controller node so users can remotely access jtag interface.
This is done by specifying a remote target in vivado hardware manager:
crdbmaster.bsc.es:3121
Users also can run a local vivado instance by running vivado in the master node and forwarding X graphics. However, this is not recommended as waveform visualization is graphics intensive and latency is not negligible.