.. index:: single: crdbmaster crdbmaster cluster installation =============================== .. _installation_crdbmaster: The `OmpSs@FPGA releases `__ are automatically installed in the crdbmaster cluster. They are available through a module file for each target architecture. This document describes how to load and use the modules to compile an example application. Once the modules are loaded, the workflow in the crdbmaster cluster should be the same as in the Docker images. General remarks --------------- * All software is installed in a version folder under the ``/opt/bsc`` directory. * During the updates, the installation will not be available for the users' usage. * Usually, the installation just takes 20 minutes. * After the installation, an informative email will be sent. .. _crdbmaster-overview: System overview --------------- Current setup consists of an x86 login node and a CRDB directly connected to it. Serial lines and jtag are connected to the login node, allowing node management as well as debug and programming. CRDB ^^^^ CRDB is a FPGA development board developed within the euroexa project. It has two discrete devices, a Zynq Ultrascale XCZU9EG and a Virtex Ultrascale+ XCVU9P. Both devices are directly connected. CRDB itself has 16GB system memory and 16x3 GB FPGA memory. .. _crdbmaster-login: Logging into the system ----------------------- Crdbmaster login node is accessible via ssh at ``crbmaster.bsc.es`` .. _crdbmaster-modules: Module structure ---------------- The ompss modules are: * ompss/arm64\_fpga/*[release version]* It requires having some vivado loaded: .. code:: bash module load vivado ompss/arm64_fpga/git To list all available modules in the system run: .. code:: bash module avail Build applications ------------------ To generate an application binary and bitstream, you could refer to :ref:`compile-ompssatfpga-programs` as the steps are general enough. Note that the appropriate modules need to be loaded. See :ref:`crdbmaster-modules`. .. _crdbmaster-running_applications: Running applications -------------------- .. _crdbmaster-access_fpga: Get access to an installed fpga ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ crdbmaster cluster uses SLURM in order to manage access to computation resources. Therefore, to be able to use the resources of an FPGA, an allocation in one of the partitions has to be made. There is 1 partition in the cluster: \* ``fpga``: EuroEXA CRDB board In order to make an allocation, you must run ``salloc``: .. code:: bash salloc -p [partition] For instance: .. code:: bash salloc -p fpga Then get the node that has been allocated for you: :: squeue The output should look similar to this: :: JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 1312 fpga bash afilguer R 17:14 1 crdb Then, you can log into your node: :: ssh crdb OmpSs Initialization ^^^^^^^^^^^^^^^^^^^^ Before the application can be executed, the ZU9 environment has to be set up in order to run ompss applications, and the VU9 design must be loaded. The following command loads the needed kernel modules and resets device to device communications. Note that the ZU9 design is already loaded upon Linux boot. .. code:: bash init_ompss-[ompss-fpga-release]-vu9-programming [vu9-bitstream.bit] For instance, to initialize environment for 3.2.0 release, run: .. code:: bash init_ompss-3.2.0-vu9-programming mybitstream.bit Loading bistreams ^^^^^^^^^^^^^^^^^ Since system has two FPGA devices, Devices can be independently. This is useful when reloading the bitstream after ompss initialization. VU9 can be loaded using: .. code:: bash load_vu9 myNewBitstream.bit Also, Zynq bitstream can be reloaded on runtime. However **this is not needed during normal operation** .. code:: bash load_zu9 myZynqBitstream.bit CRDB cold reboot ^^^^^^^^^^^^^^^^ Attach to serial ports ~~~~~~~~~~~~~~~~~~~~~~ Board serial ports and debug port are only available to the user that has the active running job on the board. Also, attached minicom instances will be killed upon job end. Devices are ``/dev/ttyUSB0`` for the management controller and ``/dev/ttyUSB2`` for the processing system serial line. .. code:: bash minicom -D /dev/ttyUSB0 # management minicom -D /dev/ttyUSB2 # serial line Serial port settings ~~~~~~~~~~~~~~~~~~~~ Minicom serial port settings in order to connect to the CRDBs Serial ports :: A - Serial Device : /dev/ttyUSB0 B - Lockfile Location : /var/lock C - Callin Program : D - Callout Program : E - Bps/Par/Bits : 115200 8N1 F - Hardware Flow Control : No G - Software Flow Control : No In order to use these settings, this has to be saved into minicom’s config file. Set ~/.minirc.dfl contents as follows: :: pu port /dev/ttyUSB0 pu baudrate 115200 pu rtscts No pu xonxoff No Hard reboot ~~~~~~~~~~~ Once connected to the management serial line, you should see a menu like this: (You may need to press enter for it to refresh) :: MainMenu 0) Interrupt config mcu 1) I2C Scan menu 2) I2C CMD menu (1 byte register) 3) I2C IO expander 1 menu 4) I2C IO expander 2 menu 5) I2C CMD menu (2 byte register) 6) Execute startup sequence eMMC 7) Execute startup sequence SD Then select 3 :: i2cIOExpander1Menu Polling: DISABLED Pin: 00, output enabled: 1, description: EN_5V Pin: 01, output enabled: 1, description: EN_3V3 Pin: 02, output enabled: 1, description: EN_12V Pin: 03, output enabled: 0, description: nSYS_RESET Pin: 04, output enabled: 0, description: nCB_RESET Pin: 05, interrupt enabled: 0, actual value: 0, description: - Pin: 06, interrupt enabled: 0, actual value: 1, description: nTHRM Pin: 07, interrupt enabled: 0, actual value: 1, description: nSMB_ALERT Pin: 10, interrupt enabled: 0, actual value: 0, description: PERST Pin: 11, interrupt enabled: 0, actual value: 1, description: PEWAKE Pin: 12, interrupt enabled: 0, actual value: 1, description: nTHRM_TRIP Pin: 13, interrupt enabled: 0, actual value: 1, description: PWR_OK Pin: 14, output enabled: 1, description: nPWRBTN Pin: 15, interrupt enabled: 0, actual value: 0, description: nTYPE(0) Pin: 16, interrupt enabled: 0, actual value: 1, description: nTYPE(1) Pin: 17, interrupt enabled: 0, actual value: 0, description: nTYPE(2) Enter [pin] to toggle the interrupt enabled/output state P to toggle polling enable status C to change input/output configuration I to initialize R to refresh Then enter ``02`` twice to switch off and on the 12V rail FPGA jtag debug ^^^^^^^^^^^^^^^ A vivado 2020.1 hw_server is running on the controller node so users can remotely access jtag interface. This is done by specifying a remote target in vivado hardware manager: ``crdbmaster.bsc.es:3121`` Users also can run a local vivado instance by running vivado in the master node and forwarding X graphics. However, this is not recommended as waveform visualization is graphics intensive and latency is not negligible.