.. index:: single: crdbmaster .. _crdbmaster-user_guide: crdbmaster user guide ===================== **crdbmaster** takes its name from *CRDB*, a compute node developed within the `EuroEXA project `__. The `OmpSs-2\@FPGA releases `__ are automatically installed in the server. They are available through a module file for each target architecture. This document describes how to load and use the modules to compile an example application. Once the modules are loaded, the workflow in the server should be the same as in the Docker images. General remarks --------------- * The OmpSs-2\@FPGA toolchain is installed in a version folder under the ``/opt/bsc/`` directory. * Third-party libraries required to run some programs are installed in the corresponding folder under the ``/opt/lib/`` directory. * The rest of the software (Xilinx toolchain, slurm, modules, etc.) is installed under the ``/tools/`` directory. Node specifications ------------------- * CPU: Intel Xeon E3-1220 CPU * https://www.intel.com/content/www/us/en/products/sku/52269/intel-xeon-processor-e31220-8m-cache-3-10-ghz/specifications.html * Main memory: 32GB DDR3-1600 * FPGAs: * Xilinx Kria KV260 * https://www.amd.com/en/products/system-on-modules/kria/k26/kv260-vision-starter-kit.html * Xilinx Zynq Ultrascale+ ZCU102 * https://www.amd.com/en/products/adaptive-socs-and-fpgas/evaluation-boards/ek-u1-zcu102-g.html * Xilinx Zynq 7000 ZC702 * https://www.amd.com/en/products/adaptive-socs-and-fpgas/evaluation-boards/ek-z7-zc702-g.html * Avnet/Digilent Zedboard (XC7Z7020) * https://www.avnet.com/americas/products/avnet-boards/avnet-board-families/zedboard/ .. _crdbmaster-login: Logging into the system ----------------------- crdbmaster login node is accessible via ssh at ``crdbmaster.bsc.es``. .. code-block:: text ssh crdbmaster.bsc.es .. _crdbmaster-modules: Module structure ---------------- The ompss-2 modules are: * ``ompss-2/arm64/*[release version]*`` This will automatically load the default Vivado version, although an arbitrary version can be loaded before ompss-2: .. code-block:: text module load vivado/2023.2 ompss-2/arm64/git To list all available modules in the system run: .. code-block:: text module avail Build applications ------------------ To generate an application binary and bitstream, you could refer to :ref:`compile-ompss2atfpga-programs` as the steps are general enough. Note that the appropriate modules need to be loaded. See :ref:`crdbmaster-modules`. .. _crdbmaster-running_applications: Running applications -------------------- .. _crdbmaster-access_fpga: Get access to an installed fpga ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The server uses Slurm in order to manage access to computation resources. Therefore, to be able to use the resources of an FPGA, an allocation in one of the partitions has to be made. You can check the number and name of partitions and nodes by running: .. code-block:: text sinfo -Nel There are 2 partitions in the node: * ``arm64``: kv260, zcu102 * ``arm32``: zedboard, zynq702 In order to make an allocation of computing resources, you must run ``srun``. For instance: .. code-block:: text srun -p arm32 --pty bash Or allocate a specific board with: .. code-block:: text srun -p arm64 --nodelist=zcu102 --pty bash These commands will allocate an FPGA and run an interactive bash inside the FPGA node with the required tools and file permissions already set by slurm. To get information about the active slurm jobs, run: .. code-block:: text squeue The output should look similar to this: .. code-block:: text JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 1312 arm32 bash afilguer R 17:14 1 zynq702 Loading bistreams ^^^^^^^^^^^^^^^^^ The FPGA bitstream needs to be loaded before the application can run. Xilinx provides the ``fpgautil`` utility in order to simplify bitstream loading. .. code-block:: text fpgautil -b bitstream.bin Get current bitstream info ^^^^^^^^^^^^^^^^^^^^^^^^^^ In order to get information about the bitstream currently loaded into the FPGA, the tool ``read_bitinfo`` is installed in the system. .. code-block:: text read_bitinfo Note that an active slurm reservation is needed in order to query the FPGA. This call should return something similar to the sample output for a matrix multiplication application: .. code-block:: text Bitinfo version: 16 Bitstream user-id: 0xDD3ABA12 AIT version: 8.2.0 Wrapper version: 13 Number of accelerators: 3 Board base frequency: 125.00 MHz Dedicated FPGA memory: not available Memory interleaving: not enabled Features: [ ] Instrumentation [ ] Hardware counter Interconnect optimization [ ] Area [x] Performance Picos OmpSs Manager [ ] AXI-Lite [x] Task creation [x] Dependencies [ ] Lock [x] Spawn queues [ ] Power monitor (CMS) [ ] Thermal monitor (sysmon) [ ] OMPIF [ ] IMP Address map: Managed rstn - address 0x80001000 CmdIn - address 0x80002000 length 256 CmdOut - address 0x80004000 length 256 SpawnIn - address 0x80006000 length 1024 SpawnOut - address 0x80008000 length 1024 Hardware counter - not enabled POM AXI-Lite - not enabled Power monitor (CMS) - not enabled Thermal monitor (sysmon) - not enabled xtasks accelerator config: type count freq(KHz) description 5839957875 1 100000 matmulFPGA 7602000973 2 100000 matmulBlock ait command line: ait --name=matmul --board=zynq702 -c=100 --interconnect_regslices --wrapper_version 13 Hardware runtime VLNV: bsc:ompss:picos_ompss_manager:7.5