.. index::
   Run OMPIF applications

.. _run-ompif-applications:

Running OMPIF applications
::::::::::::::::::::::::::

Multi-node multi-fpga applications developed using the :ref:`ompif_api`
need special setup that is not needed in regular OmpSs\@FPGA applications.

Install cluster scripts
=======================

This includes loading bitstreams into the allocated FPGAs and setting up
routing tables.

This tasks are automated for the meep machine by a series of scripts:

::

   https://pm.bsc.es/gitlab/ompss-at-fpga/rtl/meep-ompss-fpga-cluster.git

This path is referred as ``$MEEP_SCRIPTS`` throughout the document.

Application execution
=====================

This section covers cluster environment setup and application execution.
It is assumed that ``vivado`` and ``xtasks_server`` are available in the
path. The ``xtasks_server`` binary is in the ``xtasks`` installation
path ``${XTASKS_INSTALL}/bin``.

If application is to be run using multiple FPGAs in the same node,
`Start xtasks servers <#start-xtasks-servers>`__ can be skipped.

Creating cluster description file
---------------------------------

A json file describing the cluster needs to be created for the cluster
to be automatically configured. It contains, for each FPGA, its index
inside the node, the node index and the bitstream:

.. code:: js

   [
       { "fpga": FPGA_INDEX, "node": NODE_INDEX, "bitstream": BITSTREAM_PATH }
   ]

The following example configures a cluster using 4 FPGAs in 2 different
nodes (2 FPGAs each node):

.. code:: json

   [
       { "node": 1, "fpga": 1, "bitstream" : "bitstream.bit" },
       { "node": 1, "fpga": 2, "bitstream" : "bitstream.bit" },
       { "node": 2, "fpga": 1, "bitstream" : "bitstream.bit" },
       { "node": 2, "fpga": 2, "bitstream" : "bitstream.bit" }
   ]

Information regarding FPGAs, can be found in ``/etc/motd`` in each of
the FPGA nodes.

Configuring the FPGA cluster
----------------------------

Once the cluster file is created, the cluster can be configured using
the ``create_cluster.py`` script from the ``meep-ompss-fpga-cluster``
repo:

.. code-block:: text

   python3 $MEEP_SCRIPTS/scripts/create_cluster.py cluster.json

If there are FPGAs on remote nodes, the script will automatically launch
servers and connect to them. For each node, a log file
``${NODENAME}_fpga_mng.log`` is created in the home directory. You can
change the path with the ``--log_prefix`` flag. Also, for each node the
script creates the ``xtasks_devs_$(hostname).sh`` file, with the
``XTASKS_PCI_DEV`` and ``XDMA_QDMA_DEV`` environment variables. By
default it is created in the current working directory, but it can be
changed with the ``--xtasks_cluster_prefix`` flag. The use of this file
is explained in `Start xtasks servers <#start-xtasks-servers>`__. The
script also creates the ``xtasks.cluster`` file needed by your
application in the current working directory. This file must be in the
same directory where you launch the application, or also you can set the
path in the ``XTASKS_CLUSTER_FILE`` environment variable. If not, the
application will assume you are executing in single-node mode, and will
not connect to the remote servers.

Start xtasks servers
--------------------

A remote server that listens for FPGA tasks needs to be started in each
of the remote nodes:

.. code-block:: text

   python3 $MEEP_SCRIPTS/scripts/launch_servers.py cluster.json --xtasks

Log files are created in the home directory named
``${NODENAME}_xtasks.log``. You can change the path with the
``--log_prefix`` flag. The script launches the server in the current
working directory, so you must have the ``xtasks.cluster`` file in the
same directory. For the moment, it can’t be set with an environment
variable. Also, by default all ``xtasks_devs_$(hostname).sh`` files must
be in the current directory as well. However, you can set another path
with the ``--script_prefix`` flag. Then, the cluster application can be
run as usual.

Debugging
---------

There are many debug registers that can be read with QDMA, including the
number of received messages, number of corrupted messages, number of
send/receive tasks, etc. More details in
:ref:`pom_axi_lite`.