.. index::
   Ovni FPGA instrumentation

.. _ovni_instrumentation:

Ovni FPGA instrumentation
:::::::::::::::::::::::::

FPGA accelerator instrumentation is provided via ovni.
By default, dependency copies and kernel execution are traced
when instrumentation is enabled.

Prerequisites
-------------

The bitstream needs to be compiled with instrumentation support
to create trace accelerator events.
Instrumentation can be enabled by adding the ``-fompss-fpga-instrumentation`` flag
when building the bitstream.
More details on building OmpSs\@FPGA applications are available in the :ref:`compile-ompss2atfpga-programs` section.

Also, `ovni <https://github.com/bsc-pm-ompss-at-fpga/ovni>`_ and `paraver <https://tools.bsc.es/paraver>`_
need to be installed to create and visualize traces.
Ovni is automatically installed and updated in cluster installations.

Running the application
-----------------------

Enabling instrumentation
........................

Instrumentation needs to be enabled in nanos6 configuration toml file.
The default file is located in the nanos6 installation directory.

.. code-block:: text

   $OMPSS_FPGA_HOME/bsc/x86_64/ompss-2/<release>/nanos6/share/nanos6.toml

``$OMPSS_FPGA_HOME`` is defined by the OmpSs\@FPGA environment module.
The ``release`` is the specific release that is being used,
``3.2.1`` or ``git``, for instance.

This file can be copied to the working directory to edit it and
override default nanos6 settings.
For instance, to copy the configuration file from the git release, run:

.. code-block:: text

   cp $OMPSS_FPGA_HOME/bsc/x86_64/ompss-2/git/nanos6/share/nanos6.toml .

Then, set the instrument entry to ``ovni``:

.. code:: toml

   [version]
       instrument = "ovni"

Then run the application as usual.
See :ref:`running-ompss2atfpga-programs` for more details.

.. note::
   Using an instrumentation-enabled bitstream
   without enabling instrumentation at the runtime level
   will result in the application hanging.

After the program finishes, an ``ovni`` directory containing ovni traces
should be created.

Processing traces
-----------------

Ovni traces need to be converted to paraver traces to be visualized using paraver.
Paraver traces need to be generated from ovni traces for visualization.
This is done using ``ovniemu`` tool:

.. code-block:: text

   ovniemu -x myapp.xtasks.config ovni/

In this example, ``myapp.xtasks.config`` is passed to ``ovniemu``
(using ``-x`` flag) to the tool is able to read accelerator names
for properly displaying them.

The output from the emulation process should look like this:

.. code-block:: text

   ovniemu: INFO: loaded 16 streams
   ovniemu: INFO: sorting looms by name
   ovniemu: INFO: loaded 1 looms, 1 processes, 16 threads and 8 cpus
   ovniemu: INFO: generated with libovni version 1.10.0 commit unknown
   ovniemu: INFO: the following 3 models are enabled:
   ovniemu: INFO:    nanos6 1.1.0 '6' (67 events)
   ovniemu: INFO:      ovni 1.1.0 'O' (18 events)
   ovniemu: INFO:    xtasks 1.0.0 'X' (1 events)
   ovniemu: INFO: emulation starts
   ovniemu: INFO: loom.fpgan10.770230 burst stats: median/avg/max =  98/102/361 ns
   ovniemu: INFO: 100.0% done at avg 1240 kev/s
   ovniemu: INFO: processed 1446408 input events in 1.17 s
   ovniemu: INFO: writing traces to disk, please wait
   ovniemu: INFO: emulation finished ok

In the ``ovni/`` directory, two paraver traces should have been created as well as
some paraver config files:

.. code-block:: text

   cfg/                     Paraver config files
   cpu.pcf                  CPU paraver trace files
   cpu.prv
   cpu.row
   loom.fpgan10.770230/     Ovni trace directory
   thread.pcf               Thread trace files
   thread.prv
   thread.row

FPGA events are emitted to the ``thread`` trace.
See also `Paraver web page <https://tools.bsc.es/paraver>`_ for further info
on the visualization tool and download links.