How to keep HLS intermediate files generated by Mercurium?

Mercurium generates an intermediate C++ HLS source file for each FPGA task defined in the source code. Those files are used by AIT to generate the FPGA bitstream, and removed after the invocation. To keep the Mercurium intermediate files, there is the -k option. It will keep the C/C++ intermediate files for the native compiler and the C++ HLS files for AIT. Usage example:

fpgacc --ompss --bitstream-generation -k src/dotproduct.c -o dotproduct \
  --Wf,--board=zedboard

Note

HLS source files are not generated if --bitstream-generation option is not used in Mercurium call

Edit the HLS intermediate files and call AIT

It is not recommended, but under some circumstances one would need to edit the HLS intermediate files before the AIT call. The following steps shows how to do so.

1. We call Mercurium with the desired options and two extra flags: - -k which keeps the intermediate files generated by the compiler. - --verbose which enables the verbose mode of the compiler. This is needed to gather the AIT command call that Mercurium executes.

   We could add an unsupported argument in the AIT flags to make it abort, as we do not want the bitstream before the modifications. For example, we could add the --abort flag to --Wf option of Mercurium. Example of Mercurium call and generated output:

   [user@machine] $ fpgacc --ompss -k --verbose --bitstream-generation --Wf,-b=zcu102,--hwruntime=som,--abort foo.c -o foo
   Loading compiler phases for profile 'fpgacc'
   Compiler phases for profile 'fpgacc' loaded in 0.02 seconds
   Compiling file 'foo.c'
   gcc -E -D_OPENMP=200805 -D_OMPSS=1 -I/home/user/opt/ompss/git/nanox/include/nanox -include nanos.h -include nanos_omp.h -include nanos-fpga.h -std=gnu99 -D_MCC -D_MERCURIUM -o /tmp/fpgacc_zHMisA foo.c
   File 'foo.c' preprocessed in 0.01 seconds
   Parsing file 'foo.c' ('/tmp/fpgacc_zHMisA')
   File 'foo.c' ('/tmp/fpgacc_zHMisA') parsed in 0.02 seconds
   Nanos++ prerun
   Early compiler phases pipeline executed in 0.00 seconds
   File 'foo.c' ('/tmp/fpgacc_zHMisA') semantically analyzed in 0.01 seconds
   Checking parse tree consistency
   Parse tree consistency verified in 0.00 seconds
   Freeing parse tree
   Parse tree freed in 0.00 seconds
   Checking integrity of nodecl tree
   Nodecl integrity verified in 0.00 seconds
   foo.c:1:13: info: unless 'no_copy_deps' is specified, the default in OmpSs is now 'copy_deps'
   foo.c:1:13: info: this diagnostic is only shown for the first task found
   foo.c:2:13: info: adding task function 'foo' for device 'fpga'
   Nanos++ phase
   foo.c:10:3: info: call to task function 'foo'
   foo.c:2:13: info: task function declared here
   FPGA bitstream generation phase analysis - ON
   Compiler phases pipeline executed in 0.01 seconds
   Prettyprinted into file 'fpgacc_foo.c' in 0.00 seconds
   Performing native compilation of 'fpgacc_foo.c' into 'foo.o'
   gcc -std=gnu99 -c -o foo.o fpgacc_foo.c
   File 'foo.c' ('fpgacc_foo.c') natively compiled in 0.02 seconds
   objdump -w -h foo.o 1> /tmp/fpgacc_d9sKiY
   gcc -o foo -std=gnu99 foo.o -Xlinker --enable-new-dtags -L/home/user/opt/ompss/git/nanox/lib/performance -Xlinker -rpath -Xlinker /home/user/opt/ompss/git/nanox/lib/performance -Xlinker --no-as-needed -lnanox-ompss -lnanox-c -lnanox -lpthread -lrt -lnanox-fpga-api
   Link performed in 0.04 seconds
   ait -b=zcu102 --hwruntime=som --abort --wrapper_version=7 -n=foo
   usage: ait -b BOARD -n NAME
   ait error: unrecognized arguments: --abort. Try 'ait -h' for more information.
   Link fpga failed
   Removing temporary filename '/tmp/fpgacc_d9sKiY'
   Removing temporary filename 'foo.o'
   Removing temporary filename '/tmp/fpgacc_zHMisA'
   

2. Now the intermediate files are available and we could edit them as desired. In the example, 7288177970:1:foo_hls_automatic_mcxx.cpp is available:

   [user@machine] $ ls -l
   total 44
   -rw-r--r-- 1 user user  3735 Jun  5 10:27 7288177970:1:foo_hls_automatic_mcxx.cpp
   -rwxr-xr-x 1 user user 17568 Jun  5 10:27 foo
   -rw-r--r-- 1 user user   235 Jun  4 15:17 foo.c
   -rw-r--r-- 1 user user 13460 Jun  5 10:27 fpgacc_foo.c
   

3. After modifying the HLS intermediate files, the AIT call that Mercurium usually performs has to be executed. In the verbose output of the first step, there is one line with the invoked call. This call has to be repeated, removing the extra options added to make AIT abort (if any). In the example, the --abort option has to be removed and the AIT command to invoke will be:

   [user@machine] $ ait -b=zcu102 --hwruntime=som --wrapper_version=7 -n=foo
   Using xilinx backend
   Checking vendor support for selected board
   ...