Write PRV file in blocks, show progress
Currently writing the PRV file can be very slow for a large dataset. The writing is done in two phases, first putting all events ascii-encoded in a variable, then writing to file. Also, there is no progress reporting with this implementation.
Some improvement suggestions are:
-
Fill variable in smaller size chunks/blocks and write after -
Pipeline the encoding in ascii and the writing to file with multithreading -
Compute a progress percentage based on number of blocks to be written -
Find more system-aware ways to encode the events into ascii and then write