Improved creation timestamp consistency

[bieryAtFnal]

From the commit message: Reordered the creation of a unique, timestamp-based substring to put in data file names and the closing of an existing open file so that the timestamp in the filename and the timestamp in the creation_timestamp HDF5 file Attribute are created more closely in time, and therefore are more likely to agree. This is in HDF5DataStore::open_file_if_needed().

Description

While running integtests on np04-srv-029, I noticed that I would occasionally see complaints about a mismatch between the timestamp that was part of the raw data filename and the creation-timestamp that is stored in an HDF5 Attribute inside the file. Most of the time, these two timestamps agree quite well. The situation that seemed to trigger the problem was writing the output files to a (probably slow) NFS-mounted disk instead of a (much faster) locally-mounted disk.

When I looked into the problem, I saw that the timestamp that goes into the filename for raw data file N+1 was being created before file N was closed. And, a delay in the closing of file N could cause the timestamp for file N+1 to become a few seconds stale.

The fix was to reorder the code, and that is what is in this PR.

The suggested steps for reproducing the problem are the following, and they will work most reliably if they are run on np04-srv-029.

DATE_PREFIX=`date '+%d%b'`
TIME_SUFFIX=`date '+%H%M'`

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-create -n NFD_DEV_260219_A9 ${DATE_PREFIX}FDDevTest_${TIME_SUFFIX}
cd ${DATE_PREFIX}FDDevTest_${TIME_SUFFIX}/sourcecode

git clone https://github.com/DUNE-DAQ/daqsystemtest.git -b develop
git clone https://github.com/DUNE-DAQ/dfmodules.git -b develop
cd ..

dbt-workarea-env

cd pythoncode
git clone https://github.com/DUNE-DAQ/integrationtest.git -b develop
cd integrationtest; git checkout kbiery/support_filenames_without_timestamps; cd ..
pip install integrationtest/.
cd ..

dbt-build -j 12
dbt-workarea-env

mkdir -p temp_pytest_area
dunedaq_integtest_bundle.sh -k long --tmpdir `pwd`/temp_pytest_area

echo ""
echo -e "\U1F535 \U2705 Note the complaint about creation time mismatch. \U2705 \U1F535"
echo ""
echo ""
sleep 3

cd sourcecode/dfmodules
git checkout kbiery/improved_creation_timestamp_consistency
cd ../../

dbt-build -j 12
dbt-workarea-env

dunedaq_integtest_bundle.sh -k long --tmpdir `pwd`/temp_pytest_area

echo ""
echo -e "\U1F535 \U2705 Note that the timestamps now match, with the new dfmodules code. \U2705 \U1F535"
echo ""
echo ""

Type of change

• Bug fix (non-breaking change which fixes an issue)

Testing checklist

• Full set of integration tests pass (dunedaq_integtest_bundle.sh)

Further checks

• Code is commented where needed, particularly in hard-to-understand areas