Once the test annotation files and the reference heart rate annotation files have been obtained, the remainder of the evaluation procedure is straightforward. All of the information needed to characterize the analysis performed by the device under test is encoded in the test annotation files; similarly, all of the information needed to characterize the actual contents of the test signals is encoded in the reference annotation and reference heart rate annotation files. The evaluation procedure thus entails comparison of the test and reference annotation files for each record.
Four programs are provided in the DB Software Package for this purpose:
The DB Software Package also includes three related programs:
To obtain a concise summary of how to use any of these programs, including a list of any command-line options, simply run the program without any command-line arguments. Refer to the ECG Database Applications Guide, which accompanies the DB Software Package, for details.
In most cases, it will be easiest to collect all of the annotation files before beginning the comparison, and then to perform the comparison by typing:
ecgevalThe program asks for the test annotator name, the names of the databases used for testing, and what optional detector outputs should be evaluated.
Only the statistics required by ECAR and EC38 are reported by ecgeval. If more detailed evaluation data are needed, it will be necessary to run bxb, rxr, etc., separately. If file space is extremely limited, it may be necessary to delete each test annotation file after it has been compared against the reference file, before the next test annotation file can be created; in such cases, it may also be necessary to prompt the user to change media containing signal or reference annotation files, or to reset the device under test before beginning each record. Optionally, ecgeval can generate a script (batch) file of commands, which can be edited to accommodate special requirements such as these.
For example, suppose we have obtained a set of test annotation files with the
annotator name ``yow'', which we wish to compare against the reference
annotation files (annotator name ``atr'')
and reference heart rate annotation files (annotator name ``htr'').
The portion of the evaluation script generated by ecgeval for MIT DB
record 100 is:
bxb -r 100 -a atr yow -L bxb.out sd.out
rxr -r 100 -a atr yow -L vruns.out sruns.out
mxm -r 100 -a htr yow -L hr0.out -m 0
epic -r 100 -a atr yow -L -A af.out
-V vf.out -S st.out stm.out
(The last two lines shown above form a single command. The mxm command
gathers statistics on measurement number 0; if other heart rate measurements
are defined, mxm should be run once for each such measurement,
substituting the appropriate measurement numbers for 0 in the output
file name, hr0.out, and the final argument.) Statistics for the
remainder of the MIT DB are obtained by repeating these commands, substituting
in each the appropriate record names for 100. Once these commands have
been run for all of the records, the record-by-record statistics will be found
in nine files (bxb.out, sd.out, vruns.out, sruns.out,
hr0.out, af.out, vf.out, st.out, and stm.out).
The first eight of these files contain one line for each record.
sumstats can read any of these files, and calculates aggregate performance
statistics; to use it, type ``sumstats file'', where file is
the name of one of these files. The output of sumstats contains a copy
of its input, with aggregate statistics appended to the end. Typically this
output might be saved in a file to be printed later, e.g.,
sumstats bxb.out >>report.out
A scatter plot of the ST measurement comparisons performed by epic can be produced using plotstm, the output of which can be printed directly on any PostScript printer. For example, to make a plot file for stm.out, type:
plotstm stm.out >stm.ps