Comparing Annotation Files

Once the test annotation files and the reference heart rate annotation files have been obtained, the remainder of the evaluation procedure is straightforward. All of the information needed to characterize the analysis performed by the device under test is encoded in the test annotation files; similarly, all of the information needed to characterize the actual contents of the test signals is encoded in the reference annotation and reference heart rate annotation files. The evaluation procedure thus entails comparison of the test and reference annotation files for each record.

Four programs are provided in the WFDB Software Package for this purpose:

• bxb compares annotation files beat by beat; its output includes QRS, VEB, and (optionally) SVEB sensitivity and positive predictivity, as well as RR interval error and shutdown statistics.

• rxr compares annotation files run by run; its output includes ventricular (and, optionally, supraventricular) ectopic couplet, short run (3-5 beats), and long run (6 or more beats) sensitivity and positive predictivity.

• epicmp compares annotation files episode by episode; its output includes ventricular fibrillation, atrial fibrillation, and ischemic ST detection statistics as well as comparisons of ST deviation measurements.

• mxm compares measurements from a test annotation file and a reference heart rate annotation file; its output indicates measurement error.¹⁰

The WFDB Software Package also includes three related programs:

• sumstats reads certain output files generated by bxb, rxr, epicmp, and mxm, and calculates aggregate statistics for a set of records.
• plotstm generates scatter plots of ST deviation measurements collected by epicmp.
• ecgeval automates the entire comparison procedure by running bxb, rxr, epicmp, and mxm for each record, collecting their output, then running sumstats (and optionally plotstm), and finally printing the results.

To obtain a concise summary of how to use any of these programs, including a list of any command-line options, simply run the program without any command-line arguments. Refer to the WFDB Applications Guide, which accompanies the WFDB Software Package, for details.

In most cases, it will be easiest to collect all of the annotation files before beginning the comparison, and then to perform the comparison by typing:

ecgeval

The program asks for the test annotator name, the names of the databases used for testing, and what optional detector outputs should be evaluated.

Only the statistics required by EC38 and EC57 are reported by ecgeval. If more detailed evaluation data are needed, it will be necessary to run bxb, rxr, etc., separately. If file space is extremely limited, it may be necessary to delete each test annotation file after it has been compared against the reference file, before the next test annotation file can be created; in such cases, it may also be necessary to prompt the user to change media containing signal or reference annotation files, or to reset the device under test before beginning each record. Optionally, ecgeval can generate a script (batch) file of commands, which can be edited to accommodate special requirements such as these.

For example, suppose we have obtained a set of test annotation files with the annotator name ``yow'', which we wish to compare against the reference annotation files (annotator name ``atr'')¹¹and reference heart rate annotation files (annotator name ``htr''). The portion of the evaluation script generated by ecgeval for MIT DB record 100 is:

bxb -r 100 -a atr yow -L bxb.out sd.out
rxr -r 100 -a atr yow -L vruns.out sruns.out
mxm -r 100 -a htr yow -L hr0.out -m 0
epicmp -r 100 -a atr yow -L -A af.out
    -V vf.out -S st.out stm.out

(The last two lines shown above form a single command. The mxm command gathers statistics on measurement number 0; if other heart rate measurements are defined, mxm should be run once for each such measurement, substituting the appropriate measurement numbers for 0 in the output file name, hr0.out, and the final argument.) Statistics for the remainder of the MIT DB are obtained by repeating these commands, substituting in each the appropriate record names for 100. Once these commands have been run for all of the records, the record-by-record statistics will be found in nine files (bxb.out, sd.out, vruns.out, sruns.out, hr0.out, af.out, vf.out, st.out, and stm.out). The first eight of these files contain one line for each record.¹² sumstats can read any of these files, and calculates aggregate performance statistics; to use it, type ``sumstats file'', where file is the name of one of these files. The output of sumstats contains a copy of its input, with aggregate statistics appended to the end. Typically this output might be saved in a file to be printed later, e.g.,

sumstats bxb.out >>report.out

A scatter plot of the ST measurement comparisons performed by epicmp can be produced using plotstm, the output of which can be printed directly on any PostScript printer. For example, to make a plot file for stm.out, type:

plotstm stm.out >stm.ps

Please e-mail your comments and suggestions to webmaster@physionet.org, or post them to:

Updated 8 March 2019