Obtaining Test Annotation Files

For any ambulatory ECG monitor that incorporates automated analysis functions, the EC-38 standard requires the manufacturer to implement and disclose a method for producing test annotation files. Independent evaluators should seek assistance from the manufacturer in any case, since the manufacturer's interpretation of the device's outputs in the language of EC-38 is definitive (in effect, the annotation file generation technique becomes part of the system under test). Note that generation of annotation files need not be synchronous with data acquisition; a device might conceivably store all of the necessary data until the end of the test, and only then write the file. Neither does the standard require that an annotation be determined within any fixed amount of time, as would be expected of devices designed to trigger pacing, for example. Furthermore, EC-38 specifically allows for the possibility that the device under test might not produce the annotation file directly. If any external hardware or software is required to do so, however, it must be made generally available or specified in sufficient detail by the manufacturer to permit an independent evaluator to obtain test annotation files.

Annotation files contain a label (an annotation) for each beat and for certain other features of the signals, such as rhythm and ST changes. Annotations are stored in time order in annotation files. The ``time'' of an annotation is that of the sample in the signal file with which the annotation is associated.⁵ The WFDB library (included in the WFDB software package) includes C-callable functions (getann and putann) for reading and writing annotations. In a C program, annotations appear as data structures containing a 32-bit time field together with a pair of 8-bit fields that encode the annotation type and sub-type (anntyp and subtyp [sic], respectively), and a variable-length aux field usually used to store text. In annotation files, these annotation structures are usually stored in a variable-length bit-packed format averaging slightly more than 16 bits per annotation.⁶

Test annotation files may include the following:

• Beat annotations. These need not coincide precisely with the reference beat annotations, since the evaluation protocol allows a time difference of up to 150 ms between each pair of matching beat annotations. All beat annotations are mapped during the evaluation process into the set { N, V, F, S, Q } (corresponding to normal, ventricular ectopic, ventricular fusion, supraventricular ectopic, and unclassifiable or paced beats respectively); devices need not be capable of producing all of these annotations, but any beat annotations that they do produce will be translated into one of these types. The standard specifies the mapping used for the anntyp values defined in <wfdb/ecgcodes.h>. (This file is included in the WFDB Software Package.) Any beat annotations that appear in the first five minutes of a record (the ``learning period'') are ignored in the evaluation process. The remainder of the record (the ``test period'') must be fully annotated. Note in particular that the last beat of some records may be very close to the last sample; since the analyzer may reach the end of the input signals before producing an annotation for the last beat, it may be necessary to ``pad'' the input data for a few seconds at the end of the record to permit the analyzer to emit its final beat annotation.

• Shutdown annotations. If the device suspends its analysis because of poor signal quality or for any other reason, it should mark the periods during which analysis is suspended. The evaluation software tallies beats missed during such periods separately from beats missed at other times. The beginning of each period of shutdown may be marked using a NOISE annotation with ${\tt subtyp} = -1$, and the end of each period of shutdown may be marked using a NOISE annotation with ${\tt subtyp} = 0$ (see the source for bxb for notes on other acceptable methods of marking shutdown).

• Ventricular fibrillation annotations. The beginning and end of each detected episode of ventricular fibrillation should be marked using VFON and VFOFF annotations.

• Other rhythm annotations. These should include RHYTHM annotations marking the beginning and end of each detected episode of atrial fibrillation. The beginning of each episode should be marked with an ``(AFIB'' rhythm annotation, i.e., an annotation with anntyp = RHYTHM and aux = "\05(AFIB", where ``\05'' is C notation for a byte with the value 5 (ASCII control-E). Non-empty aux fields always begin with a byte that specifies the number of data bytes that follow; in this case, the five characters (( A F I B) of the string. The end of each episode should be marked with any other rhythm annotation (for example, "\02(N").

• Heart rate measurements. Each type of heart rate measurement (including any heart rate or RR interval variability measurements) made by the device under test should be assigned a measurement number, $m$, between 0 and 127. A MEASURE annotation should be recorded for each heart rate measurement, with ${\tt subtyp} = m$ and with the measurement in the aux field, as an ASCII-coded decimal number.

• ST deviation measurements. If available, these should be provided in the aux fields of beat annotations, as ASCII-coded decimal numbers indicating the deviations in microvolts from reference levels established for each signal from the first 30 seconds of each record. For example, ``25 -104'' indicates a 25 $\mu$V elevation in signal 0 and a 104 $\mu$V depression in signal 1. If ST measurements are omitted from any beat annotation, the evaluation software assumes they are unchanged from their previous values.

• Ischemic ST change annotations. These STCH annotations should mark the beginning and end of each detected episode of ischemic ST change. ST change annotations have additional information in the aux field as for rhythm annotations: the beginning of each episode is marked by an ``(STns'' annotation, and the end of each episode by a ``STns)'' annotation, where n indicates the signal affected (``0'' or ``1''), and s indicates ST elevation (``+'') or depression (``-''). n may be omitted if the episode detection criteria depend on features of both signals. The extremum of each episode may optionally be marked with an ``ASTnsm'' annotation, where n and s are defined as above, and m is the ST deviation in microvolts, relative to a reference level established as above.

• Comment annotations. Annotations with anntyp = NOTE and any desired string data in aux may be included anywhere in an annotation file. NOTE annotations are ignored by the standard evaluation software; they may be used, for example, to record the values of internal algorithm variables for debugging purposes.

Note that only beat annotations are absolutely required in test annotation files. ST deviation measurements within beat annotations, and the other types of annotations listed above, only need to be recorded for devices that are claimed by their manufacturers to provide optional features for detection of ventricular or atrial fibrillation, measurement of ST deviations, or detection of ischemic ST changes.

If the time units in the test annotation files are not the same as those in the reference annotation files (for example, because xform was used to change the sampling frequency of the signal files in a digital-domain test), the time units must be rescaled before proceeding with the comparison. This may be done by using xform to rewrite the test annotation files with the original sampling frequency.⁷

Details of the ST deviation measurement and episode detection criteria used in producing the reference annotation files for the ESC database may be found in several sources.⁸ Note, however, that many techniques for measuring ST deviation and for detecting transient ischemic ST changes are possible, and that to date the best evaluation results have been obtained for analyzers using criteria that do not attempt to mimic those used by the human experts who annotated the database.

Please e-mail your comments and suggestions to webmaster@physionet.org, or post them to:

Updated 8 March 2019