A significant constraint imposed on evaluators by the EC38 standard is that they must obtain annotation files containing the analysis results of the device under test. Although the device itself need not produce these files, EC38 specifically requires that they be produced by an automated procedure, which must be fully disclosed. The intent of this requirement is to permit reproducible independent evaluations in which neither the proprietary data of the developers (the analysis algorithms) nor that of the evaluators (the test signals and reference annotations) need necessarily to be disclosed. By defining the interface between the developer and the evaluator to be the annotation file, the responsibilities of each party are clearly defined: the developer must make certain that the device's outputs are recorded in the annotation file in the manner intended by the developer, but in the language of the standard; the evaluator must make certain that the algorithms used to compare the device's annotation files with the reference annotation files conform to the specification of the standard. The format and content of these annotation files is specified in detail below. For many existing devices, it may be difficult or impossible to obtain such annotation files without the cooperation of the developers. Newly-designed devices should incorporate the necessary ``hooks'' for producing annotation files.
Please e-mail your comments and suggestions to webmaster@physionet.org, or post them to:
PhysioNetUpdated 8 March 2019