THE EVAL_ST TOOL -------------------------------------------------------------------------------- The EVAL_ST tool [1] (PostScript and PDF versions of the reference [1] are in the man subdirectory - cinc04.ps and sinc04.pdf.) is an open source tool to evaluate and compare performance and robustness of ST episode detection algorithms. The tool supports all standard and other relevant performance measures, aggregate gross and average statistics, and bootstrap statistical procedure to predict real-world clinical performance. The tool (written in C) is compilable an a wide variety of platforms and contains an additional graphic user interface module (LessTif/Motif environment) for use on the LINUX/UNIX operating systems. -------------------------------------------------------------------------------- INTRODUCTION Assessing the performance and robustness of ST segment analysers and algorithms as well as predicting their behavior in the real-world clinical environment is a difficult task. Availability of the European Society of Cardiology ST-T Database (ESC DB) [3], gained development of transient ST segment episode detectors and allowed comparison of their performance. Newly developed Long-Term ST Database (LTST DB) [2] provides a wide variety of real-world 24-hour ambulatory records with numerous examples of transient ischemic ST segment episodes and transient non-ischemic heart-rate related ST segment episodes. It gained further development and evaluation of transient ST episode detectors. Due to relative complexity of the performance measures and of the evaluation protocol, we developed an open-source tool EVAL_ST, Version 2.0, to objective evaluate and compare the performance and robustness of transient ST episode detection algorithms. Since the LTST DB contains transient ischemic and transient non-ischemic heart-rate related ST segment episodes, we initially adapted previously developed performance measures by adding performance matrices to differentiate between these two types of ST episodes. For detailed description of the performance measures and of the evaluation protocol see [1] and related references. The tool EVAL_ST provides first- (record-by-record) and second-order (aggregate gross and average) performance statistics for evaluation and comparison of transient ST episode detection algorithms. The tool allows assessing the accuracy of: 1) detecting transient ST episodes, 2) distinguishing between ischemic and non-ischemic heart-rate related ST episodes, 3) measuring ST episode durations and ischemic ST episode durations, and 4) measuring ST segment deviations. The tool also provides generation of performance distributions using a bootstrap statistical technique for predicting real-world clinical performance and robustness. The core module of the tool (eval_st.c) supports command-line oriented (no graphic display) user interface style thus enabling possible batch processing. Input to the tool are ST segment annotation streams of a reference database (e.g., LTST DB or ESC DB) and ST segment annotation streams of the evaluated algorithms. Evaluation results are stored to output files. An additional graphic user interface module in LessTif/Motif graphic environment (geval_st.c) provides graphic display of the evaluation results on LINUX/UNIX operating system. The main window of the graphic user interface provides display of: overlapping reference and algorithm's ST episode annotation streams of the records, first- and second-order performance statistics, performance matrices regarding ST episode detection and differentiation between ischemic and non-ischemic heart-rate related ST episodes, statistics regarding ST episode and ischemic ST episode duration detection, scatter plot of ST segment deviation measurements, and bootstrap estimates of expected real-world performance with performance distributions. Detailed evaluation results are stored to output files and also graphically displayed in secondary windows of the tool. We successfully used the tool for evaluating our two ST episode detection algorithms [4,5] using the LTST DB and ESC DB. Besides the source code of the EVAL_ST tool, the reference annotations of the LTST DB and ESC DB and the annotations of the two algorithms are also provided. Availability of the EVAL_ST tool to the world community should simplify as well as promote easy and unique use of specific and complex performance measures and evaluation protocol, help to easily compare different algorithms, encourage the use of unique performance measures and evaluatio protocols in the field, and gain the consistent use of standard performance measures. -------------------------------------------------------------------------------- HOW TO COMPILE AND RUN THE EVAL_ST TOOL? If you are using the Fedora, Core 2, you may wish to use the precompiled versions of the EVAL_ST tool [1] (eval_st - command-line oriented interface, geval_st - graphic user interface in Motif environment) which are located in the bin directory (subdirectory to EVAL_ST directory). After unziping and untaring the tool, entering the EVAL_ST directory, enter the src directory. To compile the tool under Linux use the Makefile (Makefile_Linux) script, to compile it under Solaris (UNIX) use Makefile_Motif_Solaris script. To compile the tool, type following: make eval_st what will compile command-line oriented interface (eval_st.c) to eval_st. make geval_st will compile graphic user interface in Motif environment (geval_st.c) to geval_st. make will compile both, eval_st.c and geval_st.c to eval_st and geval_st. The source files eval_st.c and geval_st.c are equal. The only difference is value of a constant (motif) which allows conditional compiling and is set to 0 in the eval_st.c and to 1 in the geval_st.c source file. After that, copy the compiled tool to EVAL_ST directory: cp eval_st .. cp eval_st.hlp .. cp geval_st .. cp geval_st.hlp .. The files eval_st.hlp and geval_st.hlp contain help texts. After that enter the EVAL_ST directory which is working (or home directory) for evaluation: cd .. To run the command-line oriented interface type: ./eval_st -[a|r] _.evl [record] -[s|i] [-e|-b [NR_TRIALS]] To run the graphic user interface type: ./geval_st & Detailed description of the use of the eval_st or geval_st tool is at the bottom of this text. Working directroy (EVAL_ST), and input and output files of the tool are described below. Output files will appear in the EVAL_ST directory. It is also described how to add your own evaluation project. -------------------------------------------------------------------------------- WHAT IS IN THE EVAL_ST DIRECTORY? The directory EVAL_ST contains evaluation project files (*.evl files) and the subdirectories associated with this files. The names of the evaluation project files are of the form: _.evl where indicates the name of the database on which an algorithm was developed or tested and indicates the algorithm. Names of the databases are: ltsta, ltstb and ltstc (Long-Term ST Database, LTST DB, [2] - annotation protocols A, B and C, respectively) esc (European Society of Cardiology ST-T Database, ESC DB, [3]) Names of the algorithms are: alg01 (Karhunen-Loeve transform-based algorithm [4]) alg02 (Algorithm which tracks time-varying ST segment reference level [5]) Example: To evaluate the algorithm [5] using the protocol B of the LTST DB you select evaluation project ltstb_alg02.evl. The subdirectories associated with the evaluation projects are of the name _: indicates the name of the database indicates the algorithm These subdirectories contain reference annotation files and algorithm's annotation files for the selected pair of database and algorithm. Currently, there are following evaluation projects files in the EVAL_ST directory: esc_alg01.evl esc_alg02.evl ltsta_alg02.evl ltstb_alg01.evl ltstb_alg02.evl ltstc_alg02.evl and following subdirectories associated with these evaluation projects: esc_alg01 esc_alg02 ltsta_alg02 ltstb_alg01 ltstb_alg02 ltstc_alg02 In the other words, these directories contain reference annotation files and algorithm's annotation files of the following evaluations: 1. The algorithm [4] was developed using the ESC DB [3] (evaluation project esc_alg01.evl) and then tested using the LTST DB (protocol B) [2] (evaluation project ltstb_alg01.evl) 2. The algorithm [5] was developed using the LTST DB (protocols A, B, and C) [2] (evaluation projects ltsta_alg02.evl, ltstb_alg02.evl, and ltstc_alg02.evl, respectively) and then tested using the ESC DB [3] (evaluation project esc_alg02.evl) The files in the _ subdirectories are following: .hea .ref .cmr .dev .alg These files and the _.evl file are input files to the tool. The .hea, .ref and .cmr are reference annotation files of the database (), while .alg and .dev are annotation files of the algorithm (). Their content is described below. The subdirectory esc in the EVAL_ST directory contains reference annotation files of the ESC DB [3] only (.hea, .ref, and .cmr). The subdirectory ltsta in the EVAL_ST directory contains reference annotation files of the LTST DB, protocol A, [2] only (.hea, .ref, and .cmr). The subdirectory ltstb in the EVAL_ST directory contains reference annotation files of the LTST DB, protocol B, [2] only (.hea, .ref, and .cmr). The subdirectory ltstc in the EVAL_ST directory contains reference annotation files of the LTST DB, protocol C, [2] only (.hea, .ref, and .cmr). The subdirectory bin in the EVAL_ST directory contains precompiled versions of the EVAL_ST tool (eval_st - command-line oriented interface, and, geval_st - graphic user interface in Motif graphic environment) The subdirectory man in the EVAL_ST directory contains article [1] in the PostScript and PDF format, this README file, and help files (eval_st.hlp and geval_st.hlp). The subdirectory src in the EVAL_ST directory contains source files eval_st.c and geval_st.c, COPYING file, makefiles, this README file, and help files. -------------------------------------------------------------------------------- WHAT ARE INPUT FILES TO THE TOOL? _.evl (ASCII file) Evaluation project file which contains names of the records of the and the path to records' location in the _ directory when the is used. This file is in the working EVAL_ST directory. A row of this file is: _/, i.e., / Example: ltstb_alg02/s20011 The files connected to each record () are following: .hea .ref .cmr .dev .alg and are in the _ subdirectory. Their content is described next. .hea (ASCII file, WFDB format) The header file of the record. .ref (ASCII file) The reference annotation stream for the record . This file contains the reference annotations for ischemic or non-ischemic heart-rate related ST episodes as annotated by developers of the database in each lead separately. Each row in this file is of the form: 1. column - index of annotation [samples] 2. column - type of annotation: -1 beginnig, -2 extrema, and -3 end of the ST episode 3. column - value 0 (at the beginning and end of an episode, if this episode is ischemic) 3. column - value 1 (at the beginning and end of an episode, if this episode is non-ischemic heart-rate related) 3. column - reference ST segment deviation measurement [uV] (at the reference extrema of an episode, if this episode is ischemic or non-ischemic heart-rate related) 4. column - lead number in which the annotation was set (0, 1, or 2) .cmr (ASCII file) Combined reference annotation stream for the record . This file contains the reference annotations (ischemic or heart-rate related) combined in the sense of logical OR function while ischemic episode are more important during the combining process. It means that having an ischemic episode in one lead and simultaneously a heart-rate related episode in another lead, combined reference annotation stream will contain the ischemic episode. Each row in this file is of the form: 1. column - index of annotation [samples] 2. column - type of annotation: -1 beginning, -2 extrema, and -3 end of the ST episode 3. column - value 0 (at the beginning and end of an episode, if this episode is ischemic) 3. column - value 1 (at the beginning and end of an episode, if this episode is non-ischemic heart-rate related) 3. column - reference ST segment deviation measurement [uV] (at the reference extrema of an episode, if this episode is ischemic or non-ischemic heart-rate related) 4. column - lead number. For the algorithm's annotation stream this value is always 0, since this is combined annotated stream. .dev (ASCII file) The ST segment deviation measurements measured by the algorithm at the reference extrema annotations. This file is similar to the .ref file, while the reference ST segment deviation measurements are here replaced by algorithm's measurements. Each row in this file is of the form: 1. column - index of annotation [samples] 2. column - type of annotation: -1 beginning, -2 extrema, and -3 end of episode 3. column - value 0 (at the beginning and end of an episode, if this episode is ischemic) 3. column - value 1 (at the beginning and end of an episode, if this episode is non-ischemic heart-rate related) 3. column - algorithm's ST segment deviation measurement [uV] (at the reference extrema of an episode, if this episode is ischemic or non-ischemic heart-rate related) 4. column - lead number in which the annotation was set (0, 1, or 2) .alg (ASCII file) The algorithm's annotation stream for the record . Each row in this file is of the form: 1. column - index of annotation [in samples] 2. column - type of annotation: -1 beginning, -2 extrema, and -3 end of the ST episode 3. column - value 0 (at the beginning and end of an episode, if this episode is ischemic) 3. column - value 1 (at the beginning and end of an episode, if this episode is non-ischemic heart-rate related) 3. column - ST segment deviation measurement [uV] (at the extrema of algorithm's ST episode - within ST deviation detection function) 4. column - lead number. For the algorithm's annotation stream this value is always 0 since this is combined annotated stream. -------------------------------------------------------------------------------- WHAT ARE OUTPUT FILES OF THE TOOL? All the output files appear in the working EVAL_ST directory. __[ih|st].raw Aggregate evaluation statistics for the algorithm using the database , while ischemic and non-ischemic heart-rate related episodes are differentiated (_ih.) or are considered as episodes of single type (_st.). This file contains numbers of annotated and detected episodes, percentage of annotated and detected episode durations, sensitivity (Se) and positive predictivity (+P) matrices, number of correctly detected, falsely detected and missed episodes, and statistics on episode detection Se/+P and episode duration detection Se/+P. __[ih|st].rbr Evaluation statistics for the algorithm for each record of the database , while ischemic and non-ischemic heart-rate related episodes are differentiated (_ih.) or are considered as episodes of single type (_st.). This file contains (for each record) a numbers of annotated and detected episodes, percentage of annotated and detected episode durations, Se and +P matrices, number of correctly detected, falsely detected and missed episodes, and statistics on episode detection Se/+P and episode duration detection Se/+P. __[ih|st].ebe Evaluation statistics of the ST segment deviation measurements at the episode extrema, while ischemic and non-ischemic heart-rate related episodes are differentiated (_ih.) or are considered as episodes of single type (_st.). File contains following ST segment deviation measurement statistics: mean [uV], standard deviation [uV], correlation coefficient, regression line of measurements, the percentage of measurements for which the absolute error differs by more than 100 uV, and the value of error which 95 % of the measurements do not exceed. Additional information is: episode number, record, lead, time of the extrema [h:m:s], algorithm measurement [uV], reference measurement [uV], difference between algorithm and reference measurement [uV] and error [%]. __[ih|st].bts Evaluation statistics for the bootstrap evaluation. File contains aggregate statistics for Se and +P using the bootstrap method for the algorithm and database , while ischemic and non-ischemic heart-rate related episodes are differentiated (_ih.) or are considered as episodes of single type (_st.). The aggregate statistics included are 5%, 16%, 84% and 95% confidence limits for the performance statistics, minimum and maximum performance statistics, median, mean, and difference between the mean performance statistics and the 5% confidence limits. __[ih|st]. Evaluation statistics of the algorithm for the record of the database , while ischemic and non-ischemic heart-rate related episodes are distinguished (_ih.) or are considered as episodes of single type (_st.). File contains numbers of annotated and detected episodes, percentage of annotated and detected episode durations, Se and +P matrices, numbers of correctly detected, falsely detected and missed episodes, and statistics on episode detection Se/+P and episode duration detection Se/+P. In addition, the file contains also detailed matching report for each episode (reference and algorithm's). __[ih|st].smrYYY Summary statistics, while ischemic and non-ischemic heart-rate related episodes are differentiated (_ih.) or are considered as episodes of single type (_st.). The YYY denotes the number of demanded summary statistics (000-999). The file contains summary statistics of the performed evaluations. These can include aggregate performance statistics, bootstrap statistics and ST segment deviation measurement statistics. -------------------------------------------------------------------------------- HOW TO ADD YOUR OWN EVALUATION PROJECT? In the EVAL_ST directory add your own _.evl file and your own _ subdirectory. The _.evl file should contain the paths to the record files in the _ subdirectory. The files in the _ subdirectory should be reference annotation files (.hea, .ref and .cmr) of the selected database and annotation files of the evaluated algorithm (.alg). If you plan to evaluate also the ability to measure ST deviations, add the algorithm's .dev file. For all these files, strictly follow the file format described above. If your database is LTST DB, simply copy the reference annotation files from the ltsta (protocol A), ltstb (protocol B), or ltstc (protocol C) subdirectory (according to the selected annotation protocol) to your _ subdirectory. If your database is ESC DB, simply copy the reference annotation files from the esc subdirectory to your _ subdirectory. -------------------------------------------------------------------------------- HOW TO USE THE eval_st TOOL? Usage of the eval_st (EVAL_ST) command-line oriented user interface to evaluate transient ST segment episode detectors: To write help text: eval_st -h To evaluate an algorithm using a single record of a database: eval_st -r _.evl -[s|i] _.evl is the evaluation project file, where is database used and is the algorithm evaluated One of the options -s or -i has to be used but only one at one time. -r Performs evaluation of the algorithm using the selected record of the database -s All the episodes are considered as episodes of single type -i Ischemic and non-ischemic heart-rate related episodes are differentiated Examples: eval_st -r ltstb_alg02.evl s20011 -s Generates performance evaluation statistics for the record s20011 while all the episodes are considered as episodes of single type eval_st -r ltstb_alg02.evl s20011 -i Generates performance statistics using the record s20011 for the ischemic episodes To evaluate an algorithm using an entire database: eval_st -a _.evl -[s|i] [-e|-b [NR_TRIALS]] _.evl is the evaluation project file, where is database used and is the algorithm evaluated One of the options -s or -i has to be used but only one at one time. Only one of the options -e or -b can be used in one time. -a Performs aggregate statistics and record-by-record statistics for the algorithm using the selected database -s All the episodes are considered as episodes of single type -i Ischemic and non-ischemic heart-rate related episodes are differentiated -b [NR_TRIALS] Performs the bootstrap evaluation using NR_TRIALS bootstrap trials (default is 10000) -e Performs evaluation of ST segment deviation measurements Examples: eval_st -a ltstb_alg02.evl -s Generates aggregate statistics and record-by-record statistics while all the episodes are considered as episodes of single type eval_st -a ltstb_alg02.evl -i Generates aggregate statistics and record-by-record statistics for ischemic episodes eval_st -a ltstb_alg02.evl -s -b Generates aggregate statistics and record-by-record statistics, and performs the bootstrap evaluation while all the episodes are considered as episodes of single type eval_st -a ltstb_alg02.evl -s -e Generates aggregates statistics and record-by-record statistics, and performs evaluation of ST segment deviation measurements -------------------------------------------------------------------------------- HOW TO USE THE geval_st TOOL? To run it, type: ./geval_st & Usage of the geval_st (EVAL_ST) graphic user interface to evaluate transient ST segment episode detectors: Menu Options Mnemonic Accelerator Description - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - File Open Alt+F O Ctrl+O Opens evaluation project file _.evl which contain the list of records given the algorithm and database for testing Examine Alt+F E Ctrl+E Opens secondary window with information about statistics already perfromed given the selected evaluation project Quit Alt+F Q Ctrl+Q Quits the interface Evaluation_mode Ischemic/Heart-rate ... Alt+E I Ctrl+I Sets the evaluation mode when the ischemic and non-ischemic heart-rate related episodes are differentiated ST segment changes Alt+E S Ctrl+S Sets the evaluation mode when all the episodes are considered as episodes of single type Single_record Record Alt+S R Ctrl+R Opens the dialog to select a singe record for evaluation and then performs evaluation for the selected record Aggregate_statistics Evaluate Alt+A E Ctrl+A Performs aggregate statistics for the selected evaluation project Extrema Alt+A X Ctrl+X Performs evaluation of ST segment deviation measurements for the selected evaluation project Bootstrap Number_of_bootstrap_trials Alt+B N Ctrl+N Opens the dialog to set the nuber of bootstrap trials to be used Evaluate Alt+B E Ctrl+B Performs the bootstrap evaluation for the selected evaluation project Summary Compare Alt+M C Ctrl+C Opens a "summary" secondary window and displays the relevant statistics of the last three performance evaluation projects Differences Alt+M D Ctrl+D Comparison of algorithms Help Help Alt+H H Ctrl+H This message -------------------------------------------------------------------------------- REFERENCES [1] Jager, F., Smrdel A., and Mark, R. G. An Open Source Tool to Evaluate Performance of Transient ST Segment Episode Detection Algorithms. Computers in Cardiology, 2004. [2] Jager, F., Taddei, A., Moody, G. B., Emdin, M., Antolic, G., Dorn, R., Smrdel, A., Marchesi, C., and Mark, R. G. Long-term ST database: a reference for the development and evaluation of automated ischaemia detectors and for the study of the dynamics of myocardial ischaemia. Med. & Biol. Eng. & Comput. 2003; 41:172-182. [3] Taddei, A., Distante, G., Emdin, M., Pisani, P., Moody, G. B., Zeelenberg, C., and Marchesi, C. The European ST-T database: standard for evaluating systems for the analysis of ST-T changes in ambulatory electrocardiography. Eur. Heart J. 1992; 13:1164-1172. [4] Jager, F., Moody, G. B., and Mark, R. G. Detection of transient ST segment episodes during ambulatory ECG monitoring. Comput. Biomed. Res. 1998; 31:305-322. [5] Smrdel, A., and Jager, F. Automated detection of transient ST-segment episodes in 24h electrocardiograms. Med. & Biol. Eng. & Comput. 2004; 42:303-311. --------------------------------------------------------------------------------