PhysioNet/Computing in Cardiology Challenge 2010: Mind the Gap

In settings ranging from sleep studies to surgery to sports medicine to intensive care, real-time monitoring of a variety of physiologic signals has become an essential tool for clinicians and researchers. Transient corruption or loss of one or more signals, common in all of these settings, can be disruptive, especially when continuous observations are required in order to rule out rare events or as a basis for forecasting. Signal corruption can be particularly challenging when it mimics features that are associated with pathologic states.

Humans can be remarkably adept at dealing with transient noise and signal loss in these settings. Filling in gaps, and making use of context to recognize and ignore noise, are processes that our sensory and cognitive abilities leave us well-equipped to do. Can algorithmic solutions that take account of the same data, in broader contexts and without fatigue, do as well?

The aim of this year's challenge is to develop robust methods for filling in gaps in multiparameter physiologic data (including ECG signals, continuous blood pressure waveforms, and respiration). In a real-world monitoring application, these methods can be applied for many purposes, including:

• robust estimation of parameters such as heart rate, mean arterial pressure, and respiration when the primary signals used to derive these parameters become unavailable or unreliable;
• detection of changes in patient state, when the relationships between signals remain consistent even as individual signals change their behavior; and
• recognition of intervals of signal corruption, when a signal becomes inconsistent not only with respect to its previous history but also with respect to its relationships with other signals.

In this Challenge, participants are asked to reconstruct, using any combination of available prior and concurrent information, segments of signals that have been removed from multiparameter recordings of patients in intensive care units (ICUs).

Data for the Challenge

Participants are provided with three data sets of 100 records each. Each ten-minute record contains 6, 7, or 8 signals acquired from bedside ICU patient monitors. The recorded signals vary across records, and they include ECG, continuous invasive blood pressure, respiration, fingertip plethysmograms, and occasional other signals. In one of these signals, the final 30-second segment (the target signal) has been replaced by a gap (a flat line signal), and the goal is to reconstruct this missing 30-second target signal in each record.

• Set A is a set of 100 records for participants' use as a training set. Participants can obtain scores for Set A reconstructions at any time, but Set A scores are not included in the final rankings of Challenge entries. The target signals are provided for the records in this set. Additional training data can easily be constructed from any of the multiparameter data available in PhysioBank, such as the MIMIC II Waveform Database.
• Set B is a set of 100 records for which the target signals are withheld until the conclusion of the challenge. Participants can obtain scores for Set B reconstructions at any time, but Set B scores are not included in the final rankings of Challenge entries.
• Set C is a set of 100 records, with the target signals withheld. Participants may submit reconstructions of the target signals at any time for the duration of the challenge, but they will not receive scores except for their final submissions, which will determine the final rankings and the winners of the challenge.

Records in all three Challenge data sets are provided in standard PhysioBank (compact binary) formats; read them using the WFDB software package, use the PhysioBank ATM to convert them to text, CSV, Matlab, or EDF formats, or see How to obtain PhysioBank data in text form for notes on other methods for converting them to text. The Set A files named with the suffix .missing are the target signals; these files are in text format.

The data sets, together with the target signals for Set A, are available here. To download them efficiently, use rsync as described here (retrieve the challenge-2010 "module"). For convenience, a tarball containing the full contents of all three data sets is available here (196,792,152 bytes); see this note for information about unpacking tarballs.

Scoring

A detailed reconstruction of a signal, reproducing all characteristics of the original signal accurately, is more than is necessary in order to be useful. For example, a reconstruction of an invasive blood pressure signal may be useful if it allows a good estimation of mean pressure, or pulse pressure, even if none of the other details of the signal can be reproduced accurately; as another example, a reconstruction of an ECG signal that allows accurate estimation of RR intervals may be useful, even in the absence of other details.

For this reason, this challenge makes use of two scoring algorithms, detailed below. The first, used in event 1, measures the overall accuracy of the reconstruction, and as such will tend to favor those reconstruction methods that succeed in recovering properties of the signal that are derivable from the signal amplitude, such as MAP and pulse pressure. The second, used in event 2, is based on the correlation between the original signal and its reconstruction, and tends to favor those methods that accurately recover the timing of the major fluctuations in the original signal, such as QRS complexes in the ECG needed to derive RR interval series. Each reconstruction is scored using both algorithms, so a single submission is entered in both events of the Challenge.

Scoring for individual reconstructions

Each reconstruction, V_rec, is compared with the corresponding target (reference) signal, V_ref. Target signals are known in all cases, but those for the Set B and Set C records are not provided to participants.

Each submitted reconstruction is scored using two different methods, one for each of the two events. Since the scoring methods for the two events have different biases, a change in your reconstruction of a given record may improve your event 1 score and decrease your event 2 score, or vice versa.

Event 1

The target signal is subtracted (sample by sample) from the reconstruction to obtain the residual signal, V_res. The sum of the squares of the residuals, ssvres, is normalized by the energy (sample variance) of the target signal, E_ref.

In event 1, the quality, Q₁, of a reconstruction is defined as 1 - ssvres/E_ref, or 0, whichever is larger. If ssvres is 0, Q₁ is 1, even if E_ref is also 0.

Use of a figure of merit based on the residual signal reflects the importance in many cases of obtaining a good estimate of target signal levels (such as systolic, mean, and diastolic pressures in a continuous blood pressure signal).

[Note that Q₁ was previously defined as 1 - E_res/E_ref, or 0, whichever is larger (where E_res is the energy of the residual signal). This definition was changed because it was independent of the magnitude of any constant (DC) component of the residual signal, unlike the revised definition.]

Event 2

In event 2, the quality Q₂ of a reconstruction is defined as the correlation coefficient of V_ref and V_rec, or 0, whichever is larger. (Correlation coefficients can of course be negative; for the purposes of this challenge, an anticorrelated reconstruction is equivalent to an uncorrelated one, however.)

Use of the correlation coefficient as a figure of merit is motivated by the observation that reconstruction of a filtered signal may be useful in many cases. Such a reconstruction might, for example, provide a basis for reliable estimation of the timing of major fluctuations in a signal (such as QRS complexes in an ECG signal), even if absolute signal levels are not recovered. Unlike Q₁, Q₂ is relatively insensitive to misestimation of the amplitudes of fluctuations.

Aggregate (summed) scores

The final ranking of participants is based on summing the Q scores obtained for records in Set C. Participants are encouraged, but not required, to provide reconstructions of all records in Set C.

Both Q₁ and Q₂ are defined so they can vary between 0 and 1, and higher values are better. If you have submitted more than one reconstruction of the same record, only the last one submitted determines your Q₁ and Q₂ for that record. If you have not submitted a reconstruction for a given record, your Q₁ and Q₂ scores for that record are zero.

For event 1, your Set C summed score, C₁, is the sum of the your final Q₁ scores for each record in Set C. Similarly, for event 2, your Set C summed score, C₂, is the sum of the your final Q₂ scores for each record in Set C. Since Set C contains 100 records, C₁ and C₂ can vary between 0 and (in theory!) 100.

The summed scores are not normalized by the number of target signals reconstructed, to provide a strong incentive to submit reconstructions of as many of the Set C records as possible.

Entering the Challenge

To register as a participant in the Challenge:

1. create a PhysioNetWorks home page.
2. On your newly-created home page, click on the designated link to enter the Challenge.
3. When you next visit your home page, links for submitting reconstructions and viewing your scores will appear there.

To be eligible for an award (see below), also:

5. Submit reconstructions (see below) of at least 10 Set B target signals, no later than noon GMT on Friday, 30 April 2010.
6. Submit an abstract (about 300 words) describing your work on the Challenge to Computing in Cardiology (CinC) by Saturday, 1 May 2010. Please select "PhysioNet/CinC Challenge" as the topic of your abstract, so it can be identified easily by the abstract review committee.
7. Submit reconstructions of as many Set C target signals as you wish no later than noon GMT on Wednesday, 1 September 2010. Since your ranking is determined by the sum of the scores you receive for Set C targets, it is to your advantage to attempt as many of the 100 Set C records as possible.
8. Attend Computing in Cardiology in Belfast, Northern Ireland, 26-29 September.

If your abstract is accepted, you will be expected to prepare a four-page paper for publication in Computing in Cardiology, and to present a talk or poster about your work at CinC.

Submitting reconstructions for scoring

Reconstructions should be in the same format as the .missing files containing the Set A target signals (text format, with one sample of the reconstructed signal per line). Each file should contain 3750 samples (125 samples/second, for 30 seconds). Although the target signals consist entirely of samples with integer values, reconstructions may contain integer or floating point values. Use a decimal point (.) rather than a comma (,) to separate the integral and fractional components of floating point values.

Follow the links from your PhysioNetWorks home page (see above) to submit reconstructions and to view your Set A and B individual record scores, Q₁ and Q₂, as well as your Set A and B summed scores, A₁, A₂, B₁, and B₂.

You may replace any previously submitted reconstruction simply by submitting a new one. The corresponding scores are updated immediately.

Scores are calculated using the code in c2010-score.c, a short C program that you may download and use to compare any two time series.

Participants may submit reconstructions of as many Set A, B, and C target signals as they wish. Each reconstruction is scored separately. For each event, the final ranking of participants is based on the sum of the scores obtained for reconstructions of Set C target signals.

Keep in mind that Sets A and B are intended for experimentation with your methods for reconstruction. You can obtain scores for your Set A and B reconstructions at any time during the Challenge. Set C is the "final examination", and although you have until September to study it, you will not be able to obtain intermediate scores. "Tuning" your methods to Set C will not be possible, so learn what you can from Sets A and B and then apply it to Set C.

Final scores and Challenge awards

Aggregate and individual scores for Set C records will be calculated based on reconstructions submitted no later than noon GMT on 1 September 2010. These scores, which will appear on your page when they have been calculated, determine the final ranking of participants.

During a plenary session of Computing in Cardiology in September, four awards will be presented to the eligible participants in attendance with the best final scores as follows:

1. (Event 1, open source) Highest C₁ score for an open source entry
2. (Event 1, overall) Highest C₁ score
3. (Event 2, open source) Highest C₂ score for an open-source entry
4. (Event 2, overall) Highest C₂ score

As in previous years, the Challenge includes an open source division. You may enter the open source division by sending the source code for your challenge entry by email, before noon GMT on Wednesday, 1 September 2010, to PhysioNet. Use the subject line "Challenge 2010 entry source", and be sure to include:

• Your name and email address
• All sources needed to produce a working version of your software (except for readily available standard libraries and header files)
• A note describing how to produce a working version of your software (a commented Makefile is ideal), and how to run your software

Each source file submitted should begin with a comment block containing the names of its authors and a reference to the open source license you have chosen for it, if any; for example:

     /* fill.c - fill signal gaps using large wads of duct tape
        Copyright (C) 2010  Herman Foobar <hbar@uncertain.org>
        This software is released under the terms of the GNU General
        Public License (http://www.gnu.org/copyleft/gpl.html).
     */

Source files in C, C++, Fortran, or Matlab m-code are preferred; other languages may be acceptable, but please ask first. Do not submit any code that cannot be freely redistributed. Following the conclusion of the Challenge, selected entries will be posted, with full credit to their authors, on PhysioNet.

Questions and answers about the Challenge

Must there be an automatic process to choose the set of signals used in the reconstruction of the target signal or is it acceptable that the choice of signals is not done by a program but by the participant?

Either method is acceptable. Keep in mind that the final rankings will be determined by your scores on set C, and that you will receive only one score for each of the set C records (in other words, you will not be able to try several different signal subsets for each set C record and choose the subset that yields the best score in each case).

There are some files in the challenge dataset, where there are several signals with zeros in the last 30 seconds. The determination of the target signal is ambiguous at the following signals: c21, c54, c67. What should be the way of handling this problem?

In each of these cases, there is only one signal that becomes flat (all zero values) after 9 minutes and 30 seconds (i.e., beginning at sample 71250). Reconstruct that signal, not the one that has been flat for a longer period.

• In c21, ABP and UAP are flat, but UAP is flat throughout; reconstruct ABP.
• In c54, CVP and ICP are flat, but CVP is flat beginning at sample 49556; reconstruct ICP.
• In c67, ABP and CVP are flat, but ABP is flat throughout; reconstruct CVP.