PhysioNet/Computing in Cardiology Challenge 2011

Quick Links: Papers about the Challenge New (17 August 2012): Physiological Measurement September 2012 focus issue on signal quality in cardiorespiratory monitoring, with 9 articles by Challenge participants Final scores for all events Sources for event 2 and 3 entries Challenge Simulated Data Training Data (Set A) Test Data (Set B) Challenge Android API Challenge FAQ Help grade the challenge data New (3 February 2012): Additional follow-up event 1 entries may be submitted for scoring, even if you have previously submitted 5 event 1 entries. Visit your Challenge Participant Page for details.

Improving the quality of ECGs collected using mobile phones

According to the World Health Organization, cardiovascular diseases (CVD) are the number one cause of death worldwide. Of these deaths, 82% take place in low- and middle-income countries. Given their computing power and pervasiveness, is it possible for mobile phones to aid in delivery of quality health care, particularly to rural populations distant from physicians with the expertise needed to diagnose CVD?

Advances in mobile phone technology have resulted in global availability of portable computing devices capable of performing many of the functions traditionally requiring desktop and larger computers. In addition to their technological features, mobile phones have a large cultural impact. They are user-friendly and are among the most efficient and most widely used means of communication. Currently there is about one cell phone for every two humans in the world.

India is experiencing a double burden of disease with persistent infectious disease coupled with increasing incidence of chronic disease. Two chronic diseases, CVD and cancer, currently account for nearly 20% of the total disease burden, which is expected to double to 40% by 2016. Unfortunately, due to a lack of adequate primary care capacity, most chronic diseases are diagnosed at an advanced stage, when the cost of treatment and rehabilitation is prohibitive for the masses, particularly the poor. This is true for other middle-income developing countries such as Brazil, China, Indonesia and South Africa as well.

India's large population spread, in conjunction with the increase in heart-related diseases, is a public health concern that has led to a joint collaboration between Narayana Hrudayalaya (one of India’s leading health-care providers) and Sana (an open-source, student-managed, mobile telemedicine group at MIT; see http://www.sanamobile.org/). Sana's specific objective in this venture is to enable an inexperienced nurse or paramedic to collect and transmit electrocardiograms (ECGs) from rural patients for remote analysis by cardiologists at a city hospital. While Sana has been successful in developing open-source software for transmitting and archiving ECGs through Bluetooth recording, significant obstacles still remain. PhysioNet has partnered with Sana to identify some of the crucial obstacles involved in having an inexperienced person record ECGs usable for diagnostic interpretation from a mobile device.

The aim of the PhysioNet/Computing in Cardiology Challenge 2011 is to develop an efficient algorithm able to run in near real-time within a mobile phone, that can provide useful feedback to a layperson in the process of acquiring a diagnostically useful ECG recording. At a minimum, the software should be able to indicate within a few seconds, while the patient is still present, if the ECG is of adequate quality for interpretation, or if another recording should be made. Ideally, the software should identify common problems (such as misplaced electrodes, poor skin-electrode contact, external electrical interference, and artifact resulting from patient motion) and either compensate for these deficiencies or provide guidance for correcting them.

Data to support development and evaluation of challenge entries are being collected by the Sana project, and will be provided freely via PhysioNet. The data set will include ten-second recordings of twelve-lead ECGs; age, sex, weight, and possibly other relevant information about the patients; and (for some patients) a photo of the electrode placement taken using the mobile phone. Although some of the recordings will be identified initially as acceptable or unacceptable, challenge participants and others interested will have an opportunity to assist in establishing a "gold standard" classification of the quality of the recordings in the challenge data set.

Participants may enter the challenge by completing the classification task described below. Awards will be given to the most successful participants who attend Computing in Cardiology 2011 (18-21 September 2011 in Hangzhou, China) to present their work and discuss their findings with other participants and CinC attendees; see Awards below for details.

Data for the Challenge

The challenge data are standard 12-lead ECG recordings (leads I, II, II, aVR, aVL,aVF, V1, V2, V3, V4, V5, and V6) with full diagnostic bandwidth (0.05 through 100 Hz). The leads are recorded simultaneously for a minimum of 10 seconds; each lead is sampled at 500 Hz with 16-bit resolution.

Nurses, technicians, and volunteers with varying amounts of training recorded the ECGs for this project. In the intended application, the recordists (those making ECG recordings) will not necessarily have had experience. Since the goal of this challenge is to investigate if laypersons can be assisted via software in collecting high-quality ECGs reliably, the recordings gathered for this challenge include ECGs made by volunteers with minimal training.

Data for the Challenge are available here. To download them efficiently, use rsync as described here (retrieve the challenge-2011 "module").

The data are provided in both CSV format, compatible with the Challenge Android API, as well as standard PhysioBank (compact binary) formats, readable using the WFDB software package and the PhysioBank ATM (which can convert them to tab-separated text, Matlab, or EDF formats).

Challenge data sets

Three challenge data sets have been created from the collected ECGs:

• Set A: training data, with reference quality assessments provided to the participants (available now)
• Set B: test data for events 1 and 2, with reference quality assessments withheld; (available now)
• Set C: test data for event 3 (not released to participants)

A series of events prevented us from collecting a sufficient number of ECGs with the hardware we had originally planned to use for this Challenge, and in order to permit the Challenge to go forward, the ECGs in sets A, B, and C have been collected using conventional ECG machines. (A pilot dataset containing synthetic ECGs recorded using the target hardware was posted previously and is still available.) Our originally planned data acquisition process is continuing, and we expect to have a set of ECGs acquired using that process available soon. As soon as possible, we'll assemble a set D of these ECGs, we'll develop a set of reference classifications for them, and we will classify them using the entries for events 2 and 3 as well. We will share the results of this experiment with event 2 and 3 participants, and we will post set D as a supplement to the Challenge 2011 data sets on PhysioNet. Since participants have not had an opportunity to study samples of ECGs collected using this process, however, these results will not be used for scoring any of the Challenge events.

ECG Quality Assessment

ECGs collected for the challenge were reviewed by a group of annotators with varying amounts of expertise in ECG analysis, in blinded fashion for grading and interpretation. Between 3 and 18 annotators, working independently, examined each ECG, assigning it a letter grade (A (0.95): excellent, B (0.85): good, C (0.75): adequate, D (0.60): poor, or F (0): unacceptable) for signal quality. The average grade was calculated in each case, and each record was assigned to one of 3 groups:

• Group 1 (acceptable): If the average grade is 0.70 or more, and at most one grade is F.
• Group 2 (indeterminate): If the average grade is 0.70 or more, but two or more grades were F.
• Group 3 (unacceptable): If the average grade is less than 0.70.

Approximately 70% of the collected records were assigned to group 1, 30% to group 3, and fewer than 1% to group 2, reflecting a high degree of agreement among the annotators.

Challenge participants also have an opportunity to grade the ECGs in the challenge data sets for quality control purposes. Login to PhysioNetWorks (create an account first if you don't have one already) and follow the link from your PhysioNetWorks home page to "Diagnostic ECG Annotations" to get started. An award fund of US$2000 will be divided among participants who have contributed in this way.

Challenge events and scoring

ECGs from all three quality groups will be presented to the challenge participants in blinded fashion. Participants may enter one or more of the following challenge events:

• Event 1: (closed source, open data) Participants must submit classifications of the challenge test set recordings only. The score is the fraction of group 1 and group 3 ECGs in set B that are correctly classified by the participant's algorithm.
• Event 2: (open source, open data) Participants must submit Java software for performing the classification function compatible with a template API and Android phone emulator provided by the challenge organizers. The score is the fraction of group 1 and group 3 ECGs in set B that are correctly classified by the participant's algorithm.
• Event 3: (open source, closed data) Same as event 2, but using set C and a different scoring algorithm. The score is the product of classification accuracy (as in events 1 and 2) and a function of execution time in the reference Android phone (faster is better).

How to submit an entry

The final deadline for entries has now passed; the information in this section is for reference only.

To enter any of the three Challenge events, login to PhysioNetWorks (create an account first if you don't have one already) and follow the link from your PhysioNetWorks home page to "PhysioNet/CinC Challenge 2011" to get started. Joining the project creates a Challenge Participant Page for you, where you will submit your entries and receive your scores.

Instructions for submitting entries to event 1 are on your Challenge Participant Page. Participants may submit up to five entries in event 1, at any time until the final deadline of noon GMT on Friday, 5 August 2011; their highest-scoring entry will determine their ranking in event 1.

Events 2 and 3 are open to event 1 participants who qualified by submitting an event 1 entry no later than noon GMT on Saturday, 30 April 2011. A single submission enters both events 2 and 3. Instructions for preparing an event 2 and 3 entry are here. Participants may resubmit their event 2 and 3 entry as many times as they wish until the final deadline of noon GMT on Friday, 5 August 2011, but only the final submission received before the deadline will be scored.

In events 2 and 3, challenge participants are required to develop algorithms capable of running in a reference Android phone. An API including a working sample algorithm is provided as a framework for these algorithms. To participate in events 2 and 3 you must use this API. In events 2 and 3, each open-source algorithm that can run in an Android phone will be tested in a reference phone.

Algorithms are not required to produce a classification for each record, but only correct classifications contribute to the scores (thus a missing classification is equivalent to an incorrect classification).

Note that although a few group 2 ECGs are present in the challenge data sets, they do not influence participants' scores. Given that the expert annotators disagree about their acceptability, it is unreasonable to expect participants' algorithms to classify them in any specific way, so group 2 ECGs are not counted. Nevertheless, since participants are not told which records belong to group 2, participants can improve their chances of obtaining a high score by classifying all records.

Awards

A generous donation from the GSM Association, in addition to support from Computing in Cardiology, has allowed us to increase Challenge awards this year to amounts that will offset most or all of the costs of registration, accommodation and travel to CinC 2011.

The GSMA represents the interests of the worldwide mobile communications industry. Spanning 219 countries, the GSMA unites nearly 800 of the world’s mobile operators, as well as more than 200 companies in the broader mobile ecosystem, including handset makers, software companies, equipment providers, Internet companies, and media and entertainment organisations. The GSMA is focused on innovating, incubating and creating new opportunities for its membership, all with the end goal of driving the growth of the mobile communications industry.

To be eligible for one of the major awards, you must:

• Submit a preliminary entry in event 1 no later than noon GMT on Saturday, 30 April 2011.
• Submit an acceptable abstract (about 300 words) about your work on the Challenge to CinC no later than noon GMT on Sunday, 1 May 2011.
• Submit a final entry in at least one event no later than noon GMT on Friday, 5 August 2011.
• Submit a paper (4 pages) describing your work on the Challenge no later than noon GMT on Sunday, 11 September 2011.
• Attend CinC 2011 and discuss your work. All Challenge presentations are scheduled for Tuesday, 20 September. Challenge awards will be presented during the closing plenary session on Wednesday, 21 September.

Each of the three most successful eligible participant teams, including the winners of each of the three events, will receive an award of US$2000, but no team or individual will receive more than one such award. If an eligible team achieves top results in more than one event, they will receive one award, and the other award(s) will be distributed to the next most successful team(s). Our objective is to ensure that at least three of the best entries are represented and discussed at CinC. An additional US$2000 will be divided among other participants who have contributed to the development of the data used in the challenge.

We thank the GSMA and Computing in Cardiology for their support of this year's Challenge.