The 2017 PhysioNet/CinC Challenge aims to encourage the development of algorithms to classify, from a single short ECG lead recording (between 30 s and 60 s in length), whether the recording shows normal sinus rhythm, atrial fibrillation (AF), an alternative rhythm, or is too noisy to be classified.
There are various types of cardiac arrhythmias that may be classified by:
- Origin: atrial arrhythmia, junctional arrhythmia, or ventricular arrhythmia.
- Rate: tachycardia ( > 100 beats per minute (bpm) in adults) or bradycardia ( < 60 bpm in adults).
- Mechanism: automaticity, re-entry, triggered.
- AV Conduction: normal, delayed, blocked.
- Duration: non-sustained (less than 30 s) or sustained (30 s or longer).
AF is defined as a “tachyarrhythmia characterized by predominantly uncoordinated atrial activation with consequent deterioration of atrial mechanical function” by the American College of Cardiology (ACC), the American Heart Association (AHA) and the European Society of Cardiology (ESC) . AF is the most common sustained cardiac arrhythmia, occurring in 1-2% of the general population [2; 3] and is associated with significant mortality and morbidity through association of risk of death, stroke, hospitalization, heart failure and coronary artery disease, etc. [3; 4]. More than 12 million Europeans and North Americans are estimated to suffer from AF, and its prevalence will likely triple in the next 30-50 years . More importantly, the incidence of AF increases with age, from less than 0.5% at 40-50 years of age, to 5-15% for 80 year olds .
Despite the enormity of this problem, AF detection remains problematic, because it may be episodic. AF detectors can be thought of belonging to one of two categories: atrial activity analysis-based or ventricular response analysis-based methods. Atrial activity analysis-based AF detectors are based on the analysis of the absence of P waves or the presence of fibrillatory f waves in the TQ interval. Published methods to do this include: an echo state neural network , P-wave absence (PWA) based detection , analysis of the average number of f waves , P-wave-based insertable cardiac monitor application , wavelet entropy   and wavelet energy . Atrial activity analysis-based AF detectors can achieve high accuracy if the recorded ECG signals have little noise contamination and high resolution, but tend to suffer disproportionately from noise contamination . In contrast, ventricular response analysis is based on the predictability of the inter-beat timing (‘RR intervals’) of the QRS complexes in the ECG. RR intervals are derived from the most obvious large amplitude feature in the ECG, the R-peak, the detection of which can be far more noise resistant. This approach may therefore be more suitable for automatic, real-time AF detection . Published methods include: Poincaré plot analysis , Lorenz plot analysis , analysis of cumulative distribution functions , thresholding on the median absolute deviation (MAD) of RR intervals , histogram of the first difference of RR intervals , minimum of the corrected conditional entropy of RR interval sequence , 8-beat sliding window RR interval irregularity detector , symbolic dynamics and Shannon entropy , sample entropy of RR intervals [23; 24; 25], and normalized fuzzy entropy of RR intervals .
It is worth noting that AF detectors that combine both atrial activity and ventricular response could provide an enhanced performance by combining independent data from each part of the cardiac cycle. Such detection approaches have included: RR interval Markov modeling combined with PR interval variability and a P wave morphology similarity measure  and a fuzzy logic classification method which uses the combination of RR interval irregularity, P-wave absence, f-wave presence, and noise level . It is also worth noting that multivariate approaches based on machine learning that combines several of the above single features can also provide enhanced AF detection [29; 30; 31].
Previous studies concerning AF classification are generally limited in applicability because 1) only classification of normal and AF rhythms were performed, 2) good performance was shown on carefully-selected often clean data, 3) a separate out of sample test dataset was not used, or 4) only a small number of patients were used. It is challenging to reliably detect AF from a single short lead of ECG, and the broad taxonomy of rhythms makes this particularly difficult. In particular, many non-AF rhythms exhibit irregular RR intervals that may be similar to AF. In this Challenge, we treat all non-AF abnormal rhythms as a single class and require the Challenge entrant to classify the rhythms as 1) Normal sinus rhythm, 2) AF, 3) Other rhythm, or 4) Too noisy to classify.
- Download the training set:
training2017.zipand the sample MATLAB entry:
- Create a free PhysioNetWorks account and join the PhysioNet/CinC Challenge 2017 project.
- Develop your entry by making the following edits to
- Modify the sample entry source code file
challenge.mwith your changes and improvements. For additional information, see the Preparing an Entry for the Challenge section.
- Modify the
AUTHORS.txtfile to include the names of all the team members.
training2017.zipand move all its files to the top directory of your entry directory (where
- Run your modified source code file on the validation records in the training set by executing the script
generateValidationSet.m. This will also build a new version of
- Optional: Include a file named
DRYRUNat the top directory of your entry (where the
AUTHORS.txtfile is located) if you do not wish your entry to be scored and counted against your limit. This is useful in cases where you wish to make sure that the changes made do not result in any error.
- Modify the sample entry source code file
- Submit your modified
sample2017.zipfor scoring through the PhysioNet/CinC Challenge 2017 project. The contents of
sample2017.zipmust be laid out exactly as in the sample entry. Improperly-formatted entries will not be scored.
For those wishing to compete officially, please follow the additional four steps described in the Rules and Deadlines.
Join our community Community Discussion Forum to get the latest challenge news, technical help, or if you would like to find partners to collaborate with.
Rules and Deadlines
Participants are asked to classify one lead ECG recordings as normal rhythm, AF, other rhythm or noisy recordings.
Entrants may have an overall total of up to 15 submitted entries over both the unofficial and official phases of the competition (see Table 1). Each participant may receive scores for up to five entries submitted during the unofficial phase and ten entries at the end of the official phase. Unused entries may not be carried over to later phases. Entries that cannot be scored (because of missing components, improper formatting, or excessive run time) are not counted against the entry limits.
All deadlines occur at noon GMT (UTC) on the dates mentioned below. If you do not know the difference between GMT and your local time, find out what it is before the deadline!
|Start at noon GMT on||Entry limit||End at noon GMT on|
|Unofficial Phase||1 February||5||9 April|
|[Hiatus]||9 April||0||16 April|
|Official Phase||16 April||10||1 September|
All official entries must be received no later than noon GMT on Friday, 1 September 2017. In the interest of fairness to all participants, late entries will not be accepted or scored. Entries that cannot be scored (because of missing components, improper formatting, or excessive run time) are not counted against the entry limits.
To be eligible for the open-source award, you must do all of the following:
- Submit at least one open-source entry that can be scored before the Phase I deadline (noon GMT on Friday, 9 April 2017).
- Submit at least one entry during the second phase (between noon GMT on Sunday, 16 April 2017 and noon GMT on Friday, 1 September 2017). Only your final entry will count for ranking.
- Submit an acceptable abstract (about 300 words) on your work on the Challenge to Computing in Cardiology no later than 14 April 2017. Include the overall score for at least one Phase I entry in your abstract. Please select “PhysioNet/CinC Challenge” as the topic of your abstract, so it can be identified easily by the abstract review committee. You will be notified if your abstract has been accepted by email from CinC during the first week in June.
- Submit a full (4-page) paper on your work on the Challenge to CinC no later than the deadline of conference paper submission.
- Attend CinC 2017 (24–27 September 2017) in Rennes, France and present your work there.
Please do not submit analysis of this year’s Challenge data to other Conferences or Journals until after CinC 2017 has taken place, so the competitors are able to discuss the results in a single forum. We expect a special issue from the journal Physiological Measurement to follow the conference and encourage all entrants (and those who missed the opportunity to compete or attend CinC 2017) to submit extended analysis and articles to that issue, taking into account the publications and discussions at CinC 2017.
ECG recordings, collected using the AliveCor device, were generously donated for this Challenge by AliveCor. The training set contains 8,528 single lead ECG recordings lasting from 9 s to just over 60 s (see Table 2) and the test set contains 3,658 ECG recordings of similar lengths. The test set is unavailable to the public and will remain private for the purpose of scoring for the duration of the Challenge and for some period afterwards.
ECG recordings were sampled as 300 Hz and they have been band pass filtered by the AliveCor device. All data are provided in MATLAB V4 WFDB-compliant format (each including a .mat file containing the ECG and a .hea file containing the waveform information). More details of the training set can be seen in Table 2. Figure 1 shows the examples of the ECG waveforms (lasting for 20 s) for the four classes in this Challenge. From top to bottom, they are ECG waveforms of normal rhythm, AF rhythm, other rhythm and noisy recordings.
Please note, since all the classification was performed by a single expert, we are currently in the process of re-scoring a subset where our trust of the data is lowest. We will almost certainly update labels on some of data in both the training and test databases. We may also add new data in the future, although this is unlikely to change after we begin the official phase of the competition.
Please also note that the scoring system currently treats all classes equally. It is likely we will update this for the official phase also. The point of the unnofficial phase is for us to iron out bugs in the competition, and you are a vital part of that. Please send us suggestions/queries to email@example.com. We welcome suggestions for reclassifying a file, but please provide your reasoning.
|Type||# recording||Time length (s)|
As a starting point we have provided an example entry (post here:
sample2017.zip) which provides a state of the art detector based upon the method described by Sarkar et al . We note that this detector provides a classification of only Normal or AF rhythms. We leave it to the Challengers to add the other classes.
You may want to begin with the sample detector, or discard it completely and start from scratch using more data-driven or physiological model-based approaches. Please note that the sample entry has been patented by their original authors and, although we provide an open source version of it for benchmarking, you should not hope to create intellectual property from derivatives of it. We therefore suggest you concentrate on developing alternative methods to this benchmark.
NOTE: You do not need any additional software, apart from Matlab or GNU Octave, to run the sample entry. You can use any programming languages or libraries you like when implementing your own submission.
Preparing an entry for the challenge
To participate in the challenge, you will need to create software that is able to read the test data and output the final classification result without user interaction in our test environment. One sample entry (
sample2017.zip, written in MATLAB) is available to help you get started. In addition to MATLAB, you may use any programming language (or combination of languages) supported using open source compilers or interpreters on GNU/Linux, including C, C++, Fortran, Haskell, Java, Octave, Perl, Python, and R.
If your entry requires software that is not installed in our sandbox environment, we will work with you during Phase I to try to ensure your code can run. We will not modify the test environment after the start of Phase II of the challenge.
Participants should download the sample entry (
sample2017.zip). Entries should have the exact layout of the sample entry; specifically, they must contain:
setup.sh, a bash script runs once before any other code from the entry; use this to compile your code as needed.
next.sh, a bash script runs once per training or test record; it should analyze the record using your code, saving the results as a text file for each record.
dependencies.txt, a text file that lists additional Debian packages that must be installed prior to running your entry’s
answers.txt, a text file containing the results of running your program on each record in the validation set (part of training set, see below for details). These results are used for validation only, not for ranking entries.
AUTHORS.txt, a plain text file listing the members of your team who contributed to your code, and their affiliations.
LICENSE.txt, a text file containing the license for your software (the default is the GPL). All entries are assumed to be open source and will eventually be released on PhysioNet (for closed source entries please see below).
See the comments in the sample entry’s
next.sh if you wish to learn how to customize these scripts for your entry.
We verify that your code is working as you intended, by running it on a small subset (validation set, 300 recordings) of the training set, then comparing the
answers.txt file that you submit with your entry with answers produced by your code running in our test environment using the same records. Using a small portion of the training set means you will know whether your code passed or failed to run within a small time. If your code passes this validation test, it is then evaluated and scored using the hidden test set. The score in the hidden test set determines the ranking of the entries and the final outcome of the Challenge. Note that in the Official Phase of the challenge, more data may be added to both training and hidden test set. Your final entry in the Official Phase of the competition will be run on the entire test set, and so may take much longer than earlier entries.
In addition to the required components, your entry may include a file named
DRYRUN. If this file is present, your entry is not evaluated using the hidden test data, and it will not be counted against your limit of entries per phase; you will receive either a confirmation of success or a diagnostic report, but no scores. Use this feature to verify that none of the required components are missing, that your
setup.sh script works in the test environment, and that your
next.sh script produces the expected output for the training data within the time limits.
Closed Source Entries
Although the competition is only for open source entries, we also accept the submission of closed-source entries from industry or from individuals. If you enter closed source, we will not publish your code or score (unless you specifically request that we do so). However, the default entry is open source (GPL), so you must explicitly indicate that your entry is closed source by including with your entry a file called
CLOSEDSOURCE.txt and modifying
LICENSE.txt accordingly. If you submit an executable, it must be compiled to run in our testing environment (Debian 8.7, amd64.)
Open source entry scores will not be posted until after the close of the Official Phase, and closed source entries will not be posted. You may choose to swap between being open source or closed source at any time up to the end of the Unofficial Phase by inserting or removing the
CLOSEDSOURCE.txt file with your final entry prior to the end of the Unofficial Phase.
More on Licences and IP
We would like to note that the competition does not give the company donating the data any rights to algorithms or ideas developed by competitors. Any entity is free to contact a competitor to request a license to use their code for commercial purposes. Since the competitor must use an open source license to be eligible for a prize, it may be necessary for the competitor to produce another version of the code with a different license (as copyright holders they are at liberty to do so). This would in no way influence the posting of the open source code for the Challenge or its use for research purposes.
If your entry is properly formatted, and nothing is missing, it is tested and scored automatically, and you will receive your scores when the test is complete (depending on your entry’s run time, this may take an hour or more). If you receive an error message instead, read it carefully and correct the problem(s) before resubmitting. Missing answers are treated as noise labels.
The scoring for this challenge uses a measure, which is an average of the four values from each classification type. The counting rules for the numbers of the variables are defined in Table 3 as below:
For each of the four types, is defined as:
- Normal rhythm:
- AF rhythm:
- Other rhythm:
score2017Challenge.m in the sample entry,
sample2017.zip, generates the final challenge score as follows:
The score on the hidden data represents your algorithm’s performance on a subset of the data. We will assess your performance on all of the hidden data only on your final entry in the official phase of the competition. The reason we do this is to prevent you over training on the hidden data. To improve your algorithm, you must assess your algorithm’s performance on the training data using cross validation.
After the Challenge
As is customary, we hope to run a special issue in Physiological Measurement with a closing date of 31 January 2018. We will therefore encourage competitors (and non-competitors) to submit updates and further reworks based on the Challenge after the award ceremony at the Computing in Cardiology Conference in Rennes in September.
Obtaining complimentary MATLAB licenses
The MathWorks has kindly decided to sponsor Physionet’s 2017 Challenge providing both prize money and licenses. The MathWorks is offering to all teams that wish to use MATLAB, complimentary licenses. User can apply for a license and learn more about MATLAB support through The Mathwork’s PhysioNet Challenge link. If you have questions or need technical support, please contact The MathWorks at firstname.lastname@example.org.
Community Discussion Forum
Note: Please check the FAQ below before posting on the Forum.
- V. Fuster, L.E. Ryden, R.W. Asinger, D.S. Cannom, H.J. Crijns, R.L. Frye, J.L. Halperin, N.G. Kay, W.W. Klein, S. Lévy, R.L. McNamara, E.N. Prystowsky, L.S. Wann, D.G. Wyse, R.J. Gibbons, E.M. Antman, J.S. Alpert, D.P. Faxon, V. Fuster, G. Gregoratos, L.F. Hiratzka, A.K. Jacobs, R.O. Russell, S.C. Smith, W.W. Klein, A. Alonso-Garcia, C. Blomström-Lundqvist, G. De Backer, M. Flather, J. Hradec, A. Oto, A. Parkhomenko, S. Silber, A. Torbicki, ACC/AHA/ESC guidelines for the management of patients with atrial fibrillation: executive summary, J Am Coll Cardiol 38 (4) (2001) 1231–1265.
- G.Y.H. Lip, L. Fauchier, S.B. Freedman, I. Van Gelder, A. Natale, C. Gianni, S. Nattel, T. Potpara, M. Rienstra, H. Tse, D.A. Lane, Atrial fibrillation, Nature Reviews Disease Primers 2 (2016) 16016.
- A.J. Camm, P. Kirchhof, G.Y. Lip, U. Schotten, I. Savelieva, S. Ernst, I.C. Van Gelder, N. Al-Attar, G. Hindricks, B. Prendergast, H. Heidbuchel, O. Alfieri, A. Angelini, D. Atar, P. Colonna, R. De Caterina, J. De Sutter, A. Goette, B. Gorenek, M. Heldal, S.H. Hohloser, P. Kolh, J.Y. Le Heuzey, P. Ponikowski, F.H. Rutten, Guidelines for the management of atrial fibrillation: the Task Force for the Management of Atrial Fibrillation of the European Society of Cardiology (ESC), Eur Heart J 31 (19) (2010) 2369–2429.
- R. Colloca. (Sup. by L. Minardi & G. D. Clifford) Implementation and testing of atrial fibrillation detectors for a mobile phone application. M.Sc. Thesis, Politecnico di Milano and University of Oxford, 2013.
- I. Savelieva, J. Camm, Update on atrial fibrillation: part I, Clin Cardiol 31 (2) (2008) 55–62.
- G.V. Naccarelli, H. Varker, J. Lin, K.L. Schulman, Increasing prevalence of atrial fibrillation and flutter in the United States, Am J Cardiol 104 (11) (2009) 1534–1539.
- A. Petrėnas, V. Marozas, L. Sörnmo, A. Lukosevicius, An echo state neural network for QRST cancellation during atrial fibrillation, IEEE Trans Biomed Eng 59 (10) (2012) 2950–2957.
- S. Ladavich, B. Ghoraani, Rate-independent detection of atrial fibrillation by statistical modeling of atrial activity, Biomed Signal Process Control 18 (4) (2015) 274–281.
- X. Du, N. Rao, M. Qian, D. Liu, J. Li, W. Feng, L. Yin, X. Chen, A novel method for real-time atrial fibrillation detection in electrocardiograms using multiple parameters, Ann Noninvasive Electrocardiol 19 (3) (2014) 217–225.
- H. Pürerfellner, E. Pokushalov, S. Sarkar, J. Koehler, R. Zhou, L. Urban, G. Hindricks, P-wave evidence as a method for improving algorithm to detect atrial fibrillation in insertable cardiac monitors, Heart Rhythm 11 (9) (2014) 1575–1583.
- R. Alcaraz, C. Vaya, R. Cervigon, C. Sanchez, J.J. Rieta. Wavelet sample entropy: A new approach to predict termination of atrial fibrillation. In: Computing in Cardiology, ed A Murray: IEEE) 2006, pp. 597–600.
- J. Ródenas, M. García, R. Alcaraz, J.J. Rieta, Wavelet entropy automatically detects episodes of atrial fibrillation from single-lead electrocardiograms, Entropy 17 (9) (2015) 6179–6199.
- M. García, J. Ródenas, R. Alcaraz, J.J. Rieta, Application of the relative wavelet energy to heart rate independent detection of atrial fibrillation, Comput Methods Programs Biomed 131 (7) (2016) 157–168.
- M. Carrara, L. Carozzi, T.J. Moss, M. De Pasquale, S. Cerutti, M. Ferrario, D.E. Lake, J.R. Moorman, Heart rate dynamics distinguish among atrial fibrillation, normal sinus rhythm and sinus rhythm with frequent ectopy, Physiol Meas 36 (9) (2015) 1873–1888.
- J. Park, S. Lee, M. Jeon, Atrial fibrillation detection by heart rate variability in Poincare plot, Biomed Eng Online 8 (2009) 38.
- S. Sarkar, D. Ritscher, R. Mehra, A detector for a chronic implantable atrial tachyarrhythmia monitor, IEEE Trans Biomed Eng 55 (3) (2008) 1219–1224.
- K. Tateno, L. Glass, Automatic detection of atrial fibrillation using the coefficient of variation and density histograms of RR and deltaRR intervals, Med Biol Eng Comput 39 (6) (2001) 664–671.
- D.T. Linker. Long-term monitoring for detection of atrial fibrillation. ed U Patent (USA 2009,
- C. Huang, S. Ye, H. Chen, D. Li, F. He, Y. Tu, A novel method for detection of the transition between atrial fibrillation and sinus rhythm, IEEE Trans Biomed Eng 58 (4) (2011) 1113–1119.
- S. Cerutti, L. Mainardi, L. Sörnmo. Understanding atrial fibrillation: the signal processing contribution.2008: Morgan and Claypool Publishers).
- A. Petrėnas, V. Marozas, L. Sörnmo, Low-complexity detection of atrial fibrillation in continuous long-term monitoring, Comput Biol Med 65 (10) (2015) 184–191.
- X. Zhou, H. Ding, B. Ung, E. Pickwell-MacPherson, Y.T. Zhang, Automatic online detection of atrial fibrillation based on symbolic dynamics and Shannon entropy, Biomed Eng Online 13 (1) (2014) 18.
- R. Alcaraz, D. Abásolo, R. Hornero, J.J. Rieta, Optimal parameters study for sample entropy-based atrial fibrillation organization analysis, Comput Methods Programs Biomed 99 (1) (2010) 124–132.
- D.E. Lake, J.R. Moorman, Accurate estimation of entropy in very short physiological time series: the problem of atrial fibrillation detection in implanted ventricular devices, Am J Physiol Heart Circ Physiol 300 (1) (2011) H319-H325.
- D. DeMazumder, D.E. Lake, A. Cheng, T.J. Moss, E. Guallar, R.G. Weiss, S.R. Jones, G.F. Tomaselli, J.R. Moorman, Dynamic analysis of cardiac rhythms for discriminating atrial fibrillation from lethal ventricular arrhythmias, Circ Arrhythm Electrophysiol 6 (3) (2013) 555–561.
- C.Y. Liu, J. Oster, E. Reinertsen, Q. Li, L.N. Zhao, S. Nemati, G.D. Clifford, A review of methods of entropy for atrial fibrillation detection, BioMedical Engineering OnLine 16 (2017). Under review.
- S. Babaeizadeh, R.E. Gregg, E.D. Helfenbein, J.M. Lindauer, S.H. Zhou, Improvements in atrial fibrillation detection for real-time monitoring, J Electrocardiol 42 (6) (2009) 522–526.
- A. Petrėnas, L. Sörnmo, A. Lukoševicius, V. Marozas, Detection of occult paroxysmal atrial fibrillation, Med Biol Eng Comput 53 (4) (2015) 287–297.
- J. Oster, G.D. Clifford, Impact of the presence of noise on RR interval-based atrial fibrillation detection, J Electrocardiol 48 (6) (2015) 947–951.
- R. Colloca, A.E.W. Johnson, L. Mainardi, G.D. Clifford. A support vector machine approach for reliable detection of atrial fibrillation events. In: Computing in Cardiology, ed A Murray (Zaragoza, Spain 2013, 1047–1050.
- Q. Li, C.Y. Liu, J. Oster, G.D. Clifford. Signal processing and feature selection preprocessing for classification in noisy healthcare data. In: Machine Learning for Healthcare Technologies. ed D.A. Clifton (London: The Institution of Engineering and Technology, 2016), 33-58.
If you would like help understanding, using, or downloading content, please see our Frequently Asked Questions.
If you have any comments, feedback, or particular questions regarding this page, please send them to the webmaster.
Comments and issues can also be raised on PhysioNet's GitHub page.
Updated Tuesday, 07-Feb-2017 13:56:40 CET