Over 60 teams participated in the 2013 Challenge on noninvasive fetal ECG. Below are the scores of the top-ranked teams. (Only 6 teams participated in event 3, and only 2 produced entries that could be scored.) Each team was allowed to submit up to 11 entries; the scores shown below are the best (lowest) received by each team for any of their entries.
Event 1 (Fetal heart rate estimation, evaluated using hidden test set C)
The score for each record was the mean squared error (in bpm2) between the fetal heart rate signals estimated from the reference and test annotations. The aggregate score was the mean of the scores for each record.
| Participant | Score |
|---|---|
| Maurizio Varanini, Gennaro Tartarisco, Lucia Billeci, Alberto Macerata, Giovanni Pioggia and Rita Balocchi | 187.091 |
| Piotr Podziemski and Jan Gierałtowski | 255.989 |
| Rui Rodrigues | 278.755 |
| Luigi Yuri Di Marco, Alberto Marzo and Alejandro Frangi | 380.853 |
| Jakub Kuzilek and Lenka Lhotska | 492.412 |
| Joseph McBride, Brent McFerrin, Craig Towers and Xiaopeng Zhao (unofficial) | 505.634 |
| Aruna Deogire and Satish Hamde (unofficial) | 580.284 |
| Alessia Dessì, Danilo Pani and Luigi Raffo | 684.158 |
| Tian Wenlong (unofficial) | 931.156 |
| Joachim Behar, Julien Oster and Gari Clifford (unofficial) | (179.439) |
Event 2 (Fetal RR interval estimation, evaluated using hidden test set C)
The score for each record was the root mean squared difference (in ms) between corresponding reference and test RR intervals. The aggregate score was the mean of the scores for each record.
| Participant | Score |
|---|---|
| Maurizio Varanini, Gennaro Tartarisco, Lucia Billeci, Alberto Macerata, Giovanni Pioggia and Rita Balocchi | 20.975 |
| Piotr Podziemski and Jan Gierałtowski | 25.059 |
| Luigi Yuri Di Marco, Alberto Marzo and Alejandro Frangi | 26.085 |
| Rui Rodrigues | 28.201 |
| Joseph McBride, Brent McFerrin, Craig Towers and Xiaopeng Zhao (unofficial) | 29.680 |
| Jakub Kuzilek and Lenka Lhotska | 33.200 |
| Akshay Dhawan (unofficial) | 39.667 |
| Andrea Fanelli, Giovanni Magenes and Maria Gabriella Signorini (unofficial) | 42.859 |
| Alessia Dessì, Danilo Pani and Luigi Raffo | 47.990 |
| Joachim Behar, Julien Oster and Gari Clifford (unofficial) | (20.793) |
Event 3 (Fetal QT interval estimation, evaluated using hidden test set C)
For each record, the median FQT interval was estimated. The aggregate score was the root mean squared difference (in ms) between the reference and test FQT intervals for all records.
| Participant | Score |
|---|---|
| Piotr Podziemski and Jan Gierałtowski | 152.71 |
| Joachim Behar, Julien Oster and Gari Clifford (unofficial) | (153.07) |
Event 4 (Fetal heart rate estimation, evaluated using open test set B)
The score for each record was the mean squared error (in bpm2) between the fetal heart rate signals estimated from the reference and test annotations. The aggregate score was the mean of the scores for each record.
| Participant | Score |
|---|---|
| Fernando Andreotti, Maik Riedl, Tilo Himmelsbach, Daniel Wedekind, Sebastian Zaunseder, Niels Wessel and Hagen Malberg | 18.803 |
| Jukka A Lipponen and Mika P Tarvainen | 28.893 |
| Maurizio Varanini, Gennaro Tartarisco, Lucia Billeci, Alberto Macerata, Giovanni Pioggia and Rita Balocchi | 33.952 |
| Masoumeh Haghpanahi and David A Borkholder | 50.063 |
| Minnan Xu-Wilson, Eric Carlson, Limei Cheng and Srinivasan Vairavan | 52.496 |
| Ali Ghaffari, Seyyed Abbas Atyabi, Mohammad Javad Mollakazemi, Maryam Niknami and Ali Soleiman | 63.750 |
| Mantas Lukoševičius and Vaidotas Marozas | 66.327 |
| Martin Kropf, Günter Schreier, Robert Modre-Osprian and Dieter Hayn | 82.438 |
| Joachim Behar, Julien Oster and Gari Clifford (unofficial) | (29.619) |
Event 5 (Fetal RR interval estimation, evaluated using open test set B)
The score for each record was the root mean squared difference (in ms) between corresponding reference and test RR intervals. The aggregate score was the mean of the scores for each record.
| Participant | Score |
|---|---|
| Fernando Andreotti, Maik Riedl, Tilo Himmelsbach, Daniel Wedekind, Sebastian Zaunseder, Niels Wessel and Hagen Malberg | 4.337 |
| Jukka A Lipponen and Mika P Tarvainen | 4.844 |
| Maurizio Varanini, Gennaro Tartarisco, Lucia Billeci, Alberto Macerata, Giovanni Pioggia and Rita Balocchi | 5.098 |
| Martin Kropf, Günter Schreier, Robert Modre-Osprian and Dieter Hayn | 7.354 |
| Mantas Lukoševičius and Vaidotas Marozas | 8.239 |
| Masoumeh Haghpanahi and David A Borkholder | 9.062 |
| Christoph Maier and Hartmut Dickhaus | 9.326 |
| Minnan Xu-Wilson, Eric Carlson, Limei Cheng and Srinivasan Vairavan | 10.618 |
| Vito Starc | 10.920 |
| Joachim Behar, Julien Oster and Gari Clifford (unofficial) | (4.672) |
How the entries were scored
Entries were scored using tach, mxm, ann2rr, and custom software developed for the challenge. For each event, the software processed the reference annotations and the test annotations (i.e., those generated by the participant’s entry) to obtain a score for each test record, then calculated an average score for all records in the test set.
Participants had access to training set A (both signals and reference annotations) and open test set B (signals only); these data sets remain freely available. Reference annotations for set B, and both the signals and annotations for hidden test set C, will be kept hidden, to allow fair comparisons between entries submitted during the challenge and those developed in followup studies by challenge participants as well as other interested researchers.
Note on unofficial participants
Teams listed above as "unofficial" participated in the Challenge and achieved excellent results.
Entries from the team of Joachim Behar, Julien Oster, and Gari Clifford were unofficial since their authors were among the challenge organizers, and had major roles in developing the challenge questions and data sets.
Other teams listed as unofficial were unable to attend CinC 2013 and discuss their work, a condition of eligibility for Challenge awards.