9. Supported diagnoses, clinical safety, and performance

A clinical evaluation report (CER) assessed the clinical performance, safety and benefit of the medical device software PMcardio manufactured by POWERFUL MEDICAL s.r.o. The PMcardio mobile application is intended to be used by qualified healthcare professionals, specifically in the primary care setting for the assessment of cardiovascular diseases using ECG data and disease specific patient history forms in subjects over 18 years of age. The PMcardio mobile application provides ECG signal digitization, processing, analysis and detection of 39 diagnostic categories consisting of rhythms, arrhythmias, heart blocks, infarctions and others. PMcardio is classified as a IIb device according to MDR Annex VIII.

In summary, the potential benefits for users and patients outweigh the potential risks. The risk-benefit ratio is positive if PMcardio is used in accordance with the intended purpose and the instructions for use of PMcardio. This clinical evaluation confirms that PMcardio complies with the general safety and performance requirements (GSPR) of the MDR.

PMcardio provides detection or classification of 39 diagnostic categories. The term classification means that the goal of the process is to attribute the correct label to each data instance (ECG); the process itself is known as the classifier, or classification algorithm. One crucial aspect naturally arises concerning the outcome of the classification process: how should the classifier performance be evaluated?

The previous IEC 60601-2-51 standard stated that manufacturers of ECG analysis programs and equipment should report the sensitivity, specificity, and positive predictive accuracy of the interpretive statements for each of the major diagnostic categories (see 60601-2-51(c) IEC 2003). However, for unclear reasons, these requirements for ECG interpretation (the old clause 50.102 in IEC 60601-2-51) have been completely removed from the updated IEC 60601-2- 252011 standard. As such, this document will provide several statistical measures to report the diagnostic performance of PMcardio

Several statistical rates are available to evaluate (binary) classifications and their confusion matrices. Despite being an essential trait in machine learning, no widespread consensus has been reached on a unified elective chosen measure yet. Accuracy and F1 score computed on confusion matrices have been (and still are) among the most popular adopted metrics in binary classification tasks. However, these statistical measures can dangerously show overoptimistic inflated results, especially on imbalanced datasets (2018 Dec 22. In: Kubben P, Dumontier M, Dekker A, editors. Fundamentals of Clinical Data Science [Internet]. Cham (CH): Springer; 2019. Chapter 8.).

To address this issue, we evaluated our results using a more reliable statistical metric that yields a high score only if the prediction performed well in all categories of the confusion matrix (true positives, false negatives, true negatives, and false positives), proportional to both the size of the positive elements and the size of the negative elements in the data set – the Matthews Correlation Coefficient (MCC) (Chicco and Jurman BMC Genomics (2020) 216 https://doi.org/10.1186/s12864-019-6413-7). In addition, for completeness, we also report the PPV, NPV, Specificity and Sensitivity for the 39 supported diagnoses.

To summarize, the following list of metrics is used for the evaluation of the performance:

Sensitivity

The ability of a test to correctly identify patients with a specified disease.

Specificity

The ability of a test to correctly identify people without a specified disease.

Positive Predictive Value (PPV)

The positive predictive value is the probability that following a positive test result, that individual will truly have the specified disease.

Negative Predictive Value (PPV)

The negative predictive value is the probability that following a negative test result, that individual will truly not have the specified disease.

Matthews correlation coefficient (MCC)

A highly reliable statistical rate which produces a high score only if the prediction obtained good results in all of the four confusion matrix categories (true positives, false negatives, true negatives, and false positives), proportionally both to the size of positive elements and the size of negative elements in the dataset.

After the conversion of the diagnostic patterns to diagnoses, reporting the AUC metric is no longer possible since it cannot be calculated for boolean outputs that result from the combination of measurements and diagnostic patterns. Therefore, the reported performance metrics are PPV, NPV, Sensitivity, Specificity, and MCC. The metric is reported with the associated confidence intervals (CI) for each metric.

9.1. Supported diagnostic patterns detected by the AI-algorithms 

Pattern	PPV - 2.5 s	NPV - 2.5 s	Sensitivity - 2.5 s	Specificity - 2.5 s	MCC - 2.5 s	F1 - 2.5 s	PPV - 5 s	NPV - 5 s	Sensitivity - 5 s	Specificity - 5 s	MCC - 5 s	F1 - 5 s
Sinus rhythm	0.923 (0.908-0.938)	0.995 (0.991-0.999)	0.99 (0.984-0.996)	0.959 (0.948-0.97)	0.933 (0.925-0.94)	0.955 (0.943-0.967)	0.932 (0.918-0.946)	0.992 (0.987-0.997)	0.985 (0.978-0.992)	0.964 (0.953-0.975)	0.936 (0.928-0.943)	0.958 (0.946-0.97)
Paced rhythm	0.965 (0.954-0.976)	0.973 (0.964-0.982)	0.913 (0.897-0.929)	0.99 (0.984-0.996)	0.92 (0.911-0.928)	0.938 (0.924-0.952)	0.967 (0.957-0.977)	0.983 (0.976-0.99)	0.945 (0.932-0.958)	0.99 (0.984-0.996)	0.942 (0.935-0.948)	0.956 (0.944-0.968)
Atrial fibrillation	0.92 (0.904-0.936)	0.993 (0.988-0.998)	0.964 (0.953-0.975)	0.983 (0.976-0.99)	0.93 (0.922-0.937)	0.941 (0.927-0.955)	0.984 (0.977-0.991)	0.993 (0.988-0.998)	0.964 (0.953-0.975)	0.997 (0.994-1.0)	0.969 (0.965-0.972)	0.974 (0.965-0.983)
Atrial flutter	0.973 (0.964-0.982)	0.987 (0.98-0.994)	0.918 (0.902-0.934)	0.996 (0.992-1.0)	0.937 (0.93-0.944)	0.945 (0.932-0.958)	0.975 (0.966-0.984)	0.995 (0.991-0.999)	0.969 (0.959-0.979)	0.996 (0.992-1.0)	0.967 (0.963-0.971)	0.972 (0.963-0.981)
Other rhythm	0.931 (0.916-0.946)	0.977 (0.968-0.986)	0.836 (0.815-0.857)	0.991 (0.986-0.996)	0.866 (0.851-0.88)	0.881 (0.862-0.9)	0.94 (0.926-0.954)	0.98 (0.972-0.988)	0.856 (0.836-0.876)	0.992 (0.987-0.997)	0.883 (0.87-0.895)	0.896 (0.878-0.914)
Premature complex	0.987 (0.975-0.999)	0.871 (0.837-0.905)	0.853 (0.817-0.889)	0.989 (0.978-1.0)	0.85 (0.819-0.876)	0.915 (0.887-0.943)	0.994 (0.986-1.0)	0.958 (0.938-0.978)	0.957 (0.936-0.978)	0.995 (0.988-1.0)	0.952 (0.941-0.961)	0.975 (0.959-0.991)
2nd Degree AV Block, type Wenckebach	1.0 (1.0-1.0)	0.855 (0.824-0.886)	0.184 (0.15-0.218)	1.0 (1.0-1.0)	0.396 (0.32-0.467)	0.311 (0.271-0.351)	0.857 (0.826-0.888)	0.951 (0.932-0.97)	0.759 (0.722-0.796)	0.974 (0.96-0.988)	0.769 (0.731-0.802)	0.805 (0.77-0.84)
Higher degree AV blocks	0.716 (0.677-0.755)	0.899 (0.873-0.925)	0.843 (0.811-0.875)	0.806 (0.771-0.841)	0.631 (0.575-0.681)	0.774 (0.737-0.811)	0.892 (0.865-0.919)	0.851 (0.82-0.882)	0.714 (0.675-0.753)	0.95 (0.931-0.969)	0.702 (0.655-0.744)	0.793 (0.758-0.828)
Right Bundle Branch Block	0.937 (0.916-0.958)	1.0 (1.0-1.0)	1.0 (1.0-1.0)	0.967 (0.952-0.982)	0.952 (0.943-0.959)	0.967 (0.952-0.982)	0.903 (0.878-0.928)	0.994 (0.987-1.0)	0.989 (0.98-0.998)	0.947 (0.928-0.966)	0.916 (0.901-0.929)	0.944 (0.925-0.963)
Left Bundle Branch Block	0.989 (0.98-0.998)	0.991 (0.983-0.999)	0.984 (0.973-0.995)	0.994 (0.987-1.0)	0.98 (0.976-0.983)	0.987 (0.977-0.997)	0.989 (0.98-0.998)	0.969 (0.954-0.984)	0.942 (0.922-0.962)	0.994 (0.987-1.0)	0.947 (0.938-0.955)	0.965 (0.949-0.981)
Left Anterior Fascicular Block	0.944 (0.923-0.965)	0.94 (0.919-0.961)	0.871 (0.841-0.901)	0.975 (0.961-0.989)	0.865 (0.84-0.886)	0.906 (0.88-0.932)	0.953 (0.934-0.972)	0.957 (0.939-0.975)	0.91 (0.884-0.936)	0.978 (0.965-0.991)	0.899 (0.88-0.915)	0.931 (0.908-0.954)
Left Posterior Fascicular Block	0.963 (0.946-0.98)	0.984 (0.973-0.995)	0.969 (0.953-0.985)	0.981 (0.969-0.993)	0.949 (0.939-0.957)	0.966 (0.95-0.982)	0.958 (0.94-0.976)	0.925 (0.901-0.949)	0.846 (0.814-0.878)	0.981 (0.969-0.993)	0.854 (0.828-0.877)	0.898 (0.871-0.925)
Atrial enlargement	0.981 (0.968-0.994)	0.921 (0.895-0.947)	0.959 (0.94-0.978)	0.962 (0.943-0.981)	0.912 (0.894-0.927)	0.97 (0.953-0.987)	0.968 (0.951-0.985)	0.984 (0.972-0.996)	0.993 (0.985-1.0)	0.932 (0.907-0.957)	0.938 (0.925-0.949)	0.98 (0.966-0.994)
Suspected ventricular hypertrophy	0.934 (0.907-0.961)	0.986 (0.973-0.999)	0.988 (0.976-1.0)	0.924 (0.895-0.953)	0.916 (0.897-0.932)	0.96 (0.939-0.981)	0.933 (0.906-0.96)	0.973 (0.955-0.991)	0.977 (0.961-0.993)	0.924 (0.895-0.953)	0.903 (0.881-0.921)	0.954 (0.931-0.977)
ST-Elevation ACS	0.919 (0.892-0.946)	0.982 (0.969-0.995)	0.975 (0.96-0.99)	0.939 (0.915-0.963)	0.907 (0.888-0.923)	0.946 (0.924-0.968)	0.899 (0.869-0.929)	0.995 (0.988-1.0)	0.994 (0.986-1.0)	0.922 (0.895-0.949)	0.905 (0.885-0.921)	0.944 (0.921-0.967)
NonST-Elevation ACS	0.946 (0.924-0.968)	0.926 (0.9-0.952)	0.8 (0.76-0.84)	0.982 (0.969-0.995)	0.826 (0.792-0.855)	0.867 (0.833-0.901)	0.948 (0.926-0.97)	0.936 (0.912-0.96)	0.827 (0.79-0.864)	0.982 (0.969-0.995)	0.846 (0.815-0.872)	0.883 (0.851-0.915)
Extreme R-axis deviation	1.0 (1.0-1.0)	0.947 (0.922-0.972)	0.741 (0.692-0.79)	1.0 (1.0-1.0)	0.837 (0.8-0.868)	0.851 (0.811-0.891)	0.981 (0.966-0.996)	0.996 (0.989-1.0)	0.981 (0.966-0.996)	0.996 (0.989-1.0)	0.977 (0.971-0.982)	0.981 (0.966-0.996)
Left R-axis deviation	0.923 (0.893-0.953)	0.915 (0.884-0.946)	0.791 (0.745-0.837)	0.972 (0.953-0.991)	0.8 (0.755-0.837)	0.852 (0.812-0.892)	0.986 (0.973-0.999)	0.917 (0.886-0.948)	0.791 (0.745-0.837)	0.995 (0.987-1.0)	0.843 (0.807-0.873)	0.878 (0.841-0.915)
Normal R-axis	0.82 (0.777-0.863)	0.983 (0.968-0.998)	0.971 (0.952-0.99)	0.889 (0.854-0.924)	0.831 (0.792-0.863)	0.889 (0.854-0.924)	0.816 (0.772-0.86)	0.994 (0.985-1.0)	0.99 (0.979-1.0)	0.884 (0.848-0.92)	0.842 (0.806-0.872)	0.895 (0.86-0.93)
Right R-axis deviation	0.823 (0.78-0.866)	0.988 (0.976-1.0)	0.944 (0.918-0.97)	0.956 (0.933-0.979)	0.854 (0.82-0.882)	0.879 (0.842-0.916)	0.98 (0.964-0.996)	0.98 (0.964-0.996)	0.907 (0.874-0.94)	0.996 (0.989-1.0)	0.931 (0.914-0.945)	0.942 (0.916-0.968)
Occlusion myocardial infarction	0.822 (0.796-0.847)	0.893 (0.88-0.906)	0.742 (0.714-0.769)	0.931 (0.92-0.941)	0.693 (0.666-0.721)	0.78 (0.759-0.8)	Same as 2.5 s	Same as 2.5 s	Same as 2.5 s	Same as 2.5 s	Same as 2.5 s	Same as 2.5 s

9.2. Supported ECG measurements detected by the AI-algorithms

Measurement	Mean Difference (ms) - 2.5 s	Standard Deviation (ms) - 2.5 s	Evaluation - 2.5 s	Mean Difference (ms) - 5 s	Standard Deviation (ms) - 5 s	Evaluation - 5 s
PDuration	4.087	6.112	Pass	4.207	6.488	Pass
PRInterval	-2.337	7.288	Pass	-3.12	7.197	Pass
QRSDuration	2.022	5.668	Pass	1.141	6.334	Pass
QTInterval	-2.076	11.06	Pass	-5.304	11.291	Pass
RRInterval	1.685	13.377	Pass	-0.098	7.57	Pass

Measurement name	Definition
P Wave	PDuration if none of afib_p, aflut_p, otherhy_p Patterns predicted, otherwise 0.
Heart Rate	60000/RRInterval if nonzero RRInterval, otherwise 0.
PP Interval	rr_interval if none of afib_p, aflut_p, otherhy_p, avblock2w_p, avblockhd_p Patterns predicted, otherwise 0.
QTc Time	QTInterval + 0.154*(1000 - RR interval in milliseconds) (based on the Framingham formula (Sagie A, Larson MG, Goldberg RJ, Bengtson JR, Levy D. An improved method for adjusting the QT interval for heart rate (the Framingham Heart Study). Am J Cardiol. 1992 Sep 15;70(7):797-801. doi: 10.1016/0002-9149(92)90562-d. PMID: 1519533.)).

9.3. Supported diagnoses based on a combination of the above

Diagnosis	Explanation
Sinus bradycardia	Sinus bradycardia diagnosis is a combination of the Sinus rhythm diagnostic pattern, and low heart rate prediction derived from the RRInterval measurement.
Sinus rhythm	Sinus rhythm diagnosis is a combination of the Sinus rhythm diagnostic pattern, and standard (not high or low) heart rate prediction derived from the RRInterval measurement.
Sinus tachycardia	Sinus tachycardia diagnosis is a combination of the Sinus rhythm diagnostic pattern, and high heart rate prediction derived from the RRInterval measurement.
Paced rhythm	Paced rhythm diagnosis is derived from the Paced rhythm diagnostic pattern.
Atrial fibrillation	Atrial fibrillation diagnosis is a combination of the Atrial fibrillation diagnostic pattern, and standard (not high or low) heart rate prediction derived from the RRInterval measurement.
Atrial fibrillation - rapid	Atrial fibrillation - rapid diagnosis is a combination of the Atrial fibrillation diagnostic pattern, and high heart rate prediction derived from the RRInterval measurement.
Atrial fibrillation - slow	Atrial fibrillation - slow diagnosis is a combination of the Atrial fibrillation diagnostic pattern, and low heart rate prediction derived from the RRInterval measurement.
Atrial flutter	Atrial flutter diagnosis is a combination of the Atrial flutter diagnostic pattern, and standard (not high or low) heart rate prediction derived from the RRInterval measurement.
Atrial flutter - rapid	Atrial flutter - rapid diagnosis is a combination of the Atrial flutter diagnostic pattern, and high heart rate prediction derived from the RRInterval measurement.
Atrial flutter - slow	Atrial flutter - slow diagnosis is a combination of the Atrial flutter diagnostic pattern, and low heart rate prediction derived from the RRInterval measurement.
Supraventricular tachycardia	Supraventricular tachycardia diagnosis is a combination of the Other rhythm (Ventricular/Junctional) diagnostic pattern, heart rate threshold derived from the RRInterval measurement, and QRSDuration threshold.
Suspected junctional rhythm	Suspected junctional rhythm diagnosis is a combination of the Other rhythm (Ventricular/Junctional) diagnostic pattern, standard (not high or low) heart rate derived from the RRInterval measurement, and QRSDuration threshold.
Suspected junctional bradycardia	Suspected junctional bradycardia diagnosis is a combination of the Other rhythm (Ventricular/Junctional) diagnostic pattern, low heart rate derived from the RRInterval measurement, and QRSDuration threshold.
Suspected accelerated junctional rhythm	Suspected accelerated junctional rhythm diagnosis is a combination of the Other rhythm (Ventricular/Junctional) diagnostic pattern, high heart rate derived from the RRInterval measurement, and QRSDuration threshold.
Wide QRS rhythm	Wide QRS rhythm diagnosis is a combination of the Other rhythm (Ventricular/Junctional) diagnostic pattern, standard (not high or low) heart rate derived from the RRInterval measurement, and QRSDuration threshold.
Idioventricular rhythm	Idioventricular rhythm diagnosis is a combination of the Other rhythm (Ventricular/Junctional) diagnostic pattern, low heart rate derived from the RRInterval measurement, and QRSDuration threshold.
Wide QRS tachycardia	Wide QRS tachycardia diagnosis is a combination of the Other rhythm (Ventricular/Junctional) diagnostic pattern, high heart rate derived from the RRInterval measurement, and QRSDuration threshold.
Premature complex	Premature complexes diagnosis is derived from the Premature complex diagnostic pattern.
1st degree AV block	1st degree AV block diagnosis is derived from the PRInterval measurement threshold.
2nd degree AV block, type Wenckebach	2nd degree AV block, type Wenckebach diagnosis is derived from the 2nd degree AV block, type Wenckebach diagnostic pattern.
Higher degree AV block	Higher degree AV block diagnosis is derived from the Higher degree AV block diagnostic pattern.
Complete right bundle branch block	Complete right bundle branch block diagnosis is derived from the Right Bundle Branch Block diagnostic pattern and QRSDuration measurement threshold.
Incomplete right bundle branch block	Incomplete right bundle branch block diagnosis is derived from the Right Bundle Branch Block diagnostic pattern and QRSDuration measurement threshold.
Complete left bundle branch block	Complete left bundle branch block diagnosis is derived from the Left Bundle Branch Block diagnostic pattern and QRSDuration measurement threshold.
Incomplete left bundle branch block	Incomplete left bundle branch block diagnosis is derived from the Left Bundle Branch Block diagnostic pattern and QRSDuration measurement threshold.
Nonspecific intraventricular conduction delay	Nonspecific intraventricular conduction delay diagnosis is derived from the QRSDuration measurement threshold.
Left anterior fascicular block	Left anterior fascicular block diagnosis is derived from the Left anterior fascicular block diagnostic pattern.
Left posterior fascicular block	Left posterior fascicular block diagnosis is derived from the Left posterior fascicular block diagnostic pattern.
Bifascicular block (RBBB + LAFB)	Bifascicular block (RBBB + LAFB) diagnosis is derived from the Right Bundle Branch Block diagnostic pattern, Left anterior fascicular block diagnostic pattern, and QRSDuration measurement threshold.
Bifascicular block (RBBB + LPFB)	Bifascicular block (RBBB + LAFB) diagnosis is derived from the Right Bundle Branch Block diagnostic pattern, Left posterior fascicular block diagnostic pattern, and QRSDuration measurement threshold.
Trifascicular block (RBBB + LAFB + AVBLOCK1)	Trifascicular block (RBBB + LAFB + AVBLOCK1) diagnosis is derived from the Right Bundle Branch Block diagnostic pattern, Left anterior fascicular block diagnostic pattern, PRInterval measurement threshold and QRSDuration measurement threshold.
Trifascicular block (RBBB + LPFB + AVBLOCK1)	Trifascicular block (RBBB + LPFB + AVBLOCK1) is derived from the Right Bundle Branch Block diagnostic pattern, Left posterior fascicular block diagnostic pattern, PRInterval measurement threshold and QRSDuration measurement threshold.
Suspected long QT syndrome	Suspected long QT syndrome diagnosis is derived from the QTc Time measurement threshold.
Suspected short QT syndrome	Suspected short QT syndrome diagnosis is derived from the QTc Time measurement threshold.
Suspected atrial enlargement	Suspected atrial enlargement diagnosis is derived from the Atrial enlargement diagnostic pattern and Sinus rhythm diagnostic pattern.
Suspected ventricular hypertrophy	Suspected ventricular hypertrophy diagnosis is derived from the Ventricular hypertrophy diagnostic pattern.
Suspected ST-elevation ACS	Suspected ST-elevation ACS diagnosis is derived from the ST-elevation ACS diagnostic pattern.
Suspected NonST-elevation ACS	Suspected NonST-elevation ACS diagnosis is derived from the NonST-elevation ACS diagnostic pattern.
Occlusion myocardial infarction*	Occlusion myocardial infarction diagnosis is derived from the Occlusion myocardial infarction diagnostic pattern.

* Available only in Plus (OMI) Plan

9.4. Unsupported diagnoses by the AI-algorithms

• Asystole
• Movement artefact
• Suspected Electrode Reversal
• Sino-Atrial Exit Block (and its different types)
• Sick sinus syndrome
• Sinus pause
• Digitalis intoxication
• Pericarditis
• Pericardial effusion/tamponade
• Myocarditis, Pulmonary embolism
• Brugada syndrome
• Bundgaard syndrome
• Ashman Phenomenon
• Electrolyte imbalances (including: Hyperkalemia, Hypokalemia, Hypercalcemia, Hypocalcemia, Hypermagnesemia and Hypomagnesemia)
• Hyperthyroidism
• Hypothyroidism
• Hypothermia (Osborn wave)
• Raised intracranial pressure
• Arrhythmogenic cardiomyopathy (ACM, arrhythmogenic right ventricular cardiomyopathy (ARVC) - epsilon waves)
• Tako Tsubo cardiomyopathy
• Wellens syndrome
• De Winter ST-T
• Low QRS voltage
• Poor R-wave progression
• Persistent S-waves
• Dextrocardia
• Other medication intoxication
• Any other not explicitly supported diagnosis

9.5. Clinical benefits of PMcardio

ID	Clinical Benefit
CB1	Patients receive prompt ECG diagnostic recommendations from a spectrum of 39 cardiac pathologies while in a primary or emergency care setting.
CB2	Patients receive diagnostic recommendations with a high degree of accuracy.
CB3	Patients receive improved ECG diagnostics at the primary care level.
CB4	Patients receive appropriate, guideline-compliant clinical management recommendations for diagnoses detected from their 12-lead ECG.
CB5	Patients receive an in-depth, analysis of their 12-lead ECGs in the context of their clinical symptoms.
CB6	Patients receive clinical management recommendations according to the most up-to-date clinical practice guidelines, already at the first point of contact.