AI Governance in Practice: Implementing ISO/IEC 42001 for Medical ML
How ISO/IEC 42001 and EU AI Act principles can be translated into practical controls for medical machine learning systems.
How we integrated the ISO/IEC 42001 standard and EU AI Act constraints into a medical machine learning educational platform.
As machine learning systems transition into clinical diagnostic workflows, they face rigorous regulatory scrutiny:
- The EU AI Act (Regulation 2024/1689): Classifies medical decision-support software as high-risk, mandating detailed records of data quality, bias audits, and explainability.
- ISO/IEC 42001:2023: Specifies standard controls for establishing an AI Management System (AIMS).
During my work on DrAI—an educational ML platform for healthcare professionals funded by the EU Erasmus+ program—I served as the QA and Documentation Lead. My primary goal was to implement the ISO/IEC 42001 standard and the EU AI Act's risk management principles directly into our software design.
Here is how we translated policy constraints into software features.
1. Governance Roles (ISO 42001 Clause 5.3)
ISO 42001 requires establishing lines of organizational accountability. We mapped Scrum roles directly to compliance roles:
┌───────────────────────────────────────────────┐
│ Scrum / Compliance Mapping │
├───────────────────────┬───────────────────────┤
│ Scrum Role │ ISO 42001 Role │
├───────────────────────┼───────────────────────┤
│ Product Owner │ AI System Owner │
│ Tech Lead │ Technical Lead │
│ Developer │ Data Steward │
│ QA Lead (My Role) │ Ethics Reviewer │
└───────────────────────┴───────────────────────┘
By formalizing these roles, we ensured that every analytical choice—such as data preprocessing configuration or training splits—had a designated owner responsible for its ethical and technical implications.
2. Managing the Accuracy Paradox (Annex A.6)
A key pillar of ISO 42001 is Data Quality (Annex A.6.2). In clinical datasets, data is often highly imbalanced (e.g. only 6% of patients have active diagnoses).
A model trained on such data can achieve 94% accuracy by simply predicting "Negative" for every patient, missing all active cases. This is known as the Accuracy Paradox.
To resolve this class imbalance safely, we integrated SMOTE (Synthetic Minority Over-sampling Technique) with adaptive parameters to prevent crashes on small minority samples:
from imblearn.over_sampling import SMOTE
def apply_adaptive_smote(X_train, y_train):
minority_count = sum(y_train == 1)
if minority_count <= 1:
return X_train, y_train
# Dynamically scale k-neighbors based on minority sample size
k = min(5, max(1, minority_count - 1))
smote = SMOTE(k_neighbors=k, random_state=42)
return smote.fit_resample(X_train, y_train)
We apply SMOTE to the training split only, keeping the test split clean to ensure our final metric evaluation represents real-world clinical conditions.
3. Subgroup Bias Auditing & Performance Quality Gates
Under the EU AI Act (Article 10), high-risk systems must perform bias audits across demographic groups.
In DrAI, we built statistical validation gates. The auditor segments predictions by demographic attributes (e.g., Age or Gender) and evaluates sensitivity (recall) for each group:
from sklearn.metrics import recall_score
def audit_demographic_bias(y_true, y_pred, demographic_col):
group_scores = {}
for group in demographic_col.unique():
mask = (demographic_col == group)
# Calculate sensitivity: TP / (TP + FN)
group_scores[group] = recall_score(y_true[mask], y_pred[mask])
# Trigger high-visibility alert if sensitivity gap exceeds 10 percentage points
sens_gap = max(group_scores.values()) - min(group_scores.values())
if sens_gap > 0.10:
trigger_ui_warning(sens_gap)
return group_scores
If the sensitivity gap between subgroups exceeds 10 percentage points, the system displays a warning banner in the clinical configuration dashboard, blocking deployment until bias is resolved.
4. Understanding Medical Metrics
To help non-technical clinicians evaluate models safely, we map data science metrics to clinical concepts:
- Sensitivity (Recall): Patients correctly identified. Measures the ratio of true positive classifications over actual positives:
- Specificity: Healthy patients correctly cleared. Measures the ratio of true negatives over actual negatives:
- Positive Predictive Value (PPV/Precision): Likelihood of actual disease on positive result.
- Negative Predictive Value (NPV): Likelihood of being disease-free on negative result.
By displaying these formulas alongside concrete patient numbers (e.g., "This model missed 12 active cancer cases out of 100"), the system makes AI limitations transparent to medical professionals.
5. Privacy by Design: Volatile Memory
Medical patient records are subject to strict data storage laws (like KVKK or GDPR). To eliminate data leakage risk, DrAI uses a zero-persistence memory architecture:
- No Database Storage: Uploaded files are processed in-memory using FastAPI volatile sessions (
SessionData). - Auto-Expiration: Sessions are deleted after 1 hour of idle time or a 4-hour hard TTL.
Engineering Takeaways
- Focus on Sensitivity (Recall): In healthcare AI, raw accuracy is a misleading metric. Prioritize sensitivity to avoid the accuracy paradox and minimize missed diagnoses.
- Audit demographic subgroups: A model can look accurate on average while exhibiting significant bias against specific subgroups. Program automated subgroup validation gates.
- Minimize data retention: Enforce volatile, self-expiring memory structures for processing PII to simplify regulatory compliance.