LEVEL 1 · EXECUTIVE
Plain-language “Why?”
What happened, why it matters and what action is expected—without unnecessary model jargon.
RESPONSIBLE AI · MODEL RISK · AUDITABILITY · GOVERNANCE
Explainable AI:
from model output to accountable understanding.
Explainable AI (XAI) is not simply about making a complex model look understandable. In a regulated environment, the objective is to provide decision-relevant explanations, preserve evidence, support meaningful human challenge and make the AI lifecycle auditable.
BankingAML/CFTCredit & RiskFraud & Transaction MonitoringInternal AuditAI Governance
Banking & ComplianceReview explanation, challenge and evidence requirements.
Internal Audit & Model RiskAssess fidelity, stability, traceability and reconstruction.
AI GovernanceConnect explainability to lifecycle controls and oversight.
Students & ProfessionalsLearn XAI concepts, methods and practical examples.
Executives & BoardsUnderstand what happened, why, risk, ownership and evidence.
INTERACTIVE XAI EXPERIENCE
See the decision. Understand the drivers. Challenge the result.
This synthetic demonstration shows how an explainability layer can connect an AI signal to a human-review workflow. It is illustrative only—not a real banking, credit, AML or customer decision.
01 · Input
02 · Model signal
03 · Explanation
04 · Human review
Illustrative transaction-risk output
Risk signal78/100
Explanation statusHigh — illustrative
Review statusManual review required
ModelXAI-Demo v1.0
Decision IDXAI-DEMO-001
Why was this signal generated?
Baseline (average reference pattern)14
Transaction velocity+24
Country-risk signal+18
Amount deviation+15
Customer history+7
Plain-language explanation
The transaction was flagged primarily because transaction velocity and amount deviation were materially above the configured illustrative reference pattern.
What would change the result? If transaction frequency were about 50 instead of 82, the illustrative risk signal would fall to roughly 69.
Attribution and counterfactual signals help a reviewer challenge the result; they are not proof of causation.One decision, three explanation layers
Progressive disclosure lets different audiences reach the level of detail appropriate to their role without hiding the underlying evidence.
LEVEL 2 · RISK & COMPLIANCE
Control-relevant detail
Key drivers, thresholds, context, uncertainty, review status and challenge/escalation route.
LEVEL 3 · TECHNICAL
Reproducible evidence
Model/version, attribution method, feature context, lineage, timestamp, stability/fidelity testing and decision ID.
Human-in-the-loop: AI explains, a qualified human decides
APPROVE
Accept with evidence
Reviewer confirms the explanation, context and applicable control requirements are adequate.
CHALLENGE
Question the result
Reviewer can challenge the explanation, request supporting evidence or identify a potential model/data issue.
ESCALATE
Route the exception
Material uncertainty, suspected bias, control failure or high-impact exception moves to the defined escalation path.
Where XAI can support regulated use cases
AML/CFTExplain material risk signals while preserving investigator judgement and evidence requirements.
TBMLMake relevant transaction and trade-risk signals reviewable without presenting an automated conclusion as fact.
Fraud detectionSurface contributing signals so analysts can validate, challenge and document action.
Credit & riskSupport decision review with appropriate feature, uncertainty and counterfactual context.
Transaction monitoringConnect alerts to explainable signals, thresholds and reviewer evidence.
AI governanceSupport model documentation, testing, oversight, monitoring and audit reconstruction.
Audit evidence panel
An XAI interface becomes materially more useful for regulated environments when the explanation can be tied to reconstructable evidence.
Decision IDXAI-DEMO-001
ModelXAI-Demo v1.0
Input classSynthetic transaction
Explanation methodIllustrative feature attribution
Human reviewRequired
OverrideNot applied
Evidence statusIllustrative demo only — no real evidence collected
PurposeLearning / control demonstration
What is Explainable AI?
Explainability concerns how an AI system's operation or output can be represented and understood; interpretability concerns the meaning of that output in the context of the system's intended purpose. NIST treats explainability and interpretability as distinct but related trustworthiness characteristics and notes that explanations can support monitoring, documentation, audit and governance.
Regulated-industry principle: an explanation is useful only when its audience, purpose, timing and limitations are clear. A visually persuasive explanation is not automatically a faithful explanation of model behaviour.
01 · PURPOSE
Explain the decision
Identify what the model was asked to predict, classify, rank or recommend—and what decision the output supports.
02 · EVIDENCE
Explain the basis
Show the relevant input factors, data provenance, model/version context and material assumptions behind the result.
03 · CHALLENGE
Enable human review
Give reviewers enough context to question the result, identify uncertainty and escalate exceptions before consequential action.
XAI methods: choose for the use case
| Approach | Typical use | Control consideration |
|---|---|---|
| Intrinsic interpretability | Rules, scorecards, constrained or simpler models | Often easier to document and challenge, but may trade off predictive performance or flexibility. |
| Feature attribution | Local or global contribution analysis | Confirm what the attribution method actually measures and test stability; do not treat contribution as causal proof. |
| Counterfactual explanation | “What would need to change?” analysis | Check feasibility, actionability and whether proposed changes respect policy and data constraints. |
| Example / case-based explanation | Nearest or representative examples | Assess similarity quality, privacy, data leakage and whether the examples are genuinely relevant. |
| Global model explanation | Overall behaviour and important patterns | Useful for governance and model review, but should not be substituted for case-level evidence where individual decisions matter. |
On smaller screens, swipe horizontally to view all columns.
Important: post-hoc explanations can be approximations. Validate explanation fidelity and stability rather than assuming that an explanation method reveals the model's true causal reasoning.
Explainability as a control system
For banks and other regulated institutions, XAI should connect to model governance rather than sit as a separate visualisation layer.
GOVERN
Ownership & accountability
Define model owner, business owner, control owner, reviewer and escalation route. Record the intended use and prohibited uses.
MAP
Context & impact
Map affected customers, decisions, data sources, downstream actions, jurisdiction and potential harms.
MEASURE
Explanation quality
Test fidelity, consistency, robustness, stability across relevant populations and usefulness to the intended reviewer.
MANAGE
Exception & remediation
Define thresholds, escalation, override, rollback, incident handling and change-management requirements.
AUDIT
Reconstructability
Preserve model/version, input snapshot where permitted, explanation output, reviewer action, approval and material evidence.
MONITOR
Post-deployment assurance
Monitor drift, data changes, explanation instability, override patterns, performance deterioration and emerging failure modes.
Control-testing bridge: use the synthetic XAI experience above to vary transaction inputs, observe how the illustrative signal changes, and then review the explanation and human-review implications.
Open the interactive XAI demonstration ↗
XAI testing checklist
- Can the reviewer state what decision the model supports?
- Is the explanation appropriate for the reviewer's technical skill and role?
- Does the explanation remain materially stable under relevant perturbations?
- Does the explanation have demonstrable fidelity to model behaviour?
- Are important data limitations and missing variables visible?
- Can a reviewer distinguish correlation from causation?
- Are overrides and human challenges recorded and reviewable?
- Can the institution reconstruct the model and explanation version used?
- Have bias, privacy, security and adversarial risks been considered?
- Are material changes subject to documented validation and approval?
Minimum XAI evidence pack
| Evidence | Why it matters |
|---|---|
| Use-case statement & decision purpose | Defines scope and prevents explanations being detached from the actual decision. |
| Model / system identification and version | Allows reconstruction of the system that produced the material output. |
| Data provenance and material limitations | Shows where decision-relevant information came from and what may be missing. |
| Explanation method and configuration | Allows reviewers and auditors to understand how the explanation was generated. |
| Validation / testing results | Provides evidence that explanation quality and limitations were assessed. |
| Human review, challenge and approval record | Demonstrates accountable human oversight for consequential use cases. |
| Monitoring, incidents and remediation | Shows that explainability is maintained through the AI lifecycle. |
On smaller screens, swipe horizontally to view all columns.
Regulatory and standards context — 2026
NIST AI RMF: NIST identifies “explainable and interpretable” as a trustworthiness characteristic and frames AI risk management through GOVERN, MAP, MEASURE and MANAGE. NIST states that AI RMF 1.0 is currently being revised, so institutions should monitor the authoritative NIST source for future revisions. NIST AI RMF ↗
EU AI Act: The EU framework uses a risk-based approach and includes transparency, documentation, traceability and human-oversight requirements for relevant AI systems. Article 50 transparency obligations apply from 2 August 2026. Other AI Act obligations, including certain high-risk requirements, have different application dates depending on the system and context. Applicability depends on the system, role and use case. European Commission — AI Act ↗
ISO/IEC 42001: ISO/IEC 42001:2023 provides requirements for establishing, implementing, maintaining and continually improving an AI Management System (AIMS), including governance areas relevant to transparency, accountability and risk management. ISO/IEC 42001 ↗
Regulatory note: Explainability is not a universal legal requirement for every AI system. Its required depth should be determined by jurisdiction, applicable law or regulation, use case, impact, contractual requirements, internal policy and risk appetite.
Explainable AI Assurance Lab — practical controls and validation
This additive module extends the existing XAI guide with a practical assurance workflow for regulated use cases. It is educational guidance, not legal advice, model certification or a substitute for institution-specific validation.
01 · EXPLAIN
Explanation methods
Distinguish intrinsic interpretability from post-hoc explanation. SHAP estimates feature contributions under its assumptions; LIME approximates local behaviour around a selected instance. Neither method alone proves causality, fairness or model correctness.
02 · VALIDATE
Test explanation quality
Test fidelity to the model, stability under reasonable input perturbations, sensitivity to irrelevant features, reproducibility, coverage of important populations and consistency with the intended decision use.
03 · GOVERN
Preserve human accountability
Record the model and version, input context, explanation method/version, reviewer, challenge, override rationale, decision and escalation outcome. An explanation should inform—not automatically replace—qualified human judgement.
Banking example: transaction-monitoring alert
Use synthetic data only. Suppose a fictional alert score is influenced by unusual transaction velocity, a new counterparty and a deviation from the customer’s historical pattern. The reviewer should verify whether these features are reliable, relevant, permitted for use and correctly represented—not infer wrongdoing from a score alone.
Interpretation boundary: feature attribution describes how a model output is associated with input features under a method’s assumptions. It does not by itself establish causal effect, suspicious intent, legal breach or the correct operational action.
Minimum explanation-validation checklist
- Use case, intended user and decision impact are documented.
- Model owner, version, data scope and explanation method are identified.
- Fidelity is tested against the model being explained.
- Stability and sensitivity are assessed with controlled perturbations.
- Known limitations, missing data and out-of-distribution cases are disclosed.
- Bias/fairness testing is separate from explanation generation.
- Human review, challenge and override are logged.
- Monitoring, change control and incident escalation are assigned.
Control testing record — reusable fields
| Field | Evidence expected | Reviewer question |
|---|---|---|
| Model/use-case ID | Inventory record, purpose and owner | Is the system within approved scope? |
| Explanation artefact | Method, version, output and relevant inputs | Can the result be reproduced? |
| Fidelity and stability | Test design, results, thresholds and exceptions | Does the explanation reliably reflect model behaviour? |
| Human challenge | Reviewer rationale, override and escalation logs | Can the reviewer question the output meaningfully? |
| Limitations and monitoring | Known limitations, drift monitoring and incident records | Are failure modes detected and remediated? |
Frequently asked questions
Is explainable AI legally required?
Not universally. The required depth depends on jurisdiction, applicable law or regulation, use case, impact, contractual requirements and institutional risk appetite. Applicability should be assessed case by case.
What is the difference between explainability and interpretability?
Explainability concerns how a system's operation or output can be represented and understood; interpretability concerns the meaning of that output in the context of the intended purpose.
SHAP vs LIME: which should we use?
Both are post-hoc attribution approaches. Neither is automatically better; assess fidelity, stability, robustness and usefulness for the model and use case.
Does feature attribution prove why a model decided something?
No. Attribution describes relationships between model output and inputs; it does not establish real-world causation, and post-hoc explanations can be approximations.
What evidence should we keep?
Use-case purpose, model/version, data provenance and limitations, explanation method/configuration, validation results, human review records and monitoring or incident evidence.
Can every AI model be fully explained?
No. The practical objective is proportionate, testable and auditable explanation suited to the use case, rather than a promise of complete transparency.
RegTech Nexus AI approach
For regulated use cases, XAI should support the chain REGULATORY REQUIREMENT → USE CASE → RISK → EXPLANATION → HUMAN CHALLENGE → EVIDENCE → ASSURANCE. The objective is not to promise that every AI model can be made fully transparent; it is to establish proportionate, testable and auditable explanation practices.
Practical assurance question: if an auditor, supervisor, model-risk reviewer or affected decision-maker cannot understand what the system was intended to do, what materially influenced the output, what uncertainty or limitation existed, and who reviewed the outcome, the XAI control environment may be insufficient for the use case.
Primary reference sources
- NIST — AI Risk Management Framework
- NIST AIRC — Explainable and Interpretable AI
- European Commission — AI Act
- ISO — ISO/IEC 42001:2023
- SHAP — official documentation
- LIME — reference implementation and documentation
Putting explainability into practice?
Explore practical XAI governance, evidence and control-testing approaches for regulated environments.
Book a discussion ↗ Explore frameworks & toolkits ↗