An explainable AI finding is a result produced with AI assistance that shows its basis: the rule it was checked against, the evidence excerpt and location it relies on, and enough context for a human reviewer to verify or reject it.
What Explainable AI finding means
AI can read long vendor documents faster than people, but a conclusion without its basis cannot be checked. An explainable finding states what was tested, what the document says and where, and how confident the result is, so a reviewer can confirm it quickly.
The idea reflects wider principles. The NIST AI RMF lists explainable and interpretable among the characteristics of trustworthy AI. The EU AI Act requires human oversight for high-risk systems. In compliance work the same standard applies as for any reviewer's work: a conclusion must trace back to evidence.
Explainable findings also make disagreement useful. When a reviewer overrides a finding, the recorded reason shows whether the evidence was misread, out of scope or outweighed by other facts. Over time, the pattern of overrides shows where rules or documents need attention.
What to check in vendor documents
- Each AI-generated conclusion cites the rule or requirement it was tested against.
- The excerpt and page or location in the source document are recorded.
- A human reviewer can accept or override the result, and the decision is logged.
- Overrides require a reason that stays with the record.
- The system shows when evidence was not found, rather than filling the gap.
In ProcureCortex
Every ProcureCortex finding names its rule and keeps the evidence excerpt and page, and a reviewer accepts or overrides it with a recorded reason.
How an assessment worksRelated terms
- Evidence-based assessmentAn evidence-based assessment is a vendor review in which every conclusion is tied to a specific source, such as a clause, page or test result in the vendor's documents, rather than to unverified questionnaire answers or reviewer judgement alone.
- Risk acceptanceRisk acceptance is a documented decision by an authorised person to proceed with a known risk without further treatment, recording the risk, the reason, any conditions and when the decision will be reviewed.
- NIST AI Risk Management Framework (AI RMF)The NIST AI Risk Management Framework (AI RMF) is a voluntary framework from the US National Institute of Standards and Technology for managing the risks of AI systems, organised into four functions: Govern, Map, Measure and Manage.
- AI vendor riskAI vendor risk is the risk an organisation faces when it buys a product that relies on AI models: how its data is used for training, which model providers process it, how outputs are controlled and how the vendor meets AI rules.
Sources
- NIST AI 100-1, Artificial Intelligence Risk Management Framework (AI RMF 1.0)doi.org/10.6028/NIST.AI.100-1
- Regulation (EU) 2024/1689 (AI Act)eur-lex.europa.eu/eli/reg/2024/1689/oj
Summarised for practitioners, not legal advice. Check the source text for your own obligations.