Short answer
Anything that scores, decides or drafts inside a bank is treated as a model, which means it needs documentation of what it does, evidence it was tested, ongoing monitoring and periodic review. Model risk management practice predates generative AI by more than a decade, so the expectations are not new, only the products are.
The most common misconception about AI in a regulated institution is that supervisors are still deciding what to think. In practice the framework arrived long before these products did. Banks have been documenting, validating and monitoring statistical models for well over a decade, and an AI vendor's output slots into that same machinery. What changes is how much work landing it there costs you, and that varies enormously by product.
The four questions everything reduces to
Strip away the terminology and every examination conversation about a model comes down to the same four things. If you can answer them in writing, with evidence, the conversation is short.
Notice that none of these are about the model architecture. Nobody is going to ask you to explain a transformer. They are going to ask what it does in your institution, how you know it works, how you would notice if it stopped working, and who is accountable.
- What does it do, in the workflow, in plain language, and what decisions does it influence
- How was it tested before you relied on it, and on whose data
- How do you monitor it now, how often, and what triggers action
- Who owns it, who reviews it, and what happens when it is wrong
Vendor models are still your models
Buying rather than building does not move the obligation. The institution using the model is accountable for it, which is why third-party risk review and model risk review both apply to the same purchase and why the vendor's documentation quality is a real cost factor rather than a nice-to-have.
The practical consequence is a question to ask before signing: what does the vendor provide, automatically, that goes into our model inventory and our validation file. Vendors that answer with a specific artifact list are cheaper to own than vendors that answer with a security whitepaper.
Product design choices that reduce the burden
Three design patterns show up repeatedly among the products that are easiest to govern, and they are worth weighting in an evaluation because each one removes work your team would otherwise do by hand.
The first is documentation generated during construction rather than afterwards. In credit decisioning, that means fair-lending testing that runs as part of building the model, so the evidence exists before anyone asks for it.
The second is traceability. A calculated figure that points back to the document and page it came from, or an answer that arrives with its citation attached, can be checked by a reviewer in seconds. Output without provenance has to be re-derived by hand, which is how a time-saving tool becomes a review queue.
The third is human oversight built into the flow rather than bolted on. Some products route only low-confidence fields to a person, some keep expert reviewers behind the model output, and some escalate high-stakes questions to credentialed specialists. All three are easier to describe to an examiner than a system that produces confident output with no defined intervention point.
Fair lending is the sharpest edge
Anything influencing a credit decision carries fair-lending exposure, and that is where the most specific scrutiny falls. Two capabilities matter here and they are different from each other.
One is testing during model construction, where the process searches for less discriminatory alternatives and documents what it found. The other is testing your own portfolio outcomes after the fact, using regression that controls for legitimate credit factors to see whether observed disparities indicate bias. Some vendors do the first, some do the second, and an institution deploying AI credit decisions generally wants both.
The work itself is not new. Community banks have been buying fair-lending regression from consultants for years. What is new is that some of it now comes inside the software, which changes the cost but not the obligation.
Where the AI is easiest to defend
Not every AI deployment carries the same weight, and sequencing matters. Uses that summarise, draft or retrieve, with a human reviewing before anything leaves the institution, sit at the low end. Uses that decide, score or act autonomously sit at the high end.
That is a good argument for starting with employee-facing tools. Knowledge assistants, call summarisation, document extraction and regulatory change tracking improve throughput without putting a model between the institution and a credit decision, which buys time for the governance muscle to develop before the stakes rise.
Frequently asked questions
Is generative AI treated differently from a credit scorecard?
The framework is the same: document it, test it, monitor it, review it. What differs is that generative output is harder to test exhaustively, which is why intervention points, citation trails and human review design carry more weight in the evaluation than they would for a scorecard.
Does buying from a vendor reduce our responsibility?
No. The institution relying on the model is accountable for it. What a good vendor reduces is the effort, by supplying documentation, validation evidence and monitoring output that goes straight into your file.
What should be in the model inventory for an AI tool?
What it does and which decisions it touches, the vendor and version, the data it uses, the testing evidence, the monitoring approach and cadence, the owner, and the defined fallback when it is unavailable or wrong.
Which AI uses are lowest risk to start with?
Employee-facing ones with a human between the output and the customer: knowledge retrieval, call summarisation, document extraction and regulatory change tracking. Credit decisioning is the highest-burden starting point and rarely the right first project.