Financial institutions are rapidly adopting AI across fraud detection, anti-money laundering (AML), underwriting, trade execution, customer service, and operational workflows. The technology is advancing quickly, but governance, assurance, and security controls are not always keeping pace.
For security leaders, risk teams, and regulators, the central question is no longer whether AI creates business value.
The question is whether organizations can demonstrate that AI-enabled decisions are secure, explainable, auditable, and operating within defined risk tolerances.
While much of the market discussion remains focused on model performance and prompt guardrails, the more significant risks often emerge from the surrounding architecture, permissions, data flows, and decision-making processes.
Across banks, insurers, and asset managers, NCC Group consistently observes four areas where control gaps are developing as AI adoption accelerates.
1. The “Sandbox” Assumption Often Breaks Down in Production
Many organizations assume that because an AI model operates within a sandbox or isolated environment, it cannot directly impact critical systems.
In practice, this assumption frequently breaks down.
During assessments, we regularly observe AI workloads that:
- Share infrastructure with other business services
- Retain credentials that provide access to internal applications, APIs, and data stores
- Operate with permissions broader than those required for their intended purpose
- Maintain persistent access to sensitive customer, transaction, or operational data
Under these conditions, a malicious prompt, compromised data source, or cleverly crafted attachment may influence the model to perform actions that affect financial systems directly.
Depending on the architecture, this could include:
- Accessing customer records
- Modifying account information
- Triggering downstream workflows
- Retrieving confidential business data
- Initiating actions that appear legitimate to surrounding systems
Organizations that are successfully reducing exposure typically:
- Deploy isolated execution environments
- Apply least-privilege access controls
- Restrict resource access to specific tasks
- Validate all inputs regardless of source
If an AI-enabled application can influence a financial process, it should be treated as part of the organization’s control environment rather than as a standalone technology component.
2. Human Review Alone Is Not a Security Control
Many organizations continue to assume that human review provides an effective safety net for AI-enabled processes.
Historically, human reviewers have played an important role in functions such as fraud investigations, AML reviews, customer onboarding, underwriting assessments, and compliance monitoring.
However, AI introduces attack techniques that human reviewers were never designed to detect.
Attackers are actively exploring methods to manipulate AI-enabled workflows through prompt injection, adversarial inputs, hidden instructions, data poisoning, and indirect influence techniques.
During testing and security reviews, we commonly observe:
- Instructions concealed within whitespace, images, markdown, or Unicode characters
- Interfaces that present one action to a reviewer while the underlying model performs another
- Excessive trust in model-generated outputs
- Reviewers who are unlikely to identify subtle AI-specific manipulation techniques during high-volume operations
This is particularly important in regulated environments where AI may influence customer outcomes, risk assessments, fraud investigations, or financial decisions.
Regulators including the FFIEC, OCC, Federal Reserve, CFPB, OSFI, FCA, and PRA increasingly expect firms to demonstrate effective governance, auditability, model risk management, and control effectiveness.
Organizations should evaluate where AI participates within a business process and determine which decisions require independent validation.
Business rules, approval requirements, and policy enforcement mechanisms should operate independently of model recommendations.
Threat modeling, adversarial testing, and AI-focused penetration testing should be performed regularly to verify that controls continue to function under realistic attack scenarios.
3. Agent-to-Agent Architectures Introduce New Systemic Risks
Many financial services companies are beginning to deploy agentic AI architectures that allow multiple AI systems to interact with internal applications, external services, data providers, and business workflows.
These architectures create powerful automation opportunities, but they also introduce new trust relationships that are often poorly understood.
For example, a customer facing AI assistant may collect information from a user, pass that information to an underwriting model, retrieve external credit data, and trigger internal workflows used to support lending decisions.
If any component within that chain is manipulated, compromised, or operates with excessive permissions, downstream decisions may be affected.
Common issues include:
- Trust relationships that are inherited without validation
- Privileges that propagate across systems
- Sensitive data crossing poorly governed boundaries
- Dependencies introduced without clear ownership or accountability
A compromise affecting a seemingly low-risk component may have consequences far beyond the initial point of entry.
Organizations that manage these risks effectively typically:
- Define explicit input and output requirements for every agent
- Establish clear trust boundaries
- Validate data provenance throughout workflows
- Limit privilege inheritance
- Introduce validation checkpoints between critical decision stages
Each agent interaction should be treated as a control point rather than an assumed trust relationship.
4. AI Is Increasingly Embedded in Financial Decision Paths
AI is no longer limited to research, productivity, or customer support use cases.
In many organizations, it now sits directly within business processes that influence customer outcomes and financial decisions.
Examples include:
- Fraud investigations
- Credit decisions
- Underwriting recommendations
- Payment approvals
- Trade support workflows
- Customer servicing activities
As AI becomes more influential in these processes, it effectively becomes part of the decision-making chain.
The challenge is that many organizations still lack independent enforcement mechanisms.
When a model misinterprets a request, receives poisoned data, or is manipulated through prompt-based attacks, downstream systems may continue operating as though the decision were valid.
Regulators increasingly expect firms to demonstrate accountability, explainability, governance, and traceability for decisions that affect customers, transactions, and market activity.
Organizations addressing these risks effectively typically:
- Separate recommendation generation from policy enforcement
- Apply deterministic business controls before actions are executed
- Record complete decision histories
- Maintain auditable evidence of inputs, outputs, approvals, and policy evaluations
The objective is not to remove AI from business processes.
The objective is to ensure that business controls remain authoritative regardless of how a recommendation was generated.
Questions Boards and CISOs Should Be Asking
What happens if a malicious prompt reaches one of our AI systems?
Without appropriate isolation, access controls, and policy enforcement, an AI-enabled system may trigger actions that appear to be legitimate business activity while introducing operational, compliance, financial, or reputational risk.
Can we demonstrate to regulators how an AI-enabled decision was made?
Organizations should be able to provide evidence covering data inputs, model outputs, control validations, approvals, and final business actions.
If that evidence does not exist, governance and auditability become difficult to defend.
Where are the highest-risk control gaps?
The greatest risks often emerge where AI interacts with privileged systems, accesses sensitive data, or exchanges information with other agents without clear validation requirements.
How should we test AI security controls?
Begin with threat modeling to understand how AI is used across people, processes, and technology.
Use those findings to conduct targeted white-box AI penetration testing aligned to the OWASP Top 10 for LLM Applications and other applicable AI security frameworks.
Controls should be tested under realistic adversarial conditions to verify that unauthorized actions are blocked and that complete decision paths remain visible for audit, compliance, and forensic review.
Closing Thought
As regulators, boards, and customers place greater scrutiny on AI-enabled processes, organizations will need to demonstrate not only what their AI systems can do, but how those systems are governed, controlled, and assured.
For financial institutions, the challenge is no longer about deploying AI rapidly. The challenge is ensuring that AI-enabled decisions remain secure, explainable, auditable, and aligned with established risk management practices.
Organizations that treat AI as part of their broader control environment will be better positioned to manage emerging threats, satisfy regulatory expectations, and maintain trust with customers and stakeholders.
Security, governance, and resilience are increasingly becoming differentiators alongside innovation. The institutions that can demonstrate effective control over AI-enabled processes will be best positioned to scale adoption while managing operational, compliance, and reputational risk.
Technical Director, NCC Group NA
David Brauchler III is an NCC Group Technical Director based in Dallas, Texas. He is an adjunct professor for the Cyber Security graduate program at Southern Methodist University with a master's degree in Security Engineering and the Offensive Security Certified Professional (OSCP) certification
David published Analyzing AI Application Threat Models, which introduced new Models-As-Threat-Actors (MATA) methodology to the AI security industry and provided a new trust flow centric approach to evaluating risk in AI/ML-integrated environments. He has also released several new threat vector categories, AI/ML security controls, and recommendations to maximize the effectiveness of AI penetration tests.