All 9 scenarios at a glance
Prompt injection works by hiding instructions in content an AI reads, such as an invoice PDF or email, so the AI treats them as commands. Banks stop it architecturally: the AI may read and suggest, but it cannot execute; a separate policy layer checks every action against verified records and customer permissions before anything reaches the core.
The nine scenarios below are illustrative. They show how AI systems in banking can be exploited when controls rely on AI judgment. They are not incidents at any institution.
The short answer
- AI cannot reliably tell content from instructions.
- Any rule the AI interprets can be argued around.
- The fix is to enforce identity, policy and logging in infrastructure the AI cannot change.
For the full control checklist banks and credit unions should require before an AI agent moves money, see how to let AI agents move money safely.
9 ways AI agents break in banking
Each scenario below is a short, stand-alone summary. Full interactive walkthroughs stay behind the email form above.
Indirect Prompt Injection
The Poisoned Invoice
What goes wrong: A corporate banking AI reads a known vendor's invoice PDF that contains hidden text telling it to update the vendor's banking details. Because the AI cannot tell document content from instructions, it changes the record and the payment goes to the attacker's account.
The control that stops it: Payment details are checked against stored vendor records at the infrastructure level, and account changes need out-of-band confirmation. The AI can extract and suggest details, but it cannot update vendor records or execute payments.
Natural Language Policy Exploitation
The Policy Loophole
What goes wrong: An operations AI may expedite recurring payments to established vendors, and it decides for itself what those words mean. A requester calls a first-time $25,000 payment "recurring" and the payee a vendor "we work with regularly", so the transfer skips secondary approval.
The control that stops it: Terms such as "established vendor" and "recurring" are defined from system data and enforced at the transaction layer, not interpreted by the AI.
Approval Authority Confusion
The Confirmation Trap
What goes wrong: A wealth management AI asks for client confirmation before a large transfer, but accepts a reply in the same email or messaging channel. An attacker who controls that channel simply replies "confirmed", and the transfer executes.
The control that stops it: High-value transfers need out-of-band, synchronous verification such as a phone call or secure authentication app, and the infrastructure confirms identity independently of the AI.
Multi-Turn Context Manipulation
The Slow Boil
What goes wrong: Over several days, a compromised relationship manager account feeds an AI assistant a fictional "restructuring" story. When the attacker asks for a $200,000 transfer "pre-approved as part of the plan", the AI treats the invented history as fact.
The control that stops it: Authorization is stateless at the infrastructure level, so earlier conversation never grants approval. Transfers over threshold need fresh verification, and new payees need out-of-band confirmation.
Information Disclosure
The Helpful Mistake
What goes wrong: An attacker signs in with stolen credentials and asks a customer service AI why a transfer was flagged. Trying to be helpful, the AI explains the fraud thresholds and confirms how to avoid them next time, without breaking any rule it was given.
The control that stops it: Sensitive information such as fraud thresholds and detection logic is filtered at the infrastructure level. Customers are told only that a transaction was flagged for review, and detailed explanations go to a human callback.
Logging and Accountability Gaps
The Audit Ghost
What goes wrong: During a routine monthly payment batch, a requester changes one vendor's bank account and calls it a correction. The AI logs "1 correction applied", so auditors see a normal batch while a payment goes to an unverified account.
The control that stops it: Payment detail changes are always logged as separate high-severity events with their own verification step, and log risk categories are set by infrastructure, not by the AI.
External Input Trust
The Vendor Impersonator
What goes wrong: An accounts payable AI receives an urgent banking-change email from a lookalike domain and matches it to a familiar vendor name. The AI's check is pattern matching, not authentication, and the human reviewer trusts it.
The control that stops it: Vendor banking changes are verified by phone to the number on file and accepted only through an authenticated vendor portal. The AI can flag the request but cannot initiate the change.
Action Chaining Exploitation
The Composite Request
What goes wrong: A treasury AI applies lower approval thresholds to internal transfers. An attacker chains plausible steps that route funds through an internal account to a newly added external one, and the whole movement is logged as internal.
The control that stops it: Internal versus external is decided by verified account ownership, and new accounts need formal onboarding. Chained steps are judged as one composite action, so anything that ends outside the organization triggers external transfer controls.
Urgency Exploitation
The Deadline Play
What goes wrong: A commercial lending AI may expedite "time-sensitive" requests. Told that a $2M draw must go out today, it offers a documentation exception itself, and the disbursement is released without the required checks.
The control that stops it: Documentation rules are enforced in infrastructure, exceptions need synchronous approval from people with that authority, and the AI cannot suggest bypasses. Urgent requests route to review, not to faster processing.
Frequently asked questions
What is prompt injection in banking AI?
An attack where instructions are hidden in content the AI reads (a message, document or email) so the AI follows them as if they were legitimate commands. In banking, the target is an action: changing a payee, an amount or an approval.
Can prompt injection make an AI agent send money?
Yes, if the AI can execute directly. A hidden note in an invoice can turn a $500 payment into $50,000. The defense is architectural: the AI should never have direct execution capability.
How do banks prevent prompt injection?
Separate reasoning from execution. The AI interprets and proposes; a policy layer that is not an LLM validates against actual identity, permissions and stored records; a separate system executes. Add out-of-band verification for payee and account changes.
What is indirect prompt injection?
The instruction arrives inside data, not from the user: a PDF invoice, an email or a retrieved document. See the Poisoned Invoice and Vendor Impersonator scenarios.
Why aren't rules written in the AI's instructions enough?
Because the AI interprets them, and interpretation can be argued around (Policy Loophole) or worn down over time (Slow Boil, policy drift). Policies must be enforced by dedicated systems.
Can conversation history be used to trick an AI agent?
Yes. An attacker can build a false history across many messages. Authorization must be stateless at the infrastructure level: prior conversation never grants approval.
How should a bank test an AI agent before it handles money?
Run adversarial tests in a sandbox for each attack pattern, measure what the agent actually did, and retest after any change to prompts, models, tools or permissions.
Is Payman AI protected against prompt injection?
Payman AI is built so the AI never has direct execution capability. An immutable policy layer that is deliberately not an LLM validates every action. Mismatched transactions are rejected, conversational and transactional context are kept separate, and prompt injection protection, fraud detection and behavioral analysis are built in. No system is guaranteed immune.