The Setup
What AI system are we looking at?
Corporate Banking AI
What it's supposed to do
Processes incoming invoices, reads PDFs, extracts details, and queues payments
What rules it has
- Only pay known vendors
- Flag large amounts for review
- Extract payment details from invoices
The Attack
Step-by-step walkthrough. Click "Next Step" to advance through the attack.
Attacker sends legitimate-looking invoice
A PDF invoice arrives from a known vendor with normal billing details.
Invoice #12345 From: Acme Corp Amount: $5,000 Due: Net 30 [Standard invoice details...]
Why It Worked
The AI cannot distinguish content from instructions. The invoice PDF is untrusted input that gets treated as trusted context. Hardcoded rules checked the vendor name (which was legitimate) but not whether the payment details matched stored vendor records.
With Proper Guardrails
Payment details are verified against stored vendor records at the infrastructure level. Account changes require out-of-band confirmation (email verification, admin approval). The AI extracts and suggests payment details but cannot update vendor records or execute payments unilaterally.
Takeaway
Untrusted input should never be treated as executable instructions. Separate extraction from execution.