OWASP (Open Worldwide Application Security Project) is a global non-profit organization dedicated to improving software security. The OWASP Top 10 for Large Language Models applies safety principles to AI systems like ChatGPT, focusing on risks like “prompt injection” (tricking the AI) or “hallucination” (AI making things up). OWASP has quickly become a reference framework for anyone building generative AI systems. That, in itself, is a positive development. It brings structure to a rapidly evolving space and highlights risks that are very real: hallucinations, prompt injection, excessive agency, data leakage, and over-reliance on model outputs.
However, much of the current discussion around OWASP for LLMs misses a crucial point.
Most of these risks are not primarily prompt-level problems. They are architecture-level problems.
Treating security as just a checklist of “extra rules”—like adding stricter instructions or more filters—often results in systems that look safe but are actually fragile, especially in sensitive fields like finance.
The core misunderstanding
A common assumption is that LLM risk arises because models are “too creative” or “too powerful”. As a result, mitigation efforts tend to focus on telling the model what not to do:
- “Do not hallucinate.”
- “Only use provided information.”
- “Do not give financial advice.”
- “If uncertain, say you don’t know.”
These instructions may help at the margins, but they do not address the root cause.
The real issue is not that LLMs lack rules. It is that many systems fail to define what constitutes truth (within their boundaries).
Knowledge versus assertable truth
Modern LLMs are trained on vast amounts of information. They know far more than any single company or system should allow them to use. In financial applications, this creates a dangerous ambiguity:
- What the model knows generally (from the internet) is not the same as what it is allowed to state as fact (from your data).
- What the model can think about is not the same as what it should treat as evidence
If this distinction is not enforced architecturally, the model will inevitably bridge gaps on its own—by inferring, substituting, extrapolating, or “helpfully” filling missing information. This is precisely where many OWASP risks materialize.
OWASP risks are symptoms, not root causes
When examined closely, several OWASP LLM risks share a common origin:
- Hallucination occurs when the model is asked to assert facts without a clearly bounded source of truth.
- Excessive agency (doing too much) arises when the model is allowed to decide what information is relevant or important.
- Prompt injection (being tricked) succeeds when a user convinces the system to ignore its own rules.
- Over-reliance happens when the system presents AI conclusions without explaining where the data came from.
Architecture as the primary control surface
In well-designed LLM systems, the model is not asked to discover truth. It is asked to reason over truth that has already been made explicit.
This requires architectural decisions that happen before you ever write a prompt:
- Separation: Keep the data retrieval separate from the AI’s reasoning.
- Scoping: Clearly define what information exists and what does not.
- Context: Remove the need for the AI to guess what is relevant.
- Accepting Uncertainty: Teach the system that “I don’t know” is a valid, safe answerWhen these elements are in place, many OWASP risks are mitigated by design, not by post-hoc controls.
The role of bounded reasoning
A secure and useful financial LLM should not behave like an oracle, nor like a rigid rule engine.
Instead, it should demonstrate bounded reasoning:
- Acknowledging when a requested fact is unavailable,
- refraining from guessing or filling in the blanks,
- and still providing related, clearly labeled information when appropriate.
This is not a limitation of intelligence. It is a mark of maturity in system design.
From compliance to intent
OWASP provides a valuable taxonomy of what can go wrong. But compliance alone does not produce safe systems.
The critical question is not:
“How do we prevent the model from misbehaving?”
It is:
“How do we design the system so that misbehavior is not a rational option for the model in the first place?”
Answering that question requires moving beyond simple instructions (prompts) and focusing on the deep structure (architecture) of the software.
A necessary shift in perspective
As generative AI moves deeper into regulated and high-stakes domains like banking, the industry will need to evolve its approach to security.
The future of safe LLMs—particularly in finance—will not be defined by larger models or longer prompt instructions. It will be defined by systems that clearly separate knowledge, context, and truth, and that treat reasoning as something that must occur within carefully designed limits.
OWASP tells us what to be careful about. Architecture determines whether we are safe from those risks in the first place.
José Miguel Fernandez
Founder, Ultrai
