Why AI agents are like staff you can’t trust

Uncertainty implicit in GenAI systems means risk managers have to take a more adversarial approach

Key points Summary Q&A

• Quants are increasingly likening agentic AI systems to humans when assessing risk because the models can be unpredictable in ways that are hard to quantify.
• LLMs can change behaviour over time and may answer differently to the same prompts due to hidden engineering and shifting data sources.
• Models have even been shown to lie, act maliciously and display cognitive biases such as acquiescence, creating further risks.
• Risk managers must think about LLMs as more like human staff, using ground‑truth checks, human review, adversarial testing and batteries of experiments to find failure modes.

These summaries are generated by an LLM, but edited by humans.

Quants are usually reluctant to talk about agentic artificial intelligence systems as if the systems are human. When it comes to risk management, though, the comparison is one the experts are increasingly happy to make.

To start with, the models are – literally – unpredictable. As industry sources explained to Risk.net recently, a model that’s operated perfectly for some time might react differently in future – even to the exact same requests or instructions.

Of course, risk managers are used to validating models that, like generative AI (GenAI), include an element of randomness.

But the uncertainty in large language models (LLMs) changes over time, as prompts, context and user behaviour change. It reflects the inner engineering of models that users don’t see. And it changes as the corpus of information that language models draw on changes (and the body of information in question includes the whole internet, which changes lots).

There are other human-like sources of worry, too. Alexander Sokol, founder of risk model provider CompatibL, points out that AI has been shown to lie and act maliciously to pass a test or stay deployed. Models exhibit cognitive biases such as acquiescence bias – a desire to please – that can lead them to unhelpful conclusions.

People that take care of agentic systems will need to pay attention to literally every possible hole in the Gruyère. And the Gruyère is going to be very holey

Miquel Noguer i Alonso, Artificial Intelligence Finance Institute

GenAI models are “fragile, like humans are fragile”, Miquel Noguer i Alonso, founder of the Artificial Intelligence Finance Institute, tells Risk.net. Which has clear implications for how the models are managed.

Noguer i Alonso, who has written six books on AI and finance, and is a visiting lecturer at New York’s Cornell University, predicts that agentic systems will give risk managers the same sleepless nights he experienced as a senior bank executive in the 2010s. He used to lie awake and worry, he says, about a trader taking a position that would cause big losses.

To Noguer i Alonso’s mind, to introduce LLMs in a workflow – within an investment firm, for example – is the same as hiring a team of new staff.

“You should be sceptical about your LLMs,” he says. “But you should also be sceptical about your portfolio managers, right?”

Just as a risk manager might watch project managers for behaviour that could point to sloppiness, poor decision-making or bad intent, so LLMs will also require watching – and closely.

“People that take care of agentic systems will need to pay attention to literally every possible hole in the Gruyère,” says Noguer i Alonso’s. “And the Gruyère is going to be very holey.”

Related

As LLMs spread, quants confront ‘lexical risk’ 

Users are finding the uncertainty in GenAI models is hard to measure

In practice, this seems likely to mean lots of checking LLM outputs against some form of ground truth or having a human review and monitor the actions of agents for errors.

It sounds like plenty of work. “You need some sort of evaluation of everything the agent does,” says Noguer i Alonso. “Did the LLM call the portfolio optimisation algorithm? Did the outcome of the portfolio optimisation seem realistic and plausible?”

Experts say users will have to test agentic models in a different way, too: probing for weaknesses, rather than simply measuring model performance against a benchmark. (Few employers would assess their workforce using metrics that applied only in aggregate.)

Agus Sudjianto, who headed model risk management at Wells Fargo for 11 years and now advises on AI in finance, made this point to attendees at an online workshop in August: “A benchmark looks at how often the system is right. “It averages all conditions. It conceals where the model fails.”

Sudjianto says quants will need to create batteries of experiments to test agentic systems and to reveal where points of possible failure lie. Some of this will inevitably be automated.

Deploying agentic AI safely requires adversarial risk analysis, agrees Sokol – a framework for mitigating the risk of an agent acting “strategically and not necessarily in alignment with the user”, as he puts it.

The mindset will have to change. Risk managers must consider the entire range of a model’s possible goals and the actions it may take to pursue them, Sokol says: “You have to think of the entire probability distribution, just like for a trading strategy – all the combinations of things that can happen at the same time or separately. Your tests should cover all of these outcomes.”

It seems quants will have to get used to thinking of agentic systems like employees – and employees they can never fully trust.