Barclays built a system to spot AI agents that might go rogue
Research setup ‘reads the mind’ of LLM-based agents and flags if they’re about to do something risky
Quants at Barclays have built a prototype monitoring system to read the minds of AI agents and catch them before they do anything too risky.
Ariye Shater and Hariom Tatsat borrowed from research into so-called mechanistic interpretability to address what Shater calls “tool risk” in agentic systems – the risk that AI agents call software tools in ways that are wasteful, risky or damaging, such as
Only users who have a paid subscription or are part of a corporate subscription are able to print or copy content.
To access these options, along with all other subscription benefits, please contact info@risk.net or view our subscription options here: http://subscriptions.risk.net/subscribe
You are currently unable to print this content. Please contact info@risk.net to find out more.
You are currently unable to copy this content. Please contact info@risk.net to find out more.
Copyright Infopro Digital Limited. All rights reserved.
As outlined in our terms and conditions, https://www.infopro-digital.com/terms-and-conditions/subscriptions/ (point 2.4), printing is limited to a single copy.
If you would like to purchase additional rights please email info@risk.net
Copyright Infopro Digital Limited. All rights reserved.
You may share this content using our article tools. As outlined in our terms and conditions, https://www.infopro-digital.com/terms-and-conditions/subscriptions/ (clause 2.4), an Authorised User may only make one copy of the materials for their own personal use. You must also comply with the restrictions in clause 2.5.
If you would like to purchase additional rights please email info@risk.net
More on Risk management
Can prediction markets pass the institutional sniff test?
Regulated firms need credible event determination methodologies and cross-product margining to enable use, say experts
How enterprise risk rose to the top of the bank
Previously a support function, ERM is naturally positioned to identify strategic risks for the board
Dealers still split on XVA desks’ funding remit
Risk Benchmarking: Global dealers use XVA desks to manage swaps funding needs, but most treasuries retain some control
Most dealers run central XVA desks, but remits vary
Risk Benchmarking: Many banks struggle to track XVA costs and share of derivatives compute
EU banks in India cut contingency costs after clearing deal
Esma MoU with two Indian regulators lifts threat to direct clearing for BNP, SocGen and Deutsche
Big banks look to XVA desks to crush counterparty exposures
Risk Benchmarking: Two-thirds of global dealers run routine optimisation, compared with just 13% of smaller players
Regulators must think bigger to tackle tech risks, says Hong Kong SFC
Leung warns AI adoption and recent cyber scares show new threats require global co-operation
Investment workflows move beyond the trade
J.P. Morgan experts reveal how Vida’s digital tools enhance transparency, analytics, portfolio oversight and investment lifecycle management
