The study found that every major AI model tested was susceptible to persuasion, raising concerns about fairness and reliability as governments and justice systems increasingly explore the use of AI in legal decision-making.

Researchers used real appellate cases from the US, England and Wales, and Ireland to test how AI systems handled difficult legal disputes. In a simulated courtroom, one AI acted as an advocate for each side while a third AI served as the judge.

By changing the models used to argue each side, the researchers measured how much the quality of advocacy affected outcomes. Across 20 different model configurations, stronger advocate models prevailed in between 58 per cent and 71 per cent of cases on average. In some match-ups, the stronger advocate won more than 90 per cent of the time.

The paper, Persuadability and LLMs as Legal Decision Tools, was written by Oisin Suttle and David Lillis.

Dr Suttle said: “It is important that a judge should be open to persuasion. Otherwise there would be no point in letting the parties argue their case.

“However we don’t want a judge to be too persuadable. They should decide on the law and the facts of the case, rather than being swayed by an especially skilled advocate. Our results show that AI models in legal settings can be very persuadable. That raises worries about both reliability and fairness.”

The researchers said the findings could have implications for access to justice if AI decision-makers give an advantage to parties with better legal representation or more sophisticated AI tools.

The study found that larger and more capable AI models tended to be somewhat less persuadable than smaller systems, although all models tested showed significant sensitivity to argument quality.

Dr Lillis said: “It is essential that society and decision-makers don’t simply follow hype when it comes to the capabilities of modern AI systems.

“From my point of view as a computer scientist, collaborations with domain experts like Dr Suttle are essential in allowing us to scientifically investigate their capabilities and characteristics, particular in sensitive areas such as legal decision-making.”

The authors said persuadability should become a standard measure when assessing AI systems for legal use, arguing that policymakers should understand how readily a model can be influenced before deploying it in courts, tribunals or administrative decision-making.