The decision follows a BBC investigation which revealed that MI5 had misled three courts in a case involving a neo-Nazi informant who abused women while acting as a state agent.

A panel of three senior judges said it would be “premature” to consider contempt of court proceedings against any individuals until the fresh investigation is concluded. The judges also commended the BBC for “bringing these matters to light”.

Two official reviews, including one commissioned by the Home Secretary, Yvette Cooper, had cleared MI5 and its officers of any deliberate misconduct. However, the High Court found the reviews to be procedurally flawed and their conclusions unreliable.

The investigations by MI5 suffer from “serious procedural deficiencies,” the judges said. “We cannot rely on their conclusions,” they added.

The judgment was delivered by Lady Chief Justice Baroness Sue Carr, President of the King’s Bench Division Dame Victoria Sharp, and Mr Justice Chamberlain. They said they hope that “events such as these will never be repeated”.

The court has directed that the new inquiry be conducted under the supervision of the Investigatory Powers Commissioner, Sir Brian Leveson, who oversees MI5’s surveillance activities through the Investigatory Powers Commissioner’s Office (IPCO). That office, too, had been misled in the same case.

Sir Brian said: “It is clear that MI5’s compliance with its statutory duties fell short in this case,” and noted that he would await instruction from the Prime Minister before proceeding with the investigation.

MI5’s director general, Sir Ken McCallum, reiterated his “full and unreserved apology for the errors made in these proceedings”.

He said the matter remained “of the highest priority for MI5” and confirmed the service would cooperate fully with IPCO. “MI5’s job is to keep the country safe. Maintaining the trust of the courts is essential to that mission,” he added.

A BBC spokesperson said: “We are pleased this decision has been reached and that the key role of our journalist Daniel De Simone in bringing this to light has been acknowledged by the judges.

“We believe our journalism on this story has always been in the highest public interest.”