Supreme Court warns against unchecked AI use in courts after NCLT cites fake cases

NEW DELHI: The Supreme Court on Thursday underscored the need to regulate the use of artificial intelligence in judicial proceedings, noting that unverified AI-generated material could undermine the administration of justice.Directing the Bar Council of India (BCI) to set up a committee to examine the use of AI in legal practice and adjudication, the court said there was a clear need for regulatory safeguards. It also urged both the Bar and the Bench to exercise caution while using such technology.The court observed that while AI can assist in adjudication, it should not replace human reasoning or decision-making. It emphasised that human oversight must remain central at every stage of the judicial process.Raising concerns about the use of fabricated or non-existent AI-generated legal precedents, the court warned that such material could adversely affect the integrity of the justice system. Describing the use of fake, non-existent and AI-generated hallucinated legal precedents as "invisible, insidious and catastrophic" to justice, the court likened it to "the release of methyl isocyanate (a highly toxic, flammable, and volatile organic compound) in the field of law and justice".The observations came while the apex court set aside the National Company Law Tribunal (NCLT) and National Company Law Appellate Tribunal (NCLAT) orders in the Essel Infraprojects insolvency case after finding that the NCLT had relied on non-existent, fake and AI-generated hallucinated judicial precedents while deciding the matter. The court held that such reliance strikes at the integrity of the adjudicatory process and warranted fresh consideration of the dispute.Why the Supreme Court set aside the Essel Infraprojects ordersA bench of justices PS Narasimha and Alok Aradhe was hearing an appeal filed by suspended director Pooja Ramesh Singh against the NCLT's decision admitting Essel Infraprojects into the corporate insolvency resolution process on a petition filed by Jammu and Kashmir Bank under Section 7 of the Insolvency and Bankruptcy Code.The insolvency proceedings arose from an alleged default linked to a Rs 200 crore credit facility extended by Jammu and Kashmir Bank to Pan India Utilities Distribution Company Limited. The loan was backed by a corporate guarantee from Essel Infraprojects and a mortgage over land in Gorai, Borivali, Mumbai.The NCLT's Mumbai Bench admitted the insolvency plea in August 2024 over a claimed default of Rs 87.43 crore, and the NCLAT later upheld the order.However, the Supreme Court found that the NCLT had relied on fabricated and non-existent AI-generated judgments as legal precedents while reaching its conclusion.Holding that such reliance cannot be tolerated, the Bench said, "It is necessary for Courts to adopt a zero-tolerance mode for producing, citing or using AI-generated precedents without verification. It is a misconduct on the part of an advocate to cite such judgments without verification."The court further said it was an equally serious lapse if judges relied on fake or hallucinated AI-generated material while deciding cases."We have no hesitation in declaring that such a decision is no decision in the eyes of the law, irrespective of whether such material had a direct or indirect bearing on the decision-making. Such decisions are to be set aside even if an iota of fake or hallucinated material enters the decision-making process, as it would violate the sanctity of adjudication," the Bench observed.Accordingly, the Supreme Court set aside both the NCLT and NCLAT orders and directed the tribunal to decide the matter afresh on the facts of the case.Clarifying the scope of its ruling, the court said it was not examining the technical reasons behind AI hallucinations or ways to eliminate them. Those issues, it said, were for engineers and scientists. The judiciary's concern was the use of fake, non-existent AI-generated material as legal precedents.Get the latest India news and live updates. Download the TOI App.