Legal Insights by BRANDLEX
In the world of machine learning and generative AI, hallucinations are not science fiction—they are a real threat that can undermine trust, spread misinformation, and jeopardize critical decisions.
AI hallucinations are errors that appear to be correct answers but, in reality, have no basis in specific (real and current) data. In areas such as law, healthcare, or education, this is not just a technical glitch—it can mean losing a lawsuit, facing sanctions, or suffering an irreversible loss of trust.
Models like ChatGPT, Gemini, and Claude, among others, are powerful or “intelligent,” but they are not designed to provide legal accuracy or professional accountability. We have already seen this in real-life cases: lawyers sanctioned for citing non-existent rulings, mistakenly referencing foreign laws, or unintentionally leaking sensitive information.
The situation in Italy, where a lawyer included fabricated court rulings generated by ChatGPT in a legal brief, is not an isolated event. Similar cases of misuse of generative AI in the legal field have been documented in various countries, with different professional consequences and official warnings from authorities.
When it comes to legal decisions, accuracy is a professional obligation.
Why Do These Hallucinations Happen—and How to Avoid Them?
The goal is not to reject technology—we must acknowledge that AI brings enormous value to legal teams by speeding up response times, automating repetitive tasks, and improving operational efficiency. But we also cannot allow its uncritical use to erode trust in the work we deliver as professionals. In law, every document, every argument, every recommendation carries our signature and our responsibility. That cannot be blindly delegated to a tool—no matter how “intelligent” or fast it is.
Open generative AI (even paid versions) searches for answers in a vast ocean of data—over 45 terabytes pulled from the internet. This includes everything from academic papers to forums, blogs, social media, and unverified websites. While this breadth allows AI to answer almost anything, it also exposes it to errors, misinterpretations, and hallucinations.
Italy – Citing Non-Existent Rulings in Florence (2025)
In March 2025, an Italian lawyer filed an appeal before the Florence Court that contained references to Supreme Court rulings that did not exist.
The citations were apparently generated by ChatGPT during legal research. The lawyer admitted that a team member had used AI to draft the brief without later verifying the accuracy of the references.
When the error came to light, the opposing party requested sanctions for fraudulent use of the procedure, proposing a €5,000 fine for bad faith litigation.
However, the court ruled that there had been no deliberate bad faith and declined to impose any penalty, as the lawyer’s litigation strategy was coherent and caused no concrete harm to the other party.
In its March 14 ruling, the Florence court explicitly warned about AI hallucinations, stressing the risk that such tools can fabricate plausible-looking results if not properly verified. The case served as a wake-up call in Italy about the need to double-check any AI-generated information in legal contexts.
Australia – Fake Case Law and Regulatory Reactions (2024–2025)
Australia has also seen several incidents highlighting these same risks. In 2024, in Sydney, an immigration lawyer—pressed for time and dealing with health issues—turned to ChatGPT to prepare arguments for a client appealing an administrative decision. He input some case details into the tool, which generated a summary containing seemingly relevant case law. The lawyer then included 17 case citations in his submission without verifying them.
Weeks later, when the opposing counsel (the Ministry of Immigration) responded, it was revealed that none of the 17 cases existed. Confronted in Federal Court, the lawyer expressed deep embarrassment and apologized, explaining that he had trusted the AI output due to his inexperience with the technology and that “the summary read well,” so he copied it as is.
The minister’s counsel argued that the lawyer had failed to exercise due diligence and that, in the public interest, an example should be set to prevent AI misuse in the justice system. The judge referred the case to the New South Wales disciplinary regulator for investigation, with judicial authorities stressing that such conduct should be “nipped in the bud” to deter similar behavior.
That same year in Melbourne, in a Family Court proceeding, another lawyer submitted a list of supposedly relevant legal precedents that, upon review, were also fictitious. The judge and clerks could not find several of the cited cases in any official database. The lawyer later admitted to using legal software (Leap) with AI functionality, which “hallucinated” the citations—and he had not bothered to verify them. The hearing was suspended, and the matter was reported to the state’s legal complaints authority for possible disciplinary action.
As a preventive measure, by late 2024, the Supreme Court of New South Wales issued guidance prohibiting lawyers from using generative AI to draft affidavits, witness statements, character references, or other evidentiary materials submitted to the court. In other jurisdictions, appellate judges have gone so far as to omit AI-generated erroneous references from their rulings to avoid perpetuating the problem.
Local experts point out that these errors stem from both an “AI literacy problem” (lack of understanding of how it works and its limitations) and negligence—especially among under-resourced or inexperienced practitioners—highlighting the need for training lawyers in responsible AI use.
Other Examples – Spain and the UK
- Spain (2024): A complaint drafted with the help of ChatGPT incorrectly cited Colombia’s Penal Code instead of Spain’s Penal Code.
- UK (2023): In a tax appeal, a self-represented taxpayer—advised informally by a friend at a law firm—submitted a filing that cited nine supposedly relevant cases, all of which turned out to be fabricated by ChatGPT.
The Common Thread
In Italy, Australia, Spain, and the UK, lawyers who used ChatGPT to draft legal documents ended up citing rulings that do not exist.
Generative AI hallucinations are not science fiction—they are a real problem when used without control, context, or legal validation. They have led to financial penalties, disciplinary proceedings, public embarrassment, and stern judicial warnings.
Across all jurisdictions, the legal community agrees: technology is a useful ally only if used responsibly. Verification against official sources and human oversight remain essential.
The BrandLex AI Difference
At BrandLex AI, we anticipated these risks and built a different kind of technology.
BrandLex AI is a professional virtual legal agent, developed on secure, confidential infrastructure, using language models trained with specific legal data.
BrandLex AI reasons with real laws and sound legal judgment.
- Developed with supervised models trained by lawyers to deliver reliable, contextualized, and up-to-date answers, based on real legal sources, official legislation, and verified case law.
- Built with high standards of security, privacy, and confidentiality.
- Fully adaptable to your organization’s needs—we can deploy BrandLex AI in your own cloud, meeting your security, compliance, and privacy standards.
Custom Infrastructure
- Model trained with your internal legal information
- Secure integration into your tech ecosystem
Because in legal matters and sensitive data, trust is built from the architecture up.
Artificial intelligence is only useful if trained on reliable information. In legal contexts, that’s non-negotiable.
📧info@brand-lex.com