How to Use Large Language Models in Legal Work

If you've been paying any attention to legal tech over the last year, you've probably seen the headlines.

Lawyer sanctioned for submitting AI-generated brief citing fake cases.

The risks are real, and it's easy to see why lawyers are cautious. Large Language Models (LLMs) like ChatGPT and Claude are transforming legal research, drafting, and analysis. But they also have a notorious weakness: the tendency to hallucinate—to generate convincing-sounding information that is simply not true.

Here's why that happens—and what you can do about it.

"Treat the text returned from AI as a first draft: something for you to verify."

Why Do LLMs Hallucinate?

LLMs work by predicting the next word in a sequence.

At first glance, that sounds fine. But problems arise when they begin predicting based on patterns rather than absolute truth.

If you ask:

"Show me an example of a slip and fall case involving a dry cleaner in North Carolina in 2015."

...the model wants to be helpful. It also knows what a case citation looks like.

Instead of saying, "I don't know," the LLM might invent a plausible-looking citation:

Smith v. Jones, 246 N.C. App. 219 (2015)

And that's how a well-meaning request can turn into an embarrassing—and potentially career-damaging—filing.

The Risks Are Real

For lawyers, hallucinations are far more than an inconvenience. They create real risks:

• Professional Discipline: Courts can and have sanctioned lawyers who submit filings with fake citations.
• Client Harm: Relying on false information can weaken legal arguments, cause valid claims to be dismissed, or worse.
• Reputational Damage: Once established as a careless user of tools, it can be hard to rebuild trust with courts and clients.

Four Strategies to Avoid Hallucinations

AI is powerful, but you must treat it responsibly. Here are four methods to use it safely:

1. Give the Model More Context

If you ask a model: "Write me a brief about [topic] citing police report codes for driver impairments," it may guess or hallucinate.

Instead, give it the actual reference material:

"Here is the text of the North Carolina Crash Reporting Instruction Manual. Extract the section on alcohol impairment codes and list them here."

The more context you provide, the less the model has to improvise.

2. Validate with a Second Model

It sounds simple, but it works.

• See if another prompt (like asking it to "verify" the citation) yields a different result. Using Google Gemini or Claude to verify an OpenAI result can often spot obvious slips.
• If both agree, your confidence increases. If they differ, that's a flag to investigate further.

Even in the rare times both models hallucinate, they will do so in different ways, and still give you conflicting information.

3. Use Careful Question Framing

Asking it to "search" reduces the chance of guessing from rote association.

Frame instructions as processing tasks (summarizing, reformatting) rather than retrieval tasks. Or better yet, tell the model explicitly: "Do NOT invent facts. If the answer is not in the text, state that you do not know."

Try: "Does this text support the claim?"

Instead of asking:

"What is the treatment date?"

Ask:

1. "Does this medical report list a treatment date?"

Then, ONLY if the answer is "yes", ask it to extract it.

4. Watch Out for Example Hallucination

One of the most common ways LLMs hallucinate is when you accidentally prompt them to. Providing examples (few-shot prompting) can sometimes pressure the model to invent details that fit your example rather than actual facts.

Be careful when providing example formats—the model may try too hard to match your pattern.

"The real competitive advantage isn't just using AI—it's using it responsibly to protect your reputation and your clients."

The Bottom Line

Generative AI is here to stay, and properly used, it is a tremendous tool—faster research, automated drafting, and a serious edge over firms still billing everything by hand.

But even as these models continue to improve, the risks remain. In fact, as users get more comfortable with LLMs and grow to trust them, the risks could grow worse. This complacency is where errors can sneak in.