Just found out my chatbot hallucinated a court case reference, scary stuff

I was prepping a legal research assistant demo for a client in Phoenix and asked an AI model to pull relevant case law. It gave me a perfect citation with a docket number and everything, looked super official. Took me 20 minutes to realize the case didn't exist, I double checked on Westlaw and it was total garbage. How are any of you trusting these tools for factual work without a human eyeballing every single output? Has anyone else had a model invent a source that sounded way too real?

1 comments

1 Comment

brooke44815d agoMost Upvoted

That's genuinely terrifying, especially when it looks so polished and real.