A man who was injured on a flight to New York tried to sue an airline. The lawyer he hired used ChatGPT to write the brief submitted to the court. It was full of absolute garbage as ChatGPT had invented pretty much everything.
Mr. Schwartz, who has practiced law in New York for three decades, told Judge P. Kevin Castel that he had no intent to deceive the court or the airline. Mr. Schwartz said that he had never used ChatGPT, and “therefore was unaware of the possibility that its content could be false.”
He had, he told Judge Castel, even asked the program to verify that the cases were real.
It had said yes.
Amid all of the uncertainty surrounding ChatGPT and artificial intelligence in general, the one thing I’ve been reminding myself repeatedly is LLMs – large language models – are designed to come up with coherent (grammatically speaking) text responses following a prompt provided by the user. These responses are based on the text the LLM was trained on, which can represent large swathes of the internent, but not everything ever written. While this can be useful in a number of cases, there are other scenarios where it is downright dangerous to trust the response provided, as Mr. Schwartz found out.
My impression is that non-technical people using LLMs like ChatGPT for niche purposes have been seeing results like those produced in the above story. The situations where LLMs are proving to be much more proficient are where technical people are deploying the ChatGPT API, training it using a specific set of texts, and then using it to answer prompts querying information from those texts. Dave Winer fed ChatGPT his entire blogging history, for example, and has seen great results asking it questions about his past writings.
I would imagine if a law firm were to feed a whole bunch of case law (aviation case law, for example), and then create prompts specific to that case law, that law firm would not have ended up in the same jeopardy as described in the NYT story. Users who believe vanilla ChatGPT alone can produce a solution to their niche questions should do so at their peril.