“AI vs. Lawyers” – Interesting Result, Bad Headline

The recent clickbait headline “AI vs. Lawyers: The Ultimate Showdown” might lead some to believe that an artificial intelligence system and a lawyer were dueling adversaries or parties on opposite sides of a legal dispute (notwithstanding that an “intelligent” machine has not, as far as US jurisprudence is concerned, been recognized as having machine rights or standing in state or federal courts).

Follow the link, however, and you end up at LawGeex’s report titled “Comparing the Performance of Artificial Intelligence to Human Lawyers in the Review of Standard Business Contracts.” The 37-page report details a straightforward, but still impressive, comparison of the accuracy of machine learning models and lawyers in the course of performing a common legal task.

Specifically, LawGeex set out to consider, in what they call a “landmark” study, whether an AI-based model or skilled lawyers are better at issue spotting while reviewing Non-Disclosure Agreements (NDAs).

Issue spotting is a task that paralegals, associate attorneys, and partners at law firms and corporate legal departments regularly perform. It’s a skill learned early in one’s legal career and involves applying knowledge of legal concepts and issues to identify, in textual materials such as contract documents or court opinions, specific and relevant facts, reasoning, conclusions, and applicable laws or legal principles of concern. Issue spotting in the context of contract review may simply involve locating a provision of interest, such as a definition of “confidentiality” or an arbitration requirement in the document.

Legal tech tool using machine learning algorithms have proliferated in the last couple of years. Many involve combinations of AI technologies and typically required processing thousands of documents (often “labeled” by category or type of document) to create a model that “learns” what to look for in the next document that it processes. In the LawGeex’s study, for example, its model was trained on thousands of NDA documents. Following training, it processed five new NDAs selected by a team of advisors while 20 experienced contract attorneys were given the same five documents and four hours to review.

The results were unsurprising: LawGeex’s trained model was able to spot provisions, from a pre-determined set of 30 provisions, at a reported accuracy of 94% compared to an average of 85% for the lawyers (the highest-performing lawyer, LawGeex noted, had an accuracy of 94%, equaling the software).

Notwithstanding the AI vs. lawyers headline, LawGeek’s test results raise the question of whether the task of legal issue spotting in NDA documents has been effectively automated (assuming a mid-nineties accuracy is acceptable). And do machine learning advances like these generally portend other common tasks lawyers perform someday being performed by intelligent machines?

Maybe. But no matter how sophisticated AI tech becomes, algorithms will still require human input. And algorithms are a long way from being able to handle a client’s sometimes complex objectives, unexpected tactics opposing lawyers might deploy in adversarial situations, common sense, and other inputs that factor into a lawyer’s context-based legal reasoning and analysis duties. No AI tech is currently able to handle all that. Not yet anyway.