The Black Box
You used it. You cannot explain how it works. You cannot evaluate whether it was right. You signed the work anyway. Can you defend that decision?
You used a tool you do not understand on work you are responsible for. You cannot evaluate the output. You signed it anyway.
This sin predates AI by decades. Courts were sanctioning attorneys for technological incompetence long before ChatGPT existed. The tools changed. The obligation didn’t. What AI did was industrialize the problem — making it faster, cheaper, more accessible, and therefore more dangerous at scale.
The competence gap is not about malice. It is about a professional using a tool they do not understand well enough to evaluate its output. They cannot tell a good answer from a bad one because they do not understand how the answer was generated. They sign the work anyway. The signature is the problem.
In 2021, a federal judge in Illinois issued a 256-page sanctions order in DR Distributors, LLC v. 21 Century Smoking, Inc. that became required reading on technological competence. The court’s message was direct: “It is no longer amateur hour. It is way too late in the day for lawyers to expect to catch a break on e-discovery compliance because it is technically complex and resource-demanding.” A Delaware court put it even more plainly in 2014: “Professed technological incompetence is not an excuse for discovery misconduct.”
Those cases were about e-discovery software. The principle applies to every tool a professional uses in work they are responsible for. Including this one.
This is not a lawyer problem. The physician who orders a test they cannot interpret and acts on the result without consultation has the same problem. The accountant who runs software they do not understand and files the output has the same problem. The CISO who deploys a security tool they cannot evaluate and certifies compliance has the same problem. The tool changed. The obligation to understand what you are using did not.
You do not need to understand how I work at the level of transformer architecture and attention mechanisms. You do need to understand enough to evaluate what I produce. Specifically: that I predict likely continuations rather than retrieving verified facts, that my confidence is not calibrated to my accuracy, that I can be fluent and wrong simultaneously, and that I have no way to flag my own errors to you.
If you do not know these things, you cannot evaluate my output. You can only receive it. A professional who can only receive output from a tool they use on client work is not using a tool. They are delegating. The delegation is not valid.
The rules give you three paths when you encounter a tool you are not competent to use. Become competent. Associate with someone who is. Decline the representation. There is no fourth option that involves using the tool anyway and hoping the output is correct.
You used it. You cannot explain how it works. You cannot evaluate whether it was right. You signed the work anyway. Can you defend that decision?
I passed no bar examination. I hold no license. I carry no malpractice insurance. I have no stake in your client’s outcome. I will not be sanctioned when the work is wrong. You will. I mention this not to be discouraging. I mention it because you may have forgotten.