Of the many changes former U.S. Magistrate Judge Andrew J. Peck saw during his decades on the bench at the U.S. District Court for the Southern District of New York, there is one in particular that is continuing to cause him growing concern.
“When I got on the bench in 1995, the privilege logs in a typical case [were] two to three pages, maybe 50-100 entries,” he said during the Legalweek 2024 Judges Debate keynote in January. “Now the privilege logs are like little novels, and there may be 10,000 or more entries. That is very expensive and is often useless to the other side in figuring out what is or isn’t privileged.”
It’s no coincidence that privilege log volume grew alongside a technological transformation in the industry that saw legal communications move from paper memos and faxes to emails and messaging apps. Now, legal teams are struggling to rein in all the new electronically stored information (ESI)—and privileged content—they create.
But if technology caused this problem, can it help mitigate its impact as well? Can one use AI, for instance, to accurately and quickly flag privileged content, making privilege reviews less manual, expensive and labor-intensive?
It’s possible, but not to the extent that some would think. For all the advances in AI—especially in document review—flagging privileged documents is one area where experts say the technology isn’t there yet.
Still, there is hope on the horizon. Many are confident that AI’s shortcomings, including its limited accuracy and high cost in privilege review, will eventually be overcome. But even then, there still may be limits to just how much efficiency this technology can bring.
Where the AI seems more immediately promising, however, is in another part of the privilege process: drafting privilege logs. Some in the legal industry are already finding a level of success using generative AI to draft logs, an opportunity they see becoming even more viable in the months to come.
Such a situation underscores the nature of legal data today: it is far easier to create ESI than it is to organize or review it.
This Isn’t Your Usual Review
Attorneys have always erred on the side of caution when deciding whether to flag a document as privileged. But their circumspect nature isn’t likely to blame for the growth of privilege logs. Instead, it comes down to the new ways they communicate.
“Log volume, in my view, is entirely related to the massive growth and the overall volume of communications, both by individuals and businesses,” said Robert D. Keeling, partner at Sidley Austin, and founder and head of the firm’s e-discovery and data analytics group.
He explained, “For example, on the amount of privileged documents in a document population, we have not seen the percentages change over time. Instead, the volume, however, of communication has grown as employees and businesses transitioned first to email and later to applications and messaging applications, such as Slack or Microsoft Teams.”
ESI stored in these data repositories can be accessible to legal teams. AI-powered e-discovery tools and AI processes such as technology-assisted review (TAR) are becoming increasingly adept at searching through datasets—including those from collaboration and messaging apps—to find responsive data. And there is a place for such tools in privilege review as well.
“We will use what’s known as supervised machine learning or unsupervised machine learning and use other models and other analytics to help identify privileged communications that are not being identified by traditional manual review alone,” Keeling said. “So we augment our privilege review with the technology.”
But AI isn’t as effective in flagging privilege as it is with other attributes. Keeling added that “identifying privileged documents in a population is more difficult … than a traditional responsiveness review. And the reason is because privilege is typically a more nuanced problem and a more difficult issue for traditional AI and data analytics to confront.”
As an example, he pointed to a situation where a client sends an email to an attorney with an attached document that describes a policy on which the client requires legal advice. While an AI tool may flag the email itself as privileged, when looking at the attached document, “it just sees a policy—it’s not connected to the parent email asking the lawyer for advice,” Keeling said. “And so that document might be classified as not privileged by an algorithm.”
What’s more, AI tools have not yet tackled the more nuanced types of privilege beyond just attorney-client communication.
“We’re seeing [AI] perform better with attorney client communication, but … there’s also a number of other privileges in specific industries like the financial services industry, [such as] the bank examiner privilege and confidential supervisory information (CSI), as well as suspicious activity reports (SAR) … I’ve seen [no] models that are focused on those industry specific privileges, which obviously has an impact if you’re supporting the financial services space,” said Scott Milner, partner at Morgan, Lewis & Bockius and leader of the firm’s eData practice.
Less Expensive, More Accurate
AI’s shortcomings means that in privilege review, it is often deployed in limited ways. The reason stems not only from accuracy problems, but also cost concerns as well.
Peck, for instance, told Legaltech News that the expense of ingesting large datasets into advanced AI tools for privilege review can be prohibitive for many legal teams. “It’s sort of like where we were with TAR 12 years ago, in that it’s more expensive to run,” he said.
Milner sees the same problem, adding that “the tools just aren’t there yet from an accuracy perspective, and candidly, it’s also a cost perspective. … If I were to use it, I would be hesitant if I had to put my entire dataset through it.”
However, as with any new technology, many believe such high costs will eventually come down, and more importantly, that AI’s accuracy will increase, though not to a level where it can fully automate privilege review.
“I do think these tools will eventually get to a point where … we will be able to do a very strong first pass privilege review using these platforms,” Milner said. “We will see that for standard privileges like attorney-client and work product. [But] I do think that … certain industries will have to probably wait a little bit longer for using some of these tools to reduce or potentially eliminate first pass review for privilege.”
Keeling similarly sees such improvements on the horizon. “I think the way to look at this is that [with] AI and generative AI, we will see improvements in the ability to identify privileged information. And those improvements will help with the efficiency and the effectiveness of document review and potentially could alleviate some of cost and burden of privilege. … So it is not that they cannot be used, but typically they would be used in combination with more traditional techniques to identify privilege communication.”
Drafting Faster, but Not Shorter
While AI may, for now, have difficulty identifying privileged content, there’s more promise with using the technology—specifically generative AI—to draft the privilege logs themselves.
This isn’t, however, an easy task for AI either. “It’s a very hard problem, because in some cases, it is appropriate to summarize the entire document for your privilege log entry, and [in] other cases, it is more appropriate to summarize the portion of the document that contains or reflects the legal advice is given,” Keeling said.
In fact, determining how much information to reveal when drafting a privilege log needs to strike a delicate balance. “It becomes a Goldilocks principle—if just says communication about the ‘XYZ’ litigation, that doesn’t really tell you anything,” Peck explained. “If it says it’s a communication about the ‘XYZ’ litigation where counsel was recommending to the client doing ‘A, B, C and D,’ it then is revealing the privileged information. So it takes a certain amount of training and review of the output.”
Still, those who have used generative AI to draft privilege logs—primarily as a first pass that will then require manual review and edits—note that the technology is fast improving. “The technology is still under development,” Keeling said. “But we have seen significant improvements over the last nine to 10 months. … That said, in our testing, and especially recently over the past few months, the technology is working better than I thought it would.”
He added, “We are seeing [that] perhaps a minority of the documents that are created by generative AI would be appropriate to use in a matter … the percentage of entries that we would consider appropriate has been increasing with each iteration.”
As to when generative AI will improve enough that a majority of the privilege logs it creates would be deemed appropriate, Keeling sees that happening relativity soon. “If I have to look into my crystal ball, I think we are probably one to two years away from that.”
By: Rhys Dipshan
Leave A Comment