Livingston vs. City of Chicago: What it means for eDiscovery
Statistical Sampling Techniques in eDiscovery Workflows: Part II
September 21, 2020
Antitrust cases against GAFA
September 15, 2021
Technology Assisted Review (TAR) is now considered an essential tool in eDiscovery. Its popularity is, of course, closely tied to the increasing amounts of data used in litigation, especially when contrasted with the time, accuracy and cost of manual review.
As far back as 1985, a study by Blair and Maron found that TAR software outperformed human review in accurately identifying responsive documents. Whereas attorney-supervised paralegals believed they had found at least 75% of relevant documents, they had actually only found 20% while TAR identified 75% of the responsive documents.
However, TAR’s role is only to augment human capabilities, not replace them. Human review and software programming are vital to the success of TAR. But human reviewers now only need to review a fraction of the documents collected as opposed to the entire collection, allowing people to more quickly make better and more accurate decisions.
Courts today widely accept the use of TAR in the e-discovery process. Nevertheless, there’s still some confusion about the details of how it can be used, and whether it might be manipulated to unfairly skew the evidence in a case. This latter argument, as well as several other concerns relating to intricacies of using TAR, were recently tested in the case of Livingston vs. the City of Chicago.
On September 3, 2020 a motion to compel the City to use TAR for identifying responsive ESI was denied. Perhaps it’s not irrelevant that the case had dragged on for some four years prior to that, and that the number of pages of emails identified for review totaled some 9 million. Nevertheless, the court’s decision highlights an overarching understanding of the pragmatics of using these tools in e-discovery, and a reinforcement of findings in previous cases.
To summarize, Plaintiffs had objected to the City’s proposal to use TAR to identify relevant responsive documents from the subset of 192,000 unique emails – approximately 1.3 million pages of documents – produced as a result of keyword search. The plaintiffs tried to have the courts to block or otherwise micro-manage the use of TAR for several reasons:
Finding in favor of the City’s proposed use of TAR, the court noted that it stated that the plaintiffs’ insistence that the City must collaborate with them on the TAR process had no foothold in the federal rules. Moreover, using TAR on the entire ESI collection would be wasteful and unduly burdensome, and would further delay the resolution of this almost four-year-old case.
This ruling is supported by a number of judgments over the years. Hyles v. New York City (S.D.N.Y. Aug. 1, 2016) ruled that a party could not be forced to use a particular technology (TAR in this instance) if it preferred another, stating that the goal of using technology was not to achieve perfection but to achieve reasonable and proportional search results. Other courts have similarly found that there is no basis on which they can compel a defendant to use a particular form of ESI.
In the Livingston case, the court found that the use of TAR was both reasonable and proportional to the needs of this case. It also spotlighted the importance of not holding TAR to a higher standard than other review methods.
In Da Silva Moore v. Publicis Groupe (Southern District of New York, 2012), the judge ruled that the transparency of the process should reduce fears about using TAR. Since then courts have repeatedly held that a party may rely on TAR if it chooses to do so. The Livingston finding also underscores the importance of disclosing the responding party’s use of TAR and the general importance of transparency concerning the type of TAR tool and validation measures. The court decided that, having disclosed the TAR software it intended to use and how it intended to validate the review results, sufficient information had been provided to make the production transparent.
On Oct. 1, 2018, an amendment to the Rules of the Commercial Division of the Supreme Court codified the Commercial Division’s supports of the use of TAR discovery, encouraging parties to “use the most efficient means to review documents, including electronically stored information (ESI), that is consistent with the parties’ disclosure obligations…and proportional to the needs of the case.”
Underscoring the argument for reasonableness, the Livingston court judged that ‘While the City may dump (sic) all 1.3 million pages of documents on Plaintiffs with an entry of a Rule 502(d) order, it also has the right to perform a review to produce only those documents that are responsive and relevant.’
As to the plaintiffs’ concern that the City’s attorneys would improperly train the TAR tool, the court judged that since the City had expressed its intention to use a number of quality control applications within their selected technology, these fears were invalid.
The Livingston case showcases the courts’ ongoing focus on what’s reasonable, proportional, and practical. It also highlights their reluctance to be used as micro-managers of the parties’ internal review processes while balancing the need for transparency with the rights of a party to determine how best to meet its discovery obligation. An important case for the future of TAR.
As far back as 1985, a study by Blair and Maron found that TAR software outperformed human review in accurately identifying responsive documents. Whereas attorney-supervised paralegals believed they had found at least 75% of relevant documents, they had actually only found 20% while TAR identified 75% of the responsive documents.
However, TAR’s role is only to augment human capabilities, not replace them. Human review and software programming are vital to the success of TAR. But human reviewers now only need to review a fraction of the documents collected as opposed to the entire collection, allowing people to more quickly make better and more accurate decisions.
Courts today widely accept the use of TAR in the e-discovery process. Nevertheless, there’s still some confusion about the details of how it can be used, and whether it might be manipulated to unfairly skew the evidence in a case. This latter argument, as well as several other concerns relating to intricacies of using TAR, were recently tested in the case of Livingston vs. the City of Chicago.
On September 3, 2020 a motion to compel the City to use TAR for identifying responsive ESI was denied. Perhaps it’s not irrelevant that the case had dragged on for some four years prior to that, and that the number of pages of emails identified for review totaled some 9 million. Nevertheless, the court’s decision highlights an overarching understanding of the pragmatics of using these tools in e-discovery, and a reinforcement of findings in previous cases.
To summarize, Plaintiffs had objected to the City’s proposal to use TAR to identify relevant responsive documents from the subset of 192,000 unique emails – approximately 1.3 million pages of documents – produced as a result of keyword search. The plaintiffs tried to have the courts to block or otherwise micro-manage the use of TAR for several reasons:
- Using TAR would violate an earlier e-discovery order compelling the City to use the Plaintiff’s preferred method of keyword searches to identify relevant documents.
- They also claimed that this previous order mandated the production of all search hits) rather than responsive documents only.
- If the City were allowed to use TAR, it should do so only according to the plaintiffs’ proposed TAR protocol. This would have meant the City abandoning the pre-culled data set and instead running TAR against the entire collected set of approximately 1.5 million documents emails.
- The plaintiffs also expressed concern that document review team would improperly train the TAR tool by making incorrect responsiveness determinations or prematurely ending the review.
Finding in favor of the City’s proposed use of TAR, the court noted that it stated that the plaintiffs’ insistence that the City must collaborate with them on the TAR process had no foothold in the federal rules. Moreover, using TAR on the entire ESI collection would be wasteful and unduly burdensome, and would further delay the resolution of this almost four-year-old case.
This ruling is supported by a number of judgments over the years. Hyles v. New York City (S.D.N.Y. Aug. 1, 2016) ruled that a party could not be forced to use a particular technology (TAR in this instance) if it preferred another, stating that the goal of using technology was not to achieve perfection but to achieve reasonable and proportional search results. Other courts have similarly found that there is no basis on which they can compel a defendant to use a particular form of ESI.
In the Livingston case, the court found that the use of TAR was both reasonable and proportional to the needs of this case. It also spotlighted the importance of not holding TAR to a higher standard than other review methods.
In Da Silva Moore v. Publicis Groupe (Southern District of New York, 2012), the judge ruled that the transparency of the process should reduce fears about using TAR. Since then courts have repeatedly held that a party may rely on TAR if it chooses to do so. The Livingston finding also underscores the importance of disclosing the responding party’s use of TAR and the general importance of transparency concerning the type of TAR tool and validation measures. The court decided that, having disclosed the TAR software it intended to use and how it intended to validate the review results, sufficient information had been provided to make the production transparent.
On Oct. 1, 2018, an amendment to the Rules of the Commercial Division of the Supreme Court codified the Commercial Division’s supports of the use of TAR discovery, encouraging parties to “use the most efficient means to review documents, including electronically stored information (ESI), that is consistent with the parties’ disclosure obligations…and proportional to the needs of the case.”
Underscoring the argument for reasonableness, the Livingston court judged that ‘While the City may dump (sic) all 1.3 million pages of documents on Plaintiffs with an entry of a Rule 502(d) order, it also has the right to perform a review to produce only those documents that are responsive and relevant.’
As to the plaintiffs’ concern that the City’s attorneys would improperly train the TAR tool, the court judged that since the City had expressed its intention to use a number of quality control applications within their selected technology, these fears were invalid.
The Livingston case showcases the courts’ ongoing focus on what’s reasonable, proportional, and practical. It also highlights their reluctance to be used as micro-managers of the parties’ internal review processes while balancing the need for transparency with the rights of a party to determine how best to meet its discovery obligation. An important case for the future of TAR.
