Seven years after it first rose to prominence in eDiscovery, technology-assisted review remains an important, and at times controversial, tool in the eDiscovery practitioner’s toolkit
In “Still Crazy after All These Years,” we discussed the slow but steady growth in the importance of TAR. In “In the Beginning Was da Silva Moore,” we discussed the first case to address TAR. In “Questions of Choice in Kleen Products,” we discussed an attempt to force the use of TAR. In this Part, we review the first instance of reported TAR results.
The next significant technology-assisted review decision came from a state court in Virginia and included both the first results reported on the record and the first instance in which use of TAR was judicially approved over the objections of the requesting parties. Global Aerospace Inc. v. Landow Aviation, L.P. (Loudoun County, Va. Cir. Ct. Apr. 23, 2012) concerned the collapse of three airplane hangars and the resulting destruction of fourteen private jets.
The Defendants undertook extensive document preservation and collection activities soon after the incident occurred and collected over 8TB of ESI in their initial effort. Even after de-duplication, de-NISTing, and other filtering, 200GB of reviewable ESI remained. This was supplemented by an additional 50GB of reviewable materials gathered in supplemental collection efforts. Altogether, the Defendants estimated that they had in excess of 2,000,000 unique documents requiring review for potential production in the case.
Due to the large volume of this review set, the Defendants proposed using a technology-assisted review solution (referred to here as predictive coding) instead of doing a complete first-pass review or relying on keyword searching. The Plaintiffs did not agree to the Defendants’ proposal, so the Defendants moved the Judge for a protective order allowing them to proceed over the Plaintiffs’ objections.
In their “Memorandum in Support of Motion for Protective Order Approving the Use of Predictive Coding,” the Defendants set forth their technology-assisted review proposal and make a strong case for technology-assisted review in general. They discuss the relative reliability and costs of manual review, keyword searching, and predictive coding, using studies and the Moore case to argue that TAR will likely have lower costs and better results. Moreover, they include the results of keyword search term tests on the data to demonstrate their relative ineffectuality.
The Defendants also explain the TAR process that they will employ, which includes transparency into the training process for the Plaintiffs and a commitment to hit a minimum recall rate of 75% (i.e., to find at least 75% of the responsive materials that exist within the ESI collection). They based this target on studies showing the average human review recall rate to be 59.3% and the average predictive coding recall rate to be 76.7%. They also preemptively explain why a recall rate of 75% is sufficient to satisfy the “reasonable inquiry” requirement of the relevant procedural rules:
Ironically, what is being proposed in this case to ensure “reasonable inquiry” is far more than has ever been done with traditional discovery. Never has it been suggested that a producing party would be obligated to sample the documents determined to be irrelevant by first-pass reviewers to demonstrate the adequacy of that first-pass review. Nor is it a typical practice to review the documents left behind by a keyword search, even though upwards of 80% of the relevant documents typically are missed. The ESI protocol proposed by [the Defendants] goes well beyond what would otherwise be available to opposing counsel to ensure that relevant documents are being reviewed for production. [emphasis added]
The Plaintiffs’ primary objection to the Defendants’ proposal was to the proposed recall rate of 75%. In their opposition brief, the Plaintiffs argued that the rules required the production of all responsive materials, not just 75% of them. The Plaintiffs did not just want the protocol changed to achieve a higher recall rate; they wanted traditional human document review used instead.
The Judge granted the Defendants’ motion allowing them to proceed with their proposed TAR approach, despite the Plaintiffs’ objections. Unfortunately, the Judge’s order does not go into any useful detail regarding the arguments made for and against TAR use or for and against the proposed recall rate.
Subsequently, however, the case did provide us with reported results of the completed TAR process. At a cost hundreds of thousands of dollars less than full manual review, the TAR approach they employed achieved a recall rate of 81%, beating the target they’d set for themselves and leaving just under 32,000 missed documents among the 1.1 million documents deemed irrelevant by the process. The Plaintiffs did not object to the adequacy of these results.
Upcoming in this Series
In the next Part of this series, we will review the In Re: Actos litigation, in which the parties successfully negotiated a detailed TAR protocol.
About the Author
Matthew Verga, JD
Director, Education and Content Marketing
Matthew Verga is an electronic discovery expert proficient at leveraging his legal experience as an attorney, his technical knowledge as a practitioner, and his skills as a communicator to make complex eDiscovery topics accessible to diverse audiences. An eleven-year industry veteran, Matthew has worked across every phase of the EDRM and at every level from the project trenches to enterprise program design. He leverages this background to produce engaging educational content to empower practitioners at all levels with knowledge they can use to improve their projects, their careers, and their organizations.