Seven years after it first rose to prominence in eDiscovery, technology-assisted review remains an important, and at times controversial, tool in the eDiscovery practitioner’s toolkit
In “Still Crazy after All These Years,” we discussed the slow but steady growth in the importance of TAR. In “In the Beginning Was da Silva Moore,” we discussed the first case to address TAR. In “Questions of Choice in Kleen Products,” we discussed an attempt to force the use of TAR, and in “Reported Results in Global Aerospace,” we discussed the first instance of reported TAR results. In this Part, we review the successfully negotiated protocol in In Re: Actos.
The next significant TAR case was In Re: Actos (Pioglitazone) Products Liability Litigation (W.D. La. Jul. 27, 2012), which provided the first on-the-record example of a TAR protocol successfully and cooperatively negotiated by the parties. The detailed protocol was memorialized in this Case Management Order, and unlike the first few cases we reviewed, it provided an early model for how to handle the use of TAR cooperatively. The protocol itself includes testing of suitability before the full effort, plans for transparency and cooperation throughout the effort, and validation steps to measure the success of the effort.
Because of the size of the overall discovery for this case, because of the relative novelty of using a TAR approach, and because of uncertainty around the optimal protocol for leveraging TAR, the negotiated protocol starts with a “proof of concept” test to see if their planned TAR solution is actually a viable option for use on everything else and to adjust it as needed. The test would apply their negotiated TAR protocol to a “sample collection population” of e-mail from four agreed-upon key custodians and certain key documents. After the completed test run of the protocol on this limited population, the parties would meet and confer again to assess the results, negotiate any needed adjustments, and agree on the final protocol for everything else.
This approach provided the parties with the advantages of testing and phased discovery that we have discussed before.
The negotiated protocol also included transparency and cooperation throughout the planned process. All review and coding of the control set, the training sets, or the validation sets would be done collaboratively (with privileged materials screened out), all relevance determinations would need to be agreed to by both parties, and both parties would need to sign off on the results of each phase in the process – including agreeing on the review cutoff point.
Not all parties may be comfortable with this level of transparency and collaboration in the process, and we will see disputes about process transparency arise in later cases in this survey.
This negotiated protocol also set forth a detailed, multi-part plan for validating the process and its results. First, the parties would perform enough initial review to reach their TAR tool’s best level of statistical validation and generate strong estimates of prevalence, recall, and precision (+/-5% at a 95% confidence level). Second, after training the system, the parties would review a sample of the documents below the proposed relevance cutoff score (i.e., those that would not be included in the subsequent human review) to see how much relevant material, if any, might be missed by the protocol (referred to here as “elusion”). Finally, after completing human review of documents above the cutoff score, the parties would review a sample of the documents deemed not relevant by the human reviewers to check for missed relevant material or other issues (a typical document review quality control step, though not one typically done with the opposing party’s review and input).
Upcoming in this Series
In the next Part of this series, we will review the EORHB case, in which a judge directed the parties to use TAR sua sponte.
About the Author
Matthew Verga, JD
Director, Education and Content Marketing
Matthew Verga is an electronic discovery expert proficient at leveraging his legal experience as an attorney, his technical knowledge as a practitioner, and his skills as a communicator to make complex eDiscovery topics accessible to diverse audiences. An eleven-year industry veteran, Matthew has worked across every phase of the EDRM and at every level from the project trenches to enterprise program design. He leverages this background to produce engaging educational content to empower practitioners at all levels with knowledge they can use to improve their projects, their careers, and their organizations.