Seven years after it first rose to prominence in eDiscovery, technology-assisted review remains an important, and at times controversial, tool in the eDiscovery practitioner’s toolkit
In “Still Crazy after All These Years,” we discussed the slow but steady growth in the importance of TAR. In “In the Beginning Was da Silva Moore,” we discussed the first case to address TAR. In “Questions of Choice in Kleen Products,” we discussed an attempt to force the use of TAR, and in “Reported Results in Global Aerospace,” we discussed the first instance of reported TAR results. In “A Negotiated Protocol in In Re: Actos,” we discussed a successfully negotiated TAR protocol. In “At a Judge’s Direction in EORHB,” we discussed a Judge ordering TAR use unprompted. In “Debating Process and Transparency in Biomet,” we discussed questions of process and transparency. In “Changing Horses Midstream in Progressive and Bridgestone,” we discussed trying to switch to a TAR approach after agreeing to a traditional one. In “Seeking Approval and Perfection in Dynamo,” we reviewed the questions of whether to seek prior approval and how good is good enough. In “Bringing it Full Circle in Rio Tinto,” we discussed Judge Peck’s second prominent TAR case, and in “Trying to Compel TAR Use in Hyles,” we discussed his third and final. In this Part, we review last year’s Winfield case for its analysis of process adequacy and required transparency.
The next prominent technology-assisted review case requiring our attention is Winfield v. City of New York (S.D.N.Y. Nov. 27, 2017). The case wound up before the Magistrate Judge for the resolution of a variety of discovery disputes, including the Plaintiffs’ challenges to the City’s TAR process and their attempts to compel additional transparency regarding that process.
Before reaching the disputes over the use of TAR, discovery in this case was already contentious:
Plaintiffs have sought wide-ranging discovery, which the City has resisted vigorously. This Court has issued various rulings consistent with Federal Rule of Civil Procedure 26(b)(1) to ensure that discovery is focused on information that is relevant and proportional to the needs of this case . . . . This included directing phased discovery and limiting some of Plaintiffs’ discovery demands.
Ultimately, the City “was ordered to collect documents, including electronic documents, from 50 custodians” and to “a set of search terms that was heavily negotiated by the parties, with some guidance from the Court.”
The City began traditional manual review of these materials, but “Plaintiffs lodged numerous complaints about the pace of discovery and document review.” In response the Magistrate Judge directed the City to complete manual review of “certain custodians and begin using Technology Assisted Review (‘TAR’) software (also commonly referred to as ‘predictive coding’) to hasten the identification, review, and production of documents responsive to Plaintiffs’ document requests.”
After another dispute over the appropriate list of search terms to be applied to a particular tranche of custodians, the City agreed to add the additional search terms requested by the Plaintiffs if they could then use their now- trained TAR tool to reduce the approximately 90,000 additional results the search terms would produce to a more manageable number for human review. The Plaintiffs objected “that the City’s proposal [was] not acceptable because they [were] concerned about the reliability of the City’s predictive coding processes.”
The Plaintiffs alleged that the City had “over-designated documents as privileged and non-responsive” resulting in an improperly trained tool that was “unable to recognize documents that are truly responsive to the issues in this case.” In support of its position, the Plaintiffs pointed to several documents that had been marked as non-responsive but produced inadvertently, which they argued actually should have been marked as responsive. To address their concerns, the Plaintiffs sought the production of random samples from several pools of documents in the City’s TAR process, including non-responsive materials, as well as the production of details and documentation about the City’s TAR process.
To assess these issues, the Magistrate Judge ordered the City to submit a range of additional documents and process details for her in camera review and ordered briefing from the parties.
The Magistrate Judge reviewed the relevant rules and many of the same cases we’ve discussed in this series and came to the same conclusions we’ve seen others reach: that judges shouldn’t be in the business of micromanaging document review and that the general standard for discovery is reasonableness not perfection:
In keeping with these principles, this Court is of the view that there is nothing so exceptional about ESI production that should cause courts to insert themselves as super-managers of the parties’ internal review processes, including training of TAR software, or to permit discovery about such process, in the absence of evidence of good cause such as a showing of gross negligence in the review and production process, the failure to produce relevant specific documents known to exist or that are likely to exist, or other malfeasance. [internal citations omitted, emphasis added]
The “principles” to which she was referring were: (1) the reality that attorneys are rule-bound officers of the court; (2) the need to avoid breaching privilege or work product protection with discovery about discovery; (3) the Sedona Principle that producing parties are best situated to determine methods; and, (4) the fact that the standard is reasonableness rather than perfection.
Based on the Magistrate Judge’s review of the relevant rules and cases, and based on her in camera review of information about the City’s TAR efforts, she concluded the City had made the required reasonable efforts. In reaching this conclusion, she highlighted the training of their human reviewers, their large sample size for training the tool, the makeup of the seed set, their rounds of iterative training for the tool, and their validation process to check the results. She also emphasized how few mistakes had been made relative to the overall volume:
. . . this Court does not find the labeling of these 20 documents, only 5 of which were “incorrectly” categorized as non-responsive during the initial ESI review – out of the 100,000 documents that have been reviewed thus far in this case – sufficient to question the accuracy and reliability of the City’s TAR process as a whole. [emphasis added]
Interestingly, despite reaching the conclusion that the process as a whole is not defective, the Magistrate Judge did grant the Plaintiffs part of what they’d requested based on the minor issues that had been identified:
. . . Plaintiffs have presented sufficient evidence to justify their request for sample sets of non-privileged documents from the documents pulled from the 50 custodians. In particular, this Court agrees that the sample sets will increase transparency, a request that is not unreasonable in light of the volume of documents collected from the custodians, the low responsiveness rate of documents pulled for review by the TAR software, and the examples that Plaintiffs have presented, which suggest there may have been some human error in categorization that may have led to gaps in the City’s production. [emphasis added]
In addition to addressing the process adequacy objections, the Magistrate Judge also addressed the question of how much transparency should be required, finding the case law mixed on the subject, as we have seen in this series:
Courts are split as to the degree of transparency required by the producing party as to its predictive coding process. In some cases, parties have agreed to—or courts have ordered—transparency into the TAR or predictive coding processes. In other cases, courts have not required production of seed set documents. [internal citations omitted, emphasis added]
In this case, the Magistrate Judge concluded that the Plaintiffs were not entitled to detailed information about the TAR process employed. She concluded that the materials submitted to her in camera were “protected by the work product privilege” and that it was “unclear how this information is even potentially relevant to the claims and defenses in this litigation, as required under Federal Rule of Civil Procedure 26.”
And, although she did not compel the disclosure of any of the requested process details, she did encourage the City to disclose them voluntarily: “However, in the interests of transparency and cooperation in the discovery process, the City is encouraged to share such information with Plaintiffs.”
Upcoming in this Series
In the next Part of this series, we will touch on several additional technology-assisted review cases that are worth noting but that don’t require full posts of their own to cover.
About the Author
Matthew Verga, JD
Director, Education and Content Marketing
Matthew Verga is an electronic discovery expert proficient at leveraging his legal experience as an attorney, his technical knowledge as a practitioner, and his skills as a communicator to make complex eDiscovery topics accessible to diverse audiences. An eleven-year industry veteran, Matthew has worked across every phase of the EDRM and at every level from the project trenches to enterprise program design. He leverages this background to produce engaging educational content to empower practitioners at all levels with knowledge they can use to improve their projects, their careers, and their organizations.