Pop Quiz: How Do You Test a Search?, Testing Classifiers Series Part 1
Beyond estimating prevalence, there are other opportunities to replace informal sampling of unknown reliability with formal sampling of precise reliability. Imagine iteratively refining searches for your own use, or negotiating with another party about which searches should be used, armed with precise, reliable information about their relative efficacy. Using sampling to test classifiers can facilitate this.
Show Your Work: Contingency Tables and Error Margins, Testing Classifiers Series Part 2
Sampling can be used to test your search classifiers – whether keyword searches, TAR software, or other tools – by calculating their recall (efficacy) and precision (efficiency). Doing so requires a previously-reviewed control set, contingency tables, and some simple math.
Grading Papers: Measuring Human Review, Testing Classifiers Series Part 3
Just as a search or a TAR tool is making a series of binary classification decisions, so too are your human reviewers, and the quality of those reviewers’ decisions can be assessed in a similar manner to how you assessed the quality of a search classifier. Depending on the scale of your review project, employing these assessment methods can be more efficient and informative.