Last week I saw two different articles (at Futility Closet and DataGenetics) about exactly the same topic: suppose that two reviewers each proofread the same document. The first reviewer finds errors, and the second finds errors, of which are in common, i.e., 8 of those 12 errors had already been found by the first reviewer. Can we estimate how many additional errors remain in the document that were missed by both reviewers?
Both articles essentially reproduce the argument given by Pólya (see reference below) that a reasonable estimate for the total number of errors (both found and missed) is given by the following simple formula:
This is a “mark-and-recapture” estimation method similar to that used, for example, to estimate the number of fish in a lake. Intuitively, the first reviewer identifies and “marks” of the errors in the document (where is unknown), which should approximately equal the fraction of errors found by the second reviewer that were already marked.
However, neither article points out just how inaccurate this method of estimation can be, nor the fact that better alternatives are available. For example, continuing with the example above as originally presented in the DataGenetics article, let us assume for the moment that
- There really are a total of 15 errors in the document being reviewed.
- The first reviewer really does find each error independently with probability 10/15=2/3.
- The second reviewer really does find each error independently with probability 12/15=4/5.
Note that this example is arguably somewhat contrived to be “nice,” since the actual number of errors observed in common by both reviewers happens to equal the expected number of such errors. This need not be the case; with this model of reviewer accuracy, the number of common errors may be as large as … or as small as zero, in which case our estimator breaks down entirely.
Even if we condition against this unlikely difficulty, essentially asking the reviewers to both start over if they don’t find any errors in common (and to forget the errors they may have already found), there is still significant variance in the possible estimates that may result, as shown in the following figure.
(This is not a sample from a simulation; we can calculate this distribution exactly.) The mean of the estimate, shown in red, is approximately 15.17, which is pretty good. However, we can do better– only slightly better in this already-fortunate case, but a lot better in other cases– using a slightly different estimator due to Chapman:
This estimate has several advantages over the Lincoln-Petersen method. It has less bias and less variance, particularly in situations like this where the “population” of errors is relatively small. Also, it still works even when , i.e., when no errors are found in common by both reviewers.
Having said that, it’s not clear how really useful either method is in this particular context, given how widely the resulting estimate may vary from the true value. These estimation methods work much better when at least one of the two sample sizes and is pre-determined (e.g., first catch exactly 100 fish, mark them, then catch 100 fish again), and only varies randomly.
- Pólya, G., Probabilities in Proofreading, American Mathematical Monthly, 83(1) January 1976, p. 42 [JSTOR]