Every important and uncertain step of a text database must be measured: extraction, cleaning, duplicate detection, record matching, entity attribution. As Grimmer and Stewart (2013, sec. 2.4) put it, what should be avoided is “the blind use of any method without a validation step”. This article follows one step, matching records across two sources, through the two stages of evaluation used in the author’s projects:
- Diagnostic rounds, to find what goes wrong and fix the code. A cheap language model (in practice Haiku, through subagents) judges samples blind to the pipeline’s answer. These rounds stay private.
- Final measurement, to report a number. On one frozen sample, the authors and the model label independently, the authors settle the disagreements, and precision and recall are computed with confidence intervals. Only this human-validated measurement is published.
In this article, the judges’ answers are simulated, so that it runs on its own. In a project they come from the judges.
The population of decisions
A matching step has produced candidate pairs between two sources. It accepted some of them by rule: identical DOI, identical normalised title and year, fuzzy title. It rejected the others.
set.seed(2026)
pairs <- data.table(
pair_id = sprintf("p%04d", 1:2000),
rule = sample(c("doi", "exact_title", "fuzzy_title", "rejected"), 2000,
replace = TRUE, prob = c(0.55, 0.25, 0.08, 0.12))
)
pairs[, accepted := rule != "rejected"]
pairs[, .N, by = rule]
#> rule N
#> <char> <int>
#> 1: exact_title 516
#> 2: doi 1100
#> 3: rejected 226
#> 4: fuzzy_title 158How many pairs to check?
The precision of an estimated rate depends on the number of items checked, not on the share of the population they represent. With \(z\) the normal quantile and \(p\) the expected rate, the half-width of the interval is \(m = z\sqrt{p(1-p)/n}\). Lemercier and Zalc note that doubling a sample from 150 to 300 helps much more than going from 500 to 1,000 (Lemercier and Zalc 2019, ch. 2).
margin_of_error(c(50, 100, 200, 400), p = 0.9)
#> [1] 0.08315423 0.05879892 0.04157711 0.02939946
sample_size_for_margin(0.05, p = 0.9)
#> [1] 139To compare rules, each needs enough items. The rare fuzzy_title rule is where errors concentrate, so a stratified sample draws the same number in each stratum, and keeps the stratum sizes to weight estimates back to the population. Rejected pairs are included as negatives: without them, recall cannot be measured.
sample_1 <- draw_stratified_sample(pairs, strata = "rule", n_per_stratum = 40, seed = 101)
sample_1[, .N, by = .(.stratum, .population_n, .weight)]
#> .stratum .population_n .weight N
#> <char> <int> <num> <int>
#> 1: exact_title 516 12.90 40
#> 2: doi 1100 27.50 40
#> 3: rejected 226 5.65 40
#> 4: fuzzy_title 158 3.95 40Write the seed and the strata in the header of the evaluation script. A final evaluation uses a seed never used in a diagnostic round.
Write blind judge files
The judges see the material to judge (here, in a project, the two titles, authors and years of each pair), never the pipeline’s decision. write_judge_chunks() writes the full sample for R and, separately, chunks holding only the identifier and the shown columns. It refuses to overwrite anything: a sample is never redrawn and a verdict never overwritten.
sample_1[, shown := paste("Titles and authors of pair", pair_id)]
files <- write_judge_chunks(
sample_1, id_col = "pair_id", show_cols = "shown", dir = round_dir,
prefix = "stage1_round1", chunk_size = 40, shuffle_seed = 1,
rubric = "resources/evaluation/matching_rubric.md"
)
#> ✔ Round "stage1_round1": 160 items in 4 chunks, written to '/tmp/RtmpCRkPCg/evaluation_example'.
files[, .(chunk, n_rows)]
#> chunk n_rows
#> <char> <int>
#> 1: 01 40
#> 2: 02 40
#> 3: 03 40
#> 4: 04 40
cat(files$prompt[1])
#> Read the rubric resources/evaluation/matching_rubric.md, then the items in /tmp/RtmpCRkPCg/evaluation_example/stage1_round1_chunk01.csv. Judge every item as the rubric says, without guessing what any program decided, and write your verdicts to /tmp/RtmpCRkPCg/evaluation_example/stage1_round1_chunk01_verdicts.csv (one row per pair_id).The rubric, kept in the repository, says what the judge sees, the question, the closed list of categories (with unsure), the decision rules with worked examples, and the output format. Every judge of a stage uses it word for word.
Read the verdicts and trace the errors
Here, the verdicts of the model judge are simulated: it finds the fuzzy matches often wrong, and a few rejected pairs right.
truth <- sample_1[, .(pair_id, rule,
same = fcase(rule == "doi", runif(.N) < 0.99,
rule == "exact_title", runif(.N) < 0.95,
rule == "fuzzy_title", runif(.N) < 0.70,
rule == "rejected", runif(.N) < 0.10))]
for (i in seq_len(nrow(files))) {
ids <- fread(files$input[i])$pair_id
v <- truth[pair_id %in% ids, .(pair_id,
category = fifelse(same, "same_record", "different_records"),
reason = "simulated verdict")]
fwrite(v, files$verdicts[i])
}read_verdicts() binds the verdict files, checks them against the chunks (missing, unknown or duplicated identifiers), and joins the pipeline’s answers:
verdicts <- read_verdicts(round_dir, id_col = "pair_id", prefix = "stage1_round1")
#> ℹ 160 verdicts read from 4 chunks.
verdicts[, judged_same := category == "same_record"]
verdicts[, .(n = .N, agree_with_pipeline = mean(judged_same == accepted)), by = rule]
#> rule n agree_with_pipeline
#> <char> <int> <num>
#> 1: doi 40 0.950
#> 2: exact_title 40 0.925
#> 3: rejected 40 0.975
#> 4: fuzzy_title 40 0.750Each disagreement is traced to the step and rule that caused it, read by a human (the model is a detector of problems, not an oracle), and turned into a change of the code. The new version bumps the method version, and the next round uses a fresh sample.
The final measurement
Once the step is frozen, one gold sample is labelled independently by the authors and by the model, with the same rubric. Agreement between them is measured with Cohen’s kappa (Cohen 1960), \[\kappa = \frac{p_o - p_e}{1 - p_e},\] where \(p_o\) is the observed agreement and \(p_e\) the agreement expected by chance.
gold <- draw_stratified_sample(pairs, strata = "rule", n_per_stratum = 50, seed = 9001)
# Simulated labels: the authors' labels are taken as right, and the model
# disagrees with them on about 8% of the pairs.
gold[, author_label := fcase(rule == "doi", runif(.N) < 0.99,
rule == "exact_title", runif(.N) < 0.95,
rule == "fuzzy_title", runif(.N) < 0.70,
rule == "rejected", runif(.N) < 0.10)]
gold[, model_label := fifelse(runif(.N) < 0.08, !author_label, author_label)]
k <- cohen_kappa(gold$author_label, gold$model_label)
k[c("n", "agreement", "kappa")]
#> $n
#> [1] 200
#>
#> $agreement
#> [1] 0.915
#>
#> $kappa
#> [1] 0.8070813
k$confusion
#> b
#> a FALSE TRUE
#> FALSE 57 7
#> TRUE 10 126A low kappa usually means an ambiguous rubric: fix the rubric and relabel, rather than choosing a judge. The disagreements go to the authors. Their settled labels are the reference against which the pipeline is measured.
Precision and recall, with intervals
Precision is the share of accepted pairs that are right; recall, the share of true matches that the step accepted. Proportions get Wilson intervals (Wilson 1927), which behave well near 0 and 1:
precision_recall(truth = gold$author_label, predicted = gold$accepted)
#> metric estimate lower upper numerator denominator
#> <char> <num> <num> <num> <int> <int>
#> 1: precision 0.8866667 0.8259973 0.9280258 133 150
#> 2: recall 0.9779412 0.9371541 0.9924701 133 136
#> 3: f1 0.9300699 NA NA NA NA
#> 4: accuracy 0.9000000 0.8505942 0.9343296 180 200The sample over-represents rare rules, so these overall figures are not those of the population. Report them by stratum, or weight each stratum by its .weight:
by_rule <- gold[accepted == TRUE, .(correct = sum(author_label), n = .N,
population = .population_n[1]), by = rule]
cbind(by_rule, wilson_interval(by_rule$correct, by_rule$n)[, .(estimate, lower, upper)])
#> rule correct n population estimate lower upper
#> <char> <int> <int> <int> <num> <num> <num>
#> 1: exact_title 47 50 516 0.94 0.8378291 0.9793850
#> 2: doi 50 50 1100 1.00 0.9286524 1.0000000
#> 3: fuzzy_title 36 50 158 0.72 0.5833488 0.8252583
# Precision of the accepted pairs in the population: weighted by stratum size.
by_rule[, sum(population * correct / n) / sum(population)]
#> [1] 0.9576099What is published is the protocol (unit, sampling frame, seed, rubric, size), the authors’ metrics with their intervals, the agreement between authors when there are several, and the remaining types of error. The model’s labels and its agreement with the authors stay in the private notes of the project.