Precision, recall and F1 with confidence intervals

Compares the decisions of a pipeline step with reference labels (the reconciled labels of a gold sample) and returns precision, recall, F1 and accuracy, with Wilson intervals for the proportions.

Usage

precision_recall(truth, predicted, positive = TRUE, conf = 0.95)

Arguments

truth

Reference labels (logical, or labels compared with positive).

predicted

Labels given by the pipeline, same length.

positive

The label that counts as a positive decision (a match, a duplicate, an error found).

conf

Number. Confidence level.

Value

A data.table with metric, estimate, lower, upper, numerator, denominator.

Details

Precision is the share of positive decisions that are right; recall is the share of true positives that the step finds. Recall can only be measured if the sample contains items that the step did not select (negatives and near-misses): a sample of the step's outputs alone measures precision only. F1 is the harmonic mean of the two, given without interval. Items with a missing label are dropped. For a stratified sample, compute the metrics by stratum or weight them back to the population.

See also

Other evaluation: cohen_kappa(), draw_stratified_sample(), draw_stratified_sample_db(), margin_of_error(), read_verdicts(), wilson_interval(), write_judge_chunks()

Examples

truth <- c(TRUE, TRUE, FALSE, TRUE, FALSE, FALSE, TRUE)
predicted <- c(TRUE, FALSE, FALSE, TRUE, TRUE, FALSE, TRUE)
precision_recall(truth, predicted)
#>       metric  estimate     lower     upper numerator denominator
#>       <char>     <num>     <num>     <num>     <int>       <int>
#> 1: precision 0.7500000 0.3006418 0.9544127         3           4
#> 2:    recall 0.7500000 0.3006418 0.9544127         3           4
#> 3:        f1 0.7500000        NA        NA        NA          NA
#> 4:  accuracy 0.7142857 0.3589345 0.9177811         5           7