Cohen's kappa between two labellings

Agreement between two judges (two authors, or an author and a model) who labelled the same items, corrected for the agreement expected by chance.

Usage

cohen_kappa(a, b)

Arguments

a, b

Vectors of labels, of the same length. Pairs with a missing label are dropped.

Value

A list with n (pairs compared), agreement (observed share of identical labels), kappa, and confusion (the table of a against b).

Details

$$\kappa = \frac{p_o - p_e}{1 - p_e},$$ where \(p_o\) is the observed agreement and \(p_e\) the agreement expected if the two judges labelled independently with their own frequencies of labels. \(\kappa\) is NA when both judges always give the same single label (\(p_e = 1\)). Kappa is low when a category is rare even if raw agreement is high: report both, with the confusion table. A low kappa usually means an ambiguous rubric: fix the rubric rather than the labels.

References

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46.

See also

Other evaluation: draw_stratified_sample(), draw_stratified_sample_db(), margin_of_error(), precision_recall(), read_verdicts(), wilson_interval(), write_judge_chunks()

Examples

author <- c("ok", "ok", "error", "ok", "error", "ok")
model <- c("ok", "error", "error", "ok", "error", "ok")
cohen_kappa(author, model)
#> $n
#> [1] 6
#> 
#> $agreement
#> [1] 0.8333333
#> 
#> $kappa
#> [1] 0.6666667
#> 
#> $confusion
#>        b
#> a       error ok
#>   error     2  0
#>   ok        1  3
#>