Read the verdicts of a judging round

Reads the verdict files written by the judges of a round prepared with write_judge_chunks(), checks them against the chunks (missing, duplicated or unknown identifiers), and joins the sample with the pipeline's answers.

Usage

read_verdicts(dir, id_col, prefix = "stage1", join_sample = TRUE)

Arguments

dir

Character. Folder of the round.

id_col

Character. Identifier column.

prefix

Character. Prefix of the round.

join_sample

Logical. Join the sample file (with the pipeline's answers) to the verdicts.

Value

A data.table with the verdicts, a chunk column, and the columns of the sample when join_sample is TRUE.

See also

Other evaluation: cohen_kappa(), draw_stratified_sample(), draw_stratified_sample_db(), margin_of_error(), precision_recall(), wilson_interval(), write_judge_chunks()

Examples

dir <- file.path(tempdir(), "verdicts_example")
sample <- data.frame(
  pair_id = 1:4,
  title = c("A", "B", "C", "D"),
  decision = c(TRUE, TRUE, FALSE, TRUE)
)
files <- write_judge_chunks(sample, "pair_id", "title", dir, chunk_size = 2)
#> ✔ Round "stage1": 4 items in 2 chunks, written to /tmp/RtmpKqItGK/verdicts_example.
# Judges write their verdicts:
write.csv(data.frame(pair_id = 1:2, category = c("same", "different"), reason = c("r", "r")),
          files$verdicts[1], row.names = FALSE)
write.csv(data.frame(pair_id = 3:4, category = c("different", "same"), reason = c("r", "r")),
          files$verdicts[2], row.names = FALSE)
read_verdicts(dir, "pair_id")
#> ℹ 4 verdicts read from 2 chunks.
#>    pair_id  category reason  chunk  title decision
#>     <char>    <char> <char> <char> <char>   <lgcl>
#> 1:       1      same      r     01      A     TRUE
#> 2:       2 different      r     01      B     TRUE
#> 3:       3 different      r     02      C    FALSE
#> 4:       4      same      r     02      D     TRUE