Skip to contents

Write the files of a blind judging round

Source: R/judging.R
write_judge_chunks.Rd

Prepares an evaluation round in which judges (a model through subagents, or the authors) label a sample without seeing the pipeline's answers:

  • <prefix>_sample.csv keeps the whole sample, with the pipeline's answers, for the analysis in R;

  • <prefix>_chunkNN.csv holds only the identifier and the columns the judges must see, in chunks of chunk_size rows;

  • judges write their verdicts to <prefix>_chunkNN_verdicts.csv, read back by read_verdicts().

The function refuses to overwrite existing files: a sample is never redrawn and a verdict never overwritten; a new round goes to a new folder or gets a new prefix.

Usage

write_judge_chunks(
  sample,
  id_col,
  show_cols,
  dir,
  prefix = "stage1",
  chunk_size = 40L,
  rubric = NULL,
  shuffle_seed = NULL
)

Arguments

sample

A data frame: the sample (e.g. from draw_stratified_sample()).

id_col

Character. Identifier column of the items.

show_cols

Character. Columns shown to the judges (the material to judge, never the pipeline's answer).

dir

Character. Folder of the round (created if needed), outside the repository when the texts are private or licensed.

prefix

Character. Prefix of the file names (e.g. "stage1_r2").

chunk_size

Integer. Rows per chunk (30 to 50 suits a model judge).

rubric

Character or NULL. Path of the rubric (kept in the repository). When given, the returned table has a ready-to-use prompt per chunk.

shuffle_seed

Integer or NULL. Shuffle the rows before chunking (so that a chunk does not hold one stratum only); NULL keeps the order.

Value

A data.table with chunk, input, verdicts, n_rows and prompt, invisibly.

Details

The rubric states what the judge sees, the question, the closed list of categories (with unsure), decision rules and worked examples, and the output format (one row per identifier, a category and a one-sentence reason). The same rubric is used word for word by every judge of a stage; a new rubric starts a new stage.

See also

Other evaluation: cohen_kappa(), draw_stratified_sample(), draw_stratified_sample_db(), margin_of_error(), precision_recall(), read_verdicts(), wilson_interval()

Examples

dir <- file.path(tempdir(), "round_example")
sample <- data.frame(
  pair_id = 1:5,
  title_a = c("A", "B", "C", "D", "E"),
  title_b = c("A", "b", "X", "D.", "F"),
  decision = c(TRUE, TRUE, FALSE, TRUE, FALSE)
)
files <- write_judge_chunks(
  sample, "pair_id", c("title_a", "title_b"), dir,
  chunk_size = 3, rubric = "resources/evaluation/matching_rubric.md"
)
#> ✔ Round "stage1": 5 items in 2 chunks, written to /tmp/RtmpKqItGK/round_example.
files$prompt[1]
#> [1] "Read the rubric resources/evaluation/matching_rubric.md, then the items in /tmp/RtmpKqItGK/round_example/stage1_chunk01.csv. Judge every item as the rubric says, without guessing what any program decided, and write your verdicts to /tmp/RtmpKqItGK/round_example/stage1_chunk01_verdicts.csv (one row per pair_id)."
read.csv(files$input[1])
#>   pair_id title_a title_b
#> 1       1       A       A
#> 2       2       B       b
#> 3       3       C       X