Draw a seeded, stratified sample

Draws a random sample within strata (the rule, source, period or class that produced a decision), so that each stratum is represented, including the small ones where errors often concentrate. The sample keeps, for each row, the size of its stratum in the population and its weight, so that estimates can be weighted back to the population.

Usage

draw_stratified_sample(
  data,
  strata = NULL,
  n_per_stratum = NULL,
  n = NULL,
  allocation = c("equal", "proportional"),
  seed
)

Arguments

data

A data frame: the population to sample from.

strata

Character or NULL. Columns defining the strata; NULL for a simple random sample.

n_per_stratum

Integer, or a named integer vector (names are the stratum labels, as in the .stratum column). Items drawn in each stratum.

n

Integer. Total sample size, allocated to strata according to allocation. Give either n or n_per_stratum.

allocation

Character. "equal" gives every stratum the same size (best to compare strata); "proportional" follows the stratum sizes (best to estimate an overall rate).

seed

Integer. Seed of the draw: record it in the script header, and never reuse the seed of a diagnostic round for a final evaluation.

Value

A data.table with the sampled rows and the columns .stratum, .population_n (size of the stratum), .sample_n (items drawn in it) and .weight (.population_n / .sample_n).

Details

Only the strata columns and the sampled rows are copied, so the function works on large populations held in R; for populations held in a database, draw_stratified_sample_db() samples without loading them.

A stratum smaller than its allocation is taken whole. Over-sampling small strata and keeping their population sizes is how Lemercier and Zalc (2019, ch. 2) describe stratified sampling: equal sizes to compare groups, weights to estimate for the whole population. The draw does not change the caller's random numbers.

See also

Other evaluation: cohen_kappa(), draw_stratified_sample_db(), margin_of_error(), precision_recall(), read_verdicts(), wilson_interval(), write_judge_chunks()

Examples

matches <- data.frame(
  id = 1:200,
  rule = rep(c("doi", "exact_title", "fuzzy_title"), c(150, 40, 10))
)
s <- draw_stratified_sample(matches, strata = "rule", n_per_stratum = 8, seed = 42)
s[, .N, by = .(.stratum, .population_n, .weight)]
#>       .stratum .population_n .weight     N
#>         <char>         <int>   <num> <int>
#> 1:         doi           150   18.75     8
#> 2: exact_title            40    5.00     8
#> 3: fuzzy_title            10    1.25     8

draw_stratified_sample(matches, "rule", n = 30, allocation = "proportional", seed = 42)[
  , .N, by = .stratum]
#>       .stratum     N
#>         <char> <int>
#> 1:         doi    23
#> 2: exact_title     6
#> 3: fuzzy_title     1