Skip to contents

Score an intrusion test

Source: R/intrusion.R
score_intrusion.Rd

Compare the coders' answers with the key:

  • precision (Chang et al.'s "model precision"): share of coders who found the intruder, per question (topic or document), then averaged per model. Compare it to chance, 1 / n_options;

  • for topic intrusion, topic log odds (see make_topic_intrusion());

  • agreement between coders on the option chosen, over the questions answered by every coder: the observed share of coder pairs choosing the same option \(P_o\); the free-marginal kappa (Brennan and Prediger 1981), \((P_o - 1/m) / (1 - 1/m)\) with \(m\) options, which corrects for agreement by chance among options chosen at random; and Fleiss' kappa (Fleiss 1971), which takes chance from the observed distribution of choices. In intrusion tasks most answers fall on one option (the intruder) when topics are good, which makes Fleiss' kappa low or undefined even when coders agree almost perfectly (a known paradox of fixed-marginal kappas): prefer the free-marginal kappa.

Usage

score_intrusion(test, responses, ci_level = 0.95, ...)

Arguments

test

An intrusion_test.

responses

A table with coder, question_id and answer (e.g. from read_intrusion_responses(), or test$sheets once filled). answer is the chosen option, or its number in the coder's sheet.

ci_level

Level of the Wilson intervals of model precision.

...

Not used: an argument renamed in the development version of mintR gives an error naming its replacement, any other one an error.

Value

A list of class intrusion_scores:

  • answers: one row per coder and question, with correct;

  • questions: one row per question: precision, n_coders, and tlo for topic intrusion;

  • models: one row per model: precision, ci_lower, ci_upper (Wilson interval over all answers), chance, n_questions, n_answers, and tlo for topic intrusion;

  • coders: precision of each coder per model;

  • agreement: per model, observed agreement, free_kappa, fleiss_kappa, and the numbers of questions and coders used.

Details

Wilson intervals treat answers as independent, which they are not entirely (the same coder answers many questions, several coders answer the same question): read them as indicative.

References

Brennan, R. L. and Prediger, D. J. (1981). Coefficient kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement, 41(3), 687-699.

Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378-382. doi:10.1037/h0031619