Word-set intrusion test — make_word_set

A variant of word intrusion proposed by Ying, Montgomery and Stewart (2022) to validate topics as measures of concepts: for each topic, a coder sees n_sets - 1 sets of words drawn from the topic's most probable words and one set drawn from another topic, and must find the intruding set. Comparing sets rather than single words asks whether the topic as a whole is distinguishable from other topics, and is less sensitive to one odd word. Chance: 1 / n_sets.

Usage

make_word_set_intrusion(
  models,
  n_terms = 20L,
  n_sets = 4L,
  set_size = 4L,
  n_coders = 1L,
  seed = 1L,
  ...,
  n_topics = NULL
)

Arguments

models: A fitted model, or a named list of fitted models (e.g. the candidates kept after evaluate_topic_models()).
n_terms: Integer, number of top words the sets are drawn from (the topic's sets are disjoint, so n_terms must be at least (n_sets - 1) * set_size).
n_sets: Integer, number of sets shown (including the intruder).
set_size: Integer, words per set.
n_coders: Integer, number of coders. All coders answer the same questions (needed for agreement and for the precision of each topic), in a different order and with options shuffled.
seed: Integer seed (the global random state is restored after).
...: Not used: an argument renamed in the development version of mintR gives an error naming its replacement, any other one an error.
n_topics: Integer, number of topics drawn per model (NULL: all). Placed after ...: it must be named in full.

Value

An intrusion_test (see make_word_intrusion()). Options are the sets, as words joined by ", "; the answer is the chosen set (or its option number, see score_intrusion()).

Details

The intruding set comes from the top words of a randomly chosen other topic, excluding words also among the topic's own top words. The defaults (four sets of four words from the 20 most probable words) are a choice of this package, to adapt to your corpus and coders.

References

Ying, L., Montgomery, J. M. and Stewart, B. M. (2022). Topics, concepts, and measurement: A crowdsourced procedure for validating topics as measures. Political Analysis, 30(4), 570-589. doi:10.1017/pan.2021.33