Skip to contents

Word intrusion test

Source: R/intrusion.R
make_word_intrusion.Rd

The word intrusion task of Chang et al. (2009). For each topic, a coder sees n_shown words drawn from the topic's n_terms most probable words, plus one intruder, in random order, and must find the intruder. When a topic is coherent, the intruder stands out; when it is a mix of unrelated words, coders guess (chance: 1 / (n_shown + 1)). Chang et al. found that models with a better held-out likelihood do not necessarily produce topics that humans find more interpretable, hence the need for this kind of validation.

Usage

make_word_intrusion(
  models,
  n_terms = 10L,
  n_shown = 5L,
  intruder = c("other_topic", "within_topic"),
  within_quantiles = c(0.5, 0.75),
  match_ngram = TRUE,
  n_coders = 1L,
  seed = 1L,
  ...,
  n_topics = NULL
)

Arguments

models

A fitted model, or a named list of fitted models (e.g. the candidates kept after evaluate_topic_models()).

n_terms

Integer, number of top words the shown words are drawn from.

n_shown

Integer, number of the topic's words shown with the intruder.

intruder

How the intruder is drawn:

  • "other_topic" (Chang et al. 2009): a word improbable in the topic (below the median of its word probabilities) but among the n_terms most probable words of another topic. The intruder is then a "meaningful" word, so it cannot be spotted only because it is rare;

  • "within_topic": a word of the same topic between the quantiles within_quantiles of its word probabilities (the rule used in the history of labour economics project).

within_quantiles

Numeric vector of length 2, used with intruder = "within_topic".

match_ngram

Logical: the intruder has as many words (parts joined by _) as one of the shown words, so that a bigram is not the obvious odd one out among unigrams.

n_coders

Integer, number of coders. All coders answer the same questions (needed for agreement and for the precision of each topic), in a different order and with options shuffled.

seed

Integer seed (the global random state is restored after).

...

Not used: an argument renamed in the development version of mintR gives an error naming its replacement, any other one an error.

n_topics

Integer, number of topics drawn per model (NULL: all). Placed after ...: it must be named in full.

Value

An intrusion_test: a list with questions (one row per model and topic, options in option_1...), key (the intruder of each question), sheets (one row per coder and question, with an empty answer column to fill with the chosen word) and type. Write the sheets with write_intrusion_sheets() and score with score_intrusion().

References

Chang, J., Boyd-Graber, J., Gerrish, S., Wang, C. and Blei, D. M. (2009). Reading tea leaves: How humans interpret topic models. Advances in Neural Information Processing Systems 22 (NIPS 2009), 288-296.

See also

make_word_set_intrusion(), make_topic_intrusion(), score_intrusion()

Examples

if (FALSE) { # \dontrun{
test <- make_word_intrusion(list(k20 = fit20, k30 = fit30), n_coders = 3)
write_intrusion_sheets(test, "data/intrusion")
} # }