The topic intrusion task of Chang et al. (2009), which
validates the document-topic proportions \(\theta\) (what estimated
effects and prevalences rely on) rather than the topics' words. For each
sampled document, a coder reads an excerpt and sees n_high of its most
probable topics plus one improbable topic (each described by its top
words), in random order, and must find the topic that does not belong to
the document.
Scored by topic log odds (Chang et al. 2009): for document \(d\), true intruder \(j^*\) and the topic \(j_s\) chosen by coder \(s\), $$\mathrm{TLO}_d = \frac{1}{S}\sum_s \left(\log\hat\theta_{d,j^*} - \log\hat\theta_{d,j_s}\right) \le 0,$$ 0 when every coder finds the intruder, and very negative when coders pick topics the model deems central to the document.
Usage
make_topic_intrusion(
models,
texts,
n_documents = 20L,
n_high = 3L,
n_terms = 6L,
measure = c("frex", "prob", "lift", "score"),
n_chars = 1000,
min_chars = 200,
n_coders = 1L,
seed = 1L,
...
)Arguments
- models
A fitted model, or a named list of fitted models (e.g. the candidates kept after
evaluate_topic_models()).- texts
Character vector of the documents' texts, in the order of the model's documents (rows of
theta); for several models, the same documents in the same order.- n_documents
Integer, number of documents drawn (the same documents for every model).
- n_high
Integer, number of the document's most probable topics shown.
- n_terms
Integer, number of top words describing each topic.
- measure
Measure of the label words, as
topic_labels().- n_chars
Integer, maximal length of the excerpt shown (cut at a word boundary;
Inffor the whole text).- min_chars
Integer, documents shorter than this are not drawn.
- n_coders
Integer, number of coders. All coders answer the same questions (needed for agreement and for the precision of each topic), in a different order and with options shuffled.
- seed
Integer seed (the global random state is restored after).
- ...
Not used: an argument renamed in the development version of mintR gives an error naming its replacement, any other one an error.
Value
An intrusion_test (see make_word_intrusion()), with the
document (document_index, excerpt) in each question. Options are topic
labels; the key also holds the log proportions needed for topic log
odds.
Details
The intruder is drawn among the topics of the lower half of the
document's proportions. Long documents give long excerpts: prefer
models fitted on chunks (chunk_texts()) or a small n_chars, and
tell coders the excerpt may be partial.