Coherence of each topic's top words — topic

A topic is coherent when its most probable words occur in the same documents: a reader then sees them as belonging together. Two measures of co-occurrence are computed on the documents the model was fitted on, for the n_terms most probable words of each topic:

UMass (Mimno et al. 2011), the "semantic coherence" of stm: for pairs of top words, $$C_k = \sum_{i < j} \log \frac{D(w_i, w_j) + \epsilon}{D(w_j)},$$ with $D(w)$ the number of documents containing $w$, $D(w_i, w_j)$ those containing both, and $\epsilon = 0.01$ (as in stm; Mimno et al. use 1). Always negative; closer to 0 is better. Identical to stm::semanticCoherence().
NPMI (normalised pointwise mutual information; Bouma 2009; Lau, Newman and Baldwin 2014): for each pair, $$\mathrm{NPMI}(w_i, w_j) = \frac{\log \frac{P(w_i, w_j)}{P(w_i) P(w_j)}}{-\log P(w_i, w_j)} \in [-1, 1],$$ with document probabilities $P(w) = D(w) / D$; pairs that never co-occur get $-1$. The topic's coherence is the mean over pairs. NPMI is the automatic measure that best tracked human judgements of topics in Lau et al. (2014) and Röder et al. (2015). 0 means the words co-occur as often as chance would predict.

Usage

topic_coherence(
  model,
  documents,
  n_terms = 10L,
  measure = c("umass", "npmi"),
  ...
)

Arguments

model: A fitted stm model (or a list with beta$logbeta[[1]] and vocab).
documents: The documents the model was fitted on (stm format).
n_terms: Integer scalar, number of top words per topic (M in stm).
measure: Character vector, among "umass" and "npmi".
...: Not used: an argument renamed in the development version of mintR gives an error naming its replacement.

Value

A data.table with topic and coherence_umass and/or coherence_npmi.

Details

Co-occurrences are computed once, on a sparse document x word matrix restricted to the union of the top words, which is much faster than stm::semanticCoherence() on large corpora. Document co-occurrence favours long documents (two words of a book always co-occur): on corpora mixing very different lengths, chunk the texts first (chunk_texts()). Coherence also rewards topics made of very frequent words: read it together with exclusivity (topic_exclusivity()).

References

Bouma, G. (2009). Normalized (pointwise) mutual information in collocation extraction. Proceedings of the Biennial GSCL Conference, 31-40.

Lau, J. H., Newman, D. and Baldwin, T. (2014). Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. Proceedings of EACL 2014, 530-539.

Mimno, D., Wallach, H., Talley, E., Leenders, M. and McCallum, A. (2011). Optimizing semantic coherence in topic models. Proceedings of EMNLP 2011, 262-272.

Röder, M., Both, A. and Hinneburg, A. (2015). Exploring the space of topic coherence measures. Proceedings of WSDM 2015, 399-408. doi:10.1145/2684822.2685324

Examples

if (FALSE) { # \dontrun{
topic_coherence(fit, stm_data$documents)
} # }