Topic diversity

Share of distinct words among the n_terms most probable words of all topics (Dieng, Ruiz and Blei 2020): $$\mathrm{TD} = \frac{|\bigcup_k \mathrm{top}_n(k)|}{nK} \in [1/K, 1].$$ Close to 1, topics are described by different words; low values reveal redundant topics (several topics on the same theme), which often appear when K is too large.

Usage

topic_diversity(model, n_terms = 25L, ...)

Arguments

model

A fitted stm model (or a list with beta$logbeta[[1]] and vocab).

n_terms

Integer scalar, number of top words per topic (M in stm).

...

Not used: an argument renamed in the development version of mintR gives an error naming its replacement.

Value

A numeric scalar.

References

Dieng, A. B., Ruiz, F. J. R. and Blei, D. M. (2020). Topic modeling in embedding spaces. Transactions of the Association for Computational Linguistics, 8, 439-453. doi:10.1162/tacl_a_00325

Examples

logbeta <- log(matrix(c(0.5, 0.3, 0.2, 0.1, 0.2, 0.7), nrow = 2, byrow = TRUE))
mdl <- list(beta = list(logbeta = list(logbeta)), vocab = c("wage", "labour", "trade"))
topic_diversity(mdl, n_terms = 2)
#> [1] 0.75