Stability of topics across runs

Topic models are fitted by optimising a non-convex objective: different initialisations can give different topics (Roberts, Stewart and Tingley 2016). A topic worth interpreting should come back when the model is refitted. Given several runs of the same model (same corpus and K, different seeds), each run's topics are matched to those of a reference run (match_topics()) and the similarity of matched topics is averaged (Greene, O'Callaghan and Cunningham 2014).

Usage

topic_stability(models, reference = NULL, n_terms = 10L, ...)

Arguments

models

A list of fitted models (same corpus, vocabulary and K), e.g. fitted with init.type = "LDA" or "Random" and different seeds (spectral initialisation is deterministic: its runs are identical).

reference

Index of the reference run. Default: the run with the highest variational bound (the "best" run, as stm::selectModel() would keep).

n_terms

Number of top words for the jaccard overlap.

...

Not used: an argument renamed in the development version of mintR gives an error naming its replacement, any other one an error.

Value

A list with:

• topics: one row per topic of the reference run, with stability_cosine and stability_jaccard (means over the other runs) and min_cosine (worst run);

• model: one row, with stability_cosine and stability_jaccard averaged over topics, and reference.

References

Greene, D., O'Callaghan, D. and Cunningham, P. (2014). How many topics? Stability analysis for topic models. Machine Learning and Knowledge Discovery in Databases (ECML PKDD 2014), 498-513.

Roberts, M. E., Stewart, B. M. and Tingley, D. (2016). Navigating the local modes of big data: The case of topic models. In R. M. Alvarez (ed.), Computational Social Science: Discovery and Prediction, Cambridge University Press, 51-97.