Choosing a topic model

Fitting a topic model requires choices the data do not settle by themselves: the number of topics \(K\), the vocabulary (which words, after which filters), the unit of analysis, the covariates. There is no true \(K\): a corpus can be described with 10 broad topics or 80 narrow ones, and both can be useful (Grimmer and Stewart 2013; Roberts et al. 2016). The aim is therefore not to find the right model but to justify a choice: show that the model fits the data reasonably, that its topics are interpretable, and that they would come back if the model were refitted.

mintR organises this in three steps:

1. fit a grid of models, over corpora (e.g. vocabularies built with different filters), values of \(K\) and runs, and compute metrics (evaluate_topic_models());
2. read the metrics together, by \(K\) and in the coherence-exclusivity plane (plot_model_metrics(), plot_k_frontier()), then look at the topics of the candidates (plot_topic_diagnostics()) and at their stability (plot_topic_stability());
3. validate the few remaining candidates with humans (next guide).

The example: two vocabularies

As in the first guide, the corpus is the paragraphs of US inaugural addresses. Two vocabularies are compared: words and bigrams occurring in at least 5 paragraphs, and in at least 15.

library(mintR)
library(stm)
library(data.table)
library(ggplot2)

corpus <- quanteda::corpus_reshape(quanteda::data_corpus_inaugural, to = "paragraphs")
paragraphs <- data.table(quanteda::docvars(corpus), text = as.character(corpus))
paragraphs[, id := sprintf("%d_%s_%03d", Year, President, seq_len(.N)), by = .(Year, President)]
paragraphs <- paragraphs[stringi::stri_count_words(text) >= 40]

tokens <- extract_ngrams(paragraphs, ngrams = 2L, document_col = "id")
meta <- paragraphs[, .(id, Year, Party)]

# Paragraphs without any kept word are dropped by tokens_to_stm(): `meta`
# keeps only the paragraphs that remain.
make_corpus <- function(min_documents) {
  kept <- filter_vocabulary(tokens, document_col = "id", min_documents = min_documents, verbose = FALSE)
  tokens_to_stm(kept, meta[id %in% kept$id], document_col = "id")
}
corpora <- list(min_5_docs = make_corpus(5), min_15_docs = make_corpus(15))
sapply(corpora, function(x) c(documents = length(x$documents), vocabulary = length(x$vocab)))

           min_5_docs min_15_docs
documents        1091        1091
vocabulary       2154         582

Fitting the grid

evaluate_topic_models() fits one model per corpus and \(K\), in parallel background R processes (n_workers), and computes the metrics. Each worker holds the corpora and fits one model at a time: memory, not the number of cores, should set n_workers. With output_dir, every fit is saved as soon as it is done, and an interrupted evaluation resumes where it stopped, which matters when each fit takes hours.

evaluation <- evaluate_topic_models(
  corpora,
  K = c(5, 10, 15, 20, 25, 30),
  prevalence = ~ s(Year),
  max.em.its = 75,
  heldout_seed = 2026,
  n_workers = 4
)
evaluation$models[, .(corpus, K, heldout, residual, coherence_umass, coherence_npmi, exclusivity, diversity)]

         corpus     K   heldout residual coherence_umass coherence_npmi
         <char> <int>     <num>    <num>           <num>          <num>
 1: min_15_docs     5 -5.869123 1.364540       -85.95953     0.15243237
 2: min_15_docs    10 -5.863822 1.288261      -105.45626     0.12803802
 3: min_15_docs    15 -5.829312 1.258566      -114.15859     0.11753639
 4: min_15_docs    20 -5.827485 1.177989      -118.63984     0.11618750
 5: min_15_docs    25 -5.888239 1.180706      -129.36980     0.09005447
 6: min_15_docs    30 -5.864535 1.162635      -133.24604     0.08912917
 7:  min_5_docs     5 -7.089153 2.300867       -87.48337     0.12545589
 8:  min_5_docs    10 -7.175710 1.996739       -92.18119     0.11916055
 9:  min_5_docs    15 -7.217196 1.873019      -100.12862     0.13356206
10:  min_5_docs    20 -7.273555 1.767969      -106.10906     0.13256447
11:  min_5_docs    25 -7.389769 1.697420      -109.93131     0.11228503
12:  min_5_docs    30 -7.490134 1.651822      -113.71052     0.11570065
    exclusivity diversity
          <num>     <num>
 1:    9.188945 0.8400000
 2:    9.584845 0.7680000
 3:    9.699247 0.7840000
 4:    9.755001 0.7280000
 5:    9.808991 0.7008000
 6:    9.848138 0.6613333
 7:    8.795148 0.6880000
 8:    9.059616 0.6280000
 9:    9.258256 0.5680000
10:    9.345055 0.5860000
11:    9.301478 0.5088000
12:    9.348514 0.4946667

The result also holds one row per topic of every model (evaluation$topics), used below.

Use the settings of the model you would finally fit (covariates, iterations): the evaluation should be about the models you would use.

What the metrics measure

Fit to unseen words: held-out likelihood and residuals

Held-out likelihood (Wallach et al. 2009). Before fitting, half of the words of 10% of the documents are hidden. The fitted model then predicts them from the rest of their document: the expected log-probability per hidden word measures how well the model generalises. Higher is better. It usually rises quickly with \(K\), then levels off; it can keep rising long after topics have stopped being interpretable, and Chang et al. (2009) found it can even move against human judgements of topics.

Residual dispersion (Taddy 2012). If the model has enough topics, the variance of word counts around its predictions should be what a multinomial distribution implies: a dispersion close to 1. Values well above 1 suggest that \(K\) is too small.

Interpretability of topics: coherence and exclusivity

Coherence. A topic is interpretable when its top words go together. Coherence measures this by co-occurrence: do the top words appear in the same documents? Two versions are computed on the top 10 words of each topic:

• UMass (Mimno et al. 2011), stm’s “semantic coherence”: \(\sum_{i<j} \log\frac{D(w_i, w_j) + \epsilon}{D(w_j)}\), with \(D(\cdot)\) document frequencies. Negative; closer to 0 is better.
• NPMI (Bouma 2009; Lau et al. 2014): the normalised pointwise mutual information of each pair, \(\frac{\log P(w_i, w_j) - \log P(w_i)P(w_j)}{-\log P(w_i, w_j)}\), in \([-1, 1]\), averaged over pairs. 0 means the words co-occur as often as chance would predict. Among automatic measures, NPMI is among those that best tracked human ratings of topics (Newman et al. 2010; Lau et al. 2014; Röder et al. 2015), although later work warns that the agreement between coherence and human judgement is weaker than often assumed (Hoyle et al. 2021).

Exclusivity (Bischof and Airoldi 2012; Roberts et al. 2014). Coherence alone favours topics made of the most frequent words, which co-occur everywhere. Exclusivity measures whether a topic’s top words are specific to it (sum of their FREX scores). It rises with \(K\), as topics become narrower, while coherence falls: the two must be read together.

Redundancy of topics: diversity and similarity

Topic diversity (Dieng et al. 2020): the share of distinct words among the top 25 words of all topics. When \(K\) is too large, the model splits themes into near-duplicate topics and diversity drops. Similarity between topics: the mean (and maximum) cosine similarity of the topics’ word distributions. The mean is the criterion of Cao et al. (2009), who chose \(K\) to minimise it; the maximum points to the most redundant pair.

Two criteria from the LDA literature

The ldatuning package made popular two further criteria: Arun et al. (2010) (lower is better) and Deveaud et al. (2014) (higher is better). They were designed for LDA, rest on weaker arguments than the ones above, and are often noisy: mintR computes them as secondary evidence.

Reading the metrics by K

plot_model_metrics(evaluation, ncol = 3, base_size = 12)

Read the panels together rather than looking for one optimum:

• where held-out likelihood and residuals stop improving, adding topics no longer captures much more of the data;
• coherence falls and exclusivity rises with \(K\): the question is where the loss of one is no longer worth the gain in the other;
• a drop in diversity, or a rise in the maximum similarity, signals redundant topics.

In this small corpus (about a thousand paragraphs), the metrics do not agree on a “best” K, which is the usual situation. Held-out likelihood does not rise with \(K\): for the larger vocabulary it even falls, so additional topics do not help predict unseen words. Residuals keep improving, coherence falls, and exclusivity rises up to about 25 topics. Diversity falls steadily, faster with the larger vocabulary. Together they point to models in the range of 10 to 20 topics, to be compared by reading them, rather than to one value.

Comparing vocabularies. The two corpora cannot be compared on the levels of these curves: held-out likelihood is a log-probability over a vocabulary of a given size (a smaller vocabulary makes each word easier to predict), and coherence depends on the document frequencies of the words kept. Compare \(K\) within a corpus, and corpora on the shape of their curves and, above all, on the topics they produce and on human validation (Denny and Spirling 2018).

The coherence-exclusivity frontier

The trade-off between coherence and exclusivity is easier to read in their plane. plot_k_frontier() draws each model, joins them in order of \(K\) and draws the Pareto frontier: the models that no other model beats on both criteria. A model off the frontier is dominated: another model is at least as coherent and as exclusive. Points are coloured by a weighted harmonic mean of the two criteria, after rescaling each to \([0, 1]\) over the models compared,

\[ H = \left(\frac{w_c}{c} + \frac{w_e}{e}\right)^{-1}, \]

and the best model by this score is circled. The weights (\(w_c = 0.4\), \(w_e = 0.6\) by default) are a judgement about what matters most, to be stated and, ideally, varied.

plot_k_frontier(evaluation)

Looking inside candidate models

Model averages hide bad topics. topic_diagnostics() gives one row per topic: coherence, exclusivity, prevalence, the similarity to the closest other topic, and the distance from the corpus word distribution. A topic close to that distribution is made of common words that appear everywhere: a “background” topic, which AlSumait et al. (2009) propose to rank low. plot_topic_diagnostics() places topics in the coherence x exclusivity plane, sized by prevalence: topics at the bottom left are the ones to read first.

fit_15 <- stm(
  corpora$min_5_docs$documents, corpora$min_5_docs$vocab, K = 15,
  prevalence = ~ s(Year), data = corpora$min_5_docs$meta,
  max.em.its = 75, verbose = FALSE
)
diagnostics <- topic_diagnostics(fit_15, corpora$min_5_docs$documents)
diagnostics[order(coherence_npmi), .(
  topic, coherence_npmi = round(coherence_npmi, 3), exclusivity = round(exclusivity, 2),
  prevalence = round(prevalence, 3), background_distance = round(background_distance, 2), top_terms
)][1:5]

   topic coherence_npmi exclusivity prevalence background_distance
   <int>          <num>       <num>      <num>               <num>
1:     9          0.053        9.37      0.042                0.71
2:     7          0.054        9.46      0.064                0.69
3:     5          0.054        9.41      0.125                0.70
4:     3          0.075        9.00      0.056                0.65
5:     8          0.082        9.37      0.147                0.69
                               top_terms
                                  <char>
1: country, revenue, time, citizens, tax
2: life, purpose, common, nation, spirit
3: nation, time, america, americans, day
4:  earth, union, war, government, peace
5:  world, freedom, america, peace, free

plot_topic_diagnostics(diagnostics, coherence = "coherence_npmi")

The evaluation also keeps per-topic metrics for every model of the grid (evaluation$topics); plot_topic_diagnostics(evaluation, corpus = ..., K = ...) draws any of them.

Stability: would the topics come back?

The fitting algorithm of a topic model finds a local optimum, which depends on the initialisation (Roberts et al. 2016). Spectral initialisation (stm’s default) is deterministic, which hides the problem rather than solving it. With a random initialisation, refitting the same model can give different topics. A topic worth interpreting should come back; a value of \(K\) whose topics change from run to run gives results that depend on luck (Greene et al. 2014; Maier et al. 2018; Wilkerson and Casas 2017).

With n_runs and a random initialisation, evaluate_topic_models() fits each model several times, matches the topics of each run to those of the best run (the one with the highest likelihood bound), one to one, so as to maximise their total similarity (the assignment problem, Kuhn 1955), and reports the similarity of matched topics:

stability <- evaluate_topic_models(
  corpora["min_5_docs"],
  K = c(10, 20, 30),
  metrics = c("coherence", "exclusivity", "stability"),
  heldout = FALSE,
  n_runs = 3,
  init.type = "LDA",
  prevalence = ~ s(Year),
  max.em.its = 75,
  n_workers = 4
)
stability$models[, .(stability = mean(stability_cosine), top_word_overlap = mean(stability_jaccard)), by = K]

       K stability top_word_overlap
   <int>     <num>            <num>
1:    10 0.5798477        0.2368836
2:    20 0.5056298        0.1693866
3:    30 0.5262760        0.1621010

plot_topic_stability(stability)

Each point is a topic of the best run; its value is the mean cosine similarity with its matches in the other runs. Topics near 1 are robust; topics with low values are the model’s arbitrary splits and merges, and should be interpreted with care, whatever the \(K\) chosen.

Instability is not only a defect. Grimmer et al. (2022, ch. 21) argue that different specifications of a model often capture different aspects of the texts rather than noisy versions of one underlying truth. As two teams of human coders would design different coding schemes for the same documents, two runs can both be valid measurements of different concepts. The question is then not “is this topic stable?” but “is this topic, as measured, valid for my concept?”. Stability tells you which topics depend on the run, and therefore which ones need extra validation.

A workflow

When the topics are used as measures in later analyses (e.g. the effect of a covariate on a topic), Grimmer et al. (2022, ch. 21) recommend splitting the documents first: explore and choose the model on a training set, then apply the chosen model to the held-back test set and validate again. The choice of the model then cannot be tuned, even unknowingly, to the results it produces. Whatever the design, report the choices made at each step: preprocessing and the number of topics can change the topics found and their interpretation (Denny and Spirling 2018; Macanovic 2022).

1. Build a few corpora that differ by defensible preprocessing choices, not all possible ones.
2. Run evaluate_topic_models() over a wide range of \(K\), with the settings of the final model, and output_dir for long runs.
3. Narrow down with plot_model_metrics() and plot_k_frontier(); keep a few candidates on the frontier, at different levels of detail.
4. Read their topics (top words by several measures, top documents) and their diagnostics; check stability with n_runs.
5. Validate the candidates with humans (next guide), and report the choices, the metrics and the validation in the paper.

References

AlSumait, Loulwah, Daniel Barbará, James Gentle, and Carlotta Domeniconi. 2009. “Topic Significance Ranking of LDA Generative Models.” Machine Learning and Knowledge Discovery in Databases (ECML PKDD 2009), 67–82.

Arun, R., V. Suresh, C. E. Veni Madhavan, and M. N. Narasimha Murthy. 2010. “On Finding the Natural Number of Topics with Latent Dirichlet Allocation: Some Observations.” Advances in Knowledge Discovery and Data Mining (PAKDD 2010), 391–402.

Bischof, Jonathan M., and Edoardo M. Airoldi. 2012. “Summarizing Topical Content with Word Frequency and Exclusivity.” Proceedings of the 29th International Conference on Machine Learning, 201–8.

Bouma, Gerlof. 2009. “Normalized (Pointwise) Mutual Information in Collocation Extraction.” Proceedings of the Biennial GSCL Conference, 31–40.

Cao, Juan, Tian Xia, Jintao Li, Yongdong Zhang, and Sheng Tang. 2009. “A Density-Based Method for Adaptive LDA Model Selection.” Neurocomputing 72 (7–9): 1775–81. https://doi.org/10.1016/j.neucom.2008.06.011.

Chang, Jonathan, Jordan Boyd-Graber, Sean Gerrish, Chong Wang, and David M. Blei. 2009. “Reading Tea Leaves: How Humans Interpret Topic Models.” Advances in Neural Information Processing Systems 22, 288–96.

Denny, Matthew J., and Arthur Spirling. 2018. “Text Preprocessing for Unsupervised Learning: Why It Matters, When It Misleads, and What to Do about It.” Political Analysis 26 (2): 168–89. https://doi.org/10.1017/pan.2017.44.

Deveaud, Romain, Eric SanJuan, and Patrice Bellot. 2014. “Accurate and Effective Latent Concept Modeling for Ad Hoc Information Retrieval.” Document Numérique 17 (1): 61–84.

Dieng, Adji B., Francisco J. R. Ruiz, and David M. Blei. 2020. “Topic Modeling in Embedding Spaces.” Transactions of the Association for Computational Linguistics 8: 439–53. https://doi.org/10.1162/tacl_a_00325.

Greene, Derek, Derek O’Callaghan, and Pádraig Cunningham. 2014. “How Many Topics? Stability Analysis for Topic Models.” Machine Learning and Knowledge Discovery in Databases (ECML PKDD 2014), 498–513.

Grimmer, Justin, Margaret E. Roberts, and Brandon M. Stewart. 2022. Text as Data: A New Framework for Machine Learning and the Social Sciences. Princeton University Press.

Grimmer, Justin, and Brandon M. Stewart. 2013. “Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts.” Political Analysis 21 (3): 267–97. https://doi.org/10.1093/pan/mps028.

Hoyle, Alexander, Pranav Goel, Andrew Hian-Cheong, Denis Peskov, Jordan Boyd-Graber, and Philip Resnik. 2021. “Is Automated Topic Model Evaluation Broken? The Incoherence of Coherence.” Advances in Neural Information Processing Systems 34.

Kuhn, Harold W. 1955. “The Hungarian Method for the Assignment Problem.” Naval Research Logistics Quarterly 2 (1–2): 83–97.

Lau, Jey Han, David Newman, and Timothy Baldwin. 2014. “Machine Reading Tea Leaves: Automatically Evaluating Topic Coherence and Topic Model Quality.” Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, 530–39.

Macanovic, Ana. 2022. “Text Mining for Social Science – the State and the Future of Computational Text Analysis in Sociology.” Social Science Research 108: 102784. https://doi.org/10.1016/j.ssresearch.2022.102784.

Maier, Daniel, Annie Waldherr, Peter Miltner, et al. 2018. “Applying LDA Topic Modeling in Communication Research: Toward a Valid and Reliable Methodology.” Communication Methods and Measures 12 (2–3): 93–118. https://doi.org/10.1080/19312458.2018.1430754.

Mimno, David, Hanna M. Wallach, Edmund Talley, Miriam Leenders, and Andrew McCallum. 2011. “Optimizing Semantic Coherence in Topic Models.” Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, 262–72.

Newman, David, Jey Han Lau, Karl Grieser, and Timothy Baldwin. 2010. “Automatic Evaluation of Topic Coherence.” Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, 100–108.

Roberts, Margaret E., Brandon M. Stewart, Dustin Tingley, et al. 2014. “Structural Topic Models for Open-Ended Survey Responses.” American Journal of Political Science 58 (4): 1064–82. https://doi.org/10.1111/ajps.12103.

Roberts, Margaret E., Brandon M. Stewart, and Dustin Tingley. 2016. “Navigating the Local Modes of Big Data: The Case of Topic Models.” In Computational Social Science: Discovery and Prediction, edited by R. Michael Alvarez. Cambridge University Press.

Röder, Michael, Andreas Both, and Alexander Hinneburg. 2015. “Exploring the Space of Topic Coherence Measures.” Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, 399–408. https://doi.org/10.1145/2684822.2685324.

Taddy, Matt. 2012. “On Estimation and Selection for Topic Models.” Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, 1184–93.

Wallach, Hanna M., Iain Murray, Ruslan Salakhutdinov, and David Mimno. 2009. “Evaluation Methods for Topic Models.” Proceedings of the 26th International Conference on Machine Learning, 1105–12.

Wilkerson, John, and Andreu Casas. 2017. “Large-Scale Computerized Text Analysis in Political Science: Opportunities and Challenges.” Annual Review of Political Science 20: 529–44. https://doi.org/10.1146/annurev-polisci-052615-025542.