From texts to topics

This guide walks through a complete topic-modelling workflow with mintR, from raw texts to topics and their evolution over time. It explains what each step does and why, with the choices that matter. Two other guides cover the steps that decide whether the results can be trusted: choosing a topic model (how many topics, which vocabulary) and validating topics with humans.

What is a topic model?

A topic model is a statistical model of a collection of documents that discovers, without prior coding, groups of words that tend to occur together: the topics (Blei et al. 2003; Blei 2012). Two ideas define it:

1. A topic is a probability distribution over the vocabulary. Topic \(k\) gives each word \(v\) a probability \(\beta_{kv}\), with \(\sum_v \beta_{kv} = 1\). A topic on monetary policy gives high probabilities to inflation, interest rate, central bank; a topic on war, to army, enemy, peace. The model does not name topics: reading their most probable words and the documents where they dominate is how we interpret them.
2. A document is a mixture of topics. Document \(d\) has topic proportions \(\theta_{d1}, \dots, \theta_{dK}\) that sum to 1: a speech can be 60% about the economy, 30% about war and 10% about the constitution.

The model imagines that each word of a document is written by first drawing a topic from the document’s proportions, then a word from that topic. Fitting the model reverses this story: it finds the topics \(\beta\) and proportions \(\theta\) that make the observed documents most probable. The number of topics \(K\) is fixed by the analyst.

The structural topic model (STM) of the stm package (Roberts et al. 2014, 2016, 2019) adds document metadata (date, author, outlet…) to this framework. Prevalence covariates let topic proportions depend on the metadata, e.g. on the year, so that “how does attention to this topic change over time?” is estimated within the model rather than computed afterwards. mintR is built around stm.

A topic model is a tool for discovery and measurement, not a source of truth: there is no “true” number of topics, and the same corpus can be described at several levels of detail. Grimmer and Stewart (2013) summarise the necessary attitude in a principle: “validate, validate, validate”. Social scientists disagree on what topics are. Some take them as measurements of frames or discourses present in the texts; others take them as aids to a more systematic reading, with the researcher in charge of interpretation (Macanovic 2022). Grimmer et al. (2022) argue that there is rarely a single underlying concept to recover: different models and specifications capture different aspects of the texts, and their output has to be validated against human judgement for the purpose at hand.

Topic models of the LDA family rest on the bag of words: they ignore word order and context. Models built on contextual embeddings, such as BERTopic, take context into account. Comparing LDA and BERTopic on the same corpus of biology articles, Benz et al. (2025) find topic spaces with a similar global configuration, but clear differences for specific concepts with several meanings. They also note that LDA suits the study of topics that cut across documents, since each document mixes topics.

mintR is organised as a pipeline: a series of steps (tokens, vocabulary, model, metrics, validation), each a documented function whose choices are explicit and can be changed. Lee et al. (2020) argue that such pipelines make the design stage, where much of the critical thinking of digital research now takes place, transparent and open to revision.

The example corpus

The examples use the 60 inaugural addresses of US presidents from 1789 to 2021, included in quanteda (Benoit et al. 2018). A whole address mixes many themes; to let the model see them separately, each address is split into paragraphs, and very short paragraphs (greetings, thanks) are dropped.

library(mintR)
library(stm) # attached: stm's formulas use its s() for splines
library(data.table)
library(ggplot2)

corpus <- quanteda::corpus_reshape(quanteda::data_corpus_inaugural, to = "paragraphs")
paragraphs <- data.table(quanteda::docvars(corpus), text = as.character(corpus))
paragraphs[, id := sprintf("%d_%s_%03d", Year, President, seq_len(.N)), by = .(Year, President)]
paragraphs <- paragraphs[stringi::stri_count_words(text) >= 40]
paragraphs[, .(paragraphs = .N, words = sum(stringi::stri_count_words(text)))]

   paragraphs  words
        <int>  <int>
1:       1091 126009

The choice of the unit of analysis (address, paragraph, fixed-size chunk) is substantive: topics are defined by co-occurrence within units, so units should be small enough to be about few things. When documents have very different lengths, chunk_texts() cuts them into chunks of about the same number of words, grouping whole paragraphs when the texts have some; when they are very short (social media posts), pool_texts() joins those that share a context, such as an author’s posts in one thread.

Two checks come before counting words. Duplicated texts (a press release reproduced by several outlets, a bot posting the same message) weigh several times in the vocabulary and the topics: find_duplicate_texts() finds them, and extract_ngrams() warns when it sees some. Texts in other languages form topics of their own: detect_language() finds them, and get_stop_words() gives stop words for other languages. Links, handles or words cut by a hyphen at the end of a line (OCR) are removed or repaired by clean_texts(). The inaugural addresses need none of this.

From texts to tokens

Words and n-grams

A topic model sees each document as a bag of words: word order is ignored, only counts matter. extract_ngrams() lower-cases the texts, splits them into words, and counts words (unigrams) and sequences of two words (bigrams) by document. Bigrams keep together expressions whose meaning differs from their words (“United States”, “federal government”, “general welfare”); they are joined by _.

tokens <- extract_ngrams(paragraphs, ngrams = 2L, document_col = "id")
tokens[id == id[1]][order(-count)][1:8]

                    id        token ngram count
                <char>       <char> <int> <int>
1: 1789_Washington_002      country     1     3
2: 1789_Washington_002          day     1     2
3: 1789_Washington_002         hand     1     2
4: 1789_Washington_002        voice     1     2
5: 1789_Washington_002      retreat     1     2
6: 1789_Washington_002     citizens     1     2
7: 1789_Washington_002         dare     1     2
8: 1789_Washington_002 vicissitudes     1     1

N-grams containing a stop word (very frequent function words: “the”, “of”, “and”; by default the list of tidytext, Silge and Robinson (2017)) or a character other than a letter are dropped: they say little about topics and dominate the counts. The list of stop words is a choice: in some corpora, pronouns (“we”, “they”) carry meaning and should be kept.

Stop words are also part of some useful multiword terms (“rate of growth”, “balance of payments”). Justeson and Katz (1995) found that more than 99% of the multiword technical terms in dictionaries of four fields contain only nouns, adjectives and, rarely, the preposition “of”. They nevertheless left prepositions out of their algorithm, because the candidates containing them were mostly not terms. extract_ngrams() follows the same default (any n-gram with a stop word is dropped), and inner_stop_words_from = 3L, with inner_stop_words = "of", allows the exception: stop words inside, never at the edges, of n-grams of three words or more. For the wider literature on multiword expressions, see Constant et al. (2017).

Which words characterise a document? tf-idf

Raw counts favour words frequent everywhere. The tf-idf weight (Spärck Jones 1972; Salton and Buckley 1988) combines the frequency of a word in a document (term frequency) and its rarity across documents (inverse document frequency):

\[ \mathrm{tf}_{td} = \frac{n_{td}}{\sum_u n_{ud}}, \qquad \mathrm{idf}_t = \log\frac{D}{\mathrm{df}_t}, \qquad \text{tf-idf}_{td} = \mathrm{tf}_{td} \times \mathrm{idf}_t, \]

with \(n_{td}\) the count of word \(t\) in document \(d\), \(D\) the number of documents and \(\mathrm{df}_t\) the number of documents containing \(t\). A word in every document has \(\mathrm{idf} = 0\). Here, the “documents” are whole addresses: which words distinguish an address from the others?

by_address <- extract_ngrams(paragraphs, ngrams = 2L, document_col = c("Year", "President"))
tf_idf <- compute_tf_idf(by_address, document_col = c("Year", "President"))
top_tf_idf <- tf_idf[Year %in% c(1861, 1933, 1961, 2009)][order(Year, -tf_idf)]
top_tf_idf[, head(.SD, 5), by = .(Year, President), .SDcols = c("token", "tf_idf")]

     Year President       token      tf_idf
    <int>    <char>      <char>       <num>
 1:  1861   Lincoln      secede 0.011673114
 2:  1861   Lincoln    minority 0.011581113
 3:  1861   Lincoln      clause 0.009696928
 4:  1861   Lincoln surrendered 0.009650927
 5:  1861   Lincoln   expressly 0.008855690
 6:  1933 Roosevelt      helped 0.021186780
 7:  1933 Roosevelt    stricken 0.014961064
 8:  1933 Roosevelt  leadership 0.013864458
 9:  1933 Roosevelt    respects 0.011218441
10:  1933 Roosevelt   emergency 0.010467403
11:  1961   Kennedy   forebears 0.013632054
12:  1961   Kennedy    finished 0.013632054
13:  1961   Kennedy      pledge 0.013209767
14:  1961   Kennedy       final 0.012916439
15:  1961   Kennedy       deeds 0.012006943
16:  2009     Obama         icy 0.008296544
17:  2009     Obama        jobs 0.008231156
18:  2009     Obama      waters 0.006891991
19:  2009     Obama      storms 0.006891991
20:  2009     Obama      earned 0.006891991
     Year President       token      tf_idf
    <int>    <char>      <char>       <num>

Lincoln’s 1861 address turns on secession and the Constitution (secede, minority, clause), Roosevelt’s 1933 address on the economic emergency (stricken, emergency), Obama’s 2009 address on the crisis (jobs, storms, icy waters). Kennedy’s distinctive words (forebears, pledge) are those of his rhetoric rather than of a theme. tf-idf finds what is specific to each document, which is often a good first view of a corpus; it also shows that “specific” is not always “thematic”.

Choosing the vocabulary

Most words occur in very few documents: they cannot form co-occurrence patterns, add noise and slow the model. Denny and Spirling (2018) show that such preprocessing choices can change the results of unsupervised models substantially: they are part of the model, and the next guide compares vocabularies rather than picking one blindly. Here, the vocabulary is built in two steps, and the whole path from tokens to the model input is a table that is filtered and passed on, so that every step can be inspected and redone.

Step 1: drop the rarest tokens. filter_vocabulary() keeps the tokens that occur in at least five paragraphs, and reports what the rule removes: most distinct tokens, but a small share of the occurrences.

frequent <- filter_vocabulary(tokens, document_col = "id", min_documents = 5)

Step 2: keep the tokens that are specific to their paragraph. The tf-idf of the previous section can also be used as a filter. compute_tf_idf() is applied to the paragraphs here, and returns one row per token and paragraph, with its count and the size of the n-gram.

tf_idf_par <- compute_tf_idf(frequent, document_col = "id")
tf_idf_par[order(-tf_idf)][1:5, .(id, token, ngram, count, tf_idf)]

                   id    token ngram count   tf_idf
               <char>   <char> <int> <num>    <num>
1:     1969_Nixon_030   voices     1     6 2.010971
2: 1933_Roosevelt_016 respects     1     4 1.553520
3:      2001_Bush_005    story     1     5 1.495948
4:     1969_Nixon_074  eternal     1     2 1.486597
5: 1945_Roosevelt_011  learned     1     2 1.340647

filter_group_quantile() keeps, within each group, the rows above a quantile of a metric. The groups matter. A bigram is rarer than a word, so its idf and tf-idf are systematically higher: one threshold for all the tokens would keep almost only bigrams. With by = "ngram", unigrams and bigrams each keep their own most specific part. The metric can be any numeric column of the table, here tf_idf; count (the count of the token in the paragraph) or idf are other options. Here, the lowest 25% of each size are dropped (probs = 0.25).

specific <- filter_group_quantile(tf_idf_par, metric = "tf_idf", probs = 0.25, by = "ngram")

The function reports how many rows and distinct tokens remain. To see what each step of the vocabulary removed, vocabulary_diagnostics() takes the token table after each step, in order. Rare tokens are numerous but weigh little in the total number of occurrences; a tf-idf threshold removes no distinct token here (each survives in at least one paragraph) but a quarter of the remaining occurrences.

steps <- list(all = tokens, frequent = frequent, specific = specific)
vocabulary_diagnostics(steps, document_col = "id")

       step n_types n_occurrences n_documents share_types share_occurrences
     <fctr>   <int>         <num>       <int>       <num>             <num>
1:      all   17410         57089        1091    1.000000         1.0000000
2: frequent    2154         36356        1091    0.123722         0.6368302
3: specific    2154         27529        1090    0.123722         0.4822120
   removed_types removed_occurrences
           <int>               <num>
1:            NA                  NA
2:         15256               20733
3:             0                8827

plot_vocabulary_diagnostics(vocabulary_diagnostics(steps, by = "ngram"), base_size = 12)

A paragraph can lose all its tokens in the process; n_documents shows how many remain, and tokens_to_stm() below only keeps paragraphs with at least one token.

The input of stm

tokens_to_stm() turns the filtered table into stm’s input: a list of documents (each a matrix of word indices and counts), the vocabulary and the metadata aligned with the documents (matched by identifier, so that covariates cannot be shifted by one row). The columns id, token and count of the filtered table are all it needs.

stm_data <- tokens_to_stm(
  specific,
  meta = paragraphs[, .(id, Year, President, Party)],
  document_col = "id"
)
stm_data

The pipeline is therefore: extract_ngrams() (tokens), then any filter on the vocabulary, then compute_tf_idf() and filter_group_quantile() (specific tokens), then tokens_to_stm(). Each step takes and returns a table, so a step can be replaced or removed without touching the others.

Fitting the model

The model below has 15 topics, and their prevalence varies smoothly with the year (a spline, s(Year)). Spectral initialisation (stm’s default) is deterministic: refitting gives the same model.

fit <- stm(
  stm_data$documents, stm_data$vocab, K = 15,
  prevalence = ~ s(Year), data = stm_data$meta,
  max.em.its = 75, verbose = FALSE
)

Why 15? Here only for illustration. Choosing \(K\) is the subject of the next guide.

Reading the topics

Top words: four measures

A topic is a distribution over thousands of words; we read its top words. “Top” can mean several things, and each measure shows a different side of the topic:

• Probability (\(\beta\), "prob"): the most probable words. Frequent words of the corpus (“government”, “people”) appear in many topics.
• FREX ("frex", Bischof and Airoldi (2012); Roberts et al. (2014)): the harmonic mean of the rank of a word’s probability in the topic (frequency) and of its share of the word’s total probability across topics (exclusivity). It favours words both frequent in the topic and specific to it, and is often the most informative.
• Score ("score"): probability weighted by how much the word’s log-probability exceeds its average over topics.
• Lift ("lift", Taddy (2012)): probability in the topic divided by frequency in the corpus; it highlights rare, specific words.

topic_labels() builds short labels (the same words as stm::labelTopics()), and topic_terms() gives the top words by one or several measures in one table.

labels <- topic_labels(fit, measure = "frex", n_terms = 5)
labels

    topic                                                      label
    <int>                                                     <char>
 1:     1          1: honor, civilization, deep, obligations, belief
 2:     2                 2: friendly, navy, british, army, european
 3:     3        3: oath, president, responsibility, country, solemn
 4:     4       4: texas, institutions, population, endanger, actual
 5:     5            5: interstate, control, officers, party, public
 6:     6 6: measures, circumstances, enter, heaven, fellow_citizens
 7:     7                   7: earth, dignity, seek, weapons, energy
 8:     8             8: americans, america, forward, heart, journey
 9:     9 9: government, people, federal_government, failure, cities
10:    10            10: civil, islands, south, civil_service, lines
11:    11        11: citizen, confederacy, power, security, produced
12:    12                 12: peace, nations, peoples, world, europe
13:    13          13: constitution, surrendered, acts, slave, races
14:    14           14: currency, tariff, taxation, money, financial
15:    15          15: union, domestic, preservation, opinion, error

top_terms <- topic_terms(fit, c("prob", "frex", "score", "lift"), n_terms = 6, stm_data = stm_data)
top_terms[topic == 1, .(words = paste(term, collapse = ", ")), by = measure]

   measure                                                        words
    <char>                                                       <char>
1:    prob          honor, civilization, war, obligations, deep, duties
2:    frex        honor, deep, civilization, avail, belief, obligations
3:   score honor, civilization, deep, deliberate, preparation, devotion
4:    lift                   honor, avail, deliberate, lie, blind, deep

Documents

Top words are not enough: a topic is understood by reading the documents where it dominates. topic_documents() returns, for each topic, the documents with the highest proportion \(\theta_{dk}\) (or a random draw among the documents it dominates, method = "sampled"), and can drop duplicated texts and keep one document per author.

top_docs <- topic_documents(fit, stm_data$meta, n = 2, document_col = "id")
top_docs[paragraphs, on = "id", text := i.text]
top_docs[topic == 1, .(id, theta = round(theta, 2), text = paste0(substr(text, 1, 250), "..."))]

                  id theta
              <char> <num>
1: 1889_Harrison_035  0.95
2: 1901_McKinley_005  0.94
                                                                                                                                                                                                                                                            text
                                                                                                                                                                                                                                                          <char>
1: I do not mistrust the future. Dangers have been in frequent ambush along our path, but we have uncovered and vanquished them all. Passion has swept some of our communities, but only to give us a new demonstration that the great body of our people are...
2: Four years ago we stood on the brink of war without the people knowing it and without any preparation or effort at preparation for the impending peril. I did all that in honor could be done to avert the war, but without avail. It became inevitable; a...

Words of interest

topic_word_mass() measures how much of each topic’s probability falls on words matching a pattern: here, war and the military.

topic_word_mass(fit, "^(war|wars|army|navy|military|enemy|enemies|peace)$", n_terms = 4)[1:4]

   topic       mass                    top_terms
   <int>      <num>                       <char>
1:    12 0.06603970   peace, war, wars, military
2:     2 0.03975989    war, navy, army, military
3:    10 0.02791651    war, military, navy, army
4:     1 0.01881636 war, enemies, wars, military

Prevalence of topics

The average prevalence of a topic is the mean of its proportions over documents. plot_topic_prevalence() returns a ggplot (save it with ggplot2::ggsave()).

plot_topic_prevalence(fit, labels)

Change over time

With a prevalence covariate, stm::estimateEffect() regresses topic proportions on the covariates, taking the uncertainty of \(\theta\) into account (Roberts et al. 2019). topic_effects() turns its simulations into a table, and plot_topic_prevalence_over_time() draws expected proportions by year, with 95% intervals.

set.seed(1)
effects <- estimateEffect(1:15 ~ s(Year), fit, metadata = stm_data$meta, uncertainty = "Global")
plot_topic_prevalence_over_time(effects, labels, covariate = "Year", n_sims = 200, wrap_width = 25, base_size = 11)

The trajectories are a summary of the model, not of the texts: before interpreting one, read documents of the topic in the relevant years, and check that the topic means the same thing over the period.

Where next

• Choosing a topic model: how many topics, which vocabulary, how stable are the topics.
• Validating topics with humans: intrusion tests.
• Building the vocabulary: every choice of the vocabulary in detail, including multiword expressions merged into single tokens instead of counted on top of the words.
• Reading a topic model: all the ways of exploring a fitted model.

For a broader introduction to text as data in the social sciences, see Grimmer et al. (2022).

References

Benoit, Kenneth, Kohei Watanabe, Haiyan Wang, et al. 2018. “Quanteda: An R Package for the Quantitative Analysis of Textual Data.” Journal of Open Source Software 3 (30): 774. https://doi.org/10.21105/joss.00774.

Benz, Pierre, Carolina Pradier, Diego Kozlowski, Natsumi S. Shokida, and Vincent Larivière. 2025. “Mapping the Unseen in Practice: Comparing Latent Dirichlet Allocation and BERTopic for Navigating Topic Spaces.” Scientometrics 130: 3839–70. https://doi.org/10.1007/s11192-025-05339-6.

Bischof, Jonathan M., and Edoardo M. Airoldi. 2012. “Summarizing Topical Content with Word Frequency and Exclusivity.” Proceedings of the 29th International Conference on Machine Learning, 201–8.

Blei, David M. 2012. “Probabilistic Topic Models.” Communications of the ACM 55 (4): 77–84. https://doi.org/10.1145/2133806.2133826.

Blei, David M., Andrew Y. Ng, and Michael I. Jordan. 2003. “Latent Dirichlet Allocation.” Journal of Machine Learning Research 3: 993–1022.

Constant, Mathieu, Gülşen Eryiğit, Johanna Monti, et al. 2017. “Multiword Expression Processing: A Survey.” Computational Linguistics 43 (4): 837–92. https://doi.org/10.1162/COLI_a_00302.

Denny, Matthew J., and Arthur Spirling. 2018. “Text Preprocessing for Unsupervised Learning: Why It Matters, When It Misleads, and What to Do about It.” Political Analysis 26 (2): 168–89. https://doi.org/10.1017/pan.2017.44.

Grimmer, Justin, Margaret E. Roberts, and Brandon M. Stewart. 2022. Text as Data: A New Framework for Machine Learning and the Social Sciences. Princeton University Press.

Grimmer, Justin, and Brandon M. Stewart. 2013. “Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts.” Political Analysis 21 (3): 267–97. https://doi.org/10.1093/pan/mps028.

Justeson, John S., and Slava M. Katz. 1995. “Technical Terminology: Some Linguistic Properties and an Algorithm for Identification in Text.” Natural Language Engineering 1 (1): 9–27. https://doi.org/10.1017/S1351324900000048.

Lee, Ashley S., Poom Chiarawongse, Jo Guldi, and Andras Zsom. 2020. “The Role of Critical Thinking in Humanities Infrastructure: The Pipeline Concept with a Study of HaToRI (Hansard Topic Relevance Identifier).” Digital Humanities Quarterly 14 (3).

Macanovic, Ana. 2022. “Text Mining for Social Science – the State and the Future of Computational Text Analysis in Sociology.” Social Science Research 108: 102784. https://doi.org/10.1016/j.ssresearch.2022.102784.

Roberts, Margaret E., Brandon M. Stewart, and Edoardo M. Airoldi. 2016. “A Model of Text for Experimentation in the Social Sciences.” Journal of the American Statistical Association 111 (515): 988–1003. https://doi.org/10.1080/01621459.2016.1141684.

Roberts, Margaret E., Brandon M. Stewart, Dustin Tingley, et al. 2014. “Structural Topic Models for Open-Ended Survey Responses.” American Journal of Political Science 58 (4): 1064–82. https://doi.org/10.1111/ajps.12103.

Roberts, Margaret E., Brandon M. Stewart, and Dustin Tingley. 2019. “stm: An R Package for Structural Topic Models.” Journal of Statistical Software 91 (2): 1–40. https://doi.org/10.18637/jss.v091.i02.

Salton, Gerard, and Christopher Buckley. 1988. “Term-Weighting Approaches in Automatic Text Retrieval.” Information Processing & Management 24 (5): 513–23.

Silge, Julia, and David Robinson. 2017. Text Mining with R: A Tidy Approach. O’Reilly.

Spärck Jones, Karen. 1972. “A Statistical Interpretation of Term Specificity and Its Application in Retrieval.” Journal of Documentation 28 (1): 11–21.

Taddy, Matt. 2012. “On Estimation and Selection for Topic Models.” Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, 1184–93.