library(mintR)
library(data.table)
library(ggplot2)
corpus <- quanteda::corpus_reshape(quanteda::data_corpus_inaugural, to = "paragraphs")
paragraphs <- data.table(quanteda::docvars(corpus), text = as.character(corpus))
paragraphs[, id := sprintf("%d_%s_%03d", Year, President, seq_len(.N)), by = .(Year, President)]
paragraphs <- paragraphs[stringi::stri_count_words(text) >= 40]A topic model only sees counts of tokens in documents. Every decision taken before fitting it (what a document is, what a token is, which tokens are kept) changes those counts, and therefore the topics. Denny and Spirling (2018) show on several corpora that common preprocessing choices can change the results of unsupervised models substantially, and propose to compare the specifications rather than choose one by habit. This guide goes through the choices one by one, with what the literature says about each and the functions of mintR that implement them. The first guide shows the pipeline in its simplest form; choosing a topic model shows how to compare vocabularies once they are built.
We use the inaugural addresses of US presidents, split into paragraphs, as in the other guides.
What is a document?
Topics are defined by words that occur in the same documents, so the document should be small enough to be about few things. A whole address mixes the economy, war and the constitution; a paragraph is more often about one of them. When documents range from short notes to long reports or books, chunk_texts() cuts them into pieces of comparable size. It offers three rules, which answer different worries.
-
Paragraphs grouped into chunks (
unit = "paragraphs", the default): consecutive paragraphs are grouped until a chunk reaches aboutchunk_wordswords (400 by default), without cutting a paragraph, which is a unit of meaning. The paragraphs are the lines of each text, or the rows of the table when it has one row per paragraph (paragraph_col, as in a database of paragraphs). A paragraph longer thanmax_paragraph_words(texts whose line breaks were lost, OCR) is cut by words. -
Chunks of a fixed length (
unit = "words"): chunks of exactlychunk_wordswords, wherever the count falls. -
A fixed number of chunks (
n_chunks): each text is cut into the same number of pieces, whatever its length, and no word is lost; the pieces of a book are then much longer than those of an article.
To prevent a few long books from outweighing hundreds of articles, max_chunks keeps at most a number of chunks per text, spread evenly through it. This is a sample: the other chunks are discarded, and the function warns with the share of words lost (a 400,000-word book cut into chunks of 400 words with max_chunks = 10 keeps 1% of its text). By default every chunk is kept.
long_texts <- data.table(
id = c("short", "long"),
text = c(
paste(rep(paste(rep("word", 120), collapse = " "), 4), collapse = "\n"),
paste(rep(paste(rep("word", 90), collapse = " "), 30), collapse = "\n")
)
)
chunk_texts(long_texts, "id", chunk_words = 400)[, .(chunks = .N, words = paste(n_words, collapse = ", ")), by = id] id chunks words
<char> <int> <char>
1: short 2 360, 120
2: long 8 360, 360, 360, 360, 360, 360, 360, 180
chunk_texts(long_texts, "id", unit = "words", chunk_words = 400, max_chunks = 3)[, .(chunks = .N, total = n_chunks_total[1]), by = id] id chunks total
<char> <int> <int>
1: short 1 1
2: long 3 7Very short texts raise the opposite problem. A social media post has a dozen words, too few for the words of a topic to co-occur, and topics fitted on single posts reflect vocabulary rather than themes. pool_texts() joins the texts that share a context into one document: an author’s posts in the same thread (the choice of a study of Bluesky posts), a speaker’s interventions in one sitting, the comments of one article. Pooling changes the unit of the analysis: covariates must then be defined at the level of the pooled document. Mehrotra et al. (2013) compared ways of pooling tweets (by author, by trending term, by time, by hashtag): pooling by hashtag gave much more coherent topics than single tweets and than the other schemes.
posts <- data.table(
author = c("a", "a", "a", "b"), thread = c("t1", "t1", "t2", "t3"),
time = 1:4, text = c("First post of a thread", "and its follow-up", "Another subject", "A post by b")
)
pool_texts(posts, by = c("author", "thread"), order_col = "time") document author thread text n_texts
<char> <char> <char> <char> <int>
1: d000001 a t1 First post of a thread\nand its follow-up 2
2: d000002 a t2 Another subject 1
3: d000003 b t3 A post by b 1
n_words start end
<int> <int> <int>
1: 8 1 2
2: 2 3 3
3: 4 4 4Before counting words
Cleaning
extract_ngrams() drops tokens with digits or symbols, but some noise is made of letters: the “https”, “bsky” and “social” of links and handles in social media posts, the “employ- ment” of OCR texts whose words were cut at the end of a line. clean_texts() removes links and e-mails, keeps the name of handles and the word of hashtags (or removes them), decodes HTML entities, straightens curly apostrophes and rejoins words cut by a hyphen. The last step is ambiguous when the line breaks have been lost: “employ- ment” is a cut word, “short- and long-run” a suspended hyphen. The parts are joined only when the joined word occurs elsewhere in the corpus.
clean_texts(c(
"New speech on inflation https://t.co/x1 by @ecb.europa.eu #monetarypolicy",
"full employ- ment and short- and long-run effects of employment policy"
))[1] "New speech on inflation by ecb.europa.eu monetarypolicy"
[2] "full employment and short- and long-run effects of employment policy"Languages
Stop words, and topics, are language-specific: in a corpus of central bank speeches, the speeches in Spanish, German and French each formed a topic of their own, made of their function words. extract_ngrams() uses English stop words by default and warns when texts seem to be in another language (English stop words are less than 10% of their words, against 40-50% in English prose). detect_language() gives the language of each text (with the cld2 detector, or with the stop words of 15 languages); the corpus can then be filtered, or each text given the stop words of its language.
mixed <- data.table(
id = 1:3,
text = c(
"The central bank raised its policy rate to curb inflation in the euro area.",
"La banque centrale a relevé son taux directeur pour freiner l'inflation.",
"Die Zentralbank hat den Leitzins erhöht, um die Inflation zu bremsen."
)
)
mixed[, language := detect_language(text, method = "stopwords")]
extract_ngrams(mixed, 1L, "id", stop_words = get_stop_words(c("en", "fr", "de")), language_col = "language")[, .(tokens = paste(token, collapse = " ")), by = id] id tokens
<int> <char>
1: 1 central bank raised policy rate curb inflation euro
2: 2 banque centrale relevé taux directeur freiner l'inflation
3: 3 zentralbank leitzins erhöht inflation bremsenDuplicates
A text present twice weighs twice in the vocabulary and in its topic, and fills the list of its topic’s representative documents: press releases reproduced by several outlets, the same speech published by two institutions, a bot posting the same message with a different number. find_duplicate_texts() finds texts identical once links, digits and punctuation are removed and, with method = "near", texts that share most of their sequences of five words; extract_ngrams() warns when it finds exact duplicates. Applied to 2,490 central bank speeches, the near-duplicate search found 18 speeches present twice, in the collection of the BIS and in that of the ECB or the Bank of England.
posts <- data.table(
id = 1:4,
text = c("Poll results: 12 votes https://t.co/a", "Poll results: 31 votes https://t.co/b",
"A new release of the app is out today", "Something else entirely here")
)
find_duplicate_texts(posts, "id") id group similarity group_size keep
<int> <int> <num> <int> <lgcl>
1: 1 1 1 2 TRUE
2: 2 1 1 2 FALSE
3: 3 NA NA 1 TRUE
4: 4 NA NA 1 TRUEWhich tokens?
extract_ngrams() lower-cases the texts, splits them into words and drops tokens made of anything else than letters (numbers, symbols). Two further choices deserve a deliberate decision.
Stop words
Stop words are the very frequent function words (“the”, “of”, “and”). By default extract_ngrams() drops the list of the tidytext package (Silge and Robinson 2017); stop_words = NULL keeps every word, and any other vector can be given. Schofield et al. (2017) compared models fitted with and without stop words: removing the few most frequent words of a corpus matters, but beyond them, removing a stop word list mostly makes the top words of topics look cleaner, and removing them after fitting gives similar results. Stop words are thus less a matter of model quality than of readability, and a list can also remove meaningful words: in political speeches, “we” and “they” may carry the argument.
Short tokens
min_nchar sets the minimal length of each word of an n-gram (2 by default). In academic texts, min_nchar = 3L removes the variables of formulas and the debris of OCR (“kt”, “mt”, “ro”), which can rank among the top words of topics; it also removes abbreviations such as “uk” or “eu”. Possessives are dropped before (“bank’s” is counted as “bank”).
Stemming and lemmatising
mintR does not stem words (“economic” and “economy” stay distinct). Schofield and Mimno (2016) found that stemmers bring no meaningful improvement in likelihood or coherence and can make topics less stable; they suggest, if near-duplicates clutter the lists of top words, to group them for display after fitting.
Multiword expressions
Many concepts of the social sciences are expressions of several words: “monetary policy”, “labour market”, “rate of growth”. Counting only single words loses them, and the simplest fix, counting the bigrams in addition to the words (extract_ngrams(ngrams = 2L)), counts each occurrence of “monetary policy” three times: once as monetary, once as policy, once as monetary_policy. This inflates exactly what a topic model learns from, the co-occurrence of tokens in documents.
The literature distinguishes two tasks (Constant et al. 2017). Discovery finds which sequences of words are units; identification marks their occurrences in the texts. For topic models, the evidence comes from studies that discover collocations first, then rewrite the texts so that the selected sequences become single tokens, and the words are counted on their own only where they occur outside them. Lau et al. (2013) did so on four corpora: replacing words by the top-ranked bigrams made the models more parsimonious and the topics more coherent; about 1,000 bigrams was the best trade-off, and up to 10,000 still improved topics at the cost of a more complex model. Other routes exist: topic models with phrases built in (Wang et al. 2007), phrases found after fitting for display (Blei and Lafferty 2009), or noun phrases found with a part-of-speech tagger and added to the words (Handler et al. 2016). mintR implements the first route, which needs no tagger and leaves stm unchanged.
Scoring candidate n-grams
score_collocations() scores every candidate n-gram of the corpus. With \(c_g\) the count of an n-gram \(g\) among the \(N\) word positions of the corpus and \(p(g) = c_g / N\), its probability expected if its parts were independent is, for a bigram, \(p(w_1)\,p(w_2)\); for longer n-grams, mintR averages the ways of cutting \(g\) in two, the “fair dispersion” of Silva and Lopes (1999):
\[ \bar p(g) = \frac{1}{n-1}\sum_{i=1}^{n-1} p(w_1 \dots w_i)\, p(w_{i+1} \dots w_n). \]
Three association measures compare \(p(g)\) with \(\bar p(g)\):
- pointwise mutual information (Church and Hanks 1990), \(\mathrm{PMI}(g) = \log \frac{p(g)}{\bar p(g)}\): how many times more often than chance the words occur together;
- normalised PMI (Bouma 2009), \(\mathrm{NPMI}(g) = \mathrm{PMI}(g) / -\log p(g)\), which lies between -1 and 1 and equals 1 when the words never occur apart, so that a threshold means the same in every corpus;
- the t-score, \(t(g) = (c_g - N\bar p(g)) / \sqrt{c_g}\), which is large for sequences that are both frequent and associated.
The choice of measure matters. Nokel and Loukachevitch (2015) compared 17 measures for adding the top 1,000 bigrams to topic models: measures of the PMI family put rare, unusual bigrams at the top and changed the topics little, while measures favouring frequent bigrams, among which the t-score and raw frequency, improved topic coherence. A minimal count is therefore essential with NPMI.
scores <- score_collocations(paragraphs, ngrams = 2:3, min_count = 5)
scores[ngram == 2][order(-npmi)][1:6, .(token, count, npmi, t_score)] token count npmi t_score
<char> <int> <num> <num>
1: indian_tribes 6 0.9710941 2.449334
2: eighteenth_amendment 5 0.8736087 2.235749
3: chief_magistrate 10 0.8700726 3.161422
4: coordinate_branches 5 0.8652061 2.235720
5: nuclear_weapons 5 0.8642044 2.235717
6: fellow_citizens 105 0.8566814 10.223363
scores[ngram == 2][order(-t_score)][1:6, .(token, count, npmi, t_score)] token count npmi t_score
<char> <int> <num> <num>
1: fellow_citizens 105 0.8566814 10.223363
2: american_people 37 0.5045908 5.982351
3: federal_government 31 0.5486452 5.509474
4: foreign_nations 15 0.5122465 3.835154
5: almighty_god 13 0.7321180 3.601202
6: national_life 13 0.4865844 3.564130NPMI ranks first rare, fixed names (“indian tribes”, “eighteenth amendment”); the t-score ranks first the frequent expressions of the genre (“fellow citizens”, “american people”).
Nested expressions
A bigram may be frequent only because it is part of a longer expression. The C-value of Frantzi et al. (2000) corrects the frequency of an n-gram \(g\) by the frequency of the longer candidates \(T_g\) that contain it: \(C(g) = \log_2(n)\,\big(c_g - \frac{1}{|T_g|}\sum_{b \in T_g} c_b\big)\). score_collocations() reports it, with share_nested, the share of the occurrences of \(g\) that sit inside a candidate one word longer.
Stop words inside expressions
By default, no candidate contains a stop word, so “rate of growth” is never a candidate. Justeson and Katz (1995) found that more than 99% of the multiword terms of four technical dictionaries contain only nouns, adjectives and the preposition “of”, but also that candidates containing a preposition are mostly not terms. inner_stop_words_from = 3L allows stop words inside n-grams of three words or more, never at their edges, and inner_stop_words = "of" restricts this to “of”. The same arguments must then be given to extract_ngrams(). Stop words always count as word positions: they separate words, so “policy of the bank” never yields “policy_bank”.
scores <- score_collocations(
paragraphs, ngrams = 2:3, min_count = 5,
inner_stop_words_from = 3L, inner_stop_words = "of"
)
scores[ngram == 3][order(-t_score)][1:6, .(token, count, npmi, t_score, c_value)] token count npmi t_score c_value
<char> <int> <num> <num> <num>
1: form_of_government 13 0.5987467 3.590757 20.604513
2: blessings_of_liberty 7 0.6367862 2.640589 11.094738
3: declaration_of_independence 6 0.7971735 2.448612 9.509775
4: oath_of_office 6 0.7351302 2.447862 9.509775
5: sense_of_responsibility 6 0.6627225 2.446143 9.509775
6: time_of_peace 6 0.4505700 2.421846 9.509775Selecting: two rules
select_collocations() applies one of two rules. The "npmi" rule keeps the n-grams above an NPMI threshold (0.5 by default, a starting point to adjust by reading the n-grams around it); the "top_n" rule keeps the n best by t-score, as Lau et al. (2013) did. Both apply a minimal count and can drop n-grams that occur mostly inside longer ones (max_share_nested). compare_collocation_rules() shows what each rule keeps: how many n-grams of each size, how many occurrences they cover, how many both rules share, and examples kept by one rule only.
compare_collocation_rules(scores, list(
npmi_0.5 = list(rule = "npmi", min_npmi = 0.5),
top_100 = list(rule = "top_n", n = 100)
)) rule ngram n_selected occurrences median_count n_shared
<char> <int> <int> <int> <num> <int>
1: npmi_0.5 2 54 534 6.0 54
2: npmi_0.5 3 10 63 5.5 10
3: top_100 2 80 713 6.0 54
4: top_100 3 12 75 6.0 10
only_here
<char>
1:
2:
3: national_life, free_people, national_government, free_government, civil_war
4: time_of_peace, system_of_governmentHere every expression kept by the NPMI rule is also among the 100 best by t-score, and the t-score rule adds frequent but looser associations (“national life”, “free government”, “civil war”). On larger corpora the NPMI rule also keeps rare, fixed expressions that the t-score ranks low. Neither is right in general: the question is which expressions should be one token for the purpose of the study, and the table of scores is meant to be read.
Rewriting the texts
Given the selected n-grams, extract_ngrams(compounds = ) rewrites each text with its words in order. Where a selected n-gram occurs, its words are replaced by a single token; elsewhere, the same words are counted as usual. When two selected n-grams overlap (“monetary policy” and “policy rule” in “monetary policy rule”, or a bigram inside a selected trigram), the one with the higher score is kept. With compounds, ngrams is 1L: the result has the single words and the compounds, with their size in ngram.
selected <- select_collocations(scores, rule = "npmi", min_npmi = 0.5)
tokens_compound <- extract_ngrams(
paragraphs, ngrams = 1L, document_col = "id", compounds = selected,
inner_stop_words_from = 3L, inner_stop_words = "of"
)
tokens_compound[, .(occurrences = sum(count), distinct = uniqueN(token)), by = ngram] ngram occurrences distinct
<int> <int> <int>
1: 1 45311 8314
2: 2 532 54
3: 3 63 10How many documents should a token appear in?
Most tokens occur in very few documents: they cannot form co-occurrence patterns and slow the model. The usual filter keeps tokens occurring in at least a few documents. Very frequent tokens can be dropped too, since a token present in most documents distinguishes none of them. filter_vocabulary() applies these rules and reports what each one removes, in distinct tokens and in occurrences.
frequent <- filter_vocabulary(tokens_compound, document_col = "id", min_documents = 5, max_document_share = 0.5)Keeping the tokens specific to each document
The tf-idf of a token in a document (Spärck Jones 1972; Salton and Buckley 1988) is high when the token is frequent in the document and rare elsewhere. filter_group_quantile() keeps, within groups, the rows above a quantile of any numeric column, here tf_idf from compute_tf_idf(). The groups decide what is compared: with by = "ngram", single words and compounds each keep their own best part (compounds are rarer and have higher tf-idf); with the document identifier, each document keeps its most specific tokens and none is emptied.
tf_idf <- compute_tf_idf(frequent, document_col = "id")
specific <- filter_group_quantile(tf_idf, metric = "tf_idf", probs = 0.25, by = "ngram")Checking what each step removed
vocabulary_diagnostics() takes the token table after each step and reports the distinct tokens (types), occurrences and documents that remain. Comparing the two ways of counting expressions is instructive: counting bigrams on top of words gives many more occurrences than there are words in the texts.
tokens_bigrams <- extract_ngrams(paragraphs, ngrams = 2L, document_col = "id")
vocabulary_diagnostics(
list(words_and_bigrams = tokens_bigrams, compounds = tokens_compound,
frequent = frequent, specific = specific),
document_col = "id"
) step n_types n_occurrences n_documents share_types
<fctr> <int> <num> <int> <num>
1: words_and_bigrams 17410 57089 1091 1.0000000
2: compounds 8378 45906 1091 0.4812177
3: frequent 2129 35047 1091 0.1222860
4: specific 2129 26526 1090 0.1222860
share_occurrences removed_types removed_occurrences
<num> <int> <num>
1: 1.0000000 NA NA
2: 0.8041129 9032 11183
3: 0.6139011 6249 10859
4: 0.4646429 0 8521
steps <- list(compounds = tokens_compound, frequent = frequent, specific = specific)
plot_vocabulary_diagnostics(vocabulary_diagnostics(steps, by = "ngram"), share = TRUE, base_size = 12)
From the vocabulary to the model
tokens_to_stm() turns the final table into the input of stm. Since every step above is a choice, the honest procedure is to build a few vocabularies (with and without compounds, with two frequency thresholds…) and to compare the models fitted on each with evaluate_topic_models(), as shown in choosing a topic model.
stm_data <- tokens_to_stm(specific, meta = paragraphs[, .(id, Year, President, Party)], document_col = "id")
stm_dataReferences
Blei, David M., and John D. Lafferty. 2009. Visualizing Topics with Multi-Word Expressions. https://arxiv.org/abs/0907.1013.
Bouma, Gerlof. 2009. “Normalized (Pointwise) Mutual Information in Collocation Extraction.” Proceedings of the Biennial GSCL Conference, 31–40.
Church, Kenneth Ward, and Patrick Hanks. 1990. “Word Association Norms, Mutual Information, and Lexicography.” Computational Linguistics 16 (1): 22–29.
Constant, Mathieu, Gülşen Eryiğit, Johanna Monti, et al. 2017. “Multiword Expression Processing: A Survey.” Computational Linguistics 43 (4): 837–92. https://doi.org/10.1162/COLI_a_00302.
Denny, Matthew J., and Arthur Spirling. 2018. “Text Preprocessing for Unsupervised Learning: Why It Matters, When It Misleads, and What to Do about It.” Political Analysis 26 (2): 168–89. https://doi.org/10.1017/pan.2017.44.
Frantzi, Katerina, Sophia Ananiadou, and Hideki Mima. 2000. “Automatic Recognition of Multi-Word Terms: The C-Value/NC-Value Method.” International Journal on Digital Libraries 3 (2): 115–30. https://doi.org/10.1007/s007999900023.
Handler, Abram, Matthew Denny, Hanna Wallach, and Brendan O’Connor. 2016. “Bag of What? Simple Noun Phrase Extraction for Text Analysis.” Proceedings of the First Workshop on NLP and Computational Social Science, 114–24. https://doi.org/10.18653/v1/W16-5615.
Justeson, John S., and Slava M. Katz. 1995. “Technical Terminology: Some Linguistic Properties and an Algorithm for Identification in Text.” Natural Language Engineering 1 (1): 9–27. https://doi.org/10.1017/S1351324900000048.
Lau, Jey Han, Timothy Baldwin, and David Newman. 2013. “On Collocations and Topic Models.” ACM Transactions on Speech and Language Processing 10 (3): 10:1–14. https://doi.org/10.1145/2483969.2483972.
Mehrotra, Rishabh, Scott Sanner, Wray Buntine, and Lexing Xie. 2013. “Improving LDA Topic Models for Microblogs via Tweet Pooling and Automatic Labeling.” Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, 889–92. https://doi.org/10.1145/2484028.2484166.
Nokel, Michael, and Natalia Loukachevitch. 2015. “A Method of Accounting Bigrams in Topic Models.” Proceedings of the 11th Workshop on Multiword Expressions, 1–9. https://aclanthology.org/W15-0901.
Salton, Gerard, and Christopher Buckley. 1988. “Term-Weighting Approaches in Automatic Text Retrieval.” Information Processing & Management 24 (5): 513–23.
Schofield, Alexandra, Måns Magnusson, and David Mimno. 2017. “Pulling Out the Stops: Rethinking Stopword Removal for Topic Models.” Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, 432–36. https://aclanthology.org/E17-2069.
Schofield, Alexandra, and David Mimno. 2016. “Comparing Apples to Apple: The Effects of Stemmers on Topic Models.” Transactions of the Association for Computational Linguistics 4: 287–300. https://doi.org/10.1162/tacl_a_00099.
Silge, Julia, and David Robinson. 2017. Text Mining with R: A Tidy Approach. O’Reilly.
Silva, Joaquim Ferreira da, and José Gabriel Pereira Lopes. 1999. “A Local Maxima Method and a Fair Dispersion Normalization for Extracting Multi-Word Units from Corpora.” Proceedings of the Sixth Meeting on Mathematics of Language, 369–81.
Spärck Jones, Karen. 1972. “A Statistical Interpretation of Term Specificity and Its Application in Retrieval.” Journal of Documentation 28 (1): 11–21.
Wang, Xuerui, Andrew McCallum, and Xing Wei. 2007. “Topical n-Grams: Phrase and Topic Discovery, with an Application to Information Retrieval.” Proceedings of the Seventh IEEE International Conference on Data Mining, 697–702. https://doi.org/10.1109/ICDM.2007.86.