A collocation is a sequence of words that occurs together
more often than chance would predict ("monetary policy", "rate of
growth"). Counting such a sequence both as a unit and as separate words,
as extract_ngrams() does by default, counts each occurrence two or three
times. The usual remedy is to choose the collocations first and then to
rewrite the texts with them (Lau et al. 2013; quanteda's
textstat_collocations() and tokens_compound()).
score_collocations() is the first step. It returns, for every
candidate n-gram of the corpus, its frequency and the measures used in
the literature to decide whether it is a unit. Read the table, choose a
rule with select_collocations() (and compare rules with
compare_collocation_rules()), then give the selected n-grams to
extract_ngrams() through its compounds argument.
For an n-gram \(g = w_1 \dots w_n\) with count \(c_g\) among the \(N\) word positions of the corpus, \(p(g) = c_g / N\). Its expected probability under independence of its parts averages the \(n - 1\) ways of cutting it in two (the "fair dispersion" of da Silva and Lopes 1999): $$\bar p(g) = \frac{1}{n-1}\sum_{i=1}^{n-1} p(w_1 \dots w_i)\, p(w_{i+1} \dots w_n).$$ For a bigram this is simply \(p(w_1) p(w_2)\). Then $$\mathrm{PMI}(g) = \log\frac{p(g)}{\bar p(g)}, \qquad \mathrm{NPMI}(g) = \frac{\mathrm{PMI}(g)}{-\log p(g)}, \qquad t(g) = \frac{c_g - N \bar p(g)}{\sqrt{c_g}}.$$ PMI (Church and Hanks 1990) compares observed and expected frequency; NPMI (Bouma 2009) rescales it to [-1, 1] (1 when the words never occur apart), so that one threshold has the same meaning in every corpus. The t-score is large for collocations that are both frequent and associated.
Nesting follows the C-value of Frantzi, Ananiadou and Mima (2000): an n-gram that occurs mostly inside a longer candidate ("interest rate" inside "long term interest rate") is less of a unit by itself. With \(T_g\) the candidates of size \(n + 1\) that contain \(g\), $$C(g) = \log_2(n)\left(c_g - \frac{1}{|T_g|}\sum_{b \in T_g} c_b\right),$$ and \(C(g) = \log_2(n)\, c_g\) when \(g\) is not nested.
Usage
score_collocations(
data,
text_col = "text",
ngrams = 2:3,
min_count = 5L,
min_nchar = 2L,
stop_words = get_stop_words("en"),
inner_stop_words_from = Inf,
inner_stop_words = NULL,
language_col = NULL,
chunk_size = NULL,
...
)Arguments
- data
A
data.frameordata.tablewith the texts.- text_col
Character scalar, name of the text column.
- ngrams
Integer vector of n-gram sizes to score, between 2 and 4.
- min_count
Integer scalar, minimal number of occurrences of a candidate. Rare candidates have unstable measures; Justeson and Katz (1995) require at least 2.
- min_nchar
Integer scalar, minimal number of characters of each word of an n-gram (inner stop words excepted).
3Lremoves the one- and two-letter tokens of formulas and OCR ("kt", "mt") in academic texts, but also abbreviations such as "uk" or "eu".- stop_words
Stop words: a character vector, used for all texts (default: English,
get_stop_words()), or a named list of vectors by language (get_stop_words(c("en", "fr"))) used withlanguage_col.NULL(orcharacter(0)) uses no stop word list: only n-grams with digits, punctuation or symbols are dropped.- inner_stop_words_from
Integer scalar. From this n-gram size onwards, stop words are allowed inside an n-gram, but not as its first or last word: with
3L, "rate_of_growth" is kept but "of_growth_in" and "the_rate_of" are not. The defaultInfdrops every n-gram containing a stop word, and says so in a message (once per session) when n-grams of size 3 or more are asked for.- inner_stop_words
Optional character vector. With
inner_stop_words_from, only these stop words are allowed inside n-grams (e.g."of"), instead of every stop word.- language_col
Optional character scalar, a column of language codes (e.g. from
detect_language()). Each text is then filtered with the stop words of its language (stop_wordsmust be a named list); texts whose language has no list use all the lists together.- chunk_size
Optional integer, number of texts read at once. Counts are computed by chunk and summed (n-grams do not cross texts, so the result is identical); memory then depends on the chunk, not on the corpus (about 110 Mb of R memory per million words at once).
NULLreads all texts at once.- ...
Not used: an argument renamed in the development version of mintR gives an error naming its replacement.
Value
A data.table with one row per candidate: token (words joined
by _), ngram (size), count, n_documents (number of texts
containing it), pmi, npmi, t_score, count_nested (occurrences
inside longer candidates of size ngram + 1), share_nested
(count_nested / count) and c_value, by decreasing count.
Which measure?
Nokel and Loukachevitch (2015) compared 17 measures
for adding the top 1,000 bigrams to topic models. Measures of the PMI
family (PMI, NPMI, log-likelihood, chi-square, Dice...) put rare, unusual
bigrams first and changed the topics little; measures that favour
frequent bigrams (frequency, t-score...) improved topic coherence. Lau,
Baldwin and Newman (2013) ranked bigrams with a t-test and found that
about 1,000 bigrams was the best trade-off. With NPMI, a minimal count
is therefore essential: see select_collocations().
Stop words
Candidates are built as in extract_ngrams(), with
the same stop_words, inner_stop_words_from and inner_stop_words:
by default no candidate contains a stop word, and with
inner_stop_words_from = 3L stop words are allowed inside (not at the
edges of) longer n-grams ("rate_of_growth"). Stop words still count as
word positions: they separate words, so "policy of the bank" never gives
the candidate "policy_bank", and they enter \(N\). Use the same three
arguments here and in extract_ngrams().
References
Bouma, G. (2009). Normalized (pointwise) mutual information in collocation extraction. Proceedings of the Biennial GSCL Conference, 31-40.
Church, K. W. and Hanks, P. (1990). Word association norms, mutual information, and lexicography. Computational Linguistics, 16(1), 22-29.
da Silva, J. F. and Lopes, G. P. (1999). A local maxima method and a fair dispersion normalization for extracting multiword units. Proceedings of the 6th Meeting on the Mathematics of Language, 369-381.
Frantzi, K., Ananiadou, S. and Mima, H. (2000). Automatic recognition of multi-word terms: the C-value/NC-value method. International Journal on Digital Libraries, 3(2), 115-130. doi:10.1007/s007999900023
Lau, J. H., Baldwin, T. and Newman, D. (2013). On collocations and topic models. ACM Transactions on Speech and Language Processing, 10(3), 10:1-10:14. doi:10.1145/2483969.2483972
Nokel, M. and Loukachevitch, N. (2015). A method of accounting bigrams in topic models. Proceedings of the 11th Workshop on Multiword Expressions, 1-9. https://aclanthology.org/W15-0901
Examples
df <- data.frame(id = 1:4, text = c(
"monetary policy and fiscal policy",
"the monetary policy rule",
"monetary policy matters",
"a policy of growth"
))
score_collocations(df, ngrams = 2:3, min_count = 1)
#> token ngram count n_documents pmi npmi t_score
#> <char> <int> <int> <int> <num> <num> <num>
#> 1: monetary_policy 2 3 3 1.163151 0.6948430 1.190785
#> 2: fiscal_policy 2 1 1 1.163151 0.4195180 0.687500
#> 3: policy_rule 2 1 1 1.163151 0.4195180 0.687500
#> 4: policy_matters 2 1 1 1.163151 0.4195180 0.687500
#> 5: monetary_policy_rule 3 1 1 1.673976 0.6037594 0.812500
#> 6: monetary_policy_matters 3 1 1 1.673976 0.6037594 0.812500
#> count_nested share_nested c_value
#> <num> <num> <num>
#> 1: 2 0.6666667 2.000000
#> 2: 0 0.0000000 1.000000
#> 3: 1 1.0000000 0.000000
#> 4: 1 1.0000000 0.000000
#> 5: 0 0.0000000 1.584963
#> 6: 0 0.0000000 1.584963