Diagnostics of each topic of a model

One row per topic, to spot the topics to read carefully rather than only to choose K:

• prevalence: mean topic proportion over documents;

• tokens: expected number of words assigned to the topic, \(\sum_d \theta_{dk} N_d\);

• coherence_umass, coherence_npmi: topic_coherence();

• exclusivity: topic_exclusivity();

• background_distance: Jensen-Shannon distance between the topic's word distribution and the word distribution of the whole corpus, in [0, 1]. A topic close to the corpus distribution (small distance) is a "background" or junk topic made of common words, as in AlSumait et al.'s (2009) ranking of topic significance;

• max_similarity, most_similar: the closest other topic (cosine, topic_similarity()): a high value suggests a split theme;

• top_terms: most probable words.

Usage

topic_diagnostics(
  model,
  documents,
  n_terms = 10L,
  frexw = 0.7,
  n_label = 5L,
  ...
)

Arguments

model

A fitted stm model (or a list with beta$logbeta[[1]] and vocab).

documents

The documents the model was fitted on (stm format).

n_terms

Integer scalar, number of top words per topic (M in stm).

frexw

FREX weight of exclusivity.

n_label

Number of words in top_terms.

...

Not used: an argument renamed in the development version of mintR gives an error naming its replacement.

Value

A data.table, one row per topic.

References

AlSumait, L., Barbará, D., Gentle, J. and Domeniconi, C. (2009). Topic significance ranking of LDA generative models. Machine Learning and Knowledge Discovery in Databases (ECML PKDD 2009), 67-82.

See also

plot_topic_diagnostics()

Examples

if (FALSE) { # \dontrun{
diagnostics <- topic_diagnostics(fit, stm_data$documents)
plot_topic_diagnostics(diagnostics)
} # }