stm::stm() takes a list of documents (each a 2-row integer
matrix: indices in the vocabulary and counts), the vocabulary, and a
metadata table in the order of the documents. Building them from one
table and matching the metadata by identifier avoids silent
misalignments between documents and covariates.
Usage
tokens_to_stm(
tokens,
meta = NULL,
document_col = "document",
token_col = "token",
count_col = "count",
...
)Arguments
- tokens
Long table with one row per document and token (e.g. the output of
extract_ngrams()or ofcompute_tf_idf()after filtering). Repeated document-token rows are summed.- meta
Optional metadata, one row per document, with
document_col.- document_col, token_col, count_col
Column names of the document identifier (in both tables), the token and its count. Counts must be whole numbers.
- ...
Not used: an argument renamed in the development version of mintR gives an error naming its replacement.
Value
A list of class stm_input with documents (named by document
identifier, sorted), vocab (sorted) and, when meta is given, meta
(a data.frame in the order of documents). Documents without metadata
are an error; metadata rows without tokens are dropped, with a message.
Rows with a missing token or count, or a count of 0, are dropped with a
message. Printing it summarises the corpus.
Details
Faster than quanteda::convert(dfm, to = "stm") on large
corpora since no sparse matrix is built: tokens are sorted once, and the
documents are cut from two integer vectors.
Examples
tokens <- data.frame(
id = c("a", "a", "b"), token = c("wage", "labour", "wage"), count = c(2, 1, 3)
)
meta <- data.frame(id = c("b", "a"), year = c(2001, 2000))
tokens_to_stm(tokens, meta, document_col = "id")
#> <stm_input>: 2 documents, 2 terms, 6 tokens.
#> • Tokens per document: median 3 (min 3, max 3); 2 documents with fewer than 10.
#> • Metadata: id and year.