Skip to contents

Pool short texts into longer documents

Source: R/stm_helpers.R
pool_texts.Rd

Topic models need documents long enough for several words of a topic to co-occur: a post of 13 words is too short, and its topics then reflect vocabulary rather than themes. Pooling short texts that share a context into one document is the usual remedy: Mehrotra et al. (2013) compared pooling tweets by author, by trending term, by time and by hashtag, and found much more coherent topics than with single tweets, hashtags giving the best ones. pool_texts() joins the texts of each group, in order, and reports how long the documents become.

Which groups? The texts of a group should be about the same subject:

  • social media: an author's posts in the same thread (self-threads, and an author's replies in one conversation; the choice of socio_atproto), or by author and day; author-week documents are long enough but mix unrelated subjects;

  • debates and hearings: a speaker's interventions in one sitting or on one agenda item;

  • comments: the comments of one article, or of one author on one article;

  • paragraphs or pages back into documents (the inverse of chunk_texts()), or abstracts by journal and year when only the aggregate matters.

Pooling changes the unit of the analysis: topic proportions are then those of the pooled documents, and covariates must be defined at that level (an author's type, the first date of the thread).

Usage

pool_texts(
  data,
  by,
  text_col = "text",
  order_col = NULL,
  id_col = NULL,
  document_col = "document",
  sep = "\n",
  keep_cols = NULL,
  verbose = TRUE,
  ...
)

Arguments

data

Table with the texts and the grouping columns.

by

Character vector of grouping columns (e.g. c("author", "thread")).

text_col

Character scalar, the text column.

order_col

Optional column ordering the texts within a group (e.g. the date); its first and last values are returned as start and end.

id_col

Optional column identifying the texts: the first one of each group is returned as first_<id_col>.

document_col

Character scalar, name of the identifier of the pooled documents created (d000001, ...).

sep

Character scalar placed between texts (a line break by default, so that chunk_texts() sees the texts as paragraphs).

keep_cols

Other columns to keep, constant within a group (the first value is kept).

verbose

Logical: report the number and the length of the documents.

...

Not used: an argument renamed in the development version of mintR gives an error naming its replacement.

Value

A data.table with document_col, the by columns, keep_cols, text, n_texts, n_words, and start, end (with order_col) and first_<id_col> (with id_col).

References

Mehrotra, R., Sanner, S., Buntine, W. and Xie, L. (2013). Improving LDA topic models for microblogs via tweet pooling and automatic labeling. Proceedings of SIGIR 2013, 889-892. doi:10.1145/2484028.2484166

Examples

posts <- data.frame(
  author = c("a", "a", "b"), thread = c("t1", "t1", "t2"),
  date = as.Date(c("2026-01-02", "2026-01-01", "2026-01-03")),
  text = c("second post", "first post", "another one")
)
pool_texts(posts, by = c("author", "thread"), order_col = "date")
#> ℹ Pooled 3 texts into 2 documents by author and thread: median 1.5 text and 3 words per document (median text: 2 words).
#>    document author thread                    text n_texts n_words      start
#>      <char> <char> <char>                  <char>   <int>   <int>     <Date>
#> 1:  d000001      a     t1 first post\nsecond post       2       4 2026-01-01
#> 2:  d000002      b     t2             another one       1       2 2026-01-03
#>           end
#>        <Date>
#> 1: 2026-01-02
#> 2: 2026-01-03