Skip to contents

coRpusdb

Build, document and evaluate text databases for social science research.

coRpusdb is an R package of shared building blocks for research projects that collect texts into a DuckDB database (journal articles, central bank and EU documents, digitised archives), and for the projects that read these databases. It was extracted from functions that the author’s database projects kept rewriting. It also writes into code a few principles of how such databases should be built:

  • Data are taken, not given. Every record says where it comes from (doc_id = "<source>:<source_id>"), raw records are kept as the source returned them, and every write is logged.
  • Raw is kept, derived is versioned. Raw tables are only appended to; cleaned and derived outputs carry a stamp of the method and version that produced them, and stale outputs are refused.
  • Writes are transactions that replace what they rewrite, so that a failed run changes nothing and a rerun creates no duplicates.
  • One writer, many readers. Readers open the database read-only and fall back on a snapshot while a collection runs.
  • The schema documents itself, and is checked against the database and the README.
  • Uncertain steps are measured: stratified samples, blind judge files, Cohen’s kappa, precision and recall with Wilson intervals.
  • Memory is a constraint, not an afterthought. DuckDB’s memory is capped, large results are streamed, checks and samples run in SQL, and the functions are benchmarked on tables of 10 million texts (bench/benchmark_memory.R).

These principles draw on the literature on corpora and quantitative sources, in particular Lemercier and Zalc’s Quantitative Methods in the Humanities (2019), Grimmer, Roberts and Stewart’s Text as Data (2022), McEnery and Hardie’s Corpus Linguistics (2012), Salganik’s Bit by Bit (2018) and Guldi’s The Dangerous Art of Text Mining (2023). The article Building a text database as a research practice explains how.

Installation 

# install.packages("pak")
pak::pak("git::https://codeberg.org/agoutsmedt/coRpusdb.git")
# or
remotes::install_git("https://codeberg.org/agoutsmedt/coRpusdb.git")

What it contains

Family Functions
Connections resolve_db_path(), db_connect(), db_connect_read_only(), db_disconnect(), db_snapshot_path(), db_refresh_snapshot(), db_remove_snapshot()
Queries db_query(), db_scalar(), db_quote(), db_tables()
Schema create_schema(), add_missing_columns(), schema_dictionary(), schema_markdown(), check_schema(), check_documented_columns()
Writing check_unique(), replace_rows(), replace_documents(), append_raw_records()
Large tables and memory free_memory_gb(), check_free_memory(), db_check_unique(), db_query_chunks(), draw_stratified_sample_db()
Fingerprints and sharing db_fingerprint(), compare_fingerprints(), export_parquet(), import_parquet()
Versions write_stamp(), read_stamp(), check_stamp(), content_hash()
Files and logs write_gz_text(), read_gz_text(), dated_file_name(), file_date(), latest_dated_file(), backup_files(), compare_with_backup(), free_disk_gb(), check_free_disk(), pipeline_log(), pipeline_status()
Test mode is_test_mode(), output_path(), input_file(), cap_items()
Identifiers and normalisation make_doc_id(), split_doc_id(), hash_id(), pad_id(), make_slug(), normalise_for_matching(), escape_regex(), split_author_string(), na_if_sentinel(), normalise_sentinels(), collapse_or_na(), json_list_to_string()
Evaluation margin_of_error(), sample_size_for_margin(), draw_stratified_sample(), draw_stratified_sample_db(), write_judge_chunks(), read_verdicts(), cohen_kappa(), wilson_interval(), precision_recall()
Background jobs run_in_background(), background_jobs(), stop_background_job()

A short example 

library(coRpusdb)

schema <- list(document = "
  CREATE TABLE IF NOT EXISTS document (
    doc_id VARCHAR,  -- <source>:<source_id>
    title  VARCHAR,  -- title as published
    year   INTEGER   -- year of publication
  )")

con <- db_connect("speeches.duckdb")
create_schema(con, schema)
replace_documents(
  con,
  list(document = data.frame(doc_id = make_doc_id("ecb", "sp240101"),
                             title = "The euro at 25", year = 2024L)),
  source = "ecb", script = "collecting_ecb.R"
)
cat(schema_markdown(schema_dictionary(schema)))   # data dictionary for the README
db_disconnect(con)

# Downstream projects read without holding the lock:
db_query("speeches.duckdb", "SELECT * FROM document WHERE year = ?", params = list(2024L))

Articles

coRpusdb stores, documents and evaluates. Collecting texts politely belongs to a companion scraping package (in preparation), and analysing them to mintR (text mining and structural topic models).

Licence

MIT. See LICENSE.md.