Building a text database as a research practice
Source:vignettes/articles/text-database-research-practice.qmd
A text database built for research is not a neutral container. Choosing the sources, deciding what counts as a document, cleaning texts, linking records and deciding what to keep are research decisions. They shape every result drawn from the database, and they are usually the least visible ones. coRpusdb gathers the functions that the author’s database projects kept rewriting (harvesting JSTOR, Scopus, ISTEX, central bank and EU documents). It also turns into code a few principles of how such databases should be built.
This article states these principles and where they come from. For each one, it says what the package does and what it leaves to the researcher. The other two articles show the functions at work: From raw records to a documented database and Measuring an uncertain step.
Data are taken, not given
“Data” means “given”, but even digitised data are never simply given. Following Johanna Drucker, Lemercier and Zalc propose to speak of capta, information actively “taken”, and argue that this is what makes data interpretable for the humanities (Lemercier and Zalc 2019, ch. 1). The same argument runs through the essays collected by Gitelman (2013). A text database is a sequence of takings: a source made some documents available, a collector requested some of them, an extraction turned PDFs into text, a cleaning step decided what is body text and what is a running header.
The consequence for the database is provenance. Every record must tell where it comes from, how it was obtained and what was done to it:
- every document has an identifier
<source>:<identifier in the source>(make_doc_id()), derived from something stable in the source; - what the source returned is stored as it came, in raw tables that are only appended to (
append_raw_records()), with the time of collection; - every write to the database is logged: which script, which source, which input, how many rows written and how many replaced (
replace_rows(),replace_documents()).
A corpus is a finite, question-specific construction
“No database can ever be exhaustive and all-purpose”, write Lemercier and Zalc, “if only because its existence depends implicitly or explicitly on the definition of a corpus, that is, a finite collection of sources” (Lemercier and Zalc 2019, ch. 2). Grimmer, Roberts and Stewart start from the same point: the usefulness of a corpus depends on the research question and on the population of interest, and there is no values-free corpus construction, since selecting documents has ethical consequences (Grimmer et al. 2022, ch. 3).
Corpus linguistics distinguishes two ways of building a corpus (McEnery and Hardie 2012, sec. 1.4):
- a sample corpus follows an explicit sampling frame, so that it represents a defined population at a given time;
- a monitor corpus grows over time, adding texts as they appear, and trusts size to correct its skews.
Most of the author’s databases are monitor corpora: collectors add what the sources publish. McEnery and Hardie stress that balance and representativeness are “a matter of degree” rather than properties a corpus has or lacks, and that some corpora grow like monitor corpora while following a sampling frame. A database should therefore say which kind it is, per source, and what its frame is.
Texts are mostly found data: what exists depends on who had the resources and the incentives to write and keep it, not on a research design. Grimmer et al. (2022, ch. 4) list four sources of selection bias:
| Bias | What it means for a text database |
|---|---|
| Resource bias | Texts reflect those who could produce, record and store them (archives, digitisation budgets). |
| Incentive bias | Actors record, hide or destroy texts strategically (transcripts, transparency rules). |
| Medium bias | The medium shapes the text (length limits, what a transcript leaves out). |
| Retrieval bias | Keyword searches, APIs and search engines select documents in ways the researcher does not control. |
Salganik adds features of digital sources that matter as soon as a database covers several years: they are incomplete, nonrepresentative, drifting (their population, uses and systems change over time), algorithmically confounded and dirty (Salganik 2018, sec. 2.3). Guldi insists on the occlusions of archives. Even comprehensive collections such as parliamentary debates miss documents published separately, and “one cannot assume that the data that is easiest to find gives the most complete picture of reality in the past”. Her advice is to have a digital librarian review the datasets and their metadata before the analysis starts (Guldi 2023, ch. 2).
What the package does. Nothing here can be automated: defining the population, the frame and the known biases is the researcher’s work. The package makes room for writing it down where it will be read. The schema documents itself (schema_dictionary(), schema_markdown()), and a database description card that gathers the sampling frame, coverage and known biases of each source is planned (see the to-do list of the package).
Sample when you can, and know the precision of what you measure
Lemercier and Zalc warn against the “vogue for exhaustive databases” and recall that the margin of error of an estimate depends on the size of the sample, not on the share of the population it represents. Checking 1,000 items gives the same precision whatever the size of the population, and “there is almost no reason in historical research to gather more than 1,000 cases” (Lemercier and Zalc 2019, ch. 2). Size does not cure bias either: “a biased source remains biased, even if it yields large quantities of data”.
Exhaustive collection still makes sense for a database meant to serve several projects, which then sample from it. But every check of the database (is this match right? is this text clean?) is a sampling problem:
-
margin_of_error()andsample_size_for_margin()say how many items to check for a given precision; -
draw_stratified_sample()draws seeded samples within strata, keeping the population size of each stratum to weight estimates back; -
cap_items()tests code on a random subset rather than on the first documents, which are rarely typical.
Separate input from categorisation: raw is kept, derived is versioned
The central methodological point of Lemercier and Zalc’s chapter on data is the distinction between two stages (Lemercier and Zalc 2019, ch. 3):
- Input: record the information of the source in rows and columns, “while keeping track of the source, transcribing the information in the language of the source”.
- Categorisation: simplify it into categories that can be counted. This step can be repeated as often as necessary, in several ways.
Confusing the two stages is, for them, responsible for many of the criticisms of quantification. They are “firm advocates of ‘dirty’ datasets”: what other disciplines call cleaning is categorisation, and the heterogeneity of sources is part of their richness.
In a text database, the raw text, raw HTML and raw metadata are the input; cleaned texts, harmonised types, matches and flags are categorisations. Hence the first principle of the author’s database projects: raw is sacred, derived is versioned.
- Raw tables are appended to, never corrected (
append_raw_records()). Thetexttable of a database holds the extracted text as it came; cleaned versions go to another table. - Every derived output records the method and version that produced it and a fingerprint of its inputs (
write_stamp(),content_hash()).check_stamp()refuses an output built by another version, so that a change of rule leads to a recomputation rather than to a silent mix of old and new results.
Keep track of sources and decisions
Lemercier and Zalc’s “ten commandments of inputting data” are practical rules with substantive consequences (Lemercier and Zalc 2019, ch. 3). Several of them translate directly into database design:
| Commandment | In a text database |
|---|---|
| Retain the wording of the source; simplify in a separate column | Raw tables and raw text are kept; derived columns sit next to them (document_type and document_type_source). |
| Retain a link to the source |
doc_id carries the source; url, file and raw tables keep the origin of each record. |
| Input missing data as data | Distinguish “the source does not say” from “not collected yet”; normalise text sentinels only when they mean the same thing (na_if_sentinel()). |
| Retain the date of the information |
collected_at in raw tables; run_at in the ingestion log. |
| Save often, creating new files | Dated outputs (dated_file_name(), latest_dated_file()), backups before an overwrite (backup_files()). |
One commandment needs adapting. Lemercier and Zalc advise changing the identifiers when two individuals turn out to be the same. A database read by other projects cannot do this without breaking their joins, so the identifiers are kept forever and the decision is recorded instead: a duplicate_of column, with who decided, when and why.
They also consider the delegation of input “a risk rather than an opportunity”: consistent categorisation is hard to keep across several people, and the researcher loses the intuitions that edge cases give. The remark applies to language models used as annotators. The evaluation protocol of the package therefore treats a model judge as a detector of problems, never as the reference (see below).
Store efficiently, durably, and for others
Lemercier and Zalc recall that the data of the first wave of quantitative history were often “lost in successive generations of computer formats”, and that even open formats do not ensure durability without funding (Lemercier and Zalc 2019, ch. 1). Large text databases add practical constraints: tens of gigabytes, scripts running for hours, several projects reading the same file.
| Problem | What the package does |
|---|---|
| A writing script locks the database for every reader | Readers open it read-only and fall back on a snapshot that the writer refreshes (db_connect_read_only(), db_refresh_snapshot()). |
| A failed run leaves half-written tables | Writes are transactions that delete and insert together (replace_rows(), replace_documents()). |
| A query exhausts the memory of the machine | Connections cap DuckDB’s memory (db_connect()); queries filter in SQL and register id tables instead of loading data (db_query()). |
| A full disk corrupts writes |
check_free_disk() stops a long run in time. |
| Two copies of a database have silently diverged |
db_fingerprint() and compare_fingerprints() compare them through a small table. |
| A co-author needs the data, or the data must outlive DuckDB |
export_parquet() writes an open, column-oriented copy with its schema and row counts. |
Storing everything is not a neutral choice either. What to keep (full PDFs, page images, every intermediate version) has a cost in disk, time and legal exposure: keep what cannot be recomputed (raw material, manual decisions), and recompute the rest from versioned code.
Ethics: what to collect, what to share
Texts often contain personal information, and public availability does not settle the question of their use. Grimmer, Roberts and Stewart point to Nissenbaum’s “contextual integrity”: what matters is the norms under which the authors expected their texts to be read, not only whether the texts are public (Grimmer et al. 2022, ch. 3). Many texts are also licensed (journal articles, press) and cannot be redistributed.
The package cannot decide for a project, but it avoids making leaks easy:
- evaluation files that quote texts are written outside the repository (
write_judge_chunks()takes a folder in the data directory); -
export_parquet()exports only the tables it is given, and its documentation asks to check the content before sending; - collection belongs to the companion package for polite scraping (rate limits, identification, robots rules), and harvesting rules are stated in each project.
Validate, validate, validate
The fourth principle of Grimmer and Stewart (2013, sec. 2.4) applies to every automatic step of a database, not only to text models: “What should be avoided, then, is the blind use of any method without a validation step.” In the author’s projects, every important and uncertain step (extraction, cleaning, duplicate detection, matching, entity attribution) is measured, in two stages:
- Diagnostic rounds. On seeded, stratified samples with controls, a cheap model judges outputs blind to the pipeline’s answer. The errors are traced back to the step that caused them and the code is changed. These rounds stay private.
- Final measurement. On one frozen sample, the authors and the model label independently. Their agreement is reported (Cohen’s kappa), disagreements are settled by the authors, and precision and recall are computed with confidence intervals. Only this human-validated measurement is published.
The functions write_judge_chunks(), read_verdicts(), cohen_kappa(), wilson_interval() and precision_recall() support both stages; Measuring an uncertain step walks through them.
Pipelines that can be read and rerun
Lee et al. (2020) describe the pipeline as a modular, documented sequence of algorithms. Its transparency lets later researchers see how a result was produced and swap one step for another. Software practice reaches similar rules: record how every result was produced, avoid manual steps, keep versions of what produced what (Sandve et al. 2013; Wilson et al. 2017). These recommendations come from computational science; for texts in the social sciences, they serve the older requirement of source criticism: a reader should be able to know how the evidence was produced.
In the package, this means:
- a log of every step (
pipeline_log(),pipeline_status()) that can be read while a script holds the database; - a test mode that never touches the real outputs (
is_test_mode(),output_path(),input_file()); - long steps run as detached jobs with their own log (
run_in_background()); - a schema compared with the database and with its documentation (
check_schema(),check_documented_columns()).
What the package does not do
coRpusdb does not decide what a corpus should contain, which biases are acceptable, or when a step is good enough. It does not collect texts (see the companion scraping package) and it does not analyse them (see mintR for text mining and topic models). Its aim is narrower: once these decisions are taken, record them, apply them consistently across projects, and make their consequences measurable.