politecollect: collecting sources as research • politecollect

Polite and traceable collection of web and API sources for research.

politecollect is an R package of shared building blocks for research projects that collect documents and metadata from websites (central banks, EU institutions, journals) and from scholarly APIs (Crossref, Scopus, OpenAlex, the Internet Archive, SPARQL endpoints). It was extracted from functions that the author’s projects kept rewriting. It also writes into code a few principles of how a collection should be run:

One door for every request. A single function identifies the collector, reads robots.txt, spaces the requests to each site and backs off when a server struggles.
A failure has a reason. It is kept, and sorted into permanent, blocked or transient, so that a later run retries only what can still succeed.
A file is checked before it counts. Servers answer 200 with error pages: downloads are written under a temporary name and verified first.
A collection can stop and resume, and stops by itself when a site refuses it or has fallen over.
Collection leaves a trace: a log of every request, and a cache that lets a parser be rerun without asking the server again.
Test runs are capped, so that a script under test cannot run a whole collection.

These practices draw on the literature on web data in the social sciences, in particular Salganik’s Bit by Bit (2018), Krotov, Johnson and Silva (2020) and Freelon (2018). The article Collecting documents as a research practice explains how.

Installation

# install.packages("pak")
pak::pak("git::https://codeberg.org/agoutsmedt/politecollect.git")
# or
remotes::install_git("https://codeberg.org/agoutsmedt/politecollect.git")

What it contains

Family	Functions
Politeness	`polite_request()`, `check_robots()`, `crawl_delay()`, `clear_robots_cache()`, `last_failure()`, `request_count()`, `reset_request_count()`
Requests	`polite_get()`, `failure_kind()`, `normalise_url()`, `read_request_log()`, `request_log_summary()`
Downloads	`download_file()`, `download_pdf()`, `is_pdf_file()`, `queue_downloads()`
HTML to text	`html_text_or_na()`, `html_paragraphs()`, `read_html_blocks()`, `prefer_html_text()`
APIs	`api_request()`, `api_paginate()`, `sparql_query()`, `sparql_values()`
Crossref	`crossref_mailto()`, `crossref_request()`, `crossref_client_limits()`, `crossref_doi_path()`, `crossref_search()`
Scopus	`scopus_api_key()`, `scopus_request()`, `scopus_get_page()`, `scopus_search()`, `scopus_quota()`
OpenAlex	`openalex_api_key()`, `openalex_request()`, `openalex_search()`, `openalex_rate_limit()`
Internet Archive	`ia_search()`, `ia_metadata()`, `ia_files()`, `ia_is_restricted()`, `ia_download()`

A short example

library(politecollect)

# Once, in the project's paths_and_packages.R
options(
  politecollect.user_agent = "my_project (https://example.org; me@example.org)",
  politecollect.delay = 5,
  politecollect.log_file = "~/data/my_project/logs/requests.csv"
)

page <- polite_get("https://www.bis.org/speeches/index.htm")
if (is.null(page)) last_failure()

# A resumable queue of PDFs: stop it, start it again, it carries on
files <- data.frame(url = pdf_urls, dest = file.path("pdf", basename(pdf_urls)))
result <- queue_downloads(files, state_file = "logs/pdf_queue.csv")

request_log_summary()   # requests and failures per site

Options

Option	Default	Role
`politecollect.user_agent`	required	Who is collecting, with a contact address
`politecollect.delay`	2	Seconds between two requests to one site
`politecollect.timeout`	300	Seconds before a request is abandoned
`politecollect.max_tries`	3	Attempts per request
`politecollect.robots_unreachable`	`"allow"`	Unreadable robots.txt: `"allow"` or `"disallow"`
`politecollect.cache`, `politecollect.cache_dir`	`"off"`, none	Response cache: `"off"`, `"use"`, `"refresh"`, `"only"`
`politecollect.log_file`	none	CSV with one row per request
`politecollect.test_mode`, `politecollect.max_requests`	off, 10	Cap on the requests of a session

Keys are read from ~/.Renviron (ELSEVIER_API_KEY, CROSSREF_MAILTO, OPENALEX_API_KEY) and kept out of URLs and logs.

Articles

Collecting documents as a research practice: the principles, where they come from, and what the package leaves to the researcher.
From a list of pages to a resumable collection: a worked example of a collector.
Scholarly APIs: Crossref, Scopus, OpenAlex, the Internet Archive and SPARQL, their limits and how the package treats them.

politecollect collects. coRpusdb stores, documents and evaluates what is collected, and mintR analyses it (text mining and structural topic models). The packages do not depend on one another.

Licence

MIT. See LICENSE.md.

politecollect