From a list of pages to a resumable collection • politecollect

This article builds a collector step by step: settings, reading a page, extracting its text, downloading its files, and resuming after an interruption. Sites are not contacted when this page is built: the code that needs a server is shown without being run, and the rest uses small examples.

library(politecollect)

Say who you are, once

Settings go in the configuration file of the project, so that a script cannot change the pace by accident.

options(
  politecollect.user_agent = "my_project (https://example.org/project; me@example.org)",
  politecollect.delay = 5,                                   # seconds between requests to a site
  politecollect.log_file = "~/data/my_project/logs/requests.csv",
  politecollect.cache_dir = "~/data/my_project/cache"
)

Without a user agent, no request is sent. Choosing the delay is a judgement about the source: a large institution can bear a request every two seconds, a small server perhaps not. Five seconds is what the author’s projects use for the European Central Bank.

Read a page

polite_get() returns the content parsed, or NULL for a failure, so that one missing page does not stop a long run. last_failure() says why.

page <- polite_get("https://example.org/speeches/2024-01.html")
if (is.null(page)) last_failure()
#> [1] "HTTP 404"

A path that robots.txt disallows is an error rather than a failure: the run stops and you decide. To record the refusal and go on:

page <- tryCatch(
  polite_get(url),
  politecollect_robots_disallowed = function(e) NULL
)

Other formats are available with as = "json", "text" or "file". When a link expected to be a page turns out to be a PDF (DOI redirects often do this), the reason is "response is a PDF", and the caller can download it instead:

page <- polite_get(url)
if (is.null(page) && identical(last_failure(), "response is a PDF")) {
  download_pdf(url, "documents/this_one.pdf")
}

Extract the text

When the structure of the site is known, html_paragraphs() takes a selector for the main content and returns the paragraphs with the heading they fall under.

page <- xml2::read_html("
  <main>
    <h2>Monetary policy</h2>
    <p>Inflation has declined.</p>
    <p class='share'>Share this page</p>
    <h2>Outlook</h2>
    <ul><li>Risks remain.</li></ul>
  </main>")
html_paragraphs(page, "main", exclude = ".share")

   paragraph         section                    text
       <int>          <char>                  <char>
1:         1 Monetary policy Inflation has declined.
2:         2         Outlook           Risks remain.

html_text_or_na() reads one field, and gives NA when it is absent, so that a missing field stays one missing value in a table:

page <- xml2::read_html("<h1 class='title'>A speech</h1>")
html_text_or_na(page, "h1.title")

[1] "A speech"

html_text_or_na(page, ".author")

[1] NA

Without a known selector, read_html_blocks() reads the blocks of a saved file. When a document exists both as a page and as a PDF, prefer_html_text() tells whether the page holds the full text or only a summary.

Selectors break when a site is redesigned. html_paragraphs() returning an empty table for many pages at once is the usual symptom, so count these in the log of your collector rather than discovering them at the analysis stage.

Download files

download_pdf() saves a file only if it really is a PDF, and returns TRUE or FALSE. For another format, download_file() takes any check:

download_file(url, "data/table.csv", check = function(path) file.size(path) > 100)

A queue that can stop and resume

For a long list of files, queue_downloads() handles what a loop would get wrong:

files <- data.frame(
  url = pdf_urls,
  dest = file.path("~/data/my_project/pdf", paste0(ids, ".pdf"))
)
result <- queue_downloads(files, state_file = "~/data/my_project/logs/pdf_queue.csv")
result[, .N, by = .(outcome, kind)]

Run it again whenever you like. Files on disk are skipped, each attempt has been written to the state file, and only what can still succeed is tried:

Last outcome	Next run
File on disk	`exists`: skipped
`permanent` or `blocked` failure	`skipped`, with the reason kept
`transient` or `unknown`, fewer than `max_attempts`	Tried again
`transient` or `unknown`, `max_attempts` reached	`skipped`

After ten transient or blocked failures in a row the queue stops and warns. Long queues should run as detached jobs, which coRpusdb::run_in_background() provides.

Test before running

A collector under development should not be able to run a whole collection:

options(politecollect.test_mode = TRUE)  # at most 10 requests in this session

The cache means a parser can then be developed on the same few pages without asking for them again:

options(politecollect.cache = "use")    # first run: ask and store
options(politecollect.cache = "only")   # afterwards: never ask the server

Report on the collection

request_log_summary()
#>                  domain      kind     n median_duration_s
#> 1: https://www.bis.org        ok  1204             0.410
#> 2: https://www.bis.org permanent    37             0.312
#> 3: https://www.bis.org transient     4            30.050

The numbers above are illustrative. The log belongs with the documentation of the corpus: it is the record of what was asked and of what did not come back.