Collecting documents as a research practice

A script that collects documents from the web looks like a technical preliminary. It is in fact the first step of the method, and the one that leaves the fewest traces. What was requested, when, what the server answered and what was given up on all shape the corpus that later analyses take for granted. It also has consequences for others: every request is work done by someone else’s server.

politecollect gathers the functions that the author’s collection projects kept rewriting (central bank and EU documents, Crossref, Scopus). It turns into code a few principles of how a collection should be run. This article states them, says where they come from, and what the package leaves to the researcher. The two other articles show the functions at work: From a list of pages to a resumable collection and Scholarly APIs. What happens after the collection (storing, documenting, measuring quality) is the subject of the companion package coRpusdb.

A collection is a request to someone else’s server

Automated collection “exerts a substantial bandwidth toll on target sites” (Freelon 2018). Overloading a server is also the first way in which a collection can do damage, and the basis on which it can be held legally liable, under the theory of trespass to chattels (Krotov et al. 2020, sec. 4.4; Krotov and Johnson 2023, sec. 5.4). Politeness is therefore the ordinary condition for using a resource that belongs to someone else, not an optional courtesy.

In the package, every request goes through one function, polite_request():

• it says who is collecting: no request leaves without a user agent, which should hold a contact address, so that the administrator of a site can write to you rather than block you;
• it sends one request at a time and spaces them per site (2 seconds by default, or the crawl delay of the site when it is longer);
• it backs off when a server struggles: HTTP 429 and 5xx responses, timeouts and network errors are tried again after a growing wait, and a Retry-After header sent by the server takes precedence;
• a queue stops by itself after ten transient or blocked failures in a row (queue_downloads()), since insisting on a site that refuses you or has fallen over is both useless and impolite.

The settings are read from options set once in a project’s configuration, and the package never raises the pace by itself.

Robots.txt is a norm, not a law

“A web user is not required by any legal or technical restrictions to follow the instructions provided by the robots.txt protocol. Still, social norms generally dictate that web scrapers follow these instructions” (Krotov et al. 2020, sec. 5.1). The same authors note that site owners can have legitimate reasons to prohibit automated access, and that courts have considered these files when deciding whether scraped data could be used (sec. 4.3).

check_robots() applies the file before every request to a website. A disallowed path stops the run instead of being skipped silently: it is a decision for the researcher, not for a loop. A collector that prefers to record the refusal and go on can catch the condition, as queue_downloads() does.

The standard, RFC 9309, distinguishes a missing file (a 4xx answer: the crawler may access the site) from an unreachable one (a 5xx answer or a network error: the crawler “MUST assume complete disallow”) (Koster et al. 2022, sec. 2.3.1.3 and 2.3.1.4). The package is more lenient by default, and warns: a temporary error on one file would otherwise stop a collection of several days. The option politecollect.robots_unreachable = "disallow" follows the standard to the letter.

robots.txt says nothing about what may be done with the documents. It is not a substitute for reading the terms of use of a source, nor do these settle the ethical question. “Do not confuse TOS compliance with human subjects compliance or privacy protection” (Freelon 2018). Salganik (2018, sec. 6.4.4) makes the same distinction, treating law and contracts as one consideration among others. A website may be publicly accessible and still hold personal data that its authors did not expect to see in a corpus (Krotov et al. 2020, sec. 5.2). No function can decide this.

Prefer the door that is offered

“Other things being equal”, write Freelon’s recommendations, researchers should “exhaust all possible means of procuring your data via officially-sanctioned means” before scraping a site that prohibits it. The conventions of the author’s projects set the same order: first an API or a published dataset, then static pages, and an interactive browser only when nothing else works. The package has no browser automation. APIs have their own rules (keys, quotas, daily budgets): the package reads what the source announces, and stops when a budget is spent instead of insisting (see Scholarly APIs).

Freelon also warns that access may vanish: “APIs are easy to use and TOS-compliant but may vanish without warning”. It has happened to these clients while they were being written. The OpenAlex documentation consulted in November 2026 no longer mentions the mailto parameter that some of the author’s scripts send, and describes a daily budget instead. A collection should say which access conditions it ran under, at which date.

What did not come back is part of the data

Salganik (2018, secs. 2.3.4, 2.3.7, 2.3.9) lists among the problems of found data that it is incomplete, that it drifts, and that it is dirty. A collection adds its own holes: pages gone, links that lead elsewhere, files refused. They are rarely random. An institution that reorganised its website loses a period; a firewall that rejects a range of requests loses a series.

The package therefore keeps a reason for every failure (last_failure()) and sorts them (failure_kind()):

Kind	Examples	What to do
permanent	HTTP 404, 410, a page instead of a PDF, robots.txt	Record it, do not ask again
blocked	HTTP 403, a firewall page	Do not insist; look at why
transient	Timeout, HTTP 429 and 5xx	Try again later
unknown	Anything else	Try again, and read the reason

library(politecollect)
failure_kind(NA, c("HTTP 404", "HTTP 403", "HTTP 503", "something new"))

[1] "permanent" "blocked"   "transient" "unknown"  

With the option politecollect.log_file, every request is also written to a log (read_request_log(), request_log_summary()): what was asked, when, with which result. It answers two questions that come up late in a project: what exactly is the provenance of this collection, and was the server treated properly?

The servers also say things that are not true. A status 200 can carry an error page, which is why download_pdf() checks the signature of the file and polite_get() knows firewall pages. In one of the author’s projects, HTML pages saved under a .pdf extension would otherwise have passed for documents without text.

A collection takes time, so it must survive

The web changes while it is being collected: “by the time the web scraping task is completed, the data obtained is already somewhat inaccurate and outdated”, and a tool built today will work “only temporarily” (Krotov and Johnson 2023, sec. 3.3). A collection is rarely finished once. The package treats this as a normal condition:

• files are written under a temporary name and moved into place only once checked, so that an interrupted run leaves no truncated file that a later run would take for complete;
• queue_downloads() skips what is held, and tries again only what can still succeed;
• with cache = "use", a parser can be rerun as often as needed without asking the server again, which is both more reproducible and less demanding for it;
• in test mode a script cannot send more than a few requests (options(politecollect.test_mode = TRUE)).

What the package does not do

politecollect does not decide what should be collected, whether a source may be used, or whether a corpus is representative of anything. It does not bypass restrictions: there are no rotating addresses or disguised browsers, and ia_download() refuses the items that the Internet Archive reserves for lending. It sees only one session: several collectors running at once on one site defeat its delay. And it does not store: once the documents are collected, see coRpusdb.

References

Freelon, Deen. 2018. “Computational Research in the Post-API Age.” Political Communication 35 (4): 665–68. https://doi.org/10.1080/10584609.2018.1477506.

Koster, Martijn, Gary Illyes, Henner Zeller, and Lizzi Sassman. 2022. Robots Exclusion Protocol. RFC 9309, Internet Engineering Task Force. https://www.rfc-editor.org/rfc/rfc9309.html.

Krotov, Vlad, and Leigh Johnson. 2023. “Big Web Data: Challenges Related to Data, Technology, Legality, and Ethics.” Business Horizons 66 (4): 481–91.

Krotov, Vlad, Leigh Johnson, and Leiser Silva. 2020. “Tutorial: Legality and Ethics of Web Scraping.” Communications of the Association for Information Systems 47. https://doi.org/10.17705/1CAIS.04724.

Salganik, Matthew J. 2018. Bit by Bit: Social Research in the Digital Age. Princeton University Press.