Skip to contents

politecollect: polite and traceable collection of web and API sources

Source: R/politecollect-package.R
politecollect-package.Rd

Shared building blocks for the research projects that collect documents and metadata from websites and scholarly APIs. The functions encode a few principles (see the article Collecting documents as a research practice):

  • One door for every request. polite_request() identifies the collector, reads robots.txt, spaces requests per site and backs off on transient failures. Every other function goes through it.

  • A failure has a reason. last_failure() says why a request gave nothing, and failure_kind() whether trying again makes sense.

  • A file is checked before it counts. download_file() and download_pdf() write to a temporary name and check the content, since a server can answer 200 with an error page.

  • A collection can stop and resume. queue_downloads() skips what is held, and retries only what can still succeed.

  • Collection leaves a trace. request_log_summary() reports on the log of every request, and the response cache lets a parser be rerun without asking the server again.

  • Test runs are capped. With the test mode on, a script cannot send more than a few requests.

Politeness settings are read from options set once in a project's paths_and_packages.R. The package never raises the pace by itself.

Options

  • politecollect.user_agent: identification of the collector, with a contact address. Required: no request is sent without it.

  • politecollect.delay: seconds between two requests to one site (default 2; the crawl delay of robots.txt applies when it is longer).

  • politecollect.timeout: seconds before a request is abandoned (default 300).

  • politecollect.max_tries: attempts per request (default 3).

  • politecollect.robots_unreachable: what to do when robots.txt cannot be read because of a server or network error: "allow" (default, with a warning) or "disallow" (what RFC 9309 prescribes).

  • politecollect.cache_dir, politecollect.cache: folder and mode of the response cache ("off", "use", "refresh" or "only").

  • politecollect.log_file: CSV file receiving one row per request.

  • politecollect.test_mode, politecollect.max_requests: cap on the number of requests sent by the session (10 in test mode).

See also

Useful links:

Author

Maintainer: Aurélien Goutsmedt agoutsmedt@hotmail.fr

Authors: