Send one polite request

The single door through which every request of the package goes. It identifies the collector, checks robots.txt, waits for the turn of the site, retries transient failures with a growing back-off, records the outcome in the log and serves the response cache.

Usage

polite_request(
  x,
  pause = getOption("politecollect.delay", 2),
  path = NULL,
  robots = TRUE,
  max_tries = getOption("politecollect.max_tries", 3L),
  backoff = default_backoff,
  transient = c(429L, 500L, 502L, 503L, 504L),
  timeout = getOption("politecollect.timeout", 300),
  cache = getOption("politecollect.cache", "off"),
  on_failure = c("null", "abort", "response"),
  user_agent = getOption("politecollect.user_agent")
)

Arguments

x

A URL, or an httr2::request() already holding a method, headers or a body (POST queries, API keys). Settings already on the request (user agent, timeout, retries, throttle) are kept.

pause

Numeric. Minimum seconds between two requests to the same site. The crawl delay of robots.txt applies when it is longer. By default the option politecollect.delay, or 2.

path

Character or NULL. File the body is written to.

robots

Logical. Check robots.txt first? FALSE for APIs, which are governed by their terms of use and rate limits.

max_tries

Integer. Attempts for a transient failure. By default the option politecollect.max_tries, or 3.

backoff

Function of the attempt number returning the seconds to wait. By default 30 s, 60 s, 120 s... up to 300 s. A Retry-After header sent by the server takes precedence.

transient

HTTP codes worth retrying, or a function of a response returning TRUE or FALSE.

timeout

Numeric. Seconds before the request is abandoned; a stalled transfer would otherwise block a run for ever. By default the option politecollect.timeout, or 300.

cache

Mode of the response cache: "off", "use" (read it, and store new responses), "refresh" (ask the server, and store) or "only" (never ask the server). By default the option politecollect.cache, or "off". Needs the option politecollect.cache_dir; ignored for downloads to path.

on_failure

What to return for a failure: "null" (NULL, the reason being available from last_failure()), "abort" (an error of class politecollect_request_failed) or "response" (the response even with an error status; NULL if the server did not answer).

user_agent

Character. By default the option politecollect.user_agent, which must be set.

Value

An httr2_response, or what on_failure asks for.

Details

Pace. The wait comes before a request and is counted per site, whereas the former polite_get() of central_bank_database slept after every request. The time spent parsing a page therefore counts in the delay, and a collector alternating two sites waits less. A site never receives two requests less than pause seconds apart from one session.

Retries. With the defaults, HTTP 429 and 5xx, timeouts and network errors are tried three times, for at most about 90 seconds of waiting. Longer back-offs cost 15 minutes per unreachable URL on long runs (seen in central_bank_database). Pass a larger max_tries for an endpoint known for long outages, as sparql_query() does.

Failures are recorded with the wording that failure_kind() recognises: "HTTP 404", or the message of the network error.

One request at a time: running several collectors at once on one site defeats the delay. In test mode, a session stops after a few requests (see request_count()).

See also

Other politeness: check_robots(), last_failure(), request_count()

Examples

if (FALSE) { # \dontrun{
options(politecollect.user_agent = "my_project (me@example.org)")
resp <- polite_request("https://www.bis.org/speeches/index.htm")
if (is.null(resp)) last_failure()
} # }