Skip to contents

Classify a failed request: is a retry worth it?

Source: R/failure.R
failure_kind.Rd

Sorts failures into four kinds, from the reasons recorded by last_failure(). Vectorised over status and reason.

  • permanent: the page or file is not there (HTTP 404, 410, 451, 401, a link returning a web page instead of a PDF), or robots.txt forbids it. A retry gives the same answer.

  • blocked: HTTP 403 or a firewall page: the site refuses the collection. Retrying does not help and insisting is impolite.

  • transient: the server did not answer properly (timeouts, refused or reset connections, DNS errors, HTTP 408, 429, 5xx, empty body). The same request can succeed later.

  • unknown: no reason recorded, or one not listed here. To be treated as transient.

Usage

failure_kind(status, reason)

Arguments

status

Character. Status recorded by the collector, if any. "robots_disallowed" and "not_available" count as permanent; other values and NA are ignored.

reason

Character. Reason of the failure.

Value

A character vector.

See also

Other requests: normalise_url(), polite_get(), read_request_log()

Examples

failure_kind(NA, c("HTTP 404", "HTTP 403", "HTTP 503", NA))
#> [1] "permanent" "blocked"   "transient" "unknown"