Skip to contents

Check that robots.txt allows a URL

Source: R/robots.R
check_robots.Rd

Reads the robots.txt of a site once per session and per day, and stops if the URL is disallowed. polite_request() calls it before every request to a website.

  • crawl_delay() returns the delay asked by the site, or NA;

  • clear_robots_cache() forgets the files read in this session.

Usage

check_robots(
  url,
  user_agent = getOption("politecollect.user_agent"),
  on_unreachable = getOption("politecollect.robots_unreachable", "allow")
)

crawl_delay(url, user_agent = getOption("politecollect.user_agent"))

clear_robots_cache()

Arguments

url

Character. The URL to request.

user_agent

Character. Identification sent when fetching robots.txt (some servers reject generic agents). By default the option politecollect.user_agent.

on_unreachable

What to do when robots.txt cannot be read because of a server or network error: "allow" (with a warning) or "disallow". By default the option politecollect.robots_unreachable, or "allow".

Value

check_robots(): TRUE, invisibly; aborts with an error of class politecollect_robots_disallowed otherwise. crawl_delay(): a number of seconds, or NA if the site asks for none. clear_robots_cache(): NULL, invisibly.

Details

A disallowed path stops the run rather than being skipped silently: it is a decision for the researcher, not for a loop. A collector that prefers to record the refusal and go on can catch the condition with tryCatch().

The file is fetched with httr2 and parsed from its text, because robotstxt::robotstxt(domain = ) failed on some redirects (central_bank_database, bancaditalia.it).

RFC 9309 distinguishes two cases. A 4xx answer means that the site has no rules. A 5xx answer or a network error makes the file "unreachable", and the crawler "MUST assume complete disallow". The default here is more lenient ("allow", with a warning), as it was in the projects: a temporary error on robots.txt would otherwise stop a long run. The pace of the collection and the back-off still apply. Set on_unreachable = "disallow" to follow the RFC strictly.

Rules are read for the group *, since the descriptive agent of a research collector matches no named group. Crawl-delay is not part of RFC 9309 but is honoured when a site declares it.

References

Koster, M., Illyes, G., Zeller, H. and Sassman, L. (2022). Robots Exclusion Protocol. RFC 9309. https://www.rfc-editor.org/rfc/rfc9309.html

See also

Other politeness: last_failure(), polite_request(), request_count()

Examples

if (FALSE) { # \dontrun{
options(politecollect.user_agent = "my_project (me@example.org)")
check_robots("https://www.bis.org/speeches/index.htm")
crawl_delay("https://www.bis.org/speeches/index.htm")
} # }