Skip to contents

Search and download from the Internet Archive

Source: R/internet_archive.R
ia_search.Rd

  • ia_search() lists items with the scraping API, which is meant for walking through long result sets;

  • ia_metadata() reads the metadata of an item;

  • ia_files() lists its files;

  • ia_is_restricted() says whether it is a lending-only item;

  • ia_download() downloads one file and checks its MD5.

Usage

ia_search(
  q,
  fields = c("identifier", "title", "creator", "date"),
  sorts = NULL,
  count = 1000L,
  pause = 1,
  max_pages = Inf,
  on_page = NULL
)

ia_metadata(identifier, ...)

ia_files(metadata)

ia_is_restricted(metadata)

ia_download(identifier, file, dest, metadata = NULL, ...)

Arguments

q

Character. Query, in the syntax of the advanced search, e.g. "collection:toronto AND mediatype:texts".

fields

Character vector of metadata fields.

sorts

Character or NULL. Fields to sort by, separated by commas; identifier, if present, must be last.

count

Integer. Results per page (at least 100).

pause

Numeric. Minimum seconds between two requests.

max_pages

Numeric. Maximum number of pages.

on_page

Function called with each table of results and its page number, or NULL. See api_paginate().

identifier

Character. Identifier of the item.

...

Other arguments of polite_request().

metadata

List returned by ia_metadata().

file

Character. Name of the file in the item.

dest

Character. Destination path.

Value

ia_search(): the list returned by api_paginate(), items being a data.table of character columns, several values joined by "; ". ia_metadata(): a list, or NULL if the item does not exist or the request failed. ia_files(): a data.table with name, format, size, md5, source and private. ia_is_restricted(): TRUE or FALSE. ia_download(): TRUE if the file was saved and matches its MD5, FALSE otherwise.

Details

ia_download() refuses, with an error of class politecollect_ia_restricted, the items reserved for controlled digital lending (metadata access-restricted-item) and the private files of an item: the package only fetches what the Archive serves to everyone. Whether a text may be reused is a separate question, which depends on its copyright status and on the terms of the Archive.

The Archive warns that a long scrape may not be complete, since items can be added or deleted in the meantime: status can be "incomplete" for that reason alone.

References

Internet Archive, About search. https://archive.org/help/aboutsearch.htm

See also

Other APIs: api_paginate(), api_request(), crossref_request(), openalex_api_key(), scopus_api_key(), sparql_query()

Examples

if (FALSE) { # \dontrun{
options(politecollect.user_agent = "my_project (me@example.org)")
books <- ia_search("collection:toronto AND title:economics", max_pages = 1)
meta <- ia_metadata(books$items$identifier[1])
ia_is_restricted(meta)
files <- ia_files(meta)
ia_download(books$items$identifier[1], files[format == "Text PDF", name][1], "book.pdf")
} # }