Walk through the pages of an API

Fetches the pages of a result set by cursor until there are no more. Each client of the package gives the three functions that describe its API.

Usage

api_paginate(
  fetch,
  items,
  next_cursor,
  total = NULL,
  start = "*",
  max_pages = Inf,
  on_page = NULL
)

Arguments

fetch

Function of a cursor returning the parsed page, or NULL if the page could not be obtained.

items

Function of a parsed page returning its results.

next_cursor

Function of a parsed page returning the next cursor, or NULL at the end.

total

Function of a parsed page returning the number of results, or NULL.

start

Value of the first cursor; NULL for none.

max_pages

Numeric. Maximum number of pages.

on_page

Function called with each list of results and its page number, or NULL. Writing each page to disk there makes a long harvest safe from a failure, and keeps large results out of memory.

Value

A list: items (results of all pages, unless on_page is given, which then receives them instead), n_total (number announced by the source, NA if unknown), n_fetched, n_pages and status ("complete", "incomplete" if the announced total was not reached or a page failed, or "limited" if max_pages stopped the walk).

See also

Other APIs: api_request(), crossref_request(), ia_search(), openalex_api_key(), scopus_api_key(), sparql_query()

Examples

pages <- list(
  list(rows = list(1, 2), nxt = "b"),
  list(rows = list(3), nxt = NULL)
)
api_paginate(
  fetch = function(cursor) pages[[if (identical(cursor, "a")) 1 else 2]],
  items = function(page) page$rows,
  next_cursor = function(page) page$nxt,
  start = "a"
)$n_fetched
#> [1] 3