How good is the parsing? • anystyleR

Before using parsed references as data, you need to know how often each field is right, and whether errors are random or systematic: a parser that drops the first author of every Vancouver-style reference biases citation counts against the journals that use this style. This article measures anystyle’s parsing on references whose fields are known, describes the cleaning rules of anystyleR and what they change, and lists what remains to be done.

Data: references indexed by Scopus

For each reference of the articles it indexes, Scopus gives the reference as printed and the fields it extracted and matched: first author, year, title, journal (source title), volume, issue, pages and DOI. From a database of economics journals, we drew 251 articles in 105 journals, 50 per decade from the 1970s to the 2000s, and the few available for the 1960s (31) and the 2020s (20); the database held no references of articles from the 2010s when the sample was drawn. We kept their 7,944 references with a printed text. They mix citation styles (author-date, Vancouver, footnote-style notes in older journals) and languages.

Scopus is not a gold standard: its fields come from its own parsing and matching, and they are sometimes wrong (a title merged into the journal name, a footnote whose “author” is “op. cit.”). Agreement with Scopus is therefore a lower bound of anystyle’s accuracy, and it is above all a tool to find systematic differences, which can then be read and turned into rules. A hand-checked gold sample is the next step.

Measure

For a field $f$ (first author, year…), let $R_f$ be the references for which Scopus gives a value $y_{if}$ , and $\hat y_{if}$ the value found by anystyle. Both are normalised by a function $\nu$ that removes accents, case, spaces and punctuation, because Scopus strips accents and punctuation differs between styles. The agreement rate is

$A_f = \frac{1}{|R_f|} \sum_{i \in R_f} \mathbf{1}\left[\nu(\hat y_{if}) = \nu(y_{if})\right],$

and the disagreements are split into missing ( $\hat y_{if}$ empty) and different values. The first author is compared on the family name; the title and the journal only for references with a title in Scopus (Scopus puts book titles in the source title). The issue is compared with the number of clean_ref() when the issue is empty, as Scopus stores the number of a working paper as its issue.

We compare two versions. Raw takes the first value of each anystyle field with the processing anybody would do (the first four-digit number of the date, the page range split at the dash). Clean is the output of clean_ref().

Results

Field	References	Raw	Clean	Gain
First author	6954	78.6%	89.5%	+10.9 pts
Last page	3602	83.8%	94.7%	+10.9 pts
Issue	1599	82.9%	87.6%	+4.7 pts
Year	7311	96.6%	98.7%	+2.1 pts
First page	4389	92.0%	93.6%	+1.6 pts
Volume	3716	86.8%	88.1%	+1.3 pts
DOI	1050	99.8%	100.0%	+0.2 pts
Journal	4758	83.2%	83.2%	+0 pts
Title	5289	87.1%	87.1%	+0 pts

anystyle alone agrees with Scopus on the year of 96.6% of references and on the title of 87.1%. The cleaning rules raise the agreement on the first author from 78.6% to 89.5%, on the last page from 83.8% to 94.7%, and on the year from 96.6% to 98.7%. They do not change the title and the journal: errors there are errors of segmentation (where a title ends and the journal begins), which only a better model can fix.

Agreement depends on the period of the citing article. Older articles cite in footnote styles and their references are often notes rather than bibliographic entries. The cleaning matters most there: for the articles published before 1970, agreement on the first author rises from 14.3% to 51.6%, mostly because names printed “Family I.” are repaired.

Agreement with Scopus by period of the citing article, raw and cleaned, for the first author, the year and the title.

The rules, and the errors they correct

Each rule was written after reading samples of disagreements, and kept only if it raised agreement without creating new errors. Here are the main ones, on real cases (run live with anystyle).

Initials taken for the family name. References written “Family I.” without a comma (Vancouver style) are read by anystyle with the initials as the family name. This affected about 10% of the names in the sample, and is the largest source of first-author errors:

library(anystyleR)
example <- "Brainard W.C., Tobin J. On the internationalization of portfolios. Oxford Economic Papers. 44:1992;433-565."
raw <- parse_ref_to_df(example, clean = FALSE)
raw$author[[1]]
#> # A tibble: 2 × 2
#>   family given   
#>   <chr>  <chr>   
#> 1 W.C.   Brainard
#> 2 J      Tobin
clean_ref(raw) |> select(authors, flag)
#> # A tibble: 1 × 2
#>   authors                  flag                
#>   <chr>                    <chr>               
#> 1 Brainard, W.C.; Tobin, J family_given_swapped

Ditto marks. In many bibliographies, a dash or an underscore replaces the name of the author of the previous reference. anystyle resolves long dashes (“———”) itself, but not a single character (“_.”), which it keeps as the author’s name; anystyleR takes the authors of the previous reference of the same document.

Years. When anystyle finds no date, the year is taken from the reference itself if it is in brackets or if it is the only year the reference contains. On the references without date, this second rule recovered 90 of 107 years without a single error; a reference with several years (a title “1840-1950”, a reprint) keeps no year rather than a wrong one. anystyle also drops the letter of “1985a”, which is read back into year_suffix: it distinguishes the works of an author in a given year.

Pages and volumes. Abbreviated page ranges (“1145-64”) are expanded, roman volumes (“LIV”) converted, volume and issue glued together (“47 2”) split, and anystyle’s own error “ue 2” (from “Issue 2”) repaired.

Given names in capitals. anystyle splits “DONALD” into “D.O.N.A.L.D.”: the name is read back from the reference.

Grey literature

Grey literature has no index to compare with. We parsed the 16,601 reference paragraphs of 436 central bank documents (working papers, speeches, bulletins and reports collected from the websites of the BIS, the ECB and the Bank of England) and read samples of the output. Three errors of anystyle dominate, and rules correct them:

grey <- c(
  "Bank of England (2015) ‘The Financial Policy Committee’s powers over housing tools’, Policy Statement, July 2015",
  "Committee on the Global Financial System (2012), ‘Improving the BIS international banking statistics’, CGFS Paper No. 47, November.",
  "Mody, A. and Sandri, D. (2011), “The Eurozone Crisis: How Banks and Sovereigns Came to be Joined at the Hip”, IMF Working Papers, WP/11/269.",
  "Draghi, M. (2012), “The monetary policy of the European Central Bank and its transmission in the euro area”, speech at Università Bocconi, Milan, November."
)
raw <- parse_ref_to_df(grey, clean = FALSE)
tibble(
  anystyle_author = vapply(raw$author, function(a) paste(c(a$given, a$family), collapse = " "), ""),
  anystyle_type = raw$type
) |>
  bind_cols(clean_ref(raw) |> select(authors, title, number, type))
#> # A tibble: 4 × 6
#>   anystyle_author     anystyle_type   authors                                  title    number type 
#>   <chr>               <chr>           <chr>                                    <chr>    <chr>  <chr>
#> 1 "Bank England"      article-journal Bank of England                          The Fin… <NA>   arti…
#> 2 ""                  article-journal Committee on the Global Financial System Improvi… 47     repo…
#> 3 "A. D. Mody Sandri" article-journal Mody, A.; Sandri, D.                     The Eur… WP/11… repo…
#> 4 "M. Draghi"         chapter         Draghi, M.                               The mon… <NA>   spee…

Institutions read as persons (“Bank of England” as a person named England, with “Bank” as given name) or as titles (“Committee on the Global Financial System”): they become a single name.
Working papers read as journal articles, the series in the journal field and the number in the issue: they are typed report, with the series in container_title and the number in number.
Speeches typed as conference papers or articles become speech.

The cleaning changes the type of many references of this sample:

Type	anystyle (raw)	anystyleR (clean)
article-journal	46.0%	58.8%
(none)	22.5%	23.7%
report	16.3%	1.3%
book	6.6%	7.9%
chapter	6.2%	6.4%
speech	1.0%	0.0%
paper-conference	0.9%	1.4%
manuscript	0.5%	0.5%
thesis	0.0%	0.0%
webpage	0.0%	0.0%

and the flags show how often each rule applies:

Flag	References	Share
report_series	2534	15.3%
not_a_reference	1730	10.4%
last_page_expanded	1341	8.1%
institutional_author	1329	8.0%
given_restored	288	1.7%
several_container_title	281	1.7%
family_given_swapped	268	1.6%
year_from_full_ref	251	1.5%
speech	166	1.0%
several_issue	74	0.4%
several_volume	65	0.4%
several_pages	64	0.4%

About one paragraph in ten is flagged not_a_reference: in this database, the reference sections were cut from the PDFs by their layout, and they contain table cells, headings and disclaimers. The flag lets you drop them.

Cutting flattened bibliographies

anystyle find needs line breaks. Text from OCR or from databases often holds a whole bibliography in one block, where it finds nothing. split_ref() cuts such a block with rules on where a reference starts (an author written “Family, I.”, “Family I.”, “I. Family”, a ditto mark, a bracketed number, after the end of the previous reference). To measure it, we joined the references of each Scopus article into one block, where the true boundaries are known:

split_scores <- read_data("split_scores.csv")

On 7,693 boundaries in 251 articles, 92.4% of the cuts are right (precision) and 77.5% of the true boundaries are found (recall). Misses are mostly references that start with an institution, a title or a note (“See also…”), and false cuts come from lists of authors without commas and from places that look like names (“Cambridge, Mass.”). Real OCR text is harder than this test (page headers, hyphenation), but the rules work: on 60 pages of flattened JSTOR bibliographies, parse_ref_to_df(split = TRUE) returned 545 references with plausible authors, years and titles.

What the rules cannot fix

Segmentation within the reference. When anystyle ends a title too early, or merges a subtitle into the journal, the information is still in the reference but in the wrong field. Several titles or journals are flagged (other_title, several_container_title), but not repaired. This is the task of a better model: we plan to retrain anystyle’s parser on references generated from known records (Zotero libraries, Crossref) rendered in many citation styles, with grey literature and non-English references.
Names in “Family Given” order without comma (“Goode Richard”): the order cannot be told from the text alone.
Notes rather than references (footnotes with “op. cit.”, comments): they are not bibliographic entries.

These measures come from the scripts data-raw/evaluate_scopus.R and data-raw/inspect_grey_literature.R of the package repository, with anystyle version 1.6.0 (cli 1.5.0, data 1.3.0), on 2026-11-04. The samples themselves are not distributed (Scopus data are licensed).