Do Online Readerships Offer Useful Assessment Tools? Discussion Around the Practical Applications of Mendeley Readership for Scholarly Assessment

This methods report illustrates the relevance of Mendeley readership as a tool for research assessment. Readership indicators offer new possibilities to inform the evaluation of publications and outputs either poorly covered in citation indexes (e.g. non-English language outputs, Global South publications, Social sciences and humanities), or typically excluded from citation analysis (e.g. letters, editorial material, etc.). Mendeley readership can also inform the earlier impact of scientific outputs, as well as the impact among wider non-academic audiences. All these features are discussed in this report and the relevance of readership indicators to extend the concept of research impact beyond specific acts (e.g. citations) is highlighted. Best practical recommendations on how Mendeley readership can be used for assessment purposes are discussed. Policy highlights: • This paper illustrates practical possibilities of readership indicators for research evaluation. • Readership indicators inform impact of publications poorly covered in bibliometrics databases or excluded from citation analysis. • Readership indicators inform early impact and non-academic impact of publications. • Readership indicators can be used to inform, support, and complement (citation-based impact) decisions on research evaluation exercises.


Introduction
Developing indicators for assessing the impact and value of research has been highlighted as a crucial step to support the process of decision making in the context of research evaluation (Wilsdon et al., 2015;Wilsdon & Al., 2017). Limitations of current citation-based indicators in reflecting the broad value of research (beyond scientific impact) and its contributions to society have led to the development of alternative indicators in research evaluation, also known as altmetrics (Priem, et al., 2010) or more specifically as social media metrics (Haustein, Bowman, & Costas, 2016;Wouters, Zahedi, & Costas, 2019). These new indicators have gained significant attention in recent years, particularly in some national research assessment exercises (e.g., Research Excellence Framework (REF 1 ) in the UK or the Dutch Standard Evaluation Protocol (SEP 2 )), since they were expected to reflect a broader perspective on the impact of research. Mendeley readership metrics have been discussed as one of the altmetric sources with a stronger relevance for research evaluation (Thelwall, 2020), particularly as a complement in those aspects where citations present more weaknesses (Thelwall, 2017b). 3 Despite this strong interest on Mendeley readership, and discussions on their complementarity to citations, a reflection on the current best practical applications of readership is still missing in the literature. The main ambition of this methods report is to precisely illustrate some of these best practical applications of online readership in overcoming some of the limitations of more traditional citation analyses.

Methodological description Source database: Crossref
In this study we worked with bibliographic data from Crossref. 6 The choice of Crossref as a bibliographic data source for this illustrative exercise is motivated by the need of having a large independent bibliographic database containing all sorts of publications, and not being restricted to the coverage criteria of WoS and Scopus , or even Mendeley. The full Crossref database is stored in an in-house relational database available at CWTS. This database was downloaded in August 2018 through the public REST API. This in-house database has been used in order to collect the output of six sample universities (Leiden University, Stellenbosch University, Dalian Technology University, Sao Paulo University, University of Harvard, and Curtin University). The selection of these universities responds only to illustrative reasons (i.e. they are used as examples of the practical uses of readership indicators, thus there is no special meaning in their choice). We tried to select a university per world continent (i.e. Europe, Africa, Asia, South America, North America and Oceania) and their choice was also based on the personal knowledge of the authors of this report of some of them.
The universities were queried in the affiliation field of Crossref by using their English names. The authors' affiliation information in the Crossref database is based on the Crossref member organizations and enrichment done by Crossref itself (Hendricks et al., 2020). No time restriction (limitation to publication dates/years) in collecting Crossref DOIs 4 https://api.mendeley.com/apidocs/docs. 5 https://www.mendeley.com/catalogue/#. 6 Crossref (http://www.crossref.org) is an official DOI registration agency. It facilitates the links between distributed content hosted at different sites (e.g. publishers' websites) and provides basic metadata about the records registered.
Readership data refers to the number of users who have saved documents to their private libraries (in Mendeley). The more users that save a publication, the more readership attributed to the publication. A Mendeley user is any individual that creates a user profile on the Mendeley online reference manager application.
Users are requested to disclose their academic status (e.g. PhD, professor, etc.), disciplines and countries, which later on can be used to further characterize the readership indicators.
The Mendeley catalog provides information on the total number of readers per document as well as statistics (%) on users' academic statuses (recently called as readers 'seniority'), disciplines, and countries.
was applied, in order to illustrate how Mendeley readership can also be used to analyze publications from any time. It is important to highlight that the number of publications with affiliations in Crossref is not indicative of the true output of the universities selected, since Crossref does not index all affiliations of all publications recorded in Crossref. 7 Therefore, we do not claim any completeness in our data collection, and the same analysis could have been done using other bibliometric databases (e.g. Scopus, WoS, Dimensions) that have more curated affiliation information. However, by choosing Crossref we want to emphasize the possibilities of Mendeley of providing readership indicators to publications available in an open and free database, without initially needing the use of a commercial and more curated database. A total of 79,416 different Digital Object Identifiers (DOIs) of publications identified in Crossref for the selected universities were collected.

Readership data: Mendeley
Readership data used in this report were based on the annual Mendeley data collection carried out at CWTS of Mendeley data (annual data collection of 2018). In the annual CWTS data collection approach, the Mendeley API 8 is queried based on yearly updated lists of DOIs (obtained from Crossref, Web of Science (WoS) or Scopus, among other data sources). The downloaded JSON files resulting from querying the Mendeley API are then parsed and stored as relational tables in a SQL database environment, which allows for a more advanced use of the original data (e.g. use of the breakdowns of types of users, disciplines, or countries).

Citation data and additional bibliographic metadata: WoS & Scopus
For some of the analysis some additional metadata (e.g. citation counts, thematic classifications and document types) that were not available in either Crossref or Mendeley were necessary. In order to obtain these other metadata elements, the Crossref records were matched with the CWTS in-house databases of WoS and Scopus based on DOIs. WoS and Scopus citations counts were added to the Crossref records. Moreover, journal-based subject classifications (CWTS NOWT classification 9 ), publication years, and document types of the Crossref records were also extracted from the WoS database in order to perform some of the analysis. 10

General overview of coverage and density of Crossref DOIs across selected universities
The general descriptive values for the DOIs of the sample universities are presented in Tables 1 and 2. Descriptive analysis such as publication coverage and metrics density (i.e. the Mean Citation Score [MCS] and Mean Readership Score [MRS] 11 ) were calculated for the six selected universities, their combined set of publications, and also by disciplines, document types, and publication years. In terms of average impact (readership and citations), the total set of publications has an overall MRS of 23.0, which contrasts with the mean citation impact in the other two databases (MCS of 14.0 in Scopus and 15.3 in WoS). This higher density of Mendeley readership (over the citation densities from the other two databases) is also observable for most of 7 Journal publications, scholarly books, and conference proceedings represent the largest content in Crossref. The basic metadata in Crossref includes title, publication dates, authors, journal title, conference name, volume/issue number, author's affiliations and ORCID, abstracts and links to full text, funding metadata, license metadata, list of references, clinical trial numbers, figures and supplementary materials (Hendricks, et al., 2020). Affiliation data however is based on the input of the different member organizations -who register metadata along with the digital object identifier for their registered content -therefore there is no affiliation data for all Crossref records. 8 https://dev.mendeley.com/methods/#retrieving-catalog-documents. 9 https://www.cwts.nl/pdf/nowt_classification_sc.pdf. 10 This matching with WoS is motivated by the lack of reliable metadata about classifications, document types, etc. in Crossref (Visser, Eck, and Waltman, 2020). 11 Mean Citation Score is the ratio of the total number of citations (TCS) divided by the total number of publications (P) of a given unit, thus MCS = TCS/P. Mean Readership Score is the ratio of the total number of readership (TRS) divided by the total number of publications (P) of a given unit. MCS WoS = TCS WoS/P WoS; MCS Scopus = TCS Scopus/P Scopus; MRS = TRS/P Mendeley. 12 https://blog.scielo.org/en/2018/10/03/how-journals-can-make-the-most-of-crossref-membership/#.XmlLVqhKg2w. 13 https://crossrefbrasil.files.wordpress.com/2011/04/crossrefbrasilianconference_scielo.pdf. the selected universities, with the only exception of Dalian Technology University for which the mean citation scores in WoS (14.0) and Scopus (9.4) are higher than the MRS (9.3), probably related to the lower uptake of Mendeley in China (Fairclough, and Thelwall, 2015;Thelwall, and Maflahi, 2015).

Current applications of Mendeley readership
In this section different Mendeley-specific applications are illustrated, particularly focusing on how Mendeley readership can help to overcome some of the most common weaknesses attributed to citation databases and citation analyses, namely a) informing the impact of publications not covered in citation databases; b) informing the impact of document types typically excluded from citation analysis; c) informing the impact of publications from disciplines that are not well covered in citation databases; d) informing early impact of recent publications; and e) the possibility of refining the impact measures by further characterizing the users saving the publications. 14

Informing the impact of non-indexed publications (i.e. publications not covered in WoS, Scopus)
The restrictions of the most common bibliometric databases (e.g. WoS or Scopus), with regards to their coverage of some fields, language, and publication formats, represent one of the most common challenges in scientometric studies (Torres-Salinas, Cabezas-Clavijo & Jimenez-Contreras, 2013; Van Raan, Archambault & 14 In principle, it is possible to calculate the MRS values including also those publications that are not covered in Mendeley, assuming then a readership value of zero. However, since such an assumption cannot be applied for the publications not covered in WoS and Scopus (they may be cited but not tracked by these databases), in order to be consistent in this study we calculate MRS only for those publications covered in Mendeley. The same approach has been adopted for MCS Scopus and MCS WoS.   Larivière, 2006). In this section we illustrate how readership indicators offer practical opportunities for identifying the impact of publications that are not indexed in the most common bibliometrics databases. 15 Figure 1 depicts the share of Crossref DOIs that are covered by the different data sources (WoS, Scopus, or Mendeley) or any combination of them (see also supplementary file 1. Table A1 in the appendix 16 ). The number of overlapped DOIs across all data sources is 40,251 (51%). Overall, Mendeley has the largest coverage of publications that are not covered by any of WoS or Scopus databases (n = 18,658; 23.5% of the total publication dataset). Put differently, Mendeley offers metrics for a larger set of publications for which no metrics are available in WoS or Scopus. Share of publications covered both in Mendeley & Scopus alone (11,261; 14.2%) is higher than the share of publications covered both in Mendeley & WoS alone (5,331; 6.7%) and WoS & Scopus (273; 0.3%) (Figure 1; see also supplementary file 1. Table A1 in the appendix 17 ). These results show that Mendeley can be used to identify the impact of publications not indexed in these two citation databases.

Informing the impact of document types typically excluded from citation analysis
Document types like editorial materials, letters, news items, book reviews or meeting abstracts are types of publications that focus more on disseminating scientific debates, news, opinions, or summarized information, and typically receive relatively low numbers of citations. Due to their lower citation density they are usually deemed not suitable for robust citation analysis and are often excluded from citation analyses (Waltman et al., 2011). 18 Previous studies have shown that these document types have some coverage on Mendeley, and they also have higher readership counts than citations (Zahedi and Haustein, 2018 18 Many of these document types are also typically not peer reviewed, which can be another reason to exclude them from citation analysis. However, there could still be cases in which the analysis of the impact of these document types is relevant, e.g. a scientific journal interested in evaluating the impact of its editorial or news material, or a researcher or university department interested in discussing their impact also on these type of documents; in such cases Mendeley readership could provide relevant support evidence. 19 https://doi.org/10.6084/m9.figshare.12824201.  Table A2 in the appendix 19 ).
Since the classification of document types in Crossref has fundamental limitations (Visser, Eck, and Waltman, 2020), the WoS document type classification was used for the set of DOIs (n = 33,868) from the years 2012-2018. 20 Figure 2 (and Supplementary file 2. Table A2 in the appendix 21 ) presents the coverage and average values of the Crossref DOIs for both WoS & Scopus citations and readership for the different document types identified in WoS. While articles and reviews are the document types that are most cited in WoS and Scopus, data papers, reviews, articles, editorial material, letters, and news items are the most saved document types on Mendeley. These results support the idea that Mendeley readership can be used to identify the readership impact of these document types. 22

Informing the impact of fields that are not well covered in citation databases
Social sciences and humanities are among the research fields worst covered in citation databases (Nederhof, 2006). Their low citation density makes it more difficult to study the citation impact in these fields, as well as to compare their impact with other fields. However, Mendeley readership has been observed to have a higher density than citations in these fields (Thelwall, 2015;Thelwall, 2017b;, thus opening the possibility for more reliable and substantial studies on the impact of social sciences and humanities. In this section we show how readership indicators can offer some impact evidence when studying these fields. Since Crossref does not count with a comprehensive classification of all documents (Visser, Eck, and Waltman, 2020), the set of DOIs from the years 2012-2018 and also covered in WoS (n = 33,868) were classified into the seven main fields of science based on NOWT classification scheme 23 available from CWTS in-house database. Citation and readership indicators from CWTS in-house WoS, Scopus, and Mendeley databases were calculated and aggregated by the seven NOWT 24 main fields of science (  On the other hand, Engineering science is the field with the lowest readership density (11.7), although its readership density is still higher than their citation density (8.1 in WoS and 8.2 in Scopus). These results hint to the added value of readership indicators for reflecting the impact of the fields which are typically not very well represented by citation metrics. 25

Informing the impact of recent publications
Another important weakness in citation analysis is the need of waiting for longer periods for publications to achieve a substantial number of citations, which challenges the citation analysis of very recent publications. Although Mendeley has been observed to be a metric with a relatively slow pace (Zahedi, Costas, and Wouters, 2017), it has been reported that Mendeley readership tend to have a faster accumulation for recent publications than citations (Thelwall and Sud, 2016). This suggests that readership could play a relevant role in informing the early impact of publications (Maflahi and Thelwall, 2016). In this section this point is illustrated (Table 4 and Figure 2; see also Supplementary file 3. Table  A3 in the appendix 26 for the same analysis based on the whole dataset) by studying the temporal trend in the impact of WoS publications with at least one citation in Scopus/WoS or at least one reader in Mendeley from 2012-2018. The coverage of Crossref DOIs with at least one Mendeley readership has a steady pattern from 2012 to 2017 with a small decrease (95%) in 2018. In contrast, the coverage of Crossref DOIs with at least one citation in both Scopus (from 94% in 2012 to 62% in 2018) and WoS (from 91% in 2012 to 73% in 2018) shows a decreasing pattern over time (2012)(2013)(2014)(2015)(2016)(2017)(2018). In terms of average impact, Figure 3 shows overall decreasing pattern for MRS with a little increase from 2012 to 2013 and decreases from the year 2015 onwards while MCS decreases steadily over 2012 to 2018. However, MRS is indeed higher than MCS for the whole period of 2012-2018. This result is in line with those from previous studies (Maflahi and Thelwall, 2016;Thelwall and Sud, 2016;Zahedi, Costas, and Wouters, 2017;Thelwall, 2018) and suggests that Mendeley readership can work as an important source to reflect evidence of early impact of publications.

Informing diverse types of impact (scientific, educational, professional) by characterizing the users saving the publications
In contrast to traditional citation indicators which do not provide much information about the citers (e.g. if citations are from PhDs, Professors, etc.), Mendeley readership data includes information on the Mendeley user types, as indicated by the users themselves in their Mendeley profiles. This information provides the opportunity to identify and characterize users and potentially distinguish scholarly and non-scholarly ones.
For this purpose, Mendeley users were grouped into seven broad user types: 27 PhD students, bachelor and master students, researchers (academic and non-academic institutions), professors & lecturers, librarians, other professionals, and 25 The coverage of WoS is 100% since we are looking at the publications from Crossref that are matched in WoS in order to extract the NOWT classification from that database.  unspecified users. The dataset of Crossref DOI with at least one reader (n = 66,536) selected for this analysis. Coverage of publication with at least one reader and Mean Readership Scores (MRS) per user type were calculated. Table 5 shows coverage of publications and their average readership scores by different user types. The results show that the highest coverage of publications saved by students (PhD, bachelor, and master) than other users. Also, higher MRS for the sets of publications saved by Students (PhD, bachelor, and master), over those saved by researchers, professors and other user types is visible. The same pattern is observable across the six sample universities (see Supplementary file 4. Table A4 in the appendix 28 ). The possibility to track the use of scientific publications by different user type is an advantage that citation indicators do not have.     Based on these results and those from previous studies (Bornmann and Haunschild, 2015;Mohammadi, et al., 2015;Thelwall, 2017a;Zahedi and van Eck, 2018), the identification of patterns of readership per user type could help in informing different types of impact (e.g., scientific, educational, or professional) of publications beyond the more academic types of impact reflected by citation indicators.
However, there are some limitations that need to be considered when using Mendeley users as a proxy for different types of impact. Firstly, Mendeley users are self-reported, this means that they choose their user type from a list of predefined options which may not always correspond to their local attribution (e.g. in some countries professors and researchers may be equivalent, like the CSIC in Spain and the universities in the country), and lecturers may have a different academic consideration depending on the country (e.g. in UK a lecturer can be equivalent to an assistant professor, or to a research associate in the US 29 ). Secondly, users may change their status (e.g. a PhD may become a Postdoc or a researcher) without updating their Mendeley profile. Thirdly, the different users are not completely independent from each other, and what Bachelor & Master students read may be influenced by what is recommended by their lecturers, and PhD may read what is recommended by their senior colleagues (Postdocs and professors). 30 29 https://en.wikipedia.org/wiki/List_of_academic_ranks. 30 In a previous study it was shown thematic differences across Mendeley user types (Zahedi & van Eck, 2018), suggesting indeed different patterns in what is saved by each user type, however the limitation still needs to be observed, at least from a theoretical point of view. Percentages refer to the ratio of Crossref DOIs with specific user types to all Crossref DOIs with at least one reader.

Discussion
Mendeley readership are considered the most prominent altmetric source with evaluative value, particularly given their large coverage of scientific publications (Costas, Zahedi, and Wouters, 2015a;Thelwall and Sud, 2016;Thelwall, 2017b), the high density levels (Mohammadi et al., 2017;Zahedi and Haustein, 2018), moderate correlation levels with citations (Zahedi, Costas, and Wouters, 2014;Costas, Zahedi, and Wouters, 2015b;Thelwall, 2018), and conceptual proximity to citation indicators (Wouters, Zahedi, and Costas, 2019;Sugimoto et al., 2017). All these interesting properties of Mendeley readership for research evaluation have been discussed in multiple scattered scientific publications (Thelwall, 2020;Thelwall, 2018), and a PhD Thesis , however we were still lacking a focused discussion on the specific possibilities of Mendeley readership for evaluative purposes. In this work we focus on illustrating the practical possibilities of Mendeley readership for research evaluation in aspects in which citation analysis pose more challenges. In this regard we are adopting a relatively conservative perspective, seeing Mendeley readership as a complement to citations, which is also the most common recommendation in the literature (Haustein, Bowman, and Costas, 2016;Thelwall, 2020). There have been also discussions about the possibility of considering readership as another type of "currency of science" that could be discussed on par with citations (Costas, Perianes-Rodríguez, and Ruiz-Castillo, 2017), but we are not discussing this approach in this study. Possibilities of Mendeley readership for research evaluation can be discussed as follows:

Impact evidence for non-indexed publications
Mendeley readership can represent an important source of evidence of the impact of publications not indexed in mainstream citation databases (WoS or Scopus). This is particularly relevant for publications from the Global South and developing countries, since they usually have a lower coverage in most bibliometric databases (Alperin, 2013), with a substantial share of the output from these countries underrepresented in citation databases (Zahedi, & Costas, 2017). In a recent study it has been observed that publications from the Global South countries (Africa and South America) have relatively higher levels of readership in contrast to citation metrics (Costas, Zahedi, and Alperin, 2019), which reinforces the idea that Mendeley readership can play a relevant role in providing impact evidence for non-indexed publications. In this report the higher coverage of University of Sao Paulo (USP) publications on Mendeley reinforces this relevance of Mendeley readership for studying the impact of Global-South countries and non-indexed publications.

Impact evidence for fields with lower coverage in bibliometrics databases
A higher density of readership (over citation metrics) is typically observed for social sciences and humanities publications. In citation databases, citation impact in these disciplines is largely affected by the lower coverage of books and by the more national or local orientation of the published research (typically in other languages than English) (Moed, 2006). This study illustrates how Mendeley readership can be a valuable source to inform the impact analysis of the fields that are not well represented with citation indicators.

Impact evidence for document types that are not frequently cited
Reviews and articles are the most prevalent document types in citation analysis, since these document types capture the most important scientific findings. Usually other document types (e.g. editorial material, letters, meeting abstracts, news items, data papers, etc.) are excluded from citation analysis because they are deemed not to represent the same type of scientific contribution than articles and reviews. However, there may be situations in which the impact analysis of these other document types is necessary (e.g. a journal that wants to analyze the impact of its editorials or news items; or research teams that also want to evaluate the impact of those types of outputs). In such cases, citations are not very helpful given the low citedness of these document types. However, we have illustrated how Mendeley readership have a higher coverage and higher readership values for some of these document types (e.g., letters, data papers or editorial materials), supporting the idea of a strong relevance of Mendeley readership for the evaluation of these outputs.

Impact evidence for recent publications
We have illustrated how readership scores are more prevalent than citations in recent publications and hence they could work as an early indicators of research impact (Thelwall and Sud, 2016;Thelwall, 2018). These findings together with the fact that Mendeley readership are available openly and that they can earlier signal highly cited publications (Zahedi, Costas, and Wouters, 2016), highlight the value of Mendeley as a tool for revealing early impact of publications, particularly when substantial number of citations haven't yet been accrued.

Impact evidence beyond academic impact
We argue and illustrate how readership scores from non-academic users (such as students, librarians, or professionals) could reflect other types of impact, such as educational or professional. This more fine-grained possibility of studying the different types of users interacting with the publications on Mendeley, is something that is not possible with the most common citation indicators. This suggests the potential of Mendeley as a relevant source for expanding the notion of impact beyond the more academic impact captured by citations, although self-reported nature of Mendeley needs to be considered.

Best practice recommendations
Below we summarize some of the best practical recommendations on how Mendeley readership can be used for assessment purposes: -Use the Mendeley API or Mendeley catalog for data collection. It is recommended to retrieve readership indicators directly from Mendeley rather than using readership indicators provided by other altmetrics providers (e.g. Altmetric.com or PlumX). This is because different methodological choices are used by altmetric aggregators in collecting, processing, and reporting social media metrics Ortega, 2018). Besides, users collecting themselves their own data via the API can benefit from the more extensive information provided by Mendeley, like the breakdown of readership by users types, countries and disciplines. 31 -Use publication identifiers for data collection. Collection of readership data using Mendeley API can be done based on list of digital object identifiers (such as DOI, PubMed IDs, etc.) or manual collection of readership from Mendeley catalog. Although a manual data collection is probably more thorough, collecting data using publication identifiers is more systematic and reproducible. Other tools such as Webometric Analyst (http://lexiurl.wlv.ac.uk) can be used to retrieve readership indicator based on title of documents or DOIs. However, data collection based on titles may retrieve more duplicate records (Zahedi, Bowman, and Haustein, 2014). -Be aware of time and field differences of readership. Readership indicators differ per fields of science and publication years. These differences need to be taken into account when working with multidisciplinary datasets. As such, field-normalized indicators in the same fashion as for citations have also been proposed for readership indicators Haunschild and Bornmann, 2016). -Breakdown of readership metrics by user type can represent an interesting proxy for other types of impact, but be aware of the limitations of Mendeley user information. The additional information by Mendeley can be used to characterize the diversity of the readership audiences of a set of publication, hinting to the idea of different types of impact (e.g. educational, professional, or academic), however the limitations of Mendeley user types (self-reported, lack of updates and relationships among academic roles) need to be considered. -Use readership indicators preferably as a complement of citations. Readership indicators can be useful to inform the impact of outputs, document types, fields and outputs from recent years, which are not well covered in the more traditional citation databases (WoS or Scopus), or their impact is not well captured by citations. It is of course possible to use readership indicators in parallel to citation indicators, but their use should be better restricted to contrast and to contextualize citation analysis (e.g. for Global South countries' publications) rather than to replace them, since for citations there is a substantial literature and a much better understanding of their pros and cons. A proper framework to consider the role of the two metrics (citations and readership) in evaluative contexts is still missing (Costas, Perianes-Rodríguez, and Ruiz-Castillo, 2017). -Observe the responsible use of indicators and apply common sense. In line with the recommendations of the Leiden Manifesto (Hicks et al., 2015), the use of Mendeley readership in an evaluation is not exempted from observation of the same precautions as when more traditional scientometric indicators are applied.
There are still important open questions regarding the practical and conceptual limitations of Mendeley readership for research evaluation. Moreover, it is important to continue developing the concept of readership, which as currently operationalized as Mendeley only captures the act of saving a publication by a user in her library; however more advanced metrics, capturing different forms of engagement of users with the publications (e.g., the act of opening a document, scrolling through it, highlighting the text, commenting it, etc.) could be also captured on Mendeley (or any other online reference manager); thus opening the door to more advanced possibilities for studying different uses and impacts of scientific publications.

Data Accessibility Statement
The underlying Crossref and Mendeley metrics data that support the findings of this study are openly available in Figshare at https://doi.org/10.6084/m9.figshare.12824201.

Additional Files
The additional files for this article can be found as follows: •