Developing indicators for assessing the impact and value of research has been highlighted as a crucial step to support the process of decision making in the context of research evaluation (Wilsdon et al., 2015; Wilsdon & Al., 2017). Limitations of current citation-based indicators in reflecting the broad value of research (beyond scientific impact) and its contributions to society have led to the development of alternative indicators in research evaluation, also known as altmetrics (Priem, et al., 2010) or more specifically as social media metrics (Haustein, Bowman, & Costas, 2016; Wouters, Zahedi, & Costas, 2019). These new indicators have gained significant attention in recent years, particularly in some national research assessment exercises (e.g., Research Excellence Framework (REF1) in the UK or the Dutch Standard Evaluation Protocol (SEP2)), since they were expected to reflect a broader perspective on the impact of research. Mendeley readership metrics have been discussed as one of the altmetric sources with a stronger relevance for research evaluation (Thelwall, 2020), particularly as a complement in those aspects where citations present more weaknesses (Thelwall, 2017b).3 Despite this strong interest on Mendeley readership, and discussions on their complementarity to citations, a reflection on the current best practical applications of readership is still missing in the literature. The main ambition of this methods report is to precisely illustrate some of these best practical applications of online readership in overcoming some of the limitations of more traditional citation analyses.
One of the strongest advantages of Mendeley readership data is its free access via the Mendeley public API4 or via manual collection of readership data from the Mendeley catalog.5 The Mendeley API offers free access to diverse metadata and endpoints including among others readership statistics, as well as breakdown statistics by users’ academic statuses, disciplines, and countries. The best approach to collect Mendeley readership data is to directly use the public API, particularly over other third-party sources (e.g. vial Almetric.com or PlumX, see Robinson-García, et al., 2014; Ortega, 2018; Zahedi and Costas, 2018). Another important advantage of the Mendeley API is that it allows for data collection based on different publication identifiers (DOI, PMID, arXiv ids, Scopus ids).
Box 1: Terminology.
Readership data refers to the number of users who have saved documents to their private libraries (in Mendeley). The more users that save a publication, the more readership attributed to the publication.
A Mendeley user is any individual that creates a user profile on the Mendeley online reference manager application. Users are requested to disclose their academic status (e.g. PhD, professor, etc.), disciplines and countries, which later on can be used to further characterize the readership indicators.
The Mendeley catalog provides information on the total number of readers per document as well as statistics (%) on users’ academic statuses (recently called as readers ‘seniority’), disciplines, and countries.
Among the limitations of Mendeley readership data we can mention the lack of access to temporal and longitudinal data (i.e. the readership history of publications), the short readership window for recent publications, the lack of options for bulk data download, and the dependency on using publication identifiers (DOI, arXiv ID, Scopus ID, PMID, etc.) for querying the API (Zahedi, Bowman, and Haustein, 2014; Haustein, 2016; Zahedi and Costas, 2018). Other known limitations of Mendeley data include the biases in coverage of some geographical areas (e.g. China or Russia) (Thelwall, & Maflahi, 2015; Fairclough, and Thelwall, 2015; Alperin, 2015; Zahedi and Costas, 2017); the potential manipulability of Mendeley indicators by registering duplicate records; the inaccuracies in the proper identification of Mendeley users (e.g. wrong academic status or geographic origin); or the data qualities issues in Mendeley (e.g. the limited availability and quality of bibliographic metadata, the existence of duplicate records with different publication identifiers, etc.). These limitations emphasize that it is often necessary to resort to other bibliographic data sources (e.g. WoS or Scopus) for advanced analysis (Wouters and Costas, 2012; Haustein, 2016; Wouters, Zahedi, and Costas, 2019; Thelwall, 2020) able to overcome the lack of metadata in Mendeley (we will illustrate this issue in this work).
This methods report is structured as follows: section 2 describes the methodological approach. Section 3 provides general overviews related to the publications, coverage and impact (readership and citations) related to the six different example universities. Section 4 provides focused examples on the practical uses of Mendeley readership, and finally section 5 provides and discussion of these practical use, and section 6 condenses some specific best practice recommendations of how Mendeley readership can be used for assessment purposes.
In this study we worked with bibliographic data from Crossref.6 The choice of Crossref as a bibliographic data source for this illustrative exercise is motivated by the need of having a large independent bibliographic database containing all sorts of publications, and not being restricted to the coverage criteria of WoS and Scopus (van Eck et al., 2018), or even Mendeley. The full Crossref database is stored in an in-house relational database available at CWTS. This database was downloaded in August 2018 through the public REST API. This in-house database has been used in order to collect the output of six sample universities (Leiden University, Stellenbosch University, Dalian Technology University, Sao Paulo University, University of Harvard, and Curtin University). The selection of these universities responds only to illustrative reasons (i.e. they are used as examples of the practical uses of readership indicators, thus there is no special meaning in their choice). We tried to select a university per world continent (i.e. Europe, Africa, Asia, South America, North America and Oceania) and their choice was also based on the personal knowledge of the authors of this report of some of them.
The universities were queried in the affiliation field of Crossref by using their English names. The authors’ affiliation information in the Crossref database is based on the Crossref member organizations and enrichment done by Crossref itself (Hendricks et al., 2020). No time restriction (limitation to publication dates/years) in collecting Crossref DOIs was applied, in order to illustrate how Mendeley readership can also be used to analyze publications from any time. It is important to highlight that the number of publications with affiliations in Crossref is not indicative of the true output of the universities selected, since Crossref does not index all affiliations of all publications recorded in Crossref.7 Therefore, we do not claim any completeness in our data collection, and the same analysis could have been done using other bibliometric databases (e.g. Scopus, WoS, Dimensions) that have more curated affiliation information. However, by choosing Crossref we want to emphasize the possibilities of Mendeley of providing readership indicators to publications available in an open and free database, without initially needing the use of a commercial and more curated database. A total of 79,416 different Digital Object Identifiers (DOIs) of publications identified in Crossref for the selected universities were collected.
Readership data used in this report were based on the annual Mendeley data collection carried out at CWTS of Mendeley data (annual data collection of 2018). In the annual CWTS data collection approach, the Mendeley API8 is queried based on yearly updated lists of DOIs (obtained from Crossref, Web of Science (WoS) or Scopus, among other data sources). The downloaded JSON files resulting from querying the Mendeley API are then parsed and stored as relational tables in a SQL database environment, which allows for a more advanced use of the original data (e.g. use of the breakdowns of types of users, disciplines, or countries).
For some of the analysis some additional metadata (e.g. citation counts, thematic classifications and document types) that were not available in either Crossref or Mendeley were necessary. In order to obtain these other metadata elements, the Crossref records were matched with the CWTS in-house databases of WoS and Scopus based on DOIs. WoS and Scopus citations counts were added to the Crossref records. Moreover, journal-based subject classifications (CWTS NOWT classification9), publication years, and document types of the Crossref records were also extracted from the WoS database in order to perform some of the analysis.10
The general descriptive values for the DOIs of the sample universities are presented in Tables 1 and 2. Descriptive analysis such as publication coverage and metrics density (i.e. the Mean Citation Score [MCS] and Mean Readership Score [MRS]11) were calculated for the six selected universities, their combined set of publications, and also by disciplines, document types, and publication years. Table 1 shows that overall 95% of all publications identified in Crossref from the selected universities have some readership on Mendeley. This large coverage of Crossref publications on Mendeley contrasts with the coverage on the other databases, particularly with Scopus (66%), and WoS (59%). This higher coverage of Crossref publications in Mendeley holds true also across the six selected universities, with most of the universities having coverage values higher than 90%. A remarkable case is the University of Sao Paulo (USP), which has the largest set of publications identifiable in Crossref, probably based on the high metadata accuracy/enrichment and strong interoperability between Scielo and Crossref.1213 This larger set of Crossref publications from USP contrasts with the lower coverage of USP publications in WoS and Scopus (lower than 60% in both databases), while 96% of USP publications are covered on Mendeley.
|Pub Year||P Crossref||P WoS||P Scopus||P Mendeley||TCS WoS||MCS WoS||TCS Scopus||MCS Scopus||TRS||MRS||P CS > 0 WoS||P CS > 0 Scopus||P RS > 0 Mendeley|
|All years||79,416 (100%)||46,837 (59%)||52,535 (66%)||75,771(95%)||720,700||15.3||736,301||14.0||1,752,408||23.1||39,728 (84%)||41,836 (79%)||66,536 (87%)|
|P Crossref||P WoS||P Scopus||P Mendeley||TCS WoS||MCS WoS||TCS Scopus||MCS Scopus||TRS||MRS Mendeley|
|Curtin University||6,345 (8.0%)||4,576 (72.1%)||5,309 (83.7%)||6,170 (97.2%)||53,007||11.6||56,195||10.6||145,502||23.5|
|Dalian Technology University||8360 (10.5%)||5365 (64.2%)||7351 (87.9%)||8175 (97.8%)||75,044||14.0||69,462||9.4||75,819||9.3|
|Harvard University||15003 (18.8%)||9162 (61.1%)||9190 (61.3%)||13612 (90.7%)||277,257||30.1||288,419||31.4||621,821||45.5|
|Leiden University||9491 (11.9%)||7355 (77.5%)||7475 (78.8%)||9091 (95.8%)||130,567||17.7||135,396||18.1||262,970||28.8|
|Stellenbosch University||4157 (5.2%)||2897 (69.7%)||3217 (77.4%)||3955 (95.1%)||33,878||11.6||35,988||11.2||116,686||29.4|
|University of Sao Paulo||36253 (45.5%)||17646 (48.7%)||20163 (55.6%)||34955 (96.4%)||150,947||8.5||150,841||7.5||529,610||15.1|
In terms of average impact (readership and citations), the total set of publications has an overall MRS of 23.0, which contrasts with the mean citation impact in the other two databases (MCS of 14.0 in Scopus and 15.3 in WoS). This higher density of Mendeley readership (over the citation densities from the other two databases) is also observable for most of the selected universities, with the only exception of Dalian Technology University for which the mean citation scores in WoS (14.0) and Scopus (9.4) are higher than the MRS (9.3), probably related to the lower uptake of Mendeley in China (Fairclough, and Thelwall, 2015; Thelwall, and Maflahi, 2015).
In this section different Mendeley-specific applications are illustrated, particularly focusing on how Mendeley readership can help to overcome some of the most common weaknesses attributed to citation databases and citation analyses, namely a) informing the impact of publications not covered in citation databases; b) informing the impact of document types typically excluded from citation analysis; c) informing the impact of publications from disciplines that are not well covered in citation databases; d) informing early impact of recent publications; and e) the possibility of refining the impact measures by further characterizing the users saving the publications.15
The restrictions of the most common bibliometric databases (e.g. WoS or Scopus), with regards to their coverage of some fields, language, and publication formats, represent one of the most common challenges in scientometric studies (Torres-Salinas, Cabezas-Clavijo & Jimenez-Contreras, 2013; Van Raan, Van Leeuwen & Visser, 2011; Archambault & Larivière, 2006). In this section we illustrate how readership indicators offer practical opportunities for identifying the impact of publications that are not indexed in the most common bibliometrics databases.16
Figure 1 depicts the share of Crossref DOIs that are covered by the different data sources (WoS, Scopus, or Mendeley) or any combination of them (see also supplementary file 1. Table A1 in the appendix17). The number of overlapped DOIs across all data sources is 40,251 (51%). Overall, Mendeley has the largest coverage of publications that are not covered by any of WoS or Scopus databases (n = 18,658; 23.5% of the total publication dataset). Put differently, Mendeley offers metrics for a larger set of publications for which no metrics are available in WoS or Scopus. Share of publications covered both in Mendeley & Scopus alone (11,261; 14.2%) is higher than the share of publications covered both in Mendeley & WoS alone (5,331; 6.7%) and WoS & Scopus (273; 0.3%) (Figure 1; see also supplementary file 1. Table A1 in the appendix18). These results show that Mendeley can be used to identify the impact of publications not indexed in these two citation databases.
Document types like editorial materials, letters, news items, book reviews or meeting abstracts are types of publications that focus more on disseminating scientific debates, news, opinions, or summarized information, and typically receive relatively low numbers of citations. Due to their lower citation density they are usually deemed not suitable for robust citation analysis and are often excluded from citation analyses (Waltman et al., 2011).20 Previous studies have shown that these document types have some coverage on Mendeley, and they also have higher readership counts than citations (Zahedi and Haustein, 2018). Thus, it can be argued that the impact of these document types could be better assessed with readership indicators.
Since the classification of document types in Crossref has fundamental limitations (Visser, Eck, and Waltman, 2020), the WoS document type classification was used for the set of DOIs (n = 33,868) from the years 2012–2018.21 Figure 2 (and Supplementary file 2. Table A2 in the appendix22) presents the coverage and average values of the Crossref DOIs for both WoS & Scopus citations and readership for the different document types identified in WoS. While articles and reviews are the document types that are most cited in WoS and Scopus, data papers, reviews, articles, editorial material, letters, and news items are the most saved document types on Mendeley. These results support the idea that Mendeley readership can be used to identify the readership impact of these document types.23
Social sciences and humanities are among the research fields worst covered in citation databases (Nederhof, 2006). Their low citation density makes it more difficult to study the citation impact in these fields, as well as to compare their impact with other fields. However, Mendeley readership has been observed to have a higher density than citations in these fields (Thelwall, 2015; Thelwall, 2017b; Zahedi, 2018), thus opening the possibility for more reliable and substantial studies on the impact of social sciences and humanities. In this section we show how readership indicators can offer some impact evidence when studying these fields.
Since Crossref does not count with a comprehensive classification of all documents (Visser, Eck, and Waltman, 2020), the set of DOIs from the years 2012–2018 and also covered in WoS (n = 33,868) were classified into the seven main fields of science based on NOWT classification scheme24 available from CWTS in-house database. Citation and readership indicators from CWTS in-house WoS, Scopus, and Mendeley databases were calculated and aggregated by the seven NOWT25 main fields of science (Table 3).
|Fields of science||Engineering science||Language, information, & communication||Law, arts, & humanities||Medical & life sciences||Multidisciplinary journals||Natural sciences||Social & behavioral sciences|
|Databases||No of publication||N = 3,047||N = 481||N = 899||N = 25,414||N = 360||N = 21,469||N = 4,675|
The results show that in all fields, readership density exceeds citation density. On the one hand, Social and behavioral sciences publications exhibit the highest readership density across all fields. Publications from this field on average have 44.6 readership counts on Mendeley and are cited 10.6 times in Scopus and 8.6 times in WoS. Publications from the fields Law, arts, & humanities (13.1) and Language, information & communication (23.1) also exhibit a substantially higher density of readership in contrast to their citation densities (in both Scopus and WoS). On the other hand, Engineering science is the field with the lowest readership density (11.7), although its readership density is still higher than their citation density (8.1 in WoS and 8.2 in Scopus). These results hint to the added value of readership indicators for reflecting the impact of the fields which are typically not very well represented by citation metrics.
Another important weakness in citation analysis is the need of waiting for longer periods for publications to achieve a substantial number of citations, which challenges the citation analysis of very recent publications. Although Mendeley has been observed to be a metric with a relatively slow pace (Zahedi, Costas, and Wouters, 2017), it has been reported that Mendeley readership tend to have a faster accumulation for recent publications than citations (Thelwall and Sud, 2016). This suggests that readership could play a relevant role in informing the early impact of publications (Maflahi and Thelwall, 2016). In this section this point is illustrated (Table 4 and Figure 2; see also Supplementary file 3. Table A3 in the appendix27 for the same analysis based on the whole dataset) by studying the temporal trend in the impact of WoS publications with at least one citation in Scopus/WoS or at least one reader in Mendeley from 2012–2018.
|Pub Year||P CS > 0 WoS||P CS > 0 Scopus||P RS > 0 Mendeley||TCS WoS||TCS Scopus||TRS||MCS WoS||MCS Scopus||MRS||MRS/MCS WoS||MRS/MCS Scopus|
|2012–2018||28,671 (84%)||27,469 (83%)||32,442 (96%)||416,415||388,008||103,706||14.4||14.0||31.5||2.2||2.2|
|2012||2,215 (91%)||2,130 (94%)||2,307 (97%)||52,737||55,376||93,015||23.7||26.0||40.2||1.70||1.5|
|2013||3,150 (91%)||3,140 (94%)||3,313 (97%)||68,697||71,344||144,153||21.6||22.6||43.1||2.00||1.9|
|2014||3,863 (90%)||3,867 (93%)||4,121 (97%)||79,136||79,508||171,865||20.4||20.4||41.3||2.0||2.0|
|2015||4,366 (90%)||4,363 (92%)||4,695 (98%)||77,723||74,864||176,196||17.6||17.1||37.2||2.1||2.2|
|2016||4,618 (87%)||4,607 (88%)||5,132 (97%)||58,654||51,870||157,245||12.6||11.2||30.3||2.4||2.7|
|2017||5,515 (81%)||5,299 (80%)||6,474 (97%)||50,394||39,142||167,623||9.1||7.3||25.7||2.8||3.5|
|2018||4,944 (73%)||4,064 (62%)||6,401 (95%)||29,074||15,904||121,609||5.8||3.9||18.8||3.2||4.8|
The coverage of Crossref DOIs with at least one Mendeley readership has a steady pattern from 2012 to 2017 with a small decrease (95%) in 2018. In contrast, the coverage of Crossref DOIs with at least one citation in both Scopus (from 94% in 2012 to 62% in 2018) and WoS (from 91% in 2012 to 73% in 2018) shows a decreasing pattern over time (2012–2018). In terms of average impact, Figure 3 shows overall decreasing pattern for MRS with a little increase from 2012 to 2013 and decreases from the year 2015 onwards while MCS decreases steadily over 2012 to 2018. However, MRS is indeed higher than MCS for the whole period of 2012–2018. This result is in line with those from previous studies (Maflahi and Thelwall, 2016; Thelwall and Sud, 2016; Zahedi, Costas, and Wouters, 2017; Thelwall, 2018) and suggests that Mendeley readership can work as an important source to reflect evidence of early impact of publications.
In contrast to traditional citation indicators which do not provide much information about the citers (e.g. if citations are from PhDs, Professors, etc.), Mendeley readership data includes information on the Mendeley user types, as indicated by the users themselves in their Mendeley profiles. This information provides the opportunity to identify and characterize users and potentially distinguish scholarly and non-scholarly ones.
For this purpose, Mendeley users were grouped into seven broad user types:28 PhD students, bachelor and master students, researchers (academic and non-academic institutions), professors & lecturers, librarians, other professionals, and unspecified users. The dataset of Crossref DOI with at least one reader (n = 66,536) selected for this analysis. Coverage of publication with at least one reader and Mean Readership Scores (MRS) per user type were calculated.
Table 5 shows coverage of publications and their average readership scores by different user types. The results show that the highest coverage of publications saved by students (PhD, bachelor, and master) than other users. Also, higher MRS for the sets of publications saved by Students (PhD, bachelor, and master), over those saved by researchers, professors and other user types is visible. The same pattern is observable across the six sample universities (see Supplementary file 4. Table A4 in the appendix29). The possibility to track the use of scientific publications by different user type is an advantage that citation indicators do not have.
|All user types||Professor & Lecturer||Researcher||PhD & Postgrad Student||Bachelor & Master Student||Librarian||other Professional||Unspecified users|
|Number of pub with at least one reader||66,536 (100%)||46,487 (69%)||43,617 (65%)||55,397 (83%)||53,328 (80%)||9,768 (14%)||28,506 (42%)||42,503 (63%)|
Figure 4 shows the average readership scores classified into different categories of impact based on the user types across the six sample universities. For this, average readership scores by aggregating similar user types into related categories has been calculated as: MRS Scientific (MRS by professors, researchers, and PhD & postgraduate students), MRS Educational (MRS by bachelor and master students), MRS Professional (MRS by librarians and other professionals), and MRS unspecified (MRS by unknown users). Based on this figure different types of impact of publication could be identified which provide useful insights in the various types of impact of publications and its trend across the selected universities.
Based on these results and those from previous studies (Bornmann and Haunschild, 2015; Mohammadi, et al., 2015; Thelwall, 2017a; Zahedi and van Eck, 2018), the identification of patterns of readership per user type could help in informing different types of impact (e.g., scientific, educational, or professional) of publications beyond the more academic types of impact reflected by citation indicators.
However, there are some limitations that need to be considered when using Mendeley users as a proxy for different types of impact. Firstly, Mendeley users are self-reported, this means that they choose their user type from a list of predefined options which may not always correspond to their local attribution (e.g. in some countries professors and researchers may be equivalent, like the CSIC in Spain and the universities in the country), and lecturers may have a different academic consideration depending on the country (e.g. in UK a lecturer can be equivalent to an assistant professor, or to a research associate in the US30). Secondly, users may change their status (e.g. a PhD may become a Postdoc or a researcher) without updating their Mendeley profile. Thirdly, the different users are not completely independent from each other, and what Bachelor & Master students read may be influenced by what is recommended by their lecturers, and PhD may read what is recommended by their senior colleagues (Postdocs and professors).31
Mendeley readership are considered the most prominent altmetric source with evaluative value, particularly given their large coverage of scientific publications (Costas, Zahedi, and Wouters, 2015a; Thelwall and Sud, 2016; Thelwall, 2017b), the high density levels (Mohammadi et al., 2017; Zahedi and Haustein, 2018), moderate correlation levels with citations (Zahedi, Costas, and Wouters, 2014; Costas, Zahedi, and Wouters, 2015b; Thelwall, 2018), and conceptual proximity to citation indicators (Wouters, Zahedi, and Costas, 2019; Sugimoto et al., 2017). All these interesting properties of Mendeley readership for research evaluation have been discussed in multiple scattered scientific publications (Thelwall, 2020; Thelwall, 2018), and a PhD Thesis (Zahedi, 2018), however we were still lacking a focused discussion on the specific possibilities of Mendeley readership for evaluative purposes. In this work we focus on illustrating the practical possibilities of Mendeley readership for research evaluation in aspects in which citation analysis pose more challenges. In this regard we are adopting a relatively conservative perspective, seeing Mendeley readership as a complement to citations, which is also the most common recommendation in the literature (Haustein, Bowman, and Costas, 2016; Thelwall, 2020). There have been also discussions about the possibility of considering readership as another type of “currency of science” that could be discussed on par with citations(Costas, Perianes-Rodríguez, and Ruiz-Castillo, 2017), but we are not discussing this approach in this study.
Possibilities of Mendeley readership for research evaluation can be discussed as follows:
Mendeley readership can represent an important source of evidence of the impact of publications not indexed in mainstream citation databases (WoS or Scopus). This is particularly relevant for publications from the Global South and developing countries, since they usually have a lower coverage in most bibliometric databases (Alperin, 2013), with a substantial share of the output from these countries underrepresented in citation databases (Zahedi, & Costas, 2017). In a recent study it has been observed that publications from the Global South countries (Africa and South America) have relatively higher levels of readership in contrast to citation metrics (Costas, Zahedi, and Alperin, 2019), which reinforces the idea that Mendeley readership can play a relevant role in providing impact evidence for non-indexed publications. In this report the higher coverage of University of Sao Paulo (USP) publications on Mendeley reinforces this relevance of Mendeley readership for studying the impact of Global-South countries and non-indexed publications.
A higher density of readership (over citation metrics) is typically observed for social sciences and humanities publications. In citation databases, citation impact in these disciplines is largely affected by the lower coverage of books and by the more national or local orientation of the published research (typically in other languages than English) (Moed, 2006). This study illustrates how Mendeley readership can be a valuable source to inform the impact analysis of the fields that are not well represented with citation indicators.
Reviews and articles are the most prevalent document types in citation analysis, since these document types capture the most important scientific findings. Usually other document types (e.g. editorial material, letters, meeting abstracts, news items, data papers, etc.) are excluded from citation analysis because they are deemed not to represent the same type of scientific contribution than articles and reviews. However, there may be situations in which the impact analysis of these other document types is necessary (e.g. a journal that wants to analyze the impact of its editorials or news items; or research teams that also want to evaluate the impact of those types of outputs). In such cases, citations are not very helpful given the low citedness of these document types. However, we have illustrated how Mendeley readership have a higher coverage and higher readership values for some of these document types (e.g., letters, data papers or editorial materials), supporting the idea of a strong relevance of Mendeley readership for the evaluation of these outputs.
We have illustrated how readership scores are more prevalent than citations in recent publications and hence they could work as an early indicators of research impact (Thelwall and Sud, 2016; Thelwall, 2018). These findings together with the fact that Mendeley readership are available openly and that they can earlier signal highly cited publications (Zahedi, Costas, and Wouters, 2016), highlight the value of Mendeley as a tool for revealing early impact of publications, particularly when substantial number of citations haven’t yet been accrued.
We argue and illustrate how readership scores from non-academic users (such as students, librarians, or professionals) could reflect other types of impact, such as educational or professional. This more fine-grained possibility of studying the different types of users interacting with the publications on Mendeley, is something that is not possible with the most common citation indicators. This suggests the potential of Mendeley as a relevant source for expanding the notion of impact beyond the more academic impact captured by citations, although self-reported nature of Mendeley needs to be considered.
Below we summarize some of the best practical recommendations on how Mendeley readership can be used for assessment purposes:
There are still important open questions regarding the practical and conceptual limitations of Mendeley readership for research evaluation. Moreover, it is important to continue developing the concept of readership, which as currently operationalized as Mendeley only captures the act of saving a publication by a user in her library; however more advanced metrics, capturing different forms of engagement of users with the publications (e.g., the act of opening a document, scrolling through it, highlighting the text, commenting it, etc.) could be also captured on Mendeley (or any other online reference manager); thus opening the door to more advanced possibilities for studying different uses and impacts of scientific publications.
The underlying Crossref and Mendeley metrics data that support the findings of this study are openly available in Figshare at https://doi.org/10.6084/m9.figshare.12824201.
The additional files for this article can be found as follows:Table A1
Coverage and share of Crossref DOIs (n = 79,416) across WoS, Scopus, & Mendeley databases. DOI: https://doi.org/10.29024/sar.20.s1Table A2
Mendeley readership, WoS and Scopus citation coverage and density of Crossref DOIs 2012–2018 (n = 33,868) per document type. DOI: https://doi.org/10.29024/sar.20.s2Table A3
Distributions of MRS and MCS indicators of the Crossref DOIs. DOI: https://doi.org/10.29024/sar.20.s3Table A4
Crossref DOIs by user types across the six sample universities. DOI: https://doi.org/10.29024/sar.20.s4
3There have also been discussions about the possibility of this indicator as being another type of “currency of science” (Costas, Perianes-Rodríguez, and Ruiz-Castillo, 2017), which depending on the context could play an evaluative role on par with citation metrics.
6Crossref (http://www.crossref.org) is an official DOI registration agency. It facilitates the links between distributed content hosted at different sites (e.g. publishers’ websites) and provides basic metadata about the records registered.
7Journal publications, scholarly books, and conference proceedings represent the largest content in Crossref. The basic metadata in Crossref includes title, publication dates, authors, journal title, conference name, volume/issue number, author’s affiliations and ORCID, abstracts and links to full text, funding metadata, license metadata, list of references, clinical trial numbers, figures and supplementary materials (Hendricks, et al., 2020). Affiliation data however is based on the input of the different member organizations – who register metadata along with the digital object identifier for their registered content – therefore there is no affiliation data for all Crossref records.
10This matching with WoS is motivated by the lack of reliable metadata about classifications, document types, etc. in Crossref (Visser, Eck, and Waltman, 2020).
11Mean Citation Score is the ratio of the total number of citations (TCS) divided by the total number of publications (P) of a given unit, thus MCS = TCS/P. Mean Readership Score is the ratio of the total number of readership (TRS) divided by the total number of publications (P) of a given unit. MCS WoS = TCS WoS/P WoS; MCS Scopus = TCS Scopus/P Scopus; MRS = TRS/P Mendeley.
14In principle, it is possible to calculate the MRS values including also those publications that are not covered in Mendeley, assuming then a readership value of zero. However, since such an assumption cannot be applied for the publications not covered in WoS and Scopus (they may be cited but not tracked by these databases), in order to be consistent in this study we calculate MRS only for those publications covered in Mendeley. The same approach has been adopted for MCS Scopus and MCS WoS.
15We are aware that although these publications are not indexed in WoS or Scopus it would be technically possible to calculate their citation impact in those databases. However, conceptually speaking they are still affected by the indexing selection criteria of these databases (e.g. publications from topics not well covered in the databases would be at a disadvantage – Moed, 2006; while such a conceptual limitation does not exist on Mendeley, since all publications have in principle the same possibilities of being saved on Mendeley (except for technical issues – Zahedi and Costas, 2018 – or the geographical limitations previously discussed).
18Many of these document types are also typically not peer reviewed, which can be another reason to exclude them from citation analysis. However, there could still be cases in which the analysis of the impact of these document types is relevant, e.g. a scientific journal interested in evaluating the impact of its editorial or news material, or a researcher or university department interested in discussing their impact also on these type of documents; in such cases Mendeley readership could provide relevant support evidence.
20Out of 79,416 DOIs, a total of 45,548 DOIs were excluded from this analysis, of which 12,913 DOIs in Crossref and in WoS with a publication year (in WoS) outside the period 2012–2018; and for 32,635 DOIs the publication year was not known since these DOIs were not covered in WoS.
22Other document types such as poetry, software review and art exhibit review were excluded from the analysis due to their very low coverage (less than 3 in Mendeley and no coverage in WoS and Scopus) across all databases (see Supplementary file 2. Table A2 in the appendix – https://doi.org/10.6084/m9.figshare.12824201).
24NOWT stands for Nederlands Observatorium voor Wetenschap en Technologie (Dutch Science & Technology Observatory). NOWT is a field classification system on top of the WoS subject categories which includes 7 broad disciplines and 35 constituent research areas, for more information see here: https://www.cwts.nl/pdf/nowt_classification_sc.pdf.
27Mendeley readership statistics for users includes 15 different user categories. Here we decided to classify similar user types into 7 broad related categories as follows: Professor & Lecturer = (‘Assistant Professor’, ‘Associate Professor’, ‘Professor’, ‘Professor > Associate Professor’, ‘Lecturer’, ‘Senior Lecturer’, ‘Lecturer > Senior Lecturer’); Researcher = (‘Post Doc’, ‘Researcher’, ‘Researcher (at an Academic Institution)’, ‘Researcher (at a non-Academic Institution)’); PhD_postgrad_student = (‘Doctoral Student’, ‘Ph.D. Student’, ‘Student (Postgraduate)’, ‘Student > Doctoral Student’, ‘Student > Ph. D. Student’, ‘Student > Postgraduate’); Bachelor_master_student = (‘Student (Bachelor)’, ‘Student (Master)’, ‘Student > Bachelor’, ‘Student > Master’); Professional = (‘Other Professional’); librarian(‘librarian’); Unspecified = (‘Unspecified’).
30In a previous study it was shown thematic differences across Mendeley user types (Zahedi & van Eck, 2018), suggesting indeed different patterns in what is saved by each user type, however the limitation still needs to be observed, at least from a theoretical point of view.
The authors have no competing interests to declare.
Alperin, J. P. (2013). Ask not what altmetrics can do for you, but what altmetrics can do for developing countries. Bulletin of the American Society for Information Science and Technology, 39(4), 18–21. DOI: https://doi.org/10.1002/bult.2013.1720390407
Alperin, J. P. (2015). Geographic variation in social media metrics: An analysis of Latin American journal articles. Aslib Journal of Information Management, 67(3), 289–304. DOI: https://doi.org/10.1108/AJIM-12-2014-0176
Archambault, É., & Larivière, V. (2006). The limits of bibliometrics for the analysis of the social sciences and humanities literature. International Social Science Council: World social sciences report 2010: Knowledge divides (pp. 251–254). Paris: UNESCO.
Bornmann, L., & Haunschild, R. (2016). Normalization of Mendeley reader impact on the reader-and paper-side: A comparison of the mean discipline normalized reader score (MDNRS) with the mean normalized reader score (MNRS) and bare reader counts. Journal of informetrics, 10(3), 776–788. DOI: https://doi.org/10.1016/j.joi.2016.04.015
Costas, R., Zahedi, Z., & Wouters, P. (2015a). The thematic orientation of publications mentioned on social media. Aslib Journal of Information Management. Aslib Journal of Information Management, 67(3), 260–288. DOI: https://doi.org/10.1108/AJIM-12-2014-0173
Costas, R., Zahedi, Z., & Wouters, P. (2015b). Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. Journal of the Association for Information Science and Technology, 66(10), 2003–2019. DOI: https://doi.org/10.1002/asi.23309
Costas, R., Zahedi, Z., & Alperin. (2019). Global country-level patterns of Mendeley readership performance compared to citation performance: does Mendeley provide a different picture on the impact of scientific publications across countries? In Proceedings of the 17th INTERNATIONAL CONFERENCE ON SCIENTOMETRICS & INFORMETRICS ISSI2019 with a Special STI Indicators Conference Track 2–5 September 2019 (pp. 1195–1200). Italy: Sapienza University of Rome.
Costas, R., Perianes-Rodríguez, A., & Ruiz-Castillo, J. (2017), On the quest for currencies of science: Field “exchange rates” for citations and Mendeley readership. Aslib Journal of Information Management, 69(5), 557–575. DOI: https://doi.org/10.1108/AJIM-01-2017-0023
Fairclough, R., & Thelwall, M. (2015). National research impact indicators from Mendeley readers. Journal of informetrics, 9(4), 845–859. DOI: https://doi.org/10.1016/j.joi.2015.08.003
Haunschild, R., & Bornmann, L. (2016). Normalization of Mendeley reader counts for impact assessment. Journal of informetrics, 10(1), 62–73. DOI: https://doi.org/10.1016/j.joi.2015.11.003
Haustein, S. (2016). Grand challenges in altmetrics: heterogeneity, data quality and dependencies. Scientometrics, 108(1), 413–423. DOI: https://doi.org/10.1007/s11192-016-1910-9
Haustein, S., Bowman, T. D., & Costas, R. (2016). Interpreting Altmetrics: Viewing Acts on Social Media through the Lens of Citation and Social Theories. In C. R. Sugimoto (Ed.), Theories of Informetrics and Scholarly Communication (pp. 372–406). Berlin, Boston: De Gruyter. DOI: https://doi.org/10.1515/9783110308464-022
Hendricks, G., Tkaczyk, D., Lin, J., & Feeney, P. (2020). Crossref: The sustainable source of community-owned scholarly metadata. Quantitative Science Studies, 1(1), 414–427. DOI: https://doi.org/10.1162/qss_a_00022
Hicks, D., Wouters, P., Waltman, L., De Rijcke, S., & Rafols, I. (2015). Bibliometrics: The Leiden Manifesto for research metrics. Nature, 520(7548), 429–431. DOI: https://doi.org/10.1038/520429a
Maflahi, N., & Thelwall, M. (2016). When are readership counts as useful as citation counts? Scopus versus Mendeley for LIS journals. Journal of the Association for Information Science and Technology, 67(1), 191–199. DOI: https://doi.org/10.1002/asi.23369
Mohammadi, E., Thelwall, M., Haustein, S., & Larivière, V. (2015). Who reads research articles? An altmetrics analysis of Mendeley user categories. Journal of the Association for Information Science and Technology, 66(9), 1832–1846. DOI: https://doi.org/10.1002/asi.23286
Ortega, J. L. (2018). Reliability and accuracy of altmetric providers: A comparison among Altmetric.com, PlumX and Crossref Event Data. Scientometrics, 116, 2123–2138. DOI: https://doi.org/10.1007/s11192-018-2838-z
Priem, J., Taraborelli, D., Groth, P., & Neylon, C. (2010). Altmetrics: a manifesto. Retrieved from http://altmetrics.org/manifesto/
Robinson-García, N., Torres-Salinas, D., Zahedi, Z., & Costas, R. (2014). New data, new possibilities: Exploring the insides of Altmetric.com. arXiv preprint arXiv:1408.0135. DOI: https://doi.org/10.3145/epi.2014.jul.03
Sugimoto, C. R., Work, S., Larivière, V., & Haustein, S. (2017). Scholarly use of social media and altmetrics: A review of the literature. Journal of the Association for Information Science and Technology, 68(9), 2037–2062. DOI: https://doi.org/10.1002/asi.23833
Thelwall, M. (2015). Why Do Papers Have Many Mendeley Readers but Few Scopus-Indexed Citations and Vice Versa? Journal of Librarianship & Information Science, 49(2), 144–151. DOI: https://doi.org/10.1177/0961000615594867
Thelwall, M. (2017a). Does Mendeley provide evidence of the educational value of journal articles? Learned Publishing, 30(2), 107–113. DOI: https://doi.org/10.1002/leap.1076
Thelwall, M. (2017b). Are Mendeley reader counts useful impact indicators in all fields? Scientometrics, 113, 1721–1731 (2017). DOI: https://doi.org/10.1007/s11192-017-2557-x
Thelwall, M. (2018). Early Mendeley readers correlate with later citation counts. Scientometrics, 115(3), 1231–1240. DOI: https://doi.org/10.1007/s11192-018-2715-9
Thelwall, M. (2020). The Pros and Cons of the Use of Altmetrics in Research Assessment. Scholarly Assessment Reports, 2(1), 2. DOI: https://doi.org/10.29024/sar.10
Thelwall, M., & Maflahi, N. (2015). Are scholarly articles disproportionately read in their own country? An analysis of Mendeley readers. Journal of the Association for Information Science and Technology, 66(6), 1124–1135. DOI: https://doi.org/10.1002/asi.23252
Thelwall, M., & Sud, P. (2016). Mendeley readership counts: An investigation of temporal and disciplinary differences. Journal of the Association for Information Science and Technology, 67(12), 3036–3050. DOI: https://doi.org/10.1002/asi.23559
Torres-Salinas, D., Robinson-Garcia, N., Campanario, J. M., & López-Cózar, E. D. (2013). Coverage, field specialization and the impact of scientific publishers indexed in the Book Citation Index. Online Information Review, 38(1), 24–42. DOI: https://doi.org/10.1108/OIR-10-2012-0169
Van Eck, N., Waltman, L., Larivier, V., & Sugimotto, C. (2018, January 17). Crossref as a new source of citation data: A comparison with Web of Science and Scopus, Leiden Madtrics, https://www.cwts.nl/blog?article=n-r2s234
Van Raan, A. F. J., Van Leeuwen, T. N., & Visser, M. S. (2011). Severe language effect in university rankings: Particularly Germany and France are wronged in citation-based rankings. Scientometrics, 88(2), 495–498. DOI: https://doi.org/10.1007/s11192-011-0382-1
Visser, M., Eck, N. J., & Waltman, L. (2020). Large-scale comparison of bibliographic data sources: Scopus, Web of Science, Dimensions, Crossref, and Microsoft Academic. https://arxiv.org/abs/2005.10732
Waltman, L., van Eck, N. J., van Leeuwen, T. N., Visser, M. S., & van Raan, A. F. J. (2011). Towards a new crown indicator: An empirical analysis. Scientometrics, 87(3), 467–481. DOI: https://doi.org/10.1007/s11192-011-0354-5
Wilsdon, J., Allen, L., Belfiore, E., Campbell, P., Curry, S., Hill, S., … Hill, J. (2015). The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management. DOI: https://doi.org/10.4135/9781473978782
Wilsdon, J. R., & Al, E. (2017). Next-generation metrics: Responsible metrics and evaluation for open. Brussels. DOI: https://doi.org/10.4324/9781315206455-15
Wouters, P., & Costas, R. (2012). Users, narcissism and control – tracking the impact of scholarly publications in the 21st century. Utrecht: SURF foundation. Retrieved from http://www.surf.nl/nl/publicaties/Documents/Users narcissism and control.pdf
Wouters, P., Zahedi, Z., & Costas, R. (2019). Social media metrics for new research evaluation. In W. Glänzel, H. F. Moed, U. Schmoch & M. Thelwall (Eds.), Handbook of Quantitative Science and Technology Research (pp. 687–709). Springer. DOI: https://doi.org/10.1007/978-3-030-02511-3_26
Zahedi, Z. (12 December 2018). Understanding the value of social media metrics for research evaluation (PhD thesis). Centre for Science & Technology Studies, Social and Behavioural Sciences, Leiden University. Supervisor(s) and Co-supervisor(s): Wouters, P., Costas, R.
Zahedi, Z., Bowman, T., & Haustein, S. (2014), Exploring data quality and retrieval strategies for Mendeley reader counts. In Metrics14: ASIS&T Workshop on Informetric and Scientometric Research. Available: https://www.asis.org/SIG/SIGMET/data/uploads/sigmet2014/zahedi.pdf
Zahedi, Z., & Costas, R. (2017). How visible are the research of different countries on WoS and twitter? An analysis of global vs. local reach of WoS publications on Twitter. In Proceedings of the 16th of International Conference on Scientometrics and Informetrics. Wuhan, China.
Zahedi, Z., & Costas, R. (2018). General discussion of data quality challenges in social media metrics: Extensive comparison of four major altmetric data aggregators. PLoS ONE, 13(5), e0197326. DOI: https://doi.org/10.1371/journal.pone.0197326
Zahedi, Z., & Haustein, S. (2018). On the relationships between bibliographic characteristics of scientific documents and citation and Mendeley readership counts: A large-scale analysis of Web of Science publications. Journal of Informetrics, 12(1), 191–202. DOI: https://doi.org/10.1016/j.joi.2017.12.005
Zahedi, Z., & van Eck, N. J. (2018). Exploring Topics of Interest of Mendeley Users. Journal of Altmetrics, 1(1), 5. DOI: https://doi.org/10.29024/joa.7