Start Submission Become a Reviewer

Reading: Performance-based Institutional Research Funding in Flanders, Belgium

Download

A- A+
Alt. Display

Commissioned Reports

Performance-based Institutional Research Funding in Flanders, Belgium

Author:

Marc Luwel

Centre for Science and Technology Studies (CWTS), University Leiden, NL
X close

Abstract

Research and innovation is one of Flanders’ priorities and over the last three decades its public funding has strongly increased. Universities are key actors in this strategy. They have a large autonomy and receive a substantial share of additional R&D expenditures as lump sum funding.

The Flemish authorities use quantitative indicators to allocate these lump sums to the universities. The funding formulae take into account each institution’s size, its research performance and, if relevant, its valorization activities. Also significant is the realization of governmental priorities, such as mobility and diversity of the academic staff. This paper describes the development of the Flemish university funding model, analyses its weaknesses and its strengths, and compares it with nine national metrics-based research performance funding systems.

Policy highlights

  • Flanders, like many other regions and nations, has adopted performance-based research funding systems (PRFSs) to improve and provide accountability for its science and innovation system. The Flemish PRFS criteria have evolved considerably over the last three decades, and due to competition between universities and a consensus model of political decision making, the funding formula is comparatively complex.
  • Building on an historical background of lump sum payments to universities, supporting both education and research, special supplementary funds for blue-sky research (BOF) and for strategic applied research, innovation, and outreach activities (IOF) were introduced by the Flemish government in 1994 and 2004, respectively. The “three-legged stool” funding mechanism for research in Flanders is unique within Europe.
  • The introduction of publication and citation metrics in 2003 changed the character of the Flemish PRFS considerably, previously focused on measures of a university’s size in terms of students, degrees granted, and previous funding. Humanities and social sciences (HSS) research metrics were considered only after first adoption of bibliometric methods traditionally deployed for assessment of the natural and life sciences, but this is not unique to Flanders. HSS research has been addressed in Flanders through the creation of a special database to supplement standard citation indexes, the VABB-SHW. This special resource is a strong point of the Flemish PRFS, and other nations should appreciate the value of increased coverage of the HSS literature.
  • The PRFS in Flanders differs from that of nine European nations in several ways: analysis is undertaken annually, it includes a diversity and mobility measure (2006), and one of interdisciplinary will soon be added.
  • Flanders, like other nations using PRFSs, has generally seen increased research output and intellectual property activities after its introduction; however, in some nations the increase began before a PRFS was implemented. In almost all cases, demonstrating a causal link between a PRFS and increased output is difficult since R&D investments and the number of researchers have also increased for a variety of reasons. This is a cautionary note in not over-interpreting the effects or effectiveness of a PRFS.
  • Flanders now exhibits one of the highest levels among European nations of research funding allocations determined by a PRFS: 50% of BOF-funding and 75% of IOF-funding. By employing such a metrics-heavy scheme, Flanders is a good candidate for a detailed study of unintended consequences of a PRFS, at the national, institutional, and research group and individual researcher level. This remains a large gap in our understanding of the use of a PRFS.
How to Cite: Luwel, M., 2021. Performance-based Institutional Research Funding in Flanders, Belgium. Scholarly Assessment Reports, 3(1), p.3. DOI: http://doi.org/10.29024/sar.29
82
Views
28
Downloads
  Published on 10 Jun 2021
 Accepted on 13 Apr 2021            Submitted on 04 Oct 2020

1. Introduction

In most Western European countries universities develop an integrated strategy around their triple mission of teaching, research and service to society. As a result, different profiles are emerging: some universities are striving for internationally peer-recognized excellence in all their activities, while others are becoming more specialized or strong partners in local or regional development. But even Europe’s top universities remain deeply rooted in national education, research, and innovation systems.

Since the 1980s, public authorities in most Western countries have adopted the New Public Management (NPM) concept characterized by the ‘Economy, Efficiency, Effectiveness’ principle (Kettl, 1997). Greater autonomy of universities was counterbalanced by more emphasis on accountability. At the same time, since research carried out at universities plays a pivotal role in development of today’s knowledge economy, governments started to pay more attention to their universities’ research performance, and novel policy instruments were introduced to stimulate it.

The emergence of performance-based research funding systems (PRFSs) for universities is deeply rooted in the NPM philosophy (Herbst, 2007). Based on Hicks (2012), Zacharewicz et al. (2019) give a definition of a PRFS: Set up by national or regional authorities, it is based on the ex-post evaluation of output and impact of research carried out at universities; the results of the assessment are translated into a funding formula to allocate part of the institutional funding to the universities; and, the attribution of funding based on ex-ante evaluation and selection of research proposals and allocation of funding based on input parameters such as the number of researchers are not considered as a PRFS.

The PRFS’s objectives are translated into criteria. Initially the focus was on publications in journals indexed in two international bibliographic databases: the Web of Science (WoS), created by the Institute for Scientific Information (ISI) and currently managed by Clarivate and SCOPUS, Elsevier’s international bibliographic database. Both are citation indexes. Progressively other publication types were taken into account as well as differences in the publication culture between disciplines. Some models use a definition of impact beyond academia and include criteria for the societal and economic relevance of research.

For the assessment of the criteria three approaches are used: a metrics-based system, a (metrics informed) peer review-based system or a combination of both.

This contribution describes the PRFS of Flemish universities. In Section 2, after presenting Flanders in Belgium’s complicated institutional setting, its higher education sector and its science and innovation policy are briefly described. One of its salient characteristics is certainly the introduction and, over a period of twenty-five years, the development of the PRFS (Section 3). In the next section Flanders’ PRFS is compared with the metrics-based institutional funding model implemented in nine European countries and their (un)intended effects are briefly discussed. In the concluding section, a critical assessment of the PRFS in the broader framework of Flanders’ higher education policy is made which can also be relevant for other countries.

2. Setting the scene

2.1 Belgium: from a unitary state to a federation with uncommon characteristics

In 1830 a revolt led to the secession of the Southern Provinces from the Netherlands and the establishment of Belgium as an independent state. It unites Flanders, the Dutch-speaking northern and Wallonia, the French speaking southern region, with Brussels as its bilingual capital. In 1920 the East Cantons with a German speaking population were annexed by Belgium following Germany’s defeat in World War I and the subsequent Treaty of Versailles.

Belgium’s population is 11.6 million: 6.7 million are living in Flanders, 3.7 million in Wallonia including the nearly 80.000 German speaking inhabitants and 1.2 million in Brussels.

Tensions between the Dutch-speaking and French-speaking population runs like a thread through Belgium’s history. Often presented as linguistic quarrels, its root causes are cultural, social and economic. To keep these conflicts manageable the centralized state was gradually transformed into a federation with uncommon characteristics. (Algoed and van Nieuwenhove, 2011; Van den Wijngaert, 2011). Political power resides at three levels: the federal state, the three linguistic communities and the three regions. This complex setting is mainly due to the bilingual character of the Brussels-Capital Region that is included in both the Flemish and French communities.

A peculiar aspect of Belgium’s federal state structure is that not only the federal level but also the regions and the communities have treaty-making power for matters within their competences.

The division of competences between the three levels is not always clear-cut. To avoid frictions and even stalemate, consultative mechanisms are created and the final say lies with the Constitutional Court of Belgium.

The devolution of competences from the federal state is not finished, as regions and communities demand more autonomy to develop tailor-made solutions for their socio-economic needs.

2.2 Flanders’ higher education landscape

Since 1988, Flanders has been responsible for all aspects of higher education, except for the setting of minimum requirements for awarding qualifications that remain a federal competence.

Today, Flanders has a binary educational system. It has five universities: three comprehensive universities (Ghent University (UGent), KU Leuven, Vrije Universiteit Brussel (VUB)), the University of Antwerp (UAntwerp), and the Hasselt University (UHasselt). The latter two offer only a limited number of educational programs. Universities deliver academic bachelor’s and master’s degrees and PhD degrees; the ius promovendi is the exclusive competence of the universities.1

There are thirteen Flemish university colleges. The university colleges organize professional bachelor’s and associate degree programs. The latter is situated between secondary education and professional bachelor’s programs (European Qualification Framework Level 5). Unlike the other academic programs, the artistic programs are not integrated with the universities but within the university colleges in separate schools of arts. Research in these schools is organized jointly with the universities.

Each university college is linked to a university. The networks of university-university colleges were set up in response to the Bologna Declaration to transform the educational programs from the existing ternary degree system into the bachelor–master structure (Huisman and Mampaey, 2017). The difference between professional and academic bachelor’s programs lies in their orientation. The former is practice-oriented and focuses on the competences necessary for specific professions, whereas the latter provides stronger theoretical foundations for further studies at a master’s level. The Flemish decree (i.e. regional law) stipulates that universities and university-colleges not only provide education and training but also carry out research and provide services to the community. The scope of the research activities follows the orientation of the educational programs.

To assure that all higher education degree programs meet basic quality standards — a necessary condition for the recognition of degrees and titles, and wherever applicable, for the public funding of programs — the Dutch and Flemish Parliament has set up the Accreditation Organization of the Netherlands and Flanders (NVAO).

2.3 Flanders’ science and innovation policy

In contrast to most member countries of the Organization for Economic Co-operation and Development (OECD), during the 20th century no large public institutes for applied research were set up in Belgium to support the development of leading technologies and strengthen the industrial base. A center for nuclear research established in 1952 was the only exception. In this setting, the universities became the natural partners of industry in research and innovation activities.

Another characteristic of Belgium’s research and innovation landscape were the rather low public expenditures for research and experimental development (R&D); in the mid-1980s, 1.5% of the GDP was spent on R&D; the private sector contributed roughly 2/3 to this expenditure, whereas the public sector contributed 1/3.

Successive constitutional revisions transferred the legislative power and the public funding for education and, for the large part, science and innovation, to the regions and communities. The 1988–1989 revision was a pivotal moment, with communities receiving the lump sum funding of the universities and nearly all funding for curiosity-driven research. However, it should be mentioned that some competences remain at the federal level. Even today, the universities are still funded by the federal government to carry out research projects and to participate in initiatives it deems necessary to exercise its competences. A detailed description of the partition of the competences related to R&D between the federal level and the regions and communities is beyond the scope of this paper.

With the institutional reform of 1988–1989, Flanders received both the legal competences and the financial means to develop its own science and innovation policy. Early on, the Flemish authorities recognized the critical importance of research and innovation for the region’s economic development and its population’s well-being. At the beginning of the 1990s, a decade before the European Union (EU) in 2002 set the objective of investing 3% of its GDP in R&D, the Flemish government started to systematically increase expenditures on science and innovation. For example, between 1993 and 2019, the outlay for R&D in the Flemish budget increased six fold from 327 to 1952 million euro (EWI, 2020).

To allocate the resources, a limited number of strategic priorities were set, appropriate funding instruments and agencies were established, performance was monitored, and, if necessary, remedial measures were taken.

2.4 Lump sum funding of the Flemish universities at the turn of the century: changing formulae but persisting problems

The federal law on the universities of 1971 stipulated that each Belgian university received a lump sum based on its weighed number of students. Although slightly modified over the years, the funding formula was still used at the end of the ’80s to calculate each university’s lump sum.

It is important to emphasize that the salaries of the tenured staff were paid out of the lump sum and that there were only a few hundred other permanent positions for researchers at Belgian universities. Besides covering the cost of teaching and the general operating cost of the universities, the lump sum also provided substantial financial support for research; project-based research funding was still in its infancy.

Due to the oil shock and subsequent lax budgetary policy, successive Belgian governments were confronted with a severe financial crisis in the 1970s and 1980s. As a direct consequence, the annual adjustments of the funding formula’s elements were very often considerably below the real increase in the salary and operating costs.

Around 1985, the federal authorities became aware that R&D was key to combat the economic and financial crisis they faced. A plan was drawn up, and one of its components was the allocation of a small additional grant to each university to fund research activities. This grant had to mediate, at least partially, the impact of the erosion of the lump sum on the universities’ research capacities (Vandersmissen et al., 2015).

In 1991, the Flemish parliament used its recently obtained legislative authority to change the universities’ funding formula. The 1991 decree on the Flemish universities implemented a dual-track approach: modernizing regulation and introducing a new funding model. It not only considerably increased the autonomy of the universities and simplified the rules and regulations but also placed more emphasis on the institutions’ public accountability. The universities were given the legal obligation to systematically review all their activities and report the outcomes to the Flemish government. To comply with this regulation, the universities decided among other things to jointly set up a system of quality management at the level of the study programs. There was no similar initiative for their research activities.

Apart from substantial simplifications, the main difference between the new funding formula and the 1971 law was the considerable reduction of the impact of the evolution of the number of students: Only 50% of the total lump sum allocated to the Flemish universities still depended on the weighted number of students. Reducing the weight of the number of students was motivated by the two-fold conviction of the policymakers and the universities’ academic authorities to hedge, at least partially, the risk of decreasing student enrollment and to improve the collaborative environment among the Flemish universities with objectives to create a disincentive for aggressive student recruitment, to stimulate the introduction of inter-university study programs, and to foster the winding-down of degree programs with very low numbers of students.

The latter remained largely wishful thinking and the first assumption turned out to be erroneous. In the two decades following the introduction of this funding model the total student population at the Flemish universities increased substantially (Table 1). This trend was not uniform over the universities, with the strongest increase being witnessed at the UGent. However, due to the funding formula, the additional weighed students were only funded at a reduced rate of about 50%, resulting in a growing disadvantage to the UGent compared to the other Flemish universities.

Table 1

Evolution of the number of students enrolled at the Flemish universities in the initial bachelor’s and master’s programs. The breakpoint at the academic year 1999–2000 is due to a change in the registration; from that academic year onwards students following tailored made academic programs are only counted once (Source: VRIND – Vlaamse Regionale Indicatoren: https://www.vlaanderen.be/publicaties/vrind-20yy-vlaamse-regionale-indicatoren).


ACADEMIC YEAR STUDENTS FLEMISH UNIVERSITIES

1993–1994 50887

1994–1995 52777

1995–1996 55298

1996–1997 56416

1997–1998 56902

1998–1999 57170

1999–2000 56740

2000–2001 56118

2001–2002 56693

2002–2003 56839

2003–2004 56839

2004–2005 57005

2005–2006 59172

2006–2007 60866

2007–2008 64372

2008–2009 68601

2009–2010 72355

2010–2011 75063

2011–2012 77135

2012–2013 78717

On top of that, although less severe than in the 1980s, the annual adjustment of the lump sum below the evolution of salary and wage costs and price inflation slowly but certainly impeded the development of the Flemish universities. As already indicated, this is a structural problem originating in the 1970s and persisting still today, notwithstanding changes in the funding formula.

During the first decade of this century, the Flemish higher education landscape was profoundly transformed. Parallel with the implementation of the Bologna Declaration (1999) and the establishment of the associations of university-university colleges, in 2000, the Flemish public authorities decided to increase the lump sum allocated to the Flemish universities by more than 30 million euros over a four-year period; a substantial part was used to reduce the perverse effects of the 1991 decree. To create financial stability during this period of major institutional and educational reform, the lump sum was decoupled from the evolving number of students. As the preparation of a new funding formula for the higher education sector turned out to be more time-consuming than initially projected, the decoupling was prolonged until 2007.

The additional funding for the total lump sum only partially corrected the imbalance between the universities and the erosion of the university funding due to the insufficient correction for the increase in the personnel and operation costs. The former had the largest impact on the UGent. Claiming unequal treatment under the law, this university sued the Flemish authorities, and the Constitutional Court ruled that the Flemish government had to take action. Over the period 2005–2009, it was decided to increase the lump sum allocated to this university by 13 million euros.

In the period 2003–2006, additional funding of 15.5 million euros was also allocated to the academic bachelor’s and master’s programs organized by the university colleges to develop the integration of research and teaching in a stimulating partnership with the associated university.

3. Performance-based research funding of the Flemish universities

Starting with the allocation of funding for blue-sky research, performance-based formulae were also used for applied research and the research component in the lump sum.

3.1 Increasing research funding and greater autonomy for the Flemish universities versus growing emphasis on research performance

Since 1991, universities have been strategic partners in the conception and implementation of the Flemish ambitious science and innovation policy agenda. They received a large share of the additional public R&D funding and legislation was implemented to stimulate the collaboration between universities and firms, the valorization of academic know-how and the creation of spin-offs.

Building on the above-mentioned initiative of the federal government at the end of the 1980s to provide additional research funding to universities, the Flemish authorities created a specific funding allocation for blue-sky research at the universities. The 1994 regulation stipulated that each university had to set up a dedicated internal funding mechanism for blue-sky research and complement the governmental allocation with a minimum percentage out of its own resources: the Special Research Fund or BOF under its Dutch acronym.

Annually, the Flemish parliament allocated a global sum to the BOF. Each university received a fraction of the global sum based on a formula taking into account the institution’s share in three input parameters: the size of the university’s lump sum, the number of master’s degrees it delivered, and the number of granted PhD degrees. The latter can also be considered as an intermediate parameter, as it is a measure of a university’s research capacity, but at the same time, it provides information on the training of the next generation of researchers. For each of these parameters a university’s fraction in the Flemish universities’ total value is calculated. In the partition formula to each parameter a weight factor was assigned, a reference period taken into account and averages made (Table 2).

Table 2

The parameters used in the initial BOF partition formula and in the different revisions; the year of the revision is given in the column ‘Year’. The parameters are grouped in clusters. The weight of each cluster (Weight_1) and of each parameter (Weight_2) are given at end of the transition period between two revisions, except for the 2019 revision for which the year 2019 is used. Only from 2024 onwards the parameter for Interdisciplinary research will be taken into account.


YEAR CLUSTER WEIGHT_1 PARAMETER WEIGHT_2

1994 A 1.00 Weighted master degrees 0.50

Weighted PhD degrees 0.30

Lump sum 0.20


2003 A 0.70 Weighted initial master degrees 0.35

Weighted PhD degrees 0.50

Lump sum 0.15

B 0.30 SCIE publications 0.15

SCIE citations 0.15


2006 A 0.70 Weighed bachelor and initial master degrees 0.35

Weighted PhD degrees 0.50

Lump Sum 0.09

Mobility and diversity 0.06

B 0.30 SCIE publications 0.15

SCIE citations 0.15


2008 A 0.64 Weighed bachelor and initial master degrees 0.25

Combination raw and weighted PhD degrees 0.35

Tenured faculty and researchers 0.00

Mobility and diversity 0.04

B 0.36 Publications with weighted categories: 0.18

Publications in SCIE and SSCI with(out) IF

Publications in AHCI

Proceedings in STP/ISI Proceedings-Index

Proceedings in SSHP/ISI Proceedings-index

Publications in the VABB-SHW

Citations 0.18


2012 A minimal fraction of global BOF sum was allocated each of 3 universities (UAntwerp, UHasselt, VUB) but repartition formula remained unchanged


20191 A 0.50 Each university receives fix fraction total sum 0.50

B 0.225 Publications with weighted categories: 0.15

Publications in SCIE and SSCI with(out) IF

Publications in AHCI

Proceedings in STP/ISI Proceedings-Index

Proceedings in SSHP/ISI Proceedings-index

Publications in the VABB-SHW

Citations 0.075

C 0.275 Combination raw and weighted PhD degrees 0.09

Citation distribution 0.10

Flemish publications with international partners 0.0375

Total revenue of EU framework programs 0.0375

Parameter for Interdisciplinary research 0.00

Diversity parameter 0.01

1 In the 2019 revision the minimal fractions are maintained.

The BOF became and still is the university’s instrument par excellence to develop its blue-sky research capacity based on internal selection procedures. The regulation stipulated that each university had to set up a research council, an internal advisory board to develop its research policy and to allocate the BOF-funding based on calls for proposals, and provide candidates with transparent evaluation procedures. The research council has a large autonomy, and over the years, it has gained within each university great prestige and legitimacy. The academic authorities had to approve the research council’s funding proposals, but their rubber-stamping became a mere formality.

Confronted with the need to set up their own research policy and lacking information on the strengths and weaknesses in different research disciplines, evaluating their research capacity became high on the Flemish universities’ agenda. In the 1990s, UGent was the first Flemish university that decided to carry out a systematic review of the research performance, followed by the KU Leuven and UAntwerp. These universities commissioned the Centre for Science and Innovation Studies (CWTS) at Leiden University to carry out this work. The methodology’s basic characteristic was the use of bibliometric indicators of research output and impact combined with a validation of the results by the researchers involved. The studies covered research groups in the natural and life sciences, medical and pharmaceutical sciences, mathematics, computer sciences, and engineering — disciplines where the publication culture is primarily focused on articles in international peer-reviewed journals (Korevaar and Moed, 1996; Moed et al., 1998; Luwel et al., 1999; Van den Berghe et al., 1998). In 1999, the Flemish Rector’s Conference commissioned a study on the development of indicators for research performance in the humanities and social sciences (HSS). This pilot study by CWTS was limited to law and linguistics (Nederhof et al., 2001).

The next anchor point in the development of performance-based funding metrics is the decision by the Flemish government to identify a small number of topics around which there was a need for policy-relevant research to proactively support policy development and develop the necessary academic research capacity. As science and innovation was one of the government’s priorities, in this call, a policy support center on quantitative science and technology studies located at the KU Leuven was selected. It became operational in 2003 and started collecting bibliometric and technometric data on the Flemish region.

In 2003, the Flemish government approved a major modification of the BOF-regulation: In the repartition of the global sum among the universities next to the above-mentioned cluster of three input parameters, a performance cluster composed of two bibliometric-based parameters was introduced (Debackere and Glänzel, 2004): publications processed for the Science Citation Index Expanded (SCIE) and assigned to the main publication types and citations to publications processed for the SCIE and assigned to these publication types. The former was considered a proxy of the university’s research capacity and the latter of its impact or visibility. Table 2 gives the weight assigned to each parameter.

The policy support center on quantitative science and technology studies was tasked with providing the data for the bibliometric-based indicators using a protocol that gave each university the possibility to verify the publication and citation data underlying the partition formula and, if necessary, suggest corrective actions.

The review of the regulation was planned, and a modified version had to go into effect in 2007, together with a revised funding formula for allocation of the lump sum.

In the explicative note annexed to the regulation, the government emphasized the experimental nature of the mechanism, its limitations, and the need to develop robust indicators for the HSS. Not taken into account, research in these disciplines turned out to be a sticky issue. Furthermore, the use of raw citation counts was considered unfair by many researchers and, given the differences in the citation culture between disciplines, in contradiction with the Flemish government’s objective to increase the international visibility of its blue-sky research.

But the first major revision of the BOF regulation in 2006 focused on the input parameters. Several issues related to the recruitment of tenured staff at the Flemish universities and the appeal to public authorities to intervene came to the center of the policy debate: the tendency to appoint in vacant tenured positions researchers who often spent their entire academic career at their alma mater and the underrepresentation of women in tenured positions. The government decided not to intervene directly in the autonomy of the universities but to use financial incentives.

In the 2006 revision of the BOF-regulation, minor technical changes to two of the three input parameters in the repartition formula were made, and a fourth related to mobility and diversity was added (Table 2).

In this regulation, the government also earmarked some of the additional BOF-funding to provide long-term support to world-leading researchers. The idea is similar to that of the Max Planck Institutions in Germany. Each university’s research council has to make the selection, and these researchers receive a substantial amount of funding until their retirement.

The collection and validation of the data used in the BOF partition formula remained of critical importance both for the acceptance by the universities and for the public authorities’ accountability. After an external review, the policy support center on quantitative science and technology studies was transformed into an inter-university center in 2007 and the development of a database on PhD degrees granted by the Flemish universities was added to its mission. The center also enlarged its bibliometric databases to all journals and proceedings editions of the Web of Science Core Collection.

As it was tasked with a long-term mission, in 2009, the Flemish parliament decided to grant it permanent status as an inter-university Centre for Research & Development Monitoring (ECOOM) with a twin mission: providing policy support and carrying out research on science and innovation.

The additional data from the Web of Science Core Collection were used in the 2008 revision of the BOF partition formula (Table 2). Starting in 2011, data from the Flemish database for the registration of publications in the HSS (VABB-SHW) were also included, and will be discussed in the next subsection.

In the input cluster, the major modification was the stepwise phasing out of the size of each university’s lump sum; starting in 2012, it was no longer taken into account.

The relative weight of the input cluster, on the one hand, and the performance cluster, on the other hand, slightly changed over the period 2007–2012, and from 2012 onward, the former weighed 64% and the latter 32% (Table 2).

During the years 2000–2012, the Flemish higher education system underwent a substantial reform. At the end of a long process in 2012, all academic degree programs except the artistic academic degree programs previously organized by the university colleges were integrated into the universities. The readable binary system with the universities offering academic-oriented degree programs and the university colleges’ professionally oriented bachelor’s programs was deemed necessary for the international attractiveness of the Flemish higher education system in the gradually developing European Higher Education Area (Degroof, 2018).

Even before this integration process, there were large discrepancies between the sizes of the Flemish universities. In the academic year 2012–2013, the last before the integration, KU Leuven and UGent together had 72% of the total university student population. Of this, 26%, roughly equally divided, enrolled in two median large universities—the UAntwerp and the VUB—and 1.7% in the UHasselt (Table 3). Due to this integration, the number of students enrolled at the Flemish universities increased by 34% between the academic years 2012–2013 and 2014–2015, the first year after the integration process was realized. In absolute numbers, the dichotomy between the two large universities and the other three became even more pronounced. However, the relative weight of the institutions remained fairly stable: KU Leuven and UGent together had 73%, with an increase of 3.3% of the former’s and a decrease by 2.6% of the latter’s share; the combined share of the UAntwerp and the VUB decreased by 2.2%, and the UHasselt was the big winner, nearly doubling its share.

Table 3

Total number of students enrolled at Flemish universities and each institute’s fraction (%) in the academic years 2012–2013 and 2014–2015. The students enrolled in degree programs jointly organized by two or more Flemish universities were assigned to these institutes based on the fractions mentioned in the statistics of the Flemish ministry of education and training (Source: Flemish ministry of education and training; https://www.onderwijs.vlaanderen.be/nl/hoger-onderwijs-in-cijfers).


UNIVERSITY ACAD. YEAR 2012–2013 ACAD. YEAR 2014–2015

# % # %

UAntwerpen 12450 14.4 16962 14.7

UGent 29443 34.1 36343 31.5

UHasselt 1462 1.7 3410 3.0

KU Leuven 32773 38.0 47616 41.3

VUB 10101 11.7 10945 9.5


Total 86229 100.0 115276 100.0

As highlighted in the explicative note annexed to the 2012 revision of the BOF regulation, the supposed impact of this integration process on the three smaller universities’ research capacity was considered so important that radical measures were taken to protect their share in the BOF partition formula. For each of these institutions, a minimal fraction of the global sum allocated in the Flemish budget for the BOF was guaranteed.

In 2019, the Flemish government approved a new revision of the BOF regulation. The minimal fractions were confirmed, and until 2023, in the input cluster, each university’s fraction will no longer be calculated but fixed by the regulation. The performance parameters remained unchanged. A third cluster of six parameters was introduced in the formula. They are presented as proxies for governmental priorities (Table 2). Some parameters were in earlier versions included in the input cluster. Next to the parameter based on citations in the second cluster, in the third a new parameter using the citation distributions is introduced. The construction of this distribution is rather complex.

In third cluster a parameter for interdisciplinary research is announced. ECOOM is tasked with developing a methodology to measure interdisciplinary work at the Flemish universities and it will only be taken into account from 2024 onwards and at that moment the weights assigned to each parameter will be slightly modified.

3.2 The registration of Flemish publications in the Humanities and Social Sciences

As already mentioned, the use of Web of Science databases to construct performance indicators was severely criticized by a large part of the Flemish academic community. This approach was not only strongly biased toward the natural and life sciences and the basic disciplines in engineering, but it also gave at least the impression of neglecting the outcome and importance of scholarly work in the HSS. Its research culture is very different, and the Social Sciences Citation Index (SSCI) and the Arts & Humanities Citation Index (A&HCI) had low coverage of publications in these disciplines, especially those not written in English. In the 2008 revision of the BOF partition formula, the Flemish government took this criticism into account and tasked ECOOM to set up the VABB-SHW. The VABB-SHW includes the output of the Flemish universities in these disciplines that is not covered by the WoS. The ECOOM researchers at UAntwerp became responsible for managing this tool.

The 2008 revision of the BOF regulation specifies the publication types to be included in the VABB-SHW: articles in journals, monographs, edited books, articles or chapters in books and proceedings papers that are not part of special issues or edited books. A weight is assigned to each of the five publication types, with the weight of a monograph eight times higher than for a proceedings paper (Verleysen and Engels, 2018).

The Flemish government trusts a panel of influential scholars from the Flemish universities and university colleges working in these disciplines with the scientific management of the VABB-SHW. Based on the panel’s advice, the Flemish government laid down the procedure and detailed selection criteria. This selection is essentially done at the level of publishers and not of the individual publications.

The public authorities considered quality management as a critical success factor of this innovative policy tool. The regulation stipulates that every three years, an international panel should carry out a quality audit of VABB-SHW. The panel comprises at least five experts, of which one must be conducting research in science studies. The panel’s report with its conclusions and recommendations is sent to the Flemish government and parliament.

Flanders was not the first country or region to set up a bibliographic database to monitor the scientific publications of its universities. CRIStin, the Norwegian national research documentation system that was gradually developed by the universities from 1990 onward and became fully operational as a governmental management tool in 2010, is probably the best known and served as an inspiration for Flanders and two other Nordic countries, Denmark and Finland (Engels and Guns, 2018).

3.3 Performance-based lump sum funding for strategic applied research, innovation, and outreach activities

To develop a portfolio of strategic applied research, to stimulate collaboration with firms, and to promote the valorization of research results, in 2004, the Flemish government decided to grant each university earmarked funding for these activities. Strategic basic research is defined as applied research with a middle long-time horizon for economic or societal applications.

The regulation described in broad terms the objectives of this additional funding called the Industrial Research Fund or denoted with the Dutch acronym IOF. The only constraints were that the selection process had to be based on open calls and that at least 30% of the public funding had to be used for post-doc positions. It was stipulated that each institution could complement the public funding with proper resources such as revenues from its valorization activities. Each university had to develop its applied research and valorization strategy and report every five years on its objectives, outcomes, and adjustments. Although not mandatory, the regulation enabled the universities to set up the IOF funding jointly with the university colleges at the level of their association. To devise this strategy, to implement it, and to manage the dedicated funding, each university had to set up an IOF board. Representatives of the association’s university colleges and the business sector had to sit on this advisory body.

To determine each university’s fraction of the global sum annually allocated in the Flemish budget for the IOF, a partition formula was used. It was based on seven weighed parameters (Table 4). As with the BOF partition formula for each of these parameters, a university’s fraction in the Flemish universities’ total value is calculated, a reference period taken into account and averages made.

Table 4

The parameters used in the initial IOF partition formula and in the different revisions; the year of the revision (Year) and the weight of each parameter (Weight) are given.


YEAR PARAMETER WEIGHT

2004 PhD degrees 0.25

Publications and citations as counted in the BOF partition formula 0.25

Revenues from the Flemish Agency for applied research (IWT) 0.10

Revenues from the EU Framework Programs 0.10

Patents: 0.10

Granted by USTPO

Applications filed with and granted by EPO

Applications filed under the Patent Cooperation Treaty

Created spin-offs 0.10

Tenured faculty and researchers 0.10


2009 PhD degrees 0.15

Publications and citations as counted in the BOF partition formula 0.15

Revenues from industrial contracts 0.30

Revenues from the EU Framework Programs 0.10

Patents: 0.15

Granted by USTPO

Applications filed with and granted by EPO

Applications filed under the Patent Cooperation Treaty

Created spin-offs 0.15


2014 Each of the five associations of university – university colleges received a minimal funding from the global IOF sum; the IOF partition formula remained unchanged and was applied to the remaining amount


2019 The association UAntwerpen, the association UHasselt and the association VUB receive at least a minimal fraction of the global IOF sum and the weight of the 6 parameters in the formula is modified: 0.05, 0.05, 0.30, 0.20, 0.20 and 0.20


For publications and citations the data of the BOF partition formula were used; ECOOM provided the patent data, and for the other parameters, and the administration and the Rectors Conference made the data available. And as with the BOF partition formula for the data collected, a transparent procedure was used and the universities had access to all the data and could report on omissions or errors.

After a review, in 2009, the third parameter was replaced by each institution’s fraction in Flanders universities’ total revenues from research contracts with industrial partners, and from 2011 onwards, the parameter based on the number of tenured faculty and researchers was no longer taken into account. Some technical changes were made to the regulation, and the weight of the parameters was modified gradually over a three-year period (Table 4).

To incentivize an integrated strategy for strategic applied research and valorization in each association of university-university colleges, the parameters were no longer based on university but on association data. The 2009 revision of the regulation also made mandatory the organization of IOF funding at the level of the associations.

The regulation also stipulated that each association had to develop a five-year strategic plan for its strategic applied research and valorization activities and submit it to the Flemish government. As for all its funding instruments for science and innovation, the Flemish government commissions an external review of the IOF funding every five years.

In the 2014 revision of the IOF-regulation a major change was made in the repartition formula. As with the BOF regulation, from 2014 onward, a minimum fraction of the global IOF funding is guaranteed for the Association UAntwerp, the Association UHasselt, and the Association VUB. Together with slight changes in the weights, in the 2019 revision this modification was confirmed.

3.4 From 2008 onward, incentive mechanisms for research in the lump sum funding

In 2008, the Flemish parliament approved a new funding formula for the lump sum of the universities and the university colleges. The model is rather complex. A detailed description and analysis are outside the scope of this contribution that focuses on its research components (See: De Cock and Vercruysse, 2009).

The main lump sum of each university has comprised four components:

  • A base funding for education;
  • A variable funding for education;
  • A base funding for research;
  • A variable funding for research.

To each of these four components, a global sum is allocated, and a partition formula is used. Each university receives base funding for education taking into account the number of awarded study-credits with some degree of degressivity. With the implementation of the Bologna Declaration study-credits were introduced as a standard for the degree programs (Huisman and Mampaey, 2017).

The funding formula for the partitioning among the universities of the global sum for the variable funding for education is, to a large extent, based on the awarded study-credits and the bachelor and master degrees delivered by each institution.

The distribution of the global sum for the base funding for research among the universities is based on the number of publications and granted PhD degrees. Both parameters are the same as those in the partition formula for the BOF.

To offset the effects on its research capacity of underfunding of UGent’s lump sum over the previous decades, it received a temporary increase of its base funding for research in the period 2008–2013.

The distribution of the global sum for the variable funding for research among the universities is also based on parameters of the 2008 version of the partition formula for the BOF:

  • the number of weighted academic bachelor’s and weighted academic initial master’s degrees;
  • the number of granted PhD degrees;
  • the number of publications and citations;
  • the mobility and diversity parameter.

In the period 2008–2013, a gradual transition was made from the existing to the new funding model. Only from 2014 onward and with a structural increase of the total amount allocated in the Flemish budget was the new funding formula used to calculate each institution’s lump sum.

The decree has a provision to adapt the variable funding for education to changes in the weighted number of study-credits. There is no equivalent provision for changes in the parameters used to determine the global sum allocated in the model to the variable funding of research.

4. Flanders’ metrics-based PRFS in an international context

In this section the Flemish model2 is compared with the metric-based PRFS (recently) applied in nine European countries: Croatia, Czech Republic, Denmark, Estonia, Finland, Norway, Poland, Slovakia and Sweden.

Table 5 gives an overview of the criteria used in these PRFSs. It is based on Table 1 in Zacharewicz et al. (2019)3 but takes into account the most recent version of the BOF and IOF partition formula. The latter was not included in the analysis of Zacharewicz et al. (2019).

Table 5

Criteria used in the metrics-based PRFS of Flanders (FL), Czech Republic (CZ), Denmark (DK), Estonia (EE), Finland (FI), Croatia (HR), Norway (NO), Poland (PL), Sweden (SE) and Slovakia (SK).


CRITERIA OPERATIONALISATION FL CZ (a) DK EE FI HR (b) NO PL SE (c) SK


Historical and educational × × × × × × × ×


Biblometrics CRIS × × × × × × × × ×

WOS/SCOPUS × × × × × × ×

Citations × × ×


Other PhD degrees × × × × × × ×


Project funding National public × × × × × × × ×

International/EU × × × × × × ×

Business sector × × × × × × × × × ×


Patents and other IPRs × × × × ×

Revenues valorisation IPRSs × × × ×


Internationalisation, other × × × ×

than internat./EU funding


Diversity/gender ×


interdisciplinarity ×

(a) Addition to Table 1 in Zacharewicz et al. (2019): Funds abtained from private sources are included in the PRFS (Debackere et al., 2018).

(b) Addition to Table 1 in Zacharewicz et al. (2019): Awarded PhD degrees are not included in the PRFS (Debackere et al., 2018).

(c) Addition to Table 1 in Zacharewicz et al. (2019): 20% of the institutional research is based on the PRFS (Hammarfelt et al., 2016).

4.1 Architecture of the PRFS within the national science and innovation system

First and foremost it should be emphasized that not only Flanders but also the other countries regularly update the funding formula. As illustrated below, these changes are often deemed necessary to remediate unintended consequences or short-circuit attempts to game the system. The underlying motives behind these modifications are an interesting research topic in itself.

The PRFSs are used to allocate research funding based on the universities’ relative performance in a zero-sum game. The frequency with which the PRFS is run differs among the countries. Models based on information collected at institutional level and peer assessment generally work with 3–5 year cycles and those relying on publicly available data have shorter cycles (Zacharewicz et al., 2019). Although it partially depends on information provided by the universities, the Flemish PRFS is implemented on a year-by-year basis. To avoid problems due to too strong fluctuations in allocated sums, as is the case in the Czech PRFS (Good et al., 2015), in the Flemish formulae long-term averages are applied.

Although publications play an important role in the ten PRFSs, the bibliometric indicators differ among the countries.4 They have all set up either a comprehensive quality assured current research information system (CRIS) or a quality controlled publication database to complement the WoS and/or SCOPUS. Some countries also include artistic creations (Poland (Kulczycki et al., 2017), Slovakia (Balasz, 2015)). Except Sweden5 that uses WoS data (Carlsson, 2009; Jacob et al., 2016), one of their functions is to provide the bibliometric data included in the PRFS.

In all the models differences in the publication culture among research fields are taken into account by applying elaborated field weighted normalizations of WoS or SCOPUS publication (and citation) data (e.g. the Swedish PRFS (Carlsson, 2009)) or by assigning weights or points to the scholarly work included in the CRIS. Researchers recognized as leaders in their discipline are tasked with the assessment of publishers and publications (e.g. the Norwegian PRFS (Sivertsen, 2018)). As the Flemish PRFS is based on the WoS and a complementary database for the HSS, both techniques are applied.

To complement the bibliometric criteria all 10 countries include in their PRFS a wide range of other indicators. Flanders and six other countries take into account the (weighted) number of granted PhD degrees.

All the countries use external funding as a criterion: competitive project funding from public national and/or international, mainly European Union’s research programs, and initiatives and/or revenues from research collaboration with the business sector. Countries use also criteria they consider important for their national policy agenda. As the strengthening of the national identity is one the Estonian government’s priorities, the participation in scholarly activities essential to sustain the Estonian language and culture is included in this indicator (Debackere et al., 2018). To develop the general public’s better understanding of the impact of science on everyday live, in the Croatian PRFS participation in activities related to the popularization of science is taken into account (Debackere et al., 2018).

Intellectual property rights and their valorization form the next cluster of indicators:

  • Patents and patent applications (Flanders and four other countries) but also other forms such as trademarks (Poland (Kulczycki et al., 2017)) and utility models, industrial designs and prototypes (Czech Republic (Good et al., 2015));
  • Other activities related to intellectual property rights (IPRs): Spin-offs (Flanders) and revenues from licenses and business consultancy (Poland (Kulczycki et al., 2017)).

Flanders is one of the seven countries to include internationalization as a criterion in its PRFS. It is operationalized by the above mentioned participation in international research programs. Additionally Flanders (2019 revision of the BOF partitioning formula) and Norway (Debackere et al., 2018) give a higher weight to internationally co-authored publications; Finland takes into account both the PhD degrees granted to foreign researchers and the international teaching and research staff at its universities (de Boer et al., 2015); and Croatia uses information on incoming and outgoing mobility of the academic staff (Debackere et al., 2018).

Flanders is the only country to include since 2006 diversity as a criterion in the PRFS and it plans to introduce in the next couple of years a parameter quantifying interdisciplinary research activities.

Except for the latter two, the criteria used in the 10 countries’ PRFS are fairly similar aiming to strengthen the international competitiveness of university research and to enhance its contribution to the development of a competitive knowledge-based society.

While there is a fairly broad consensus on the criteria, among the countries there are large differences in the allocation of public research funding between:

  • Performance-oriented allocation of project funding based on ex-ante evaluation criteria;
  • Performance-based allocation of institutional research funding based on ex-post evaluation criteria; and
  • Non-performance oriented institutional research funding (block grant) based on the funding allocated in the preceding years (‘historical funding’) or educational criteria such as the number of students and/or graduates.

For the Czech Republic, Denmark, Finland, Norway, Poland and Sweden, Reale (2017)6 gives an overview of the relative weight of these types of funding in the total public outlays for R&D. However no breakdown is given by institutional type, making it difficult to estimate these ratios for the university sector.

For the universities’ institutional research funding more information is available on the ratio between the performance-based and block funding. For the 10 countries this ratio differs between 2% (of the total institutional funding) for Norway (Aagaard, 2015) and 100% (of the institutional research funding) in the Czech Republic (Good et al., 2015). In Flanders the allocation of 50% of the BOF-funding and 75% of the IOF-funding is metrics-based and the allocation of the research component of the lump sum is fully metrics-based7 (De Cock and Vercruysse, 2009). Together they represent the institutional research funding of the Flemish universities. With 75% of the sum total allocated through PRFS, Flanders is among the countries with the highest ratio (EWI, 2020).

4.2 (Un)intended effects of the ten countries’ PRFS

The different trade-offs made in the design of the PRFSs lead to the question of the realization of their objectives and their potentially unintended consequences on three levels: the national level as the mechanisms’ primary objective is the allocation of research funding among universities, the institutional level (university, faculty, department and research groups) and the individual researcher.

Over the last two decades empirical research has been done using two approaches: qualitative and quantitative. The former are based on document analysis, surveys and interviews with stakeholders. Although as shown in Table 5 in most PRFSs a variety of criteria are used, the latter are mostly limited to bibliometric studies of the publication output and impact.

In a pioneering study on performance-based funding in Australia, Butler (2003) showed that after the introduction of a PRFS based on publications covered by WoS aggregate national publication output increased but the country’s citation impact declined. This observation could have been one of the reasons the Australian government replaced it by a metrics-informed peer review assessment8 (Jonkers, 2016). However recently van den Besselaar et al. (2017) contested Butler’s observation leading to a discussion about assumptions and causality (Aagaard and Schneider, 2017; Butler, 2017; Gläser, 2017; Hicks, 2017). As already observed by Aagaard and Schneider (2016) in their analysis of the Danish system, this discussion highlights the difficulty to separate effects of the PRFS from other potential drivers in the science and innovation system and to identify causal relationships.

In a similar analysis of the long-term developments of the Norwegian publication activity, Schneider et al. (2016) observed an increase in publication output while the impact remained stable. The authors indicate that the introduction of the PRFS stimulated more research work and resulted in more publications, but no causal link could be made as during the same period R&D investments and the number of researchers increased also.

Ingwersen and Larsen (2014) analyzed the Danish publication output in the period 2000–2012, prior to and after the introduction of the PRFS in 2008. The number of journal articles covered by the CRIS grew during the whole period but progressed more steeply in the years 2008–2012. This growth rate outperformed the increase of academic staff. The number of articles processed for the WoS was also steadily increasing and the growth was even accelerating in 2010–2012. On the other hand, the number of the proceedings papers was nearly halved between 2007 and 2012. The average citation impact of all WoS items combined was linearly rising.

Korytkowski and Kulczycki (2019) analyzed the publication and patent output of the Polish universities and public research institutions during the period 2009–2016. The number of WoS publications increased without a decrease of the quality (in terms of JCR quartiles). It was found that this increase was not only due to a select group of researchers that became more productive, but that more researchers started publishing in journals indexed in the WoS. The authors link this result to the funding model and the stability over time in the assignment of weights to journal articles. Although during the same period the evaluation criteria of monographs were several times modified, their number remained stable.

Pisar et al. (2019) compared the number of Slovak publications in the WoS and SCOPUS between 2007 and 2016, four years before and five years after the introduction of the PRFS. During this period a significant increase in the publication output is observed. However, compared with Austria where after 2013 funding of universities was based on negotiations between the government and individual institutions, the publishing activity in both countries was fairly similar. Pisar et al. (2019) shows that the introduction of the PRFS increased the publication output in higher quality journals of universities that were initially less productive. But at the same time they provide evidence that less well performing universities also started gaming the rules to increase their output: launching new Central and Eastern European focused regional journals often published by the universities themselves which subsequently got indexed in WoS or SCOPUS and publishing in predatory journals that for a number of years managed to get indexed in these databases. To counter the tendency to increase output at all costs, the Slovak authorities started in 2017 to apply stricter rules and to take the journal impact factors into account.

The Czech PRFS is fairly similar to the Slovakian. Vanecek (2014) analyzed the publication output of the Czech universities and publicly funded research institutions in the period 2000–2011; the PRFS being introduced in 2009. The publication output (journal articles, books, book chapters and conference proceedings) covered by the Czech CRIS stagnated after 2007, due to a steep decrease of the number of contributions in conference proceedings. The subset of Czech publications indexed in the WoS grew rapidly between 2000 and 2010 followed by a small decrease in 2011. As for Norway, no substantial changes of the publishing journals’ impact factor were observed. The increase in the annual number of WoS-publications started in 2005 before the implementation of the PRFS. The growth in public R&D funding and/or the systematic evaluation of universities and research institutes in the period 2004–2008 could explain (at least partially) this trend. It should also be remarked that Pisar et al. (2019) formulated similar criticism on the Czech PRFS as on the Slovak claiming that a clear majority of the local and Eastern European journals indexed in the WoS or SCOPUS have only a ‘provincial status’ but they were taken into account in the allocation model. To avoid the overemphasis on local journals the Czech authorities are changing the PRFS combining a quantitative and qualitative approach.

For Flanders no similar studies are available. The biannual Flemish Indicator Book provides data on trends in the total Flemish publication output and impact. The number of WoS publications with a Flemish address per 10.000 inhabitants nearly doubled between 2006 and 2017 and the citation impact increased (Debackere and Veugelers, 2019). A large share of these publications is authored by researchers working at Flemish universities. However, no causal link can be made between the increase of the Flemish publication output and the use of the PRFS as the public research funding of universities and the number of researchers also considerably increased.

Compared to the WoS publication output, Flemish universities’ publications in the SSH have been comprehensively analyzed and benchmarked with other countries. For the period 2004–2015 Engels et al. (2018) studied trends in books and book chapters in the VABB-SHW and in the Finnish, Norwegian, Polish and Slovenian CRIS. For the humanities the share of monographs and book chapters in the peer reviewed publications indexed for VABB-SSH was fairly stable. Compared to Finland, Norway and Poland both shares were lower.

In Flanders’ peer-reviewed publication output in social sciences, the share of monographs was also fairly stable and increasing somewhat in 2014 and 2015 while share of book chapters was steadily increasing. The authors link the increase in book chapters to the introduction of the Guaranteed Peer Reviewed Content (GPRC) label (Verleysen and Engels, 2013). As for the humanities both shares were lower than for Finland, Norway and Poland.

For the HSS Ossenblok et al. (2012) compared trends in the coverage by WoS of publications included in the Flemish VABB-SHW and in the Norwegian CRISTin and the evolution of the share of articles published in English. Over the period 2005–2009 the fraction of peer reviewed journal publications of both countries written in English increased by 10%; in 2009 more than half were in English. Ossenblok et al. (2012) identified this trend as an indication of the growing internationalization.

Between 2005 and 2009 the fraction of publications in the VABB-HSW covered by the WoS increased steadily, while for Norwegian CRISTin it remained stable. For Flanders two factors could explain this trend. The tendency of Flemish scholars in SSH to publish more in WoS-covered journals and by the expansion of the WoS indexed more journals in these disciplines with local ties to Flanders and the Netherlands (Engels et al., 2012). The former could be related to the version of the BOF partition formula used up to 2008 that took only publications in journals indexed in the WoS into account. Contrary to the suggestion by de Rijcke et al. (2016), these results do not validate the claim that Flemish scholars’ quality considerations were displaced by incentives to increase their publication output. In fact, citations and impact were not analyzed by Ossenblok et al. (2012).

In a follow-up study Kulczycki et al. (2018) analyzed the publication types and languages used in the HSS in eight countries that have set up a CRIS. Although the coverage of these databases is not fully identical, broad patterns emerged. In 2014 79% of Flemish publications were written in English, which was the highest among the seven countries, followed by Finland (68%) and Denmark (63%). In the same year, 40% of the publications in these disciplines covered by the VABB-HSW were published in journals indexed in the WoS. Denmark (51%) and Norway (44%) have the highest share and Poland (15%) the lowest.

In quantitative studies of PRFSs little attention is paid to the other parameters used alongside bibliometric criteria. Patents and the valorization of IPRs are maybe the exception confirming the rule. Vanecek (2014) observed a strong increase since 2006 in patents and utility models granted by the national patent office to Czech universities and public research institutes. In the period 2002–2011 there was also an upward trend in the number of Czech patents granted by EPO and USTPO. Although no breakdown is given by sector, it seems reasonable to assume that the Czech universities have contributed to this increase. However, the introduction of the PRFS which was announced in 2008, could only partially explain this upward trend as the highest increase in the growth rate occurred already before. Moreover Good et al. (2015) makes some critical remarks about the role of national patents, utility models and prototypes granted by the Czech authorities. They should be easy to produce and no quality checks such as originality and functionality were required, yielding ‘cheaply earned’ points in the funding formula.

In the period 2009–2016 the number of patents submitted for the PRFS by the Polish universities and public research institutes increased until 2012 and then remained fairly stable (Korytkowski and Kulczycki, 2019).

The biannual Flemish Indicator Book (Debackere and Veugelers, 2019) provides information on Flemish patents and patent applications. Although the raw numbers are not published, the evolution of the fraction of the Flemish universities in the EPO patent applications with only Flemish applicants increased spectacularly: using two year averages from 3% in 1991–1992 to 12% in 1998–1999 and 18% in 2016–2017. Again this trend must be seen against the background not only of the introduction of the PRFS but also the strong growth in public research funding.

The ten countries use research funding from international, national and regional public authorities and/or from the private sector in their PRFSs. In the literature survey no study on project funding by public authorities and other R&D revenues for a country’s university sector or individual universities was found. Most probably these data are published in the local language in universities’ annual accounts, official documents, reports of commissioned studies and other grey literature.

Even for Flanders these data are not readily available. The only exceptions are the participation of Flanders at the different EU Framework Programs for Research and Innovation (FP) and the number of PhD-degrees granted by its universities.

Since 1991 for the successive FPs and by organization type the information on the number of projects and on the granted EU funding have been collected. Of the funding available for the 4th FP (1994–1998) the Flemish universities received 1.01%; for the 5th FP (2002–2006) 0.94%, for the 6th FP (2002–2006) 0.86%, for the 7th FP (2007–2013) 1.15% and for Horizon 2020, the current FP 1.12% (based on allocation of 58% of the available EU funding) (Debackere and Veugelers, 2019). As the overall success rate for obtaining FP funding is low, these data could be an indication of an increasing competitiveness of the Flemish universities. However, a more detailed benchmarking analysis is needed.

One of the tasks of ECOOM is the registration of information on doctoral researchers at the Flemish universities such as funding source, gender, success rate and time to degree (Groenvynck et al., 2013). Between the academic year 1991–1992 and 2017–2018 the number of awarded PhD-degrees increased more than fourfold to 2155. The share of women grew steadily to 45% (Debackere and Veugelers, 2019). Over the years the median time to degree is quite stable at 4.7 years but the doctoral completion rates have increased. Again these data have to be put in perspective as the public R&D funding grew also substantially during this period.

To complement these macro-level studies, for the ten countries there are only a few quantitative studies on the effects of PRFSs on meso- and micro-level. Bloch and Schneider (2016) analyzed the influence of the Norwegian PRFS at the individual level. In 2011–2012 compared to 2004–2005 a Norwegian researcher published more papers but at the same time the average number of coauthors per publication also increased. The lack of a clear definition of the publication activity of an individual researcher makes it challenging to link these results to the use of the PRFS.

Fiala (2013) analyzed the publication output of the Czech universities in the period 2008–2011. Although of each university both the total publication output in the CRIS and the fraction in journals indexed in the WoS increased, their relative weight in the Czech total (WoS) publication output changed to the detriment of the largest institutions.

Hammarfelt and de Rijcke (2015) studied research practices and publication patterns of scholars at the faculty of Arts at Uppsala University, Sweden using bibliometric data and a survey. Publication statistics were collected for the period 2006 to 2013 covering the introduction of both the Swedish PRFS in 2009 and the system for allocating resources based on performance measures at Uppsala University in 2011. Over the period the fraction of publications in English and of peer-reviewed publications increased. However the authors were not able to make a causal link with the implementation of the PRFS and/or the university level allocation model.

Sile and Vanderstraeten (2019) made a similar study on publication patterns within the field of educational research at the University of Gothenburg, Sweden during the period 2005–2014. This university introduced an institutional PRFS in the same year as the national one. The publication output rose, but the increase predated the introduction of the two PRFSs. Differences between publication types were observed: the number of peer-reviewed journal articles, especially those written in English grew rather steeply, but also the number of articles addressing a non-academic audience increased. The share of monographs remained stable.

For the Flemish model no similar studies have been published.

Although interesting, the above-mentioned publications focus on a single component of the PRFS. Comprehensive and international comparative quantitative analyses of the realization of all the objectives of the metrics-based PRFSs and their (un)desirable effects on the science and innovation system are necessary. Their result should be compared with those of other models for institutional research funding. They should go beyond the purely descriptive work commissioned by the EU ((Debackere et al., 2018), (Jonkers and Zacharewicz, 2016)). With the exception of Checchi et al. (2019), to the best of our knowledge no such studies have been done, let alone results published for scrutiny in the open literature. Using econometric techniques Checchi et al. (2019) examined to what extent the use of a PRFS for universities had an impact on their publication output and its quality. They found that the introduction of a PRFS increased a country’s publication output, however this result is only temporary. The effect of a peer review-based system is larger compared with a metric-based system. Using as an indicator for impact the number of publications in top journals the PRFS had no sizeable effect. One of the limitations of this study is its narrow focus on only one aspect of a university’s mission: publishing scientific results in the open literature.

Additional quantitative work could complement or serve as background material for qualitative research on the use of PRFSs and their implications on knowledge production, institutional management and (the careers of) individual researchers. Although based on the literature review of the much broader question of evaluation practices and effects of indicator use, de Rijcke et al. (2016) report that overall the knowledge of the implications of quantitative assessments and more broadly of how performance measures shape the production of knowledge is sketchy.

In the recent critical literature on the ‘state-of-the-art’ of institutional performance-based research evaluation arrangements, Thomas et al. (2020) identify limitations in our knowledge of their effects. Based on these observations and their belief that whatever the shortcomings performance-based funding arrangements are ‘here to stay’, the authors propose a novel research agenda to gain additional insights to support the development of more performant funding mechanisms.

5. Concluding remarks

The most striking difference between the institutional research funding in Flanders and the other nine countries is the use of three instead of one lump sum. For the three lump-sum funding mechanisms, the Flemish government has tried continuously to find the right balance between institutional autonomy on the one hand and enhancing research quality and realizing its policy priorities on the other. Stepwise, new indicators were introduced and existing ones fine-tuned and made more robust, often based on new research findings of ECOOM.

A Flemish university cannot use an institutional PRFS for allocating BOF- or IOF-funding to departments or research groups. Instead each university must set up its own research and valorization policy by providing funding to a mix of promising young researchers often in emerging research fields and internationally leading research groups. Together with the sustained increase in public R&D outlays over the last thirty years, as shown by a number of indicators in the previous section, these policy instruments played a role in bolstering the Flemish universities’ research capacity and in increasing their contribution to the development of Flanders’ knowledge economy.

This somewhat odd three-legged construction is rooted in the erosion of the lump sum due to the strained public budget, the financial claims of other educational sectors and the broad societal consensus on the critical importance of research and innovation. The wages of all the tenured faculty are paid out of the lump sum, and at each university, there are only a small number of other indefinite academic appointments that can be terminated only for a specific cause or under extraordinary circumstances. The final responsibility for providing education, managing research activities, and supervising PhD students rests on the tenured faculty.

Due to governmental regulations, the IOF- and BOF-funding can only be used for financing short-term research grants, PhD scholarships and post-doctoral fellowships. Additional research funding provided by public authorities, such as the EU, is also mostly allocated through competitive short-term project grants; the work is carried out by researchers with a temporary contract. Short-term research contracts are also the main instrument for bilateral collaboration with firms and other public and private partners.

In the period 1990–2019 the ratio between the number of researchers on temporary contracts and the tenured staff increased from 0.98 to 3.67.9 The ratio dissimulates large underlying differences among disciplines. As contract research is mainly concentrated in the faculties of natural and life sciences and engineering and medicine, in these disciplines, this increase is even more pronounced. An enlightened policy discussion about the use of metrics-based PRFS has to take as a starting point this still increasing imbalance, maybe the greatest challenge to address in Flanders’ science and innovation policy.

A similar remark holds for the ‘closed envelope’ principle. In the ten countries the implementation of the PRFS is based on this principle: Global sums are annually entered in the public budget, and based on partition formulae, they are allocated to each university. Flemish researchers repeatedly and again in the 2018 evaluation of the BOF regulation (Dialogic, 2018) signaled the perverse effects of the use of closed envelops for institutional research funding. A university that succeeds in increasing over the years in absolute numbers the parameters used in the partition formulae is not necessarily rewarded with additional funding, as its relative position with respect to the other universities is taken into account. Moreover this benchmarking is only at national level. Limiting to the Flemish setting, this option was maybe very valuable at the turn of the century in starting up the funding mechanisms, but it is becoming less and less relevant today. With the emergence of the European Higher Education and Research Area, the Flemish universities’ playing field is no longer restricted to its own region but has become European and even international. Flemish universities participate in international networks and adapt their mission statements and strategic goals to new challenges. Thus, Flemish universities become incommensurable, with the risk of comparing apples and oranges in funding formulae.

In conclusion, it is maybe an expression of Flanders’ decision-making culture, but it is remarkable that the use of a metrics-based PRFS has created such an intense controversy among academic staff10 and in discussions between university authorities,11 since only superficial assessments of the impact of its use on the development of the academic system have been made (Dialogic, 2018). The in-depth analysis of the Flemish publication output in the HSS is the exception rather than the rule.

To further develop Flanders’ ambitious science and innovation policy agenda, a cost-benefit analysis of the funding model should periodically be carried out as every system has its (dis)advantages (van den Besselaar and Sandström, 2020) and the policy landscape is continuously shifting. Given the available know how on science of science policy Flanders could even take the lead and kick start work on the research agenda mentioned in section 4.

Notes

1By a treaty between the Netherlands and Flanders, the transnational University Limburg (tUL) was established in 2001 as a tool for the Dutch Maastricht University and UHasselt to deepen their collaboration. It has only a small number of students, and the treaty stipulates the quite complicated funding rules. Given its specific legal status, it is outside this contribution’s scope. 

2Although Flanders is formally a region, for the sake of simplicity, in this section it is listed as a country. 

3The French speaking community of Belgium is not included in the section. By the regional law of February 27, 2014 the decision in principle is taken to use bibliometric indicators to partition a small amount of research funding among the universities. 

4Only Croatia, Flanders and Sweden use citation data in their PRFS. 

5In commissioned advice to the government, the Swedish Research Council proposed in a 2014 report to move for the allocation of research funding from a metrics-based to a peer assessment-based model with panels allocating scores and providing explanatory statements for their grading (Jacob et al., 2016). 

6No breakdown of Belgian data per region is given. 

7All criteria are research related, expect one based on the number of bachelor and initial master degrees. 

8See: Excellence in Research for Australia | Australian Research Council (https://www.arc.gov.au/excellence-research-australia). 

9VLIR, Statistische gegevens betreffende het personeel aan de Vlaamse universiteiten, 2020 (https://vlir.be/publicaties/personeelsstatistieken/). 

10Beste rectoren, red ons van de ratrace – De Standaard (https://www.standaard.be/cnt/dmf20180131_03332483). 

Acknowledgement

The author gratefully thanks the anonymous referees for their constructive comments and recommendations which substantially helped to improve the quality of the paper.

Competing interests

Throughout his career and until the end of 2015 the author has been involved in the development of the Flemish PRFS and more generally of Flanders’ science and innovation policy.

References

  1. Aagaard, K. (2015). How incentives trickle down: Local use of a national bibliometric indicator system. Science and Public Policy, 42(5), 725–737. DOI: https://doi.org/10.1093/scipol/scu087 

  2. Aagaard, K., & Schneider, J. W. (2016). Research funding and national academic performance: Examination of a Danish success story. Science and Public Policy, 43(4), 518–531. DOI: https://doi.org/10.1093/scipol/scv058 

  3. Aagaard, K., & Schneider, J. W. (2017). Some considerations about causes and effects in studies of performance-based research funding systems. Journal of Informetrics, 11(3), 923–926. DOI: https://doi.org/10.1016/j.joi.2017.05.018 

  4. Algoed, K., & van Nieuwenhove, J. (2011). Vijf staatshervormingen achter de rug, één in de wachtkamer (p. 46). Acco, Leuven. ISBN: 9789033484902. 

  5. Balasz, V. (2015). RIO country report Slovak Republic 2014. Report EU 27298. 

  6. Bloch, C., & Schneider, J. W. (2016). Performance-based funding models and research behavior: An analysis of the influence of the Norwegian Publication Indicator at the individual level. Research Evaluation, 25(4), 371–382. DOI: https://doi.org/10.1093/reseval/rvv047 

  7. Butler, L. (2003). Explaining Australia’s increased share of ISI publications – the effects of a funding formula based on publication counts. Research Policy, 32(1), 143–155. DOI: https://doi.org/10.1016/S0048-7333(02)00007-0 

  8. Butler, L. (2017). Response to van den Besselaar et al.: What happens when the Australian context is misunderstood. Journal of Informetrics, 11(3), 919–922. DOI: https://doi.org/10.1016/j.joi.2017.05.017 

  9. Carlsson, H. (2009). Allocation of research funds using bibliometric indicators – asset and challenge to Swedish higher education sector. InfoTrends, 64(4), 84–90. DOI: https://doi.org/10.1111/hequ.12185 

  10. Checchi, D., Malgarini, M., & Sarlo, S. (2019). Do performance-based research funding systems affect research production and impact? Higher Education Quarterly, 73(1), 45–69. 

  11. Debackere, K., Arnold, E., Sivertsen, G., Spaapen, J., & Sturn, D. (2018). Performance-based funding of university research. Mutual Learning Exercise, Horizon 2020 Policy Support Facility, European Commission, DG, Research and Innovation. 

  12. Debackere, K., & Glänzel, W. (2004). Using a bibliometric approach to support research policy making: The case of the Flemish BOF-key. Scientometrics, 59(2), 253–276. DOI: https://doi.org/10.1023/B:SCIE.0000018532.70146.02 

  13. Debackere, K., & Veugelers, R. (Eds.) (2019). Vlaams Indicatorenboek 2019 Wetenschap – Technologie – Innovatie. <https://www.vlaamsindicatorenboek.be/> accessed December 30, 2020. 

  14. de Boer, H., Jongbloed, B., Benneworth, P., Cremonini, L., Kolster, R., Kottmann, A., Lemmens-Krug, K., & Vossensteyn, H. (2015). Performance-based funding and performance agreements in fourteen higher education systems – Report for the Ministry of Education, Culture and Science, CHEPS, University of Twente. 

  15. de Cock, L., & Vercruysse, N. (2009). De financiering van het hoger onderwijs in Vlaanderen. Documentatieblad van de Federale Overheidsdienst Financiën – België, 69(4), 91–114. 

  16. de Rijcke, S., Wouters, P. F., Rushforth, A. D., Fransen, T. P., & Hammerfelt, B. (2016). Evaluation practices and effects of indicator use – a literature review. Research Evaluation, 25(2), 161–169. DOI: https://doi.org/10.1093/reseval/rvv038 

  17. Dialogic. (2018). Evaluatie van de Bijzondere Onderzoeksfondsen (BOF) – Managementsamenvatting. Available at: https://www.dialogic.nl/wp-content/uploads/2018/07/evaluatie_van_de_bijzondere_onderzoeksfondsen_-_managementsamenvatting.pdf 

  18. Engels, T. C. E., & Guns, R. (2018). The Flemish performance-based research funding system: a unique variant of the Norwegian model. Journal of Data and Information Science, 3(4), 45–60. DOI: https://doi.org/10.2478/jdis-2018-0020 

  19. Engels, T. C. E., Ossenblok, T. L. B., & Spruyt, E. H. J. (2012). Changing publication patterns in the social sciences and humanities, 2000–2009. Scientometrics, 93(2), 373–390. DOI: https://doi.org/10.1007/s11192-012-0680-2 

  20. Engels, T. C. E., Starcic, A. I., Kulczycki, E., Polonen, J., & Sivertsen, G. (2018). Are book publications disappearing from scholarly communication in the social sciences and humanities? Aslib Journal of Information Management, 70(6), 592–607. DOI: https://doi.org/10.1108/AJIM-05-2018-0127 

  21. EWI – Department Economie, Wetenschap & Innovatie. (2020). Speurgids 2020 Ondernemen en Innoveren. <https://www.ewi-vlaanderen.be/sites/default/files/speurgids2020.pdf> accessed December 27, 2020. 

  22. Fiala, D. (2013). Science evaluation in the Czech Republic: The case of universities. Societies, 3(3), 266–279. DOI: https://doi.org/10.3390/soc3030266 

  23. Gläser, J. (2017). A fight on epistemological quicksand: Comment on the dispute between van den Besselaar et al. and Butler. Journal of Informetrics, 11(3), 927–932. DOI: https://doi.org/10.1016/j.joi.2017.05.019 

  24. Good, B., Vermeulen, N., Tiefenthaler, B., & Arnold, E. (2015). Counting quality? The Czech performance-based research system. Research Evaluation, 24(2), 91–105. DOI: https://doi.org/10.1093/reseval/rvu035 

  25. Groenvynck, H., Vandevelde, K., & Van Rossem, R. (2013). The PhD track: who succeeds, who drops out? Research Evaluation, 22(4), 199–209. DOI: https://doi.org/10.1093/reseval/rvt010 

  26. Hammarfelt, B., & de Rijcke, S. (2015). Accountability in context: Effects of research evaluation systems on publication practices, disciplinary norms, and individual working routines in the faculty of Arts at Uppsala University. Research Evaluation, 24(1), 63–77. DOI: https://doi.org/10.1093/reseval/rvu029 

  27. Herbst, M. (2007). Financing public universities – the case of performance funding. Dortrecht, the Netherlands: Springer. 

  28. Hicks, D. (2012). Performance-based university research funding systems. Research Policy, 41(2), 251–261. DOI: https://doi.org/10.1016/j.respol.2011.09.007 

  29. Hicks, D. (2017). What year? Difficulties in identifying the effect of policy on university output. Journal of Informetrics, 11(3), 933–936. DOI: https://doi.org/10.1016/j.joi.2017.05.020 

  30. Huisman, J., & Mampaey, J. (2017). Setting up associations in Flemish higher education. In: de Boer, H., File, J., Huisman, J., Seeber, M., Vukasovic, M., & Westerheijden, D. F. (Eds.), Policy analysis of structural reforms in higher education: processes and outcomes (pp. 205–223). Palgrave studies in global higher education book series. DOI: https://doi.org/10.1007/978-3-319-42237-4_10 

  31. Ingwersen, P., & Larsen, B. (2014). Influence of a performance indicator on Danish research production and citation impact 2000–12. Scientometrics, 101(2), 1325–1344. DOI: https://doi.org/10.1007/s11192-014-1291-x 

  32. Jacob, M., Dahlstrand, A. L., & Sprutacz, M. (2016). RIO country report 2015: Sweden. EUR 27859. 

  33. Jonkers, K., & Zacharewicz, T. (2016). Research performance based funding systems: a comparative assessement. EUR 27837. 

  34. Kettl, D. F. (1997). The global revolution in public management: driving links, missing links. Journal of Policy Analysis and Management, 16(3), 446–462. DOI: https://doi.org/10.1002/(SICI)1520-6688(199722)16:3<446::AID-PAM5>3.0.CO;2-H 

  35. Korytkowski, P., & Kulczycki, E. (2019). Examining how country-level science policy shapes publication patterns: the case of Poland. Scientometrics, 119(3), 1519–1543. DOI: https://doi.org/10.1007/s11192-019-03092-1 

  36. Kulczycki, E., Engels, T. C. E., Polonen, J., Bruun, K., Duskova, M., Guns, R., Nowotniak, R., Petr, M., Sivertsen, G., Starcic, A. I., & Zuccula, A. (2018). Publication patterns in the social sciences and humanities: evidence from eight European countries. Scientometrics, 116(1), 463–486. DOI: https://doi.org/10.1007/s11192-018-2711-0 

  37. Kulczycki, E., Korzén, M., & Korytkowski, P. (2017). Towards an excellence-based research funding system: Evidence from Poland. Journal of Informetrics, 11(1), 282–298. DOI: https://doi.org/10.1016/j.joi.2017.01.001 

  38. Luwel, M., Noyons, E. C. M., & Moed, H. F. (1999). Bibliometric assessment of research performance in Flanders: policy background and implementations. R&D Management, 29(2), 133–141. DOI: https://doi.org/10.1111/1467-9310.00124 

  39. Korevaar, J. C., & Moed, H. F. (1996). Validation of bibliometric indicators in the field of mathematics. Scientometrics, 37(1), 117–30. DOI: https://doi.org/10.1007/BF02093488 

  40. Moed, H. F., Luwel, M., Houben, J. A., Spruyt, E., & Van Den Berghe, H. (1998). The effects of changes in the funding structure of the Flemish universities in their research capacity, productivity and impact during the 1980’s and early 1990’s. Scientometrics, 43(2), 231–255. DOI: https://doi.org/10.1007/BF02458409 

  41. Nederhof, A. J., Luwel, M., & Moed, H. F. (2001). Assessing the quality of scholarly journals in linguistics: an alternative to citation-based journal impact factors. Scientometrics, 51(1), 241–265. DOI: https://doi.org/10.1023/A:1010533232688 

  42. Ossenblok, T. L. B., Engels, T. C. E., & Sivertsen, G. (2012). The representation of the social sciences and the humanities in the Web of Science – a comparison of publication patterns and incentive structure in Flanders and Norway (2005–9). Research Evaluation, 21(4), 280–290. DOI: https://doi.org/10.1093/reseval/rvs019 

  43. Pisar, P., Sipikal, M., Jahoda, R., & Spacek, D. (2019). Performance based funding of universities: Czech Republic and Slovakia. In M. S. de Vries et al. (Eds.) Performance-based budgeting in the public sector (pp. 237–254). Cham, Switzerland: Palgrave MacMillan. DOI: https://doi.org/10.1007/978-3-030-02077-4_13 

  44. Reale, E. (2017). Analysis of national public research funding – PREF. Final report. EU JRC107599. 

  45. Schneider, J. W., Aagaard, K., & Bloch, C. W. (2016). What happens when national research funding is linked to differentiated publication counts? A comparison of the Australian and Norwegian publication-based funding models. Research Evaluation, 25(3), 244–256. DOI: https://doi.org/10.1093/reseval/rvv036 

  46. Sile, L., & Vanderstraeten, R. (2019). Measuring changes in publication patterns in a context of performance-based research funding systems: the case of educational research in the University of Gothenburg (2005–2014). Scientometrics, 118(1), 71–91. DOI: https://doi.org/10.1007/s11192-018-2963-8 

  47. Sivertsen, G. (2018). The Norwegian Model in Norway. Journal of Data and Information Science, 3(4), 3–19. DOI: https://doi.org/10.2478/jdis-2018-0017 

  48. Thomas, D. A., Nedeva, M., Tirado, M. M., & Jacob, M. (2020). Changing research on research evaluation: A critical literature review to revisit the agenda. Research Evaluation, 29(3), 275–288. DOI: https://doi.org/10.1093/reseval/rvaa008 

  49. Van Den Berghe, H., Houben, J. A., de Bruin, R. E., Moed, H. F., Kint, A., Luwel, M., & Spruyt, E. H. J. (1998). Bibliometric indicators of university research in Flanders. Journal of the American Society for Information Science, 49(1), 59–67. DOI: https://doi.org/10.1002/(SICI)1097-4571(1998)49:1<59::AID-ASI8>3.0.CO;2-2 

  50. van den Besselaar, P., Heyman, U., & Sandström, U. (2017). Perverse effects of output-based research funding? Butler’s Australian case revisited. Journal of Informetrics, 11(3), 905–918. DOI: https://doi.org/10.1016/j.joi.2017.05.016 

  51. van den Besselaar, P., & Sandström, U. (2020). Bibliometrically disciplined peer review: on using indicators in research evaluation. Scholarly assessment reports, 2(1), 5: DOI: https://doi.org/10.29024/sar.16 

  52. Van den Wijngaert, M. (2011).Van een unitair naar een federaal België: 40 jaar beleidsvorming in gemeenschappen en gewesten (1971–2011) (p. 242). Brussel: VUBPress. ISBN: 9789054879473. 

  53. Vandersmissen, J., Halleux, R., Xhayet, G., Pirot, R., Raedschelders, R., & Mettens, P. (2015). Tant qu’il y aura des chercheurs: science et politique en Belgique de 1772 à 2015. (p. 340). Editions Luc Pire, Liège. ISBN: 978-2-87542-117-3. 

  54. Vanecek, J. (2014). The effect of performance-based research funding on output of R&D results in the Czech Republic. Scientometrics, 98(1), 657–681. DOI: https://doi.org/10.1007/s11192-013-1061-1 

  55. Verleysen, F. T., & Engels, T. C. E. (2013). A label for peer-reviewed books. Journal of the American Society for Information Science and Technology, 64(2), 428–430. DOI: https://doi.org/10.1002/asi.22836 

  56. Verleysen, F. T., & Engels, T. C. E. (2018). How arbitrary are the weights assigned to books in performance-based research funding? An empirical assessment of the weight and size of monographs in Flanders. Aslib Journal of Management, 70(6), 660–672. DOI: https://doi.org/10.1108/AJIM-05-2018-0110 

  57. Zacharewicz, T., Lepori, B., Reale, M., & Jokers, K. (2019). Performance-based research funding in EU member states. Science and Public Policy, 46(1), 105–115. DOI: https://doi.org/10.1093/scipol/scy041 

comments powered by Disqus