A manifesto for fair and equitable research funding in ecology

The way in which research funding is allocated by both governmental and non-governmental research agencies needs to be revamped to avoid bias and encourage innovation. Known biases in allocation of funding include those driven by gender, race, institution size, geographic location and interdisciplinary study. We also contend that the peer-review process itself provides an apparently fair process, but that the flaws within it work against funding innovative science. We propose an unbiased process that combines the use of short proposals, blinded review and a lottery to allocate funding.


INTRODUCTION
Research scientists spend substantial amounts of time preparing research grants for submission.For example, Herbert et al. (2013) report that in 2012 the Australian National Health and Medical Research Council alone received 3727 applications.Each proposal took an average of 38 working days to prepare, with an estimated effort equivalent to 550 working years at a cost of AU$66 million.While this effort is to some extent inevitable given the funds available and the competitive nature of gaining grants, it can be a frustrating process, and particularly for those who are new researchers or marginalised by biases that reduce their chances of success (see below).
We believe that the way funds are distributed requires a complete overhaul to make the process both fair and equitable, and to drive innovation and ambition.Innovation and ambition is constrained by a risk-averse approach to awarding grants, maintaining the status quo.This overhaul is necessary for two main reasons:

The process is biased
There is substantial evidence that bias in many forms impacts on the grant-review process, leading to outcomes that are neither fair nor in the interest of science itself.Known negative biases include those driven by gender (Pohlhaus 2011;Tricco et al. 2017), race (Ginther et al. 2011), institution size (Murray et al. 2016), geographic location (Wahls 2016), and interdisciplinary study (Bromham et al. 2016).Additionally, Boudreau et al. (2016) found a tendency for reviewers with a shorter intellectual distance from an application to provide harsher evaluations even when they were highly innovative.Other factors can also influence decision making, such as the bias towards the perceived value of high-income-country research (Harris et al. 2017).
Known positive biases include the distribution of funding to a disproportionate minority of investigators (Wahls 2017).The tendency to reward eminence or track record by giving them more weight to funding applications is, as Vazire (2017) points out, "… like giving Usain Bolt a 10 meter head start in his next race".This results in elite scientists over-attracting resources (Ma et al. 2015;Szell and Sinatra 2015), with a bias towards an orthodoxy where well-established laboratories are favoured over those that are small or newly established (Alberts 1985, Alberts et al. 2014;Daniels 2015) even when established research units might deliver reduced productivity ovr time (Lorsch 2015).
Such positive and negative biases in the funding process does not mean that poor science is being funded, but it does result in a culture where some researchers are marginalised and innovation is stifled.In a newspaper interview, the Nobel Prize winner, Sir James Black (who created medicines for the prevention of heart attacks and stomach ulcers), stated that "… the process of peer review is the enemy of scientific creativ-ity… it tends to reinforce the majority research program by withdrawing funding from its competitors" (see Labini 2016).

Peer review is not sufficiently reliable for decision-making purposes
Putting scientists into a room to rank, debate and award research funding appears like a fair process.Yet there is substantial evidence that this process has weaknesses (Ballabeni et al. 2016;Cicchetti 1991;Fang and Casadevall 2009;Nicholson and Ioannidis 2012;Staats 2014;Danthi et al. 2015;Gallo et al. 2014).Decisions regarding the funding of grant applications can be influenced by small differences in the scores given by reviewers (Box 1).Evidence of poor precision when making judgements (Kaplan et al. 2008) and the impact of expertise on evaluation (Gallo et al. 2016) compromise the process further, whilst panels' reliance on bibliometric measures of success (impact factor, etc.) tend to deliver risk-averse funding decisions (Stephan 2017).
When the grant-awarding process has been systematically evaluated, there appears little evidence that peer-review scores reliably predict the subsequent success of research projects (Danthi et al. 2014;Kaltman et al. 2014;Danthi et al. 2015).Although a study by Li and Agha (2015) indicated that panel rankings predicted good research outcomes (in terms of publication record), a subsequent reanalysis by Fang et al. (2016) found no such effect.Even if panel discussion rather than ranking is used to evaluate proposals, there is little evidence that the reliability of peer review is improved (Fogelholm et al. 2012).In fact, committees are often forced into choices where essentially arbitrary decisions are made from a pool of applications that are all considered more or less equally fundable (Powell 2010).
Such observations might be uncomfortable for the scientific establishment, because, the peer review process for grant allocation is embedded in our collective research culture.Our trust in a small panel of scientists being able to evaluate and assess often large numbers of varied proposals is not warranted.This problem is magnified by the use of 'introducing members' (where one or two panel members are asked to evaluate an application fully), which further restricts scrutiny and is likely to introduce personal bias.Ballabeni et al. (2016) found strong support for a process that distributes funds equitably among individuals and establishments, preventing marginalisation of smaller or new research groups, and thus avoiding the "… incumbency advantage" that favours "… insiders and the familiar" over the unknown.

THERE MUST BE A BETTER WAY
There has been recent interest in introducing a lottery when allocating research funding, which, if combined with blinded review, could result in a much fairer, transparent and, importantly, unbiased system (Avin 2017;Fang and Casadevall 2016a, b, c;Solans-Domenech 2017).A transparent process could allow applicants to be confident that they have been treated fairly (Gurwitz et al. 2014), and these modifications could "… increase the number of funded investigators and harness a greater diversity of tools, perspectives and creative ideas" (Fang et al. 2016a).It would also give all those within the scientific community confidence that "… if my idea is good enough, I'm good enough".
We propose that the essential components for such a funding system are as follows: a) Using concept notes (i.e., expressions of interest) to reduce the time spent preparing full grant applications (Barnett et al. 2015).b) Assessing anonymised (blinded) concept notes so that the scientific idea is paramount, reducing the possibility of bias.
c) Evaluating a concept note using a simple binary 'meritorious/non-meritorious' system, with the objective of eliminating "… infeasible, poorly conceived, unoriginal or otherwise flawed applications" (Fang and Casadevall 2016 a, b, c).
The assessment is based on the validity and novelty of the concept.All proposals that meet this threshold advance.d) Requesting a full application following this triage.e) Screening the full proposal for budget, facilities, methodology, etc. to ensure that it is viable with an opportunity to correct any aspect of the proposal.This process is 'peer support', not 'peer review', and is focussed on supporting the researcher to deliver their 'meritorious' research idea.f ) If the proposal cannot be advanced, feedback is given to promote progression at the next funding round.This is a technical evaluation and is not ranked.g) Once screened and approved, forwarding a full proposal to a lottery.All 'meritorious', full proposals are included.h) Additional features could include limiting the number of applications that an individual researcher can have within the assessment system, and providing robust feedback for both the concept notes and the full applications.Separate calls for established and new entrant funding could also be considered.
Some of these components are already widely implemented (e.g., using concept notes and new researcher awards) but not adopted holistically.Such a scheme would be fairer to all applicants and be a catalyst for new thinking.All researchers could be confident that the forces of bias are minimised, and that decisions made are less risk-averse.Such a randomisation approach would broaden the number of institutions, and the number of individuals and range of researchers that gain funding (Box 2).Those with established track records will still have an advantage; they have the platform to submit well-argued and innovative applications from a position of strength.However, in this system their advantage is implicit as it is the idea itself rather than the cyclical, self-fulfilling grading of their proposal based on their previous track record.

CONCLUSION
Although our proposed approach would not lead to an increase in funded projects, it could provide a platform for research funding to be more equitably distributed.We are not arguing that the current system results in poor science.Rather, we argue that systems should encourage scientists to submit their ideas in the knowledge that they have a genuine chance of gaining funding, where bias, prejudice, and caution are minimised.This would encourage innovation and the development of new ideas.While no system of allocating research funds can be perfect, such an impartial and transparent funding system should be welcomed by the scientific community.

Appendix 1
Box 1: Sensitivity of the scoring process for funding decisions Grant applications to an anonymous United Kingdom funding body (UKFB) are scored by reviewers from 0 to 7, and a mean score is obtained (Anon. 2017).We obtained these funding data from eight panel meetings, encompassing 905 grant applications.The highest scores each ranged between 5.8 and 6.6, and the lowest between 1 and 2.9, respectively.In total, 192 applications (21.2%) were funded (Table 1).Individual reviewers gave scores to one decimal place, but differences between scores for the lowest-scoring funded application and highest scoring non-funded application were always to at least the second decimal place.These small differences suggest that a small variation in the score given by any one reviewer, or inter-application variation in reviewers, could dictate whether an application was funded or not.
To assess how small, random changes to scores influenced the likelihood of funding, we ran a simulation (40 runs) on the Panel 8 data.To each score, we added a random number based on a normal distribution with mean = zero and standard deviation (SD) ranging from 0.01 to 0.35.After adjusting the scores and re-ranking, we assessed the proportion of simulations that resulted in all the originally funded applications being 're-funded'.There was a clear negative relationship with consistency of funding and the score change SD; with an adjustment SD > 0.25, we recorded no occasions when all the previously awardees received funding (Figure 1).With a SD = 0.25, the average score change was approximately ± 0.2 marks (Table 1), which represented 3% (0.2 ÷ 6.6 × 100) of the highest score awarded).This suggests that even small changes in scores results in consistent changes to the pool of applicants that were awarded funding.Given that applications to the UKFB considered 'fundable', but not actually funded, were separated by very small score differences from those that were successful (Table 1), the process is clearly sensitive to even small fluctuations in marks awarded by evaluators (Box 1).So, rather than fund applications strictly based on the final funding score, what if various rules were applied so that more or less equally scored applications all had a chance of being funded?It appears that some rule other than just funding the top-scoring applications was the case for these data; in each year, when the applications were ranked in terms of scores, a break in the funding sequence occurred.This indicates that some applications, although scoring highly, were overlooked in favour of applications with lower scores (Table 1).In one panel (Panel 6) the lowest scoring application was ranked 58 out of 94 applications when only 22 projects were funded; 36 applications scored more highly than this application and yet were not funded.
What happens when we introduce more randomness to the decision process so that applications in the 'fundable zone' all had a chance of receiving funding?In Panel 8 of the data, there were 150 applications, of which 147 were deemed fundable after an initial screening process, and 29 were actually funded.By using a bootstrap-resampling process (1000 iterations), we applied a series of selection rules to these data, each using various randomness to select 'winners' from the 147 fundable applications.We then ascertained how many previously unfunded applications would now be funded in each case (Table 2).Naturally, if all 29 funded grants were selected at random from all those considered fundable, a high percentage (80.1%) of previously unfunded grants would be funded (Scenario 1).However, even when we applied much stricter rules such as awarding all applications with a score ≥ 6 (only six applications), and then selecting the remainder at random from those with scores of ≥ 5, nearly half (44.8%) of the applications now awarded funding did not receive funding in the actual funding round (Table 2; Scenario 6).

Figure 1 .
Figure 1.Results of adding a small random number to funding application scores for Panel 8 of the anonymous United Kingdom funding data: percentage (%) of 40 simulations that resulted in all previously funded applications being awarded funding.

Table 1 .
Summary of grant applications, score evaluation, and numbers funded by an anonymous United Kingdom funding body from eight panels.