The way in which research funding is allocated by both governmental and non-governmental research agencies needs to be revamped to avoid bias and encourage innovation. Known biases in allocation of funding include those driven by gender, race, institution size, geographic location and interdisciplinary study. We also contend that the peer-review process itself provides an apparently fair process, but that the flaws within it work against funding innovative science. We propose an unbiased process that combines the use of short proposals, blinded review and a lottery to allocate funding.

Funding, research, grants, lottery, bias, ballot, fairness

Research scientists spend substantial amounts of time preparing research grants for submission. For example, Herbert et al. (2013) report that in 2012 the Australian National Health and Medical Research Council alone received 3727 applications. Each proposal took an average of 38 working days to prepare, with an estimated effort equivalent to 550 working years at a cost of AU$66 million. While this effort is to some extent inevitable given the funds available and the competitive nature of gaining grants, it can be a frustrating process, and particularly for those who are new researchers or marginalised by biases that reduce their chances of success (see below).

We believe that the way funds are distributed requires a complete overhaul to make the process both fair and equitable, and to drive innovation and ambition. Innovation and ambition is constrained by a risk-averse approach to awarding grants, maintaining the status quo. This overhaul is necessary for two main reasons:

There has been recent interest in introducing a lottery when allocating research funding, which, if combined with blinded review, could result in a much fairer, transparent and, importantly, unbiased system (Avin 2017; Fang and Casadevall 2016a, b, c; Solans-Domenech 2017). A transparent process could allow applicants to be confident that they have been treated fairly (Gurwitz et al. 2014), and these modifications could “… increase the number of funded investigators and harness a greater diversity of tools, perspectives and creative ideas” (Fang et al. 2016a). It would also give all those within the scientific community confidence that “… if my idea is good enough, I’m good enough”.

We propose that the essential components for such a funding system are as follows:

a) Using concept notes (i.e., expressions of interest) to reduce the time spent preparing full grant applications (Barnett et al. 2015).

b) Assessing anonymised (blinded) concept notes so that the scientific idea is paramount, reducing the possibility of bias.

c) Evaluating a concept note using a simple binary 'meritorious/non-meritorious' system, with the objective of eliminating "... infeasible, poorly conceived, unoriginal or otherwise flawed applications" (Fang and Casadevall 2016a, b, c). The assessment is based on the validity and novelty of the concept. All proposals that meet this threshold advance.

d) Requesting a full application following this triage.

e) Screening the full proposal for budget, facilities, methodology, etc. to ensure that it is viable with an opportunity to correct any aspect of the proposal. This process is "peer support", not "peer review", and is focussed on supporting the researcher to deliver their "meritorious" research idea.

f) If the proposal cannot be advanced, feedback is given to promote progression at the next funding round. This is a technical evaluation and is not ranked.

g) Once screened and approved, forwarding a full proposal to a lottery. All "meritorious", full proposals are included.

h) Additional features could include limiting the number of applications that an individual researcher can have within the assessment system, and providing robust feedback for both the concept notes and the full applications. Separate calls for established and new entrant funding could also be considered.

Some of these components are already widely implemented (e.g., using concept notes and new researcher awards) but not adopted holistically. Such a scheme would be fairer to all applicants and be a catalyst for new thinking. All researchers could be confident that the forces of bias are minimised, and that decisions made are less risk-averse. Such a randomisation approach would broaden the number of institutions, and the number of individuals and range of researchers that gain funding (Box 2). Those with established track records will still have an advantage; they have the platform to submit well-argued and innovative applications from a position of strength. However, in this system their advantage is implicit as it is the idea itself rather than the cyclical, self-fulfilling grading of their proposal based on their previous track record.

Although our proposed approach would not lead to an increase in funded projects, it could provide a platform for research funding to be more equitably distributed. We are not arguing that the current system results in poor science. Rather, we argue that systems should encourage scientists to submit their ideas in the knowledge that they have a genuine chance of gaining funding, where bias, prejudice, and caution are minimised. This would encourage innovation and the development of new ideas. While no system of allocating research funds can be perfect, such an impartial and transparent funding system should be welcomed by the scientific community.

Linton Winder conceptualised the idea, prepared the first draft of the main text and edited text in Box 1 and Box 2. Simon Hodge conducted all the analyses in Boxes 1 and 2, prepared the first drafts of these sections and edited the main text. The authors contributed equally (50:50) to this paper.

The authors are grateful to a UK research funding agency that kindly provided data used in Box 1 and Box 2. The authors would also like to thank two referees, the Subject Editor and the Editor in Chief for helping us to substantially improve this paper.

Box 1: Sensitivity of the scoring process for funding decisions

Grant applications to an anonymous United Kingdom funding body (UKFB) are scored by reviewers from 0 to 7, and a mean score is obtained (Anon 2017). We obtained these funding data from eight panel meetings, encompassing 905 grant applications. The highest scores each ranged between 5.8 and 6.6, and the lowest between 1 and 2.9, respectively. In total, 192 applications (21.2%) were funded (Table 1). Individual reviewers gave scores to one decimal place, but differences between scores for the lowest-scoring funded application and highest scoring non-funded application were always to at least the second decimal place. These small differences suggest that a small variation in the score given by any one reviewer, or inter-application variation in reviewers, could dictate whether an application was funded or not.

Table 1.Download as CSV 

Summary of grant applications, score evaluation, and numbers funded by an anonymous United Kingdom funding body from eight panels.

Panel	Applications	Funded	Applications Funded (%)	Highest score	Lowest score	Lowest score funded	Lowest rank funded	Break in funding sequence	Applications above last funded	Score difference: (lowest funded – highest not funded)
1	103	23	22.3	5.8	2.6	5.00	31	Yes	8	0.01
2	83	22	26.5	6.2	2.0	5.25	33	Yes	11	0.05
3	97	19	19.6	5.8	2.9	5.23	20	Yes	1	0.01
4	118	26	22.0	6.5	2.8	5.14	46	Yes	20	0.01
5	129	27	20.9	5.9	2.0	5.10	34	Yes	7	0.01
6	94	22	23.4	6.6	2.9	4.84	58	Yes	36	0.01
7	131	24	18.3	6.3	1.0	5.00	46	Yes	22	0.0001
8	150	29	19.3	6.3	2.6	5.32	41	Yes	12	0.07

To assess how small, random changes to scores influenced the likelihood of funding, we ran a simulation (40 runs) on the Panel 8 data. To each score, we added a random number based on a normal distribution with mean = zero and standard deviation (SD) ranging from 0.01 to 0.35. After adjusting the scores and re-ranking, we assessed the proportion of simulations that resulted in all the originally funded applications being ‘re-funded’. There was a clear negative relationship with consistency of funding and the score change SD; with an adjustment SD > 0.25, we recorded no occasions when all the previously awardees received funding (Figure 1). With a SD = 0.25, the average score change was approximately ± 0.2 marks (Table 1), which represented 3% (0.2 ÷ 6.6 × 100) of the highest score awarded). This suggests that even small changes in scores results in consistent changes to the pool of applicants that were awarded funding.

Figure 1.

Results of adding a small random number to funding application scores for Panel 8 of the anonymous United Kingdom funding data: percentage (%) of 40 simulations that resulted in all previously funded applications being awarded funding.

Box 2. What if funding bodies played dice? Sensitivity of the funding process to varying random selection

Given that applications to the UKFB considered ‘fundable’, but not actually funded, were separated by very small score differences from those that were successful (Table 1), the process is clearly sensitive to even small fluctuations in marks awarded by evaluators (Box 1). So, rather than fund applications strictly based on the final funding score, what if various rules were applied so that more or less equally scored applications all had a chance of being funded?

It appears that some rule other than just funding the top-scoring applications was the case for these data; in each year, when the applications were ranked in terms of scores, a break in the funding sequence occurred. This indicates that some applications, although scoring highly, were overlooked in favour of applications with lower scores (Table 1). In one panel (Panel 6) the lowest scoring application was ranked 58 out of 94 applications when only 22 projects were funded; 36 applications scored more highly than this application and yet were not funded.

What happens when we introduce more randomness to the decision process so that applications in the ‘fundable zone’ all had a chance of receiving funding? In Panel 8 of the data, there were 150 applications, of which 147 were deemed fundable after an initial screening process, and 29 were actually funded. By using a bootstrap-resampling process (1000 iterations), we applied a series of selection rules to these data, each using various randomness to select ‘winners’ from the 147 fundable applications. We then ascertained how many previously unfunded applications would now be funded in each case (Table 2). Naturally, if all 29 funded grants were selected at random from all those considered fundable, a high percentage (80.1%) of previously unfunded grants would be funded (Scenario 1). However, even when we applied much stricter rules such as awarding all applications with a score ≥ 6 (only six applications), and then selecting the remainder at random from those with scores of ≥ 5, nearly half (44.8%) of the applications now awarded funding did not receive funding in the actual funding round (Table 2; Scenario 6).

Table 2.Download as CSV 

The effect of using different random selections by including new awardees from those previously funded in Year 8 of the anonymous United Kingdom funding body data.

	Scenario	New grants funded (%) (95% bootstrap CI)
1	29 funded at random from all 147 fundable applications	80.1 (86.2–75.9)
2	29 funded at random from Top 50 scores	41.9 (48.3–34.5)
3	29 funded at random from those with score ≥ 5 (n = 59)	50.9 (55.2–44.8)
4	Top 10 scores all funded. 19 selected at random from remainder (n = 137)	56.2 (58.6–51.7)
5	Score ≥ 6 all funded. 23 funded at random from remainder (n = 141)	66.4 (72.4–62.1)
6	Score ≥ 6 all funded. 23 funded at random from those with score ≥ 5 (n = 53)	44.8 (51.7 - 37.9)