Hauke Hillebrandt

Apr 2020

Ideas for UK ARPA science funding proposals

Open in Google Docs

Quick note on what one should propose to the UK government what ARPA should focus on

  1. Biosecurity: broad spectrum antivirals and universal viral diagnostics

“The risk of a globally catastrophic pandemic would be substantially reduced:

  1. A better selection of well-stocked, broad-spectrum antiviral compounds with low potential for development of resistance
  2. Ability to confer immunity against a novel pathogen in fewer than 100 days
  3. Widespread implementation of intrinsic biocontainment technologies that can reliably contain viral pathogens in the lab without impairing research
  4. Improved countermeasures for non-viral conventional pathogens
  5. Rapid, inexpensive, point-of-care diagnostics for all known pathogens
  6. Inexpensive, ubiquitous metagenomic sequencing
  7. Targeted countermeasures for the most dangerous viral pathogens

Of the areas discussed above, our impression based on our research is that:

Broad-spectrum antivirals are receiving limited attention. We’re planning to fund work in this area.

Vaccine R&D is generally a crowded space, though it seems possible to us that a deeper investigation into more of the specific subtopics would reveal additional opportunities.

Diagnostics R&D seems highly crowded.”

https://www.openphilanthropy.org/research/cause-reports/research-and-development-decrease-biosecurity-risks-viral-pathogens

  1. The Open Philanthropy Project announced a grant and an investment in Sherlock Biosciences to develop a universal viral diagnostic

The Open Philanthropy Project recommended a grant of up to $17,500,000 over five years and an additional investment in Sherlock Biosciences to support the development of a diagnostic platform to quickly, easily, and inexpensively identify any human virus present in a patient sample. Development of this technology would represent a significant advance in viral diagnosis, and could both reduce threats from viral pandemics and also benefit health care broadly. Sherlock recently spun out of Harvard University’s Wyss Institute and the Broad Institute of MIT and Harvard.

Example UK researcher to fund: Sam Jones (Imperial) Jones Lab 

2. AI: Designing robust and reliable AI systems

“A lot of people see ML safety work as a sort of tax. That you have to pay this tax to make sure that you are doing things correctly. Safety is something that is necessary, you have to do it, but if you’re not driven by it. I don’t see it that way in some sense. So it’s not … I don’t think basically an organization which is doing safety work is paying a tax. In fact, it is to its advantage. How do we explain this?

Pushmeet Kohli: Suppose, we were, you and me, took a unique mission and the mission was that we’re going to drive a car around the planet. One approach is you sit in your car seat and you drive off, without putting your seatbelt. The other issue would be, you put your seatbelt on. If we were just going from point A to point B and like maybe one kilometer, the probability of an accident is so low that you might actually reach the destination without putting the seatbelt on. But if your destination is so far, we are circumnavigating the whole world, if we think about the probability of who reaches the end, you will see the person who put on the seatbelt.”[1]

See Identifying and eliminating bugs in learned predictive models 

Example UK researcher: Stuart Armstrong (Oxford)

3. Clean Energy Innovation

UK should invest in Clean Energy Innovation - modelled after ARPA-E

“Modeled after the highly successful Defense Advanced Research Projects Agency (DARPA), the Advanced Research Projects Agency-Energy (ARPA-E) advances high-potential, high impact energy technologies that could radically improve U.S. economic prosperity, national security, and environmental well-being, but are too early for private-sector investment. Its grants help fund energy innovators that are developing technologies to solve critical cross-cutting, real-world problems in transportation, electricity, building, and other sectors.“[2]

UK relevance: Oxford Photovoltaics 

4. MetaScience

It is now standard for many grant awarding bodies to require preregistration of clinical trials.

Yet, preregistration still leaves a lot of room for so-called questionable research practices.

All hypothesis driven science funding from UK ARPA should be awarded in the novel Registered Reports format pioneered by UK Professor Chris Chambers.

Improving the Efficiency of Grant and Journal Peer Review: Registered Reports Funding

But from climate science to cancer biology, science is facing a replication crisis. Lots of research findings can’t be repeated and billions of valuable research funding dollars might be wasted in the process.

There’s a new and better way of doing and publishing research, called Registered Reports, where research is peer-reviewed before the results are known.

While not a ‘panacea’, Registered Reports might:

Currently, science is only peer-reviewed after the results are known. In contrast, in the Registered Reports format the peer reviewers accept scientific papers based on the methodology:

registeredreports_graphic

This format is applicable to a lot of areas in science and might help with the replication crisis.

Grant funding that could be awarded based on the first stage of peer review. Researchers could submit their introduction and methodology as well as their analysis plan, and peer reviewers could be responsible for awarding funding relative to the cost of the study. This could lead to more meritocratic funding decisions where the most interesting studies are run instead of those that are run by researchers with the best track record (who will be most likely to be awarded grants). One paper argued that “Combining grant funding and publication decisions into a single, two-stage process promises to dramatically reduce the burden on reviewers”.[209] There are currently several tests run by a few funders and journals with awarding grant funding for the first stage of Registered Reports. For instance, the journal Nicotine and Tobacco Research has partnered with Cancer Research UK [210], the journal PLoS One has partnered with the Children’s Tumor Foundation for the “Drug Discovery Initiative Registered Report (DDIRR) 2017 Awards”. [211]

5. Data-driven Scientometrics to find grantees and evaluate output and prizes

Scientometrics is the field of study which concerns itself with measuring and analysing scientific literature such as the impact of research papers and academic journals.

Grant selection peer review scores are a poor predictor of grant productivity.[3]

Scientometrics should be used both to assist in grant making and grant evaluation.

“Of course, such tools cannot substitute for substantive knowledge of human experts, but they can be used as powerful decision support systems to structure humans’ effort and augment their capabilities/efficiency to handle the enormous volume of data on research input and output”

https://link.springer.com/article/10.1007%2Fs11192-020-03466-w 

Focus on early career researchers

Career age can sometimes negatively affect the quality of scientific work and young researchers who work in large teams are more likely to produce high quality publications.[4] Awarding grants to postdocs instead of PI can increase research output.[5]

Premature deaths of eminent life scientists can improve scientific productivity.[6]

Use of Scientometrics

Scientometric indicators and machine learning-based models can be used to predict the ‘rising stars’ in academia [7], [8]—identifying these junior researchers and awarding them prizes would greatly improve research output.

“Grant programs must be designed in ways that recognize the possibility of failure. This entails encouraging recipients to take on scientific and technological risks, exploring new research avenues rather than sticking with safer and more conventional trajectories. “[9]

“Grant systems also face implementation challenges. Scholars have noted the inefficiency inherent to a system where much of the effort sunk into writing unfunded proposals appears to be wasted (Gross and Bergstrom 2019); they have commented on the unfairness of a system within which female and minority applicants appear to fare less well on average than white, male, or Asian applicants (Ginther et al. 2011); they have provided evidence that peer review sometimes filters out the most novel or creative proposals (Boudreau et al. 2016), or worse, induces scientists to skew their agenda towards projects more likely to generate results in the short term (Azoulay et al. 2011).

Why, then, do grants exist?

We argue that grants are likely to be the most effective—and feasible—way to fund basic research when two fundamental conditions simultaneously hold. First, when the social value of a scientific finding likely exceeds its privately appropriable value. Second, when specifying the parameters of a desired research solution ahead of time is impossible. These twin conditions would appear to characterize much exploratory and early-stage research that is often labeled “basic” or “pure.” We will also discuss two subsidiary arguments in favor grant funding over alternative mechanisms: when potential research performers face financial constraints, and when investments take the form of general-purpose research infrastructure (as opposed to specific projects).”

This paper is interesting:

Highlights

“The STM algorithm, in particular, allowed us to incorporate document-topic proportions into the model and analyze topic prevalence over time. Our findings suggest that in general the projects of Canadian researchers in natural sciences and engineering were in line with the top research priorities of the country as well as NSERC’s. The appearance of environmental and climate projects among the important topics was notable, which indicates the importance of environmental issues in the Canadian scientific community. The country’s intensive investment in AI and emerging technologies that was reflected as the rise of analytics in the projects is also noticeable.

“Bug bounty” prizes

A bug bounty program is a deal offered by organizations such as Facebook, Google, and  Microsoft, by which individuals can receive compensation for reporting software bugs, especially those pertaining to security exploits and vulnerabilities.

Similarly, in science, serious flaws in widely used scientific methods are found - which sometimes invalidates the findings of whole scientific subdisciplines.[10],[11] This is not only the case for softer sciences like psychology: a surprisingly high number of scientists from harder sciences such as physics and chemistry report having failed to reproduce experiments by others in their own fields.[12] However, there is little incentive for researchers to point out and prizes should be awarded based on pointing out common errors in different fields.

Other points that might be of interest

Notes

[1] "DeepMind's plan to make AI systems robust & reliable, why it's ...." 3 Jun. 2019, https://80000hours.org/podcast/episodes/pushmeet-kohli-deepmind-safety-research/. Accessed 24 Apr. 2020.

[2] "Federal Energy R&D: ARPA-E - ITIF." 16 Mar. 2020, http://www2.itif.org/2020-budget-arpa-e.pdf. Accessed 24 Apr. 2020.

[3] "Research: NIH peer review percentile scores are poorly ... - eLife." 16 Feb. 2016, https://elifesciences.org/articles/13323. Accessed 25 Apr. 2020.

[4] "How to boost scientific production? A statistical analysis of ...." 4 Jan. 2016, https://link.springer.com/article/10.1007/s11192-015-1825-x. Accessed 25 Apr. 2020.

[5] "The Impact of NIH Postdoctoral Training Grants on Scientific ...." https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3158578/. Accessed 25 Apr. 2020.

[6] "Does Science Advance One Funeral at a Time? - American ...." https://www.aeaweb.org/articles?id=10.1257/aer.20161574. Accessed 25 Apr. 2020.

[7] "Scientometric Indicators and Machine Learning-Based Models ...." https://ieeexplore.ieee.org/abstract/document/8843686/. Accessed 25 Apr. 2020.

[8] "Finding rising stars in bibliometric networks | SpringerLink." 10 Apr. 2020, https://link.springer.com/10.1007/s11192-020-03466-w. Accessed 25 Apr. 2020.

[9] "Scientific Grant Funding - NBER." 6 Mar. 2020, https://www.nber.org/papers/w26889. Accessed 25 Apr. 2020.

[10] "Erroneous analyses of interactions in neuroscience: a ... - Nature." 26 Aug. 2011, https://www.nature.com/articles/nn.2886. Accessed 25 Apr. 2020.

[11] "Controversial software is proving surprisingly accurate at ...." 28 Nov. 2017, https://www.sciencemag.org/news/2017/11/controversial-software-proving-surprisingly-accurate-spotting-errors-psychology-papers. Accessed 25 Apr. 2020.

[12] "1,500 scientists lift the lid on reproducibility: Nature News & Comment." 25 May. 2016, http://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970. Accessed 16 Jan. 2017.