The rapid advancement of AI creates dual-use risks that current research governance mechanisms are ill-equipped to address.

Published research can inadvertently proliferate capabilities that empower malicious actors. Examples are wide ranging across scientific disciplines and include:

• LLM Agents can Exploit Zero-Day Vulnerabilities

• AI can synthesize high-quality personalized speech with only a 3-second enrolled recording of an unseen speaker scams

• LLMs to assist non-experts in designing pathogens

• Biological design tools (BDTs) can reduce speed up testing of molecules in the lab, thereby proliferating e.g. the capability to design and manufacture bio weapons. BDTs present fundamentally distinct governance challenges to frontier AI models for various reasons: most models are open source; there are strong open science norms in the biomedical sciences; the space is highly decentralised and not dominated by a few actors; some powerful models are relatively cheap to train; it is not clear that compute used in training is a good proxy for risk; and biorisk is highly offence-dominant.

• An LLM chemistry agent, ChemCrow, designed to automate a diverse set of chemical tasks might have dual-use concerns

Research Ethics Committees (RECs) have a well-established history in ethics governance, but there have been concerns about their capacity to adequately govern research - especially AI research with dual-use concerns.

Current ethical review of research has key weaknesses, in terms of safeguarding against the publication of potentially dangerous research:

1. Expertise Gap: Traditional institutional review boards are often internal, sometimes by consisting of non-experts, and often lack specific expertise in emerging capabilities and their potential dual-use implications- particularly at the intersection of AI and synthetic biology. In some countries, ethics review might have low standards.

2. No special focus on dual-use risks: Ethical review boards might 'bikeshed' around issues they understand well (e.g. animal welfare in biological research) and not focus enough on dual-use risks.

3. Timing of Review: Ethical review often occurs after preliminary research has already been carried out, or significant resources have been committed creating institutional pressure to proceed despite identified risks.

While some ethics review committee conduct internal reviews pre-publication and some journals like NeurIPS have introduced a requirement that all paper submissions include a statement of the “potential broader impact of their work, including its ethical aspects and future societal consequences”, there is as of yet no standardized framework for pre-registration of research that integrates scientific merit review with rigorous security and ethics assessment by independent experts assisted by AI tools.

This creates an urgent need for improved governance mechanisms where academic subject-matter experts are nudged to identify dual-use concerns early in the research process. The problem requires an international solution that can work within existing scientific institutions.

Registered Reports introduce a new approach to scientific practice. They have the potential to impact various scientific fields due to the commonalities in research paper structure, methodology, and statistical tests. This cross-cutting tool enhances systemic safety by prioritizing indirect, broadly applicable contributions that would reduce dual-use concerns across all scientific disciplines. Funding such meta-research is more effective than funding individual science projects (just as a lake can contain a lot more water than a well, even if the well is deeper). 

I propose to extend Registered Reports to the "Trinity Review" system to improve the efficiency and safety of scientific research. This combines into a single process not only scientific peer review and research funding reviews (i.e. the introduction and proposed methodology of a paper works is a grant application), but, most crucially, also ethics review. This ensures that the most relevant experts with the help of AI evaluate the ethical aspects of research (with focus on dual-use risks), improving the safety of research, particularly AI-enabled research, and lowering the risk of dual-use research. 

It achieves this by:

• Enabling subject matter experts (perhaps assisted AI tools and LLMs) to recommend specific risk mitigation measures or, when necessary, advise against conducting the research entirely

• Integrating funding review only after scientific and ethical approval, ensuring resources aren't committed to potentially dangerous research

Feasibility

Researchers and publications have strong incentives to implement this change as it reduces paperwork and they’re incentivized by grant funding via this new format.

Innovation

• Aird et al's survey suggests AI governance experts want to change AI publication norms to encourage less immediate and complete publication.

• Sandbrink et al propose that early-stage peer review in biology, similar to registered reports, could include biosecurity experts to assess and mitigate potential risks, possibly advising against certain research altogether. This approach aims to address dual-use risks more effectively than suppressing information after research is completed.

References

• Ethical AI governance: mapping a research ecosystem

• The Asilomar Conference: A Case Study in Risk Mitigation Katja Grace

• Artificial intelligence and biological misuse: Differentiating risks of language models and biological design tools

• Rewarding Chatbots for Real-World Engagement with Millions of Users 

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

• The offense-defense balance of scientific knowledge: does publishing AI research reduce misuse? 

• Artificial Intelligence Research Needs Responsible Publication Norms

• Publication Norms for Responsible AI

• Most AI research shouldn't be publicly released

• The Artefacts of Intelligence: Governing Scientists' Contribution to AI Proliferation | GovAI