Overview
We will run a two-stage review process consisting of a peer review and sift followed by an expert assessment panel. For the sift stage, we may use partial randomisation in the process of shortlisting which applications advance to the expert assessment panel.
Each application during the peer review and sift stage will be assessed by individuals outside of UKRI and AISI. All reviewers are expected to uphold principles of peer review assessment and decision making.
You will be notified of the outcome of the sift stage by mid-December 2024.
If your application advances to the expert assessment panel, then you must provide additional information to EPSRC by 9th January 2024; see list below.
The expert assessment panel will review all shortlisted applications against all the assessment criteria. Final selection will be made by AISI based on the expert assessment panel’s recommendations and in line with the programme’s objectives.
Depending on the level of demand for this opportunity, the delivery partners reserve the right to modify the assessment process as needed.
How will applications be shortlisted?
The shortlisting of proposals during the sift stage will be based exclusively on questions 7 and 9 of the application form, which are stated below
7) Tell us about your background, your track record, and why you are a good person or team to conduct impactful work on systemic AI safety
9) What problem does your idea solve / what risk does it address / what question does it answer?
This step will be taken to exclude proposals for which the problem/risk being addressed is out of scope for our call, or where the applicants do not have a track record of conducting research relevant to the proposal. All proposals will be subjected to peer review based on the above criteria, to deem if the proposal is likely to be competitive at the expert assessment panel. Depending on application volumes, we reserve the right to use partial randomisation when shortlisting applications for the expert panel selection stage, based on the ranking that emerges from reviewers’ scores.
Please note that by applying you agree to allow us to make public the answer to question 9 of the proposal (“What problem does your idea solve / what risk does it address / what question does it answer?”). In order to maximise the visibility of the problems in systemic AI safety, we reserve the right to publish all the proposed answers to this question (anonymously) on our website. Our goal here is to create a repository of relevant questions in systemic AI safety, that future researchers might choose to address. Note that we will NOT publish your proposed solution, or link researcher identities to any published text.
You will be notified of the outcome from the initial sift process regardless of outcome by mid-December 2024. If your application advances to the expert assessment panel, then your full proposal will be evaluated by an expert assessment panel that is recruited by UKRI and AISI.
How will the final selection be made?
If you successfully pass to panel stage, then you must provide to UKRI by the 9th of January 2025 the following additional information for the panel meeting or your application will not be considered further for funding:
- a detailed budget of the resources and costs for your proposed work and
- information on your project partners and their contribution (to enable UKRI to perform due diligence checks).
- Additional information as required to perform due diligence.
UKRI and AISI will be available to answer your questions and provide advice where possible, as you finalise your budget. Further information on eligible costs can be found on the application form the FAQ document.
Alongside an additional evaluation of the team track record and the requested budget, the expert panel assessment will be judging research proposals on the following criteria, and expect successful applications to score highly on each of them:
1. Relevant. Proposals should address a pressing risk that arises when AI is deployed in society, for private, professional or industrial use.
- Our focus is on “Frontier” AI models - those capable of the generation of text (including code), realistic images, audio, and video, including those that can use tools to take actions on digital platforms, and large-scale simulators for science-based applications (e.g. climate simulation or protein folding).
2. Feasible. We want feasible proposals that address plausible risks. Risks should be those that could plausibly arise from AI over the near term (e.g., a 2–5-year time horizon). We imagine that over this time horizon, there will be significant progress in AI development, that will make Frontier AI systems:
- More personalised to the user.
- Better able to take sequences of actions on digital platforms.
- More densely interconnected, and likely to share information with each other.
3. Innovative. Proposals that address risks or issues that are currently under-explored or under-considered are particularly welcome.
4. Actionable. Proposals should result in clear, actionable deliverables. For example, this could be:
- a dataset that helps us understand an emerging trend in AI usage.
- an application that helps us track or monitor for hazards.
- a system map that identifies vulnerabilities in critical infrastructure.
- a computer simulation that surfaces new ideas about the threats posed by AI.
- a detailed analysis of stakeholders, capacities and incentives for AI adoption in a sector, that can help prioritise and deliver interventions.
- a technical protocol that enables the interoperable identification of AI agents.
- a policy paper that rigorously evaluates downstream impacts, weighs the proposal against other options, and draws lessons from case studies.
- A white paper or policy note for regulators.
Proposals for which the only deliverable is a literature review are unlikely to be successful. Please see the applicant guidance for more details on the assessment criteria and scoring.
Final selection will be made by AISI based on the expert assessment panel’s recommendations and in line with the programme’s objectives. Depending on the level of demand for this opportunity, UKRI reserves the right to modify the assessment process as needed.
If your proposal is selected for a potential award, AISI and the applicant will discuss, negotiate and agree on the project activities and milestones. We will conduct due diligence and assurance checks before an award is made, in accordance with our internal guidelines. This might include verifying the identity and institutional associations of applicants and project partners, the capacity within the awarding institution to administer and deliver the award, and any additional financial due diligence checks.
The successful applicants should be prepared to start their grants on the 5th February 2025.
See the full assessment criteria here.
