At the occasion of the 25th PRACE Council Meeting in Amsterdam, the PRACE Members ratified a Resolution to proceed with the second phase of their Partnership: PRACE 2. The PRACE 2 programme defines the second period of PRACE from 2017 to 2020. With this agreement, PRACE will strengthen Europe’s position as world-class scientific supercomputing provider, a technology considered a key enabler for knowledge development, scientific research, big data analytics, solving global and societal challenges, and European industrial competitiveness.
See the corresponding press release here.
Annex IV of this Resolution explains the Peer Review Process under the PRACE 2 Programme.
The PRACE peer review process is based on the following principles:
- Transparency: The peer review process is transparent and clear to all stakeholders in PRACE including funding agencies of all member countries and users from research institutions and industry
- Fairness: Proposals are evaluated on merit and potential high impact on European and international science and economy
- No parallel assessment: The PRACE peer review process builds on the experiences and best practices of national and international institutions and constitutes a centralised peer review exercise recognised by all PRACE partner countries and scientific communities
- Reviews are done by experts in the scientific field of the proposal, with no declared conflict of interest, based on criteria published in the PRACE Calls for Proposals and with a periodic reshuffling of reviewer’s appointments
- Confidentiality: Proposals will be treated with the needed confidentiality by PRACE staff and reviewers. The identities of the peer reviewers shall not be disclosed
- Right to reply to technical and scientific evaluations.
The PRACE Peer review includes two types of assessment: technical and scientific. The two assessments are carried out separately by different groups of experts with the applicant having the right to reply to the evaluation outcome of both the technical and the scientific reviewers (if necessary). The technical review seeks to assure that the proposal is technically feasible for the system(s) requested. At the end of the scientific review, all applicants receive documentation of both the technical and scientific evaluation of their proposal.
1. Technical assessment
All proposals will undergo a technical assessment. The technical assessment can result in three outcomes:
- Accepted. The application fulfills all technical requirements to run on the selected system(s).
- Conditionally accepted. The application does not meet the technical requirements to run in the selected system(s), but the technical reviewers can identify the measure and time frame necessary to meet them.
- Rejected. The application does not meet the minimum technical requirements to run in the selected system(s).
Criteria for technical assessment
A proposal must convincingly address the following criteria:
- The need to use PRACE resources
- Software availability on the requested system(s). The codes necessary for the project must already be available on the system requested or, in case of codes developed by the appli-cants, they must have been sufficiently tested for efficiency, high scalability, and suitability. For Project Access, proposals proof of successful tests must be submitted together with the proposal. Applicants usually provide this proof after a Preparatory Access application.
- Feasibility of the requested resource. The requested system(s) must be suitable for the proposed project. The technical assessment may redirect projects to a more appropriate system.
These criteria should be fully addressed in the application. Technical reviewers will assess proposals against these criteria.
2. Scientific assessment
Scientific review is performed by internationally recognized experts in the field of research of the proposal. During the scientific assessment an increase or decrease in the requested resources can be recommended for consideration during resource allocation.
Criteria for scientific assessment
The proposals must address the following scientific criteria:
- Scientific excellence. The proposed research must demonstrate scientific excellence and a potential for high European and international impact
- Novelty and transformative qualities. Proposals should be novel, develop an important scientific topic of major relevance to European research, describe possible transformative aspects, and expected advances
- Relevance to the call. Proposal needs to address how the proposed project is addressing the scope of the call if a specific scope is stated in the call
- Methodology. The methodology used should be described and appropriate to achieve the goals of the project
- Dissemination. The planned channels and resources for dissemination and knowledge exchange should be described
- Management. There must be a solid management structure which will ensure that the project will be successfully completed.
These criteria should be fully addressed in the application. Reviewers will assess proposals against these criteria. Find here the guidelines for scientific reviewers.
Role of reviewers
The PRACE staff handling peer review is responsible for obtaining high quality non-conflicted technical and scientific reviews for each proposal.
The technical reviewers are experienced technical experts in PRACE systems and software. Technical reviewers will provide a report on the technical quality (code, methodology, etc.) of proposals and on the suitability of the project for the system/architecture requested, and the extent of the request.
Scientific reviewers will provide a report on the novelty and quality of the science proposed, its potential impact and on the applicant’s ability to carry it out. They will also assess the proposal based on its relevance to any pre-defined criteria for a particular call.
The scientific reviewers are chosen from among researchers with expertise in the scientific field of the proposal. One of the reviewers, but no more than one, may be chosen from the list of reviewers suggested by the applicant. The PRACE staff managing the peer review process will seek to assure that each proposal is evaluated by non-conflicted reviewers that are experts in at least one aspect of the proposal and together can provide an expert view in the whole of the scientific case presented.
The identity of the technical and scientific reviewers will be known only to the PRACE peer review staff and anonymous to the applicant(s) and others involved in the process.
The reviewers will be required to keep personal details of the applicants and the details of their proposals confidential. In exceptional cases it may be necessary for all personnel and reviewers to sign a Non-Disclosure Agreement (NDA).
Applicant right to reply
The reviewer’s report (technical and scientific) will be sent to the applicants for an opportunity to respond, correcting any factual inaccuracies or providing any necessary further information in response to issues that the reviewers may have raised. The reviewer report will be made anonymous before being sent to applicants. The applicant will be given a deadline to respond to the reviewers’ comments. Applicant’s responses to the reviewer’s reports will be sent for resource allocation. Reviewer’s reports will also be sent to the Access Committee that will perform an assessment. During the AC meeting the PRACE Committee will take both reviews and the response to reviews into account to elaborate the recommendation for PRACE on the score and amount of resources to be allocated.
Prioritisation of proposals
After the technical and scientific review steps, the proposals, reviews and replies from the applicants are used by the PRACE Access Committee to rank them according to scientific and technical excellence.
The PRACE Access Committee is composed of eminent scientists from across the remit of the PRACE scientific applications. Its members are selected annually by the PRACE Scientific Steering Committee. The composition of the PRACE Access Committee is available here.
The Access Committee will analyse the proposal, the technical and scientific review reports together with any applicants’ response, in order to produce a single ranked list for each call of proposals that have been sent to the Committee. Multi-Year and Single-Year Project Access proposals are ranked in the same list. The Committee takes into account the advice of the technical and scientific assessments regarding amounts of resources requested or desirable for each proposal in making its recommendation on the resources to be allocated to each proposal.
Allocation of resources to proposals for Project Access will follow the prioritization of the PRACE Access Committee. Resources will be allocated by the Board of Directors of PRACE until they are exhausted, or until the list of proposals deemed of sufficient quality by the PRACE Access Committee is exhausted. The applicants will be promptly informed if their proposal has been successful or not. They will also be informed of the amount of resources that they have been awarded, which may have been modified by the PRACE Access Committee or the Board of Directors of PRACE.
Changes and extensions to awards
Applicants must inform PRACE promptly of any changes to a successful proposal, either regarding the amount of resources needed or the distribution of the usage time within the schedule. Requests for the extension of the allocation period need to be fully justified and will be analyzed by PRACE on a case by case basis. Extension will only be considered in the event of unforeseen technical issues at the HPC hosting site which would prevent the user from accessing the awarded HPC resources. The awarded resources (total computer time and/or expert support) cannot be increased. All notifications of changes or requests for extensions should be sent to peer-review[at]prace-ri.eu.
Applicants right to appeal
All PRACE allocation/rejection decisions are final. Nevertheless, applicants that have concerns about the review of their applications should contact via email the PRACE Peer Review Team peer-review[at]prace-ri.eu within two weeks after receiving the decision. The PRACE peer review staff will respond by providing clarifying information as needed about the peer review process and the PRACE final decision.
Please only contact the PRACE peer review staff if it is not clear, from the reviewers’ comments, where the flaws in the proposal were. High quality proposals may not be allocated resources because of resources limitations.
If, following clarifications from the PRACE peer review staff, the applicant still has concerns about procedural aspects of the review, a formal letter of appeal can be submitted via email to the PRACE BoD bod[at]prace-ri.eu no later than one month after receiving the clarifications from the peer review staff. The PRACE BoD will analyze the applicant’s reasons for complaint on a per case basis and will send a formal reply to the applicant within one month of receiving the appeal.