Topic 4.3.2.1: Theory-informed policy guidelines for ex ante impact evaluation
Host institution: Norwegian Research Council (Forskningsrådet) + remote supervision via Høgskulen på Vestlandet.
Description of the topic: Much of the focus on evaluation lies in ex post evaluation of the quality of research whether through published outputs or completed projects, and ensuring that evaluation provides a strong steering effect towards delivering quality research. But
another key element of the research steering cycle played by evaluation is in the ex ante evaluation of research proposals, deciding which proposals will be funded. Research proposal evaluation has become a highly professionalised business in which many competing proposals are assessed on the basis of a set of criteria such as scientific excellence and quality of project management. Funding is awarded to those which score the highest against those criteria, in anticipation that it will correlate with the highest ex post quality of project.
A number of research councils have in recent years introduced specific criteria for ‘societal impact’, with proposals being evaluated in terms of how they will contribute to creating impact in society as well as generating scientific excellence. But it is widely understood that creating this societal impact from research is an uncertain business, highly dependent upon partners outside the scientific production system (societal partners). There can be no guarantee that those proposals that make claims that their research will create impact ex ante will in practice live up to those claims in creating impact in society (ex post).
Policy-makers create guidelines for evaluators in order to help steer their funding towards their overall goals, and as creating societal impact has become more important for research councils, policy-makers have also set out guidelines for evaluating impact in assessing research proposals. But this process has tended to be pragmatic and intuitive rather than based upon a deep understanding of what the characteristics are of research proposals that have the best chances of this ex post success.
In this Short-term science mission, the candidate will work with the Norwegian Research Council (Forskningsrådet) and the Working Group Leader to develop a conceptual framework and developing guidelines to assist evaluators in evaluating ex ante evaluation.
Objectives: The STSM aims to create a set of guidelines for the ex ante evaluation of research impact in research proposals that is strongly rooted in a conceptual understandings of the creation of research impact. The STSM seeks to support policy-makers to better stimulate the contributions that SSH research makes in society, and to stimulate the flow of that knowledge to society at large (WG2 task 3). The Guidelines represent a clear set of measures for valuing SSH (Task 5), by ensuring that SSH research that is better committed to creating societal impact is funded. By undertaking the work jointly between the WG leader and a user (policy-maker), and creating a concrete set of guidelines of use to policy-makers for evaluating impact, the STSM contributes to the overall goals of the Action to create policy impact.
Special criteria for this STSM: The applicant should have a broad knowledge of science studies and scientific evaluation processes, ideally with some knowledge about ex ante evaluation of research proposals (either academic or practical).
Results: The proposed STSM will provide guidelines for the ex ante evaluation of societal impact in research proposal evaluation which are sensitive to the specific needs of SSH research. As far as possible, the guidelines will be formulated in such a way that they may be used as a conceptual framework for the ex ante evaluation of societal impact in research proposals more generally, not limited to projects within SSH.
Practical details:
Working group: WG2 (Societal impact and relevance of SSH Research)
Duration and timing: between 2 weeks and 2 months from 1st September to 30th November 2019.
Location:
Forskningsrådet, Drammensveien 288, 1327 Lysaker, Norway
Contact:
Jon Holm (joh@forskningsradet.no)
Paul Benneworth (paul.benneworth@hvl.no)