Evaluation of research involves ethical issues. While research on assessment procedures, especially indicators is abundant, there is not much progress in the discussion of ethics in research evaluation. The analysis of the existing scientific and grey literature including many guidelines on research ethics and ethics of evaluation showed that the two research streams can be used to develop an ethics of research evaluation. This STSM will continue the ongoing project examining ethical aspects of research assessment, with the goal to develop guidelines for ethics of research evaluation by combining ethical theories, research ethics, evaluation ethics. The goal of the study is to stimulate a reflection on ethical principles among people active in research evaluation, be it SSH scholars, evaluators or policy makers, to make efforts to make these principles transparent.
Objectives: The STSM aims to further the understanding knowledge production and dissemination in the SSH (Task 1 of WG1) from an ethical perspective; it will analyse assumptions underlying peer review (Task 2 of WG1); it will not only observe (national) regulations (Task 3 of WG1) but also propose new guidelines. It will add to the deliverable “recommendations for better adapted criteria and indicators” due in GP4. It will lead to collaborative publications and to guidelines for ethical behavior for the evaluation of research.
Special criteria for this STSM: the applicant should have knowledge of the different evaluation situations as well as of the characteristics of peer-review and should have knowledge of the fundamentals of ethical behavior.
Results: The proposed STSM will provide guidelines for ethical behavior for the evaluation of research, and will produce co-authored publications.
Practical details:
Working group: WG1 (Conceptual frameworks for SSH research evaluation)
Duration and timing: between one and two weeks
Location: Sapienza Rome University, Faculty of “Lettere e Filosofia”, piazzale Aldo Moro 5, 00185 Roma – Italy.

Contact: Maria Teresa Biagetti (mariateresa.biagetti@uniroma1.it) and Michael Ochsner michael.ochsner@fors.unil.ch

Publications in social sciences and humanities (SSH) present important differences with respect to publications in other areas; for example, the scope of research (more local), the publication typology (mainly book chapters and monographs), the language (mainly the language of the country of publication) and the habits of collaboration (more single authored publications). These different characteristics require the need to develop specific methodologies for the analysis and evaluation of SSH publications.
In particular, legal science and included disciplines presents the most considerable difficulties for evaluation based on a quantitative approach. In our study we consider ‘legal science’ as the set of disciplines that deal with law: science of law, legal theory, jurisprudence, legal dogmatics, the sociology of law, legal anthropology, comparative law, history of law, etc. The evaluation of research in law and law related disciplines is complex because there are profound differences among them. The evaluation of research in law and law related disciplines is more complex because there are profound differences in the dissemination of research results among them. There is no European ranking of law journals or legal publishers, no generally accepted system of peer review, no bibliometric databases, and no consensus on quality indicators for academic legal publications.
In recent years, European countries have participated in an intense debate on the evaluation of the results of legal research. The discussion tends to focus on the new appeal of law, moving from a national towards a transnational approach, from mono-disciplinarity to multidisciplinarity. All these specific aspects are very relevant for the research that will be carried out in this STSM.
This STSM is addressed to explore methodological approaches for evaluation of legal journals published in three different countries (Italy, Spain and Croatia), in order to find criteria that allow establishing a classification model of national legal journals based on qualitative and quantitative indicators. In this regard, the idea is to continue the work of the Spanish Foundation for Science and Technology (FECYT) on the evaluation of journals in SSH disciplines. The methodology used by FECYT is based on two dimensions: the analysis of journals’ impacts and the analysis of their visibility. During the STSM mission, we are going to further develop this methodology, including new quantitative and qualitative criteria that show well the specificities of the research reported in the SSH journals of these three counties.
All the mentioned countries are characterized to have a quite good production of law journals and to be the place of lively national debate among legal scholars who have a strong preference for qualitative evaluation methods over quantitative methods.
Objectives:
• To provide an accurate survey of the national legal journals in Italy, Spain and Croatia, as well as their impact and visibility in the international bibliographic databases.
• After literature review on different approaches used for the evaluation of legal journals, to identify at quantitative (citation, visibility, etc.) and qualitative indicators (type of peer review, presence of internal or/and external referees, openness, etc.) for the assessment of legal journals.
• To propose a methodology for assessing national legal journals of the basis of the methods used in three different countries (Italy, Spain and Croatia), capable of being used for national legal journals in other countries.
• To get an insight in the peer review process of the legal journals by a content analysis of the instructions for peer reviewers and other documents describing editorial processes.
Expected Results:
● A new specific methodology to evaluate the national legal journals in different countries.
● In-depth literature survey in order to record all possible indicators and their characteristics
● A review of national practices in the legal science
● An analysis of editorial processes and peer review in the national legal journals
● Design of a possible structure for building a database of scientific papers published in legal journals with particular interests on open access publications.
● Scientific papers published, including research results on comparative analysis of legal journals in Italy, Spain and Croatia, journals’ quality criteria, classification model and peer review process.

Special criteria for this STSM:
The applicant must have knowledge of SSH research in his or her country and ideally possess the following:
● Excellent skills related to bibliometric research with particular interest in evaluation of legal outputs;
● Good knowledge in SSH scholarly publishing;
● High interdisciplinary approach.

Practical details:
Duration and timing: 10 days from the period of November 2019 to December 2019.
Location: Institute of Legal Informatics and Judicial Systems – National Research Council (IGSG-CNR) Via de’ Barucci 20, 50127 Firenze, Italy
Contact: Ginevra Peruginelli (ginevra.peruginelli@igsg.cnr.it)

Topic 4.3.2.1: Theory-informed policy guidelines for ex ante impact evaluation

Host institution: Norwegian Research Council (Forskningsrådet) + remote supervision via Høgskulen på Vestlandet.

Description of the topic: Much of the focus on evaluation lies in ex post evaluation of the quality of research whether through published outputs or completed projects, and ensuring that evaluation provides a strong steering effect towards delivering quality research. But
another key element of the research steering cycle played by evaluation is in the ex ante evaluation of research proposals, deciding which proposals will be funded. Research proposal evaluation has become a highly professionalised business in which many competing proposals are assessed on the basis of a set of criteria such as scientific excellence and quality of project management. Funding is awarded to those which score the highest against those criteria, in anticipation that it will correlate with the highest ex post quality of project.
A number of research councils have in recent years introduced specific criteria for ‘societal impact’, with proposals being evaluated in terms of how they will contribute to creating impact in society as well as generating scientific excellence. But it is widely understood that creating this societal impact from research is an uncertain business, highly dependent upon partners outside the scientific production system (societal partners). There can be no guarantee that those proposals that make claims that their research will create impact ex ante will in practice live up to those claims in creating impact in society (ex post).
Policy-makers create guidelines for evaluators in order to help steer their funding towards their overall goals, and as creating societal impact has become more important for research councils, policy-makers have also set out guidelines for evaluating impact in assessing research proposals. But this process has tended to be pragmatic and intuitive rather than based upon a deep understanding of what the characteristics are of research proposals that have the best chances of this ex post success.
In this Short-term science mission, the candidate will work with the Norwegian Research Council (Forskningsrådet) and the Working Group Leader to develop a conceptual framework and developing guidelines to assist evaluators in evaluating ex ante evaluation.

Objectives: The STSM aims to create a set of guidelines for the ex ante evaluation of research impact in research proposals that is strongly rooted in a conceptual understandings of the creation of research impact. The STSM seeks to support policy-makers to better stimulate the contributions that SSH research makes in society, and to stimulate the flow of that knowledge to society at large (WG2 task 3). The Guidelines represent a clear set of measures for valuing SSH (Task 5), by ensuring that SSH research that is better committed to creating societal impact is funded. By undertaking the work jointly between the WG leader and a user (policy-maker), and creating a concrete set of guidelines of use to policy-makers for evaluating impact, the STSM contributes to the overall goals of the Action to create policy impact.

Special criteria for this STSM: The applicant should have a broad knowledge of science studies and scientific evaluation processes, ideally with some knowledge about ex ante evaluation of research proposals (either academic or practical).

Results: The proposed STSM will provide guidelines for the ex ante evaluation of societal impact in research proposal evaluation which are sensitive to the specific needs of SSH research. As far as possible, the guidelines will be formulated in such a way that they may be used as a conceptual framework for the ex ante evaluation of societal impact in research proposals more generally, not limited to projects within SSH.

Practical details:
Working group: WG2 (Societal impact and relevance of SSH Research)
Duration and timing: between 2 weeks and 2 months from 1st September to 30th November 2019.
Location:
Forskningsrådet, Drammensveien 288, 1327 Lysaker, Norway
Contact:
Jon Holm (joh@forskningsradet.no)
Paul Benneworth (paul.benneworth@hvl.no)