Objectives:

• A recent comparative study of eight European countries (Kulczycki et al, 2018) revealed important differences across countries and SSH disciplines in terms of language use and publication types. Building on this work this STSM will contribute to the further understanding of those differences by studying the coverage of journal publications from Central European Countries in major international indexes;
• Identify trends in terms of coverage by a major international index and disentangle them in terms of changes in publication patterns and indexing of additional journals (cf. Ossenblok, Engels and Sivertsen, 2012).

Results:

• Analysis of the coverage in a major international index of journal publications from two or more CEE countries in recent years;
• Analysis of the scope (regional vs international) and the reputation indicators of the indexed journals that have been published from CEE countries;
• Conference presentation and publication in peer-reviewed journal.

Special criteria for this STSM: The applicant has access to and is familiar with the data infrastructure in a central European country; the applicant is familiar with the interfaces of major international indexes; data collection is anticipated to take place prior to the STSM.

Practical details: Working group: WG3 (databases and uses of data for understanding SSH research);

Duration: between 2 weeks and 2 months;

Location: Centre for R&D monitoring (ECOOM), University of Antwerp, Antwerp, Belgium;

Contact: Raf Guns (raf.guns@uantwerpen.be).

Description of the topic: During this STSM, the grantee will develop a standardized questionnaire on the topic of evaluation and career. She or he will also develop a concept for the fieldwork for the international survey that will be conducted shortly after the STSM. The grantee will use the preliminary analyses of the qualitative interviews conducted within the SIG EIC and the draft questionnaire developed in the SIG ECI-STSM of GP2 as a basis for the final questionnaire. FORS will provide expert knowledge on questionnaire development and survey design (question formulation, scale development, translation, sampling, fieldwork management, web-survey programming and management). The questionnaire development will be driven by findings of the qualitative interviews and the potential future analyses with the quantitative data in cooperation with the host and SIG EIC.

Objectives: This STSM will lead to a prepared fieldwork for a survey on evaluation and career among early career researchers. It is necessary for achieving the GAGP 10 “To release a questionnaire about the evaluation of ECI in various COST countries” and GAPG 2 “To understand the impact of evaluation practices on ECI careers”. It further helps to advance the understanding of knowledge production and dissemination in the SSH (Task 1 of WG1).

Special criteria for this STSM: the applicant is willing to, and capable of, developing a survey on evaluation and (early) career with the support of the host institution.

Results: The proposed STSM will result in a questionnaire. Fieldwork for the survey will be prepared. Thus, the STSM contributes to the policy papers “Recommendations for better adapted approaches to research evaluation” (GAPG 6) and “Recommendations for better adapted criteria and indicators” (GP4). It will be a crucial input for the second Training School.

Practical details:

Working group: WG1 (conceptual frameworks for SSH research evaluation)

Duration and timing: between 2 to 4weeks in September/October 2018.

Location: FORS, Géopolis, CH-1015 Lausanne, Switzerland

Contact: Michael Ochsner (ochsner@evalhum.eu; michael.ochsner@fors.unil.ch)

Description of the topic: The role of peer review in SSH is to give scholars feedback on un-published manuscripts so that they can improve their research. Often it is referred to as the ‘gold standard’ of evaluation; yet, as a vetting process, it is time consuming, difficult to reproduce, subject to bias, and understudied (Campanario, 1998; Lee et al., 2013; Sayer, 2014). Little is known about the ‘quality’ of reviewing techniques and the effect that it is having on research products. In fact “there can be a huge variation” in how reviews are written, “from deep and insightful to shallow and misinformed” (Flier, 2016).2 Across the humanities in particular, scholars generally believe that “there is a lack of consensus on quality criteria” and that “comparing or assessing research quality is impossible” (Hug et al., 2013, p. 370).3 Guidelines are usually available to scholars concerning review practices; however without a sound body of research related to this, focused on large datasets of review texts, we have no evidence-base proxies for what we might call review ‘quality’ in the SSH.

The aim of this STSM is to investigate the peer review process both in terms of authors’ observations concerning manuscript reviews, as well as the thematic content of review texts themselves (i.e., manuscript referee reports). The research is designed to operate inductively (i.e., identifying themes/concepts as they occur in full-text) rather using a hypothetico-deductive approach to the study of peer review. Our project will be guided; therefore; by three major research questions: 1) What ‘proxies’ of quality are prevalent in referee reports produced for journal articles/books in the SSH? 2) Do these proxies of ‘quality’ differ for various disciplinary/subject areas?   3) Which ‘proxies’ of quality do SSH scholars perceive to be helpful in terms of stimulating specific types of actions?

Objectives: The first stage of data collection will involve the development and implementation of a questionnaire survey, with a specific crowd sourcing function: scholars from the SSH will be asked to respond to questions concerning their experience of a manuscript review process, and then submit copies of their received referee reports.   Use of this data will involve tagging, coding, and extracting concepts from the referee reports, and comparing them to the questionnaire responses. It is based on the notion that “text mining [via natural language processing] and qualitative research are epistemologically compatible” (Yu et al., 2011).6   The second stage will involve analysing the survey results and evaluating them in connection with the referee reports using a combination of statistics, computational linguistic methods, with the integration of hermeneutics and social simulation to the mixed-methods design..

Special criteria for this STSM: The applicant must have knowledge of SSH research in his or her country and ideally possess the following:

• Masters degree or Ph.D. in social science/humanities research, relevant to the study description and objectives outlined above;
• A sound scientific background relevant to peer review research;
• Excellent skills related to the development of and analysis of questionnaire-based surveys;
• Excellent skills in quantitative or computational research methods;
• High interdisciplinary openness;
• Excellent English language skills (oral and written);
• High communicative and inter-cultural competence.

Results: The proposed STSM will a) contribute to new insights and indicators related to peer review and SSH research, b) generate empirical evidence for developing better approaches to peer review, and c) result in co-authored publications, as well as further applications for project funding with European granting agencies.

Practical details:

Working group: WG1 (conceptual frameworks for SSH research evaluation);

Duration and timing: between 2 weeks and 1 month from June to November 2018;

Location: University of Copenhagen (Denmark) – http://www.ku.dk/ENGLISH/;

Contact: Alesia Zuccala (mailto:a.zuccala@hum.ku.dk).

Description of the topic:

Identifying the societal impact of public funded research has become a hot topic that is receiving more attention in both the academic and policy arena. However, to initiate a well-directed debate/discussion on the societal impact of research, a pre-condition is to establish a common vocabulary and understanding of what constitutes research societal impact and what does not. This STSM focuses specifically on deliminating the boundaries of SSH as disciplinary fields and their orientations towards societal impact, asking whether research impact can be provided by the SSH field itself or in combination with other science fields. The STSM will conduct an exploratory analysis of researchers’ reply to a questionnaire in order to identify the myths surrounding what constitutes research societal impact in SSH, to develop a common terminology and understanding for addressing this issue.

Objectives: This STSM aims to advance the understanding of the contextual conditions of SSH societal impact creation (Task 1 of WG2). In particular, the STSM seeks to develop a common language and vocabulary for talking about what matters to SSH impact generation in ways that are relevant to the field and not drawn from other fields. This analysis will complement the work already taken of specific examples of impact creation (STSM 1_1, 2_1) and also provide empirical material to support the theoretical framework developed setting out the problems of complementarity vs contradictions (STSM 2_2). The final output will be an analysis of the most common definitions of various key concepts within SSH valorisation.

Special criteria for this STSM: the applicant has knowledge of the theories, practices and policies for the stimulation of societal impact from SSH research activities. The applicant must also have knowledge of qualitative analysis (ideally narrative analysis or be willing to learn that skill.

Results: The proposed STSM will supplement the existing empirical work undertaken in STSMs 1_1, 2_1 and 2_3, and help to complement the theoretical work undertaken in STSM 2_2. This will result in co-authored publications.

Practical details:

Working group: WG2 (Societal Impact of SSH Research);

Duration and timing: between 2 weeks and 1 month;

Location: CHEPS, University of Twente, the Netherlands;

Contact: Paul Benneworth (p.benneworth@utwente.nl).

Description of the topic: ENRESSH-VIRTA is a collaborative project conducted as part of the activities of the working group 3. The project aims at integration of publication information from various countries and organizations. The grantee of this STSM will contribute to both technical and contextual development of the proof of concept VIRTA-ENRESSH-POC into a European research information service. Particularly, he or she will support the project in the context of metadata standards and harvesting and integration of information from various data sources.

Objectives:

• Explore the current technology and data model of ENRESSH-VIRTA;
• Investigate the potential use of international metadata standards, CERIF in particular in harvesting and exporting publication data in the frame of the European research information service.

Results:

• A review on potential of CERIF data model in data harvest and export in ENRESSH-VIRTA information service
• A protocol for mapping heterogeneous data from various national, institutional or local sources into a common standardized data model;
• A plan for automatic generation of publication type information.

Special criteria for this STSM: Preferably the applicant has access to and is familiar with the data infrastructure in their own country or region. Understanding of the CERIF data model is an advantage.

Practical details: Working group: WG3 (databases and uses of data for understanding SSH research);

Duration: between 2 weeks and 2 months;

Location: CSC – IT Center for Science Ltd, Espoo, Finland;

Contact: Hanna-Mari Puuska (hanna-mari.puuska@csc.fi).

Objectives:

• A recent comparative study of eight European countries (Kulczycki et al, 2018) revealed important differences across countries and SSH disciplines in terms of language use and publication types. Building on this work this STSM will contribute to the further understanding of those differences by studying the coverage of journal publications from Central European Countries in major international indexes;
• Identify trends in terms of coverage by a major international index and disentangle them in terms of changes in publication patterns and indexing of additional journals (cf. Ossenblok, Engels and Sivertsen, 2012).

Results:

• Analysis of the coverage in a major international index of journal publications from two or more CEE countries in recent years;
• Analysis of the scope (regional vs international) and the reputation indicators of the indexed journals that have been published from CEE countries;
• Conference presentation and publication in peer-reviewed journal.

Special criteria for this STSM: The applicant has access to and is familiar with the data infrastructure in a central European country; the applicant is familiar with the interfaces of major international indexes; data collection is anticipated to take place prior to the STSM.

Practical details: Working group: WG3 (databases and uses of data for understanding SSH research);

Duration: between 2 weeks and 2 months;

Location: Centre for R&D monitoring (ECOOM), University of Antwerp, Antwerp, Belgium;

Contact: Raf Guns (raf.guns@uantwerpen.be).